Your best salesperson had a great call on Tuesday. By Thursday she cannot remember whether the prospect said the budget was approved or just “likely”, the follow-up email is still in drafts, and the CRM says the deal is where it was a month ago. Nothing went wrong on the call; everything went wrong after it.
An AI sales call summary fixes the part after the call. The call is recorded, turned into text, and a language model writes a structured summary and a draft follow-up email within minutes of hanging up. The salesperson reads both, fixes anything wrong, and sends.
This article explains each piece: how calls get recorded, what transcription gets wrong, the summary format that stops the model inventing, the follow-up draft, the CRM update, and the consent rules that differ by state.
What this actually is
Three steps happen in sequence. Recording captures the audio. Transcription turns it into text with speaker labels and timestamps, using a speech-to-text model, a separate kind of AI from the one that writes. Summarization is where a language model (software that reads and writes text, the engine behind Claude, ChatGPT and Gemini) reads the transcript and produces a structured note and a draft email.
The ordinary-business analogy is a junior colleague taking minutes. Good minutes record who was there, what was decided, what was asked and not answered, and what happens next with a name against it. Bad minutes are a paragraph of impressions with a number in it that nobody actually said. The whole discipline is making sure you get the good minutes.
Where it goes wrong is where every AI application goes wrong: the model is fluent and confident, and if the instructions let it, it fills gaps with plausible content. A summary that says “budget approved at $40,000” when the prospect said “we could probably find forty” is worse than no summary.
1. Record every call where you may, and announce it every time
Call recording consent law in the United States is set state by state. In most states, one party to the call (you) consenting is enough. In around a dozen, every party must consent; California, Florida, Illinois, Maryland, Massachusetts, Pennsylvania and Washington are among those usually cited, and the list changes, so confirm the current rules for the states you call into. The safer rule is to treat every call as all-party: a clear announcement at the start, every time. We are not lawyers and this is not legal advice.
The announcement is not a burden in practice. A short line (“I record my calls so I can send you accurate notes afterward, is that all right?”) is accepted almost universally. If someone declines, the recording stops and the salesperson takes notes by hand. For video meetings, Zoom, Meet and Teams display a recording notice, and note-taker bots announce themselves; use those rather than capturing audio silently. A recording nobody was told about is a liability, whatever the state.
2. Capture the recording where the call actually happens
The recording has to come from the system the call runs through, and small businesses run calls through several. A business phone system (RingCentral, Dialpad, Aircall, a Twilio-based setup) records calls natively. Mobile calls on a personal phone are hard to record; route sales calls through the business system’s mobile app instead. Video meetings record inside the meeting tool or via a note-taker bot.
Each source should deposit the recording in one place with consistent metadata: who called, when, how long, which deal or contact it belongs to. A recording that cannot be attached to a CRM contact is a file nobody will ever open. Set retention rules from the start: transcript and summary for the life of the deal, audio for a shorter window unless someone marks it to keep. What you may hold, and what happens when it is sent to an AI provider, is covered in AI Data Privacy for Business, What Happens to Customer Data.
3. Transcribe with speaker labels and expect specific errors
Modern speech-to-text is very good on clear audio and worse on speakerphone, crosstalk, unfamiliar accents, and industry vocabulary. Product names, part numbers and dollar amounts are where errors cluster, and they are exactly the details a sales summary needs. Choose a transcription service that supports speaker labels and custom vocabulary, and feed it your term list.
Keep the timestamps. A summary that cites “at 14:20 the prospect said…” lets the salesperson jump to that moment and check. Without timestamps, checking a doubtful line means listening to the whole call, and nobody does that, so the doubtful line gets believed.
4. Summarize into a fixed format that cannot invent
This rule decides whether the system is trustworthy. The summary is not “summarize this call”. It is a template with fixed headings, each with a rule about what may go in it. In the systems we build, the headings run: participants; the prospect’s stated problem, in their words; timing; budget, quoted; objections; questions not answered; what was agreed; next steps with an owner and a date; and anything promised.
Every specific (a number, a date, a commitment) must be tied to a quote from the transcript with a timestamp. If nothing supports a heading, it says “not discussed”. The model is told explicitly that leaving a heading empty is correct and filling it with an inference is a failure (AI Hallucination Guardrails for Business Applications). Keep it short enough to read in under a minute, with next steps and unanswered questions near the top.
5. Draft the follow-up email, with a person’s finger on the send button
From the confirmed summary, the model drafts the follow-up. Its rules are strict. It thanks the person briefly. It restates, in one short paragraph, what they said their problem is. It lists what was agreed and what the salesperson will send, with dates. It answers, or promises a date for, any unanswered question. It ends with the single next step. It adds nothing that was not on the call, does not pitch, and stays under roughly one hundred and fifty words.
The draft goes to the salesperson, not the prospect. The salesperson reads it against the summary, fixes anything wrong, adds the personal line only a human can add, and sends. That takes a minute or two and it is where mistakes get caught; sending drafts unread is how a prospect receives an email promising a discount that was never offered. The draft should be waiting within minutes so the email goes out the same hour (Automated Lead Follow-Up That Replies in Minutes, Not Days). A follow-up the same afternoon reads as attentiveness; three days later it reads as a to-do list.
6. Update the CRM from the confirmed version, not the raw one
Once the salesperson has confirmed the summary, it becomes an activity on the deal, with the transcript attached and the recording linked. Next steps become tasks with owners and due dates. All of this is written by the automation under its own account, so it is clear what came from the system and what came from a person.
What the automation does not do is move the deal’s stage, change its value, or set the close date. Those fields drive the pipeline report and are changed by a person, prompted by the summary if appropriate (AI CRM Automation That Makes the CRM Do the Work for You). If the summary was corrected before confirmation, keep both versions; the difference is the feedback that improves the prompt, and the evidence you want if anyone asks what the system said versus what was true.
7. Handle calls that are not sales calls, and calls that go badly
Not every recorded call is a sales conversation. Some are support, some are vendor calls, some are the salesperson’s dentist. The system needs a first pass that classifies the call: sales calls get the sales summary, support calls go to the support process, personal calls are deleted.
Some sales calls go badly: the prospect is angry, a complaint is raised, something legally sensitive comes up. The summary should flag these and route the flag to a manager, and the follow-up draft should be held for review, the same escalation logic a good inbound intake system uses (AI Call Intake Best Practices: Twelve Rules That Hold Up). And some calls have terrible audio. The system should say so, with a note at the top, rather than produce a clean-looking summary built on a broken transcript.
8. Review a sample monthly against the recordings
Once a month, pick a handful of calls at random, listen to each, and read the summary the model produced and the version the salesperson confirmed. Count the errors: invented specifics, missed next steps, misheard numbers. Check how many follow-up drafts were sent unchanged, edited, or discarded.
The pattern of edits tells you what to fix. If salespeople keep deleting a sentence the model adds, remove it from the prompt. If numbers are wrong often, add vocabulary or tighten the quote rule. If “not discussed” is rarely used, the model is filling gaps. Change one thing and check next month. Also check consent compliance: are announcements being made, are declined recordings actually not recorded, is retention deleting audio on schedule.
Picture a business like this one
The business below is a composite of the kind of company that writes to us, not a client. The numbers describe the shape of the problem, not a case study.
Picture a business like this one: a commercial insurance brokerage with eighteen staff and five producers who each take four to eight prospect calls a day. Follow-up emails go out a day or two later from memory, CRM notes are a line or two, and when a producer is out sick nobody can pick up their conversations. Two states the brokerage sells into require all-party consent.
What a brokerage like this would build:
- Every producer’s calls routed through the business phone system’s app, with a recording announcement on every call and a one-key stop if the caller objects.
- Recordings deposited with caller, time, and matched CRM contact; transcription with speaker labels and a custom vocabulary of carrier names; audio retained ninety days.
- A summary template with nine headings, quotes required for every specific, “not discussed” allowed, complaint flags routed to the office manager.
- A follow-up draft under one hundred and fifty words, waiting in the producer’s drafts folder within five minutes, sent only after the producer reads it.
- Confirmed summaries written to the CRM under the automation’s account; next steps as tasks; stage and value untouched unless the producer changes them.
- A monthly review of six random calls by the office manager.
What changes: follow-up emails go out the same hour, and prospects comment that the notes are accurate. When a producer is out, a colleague reads the last summary and picks up the thread. In month two, the review finds the model quoting an approximate budget as a firm one; the quote rule is tightened. The consent announcement, which producers were nervous about, is declined by almost nobody.
What it costs to run
Call recording is included in most business phone systems at standard tiers, or available as an add-on for a few dollars per user per month; check the current pricing page. Meeting recording is built into Zoom, Meet and Teams on business plans, and note-taker bots are typically ten to thirty dollars per user per month.
Transcription is priced per minute of audio and is cheap: current rates work out to a fraction of a cent to a cent or two per minute, so a producer doing thirty hours of calls a month costs a few dollars. The summary and follow-up draft cost a few cents per call with current models. A workflow tool or small server adds ten to twenty dollars a month. The real cost is the two minutes per call the salesperson spends confirming the summary and reading the draft, and the hour a month for the review.
The mistakes we see most
Recording without announcing. Whatever your state’s rule, say it every time. The one call you needed to record without asking is the one that becomes a problem.
“Summarize this call” as the whole prompt. Fluent, confident, and wrong about the number that mattered.
Sending the follow-up automatically. A model will eventually promise something that was not offered, and the person reading before sending is what catches it.
Recordings that cannot be matched to a contact. Fix the metadata at the capture point.
No review against the audio. Only listening to a sample tells you whether the summaries are right.
When to bring in help
If your team uses one phone system and one meeting tool, you can get a basic version of this from off-the-shelf features: most business phone systems now offer recording plus an AI summary, meeting note-takers produce notes, and several CRMs log those notes against the contact. The summaries are generic and the follow-up is on you, but an owner can switch it on in an afternoon. Sort out the consent announcements first.
A developer becomes worth it when you want your summary format with your quote requirements, when calls come from several systems and all need to land on the right CRM record, when the follow-up draft has to follow your voice, when sensitive calls need routing to a manager, when retention has to be enforced rather than remembered, and when you want a log of what the model wrote versus what a person confirmed. That is the difference between a feature that produces notes and a system you can rely on when a prospect disputes what was said.
Levelbrook builds call summary and follow-up systems like this for businesses, at a fixed price from a written scope. The recordings, prompts and logs all live in accounts you own. The form below is how a conversation starts.