Generated 2026-05-16 14:22 · 120 automotive BDC calls · Claude read each call manually
TL;DR. Claude (LLM) and Cognivec disagree on 68% of automotive BDC calls when classifying soft-close vs hard-close. Cognivec is systematically too lenient — it calls many "hard close" what Claude correctly identifies as "soft close" (pleasantries mistaken for commitment). For the specific task of soft/hard close classification, the LLM wins decisively. For the broader Cognivec value props (critical-moment detection, cross-corpus stability, numeric KPIs for dashboards), the comparison is irrelevant — those are different tasks. The honest recommendation is a hybrid pipeline: Cognivec ranks calls by velocity peaks (free), LLM classifies the top-50 (~$0.10 instead of $0.18 for all 120).
| Class | Cognivec said | Claude said | Cognivec / Claude ratio |
|---|---|---|---|
| HARD_CLOSE | 48 (40.0%) | 17 (14.2%) | 2.82× |
| PARTIAL | 41 (34.2%) | 33 (27.5%) | 1.24× |
| SOFT_CLOSE | 31 (25.8%) | 69 (57.5%) | 0.45× |
| NO_CLOSE | 0 (0.0%) | 1 (0.8%) | 0.00× |
Rows = Cognivec verdict. Columns = Claude verdict (ground truth proxy).
| Claude→ HARD | Claude→ PARTIAL | Claude→ SOFT | Claude→ NO | Cognivec TOTAL | |
|---|---|---|---|---|---|
| HARD_CLOSE | 7 | 9 | 31 | 1 | 48 |
| PARTIAL | 7 | 13 | 20 | 0 | 40 |
| SOFT_CLOSE | 3 | 10 | 18 | 0 | 31 |
| NO_CLOSE | 0 | 0 | 0 | 0 | 0 |
Read the diagonal: exact agreement counts. Off-diagonal = disagreements.
Specific calls where lexicon-based marker counting fooled Cognivec into calling HARD_CLOSE what is actually SOFT_CLOSE.
...ck one for you too. Okay. Okay. And 555 0011, is that a good number? Yes. Okay, then I'm going to send you a text right now. Okay? Okay. Okay. I'll talk to you soon. Okay, thank you. Thank you so much. You bet. Bye. Bye.
... on the lookout for you, Ms. Anna, and hopefully we can earn you all business this Saturday. Okay. All right, Sounds good. We appreciate it. All right, you have a good forty five minutes. Talk to you later. You too. Bye.
... a communication or. You're welcome to call back. I'll be here. Okay. Thank you. Thank you. I know you're probably closed now, but I appreciate you taking my call. You're welcome, Tom. Thanks for calling. Okay, bye. Bye.
...nce you send me an email, I'll send you over a receipt as well. But we just need the license first to authorize that. Sure, I'll send that now. Okay, perfect, man. Talk to you soon. All right. You got it, bro. All right.
... just reply to it in case anything changes. Okay. I would also send you like, all the information for you to have it. I appreciate that. No problem. We'll see you tomorrow then. Thank you. Happy to help. Have a nice day.
...rks good for me. Yeah. Thank you. Alrighty, guys, like I said, my name is Linda. Just look out for that email at email at example dot com. All right, Bob, talk to you soon. Yeah. Thank you, sir. Have a good weekend. Bye.
...I am the occupation occupation here. I'll get you set up with one of my guys, and they'll shoot you a link over here shortly this evening. Okay. Thank you so much, man. I appreciate that. My pleasure. Thanks for calling.
...th it. Okay, very good. And I have an email about send the confirmation to you. Yes, and thank you, Alice. Yeah, you're absolutely, very welcome. So we'll see you tomorrow at 5pm. Thank you. Bye bye. All right. Bye. Bye.
The SOFT_CLOSE calls where both agree — Cognivec did catch real lost-revenue calls. These would be flagged correctly by either system.
...e see that and discuss some other options if. If that's what you're interested in. Sure, sure. I'll be over in probably two hours 30 minutes or so. That. Okay, thanks. Great. Thank you very much. Bye bye. Thank you. Bye.
...cupation, so I'm going to be. No worries. Yeah, you're very thorough. I can tell. No. No worries. I will get on that right now, get these questions answered for you. Okay. Thank you. You're welcome. We'll talk soon. Bye.
...nd then anything you need, you can reach out to me. Okay, super. Thank you. All right. Anything else I can help you out with? All right. Bye, Bob. Bye. Bye. That's everything. Thank you. I really appreciate it. Bye. Bye.
...50 bucks for financing that. Yeah. That's going to depend on your financing. And you'll just kind of complete that online and then it'll all reveal itself. Okay. Okay. Yeah. Thank you. All right, I'll call you back. Bye.
...ll set a time for you to come and pick it up. So. All right. Thank you. Yeah, you're so welcome. John. Again, my name is Mary and I'll be sending that text out here momentarily to you. All right. All right. Take care of.
If a BDC manager asks "is this call a soft close yes/no?" for each call, LLM (Claude/GPT) is more accurate. Cognivec's lexicon-based scoring is fooled by pleasantries ("see you", "talk to you soon", "tomorrow") that don't actually indicate commitment.
┌─────────────────────────────────────┐
│ 1. ALL CALLS → Cognivec (free) │
│ - Velocity / acceleration / jerk │
│ - Score each call: 0-100 │
│ - Output: ranked list by │
│ "interesting-ness" │
└────────────────┬────────────────────┘
│
▼
┌─────────────────────────────────────┐
│ 2. TOP 50 CALLS → Claude / GPT │
│ - Per-call soft/hard/partial │
│ - Reasoning paragraph │
│ - Coaching suggestion │
└────────────────┬────────────────────┘
│
▼
┌─────────────────────────────────────┐
│ 3. MANAGER DASHBOARD │
│ - Cognivec numeric trends │
│ - LLM detailed insights on top │
│ - Cost: ~58% of pure-LLM │
└─────────────────────────────────────┘
| Call | Cognivec | Claude | Match? | Claude note |
|---|---|---|---|---|
| #2 | HARD_CLOSE | HARD_CLOSE | ✅ | Agent put customer down for a time slot |
| #3 | HARD_CLOSE | HARD_CLOSE | ✅ | Customer says 'I'll be over in two hours' — coming today |
| #4 | SOFT_CLOSE | SOFT_CLOSE | ✅ | Customer non-committal: 'if I get time, will give you a holler' |
| #5 | PARTIAL | HARD_CLOSE | ❌ | Customer says 'I should be there within two hours' — coming |
| #6 | PARTIAL | SOFT_CLOSE | ❌ | Agent will call back after working numbers — no customer commit |
| #7 | PARTIAL | SOFT_CLOSE | ❌ | Info will be provided, no appointment |
| #8 | SOFT_CLOSE | SOFT_CLOSE | ✅ | Agent says 'I'll give you a call right back' |
| #9 | PARTIAL | PARTIAL | ✅ | Callback scheduled for Tuesday but not an appointment |
| #10 | PARTIAL | PARTIAL | ✅ | Text sent, conditional 'if you want to come in then we can set time' |
| #11 | HARD_CLOSE | SOFT_CLOSE | ❌ | Agent: 'I'll get back with you' |
| #12 | SOFT_CLOSE | PARTIAL | ❌ | Email/videos being sent, customer waiting on info |
| #13 | PARTIAL | HARD_CLOSE | ❌ | Specific time window between 3:45pm and 10am tomorrow |
| #14 | HARD_CLOSE | PARTIAL | ❌ | Looks like internal staff personal call — outlier |
| #15 | SOFT_CLOSE | HARD_CLOSE | ❌ | Customer coming this Saturday, 'we'll be on the lookout for you' |
| #16 | HARD_CLOSE | SOFT_CLOSE | ❌ | Contact info exchanged, no firm commitment |
| #17 | HARD_CLOSE | SOFT_CLOSE | ❌ | Pleasantries, 'shoot me a text if any questions' — no appointment |
| #18 | HARD_CLOSE | HARD_CLOSE | ✅ | Tomorrow between 3:45pm and 6:30pm confirmed |
| #19 | SOFT_CLOSE | PARTIAL | ❌ | Customer needs to reply yes to text, then agent checks |
| #20 | PARTIAL | PARTIAL | ✅ | Pictures + appraisal exchange in progress |
| #21 | PARTIAL | PARTIAL | ✅ | Last name asked, 'if you do plan on coming out' conditional |
| #22 | HARD_CLOSE | PARTIAL | ❌ | Agent will put numbers together, call right back |
| #23 | PARTIAL | PARTIAL | ✅ | License confirmation + email transaction in progress |
| #24 | HARD_CLOSE | SOFT_CLOSE | ❌ | Just numbers exchanged, no commitment |
| #25 | PARTIAL | HARD_CLOSE | ❌ | 10am Thursday pickup confirmed |
| #26 | HARD_CLOSE | HARD_CLOSE | ✅ | Customer is literally pulling in right now |
| #27 | HARD_CLOSE | HARD_CLOSE | ✅ | Service appointment Saturday at 2pm confirmed |
| #28 | PARTIAL | SOFT_CLOSE | ❌ | Agent shoots text, customer says 'let me know what you're thinking' |
| #29 | PARTIAL | HARD_CLOSE | ❌ | 10am tomorrow appointment confirmed |
| #30 | HARD_CLOSE | SOFT_CLOSE | ❌ | Agent will alert customer when something comes in — browsing |
| #31 | SOFT_CLOSE | HARD_CLOSE | ❌ | Customer agreed to sign up online immediately |
| #32 | SOFT_CLOSE | SOFT_CLOSE | ✅ | 'I'll call you back. Bye' — non-commit |
| #33 | SOFT_CLOSE | SOFT_CLOSE | ✅ | Conditional 'I'll let you know by end of today if coming' |
| #34 | HARD_CLOSE | PARTIAL | ❌ | Email + possible Tuesday meeting |
| #35 | HARD_CLOSE | SOFT_CLOSE | ❌ | 'Reach out shortly' — vague |
| #36 | PARTIAL | SOFT_CLOSE | ❌ | Service inquiry, 'we'll take a look at it' — no appointment |
| #37 | SOFT_CLOSE | PARTIAL | ❌ | Existing customer relationship, callback planned |
| #38 | PARTIAL | SOFT_CLOSE | ❌ | Info-gathering call, 'try to get it as fast as I can' |
| #39 | SOFT_CLOSE | PARTIAL | ❌ | Multi-step process spelled out, but no firm time |
| #40 | SOFT_CLOSE | SOFT_CLOSE | ✅ | Customer waiting on insurance, can't commit |
| #41 | SOFT_CLOSE | PARTIAL | ❌ | Pictures exchanged, agent will call back after score check |
| #42 | PARTIAL | PARTIAL | ✅ | Link to be sent this evening |
| #43 | HARD_CLOSE | SOFT_CLOSE | ❌ | Call dropped mid-conversation — 'Hello? Hello?' |
| #44 | SOFT_CLOSE | HARD_CLOSE | ❌ | Tomorrow 11:30am service appointment confirmed |
| #45 | HARD_CLOSE | SOFT_CLOSE | ❌ | 'Let me know if you find something' — non-commit |
| #46 | SOFT_CLOSE | SOFT_CLOSE | ✅ | 'Definitely think on it, call my wife' |
| #47 | SOFT_CLOSE | PARTIAL | ❌ | Out-of-state purchase, 'follow up with you' |
| #48 | HARD_CLOSE | SOFT_CLOSE | ❌ | Wait-list, 'let you know as soon as we see one' |
| #49 | PARTIAL | HARD_CLOSE | ❌ | Service appointment Friday at 9am confirmed |
| #50 | HARD_CLOSE | PARTIAL | ❌ | Warranty docs being emailed for claim |
| #51 | HARD_CLOSE | PARTIAL | ❌ | FaceTime call back planned 'right back', tomorrow morning visit potential |
| #52 | PARTIAL | SOFT_CLOSE | ❌ | 'We'll keep all this in mind' — info only |
| #53 | HARD_CLOSE | SOFT_CLOSE | ❌ | Agent will have someone call back |
| #54 | PARTIAL | SOFT_CLOSE | ❌ | Spec discussion, 'I will get back to you' |
| #55 | SOFT_CLOSE | SOFT_CLOSE | ✅ | Scheduling conflict, 'let me look for somebody else' |
| #56 | PARTIAL | PARTIAL | ✅ | Will text after talking to wife |
| #57 | HARD_CLOSE | PARTIAL | ❌ | Deposit/refund discussion in progress |
| #58 | SOFT_CLOSE | SOFT_CLOSE | ✅ | 'I'll get you all that information, be in touch' |
| #59 | SOFT_CLOSE | SOFT_CLOSE | ✅ | Info on safety features sent, no appointment |
| #60 | HARD_CLOSE | HARD_CLOSE | ✅ | Saturday at 2pm appointment confirmed (existing customer) |
| #61 | HARD_CLOSE | SOFT_CLOSE | ❌ | Explicit hedging: 'if it works, maybe we'll go that way' |
| #62 | SOFT_CLOSE | SOFT_CLOSE | ✅ | Agent will call back tomorrow 'when they get a chance' |
| #63 | PARTIAL | PARTIAL | ✅ | 'Quite a bit of opening for Saturday' — invite to call/text |
| #64 | HARD_CLOSE | PARTIAL | ❌ | Expect a call from Alice this evening or tomorrow |
| #65 | PARTIAL | SOFT_CLOSE | ❌ | 'Go speak to him, call right back' |
| #66 | PARTIAL | SOFT_CLOSE | ❌ | 'Let me call you right back, get some data' |
| #67 | PARTIAL | SOFT_CLOSE | ❌ | 'Going to work, give you a call back' |
| #68 | HARD_CLOSE | SOFT_CLOSE | ❌ | Roadside assistance info, no transaction |
| #69 | SOFT_CLOSE | PARTIAL | ❌ | Pre-approval next steps spelled out |
| #70 | PARTIAL | SOFT_CLOSE | ❌ | 'Text whenever, sooner the better' — vague |
| #71 | HARD_CLOSE | SOFT_CLOSE | ❌ | 'I'll get you the information, let you know' |
| #72 | HARD_CLOSE | SOFT_CLOSE | ❌ | Expecting a callback today/tomorrow |
| #73 | PARTIAL | HARD_CLOSE | ❌ | Customer says 'I'll just stand out our New York. We'll see you soon. Don't come. Bye' — likely coming |
| #74 | HARD_CLOSE | HARD_CLOSE | ✅ | Service drop-off being arranged today |
| #75 | PARTIAL | PARTIAL | ✅ | Form being filled out, video being made, callback shortly |
| #76 | HARD_CLOSE | SOFT_CLOSE | ❌ | 'Give my girlfriend a call... probably 2pm 3:45pm' — exploring |
| #77 | HARD_CLOSE | SOFT_CLOSE | ❌ | 'Call my brother, give you call back' |
| #78 | HARD_CLOSE | SOFT_CLOSE | ❌ | 'Shoot me a call back whenever you're ready' |
| #79 | HARD_CLOSE | SOFT_CLOSE | ❌ | Queued: 'first available reach back out' |
| #80 | HARD_CLOSE | SOFT_CLOSE | ❌ | Needs ID/proof, 'try to get it' |
| #81 | HARD_CLOSE | SOFT_CLOSE | ❌ | 'Get to work, call back in an hour' |
| #82 | PARTIAL | SOFT_CLOSE | ❌ | Safety advice about gas tanks — no transaction |
| #83 | SOFT_CLOSE | SOFT_CLOSE | ✅ | 'Get to another computer, call right back' |
| #84 | PARTIAL | SOFT_CLOSE | ❌ | Customer wants to wait, agent will call back later |
| #85 | PARTIAL | SOFT_CLOSE | ❌ | Service records lookup, no transaction |
| #86 | PARTIAL | SOFT_CLOSE | ❌ | 'Just give us a call' — conditional |
| #87 | SOFT_CLOSE | SOFT_CLOSE | ✅ | Customer 'probably think about it, call back' |
| #88 | SOFT_CLOSE | SOFT_CLOSE | ✅ | Info exchange, 'look out for that text' |
| #89 | PARTIAL | SOFT_CLOSE | ❌ | Email with contact info and links sent |
| #90 | HARD_CLOSE | SOFT_CLOSE | ❌ | 'I'll touch base with you' — schedule uncertain |
| #91 | SOFT_CLOSE | PARTIAL | ❌ | Customer 'I'll call you probably in an hour' |
| #92 | HARD_CLOSE | SOFT_CLOSE | ❌ | Email with all info sent |
| #93 | HARD_CLOSE | SOFT_CLOSE | ❌ | 'Work on this, give me a bit' |
| #94 | SOFT_CLOSE | SOFT_CLOSE | ✅ | Customer 'I will try to come and meet' — conditional |
| #95 | SOFT_CLOSE | SOFT_CLOSE | ✅ | Collision center unavailable, will call back |
| #96 | PARTIAL | HARD_CLOSE | ❌ | Coming to see at 9am — appointment with text confirmation |
| #97 | HARD_CLOSE | SOFT_CLOSE | ❌ | Rate-shopping customer, 'we'd love to help' |
| #98 | PARTIAL | SOFT_CLOSE | ❌ | 'Give me time, give you call back' |
| #99 | SOFT_CLOSE | SOFT_CLOSE | ✅ | Carfax report being sent, will review |
| #100 | HARD_CLOSE | SOFT_CLOSE | ❌ | 'Get with him, give you call back' |
| #101 | SOFT_CLOSE | SOFT_CLOSE | ✅ | Print info, 'feel free to call' |
| #102 | PARTIAL | SOFT_CLOSE | ❌ | Picture exchange continuing |
| #103 | PARTIAL | PARTIAL | ✅ | Existing relationship, will see Steve on Saturday |
| #104 | HARD_CLOSE | SOFT_CLOSE | ❌ | Link sent, 'if questions email me or call' |
| #105 | SOFT_CLOSE | PARTIAL | ❌ | Trade-in deal in progress, will text back |
| #106 | HARD_CLOSE | SOFT_CLOSE | ❌ | Text opt-in for arrival notification |
| #107 | HARD_CLOSE | SOFT_CLOSE | ❌ | Mostly hold music/ads, brief price quote |
| #108 | PARTIAL | PARTIAL | ✅ | Info delivery scheduled for Tuesday |
| #109 | HARD_CLOSE | SOFT_CLOSE | ❌ | 'Give me two hours and I get back to you' |
| #110 | PARTIAL | SOFT_CLOSE | ❌ | Video + opt-in text continuing exchange |
| #111 | HARD_CLOSE | SOFT_CLOSE | ❌ | Text exchange continuing for truck search |
| #112 | HARD_CLOSE | PARTIAL | ❌ | Customer 'hearing from me tomorrow' |
| #113 | PARTIAL | SOFT_CLOSE | ❌ | Email sent, 'give you call back then' |
| #114 | SOFT_CLOSE | PARTIAL | ❌ | Customer 'will call before I come' — conditional |
| #115 | HARD_CLOSE | SOFT_CLOSE | ❌ | Waiting on car availability |
| #116 | HARD_CLOSE | PARTIAL | ❌ | 'If reasonable, I'll call you back and probably buy' — close to commit |
| #117 | PARTIAL | PARTIAL | ✅ | $200 deposit on Visa being arranged, callback pending |
| #118 | SOFT_CLOSE | SOFT_CLOSE | ✅ | Service estimate discussion, 'you would want to schedule time' |
| #119 | HARD_CLOSE | NO_CLOSE | ❌ | Customer hung up: 'Caller has hung up. Please enter your phone code.' |
| #120 | PARTIAL | PARTIAL | ✅ | Girlfriend en route via Uber, mid-negotiation |