Why AI sounds confident when it's wrong (and how to catch it)

Lawyers keep getting sanctioned for filing briefs full of court cases that don't exist. The AI didn't flag a single one. It wrote them the way it writes everything: fluent, specific, and dead sure.

You asked the model a question and it gave you a clean answer. Names, a date, a citation, no hedging. It read like it knew. Half the time it does. The other half it's guessing, and it sounds exactly the same either way.

That's the whole problem. A person who isn't sure usually shows it. The model doesn't. The confidence is the danger.

The honest answer

A large language model is a fluent next-word predictor, not a truth engine.

It generates the most plausible next words from patterns it learned, optimizing for sounding right, not for being right. It has no built-in "I don't know." So when it has the answer and when it's bluffing, the output looks identical: well-structured prose, specific details, real-looking citations, zero hedging.

A 2025 OpenAI paper put a sharp point on why it bluffs. Models get graded like students on a test, where a confident guess scores better than admitting you don't know, so they're effectively trained to guess rather than say "I'm not sure." The paper's own example: ask three top models for a researcher's dissertation title and you get three different confident answers, all wrong.

That's the shape of it. A bluff isn't honestly vague ("sometime in autumn"). It's overconfident and specific ("September 30"). The specificity is what fools you.

Why it's harder than it looks

The output is built to be believed. Fluent, well-formatted, sourced-looking, no doubt anywhere. Research suggests models are systematically overconfident about how right they are, and people over-trust polished output from a machine, the same way we over-trust anything that looks authoritative. As one person put it on X, hallucination is "confident guessing, trust-me-bro technology."

The marquee example is sitting in court records. Lawyers keep getting sanctioned for briefs containing AI-fabricated case citations, a pattern that has recurred from 2023 into 2026. A database that tracks these has logged more than a thousand filings with AI-hallucinated content, and courts have handed down real sanctions. The point isn't the count. It's that the fake citations looked real, complete with case names, courts, and page numbers. That's exactly why they slipped past trained lawyers and into a filing. Fabricated statistics and quotes do the same thing in board decks and memos.

How often does it happen? It varies wildly by task. Independent leaderboards put the best models around 1-2% on easy grounded summaries, where the model just has to restate a document in front of it. On harder, long real-world documents, law, medicine, finance, top models reportedly still blow past 10%. It has improved on average. It has not disappeared. Counterintuitively, some newer "reasoning" models hallucinate more on factual recall, not less. And as one person put it on X, the rarer it gets, the more dangerous it becomes, because a made-up citation looks exactly like a real one.

What to do this week

You don't need a better model. You need to know where you can't trust it blind, and build a check into the workflow. Six moves:

• Never trust it blind on anything you'd stake money or reputation on. Names, numbers, citations, legal, medical, financial facts. Confidence is not correlated with correctness, so the sure tone tells you nothing about whether it's right.
• Ask for sources, then actually open them. A formatted citation is the single easiest thing to fabricate. The link or the case has to exist and say what the model claims it says.
• Use it where you can verify the output or where an error is cheap. Drafting, brainstorming, reformatting: you can see if it's good. Novel facts you can't check: don't.
• Cross-check important facts against a second source or a second model. Two independent answers that agree is a signal. One confident answer is not.
• Make "where did you get this?" a standing habit. Ask it every time, of the model and of your team, until it's reflex.
• Say the rule out loud to your team. The real failure isn't the model being wrong. It's a person shipping confident AI output unread.

This is the live state of play in 2026, not a 2023 problem you can wave off. Grounding and citations help. They don't eliminate it. The legal-sanction wave running through this year is the proof.

Want to maximize your AI leverage? Upgrade to Pro.

Pick the one place AI output already flows into a real decision in your company: a memo, a number in a board deck, a customer answer. Add one verification step this week, a human who checks the facts against a real source before it ships. Confidence is not correctness, and the gap is your liability.

Related

• What is a large language model?
• Why most AI agents fall apart in real work
• What is a context window?
• Which AI releases actually matter