Blog

Researchers have long known that AI models generate hidden “confidence signals” when they process information. But a fundamental question remained: Does the AI actually use this internal sense of confidence to decide whether to answer a question or simply say, “I don’t know”? 3. The Proof: To prove this wasn’t just a coincidence, researchers directly intervened in the AI’s internal workings, essentially turning its “confidence volume” up or down. When they artificially boosted the AI’s confidence, it answered more questions. When they suppressed it, the AI abstained more often. This provided direct, causal proof that the model’s confidence level is what drives its decision to speak up or stay quiet. Interestingly, the study found that the AI expresses confidence in two ways: a mathematical one (based on how it calculates the probability of its next word) and a verbal one (when it explicitly states, “I am 80% sure”). The researchers discovered that both of these are actually just simplified glimpses into a much richer, more complex internal understanding of the AI’s own uncertainty. 1. “Saying” it is confident isn’t as accurate as “Being” confident The study makes a crucial distinction between the AI’s internal mathematical confidence and its *verbal* confidence (when the AI literally types out “I am highly confident in this answer”). The researchers found that verbal confidence is a “lossy read-out” and is actually less accurate at predicting whether the AI is right or wrong than its hidden internal probabilities. In short: just because an AI tells you it is certain doesn’t mean its internal mechanics actually back that up. We shouldn’t blindly trust an AI’s explicit claims of certainty. 2. We still can’t see the full “Black Box” Large language models are trained heavily on human feedback (a process called Reinforcement Learning from Human Feedback, or RLHF). During training, human reviewers often reward the AI for saying “I don’t know” when a question is too difficult, rather than making up a false answer (hallucinating). A major ongoing debate in AI research is whether these models have genuinely developed an organic internal sense of uncertainty, or if they are simply executing highly advanced pattern-matching to mimic the abstention behaviors that human trainers rewarded them for during development.

Kumaran et al. show that large language models making decisions on when to answer a question or abstain from answering can be influenced by boosting or suppressing confidence signals in the model.