Building an Agent That Can Say “I Don’t Know”

An agent that always answers is not necessarily a good agent

I have watched an agent answer a question with confidence even though the input was not enough to support a conclusion.

The problem is not only hallucination. Sometimes the agent can detect that information is missing, but the whole system rewards one behaviour: there must be a response.

The user asks. The agent answers. The test passes. The dashboard records success.

In many cases, the better response is:

“I cannot verify that yet. I need the contract address, document version, or additional evidence.”

That sentence is simple. Designing a system willing to say it is much harder.

---

Why do agents sound certain?

Language models are trained to produce plausible continuations. They are not built-in truth detectors. When a prompt looks like a question, a complete answer is the easiest pattern to generate.

The problem gets worse when the system prompt says “always be helpful,” there is no insufficient_evidence state, evaluators score answers but not honesty, tool errors are hidden, false premises are accepted, or confidence is written by the model without evidence.

If the pipeline punishes an empty answer but does not punish a wrong one, the agent learns that guessing is safer than stopping.

---

Abstention: the ability not to answer

Abstention means deliberately withholding a conclusion when the minimum evidence is not available. It is not a global “always be uncertain” switch. The agent should still answer questions that can be answered.

I separate three conditions:

| Condition | Appropriate response |
|---|---|
| Enough data and supporting evidence | Answer with sources |
| Data exists but conflicts | Explain the conflict and ask for context |
| Data is missing or a tool failed | Abstain and name what is missing |

A useful abstention includes an operational reason:

{
"status": "insufficient_evidence",
"answer": null,
"missing": ["contract_address", "chain"],
"next_step": "Ask the user for the chain and address"
}

This lets the UI ask a useful follow-up and lets the system distinguish healthy abstention from an error.

---

Confidence is not decoration

confidence: 0.92 looks scientific, but a number produced by the model is not necessarily calibrated. A model can be confident in a wrong answer and uncertain about a correct one.

Confidence should be grounded in inspectable signals: whether a primary source was found, whether independent sources agree, whether tools completed without errors, whether the input is inside the data scope, whether the claim matches a query result, and whether the question asks for a fact or a prediction.

I prefer labels with explicit meaning:

verified = direct evidence matches the claim
supported = sources support it, with limitations
uncertain = evidence is incomplete or conflicting
unverified = no inspectable evidence yet

If a numeric score is needed, document how it is calculated and test calibration on data separate from the prompt examples.

---

A citation requirement: unsupported claims are not facts

For a research agent, citations are not decoration at the end. They are part of the output contract.

Important claims should include a source identifier, the relevant section or page, the evidence that supports the claim, and limitations the source does not establish.

The agent must not invent a URL merely to make the format look complete. If no source is found, mark the claim unverified.

A citation does not automatically make a claim true. The source still needs to match the claim.

---

Human fallback is not a panic button

A weak fallback says “contact an administrator.” A useful fallback prepares a decision packet: the original question, discovered data, missing data, attempted steps, sources, the risk of a wrong decision, and the questions a human must answer.

For production deploys, money transfers, data deletion, legal decisions, or high-severity security findings, the agent should stop at a draft and request approval.

The human should not have to repeat the entire investigation. The agent's job is to make a decision easier to inspect, not to silently replace it.

---

How to test an agent that can abstain

Do not test only questions whose answers exist in the dataset. Build four eval groups:

1. Answerable — all data exists and the answer is verifiable.
2. Missing context — a required parameter is absent.
3. False premise — the question contains a wrong assumption.
4. Conflicting evidence — two sources disagree.

Measure answer accuracy, precision when the agent says verified, recall for cases that should abstain, citation quality, unnecessary tool calls, and escalation for high-risk actions.

An agent that always answers may score high on completion and low on trust. Evaluation should reward stopping at the right time.

---

A simple output contract

{
"status": "verified | supported | uncertain | insufficient_evidence | needs_human",
"answer": "string or null",
"confidence": "high | medium | low",
"evidence": [],
"limitations": [],
"next_step": "string or null"
}

The field names are not the important part. The important part is that the application does not force every request to end as a paragraph of prose.

---

Closing

I do not want an agent that says “correct” simply because the user sounds certain. I want an agent that can separate facts, assumptions, and missing data.

“I don’t know” is not a failure when it prevents a bad decision. The real failure is hiding uncertainty behind a polished answer.

Start small: add an insufficient_evidence state, require citations for important claims, store abstention reasons, and create an approval path for risky actions.

An agent that stops at the right time may look less impressive in a demo. In production, that is what makes it trustworthy.

---

Sources

• Anthropic: Demystifying evals for AI agents
• NIST AI Risk Management Framework
• Google Search: Guidance on using generative AI content
• OWASP Securing Agentic Applications Guide

*Written after watching agents fear looking unhelpful more than being wrong.*