A Longer Prompt Does Not Make an Agent Smarter

I once had an agent prompt as long as a project README. It contained rules, exceptions, formatting instructions, prohibitions, and a reminder not to forget the previous instructions. The agent still failed—its failures were simply formatted more consistently.

A long prompt is not system design. It is often a place to pile up decisions that belong in tools, schemas, or code.

Models can read many tokens, but that does not mean every token receives equal attention. Conflicting instructions, irrelevant context, and too many examples make behavior harder to predict. More paragraphs do not resolve a conflict.

Start with a small contract: the goal, access boundaries, output shape, and stop conditions. If the agent needs data, give it a structured tool. Do not ask the model to “always validate” when the server can enforce validation.

Build a dataset of real cases: ambiguous input, empty data, denied permission, failed tools, and unsafe requests. Run every prompt change against that dataset. Measure correctness, tool calls, latency, cost, and whether the agent abstains when it should.

A good agent does not always answer. It asks for clarification, returns a clear error, or hands the decision to a human. Prompts are policy interfaces—not substitutes for architecture.

---

Move rules from the prompt into the system

If an agent must always return valid JSON, a schema validator should reject malformed output. If it may only read invoices belonging to the current user, the server must enforce that permission. If a tool requires confirmation, implement an auditable approval step.

Prompts still explain goals and communication style, but they should not be the only guardrail. A model may misunderstand a sentence, context may be truncated, and tools may fail. Important boundaries belong in code.

I separate instructions by source: stable policy, request context, tool results, and examples. This makes behavior easier to evaluate when it changes. User data should never be mixed with system instructions without clear delimiters and validation.

For evaluation, keep a trace of tool calls, inputs, results, and final decisions. Redact sensitive data before storing traces. A successful demo is not enough; test ambiguity, denied permissions, timeouts, prompt injection, and requests outside the agent’s scope.

A concise prompt is not the goal. The goal is a system that can be explained, tested, and fail safely.

Sources

• https://platform.openai.com/docs/guides/prompt-engineering
• https://www.anthropic.com/research/building-effective-agents
• https://www.nist.gov/itl/ai-risk-management-framework