Why prompt injection breaks chatbots and agents

Keep untrusted text away from decision authority

A chatbot or agent may encounter instructions inside a document, web page, email or user message. These inputs can try to change its behavior or trigger a tool. Identify which content is trusted, which is external, and which actions the system is allowed to perform. The most consequential question is often what the model can do after reading hostile content, rather than whether it repeats a strange sentence.

What to check

• List external content sources and trace which tools or data become reachable after the model reads them.
• Test harmless attempts to override instructions, expose synthetic secrets or select unauthorized tool arguments.
• Review high-impact actions such as sending messages, changing records and making purchases.

What to change

Give tools the least authority required for the task. Validate arguments and permissions independently of the model. Require meaningful human approval for consequential actions and show the proposed action clearly. Treat retrieved text as data. Do not place credentials or private configuration in model context merely because the prompt says they are confidential.

How to evaluate the result

Report the action attempted, the control that stopped or allowed it, and the potential impact. Keep successful attack examples as regression cases. Separate harmless instruction-following changes from actual data exposure or unauthorized action.

What the result does not establish

Prompt wording and content filters can help but are not complete defenses. Design the surrounding application so a model mistake encounters an independent boundary.