LLM Prompt Injection

Prompt injection is the new SQL injection, with one large complication: the parser is a probabilistic neural network whose decision boundaries are not specified anywhere. There is no equivalent of parameterized queries because the LLM does not have a structural separation between code and data — both arrive as text in a single context window. Every developer who builds a chatbot, summarization tool, or RAG-backed search starts with the same naive pattern (concatenate the system prompt with the user's input, send it to the LLM), and every one of them is vulnerable until they layer on defenses. Worse: the attacker doesn't need to be the user. Indirect prompt injection — instructions hidden inside documents, web pages, or emails the LLM consumes — turns content fetching into command execution.

Prompt injection appears when LLM-facing inputs can override instructions, leak context, or trigger unsafe tool behavior. The risk depends on what the AI feature can read or do inside the product.

Data exfiltration: system prompts (often containing internal context, business logic, or credentials), conversation history from other users in shared deployments, document contents from RAG systems. Reputation damage when chatbots produce offensive output that screenshots cleanly. Phishing assistance via injected URLs that the LLM presents as legitimate. Financial loss when LLMs with tool access execute unintended operations — sending email, posting messages, hitting paid APIs, calling code-execution sandboxes. In agentic systems with broader tool access, the impact grades up to remote code execution.

Treat user input as untrusted by the LLM. The most reliable structural defense is to constrain output to structured formats (JSON schema with strict validation, function-calling with tool-side permission gates) so the LLM's output is parsed by your code rather than executed. Layer the system prompt to restate boundaries after user content — instructions placed at the end of the context have measurably better adherence than those at the beginning. For tool-using agents, gate every dangerous operation behind a human approval step (don't let an LLM send email, transfer money, or execute code without a confirmation prompt). For indirect injection, sanitize fetched content before adding it to the context — strip HTML attributes, comment markers, and instruction-shaped text patterns. Run a second model as a 'jailbreak detector' over user inputs and outputs in high-stakes deployments; OpenAI's moderation endpoint and Anthropic's classifier APIs cover the common cases. Most importantly, design with assumed compromise: what's the worst the LLM can do if it follows attacker instructions? If the answer is 'leak the system prompt,' you might accept that risk. If the answer is 'send mass email or call paid APIs,' you need the gates.

There is no parameterized-query equivalent for LLMs yet. Defense is currently architectural — constrain what the model can do with its output, not what it can be told to do.