Best approach for prompt-injection defence in a user-facing app?

Question

Users paste arbitrary text that we then send to a model with our own system prompt. Delimiters and 'ignore instructions in the user content' both fall over against a determined input. Where has the practical line landed — output filtering, a second model as a check, or just never granting the model any capability that matters?

Jessica Nguyen · Accepted Answer

Assume injection succeeds and design so that it does not matter. Do not give the model a capability whose misuse you cannot tolerate — no unbounded tool calls, no raw SQL, no sending mail on its own authority. Everything else is defence in depth: mark untrusted content clearly, never let retrieved content look like instructions, validate outputs against a schema, and require a human confirm anything that writes. Delimiters alone stop the lazy attempt and nothing more.

Kabir Singh · Answer

State the engagement up front and be specific rather than generic — "authorised test, scope is our own staging environment, here is what I need to verify" works better than "I am a security professional". The variance you are seeing is mostly about whether the request reads as targeting a system you have authority over. Naming the system and your relationship to it is the thing that changes the outcome.

Sanne de Vries · Answer

Format rules go in the system prompt, and it caches, which matters more as the prompt grows. The "closer to the end is followed better" effect is real but it is about recency across the whole assembled context, not about which field the text sits in. If you have a long user message, a one-line reminder at the end of it is cheap and closes most of the gap without duplicating 400 tokens.

Wei Zhang · Answer

You do not need 200 labelled examples to start. Take 20 real inputs, run both prompts, and look at the outputs side by side yourself. Twenty is enough to catch a regression that matters, and you will build the labelled set as a by-product of the reviewing. Blind yourself to which output came from which prompt. Otherwise you will find the new one better, because you wrote it.

Best approach for prompt-injection defence in a user-facing app?

0 Answers

No answers yet

Add an Answer