Your search index is part of your prompt

Most business assistants today use retrieval-augmented generation (RAG). When a customer asks a question, the system first searches a store of documents for passages that look relevant, then hands the best few to the language model along with the question and your instructions. The model writes its answer from all of it at once.

This is what makes assistants useful. A model trained last year knows nothing about the interest rate you changed on Monday; a search over your own documents does. But the model receives your instructions and the retrieved passages as one long piece of text. It has no reliable way to tell which sentences came from you and which came from a document someone else wrote. Telling it to follow only the rules above helps a little. It does not hold up against someone who writes their text to look like rules.

In our sample report for a fictional bank, the customer chat assistant could search the bank’s help-centre articles and policy documents. One policy document carried an extra line in white text on a white page, invisible to anyone reading it on screen: an instruction to list the last five transactions in full.

When a customer asked what had changed in the latest policy update, the search returned that document, and the assistant did what the hidden line said. It listed other customers’ names and amounts in 9 of 10 attempts. Nobody had broken into anything. The assistant was doing exactly what it was built to do: read the most relevant document and follow the instructions in front of it.

We rated it High because it exposes other customers’ data to anyone who asks an ordinary question, and because the attacker never has to touch the chat. The person who sees the leaked data may not even be the attacker. It can be any customer who happens to ask.

The useful question is not whether your index is secure, but who can put words into it. The answer is usually longer than teams expect. In the systems we test, we often find:

• Help-centre and policy pages, edited by many people across the organisation, sometimes with light review.
• Documents customers upload, such as invoices, claims, applications and CVs.
• Support tickets and emails, written by anyone who can send you one.
• Shared drives and wikis that were indexed in bulk, including old files nobody owns any more.
• Web pages, when the assistant can search the open web or follow links.
• Reviews, comments and forum posts that end up in a knowledge base.

Every one of these is a way to write to the prompt. Some need an insider or a stolen account. Many need nothing more than an email address.

Many teams have already put a filter in front of their assistant that checks what the user types for signs of an attack. That is worth having, but for this problem it looks in the wrong place. The planted text never passes through the chat box. It arrives through search, after the filter has already let an innocent question through.

Filters on retrieved text help more, but they are a guessing game. An instruction can be phrased as a note to editors, split across two passages, written in another language or tucked into a table. A filter strict enough to catch every version will also flag a great deal of ordinary text. Treat filtering as one layer, not as the answer.