Insights · Trend · AI
What Chatbots Must Never Say: Hallucinations & Law
The fear of a bot promising a price out of thin air, or leaking data, is enough to stall an AI rollout at board level. Reasonably so — because safety does not live in the prompt but in the architecture: in where the model gets its numbers and what happens to an answer before the customer sees it. And the company answers for it, not the bot.
What's changing right now
The era of 'sandbox' AI testing is over – companies now demand full legal accountability for bot responses. Guardrail systems are gaining prominence, filtering AI outputs before they reach the customer to eliminate the risk of erroneous financial declarations.
What you can do better
Concretely, starting today
- Implement a price verification layer (hard filters) that blocks any numbers not originating from your official database.
- Before a conversation reaches the model, swap personal data for identifiers — “customer 4471” instead of “John Smith”. That genuinely lowers the risk, but it does not take you outside the GDPR: as long as a key exists anywhere that leads back to a specific person, this is still personal data. Only irreversible anonymisation falls outside the GDPR — and you rarely want that, because it also costs you the knowledge of whose conversation it was.
- Use 'query-response' logging in a secure, isolated environment to enable auditing of every interaction.
- Enforce a restricted mode (Grounding) where the bot can only use the provided corporate documentation.
Proof from our work
In FlashAI we solved this with architecture, not an incantation in the prompt. Anti-hallucination rules are appended at runtime and only for the tools a given bot actually has — we never tell the model about functions it will not find. For pricing the wording is blunt: the bot must query the catalogue before quoting any price and never quotes from memory. It may not list an item the catalogue did not return — the list is limited to the company’s offering, not to its general industry knowledge. When it finds nothing it has to say plainly that the item is not in the online catalogue and hand over to a human rather than guess. Ranges are quoted verbatim, with no averaging. That is exactly the separation of business logic from the model described in the questions below.
Frequently asked questions
How to stop chatbot price hallucinations?
Separate business logic from the language model. The AI should only identify the product category, while the actual price must be fetched by an external script from your ERP/PIM system.
Is logging conversations GDPR compliant?
Yes, but not by default. Start with the legal basis for processing and with telling the user why the conversation is stored — that is a duty, not good manners. Then: a defined retention period instead of keeping logs indefinitely, restricted access, and a way to handle deletion requests. Mind the wording: under the GDPR “sensitive data” is a separate category (health, beliefs and so on) under a stricter regime — if your bot serves a clinic, that is where you are. An ordinary sales chat is personal data, but not sensitive. We cover this in more depth in our piece on consent and GDPR audits.
Is a company liable for what its chatbot says?
Practice is moving towards yes. The best-known precedent is Moffatt v. Air Canada (2024): the bot gave a passenger incorrect discount terms, and the airline argued the chatbot was a separate entity responsible for its own words. The tribunal rejected that defence and held the carrier bound by the information on its own site. That is a Canadian ruling, not Polish, but the direction is clear: a bot is a company’s communication channel, not a separate being. Which makes output control a question of financial exposure, not of service quality.
Secure your AI implementation
Don't let the fear of AI errors block your growth. We will help you set up secure guardrails.
