Chat Completions

Guardrails Filter Tackles Complex LLM Streaming and Tool Call Challenges to Protect Sensitive Data

Developers at Cloud.ru built Guardrails Filter to mask personal data such as phone numbers, emails, passport details and names before they reach large language models. The system replaces detected values with consistent placeholders like <PHONE_1> and maintains a mapping table so original data can be restored after the model responds. Simple replacement proved insufficient because identical values must receive the same placeholder across an entire conversation history, and the model receives the full message array on every request. Streaming responses using SSE create additional difficulties since placeholders can be split across multiple chunks, requiring buffering of 10-15 characters and state tracking for reasoning, content and tool_calls. The team also had to handle JSON-inside-JSON arguments for tool calls, different field names across providers, and edge cases such as escaped newlines matching email patterns. Separate implementations were written for OpenAI Chat Completions and Anthropic Messages APIs, resulting in roughly 1,500 lines of streaming code and more than 4,000 lines of tests to ensure agent pipelines remain intact.