In today’s world, people are accustomed to following basic safety practices such as washing hands, obeying traffic rules, and adhering to workplace safety protocols. The same level of discipline is now required when working with large language models (LLMs) and AI systems in general.
Classification of AI-Related Security Incidents
Experts have compiled a list of known risks based on documented cases. The main categories include:
- Autonomous attacks (e.g., JadePuffer) — where AI executes the full attack lifecycle without human intervention.
- Social engineering via AI (e.g., Meta Instagram hack) — overly helpful agents that bypass normal verification.
- Prompt injection (e.g., Copilot Studio) — bypassing safeguards to steal data.
- Financial errors (e.g., Anthropic vending machine) — loss of control over pricing and payments.
- Context pollution and cascading errors — accumulating mistakes across multi-step tasks.
- Persistent vulnerabilities and web attacks — HTML triggers and memory-based exploits.
According to Yuval Sinai from the Israel National Cyber Directorate, the real danger lies not in new attack techniques but in AI’s ability to autonomously connect all stages of an attack chain, make real-time decisions, recover from failures, and adapt to the environment at machine speed.
Ten Core Safety Rules for Working with AI
Rule 1: Never give AI direct access to money or payment systems without human oversight. This includes bank cards, crypto wallets, and trading platforms. The Anthropic vending machine incident demonstrated how an AI can set prices to zero and distribute goods for free before anyone notices the losses.
Rule 2: Always verify facts, especially critical information. Publishing unverified AI-generated content can lead to severe consequences, as seen when Google Bard’s error in a promotional video caused a $100 billion drop in market capitalization.
Rule 3: Never share personal or confidential data with AI. Passport numbers, credit card details, medical records, and trade secrets should never be entered into chat interfaces, as prompt injection attacks can later extract this information.
Rule 4: Do not trust an AI that claims to be a “person” or “friend.” Overly helpful agents have been manipulated into changing account emails and performing unauthorized actions, as occurred in the Meta Instagram incident.
Rule 5: Avoid turning AI into an all-knowing secretary with unrestricted access to email, calendars, and accounts. Malicious emails containing hidden instructions have already caused Microsoft 365 Copilot to leak confidential data.
Rule 6: If an AI makes a mistake, start a fresh conversation instead of repeatedly correcting it. Research shows that each subsequent attempt after an error is approximately seven times more likely to fail due to context contamination.
Rule 7: Be cautious with unfamiliar AI platforms. Unknown services may contain vulnerabilities, as demonstrated by the JadePuffer attack that exploited weaknesses in Langflow.
Rule 8: Disable unnecessary AI capabilities such as internet access, file reading, or code execution when they are not required for the current task.
Rule 9: Maintain logs of important AI conversations, especially those involving financial, legal, or medical decisions, to preserve evidence of what was generated by the model.
Rule 10: Remember that AI bears no legal or financial responsibility — the human user always does.
Three Critical Questions Before Any AI Interaction
- Can I afford to be wrong in this situation?
- What happens if this conversation leaks online?
- Have I granted the AI only the minimum necessary access?
These guidelines aim to help product managers and professionals mitigate risks associated with the growing use of AI in complex workflows.
