AI hasn't replaced traditional cyberthreats. Instead, it is changing the speed, scale and sophistication of attacks while introducing new risks that specifically target AI systems, models and the data they rely on.
That means organizations now have to manage two challenges: familiar cyberthreats that can be amplified by AI and new vulnerabilities related to the AI tools and environments adopting the tools.
In Episode 8 of ZeroDay Law's Decoding Cyber Law podcast, I break down three categories of AI-related cyber incidents and what organizations can do to strengthen their defenses. Here’s a review of the key takeaways from the episode.
Let’s review what general cyber incidents look like. Most cyber incidents fall into a handful of categories:
Stealing data
Conducting espionage
Disabling systems
Interrupting IT operations
If a security event involves an AI tool, it may not fundamentally differ from cyber incidents that came before. For example, if a software vulnerability or misconfiguration results in confidential information being stolen & put up for sale on the dark web, the mechanics of the incident and investigation are fundamentally the same whether the software was an AI tool or traditional SaaS.
Beyond these familiar cyber risks, however, organizations need to understand and prepare for new types of attacks that have arisen because of AI. Let’s look at three.
Creating appropriate AI governance policies requires considering several important areas. Learn more in our blog post, “Why Governance, Security, and Legal AI Risk Are Converging.”
The first AI-specific category is traditional cyber threats made more effective and efficient by AI. Attackers can buy cheap exploit kits and use them to scale cyber attacks exponentially.
Specific examples include:
Phishing
Credential abuse
Credential stuffing
Vulnerability exploitation
Realistic disinformation
AI has made these attacks significantly cheaper to run and easier to scale, which is what makes this category worth watching even for organizations already familiar with the underlying tactics.
The second AI-specific category is adversarial attacks aimed at the AI system itself. These attacks are designed to make the AI produce wrong or malicious output, or to steal the confidential data users have already entered.
The three most common types are
Data poisoning
Data breach or theft from the AI company
Evasion, which involves tricking the model.
Let's take a closer look at each AI-specific adversarial attack.
Data poisoning is when attackers manipulate AI engines to make outputs wrong or to create malicious outputs. Attackers do this by corrupting the training data to mislead the AI engine's intelligence or logic, or by compromising its integrity, so they can continue to inject incorrect or malicious information into the engine.
For example, you ask Claude to generate code that will act for you, and it introduces a vulnerability into whatever product you're building.
The easy way to understand data poisoning: bad data, injected by a threat actor, which leads to wrong decisions and harmful outcomes.
AI tools can expose sensitive data at more than one point. Attackers may target the documents and information users upload, or go after data stored within the AI provider's own systems — which can include documents uploaded from your organization.
Consider any major technology company. They run robust security controls, but they're massive organizations running many tools, and their AI products are often released and updated faster than security review can keep pace. Attackers know these providers hold enormous repositories of confidential data belonging to their customers — organizations like yours — in their AI data stores.
The risk isn't limited to external hackers. An insider threat, a software flaw, or corporate or geopolitical espionage targeting an AI provider can all affect where your data lives and what outputs you receive.
The last of the adversarial attacks is evasion, or tricking the model. These attacks are some of my favorites to analyze.
Take a self-driving car controlled by AI. It's trained to look at a stop sign, recognize it and stop. By putting a piece of masking tape on the corner of the stop sign, an attacker can trick the model, causing the car to go flying through the intersection instead of stopping. Even little changes to a stop sign can trigger a problem with the AI car driving model. Adversarial attacks target the AI system directly.
The last AI-specific category shifts the focus to the environments around the AI.
The last category includes operational and environmental vulnerabilities.
Operational and environmental vulnerabilities can exist anywhere an AI system is developed, tested or deployed. Weak security controls in development, testing or production environments can create opportunities for hackers. Risks can also enter through the software supply chain, including through third-party components or code that contains intentional or unintentional weakness.
All of these vulnerabilities can affect:
Models
Prompts
Training data
Outputs
We’ve just reviewed the three AI-specific categories organizations should be watching for. The next question is what to do about them.
Defending against AI-specific cyber incidents requires organizations to create a strong AI attack defense. A good AI cyber defense system includes:
Continuous monitoring
Adversarial testing
Treating AI problems (like hallucinations, model drift and bias) as potential security incidents
These AI incidents often masquerade as mistakes, which they could be. But further cybersecurity investigation is always warranted, as these “mistakes” could also be an intentional, malicious act.
From an information security perspective, industry standards provide a baseline for a proactive AI incident approach. Start by initially consulting industry standards, reviewing the recommendations, following them and then figuring out what controls make the most sense for your organization.
Here are several to consider:
NIST’s AI Risk Management Framework
The ISO Information technology — Artificial intelligence — Management system
Guidance from these organizations also provides a glimpse into how other businesses and entities are protecting their AI systems. Cyber industry experts stay current with new cybersecurity incidents and emerging threats, and will keep you informed when new attacks surface and in regards to new playbooks malicious actors are following.
Defense controls will handle your ongoing threat picture. But incident response plans handle what happens when threats breach your controls, which is why they are also an important element to consider in your AI risk management approach.
Incident response is one of my favorite cybersecurity topics because preparation matters when a security incident occurs. Today, an effective incident response plan also needs to account for AI-related risks.
If you're a small or medium-sized organization, make modifications to your existing incident response planning to account for AI tools being used. If you're a larger organization with an entirely separate AI landscape, you may need an AI-specific incident response plan.
Regardless of your business size, including incident response considerations in your incident response plan can help mitigate the damage that can be caused by an AI cyber incident.
Learn about how engaged your board should be in incident response planning with our blog post, ”The Important Role Boards Should Take in Incident Response Planning.”
Incident response work looks different for every organization, and it's easier to think it through with a partner who does it every day.
For help evaluating your AI cyber risk profile and building AI-aware incident response into your program, reach out to ZeroDay Law.
Listen to the Decoding Cyber Law podcast for the full discussion.