- AI systems are taught to recognize patterns, but they do not always have the flexibility to recognize slight deviations from those patterns.
- AI-powered cybersecurity tools are susceptible to data poisoning and evasion attacks, where attackers manipulate training data or inputs for malicious aims.
- Whether building or buying AI-based security tools, organizations must control access, ensure resilience, and define incident plans.
If you wanted to build an autonomous vehicle, part of the training would involve feeding thousands of pictures of road signs into its AI brain. The car could become very good at reading signs. It would stop, yield, and observe the speed limit with great accuracy—and great confidence.
But stick a few pieces of electrical tape on a stop sign, so it roughly looks like a green light instead of a stop sign, and the car will sail right through an intersection.1
That AI brain never learned what a stop sign means. It learned what stop signs usually look like. If you change the way a stop sign looks, even with a few strips of tape, that brain might continue to be confident in its knowledge, even though it’s completely wrong.
Enterprise AI systems—including those used for cybersecurity—can be fooled by attackers in similar ways. For example, attackers could provide inputs that appear safe (they do not match a template for a threat), but they nevertheless cause the AI system to malfunction in some way, possibly outputting sensitive information or executing unauthorized actions.
How can organizations guard against these kinds of attacks, which are multiplying as more organizations adopt AI systems? Some key best practices for securing AI systems and evaluating AI vendors can help ensure that organizations are prepared for these deceptive threats and other attacks.
The Weaknesses of AI-Powered Tools
Just as the AI brain of an autonomous vehicle learns how to recognize a stop sign, AI-powered cybersecurity tools learn how to identify threats. Tools are often fed large volumes of data so they can recognize both normal behavior and anomalous—potentially threatening—behavior.
There are a few problems with this approach. First, the training data could be poisoned. If an attacker provides a model with incorrect data about normal behavior, the model will have trouble recognizing a threat. Second, even if the model receives only clean data, it still might not detect slight variations in inputs that indicate a threat. Just like the stop sign example, an attacker could very subtly change input data so it looks safe when in fact it is not.
Attackers understand these weaknesses and are becoming very adept at exploiting them. They are poisoning training data and manipulating inputs to capitalize on the brittleness of models.
AI-powered cybersecurity solutions, then, can present serious vulnerabilities to an organization’s security posture. Security teams might thoroughly harden their firewalls and identity layers, but if an AI-based detection model is not tested and hardened with the same rigor, it could render all that other work meaningless.
Closing the Security Gap for AI-Powered Tools
AI-powered cybersecurity solutions are already becoming essential tools for combating AI-enhanced threats. But organizations must implement policies and processes that control access to systems, ensure resilience of tools, and prepare for failures.
Control who touches the model and the training data. To make sure the model is trained correctly, organizations must carefully guard access to the model and the training data. They should treat the training data pipeline as a production pipeline, instituting the same level of access control. Logging changes is also key: If a team finds a problem with training data, that team needs to quickly determine who introduced the problem.
Test for resilience. An AI model—or AI-based tool—might be perfectly suitable for a task in normal conditions. For example, it might effectively detect threats. But what happens when its training data is poisoned or it receives malicious inputs? It might no longer deliver the intended (accurate) results. Thoroughly testing the model or tool to see if it can continue to operate under adverse conditions is imperative before putting it into production.
Plan for a detection failure. When an attacker successfully breaches a traditional hardware tool, software engineers will write a patch that repairs the vulnerability. If attackers use data poisoning on an AI-based tool, however, the security team might not even know there is a problem until a significant security occurs. And when it does, there is no patch that can fix the model—it has to be retrained. Planning ahead for this scenario can save valuable time getting the system back up and running after a failure.
Evaluating Cybersecurity Vendors
Of course, not all organizations are building and implementing their own AI-powered cybersecurity tools. Consequently, they need to scrutinize the vendors and solutions they are considering implementing.
Buyers expect security products to plug into an existing tech stack without significant customization or downtime. At the same time, they expect the AI model at the heart of the AI tool to work—and to keep working—no matter what.
To help ensure they can count on these tools, they should ask vendors three things:
- Who on your side can modify the model? Even if an organization is not building the model and tool themselves, they should know who has access to it and could modify it, potentially maliciously.
- What happens to threat detection quality under adversarial input? This is essentially the resilience test. Will the tool stop working when it is attacked?
- What is your incident path when the model is the compromised component? Just as they build an internal plan for failure, they should be sure that a vendor similarly has a detailed process in place for when problems arise.
Gaining Executive Buy-In
Despite the speed of AI development and adoption, AI systems still have potential vulnerabilities. Just as a few pieces of tape on a stop sign can cause an autonomous car to fail, very slightly altered inputs can cause problems for an AI-powered cybersecurity tool. Those vulnerabilities require some extra precautions whether an organization is building or buying capabilities.
The executives, board members, or other decision-makers who need to sign off on an AI-powered cybersecurity solution might not want all the details about how it works or even how it can fail. Instead, they need to know that it has been hardened, just like any other security solution. Making sure the solution has access controls in place, will continue to function in the event of an attack, and is backed by a detailed incident plan of action can help reassure all decision-makers that it is worthy of investment.
A: AI-powered cybersecurity tools learn to recognize certain patterns as threats. But if attackers “poison” (maliciously alter) training data, the tool might be unable to detect threats. Alternatively, attackers can provide subtly altered inputs that evade detection. Organizations must have controls and plans in place to address these types of vulnerabilities.
A: When building AI models and tools, organizations should control who can access the model and its training data. They should also evaluate whether the model will remain operational for a job even when it is attacked. And they must plan for failures, ensuring that the system can be restored or retrained quickly.
A: Organizations should ensure that AI-powered security solution vendors have tightly controlled access to models and training data. Solutions should also continue to function even in the event of receiving malicious input. The vendor should also have a clear incident path defined for security events.
- Marcus Comiter, Belfer Center for Science and International Affairs, Harvard Kennedy School, Attacking Artificial Intelligence: AI’s Security Vulnerability and What Policymakers Can Do About It, August 2019
