Imagine yourself being in an autonomous car along with your family for a joyous ride. The weather seems fine, chit-chats are on and you are about to reach the beautiful farmhouse on the outskirts of the city. Now, suddenly with applications of a few pieces of tape on the road, the vehicle is induced to swerve into the wrong lane or speed through the stop sign, as some research also shows – the entire episode holds the potential to produce catastrophic consequences.

With several high-stake applications of AI including self-driving cars, control of power grid with AI, automated surgical assistants, autonomous weapons, etc., robust AI systems are of paramount importance. Unsurprisingly, the concern regarding the stability and safety of these systems increases manifold. In 2015, there were about 168 papers on robust machine learning submitted to arXiv (the open research-sharing platform), while it jumped to 3,318 in 2021. Moreover, adversarial attacks and various defence mechanisms have become a highlight of major conferences including NeurIPS, DEF CON, ICLR, Usenix and Black Hat.  

Decoding the weaknesses 

The surge of adversarial attacks is likely to coincide with the rising deployment of AI. It's a never-ending arms race, but, thankfully several advancements have been made to limit the worst of the attacks. But the very first requirement is to understand the nature and extent of threats. To start with, the attacks against AI systems can be simply categorised as “black box” or “white box.” In white-box attacks, the attacker has access to the parameters of the model, while black-box attacks are those where attackers have no access to these parameters.  

Data poisoning aka adversarial contamination of data is another type of attack. Machine learning systems are fed with training data as input and prepare ML models to produce valuable outputs. An attacker can contaminate this data by injecting malicious samples, causing the process to be disrupted. During the training phase, an adversary may enter data that is mistakenly tagged as harmless but is actually malicious. Last year, Google in collaboration with OpenAI, Stanford, Berkeley, Apple and Northeastern University, demonstrated that large language models like GPT-3 from OpenAI can reveal private and sensitive information when fed with certain words and phrases. 

Data is manipulated to elude detection or to be regarded as genuine in one of the most common tricks – evasion attacks. Although evasion does not imply having control over the data used to train a model, it is similar to how hackers and spammers conceal the substance of spam emails and malware.  

One of the emerging ML frameworks, Vertical federated learning (VFL) allows a model to be trained utilising data from multiple sources on the same set of subjects. For example, using data from several hospitals about the same patients during training could yield richer insights while keeping the real data private. However, to protect data privacy only the model parameters and their gradients are communicated during training. But what if, during VFL, private data could be "recovered" from gradients, as rightly mentioned by the IBM team. 

Need to deal with the mess 

Adversarial attacks put the stability and safety of AI and robotic technology in jeopardy. As the specific conditions for such attacks are often incomprehensible to humans, it is difficult to forecast when and where the attacks could take place. “Through 2022, 30% of all AI cyberattacks will leverage training-data poisoning, AI model theft, or adversarial samples to attack AI-powered systems,” as per Gartner’s report. 

Challenges are real, but several steps have been taken towards scrutinising the security of ML systems. To that end, Microsoft, MITRE the non-profit organisation, and 11 other organisations including the big tech giants NVIDIA, IBM, and Bosch released the Adversarial Threat Matrix - an industry-focused open framework designed to help security analysts detect, respond to, and remediate threats against machine learning systems. Additionally, the IBM team has come up with several solutions and provided the first formal analysis of the robustness of neural networks against ML models’ parameters perturbations. The team further developed an attack method called Catastrophic Data Leakage in Vertical Federated Learning (CAFE) to recover sensitive data, and two new algorithms to enhance robustness in AI models. 

The lack of robustness of the AI systems can be a fatal blow to the adoption of this emerging technology in the future to come. To conclude, the time calls for a concrete effort towards making AI hack-proof, else the incredible transformations across healthcare, finance and defence, brought by AI will lose ground and may take us towards another “AI winter.” 

Image Credits: Unsplash