In a utopia, things are as good as how they seem to be. However, that isn't the reality we live in. There are errors of judgement.Researchers wanted Artificial Intelligence (AI) to be of better judgement! For example, collision avoidance systems in self-driving cars may have 'adversarial inputs' due to the slightest of factors such as an image shifting a few pixels due to a glitch in the camera. 

MIT researchers have developed an algorithm that attempts to beat these adversarial inputs by ensuring that the machines 'understand' that they operate in the imperfect, real world. This is done by ensuring that machines possess a healthy 'scepticism' towards inputs such as measurements that they've received. 

The team has built CARRL, as in Certified Adversarial Robustness for Deep Reinforcement Learning, by combining two approaches - a reinforcement-learning algorithm and a deep neural network - that are used as trainers for computers to enable them to play video games such as chess and go. 

CARRL has been tested in various settings such as a stimulated collision-avoidance test and Pong, a video game. The researchers found that when compared to other standard machine-learning techniques, CARRL did much better, even when it faced adversarial inputs. 

"You often think of an adversary being someone who's hacking your computer, but it could also just be that your sensors are not great, or your measurements aren't perfect, which is often the case," says Michael Everett, the lead author of a study outlining the new approach and a postdoc in MIT's Department of Aeronautics and Astronautics (AeroAstro). "Our approach helps to account for that imperfection and make a safe decision. In any safety-critical domain, this is an important approach to be thinking about." The study originated from MIT PhD student Björn Lütjens' master's thesis and MIT AeroAstro Professor Jonathan How guided the team. This study, outlining the new approach, first appeared in IEEE's Transactions on Neural Networks and Learning Systems.

Traditionally, a neural network associates certain labels and actions with a set of inputs. For example, a neural network that has been fed thousands of images labelled as cats would be able to 'identify' an image of a cat. This is a type of supervised-learning technique. To ensure that an AI system is truly robust, ideally, datasets should introduce with several slightly altered versions of the cat images. But running through every possible image alteration is computationally exhaustive and difficult to apply successfully to time-sensitive tasks such as collision avoidance. "In order to use neural networks in safety-critical scenarios, we had to find out how to take real-time decisions based on worst-case assumptions on these possible realities," Lütjens says.

The team then looked beyond the neural network and towards reinforcement learning, which is used to train computers to play and win at chess and Go, since it doesn't associate labelled inputs with outputs. It focuses on producing outputs based on a response to certain inputs to get the best reward possible. 

However, CARRL employs a deep-reinforcement-learning algorithm to train a deep Q-network, or DQN—a neural network with multiple layers that associates an input with a Q value, or level of reward. Through this approach, CARRL takes an input, for example, a dot and considers how adversarial inputs could have slightly changed the dot from its actual position. Therefore, CARRL will evaluate an output after considering every possible position that the dot could have been in the given region, and come up with the ideal most action. This technique has been a contribution of yet another MIT graduate student, Tsui-Wei "Lily" Weng, who successfully received a PhD in 2020. 

The team pitched CARRL to play Pong, a video game in which two opponents are pitted against each other and use paddles to bass the ball to and fro; and introduced a glitch that placed the ball further down that where it was supposed to be. The results were satisfactory according to the creators. "If we know that a measurement shouldn't be trusted exactly, and the ball could be anywhere within a certain region, then our approach tells the computer that it should put the paddle in the middle of that region, to make sure we hit the ball even in the worst-case deviation," Everett says.

To test the collision avoidance capacity of CARRL, the team created a blue and orange agent that want to switch positions without colliding into each other. Then the team introduced glitches in the orange agent's record of the blue agent's placement. CARRL was successful in steering the orange agent around the blue agent. Each time the blue agent's location became more doubtful, the organe agent responded by creating a wider berth. So much so that at one point, CARRL avoided the destination all together because it assumed that the blue agent can be anywhere in its vicinity. 

The researchers say that the acute conservatism is also beneficial since researchers can fine tune an algorithm's robustness by using the extremes as a limit. "People can be adversarial, like getting in front of a robot to block its sensors, or interacting with them, not necessarily with the best intentions," Everett says. "How can a robot think of all the things people might try to do, and try to avoid them? What sort of adversarial models do we want to defend against? That's something we're thinking about how to do."