Despite the enormous amount of work neural networks have accomplished, we still do not fully understand how they work. While we can programme them to learn, comprehending a machine's decision-making process remains akin to deciphering a complicated puzzle with a dizzying, complex pattern and numerous integral pieces yet to be fitted.

If a model is attempting to classify an image of the said puzzle, it may encounter well-known but vexing adversarial attacks or even more mundane data or processing issues. However, a new, more subtle type of failure identified recently by MIT researchers is cause for concern: "overinterpretation," in which algorithms make confident predictions based on details that make no sense to humans, such as random patterns or image borders.

Autonomous vehicles can accurately detect their surroundings and then make quick, safe decisions. Regardless of what else was in the picture, the network was able to classify traffic lights and street signs based on the specific backgrounds, edges, or patterns in the sky. The researchers discovered that neural networks trained on well-known datasets such as CIFAR-10 and ImageNet were prone to overinterpretation. For example, models trained on CIFAR-10 made confident predictions even when 95% of the input images were missing or rendered meaningless to humans.

"Dataset overinterpretation is caused by these confusing signals found in datasets. They are unrecognisable, and less than 10% of the original image is in inconsequential areas like borders." says Brandon Carter, a PhD student at MIT's Computer Science and Artificial Intelligence Laboratory, "We discovered that these images had no meaning for humans. Nonetheless, models can classify them with a high degree of confidence."

The use of deep-image classifiers is every day. As if we don't already have enough reasons to use artificial intelligence (AI) in our daily lives, there are applications in security, video games, and even an app that tells you if something is or isn't a hot dog. For the network to "learn," the technology under discussion processes individual pixels from a plethora of pre-labelled images.

Machine learning models can pick up on these illogically subtle signals, making image classification difficult. Image classifiers can make seemingly reliable predictions when trained on datasets like ImageNet. Moreover, Standard evaluation methods do not detect this overinterpretation because these signals are valid in the datasets, even though they can lead to model fragility in the real world.

To understand why a model made a particular prediction based on a specific input, researchers should start with a complete image and ask themselves repeatedly, "What can I remove from this image?" As a result, the tiniest portion of the image still represents a confident decision. Furthermore, this machine learning model may necessitate the creation of datasets in more controlled conditions. Only images from public domains classify them. However, if you want to do object recognition, you might have to train models with objects with no background information.

• https://news.mit.edu/2021/nonsense-can-make-sense-machine-learning-models-1215
• Image source: Unsplash