A group of MIT researchers in collaboration with Harvard University and Fujitsu Ltd. sought to understand when and how a machine-learning model can overcome bias in the dataset on which they are trained.

One of the finest examples and the widely criticised facial recognition technology, when trained on biased data, exhibits how unfair AI systems can be, despite producing results in no time.

“A neural network can overcome dataset bias, which is encouraging. But the main takeaway here is that we need to take into account data diversity. We need to stop thinking that if you just collect a ton of raw data, that is going to get you somewhere. We need to be very careful about how we design datasets in the first place,” says Xavier Boix, the senior author of the paper.  

The team used an approach from neuroscience to study how training data affects whether an artificial neural network can learn to recognise objects it has not seen before. They built datasets that contained images of different objects in varied poses and carefully controlled the combinations so some datasets had more diversity than others. In this case, a dataset had less diversity if it contains more images that show objects from only one viewpoint. A more diverse dataset had more images showing objects from multiple viewpoints. Each dataset contained the same number of images.

The researchers trained a neural network for image classification using these carefully generated datasets, then tested how well it could recognise subjects from angles the network had not seen during training. The researchers discovered that a more diversified dataset allows the network to generalise to new images or viewpoints.

“Data diversity is key to overcoming bias. But it is not like more data diversity is always better; there is a tension here. When the neural network gets better at recognizing new things it hasn’t seen, then it will become harder for it to recognize things it has already seen,” says Boix.