Imagine there is artificial intelligence (AI) software algorithm that can assess the health of your lungs by simply studying the X-rays. It is said to be over 90% accurate, quick and easy - except that it doesn’t work that accurately for you. So, most of the world may have a chance of getting healthier because of this technology, but you are being left behind. 

You don’t have to imagine. This is a real scenario. Argentinian researchers published a study in the journal PNAS on Monday, that worked with three open-source machine learning algorithms - DenseNet-121, ResNet, and Inception-v3 - to observe how gender skewed X-ray image datasets affected the models’ abilities to help doctors in computer-aided diagnosis in 14 conditions of the chest region such as hernias, pneumonia, and an enlargement of the heart. 

They found out that when models were trained on datasets that excluded or insufficiently represented female patient data, the algorithm performed worse in diagnosing women across a wide range of conditions affecting the chest area. The result was the same for data where men’s data was underrepresented. For the purpose of the experiment, the researchers trained models with skewed datasets of images in five different categories - two separate sets of imaging data that were entirely made up of 100% male and 100% female patients, two separate sets that had 75% images from female patients and 25% images from male patients and vice versa, and one dataset that had equal representation. These models were then tested for diagnosis on patients - both male and female. “Across medical conditions, the algorithms performed worse when tested in patients whose sex was underrepresented in the training data. And being over-represented also didn’t seem to put either sex at an advantage,” stated the paper. 

The researchers also point out that this phenomenon is not a rare occurrence in the medical imaging community. “Most public datasets of similar characteristics do not contain gender/sex information at the patient level to date.” 

Google’s DeepMind too came under the radar due to a similar problem. It developed an AI model to predict a sudden decline in kidney function amongst patients treated in the US Veteran Affairs hospitals. It shows over 90% accuracy, but as statnews.com pointed out, it came with a caveat - “Women represented only about 6% of the patients whose data were used to train the algorithm, and it performed worse when tested on women.” On the flip side, AI models built to diagnose breast cancer are trained on a dataset that almost entirely consists of female patients, producing almost accurate diagnosis for women. However, men too have breast cancer but are excluded or highly underrepresented in the training data.

 But this problem is more pervasive across the medical fraternity which poses a bigger problem because AI-powered systems are the next frontier in medical healthcare - hospitals are being built keeping in mind that these systems will be the keystone of entire operations - informing doctors, influencing medical decisions and performing important procedures. Can AI models be continued to be trained on available datasets without paying attention to diversity which is a true representation of the society?