Researchers observed that machine-learning algorithms replicating human decision-making are frequently harsher than humans. They found that how data is collected and labelled affects how accurately a model can be trained to determine if a rule has been broken.

"Before we work on AI, why don't we do something about natural stupidity?" - Steve Polyak

Machine-learning models are occasionally created to simulate human decision-making, such as determining whether or not a social media post violates harmful content regulations to increase fairness or decrease backlogs.

However, MIT and other researchers have found that these models cannot be relied upon to accurately reflect how humans judge when faced with potential rule infractions. Without adequate training data, algorithms will likely render verdicts that differ from and are often more severe than those reached by people.

Training data

In this context, the "right" data are labelled by people who were explicitly asked if objects violated a rule. For a machine learning model to learn a task, it must be exposed to millions of examples of this "normative data" during training. Machine-learning models require factual labels. Thus people are usually asked to determine whether or not a given image contains, for example, fried food. Over-prediction happens when "descriptive data" are used to train algorithms that determine rule violations, such as whether a school meal violates a fried food prohibition.

This decline in precision may have significant consequences in the actual world. If, for instance, a descriptive model is used to determine if an individual is likely to re-offend, the results may lead to more severe punishments (such as more significant bail amounts or longer prison terms) than would be imposed by a human court.

Labelling discrepancy

This research stemmed from an earlier investigation into the justification capabilities of machine learning models. Data collection for that study revealed that people's responses could vary depending on whether they were asked to describe or evaluate a set of variables.

Researchers ask labellers questions like "Does this text contain obscene language?" to collect descriptive labels. Finally, researchers collect normative labels by providing a rule and asking labellers to indicate whether or not the data in question violates that rule, such as "Does this text violate the platform's explicit language policy?" Researchers were taken aback by this result and conducted a user study to learn more. 

First, they assembled four datasets to serve as stand-ins for a variety of regulations, including one including pictures of dogs that would be considered violent by a landlord. The next step was to have participants form groups and provide normative or descriptive labels to the data.

Humans were shown to be far more inclined to categorise an object as a violation in the descriptive environment, as discovered by the researchers. They found that the discrepancy, measured by taking the average absolute difference in labelling, varied from 8% for a collection of photographs used to determine dress code infractions to 20% for the dog images.

Training troubles

The researchers trained two models to judge rule infractions in one of their four data sets to examine the potential effects of repurposing descriptive data. One model was trained with descriptive data, while the other was trained with normative data, and then their results were compared.

One solution to this issue is to increase dataset visibility. Researchers can better apply collected data if they understand the methods utilised to manage them. Another approach is to fine-tune a model that has been descriptively trained using a small sample of normative data. The researchers plan to investigate this concept of transfer learning further in their future studies.

They are also interested in conducting a similar study with expert labellers, such as doctors or lawyers, in determining if this same labelling discrepancy occurs with these professionals.

Conclusion

As governments and businesses increasingly rely on automated decision systems, it's crucial to ask how accurately they can mimic human judgement. The researchers found that annotators categorise items differently depending on whether they are asked a factual or normative question, revealing a significant issue. Moreover, it presents a problem for many common practises in machine learning data acquisition, which assumes that predicting the accurate classification of an item and exercising judgement about whether an object violates a rule dependent on those facts are equivalent. 

They discover a significant measurement error is introduced when factual labels train models designed for normative judgements. The study's findings demonstrate that models trained with accurate labels produce significantly different judgements than those trained with normative labels. This effect on model performance can outweigh other factors (such as dataset size) that routinely attract attention from ML researchers and practitioners.

Image source: Unsplash