The questions regarding machine learning models' robustness, quality and privacy have become a matter of the moment. Machine learning offers a vast amount of opportunities to increase productivity. But the machine learning systems are only valued under the quality of the data that informs the training of machine learning models.

Training machine learning models requires more data than an individual or an organization can offer. At the same time, building explainable models is a broad concept as it helps people to understand the results provided by ML models. Building trustworthy and good-quality ML models is what Dr. Anupam Datta aims to do with TrueEra.

Dr. Anupam Datta, the Co-founder, President and Chief Scientist of TruEra, says they focus on accountable data-driven systems that employ machine learning and other statistical and Artificial Intelligence methods in a conversation with Jibu Elias, Head of Content and Research of INDIAai.

Dr. Anupam Datta, also the faculty at Carnegie Mellon University for 15 years, worked as a Professor and Director of the Accountable Systems Lab. He predominantly worked in other aspects of privacy, fairness, security, and compliance and has a Master's degree from Stanford University and a B.Tech from IIT Kharagpur, in Computer Science. 

The impeccable journey of Datta and TruEra

Datta says the inception of TruEra can be traced to the time after earning the PhD thesis, he developed a diversified outlook and steered his focus on the topics which had greater impacts on society. "As I started getting deeper into privacy, the realization that the intersection of privacy and machine learning was emerging as an important area with unsolved problems that were going to be impactful took me there", Datta says about his transition from cryptography to privacy.

TrueEra, from the genesis, conversed with many potential product users in different verticals like financial services, healthcare, tech native companies etc. and started the company in early 2019, which marks three and half years of trustworthy service.

Datta’s emphasis on the explainability of AI models

Understanding a model will increase trust in it, especially in scenarios involving life and death situations such as health, credit lending, and law explains why model explainability is required. Only if we understand the model we could detect if it has any bias present in the model. It is significant while debugging a model during its development phase.

Datta says, “after digging into fairness, I started understanding what causes unfairness that requires you to ponder questions of explainability. He finds it as the progression into explainability, and on that basis, we can also raise similar questions on machine learning behaviour about its fairness, model performance, robustness and quality. Datta says, “part of this insight leads me to think how I could get it out in the world and have a direct impact.” As AI adoption in different verticals is consistently increasing, it also impacts the business. According to Datta, being responsible and ensuring a positive impact through these models have been the cornerstone for the past decade. 

What does TruEra aim to solve?

Regarding Artificial Intelligence, Datta says, “quality and trustworthiness matter. They are like the dual sides of the same coin”. Considering the first-generation tools to build and deploy the machine learning models, people focused on getting these large-scale data into a data management system, training models, and finally deploying models.

However, these basics are already established in many places nowadays. Datta says people raise questions about the impact of these AI models on their businesses. He considers people’s worries about whether the AI models that TruEra is building and deploying impact their business positively. Getting business value from AI has become a dominant question. Datta refers to it as a core quality challenge. Datta and the team tend to be responsible for protecting the systems from potential risks during the deployment of AI, as people are getting business value from it. Hence TruEra tries to balance and ensure robustness and guard against the privacy harms of AI systems simultaneously.

Back then, there weren’t systematic tools for testing, debugging, and monitoring software to ensure its quality, but over time, that has changed with the emergence of companies like Harness. These tools do a great job of automating the testing and deployment of software. But, Datta says in the software world, the analogue for that in machine learning is missing, which makes TruEra significant, as they help train good-quality AI models.

Addressing the biased models

Datta says we can’t just blame the tool when it comes to ‘accountability or responsibility’. Ultimately the responsibility lies with people. The model’s bias can be the data bias or the way the model is built. The bias can emerge over time as the data keep changing. According to Datta, organizations should think structurally about assigning responsibility and accountability to individuals within the organizations to execute the processes and ensure good quality.

Datta shares his thought on one of the emerging areas where there is good precedence of this accountability and responsibility: financial services and banking in the US. The regulators created a chain of command for model development which leads up to a senior person in the organization. Still, in parallel, they created an independent group called model risk management groups. There will be a senior person at the head of the model development team, and the second team is charged with providing independent validation of high-risk models. Hence, the credit risk models or fraud risk models are validated independently by a second team, and accountability is assigned across the two teams at a senior leadership level. Because of this regulatory structure, they will ultimately become responsible for ensuring their models’ robustness. Datta considers it an excellent example of how an organizational structure should work.

Need for regulation and frameworks

About the idea of developing an ideal framework and regulation for AI models for each sector, Datta says, “some amount of thoughtful regulation potentially somewhat in a sectoral manner might be the way to go”. He adds that credit, employment, and housing are the three sectors where there has been thoughtful regulation in the US.

The foundation models that act as the building block of various applications will have sectoral boundaries. Datta sees it useful to have regulations for these kinds of foundational technologies. And when it comes to the foundation model like GPT 3 or ChatGPT, they are trained on huge volumes of data from all over the world. Hence initially, bias exists in the data that it’s trained on. This paves the way for careful evaluation to find them, understand their root causes, and work on mitigations.

About the hype on ChatGPT

Datta mentions, with ChatGPT, another paradigm got added. Instead of simple data training, here, human feedback is valued for reinforcement learning. Here the technological advancement helped in a bigger step from GPT3 to ChatGPT, where the model size decreased, but the quality increased tremendously. Datta considers it a milestone. He believes the sequence of this work will improve in the coming days by producing better results. At the same time, Datta also reminds us to be careful about how truthful ChatGPT is. It has toxicity and bias, and sometimes it isn’t robust.

The AI quality and trustworthiness

Datta, while sharing his outlook on the end-to-end life cycle for machine learning and the place for AI quality and trustworthiness in it, he advises that, while you build these models, you should be very careful about rigorously testing them across all of these different kinds of attributes around performance and fairness and robustness, and so on. And then, once the model is in production, you keep on monitoring and tracking all of these metrics that you originally tested during development. But once you are in production, things can change and go beyond control as the world might change, a shift develop in data distribution as the interaction of people with the system changes, bugs in the data pipeline that result in poor or incorrect data and so on. Datta quotes the Microsoft chatbot and says these can make way for bias, toxicity and unfairness.

Anupam Datta concludes by sharing the core focus of TruEra, which is ultimately “quality and trustworthiness”. For that, they must work a lot to ensure that these machine learning models stay truthful to the purpose and deliver value and guard against the social harms that prevail. He adds that their vision here is similar to every machine learning team building and deploying models and maintaining them till the end. They also intend to build and deploy high-quality models and maintain their quality in production.