Imagine this: you’ve just landed on a popular website (or app) to book your vacation flights. As you browse, you notice a “smart travel assistant”—an AI chatbot that promises a seamless, one-click experience. By simply entering your personal details and travel plans, the website automatically suggests the best travel and accommodation package at the lowest fare. All that’s left is to hit the payment button, and your booking is complete.

Need of AI in consumer services

In recent years, the customer service sector has faced significant challenges in meeting the growing demand for timely and personalized service .

As the volume of customer service requests has surged, the industry has grappled with a shortage of skilled labor capable of making quick, informed decisions while delivering increasingly tailored customer experiences. This shortage is particularly pronounced in environments where customer interactions require both speed and a high degree of personalization. Many service professionals lack the requisite training and expertise needed to effectively navigate complex queries, leaving organizations struggling to meet consumer expectations. The pressing need for qualified personnel has become a critical issue, hindering the industry's ability to scale operations while maintaining service quality. Addressing this gap will require targeted investment in workforce development, including comprehensive training programs aimed at equipping customer service professionals with the skills necessary to adapt to the fast-paced, evolving nature of modern customer interactions. 

It is here that the utility and efficiency in the use of AI for personalisation in consumer experience comes to the fore. AI chatbots serving as customer service channels can save cost and time, consolidate information at one place and make quick decisions. It allows for 24x7 customer support even for small businesses. Use of AI would also allow companies to leverage proprietary customer data for low cost hyper personalisation for its customers. This is one of the most important use cases of digital AI agents as every business has a customer facing interface.  

Intelligent but not wise!

Over-reliance on AI and dehumanization of the customer service process comes as a risk. In our travel booking example, consider the scenario where the chatbot made a mistake, your flight was not booked at the cheapest fare. 

Would you trust this AI-driven service in the first instance? How is this different from using the “filter by preference” option on the website? What would you do when the chatbot made a mistake? These are the kinds of questions that arise as AI becomes increasingly integrated into consumer-facing interactions. These concerns have not missed the eyes of the regulators. 

The risk of breach of customer trust can extend to exacerbation of societal inequalities and promotion of bias; while the organizations can suffer from loss of customer loyalty, reputational damage, potential legal & regulatory penalties, and financial losses. Further, consumers may seek more responsible and trustworthy alternatives.

As the adoption of AI-powered digital agents increases, it is essential for AI system developers to carefully examine instances where customer-facing algorithms negatively impacted a company's reputation. Following examples illustrate that AI integration in customer experience is not limited to chatbots. They also show how the improper integration of autonomous customer interactions into business operations can lead to significant challenges, including reputational damage and legal issues.

The first example involves Home Depot, which used Google's AI system in its customer service workflow. Customers were allegedly unaware that their conversations were being recorded and analyzed by the AI without consent. This led to a class-action lawsuit claiming that the company violated privacy laws. The lawsuit highlighted the risks of improper implementation of AI in customer service, particularly when transparency about data usage is lacking.

Microsoft’s AI chatbot, Tay, was launched on Twitter to interact with users and learn from those conversations. Tay was designed as a fun, conversational chatbot targeting younger audiences. However, within just 24 hours of its release, the chatbot began posting offensive, racist, and inflammatory remarks. This behavior was the result of Tay being manipulated by Twitter users who fed it inappropriate and harmful content. The chatbot’s machine learning algorithm allowed it to learn from the conversations it was exposed to, and it began to mimic and reproduce the hateful language it encountered. As a result, Tay started tweeting offensive statements, which led to widespread backlash. Microsoft quickly shut down Tay and issued an apology (see here), explaining that the bot had been compromised by malicious users. This controversy highlighted the vulnerabilities of AI systems that rely on open, unsupervised learning and underscored the importance of safeguards when deploying AI in public spaces.

Consumer Trust As Public Asset 

In Spite of these widely reported incidents, a recent research highlighted that consumers are optimistic about AI which means that the companies can capitalize on this trust to come up with innovative offerings (see here) and this consumer trust (a public resource) cannot be squandered at the hands of a product gone wrong (private action). This thought process has encouraged governments across the world to step in and formally recognise the risk of erosion of consumer trust in AI enabled services and set responsibility for any wrongful conduct.  

‘Indifference of organizations to the societal impact of AI leading to erosion of consumer trust’ is one of the risk factors identified in the IndiaAI Risk Identification and Assessment Tool, cataloged as risk number 41 in the ‘societal’ risk category. This risk generally refers to the phenomenon where companies, in their pursuit of profit and innovation, actively neglect, or fail to consider the broader ethical, social, and human consequences of their AI technologies. This disregard can manifest in the deployment of biased algorithms, invasion of privacy, or the promotion of harmful content, leading to public backlash and a loss of trust in the organization.

When consumers perceive that companies are not prioritizing their well-being or societal values, they become more skeptical and wary of engaging with those companies' products or services, ultimately damaging the company's reputation and long-term success. In the worst case scenario, this can even lead to a general consumer distrust in technological solutions. The Government of India has identified protection of consumer trust as one of the cornerstones of India’s evolving AI policy. It has suggested that India's approach to regulate AI systems would be a risk-based approach to encourage innovation and safeguard the consumer and citizen interests.

Conclusion 

As AI continues to weave itself into the fabric of consumer services, striking a balance between leveraging its transformative capabilities and safeguarding consumer trust becomes paramount. The rapid advancements in AI technology present unprecedented opportunities for personalization and efficiency, as evidenced by its integration into various business sectors.

Expediagroup.com, McKinsey.com, BBC.com, Skadden.com, Microsoft.com, nytimes.com, theguardian.com, technologyreview.com, kpmg.com