We always refer to the digital divide and what can be done to address it. Unsurprisingly, it isn’t only about not having access to devices, bandwidth, and relevant skills, but also about the language divide. Last year at RAISE 2020, one of the sessions delved into this aspect and how technology can be levered to bridge it.

In 2019, the top 10 million internet sites were used for research, and the findings indicated that 54.4% of the web content was in English. By now, it may have inched closer to 60%. There are about 7K major languages that are “alive” today out of which only about a hundred are supported by technology. Even then, only about 15 are meaningfully supported and have relevant web content. The disparity is obvious – the ones who are proficient in one/many of those 15 languages (it can be stretched to 50, perhaps) and others who are part of the 6850-odd “insignificant” cohort who cannot communicate in those 15 - 20 languages. Incidentally, Hindi with the highest rank amongst Indian languages is at #55. 

It sounds archaic but should Macauly’s children continue to wield disproportionate power over others? 

The archetypal bottom of the pyramid can be said to comprise humans who earn less than 5 dollars a day, and there are approximately 3 billion of them, globally. A significant number, about 1 billion, are illiterate. They can only follow audio-visual clues in their native languages. In a country of India’s size, the semi-literate population (number) could be as high as 400 million. At 5 dollars a day, it’s about 2 billion dollars (daily) and ~700 billion dollars in a year. That’s close to a trillion dollars: too big an empowerment opportunity to be shelved.

Can we use language-enabled conversational AI (NLP-based) that empowers this vast multitude so that humans can communicate in any language in real-time, be understood by someone who doesn’t speak the same language, thereby getting a better shot at being a part of the mainstream? Someone speaks in Telugu and the hand-held device instantly translates the speech into Hindi and a seamless conversation is made possible between two people who speak and understand two different languages. Another study by linguists revealed that all Indian languages have 70% commonality in structure, etc. In theory, if it’s possible for a few, it can be done for all Indian languages. Most Indians are bilingual or even trilingual anyway.    

Siri, Alexa are prominent examples of how this technology works – the voice-enabled AI – but of course with limited applications only. And, largely in English. For a machine to understand the nuances of any language even at a basic level, it requires `100k words of speech and in diverse contexts with syntax, intonation, pause, and everything else that are required to bring in the desired level of clarity and understanding. When we speak into the device, we should not have to say, “Alexa please say good morning to my friend” or something equally silly. It has to be like a normal conversation where we don’t “type or tap” at each other. We aren’t there yet but the technology is developing fast enough.

The challenge is about getting access to data (annotated, labeled) in Indian languages that can be used to make machines learn. Who will do it? And, we need domain-specific data which can be in the form of applications sitting atop the voice-enabled platform. We have seen some good and not-so-good translations of Wikipedia pages. But, we cannot wait for the technology to perfect itself and neither for the quality of data to improve, so we have to make do with what we have.        

Firstly on design, there are certain prerequisites (tech aspect):

•  It has to be non-intrusive.
•  Always-on & always working mode.
• Device-independent.  
• Technology should be able to self-learn 24/7 and perfect itself over time. 

There are caveats too:

•  Normal conversation happens with background noise and the technology has to distinguish that.
• Anonymizing, Annotating, and labeling voluminous data require massive computing power & resources and this exercise cannot be repeated many times over. 
•  Algorithmic biases will creep in too and will have to be addressed. 
•  The privacy & data security angle will soon hit us – are we comfortable with machines listening to our conversations all the time?  

The data collection part is tricky and has to be partnered by the government, industry, and academia alike. There were several suggestions made at RAISE 2020. For example, in India, there are 600 million Whatsapp users – can we get access to one minute's worth of data for every user, of course, by not infringing on user rights and privacy? Or, what if we have a volunteer-based model designed for school students so that they can translate Wikipedia pages into Indian languages and get a token of recognition? We can also have a paid model for professionals who translate pages to do with higher learning. A continuous feedback loop will be required. 

It is estimated that AI-enabled NLP-based systems that enable many languages can add value that is 10% of the global GDP i.e. a 10 trillion dollar opportunity. In India, we would need to look at 22 languages and obviously, this is a mammoth-size exercise so we have to prioritize the areas that need to be covered. 

When language ceases to be a barrier, straightaway, we empower millions of people who will then have access to better education, healthcare, infra support, and the judiciary, to start with. We must bear in mind that these models will not be 100% accurate right away but we should be able to identify areas in which 90 – 95% of accuracy & sensitivity is acceptable. 

Technology doesn’t perfect itself overnight but we must be prepared to use what is already available to drive significant outcomes and drive greater empowerment for the less fortunate. 

Image by Willi Heidelbach from Pixabay