In just the three weeks since the beginning of the year, some noteworthy research advancements in language technology have emerged, including a model to build new vaccines for the latest variant of the coronavirus.

Here are the top advancements in Natural Language Processing research so far in 2021.

• Framework for robustness testing of NLP models developed: Robustness Gym is a simple and extensible toolkit for robustness testing of NLP models that supports the entire spectrum of evaluation methodologies, from adversarial attacks to rule-based data augmentations. Developed jointly by researchers from Stanford, Salesforce and UNC Chapel Hill, this advance addresses the challenges in evaluating machine learning models today. Robustness Gym can be used to conduct new research analyses with ease. To validate this, the researchers have conducted the first study of academic and commercially available named entity linking (NEL) systems, as well as a study of the fine-grained performance of summarisation models. Robustness Gym supports a broad set of evaluation idioms and can be used for collaboratively building and sharing evaluations and results. A promising tool for researchers and practitioners, it has been embedded into the Contemplate → Create → Consolidate continual evaluation loop. Read more...
• DALL.E - A neural network that creates images from text: OpenAI's new neural network combines NLP with image recognition to give AI a better understanding of concepts in natural language. DALL·E, a pun on the Pixar movie WALL.E and the surreal artist Salvador Dali, is a newer version of GPT-3 which combines textual understanding with the ability to generate images. The smaller version of GPT-3 has 12-billion parameters to generate images from text descriptions by accessing a dataset of the combined text-image descriptions. It has a diverse set of capabilities, including creating anthropomorphised versions of animals and objects, combining unrelated concepts in plausible ways, rendering text, and applying transformations to existing images. DALL·E considers text and images as one stream of data which allows DALL·E to create an image based on a caption and recreate an image that is in line with the caption. In addition, DALL·E can create images based on specifications based on various angles, zoomed, x-rayed, in 2D and 3D, optical distortions, etc. Read more...
• Google trains an industry first trillion-parameter AI model: Google researchers have developed techniques that enable them to train a language model containing more than a trillion parameters. Their 1.6 trillion parameter model – supposedly the largest to date – has achieved upto 4 times speed over the previously developed language model (T5-XXL). This model is about six times bigger than OpenAI's GPT-3, which uses about 175 billion parameters. Large scale training is the effective path towards powerful models – simpler architectures with large datasets and parameter counts are far superior to complex algorithms. This led to researchers pursuing the Switch Transformer, which builds on a mix of experts, keeping them specialized in different tasks within a gating network and choose experts for a given set of data. In future work, the researchers plan to apply the Switch Transformer to “new and across different modalities,” including image and text. They believe that model sparsity can confer advantages in a range of different media, as well as multimodal models. Read more...
• Language-processing AI to track viral escape: MIT researchers have used NLP model to identify viral protein sequences that could be targeted by vaccines. Since viruses mutate rapidly, they continue to confuse and elude the antibodies generated by vaccines; this process is called 'viral escape'. To understand and predict these 'escapes', MIT researchers have devised a computational model based on Natural Language Processing (NLP) models that were originally created to decrypt and predict languages. The model can predict the sections of viral surface which have the most likely to mutate, thus enabling a 'viral escape'. Luckily, it also predicts the sections that are less likely to mutate which can be helpful to create new vaccines. To apply a language model to understanding biological information such as genetic sequences, the researchers drew a few parallels between the two subjects - to be able to maintain the sequence which changing the protein structures. Grammar became analogous to rules that decide whether a protein encoded by a sequence functional or not; the semantics became analogous with understanding whether a protein can change shapes to evade antibodies. The study identified possible targets for vaccines towards influenza, HIV, and SARS-CoV-2. The researchers have applied their model to study the new variants of the SARS-CoV-2 that have recently surfaced in the United Kingdom and South Africa. Read more...

Image by Anas Alshanti on Unsplash