Microsoft's BioGPT is a generative pre-trained Transformer language model in the biomedical domain. We can use it to make and mine text from life science literature.

When applying BioGPT to tasks further down the line, the researchers looked at the design of prompts and target sequences. They found that target sequences with natural language semantics are better than structured prompts.

The researchers made the prompt and the target sequence format and tested them while applying pre-trained BioGPT to downstream tasks based on GPT-2 and pre-trained on 15 million PubMed abstracts corpus. It does better on most of the six biomedical NLP tasks that it tests than earlier models. We can use the trained model for biomedical NLP tasks like answering questions, classifying documents, making up text, and extracting end-to-end relationships.

BioGPT gets SOTA on one question-answering task and three end-to-end relation extraction tasks. It also does better than GPT-2 on the text generation task when writing about life sciences. It sets a new record with F1 scores of 44.98 per cent on the BC5CDR, 38.42 per cent on the KD-DTI, and 40.76 per cent on the DDI end-to-end relation extraction tasks, and 78.2 per cent accuracy on the PubMedQA task.

In this work, the researchers developed BioGPT, a generative pre-trained Transformer language model for creating and mining biomedical text. The researchers used GPT-2 as their primary model and trained it on 15 million PubMed abstracts before using it in the real world. When they put pre-trained BioGPT to work on downstream tasks, they carefully planned and tested the prompt and the target sequence format. The researchers used the BioGPT that had already been trained to do biomedical NLP tasks like end-to-end relation extraction, answering questions, classifying documents, and making new text.

Furthermore, BioGPT gets SOTA results on three tasks for extracting end-to-end relationships and one task for answering questions. On the text generation task, it also does a better job than GPT-2 at making biomedical texts. For future work, the researchers want to train BioGPT on more extensive sets of biomedical data and use it for more tasks further down the line.