However, instead of simply masking random sentences, the training algorithm uses a heuristic to choose the most important sentences in the text to mask, which improves the model's performance on summarization. PEGASUS extends this objective by masking entire sentences. Because these models need large amounts of training data, most use pre-trained Transformers as components typically the pre-training objective is a masked language model (MLM), where random tokens in the input are masked, and the model must predict the correct value for the masked tokens. We hope the automatic suggestions now offered in Google Workspace make it easier for writers to annotate their documents with summaries, and help readers comprehend and navigate documents more easily.Ī common technique for NLP tasks, including abstractive text summarization, is to train a sequence-to-sequence model, which takes as input a sequence of tokens (for example, letters or even whole words), feeds the input sequence into an encoder that produces a latent representation of the sequence, then uses a decoder to convert that latent representation to an output sequence. The current model used by Google Docs features several improvements on PEGASUS, including fine-tuning on a high-quality dataset and knowledge distillation to improve latency and reduce memory footprint. PEGASUS uses a pre-training scheme called Gap Sentence Prediction (GSP), which teaches the model to re-generate full sentences that have been masked from input text in particular, the masked sentences are chosen based on how important they are for generating a summary of the text. The model is based on PEGASUS, an NLP system for abstractive text summarization developed by the Brain Team. The model was described in a blog post written by Mohammad Saleh, a software engineer from Google Research's Brain Team, and Anjuli Kannan, a software engineer from Google Docs. The summarization is powered by a natural language processing (NLP) AI model based on the Transformer architecture. Google has announced a new feature for their Docs app that will automatically generate a summary of the document content.
0 Comments
Leave a Reply. |