What is best model for summarization? Which model should you use for a particular type of summarization? What if your computational resources are limited? These are just a few considerations that might come to mind when deciding on a model for a particular functionality. There are so many models out there that it can be overwhelming and every model has certain strengths and weaknesses. The below elaborate on certain areas to think about when choosing the right model for summarization.
- Type of Summarization: Is this for extractive (picking existing sentences) or abstractive (generating new sentences) summaries?
- Document Length: Are summaries from short text, long articles, or entire books?
- Computational Resources: Do you need a model that is fast and efficient? Can you afford to run a large model or a complex model? Are you comfortable with spending time and resources for fine-tuning?
- Specific Tasks: Is it general summaries or more specific? Are these over dense or sparse documents?
Models with High Quality Summarizations:
- BigBirdPegasus: Designed for abstract summarization for long documents, combination of BigBird attention and Pegasus pre-training, good for concise and informative summaries. But, is computationally expensive.
- LongT5: Designed to handle long sequences, good for summarization of lengthy texts with strong performance and very versatile. But, not as specialized for summarization.
- Pegasus: Powerful for summarization. But, not explicitly designed for long documents. Might struggle with very long documents.
- GPT3/GPT4: Produces human-like summaries as a generation task. But, computationally expensive. Requires careful prompt engineering.
- Mistral/Llama: Flexible models allowing for greater degree of fine-tuning for summarization. But, this requires additional engineering effort.
- Longformer: Super flexible for long document processing in understanding and classifying. But, not specifically designed for summarization task.
- Gemini: Good contextual summaries with generally high performance. But, further evaluations are necessary.
- Bart: Good for abstractive summarization using denoising autoencoder for text generation.
- T5: Versatile model that can be fine-tuned for various tasks.
Task Considerations:
- Abstractive Long Documents: BigBirdPegasus, LongT5, GPT3/4, Gemini, Mistral, and Llama
- Abstractive Short Documents: Pegasus, Bart, T5, Gemini, GPT3/4, Mistral, and Llama
- Computational Resources: Higher computational cost requirements with BigBirdPegasus, Gemini, and GPT3/4 models
- Fine-Tuning: Most of these models can be fine-tuned
- Top Performers: BigBirdPegasus and LongT5 for abstractive summaries over long documents with greatest flexibility on summarization focus and quality.
- Multimodal Summaries: Better with Gemini, GPT 4, and Llama. But, support may be limited as this functionality is actively under development. The assumption of a lot of the models is summarization is over text only. Likely would need to create your own. Approaches could vary between multimodal transformers, hierarchical multimodal models, or graph-based models.
- Extractive Summaries: TextRank, LexRank and other types of methods
- Very Limited Resources: DistilBert
Summarization Model Selection Steps:
- Start simple
- Iteratively experiment
- Fine-tune on dataset relevant to task
- Iteratively evaluate (ROUGE, BLEU, among others) + human evaluation
- Rinse and repeat
