Research Directions for Self-Attention Efficiency

• Low-Rank Kernels
• Fixed/Factorized Random Patterns
• Learnable Patterns
• Memory
• Recurrence