Publication
Beyond Transformers: Assessing Next-Generation Neural Architectures for Multimodal Sequence Modeling
| Summary: | The Transformer has been the dominant architecture for sequence modelling for almost a decade, but its self-attention mechanism compares every position with every other, so its cost grows rapidly with sequence length and its inference memory grows with the length of the history. These costs are especially burdensome in multimodal settings, where text, audio, video, and sensor streams are combined into very long sequences. A new generation of architectures, including structured state space models, linear and retentive attention, long convolutions, modernised recurrent networks, and hybrids that mix several of these with a few attention layers, claims to remove the bottleneck. This paper provides a systematic assessment of these families under a controlled protocol. Six architectures and two hybrid designs were trained with matched parameter budgets and identical data on five multimodal tasks: video-text retrieval, audio-visual speech recognition, long-document question answering with figures, wearable sensor and text fusion, and long-horizon multimodal....... |
|---|---|
| Main Authors: | Rambabu Nalluri |
| Subject: | Transformers, State Space Models, Linear Attention, Recurrent Networks, Hybrid Architectures, Multimodal Learning, Long-Context Modeling, Efficient Inference |
| Year: | 2025 |
| Country: | Portugal |
| Document type: | article |
| Access type: | unknown |
| Associated institution: | Street Art & Urban Creativity Scientific Journal |
| Language: | English |
| Origin: | Street Art & Urban Creativity Scientific Journal |
| Summary: | The Transformer has been the dominant architecture for sequence modelling for almost a decade, but its self-attention mechanism compares every position with every other, so its cost grows rapidly with sequence length and its inference memory grows with the length of the history. These costs are especially burdensome in multimodal settings, where text, audio, video, and sensor streams are combined into very long sequences. A new generation of architectures, including structured state space models, linear and retentive attention, long convolutions, modernised recurrent networks, and hybrids that mix several of these with a few attention layers, claims to remove the bottleneck. This paper provides a systematic assessment of these families under a controlled protocol. Six architectures and two hybrid designs were trained with matched parameter budgets and identical data on five multimodal tasks: video-text retrieval, audio-visual speech recognition, long-document question answering with figures, wearable sensor and text fusion, and long-horizon multimodal....... |
|---|
Funded activities
No funded projects found.