A. Filippakopoulos, N. Xiros, E. Kallioras, et al. · Interspeech 2026 (accepted)
Multimodal language models combine text, audio, and visual streams, and most architectures treat fusion as a single design decision. This work argues that interaction topology (which modalities are allowed to interact, in what order, and at what depth) is an independent design axis that deserves to be studied on its own rather than folded into the choice of fusion operator.
We frame the space as three stages. Segregate keeps each modality's representation intact so nothing is collapsed early, refine processes each stream before it is mixed, and integrate combines them once each carries a well formed representation. Varying the topology while holding capacity and training data fixed isolates its effect from the usual confounds.
The design is evaluated on standard multimodal benchmarks, with the clearest gains on samples where cross modal cues are complementary rather than redundant.
Tools & methods
Multimodal LearningInteraction TopologyDeep FusionPyTorchTransformers