Publications by Mohammadreza Banaei

Parameter-efficient Methods for Training and Inference of Transformer-based Models

Recent advancements in natural language processing, driven by the success of large language models, have led to the scaling up of models to unprecedented sizes. While these models have achieved remarkable performance across various tasks, their computation ...

EPFL2025

Breaking the Language Barrier: Improving Cross-Lingual Reasoning with Structured Self-Attention

Karl Aberer, Antoine Bosselut, Negar Foroutan Eghlidi, Mohammadreza Banaei

In this work, we study whether multilingual language models (MultiLMs) can transfer logical reasoning abilities to other languages when they are fine-tuned for reasoning in a different language. We evaluate the cross-lingual reasoning abilities of MultiLMs ...

Association for Computational Linguistics2023

Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Models

Karl Aberer, Rémi Philippe Lebret, Mohammadreza Banaei

Recent transformer language models achieve outstanding results in many natural language processing (NLP) tasks. However, their enormous size often makes them impractical on memory-constrained devices, requiring practitioners to compress them to smaller net ...

Assoc Computational Linguistics-Acl2023

Discovering Language-neutral Sub-networks in Multilingual Language Models

Karl Aberer, Rémi Philippe Lebret, Antoine Bosselut, Negar Foroutan Eghlidi, Mohammadreza Banaei

Multilingual pre-trained language models transfer remarkably well on cross-lingual downstream tasks. However, the extent to which they learn language-neutral representations (i.e., shared representations that encode similar phenomena across languages), and ...

Association for Computational Linguistics2022

Direction is what you need: Improving Word Embedding Compression in Large Language Models

Karl Aberer, Rémi Philippe Lebret, Mohammadreza Banaei

The adoption of Transformer-based models in natural language processing (NLP) has led to great success using a massive number of parameters. However, due to deployment constraints in edge devices, there has been a rising interest in the compression of thes ...

ASSOC COMPUTATIONAL LINGUISTICS-ACL2021