Publication
Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages
Related publications (35)
Direction is what you need: Improving Word Embedding Compression in Large Language Models
Karl Aberer, Rémi Philippe Lebret, Mohammadreza Banaei
The adoption of Transformer-based models in natural language processing (NLP) has led to great success using a massive number of parameters. However, due to deployment constraints in edge devices, there has been a rising interest in the compression of thes ...
ASSOC COMPUTATIONAL LINGUISTICS-ACL2021Fidelity Estimation Improves Noisy-Image Classification with Pretrained Networks
Sabine Süsstrunk, Majed El Helou, Deblina Bhattacharjee, Xiaoyu Lin
Image classification has significantly improved using deep learning. This is mainly due to convolutional neural networks (CNNs) that are capable of learning rich feature extractors from large datasets. However, most deep learning classification methods are ...
2021On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines
Fine-tuning pre-trained transformer-based language models such as BERT has become a common practice dominating leaderboards across various NLP benchmarks. Despite the strong empirical performance of fine-tuned models, fine-tuning is an unstable process: tr ...
2021Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems
Boi Faltings, Fei Mi, Lingjing Kong, Fengyu Cai, Wanhao Zhou
As the labeling cost for different modules in task-oriented dialog (ToD) systems is expensive, a major challenge is to train different modules with the least amount of labeled data. Recently, large-scale pre-trained language models, have shown promising re ...
ASSOC COMPUTATIONAL LINGUISTICS-ACL2021The Unstoppable Rise of Computational Linguistics in Deep Learning
In this paper, we trace the history of neural networks applied to natural language understanding tasks, and identify key contributions which the nature of language has made to the development of neural network architectures. We focus on the importance of v ...
Association for Computational Linguistics2020Learning Entailment-Based Sentence Embeddings from Natural Language Inference
James Henderson, Rabeeh Karimi Mahabadi, Jan Frederik Jonas Florian Mai
Large datasets on natural language inference are a potentially valuable resource for inducing semantic representations of natural language sentences. But in many such models the embeddings computed by the sentence encoder goes through an MLP-based interact ...
Idiap2019Overcoming Multi-model Forgetting
Mathieu Salzmann, Anthony Christopher Davison, Martin Jaggi, Yassine Benyahia, Claudiu-Cristian Musat, Kaicheng Yu
We identify a phenomenon, which we refer to as multi-model forgetting, that occurs when sequentially training multiple deep networks with partially-shared parameters; the performance of previously-trained models degrades as one optimizes a subsequent one, ...
JMLR2019EMPIRICAL EVALUATION AND COMBINATION OF PUNCTUATION PREDICTION MODELS APPLIED TO BROADCAST NEWS
Natural language processing techniques are dependent upon punctuation to work well. When their input is taken from speech recognition, it is necessary to reconstruct the punctuation; in particular sentence boundaries. We define a range of features from low ...
2019EMPIRICAL EVALUATION AND COMBINATION OF PUNCTUATION PREDICTION MODELS APPLIED TO BROADCAST NEWS
Natural language processing techniques are dependent upon punctuation to work well. When their input is taken from speech recognition, it is necessary to reconstruct the punctuation; in particular sentence boundaries. We define a range of features from low ...
Idiap2019How to Tell Ancient Signs Apart? Recognizing and Visualizing Maya Glyphs with CNNs
Daniel Gatica-Perez, Jean-Marc Odobez, Gülcan Can
Thanks to the digital preservation of cultural heritage materials, multimedia tools (e.g., based on automatic visual processing) considerably ease the work of scholars in the humanities and help them to perform quantitative analysis of their data. In this ...
2018