Automatic Learning and Optimization Laboratory

Laboratory

Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems

Fengyu Cai, Boi Faltings, Lingjing Kong, Fei Mi, Wanhao Zhou

As the labeling cost for different modules in task-oriented dialog (ToD) systems is expensive, a major challenge is to train different modules with the least amount of labeled data. Recently, large-scale pre-trained language models, have shown promising results for few-shot learning in ToD. In this paper, we devise a self-training approach to utilize the abundant unlabeled dialog data to further improve state-of-the-art pre-trained models in few-shot learning scenarios for ToD systems. Specifically, we propose a self-training approach that iteratively labels the most confident unlabeled data to train a stronger Student model. Moreover, a new text augmentation technique (GradAug) is proposed to better train the Student by replacing non-crucial tokens using a masked language model. We conduct extensive experiments and present analyses on four downstream tasks in ToD, including intent classification, dialog state tracking, dialog act prediction, and response selection. Empirical results demonstrate that the proposed self-training approach consistently improves state-of-the-art pre-trained models (BERT, ToD-BERT) when only a small number of labeled data are available.

ASSOC COMPUTATIONAL LINGUISTICS-ACL2021

Masked Training of Neural Networks with Partial Gradients

Martin Jaggi, Amirkeivan Mohtashami, Sebastian Urban Stich

State-of-the-art training algorithms for deep learning models are based on stochastic gradient descent (SGD). Recently, many variations have been explored: perturbing parameters for better accuracy (such as in Extra-gradient), limiting SGD updates to a subset of parameters for increased efficiency (such as meProp) or a combination of both (such as Dropout). However, the convergence of these methods is often not studied in theory. We propose a unified theoretical framework to study such SGD variants-encompassing the aforementioned algorithms and additionally a broad variety of methods used for communication efficient training or model compression. Our insights can be used as a guide to improve the efficiency of such methods and facilitate generalization to new applications. As an example, we tackle the task of jointly training networks, a version of which (limited to sub-networks) is used to create Slimmable Networks. By training a low-rank Transformer jointly with a standard one we obtain superior performance than when it is trained separately.

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Natural Language Processing (NLP) driven categorisation and detection of discourse in historical US patents

Jérôme Baudry, Nicolas Christophe Chachereau, Prakhar Gupta, Bhargav Srinivasa Desikan

Patents have traditionally been used in the history of technology as an indication of the thinking process of the inventors, of the challenges or “reverse salients” they faced, or of the social groups influencing the construction of technology. More recently, historians of science and technology also read them to interpret the way people described technology and how the specific inscriptions of inventions mattered for the justification and operation of the patent system. The digitization of historical patents opens up unique opportunities to assess the feasibility of unsupervised machine learning and natural language methods for such explorations. In this project, we analyze over a million US historical patents from 1830-1930 using a variety of text-based methods, with two major aims: 1) categorizing patents into coherent technical categories, 2) identifying discourses of safety, reflexivity, and environmental concern in technological innovation. We use both frequency-based and context-based methods, and find that a bag of words-based methods such as TDF-IDF and topic modeling do not perform well on semantic categorization due to the linguistic peculiarities of patent specifications. This suggests that a successful approach to categorizing patents would require contextual semantic representations such as Transformers-based methods (e.g. BERT), or static embedding based methods (e.g Word2Vec, Doc2Vec) which have relatively low computational costs but less expressive in some scenarios. We run early experiments using these methods and find that word embedding models are effective in learning semantics from the descriptions of the patents. In this poster, we will describe our early results, as well as exploratory data analysis on this massive historical patents dataset.

2022

Stochastic distributed learning with gradient quantization and double-variance reduction

Konstantin Mishchenko, Sebastian Urban Stich

We consider distributed optimization over several devices, each sending incremental model updates to a central server. This setting is considered, for instance, in federated learning. Various schemes have been designed to compress the model updates in order to reduce the overall communication cost. However, existing methods suffer from a significant slowdown due to additional variance omega > 0 coming from the compression operator and as a result, only converge sublinearly. What is needed is a variance reduction technique for taming the variance introduced by compression. We propose the first methods that achieve linear convergence for arbitrary compression operators. For strongly convex functions with condition number kappa, distributed among n machines with a finite-sum structure, each worker having less than in components, we also (i) give analysis for the weakly convex and the non-convex cases and (ii) verify in experiments that our novel variance reduced schemes are more efficient than the baselines. Moreover, we show theoretically that as the number of devices increases, higher compression levels are possible without this affecting the overall number of communications in comparison with methods that do not perform any compression. This leads to a significant reduction in communication cost. Our general analysis allows to pick the most suitable compression for each problem, finding the rig ht balance between additional variance and communication savings. Finally, we also (iii) give analysis for arbitrary quantized updates.

TAYLOR & FRANCIS LTD2022

Predictive Analysis of Errors During Robot-Mediated Gamified Training

Pierre Dillenbourg, Nevena Dresevic, Arzu Güneysu Özgür, Marija Katanic, Nihal Ezgi Yücetürk

This paper presents our approach to predicting future error-related events in a robot-mediated gamified phys- ical training activity for stroke patients. The ability to predict future error under such conditions suggests the existence of distinguishable features and separated class characteristics between the casual gameplay state and error prune state in the data. Identifying such features provides valuable insight to creating individually tailored, adaptive games as well as possible ways to increase rehabilitation success by patients. Considering the time-series nature of sensory data created by motor actions of patients we employed a predictive analysis strategy on carefully engineered features of sequenced data. We split the data into fixed time windows and explored logistic regression models, decision trees, and recurrent neural networks to predict the likelihood of a patient making an error based on the features from the time window before the error. We achieved an 84.4% F1-score with a 0.76 ROC value in our best model for predicting motion accuracy related errors. Moreover, we computed the permutation importance of the features to explain which ones are more indicative of future errors.

2022

Genomic Diversity of Torque Teno Virus in Blood Samples from Febrile Paediatric Outpatients in Tanzania: A Descriptive Cohort Study

Mary-Anne Hartley

Torque teno virus (TTV) is considered to be an ubiquitous member of the commensal human blood virome commonly reported in mixed genotype co-infections. This study investigates the genomic diversity of TTV in blood samples from 816 febrile Tanzanian children. Metagenomic next-generation sequencing was used to screen for TTV in individual blood samples from a cohort of 816 febrile Tanzanian paediatric outpatients. For positive samples, the number of TTV species and genotypes present were evaluated. We investigate the linear relationship between individual TTV diversity and the patient age by linear regression. TTV was detected in 97.2% of sera. ORF1 analysis revealed the presence of 149 genotypes from 38 species, suggesting the presence of 13 new species. These genotypes were mostly present as co-infections with a median of 11 genotypes/subject (range: 1-71). In terms of species, we found a median of nine species/subject (range: 1-29). We further show a significant association between the diversity of co-detected TTV and the age of the subjects (p value < 0.0001). This study shows that significant TTV genomic diversity is acquired by the age of five and that this diversity tends to increase with age, which indicates a repetitive TTV acquisition during the first months/years of life.

MDPI2022

MoDN-Flip: Optimizing Modular Clinical Decision Support Networks with Decision Flippability Score

Kuan Tung

BACKGROUND. Clinical Decision Support Systems (CDSS) are point-of-care questionnaires that guide consultations to evidence-based predictions with the aim of standardizing care and optimizing resource allocation. The data collected in these tools contain potentially useful insights to further train its predictive algorithms, however, the questionnaire structure biases data with systematic missingness, making it unusable in traditional models. Further, privacy and interoperability issues make it challenging to synergize with data collected in related CDSS tools. The Modular Decision Support Network (MoDN) was previously proposed by our group to address these issues. It can predict diagnoses at each step in a consultation using sequentially encoded questions. However, it lacks the ability to provide information on unencoded questions, such as predicting their importance, which could potentially optimize questionnaire structure. AIM. In this work, we aim to optimize MoDN by deriving a flippability score (FS) which computes the importance of a question in the form of how likely its answer would flip (change) the predicted diagnoses. METHODS & FINDINGS. The FS is computed with the help of MoDN’s feature decoders that predict unknown feature values in the data. The optimal questionnaire order is then derived by feeding MoDN the question with the highest FS. This approach—-MoDN-flip—- is validated on two independent real world CDSS-derived data sets of pediatric outpatients. We show that FS-optimized question order can achieve the baseline predictive performance 14.71% faster (or with 3.53 fewer questions) compared to MoDN with the original question order. FS-optimized questionnaires are also able to improve the cumulative F1 score by 5.24%. Furthermore, we test the utility of FS-optimization when porting MoDN to an imperfectly interoperable data set (i.e. with slightly different questionnaire structures and partially-overlapping feature sets). The results show that the score is valid in a new dataset and can provide better predictive performance at an earlier stages, albeit a slight cost to final performance. CONCLUSION. With the aim of providing data-driven questionnaire optimization for CDSS tools, we propose the flippability score, which indicates the importance of each potential question in a consultation. The experimental results show that FS can express predictive importance of questions and has the potential to significantly optimize questionnaire structure. It is now necessary to evaluate this ordering against human consultation logic.

2022

Determining crack kinematics from imaged crack patterns

Katrin Beyer, Ketson Roberto Maximiano Dos Santos, Bryan German Pantoja Rosero, Amir Rezaie

Determining the relationship between the cause of damage and the subsequent structural behavior of infrastructure systems requires an accurate characterization of the propagation of cracks, which represents the evolution of the damage state. When no information about the cause of damage is available, kinematic approaches can be used to describe the motion of crack contours. Current image-based approaches to derive crack kinematics use digital image correlation (DIC) on a set of sequential images as the crack propagates. However, DIC is invasive in that the structure surfaces must be painted with random speckle patterns, limiting its use primarily to controlled experiments. In this paper, we propose a novel image-based methodology for computing crack opening in Mode I or Mode II. As an input, this method takes a binary image from a semantic segmentation of an image of a crack pattern. This binary image is used to detect the opposite edges along the crack, which are then registered using an optimization algorithm based on the Euclidean transformation model and non-linear least squares. As a final output, this method produces displacement maps in the tangential and normal directions to the crack skeleton. To demonstrate its performance, we validate our methodology first with synthetic crack patterns and then with real crack patterns. Because this methodology for determining crack openings requires only simple data (just a binary crack pattern image), it is straightforward, robust, and adaptable, thus contributing to the development of structural image-based damage assessments. The computational codes and datasets are available to the public for future research and benchmarking on https://github.com/eesd-epfl/crack_kinematics and https://doi.org/10.5281/zenodo.6632071.

ELSEVIER SCI LTD2022

Point-of-care lung ultrasonography for early identification of mild COVID-19: a prospective cohort of outpatients in a Swiss screening center

Luca Bosso, Jean-Baptiste Francis Marie Juliette Cordonnier, Tanguy Espejo, Mary-Anne Hartley, Jean-Yves Meuwly, Siméon Schaad

Objectives Early identification of SARS-CoV-2 infection is important to guide quarantine and reduce transmission. This study evaluates the diagnostic performance of lung ultrasound (LUS), an affordable, consumable-free point-of-care tool, for COVID-19 screening. Design, setting and participants This prospective observational cohort included adults presenting with cough and/or dyspnoea at a SARS-CoV-2 screening centre of Lausanne University Hospital between 31 March and 8 May 2020. Interventions Investigators recorded standardised LUS images and videos in 10 lung zones per patient. Two blinded independent experts reviewed LUS recording and classified abnormal findings according to prespecified criteria to investigate their predictive value to diagnose SARS-CoV-2 infection according to PCR on nasopharyngeal swabs (COVID-19 positive vs COVID-19 negative). Primary and secondary outcome measures We finally combined LUS and clinical findings to derive a multivariate logistic regression diagnostic score. Results Of 134 included patients, 23% (n=30/134) were COVID-19 positive and 77% (n=103/134) were COVID-19 negative; 85%, (n=114/134) cases were previously healthy healthcare workers presenting within 2-5 days of symptom onset (IQR). Abnormal LUS findings were significantly more frequent in COVID-19 positive compared with COVID-19 negative (45% vs 26%, p=0.045) and mostly consisted of focal pathologic B lines. Combining clinical findings in a multivariate logistic regression score had an area under the receiver operating curve of 80.3% to detect COVID-19, and slightly improved to 84.5% with the addition of LUS features. Conclusions COVID-19-positive patients are significantly more likely to have lung pathology by LUS. However, LUS has an insufficient sensitivity and is not an appropriate screening tool in outpatients. LUS only adds little value to clinical features alone.

BMJ PUBLISHING GROUP2022