Bias at a Second Glance: A Deep Dive into Bias for German Educational Peer-Review Data Modeling

Natural Language Processing (NLP) has become increasingly utilized to provide adaptivity in educational applications. However, recent research has highlighted a variety of biases in pre-trained language models. While existing studies investigate bias in different domains, they are limited in addressing fine-grained analysis on educational and multilingual corpora. In this work, we analyze bias across text and through multiple architectures on a corpus of 9,165 German peer-reviews collected from university students over five years. Notably, our corpus includes labels such as helpfulness, quality, and critical aspect ratings from the peer-review recipient as well as demographic attributes. We conduct a Word Embedding Association Test (WEAT) analysis on (1) our collected corpus in connection with the clustered labels, (2) the most common pre-trained German language models (T5, BERT, and GPT-2) and GloVe embeddings, and (3) the language models after fine-tuning on our collected data-set. In contrast to our initial expectations, we found that our collected corpus does not reveal many biases in the co-occurrence analysis or in the GloVe embeddings. However, the pre-trained German language models find substantial conceptual, racial, and gender bias and have significant changes in bias across conceptual and racial axes during fine-tuning on the peer-review data. With our research, we aim to contribute to the fourth UN sustainability goal (quality education) with a novel dataset, an understanding of biases in natural language education data, and the potential harms of not counteracting biases in language models for educational tasks.

Cognitive language engineering towards robust human-computer interaction

Vincenzo Pallotta

Intelligent information processing seems to be one of the most challenging task among those involved in human-computer interaction. A central issue is how to model the various types of interaction among artificial and natural entities at different levels of abstraction. On the one hand, models of interaction are required to better understand the communication phenomena. On the other, suitable languages and paradigms should provide powerful frameworks for developing computer-based applications. In this dissertation I focus on different aspects of the second problem, trying to develop a methodology for the design of interactive natural language applications (e.g. from question-answering to mixed-initiative dialogue). One of the main aspects I am concerned with in this work is the problem of their robustness. Several methods have been proposed for achieving robustness in natural language understanding, but these methods are sometimes hard to scale up or re-use in different applications. Moreover, they often concentrate on a single linguistic level of the processing rather than offering a global solution. I will set up a Language Engineering environment whose goal is to combine software engineering and cognitive aspects (e.g. aspects related to representation of a mental model of the speaker). Given its complexity, it is apparent that the problem can be solved only partially. I want to stress here that the main contribution of my work is a holistic perspective on the problem of natural language understanding. Rather than focusing on a particular aspect of natural language processing, I tried to benefit from the big amount of work that has been already done in Computational Linguistics and Computer Science merging different ideas and techniques. In the first part of the dissertation I explore the universe of Language Engineering in order to clarify how my contribution can be situated. After a survey on the state of the art on robust methods in analysis of natural language data, I focus on the role that Computational Logic plays in relating the syntactic and semantic analysis of natural language to its practical understanding within specific applications. Robustness is considered from two complementary perspectives, borrowing the terminology from modern software engineering: robustness "in the small" and robustness "in the large". The first perspective is discussed while presenting an application for the Interaction through Speech with Information Systems, where robust semantic parsing is used to extract queries from spoken natural language utterances.The second perspective is examplified by the re-engineering of an existing text analysis system using a new Language Engineering methodology: Agent-Oriented Language Engineering. In the second part of the thesis I discuss how cognitive aspects can be integrated into a Language Engineering environment leading to the notion of Cognitive Language Enginering. I tackle the difficult problem of robust dialogue management from both a cognitive and computational perspective. I propose two frameworks for the semantic representation and assimilation of information into the dialogue information state. The first framework allows us to represent and reason about the dynamic aspects of objects and events. The second framework is centered on the notion of mental space and it is used to build representations of the cognitive processing of information during communication.

EPFL2002

Using Educational Data From Teaching And Learning To Inform Teachers’ Reflective Educational Design In Inquiry-Based STEM Education

Denis Gillet, Maria Jesus Rodriguez Triana

Science, Technology, Engineering and Mathematics (STEM) education is recognized as a top priority for school education worldwide and Inquiry-based teaching and learning is identified as one of the most dominant approaches. To effectively engage individual students in Inquiry tasks, appropriate guidance needs to be provided, usually by combining different digital tools such online labs, data analysis tools and modelling tools. This is a cumbersome task for teachers to perform manually since it involves (a) assessing during the education design, the type and level of tool-supported guidance to be provided to students and (b) potentially refining this level and types to meet the guidance needs of individual students based on educational data from the delivery of the educational design. Thus, in our research we target to investigate how to support this process with educational data analytics methods and tools from both the design and the delivery of educational designs, that inform teachers’ decision making for systematic reflection. To this end, the contribution of this paper is the design and evaluation of a novel “Teaching and Learning” Analytics method and supporting research prototype tool, extending the scope of purely learning analytics methods, to (a) analyze inquiry-based educational designs in terms of the tool-supported guidance they offer and (b) relate these analyses to students' educational data that are already being collected by existing learning analytics systems, so as to increase teachers’ awareness and understanding and scaffold their reflection. A two-layer evaluation methodology was adopted to evaluate both the capacity of our method to analyze educational designs in terms of appropriate guidance as well as to investigate whether the insights generated by the method offer statistically significant indicators that impact students' activity during the delivery of these educational designs. The results obtained, based on real-life educational data, argue that the proposed method and tool can support teachers to accurately analyse Inquiry-based educational designs and receive meaningful insights to improve and tailor students' learning experiences. The insights of this work aim to contribute in the research field of cognitive data analytics for teaching and learning, by investigating new ways to combine analyses of the educational design and the students’ activity, so as to inform teachers’ reflective decision making from a holistic perspective

2019

Evaluating the Effects of Introducing Three Gamification Elements in STEM Educational Software for Secondary Schools

Juan Carlos Farah, Denis Gillet

Gamification aims to enhance the educational experience by enabling students to have fun with technology. Although research in the field has previously looked into the effectiveness of gamification, reviews of existing studies show that in the context of education, gamification has been largely applied at the university level, with diverse pedagogical approaches and outcomes. Furthermore, reports often refer to interactions with gamification elements in contexts outside STEM education. To bridge these research gaps, this paper reports on an empirical study involving 199 students from two secondary schools. Students had the opportunity to receive a digital reward (in the form of points, badges, or rankings) for their participation in an online physics lesson. Although no significant differences were found regarding student motivation, results confirm an impact—depending on the gamification element being introduced in the software—on three other important aspects: (i) perceived usability, (ii) student engagement, and (iii) learning performance.