Heuristics for the Development and Evaluation of Educational Robotics Systems

Contribution: While previous research has studied the use of educational robotics in classrooms, there is still a lack of methods to support the development and evaluation of such tools. To this end, this paper presents an evaluation framework and a corresponding set of heuristics, which are specifically adapted to the needs and expectations in formal education settings. Background: The increased usage of educational robots in classrooms, as well as the steadily growing number of alternatives to choose from, raises the question of finding appropriate methods for the development and evaluation of such tools. Yet the current body of literature does not provide comprehensive frameworks that allow to adequately address this question. Intended outcomes: The work aims at providing an evaluation framework, which could support researchers and engineers, as well as educators and decision makers in taking informed decisions about educational robotics systems. Application design: This paper proposes to consider activities involving educational robotics systems as a kind of “educational augmented tabletop game”. Within this framework, a set of fourteen heuristics was devised based on literature about games and learning tools. The validity of the devised heuristics was examined with a heterogenous group of twelve compulsory school teachers, who tested five different educational robotics systems. Findings: The experimental results illustrated high approval for the devised heuristics by the participating teachers. A heuristic evaluation based on the proposed framework appeared to be more appropriate to reflect the teachers’ needs than conventional methods, i.e., the isolated comparison of system characteristics.