Component-based methodology and development framework for virtual and augmented reality systems

The very recent revolutionary advancements and wide availability of the real-time 3D graphics technology results in the overwhelming and still quickly growing number of toolkits, game engines, and VR/AR frameworks, which offer very broad collections of functional features. It becomes apparent that soon the winning factor will not be related to the number of features provided, but rather to the availability of a flexible component-based process and architecture able to curb exploding complexity, supporting seamless integration, and assuring broad design and code reuse. Since the late 90's Component Based Development (CBD) is a very active area of research and development. However, most of the current efforts and component standards are strongly biased towards enterprise information management systems focusing on distributed, secure, and transactional business logic. Concerning Game/VR/AR (GVAR) system engineering domain component-orientation, although highly demanded, currently is poorly understood and still in the pioneering phase. The work presented in this thesis consists of the three main parts. The first part focuses on the systematic analysis, mapping and adaptation of the current understanding of the CBD methodology to the needs of the GVAR system engineering. The resulting GVAR specific CBD methodological template is then validated by confrontation with the set of existing GVAR system engineering solutions. Mapping to the uniform CBD semantics yields detailed taxonomy and demonstrates the evolutionary convergence of initially isolated architectural (design related), functional (system operation and mechanism related), and development (process related) patterns towards the common CBD methodological denominator. As a result, the second part proposes GVAR specific component model and the respective component framework implementation (VHD++). In context of the VHD++ component model, we study consequences of separation between content (storing) and software (computing) side components and the role of the multi-aspect- graph concept. In context of the VHD++ component framework, we specify the architecture and identify an ensemble of fundamental coordination mechanisms necessary to support and enforce the VHD++ component model. In the third part we focus on the validation of the proposed CBD methodology from the perspective of the main three actors of the CBD process (component framework developer, component developer, application composer). In particular, we study examples of concrete components, inter-component collaborations, and instances of VR/AR storytelling systems featuring various combinations of advanced virtual character simulation technologies, immersion, and interaction paradigms.

Augmented Reality to Facilitate a Conceptual Understanding of Statics in Vocational Education

Lorenzo Lucignano

At the core of the contribution of this dissertation there is an augmented reality (AR) environment, StaticAR, that supports the process of learning the fundamentals of statics in vocational classrooms, particularly in carpentry ones. Vocational apprentices are expected to develop an intuition of these topics rather than a formal comprehension. We have explored the potentials of the AR technology for this pedagogical challenge. Furthermore, we have investigated the role of physical objects in mixed-reality systems when they are implemented as tangible user interfaces (TUIs) or when they serve as a background for the augmentation in handheld AR. This thesis includes four studies. In the first study, we used eye-tracking methods to look for evidences of the benefits associated to TUIs in the learning context. We designed a 3D modelling task and compared users' performance when they completed it using a TUI or a GUI. The gaze measures that we analysed further confirmed the positive impact that TUIs can have on the learners' experience and enforced the empirical basis for their adoption in learning applications. The second study evaluated whether the physical interaction with models of carpentry structures could lead to a better understanding of statics principles. Apprentices engaged in a learning activity in which they could manipulate physical models that were mechanically augmented, allowing for exploring how structures react to external loads. The analysis of apprentices' performance and their gaze behaviors highlighted the absence of clear advantages in exploring statics through manipulation. This study also showed that the manipulation might prevent students from noticing aspects relevant for solving statics problems. From the second study we obtained guidelines to design StaticAR which implements the magic-lens metaphor: a tablet augments a small-scale structure with information about its structural behavior. The structure is only a background for the augmentation and its manipulation does not trigger any function, so in the third study we asked to what extent it was important to have it. We rephrased this question to whether users would look directly at the structure instead of seeing it only through a tablet. Our findings suggested that a shift of attention from the screen to the physical object (a structure in our case) might occur in order to sustain users' spatial orientation when they change positions. In addition, the properties of the gaze shift (e.g. duration) could depend on the features of the task (e.g. difficulty) and of the setup (e.g. stability of the augmentation). The focus of our last study was the digital representation of the forces that act in a loaded structure. From the second study we observed that the physical manipulation failed to help apprentices understanding the way the forces interact with each other. To overcome this issue, our solution was to combine an intuitive representation (springs) with a slightly more formal one (arrows) which would show both the nature of the forces and the interaction between them. In this study apprentices used the two representations to collaboratively solve statics problems. Even though apprentices had difficulties in interpreting the two representations, there were cases in which they gained a correct intuition of statics principles from them. In this thesis, besides describing the designed system and the studies, implications for future directions are discussed.

EPFL2018

Augmented Reality to Facilitate a Conceptual Understanding of Statics in Vocational Education

Lorenzo Lucignano

EPFL2018