All relevant details about the OpenInterface Open Source platform.

Main Objective

The main objective of the OpenInterface project is to design, implement and test an open source platform for developing multimodal interaction:

 that handles a rich and extensible set of modalities ,
 that enables quick replication ,
 that enables a focus on innovation (new modalities or forms of multimodality),
 that supports dynamic selection and combination of modalities to fit the ongoing context of use,
 that enables iterative user-centred design .

More specifically, the foreseen platform will embed a set of pure and combined modalities as reusable components and generic mechanisms for combining modalities.

 At design-time, for specifying the multimodal interaction of a particular interactive system under design, the designer will reuse and assemble OI components. From this assembling, the code of the interaction part of the system will be generated. The open source platform will therefore allow rapid development of multimodal interaction by assembling components. It is then a central tool for supporting a truly iterative User-Centred Design (UCD) process by enabling the rapid development of early prototypes for exploration, prototypes of different design options, and testbeds for experimental evaluation. Moreover by supporting the exploration of new forms of multimodality, the platform will be an instrument for defining extensions to existing standards. As part of the OI project, our objective is to implement this path transferring research results to industry by extending existing standards.

 At run-time, the platform must include mechanisms for dynamic reconfiguration of the components in order to support dynamic adaptation of the interaction, such as changing the run-time parameter of a component (e.g. the sampling rate of an audio input device), replacing one component for a functionally equivalent one (e.g. functional equivalence of modalities).

OI detailled objectives

1. First, in order to populate the platform with innovative pure or combined modalities, our objective is to gain understanding of multimodal interaction. The project will first make significant advances to theoretical understanding of the concept of multimodality and towards the specification of an operational characterization of a modality and of a combination of modalities.
2. Based on the theoretical results on multimodality, the second objective is to design, development and test a platform for rapidly developing multimodal interaction. The platform will embed modalities as reusable building blocks and generic mechanisms for combining the modalities. While few modalities can be developed by the OI partners for the need of the project (for example specific identified user’s need for one of our application domains), the modalities embedded in the platform will mainly be reused and adapted. Instead of focusing on pure modalities including robust trackers and recognizers, the work will focus on combined usable modalities and generic mechanisms for combining them. The objective is that the platform defines an open source repository of modalities along with generic mechanisms dedicated to the design of multimodal interaction. For testing the platform and understanding the impact of the platform on the user-centred design process, our objective is to design and develop two testbeds for different application domains. Two different application domains are considered for proving the unavoidable generic aspect of the platform. One testbed is dedicated to multimodal interaction with a large information space while the second one will enable us to study multimodal interaction in the context of a game. The two testbeds will consider multimodal interaction on a mobile device as well as with an augmented pervasive environment, e.g. home environment. Mobile device and pervasive environment define the two action lines of the project. As part of a user-centred design, it is important to note that we cannot define before the beginning of the project, the pure and combined modalities and therefore the building blocks of the platform that will be studied and developed.
3. Based on the iterative design of the two testbeds developed with the platform, our third objective is to identify standard extensions and to develop an interpreter for those standard extensions. The interpreter will exploit an XML specification generated by the platform: by doing so we materialize the transfer from research results to industrial standards. For testing the interpreter and the coupling with the platform, two validators for the same application domains as the testbeds will be developed and evaluated. As opposed to the testbeds that do not consider the limitations of the current mobile terminals, the two validators will run on existing mobile terminals (e.g., a phone).