Cross-Device Taxonomy: Survey, Opportunities and Challenges of Interactions Spanning Across Multiple Devices
Frederik Brudy, Christian Holz, Roman Rädle, Charles Wu, Steven Houben, Clemens Klokmose and Nicolai Marquardt. CHI 2019.
Microsoft Research, Redmond, WA.
Abstract
Designing interfaces or applications that move beyond the bounds of a single device screen enables new ways to engage with digital content. Research addressing the opportunities and challenges of interactions with multiple devices in concert is of continued focus in HCI research. To inform the future research agenda of this field, we contribute an analysis and taxonomy of a corpus of 510 papers in the cross-device computing domain. For both new and experienced researchers in the field we provide: an overview, historic trends and unified terminology of cross-device research; discussion of major and under-explored application areas; mapping of enabling technologies; synthesis of key interaction techniques spanning across multiple devices; and review of common evaluation strategies. We close with a discussion of open issues. Our taxonomy aims to create a unified terminology and common understanding for researchers in order to facilitate and stimulate future cross-device research.
Publication
Figure 1: Ontology of cross-device research terminology
We untangle the diverse terminology of cross-device work and map terms out into a single ontology. The main goal of this synthesis is to provide better guidance about scope and specialisations of research within our field. The unified ontology is formed of two parts of key terms used to refer to cross-device literature.
Top part: key term categories of cross-device sub disciplines. The nested categories begin at the bottom with dual-display and multi-monitor work (1a), extend to work with multiple mobile devices, tablets or slates (1b), further to the category of cross/multi-display (1c) and cross/multi/trans-surfaces (1d), and finally to cross/multi-device and distributed covering the broadest scope (1e). The nested structure in the figure indicates focus areas associated with each term as well as the often included device form factors. It is important to note that a large subset of the terms are used interchangeably.
Bottom part: a list of terms indicating focus areas of research projects (1f): interactions and collaboration; interfaces; applications or systems; platform or middleware; environments or ecologies; and computing. We annotated these terms with the most common research focus of papers using each term.
Other figures
Figure 2. Taxonomy of cross-device design space dimensions: temporal, configuration, relationship, scale, dynamics and space.
Table 2. Tracking characteristics and modalities of the cross-device papers with tracking as a main contribution. Our tracking classification directly relates to surveys of tracking technologies in ubicomp.
Table 3. Overview of interaction techniques for cross-device computing.
Conclusions
Over the past three decades, interaction with computers has progressed from single-screen mainframe computing, to dual-screen desktop PCs, to advanced multi-display devices with gesture interactions, to the proliferation of today’s mobile and wearable devices. Multi- and cross-device computing has become a fundamental part of human-computer interaction research. Despite the great variety in research agendas and focus points, the common ground in our community is to understand, create, and deliver experiences that transcend the individual device. Surveying over 30 years of cross-device research in a single paper does not do justice to the many researchers who have actively created, developed, contributed to, and shaped this field. We acknowledge that there are many different ad to be drawn from this survey and we hope that our interpretation is one further step for a wider discussion within the HCI community. By reflecting on the terminology we are using, and by identifying and addressing the fundamental challenges, the community can shape the future of cross-device computing together. We are looking forward to continuing discussions about where we are heading, and invite cross-device researchers and practitioners – new and established alike – to contribute to our open dataset.