Multimodal HCI in User Interface Design

Multimodal HCI is at the intersection of several disciplines that include psychology, computer vision, user interface design , and artificial intelligence just to name a few. Computers have become integrated into everyday objects and tasks (often termed ubiquitous and pervasive computing) and so it has become important to facilitate a natural interaction between users and computers that simulate the way humans interact with each other face-to-face. Multimodal HCI aims to discern the ways in which humans interact with each other in order to translate these processes to the human and computer interaction.

Multimodal HCI encompasses a wide range of modalities. A modality is a mode of communication according to human sense or type of computer input device. Typically modalities are divided into two categories:

1. Human senses: sight, touch, hearing, smell, taste, and cognitive/emotional abilities

2. Computer input devices: cameras, haptic sensors, microphones, olfactory (all meant to emulate the human senses)

Together these modalities can be used to construct various types of user interface designs in order to increase usability. In this article I will briefly overview the functions of each modality and then discuss the benefits of multimodal user interface design.

Multimodalities and their application in user interface design

The above mentioned categories exist to understand how, particularly related to user interface design, humans and computers utilize the same functions and can thus complement each other for a more positive user experience.

The human senses category helps designers understand how humans take in and process information. Sight has to do with what a user sees on an interface – words, images etc. Hearing has to do with the audio aspect of a user interface, whether it be music, commands, or conversation between two people. While smell and, especially, taste are less important in relation to traditional user interface design, they are still important for designers to consider since they can affect the other senses, and the synergy of all the senses ultimately affects the user’s cognitive and emotional state. From a design perspective, it is critical to understand the cognitive and emotional state that results from a user employing a user interface. This can help to detect and track the user’s task execution, motivations, and level of satisfaction with a design.

Computer input devices mimic the human senses in order to help people relate to computer interfaces (and conversely to help computers relate to people). Cameras function like an eye and mimic the visual sense, and haptic sensors are a tactile feedback technology (think of the mouse) mimicing the sense of touch. Microphones and speakers enable a user to hear and make sounds. Olfactory devices can stir up a user’s sense of smell; a process called machine olfaction employed by SMBO instruments (commonly referred to as electronic noses) uses a complex array of chemical sensors and pattern recognition algorithms to allow computers to mimic the sense of smell.

Why is multimodal HCI important to user interface design?

Multimodality is an incredibly wide-ranging and complex aspect of HCI, and we have only given a very brief overview of it in this article. However, it is important to have at least a rudimentary understanding of it in relation to interface design because at its core it attempts to understand how humans function as an integrated system of senses and cognitive abilities, thus allowing UI designers to let computers mimic and/or complement these processes. Understanding the full range of human senses and emotions can allow designers to construct user interfaces that capitalize on what senses and emotions their target group is most reliant upon or most influenced by. The user interface can then facilitate the user’s ability to perform tasks and achieve goals at a much more efficient rate, and ultimately lead towards a more optimally usable system. Stating that optimal usability is one of the ultimate goals of multimodal HCI and interface design may seem like a cop out, and probably a bit redundant, but in terms of user interface design optimal usability is always the ultimate goal, but it’s hard to reach if designers don’t grasp the critical methods and paths of getting there.