Usability of next-generation interfaces - ubiquitous computing

Apple’s iPhone isn’t the first device to use a multi-touch sensing interface, but it certainly is the most hyped. Using a similar interface in a wildly less portable way, Microsoft’s ‘Surface’ is a glorified coffee table that lets you manipulate photos in a hypernatural way, resizing them as though they were made of proportion-restrained rubber. And multi-touch is only the tip of the interface technology iceberg, with context-aware and ubiquitous computing floating right below the surface. Is the next generation of HCI upon us? And if so, how can we test or even determine the usability of these new paradigms?

The simple answers are: not quite, and quite easily. If a system is meant to be used, its usability can be measured. The real issue is to determine what exactly has changed. Future interface trends tend to fall into one of three categories: new means of input, new means of display, or an unconscious interface.

Nontraditional Input

The most significant alternative means of input lately has been gestural input devices, sometimes as part of a multimodal approach. This technology has its roots in the cumbersome wired gloves and suits of the 70s and 80s and now allows users to control large-scale applications like Google Earth or, to a certain extent, more common desktop applications, though to date these technologies lack the subtleties needed for fine control and interpretation.

Gestural interfaces are looking especially promising for working with large 3D or virtual reality models of complex systems, such as representations of molecular interactions or Fakespace’s immersive visualization systems. And of course gestural interfaces have entered the consumer market with a vengeance. Nintendo’s Wii, with its noncommand interface, lets users practice their golf or tennis swings using a small plastic controller in place of a putter or racket, and adapted dance pads let users hop around to control simple activities such as reading email or organizing photos with their feet.

Multimodal approaches tend to combine these inputs with speech recognition, most often using the X+V (XHTML plus voice) markup language. The Multi-Modal Interface for Man-Machine Interaction, or MMI2, combines direct manipulation of graphics with mouse gestures, command language and natural language entered through a keyboard.

Other systems, such as Iowa State University’s creation for infrastructure planning, use a variety of outputs – visual, audio and haptic – to convey information that would be hopelessly jumbled in a purely visual representation. A similar project at the University of Maryland’s Human Computer Interaction Laboratory combines visual and virtual spatial sound outputs to solve information overload, with obvious implications for the blind and partially sighted.

Tactile and/or kinaesthetic displays using vibrations, pin arrays and other alternative and multimodal approaches overlap with accessibility solutions, including people suffering from ‘situational impairment’ – such as those who need to operate a GPS system while driving, or who don’t want to announce our credit card numbers out loud while in public.

In the case of these variations on input and output, the issue of usability boils down to the gadgets themselves. What exactly are they intended to do, and do they succeed? Success seems to be linked to intent. The intent of the Wii is to provide a naturalistic entertainment experience to all users, and it seems to be thriving.

Not quite so many people are lining up to teach their computers how to recognize more than the eight preset gestures for controlling desktop applications that are designed for keyboard input (and will probably require keyboard input along the way no matter how many gestures it learns).

Multi-touch

More promising is the multi-touch interface appearing on the ubiquitous (or ubiquitously marketed) iPhone. Its ‘intuitive’ interface has been hailed as a key selling point, indicating good usability for such functions as zooming, scrolling and deleting. Similar technology, such as that developed by Jeff Han, encourages multiple users to interact and uses pressure sensitivity to enhance fine control. For manipulating virtual objects such as photographs, multi-touch seems to be a usability expert’s dream.

The real challenge will lie in attempts to integrate multi-touch into a wider range of applications. Speculation is mounting that Apple will make its move by introducing a tablet computer that uses a new operating system specifically designed for the multi-touch interface, or at least rework the Leopard operating system to handle multi-touch.

Usability will make or break any such attempts. It seems likely that multi-touch interfaces will succeed in two of the five generally accepted realms of usability: efficiency and satisfaction. But when it comes to complex tasks, will multi-touch provide enough flexibility to be truly easy to learn and remember?

And how will the interface prevent users from making a frustrating number of errors? Simply touching the screen rather than pointing with a mouse will not automatically remove these challenges. What it will do is require an unprecedented effort across the IT and multimedia industries to examine the usability implications of these changes and bridge the gaps as quickly and profitably as possible.

Nontraditional Interfaces

This brings us to the second new frontier in human-computer interaction: revolutionary ways of representing and interacting with information in a post-WIMP (window, icon, menu, pointing device) world. Traditional function-oriented interfaces have failed to latch onto such simple advances as 1998’s Data Mountain, a 2D representation of a 3D desktop, with files placed in stacks rather than folders. This approach uses spatial cognition, audio and visual cues and pattern recognition to reduce cognitive load, consequently reducing user errors and time spent on information retrieval, and yet most people have never heard of it.

So is there any practical hope for object-oriented user interfaces, where menus of actions are thrown over in favour of direct manipulation, not just of photos or maps but of every item a computer can contain? A good portion of the exciting, just-around-the-corner literature was written in the 1990s, and bloggers are mostly quiet on the subject. It seems there are far more important things to worry about than the potential usability issues for unknown paradigms.

Ubicomp

New inputs, new interfaces – is that all there is? Of course not. As always, technology is advancing diagonally, from unexpected angles. It pops up everywhere – or everyware. Mobile phones are beginning to use information about their own location and position to your advantage: iPhones save battery life by shutting down their displays when you place the phone next to your ear, and PARC’s Magitti software learns its user’s preferences for dining and shopping, offering targeted recommendations based on its current location.

In a sense, these devices are using alternative input methods, from position sensors in the iPhone to GPS and inference algorithms in Magitti-enabled mobiles. What’s important is that the input isn’t coming from the user, or at least not intentionally, which removes the need for an interface altogether. Ubiquitous computing is sneaking up on us, and we hardly even know it. Even the RFID chips in your Oyster card or library book are part of this network of interoperating devices that we interact with, even depend on, without being required (or allowed) to formulate a direct instruction.

These noncommand non-interfaces are clearly a goal for a number of influential researchers and designers. In much the same way that we’re so excited about object-oriented interactions on our computers, like resizing a photo with the touch of two fingers, we’re also supposed to be excited about the prospect of goal-oriented interactions with other devices, like remote controls that guess what we want to watch next. Both of these shift the orientation away from the functions of the device – cut, copy, paste, volume up, volume down – and toward the intentions of the user. This is fantastic news, right?

Ubi-usability

In the best of all possible worlds, of course it is. When I tell my phone or computer that I don’t want Chinese, it will understand that by next week I might well have insatiable cravings for chow mein – but when I tell it that I don’t want Polish, it will understand that I have an eternal horror of pickles. And if I were to fall pregnant and wish for nothing more than pickles morning, noon and night, it would adapt to my new requirements. This level of seamless intuition isn’t impossible, but as of today it certainly isn’t common.

Ubicomp will need to either follow or sidestep today’s critical usability principles. It seems well poised to avoid several of them altogether: the systems match the real world, require no separate standards and little user recall, enjoy the most minimal of minimalist interface design, and provide amazing opportunities for efficiency of use. It remains to be seen how seamless automation might negatively affect user expectations, though, especially in terms of user control. How can a user choose to exit from an interaction which he hasn’t even consciously initiated? How can she prevent, correct, recover from or possibly even notice an error that the system has not recognized itself? How can he confirm the system’s status? Most importantly, what flexibility will there be for the user to make adjustments based on his personal desires and expectations rather than the system designer’s?

Total User Experience

Any system can be analysed for usability by determining the user’s expectations before she encounters or even hears about it, then tracking her reactions and interactions as she learns the interface and makes use of it in ways either intended or unintended, following through to her likelihood of using the system again.

There are a number of complementary ways of gathering data on this process, including eye tracking, interviews, field visits, prototyping, modeling, audits, analyses, reviews, questionnaires and training – and the more the merrier. With a detailed and interconnecting picture of user expectations and reactions, any system can be revised to take optimal advantage of the human side of the human-device interaction, including those that are far from the current norm of typing commands at a workstation.

Some of the issues raised here will surely prove to be tiny molehills, while new issues will appear from nowhere. And even correctly functioning systems can cause heated debates stemming from personal preferences or opposing goals. The usability professional’s toolkit will need to be restocked and reconfigured over the coming years, but it will certainly not sit in a corner gathering dust.

References

Fakespace’s immersive visualisation systems: http://www.fakespace.com/displays.htm

Iowa State University’s creation for infrastructure planning: http://www.springerlink.com/content/bnq534j3256108w2/

Jeff Han: http://cs.nyu.edu/~jhan/ftirtouch/

Usability criteria: http://www.useit.com/alertbox/20030825.html

Usability heuristics: http://www.useit.com/papers/heuristic/heuristic_list.html
http://arstechnica.com/journals/apple.ars/2007/1/19/6688