Usability Evaluation Considered Harmful

Usability Evaluation Considered Harmful (Some of the Time)

Saul Greenberg Bill Buxton

In this paper, Greenberg and Buxton discuss some problems that are associated with current methods of user interface evaluation, namely an overuse of quantitative usability studies in cases where other methods of evaluation might be more appropriate.

I find this particularly interesting as it brings up the question (to me) of whether CHI truly is a "hard science" where specific testing methods are generally sufficient and indeed necessary to evaluate a solution. Because of the great variety of goals in systems developed in CHI it is easy to see that one testing methodology is not going to be sufficient to test the usefulness of all systems.

A somewhat disturbing point that the authors bring up to support this claim, is the trend of researchers designing systems with the idea of qualitative evaluation in mind. If you are designing a system so that it specifically does well in the tests you plan to put it through, doesn't that really invalidate the tests you are going to perform? What does that prove about the system, that it performs exactly as hoped in the specific highly controlled environment you put it in. Even then, is it actually a system that should be evaluated in this way?

I agree with the authors that testing should be more flexible and actually fit the problem that is being addressed, and thus not necessarily quantifiable. Perhaps a new testing system should be created for each new system instead of recycling old test and trying to recast the problem into an existing mold.

Ethnography Considered Harmful

CHI 2009

Ethnography Considered Harmful

Andy Crabtree, Tom Rodden, Peter Tolmie Graham Button

This paper is essentially a caution written towards interactive designers about the dangers of the wholesale use of sociological ethnographic methods. The authors feel that even though computing is moving into a new level of ubiquitousness it does not mean that designers should throw away old task and work specific study methods. They feel that the new methods being employed are too centered on creating a story and not on focusing on applications and uses of technology.

I feel that these arguments are certainly fair, but that in finding new places to apply technology designers can't just look for existing tasks and work but have to create a broader picture of their target. Just as it makes no sense to only ask a user what they want and not considering what they might need, it makes little sense to study only what people are currently doing without asking what their goals and beliefs are and whether there is some system or solution that could fit those goals better than adapting what they already use.

Still, I can see that it is dangerous to only rely on a highly qualitative study of a group as this can lead to conclusions that are difficult to back up. Without well aimed quantitative study, working results that are gleaned from an ethnography could essentially be considered conjecture and do little to provide scientific proof.

CHI 2007: Software or Wetware

CHI 2007

Software or Wetware? Discovering When

and Why People Use Digital Prosthetic Memory

Vaiva Kalnikaitė , Steve Whittaker

This paper investigates the use of prosthetic memory (both electronic and traditional) vs the use of organic memory. They compared usage of their journaling system, ChittyChatty, an audio recorder, organic memory and pencil and paper notes. They evaluated the four over a period of 1 to 30 days, asking users about the perceived efficiency, usability and accuracy of the systems.

As a result, the studies found that people like the ease of use of organic memory and pencil and paper over the short term, but that these degraded quickly in their usability and accuracy over time. They also found that their system ChittyChatty was generally well liked.

While their studies are interesting, especially for design of memory aid devices, I found their ChittyChatty system most interesting. It is a free hand note taking software that also records audio simultaneously, associating text notes with the audio recorded at the time. I think that would be extremely useful in the context of classes because you could essentially manually index the audio so you know where certain events occurred. Then you can quickly go back and listen to the audio for more details instead of having to write all the details at once.

UIST 2007: Gaze-enhanced Scrolling Techniques

UIST 2007

Gaze-enhanced Scrolling Techniques

Manu Kumar, Terry Winograd

In this paper the authors present several possible ways of using eye gaze detection to make reading text easier, and also present a few other interactions made possible by gaze detection. For the text reading system they had two main types: manual scrolling with gaze enhancement and automatic scrolling of various types.

Manual scrolling with gaze enhancement lets the user use page up and page down to manually go to the next section of text. However as the user pages, the system detects which line they were reading before they paged and highlights it in its new position once the text has finished moving.

Automatic scrolling came in a few different forms. The first is "Eye in the Middle" where the system measures the user's reading speed and adjusts the text scroll to match it. Another technique was smooth autoscrolling which had specific zones of control. If the reader was reading towards the top of the page, scrolling would stop. As the reader moves down the page begins scrolling and if the reader is near the bottom the page scrolls even faster. Finally there is an automatic page up and page down, where if the reader looks at the bottom of the screen for a long enough period (150-200 ms) then the page would page down.

In addition to the text, the researchers also created buttons that would be activated by the user looking off the screen in certain directions.

In a very small user study, the researchers found that the gaze enhanced methods were preferred, specifically the ones associated with discrete page up and page down, as the users found it somewhat difficult to read moving text.

I found the whole gaze assisted scrolling techniques interesting, particularly the use of a short activation period when gazing to determine if that was indeed the user's intention to scroll. There could be some very interesting uses for this (for example switching between windows in a desktop environment or somesuch)

Don Norman gets all Emotional

Emotional Design: Donald Norman

In this book Don Norman goes out to cover how designers can design with the emotions of users in mind. He talks about a number of ways designers can approach and affect the visceral, behavioral and reflective levels. He also talks about emotions in robots both as a human interface and as an AI goal/reward system.

I like that Norman acknowledges the large role that emotional connection plays in how people use objects, and more obviously how they feel about how well those objects/devices work.

Just as Norman says, it not very often that people use the reflective part of their mind, and as such it is rare that people truly use logic to make decisions in real life. Often people are doing things where there is not enough time or information to make a decision based solely on logic, and that is where emotion plays a large part.

Say you are looking to buy a new laptop. If you went in to the process just looking for a complete comparison based numbers and price you would likely go crazy with all the options presented. But then you decide that one laptop looks better than another. Perhaps it is pleasing to look at, this could signify that someone put a lot of thought into the construction. Maybe they keyboard feels good, something that might be important when using it to write a long paper. Maybe when you pick it up, it feels sturdy, possibly indicating good construction. All of these aspects affect how you feel about the laptop, and while these feelings may not be indicative of reality, they are often good heuristics to live by.

Of course if we make the jump and say that emotions are just subconscious, self-adjusting mental heuristics, then we could say that Don Norman's vision of robots with emotions is already a reality. Modern AI systems already have rudimentary learning systems that update heuristics that in essence act in the same way as emotions.

My final thought is on Norman's suggestion that we use emotion as output from systems as a way to quickly convey messages to people. This is pretty much the best idea in the book. Taking advantage of the emotional connotations that people place on everything from colors, sounds and facial expressions, we could create a subconscious vocabulary that would create an efficient way for machines to communicate their status to people.

Say your computer is running out of hard drive space, then perhaps a small face or something representing your computer in the corner of your screen begins to look worried, with its expression getting more severe as the problem gets worse. Now obviously the expressions would have to fit the situation, otherwise a person would just get irritated at the false positives and turn the system off. But at a quick glance the system could quickly inform you as to the status of your machine with more detailed information available on request.

Shocking

The Man Who Shocked The World

by Thomas Blass

This book is a biography of Stanley Milgram, the famed experimental psychologist. As far as biographies goes it was certainly fairly standard, with a mix of interesting character analysis and typical (if bland) biographical details.

I find Milgrams ideas and experimental methods interesting, but I don't know if I feel very sure as to whether his experiments were the best way to empirically prove research points. While his shock experiments were certainly enlightening, it seems unclear as to what exactly he thought this would prove about obedience. It showed that within the circumstance of a procedure perceived to be some sort of experiment people would be obedient to a distressing degree, but he did not seem to dig deeper to find out why people acted like that and whether the results could be reproduced in different sorts of situations.

Milgram most certainly shocked the world, but what he meant to do with his controversial experiments is still somewhat unclear to me.

Robot control: wiimotes

Exploring the Use of Tangible User Interfaces for Human-Robot Interaction: A Comparative Study

Cheng Guo, Ehud Charlin. University of Calgary

This paper was a comparison of the efficiency in controlling a robot using two different types of input devices: a set of buttons (aka a keyboard) and a 3d Tangible User Interface (aka a pair of Wii-motes).

There were two tasks that users had to perform for the study using either a keyboard system or the wii-mote+nunchuck system to control a sony AIBO. The first was to navigate through a maze, the second was to make the robot pose in a variety of ways.

Overall the researchers found that the Wii-motes worked slightly better than the keyboard for the navigation task and much better for the posing task. This is likely because there was more of a direct conceptual mapping from the wii-mote motions to the actual motions of the AIBO.

Additionally people preferred the Wii-motes because they generally felt it was easier to remember what commands did what.

In summary I would say that it makes sense to use an input device that has similar constraints to what you are trying to do. If you are controlling something in three dimensions you want something that senses 3d control input. If you are performing a task in 2d then all you need is 2d control ability.