Usability Evaluation Considered Harmful

Usability Evaluation Considered Harmful (Some of the Time)

Saul Greenberg Bill Buxton

In this paper, Greenberg and Buxton discuss some problems that are associated with current methods of user interface evaluation, namely an overuse of quantitative usability studies in cases where other methods of evaluation might be more appropriate.

I find this particularly interesting as it brings up the question (to me) of whether CHI truly is a "hard science" where specific testing methods are generally sufficient and indeed necessary to evaluate a solution. Because of the great variety of goals in systems developed in CHI it is easy to see that one testing methodology is not going to be sufficient to test the usefulness of all systems.

A somewhat disturbing point that the authors bring up to support this claim, is the trend of researchers designing systems with the idea of qualitative evaluation in mind. If you are designing a system so that it specifically does well in the tests you plan to put it through, doesn't that really invalidate the tests you are going to perform? What does that prove about the system, that it performs exactly as hoped in the specific highly controlled environment you put it in. Even then, is it actually a system that should be evaluated in this way?

I agree with the authors that testing should be more flexible and actually fit the problem that is being addressed, and thus not necessarily quantifiable. Perhaps a new testing system should be created for each new system instead of recycling old test and trying to recast the problem into an existing mold.