In business software applications, a key factor whether or not to invest is whether or not the productivity of employees can be raised. ROI is measured in productivity, and productivity is measured in time. In usability tests, we routinely measure task completion time. This gave me an opportunity to have a closer look, and I’d like to share what I learned: it’s the slow users who matter most when you want ROI.
Most processes in the time domain follow an exponential distribution (Oops, statistics – bear with me for 3 more lines). The Exponential has an interesting property: at each time interval, the same percentage of units is affected by the process you’re looking at. In physics, in radioactive decay, you can tell exactly at which time half of the atoms have decayed. After the same time interval, again half of the remaining units will decay.
The same holds true for completion times in many tasks; with one difference however: typically, in usability test tasks, systems have a fairly constant response time, and users also need time to execute keystrokes and mouse clicks. This mostly constant time leads to a so-called translation of the exponential distribution. You may say that it starts only after a certain offset time. When the offset time has passed, the proportion of users completing the task in a given time interval is constant.
Sounds complicated? Here’s an example.
Suppose an expert user needs 80s for a task, say, to create a business document. For the expert, most of this time will be consumed by system response time and punching in the data – basically, this is the offset time. An “average” user may take longer of course. Let’s consider a really “average” user (technically, the median) who is faster than exactly 50% of users, and slower than the other 50%. Now suppose our average user takes 120s to complete the task, that is, 40s more than the expert. In the typical distribution of task completion times, which follows a translated exponential distribution, the next half of users will take again 40s more to solve the task: within 160s, ¾ or 75% are done. Compared to school grades, a percent rank of 75% would correspond to a B-. Not spectacularly bad, yet in our example this user would take twice as long as the expert to solve the task!
The next half, i.e. 87,5%, will be done in 200s. After 240s, three times the expert time, 6,25% users will still be struggling with the task.
As a rule of thumb for estimating task completion times, you can proceed like this:
- Determine the minimum time (A) needed to fulfill the task.
- Determine the “average” (correctly: median) time (B) to fulfill the task.
- Subtract A from B. Let’s call the result the Delta time D.
- You can expect ¾ of users to complete the task at time A+2D.
- Roughly 1/8 (13%) of users will still be working at time A+3D.
- Roughly 1/16 (6%) of users will still be working at time A+4D.
What this has to do with User Experience? The point is that if you really want ROI, you should focus on the average or slow users. Tweaking the expert time A may have an effect, but the amount of time D “average” people take longer than the expert is what actually drives ROI. Catering to slow users is what makes applications efficient.