You need a quick usability assessment of an already deployed system? For enterprise software, I’d start with a SUS survey campaign. It can give you exactly what you need in order to drill deeper into your users’ experience.

As a user, you know perfectly well how it feels when one of those survey popups jumps at you and asks you “to help improve your experience”. Surveys are a nuisance, if not worse, and most get clicked away immediately. So if you want to really know how good or bad your users’ experience is, you need to focus – ask efficient questions, ask the right people, and ask more only when you have enough results to determine where exactly to zoom in next.

The System Usability Scale (SUS) is a 10-item questionnaire that was published by John Brooke in 1996 as “A quick and dirty usability scale”. In spite of its age, over the years it turned out to be quick but not at all dirty. As of today, it’s the best-researched usability questionnaire in the field, among the best in terms of reliability and validity, available in several languages (also in German), and backed up with extensive benchmarking research (example). It has become an industry standard for measuring user satisfaction in a simple and quantitative way. You can deploy it on paper, or any electronic survey platform. What you eventually get as a result is a score between 0 and 100, which you can compare to published benchmark data.

Will this be sufficient? Of course not, but definitely yes as a first step. Let me explain.

There are numerous usability questionnaires out there that do a good job on assessing the user experience in some way (SUMI, UEQ, NPS, etc.) – each has its specific strengths and weaknesses. A common weakness in all usability questionnaires is the fact that they are not very specific. In comparative studies, most questionnaire scores are highly correlated somewhere in the .80s – this means that, to a large extent, they actually measure the same thing. You may ask about several things, for instance, what users feel about a software’s efficiency. What they will tell you is whether they liked it or not.

The interesting fact is that this is exactly what you need to know – first.

In user experience research, typically the variability between users is higher than between designs. In more common words, for any given product you will always find people who hate it and people who love it. If you want to know how good your product works for a larger audience, such as your company, you need to figure out how large each group is. Once you know that, you can drill down into each group and find out what makes them feel the way they do.

But first things first – you’ll need an efficient and reliable instrument for measuring the overall satisfaction of individual users. As of today, SUS offers the best trade-off between brevity and reliability I know of, and a huge body of research indicates that it indeed does measure our ominous “user satisfaction”. With SUS, you don’t have to annoy the whole company with a questionnaire only to get a result that is biased by skewed return rates. You can get a fairly precise measurement from 30 respondents or even less – as long as they are carefully selected and motivated to participate. Regardless of how many respondents you have, you can always estimate how accurate your measurement is by calculating a confidence interval, that is, the interval in which the “true” value for the entire population would be located with a defined likelihood.

In a recent analysis of an internal SUS survey, I found that the size of the SUS score 95% confidence interval dropped below 5 score points already between 20 and 30 respondents. To reduce the confidence interval further to 2.5 points, you would need as many as 160 respondents, and it would not get much better than that even when adding hundreds more.

So here’s the bottom line: if you want to get a differentiated picture of your users’ experience, forget big questionnaires. You can get a pretty accurate quantitative assessment of the overall quality of this experience with as few as 30 SUS respondents. So rather put your effort in selecting those 30 and proper administration of the questionnaire. Then scrutinize your findings – where scores are low, spend the rest of your research budget on investigating why. For this, at the end of the day there will be only one way – go and see users.

Not logged in