After explaining what A/B testing is and how powerful it can be, I would like to stress how important it is to have the right persons, users, and customers to test.
You can for sure test your product or service with all users / all visitors, but in almost all cases we are in a world where and when we must show and offer customized elements, interactions, and interfaces. That leads you to the need to define target groups and participants to test for specific usage and expectation.
Recruiting the right persons is crucial for getting valuable results from your testing and research. These ‘right persons’ have to be able to represent your consumers, customers, and end-users. That means that you must consider what criteria you have for your users before you can start really looking and hire some.
When you have only a high level and general characteristics and attributes, like age and location, getting such persons together is relatively easy. However, when you have more specific needs and attributes concerning whom you need to conduct your testing, it can be more difficult and time-consuming.
In other cases, you need people who can provide more specific insights into the service, application, or use case you are designing and developing for – e.g., persons who have experience using a certain technology or even likely to use it. Think, e.g., about VR or AR – or someone who has been through a specific type of situation or life experience.
Within the group of persons you choose you can go for sure with mainstream users – but please involve extreme users, experts, and analogous users. What are extreme users and experts? Imagine you develop e-commerce for a DIY hardware store. For your project of a DIY – an extreme user might be someone who never really build something himself. And an expert is someone who knows everything about do-it-yourself – from countersunk bolts to special boring and milling machines. An analogous user might be someone like a model builder with very special needs. Or a disabled person who likes to fix something. The dimension or quantity of their needs and knowledge is most often the contrary reflection of the number of users. You can learn so much and you can improve your chances for meeting the expectation of your true target users later, as the extreme users will consist of a blanket of experience, and lack thereof, that will show you where your service and application really stands or should be. And as always, please keep accessibility issues and users with disabilities in mind as well.
Testing the right people also includes the aspect of time. If we are talking for instance about a running e-commerce store and you like to test the performance, the usage, and in the best case the conversion rate – purchasing your advertised product on your promotion, of your campaign, or the carousel of products – keep the time of the day on your radar.
In certain cases, all of us interact differently depending on the time of the day. In the morning hours of a workday, we are often in a hurry; so, we interact, view, recognize things differently and we emphasize other interests. This indicates that when you want to test something in a running environment, it might make a lot of sense to define a time frame. There are hormones that help us wake up, there are hormones that help us go to sleep, and the changes in these various factors have impacts on the choices and judgments we make. In the fall and the winter season, there’s a lot less daylight. Or there are some people who are early birds, and others who consider themselves night owls. Each of these factors has an impact on how we do things and consume and buy things. And we can and should have on our radar who, when (running system), and how we test.
Finally, deciding on the right number and type of criteria for research participants is a balancing act. Here it might make sense to look what the Nielsen Norman Group suggests: https://www.nngroup.com/articles/how-many-test-users/
Conclusion and final hint
Don’t wait to think and plan for testing. As soon as you have some idea of what you want to test, start recruiting. It takes a little lead time, so start even before your design is finalized. Good luck and much fun!