Conducting moderated user research activities remotely is a great way to get valuable feedback from users if travel budget is tight or there are other reasons why travelling to meet users in person is not viable. There are a lot of methods that can be conducted remotely, for example interviews, story board validations, or formative usability tests. In a remote set-up the moderator and the participant take part in the session from different locations by using a telephone or an online conferencing tool.

While remote sessions have appealing benefits, such as saving travel costs, there are disadvantages, such as missing out on non-verbal cues from the participants.

In the first post, I’ll summarize the advantages and disadvantages of remote user research sessions. In a second post, I give some tips and tricks about what you have to consider before, during, and after the sessions. In case you’re already planning your remote sessions and are looking for an overview of the most important tasks, take a look at the post Checklist for Remote Usability Tests.

Summary of Advantages & Disadvantages

Advantages

  • One of the biggest advantages of remote sessions is that they are inexpensive, since no travel is required for your team or the participant. This also means that you don’t have to put aside time for travelling.
     
  • Customers are rather inclined to take part in remote sessions than accepting a team from another company visiting them on site.
     
  • The chances are higher that you will get permission from the participant to record the remote session than, for example, recording an interview during a site visit.
     
  • A remote session can be joined by many members of your project team, as opposed to, for example, only a couple of team members going on a site visit.
     
  • The chances are high to get the participant’s approval for follow-up research activities (for example validations) at the end of a remote session.

Disadvantages

  • Planning remote sessions can be very time-consuming as you need to carry out all of the following administrative tasks:
    • Intensive communication with participants ahead of the sessions to explain the technical setup and how the sessions will be conducted, to ensure that any required legal documents are signed, and to clarify any open issues from the participants.
    • Scheduling of the remote session with each participant and with the project team. Rescheduling some of the sessions might also be necessary if participants request an already scheduled session to take place at another date and time. 
       
  • You do not have full control over the technical setup, for example, the participant might get thrown out of the telephone conference, or the online conferencing tool can crash completely and not recover.
     
  • In a remote session you have less time for the actual interview or test than in a face-to-face session. Here are some reasons why:
    • The first 10-15 minutes need to be reserved for the introduction (participant to join the session; moderator to build trust and rapport, to make the participant feel at ease, and to explain how the session will run).
      In a face-to-face session you do not need to spend that much time on the introduction, because building trust and rapport is much easier if you are in the same room.
    • Be prepared to spend about 30 minutes of the entire remote session for the technical setup, the introduction, technical difficulties (e.g., audio connection is interrupted), and additional explanations to the participant. That is, in a 90 minutes remote session, only 60 minutes should be reserved for the actual test or interview.
    • The moderator needs to provide more verbal feedback throughout the session than in a face-to-face session.
       
  • You can miss out on feedback from the participant.
    • Non-verbal cues of a participant, such as facial expression, head movements, or body language, are important signs that a moderator can interpret and react on in a face-to-face session. Many of these cues are missing in a remote session, even if you’re using a webcam.
    • Moments of hesitation and confusion on the participant’s side are more difficult to detect.
    • A bad phone connection or a participant mumbling or speaking softly can make it hard for you to understand everything the participant says. Also, participants who speak with a heavy accent or dialect are more difficult to understand via phone.
       
  • A remote session can be more exhausting for all involved parties (e.g., moderator, participant, note takers) than a face-to-face session.
    In a remote setting, you need to listen more carefully and stay highly focused throughout the whole session to not miss out on anything. Apart from the social nature of a face-to-face session, such a setting also gives you extra cues, such as body language or facial expression, that is, more senses are involved than just listening (the information load is distributed to multiple senses, which makes it easier to handle).
     
  • The participant might not be able to sit in a quiet, extra meeting room during the remote session.
    If the participant sits at his or her working place, he or she might get distracted by background noises, ringing of telephones or other colleagues. This in turn can influence the quality of the results of the session.
     
  • You never know who else will join the remote session together with the participant, that is, you don’t know if colleagues of the participant will also observe the session. This can lead to the following undesired outcomes:
    • The participant being self-conscious und inhibited if his or her manager is in the room.
    • Legal issues, for example, if only the participant has signed a required legal document, such as a non-disclosure agreement.
    • Not only the participants giving feedback, but also the other persons who joined the session. Worst case: The other persons do not match the role you needed for the session and take over the conversation.
       
  • The online conferencing tool can be a hurdle and not easy to handle for participants who have never used this tool before.
     
  • If you are conducting a remote formative usability test on a test system, the response time of the system might be really slow via the online conferencing tool.
     
  • You do not have full control over the way how the participant sees the screen that you’re sharing. For example, if you’re showing a prototype, it can happen that colors are not displayed the same way on the participant’s screen as on yours. Moreover, parts of the screen might be cut off on the participant’s screen.
     
  • It does not make sense to conduct context interviews remotely. Context interviews are traditionally conducted at the workplace of the interviewees to get information also about the context of the work. Therefore, context interviews are not well suited for remote sessions since you’ll be lacking the entire context remotely. Even if the participant describes to you his or her working environment, it is likely that you will miss out on many aspects, esp. the ones that are not obvious to the participant anymore or the ones that are not important to him or her.
    Another disadvantage of remote context interviews is that you don’t see anything that the participant needs to do without the PC; for example, the participant might need to print out a list and work with this list at another PC. Of course, the participant can explain the situation to you, but it takes longer to understand than observing the user directly on site.
    If an on-site visit is not viable, you might want to consider using other methods likely to provide you the data you need, such as a diary study.
     
  • You might need to conduct sessions during the night, for example if you are located in Germany and the participant in Australia. For night sessions it is more difficult to get the buy-in of the project team.
     
  • It is more difficult to hand the participant a gift after a remote session than after a face-to-face session. If you wish to provide the participant a gift, you can send him or her a voucher.
     

Judging from the high number of disadvantages, remote sessions are a second choice for conducting user research activities; whenever possible, it’s best to meet the users in person and conduct face-to-face sessions. However, if there is no chance at all to go on site, remote sessions can be a good method to collect user feedback.

If you fancy conducting remote sessions to collect user feedback, you might want to check out the second post on the subject; there you can find plenty of hands-on tips and tricks about what you have to consider before, during, and after the sessions.

 

Not logged in
  • Anonym  5 years ago

    Excellent articles (both part 1 and 2). I’ve conducted 16 remote exploratory interview sessions with SAP users last year and you’ve captured the preparation expected, required conduct, and resulting value. Me experience has been to always use remote interviews new customers as a preliminary first step to on-site face-to-face engagement. Learning is always achieved and the research team can then decide whether to engage directly or move elsewhere. Thank you.