When you design an intelligent system based on a machine learning algorithm, the data model it creates is typically dynamic. After being trained with an initial set of data, the model needs to be retrained on an ongoing basis as new data becomes available during usage. This new data also includes feedback from end users.

Usage

In the context of intelligent systems, we use the term “user feedback” to refer to collecting, interpreting, and integrating user feedback into the system to improve the software.

Collecting data for further manual analysis for other purposes (such as marketing or usability research) is out of scope.

Feedback Components

The feedback loop comprises three steps: collection, interpretation, and integration.

Collection – At this stage, the choice of UI pattern controls the user input and impacts the quality of the feedback. So optimizing the collection process means determining the most appropriate user input format.

Interpretation – To address typical feedback issues, feedback data may require additional processing before it can be used to retrain a model. For example, implicit feedback might be subject to event-related bias, or explicit feedback might only reflect the opinion of a specific group of users. In these cases, the feedback may first need to be assessed by a user in a different user role (such as a data scientist), using a separate user interface.

Integration – The final step is to integrate the feedback you have gathered to improve your product and make future AI results more reliable, precise, and relevant for the user. You may also find evidence that certain information is not required, or that steps in your process are redundant. This can lead to changes in the appearance of your UI over time.

Feedback Collection

There are four main aspects to consider when deciding how to collect feedback:

Feedback Collection Method
Feedback Collection Context
Type of User Input
Type of Interaction

The sections below look at each aspect in more detail.

Feedback Collection Methods

Feedback collection can be implicit or explicit.

Implicit Feedback

Implicit feedback is collected by tracking the user’s activities, including navigation paths, search queries, inputs, and other interactions. This methodology gathers feedback while the user is interacting with the system, and is non-disruptive.

Implicit methods serve most collection needs.

Explicit Feedback

Explicit feedback is given when the users interrupt their regular course of action to give feedback, or when the system interrupts the user to ask for feedback. Use explicit feedback methods judiciously, and only if implicit methods aren’t suitable.

Explicit methods serve special collection needs.

For more information about explicit feedback, see the Behavior and Interaction section.

Feedback Collection Context

The position of the trigger reflects the context or purpose for collecting the feedback. We can broadly classify the context as either “local” feedback or “global” feedback.

Characteristics of local feedback

• The feedback trigger is close to the UI element and contextual information.
• User can give direct feedback on a specific component or value.
• Local feedback has a clear focus, therefore the feedback interpretation process afterwards is easier.

Characteristics of global feedback

• The context for the feedback is at page level, or even app level.
• The focus of the feedback is not limited to a specific component or value.

Types of User Input

To choose the appropriate input control, think about different ways of asking for the feedback and evaluate how well you will be able to utilize the insights. The method and format determine the quality of the feedback.

We differentiate between the following types of user input:

• Structured input
• Unstructured input
• Absolute input
• Relative input

Structured Input

You can collect very specific feedback by asking a closed question with a given set of options. The user can answer only in the predefined format and can’t give feedback freely.

Examples:

• You want feedback about the quality of your recommendation   Ask users to submit their own rating
• You want to know about the perceived accuracy of your results  Ask about user satisfaction
• You want to learn whether a proposal you suggested was adopted  Ask why an item was closed or completed

Characteristics of structured feedback:

• Questions have a clear focus.
• Questions can be answered with “yes” or “no”.
• The format of the input or answer is given (for example, a price, amount, time, length, or category).
• The meaning and impact of the feedback is clear.
• The feedback can be based either on fact or opinion.
• Narrowly-defined feedback can refer only to internal system information.

Unstructured Input

You also can collect unrestricted feedback by asking open questions. Here, the user can provide complex answers and include feedback on aspects you hadn’t thought of. This allows you to collect insights on external factors from the real world that aren’t part of your system.

Examples:

• How do you like the proposal?  Open text area
• What prevented you from completing the task?  Text field (provide a reason)
• Was any information missing  Upload collection control

Characteristics of unstructured feedback:

• Questions are formulated openly, provoking unlimited possible answers.
• Questions can’t be answered with “yes” or “no”.
• The format of the user’s input/answer is not limited.
• The meaning or impact of the feedback provided is not known.
• Broad feedback can include external information from the real world.
• Broad feedback requires additional postprocessing to enable interpretation by the system.

Absolute Input

You can collect feedback based on properties or values that are part of the prediction object. In practice, you start with a given value and the users respond with a value of the same type. For example, if your output was a price suggestion, the user returns an alternative price. This type of feedback enables you to improve your AI model very easily, but incorrect user feedback can also infuse bias.

Examples:

• Do you agree with the given rating?   Users provide their own rating
• Do you agree with the suggested pricing?   Users suggest a different amount
• Do you agree with the proposed time?   Users suggest another time

Characteristics of absolute feedback:

• Returned values are a property of the object in focus.
• Users can provide concrete reference to indicate why the value should be different.

Relative Input

Relative feedback indicates more of a trend or user preference. This is a great way to collect generic information about your AI output and collect opinions from non-expert users to reinforce your AI outputs.

Examples:

• How did you like it?   Thumbs up or thumbs down
• How would you rate it?    +1 or -1 / higher or lower
• Was the proposal useful?    It was useful or it was not useful

Characteristics of relative feedback:

• Returned values are not a property of the object in focus but a judgement about it.
• Such values always indicate a tendency or trend.

You might want to collect feedback about the output of your machine learning algorithm in different ways. For example, if your user is not an expert in the domain, or can’t provide qualified feedback for any other reason, “relative” feedback might be more helpful than “absolute” feedback. However, if you can collect from reliable sources, it is preferable to get exact values that can be translated into direct insights more easily.

Type of Interaction

The interaction for collecting feedback depends on the use case and the user’s motivation. We differentiate between active and passive users:

• Active users take the initiative to give feedback if they are offered a feedback option. Active users are prepared to give feedback if they trust that their input will be used to improve their user experience.
• Passive users are unlikely to give feedback unless they receive a trigger from the system. Passive users don’t see an obvious benefit from giving feedback.

Interaction for Active Users

For active users, offer a feedback trigger on the UI so that users can give feedback whenever they want.

Interaction for Passive Users

For passive users, prompt the user to provide feedback using a feedback form or notification.

Feedback Form

Notification / Delayed Feedback

With this approach, the system simply notifies the user that feedback is required. The user can choose whether to react immediately or continue working give feedback later (“delayed feedback”). This allows you to gather feedback from passive users without interrupting their flow. In this case, however, you must be able to reconstruct the context of the user’s decision when the user provides the feedback to the system at a later point in time.

Choosing the Right Interaction Type

• Where possible, avoid prompting the user to give feedback. If your use case allows, offer feedback options directly on the respective UI instead.
• If the system needs feedback from the user, consider notifying the user that feedback is needed, rather than interrupting the user with a feedback form.
  Example: You want to collect feedback on why an approved amount exceeds the system recommendation, but don’t need the information immediately.

Top Tips

Before integrating feedback features into your product, think about trust, motivation, and quality:

Trust – By participating and exerting influence, users experience the rational sense of control they need to start trusting in your service. With feedback, your service becomes more of a partner than a dictator.

Motivation – You want your users to be active participants and use your service. Highlight the positive impact of their contribution and show them that they are a relevant part of the process. Motivated users deliver better and more profound data for your AI model.

Quality – AI models are abstract approximations of reality. But circumstances change, processes transform, and businesses evolve. As a result, the models need to be adapted from time to time. To keep abreast of change, you need to collect the relevant parameters systematically.

Research Questions

Before you start with your design, conduct research to figure out the answers to the following questions:

If you are using machine learning models, what does the retraining process (feedback loop) look like? If the model requires end user feedback, what kind information do you need?

Background: Depending on the nature of the feedback loop, there may be specific requirements for the feedback UI (explicit feedback, implicit feedback). You may also need administrative UIs to aggregate and evaluate the feedback data (for the data scientist role, for example).

What could prevent the user from providing (explicit) feedback in your case?

Background: Knowing the potential issues will help you to address them with your design, and may even involve adjustments to the business process.

What level of detail is expected for explicit feedback?

Background: On the one hand, it is beneficial for an algorithm to acquire as much structured information from the user as possible. On the other hand, this can easily impair the user experience.

What type of information can be considered as implicit feedback?

Background: Depending on your use case, you will be able to capture different aspects of user behavior (search, navigation, input, …). Some types of information are specific to the use case, and can be gathered only by the application, while other types can be gathered centrally.

Related Links

• Designing Intelligent Systems
• Explainable AI
• Matching
• Ranking
• Recommendations
• Situation Handling