Intro

In the context of intelligent systems, “user feedback” refers to the process of collecting, interpreting, and integrating user feedback into the system to improve the intelligence and the results.

Feedback is always triggered by the user. The feedback is based on the user’s judgement and experiences while reviewing and responding to the AI output.

Collecting data for further manual analysis for other purposes (such as marketing or usability research) is out of scope.

When to Use

Collect user feedback on AI output if you want to enable dynamic learning behavior to enhance the intelligence of your product.

Prerequisites

  • You have implemented a retraining infrastructure.
  • There is sufficient data to run instant or periodic model training.

User Consent

Some user feedback types can be legally sensitive and require user consent. When applying user feedback elements to your product, validate your use case with users, customers, and legal team. Local data protection regulations may also put constraints on how user feedback is processed.

Feedback Collection

There are four main aspects to consider when deciding how to collect feedback:

The sections below look at each aspect in more detail.


Feedback Collection Methods

Feedback collection can be implicit or explicit.

Implicit vs. explicit user feedback
Implicit vs. explicit user feedback

Implicit Feedback

Implicit feedback is collected by tracking the user’s activities, including navigation paths, search queries, inputs, and other interactions. This methodology gathers feedback while the user is interacting with the system, and is non-disruptive.

Implicit methods serve most collection needs.

Guidelines
Use implicit feedback methods if:

  • Adequate feedback collection is critical for retraining the model on a regular basis and explicit feedback methods fail to meet the requirements.
  • Users are unlikely to register the feedback accurately using the explicit feedback methods.
  • It is part of the user’s main task to make the decisions that are being tracked.

Note: Users are not aware when implicit feedback is collected. Evaluate carefully whether you need to request consent from each user in advance.

Explicit Feedback

Explicit feedback is given when the users interrupt their regular course of action to give feedback, or when the system interrupts the user to ask for feedback. Use explicit feedback methods judiciously, and only if implicit methods aren’t suitable.

Explicit methods serve special collection needs.

Guidelines
Use explicit feedback methods if:

  • Giving feedback is a choice, and not mandatory.
  • Providing the feedback doesn’t demand too much cognitive effort.
  • Giving feedback doesn’t distract users from performing their primary tasks.


Feedback Collection Context

The position of the triggers reflects the context or purpose for collecting the feedback. We can broadly classify the context as either “local” feedback or “global” feedback.

Characteristics of Local Feedback

  • The feedback trigger is close to the UI element and contextual information.
  • Users can give direct feedback on a specific component or value.
  • Local feedback is clearly focused, which makes it easier to interpret the feedback afterwards.

Characteristics of Global Feedback

  • The feedback applies for an entire page, or even an entire app.
  • The focus is not limited to a specific component or value.
  • The feedback is collected with the intent to improve the overall ML model, not a single inference.

Guidelines
To avoid cluttering table line items with multiple feedback triggers, use the table toolbar to collect both local and global feedback.

Types of User Input

To choose the appropriate input control, think about different ways of asking for the feedback and how well you will be able to utilize the insights. The method and format determine the quality of the feedback.

We differentiate between structured user input and unstructured user input.

Structured Input

You can collect very specific feedback by asking a closed question with a given set of options. The user can answer only in the predefined format and can’t give feedback freely.


Characteristics of structured feedback:

  • Questions have a clear focus.
  • Questions can be answered with “yes” or “no”.
  • The feedback collected usually relates to the quality of recommendations.

Unstructured Input

You also can collect unrestricted feedback by asking open questions. Here, the user can provide complex answers and include feedback on aspects you hadn’t thought of. This allows you to collect insights on external real-world factors that aren’t part of your system.

Characteristics of unstructured feedback:

  • Questions are formulated openly, inviting unlimited answers.
  • Feedback is collected to gain information on the quality and performance of intelligence services.
  • Feedback is collected to understand user behavior on the UI while performing a task.
  • Feedback processing takes time and needs detailed investigation to be interpreted accurately.

Feedback Interaction Methods

The interaction method for collecting explicit feedback depends on the use case and the user’s motivation. We differentiate between active users and passive users:

Interaction Methods for Active Users

For active users, offer an optional feedback trigger on the UI so that users can give feedback whenever they want.

Interaction Methods for Passive Users

For passive users, prompt the user to provide feedback using a feedback form or notification. With this approach, the system interrupts the user’s workflow to ask for feedback.

Guidelines

  • Do not interrupt the user’s workflow in the middle of a process to ask for feedback.
  • For active users, place the feedback trigger close to the content you are collecting feedback about (context).
  • For passive users, request the feedback towards the end of the process. After the user has submitted feedback, display a message to confirm that the feedback has been received.
  • Give the user an option to skip the feedback step.
  • To gain the user’s trust, provide a timely system response to the user’s feedback and make sure the feedback can be tracked.

AI Feedback Patterns

An AI feedback pattern is a set of recommended UI controls that can be used to collect user feedback implicitly or explicitly to improve the AI output. The purpose of an AI feedback pattern is to help the designer and the product team understand the use case requirements and choose suitable UI interactions for the task.

The feedback patterns have been carefully designed to support different use cases. The recommended UI controls need to be implemented at app level to collect the user feedback and deliver it back to the AI service.

AI feedback patterns help UX designers choose a suitable user interaction for the feedback requirements
AI feedback patterns help UX designers choose a suitable user interaction for the feedback requirements

Steps for Implementing AI Feedback Patterns 

  1. Choose the AI feedback pattern for your AI implementation.
  2. Narrow down the required user interactions.
  3. Incorporate the feedback pattern using the appropriate UI controls.
3-step process for incorporating an AI feedback pattern into your design
3-step process for incorporating an AI feedback pattern into your design

Pattern Types

So far, we have identified three AI feedback pattern types:

  • Confirm or correct: The user confirms or corrects the solution provided by the AI.
  • Track user actions: The system tracks user actions performed on the AI results.
  • Measure observable behavior: The system measures the observable behavior of a user experiencing AI results.

Confirm or Correct

Users can actively give structured feedback on the AI output to help the system learn and improve its intelligence over time. Examples of this type of feedback include confirming and correcting the AI results. We distinguish between binary and non-binary feedback collection methods.

Binary feedback collection

Examples of simple binary feedback collection methods are action buttons, toggle buttons, or toggle controls like checkboxes (selected, deselected).

Toggle buttons for binary feedback collection
Toggle buttons for binary feedback collection
Primary action buttons for binary feedback collection
Primary action buttons for binary feedback collection

Non-binary feedback collection

In non-binary feedback collection methods, multiple selection categories are offered. Possible controls include combo boxes or a combination of list selection and radio buttons.

Combo box for non-binary feedback collection
Combo box for non-binary feedback collection
Guidelines
Use this pattern if you want to:

  • Collect bias-reduced user feedback
  • Prepare your AI model for supervised learning with predefined user feedback
  • Create an AI model that can only consume structured feedback
  • Collect feedback for multi-class classifications or clustering problem types


Track User Actions

Intelligent systems can also learn from the UI actions users perform on the AI results. These actions include:

  • Selecting, sorting, filtering, and grouping the AI results
  • Rating the AI results
  • Searching the AI results

All of these interactions can be tracked implicitly for retraining to display refined and optimized AI results.


Selecting, sorting, filtering, and grouping the AI results 

Users can personalize AI results by selecting, sorting, or filtering items. They can also perform actions to group the results, such as:

  • Add to cart: The user moves the item to a more focused list, which will be processed soon.
  • Not relevant: The user isn’t interested in the item determined by the AI but would like to see the item in the future in other contexts.
  • Save for later: The item isn’t relevant in the current context but is an interesting find. The user moves it to a less focused list to revisit it later.

Here, you can use a grid list to display the recommended list of items. Each block contains one item, with enough details for user to make a decision. To collect local feedback, you can place a primary action button in the block to record the user’s actions. If you need to accommodate more actions, use an overflow menu.

Recommended products displayed in a grid list
Recommended products displayed in a grid list

Alternatively, you can use a responsive table to display the list of recommended items. In this case, you can offer a primary action button at line item level to record user actions. If you need to accommodate more actions, use an overflow menu.

Recommended products displayed in a responsive table
Recommended products displayed in a responsive table

Rating the AI results

You can give users the option to prioritize rate AI results using a rating indicator. The rating can then be used to determine the usefulness of individual AI results, based on a numeric scale for all the results (highest to lowest).

A rating indicator for collecting user feedback
A rating indicator for collecting user feedback

Searching the AI results 

When a user searches for a specific item in a list of AI results, the system can use the search information to return similar results or optimize the order of results based on user’s interests and search history.

Guidelines
Use this pattern if you want to:

  • Primarly collect feedback about the quality of an AI result 
  • Collect feedback for a multi-class classifications or clustering problem types


Measure Observable Behavior

Once a user starts examining the AI results, the system can learn by observing the user’s behavior as they browse through the results. The system can track a variety of interaction variables, known collectively as clickstream data analysis. This includes tracking viewed itemsbrowsing patternsitems retained for future reference, items recommended to a friendthe time spent on individual tasks, and more. Insights derived from clickstream data analysis are used to improve the quality of future AI output.

Since this feedback is entirely implicit, no UI interactions are defined for this pattern. It is up to individual application domains to explore how clickstream data analysis could feed into their own AI models.

Note: This implicit feedback collection method is not transparent to end users and is likely to require advance user consent. Users are more likely to trust and accept SAP intelligence if you always ask for explicit user consent before collecting feedback data and using it to optimize AI results.

Guidelines
Use this pattern if you want to:

  • Collect feedback for a UI with a list of recommended products or services
  • Collect user feedback implicitly because explicit feedback collection methods cannot assemble enough adequate data to train the AI


Related Links