Updated: February 26, 2024

Explainable AI

Intro

To help build trust between human and machine, we need to provide sufficient information about the underlying model and the reasoning behind the results of an algorithm. This is even more critical in business applications, where users are held accountable for the decisions they make.

“Explainable AI” means explaining the reasoning behind intelligent system proposals in context and at the right time. 

Usage

Consider using AI explanations if one or more of the following conditions apply:

  • Criticality – There is a high risk associated with making the wrong decision. Once taken, actions are hard to reverse.
    By contrast, if the risk is low and actions can easily be rolled back, users might not need an explanation for the system proposal.

 

  • Complexity – It’s hard for users to assess the impact and quality of their decisions. This might be due to a lack of process insight because of the automation of this task with machine learning.
    By contrast, if users can easily tell when a proposal fits (without training), they probably don’t need additional input.
  • Transparency – Your business case is subject to extensive auditing requirements. Auditors must be able to trace transactions back and see the underlying reasoning for each execution step. Auditors may require more information than end users.
    By contrast, if there are no auditing requirements, explanations might not be necessary (assuming they are also not required by end users).

 

  • Volatility – To adapt your machine learning model to changing conditions or requirements, your AI application relies on continuous feedback.
    By contrast, if the feedback has little or no effect on the output of the algorithm or the user experience, it can be more distracting than helpful to provide an additional explanation.

What Needs to be Explained?

When AI (machine learning) is introduced into business applications, systems originally designed to “react” can become active players that propose, change or create elements in the interface. Objects, actions, or attributes are manipulated not only by humans, but also by the AI algorithms.

To avoid confusion and loss of control, users need to be aware of the working principles, actions, and consequences of AI at any point in time. When designing AI services, we therefore recommend following our guiding principles for intelligent systems.

How Much Detail Do Users Need?

The amount of information users need to understand a system proposal can vary. This depends on the applied AI, usage context, user task, and user role. We differentiate between three explanation levels: minimum, simple, and expert.

Explanation level 1
Explanation level 1

Level 1: Indicator (What?)
The minimum explanation level. An indicator is required whenever AI (machine learning) output is provided. The indicator is also the access point for the next explanation level (if required).

Explanation level 2
Explanation level 2

Level 2: Abstract (Why?)
A condensed view of the relevant properties, amounts, and contextual information. An abstract helps users to better understand the AI proposals. It can contain links to the last and most detailed explanation level.

Explanation level 3
Explanation level 3

Level 3: Detail (How?)
An extended report specifically for advanced users. It covers all aspects processed by the intelligent system, the AI performance, and any further context and conditions that help users to monitor AI operations.

Guidelines

  • Decide which explanation levels are relevant for your user profile. Some user groups and roles might benefit from seeing all levels. Typically, these will be AI subject matter experts, such as data scientists and software engineers. For most users, the first two levels will be sufficient.
  • Use the explanation levels to apply progressive disclosure principles to your AI explanations.
  • Think about how users will access each level. The indicators can look and behave differently, depending on the user’s permission to access the next explanation level.

Progressive Disclosure

Progressive disclosure avoids overwhelming users with too much information at once. Initially, the user sees only an explanation indicator, and can then drill into two further layers of detail. The benefit of this approach is that the explanations don’t clutter the existing UI. Users only need to concern themselves with details if they actually need them.

Overview of Progressive Disclosure Levels

Level Type of Information
Level 0 No additional explanation elements. For cases where users do not expect or require an explanation from the system.
Level 1 Briefly explains the context, with or without the option to get more detailed information.
In simple cases where a short explanation clarifies the issue, a level 1 explanation may be enough.
Level 2 Presents condensed information, including tables, charts, or any other UI elements that offer more insight into the issue. Users access this level by clicking the explanation provided as a link in Level 1.
Level 3 Presents detailed information, including tables, charts, or any other UI elements that offer more insight into the issue.
Users access this level by clicking the explanation provided as a link in Level 2.
Via Conversational UI Not really part of the explanation concept, but gives users the option to engage SAP CoPilot (Ask CoPilot button or input field).

Level 1
Explanation indicator

Global / local indicator in lists and tables
Global / local indicator in lists and tables
Global / local indicator for cards
Global / local indicator for cards

Level 2
Simple explanation

Explanation popover, conversation items
Explanation popover, conversation items
Overlay
Overlay

Level 3
Extended explanation

Explanation page
Explanation page
Dedicated explanation application
Dedicated explanation application
Guidelines

  • Show the most frequently-used information in a concise form on the main screen (Level 1 – Explanation Indicator), with an option to drill down to more detailed information on secondary screens (Level 2 – Explanation Popover/Overlay). 
  • Use a separate page or app to offer a more demanding analysis of the AI, such as benchmarking or monitoring, or to let users make improvements (Level 3 – Extended Explanation Page/Monitor). Give access to this page or app via the explanation popover or by means of a dedicated tile on the SAP Fiori launchpad.

Explanation Components

The UX pattern for explanations builds on the explanation levels above (What?/ Why?/ How?). We use dedicated UI components for each of these levels.

Level 1: Explanation Indicator

The explanation indicator marks elements or combinations of elements that are created or manipulated with the help of AI. The indicator provides context about the “nature” and “quality” of a service provided. Clicking the indicator reveals a more detailed explanation (Level 2).

The following sections describe the different explanation indicator variants and when to use them.

Explanation Type

Explanation indicators can be global or local. This affects both the type of information they contain and their placement on the UI.

Global – Model level
Explains the AI output for a group of elements, such as a list, table, or whole section. The global explanation provides the overall context for more complex constellations affecting multiple objects or aspects.

Local – Item level
Explains the AI output for a single value or item.

Global Indicator

Example: Global indicator for a table or list
Example: Global indicator for a table or list

Local Indicator

Example: Local indicator in a table or list
Example: Local indicator in a table or list

It isn’t always necessary to provide explanations at both levels, but can be necessary if AI affects both the individual object and its position in the group.


Indicator Styles

The styling of the indicator influences how the user perceives and interprets the displayed output. We distinguish between a default style and an extended style.

Default Style
The default style provides the minimum amount of indication about the type of AI output. It uses explicit and descriptive labels and blue coloring for clickable explanations.

The default style offers sufficient visual affordance for non-critical AI applications.

Extended Style
The extended style uses semantic colors and to offer more contextual insight at the first explanation level. It can also include other semantic elements, such as an icon to indicate a trend.

The extended style can further improve the affordance for the user and emphasize the importance of the information provided.

Guidelines

  • If users need to be fully aware of the nature and quality of the output from the start, add semantic colors to the explanation indicator (extended style).
  • Don’t use icons or other “smartness” labels to show that the result was provided by a machine learning algorithm. Currently, we don’t have any proven research to indicate that highlighting “intelligence” improves the user experience.


Explanation Indicator Patterns / Examples

The table below outlines the recommended controls, placement, and patterns for different local/global explanation scenarios. The Examples column shows both a default explanation indicator and alternative (extended) indicators with semantic components.

Scenario Pattern Examples
Global Indicator

Ranking for table items
Placement: Table toolbar
sap.m.Link
sap.m.Button

Link/Button Text:
View <AI Operation> <Value Name>
Default:
View Ranking Criteria
Alternative:
  
How does this work?

 
ML Ranking
Global Indicator

Ranking for table items
Placement: Next to table title
sap.m.Label
sap.m.Link

Label Text:
<AI Operation> By:
Link (value) Text:
<Decision C
riterion>
<Criterion1, Criterion2, Criterion3> …
Default:
Ranked By: Price-Performance
Alternative:
  
Ranked By: 
Price, Delivery Time, Minimum Order Quantity, … 
 
Ranked By: Ranking Criteria
Local Indicator

Predicted priority of an item
sap.m.ObjectStatus
sap.m.ObjectNumber

Column Title:
 <AI Operation> <Value Name>:
Cell Value:
 <AI Output Value>
Default:
Column Header: Recommended Priority:
Cell: Very High (on a scale from very high to very low)
Alternatives:
  Very High
  9.6  Very High
    9.6  Very High

  9.6
  90%
Local Indicator

Quality classification for a product (such as wine)
sap.m.ObjectStatus
sap.m.ObjectNumber

Column Title:
 <AI Operation> <Value Name>:
Cell Value:
 <AI Output Value>
Default:
Column Header: Predicted Product Quality:
Cell: 
Premium
 (scale depends on use case)
Alternatives:
  Premium
  9.6  Premium
    9.6  Premium 

  9.6
  90%
Local Indicator

Price regression or prediction in a table
sap.m.Label
sap.m.ObjectStatus
sap.m.ObjectNumber

Column Title:
 <AI Operation> <Value Name>:
Cell Value:
 <AI Output Value>
Default:
Column Header: Predicted Achievable Price:
Cell: 100.00 EUR (scale depends on use case)
Alternatives:
  100 EUR
  100 EUR (+/- 10%)
    100 EUR

  100 EUR

Styling Conditions

The length of the explanation indicator and whether or not you display a link to a simple explanation (level 2) depends on your use case:

Form Factor Length Action Conditions
All form factors
(S, M, L, XL, >)
Short (default) Display only

Clickable

Display permission limited to level 1 or level 2 explanation is missing.

Display permission for levels 1 and 2 and level 2 explanation is available.

Large
(L, XL, >)
Long (optional) Display only

Clickable

Display permission limited to level 1 or level 2 explanation is missing.

Display permission for levels 1 and 2 and level 2 explanation is available.


Guidelines
Display clickable explanation indicators if:

  • The context is explained in more detail in level 2.
  • The user needs access to the level 2 explanations and has the appropriate authorization.

Ensure that the length of the explanation indicator is suitable for the screen size. To prevent truncated explanation indicators, apply the following rules:

  • Use the optional long style only if there is sufficient space and no collision with other UI elements.
  • If you are listing criteria, show 3 criteria and use the ellipsis to indicate that there are more criteria. If there is not enough space to show 3 criteria, use a short explanation indicator instead.
      Ranked By: Price, Delivery Time, Minimum Order Quantity, …
     
    Ranked By: Price, De…
  • If you use the long style, always offer a fallback to the default short indicator for smaller screens or form factors.
  • If you have no other options to free up space on the UI (for example, by moving other actions into the overflow), wrap the text.

For more information about responsive and adaptive design, see Multi-Device Support and Responsive Spacing System.


Confidence Indicator

A confidence indicator is a variant of the explanation indicator, and is used to indicate the quality for an individual result. Rather than percentage-based confidence levels, the explainable AI concept uses confidence categories.

Guidelines

  • Use simplified prediction ratings (confidence categories), such as “Recommended”, “Preferred”, or “Best …”.
  • Supply a reference system for interpreting the rating.
  • Exception:
    in some scenarios you may want to emphasize the rating or ranking of items. In such cases, you can also use scoring indicators for local explanations (see the examples for indicator styles above). This is because the AI output (which is a score or rating) and the explanation indicator are the same element in this scenario. For more information, see Ranking.

Confidence Indicator – Best Practices 

 Do    Don’t  
Approach Use confidence categories Use confidence levels
 

High level of accuracy
Acceptable accuracy

Examples:

Recommended
Alternative 

Examples:

89.68% 
77.53%

Reference system Provided Not provided
Design Rationale
A common approach in AI applications is to use percentage values to indicate the quality of a prediction. However, there is a usability issue with this method. People interpret confidence percentages differently, depending on their character and mood and the current situation.

Some users might reject predictions with a relatively high confidence rating (like 84%). Others will go with a proposal despite a much lower confidence rating (like 64%).

The percentage value for the confidence level has no bearing on the actual output. Poor user choices can also adversely affect learning algorithms. In this case, the machine learns from inefficient or misleading feedback.


Level 2: Simple Explanation

Explanation Popover/Overlay

Users access the explanation popover by clicking the explanation indicator. The popover is a canvas to provide contextual information beyond quality ratings, and to list or visualize key factors, inputs, or criteria that led to the AI result (for example, you could use a model induction algorithm or “Explainer” to reveal which data best supports the prediction).

Explanation popovers, screen overlays or conversational items can include the following elements:

  • Self-explanatory header title
  • Minimum: Short explanation of what the AI has performed in natural language.
  • Minimum: For AI outputs that are part of a scale, offer a key or legend to explain how the current value should be read.
  • Recommended: A timestamp indicating when the result or prediction was computed. If the output is time-critical, show when the result or prediction was last updated.
  • Recommended: A list of the 3-5 most important parameters that influenced the result being explained. This can include a comparison with averages or reference values, if required.
  • Optional: A short explanation in natural language of how the AI has performed in comparison to a given reference (historical cases, given data, average prediction accuracy, or similar). This can be combined with performance indicators or a trend.
  • Optional: In-place feedback controls if you want to collect explicit user feedback. For more information, see User Feedback.
Guidelines
Structure

  • Limit the breakdown to 3-5 factors, unless there is a critical reason to include more. Users may not benefit from greater detail – more conditional facts also trigger more questions and cross-checks.
  • To ensure a consistent experience, follow the suggested structure above.

Content

Make it easy to identify the most relevant information.

  • Use the most suitable format for conveying information. This can be language, visual elements, or a combination of both.
  • In longer texts, emphasize important information. This makes it easier for users to scan the content quickly.
  • Be careful with text formatting and visuals; do not use them for decoration only.
  • Try to restrict text explanations to no more than 3 short sentences.

Reveal information progressively.

  • Add value with each disclosure level. Do not just repeat information from the previous level.
  • Make sure that the information on each level is self-contained and that all elements form a natural flow.
  • If your use case offers the Ask CoPilot entry field, think of a smooth transition to the CoPilot panel. Guide the user and offer a useful hint. Use the phrase “Try asking”, followed by the hint.

Natural Language Explanations

Natural language explanations are a variation of AI explanations that apply conversational interaction principles as they are used in conversational interfaces. Reasoning and relations are represented in the form of sentences and non-technical speech.

The usage of natural language explanations is limited to the components explanation popover/overlay, explanation page, recommendation items, and conversational UI/SAP CoPilot. Currently, we have no proven research that natural language explanations are beneficial in other locations. For more complex situations, also see Situation Handling.

Natural Language Explanations – Examples

Scenario Examples
Adding value at each disclosure level Level 1: Ranked By: Price, Delivery Time, Minimum Order Quantity, …

Level 2:
  The price has the biggest impact on the score, followed by the delivery time and then the minimum order quantity.
  Suppliers are ranked by price, delivery time, and minimum order quantity.

Specific information   “Polaris Lights” offers the best price (470 USD) for the product “Taillight”, with the lowest minimum order quantity (300 PC). Delivery (14 days) takes longer than average.
  Best price with lowest minimum order quantity. Delivery takes longer than average.
Just enough formatting   This is high priority because the net payment is due soon (in 3 days) and the amount is small (1.200,00 USD). Customer Electronics usually pays on time.
  This is high priority because the net payment is due soon (in 3 days) and the amount is small (1.200,00 USD). Customer Electronics usually pays on time.

Guidelines

  • Use natural language to summarize the overall situation and to explain dependencies, origins or principles. For more information about the conversational user interaction, see the guideline for SAP CoPilot.
  • To display the simple description, we recommend using plain text.

Level 3: Extended Explanation

Explanation Page

You can use the extended explanation page to offer a more in-depth analysis. You can either direct users to a separate, dedicated application, or provide an additional page within your own app. This will depend on how much control and insight you are willing or able to provide for the technology being used. In most cases, extended explanations have a distinct use case that requires a separate implementation.

Explanation Page

Detail explanation page in a flexible column layout
Detail explanation page in a flexible column layout

Dedicated Explanation App

Guidelines

  • Only provide links from explanation level 2 to the explanation page or app if the user has appropriate authorization. AI analysis is relevant for only very few roles, and should only be offered on a need-to-know basis.
  • Do not offer the link to regular business users.

Explanation Page Content

The content on the explanation page depends on the needs of the user. At this point in time, the user probably has a specific reason for opening the detailed explanation.

  • If the explanation page is opened from a global explanation popover, the user focus is on the service as a whole. In this case, the user might want to learn more about the overall reliability of the service (business user), or the performance of the AI model (AI expert).
  • If the explanation page is opened from a local explanation popover, the user focus is on how a specific result was inferred. In this case, the user might want to see which other parameters were used for this prediction (business user), or review the log entries for the system reasoning (AI expert).
Guidelines
Identify the intention of your user.

  • Focus on the aspects that users want to explore in more detail, based on the preceding local or global level 2 explanation.

Display only relevant information.

  • Only expose the information users need to get clarity on their particular issue.
  • Don’t disclose information that users can’t utilize in their work. Hide elements and sections that are irrelevant for the current user role.

Provide separate views for each AI experience level.

You can roughly categorize users into three larger groups:

  • Business users (AI consumers)
  • AI lay users (such as configuration experts)
  • Subject matter experts (such as data scientists)

Behavior and Interaction 

When to Display Explanations

Display explanation components only when the AI service is able to provide predictions with reasonable confidence/accuracy. The threshold for acceptable prediction quality must be defined on a case-by-case basis, and depends on the given use case and AI capabilities. Nevertheless, we do not recommend exposing any predictions that have only poor reliability. Unreliable results lead to damaged user trust and hinder adoption of the AI functionality.

Example: Definition of quality thresholds
Example: Definition of quality thresholds

How Long is an Explanation Valid?

The validity of an explanation depends on the machine learning cycle. We differentiate two cases based on the explanation lifespan.

Static Explanations

Static explanations are suitable for “offline learning” or instance-based AI services. We assume that the need for a repeated explanation of the model can decrease or even disappear over time as the user gains more experience with the system.

Guidelines

  • Once the user understands the main principle of the underlying service (algorithm), consider leaving out explanations at the model level.
  • Always display the explanation again if the AI model is changed or updated. This applies to explanations that are hard-coded or maintained manually.

Dynamic Explanations

Dynamic explanations are required for “online learning” AI services. This type of explanation is generated (refined) every time the AI has learned and adapted its model. In most cases, it’s not necessary to change the appearance and content of the explanations completely. However, if an AI service discovers new rules to process items in the system, this newly-acquired knowledge must be reflected in its explanations.

Guidelines

  • Adapt the main AI description to reflect newly-applied processing principles.
  • Change the list of decision criteria whenever objectives/directions change.

Responsiveness

Fully Responsive

The explanation components explanation indicator and explanation popover must be fully responsive and available on all devices.

Partially Responsive

We recommend limiting the extended explanation (level 3) to larger form factors (desktop, large tablets) due to its complexity and type of use. If your scenario requires mobile access to detailed explanations, try to limit the amount of information and options to the essential elements.

Partially Adaptive

Explanation indicators are partially adaptive due to the two possible indicator styles (default and extended). The default style is the mandatory format. The extended style is optional for cases where decision criteria are available or beneficial for the user. If the information for the extended display is missing, or there is insufficient screen real estate, revert back to the default state.

For more information about responsiveness, see Multi-Device Support.

Top Tips

Before you start with your design, conduct research to figure out the answers to the questions below.

If extensive research is not an option, use a heuristic evaluation method to test your design. This should be the minimum investment.

  • Does the user expect an explanation?
    If the risks associated with an action are quite low and the results can easily be rolled back, users are not normally interested in an explanation of the system proposal.
  • What level of automation are you aiming for?
    Depending on the automation level, the use case and target user roles and app functionality can change dramatically.
  • Which other roles are involved in the experience besides your target business user?
    The interaction with AI systems also involves technical, non-business roles. Also consider roles involved in development, support, or maintenance.
  • How transparent or traceable does your machine learning service have to be?
    Some use cases are more critical than others. Due to legal and regulatory provisions, enterprise applications are often subject to stricter auditing requirements than consumer software.
  • Are your users generally able to relate to the displayed information and data, and can they infer the follow-on actions and impact (detail and explicitness)?
    Although AI can help optimize even simpler tasks, it might be overengineered to provide AI explanations when the user fully understands the results and implications.
  • How harmful can wrongly processed data be for the business of your client?
    Some actions in processes are critical by design and can have serious side effects. AI explanations must safeguard such scenarios and help prevent any damage or disruption.
  • How easy or difficult is it to revert changes applied to the system or process?
    In scenarios where the user faces deadlines, period-end closing, or other actions that must succeed first time, providing information to support the user is crucial. However, may not need explanations if it is possible to just revert everything instantly in case of failure.
  • Does your business case require continuous adaptation?
    We assume that the need for a repeated (static) explanation of the model can decrease or even disappear over time as the user gains more experience. However, if the AI learns dynamically, users must always be informed of changing conditions (dynamic explanations).

Related Links