Explainable AI: making decisions understandable

People have always tried to improve their future by making sensible decisions. In doing so, they apply various rules and principles that they derive from their experience. In the course of technological, social and cultural progress, however, decisions have become increasingly complex.

They often include significantly more parameters than people can take into account in their subjective perception. For this reason, efforts are being made to develop learning systems that can independently recognize the relevant rules in large amounts of data. This is known as machine learning, the most important area of artificial intelligence (AI).

It can happen that the decision or forecast of an AI system differs significantly from expectations. In these cases, parts of the applied rules are unknown. But it is only when there are significant consequences associated with this that it becomes necessary to bring these unknown dimensions to light. Explainable AI is the approach that sets itself this goal. This creates confidence in critical decisions and at the same time helps to improve the human assessment of the situation.

What does explainable AI (XAI) mean?

Explainable artificial intelligence comprises methods and approaches that aim to better understand AI decisions. The focus here is on the question of which unknown influencing factors have led to a certain result. Because the models have recently become much more complex and often resemble a much-cited "black box" in the field of machine and deep learning, it is no easy task to find the necessary explanations. A typical example is large language models (LLMs), where even the developers themselves are often unable to say with certainty why their model has generated a given output.

Explainable AI enables understandable answers to the question "Why?"

Developers invest a lot of time and effort in explainability, especially in order to be able to adequately deal with possible biases or expected effects of critical AI decisions. This is particularly the case when the effects affect important decision-making processes in companies, compliance with laws and guidelines and general trust in artificial intelligence. Appropriate measures can be taken throughout the entire course of the projects, from the creation and processing of the database to the interpretation of the results. A distinction is made between Ad hoc methodswhich are already taken into account in the design of the model, and Post-hoc methods with the later addition of supplementary techniques and algorithms.

Why is explainability important?

One of the biggest goals in the development of artificial intelligence is to achieve an optimal compatibility of the People and technology skills to achieve this. In general, society and many companies are still in a discovery phase, while the proportion of processes involving advanced algorithms continues to grow. However, this now also affects many sensitive Processes where less human control through automation is only permitted under strict conditions. Explainable AI has become a key element that has a positive effect on the following aspects in particular:

Transparency

In many processes, the traceability of intermediate statuses is important. This allows errors and deviations to be detected in good time and only requires intervention in the affected work steps without any further consequences. Explainable AI helps to ensure that this form of transparency is also present in automation. This also has a resource-saving side effect, as the entire process does not necessarily have to be repeated in the event of errors. Instead, targeted maintenance can be carried out, which also increases the robustness of the system. Transparency therefore also strengthens trust and makes it easier for people to compare individual or general values and principles of action.

However, transparency and explainability do not necessarily have to be linked: A system can be transparent, but still be so complex that the reasons for a decision remain difficult to explain.

Compliance

In the context of companies today, adherence to certain values usually falls under the term compliance. It often takes place at every stage of human decision-making - often unconsciously. However, if the whole process is automated, this does not exempt it from these values, especially when it comes to legal regulations. If a process could come into conflict with these values, there is a duty of accountability and responsibility towards the legislator or even corporate authorities. The properties of explainable AI enable auditability, i.e. records and insights into the processes that can be referred to later.

Fairness

A major risk when using AI is distorted forecasts if the training data is not representative enough. In such cases, the nature of the production data may differ too much from the training data. In the case of personal decisions such as the granting of a loan, discriminatory biases relating to gender or ethnicity can even be reproduced. If this possibility exists, explanations are important so that the result of the algorithms can be contested at any time and corrected if necessary. Without the methods of explainable AI, such incorrect behavior of the systems often remains unknown.

Methods of explainable AI

There are many different approaches to achieving explainable artificial intelligence (XAI). On the one hand, the specific objective can provide for the traceability of decision-making processes and examine model accuracy - in other words, this concerns the technical requirements. On the other hand, understanding the results is aimed at human access to them. The specific methods can also be divided into two paradigms:

Ante-hoc explanations

For many use cases in companies, it is not necessary to set up complex black box models. On the contrary: if these do not fit the use case, the result is less precision and you also lose easy access to the required explanations. Instead, there are many classic machine learning models that can solve simple problems and are easy to explain due to their nature per se. One could also speak of White box models speak. This concerns, among other things:

Regression analysis

Whether linear, multiple or logistic regression: all of these methods examine relationships between variables. In many cases, they can be used to model specific, individual decisions, as well as the probability that a certain event will or will not occur. Other coefficients such as the confidence of a forecast are also included. This means that you not only get a blank result, but also further information on the respective forecasts, which also always remain the same for matching inputs.

Decision Trees

These models are characterized by their visual representability in the form of a tree diagram that depicts the entire decision-making process. Each branch follows an if-then logic that is easy for people to understand. In complex processes, the individual steps can be viewed and understood in isolation, which also facilitates troubleshooting. In addition, the rules applied can be extracted and analyzed. Decision trees are a good example of how explainability can be achieved through Transparency and Simplicity: is created.

Post-hoc explanations

It becomes particularly difficult when there is a lack of insight into the internal decision-making instances of the models. In these cases post-hoc - After model training, additional techniques are used to learn about the derivation of the results. This is where most publications and projects take place, as the recently intensively used deep learning models in particular require such elaborate approaches. These are often model-agnostic methods, i.e. they can be applied independently of the underlying architecture.

LIME

Local Interpretable Model-agnostic Explanations is probably the most prominent technique and was introduced in 2016 to approach black box models via a local, more easily explainable model. To learn about the unknown decision function of the model to be explained, LIME draws samples around the data point to be explained. These are fed into the original model to determine the effects of slight changes in the features on the result. This is repeated until a local, explainable model can be formed around the analyzed instance from the resulting data. This makes individual predictions comprehensible.

SHAP

Shapley Additive Explanations approaches individual forecasts via so-called Shapley values. These correspond to a common concept in cooperative game theory and specify "desirable" or positively or normatively interpreted target values depending on a coalition function. The outcome of the "game" - in this case the model prediction - depends on the participating players, who are to be understood here as features. The Shapley value measures the average marginal contribution of a feature across all possible combinations. In this way, it is possible to learn more about which combinations and which strengths of influence affect the result to be explained.

Grad-CAM

Gradient-weighted class activation mapping is one of the most important visualization techniques and, compared to SHAP, is particularly impressive due to its good applicability to convolutional neural networks (CNNs). Accordingly, it is also suitable for explanations in the field of image processing. The basis is the consideration of each position in the last convolutional layer of the deep network by calculating linear combinations of the activations from the previous layer. A visual map composed of the weights of the loss function gradients shows regions in the distribution of the activations that are significant for the final result - similar to a heat map. This provides information about the occurrence of image classifications.

Explainability for Large Language Models

The most relevant models now undoubtedly include large language models, which are used in countless AI systems for both business and private use - think ChatGPT. These are particularly complex neural networks based on the Transformer architecture. Due to their complex nature, their individual voice outputs are anything but easy to explain - classic techniques such as SHAP, for example, reach their limits due to the computing power required. However, since the significance and influence of LLMs are very high, the problem is logically the subject of scientific publications:

Zhao et alt. (2023) have developed an entire taxonomy of explainability techniques for LLMs. The two training paradigms - fine-tuning or prompting - are initially decisive for the choice of the respective method. The locally and globally applicable principles largely make use of already known approaches, but necessarily take into account the characteristics and special behaviors of LLMs, especially hallucinations.

One particularly common method is to analyze the activations that emanate from the attention modules of the transformers. This form of interpretation is discussed in various publications, with the question being raised, among others, whether this is explainable AI in the true sense of the word.

Meanwhile, various open-source tools are also available on GitHub to interpret LLMs or visualize word contexts:

• The Learning Interpretability Tool (LIT)
• Comgra
• Phoenix
• BertViz

Application areas

In recent years, XAI has developed from a niche topic to an indispensable part of technology. It is noticeable that the most relevant technologies (LIME, SHAP, Grad-CAM) were all developed around 2016-2017. During this time, the use and implementation of these technologies in the economy intensified in leaps and bounds. The risk and damage potential of AI errors increased, the importance of Trust and transparency which presumably provided the impetus for the development of the aforementioned techniques.

However, it is important to understand that explainable AI is only necessary for sensitive, risky decisions and processes. This can be approached probabilistically:

AI risk = (probability of error) x (potential effect)

It is therefore not surprising that the most important areas of application for explainable AI are primarily those that involve decisions with a direct impact on human life:

Medicine

AI systems are increasingly being used to optimize disease prognoses, risk profiles and diagnostics on the basis of very extensive medical data collection. In such matters, as little as possible should be left to chance. The final decision must for ethical reasons The responsibility for the treatment of patients lies with people - in many cases doctors, who are responsible for their patients and their treatment. Only comprehensible AI results are suitable as a basis for this.

Finance

The financial sector is a breeding ground for countless sensitive processes based on financial and personal data. Explainable AI is used for fraud detection, risk management and, in particular, the assessment of creditworthiness. Fairness and transparency are particularly important here, which is only possible through appropriate traceability. In addition, banks already have to comply with legally binding regulations by the AI Act get ready.

Justice

Here, too, the benefits of explainable AI are used to learn more about the basis for decision-making in risk assessment, including crime predictions. In addition, there are DNA analyses in the investigation of perpetrators, increased efficiency in case processing and much more. As the process results have a major impact on people, it is important to scrutinize and ensure that no discriminatory biases are reproduced or other serious errors occur.

Intelligent document processing is an area of application that plays a role in every industry and in processes of every sensitivity. Konfuzio is the partner at your side and enables data-based insights exactly where they make sense - with full legal certainty.

Conclusion

As explainable AI creates human access to complex algorithm-based decisions and forecasts, the approach has become a valuable part of technology development. Its importance has increased rapidly in recent years in line with the intensified use of AI. Since 2016, it has reached a level that has raised completely new questions about Responsibility and ethics in connection with AI. After all, the effects are now affecting large parts of the economy and society.

As a result, various publications and techniques such as LIME or SHAP were developed in order to better understand even complex black-box models and build trust. The necessity of XAI depends heavily on the sensitivity and scope of a process. In any case, it is essential to examine them carefully in detail and always act in the interests of the people affected. This applies today as well as for all future developments that we will go through in connection with this technology.

Are you concerned about the careful automation of sensitive business processes or would you like to use artificial intelligence in a transparent and comprehensible way? Contact us directly. Our experts will be happy to advise you.