Open Collections

An experimental investigation of the use of explanations provided by knowledge-based systems

University of British Columbia

Ever since MYCIN introduced the idea of computer-based explanations to the artificial intelligence community, it has come to be taken for granted that all knowledge-based systems (KBS) need to provide explanations. While this widely-held belief has led to much research on the generation and implementation of various kinds of explanations, there has however been no theoretical or empirical evidence to suggest that 1) explanations are used by users of KBS, and 2) the use of explanations benefits KBS users in some way. In view of this situation, this study investigates the use of explanations that are provided by a knowledge-based system, from the perspective of understanding both the specific factors that influence it, as well as its effects. The first part of this dissertation proposes a cognitive learning theory based model that both clarifies the reasons as to why KBS need to provide explanations and serves as the basis for conceptualizing the provision of KBS explanations. Using the concepts of the feed forward and feedback operators of cognitive learning it develops strategies for providing KBS explanations and uses them to classify the various types of explanations found in current KBS applications. The roles of feedforward and feedback explanations within the context of the theory of cognitive skill acquisition and a model of expert judgment are also analyzed. These, together with past studies of KBS explanations, suggest that user expertise, the types of explanations provided, and the level of user agreement are significant factors that influence the explanation seeking behavior of users. The dissertation also explores the effects of the use of KBS explanations in judgmental decision making situations supported by a KBS. It identifies and considers four distinct categories of potential effects of the use of explanations --- learning effects, perceived effects, behavioral effects, and effects on judgmental decision making. The second part of the dissertation empirically evaluates the explanation provision strategies in a laboratory experiment in which 80 novice and expert subjects used a KBS for financial analysis to make judgments under conditions of uncertainty. The experiment was designed specifically to investigate the following fundamental research questions: 1) To what extent are the various kinds of explanations used? 2)How does user expertise, the feedforward and feedback provision of explanations, and the level of user agreement influence the amount and the types of explanations that are used? and 3) Does the use of explanations affect the accuracy of judgmental decision-making and user perceptions of usefulness? Some of the major results relating to the determinants of the use of KBS explanations include:1) user expertise is not a determinant of the proportion of explanations used but influences the types of explanations that are used, 2) explanation provision strategy is a critical determinant of the use of KBS explanations with feedback explanations being used significantly more than feedforward explanations, and 3)the three types of explanations are used in different proportions with the Why and How explanations being used significantly more than the Strategic explanations. It was also found that the level of user agreement with the KBS had an "inverted-U" shaped relationship with the use of explanations. The least number of explanations are used when the level of user agreement is either very high or very low. The major results relating to the effects of the use of explanations include the following: 1) the increased use of feedback explanations improves the accuracy of judgmental decision-making but has no effect on user perceptions of usefulness, 2) the increased use of feedforward explanations while having no impact on the accuracy of judgments is positively correlated with user perceptions of usefulness, 3) the use of the Why explanation as feedback improves the accuracy of judgmental decision-making. As well, there was also evidence that the use of the KBS benefited both experts and novices. Considering that an understanding of the determinants and effects of the use of KBS explanations is a critical prerequisite for the design of KBS explanations, these and other findings of the study contribute both towards the development of a theoretical basis for the provision of KBS explanations, as well as the practical design of such explanation facilities.

Thesis/Dissertation

application/pdf

2008-09-16

For non-commercial purposes only, such as research, private study and education. Additional conditions apply, see Terms of Use https://open.library.ubc.ca/terms_of_use.

http://hdl.handle.net/2429/2072

Business Administration - Management Information Systems

University of British Columbia

UBCV

DSpace