UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

An empirical investigation of knowledge acquisition Chan, Christine Wai-Chi 1988

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


831-UBC_1988_A4_6 C42.pdf [ 5.55MB ]
JSON: 831-1.0051903.json
JSON-LD: 831-1.0051903-ld.json
RDF/XML (Pretty): 831-1.0051903-rdf.xml
RDF/JSON: 831-1.0051903-rdf.json
Turtle: 831-1.0051903-turtle.txt
N-Triples: 831-1.0051903-rdf-ntriples.txt
Original Record: 831-1.0051903-source.json
Full Text

Full Text

AN EMPIRICAL INVESTIGATION OF KNOWLEDGE ACQUISITION by CHRISTINE WAICHI CHAN A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE in THE FACULTY OF GRADUATE STUDIES Commerce & Business Administration We accept this thesis as conforming to the required standard THE UNIVERSITY OF BRITISH COLUMBIA 16 September 1988 © Christine Waichi Chan, 1988 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission. Department of ^m/ME/QCe <£ BuSfAJESS /flO^l/A//Sr/l^T/0Af The University of British Columbia 1956 Main Mall Vancouver, Canada V6T 1Y3 Date Poroses /o. /f<f<f r-\r /- / *> /ft -« \ ABSTRACT Expert systems are being developed despite the widely acknowledged problem of acquiring knowledge from experts. This study attempts to understand how knowledge acquisition is conducted in practice by investigating three expert system development projects. The variables examined include the expert, knowledge engineer, problem domain, organizational setting, the knowledge acquisition process, the expert system construction process, and the expert system itself. A case research methodology is adopted and data is collected through observation and taped protocol of knowledge acquisition sessions, post facto interviews with the participants involved, journalistic accounts kept by the subjects, and deliverables produced. Three cases on expert systems built in the domains of law of negligence, telephone line fault diagnostic, and wastewater treatment have been investigated. By juxtaposing the observations drawn from these cases with the findings reported in the literature, this inquiry contributes to the current understanding of the knowledge acquisition process. ii TABLE OF CONTENTS Abstract ii Table of Contents iii List of Figures . iv Chapter Introduction 1 1. Chapter Research Model 4 2. Chapter Research Methodology and Design 12 3. Chapter Case on the Remoteness Advisor 18 4. Chapter Case on the Test Desk Decision Support Tool 51 5. Chapter Case on the Wastewater Treatment Expert System 77 6. Chapter Conclusions 88 7. Bibliography 98 Appendix A 105 Appendix B 106 Appendix C 107 Appendix D 108 iii List of Figures Figure Page 1 Research Model: Manual Development of an 9a Expert System 2 A Validation-Based Life Cycle Model of the 89a Knowledge-Based System Development Process 3 KA Process for the Remoteness Advisor 91a 4 KA Process for the Test Desk Decision Support 91b Tool 5 KA Process for WASTES 91c 6 Summary of Characteristics of the Three Cases 9 Id D.l Physical Layout of Smith's Office 108a D.2 Types of Remoteness 108b D.3 Types of Activities 108c D.4 Risk of Harm 108d D.5 Risk 108e D.6 Harm 108f D.7a Structure of Nervous Shock Advisor 108g D.7b Structure of Negligence Advisor 108h D.8 Structure of Remoteness 108i D.9 Structure of Remoteness 108j D.10 Remoteness Advisor (Epsilon Project) 108k D.l l Structure of Remoteness Advisor 1081 D.12 Job Profile 108m D.13 Trouble Analysis Centre Current and Future 108n i v D. 14 Physical Layout of Demonstration Room 108o D.15 Draft Diagram of Test Desk Advisor 108p D.16 Agenda for Test Desk Course Task Analysis 108q Sessions, 30 Nov 87 - 4 Dec 87 D.17 Fault Analysis Flow Diagram 108r D.18 Redrawn Flowchart for No Dial Tone Type of 108s Trouble D.19 Sample Rule Base for the Test Desk Decision 108t Support Tool D.20 Conventional Treatment Plant Layout and 108u WASTES System Domain D.21a Racks.KB Logic Structure 108v D.21b Grit.KB Logic Structure 108w D.22 WASTES File Structure 108x v Acknowledgement I wish to thank my thesis supervisor Professor Izak Benbasat for introducing me to the field of knowledge acquisition and for his guidance throughout the various stages of the research process. I am grateful also to Professor Robert Goldstein and Professor J.C. Smith for their careful reading and comments on earlier drafts of the thesis. I am especially indebted to the participants in the three cases who shared with me their experiences in knowledge acquisition and system construction and gave generously of their time and insights. These individuals include Professor J.C. Smith, Cal Deedman, Jeffrey Howell, Dan Brophy, Marie Burlinson, Barry M. Chilibeck, and Dr. William Oldham. This work could not have been completed without their help. v i CHAPTER 1. INTRODUCTION Expert systems (ES) are widely acknowledged to be the fastest growing systems within Artificial Intelligence (Al) applications to reach the business and industrial sectors. The expert system portion of the Al market is estimated to climb to $350 million by 1990 [Sviokla, 1986]. Knowledge acquisition (KA) is an essential phase in the development of expert systems. It involves eliciting, analyzing, and interpreting knowledge that a human expert uses when solving a particular problem and then transforming this information into a suitable machine representation. This stage is important because the power and utility of the resulting expert system depend on the quality of the acquired expert knowledge which is later represented in the system [Kidd, 1987:1]. Knowledge acquisition is a major bottleneck in the efficient development of expert systems primarily because of the difficulty experts have in verbalizing their knowledge [Breuker et al.,1987; Abrett et al.,1986; Gaines, 1987d]. Experts generally find it difficult to formulate their knowledge explicitly but find it easy to demonstrate their expertise in specific situations [Witten et al.,1987]. The phenomenon is also referred to as the paradox of expertise [Johnson, 1984]. Despite the difficulty of accessing expertise widely acknowledged in the literature, expert systems are being built everyday. Dupont alone projects that by the year 1991, over 2000 systems will be in place. [Watkins et al., 1987] It has been 1 Introduction / 2 pointed out that most knowledge engineers use an "intuitive" approach to knowledge acquisition [Alexander et al.,1986]. This intuitive approach has in many cases succeeded in sufficiently covering the problem domain and providing a basis for expert systems that perform satisfactorily. So how did they do it? The objective of this study is to empirically investigate the knowledge acquisition efforts currently underway. The need for empirical research into expert systems development and use has been suggested in the literature. Gaines advocated more variety, detail and evaluation in "the case histories of applications of the techniques and tools" [Gaines, 1987e]. Referring to the lack of understanding of expert system implementation issues, Sviokla wrote as follows: ...more empirical evidence is needed from which more generalizable inferences can be drawn. Experience is still scant and generalizations are hard to create with confidence. [Sviokla,1986:363]. The study attempts to address this need. The question the study poses is as follows: How is knowledge acquired in developing an expert system? It is expected that the inquiry would shed light upon the process by providing a set of useful observations to guide future attempts on a practical level and data for hypothesis generation for research purposes.. The contents of the following chapters are outlined as follows: Chapter 2 presents a brief overview of expert system lifecycle models described in some existing literature, a model that guided the investigation, and some Introduction / 3 definitions of knowledge acquisition; Chapter 3 explains the research methodology and design; Chapter 4 reports on the case on the Remoteness Advisor; Chapter 5 reports on the case on the Test Desk Decision Support Tool; Chapter 6 reports on the case on the Wastewater Treatment Expert System; and Chapter 7 concludes the inquiry by presenting some lessons learnt, limitations of the study, and suggestions for future research. C H A P T E R 2. R E S E A R C H M O D E L Literature descriptions on the expert sj'stem development lifecycle provide some guidelines on the steps involved in acquiring knowledge and building an expert system. This chapter presents a brief overview of the expert system development lifecycle described in the literature, and explains a model of expert system development that has guided the identification of variables and the compiling of a questionnaire. In contrast to the popular acceptance of the system development life cycle (SDLC) which describes development processes of a conventional data processing software, general agreement in the literature on the development lifecycle for expert systems does not exist. A brief summary of some of the lifecycle models that have been advanced is presented in the following discussion. 2.1 B r i e f O v e r v i e w of E x p e r t System L i f e c y c l e Models. Buchanan et al.[1983] outlined the "major stages of knowledge acquisition" to include the following: 1. identification stage: to identify a. participants in the process and their roles, b. the problem domain, c. resources needed, 4 Research Model / 5 d. the goals of building the system. 2. conceptualization stage: to explicate the key concepts and relations in the problem domain, the result of this stage is a conceptual base for the system; 3. formalization stage: to map the key concepts, subproblems, and information flow characteristics isolated during conceptualization into more formal representations based on various knowledge engineering tools or frameworks. The result of this stage is a partial specification for building a prototype knowledge base. 4. implementation stage: this involves the mapping of the formalized knowledge from stage (3) to a representational framework associated with the tool chosen for the problem and development of a prototype; 5. testing stage to evaluate the prototype system. Buchanan et al. supported the notion of "rapid prototyping" [Gaines, 1987b] and considered it advisable to move directly to the formalization and implementation stages as soon as key concepts and relations had been identified. Their view is expressed as follows: It will be tempting to try to analyze the problem correctly and completely before implementing a trial system, but experience has shown that to be a mistake. Once the key concepts and relations are written down, much can be gained from formalizing them and working toward an initial implementation [Buchanan, 1983:144]. Grover [1983] described a KA cycle to comprise of the following three stages: 1. domain definition - this stage aims to produce a "Domain Definition Handbook" which consists of "a general problem description, a bibliography of reference documents, a glossa^ of terms, acronyms, and symbols, the Research Model / 6 identification of authoritative "experts", the definition of appropriate and realistic performance metrics, and a description of sample reasoning scenarios. 2. fundamental knowledge formulation - in this stage, scenarios chosen by the expert are reviewed to form a "baseline for minimum performance" which include the following : a. an ontology of domain entities, object relationships (classes) and object description; b. a selected lexicon; c. a definition of input sources and formats; d. an initial state description including static "background knowledge"; e. a list of human strategies or meta-rules which may become control rules for the system [Grover, 1983:437]. 3. basal knowledge consolidation - basal knowledge refers to the set of rules and definitions adequate to produce the lowest level of activity or system behaviour essential for maintenance of vital functions. At this stage, the fundamental knowledge corpus is reviewed and enhanced by appropriate reconstruction of rules. Issues considered include control strategies, rule sequencing, rule adequacy, rule consistency, and confidence values; additional experts maj' also be consulted. This stage is repeated with introduction of novel scenarios until the system performs satisfactorily [Grover, 1983:438]. The Guide to Expert Systems Program Management developed by Digital Electronics Corporation (DEC) outlined the knowledge engineering project to consist of the following steps: Research Model / 7 1. develop inital program definition; 2. build initial prototype; 3. develop program plan; 4. develop education plan and user involvement; 5. create design document; 6. build the basic shell; 7. develop adaptively; 8. conduct field test; 9. release product to production; 10. enhance and adapt the system; For a detailed discussion of the steps, see [Boose, 1986]. Fellers [1987] advanced a simpler model of the knowledge engineering process which consisted of two phases: knowledge acquisition and expert system construction. The former process involves "one or more knowledge engineers (KEs) interacting with one or more domain experts (DEs)", and the objective of this interaction is to develop a "shared representation" or model of the DE's problem solving process. Once the shared or external representation has been developed, the system construction phase begins. The KE takes the external representation and maps that to the actual knowledge base (KB). The KA process then continues with the expert interacting with the prototypical expert system to develop the internal representation or mental model of the system. The objective is to increase the congruence between this internal representation of the system and the mental model of the expert [Fellers, 1987]. Research Model / 8 Benbasat and Dhaliwal [1988b] propose a lifecycle model of the knowledge-based system development process which differs from other similar models [Buchanan et al., 1983, Fellers, 1987] in that "it specifically isolates, • at different stages of the lifecycle, three progressively more developed knowledge bases". The model consists of four stages : 1. modelling phase to capture knowledge from the source experts to a conceptual knowledge base; 2. knowledge elicitation phase which produces an elicited knowledge base, i.e., "a 'complete' expertise specification formalized to some selected knowledge representation (for example, rules, frames, or semantic nets) but in a nonmachine-readable form" [Benbasat et al., 1988b: 12]; 3. the system construction phase that maps the elicited knowledge base into an implemented knowledge-based system, and 4. experimentation and use phase which refines the knowledge base. Sviokla summarized the few ES project lifecycles he gleaned from the literature [Sviokla, 1986:354] and noted the absence of the specification or requirements analysis and maintenance stages in the ES project lifecycle [Sviokla, 1986:354]. In the DEC ES program management cycle, "build initial prototype" takes place immediately following "intial program definition." Implicitly, "rapid prototyping" is the norm. Rapid prototyping is a well-established technique employed in the development process of knowledge-based systems [Gaines, 1987b]. It describes knowledge elicitation to take place with the knowledge engineer interviewing the expert and Research Model / 9 then encoding the verbal knowledge into rules. Then through interative interviewing sessions in which the expert comments on and helps debug the prototype, the expert system knowledge base expands and is refined until satisfactory performance results [Young et al.,1987]. This approach is supported by the models described by Buchanan [1983] and in the DEC expert system program management cycle [Boose, 1986]. Focusing specifically on the KA process, Gaines illustrated it with a model of interaction, which is used to guide the initial identification of variables in this study. The model suggests the actors in the process to include the expert, the knowledge engineer (KE), and the users, and the software involved to include the expert system shell and the end product, a knowledge base. The study focuses on the two processes, primarily on the knowledge acquisition and secondarily on the expert system construction process. A slightly modified version of Gaines' model is shown in figure 1 [Gaines, 1987a]. Based on this simple model, the variables to be addressed include the following: 1. problem domain, 2. actors involved: a. expert, b. KE, 3. knowledge acquisition process, 4. expert system construction process, 5. potential users, 6. organization setting, Fi t . 1 Manual development of aa expert system Research Model / 10 7. expert system itself. A questionnaire covering these variables was compiled, which guided data collection. As can be seen from the questionnaire, the main emphasis of the study is on the knowledge acquistion process. For a copy of the questionnaire, see appendix A. 2.2 Knowledge Acquisition : Definitions. Knowledge acquisition has been defined in a variety of ways, some sample definitions that have been proposed include the following: "A process of gathering knowledge from one or more domain experts for the utilization in a knowledge based system" [Jacobson et al.,1987]. "The transfer and transformation of problem solving expertise from some knowledge source to a program. Potential sources of knowledge include human experts, textbooks, databases, and even one's own experience"[Belkin et al.,1986]. An attempt to differentiate knowledge acquisition and knowledge elicitation has also been made as follows: "Elicitation is the process of developing a model by interviewing the experts and observing their environment; it is theory formation or model design. Acquisition is the process of collecting the detailed information (facts) that fit into the framework defined by the model" [Addis, 1987]. The distinction between the two terms is not observed in this study and the definition of knowledge acquisition adopted is stated as follows: Research Model / 11 "Knowledge acquisition consists of the elicitation and interpretation of data on the functioning of expertise in some domain, in order to design, build, extend, adopt or modify a knowledge based/expert system (KBS). In this view, knowledge acquisition is a permanent and crucial activity throughout all stages of designing, implementing, and maintaining an expert system" [Anjewierden,1987; Breuker et al,1987J. Hence, in the following discussion, knowledge acquisition involves not just elicitation of knowledge, but also analysis and interpretation of the acquired expertise. CHAPTER 3. RESEARCH METHODOLOGY AND DESIGN To study the KA and ES construction processes, a case research methodology is adopted. The purpose of this chapter is to explain why a case research methodology is appropriate for the present study and describe the research design. 3.1 Research Methodology. Bonoma [1985] suggests case research to be appropriate for investigations into areas "where the existing body of knowledge is insufficient to permit the posing of causal questions, and when a phenomenon cannot be studied outside the context in which it naturally occurs" [Bonoma, 1985,207]. Benbasat et al. noted three reasons for the use of case research as follows [Benbasat et al.,1987:370]: 1. the researcher can study information systems in a natural setting and learn about the state of the art, and generate theories from practice. 2. the case method allows the researcher to answer 'how' and 'why' questions and to understand the nature and complexity of the phenomenon being studied. 3. a case approach is appropriate for an area in which few previous studies have been carried out. An empirical investigation into how an expert system is developed satisfies the criteria for case research advanced in the above studies. The lack of scientific and engineering foundation for expert systems has been suggested as the major 12 Research Methodology and Design / 13 obstacle in establishing professional standards for expert system development, and the lack of well-established techniques for knowledge engineering poses a major problem for new development teams [Gaines,1987a]. The phenomenon of interest, i.e. the acquisition process, takes place in organizations that currently develop such systems; an inquiry into the undertakings would enable the researcher to answer the question of how the systems are produced. Bonoma poses the four stages in the case research process to be drift, design, prediction, and disconfirmation [Bonoma, 1985:206]. This investigation focuses on primarily the first stage and aims to understand and describe. From the narration, lessons drawn from the cases form the basis for future hypothesis generation. The procedure closely approximates Bonoma's statement on the objective of case research: The goal of data collection in case research is not quantification or even enumeration, but rather (1) description, (2) classification (typology development), (3) theory development, and (4) limited theory testing. In a word, the goal is understanding [Bonoma,1985:206]. 3.2 Research Design. In contrast to the scant work on expert systems reported to date which deals with single cases or action reports [Benbasat & Nault, 1988], this study attempts to empirically investigate three expert system development projects. Adopting an empirical approach to observational methodology [Weick, 1968:401], the questionnaire outlined the variables to address when the researcher observed the Research Methodology and Design / 14 sessions for the first two cases and interviewed the subjects for the third. 3.2.1. Unit of Analysis and Site Selection. The unit of analysis consisted of the knowledge acquisition process for an individual expert system. In practice, however, a clear distinction between this and the development process often cannot be made. Therefore, whenever necessary, the unit of analysis is extended to include the development process as well. The sites were selected based on two criteria : 1. an expert system was being developed, 2. permission to conduct the study was procured. Since May of 1987 when the researcher embarked upon the study, 7 development projects have been identified and approached. One was later found to be an unsuitable candidate for the study because it involved an intelligent front end for an existing system rather than an expert system. Another system was being developed in a commercial context and the company involved was unwilling to co-operate. Two other systems were shelved two months after the researcher had started collecting data from them. The three cases finally investigated include the Remoteness Advisor developed at the Law Faculty of the University of British Columbia, the Test Desk Decision Support Tool developed at Microtel Learning Services Centre, and the Wastewater Treatment Expert System developed at the Civil Engineering Department of the Universitj' of British Columbia. In two of the three cases, the researcher learned of the systems Research Methodology and Design / 15 through publications or word of mouth, and obtained approval to conduct the study from the participants. In only one case did the persons involved volunteered their project to be studied. 3.2.2. Data Collection. The study aims to track the knowledge acquisition and development processes as they occurred. This allows the researcher to study the phenomenon within its natural setting and also ensures maximum currency of data. Bonoma noted the major weakness in case research to be low data integrity [Bonoma, 1985:200]. Triangulation of data is a technique, which uses different types of data to capture a single event. To increase the validity of data, this study adopted multiple data collection methods which include the following : 1. direct (unobtrusive) observation of the knowledge acquisition and development processes between the expert and knowledge engineer (KE); 2. taped protocols of the sessions; 3. retrospective interviews with experts and KEs; 4. diary or journalistic record of events kept by the KE; 5. documentary evidence or deliverables produced during the process. Since subjects' co-operation was essential, the above list was consistently presented to the KEs in the three projects. In two of the three cases, they agreed to co-operate on all methods except keeping a diary. In the third case, since the researcher did not learn of the project until it was almost finished, only post facto interviews were conducted with the expert and KE, and some Research Methodology and Design / 16 deliverables were also obtained. A major weakness in observation methods is its susceptability to biases introduced by the observer [Weick,1968:428]; having multiple observers could potentially reduce the error. This proves impossible in the present study for subjects often were relunctant to accomodate more than one observer while they worked, whether within the same or across different sessions. While the researcher began with the intent of observing the interaction between the KE and expert in an unobtrusive manner [Webb et al.,1966], this approach was likewise found to be infeasible. Weick correctly suggested that a non-participant observer causes reactions on the observed subjects [Weich, 1968:370], and that some interference might be needed to maintain rapport [Weich, 1968:311]. Often the subjects directed comments to the researcher during the sessions but such interactions were kept to the minimum as much as possible. A method adopted to "to enhance precision and validity of observational studies" is to introduce interference in the observation setting [Weich, 1968:366]. In this case, a tape recorder was used. Subjects proved to be unaffected by its presence, and were exceedingly co-operative in helping to tape the sessions whenever the researcher could not personally attend them as long as sufficient tapes and a machine were provided. For a sample of the transcript of some taped protocol, see Appendix B. Interviews with the KE and the experts were the main source of data in the case on the Wastewater Treatment Expert System. The interviews were Research Methodology and Design / 17 semi-structured and the questionnaire guided the discussion. Interviews conducted for the other two cases were primarily for clarification purposes. A diary form was developed and used successfully in the two cases in which the projects were shelved. In the other cases, the KE was reluctant to keep a diary for lack of time. See Appendix C for a copy of the diary form. Some deliverables were obtained and proved to be indispensable for the researcher was initially completely unfamiliar with the three domains addressed by the systems. All figures referenced in the narrative description are included in Appendix D. The data collection methods were adapted for each case depending on circumstantial factors and willingness of subjects to co-operate; this will be described in greater detail in the following chapters. CHAPTER 4. CASE ON THE REMOTENESS ADVISOR 4.1 Introduction. The Remoteness Advisor is an expert system developed by Joe Smith and Cal Deedman at the Law Faculty of the University of British Columbia. Smith, the expert, is a professor of law at the Faculty who wrote a book on Liability in Negligence [Smith, 1984], in which he laid out a coherent theory of negligence developed after 23 years of teaching, research, and writing in the area of the law of negligence. Deedman, the knowledge engineer (KE), is currently pursuing a doctorate in the interdisciplinary program of law and computer science. He is also a lawyer who has practiced for 10 years and decided to return to school. The two had previously co-operated in developing another expert system called the Nervous Shock Advisor (NSA), which assists the user in determining if a nervous shock case can be established [Deedman, 1986]. Deedman learned the importance of developing the knowledge structure before starting to code, otherwise, much time is wasted changing the code. The ss'Stem was used by Teknowledge Inc. as a demonstration system at the AAAI Conference in Seattle in July of 1987 and gave both developers their first experience of building a system. 4.2 Negligence: Definitions. A definition of negligence is quoted in Smith[1984] as follows: 18 Case on the Remoteness Advisor / 19 Negligence is the omission to do something which a reasonable person, guided by those considerations which ordinarily regulate the conduct of human affairs, would do, or doing something which a prudent and reasonable person would not do [Smith, 1984:129]. A central issue in negligence is the problem of remoteness, i.e. how far down the chain of causality could a person be liable for a negligent act so that a damage would be recoverable. Negligence usually involves the creation of risk of harm to someone. The injuries of the primary subject of the risk are generally not too remote; the remoteness issue arises when other persons suffer a loss as a result of the negligence to the primary subject [Smith, 1984:127]. Foreseeability is that which is natural and probable. Foreseeability of harm is a necessary condition for culpability and consequently for the liability derived [Smith, 1984:132]. The obvious case of foreseeability and culpability is where the harm caused is the very intent of the actor, in which case the actor is held liable for the intentional causing of harm by the law [Smith, 1984:130]. So for any negligent act there are certain harmful effects which are foreseeable, which when they materialize as a result of an action, the remoteness issue is not involved. There are other effects, however, which could not be said to be reasonably foreseeable, and it is in regard to these that the issue of remoteness of damage may be raised [Smith, 1984:132]. Traditional legal doctrine has posited two tests on when a damage is recoverable. The first is to allow recovery for damage foreseeable as probable, and the second is to allow recovery for damage foreseeable as possible. Basically, the Case on the Remoteness Advisor / 20 judge can choose between the two tests based on his discretion. If his decision is to award recovery, the latter test can be applied for anything is possible; otherwise, the former is used. Smith describes this situation as follows: "Foreseeability...becomes the justification for a conclusion about remoteness reached on other grounds, rather than the real test or basis of the decision" [Smith, 1984:120]. Smith argues instead that a rational and systemactic basis for predicting recovery in remoteness cases can be formulated which allows us not to have to "swing between the poles of the direct causation and the foreseeability test, or between the poles of the 'foreseeable and probable' test and 'foreseeable and possible' test" [Smith, 1984:131]. And he believes this rational approach is in fact used by most judges : "It probably is the instinctive feeling of judges which is the bottom line in remoteness cases. The fact that...the instinctive feeling of judges has a 90 percent conformity would indicate that some basic moral principles about responsibility and blame are at work at an unconscious level. The intention has been to articulate that principle in the form of the following restatement of the foreseeability rule : Damages resulting from a negligent action are not too remote if they are reasonably foreseeable in the particular, or are one of the reasonably foreseeable class of injuries [Smith, 1984:160]. Hence, Smith proposes a new test for remoteness which involves two concepts : (1) damages foreseeable in the particular, and (2) damages that are one of the reasonably foreseeable class of injuries. 4.3 Data C o l l e c t i o n Methods. Knowledge acquistion for the Remoteness Advisor took place at Smith's office at Case on the Remoteness Advisor / 21 the Law faculty of UBC. This office is a room about 10' by 12' (see Appendix D, figure 1). Smith and Deedman, usually worked in front of the flip chart, either standing or sitting. Smith regularly paced around during the discussion. There were a total of six KA sessions, each took on average 1.5 hours. The first KA session took place on October 8, 1987 and the last on February 4, 1988. At all except one session, the researcher was present to observe as well as tape record the discussion between the expert and the KE. The one session when the researcher was absent, Deedman taped the session. Before or after the sessions and at occasions when the researcher found it difficult to follow the legal jargon, independent or joint interviews with the expert and the KE took place. The following is a narrative description of what transpired at each KA session. At the end of each description, some observations the researcher had made during the session plus relevant comments made by the participants during the interviews are presented. 4.4 Knowledge Acquisition. 4.4.1. KA Session I. At the first KA session on October 8, 1987, two distinct topics were discussed; they included structure of the sj'stem and presentation format. Structure of the System. Case on the Remoteness Advisor / 22 The structure of the expert system was different in the minds of the expert and the KE. They had obviously spent time pondering about the issue before coming to the first KA session. It was equally obvious that they approached it differently. Deedman suggested posing a question at the start to ask if there had been damage suffered as a result of a negligent act. Smith however, chose to bypass the difficulty for most users of trying to determine if they had a remoteness problem. He avoided words like "reasonably foreseeable" and instead posed questions which guided the user to think in concrete terms such as the activity which caused the damage, the damage itself, and the risk that justified the possibility of positively concluding causal relationship between the activity and the damage. The structure underlying this approach was to find out if the damage was foreseeable in the particular, in which case there would be no remoteness problem. For example, it is reasonably foreseeable that if a plank is negligently dropped into the hold of a load ship, the cargo would be damaged [Smith, 1984:132]. If the damage was not foreseeable in the particular, then remoteness may be involved and the problem would be further clarified in detail. The structure of the system was a giant categorization scheme, hence, a major portion of the time during the KA session was spent on clarifying the activity, damage, and risk involved in a case. A rudimentary theory of negligence is in the appendix of [Smith, 1984] ( see Appendix IV and V of [Smith, 1984] ). Deedman, having read Smith's book, suggested adding some new categories to the basic scheme. For example, a new categor}' of radioactivity was added to the list Case on the Remoteness Advisor / 23 of causes of damage which previously only had an item on inflammable substances. They discussed how different items could be classified. For example, Deedman asked whether an animal-drawn vehicle ( which was involved in the negligent activity ) should be categorized as vehicles, or as negligent supervision of animals. Smith decided since the system was a negligence advisor, negligence was assumed, so a category of damage caused by animals should be sufficent. Deedman also posed questions to clarify Smith's definitions. Smith had one item called "high voltage electricitj'" and Deedman suggested simply calling it "electricity". Another category was termed "dangerous objects" which they agreed was too vague and was subsequently changed to "weapons". Format of Presentation. There was substantial discussion on software design issues. They considered different formats for presenting the list of possible causes of damage to users. Deedman suggested listing the options exhaustively and then split them up for menu selection because it was not "intuitively appealing to have animals and radioactivity together." Smith responded by noting that the options overlap so simple yes/no answers was insufficient. The user should be able to choose say option one, two and any number of others. For that reason, he preferred keeping all alternatives on one level on the screen and let the user put together any combination. Then Deedman decided that focusing on one piece of information at Case on the Remoteness Advisor / 24 a time would be easier to prototype than a menu selection. Smith, on the other hand, pointed out it would be more elegant if the user chose motor vehicle and the system responded by posing the options under this category, but if the user chose weapons, then nothing more was asked. Finally, it was decided that the categories would be presented one at a time, and cases that would be generated as a result of one option would include cases where others were involved so multiple selection would not be necessary. Observations & Post-Session Interview with KE. The researcher made the following observations and asked the KE for verification at a post-session interview on October 23, 1987. 1. Style of interaction - It was obvious to the researcher that Smith and Deedman had a close working relationship. Often they finished off sentences for each other. Deedman agreed in the interview that since he was a lawyer by profession, he and Smith had a shared vocabulary which was a considerable advantage for him as a KE. 2. Conflicts of opinion - Conflicts of opinions arose but were resolved when each tried to understand what the other meant. On issues dealing with the domain of negligence, Deedman deferred to Smith's opinions but Deedman's suggestions and queries often clarified the matter in the expert's mind. Deedman conceded in the interview with the researcher that the gap between their expertise on that topic was "enormous." On issues where they differed in opinion, he relied on Smith's definitive statements after making sure Smith had understood what his point was. Deedman tried to Case on the Remoteness Advisor / 25 pose hypothetical cases to test Smith's categorization scheme and he used current news topics such as AIDS, the space shuttle disaster, and danger of pit bull terriers. He needed to attach Smith's ideas to concrete examples in order to make the ideas meaningful to him. 3. Presentation details discussed - Although the purpose of the KA session was explicitly to clarify on a conceptual level the domain of negligence, much discussion was on presentation details. Smith had definite ideas on what the resulting system would be like and often suggested to Deedman design details on presentation formats. Deedman, on the other hand, admitted he had no idea what the system would look like. 4. The KE further pointed out it was impossible to create rules at this stage. They were primarily investigating the options by posing if-then situations. Case on the Remoteness Advisor / 26 4.4.2. KA Session II. The second KA session for the Negligence Advisor took place on 29 October 1987, exactly three weeks after the first session. Smith, Deedman, and the researcher met at Smith's office. Once again, their discussions focused on the issues of the structure of categories, and presentation format. But at this session, it was difficult to separate out the three issues as they were closely interwined. Hence, they will not be separately discussed in the following presentation. Deedman began by first summarizing what was accomplished in the previous session. First, they agreed to have the Remoteness Advisor establish if negligence was involved by asking whether it was a negligent or deliberate act. Then there would be an introduction to each category of possible causes of the damage and the user would be led through the categories. Secondly, if negligence was involved and the damage was caused by one of the combinations of these categories, then there would be recovery. Lastly, the format of presentation would be to present one category at a time, and multiple selection would not be necessary because cases that would be generated as a result of one option would include cases where others were also involved. For example, the list of cases retrieved on the basis of "explosives" would include a case involving "a truckload of explosives," which would appear under both the category of "explosives" and that of "transportation," specifically, the sub-category of "truck". Deedman had thought about what happened during the three weeks and came up with a few questions for the expert. Basicallj', he questioned if the categories Case on the Remoteness Advisor / 27 they discussed covered the whole field of remoteness. Smith affirmed that they did. The only qualification was in the case of nervous shock. Since it was handled by another system, the Nervous Shock Advisor, they did not have to worry about it. Furthermore, anything that did not fall into these categories would be recoverable provided it was foreseeable in the particular. Deedman then suggested having a screening mechanism to determine if it was remote before leading the user through the categories. Three levels of questions would be posed : first, if negligence was involved, secondly, if remoteness was involved, and thirdly, if the cause of damage could be captured by the categories. If it could not, then the cause was too remote and the damage not recoverable. Smith preferred to take the user through the categories without first concluding it was too remote and then see if the cause fell within the risk. In other words, the front end screening mechanism Deedman suggested was to be put at the back. Deedman thought this approach might have the system asking about cases where no remoteness issue was involved and the user needed not have bothered with answering all the questions. Smith agreed. Using this approach, the user would be told at the end s/he could not recover because the damage fell within the risk. In most cases, smart lawyers would know there was no remoteness issue if the damage was foreseeable in the particular. In which case, they should not even use the system. (He said : "this is to catch the dummy at the end." ) Deedman agreed and they moved onto discuss the categorization scheme. Case on the Remoteness Advisor / 28 In discussing the categories of causes for the damage, Smith and Deedman were wrestling with the underlying structure of the expert system. This perhaps explained the large percentage of time devoted to the topic. In the previous KA session, they already noted down a number of causes. They started by reviewing the categories and discussing various detailed issues such as ordering of the categories, specialization within general headings, groupings of various items, and presentation formats. Means of transportation constituted the major group of causes for damage, hence both Smith and Deedman began their discussion with this cause. Smith asked what the specific negligent acts involved in the operation of means of transportation were that caused the damage. The specific categories under means of transportation were dictated by the cases and these included vessels, railway (with the sub-category of engines), motor vehicles (with the sub-category of cars), and planes. Then they considered the ordering of the categories. Deedman suggested putting machinery behind motor vehicles because thej' both belonged under "mechanization." Then, fire and inflammable substances, followed by toxic substances, electricity, and radioactivity, which was "a modern form of electricity." Presentation of the categories was also addressed in the discussion. Smith preferred posing an over-riding question such as : "was the negligent act that caused the damage related to dangerous activity such as explosives, fire, Case on the Remoteness Advisor / 29 machinery, etc." And a list of these would follow. Deedman pointed out it might not be good to present the user with too much information because while it might "take them to the end faster", directing people's minds to individual things might be easier for the user. He then suggested a compromise : instead of going through each eategory, group them together in pairs by asking explosives OR fire, electricity OR radioactivity, etc. If the user chose one or the other, the system could branch to that, if s/he chose neither, s/he simply continue on. The rationale for this presentation method was to allow the user to proceed in an incremental manner. He compared it to talking on the phone, when one would not "list but ask them was it explosives or fire." Smith disagreed noting actually on the phone, one would simply "ask them what it was." Deedman agreed. Smith apparently changed from an earlier position at this point. He had earlier remarked that the categories were discrete and that "nothing needed anything else." At this point, however, he noted that "overlap existed." For example, in a case where motor vehicle was the cause, thin skull or rescuer might also be involved. There were two major divisions : dangerous versus non-dangerous activities. The latter such as thin skull or medical complications apply to any negligent act. Smith then pondered on the possibility of starting with these generally applicable categores and "get them out of the way." Deedman encouraged him in this endeavor by posing more scenarios that involved combinations of categories such as a car crashing into explosives, or a train carrying radioactive stuff. Deedman suggested ordering by starting with the "most devastating." Case on the Remoteness Advisor / 30 Smith proposed a solution. He decided to "avoid overlap by getting the right order," which meant to begin with clear cut cases of recovery and then proceed with the categories. The order he put down was as follows : 1. nervous shock, which would exit to the nervous shock advisor; 2. rescuer, who would always recover; 3. thin skull, or victim with particular susceptibility; 4. medical complications; since they usually happened away from the actual event, they should be last. Deedman noted down this ordering. The system would elicit information from the user by posing questions such as "was the victim or did the victim" fall into one of these categories. Relevant cases would then be retrieved. Hence, the structure that emerged had the system go through the four categories, if none of these was invoked, it would proceed to the categories discussed earlier. Deedman flipped back to the pages on the categories and re-ordered them. He noted at this point that two major groupings emerged : 1. special types of people and 2. special types of activities. And the. structure became clear in his mind : "the law allows recovery if it is not too remote, if it is remote, then it needs to be special types of people or special types of activity. So it is foreseeable if it is a particular class of person or particular class of activity." Based on this, he reordered the categories under special t3rpes of activities on the flip chart as follows : 1. motor vehicle, 2. machinery, Case on the Remoteness Advisor / 31 3. explosives, 4. fire 5. inflammable substances, 6. toxic substances, 7. electricity, 8. radioactivity, 9. weapons - firearms, others, 10. animals. Smith consented this was the skeleton of the system. Observations & Post-Session Interview with Expert and KE. The most interesting point about this KA session was the gradual emergence of the underlying "skeleton" for the expert system as the discussion developed. It seemed that the overall structure emerged despite the explict focus of the discussion on specific details within the framework. The researcher also found the method they used of bypassing the difficult and abstract and always posing concrete questions to the user remarkable. At the post-session interview, Smith further elaborated upon this method by emphasizing they left out legal rules, philosophy, and anything disputational out of the system. In doing so, they avoided the risk of users not understanding the carefully formulated and articulated questions on legal philosophy. Rather, the system posed questions generated by the philosophy pertaining to the conrete situations. Deedman pointed out that users, being lawyers, were not interested in Case on the Remoteness Advisor / 32 theory but only in cases to give them some bases for making successful arguments in court. The essence of their approach was to ascertain facts in cases by relating them to precedence which had similar facts. To do that one had to reason by analog}' to find alike or relevant cases. This implied the need for a criteria of relevance. All this was done outside of the system. Smith summarised their approach succinctly when he said : "the kejr to our success is to capture reasoning by analogy in heuristic rules and that is where the dispute takes place and you trust the expert on that." Ultimately, the empirical test was whether the user obtained the right cases. If the group of lawyers agreed the system presented to the user the relevant cases and nothing was missing, then the system had performed satisfactorily. The domain of law is distinctive in that the set of court cases which embodies all the facts to be considered is readily available. In other words, the scope of the cases relevant to the domain is clearly defined. 4.4.3. KA Session III. The third KA session for the Remoteness Advisor took place on December 7, 1987. Smith and Deedman met at the former's office. The researcher was absent and Deedman taped the session. The First Categorization. Case on the Remoteness Advisor / 33 Deedman started the session by reviewing what was accomplished the last time they met. They concluded there were two basic categories : 1. special types of people, which included rescuer, thin skull, nervous shock, and medical complications and, 2. special types of activities, which included weaponry, firearms, toxic substances, animals and motor vehicles, etc. Based on the conviction that the categorization was derived from the cases, Deedman had gone through the cases again to see how well they fit into this scheme. He came away with two proposals. Firstly, motor vehicle was a major category but Deedman thought a more generic category of transportation may be more appropriate because motor vehicles could be considered a subset under means of transportation. Secondly, transportation cases could be broken down into four groups: 1. motor vehicles, 2. railways, 3. animal-drawn, 4. vessels. Deedman wanted to get Smith's reactions to his suggestions. Smith thought that was fine but would like to add the category of "other" to capture transportation cases dealing with snowmobiles, bicycles, or other "far-fetched ones." Deedman remarked there were no cases on airplanes. Smith agreed and suggested handling that by having the system inform the user at the end that there were no remoteness involved in plane cases. A general introductory question would take them into the transportation category. Case on the Remoteness Advisor / 34 After agreeing on the general categories, they continued to discuss more detailed breakdown within each class. If the user responded positively to the general question on transportation, another question would ask which type of transportation was involved. The user could choose among the four plus the "other" categories. The cases within the motor vehicle category included cars, motorcycles, trucks, buses, vans, track and trailers, trailer snowmobiles, etc. Deedman counted that among the 58 motor vehicle cases, nine were on trucks, four on buses, one on motorcycle; etc. Among the railway cases, four were on the whole train, two on railcars, which included one on trams, and one on electric street cars. There were two carriages or animal drawn cases and three cases on boat or steamships, which would fall under vessels. A large number of cases in one category might indicate the need to further break it down into sub-categories; e.g., 58 motor vehicle cases indicate that sub-categorization may be useful. A Second Categorization Scheme. At this point, Smith proposed a second categorization scheme. Instead of looking at the nature of the vehicle, a second categorization scheme that cut across all vehicles was needed. Since these were all cases that deal with the remoteness issue, negligence was assumed. Given negligence, there was a primary risk to somebod3\ Hence, for each case, they needed to clarify the following : 1. What the risk was; 2. Did the risk materialize or not; 3. Was the damage or injury in question the same as that expected in the Case on the Remoteness Advisor / 35 risk; 4. Was the plaintiff at risk or was it somebody else. Smith felt that this second level categorization, which was independent of the kind of vehicle involved, was important and needed to be clarified on top of the first level categorization that Deedman had formulated. It would specify the relationships that cut across all motor vehicle categories. To illustrate his point, Smith gave the example of a plaintiff who was a rescuer hit by a motor vehicle. Then, regardless of whether he was rescuing on a bus, van, or motorcycle, the four questions on the risk, the damage, and the plaintiffs relationships to them needed to be answered. Therefore the3' should go through all transportation cases, and identify the plaintiffs' relationship to the risk in order to seek the patterns among them. Smith and Deedman sat down together and leafed through a thick book of cases to weed out irrelevant ones. The case book includes 94 cases and was compiled based on the criteria presented in Liability in Negligence [Smith, 1984]. As they were reviewing the cases, thej' noticed there were two types of remoteness which they had started out thinking were the same. First, given any type of car, was it too remote that this person suffered harm? Secondly, was it this particular kind of harm that was remote? Smith suggested thej' go over all cases on means of transportation and identify the plaintiffs' relationships to the risks and classify them in order to typify the kind of remoteness. This would reveal the basic structure of the knowledge base. Case on the Remoteness Advisor / 36 Observations and Post-Session Interview with the KE. 1. Since the researcher was not present at the session, she needed some explanation about points raised at the KA session. In the post-session interview with Deedman on December 15, 1987, he further explained the two types of remoteness: first, remoteness for victim, and secondly, remoteness for damage (see figure 2). The permutations seen in the cases were derived from these two basic types. A remote damage case involved an unexpected harm. If the harm occurred to somebody other than to one whom you expected then that would be a remote victim case. He cited the following example to illustrate this distinction. A driver hit a fire hydrant and water from the hydrant flooded stocks at the basement of a nearby store. The victim was not remote since s/he was in the proximity of the accident; but flooding damage resulting from a car accident was unexpected. Hence this would be a remote damage case. Another example was: a car hit a fence around the farmer's field, the staples on the fence popped into a field and some cows grazing on the field ate them. The cows died and the farmer who owned them was compensated in court. This would be a remote victim case because usually a car accident victim would be somebody in its vicinity and not a farmer physically far from the scene. The distinction between the two types of remoteness was derived from reading the cases. Deedman also felt the time was right to start prototyping. So far at the KA sessions, they had discussed in conceptual terms but discussion could become "interminable" and the KE believed it best to embody what Case on the Remoteness Advisor / 37 structure they had in a concrete object. He was in complete agreement with Weiss and Kulikowski [1984] who stated : "The single most important piece of advice that one can give a model designer is to build a prototype model as soon as possible. Because expert reasoning problems are frequently poorly specified, one needs to have something concrete to view and 'lay hands on.' It is particularly important for the expert to see something running early. A running program is worth thousands of words from an unformalized interview with the expert" [Weiss et al.,1984:106]. 2. It is significant that Smith proposed a second categorization which captures the causal relationships among the risk, damage, and victim. This scheme of relationship is distinct and different in nature to the first superset-subset categorization implicit in the means of transporation and its sub-categories. 4.4.4. KA Session IV. Case on the Remoteness Advisor / 38 The fourth KA session for the Remoteness Advisor occurred one week later on December 14, 1987. Smith, Deedman and the researcher met at Smith's office. Structure. The expert and the KE continued their struggle with the underlying structure of the system. To recapitulate, they had decided at the previous meeting there were two general categories : 1. special types of people and 2. special types of activities. And lists of both had been compiled. Smith and Deedman had obviously thought about the problem since they last met. Deedman suggested there were three key labels and wrote his ideas on the flip chart as follows : ACTION - known identifiable (smoking) RISK - of particular type of harm (fire explosion) VICTIM - person (bystanders at the scene) - property Smith added "HARM" to the chart. He felt that harm was a distinct category because he wanted the system to ask the user about both risk and actual harm. Case on the Remoteness Advisor / 39 The remoteness issue arose when what happened fell outside the risk of negligence. Smith added that harm might be to person, property, and economic. Deedman suggested the traditional division would be just between personal and property, and economic loss was not "on the agenda." Smith however, saw this as "a map," and if economic loss was chosen, the user would be directed to another system. Smith envisioned the system to be collecting information through the first series of questions which required the user to choose among the categories. Then the system would digest the collected information and "shoot" the user into series of questions on the basis of the digested answers. Using the labels on the flip chart as items on the underlying scheme, a series of questions could be posed and from the answers obtained, much information would be gathered. But first they had to make up the lists of action and harm. Since the list of types of activity (action) was ready-made from the previous session, Smith recorded the following question : did activity which cause the harm involves the use of any of the following? And then he copied the ready-made list under it (see figure 3 ). For risk of harm, Smith put down on paper the following : (see figure 4) He posed the question of whether the plaintiff was outside or within the risk, whether the risk materialize or not, and various combinations of these options. As he was talking, he drew the following flowchart on paper : ( see figure 5 ) Case on the Remoteness Advisor / 40 For victim, the question was whether the plaintiff was rescuer, nervous shock victim, medical complication, or thin skull. For nervous shock victims, the Nervous Shock Advisor would be loaded. And Deedman put down on the flip chart the list of victims as follows: 1. rescuer, 2. nervous shock, 3. medical complications, 4. thin skull - mental or physical. For harm, he put down the following : 1. mental, 2. physical, 3. thin skull. After some more discussions, Smith drew a different version and threw out the above. He put down under actual harm the following : 1. to person a. mental b. physical -1) thin skull, 2) medical complications, 2. to property, 3. to both, (see figure 6) By posing questions under these categories, all the necessary information would be gathered and rules based on the information can be formulated. Case on the Remoteness Advisor / 41 At this point Smith showed the researcher a chart of categories for the Nervous Shock Advisor (see figure 7). He explained that the 255 rules of the Nervous Shock Advisor stood behind the structure diagram. And a similar diagram for the Remoteness Advisor was their present objective. Deedman pointed out the diagram had been derived from the exercise of developing the system. But Smith felt that if they could arrive at a structure diagram before developing the system, Deedman could work backwards from the structure and expound it into a massive set of rules and questions. In this way, the process of KE would be accomplished in half the time it took for the Nervous Shock Advisor. Or alternatively, if, they could arrive at the structure and work in both directions, the availability of the structure would help clarify to the students working on the knowledge base of cases what information to look for. Returning to the issue at hand, Deedman remarked the risk of harm looked fine to him. However he thought actual harm and whether the harm materialized were different approaches to the same problem and suggested that Smith try to integrate the two. Smith agreed to attempt that and they both decided to go away and think about the problem some more. Observations & Post-session Interviews with Expert and KE. 1. In the post-session interview with Smith, he explained further the structure diagram. This structure diagram replaced a tree diagram which was the usual mechanism in clarifying knowledge for expert system development. He preferred a structure diagram because too much overlap existed among the categories and a tree diagram could not adequatel}' represent the overlaps. Case on the Remoteness Advisor / 42 The structure could be converted to a tree, but there was "no point" in doing so. The structure chart summarized everything the user needed to know about the cases to fit them into the 255 rules of the system. By following the instructions and checking off the appropriate categories for each case, the user in fact identified the key variables in the case and link it to the applicable rules. They presently aimed at developing a similar chart and then developing the rules from it instead of doing the reverse as they had done for the Nervous Shock Advisor. 2. In this session, the expert and the KE tried to arrive at the second level categorization that Smith thought was necessary in the previous KA session. The researcher observed that this attempt had not been completely successful as the second level categoization was by no means obvious. There were times during the session when the researcher observed the participants seemed stuck. Considering Smith's objective of arriving at the underlying structure of the system before the sj'stem was developed, it seemed Smith was embarking upon an innovative and indeed, in the researcher's opinion, courageous effort to short circuit their original KE efforts. It remained to be seen if a structure could be derived in this manner without what the KE felt was the necessar}' but painful experience of fumbling though the expert system developement process. In developing the Nervous Shock Advisor, the structure was developed after the rule base. As a result, fundamental structural changes were made during coding. The lesson learned reinforced the importance of diagramming the structure before beginning to code. Case on the Remoteness Advisor / 43 4.4.5. KA Session V. Pre-KA Session V Interview On January 7, 1988, the researcher met for half an hour with Deedman at his office before the fifth KA session. This was the first meeting after the Christmas holidays. During the holidays, Deedman had drawn a schematic diagram to categorize all negligence cases, and he had wanted to show this to the researcher. The diagram represented a schematic skeleton of^the^ Remoteness Advisor ( see figure 7 ). He explained the diagram as follows: the top-most level question in the system decided whether the harmful act was an act or an omission, in the latter case, the user exits to another system. The next level question decided if the act was deliberate or negligent, if it was deliberate, no remoteness issue arose. Otherwise, the user would continue to ascertain the kind of damage and choose among the three types possible : psychological, physical, and economic loss. In the first case, s/he would exit to the NSA, in the third, to the Economic Loss Advisor. If it was physical damage, then it might be plrysical damage to person or property. The basic distinction was between unusual victim or unusual activity. Unusual victim could be rescuer, thin skull, or exacerbation; whereas unusual activity might involve the use of inherentlj' dangerous substances such as explosives, toxic or inflammable substance, etc., or involve machinery and means of transportation, which included the subsets of motor vehicles, railway, vessels, Case on the Remoteness Advisor / 44 airplanes, and animal-powered vehicles. For a specific case, the question was whether it could be disposed of by matching the facts to particular types of activities that fell into the two categories of unusual people or unusual victim. Deedman expected that most cases would be taken care of this way. For cases whose facts did not match with the particular activities, the system would adopt a philosophical approach and pose the question to ask if the consequences of the harmful act might have been reasonably anticipated. If the answer was affirmative, no remoteness issue was involoved and recovery would result. Otherwise, the consequence of the harmful act was unusual and remoteness issue arose. The unexpected consequence might be due to a different victim, or a different damage; and the difference might be in type or in extent. For example, the "cow case" involved an unexpected extent of damage. Smith had already seen the diagram and liked the categorization. The agenda for the session was to go over the diagram with him. This represented the structure of the system, the next step was to represent it in terms of rules. Then a prototype could be produced. Deedman felt that Smith had the elements but they had not been hitherto organized in a diagram. KA Session V. Case on the Remoteness Advisor / 45 Smith, Deedman and the researcher sat together over a redrawn version of the diagram at Smith's office ( see figure 8 ). The purpose of the session was to further refine the diagram. Refinement of Motor Vehicle Category. Smith had thought about further refinement of the motor vehicle category before coming to the session. He went through all the cases and decided they fall under two categories. First, when one motor vehicle accident triggered another, and secondly, where a panic reaction to risk of harm was involved. He put down the following on the flipchart : motor vehicle (1) one motor vehicle accident triggers other (includes debris on highway, must include other vehicles) (2) Panic reaction ( to risk of harm ) Then they sat down and reviewed the case book, categorizing the cases according to the two groupings. Smith was uneasy about what Deedman had referred to as the "philosophical module" (see figure 8) : "I know something (is) not quite right and I don't know what it was." He had a list of questions which he thought were relevant but was unsure on how to put them together. Smith thought the concepts embodied in his questions were different from those represented in Deedman's diagram. And he would like to clarif}' them in their minds at the present session. His list of questions was as follows : Case on the Remoteness Advisor / 46 1. did the risk materialize? 2. were there damages that fall outside the parameters of the risk? 3. did the person who suffer the damage fall within the risk? 4. were the damages of a different kind than the risk, if ( they were of the ) same kind were they in excess of the risk? Deedman apparently did not think Smith's questions captured more possible combinations than his diagram. He objected to the first question because while it might be more precise, "somebody from the street" might not know what it meant. Their difference in opinion could be illustrated in the following exchange : Deedman : you know you are dealing with difference both in damage and victim. Smith :so ask first about damages... Deedman :does it matter? Smith :yes it does. Deedman :yeah you're right. Smith :(The) damage is same but (the) victim is different, a different victim suffered the risk... Deedman :Logically, ask was the damage or victim different, or both? Wouldn't that capture it? Smith :No...okay let's go back and see (examine the diagram again). Smith felt the philosophical classification of cases was more important than the "motor vehicle" ( or physical ) classification because it would supply all cases relevant by analogy to the user. Speaking as a lawyer, Deedman agreed and emphasized the importance of presenting to the judge cases closely related to the Case on the Remoteness Advisor / 47 case being argued. Then they moved onto discuss if the intervening act which created the risk was criminal or not. Deedman pointed out a clear cut answer could not be given for some cases, and other criteria might be relevant. Smith cited some factors such as date of the case, their country of origin, etc. But he had to go back to the cases again to decide. Further refinement of the psychological damage branch of the diagram was also discussed. Smith distinguished between a mother who went into depression because her child was killed versus a mother who saw her child killed, suffered nervous shock due to the incident and went into depresssion. In the former case, no shock was incurred and hence the plaintiff could not recover. In the latter, nervous shock was suffered and the NSA would be invoked. The label for that branch became "psychological only" instead of the former "psychological", and all regular nervous shock cases should follow down the branch. If only two categories of psychological versus physical were set up, it would have been impossible to differentiate between shock and non-shock cases. In clearly differentiating the categories, Smith was trying to ensure the user would "come out right " from the advisor. Other refinements of complications into medical and non-medical categories, thin skull into physical versus physical and psychological groupings, and further subcategorization of medical complications into treatment versus deterioration were added to the diagram. Case on the Remoteness Advisor / 48 At the end of the session, Smith commented the structure was almost close to finish and that Deedman might soon be ready to prototype. Deedman said he wanted to test the control flow by implementing a small version on Ml. Observations. The researcher observed in this session two people struggling to categorize a sea of cases to ensure the user would obtain the right answer from the system. Both thought in terms of the overall expert system design and often put themselves in the shoes of the user travelling down the system. The expert especially often took the initiative in assuming the role of the user. 4.4.6. KA Session VI. On February 4, 1988 Smith, Deedman and the researcher met for about 40 minutes in Smith's office. The expert and the KE continued their refinement of the structured chart. They posed actual or hypothetical cases to each other and tried to think through how a particular case would travel through the diagram, whether it would fall into one category versus another. The objective was apparently twofold : first, to label the categories general enough to capture all relevant cases, and secondly, if the category was found to be inadequate, insert another one. For example, at Case on the Remoteness Advisor / 49 one point, they discussed the merits of adding an extra category of "secondary causal factors" under "aggravation of existing conditions" which was the new label for "exacerbation" (see figure 9). Deedman was concerned that the term should be general enough to cover all possible cases of that type, but also that it should not be overly-academic for lawyers. He suggested alternative labels like "other acts that cause further harm" and preferred to avoid terms like "intervening acts." A usual mode of interaction was for Deedman to ask Smith for a sample case for a particular category and Smith would respond almost immediately with a case to illustrate it. The philosophical module was to characterize the user's case and inform the user the precise reason why recovery would not be allowed. Then similar or relevant cases could be retrieved. They also further refined the branches on psychological only versus physical and psychological, and that on medical versus non-medical. Once again, they confirmed that medical would break down to treatment and natural deterioration. And Smith cited cases to illustrate the subcategories as well as to ensure the categories adequately classify them. At the end of the session, Deedman agreed to redraw the diagram and asked Smith to "live with it for awhile" to see if changes were needed. He also proposed this session to be a milestone to mark the end of KA and the Case on the Remoteness Advisor / 50 beginning of the prototype phase. Smith agreed and said they would just spend one more session on the final diagram. Epilogue. At the last meeting, the "method of infliction different" branch was further subdivided into the two branches of "agent radically different ( e.g. fire vs water )" and "agent same but magnitude greater (e.g. massive conflagration vs small fire)". Deedman did not agree with Smith on the necessity of this addition. Deedman then worked independently from the structure chart and derived the list of variables which would be used to characterize the cases ( see figure 10 ). Based on the structure chart in figure 11, Deedman developed a prototype of the Remoteness Advisor on Ml, which was shown to the researcher on April 22, 1988. CHAPTER 5. CASE ON THE TEST DESK DECISION SUPPORT TOOL 5.1. Introduction. The Test Desk Decision Support Tool is an expert system developed by Jeffrey Howell of Microtel Learning Services Centre. Howell is a Senior Instructional Designer at the centre and acted as manager of the expert system project. Howell had a Master's degree in Instructional Systems Technology and worked for over 10 years as a courseware designer. The chief expert on the project is Dan Brophy, who is a senior instructor at the B.C. Telephone Education Centre (B.C. Tel Ed Centre) and belongs to the Staff Group of B.C. Tel.. He has over thirty years of experience in the industry and is described by Howell as "a natural teacher." Brophy is the chief resident expert who consistently worked with Howell over various phases of the development process; but as will become clear later, five other experts participated in the project. 5.2. Test Desk Operation. The expert s}'stem serves as a job aid for the telephone plant tester, alias test desk person or analyzer. A test desk person identifies and diagnoses faults in the telephone lines, clears or routes trouble reports, and assists and directs maintenance forces in remedying the trouble (for detailed discussion, see figure 12, "Job Profile"). A job aid is important as a tool to help upgrade the service that test desk persons provide to customers. Presently, three to four persons are 51 Case on the Test Desk Decision Support Tool / 52 involved in processing the touble reports produced by the 4 way telephone testing equipment which performs daily routine checks on the telephone lines (see figure 13 for diagrams on current and future operations at the Trouble Analysis Centre). When the reports arrive at the Trouble Analysis Centre (TAC), the TAC supervisor assigns a tester to investigate the trouble reported. The tester gives his analysis report to the clerk, who in turn makes up a dispatch list to send out a TAC isolator to the site. The isolator calls from the site and the test desk person sends out test signals to the isolator. They interactively test the line until a diagnosis can be made and the problem remedied. If a wrong diagnosis has been made then the time and cost incurred is substantial. An expert system could potentially incorporate the knowledge that the best test desk persons have and aid a poor or an average tester to more accurately diagnose the problem. Not only is the time (and money) saved in making accurate diagnosis important to the company, but successful and speedy line repairs improve customer satisfaction. The domain of the expert system is the job of the test desk person which involves three major functions: 1. to identify and diagnose faults; 2. to clear and route trouble reports; 3. to assist and direct maintenance forces. To accomplish these objectives, the test desk person needs to interact with the customer in say, clarifying a trouble report, as well as with staff of the Central Office and rack, the installation and repair department, the cable maintenance division, the service center supervisors, the terminal equipment suppliers, lease Case on the Test Desk Decision Support Tool / 53 wire customers, and other service centre or telephone company personnel. In addition, s/he has to be familiar with a number of tools and equipment which include the following : 1. touch tone telephone set, 2. CRT terminal which provides access to other databases or departments in the company, 3. 4 way telephone testing system (4tel) which automatically tests telephone lines of both business and residences (the 4tel system was installed in 1986; previous to that, the Badger system became operational in 1983, both systems are in use today), 4. Trouble Administration System (TAS) which holds and routes trouble reports, 5. Customer Record Information System (CRIS), which contains all service order, customer, facility, and billing data for both customer and residence, 6. Service Order Update and Locate (SOUL) which holds business and leasewire orders, 7. Mechanized Assignment Information System (MAIS) which holds facility information, 8. Facility Management System (FMS) which holds detailed facility data (from figure 12, "Job Profiles"). Hence it can been seen that the domain of the expert system is complex and represents one link in a network of operations vital to the telephone company. As Howell put it, the test desk person has "to know everything out there, where a problem can occur...in all this mess, and figure out what to do with all this as well as interface with the customer and make them happy and aware that ...we will take care of your needs...and then make the forces go out and Case on the Test Desk Decision Support Tool / 54 fix the problems." Despite the wealth of information a tester has to analyze, there are only fourteen types of trouble and six dispatches to handle them. Given a trouble type, only a finite number of procedures could reasonably apply. Hence the domain is definable in this respect. To better illustrate the importance of the afore mentioned databases to the test desk job, consider the following example. The 4tel testing system gives "window parameters" which indicate where the problem may be with a line. But if a customer call arrives at time A, and testing done at time B reveals nothing is wrong, then some data is missing. By checking into the TAS and other databases, the timing of the call can be obtained. The test desk person may then discover that problems on a line coincide with rain in the area and the line is wet. This is an important factor in the final diagnosis of the trouble. A domain expert, according to Howell, is one well versed in operating within this complex environment of tools, databases, and people and in accurately diagnosing the trouble reported. He called such a person a "good practitioner, the person you want all employees to be like." Instead of relying on any single expert, he planned to draw "the aggregate of all sources," which include the manuals for the two testing systems: Badger and 4tel, descriptions of office procedures, human informants such as supervisors of test desk persons, expert test desk operators, as well as poor tesk desk persons (to see why they fail in their work). Furthermore, one third of the information for the expert system already Case on the Test Desk Decision Support Tool / 55 resides in a simulation program developed previously, and the information could be transferred directly to the new system. The expert system was being developed simultaneously as the materials for a course on the testing equipment was being updated. Brophy would be teaching the course and his primary interest laid in revising the materials. Howell on the other hand, helped him with the revision while at the same time extracted materials from it for the expert system. Since Brophy would not be available as much as Howell needed, one of his former students who was present^ a field specialist from one of the telephone company's service offices, would act as his proxy and concentrate on the information for the advisory system. 5.3 Data Collection Methods. Organizational funding and availability of personnel were obstacles with which Howell contended since day one of the project. Several times development was suspended due to problems from either sources, while other times, progress was made on a daily basis. The researcher tracked the development by staying in touch with Howell by telephone and attended sessions personally approximately once a week from November of 1987 to early February of 1988. The data was collected either over the telephone from conversations with Howell and sometimes with Brophy, or through observations when the researcher attended the sessions Case on the Test Desk Decision Support Tool / 56 and tape recorded them. Each session lasted from 2.5 to 5 hours. The following description traces the researcher's interactions with the Microtel development team from initial contact to demonstration of the first prototype, which spanned a period of six months from October of 1987 to February of 1988. It presents both telephone conversations in which Howell reported on progress to date and accounts of sessions the researcher attended personally. At the end of some sessions, observations made by the researcher and relevant comments made by the participants are also included. Since the expert system project team was responsible for both administrative as well as developmental details, the discussion touches upon both aspects. 5.4 Knowledge A c q u i s i t i o n . On October 28, 1987, the researcher had a first orientation meeting with Howell. The discussion took place at Microtel Learning Services Centre, on the third floor of the B.C. Tel Ed Centre in Burnabj' and lasted three hours. Howell explained to the researcher the test desk job and its importance to the company operations, the contents of the discussion has been presented in Section 5.2. The expert system was to serve as a job aid and training tool, intended users included testers in the operational environment and analyzer trainees in the classroom at the Education Centre. knowledge acquisition for the test desk decision support system was carried out Case on the Test Desk Decision Support Tool / 57 under the rubric of "task analysis." The objective was to decompose the job of the test desk person to its component duties, tasks, subtasks, and prerequisite knowledge. The first phase of the analysis involved verifying and updating the materials in the two task analysis manuals on the Badger and the 4tel systems compiled in 1981 and 1986 respectively. * * * On November 4, the researcher spoke with Howell over the phone. He and Brophy had spent 5 to 6 hours cross referencing the two task analysis manuals to see how the job had changed. They updated the material and decided what was to be included in the expert S}'stem. He expected to spend another five to six hours the next day doing the same and planned to create a fast prototype by 25 or 26 of November. Tentatively he set up meetings for task analysis on 12, 16, 18, and 19 of the month. * * * On November 5, 1987, Howell met with Brophy and the researcher for 2.5 hours at the demonstration room on the second floor of the Microtel Learning Services Centre to continue the cross-referencing. (See figure 14 for a diagram on the physical layout of the room). The documents included the Badger instructional material which was developed for an instructor-led situation, a course audit done in 1983, and the 4tel instructional software document developed in 1986. Although some of the materials were seven years old, and the tools used today were different, most of the knowledge was still valid. However, there was considerable variety across different areas of British Columbia; 55% of the Case on the Test Desk Decision Support Tool / 58 company sites used the 4tel testing equipment and the rest had either the Badger system or none at all. Since the sites varied in size and complexity, for example, the downtown Mutual Centre had more than 20 testers while the Vernon centre was operated by only one to two men, the office procedures were different. As Brophy pointed out, the company encouraged people to be "innovative," and when they responded, there was great variety in the methods used. To standardize the procedures among the diverse methods adopted was difficult. At times Brophy made a note to send memos to other people in the company for clarifications. There were altogether three categories of information: 1. central office information on the type of switch, test equipment (Badger or 4tel), station equipment at customer premise (PBX, dial, or outside plant equipment); 2. IMS packages (TAS, CHRIS, or FMS); 3. trouble type information and test analysis procedures (including 4tel parameters) on what to do for different types of trouble. Typically, Brophy would read through the 1986 manual, checking each step with his notes and the 1981 manual. Sometimes he raised a point or Howell asked him if some step was indeed currently implemented, and they discussed it. Either they arrived at a conclusion and it was noted down, or one of them made a note to call somebody for clarification. An example of a task decomposition is shown as follows: 1. Duty : Identify and Diagnose Faults Case on the Test Desk Decision Support Tool / 59 Task : 1.1 read trouble report (TAS) 1.2 interpret trouble report 1.3 perform basic line test 1.4 interpret basic line test 1.5 determine type of trouble 1.7 determine what test to perform 1.9 perform 4tel tests 1.9 interpret test result 1.10 route/write off Later, Howell copied the updated information onto a word processor to produce a new document. At the end of the meeting, Howell and Brophy discussed the timing of presenting what they were doing to the staff personnel. Such a presentation was necessary both to obtain the staffs verification that indeed Howell and Brophy had captured the correct and current method of conducting tasks and to reconcile the different methods that existed in the field. These meetings would take place at the end of November. For the next two weeks, since Brophy would be unavailable, Howell planned to meet with Broprry's proxy every Monday, Wednesday and Friday morning to continue knowledge clarification. Observations and Interviews with the Expert and KE. 1. The researcher observed that as they discussed the substantive issues of Case on the Test Desk Decision Support Tool / 60 what to include in the expert system, the expert and KE were simultaneous pondering implementation methods which could be applicable to the materials. Since some of the materials to be included in the new system was already in the 4tel simulation software, they could simply be transferred over to the new system. Howell recalled developed screens which could be used or pondered whether to acquire data through an interactive query system or by accessing an existing database. Brophy was likewise involved in such implementation details; and there was no effort made to keep their discussion at a conceptual level. 2. The main concern was whether the method or procedure of doing a certain task was indeed correct and current. In a casual discussion with the researcher before their session began, Brophy remarked that it was inevitable that by the time the software was developed, the procedures for doing something which the software aimed to teach was already outdated. In the telecommunications industry, it was often difficult to keep up with the changes. Moreover, there was the difficulty of reconciling wide variety in the methods that different centres adopted in doing a similar task. * * * On November 6, Brophy and Howell met in the morning to continue their cross-referencing. They discussed structuring of the knowledge at the meeting. Howell pointed out four possible "points of identification" which could be indices to the information : the trouble type, the telephone number, the central office, and whether the Badger or 4tel system was used. Since the existing database like TAS and FMS were indexed by telephone numbers, using this as the index Case on the Test Desk Decision Support Tool / 61 could easily retrieve information from the database. If they used the switch type, parameter values could be easily accessed with either the Badger or 4tel testing tools. Outside plant equipment and line information were indexed in FMS by both central office and telephone number; while operational procedures followed directly from trouble types. Hence there were advantages to using any one of these possible indices; Howell decided trouble types were the most straight forward index for the system. The 4tel simulation software was contracted to an outside software development company, and he was contemplating doing the same for the expert system. He had discussed the project with the software company personnel and found out their schedules. They also prepared for the group verification meetings to be held at the end of the month. Materials had been sent out to the participants and Brophy had requested the supervisors to bring in as much information on local procedures as possible. * * * On 10th of November, Howell met with the researcher at Microtel Learning Services Centre for he had planned to see the other expert (Brophy's proxy) to inform him about the progress to date. However, the expert had the flu and failed to show up. Instead, Howell spent the time explaining to the researcher the design of the expert system. Case on the Test Desk Decision Support Tool / 62 The top level design of the expert system is presented in figure 15, "Draft Design of Expert System." As can be seen from the diagram, knowledge for the system is categorized into dynamic and static information. Dynamic information varies with each trouble case that the expert system attempts to diagnose. They are information that the user has to gather and enter into the system, which is primarily the NNX code. The NNX code is the first three digits of the telephone number which provides access to information on the plant (e.g. if the cable is wet), the company (i.e. type of switch), and the station or the customer premise equipment (e.g. whether one or two private lines were installed). This information does not change once it has been entered into the system. The static information refers to the knowledge base of heuristic rules, which are indexed on the trouble type. When the information about the plant, company and station is entered, then the 4tel parameters associated with the given NNX can be retrieved, and information on switch, voltage, cable, and basic line tests can be determined. Hence NNX and trouble type are the key indices. The design is based on the 4tel testing equipment; Badger has not been included because it is a less sophisticated system and provides less information. Observations Howell had considered building an expert sj'stem for a long time. He reported the ideas presented in the draft represent the results of his deliberations for the past 1.5 years. Case on the Test Desk Decision Support Tool / 63 On 12th of November, Howell met with Brophy again mainly to discuss strategies of presenting the material to the staff group that would be coming in from the other areas in B.C. for the week of the 30th. They finalized the list of people to be brought in and other administrative details. Howell considered these steps important to ensure the eventual successful adoption of the system as a job aid for the service centre personnel. Howell reported in a telephone conversation with the researcher on November 22 that Brophy's proxy was unavailable for the project. Howell planned to continue the cross-referencing with Brophy and shift the focus from the expert system to the course for the other expert was to be primarily responsible for the information in the expert system. The prototype was delayed until January of next year. Howell also discussed with staff from the Management Information Systems (MIS) division their involvement in the project. * * * In the week of November 30, Howell conducted group KA sessions with four experts brought in from various areas in British Columbia. They included Steve Lewis from Dawson Creek, Terry Brand from Victoria, Herb Young from Abbotsford, and Grant Coplitt from downtown Vancouver. Together with Brophy, thej' met in Room 211 of the Learning Centre from 8 to 4 everydaj' from 30 November to 4 December. The researcher attended the meetings during two of the five days. See figure 16 for "Agenda for group meeting" for a schedule of the events at the sessions. Case on the Test Desk Decision Support Tool / 64 The purpose of the meeting was primarily to gather and verify information to be included in the course and the expert system for training the testdesk persons. Again, they based their discussion on the two existing manuals compiled in 1981 and 1986. Howell and Brophy had begun this work and now the four local experts were brought in to continue the verification process. This process involved a verification and update of the two manuals. One person from the group would read from a copy of the manual and as he was doing so, the others made suggestions on appropriate changes and clarifications. In this process, substantial discussion arose due to diversity in equipment and procedures among the various centres. The downtown Vancouver office (Mutual) is much busier than that at Abbotsford. On a typical Monday morning, a maximum of 65 trouble report calls would come in at the Abbotsford centre in comparison to several times that for the downtown office. Hence c o n s i d e r a b l e t i m e was spent w h e n the p a r t i c i p a n t s e x c h a n g e d i n f o r m a t i o n on p r o c e d u r a l d i f f e r e n c e s . Even though Brophy acted as the arbitrator in the event of disagreements, he often was at a loss on what the company standards were since procedures vary from region to region. An example to illustrate this is the use of acronyms. While the company has a standard set of acronyms for the various tools and equipment, local personnel at the different centres often invented new ones which were totally incomprehensible to outsiders. Howell suggested including a glossary in the course and expert system to overcome this problem. Despite these differences, the basic duty of the test desk person was to "test Case on the Test Desk Decision Support Tool / 65 reported trouble" and the six major dispatches involved to achieve this were generic and common across all sites regardless of equipment and tools. Similarly, there existed only fourteen trouble types possible. Focusing on the generic, they updated the materials in the manuals. After verifying the information on the documents to be accurate and current, they proceeded to integrate the three documents by listing each duty and its subtasks on the board. Discussion arose over the proper decomposition of the tasks, whether a specific task should be done elsewhere, or some tasks deleted. The integration that emerged represented what the 5 experts consented to be the best way of conducting the duties. The new arrangement was noted down on the master manual and sent to the typist so that a new version of the manual could be produced. The sessions were also used as an information dissemination forum so the testdesk personnel from different districts had a chance to exchange insights and identify problem areas. Brophy noted down these and subsequently wrote letters to the staff personnel to request further investigation into identified problem areas and also to make recommendations that certain procedures could be passed onto the other districts' testdesk personnel. They also pondered implementation issues such as whether the PC was the best environment for the expert system and if testers were familiar with it. The next stage of the KA process had the experts together discuss how they would handle different trouble types. The crucial duty was when the test desk Case on the Test Desk Decision Support Tool / 66 person had obtained the trouble reports and other relevant data and proceeded to analyze them to identify the trouble type. There were the trouble reports, trouble types and dispatches and the trick was to correctly interprets the trouble reports, accessing the necessary databases and anyalyzing the data to arrive at a diagnosis, which would point to a type of trouble, and then to send out the appropriate dispatch. Again working from the flowcharts in the manuals (see figure 17 for sample flowcharts), they described the steps necessary to verify that the problem was indeed caused by a particular type of trouble. They worked from the existing flowcharts and the experts would comment on them. At one point, one expert suggested starting from scratch again because the flowcharts were not good. Howell went over to the board and proceeded to draw out the flowchart as they indicated. One expert commented that the flowchart did not capture all necessary details. For example, when a tester called the customer and there was no answer, s/he had to decide if the customer was out or was the line not working. Brophy defended their use among experts and referred to them as "working documents" to "figure out where everything is, and how it is ending up, and how it came in...and where you have problem." Howell on the hand, promised to send the final document to them for verification. (See figure 18 for a sample redrawn flowchart). On the last day of the group meetings, Howell explained the top level design of the expert system to the group. Working out the procedures to be encoded in the knowledge base and clarifying part of the static information had been the Case on the Test Desk Decision Support Tool / 67 objective for the last five days. Howell explained they aimed to build a prototype based on the no-dial-tone type of trouble by January or February of next year and then expand upon the knowledge base. By the time the prototype is done they would meet again to evaluate it. Meanwhile the MIS department was responsible for finding a way to retrieve the necessary data from the databases without having to enter them anew into the PC. Howell also presented the options of structuring the knowledge base so that a mini expert system is developed for each trouble type and have them linked together or have one expert system handle all trouble types within one knowledge base. The experts preferred the latter for they thought a menu-driven interface that directed the user to the appropriate rule base was best for the "new guy." Observations and Interviews with the KE. 1. Howell explained to the researcher that the difficulty in the group sessions was reconciling the differences across the sites : what worked at Victoria might not be the best for Vancouver or Abbotsford. For the expert system, the objective was to cover the generic problem and local variations would be added later to this basic framework. He envisioned a series of mini-expert knowledge bases on the different trouble types and then linkages made to connect them together. Such an approach would facilitate later modifications to the knowledge base. 2. Interaction among the four experts during flowcharting was noticeably more Case on the Test Desk Decision Support Tool / 68 intense than before. The method to handle different trouble types varied from person to person, and the discussion at times became heated as they could not agree on one best way to handle a trouble type. In general, however, the experts co-operated in an open, serious yet jovial manner. The atmosphere was relaxed and the researcher felt they were like a group of old friends meeting to discuss an issue they all felt was important and interesting. Some heated arguments erupted over fine points in the flowchart, but they were either resolved by the group or by Brophy's writing memos to check with some other persons in the company. 3. Howell was skillful as the mediator of the meeting and in making sure everyone had a chance to speak. * * * On 16 December 1987, Howell and Brophy met for the first time since the group sessions two weeks ago. Their current objective was to concentrate on the flowchart for the trouble type of no dial tone, which covered 40% of all trouble reports and whose diagnosis was considered relatively straight forward. Howell planned to meet with the software developer the next day with Jack Tom who was a former analyst with the MIS department and newly appointed to be in charge of Artificial Intelligence (Al) applications at B.C. Tel. Tom was familiar with the databases (e.g. TAS, FMS) necessary for the expert system. 5.5 Development of the Rule Base. Case on the Test Desk Decision Support Tool / 69 On 17 December 1987, Brophy, Howell and the researcher met for five hours with Marie Burlinson, owner of RJM Computer Systems, Ltd., a software development firm responsible for the simulation program on the 4tel testing equipment. Howell had some ideas on the design of the expert system and spent a good portion of the day explaining to Burlinson his ideas. Detail implementation issues like number of data files accessible from the shell (PC LEVEL 5), structure of knowledge bases, as well as administrative issues like work schedule, and tentative time commitments were discussed. Brophy explained the mechanical details of the domain and Howell the software design considerations to Burlinson, who would redesign the system after she had gained sufficient familarity with the shell. Burlinson anticipated problems with writing the rules as she was unfamiliar with the test desk problem and Howell promised to help her in writing the rule base. They agreed that the first phase was to write the design specifications, and then they would develop a prototype to correctly diagnose no-dial-tone type of troubles. Burlinson asked for one week to learn the shell. Since Christmas was approaching, they agreed to meet again after the New Year. By then she would have the design specifications ready. A tentative schedule was set up as follows: 8 January specs done 22 January prototype I done 5 February revised prototype I done 15 - 18 Februarj' group revision of prototype I Case on the Test Desk Decision Support Tool / 70 On 8 January 1988, Howell, Burlinson, Tom and the researcher met to outline the function and structure of the system. Burlinson explained what she knew about the workings of the shell to Howell. They proceeded to write the rules based on the flowchart on "no-dial-tone" produced at the group sessions but soon agreed it would be more productive for Howell to write the rules on his own and then meet again to edit them. A tentative work schedule was set up as follows: 12 Jan 8:30 - 11:30 a.m. write rules 13 Jan 8:30 - 2:30 p.m. program rules 19 Jan 8:30 - 12:30 p.m. program rules 20 Jan 8:30 - 12:30 p.m. program rules 21 Jan 8:30 - 12:30 p.m. program rules 22 Jan whole day finish prototype I Observations 1. Howell and Burlinson had co-operated previously on the simulation program and had a close working relationship. Burlinson has Masters' degrees in Computer Science and Statistics and supplied the technical know-how needed in the expert system development team. 2. Constrained by funding approved for the project, the participants had to set Case on the Test Desk Decision Support Tool / 71 timetables for themselves by estimating the time needed to do the work and the number of man-days for which the budget allowed. * * * On 13 January, Howell reported they concentrated on writing the rules for the no-dial-tone flowchart up to the test/no test decision point. He spent an hour familarizing himself with the shell and expected to continue doing the same on 14 and 15. * * * On 18 January, Burlinson tried out the shell and entered some dummy rules. She tested out how to initialize variables and access dbase files. (See figure 19 for some sample rules). * * * On 19 January, the researcher attended the development session. Howell recounted he spent five to six hours learning the software on 14. On 18 he and Brophy tapped into CRIS, FMS, and TAS and transferred all cases and screens related to the trouble types into files which he passed onto Burlinson. He also showed Burlinson the rules he wrote. The rules were syntactically incorrect for the shell and Burlinson edited them. Again they based the rule generation on the flowchart. For every decision point on the flowchart, they tried to consider all information relevant to the decision. For example, on dial tone versus no-dial-tone, both conditions external and Case on the Test Desk Decision Support Tool / 72 internal to the plant had to be considered. The information might be from the databases, memos, or on line obtained by the test desk persons. For a first attempt, they categorized the rules into out-of-company and in-company rules. As Howell explained the syntactically-incorrect rules to Burlinson and the latter edited and entered them into the computer, they were simultaneous^ discussing the structure of the rule base. They worked through iterative refinement of the rule-base. After putting down a few rules, they backtracked and ensured all the antecedants are covered and added a rule to explain them if necessary. They planned on expanding the rule base in the next few days. Burlinson would enter them and work on the randomization program to pull out random cases from the simulation program and Howell would continue to enter rules on 20 and 21. On Friday, he would bring an expert to review the rules. * * * Instead of producing the first demonstration prototype on 22 January as planned, both 21 and 22 Januarj' were spent expanding the rule base. In a telephone conversation with the researcher, Howell reported a shortage of experts as B r o p h j r was unavailable for the project this week. Another tesk desk person, Hans Enns, who used to work for Brophy and was a senior technician with 7 years test desk and 2 years 4tel maintenance experience, would be assigned to the project instead. Enns would be responsible for entering and refining the rules with Burlinson, a process which was expected to take up the whole of next week. Meanwhile Burlinson was working on the random number generator Case on the Test Desk Decision Support Tool / 73 program. * * * On 26 January, Enns, Howell, and Burlinson worked together on the rule base. Enns and Howell focused on refining and adding rules and Burlinson on debugging them. Enns also collected cases from CRIS, TAS, and FMS and set up the file structure to transfer them over to dbase files. Meanwhile Howell reworked the budget and work plan. * * * On 27 January, they again worked on pulling cases from the simulation program and screens from CRIS over to the expert system. Howell's boss Gordon Baker met with the MIS division to discuss future field installation plans. Meanwhile they continued to add, refine, and verify the rule base. Howell demonstrated the system to Enns and showed him the hardcopy of rules. Enns then modified the rules on the hardcopy. * * * On 28 January, the researcher was present when they continued the revision process. Aside from entering rules, Enns also modified the wording of the explanatory text with Howell. Burlinson was correcting the syntax of the rules and ensuring the control structure worked. She probed for clarifications on how something was done and Howell and Enns would check the hardcopy rule base in response to her queries. Case on the Test Desk Decision Support Tool / 74 Enns tested the system. As he tried different options, he compared the system's responses with his experience to verify that it reacted as he would at the test desk. In his judgement, the system contained all the rules necessary to handle no-dial-tone type troubles up to the test/no test decision point : "the system does not take short cut up to that part." Since the company demonstration of the system was scheduled for 9 February, they spent substantial amount of time discussing how to prepare for it. Burlinson pointed out the interface was "terrible." Presentation of the line of reasoning was difficult for users to understand and she planned to "pretty it up." The prototype in general was not user friendly and did not have good displays. A schedule was set up to have the system thoroughly debugged by 29 January, and devote 1 February to the interface. On 2 February Brophy would test the system; Burlinson would spend the rest of 2 and 4 making the changes Brophy deemed necessary. Then on 5 February, Enns would do a last check on the system. By 8 February, the demonstration-level prototype would be ready for the presentations scheduled on 9 and 10 of February. They also discussed starting to write the rules for prototype II and the pros and cons of having a Microtel programmer on the team instead of hiring Burlinson's assistant. Together with Enns, they decided to call the system "Test Desk Decision Support Tool." Observations & Interviews with the Software Developer. Case on the Test Desk Decision Support Tool / 75 Burlinson was enthusiastic about the shell. The most attractive feature she found was the ease with which it allowed users to exit to the environment. a= * * On 2 February, Brophy tested the prototype and the researcher observed his reactions. He assumed the user's role and selected different options to test the system's logic. "Neat", "pretty good" were some of his comments. Specifically he detected a logic error in the program, some non-standard acronyms and typos, and that two fields were missing in the database. Recognizing the prototype only dealt with simple problems within one trouble type, Brophy suggested incorporating a more involved trouble type and providing window access to more than one TAS report screen whenever necessary. Howell acknowledged these areas to be important but decided to postpone incorporation of more rules to handle other trouble types. Presently, thej' would concentrate on providing access to the TAS screens during the run, more user friendly interface and reporting features in preparation for the demonstration. 5.6 Epilogue. On 9 February, the researcher called Brophy for a report on the demonstration. He reported that on 8 Februar3% a presentation was made to the Staff group at B.C. Tel, some MIS personnel, some service centre managers, and union representatives. The MIS people were most receptive and interested in using the technology in other applications. They tentatively agreed to commit people and Case on the Test Desk Decision Support Tool / 76 money to support the project. On 9 February, another demonstration was made to the service office managers from all areas in the province. Despite materials on Al and expert systems distributed to them several days before the meeting, basic knowledge about the new technology was lacking. The consensus so far was to use the expert S3'stem for internal training and in the courses. But district managers showed reservation in using it at the operational level. They were unsure how it would fit into current and future test desk operations, and deferred decision to the Staff Manager of Operational Support Services, Steve Brandner, who had championed the FMS before. In Brophy's opinion, the expert system had been developed using a "grass-root" or "bottom-up" approach, and Howell had been the champion behind the project. The time had come for top management to accept the technology :" now (we need) to sit back and let it sink in, and make sure people can accept it, and let everybody have time to digest it." * * * On 10 February, Howell reported more presentations to senior management of the company and the reaction had been "favorable." The MIS group showed keen interest in expanding the expert system technology to other applications and he was making plans to bring the S3'stem to the other areas in B.C. for feedback. CHAPTER 6. CASE ON THE WASTEWATER TREATMENT EXPERT SYSTEM 6.1. Introduction. The Waste Treatment Expert _System ( WASTES ) was developed by Barry Chilibeck who was a Master's student in environmental engineering at the Department of Civil Engineering of the University of British Columbia. The expert system project was his Master's thesis, and the objective was "to produce a prototypical demonstration tool from which further research could be advanced from the Civil Engineering Department and in the Environmental Engineering Group" [Chilibeck, 1988:100]. Chilibeck took a number of courses on wastewater treatment and was familiar with the domain. The experts who contributed knowledge for the system included his thesis supervisor, Dr. William Oldham, and three plant operators. Dr. Oldham is a recognized authority on wastewater treatment plant design engineer and chairman of the Civil Engineering Department at the University of British Columbia. The three operators were chosen based on their extensive experience in the field. Another professor at the department, Dr. Alan Russell, counselled Chilibeck on software issues. The system was developed using a shell FRO, developed by a fellow graduate student Thomas Froess. 6.2 Data Collection Methods. 77 Case on the Wastewater Treatment Expert System / 78 Data on the development process was obtained primarily from post facto interviews with Chilibeck and the chief domain expert, Dr. Oldham. Concurrent observations on the process was impossible because the researcher did not learn of the project until six months after its initiation. Chilibeck began the project in September of 1987 and by February of 1988, which was when the researcher heard about it, he had only two days of coding left for the prototype. A first draft of his thesis was completed by the end of June. The following narrative description has been compiled from the interviews and from the draft of his thesis. 6.3 Wastewater Treatment Operations The system aims to aid the operation of conventional secondary activated sludge wastewater treatment facilities and diagnose problems that may arise. Treatment facilities are called preliminary, primary, secondary and tertiary depending on the level of waste water treatment provided by the combination of unit operations. A treatment plant consists of an interconnected string of unit operations, which are physical, chemical, and biological processes that remove waste materials from the water. A secondary facility includes all lower degrees of treatments, preliminary, and primary [Chilibeck, 1988:17]. A plant operator controls conditions such as temperature, air, nutrients, and hydraulic flows in the pipes in order to maintain the system to facilitate or retard bacteria growth. Other variables that can be controlled include age and type of bacteria. The secondary treatment facility is chosen as the model for the system because Case on the Wastewater Treatment Expert System / 79 the largest proportion of plants in use in North America belongs to this category, and developing a system for this level of treatment would describe preliminary, primary, and secondary treatment operations. The treatment of the municipal wastewater is usually carried out by the activated sludge process in which microorganisms metabolize the waste material that would otherwise consume oxygen following discharge into the receiving river, lake estuary, or sea. Too low an oxygen level will kill fish and other living things, and cause the body of water to "die." The discharge of inadequately treated wastewater by the city of Philadelphia, for example, has resulted in oxygen levels in the Delaware River estuary that are inadequate to sustain migrating fish during the season of low flow [Denn, 1986, p.28]. The activated sludge process was suitable for expert system technology because it requires substantial operator knowledge and input for process control as compared to other secondary processes [Chilibeck, 1988:23]. The layout of a secondar3' sewage plant is shown in figure 20 [Chilibeck, 1988:25]. The municipal wastewater or influent first passes through a lift pump to the racks, shredders, and grit chambers where large objects are removed. The sewage then enters the primary clarifiers where 60% of suspended solids settle out. Some of the suspended solids are carbonaceous and have a biological oxygen demand (BOD). The water then passes into the aeration tanks, where microorganisms attack the dissolved and suspended carbonaceous material (as well as nitrogen and phosphorous compounds); the microorganisms grow on this substrate, and produce carbon dioxide and water. Air and oxygen must be added continuously to keep the dissolved oxygen level above a critical level [Denn, 1986:28]. The treated sewage finally enters the secondary clarifiers and the Case on the Wastewater Treatment Expert System / 80 clarified effluent can be chlorinated and discharged. The primary clarifier sludge and part of the secondary clarifier sludge is pumped into the sludge treatment and disposal works. The sludge could undergo a series of unit operations before disposal, but these operations are not covered in WASTES. Similarly the chlorination system is omitted [Chilibeck, 1988:24]. WASTES covers the following unit operations: 1. raw sewage pumping, 2. racks and screening, 3. shredding, 4. grit removal, 5. primary clarification, 6. microscreening, 7. biological treatment, a. activated sludge, b. rotating biological contactors, 8. secondary clarification. Among these, the activated sludge basins and operations of the secondary clarifiers are the most important. For detailed explanation of the components of the treatment system, see sections 4,5, and 6 of [Chilibeck, 1988]. Traditional process control of waste treatment plants has been done algorithmically by manual or automatic control systems. WASTES however, uses heuristic rules based on observational data on the color and condition of the mixed liquor and aeration tanks, and its microscopic inhabitants [Chilibeck, 1988:50]. The literature has isolated a number of visually observable Case on the Wastewater Treatment Expert System / 81 variables to identify problems in the plant, which include: 1. activated sludge floe type and settleability, 2. aeration tank forming and foaming type, 3. secondary clarifier observations, 4. microbiological examination. In conjunction with other numeric variables such as effluent BOD and total solid suspension (TSS) levels, and BOD and food-to-micro-organism (F/M) loading rates, these are used to determine the conditions of the activated sludge process. These variables serve as indices to the knowledge. [Chilibeck, 1988:50]. In a discussion with the researcher, Chilibeck suggested expert system technology improved upon numerical or algorithmic process control for the former was more adaptable to unforeseen conditions and often cheaper. Numerical control programs required installation of digital meters at critical sites at the plant so that data on the conditions could be fed into the program. This involved considerable investment and may not be suitable for plant operators who are generally not technically sophisticated. Their knowledge of the plant is mostly gained through experience and their method of control is "an art more than a science." An expert system allows heuristic rather than numeric control and is more flexible. In an interview with the researcher, Dr. Oldham likewise suggested numerical equations were inadequate for process control. Equations for individual processes existed but there were none that related all the processes together for the entire plant and interactions between unit operations were too complex to be described by equations. Moreover, an equation valid under one set of external conditions Case on the Wastewater Treatment Expert System / 82 became invalid when the conditions changed. For example, domestic waste water in Vancouver differed from that in Edmonton due to variances in housing development, and the treatment processes for the two sites could not be identical. Hence the experience of operators at the two locations served as a more reliable basis for control than equations. For Chilibeck, motivation for developing the system also came from the need to fill the gap between research theory and practical treatment plant application. The experience that operators possess fill this gap and encoding it in heuristic rules is one means to capture this valuable asset. A new operator learns his trade by example and from the knowledge of the experienced operators. The system is therefore useful as an operational as well as teaching aid. Chilibeck indicated the heavy emphasis on design in the engineering faculties in the university meant that students often were unfamiliar with operational realities. Hands-on experience with the expert system would enable students to have a "feel" of what it was like to operate a plant. Anticipated users of the system include experts such as waste treatment plant design engineers as well as non-experts like municipal engineers who are unfamiliar with the domain and students. Chilibeck described his potential users as follows: One of the goals of the system was to make it simple enough that a person unfamiliar with wastewater treatement facilities could operate the expert system, understand what the system is doing, and learn about the treatment process. At the same time the system should be designed to be useful to someone familiar with wastewater treatment. That person should be able to quickly find an area that he or she is interested in and examine it thoroughly with the system" [Chilibeck, 1988:70]. Case on the Wastewater Treatment Expert System / 83 6.4. Knowledge Acquisition. There were three sources of information for the system and their relative importance were as follows : 1. 50% from published works on process control of waste treatment plants, these supply primarily diagnostic information. 2. 25% from the chief domain expert, Dr. Oldham, and 3. 25% from the plant operators with whom he had short conversations over the telephone, primarily when he needed to clarify process control information. The written sources of information consisted of texts specifically written for plant operations and process control, articles from magazines, and research publications from journals. The Environmental Protection Agency has published troubleshooting guides and process control manuals for municipal treatment such as Field Manual for Performance Evaluation and Troubleshooting at Municipal Wastewater Treatment Facilities[197 8] and Process Control Manual for Aerobic Biological Treatment Facilities [1977] published by the United States Environmental Protection Agency. Additional troubleshooting information was gained from the Water Pollution Control Federation publications on design and operation of treatment facilities as well as their monthly series, Operations Forum [Chilibeck, 1988:68]. From the literature, Chilibeck obtained information which he represented in logic trees. Then he asked the expert-operators for verification of the information presented in the logic trees. Since every plant was different and no model-plant Case on the Wastewater Treatment Expert System / 84 existed, the analysis presented in the written works often did not coincide with reality. For example, he presented to the expert-operators three different ways of controlling a process derived from three different schools of thought. He would ask them which method they adopted at their plants, and the operators chose "the easiest method." Chilibeck then modified the logic tree accordingly and the corresponding rules were incorporated into the knowledge base. 6.5. Development of Rule Base. After the initial phase of acquiring knowledge by extensive research into the literature, Chilibeck consolidated the information he needed for the expert system by identifying the following : 1. unit processes and operations to be included (see Section 6.3 above for a list); and 2. common plant problems. For each plant problem, key information or parameters both inside and outside of the unit operations were identified and the corrective actions obtained. The result of this consolidation process was a logic structure or flowchart for each unit operation relating all possible problems with the parameters to be examined which determine what precise problems existed, and solutions for them. Chilibeck then proceeded to convert the flowcharts, starting with the smallest and simple ones, into rules, which were subsequently coded and debugged for logic and syntax errors (for sample trees, see figure 21) [Chilibeck, 1988:79]. A series of mini-knowledge bases were developed in this manner, one for each unit Case on the Wastewater Treatment Expert System / 85 process. Then they were linked to the menu structure and to the main knowledge base. Rule bases on problems of floating materials and odor, which affected the entire plant, were also included. For a diagram of the file structure of the system, see figure 22 [Chilibeck,1988:61]. As the knowledge base was being developed, the expert's contribution of knowledge played a significant role. Some processes were not well documented in the literature, and Dr. Oldham supplied the necessary clarifications. According to Chilibeck, knowledge acquisition sessions with the expert were highly informal and usually lasted half an hour. For simple problems, Chilibeck described the scenario to Dr. Oldham and through discussion arrived at a conclusion. For more complex problems, Chilibeck demonstrated the system to Dr. Oldham, who assumed the user's role and suggested changes when necessary. Chilibeck reported he consciously maintained the expert's interests by showing him the system. Their interaction was facilitated by a shared vocabulary. Hence, continual feedback was obtained from the expert primarily by posing scenarios and letting him see how the system functioned. On 5 to 6 occasions, Dr. Oldham also took the system home over the weekend when he reviewed it as a user to check if all relevant variables have been considered. In situations when the three operators did not agree among themselves, Chilibeck relied on Dr. Oldham's judgement. With years of experience as a plant design engineer, Dr. Oldham was able to resolve differences or attribute them to circumstantial variances. If operators disagreed on the optimal procedures because Case on the Wastewater Treatment Expert System / 86 of differences in plant size and conditions, they might be all correct for their own plants. In such cases, the system had to accomodate all situations. While the operators were knowledgeable about the plant operations, Dr. Oldham had an overall appreciation for the entire operation of the treatment facilities as well as how the environment, such as the quality of the incoming sewage, impacted the plant. At an interview with the researcher, the expert stressed the importance of the operators' input for "a good operator can rectify a lot of design errors in a treatment plant; a good designer still needs a good operator at the other end or you wouldn't get results." The expert added that while he was helping Chilibeck with his system, he was simultaneously working closely with a plant operator to improve the waste water treatment operations in Kelowna. Discussions with Chilibeck often clarified his own understanding of the problem at Kelowna. In the expert's opinion, WASTES is usable but not foolproof. It is complete in that it has addressed all the important variables that should be considered in a conventional secondary treatment plant. But for an esoteric facilit}' that involves processes not currently understood, the system, would have problems. More basic research is needed before a "complete" sj'stem can be developed. 6.6. Epilogue. On April 7 1988, Chilibeck gave a demonstration of WASTES to his fellow Case on the Wastewater Treatment Expert System / 87 students at the Civil Engineering Department. At the time of writing, he planned to send the system out for evaluation by the field operators. CHAPTER 7. CONCLUSIONS The study has reported on an empirical investigation of three expert system development projects. This chapter presents a number of observations drawn from the inquiry and discusses some limitations of the study as well as suggestions for future research. 7.1 General Observations. 1. The "rapid-prototyping" approach is a well-established technique in building expert systems [Gaines, 1987b; Buchanan et al.,1983; Young et al., 1987]. The notion of "rapidness" however, is much harder to precisely define. Buchanan et al., for example, suggested implementation should start once "a few key concepts and relations" are noted down. Similarly, Naumann and Jenkins emphasized the importance of constructing a working prototype within a short time [Naumann et al.,1982; Jenkins, 1983]. Observations of the law and tesk desk cases suggest that while prototyping is a valid approach in building expert systems, implementation should not begin until the domain is reasonably understood. Referring to information systems, Jenkins opined that prototyping rests on the premise that development of complete and correct information requirements specifications prior to implementation is both technically and behaviourally infeasible [Jenkins, 1980]. For expert sjrstems, instead of 88 Conclusions / 89 specifying the information requirements, expertise is to be represented. The mapping between expertise and the conceptual model requires a series of modelling processes, and prototyping is appropriate for ES construction precisely because it is a tool for modelling which facilitates communication between the KE and the expert [Sol; Jorgensen, 1984]. It is useful in both the latter part of the modelling as well as the knowledge elicitation phases (see figure 2) [Benbasat & Dhaliwal, 1988b]. A clear definition of prototyping is often lacking in the literature, in the present context, it refers to a system that is to be expanded, supplemented, or supplanted [Jenkins, 1980], and not a system that is "created as quickly and as cheaply as possible to test the feasibility of a new concept or approach [Asner et al., 1987]. In fact, empirical observation from all three cases suggests that the latter notion of prototyping is to be avoided and implementation should begin only when the domain addressed is substantially clarified. In the case of the Remoteness Advisor, the expert and knowledge engineer spent seven sessions formalizing the structure of the domain (and the system) before the KE developed a prototype. At that point, both were reasonabfy assured that the structure would remain stable. Speaking from experience, Smith and Deedman stressed the importance of coding only a f t e r the structure of the domain had been formulated. Prototyping early resulted in changing the code in the prototype system, which caused conflict between the expert and the knowledge engineer for the latter found it difficult to 89a Fig. 2. A Validation-Based Life Cycle Model of the Knowledge-Based System Development Process (Benbasat A DhaliwsJ, 1988bI IDEALIZED* KNOWLEDGE BASH <aooeast<at • (cafity> OTHER ALTERNATIVE KNOWLEDGE-BASES W REALITY <t | , ocbei cxpe/t»> "SOURCE" KNOWLEDGE BASE IN REALITY <t j . , louxce expert > IMPLEMENTED KNOWLEDGE- BASED SYSTEM CONCEPTUAL KNOWLEDGE-BASE ELICITED KNOWLEDGE-BASE N O M 'Concafflu* and * M « .•trtrton mwtkM Kmmlm\gt l*rfj*%m M M n Snpiwnantadan «Msoon can b« Mrmd V M a f a 1Bo4t KjrxrSonX vol rapnanaSonol nSdclon conoftut* K*imtnifl-9fwm VMMjfew, Conclusions / 90 give up what he had spent time working on. Similarly in the case of the Test Desk Decision Support Tool, the project team spent two months updating and verifying the knowledge in the manuals before thej' took the design of the expert system to the software developer. In addition, the cost devoted to clarifying the knowledge was substantial for the four test desk persons were released from their jobs for a week and brought to the Microtel Learning Services Centre for the group sessions. 2. The KA techniques that were observed in the three cases include the following : a. Interview in the form of question-answer between the expert and KE; typically, structure interviews predominate at the early and unstructured ones at the later KA sessions. This technique was observed in all three cases. b. Scenario-elicitation where the KE posed a scenario and the expert commented on it; this method was observed in the two engineering cases. c. Task analysis by decomposing the domain into goals and subgoals; this was used in test desk case. d. Group discussions among the experts, also observed in the test desk case. e. Prototype or system refinements, observed in the two engineering cases. Among these, scenario-elicitation was recommended in two out of the three cases. This finding supports the literature suggestion that experts find it Conclusions / 91 easier to demonstrate their expertise in a particular situation rather than formulate it explicitly [Witten et al.,1987]. It is noted that the techniques are by no means mutually exclusive : prototype or system refinement might be a means to elicit scenarios. 3. Diagramatic representations of the three KA processes summarize the steps involved in each case and are presented in figures 3, 4 and 5. Characteristics of the three cases are summarized in figure 6. 4. The problem domain emerges as a critical factor in the KA process, which determines the number of experts to be consulted. In the two engineering cases, since the tasks involved diverse locations and could not be adequately modelled as a single process, multiple experts were required [Henderson, 1987]. In the Test Desk Decision Support Tool case, for example, five experts were involved in the knowledge acquisition process and another one during development of the rule-base. In the Wastewater Treatment case, four experts were involved in clarifying the domain. Experts from different areas in British Columbia were consulted to ensure local knowledge was incorporated into the system. Only the Remoteness Advisor case falls within the traditional mode of eliciting from one expert. In this case, the domain addressed is more subjective than the other two and Smith was one of the few (if not only) proponent of a "deep structure" interpretation of the law of negligence. 5. The domain is also important in another respect. Both the test desk and read Appendix of [Smth. 84] c la r i fy phy. categories c la r i fy philo categories ~T~ combine in struct, chart develop kb Fig. 3 KA Process for the Remoteness Advisor 91b read Manuals group update of Manuals flowchart strategies develop kb 4 i systen refinement Fig. 4 K A Process for the Test Desk Decision Support Tool r e a d EPA Manuals zrz develop l o g i c t r e e s T" v e r i f y w i t h o p e r a t o r s develop kb * 1 x s y s t e n r e f i n e m e n t Fig. 5 KA Process for WASTES Fig . 6 Summary Table of Characteristics of 3 Cases Sample 1 Sample 2 Sample i Bxpert System Remoteneas Advleor Teat Desk Decision Support Tool Wastewater Treatment Expert 8ye Nuabil of K l l l l 1 1 1 Number of Bxpert(s) 1 5 4 previous experience of KB yea, 1 ayetem took a course In B8 no previous experience of Expert yea, 1 system no no Bxpert-KB reletlonehlp personal,shared vocab professional, shared some vocab advlaor-student, shared vocab Problem DoMln remoteness In law teat deak diagnostics waatawater treatment Problem Category c l a s s i f i c a t i o n , bottom up diagnostic, top down diagnostic, top down Objectlve/Subjectlve subject 1ve objective objective Olobal va Local Domain qlobal global qlobal Closed/Open Domain r e l a t i v e l y closed - cases 1 of trouble types set • of unit proceasea aet Qualitative va Quantitative qualitative quant Itlve * q u a l i t a t i v e q u a l i t a t i v e • quantitative KA Procesa: aouicaa of Info caae book, (Smith,1964) manuals for equipment BPA manuals, journals IS ConatructIon: tool s h e l l :H1 PC Level 5 PRO Orq. Setting Law Faculty, UBC Hlcrotel Learning Services Dept. of C i v i l BnqlnearIng, UBC Usee Instructional, advisory job aid. Instructional tool Job Aid, Instructional tool Conclusions / 92 wastewater treatment cases involved diagnostic tasks while the law system addresses a classification problem [Hayes-Roth, 1983]. In the former instance, the number of parameters to be examined varies with the location while in the latter, the number of relevant facts are dictated by the set of court cases for this domain. 6. It has been observed that the stages in the expert system lifecycle in fact overlapped [Buchanan, 1983]. The inquiry supports this finding. Despite the KE's efforts to keep the discussion to the conceptual level, the expert in the Remoteness Advisor case frequently considered implementation issues during the KA sessions. The expert and KE of the Test Desk Decision Support Tool recalled screens or data that could be used from the simulation program. A segregation of the stages of conceptualization and implementation rests on the assumption that the domain expert is ignorant about technology. In reality, however, experts who are willing to participate in expert system projects are individuals highly interested in technology and eager to be involved in developing the systems. In the case of the Wastewater Treatment Expert System, the expert claimed to be non-computer-oriented and the scheme of separating the conceptualization and implementation stages prevailed. 7. Close rapport between the expert and KE directly and positively influences the quality of the conceptual model produced. The law case is distinctive in that personal friendship existed between the expert and the KE and the conceptual structure that resulted from the KA sessions was a product of the intellectual as well as personal dynamics that took place [Smith,1988]. Conclusions / 93 8. The Remoteness Advisor and the Wastewater Treatment Expert System were developed in an academic while the Test Desk Decision Support Tool was produced in a commercial environment. The pressure of building a prototype to win managerial support applied in the latter but not in the former two cases. 9. The law and civil engineering cases are similar in that both KE and experts were knowledgeable in the domains. Deedman is a lawyer by training and Chilibeck a Master's student in Environmental Studies. Hence they shared the experts' language as well as knowledge. In the law case for example, the KE was often able to cite relevant cases to test the expert's categorization schemes; a non-lawyer would not have had such facility. 7.2 O b s e r v a t i o n s on the Methodology Conducting longitudinal observation of multiple cases requires a research team [Yin, 1984]. This theoretical necessitj' however, is impossible to practice in the present study because the subjects involved did not agree to having alternative observers. Having familiarized themselves with the researcher and having expended the time and effort to acquaint her with the basic jargon, subjects were understandably reluctant to re-invest their energy. While cases do provide the benefits of current data and observation facilitates understanding, the Conclusions / 9 4 demands of keeping track of multiple cases over a period of almost a year were onerous for one person. Observation of the KA sessions was a valuable means to obtain data. Maintaining rapport and developing trust with the subjects were essential so that they did not feel uncomfortable about being watched. Taped protocols of the sessions were useful if the quality of the tapes could be assured. This proved difficult especially since it was impossible to attend every single session and subjects who took over the responsibility might have been less careful. Group sessions were difficult to tape and the recordings were hard to transcribe as multiple voices might overlap. In the law case however, since there were only two participants in a reasonably small room, the tapes were invaluable when legal jargon became hard to follow during the session. Retrospective interviews were useful for clarification purposes and less so as a primary data gathering tool. In the case on the Wastewater Treatment Expert System for example, the researcher had no means of validating the responses obtained. The interviewees might have succumbed to retrospection biases. Not all subjects responded favorably to keeping a diary and in the three cases described, it was not adopted. In the two cases where it was used, the diary proved extremely informative in tracking the process of how the KE approached the task. Deliverables were in general useful. For example, the structure chart in the law Conclusions / 95 case and the Job Profile document in the Test Desk case were indispensable in facilitating understanding of the domains. The third case on treatment plant could not have been completed without information from Chilibeck's thesis. 7.3 Limitations of the Study. 1. This study depended heavily on observational data. Despite the use of taped protocol and post facto interviews with subjects to triangulate the collected data, reactive bias due to presence of the researcher was minimized but not avoided. Since the researcher observed the KA sessions for the Negligence and the Test Desk cases over a period of over six months, reactive bias introduced likely decreased as the subjects became more accustomed to being watched. Another threat to internal validity came from the researcher's record and interpretation of data, which was controlled by showing the reports on each case to the subjects involved for verification. 2. External validity is constrained by the fact that only three cases were studied. Attempts to generalize the lessons to other development efforts can be made only with due consideration to the domain addressed and the human actors involved. 3. The emphasis of this study has been on the KA process. In all three cases, only a demo-level prototype had been developed by the time observation terminated. Hence the entire development lifecycle was not documented. 4. The sample is biased in that the reported projects all succeeded in producing sj'stems. However, since these systems have not been evaluated, Conclusions / 96 quality of the resulting systems remain unknown. 7.4 Suggestions f o r F u t u r e R e s e a r c h Empirical research in expert system implementation is only in its early stages [Benbasat et al., 1988a], much work in this area needs to be done in the future. Some suggestions for further research are presented as follows: 1. Validation of the findings made in this study can be conducted by either investigating other cases of expert system development [Bonoma, 1985] or through laboratory experiments. The following are some hypotheses that can be generated from the inquiry: a. What are the domain characteristics that necessitate the involvement of multiple vs single expert(s)? b. Does a detailed study of the domain which result in a well-formulated problem space provide a better quality knowledge base than "rapid prototyping"? c. Is a person familiar with the domain who later acquires computer knowledge a better candidate for the knowledge elicitation task than one who has computer knowledge and then learns about the domain? d. Does user involvement improve the quality of knowledge base? e. Can expert-user replace a real user in providing input in the development process? 2. A longer observational period could be undertaken to cover the entire development lifecycle including the maintenance phase to better understand Conclusions / 97 the expert system development lifecycle. The issue of how to manage an expert system development project has seldom been studied and is a logical next step in the research agenda. BIBLIOGRAPHY [Abrett et al., 1986] Abrett Glenn, M.H. Burslein, "The KREME Knowledge Editing Environment," in Proceedings of the First Knowledge Acquisition for  Knowledge-Based Systems Workshop, Banff, Canada, November 1986. [Addis, 1987] Addis, T.R., "A Framework for Knowledge Elicitation", in Proceedings of the First European Workshop on Knowledge Acquisition for  Knowledge-Based Systems, Reading University, September, 1987. [Alexander et al., 1986] Alexander J.H., M.J. Freiling, S.J. Shulman, S. Rehfuss and S.L. Messich, "Ontological Analysis, An Ongoing Experiment," in Proceedings of the First Knowledge Acquisition for Kno wledge-B ased Systems  Workshop, Banff, Canada, November 1986. [Anjewierden,1987] Anjewierden, A., "The KADS System", in Proceedings of the  First European Workshop on Knowledge Acquisition for Knowledge-Based  Systems, Reading University, September, 1987. [Asner et al.,1987] Asner, M., King, A., "Prototyping : A Low-Risk Approach to Developing Complex Systems", unpublished paper from Gellman Hayward & Partners, Ltd., 1987. [Bell, 1987] Bell, Jill, "Teaching Knowledge Engineers, the Human Side of Knowledge Engineering", in Proceedings of the First European Workshop on  Knowledge Acquisition for Knowledge-Based Systems, Reading University, September, 1987. [Belkin et al.,1987] Belkin, N.J., H.M. Brooks and P.J. Daniels, "Knowledge Elicitation Using Discourse Analysis," in Proceedings of the First Knowledge  Acquisition for Knowledge-Based Systems Workshop, Banff, 1986. 98 / 99 [Benbasat et al.,1987] Benbasat, Izak, David K. Goldstein, Melissa Mead, "The Case Research Strategy in Studies of Information Systems", in Management  Information Systems Quarterly, Vol.11, No.3 (September 1987), 369-386. [Benbasat et al., 1988a] Benbasat, Izak, Barrie R. Nault, "An Evaluation of Empirical Research in Managerial Support Technologies : Decision Support Systems, Group Decision Support Systems, and Expert Systems, Faculty of Commerce & Business Administration, University of British Columbia, February 1988. [Benbasat et al., 1988b] Benbasat, Izak, Jasbir S. Dhaliwal, "A Framework For the Validation of Knowledge Acquisition", Working Paper No.88-Management Information Systems-006, Faculty of Commerce & Business Administration, University of British Columbia, May 1988. [Bonoma, 1985] Bonoma, Thomas V., "Case Research in Marketing : Opportunities, Porblems, and a Process," in Journal of Marketing Research, Vol.XXII (May 1985), 199-208. [Boose, 1986] Boose, John, Expertise Transfer for Expert System Design, New York: Elservier, 1986. [Brueker et al., 1987] Brueker, J., B. Wielinga, "Knowledge Acquisition as Modelling Expertise : the KADS Methodology", in Proceedings of the First  European Workshop on Knowledge Acquisition for Knowledge-Based Systems, Reading University, September, 1987. [Buchanan et al.,1983] Buchanan, B.G., et al., "Constructing an Expert System", in Hayes-Roth, R., Waterman, D.A., and Lenat, D.B., (eds.), Building  Expert Systems, London : Addison-Wesldy, 1983. / 100 [Burton et al., 1987] Burton, A.M., N.R. Shadbolt, A.P. Hedgecock, G. Rugg, "A Formal Evaluation of Knowledge Elicitation Techniques for Expert Systems : Domain 1", in Proceedings of the First European Workshop on Knowledge  Acquisition for Knowledge-Based Systems, Reading University, September, 1987. [Chandrasekaran,1984] Chandrasekaran, B., "Expert Systems : Matching Techniques to Tasks", in Reitman, W. (Ed.) Artificial Intelligence  Applications in Business, New Jersey : Ablex Publications Corp., 1984. [Chandrasekaran et al.,1985] Chandrasekaran, B., Bylander, T., "Al Research at the Ohio State University", Al Magazine, Summer 1985, p.74-79. [Chilibeck, 1988] Chilibeck, Barry M., Operation and Diagnostics of Wastewater Treatment Facilities Using An Expert System, unpublished Master's thesis completed at the Department of Civil Engineering, University of British Columbia, 1988. [Deedman, 1986] Deedman, Cal, Building Rule-based Expert System in Case-Based Law, unpublished Masters' thesis, Faculty of Law, University of British Columbia, 1986. [Denn, 1986] Denn, Morton M., Process Modelling, New York, Longman Inc., 1986. [Dhaliwal et al., 1988] Dhaliwal, Jasbir S., Izak Benbasat, "A Model for Empirical Investigation of Knowledge Acquistion," Working Paper No. 88-Management Information Systems-008, Faculty of Commerce & Business Administration, University of British Columbia, July 1988. [Fellers, 1987] Fellers, Jack W., Key Factors in Knowledge Acquisition, Computer  Personnel, Vol. 11, No.l, May 1987. / 101 [Gaines, 1987a] Gaines, Brian R., Integrated Knowledge Support Environments for Rapid Prototyping of Expert Systems, Invited Paper. [Gaines, 1987b] Gaines, Brian R., "Rapid Prototyping for Expert Systems," in Oliff, M. (ed.), Proceedings of the International Conference on Expert  Systems and the Leading Edge in Production Planning and Control, University of Southern California, 1987. [Gaines, 1987c] Gaines, Brian R., "How Do Experts Acquire Expertise?," in Proceedings of the First European Workshop on Knowledge Acquisition for  Knowledge-Based Systems, Reading University, September, 1987. [Gaines, 1987d] Gaines, Brian R., "Knowledge Acquisition for Expert Systems," in Proceedings of the Second Knowledge Acquisition for Knowledge-Based  Systems Workshop, AAAI, Banff, November, 1987. [Gaines, 1987e] Gaines, Brian R., "Advanced Expert System Support Environments," in Proceedings of the Second Knowledge Acquisition for  Knowledge-Based Systems Workshop, AAAI, Banff, November, 1987. [Grover, 1983] Grover, M.D., "A Pragmatic Knowledge Acquisition Methodology," Proceedings from the International Joint Conference On Al, Karlsruhe, West Germany, 1983, P.436-438. [Hayes-Roth et al. 1983] Hayes-Roth, R., Waterman, D.A., and Lenat, D.B., Building Expert Systems, Reading, MA: Addison-Wesley, 1983. [Henderson, 1987] Henderson, H.C., "Finding Synergy between Decision Support Systems and Expert Systems Research, Decision Sciences, Vol. 18, No.3, Summer 1987, p.333-349. / 102 [Jacobson et al, 1987] Jacobson, Chris, Michael J. Freiling, "ASTEK : A Multi-Paradigm Knowledge Acquisition Tool for Complex Structured Knowledge," in Proceedings of the Second Knowledge Acquisition for  Knowledge-Based Systems Workshop, AAAI, Banff, November, 1987. [Jenkins, 1980] Jenkins, A.M., "The Prototype Model As A Management Information Systems Design Technique", Discussion Paper #1663, School of Business, Indiana University, 1980. [Jenkins, 1983] Jenkins, A.M., "Prototyping : A Methodology for the Design and Development of Application Systems," Discussion Paper #227, School of Business, Indiana University, April 1983. [Johnson, 1984] John, P.E., "What Kind Of Expert Should A System Be?" The  Journal of Medicine and Philosophy, Vol.8, 1983, p.77-97. [Jorgensen,1984] Jorgensen, A.H., "On the Psychology of Prototyping", in Budde, R., Kuhlenkamp, K., and Mathiassen, L. (Eds.), Approaches to Prototyping, Berlin : Springer-Verlag, 1984. [Kidd,1987] Kidd, Allison L. (Ed.), Knowledge Acquisition for Expert Systems : A Practical Handbook, New York : Plenum Press, 1987. [Kraushaar et al., 1985] Kraushaar, J.K., and Shirland, L.E., "A Prototyping Method for Applications Development by End Users and Information Systems Specialists", Management Information Systems Quarterty, Vol. 9, No. 3, September 1985, p. 189-197. [Naumann et al.,1982] Naumann, J.D., and Jenkins, A.M., "Prototyping : The New Paradign for Systems Development," Management Information Systems  Quarterly, Vol. 6, No. 3, September 1982, p.29-44. / 103 [Pau,1987] Pau, L.F., "Prototyping, Validation and Maintenance of Knowledge Based Systems Software", in the Third Annual Expert Systems in Government Conference Proceedings, Washington, D.C.: The Computer Society of IEEE, 1987. [Smith, 1984] Smith, J.C., Liability and Negligence, Toronto, Carswell Legal Publications, 1984. [Smith, 1988] from a discussion the author had with Professor J.C. Smith and Cal Deedman on August 19, 1988. [Sol] Sol, H.G., "Prototyping : A Methodological Assessment", unpublished paper from Faculty of Economics, University of Groningen. [Sviokla, 1986] Sviokla, John J., Planpower, Xcon, and Mudman : An In-Depth Analysis Into Three Commercial Expert Systems In Use, Harvard University, unpublished doctoral thesis, 1986. [Watkins et al., 1987] Watkins, P., D.E. O'Leary, "Knowledge Acquisition for Small Scale Expert Systems from Consulting Experts : A Field Study Approach," in Proceedings of the First European Workshop on Knowledge  Acquisition for Knowledge-Based Systems, Reading University, September, 1987. [Webb et al., 1966] Webb, Eugene J., Donald T. Campbell, Richard D. Schwartz, Lee Sechrest, Unobtrusive Measure j _ Nonreactive research in the Social  Sciences Chicago, Rand McNally, 1966. [Weick, 1968] Weick, K.E., "Systemactic Observational Methods" in G. Lindzen and E. Aronson (Eds.), The Handbook of Social Psychology (Vol. 2), Reading, Mass: Addison Wesley, 1968. / 104 [Weiss et al.,198?] Weiss, S.M. and CA. Kulikowski, A Practical Guide to Designing Expert Systems, Totowa, New Jersey, Rowman & Allanheld Publications, 1984. [Winston et al., 1986] Winston, Philip H., Karen A. Prendergast, The Al Business The Commercial Uses of Al, Cambridge: The MIT Press, 1986. [Witten et al., 1987] Witten, Ian H., Bruce MacDonald, "Using Concept Learning for Knowledge Acquisition," in Proceedings of the Second Knowledge  Acquisition for Knowledge-Based Systems Workshop, AAAI, Banff, November, 1987. [Yin, 1984] Yin, R.K., Case Study Research, Design, and Methods, California : Sage Publications, 1984. [Young et al., 1987] Young, R.M., J. Gammack, "Role of Psychological Techniques and Intermediate Representations in Knowledge Elicitation," in Proceedings of the First European Workshop on Knowledge Acquisition for  Knowledge-Based Systems, Reading University, September, 1987. APPENDIX A Questionnaire 105 1. General C h a r a c t e r i s t i c s of the Expert System 1 What i s the problem domain of the system? 2 What are the majo^ c a p a b i l i t i e s of the system? e x p l a i n reasoning modify database access database g i v e a d v i c e o t h e r s 3 Who are the users of the system? gen e r a l (non-expert) users experts 4 Does the ES access an e x i s t i n g database or i t s own database? how does the i n t e r f a c e between the ES and e x i s t i n g database work? are there j u s t one or more than one database? 5 Could you g i v e me some background on y o u r s e l f ? why i s there the i n t e r e s t or n e c e s s i t y t o b u i l d the ES? Is t h e r e any p r i o r r e l e v a n t e x p e r i e n c e / t r a i n i n g ? 6 Does the system use forward or backward c h a i n i n g and why? 7 What machine does i t run on? 2. KA Process 1 What are the sources of i n f o r m a t i o n : your own e x p e r t i s e , textbooks, or other experts? 2 I f you draw from your own e x p e r t i s e , how d i d you decide whether something should be included? d i d you c o n s u l t other people or resources 3 I f you used textbooks, were they the only source of information? 4 I f you i n t e r a c t e d with other e x p e r t s , c o u l d you d e s c r i b e the KA process? how long was a session? what happened? d i d you use automated i n t e r v i e w i n g a i d s ? d i d you use "common sense" to guide your q u e s t i o n s d u r i n g the interview? how d i d you record the information? how d i d you organize the i n f o r m a t i o n , what method d i d you use? d i d you t r y to i l l i c i t i n f o r m a t i o n t o f i t i n t o your o r g a n i z a t i o n scheme or v i c e versa? 5 Can the KA process be d i v i d e d i n t o stages or phases? 6 What are the items of knowledge e x t r a c t e d from the expert? ES C o n t r a c t i o n 1 Could you d e s c r i b e the process of b u i l d i n g the ES? how d i d you s t a r t ? d i d the development i n v o l v e KA and then prototype and refinement or d i d i t f o l l o w the" t r a d i t i o n a l c y c l e of d e s i g n - c o d e - t e s t - d e l i v e r ? 2 How d i d you s t r u c t u r e the knowledge? 3 How d i d you t r a n s l a t e knowledge i n t o machine r e p r e s e n t a t i o n ? what kind of r e p r e s e n t a t i o n d i d you use : schema, s c r i p t , frame, i f - t h e n r u l e s or others? 4 Did you use any ES s h e l l or t o o l s to b u i l d the system? Outcomes 1 How do you judge whether a system i s complete? c r i t e r i a f o r judgement : speed accuracy q u a l i t y of advic e user acceptance of advice o t h e r s 2 Have m o d i f i c a t i o n s been made or new r u l e s added to the system? how easy i s i t to add,delete, or modify the r u l e s ? 3 How long d i d the e n t i r e development process take? 4 How much d i d i t c o s t ? 5 How would you judge performance of a system? observed improvement on p r o d u c t i v i t y / a c c u r a c y does i t r e p l a c e or support human e f f o r t how do you know the advice i t g i v e s i s sound would you say the system outperforms a person Do you a n t i c i p a t e problems with user acceptance? If that happens, what do you plan to do? APPENDIX B Sample Tape Protocol 106 I ' l l be going to Microtel In the morning around 10:30 -- i t w i l l be on the 30th of November. They have having a group session where they are bringing outside experts and they are meeting at Microtel Learning Centre Room 211. Then he started testing. Part of the problems that he had It's the f a s t e s t part of that course now. What you're saying i s i t doesn't ... occur i f you won't abound and concept e n t i r e l y , i t ' s that you are not going to use those. (Well we might end up on the i s l a n d because they haven't been quite s e l f with what the workload i s on the is l a n d i n areas. We might end up having permanent screen and the things w i l l be sensitive, everything w i l l be s e n s i t i v e around here and h e ' l l do i t and i t ' s h i s job that sorting this s t u f f out to where i t ' s supposed to go. We might end up doing i t but r i g h t now i t ' s not being done, i t ' s being talked about and there i s a p o s i t i o n being made a v a i l a b l e i f we want to use i t for that. But they haven't said we want to ... I t ' s unscheduled as yet. We're going to schedule I think i n the end we w i l l because of the fact we're t r y i n g to work a whole is l a n d -- there's a vast amount things that could be eliminated by having the screen ri g h t there j u s t to f i l t e r i t a l l out. Probably and y o u ' l l see i t i n February and March, there won't be any s i l e n t screen. We'll change our flow chart on how these things pass through; we'll send i t through a screener, and we then w i l l put i t In a vat unless i t ' s you know , doesn't have to go through a screener .) For the purpose of what we are doing t h i s , i s that enough or should we .... I think at t h i s point i t ' s going to have to be l e f t . You know I think we're s t i l l going to work at i t Let us go out and have a look at i t j u s t for a minute and play with what we have. With Steve he's everybody. He's already everybody. I think that some reading c o r r e c t l y to what was going on, or talking to John, too, i s that the enemies would have got i t . We'll f o r e t e l l working i n the o l d well was that depth takes a l i t t l e b i t of the pressure o f f , depth of the screen you're taking a l i t t l e b i t of the pressure o f f , for the soup to go through the data base areas, to check the re t a i n i n g , and get that over there and clean up that. I t depends l i k e , t h e i r system i s a l i t t l e d i f f e r e n t than ours. In where I am I'm i t , I'm the guy that signs on and runs the queues and that s o r t of t r i e s to keep everything together. Well you don't r e a l l y have a heck of a l o t of time to test -- you might pick up the phone a few times and r i p i t outside maybe do i t t h a t way. Your main f u n c t i o n i s to make sure that everyfields gets c l e a r , and you know that things get done and that's what you are there f o r . (So that's what you guys are here f o r . ) Well, Doug and I, we p u l l , each of us sign on as a .) I t says no question purging out. No question then that we can view i t as serious -- some ki n d of screening that i s You would have to take courses i n someone e l s e ' s j o b or actual whatever. g e t t i n g on i n your area. And s t i l l there i s an analyzing , something that's g i v i n g us the tough or being p r e f e r r e d depends on the s i z e of the operation and / trouble with and mutuals that s t i l l what's going under. What's that. The process that.... They have a -- guys that j u s t view of the outside guys too. It's completely separate you're i n a d i f f e r e n t end of the b u i l d i n g among others. Where the i n i t i a l t e s t i n g i s done and we can scratch i n a d i f f e r e n t area and there's three guys there Just answer the outside guys. I t a l l they do; i t ' s t h e i r f a u l t an i n t e r a c t i v e r o l e They've got s p e c i f i c t a s k s , s p e c i f i c people to d i s p a t c h on In an operating centre there's three guys i n mutual, two guys i n r e a l . . . . two guys i n North Vancouver. A l l they do i s work outside. That's a l l Assuming. It s o r t of depends on the s i z e of the operation, the functions do phone them, with .... screening those running c a l l s . We don't have a screen on. Okay, then screen i t on. Now i t ' s j u s t saying pay i t on the area. You have to look at i t that way -- i t ' s important mentioning yes. That i s a function for somebody who doesn't understand the system by a l l means. Brings them a l o t more to t r y and can use this type of function i n A u s t r a l i a . I can say i t ' s You can't be with that functions as near as yeh they're organized. Yeh, i t ' s a l a t e s p l i t u p . . . . s p l i t up they can separate except the job functions. The functions that are there are j u s t people, depends on the s i z e of their operation. Steve gets to And i t screws up on them. That's r e a l l y You phone y o u r s e l f and give yourself h e l l . Yeh. Okay, does that give us a front-end p o r t i o n or something to hang anything on them? We can do i t that way? Is i t s t i l l v a l i d or i s i t s t i l l a good thing to go ahead and work through the o l d system i s t r y i n g to cross out the one? APPENDIX C Diary Form 107 1 Date Time Place Initiator (person initiating the event) What triggered (caused) the eventi (i.e. what type of information, e.g. wish for clarification, new information obtained, somebody's comments, a bug in the system, a new idea, wish to improve code etc.) Person(s) Involved Machine(s) Involved Description of what took place Duration of Activity Objective you wished to achieve before start of activity Outcome : what was accomplished, what was not What do you plan to do next? Thoughts or Comments APPENDIX D Deliverables 108 0 CjJL 108a j a r - 3 a j r v A ^ e . jcrr VICTIM L JtGuA£ Z ) 108b 1 0 8 c 1 0 8 d . 11 ' 1 V. 108e I c -(So V ^ O v ^ 108f Case: A'* » ) a. b. c. d. e. f. a. o. P-q. r. a. L a. v. w. X. y. yy. zz. loteationally Induced Wrongful Act False Report Insufficiently Serious Plaintiff Only Harm Plaintiff Only Risk Plaintiff Harm, Third Party Harm A -Plaintiff Risk, Third Party Risk Plaintiff Harm, Third Party Risk Plaintiff Risk, Third Party Harm Third Party Only Death Third Parry Only Injury Third Party Only Risk _ _1 1. I T Physical 2. Mental 3. Sleep 4. , No Proof 5. Held for Pla int i f fXLI 6. Held for Defendant 1. Check t, II, and III. 2. If check In IA then check C 3. If check In IB then check D. Child Spouse Parent Sibling Living Together Engaged Rescuer Not Close Relationship .Witness , Nearby and Attended Imminent Accident Traces of Incident Hospital Visit Disfigurement Insufficient Exposure D. 108 g 1 0 8 h 108 j Cat*: A . JurlsdWtloa 1. Australia 2. Canada 3. Great Britain 4. New Zealand Holding 1. Plaintiff 2. Defendant VlctU Incidents 1. Deliberate Harmful Act 2. Negligent Harmful Act D. Act iv i ty 1. Rescuer 2. Thin skull physical 3. Thin skull physical & psychological 4. Aggravated a. medical natural deterioration b. medical negligent treatment medical non-negligent treatment criminal secondary acts non-criminal secondary acts P's own secondary act c. d. t. 1. Weapons 2. Machinery 3. Explosives 4. Fire 5. Inflammable substances 6. Toxic substances 7. Electricity 8. Radioactivity 9. Animals 10. Transportation: a. car b. motorcycle c. truck d. van e. bus f. tractor unit g. trailer h. tractor trailer L snowmobile j. railway engine k. railway car I. train m. vessel n. airplane o. animal power S. Nothing unusual F. Nature of Remoteness a. Victim b. Method c. Damages -o « c « u X M 11. Not Dangerous 108 1 JOB PROFILE TOOLS t EQUIPMENT The tools and equipment available to the TEST DESK PERSON include: 1.0 Touch Tone Telephone Set (All exchanges) 2.0 CRT TERMINAL ( a l l exchanges) Locally based (IBM PC workstation) Tool for addressing 4 TEL, TAS, CRIS, MAIS, FMS, SOUL and other MIS information or other Telco Depts. Other Software DOS, Crosstalk, Desk View, Pascal subrouting for accessing 4 TEL SAC UNITS through MUX 3.0 4 TEL - 4 WAY TELEPHONE TESTING SYSTEM (All exchanges) Locally (service centre) Based Testing System Business and Residence 4.0 TAS - Trouble Administration System - (All exchanges) MIS based" Holds and routes - Trouble reports - business and residence 5.0 CRIS - Customer Records Information System ( A l l exchanges) MIS based Holds Residential - Service Orders - Customer Data (Name, Address Etc.) - F a c i l i t y Data (Cable pair, Exchange Hardware I.D. Etc.) - B i l l i n g Data - Comments Holds Busines - Customer Data - B i l l i n g Data - Comments 108m JOB PROFILE 6.0 SOUL - SERVICE ORDER UPDATE AND LOCATE (All Exchanges) HIS based " Holds - Business Orders - Leasewire Orders 7.0 MAIS - MECHANIZED ASSIGNMENT INFORMATION SYSTEM (Not a l l Exchanges) Holds f a c i l i t y information (New York Based) Residence and Business 8.0 FMS - FACILITY MANAGEMENT SYSTEM (Not a l l Exchanges-Yet) Holds Detailed Facilty Data (MIS Based) Residence and Business 108m f i g . 13 , TROUBLE ANALYSIS CENTER CURRENT (Not lor OvwniQM S—«•») f i g . 13 TROUBLE ANALYSIS CENTER FUTURE f i g . 15 108p f l a i g AGENDA TOR TEST DESK COURSE y ' TASK ANALYSIS SESSION 30 NOV 87 - 4 DEC 87 30 NOV -1. I n t r o d u c t i o n s - Dan Brophy 2. Purpo i t of s e s s i o n - Gord Bacon 3. Information about the p r o c e s s and outcomes - J e f f Howel l 4. Quest ions & comments 5. Short Break - 9:15 6. Update 1981 Tes t Desk Task A n a l y s i s Document LUNCH 11:45 - 12:45 7. Continue wi th 1981 TA Update 8. Short Break - 2:00 9 . Continue wi th 1981 TA Update 10. Summary 4 c leanup - 3:45 11. End Session f o r day - 4:00 1 DEC -1. Review of l a s t s e s s i o n t q u e s t i o n s - 8:00 2. Continue wi th 1981 TA Update 3. Short Break - 9:15 4. Continue wi th 1981 TA Update LUNCH 11:45 - 12:45 5. Continue wi th 1981 TA Update 6. Short Break - 2:00 7. Complete 1981 TA Update 8. Summary t c l e a n u p - 3:45 9. End Session f o r day - 4:00 108q 2 DEC -1. Review of l a s t s e s s i o n t q u e s t i o n s - 8:00 2. Update 1986 4TEL Task A n a l y s i s 3. Short Break - 9:15 4. Cont inue with 1986 TA Update LUNCH 11:45 - 12:45 5. Cont inue with 1986 TA Update 6. Short Break - 2:00 7. Complete 1986 TA Update 8. Summary t c leanup - 3:45 9. End S e s s i o n fo r day - 4:00 3 DEC -1. Review o f l a s t s e s s i o n 4 q u e s t i o n s - 8:00 2 . I n t e g r a t e 1981 and 1986 Task A n a l y s i s 3. Shor t Break - 9:15 rA. Cont inue TA i n t e g r a t i o n 5. Complete TA i n t e g r a t i o n 6. Short Break - 2:00 7. Review ongoing course and ES development p r o c e s s f o r upcoming y e a r , and d i s c u s s any other o u t s t a n d i n g i t e m s , i s s u e s or q u e s t i o n s 8. Summary « c leanup - 3:45 9 . End S e s s i o n fo r day - 4:00 1 0 8 q DEC -1. Review of week and l a s t s e s s i o n 4 q u e s t i o n s - 8:00 2. V a l i d a t i o n 4 s i g n - o f f of r e v i s e d T e s t Desk Task Ana lys 3. Short Break - 9:15 4. Cont inue v a l i d a t i o n 4 s i g n - o f f LUNCH 11:35 - 12:55 5. Complete v a l i d a t i o n 4 s i g n - o f f 6. Summary of weeks a c t i v i t i e s , q u e s t i o n s , comments 7. End Task A n a l y s i s S e s s i o n - 2:30 1 0 8 q f i g . 17 Fault Analysis Flow Diagram Part Five ^ CWTO ^ f^CU»TOtl C A U ^ Q DATA J OM.TCM TUT COUPAflC CUSTOMS". Ae<x*=D» WTTH DATABASE PERFORM MOMTCM TEST-WAT C O U PUUEO ROUTE TO CO. CAU.CUSTCAS-" OOlCOPAfCG* TESTwrm CUSTOMER fOUTt AOUTi TO TO? OAT MSPATCM FOUiOWU* 1 0 8 r f i g . 18 1 0 8 s 108s 1 0 8 s OR t h e r e IS an a v a l a n c h e OP t h e r e a r e h i g h w i n d s ! 00 im fir OP' t h e r e ARE o t h e r A c t s o f God THEN c u t of c o m p a n y - f a c t o r s e j e c t i n g p l a n t c o n d 1 1 i o n s R U L E -for d e t e r m i n i n g A c t s o f Man e f f e c t i n g p l a n t c o n d i t i o n s IF t h e r e IS h e a v y c o n s t r u c t i o n i n t h e a r e a OR t h e r e IS a l a r g e a u t o a c c i d e n t OR t h e r e IS s a b o t a g e OR t h e r e ARE o t h e r A c t s o-f Man THEM o u t o-f c o m p a n y f a c t o r s e - f - f e c t i n g p l a n t c o n d i t i o n s R U L E -for d e t e r m i n i n g h o l i d a y e - f - f e c t i n g p l a n t c o n d i t i o n s IF i t IS a h o l i d a y THEN o u t o-f c o m p a n y f a c t o r s e - f - f e c t i n g p l a n t c o n d i t i o n s R U L E -for d e t e r m i n i n g i-f d i s p l a y s c r e e n i s c u s t o m e r o r c o m p a n y r e p o r t e d IF TAS s c r e e n i s l a b e l e d c u s t o m e r r e p o r t e d THEN t r o u b l e i s c u s t o m e r r e p o r t e d E L S E t r o u b l e i s c o m p a n y r e p o r t e d R U L E f o r d e t e r m i n i n g i f t r o u b l e t y p e i s c o r r e c t I F t r o u b l e t y p e i i s t e d IS NDT OR t r o u b l e t y p e 1 i s t s d IS CCO OR t r o u b l e t y p e 1 i s t e d IS CBC OP t r o u b l e t y p e 1 i s t e d IS RWN CR t r o u b l e t y p e 1 i s t e d IS 00L OR t r o u b 1 e t y p e l i s t e d IS N8E OF t r pi ib 2 e t - B ? 1 i s t e d IS CH OR t r o u b l e t y p e 1 i s t e d IS C&H CR t r o i t l e t v p e 1 i s t e d 13 CUTC OR t r OL b 1 s t y p e i i s t e d IS FHY 3 OR t r nu t y p e 1 i s t e d IS- CUSTOM C A L L OR t r o u b i e t y p e 1 i s t e d IS O a t a THEN t . - o u b l e t y p e r e p o r t e d l s v a l I d R U L E i o r p r o o f r e a d i n g t h e TAS t r o u b l e r e p o r t I~ t r o u - l e t y p e i s c o r r e c t AND OS NGS f i e l d IS c o m p l e t e AND p a r t v l i n e f i e l d IS c o m p l e t e AND t i m e r e p o r t e d IS c o m p l e t e AND name f i e l d IS c o m p l e t e AND a d d r e s s f i e l d IS c o m p l e t e AND c i t y f i e l d IS c o m p l e t e AND who r e p o r t e d t h e t r o u b l e f i e l d IS c o m p l e t e AND C P E f i e l d IS c o m p l e t e AND a l l p h o n e s f i e l d IS c o m p l e t e AND c r i t i c a l r e a s o n IS c o m p l e t e AND a l l c a l l s f i e l d IS c o m p l e t e AND c a b l e f i e l d IS c o m p l e t e AND p a i r f i e l d IS c o m p l e t e T H E N a l l n e e d e d f i e l d s a r e c o m p l e t e d E L S E f->ere i s an e r r o r i n t h e T A S r e p o r t R U L E f o r p r o o f r e a d i n g b u s i n e s s c u s t o m e r T A S t r o u b l e r e p o r t IF a l l n e e d e d f i e l d s a r e c o m p l e t e d AND l i - . e number f i e l d IS c o m p l e t e 1 0 8 t THEN l i n e s t a t i o n i n f o r m a t i o n i s p r e s e n t E L S E l i n e o r s t a t i o n i n f o ma> b e n e e d e d f r o m c . s t o n i e r •ecorc ' END 1 0 8 t WASTES S Y S T E M DOMAIN RAW SEWAGE PRIMARY SlUOCC RAW 3EWACC UFT SCREENING CH.ORWE CONTACT TANK T EmUENT WASTE SLUOCC • SLUOCC DISPOSAL FILTRATE FIGURE ZO- Conventional Treatment P l a n t Layout and WASTES Syatem Domain 108u FIGURE 11 - RACKS.KB Logic S t r u c t u r e 108v • ( a r m iiiiiiiiin f M I 7^— LOADED IN MEMORY WASTE. K B R B C . K B FILES ON DISK LI FT. K B R A C K S . K B GRIT.KB MICRO.KB I L . FLOAT. K B O D O R . K B FICURE 2Z - WASTES F i l e S t r u c t u r e 108x FIGURE 2-1.- GRIT.KB Lo g i c S t r u c t u r e 108w 


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items