UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

An empirical investigation of knowledge acquisition Chan, Christine Wai-Chi 1988

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


831-UBC_1988_A4_6 C42.pdf [ 5.55MB ]
JSON: 831-1.0051903.json
JSON-LD: 831-1.0051903-ld.json
RDF/XML (Pretty): 831-1.0051903-rdf.xml
RDF/JSON: 831-1.0051903-rdf.json
Turtle: 831-1.0051903-turtle.txt
N-Triples: 831-1.0051903-rdf-ntriples.txt
Original Record: 831-1.0051903-source.json
Full Text

Full Text

AN EMPIRICAL INVESTIGATION OF KNOWLEDGE ACQUISITION by CHRISTINE WAICHI CHAN  A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE  in THE FACULTY  OF GRADUATE  STUDIES  Commerce & Business Administration  We accept this thesis as conforming to the required standard  THE UNIVERSITY OF BRITISH COLUMBIA 16 September 1988  © Christine Waichi Chan, 1988  In  presenting  degree  this  at the  thesis  in  University of  freely available for reference copying  of  department  this or  publication of  thesis for by  his  or  partial  fulfilment  of  the  requirements  British Columbia, I agree and study. I further  that the  representatives.  It  is  this thesis for financial gain shall not  granted  <£ BuSfAJESS  The University of British Columbia 1956 Main Mall Vancouver, Canada V6T 1Y3 Date  r-\r /- / *> /ft -« \  Poroses /o. /f<f<f  advanced  Library shall make  by the  understood  that  it  extensive  head of copying  my or  be allowed without my written  permission.  Department of ^m/ME/QCe  an  agree that permission for  scholarly purposes may be her  for  /flO^l/A//Sr/l^T/0Af  ABSTRACT Expert systems are being developed  despite the widely acknowledged problem of  acquiring  This  knowledge  from  experts.  study  attempts  to  understand  how  knowledge acquisition is conducted in practice by investigating three expert system development  projects.  The  variables  examined  include  the  expert,  knowledge  engineer, problem domain, organizational setting, the knowledge acquisition process, the expert system  construction process, and  research methodology is adopted  and  data  the expert system  is collected through  itself. A  case  observation and  taped protocol of knowledge acquisition sessions, post facto interviews with the participants involved, journalistic accounts kept by  the subjects, and  deliverables  produced.  in the  of law  Three  cases  on  expert  systems  built  negligence, telephone line fault diagnostic, and investigated. By findings  reported  wastewater treatment  juxtaposing the observations drawn from in  the  literature,  this  inquiry  understanding of the knowledge acquisition process.  ii  domains  of  have been  these cases with the  contributes to  the  current  T A B L E OF CONTENTS Abstract  ii  Table of Contents  iii  List of Figures  .  iv  Chapter Introduction 1.  1  Chapter Research Model 2.  4  Chapter Research Methodology and Design 3.  12  Chapter Case on the Remoteness Advisor 4.  18  Chapter Case on the Test Desk Decision Support Tool 5.  51  Chapter Case on the Wastewater Treatment Expert System 6.  77  Chapter Conclusions  88  7. Bibliography  98  Appendix A  105  Appendix B  106  Appendix C  107  Appendix D  108  iii  List of Figures Figure  Page  1  Research Model: Manual Development of an Expert System  9a  2  A Validation-Based Life Cycle Model of the Knowledge-Based System Development Process  89a  3  K A Process for the Remoteness Advisor  91a  4  K A Process for the Test Desk Decision Support Tool  91b  5  K A Process for WASTES  91c  6  Summary of Characteristics of the Three Cases  9 Id  D.l  Physical Layout of Smith's Office  108a  D.2  Types of Remoteness  108b  D.3  Types of Activities  108c  D.4  Risk of Harm  108d  D.5  Risk  108e  D.6  Harm  108f  D.7a  Structure of Nervous Shock Advisor  108g  D.7b  Structure of Negligence Advisor  108h  D.8  Structure of Remoteness  108i  D.9  Structure of Remoteness  108j  D.10  Remoteness Advisor (Epsilon Project)  108k  D.ll  Structure of Remoteness Advisor  1081  D.12  Job Profile  108m  D.13  Trouble Analysis Centre Current and Future  108n  iv  D. 14  Physical Layout of Demonstration Room  108o  D.15  Draft Diagram of Test Desk Advisor  108p  D.16  Agenda for Test Desk Course Task Analysis Sessions, 30 Nov 87 - 4 Dec 87  108q  D.17  Fault Analysis Flow Diagram  108r  D.18  Redrawn Flowchart for No Dial Tone Type of Trouble  108s  D.19  Sample Rule Base for the Test Desk Decision Support Tool  108t  D.20  Conventional Treatment Plant Layout and WASTES System Domain  108u  D.21a  Racks.KB Logic Structure  108v  D.21b  Grit.KB Logic Structure  108w  D.22  WASTES File Structure  108x  v  Acknowledgement  I wish to thank my thesis supervisor Professor Izak Benbasat for introducing me to the field of knowledge acquisition and for his guidance throughout the various stages of the research process. I am  grateful also to Professor Robert Goldstein and Professor J.C.  Smith for their careful reading and comments on earlier drafts of the thesis. I am  especially indebted  their experiences  to the participants in the three cases  who shared  in knowledge acquisition and system construction and gave  with  me  generously  of their time and insights. These individuals include Professor J.C. Smith, Cal Deedman, Jeffrey  Howell, Dan Brophy, Marie Burlinson, Barry  M. Chilibeck, and Dr. William  Oldham. This work could not have been completed without their help.  vi  C H A P T E R 1. INTRODUCTION  Expert systems (ES) are widely acknowledged to be the fastest growing systems within Artificial Intelligence (Al) applications to reach the business and industrial sectors. The expert system portion of the A l market is estimated  to climb to  $350 million by 1990 [Sviokla, 1986].  Knowledge acquisition (KA) is an essential phase in the development of expert systems. It involves eliciting, analyzing, and interpreting knowledge that a human expert  uses  information  when into  solving  a  suitable  a  particular machine  because the power and utility  problem  and then  representation. This  of the resulting expert  transforming stage  system  this  is important  depend  on the  quality of the acquired expert knowledge which is later represented in the system [Kidd, 1987:1].  Knowledge acquisition is a major bottleneck in the efficient development of expert systems  primarily  knowledge  because  [Breuker  of the difficulty  et al.,1987;  Abrett  experts  have  et al.,1986;  in verbalizing  Gaines, 1987d].  their  Experts  generally find it difficult to formulate their knowledge explicitly but find it easy to demonstrate  their  expertise in specific  situations  [Witten  et al.,1987]. The  phenomenon is also referred to as the paradox of expertise [Johnson, 1984].  Despite the difficulty of accessing expertise widely acknowledged in the literature, expert systems are being built everyday. Dupont alone projects that by the year 1991,  over  2000 systems will be in place. [Watkins et al., 1987] It has been  1  Introduction / 2 pointed  out that  knowledge  most  acquisition  knowledge  [Alexander  engineers  use an "intuitive"  et al.,1986]. This intuitive  approach to  approach  has  in  many cases succeeded in sufficiently covering the problem domain and providing a basis for expert systems that perform satisfactorily. So how did they do it?  The objective of this study is to empirically investigate the knowledge acquisition efforts currently underway. The need for empirical research into expert systems development and use has been suggested in the literature. Gaines advocated more variety,  detail  techniques and  and evaluation  in "the case  histories  of applications  of  the  tools" [Gaines, 1987e]. Referring to the lack of understanding of  expert system implementation issues, Sviokla wrote as follows: ...more empirical evidence is needed from which more generalizable inferences can be drawn. Experience is still scant and generalizations are hard to create with confidence. [Sviokla,1986:363].  The study attempts to address this need.  The question the study poses is as follows: How is knowledge acquired in developing an expert system?  It is expected that the inquiry would shed light upon the process by providing a set of useful observations to guide future attempts on a practical level and data for hypothesis generation for research purposes..  The contents of the following chapters are outlined as follows: Chapter 2 presents a brief overview of expert system in  some  existing  literature, a model that  guided  lifecycle models described  the investigation, and some  Introduction / 3 definitions of knowledge acquisition; Chapter 3 explains the research methodology and design; Chapter 4 reports on the case on the Remoteness Advisor; Chapter 5 reports on the case on the Test Desk Decision Support Tool; Chapter 6 reports on the case on the Wastewater Treatment Expert System; and Chapter 7 concludes the inquiry by presenting some lessons learnt, limitations of the study, and suggestions for future research.  CHAPTER  2. R E S E A R C H  MODEL  Literature descriptions on the expert sj'stem development lifecycle provide some guidelines on the steps involved in acquiring knowledge system. This chapter presents a brief overview lifecycle  described  in the literature,  and building an expert  of the expert system development  and explains  a model  of expert  system  development that has guided the identification of variables and the compiling of a questionnaire.  In  contrast to the popular  acceptance  of the system  development  life cycle  (SDLC) which describes development processes of a conventional data processing software,  general  agreement  in the literature  on the development lifecycle for  expert systems does not exist. A brief summary of some of the lifecycle models that have been advanced is presented in the following discussion.  2.1 B r i e f O v e r v i e w o f E x p e r t S y s t e m L i f e c y c l e M o d e l s .  Buchanan  et al.[1983] outlined the "major stages  include the following: 1.  identification stage: to identify a.  participants in the process and their roles,  b.  the problem domain,  c.  resources needed,  4  of knowledge  acquisition" to  Research Model / 5 d. 2.  the goals of building the system.  conceptualization stage: to explicate the key concepts problem  domain, the result of this  stage  and relations in the  is a conceptual  base for  the  system; 3.  formalization stage: to map the key concepts, subproblems, and information flow  characteristics  isolated  during  conceptualization  into  more  formal  representations based on various knowledge engineering tools or frameworks. The  result of this stage is a partial specification for building a prototype  knowledge base. 4.  implementation from  stage  stage: this involves the mapping of the formalized knowledge  (3) to a representational framework  associated with  the tool  chosen for the problem and development of a prototype; 5.  testing stage to evaluate the prototype system.  Buchanan et al. supported  the notion of "rapid prototyping" [Gaines, 1987b] and  considered it advisable to move directly to the formalization and implementation stages as soon as key concepts and relations had been identified. Their view is expressed as follows: It will be tempting to try to analyze the problem correctly and completely before implementing a trial system, but experience has shown that to be a mistake. Once the key concepts and relations are written down, much can be gained from formalizing them and working toward an initial implementation [Buchanan, 1983:144].  Grover [1983] described a KA cycle to comprise of the following three stages: 1.  domain  definition  - this  stage  aims  to produce  a  "Domain  Definition  Handbook" which consists of "a general problem description, a bibliography of reference documents, a glossa^ of terms, acronyms, and symbols, the  Research Model / 6 identification realistic  of authoritative  performance  "experts", the definition  metrics,  and  a  description  of appropriate and  of  sample  reasoning  scenarios. 2.  fundamental knowledge formulation - in this stage, scenarios chosen by the expert are reviewed to form a "baseline  for minimum  performance" which  include the following : a.  an ontology of domain entities, object relationships (classes) and object description;  b.  a selected lexicon;  c.  a definition of input sources and formats;  d.  an initial state description including static "background knowledge";  e.  a list of human  strategies or meta-rules which may  become control  rules for the system [Grover, 1983:437]. 3.  basal knowledge consolidation and  - basal knowledge refers to the set of rules  definitions adequate to produce the lowest level of activity or system  behaviour essential for maintenance of vital functions. fundamental reconstruction  knowledge of rules.  corpus Issues  is reviewed considered  At this stage, the  and enhanced include  control  by appropriate strategies,  rule  sequencing, rule adequacy, rule consistency, and confidence values; additional experts maj' also be consulted. This stage is repeated with introduction of novel scenarios until the system performs satisfactorily [Grover, 1983:438].  The  Guide  to Expert  Systems  Program  Management  developed  by  Digital  Electronics Corporation (DEC) outlined the knowledge engineering project to consist of the following steps:  Research Model / 7 1.  develop inital program definition;  2.  build initial prototype;  3.  develop program plan;  4.  develop education plan and user involvement;  5.  create design document;  6.  build the basic shell;  7.  develop adaptively;  8.  conduct field test;  9.  release product to production;  10.  enhance and adapt the system;  For a detailed discussion of the steps, see [Boose, 1986].  Fellers [1987] advanced which  consisted  of  a simpler model of the knowledge engineering process  two  phases:  knowledge  acquisition  and  expert  system  construction. The former process involves "one or more knowledge engineers (KEs) interacting with one  or more domain experts (DEs)", and  the objective of this  interaction is to develop a "shared representation" or model of the DE's solving process. Once the shared or external representation has the system construction phase begins. The and  maps  continues  that with  to the  the  actual  expert  knowledge  interacting  with  KE  problem  been developed,  takes the external representation  base the  (KB).  The  prototypical  KA  process  expert  then  system  to  develop the internal representation or mental model of the system. The objective is to increase the congruence between this internal representation of the system and the mental model of the expert [Fellers, 1987].  Research Model / 8 Benbasat and Dhaliwal [1988b] propose a lifecycle model of the knowledge-based system development process which differs from other similar models [Buchanan et al., 1983, Fellers, 1987] in that "it specifically isolates, • at different stages of the lifecycle, three progressively more developed knowledge bases". The model consists of four stages : 1.  modelling  phase  to capture  knowledge  from  the source  experts  to a  conceptual knowledge base; 2.  knowledge elicitation phase which produces an elicited knowledge base, i.e., "a 'complete' expertise specification formalized representation  (for example,  rules,  to some  selected  knowledge  frames, or semantic nets) but in a  nonmachine-readable form" [Benbasat et al., 1988b: 12]; 3.  the system construction phase that maps the elicited knowledge  base into  an implemented knowledge-based system, and 4.  experimentation and use phase which refines the knowledge base.  Sviokla summarized  the few ES project lifecycles he gleaned from the literature  [Sviokla, 1986:354] and noted the absence of the specification  or requirements  analysis and maintenance stages in the ES project lifecycle [Sviokla, 1986:354]. In the DEC  ES  program  management cycle, "build  initial prototype" takes place  immediately following "intial program definition." Implicitly, "rapid prototyping" is the norm.  Rapid process  prototyping of  is a well-established  knowledge-based  systems  technique  employed  [Gaines, 1987b].  in the development  It describes  knowledge  elicitation to take place with the knowledge engineer interviewing the expert and  Research Model / 9 then  encoding  interviewing prototype,  the  verbal  knowledge  into  rules.  Then  sessions in which the expert comments the expert  system  knowledge  base  through  interative  on and helps debug the  expands  and  is refined  until  satisfactory performance results [Young et al.,1987]. This approach is supported by  the models described by Buchanan [1983] and in the DEC  expert system  program management cycle [Boose, 1986].  Focusing specifically on the KA  process, Gaines illustrated it with a model of  interaction, which is used to guide the initial identification of variables in this study. The model suggests the actors in the process to include the expert, the knowledge engineer (KE), and the users, and the software involved to include the expert system shell and the end product, a knowledge base. The study focuses on the two processes, primarily on the knowledge acquisition and secondarily on the expert  system  construction process. A  slightly  modified  version of Gaines'  model is shown in figure 1 [Gaines, 1987a].  Based on this simple model, the variables to be addressed include the following: 1.  problem domain,  2.  actors involved: a.  expert,  b.  KE,  3.  knowledge acquisition process,  4.  expert system construction process,  5.  potential users,  6.  organization setting,  F i t . 1 Manual development of aa expert system  Research Model / 10 7. A  expert system itself. questionnaire  covering  these  variables  was  compiled,  which  collection. As can be seen from the questionnaire, the main  guided  data  emphasis of the  study is on the knowledge acquistion process. For a copy of the questionnaire, see  appendix A.  2.2 Knowledge  Knowledge  Acquisition : Definitions.  acquisition  has been  defined  in a variety  of ways, some  sample  definitions that have been proposed include the following:  "A process of gathering knowledge from one or more domain experts for the utilization in a knowledge based system" [Jacobson et al.,1987]. "The transfer and transformation of problem solving expertise from some knowledge source to a program. Potential sources of knowledge include human experts, textbooks, databases, and even one's own experience"[Belkin et al.,1986].  An  attempt to differentiate knowledge  acquisition and knowledge elicitation has  also been made as follows:  "Elicitation is the process of developing a model by interviewing the experts and observing their environment; it is theory formation or model design. Acquisition is the process of collecting the detailed information (facts) that fit into the framework defined by the model" [Addis, 1987].  The  distinction between  the two terms is not observed in this study  definition of knowledge acquisition adopted is stated as follows:  and  the  Research Model / 11 "Knowledge acquisition consists of the elicitation and interpretation of data on the functioning of expertise in some domain, in order to design, build, extend, adopt or modify a knowledge based/expert system (KBS). In this view, knowledge acquisition is a permanent and crucial activity throughout all stages of designing, implementing, and maintaining an expert system" [Anjewierden,1987; Breuker et al,1987J.  Hence,  in the  elicitation expertise.  following  discussion,  of knowledge, but also  knowledge  analysis  and  acquisition  involves  interpretation  not just  of the acquired  C H A P T E R 3. R E S E A R C H METHODOLOGY AND  To study the KA adopted. The methodology  and ES  purpose  DESIGN  construction processes, a case research methodology is  of  this  is appropriate  chapter  for the  is to  present  explain  study  and  why  a  case  research  the  research  describe  design.  3.1 Research Methodology.  Bonoma [1985] suggests  case  research to be  appropriate for investigations into  areas "where the existing body of knowledge is insufficient to permit the posing of causal questions, and  when  a  phenomenon  cannot be  studied  outside  the  context in which it naturally occurs" [Bonoma, 1985,207]. Benbasat et al. noted three reasons for the use of case research as follows [Benbasat et al.,1987:370]: 1.  the researcher can study information systems in a natural setting and learn about the state of the art, and generate theories from practice.  2.  the case method allows the researcher to answer 'how' and  to understand  the  nature  and  complexity  and 'why'  of the  questions  phenomenon being  studied. 3.  a case approach is appropriate for an  area in which few  previous studies  have been carried out. An  empirical investigation into how  an  expert system is developed  criteria for case research advanced in the above studies. The and  lack of scientific  engineering foundation for expert systems has been suggested  12  satisfies the  as the major  Research Methodology  and Design / 13  obstacle in establishing professional standards for expert system development, and the lack of well-established techniques for knowledge engineering poses a major problem for new development  teams [Gaines,1987a]. The phenomenon of interest,  i.e. the acquisition process, takes place in organizations that currently develop such systems; an inquiry into the undertakings would enable the researcher to answer the question of how the systems are produced.  Bonoma poses the four stages in the case research process to be drift, design, prediction, and disconfirmation primarily  the first  stage  and aims  narration, lessons drawn from generation.  The procedure  [Bonoma, 1985:206]. This to understand  the cases form  closely  investigation  focuses on  and describe.  From the  the basis for future hypothesis  approximates  Bonoma's  statement  on the  objective of case research: The goal of data collection in case research is not quantification or even enumeration, but rather (1) description, (2) classification (typology development), (3) theory development, and (4) limited theory testing. In a word, the goal is understanding [Bonoma,1985:206].  3.2  Research Design.  In contrast to the scant work on expert systems with  single  attempts  cases  or action  to empirically  reports  investigate  [Benbasat  three  expert  reported to date which deals &  Nault,  system  1988],  this  development  study  projects.  Adopting an empirical approach to observational methodology [Weick, 1968:401], the questionnaire outlined the variables to address when the researcher observed the  Research Methodology and Design / 14 sessions for the first two cases and interviewed the subjects for the third.  3.2.1.  The  Unit  unit  of Analysis  and  Site  Selection.  of analysis consisted of the knowledge  acquisition  process  for  an  individual expert system. In practice, however, a clear distinction between this and  the development  process  often  cannot  be made.  Therefore,  whenever  necessary, the unit of analysis is extended to include the development process as well.  The sites were selected based on two criteria : 1.  an expert system was being developed,  2.  permission to conduct the study was procured.  Since  May  of 1987 when  the researcher  embarked  upon  the study,  7  development projects have been identified and approached. One was later found to be an unsuitable candidate for the study because it involved an intelligent front end for an existing system rather than an expert system. Another system was being developed  in a commercial context and the company involved was unwilling  to co-operate. Two other systems were shelved two months after the researcher had started collecting data from them. The three cases finally investigated include the  Remoteness Advisor  British  developed  Columbia, the Test  Learning  Services  Centre,  developed  at the Civil  Desk  at the Law Faculty of the University of Decision Support  and the Wastewater  Engineering  Department  Tool  developed  Treatment  at Microtel  Expert  System  of the Universitj' of British  Columbia. In two of the three cases, the researcher learned  of the systems  Research Methodology and Design / 15 through study  publications or word of mouth, and from  the  participants.  In  only  one  obtained case  approval to conduct  did  the  persons  the  involved  volunteered their project to be studied.  3.2.2. Data  Collection.  The study aims to track the knowledge acquisition and development processes as they occurred. This allows the researcher to study natural setting and  also ensures maximum currency of data. Bonoma noted the  major weakness in case Triangulation capture  a  of data  single  the phenomenon within its  research to be  is a  event. To  low  technique, which increase the  data integrity  [Bonoma, 1985:200].  uses  types  validity  different  of data, this  of data to  study  adopted  multiple data collection methods which include the following : 1.  direct (unobtrusive) observation of the knowledge acquisition and development processes between the expert and knowledge engineer (KE);  2.  taped protocols of the sessions;  3.  retrospective interviews with experts and  4.  diary or journalistic record of events kept by the KE;  5.  documentary evidence or deliverables produced during the process.  Since  subjects' co-operation  presented  to the KEs  was  essential,  KEs;  the  above  in the three projects. In two  list  was  consistently  of the three cases, they  agreed to co-operate on all methods except keeping a diary. In the third case, since the researcher did not learn of the project until it was  almost finished,  only post facto interviews were conducted  KE,  with the expert and  and  some  Research Methodology and Design / 16 deliverables were also obtained.  A  major  weakness  in observation  introduced  by  the observer  potentially  reduce the error. This  methods  is its susceptability  [Weick,1968:428]; having proves impossible  multiple  to biases  observers  in the present  could  study for  subjects often were relunctant to accomodate more than one observer while they worked,  whether  within  researcher began with and  expert  same  or  across  the intent of observing  in an unobtrusive  likewise found observer  the  manner  different  sessions.  While  the  the interaction between the KE  [Webb et al.,1966], this approach was  to be infeasible. Weick correctly suggested  causes reactions on the observed  that a non-participant  subjects [Weich, 1968:370], and that  some interference might be needed to maintain  rapport [Weich, 1968:311]. Often  the subjects directed comments to the researcher during the sessions but such interactions were kept to the minimum as much as possible.  A method adopted to "to enhance precision and validity of observational studies" is to introduce interference in the observation setting [Weich, 1968:366]. In this case, a tape recorder was used. Subjects proved to be unaffected by its presence, and  were exceedingly  co-operative in helping to tape the sessions whenever the  researcher could not personally attend them  as long as sufficient tapes and a  machine were provided. For a sample of the transcript of some taped protocol, see Appendix B.  Interviews with the KE case  on  the  and the experts were the main source of data in the  Wastewater  Treatment  Expert  System.  The  interviews  were  Research Methodology and Design / 17 semi-structured and the questionnaire  guided the discussion. Interviews conducted  for the other two cases were primarily for clarification purposes.  A diary form was developed and used successfully in the two cases in which the projects were shelved. In the other cases, the KE was reluctant to keep a diary for lack of time. See Appendix C for a copy of the diary form.  Some  deliverables  researcher was  were  obtained  and  proved  initially completely unfamiliar  by the systems. All figures referenced  to  be  indispensable  for the  with the three domains addressed  in the narrative description are included in  Appendix D.  The  data  collection  circumstantial  factors  methods and  were  willingness  adapted  for each  of subjects  described in greater detail in the following chapters.  case  depending  to co-operate; this  on  will be  C H A P T E R 4. CASE ON T H E  4.1  The  REMOTENESS ADVISOR  Introduction.  Remoteness Advisor is an expert system developed  Deedman at the Law  by Joe Smith and Cal  Faculty of the University of British Columbia. Smith, the  expert, is a professor of law  wrote a book on Liability  at the Faculty who  in  Negligence [Smith, 1984], in which he laid out a coherent theory of negligence developed  after 23  years of teaching, research, and  writing in the area of the  law of negligence. Deedman, the knowledge engineer (KE), doctorate in the interdisciplinary also a lawyer who The  two  had  program  of law  and  is currently pursuing a computer science. He is  has practiced for 10 years and decided to return to school.  previously co-operated  in developing another  expert system called  the Nervous Shock Advisor (NSA), which assists the user in determining if a nervous shock case can be established [Deedman, 1986]. Deedman learned importance of  developing the  knowledge structure before starting to  otherwise, much time is wasted changing the Teknowledge Inc. as a demonstration in July of 1987  and  code. The ss'Stem was  system at the AAAI Conference  code,  used by in Seattle  gave both developers their first experience of building a  system.  4.2  the  Negligence: Definitions.  A definition of negligence is quoted in Smith[1984] as follows:  18  Case on the Remoteness Advisor / 19 Negligence is the omission to do something which a reasonable person, guided by those considerations which ordinarily regulate the conduct of human affairs, would do, or doing something which a prudent and reasonable person would not do [Smith, 1984:129].  A  central  issue in negligence is the problem of remoteness, i.e. how far  down the chain of causality could a person be liable for a negligent act so that a damage would be recoverable. Negligence of harm  to someone.  The injuries  usually involves the creation of risk  of the primary  generally not too remote; the remoteness  subject  of the risk are  issue arises when  suffer a loss as a result of the negligence to the primary  other  persons  subject  [Smith,  1984:127].  Foreseeability is that which is natural and probable. Foreseeability of harm is a necessary  condition  for culpability  and consequently  for the liability  derived  [Smith, 1984:132]. The obvious case of foreseeability and culpability is where the harm caused is the very intent of the actor, in which case the actor is held liable for the intentional causing of harm by the law [Smith, 1984:130]. So for any  negligent act there are certain harmful effects which are foreseeable, which  when they  materialize as a result of an action, the remoteness issue is  involved. There reasonably  are other  effects,  however, which could  not be said  not  to be  foreseeable, and it is in regard to these that the issue of remoteness  of damage may be raised [Smith, 1984:132].  Traditional legal doctrine has posited two tests on when a damage is recoverable. The  first  is to allow  second is to allow  recovery  recovery  for damage foreseeable  for damage foreseeable  as probable,  and  the  as possible. Basically, the  Case on the Remoteness Advisor / 20 judge can choose between the two tests based on his discretion. If his decision is to  award  recovery, the latter  test  can be applied  for anything is possible;  otherwise, the former is used. Smith describes this situation as follows: "Foreseeability...becomes the justification for a conclusion about remoteness reached on other grounds, rather than the real test or basis of the  decision" [Smith, 1984:120].  Smith argues instead that a rational and systemactic basis for predicting recovery in remoteness cases can be formulated which allows us not to have to  "swing  between the poles of the direct causation and the foreseeability test, or between the poles  of the 'foreseeable and probable'  [Smith, 1984:131].  test and 'foreseeable and possible' test"  And he believes this rational approach is in fact used by  most judges : "It probably is the instinctive feeling of judges which is the bottom line in remoteness cases. The fact that...the instinctive feeling of judges has a 90 percent conformity would indicate that some basic moral principles about responsibility and blame are at work at an unconscious level. The intention has been to articulate that principle in the form of the following restatement of the foreseeability rule : Damages resulting from a negligent action are not too remote if they are reasonably foreseeable in the particular, or are one of the reasonably foreseeable class of injuries [Smith, 1984:160].  Hence, Smith proposes a new test for remoteness which involves two concepts : (1) damages foreseeable in the particular, and (2) damages that are one of the reasonably foreseeable class of injuries.  4.3 D a t a  Collection  Methods.  Knowledge acquistion for the Remoteness Advisor took place at Smith's office at  Case on the Remoteness Advisor / 21 the Law faculty of UBC. This office is a room about 10' by 12' (see Appendix D, figure 1). Smith and Deedman, usually worked in front of the flip chart, either standing  or sitting. Smith  regularly paced around during the  There were a total of six KA sessions, each took on average  discussion.  1.5 hours. The  first KA session took place on October 8, 1987 and the last on February 4, 1988.  At all except one session, the researcher was present to observe as well  as tape record the discussion between the expert and the KE. The one session when the researcher was absent, Deedman taped the session.  Before  or after  the sessions  and at occasions  when  the researcher  found it  difficult to follow the legal jargon, independent or joint interviews with the expert and  the K E  took  place.  The following is a narrative  description of what  transpired at each KA session. At the end of each description, some observations the researcher had made during the session plus relevant comments made by the participants during the interviews are presented.  4.4 Knowledge Acquisition.  4.4.1. K A Session I.  At the first KA session on October 8, 1987, two distinct topics were discussed; they included structure of the sj'stem and presentation format.  Structure  of the  System.  Case on the Remoteness Advisor / 22 The structure of the expert system was different in the minds of the expert and the KE. They had obviously spent time pondering about the issue before coming to  the first  KA  session. It was  equally  obvious  that  they  approached it  differently. Deedman suggested posing a question at the start to ask if there had been damage suffered as a result of a negligent act. Smith however, chose to bypass  the difficulty  for most  users  of trying  to determine  if they  had a  remoteness problem. He avoided words like "reasonably foreseeable" and instead posed questions which guided the user to think in concrete terms such as the activity which caused the damage, the damage itself, and the risk that justified the possibility of positively concluding causal relationship between the activity and the damage.  The  structure underlying this  foreseeable  in the particular,  approach in which  was  to find  case  there  out if the damage would  was  be no remoteness  problem. For example, it is reasonably foreseeable that if a plank is negligently dropped  into  the hold of a load ship, the cargo  would  be damaged [Smith,  1984:132]. If the damage was not foreseeable in the particular, then remoteness may be involved and the problem would be further clarified in detail.  The  structure of the system was a giant categorization scheme, hence, a major  portion of the time during the KA  session was spent on clarifying the activity,  damage, and risk involved in a case. A rudimentary the  appendix  of [Smith, 1984]  ( see Appendix  theory of negligence is in  IV and V  of [Smith, 1984] ).  Deedman, having read Smith's book, suggested adding some new categories to the basic scheme. For example, a new categor}' of radioactivity was added to the list  Case on the Remoteness Advisor / 23 of  causes  of damage  which  previously  only  had an item  on  inflammable  substances.  They  discussed how different items  asked  whether an animal-drawn vehicle ( which was involved in the negligent  activity  ) should  be categorized  could be classified. For example, Deedman  as vehicles,  or as negligent supervision of  animals. Smith decided since the system was a negligence advisor, negligence was assumed, so a category of damage caused by animals should be sufficent.  Deedman also posed questions to clarify Smith's definitions. Smith had one item called  "high  voltage  electricitj'"  and Deedman  suggested  simply  calling it  "electricity". Another category was termed "dangerous objects" which they  agreed  was too vague and was subsequently changed to "weapons". Format  There  was  of  Presentation.  substantial  different formats  discussion on software  issues. They  considered  for presenting the list of possible causes of damage to users.  Deedman suggested  listing the options exhaustively and then split them up for  menu selection because it was not "intuitively radioactivity  design  together." Smith  responded  appealing to have animals and  by noting that the options overlap so  simple yes/no answers was insufficient. The user should be able to choose say option one, two and any number of others. For that reason, he preferred keeping all alternatives on one level on the screen and let the user put together any combination. Then Deedman decided that focusing on one piece of information at  Case on the Remoteness Advisor / 24 a time would be easier to prototype than a menu selection. Smith, on the other hand, pointed out it would be more elegant if the user chose motor vehicle and the system responded  by posing the options under this category, but if the user  chose weapons, then nothing more was asked. Finally, it was decided that the categories would be presented one at a time, and cases that would be generated as a result of one option would include cases where others were involved so multiple selection would not be necessary. Observations  The  & Post-Session  Interview  with KE.  researcher made the following observations and asked the K E for verification  at a post-session interview on October 23, 1987. 1.  Style of interaction - It was obvious Deedman  had a close  sentences  for each other. Deedman agreed  was  a lawyer  working  to the researcher that Smith and  relationship.  Often  they  finished off  in the interview that since he  by profession, he and Smith  had a shared  vocabulary  which was a considerable advantage for him as a KE. 2.  Conflicts of opinion - Conflicts of opinions arose but were resolved when each tried to understand  what the other meant. On issues dealing with the  domain of negligence, Deedman deferred to Smith's opinions but Deedman's suggestions Deedman  and queries often clarified conceded  between their they  differed  in the interview  the matter with  in the expert's mind.  the researcher  that  the  gap  expertise on that topic was "enormous." On issues where in opinion, he relied  making sure Smith had understood  on Smith's definitive  statements  after  what his point was. Deedman tried to  Case on the Remoteness Advisor / 25 pose hypothetical cases to test Smith's categorization  scheme and he  used current news topics such as AIDS, the space shuttle disaster, and danger of pit bull terriers. He needed to attach Smith's ideas to concrete examples in order to make the ideas meaningful to him. 3.  Presentation  details discussed  - Although  the purpose of the KA  session  was explicitly to clarify on a conceptual level the domain of negligence, much discussion was  on presentation details. Smith had definite ideas on  what the resulting system would be like and often suggested to Deedman design  details  on  presentation  formats.  Deedman,  on  the other  hand,  admitted he had no idea what the system would look like. 4.  further pointed out it was impossible to create rules at this  The K E stage.  They  situations.  were  primarily investigating the options  by  posing  if-then  Case on the Remoteness Advisor / 26 4.4.2. K A  The  Session II.  second KA  1987,  exactly  researcher  session for the three  met  Negligence Advisor  weeks after  the  first  took place  on  29  October  session. Smith, Deedman, and  at Smith's office. Once again, their discussions focused  on  the the  issues of the structure of categories, and presentation format. But at this session, it was  difficult to separate out the three issues as they were closely interwined.  Hence, they will not be separately discussed in the following presentation.  Deedman began by  first  summarizing  what was  accomplished  in the  previous  session. First, they agreed to have the Remoteness Advisor establish if negligence was  involved by asking whether it was  a negligent or deliberate act. Then there  would be an introduction to each category the  user  involved  would and  be  the  led  through  damage  was  the caused  of possible causes of the damage and  categories. by  one  Secondly, of the  if negligence  combinations  was  of these  categories, then there would be recovery. Lastly, the format of presentation would be  to present  one  category  at  a  time, and  multiple  selection  would  not  be  necessary because cases that would be generated as a result of one option would include cases where others  were also involved. For  example, the list of cases  retrieved on the basis of "explosives" would include a case involving "a truckload of explosives," which would appear under both the category  of "explosives" and  that of "transportation," specifically, the sub-category of "truck".  Deedman had with  a few  thought about what happened during the three weeks and came up questions  for the expert. Basicallj', he  questioned  if the categories  Case on the Remoteness Advisor / 27 they discussed covered the whole field of remoteness. Smith affirmed that they did.  The only qualification  handled by another worry  was  in the case  of nervous shock. Since it was  system, the Nervous Shock Advisor, they did not have to  about it. Furthermore,  anything  that did not fall into  these categories  would be recoverable provided it was foreseeable in the particular. Deedman then suggested  having  a screening mechanism to determine  if it was remote before  leading the user through the categories. Three levels of questions would be posed : first, if negligence was involved, secondly, if remoteness was involved, and thirdly, if the cause of damage could be captured by the categories. If it could not, then the cause was too remote and the damage not recoverable.  Smith preferred to take the user through the categories without first concluding it was too remote and then  see if the cause fell within the risk. In other  words, the front end screening mechanism Deedman suggested was to be put at the back.  Deedman thought this approach might have the system asking about cases where no remoteness issue was involved and the user needed not have bothered with answering  all the questions. Smith agreed. Using this approach, the user would  be told at the end s/he could not recover because the damage fell within the risk. In most cases, smart lawyers would know there was no remoteness issue if the damage was foreseeable in the particular. In which case, they should not even use the system. (He said : "this is to catch the dummy  at the end." )  Deedman agreed and they moved onto discuss the categorization scheme.  Case on the Remoteness Advisor / 28 In discussing the categories of causes for the damage, Smith and wrestling  with  the  underlying  structure  of  the  expert  Deedman were  system. This  perhaps  explained the large percentage of time devoted to the topic. In the previous session, they already noted down a number of causes. They started by the  categories  and  discussing  various  detailed issues  such  as  KA  reviewing  ordering  of the  categories, specialization within general headings, groupings of various items, and presentation formats.  Means of transportation constituted the major group of causes for damage, hence both Smith and what  the  Deedman began their discussion with this cause. Smith asked  specific  negligent  acts  involved  in  the  operation  of  means  of  transportation were that caused the damage. The  specific categories under means  of transportation were dictated by  these included vessels, railway  the cases and  (with the sub-category of engines), motor vehicles (with the sub-category of cars), and  planes.  Then they considered machinery  behind  "mechanization." substances,  the ordering of the categories. Deedman suggested putting motor  Then,  electricity,  vehicles  fire and  and  because  inflammable  radioactivity,  which  thej'  both  substances, was  belonged followed  "a  modern  by  under toxic  form  of  electricity."  Presentation  of  preferred posing caused  the  the an  damage  categories  was  also  over-riding question related  to  addressed  such as  dangerous  in  : "was  activity  the  discussion.  Smith  the negligent act that  such  as  explosives,  fire,  Case on the Remoteness Advisor / 29 machinery, etc." And a list of these would follow. Deedman pointed out it might not be good to present the user with too much information because while it might "take them to the end faster", directing people's minds to individual things might be easier for the user. He then suggested a compromise : instead of going through  each eategory, group them together in pairs by asking explosives OR  fire, electricity OR  radioactivity, etc. If the user chose one or the other, the  system could branch to that, if s/he chose neither, s/he simply continue on. The rationale for this presentation method was to allow the user to proceed  in an  incremental manner. He compared it to talking on the phone, when one would not "list but ask them was it explosives or fire." Smith disagreed noting actually on the phone, one would simply "ask them what it was." Deedman agreed.  Smith apparently changed from an earlier position at this point. He had earlier remarked that the categories were discrete and that "nothing needed  anything  else." At this point, however, he noted that "overlap existed." For example, in a case where motor vehicle was the cause, thin skull or rescuer might also be involved.  There  were  two  activities. The latter such negligent  act. Smith  generally  applicable  encouraged combinations  him  then  major  : dangerous  as thin skull or medical pondered  categores  in this  divisions  and  endeavor  of categories such  versus  non-dangerous  complications apply to any  on the possibility  of starting  "get them  out of the way."  by  more  posing  scenarios  with  these  Deedman  that involved  as a car crashing into explosives, or a train  carrying radioactive stuff. Deedman suggested ordering by starting with the "most devastating."  Case on the Remoteness Advisor / 30 Smith  proposed  a solution. He  decided to "avoid overlap by  getting the right  order," which meant to begin with clear cut cases of recovery and then proceed with the categories. The order he put down was  as follows :  1.  nervous shock, which would exit to the nervous shock advisor;  2.  rescuer, who  3.  thin skull, or victim with particular susceptibility;  4.  medical complications; since they usually  would always recover;  happened  away  from  the actual  event, they should be last. Deedman noted down this ordering. The system would elicit information from the user by posing questions such as "was  the victim or did the victim" fall into  one of these categories. Relevant cases would then be retrieved.  Hence,  the  structure  that  emerged  categories, if none of these was discussed  earlier. Deedman  re-ordered them. He  flipped  had  the  system  invoked, it would back  to the  go  proceed  pages  on  through  the  four  to the categories the  categories and  noted at this point that two major groupings emerged :  1.  special types of people and  2.  special types of activities.  And  the. structure became clear in his mind : "the law allows recovery if it is  not too remote, if it is remote, then it needs to be special types of people or special types of activity. So it is foreseeable if it is a particular class of person or  particular class of activity." Based  on this, he reordered the categories under  special t3 pes of activities on the flip chart as follows : r  1.  motor vehicle,  2.  machinery,  Case on the Remoteness Advisor / 31 3.  explosives,  4.  fire  5.  inflammable  6.  toxic substances,  7.  electricity,  8.  radioactivity,  9.  weapons - firearms, others,  10.  animals.  substances,  Smith consented this was the skeleton of the system.  Observations  & Post-Session  Interview  with Expert  and  KE.  The most interesting point about this KA session was the gradual emergence of the underlying "skeleton" for the expert system as the discussion developed. It seemed  that  the overall  structure emerged  despite the explict  focus  of  the  discussion on specific details within the framework. The researcher also found the method  they  used  of bypassing  the difficult  and abstract and always posing  concrete questions to the user remarkable.  At  the post-session interview, Smith  emphasizing  further  elaborated upon this method by  they left out legal rules, philosophy, and anything disputational out  of the system. In doing so, they avoided the risk of users not understanding the carefully system  formulated posed  and articulated  questions generated  questions on legal philosophy. Rather, the by the philosophy  pertaining to the conrete  situations. Deedman pointed out that users, being lawyers, were not interested in  Case on the Remoteness Advisor / 32 theory  but only  in cases  to give  them  some  bases  for making  successful  arguments in court.  The  essence of their approach  was to ascertain facts in cases by relating them  to precedence which had similar facts. To do that one had to reason by analog}' to find alike or relevant cases. This implied the need for a criteria of relevance. All  this  was done outside of the system.  succinctly when he said : "the kej  r  analogy  Smith  summarised  their  approach  to our success is to capture reasoning by  in heuristic rules and that is where the dispute takes place and you  trust the expert on that." Ultimately, the empirical test was whether the user obtained the right cases. If the group of lawyers agreed the system presented to the  user  the relevant cases  and nothing was missing, then  the system had  performed satisfactorily. The domain of law is distinctive in that the set of court cases which embodies all the facts to be considered is readily available. In other words, the scope of the cases relevant to the domain is clearly defined.  4.4.3. KA Session  The  third  KA  III.  session for the Remoteness Advisor took place on December 7,  1987. Smith and Deedman met at the former's office. The researcher was absent and Deedman taped the session.  The First  Categorization.  Case on the Remoteness Advisor / 33 Deedman started the session by reviewing what was accomplished  the last time  they met. They concluded there were two basic categories : 1.  special types of people, which included rescuer, thin skull, nervous  shock,  and medical complications and, 2.  special  types  of  activities,  which  included  weaponry,  firearms, toxic  substances, animals and motor vehicles, etc. Based  on the conviction  that the categorization  was  derived from  the cases,  Deedman had gone through the cases again to see how well they fit into this scheme. He came away with two proposals. Firstly, motor vehicle was a major category but Deedman thought a more generic category of transportation may be more  appropriate because  motor  vehicles  could be considered  a subset under  means of transportation. Secondly, transportation cases could be broken down into four groups: 1.  motor vehicles,  2.  railways,  3.  animal-drawn,  4.  vessels.  Deedman wanted to get Smith's reactions to his suggestions.  Smith  thought that was fine but would like to add the category of "other" to  capture  transportation  cases  dealing  with  snowmobiles,  bicycles,  or  other  "far-fetched ones." Deedman remarked there were no cases on airplanes. Smith agreed and suggested handling that by having the system inform the user at the end  that  there  were  no  remoteness  involved  in plane  cases.  introductory question would take them into the transportation category.  A  general  Case on the Remoteness Advisor / 34 After agreeing on the general categories, they continued breakdown within question  on  transportation "other"  each class. If the user  transportation,  another  was involved. The user  responded  question could  to discuss more detailed positively  would  ask  to the general which  trucks, buses, vans, track  of  choose among the four plus the  categories. The cases within the motor vehicle category  motorcycles,  type  and trailers,  trailer  included cars,  snowmobiles,  etc.  Deedman counted that among the 58 motor vehicle cases, nine were on trucks, four on buses, one on motorcycle; the  etc. Among the railway cases, four were on  whole train, two on railcars, which included  one on trams, and one on  electric street cars. There were two carriages or animal drawn cases and three cases on boat or steamships, which would fall under vessels. A large number of cases in one category  might indicate the need to further break it down into  sub-categories; e.g., 58 motor vehicle cases indicate that sub-categorization may be useful. A Second  Categorization  Scheme.  At this point, Smith proposed a second categorization scheme. Instead of looking at the nature of the vehicle, a second categorization scheme that cut across all vehicles was needed. Since these were all cases that deal with issue, negligence  the remoteness  was assumed. Given negligence, there was a primary risk to  somebod3\ Hence, for each case, they needed to clarify the following : 1.  What the risk was;  2.  Did the risk materialize or not;  3.  Was the damage or injury in question the same as that expected in the  Case on the Remoteness Advisor / 35 risk; 4.  Was  the plaintiff at risk or was  it somebody else.  Smith felt that this second level categorization, which was kind of vehicle involved, was first  level  categorization  important and  that  Deedman  independent of the  needed to be clarified on top of the  had  formulated. It would  relationships that cut across all motor vehicle categories. To Smith  gave the  vehicle.  Then,  motorcycle, the  example regardless four  of a of  plaintiff who  whether  questions  on  relationships to them needed to be all transportation cases,  and  he  the  was  risk,  was  a  hit by  a motor  a  bus,  van,  damage, and  the  plaintiffs  answered. Therefore  identify the  the  illustrate his point,  rescuer  rescuing the  specify  on  the3' should  or  go through  plaintiffs' relationship to the  risk in  order to seek the patterns among them.  Smith and  Deedman sat down together and leafed through a thick book of cases  to weed out irrelevant ones. The  case book includes 94 cases and  was  compiled  based on the criteria presented in Liability in Negligence [Smith, 1984]. As they were reviewing the cases, thej' noticed there were two they had  types of remoteness which  started out thinking were the same. First, given any  it too remote that this person suffered harm? Secondly, was kind  of harm  that was  remote? Smith  means of transportation and  suggested  thej' go  type of car,  was  it this particular over all cases  on  identify the plaintiffs' relationships to the risks and  classify them in order to typify the kind of remoteness. This would reveal the basic structure of the knowledge base.  Case on the Remoteness Advisor / 36 Observations  1.  Since  and Post-Session  the researcher  explanation  about  Interview  was not present  points  types  at the session, she needed  raised at the KA  interview with Deedman on December two  with the KE.  of remoteness:  first,  some  session. In the post-session  15, 1987, he further explained the  remoteness  for victim, and secondly,  remoteness for damage (see figure 2). The permutations seen in the cases were derived from these two basic types. A remote damage case involved an unexpected harm. If the harm occurred to somebody other than to one whom you expected then that would be a remote victim case. He cited the following example to illustrate this distinction. A driver hit a fire hydrant and  water from the hydrant flooded stocks at the basement of a nearby  store. The victim was not remote since s/he was in the proximity of the accident; but flooding damage resulting from a car accident was unexpected. Hence this would be a remote damage case. Another example was: a car hit a fence around the farmer's field, the staples on the fence popped into a field and some cows grazing on the field ate them. The cows died and the farmer who owned them was compensated in court. This would be a remote  victim  case  because  usually  a car accident  victim  would be  somebody in its vicinity and not a farmer physically far from the scene. The  distinction  between  the two types  of remoteness was derived  from  reading the cases.  Deedman also felt the time was right to start prototyping. So far at the KA  sessions, they  become  had discussed in conceptual  "interminable"  and the K E  believed  terms but discussion could it best  to embody  what  Case on the Remoteness Advisor / 37 structure they  had in a concrete object. He was in complete agreement  with Weiss and Kulikowski [1984] who stated :  "The single most important piece of advice that one can give a model designer is to build a prototype model as soon as possible. Because expert reasoning problems are frequently poorly specified, one needs to have something concrete to view and 'lay hands on.' It is particularly important for the expert to see something running early. A running program is worth thousands of words from an unformalized interview with the expert" [Weiss  et al.,1984:106].  2.  It is significant that Smith proposed a second categorization which captures the causal relationships among the risk, damage, and victim. This scheme of relationship is distinct and different in nature to the first superset-subset categorization implicit in the means of transporation and its sub-categories.  Case on the Remoteness Advisor / 38 4.4.4. KA Session  The  IV.  fourth KA session for the Remoteness Advisor occurred one week later on  December 14, 1987. Smith, Deedman and the researcher met at Smith's office.  Structure.  The expert and the K E continued their struggle with the underlying structure of the system. To recapitulate, they had decided at the previous meeting there were two general categories : 1.  special types of people and  2.  special types of activities.  And  lists of both had been compiled.  Smith and Deedman had obviously thought  about the problem  since they  last  met. Deedman suggested there were three key labels and wrote his ideas on the flip chart as follows : ACTION - known identifiable (smoking) RISK - of particular type of harm (fire explosion) VICTIM - person (bystanders at the scene) - property Smith added "HARM" to the chart. He felt that harm was a distinct category because he wanted the system to ask the user about both risk and actual harm.  Case on the Remoteness Advisor / 39 The  remoteness  issue  arose  when  what  happened  fell  outside  the risk of  negligence. Smith added that harm might be to person, property, and economic. Deedman suggested  the traditional division would be just between personal and  property, and economic loss was not "on the agenda." Smith however, saw this as "a map," and if economic loss was chosen, the user would be directed to another system.  Smith envisioned the system to be collecting information through the first series of  questions which required the user to choose among the categories. Then the  system would digest the collected information and "shoot" the user into series of questions  on the basis of the digested answers. Using  the labels on the flip  chart as items on the underlying scheme, a series of questions could be posed and  from the answers obtained, much information would be gathered. But first  they had to make up the lists of action and harm.  Since  the list of types  of activity  (action) was ready-made from the previous  session, Smith  recorded  harm  the use of any of the following?  involves  the following question  : did activity And  which  then  cause the  he copied the  ready-made list under it (see figure 3 ).  For risk of harm, Smith put down on paper the following : (see figure 4) He posed the question of whether the plaintiff was outside or within the risk, whether the risk materialize or not, and various combinations of these options. As he was talking, he drew the following flowchart on paper : ( see figure 5 )  Case on the Remoteness Advisor / 40 For victim, the question victim,  medical  was  complication,  Nervous Shock Advisor  whether the plaintiff was or  thin  would be  skull.  loaded. And  For  rescuer, nervous shock  nervous  Deedman put  shock  victims, the  down on  the flip  chart the list of victims as follows: 1.  rescuer,  2.  nervous shock,  3.  medical complications,  4.  thin skull - mental or physical.  For harm, he put down the following : 1.  mental,  2.  physical,  3.  thin skull.  After some more discussions, Smith drew a different version and above. He 1.  threw out the  put down under actual harm the following :  to person a.  mental  b.  physical 1)  thin skull,  2)  medical complications,  2.  to property,  3.  to both,  (see figure 6)  By  posing questions under these categories, all the necessary  be gathered and rules based on the information can be  information would  formulated.  Case on the Remoteness Advisor / 41 At this point Smith showed the researcher a chart of categories for the Nervous Shock Advisor (see figure 7). He explained that the 255 rules of the Nervous Shock Advisor stood behind the structure diagram. And a similar diagram for the Remoteness Advisor was their present objective. Deedman pointed out the diagram had been derived from the exercise of developing the system. But Smith felt that if  they  Deedman  could could  arrive work  at a structure backwards  diagram  from  before developing the system,  the structure  and expound  it into a  massive set of rules and questions. In this way, the process of K E would be accomplished  in half  the time  it took  for the Nervous  Shock  Advisor. Or  alternatively, if, they could arrive at the structure and work in both directions, the availability of the structure would  help clarify to the students working on  the knowledge base of cases what information to look for.  Returning to the issue at hand, Deedman remarked the risk of harm looked fine to him. However he thought actual harm  and whether  the harm  materialized  were different approaches to the same problem and suggested that Smith try to integrate the two. Smith agreed to attempt that and they both decided to go away and think about the problem some more. Observations  1.  & Post-session  Interviews  with Expert  and  KE.  In the post-session interview with Smith, he explained further the structure diagram. This structure diagram  replaced a tree diagram  which  was  the  usual mechanism in clarifying knowledge for expert system development. He preferred a structure diagram because too much overlap existed among the categories and a tree diagram could not adequatel}' represent the overlaps.  Case on the Remoteness Advisor / 42 The  structure could be converted to a tree, but there was "no point" in  doing so. The structure chart summarized  everything the user needed to  know about the cases to fit them into the 255 rules of the system. By following  the instructions  and checking  off the appropriate categories for  each case, the user in fact identified the key variables in the case and link it to the applicable rules. They presently aimed at developing a similar chart and then developing the rules from it instead of doing the reverse as they had done for the Nervous Shock Advisor. 2.  In this session, the expert and the K E tried to arrive at the second level categorization that Smith thought was necessary in the previous KA session. The  researcher  observed  that  this  attempt  had  not been  completely  successful as the second level categoization was by no means obvious. There were times during the session when the researcher observed the participants seemed  stuck. Considering Smith's objective of arriving  at the underlying  structure of the system before the sj'stem was developed, it seemed Smith was embarking upon an innovative and indeed, in the researcher's opinion, courageous effort to short circuit their original KE efforts. It remained to be seen if a structure could be derived in this manner without what the KE  felt was the necessar}' but painful experience of fumbling though the  expert  system  developement  Advisor, the structure was  process. developed  In developing after  the Nervous  Shock  the rule base. As a result,  fundamental structural changes were made during coding. The lesson learned reinforced the importance code.  of diagramming the structure before beginning to  Case on the Remoteness Advisor / 43 4.4.5. K A Session V.  Pre-KA  On January  Session  V  Interview  7, 1988, the researcher met for half an hour with Deedman  at his office before the fifth KA session. This was the first meeting after the  Christmas  holidays. During  the holidays, Deedman  had drawn  a  schematic diagram to categorize all negligence cases, and he had wanted to show this to the researcher. The diagram represented a schematic skeleton of^the^ Remoteness Advisor ( see figure 7 ).  He explained the diagram  as follows: the top-most level question in  the  system decided whether the harmful act was an act or an omission, in the latter  case, the user  decided  exits  to another  if the act was deliberate  system. The next  level question  or negligent, if it was deliberate, no  remoteness issue arose. Otherwise, the user would continue to ascertain the kind of damage and choose among the three types possible : psychological, physical, and economic loss. In the first case, s/he would exit to the NSA, in the third, to the Economic Loss then  it might  be plrysical  damage  distinction was between unusual  Advisor. If it was physical damage, to person  or property.  The basic  victim or unusual activity. Unusual victim  could be rescuer, thin skull, or exacerbation; whereas unusual activity might involve the use of inherentlj' dangerous substances such as explosives, toxic or  inflammable  substance,  etc., or involve  machinery  and means of  transportation, which included the subsets of motor vehicles, railway, vessels,  Case on the Remoteness Advisor / 44 airplanes, and animal-powered vehicles.  For a specific case, the question was whether it could be disposed of by matching the facts to particular types of activities that fell into the two categories of unusual people or unusual victim. Deedman expected that most cases would be taken care of this way. For cases whose facts did not match with the particular activities, the system would adopt a philosophical approach and pose the question to ask if the consequences of the harmful act might have been reasonably anticipated. If the answer was affirmative, no remoteness issue was  involoved and recovery would result. Otherwise,  the consequence of the harmful act was  unusual and  remoteness issue  arose. The unexpected consequence might be due to a different victim, or a different damage; and the difference might be in type or in extent. For example, the "cow case" involved an unexpected extent of damage.  Smith agenda  had  already  seen the diagram and  for the session  was  to go  liked  the categorization.  over the diagram with  The  him. This  represented the structure of the system, the next step was to represent it in terms of rules. Then a prototype could be produced.  Deedman felt that Smith had the elements but they had not been hitherto organized in a diagram. KA  Session  V.  Case on the Remoteness Advisor / 45 Smith, Deedman and the researcher sat together over a redrawn version of the diagram at Smith's office ( see figure 8 ). The purpose of the session was to further refine the diagram.  Refinement  Smith had thought  of Motor  Vehicle  Category.  about further refinement  of the motor  before coming to the session. He went through they  fall  triggered  under  two categories.  another,  First,  and secondly, where  when  vehicle category  all the cases and decided one motor  a panic reaction  vehicle to risk  accident of harm  was involved. He put down the following on the flipchart : motor vehicle (1) one motor  vehicle accident triggers  other  (includes debris on highway,  must include other vehicles) (2) Panic reaction ( to risk of harm ) Then  they  sat down  and reviewed the case book, categorizing the cases  according to the two groupings.  Smith  was  uneasy  "philosophical module"  about  what  (see figure  Deedman 8) : "I  had  know  referred  something  to  as the  (is) not quite  right and I don't know what it was." He had a list of questions which he thought were relevant but was unsure on how to put them together. thought  the concepts embodied in his questions were  different  from  Smith those  represented in Deedman's diagram. And he would like to clarif}' them in their minds at the present session. His list of questions was as follows :  Case on the Remoteness Advisor / 46 1.  did the risk materialize?  2.  were there damages that fall outside the parameters of the risk?  3.  did the person who suffer the damage fall within the risk?  4.  were the damages of a different kind than the risk, if ( they were of the ) same kind were they in excess of the risk?  Deedman  apparently  did not think  Smith's  questions captured more possible  combinations than his diagram. He objected to the first question because while it might be more precise, "somebody from  the street" might  not know  what it  meant. Their difference in opinion could be illustrated in the following exchange : Deedman : you know you are dealing with difference both in damage and victim. Smith :so ask first about damages... Deedman :does it matter? Smith :yes it does. Deedman :yeah you're right. Smith  :(The) damage is same but (the) victim  is different, a different victim  suffered the risk... Deedman  :Logically, ask was the damage or victim different, or both? Wouldn't  that capture it? Smith :No...okay let's go back and see (examine the diagram again).  Smith felt the philosophical classification of cases was more important than the "motor vehicle" ( or physical ) classification because it would  supply all cases  relevant by analogy to the user. Speaking as a lawyer, Deedman  agreed and  emphasized the importance of presenting to the judge cases closely related to the  Case on the Remoteness Advisor / 47 case being argued.  Then they moved onto discuss if the intervening act which created the risk was criminal or not. Deedman pointed out a clear cut answer could not be given for some cases, and other criteria might be relevant. Smith cited some factors such as date of the case, their country of origin, etc. But he had to go back to the cases again to decide.  Further refinement of the psychological damage branch of the diagram was also discussed.  Smith  distinguished  between  a  mother  who  went  into  depression  because her child was killed versus a mother who saw her child killed, suffered nervous shock due to the incident and went into depresssion. In the former case, no  shock was incurred and hence the plaintiff could not recover. In the latter,  nervous shock was suffered and the NSA  would be invoked. The label for that  branch became "psychological only" instead of the former "psychological", and all regular  nervous  categories impossible  shock  cases  of psychological  should  versus  follow  physical  to differentiate between  shock  down  were and  the branch.  If only  two  set up, it would  have been  non-shock  In  cases.  clearly  differentiating the categories, Smith was trying to ensure the user would "come out right " from the advisor.  Other refinements of complications skull  into  physical  versus  into medical and non-medical categories, thin  physical  and psychological  subcategorization of medical complications added to the diagram.  groupings,  and further  into treatment versus deterioration were  Case on the Remoteness Advisor / 48 At the end of the session, Smith commented the structure was almost close to finish and  that Deedman might soon be ready  wanted to test the control flow by implementing  to prototype. Deedman said he a small version on M l .  Observations.  The researcher observed in this session two people struggling to categorize a sea of cases to ensure Both  thought  themselves  the user would obtain the right answer from  in terms  of the overall  expert  in the shoes of the user travelling  system  design  the  system.  and often put  down the system. The expert  especially often took the initiative in assuming the role of the user.  4.4.6. KA Session VI.  On February  4, 1988 Smith, Deedman  and the researcher met for about 40  minutes in Smith's office.  The expert and the K E continued their refinement of the structured chart. They posed actual or hypothetical cases to each other and tried to think through how a particular case would travel through one  category  versus  the diagram, whether it would fall into  another. The objective was apparently twofold  : first, to  label the categories general enough to capture all relevant cases, and secondly, if the category was found  to be inadequate, insert another  one. For example, at  Case on the Remoteness Advisor / 49 one point, they discussed the merits of adding an extra category of "secondary causal factors" under  "aggravation of existing conditions" which  label for "exacerbation" (see figure 9). Deedman was  concerned  was  the  new  that the term  should be general enough to cover all possible cases of that type, but also that it should not be overly-academic for lawyers. He "other  acts  that  cause  further  harm"  and  suggested alternative labels like preferred  to  avoid  terms  like  "intervening acts."  A  usual mode of interaction was  for  a particular category and  for Deedman to ask Smith for a sample case  Smith  would respond  almost immediately  with a  case to illustrate it.  The philosophical module was the precise reason why  to characterize the user's case and inform the user  recovery would not be allowed. Then similar or relevant  cases could be retrieved.  They also further refined the branches on psychological only versus physical and psychological, and that on medical versus non-medical. Once again, they confirmed that Smith  medical cited  would cases  break  down  to illustrate  to treatment the  and  natural  subcategories as  well  deterioration. as  to ensure  And the  categories adequately classify them.  At the end of the session, Deedman agreed to redraw the diagram Smith  to "live  proposed  this  with  and  it for awhile" to see if changes were needed. He  session to be  a  milestone  to mark  the  end  of KA  and  asked also the  Case on the Remoteness Advisor / 50 beginning of the prototype phase. Smith agreed and said they would just spend one more session on the final diagram.  At  Epilogue.  the last  meeting, the "method  of infliction  different" branch  was  further  subdivided into the two branches of "agent radically different ( e.g. fire vs water )" and "agent same but magnitude  greater (e.g. massive conflagration vs small  fire)". Deedman did not agree with Smith on the necessity of this addition.  Deedman then worked independently from the structure chart and derived the list of variables which would be used to characterize the cases ( see figure 10 ).  Based on the structure chart in figure 11, Deedman developed a prototype of the Remoteness Advisor on Ml, which was shown to the researcher on April 22, 1988.  C H A P T E R 5. C A S E ON T H E  TEST DESK DECISION SUPPORT TOOL  5.1. Introduction.  The  Test Desk Decision Support Tool is an expert system developed by Jeffrey  Howell  of Microtel Learning  Designer Howell  at the had  for over Dan  centre and  a Master's  Services Centre. Howell acted as  manager  Senior Instructional  of the expert  system project.  degree in Instructional Systems Technology  10 years as a courseware designer. The  Brophy, who  is a  and  worked  chief expert on the project is  is a senior instructor at the B.C.  Telephone Education Centre  (B.C. Tel Ed Centre) and belongs to the Staff Group of B.C. Tel.. He thirty years of experience in the industry and natural teacher." Brophy  is described by  is the chief resident expert who  has over  Howell  as "a  consistently worked  with Howell over various phases of the development process; but as will become clear later, five other experts participated in the project.  5.2.  Test Desk Operation.  The expert s}'stem serves as a job aid for the telephone plant tester, alias test desk person or analyzer. A the  telephone  lines, clears  test desk person identifies and or  routes trouble reports, and  diagnoses assists  faults in  and  directs  maintenance forces in remedying the trouble (for detailed discussion, see figure 12, "Job Profile"). A job aid is important as a tool to help upgrade the service that test desk persons provide to customers. Presently, three to four persons are  51  Case on the Test Desk Decision Support Tool / 52 involved in processing the touble reports produced by the 4 way  telephone testing  equipment which performs daily routine checks on the telephone lines (see figure 13  for diagrams  on  current  and  future  operations  at  the  Trouble  Analysis  Centre). When the reports arrive at the Trouble Analysis Centre (TAC), the supervisor assigns a tester to investigate the trouble reported. The his analysis report to the clerk, who out a TAC  in turn makes up  isolator to the site. The  TAC  tester gives  a dispatch list to send  isolator calls from the site and  the test  desk person sends out test signals to the isolator. They interactively test the line until a diagnosis can be made and  the problem remedied. If a wrong diagnosis  has been made then the time and cost incurred is substantial. An  expert system  could potentially incorporate the knowledge that the best test desk persons have and  aid a poor or an  Not  only is the time (and money) saved in making accurate diagnosis important  to  the  company,  but  average tester to more accurately diagnose the problem.  successful  and  speedy  line  repairs  improve  customer  satisfaction.  The  domain of the  expert  system  is the job of the  test desk person which  involves three major functions: 1.  to identify and diagnose faults;  2.  to clear and route trouble reports;  3.  to assist and direct maintenance forces.  To  accomplish these objectives, the test desk person needs to interact with  the  customer in say, clarifying a trouble report, as well as with staff of the Central Office and division, the  rack, the installation service center  and  repair department, the cable maintenance  supervisors, the  terminal equipment suppliers, lease  Case on the Test Desk Decision Support Tool / 53 wire  customers,  and other  service  centre or telephone  company  personnel. In  addition, s/he has to be familiar with a number of tools and equipment which include the following : 1.  touch tone telephone set,  2.  CRT terminal which provides access to other databases  or departments in  the company, 3.  4 way telephone lines of both 1986;  testing system  (4tel) which automatically tests telephone  business and residences (the 4tel system  previous to that, the Badger  system  was installed in  became operational in 1983,  both systems are in use today), 4.  Trouble Administration System (TAS) which holds and routes trouble reports,  5.  Customer  Record  Information  System  (CRIS),  which  contains  all service  order, customer, facility, and billing data for both customer and residence, 6.  Service  Order  Update  and  Locate  (SOUL)  which  holds  business and  (MAIS)  which  holds  leasewire orders, 7.  Mechanized  Assignment  Information  System  facility  information, 8.  Facility Management System (FMS) which holds detailed facility data (from figure 12, "Job Profiles").  Hence it can been seen that the domain of the expert system is complex and represents one link in a network of operations vital to the telephone company. As  Howell  put it, the test desk person  has "to know everything out there,  where a problem can occur...in all this mess, and figure out what to do with all this as well as interface with the customer and make them happy and aware that ...we will take care of your needs...and then make the forces go out and  Case on the Test Desk Decision Support Tool / 54 fix the problems."  Despite  the  wealth  fourteen types type, only  a  of information  of trouble and finite  number  a  tester has  six dispatches of procedures  to  analyze,  there  are  only  to handle them. Given a trouble could  reasonably  apply.  Hence  the  domain is definable in this respect.  To better illustrate the importance of the afore mentioned databases to the test desk job, consider the following example. The  4tel testing system gives "window  parameters" which indicate where the problem may customer call arrives at time A, and wrong, then  some  data  with  testing done at time B  is missing.  By  databases, the timing of the call can  be  then discover that problems on  be  checking  into  obtained. The  the  a line. But  if a  reveals nothing is TAS  and  other  test desk person  a line coincide with rain in the area and  may the  line is wet. This is an important factor in the final diagnosis of the trouble.  A domain expert, according to Howell, is one well versed in operating within this complex environment of tools, databases, and the trouble reported. He you  people and  in accurately diagnosing  called such a person a "good practitioner, the person  want all employees to be like." Instead of relying on any  single expert, he  planned to draw "the aggregate of all sources," which include the manuals for the  two  testing  human informants operators, as  systems: such as  well as  work). Furthermore, one  Badger  and  4tel,  descriptions of office  supervisors of test desk persons, expert  poor tesk desk persons (to see  why  they  procedures, test desk  fail in their  third of the information for the expert system already  Case on the Test Desk Decision Support Tool / 55 resides in a simulation program developed be transferred directly to the new  The  expert system was  course on  previously, and  the information could  system.  being developed  the testing equipment was  simultaneously as the materials for a  being updated. Brophy would be teaching  the course and his primary interest laid in revising the materials. Howell on the other  hand, helped  him  with  the revision  while  at the  same time extracted  materials from it for the expert system. Since Brophy would not be available as much as Howell needed, one specialist from  of his former  students who  was  present^ a field  one of the telephone company's service offices, would act as his  proxy and concentrate on the information for the advisory system.  5.3 Data Collection Methods.  Organizational funding Howell  contended  suspended due  and  since day  was  to problems from either sources, while other times, progress  was  with Howell  by  one  of personnel were obstacles with which  of the project. Several times development  made on a daily basis. The touch  availability  researcher tracked the development by  telephone  and  once a week from November of 1987  attended  staying in  sessions personally approximately  to early February of 1988. The data  was  collected either over the telephone from conversations with Howell and sometimes with Brophy, or through observations when the researcher attended the sessions  Case on the Test Desk Decision Support Tool / 56 and tape recorded them. Each session lasted from 2.5 to 5 hours.  The  following description traces the researcher's  development team  from  initial contact to demonstration  which spanned a period of six months from 1988.  It presents  both  interactions with  telephone  the Microtel  of the first  prototype,  October of 1987 to February of  conversations  in which  Howell  reported on  progress to date and accounts of sessions the researcher attended personally. At the  end of some sessions, observations  made by the researcher  and relevant  comments made by the participants are also included. Since the expert system project team  was responsible for both  administrative as well as developmental  details, the discussion touches upon both aspects.  5.4  Knowledge Acquisition.  On  October 28, 1987, the researcher had a first orientation meeting with Howell.  The discussion took place at Microtel Learning Services Centre, on the third floor of the B.C. Tel Ed Centre in Burnabj' and lasted three hours. Howell explained to  the researcher  the test  desk  job and its importance  to the company  operations, the contents of the discussion has been presented in Section 5.2. The expert included  system  was to serve  as a job aid and training  testers in the operational environment  tool, intended  and analyzer  users  trainees in the  classroom at the Education Centre.  knowledge acquisition for the test desk decision support system was carried out  Case on the Test Desk Decision Support Tool / 57 under the rubric of "task analysis." The  objective was  to decompose the job of  the test desk person to its component duties, tasks, subtasks, and prerequisite knowledge. The  first phase of the analysis involved verifying and  updating the  materials in the two task analysis manuals on the Badger and the 4tel systems compiled in 1981 and 1986 respectively. * * *  On  November  4, the researcher spoke with Howell  over the phone. He  and  Brophy had spent 5 to 6 hours cross referencing the two task analysis manuals to see how  the job had  changed. They updated  to be included in the expert S}'stem. He  was  the material and  expected to spend  decided what  another five to  six hours the next day doing the same and planned to create a fast prototype by 25 or 26 of November. Tentatively he set up meetings for task analysis on 12, 16, 18, and 19 of the month. *  On  November  5, 1987,  Howell  met  *  *  with Brophy  and  the researcher for 2.5  hours at the demonstration room on the second floor of the Microtel Learning Services Centre to continue the cross-referencing. (See figure 14 for a diagram on the physical layout of the room). The documents included the Badger instructional material which was in  1983,  and  developed for an instructor-led situation, a course audit done  the  4tel  instructional  software  document  developed  in  1986.  Although some of the materials were seven years old, and the tools used today were  different, most  considerable variety  of the across  knowledge different  was  areas  still  valid.  of British  However, there Columbia;  55%  was  of the  Case on the Test Desk Decision Support Tool / 58 company  sites  used  the 4tel  testing  equipment  and  the rest  had  either the  Badger system or none at all. Since the sites varied in size and complexity, for example, the  downtown  Vernon centre was different.  As  Mutual  Centre  had  more than  operated by only one to two men,  Brophy  pointed  out,  the  company  "innovative," and when they responded, there was used. To  standardize the procedures  20  testers while the  the office procedures were encouraged  people  to  be  great variety in the methods  among the diverse methods adopted  was  difficult. At times Brophy made a note to send memos to other people in the company for clarifications.  There were altogether three categories of information: 1.  central office information on the type of switch, test equipment (Badger or 4tel), station equipment at customer  premise  (PBX,  dial, or outside plant  equipment); 2.  IMS  packages (TAS, CHRIS, or FMS);  3.  trouble  type  information  and  test  analysis  procedures  (including  4tel  parameters) on what to do for different types of trouble.  Typically, Brophy would read through the 1986 his  notes and the 1981  him if some step was  manual. Sometimes he raised a point or Howell asked  indeed currently implemented, and they discussed it. Either  they arrived at a conclusion and  it was  note to call somebody for clarification. An shown as follows: 1.  manual, checking each step with  Duty : Identify and Diagnose Faults  noted down, or one of them made a example of a task decomposition is  Case on the Test Desk Decision Support Tool / 59 Task : 1.1  read trouble report (TAS)  1.2  interpret trouble report  1.3  perform basic line test  1.4  interpret basic line test  1.5  determine type of trouble  1.7  determine what test to perform  1.9  perform 4tel tests  1.9  interpret test result  1.10  route/write off  Later, Howell copied the updated information onto a word processor to produce a new  document.  At  the end of the meeting,  Howell  and Brophy  discussed  the timing of  presenting what they were doing to the staff personnel. Such a presentation was necessary  both to obtain the staffs verification that indeed  Howell and Brophy  had captured the correct and current method of conducting tasks and to reconcile the different methods that existed in the field. These meetings would take place at  the end of November. For the next  unavailable,  Howell  planned  to meet  two weeks, since Brophy  with  Broprry's  proxy  every  would be Monday,  Wednesday and Friday morning to continue knowledge clarification.  Observations  1.  and  Interviews  The researcher  with  the Expert  observed that as they  and  KE.  discussed  the substantive issues of  Case on the Test Desk Decision Support Tool / 60 what  to  include  in  the  expert  system,  the  expert  and  KE  were  simultaneous pondering implementation methods which could be applicable to the  materials. Since  system was  some of the  materials  to be  included  in the  new  already in the 4tel simulation software, they could simply  transferred over to the new  be  system. Howell recalled developed screens which  could be used or pondered whether to acquire data through an interactive query system or by  accessing an  existing database. Brophy was  involved in such implementation details; and  there was  no  likewise  effort made to  keep their discussion at a conceptual level. 2.  The task  main concern was was  researcher  whether the method or procedure of doing a certain  indeed  correct and  before  their  current. In  session  began,  was  discussion with  remarked  that  it  the was  developed, the procedures for  aimed to teach was  In the telecommunications industry, it was the changes. Moreover, there was  casual  Brophy  inevitable that by the time the software doing something which the software  a  already  outdated.  often difficult to keep up  with  the difficulty of reconciling wide variety  in the methods that different centres adopted in doing a similar task. *  On  November  6,  Brophy  cross-referencing. They  and  discussed  Howell  * *  met  in the  structuring of the  morning knowledge  to continue at the  their  meeting.  Howell pointed out four possible "points of identification" which could be indices to the information : the trouble type, the telephone and  whether the Badger or 4tel system was  like TAS  and FMS  were indexed by telephone  number, the central office,  used. Since the existing database numbers, using this as the index  Case on the Test Desk Decision Support Tool / 61 could easily retrieve information from the database. If they used the switch type, parameter values could be easily accessed with either the Badger or 4tel testing tools. Outside plant equipment and line information were indexed in FMS  by both  central office and telephone number; while operational procedures followed directly from  trouble types. Hence there were advantages to using any one of these  possible indices; Howell  decided  trouble types  were the most straight  forward  index for the system.  The  4tel simulation software was contracted to an outside software development  company, and he was contemplating doing the same for the expert system. He had  discussed the project with the software  company  personnel and found out  their schedules.  They also prepared for the group verification meetings to be held at the end of the month. Materials had been sent out to the participants and Brophy had requested the supervisors to bring in as much information on local procedures as possible. *  On  10th of November, Howell  * *  met with  the researcher at Microtel Learning  Services Centre for he had planned to see the other expert (Brophy's proxy) to inform him about the progress to date. However, the expert had the flu and failed to show up. Instead, Howell the design of the expert system.  spent the time explaining to the researcher  Case on the Test Desk Decision Support Tool / 62 The  top level design  of the expert  system  is presented  in figure 15, "Draft  Design of Expert System." As can be seen from the diagram, knowledge for the system is categorized into dynamic and static information. Dynamic information varies with each trouble case that the expert system attempts to diagnose. They are information that the user has to gather and enter into the system, which is primarily the NNX code. The NNX code is the first three digits of the telephone number which provides access to information on the plant (e.g. if the cable is wet), the company (i.e. type of switch), and the station or the customer premise equipment (e.g. whether one or two private lines were installed). This information does not change once it has been entered into the system. The static information refers to the knowledge base of heuristic rules, which are indexed on the trouble type. When the information  about the plant, company  and station is entered,  then the 4tel parameters associated with the given NNX can be retrieved, and information  on switch, voltage, cable, and basic line tests can be determined.  Hence NNX  and trouble type are the key indices. The design is based on the  4tel  equipment; Badger  testing  has not been  included  because  it is a less  sophisticated system and provides less information.  Observations  Howell had considered the ideas presented past 1.5 years.  building an expert sj'stem  for a long time. He reported  in the draft represent the results of his deliberations for the  Case on the Test Desk Decision Support Tool / 63 On  12th  of November,  Howell  met  with  Brophy  again  mainly  to discuss  strategies of presenting the material to the staff group that would be coming in from the other areas in B.C. for the week of the 30th. They finalized the list of people to be brought in and  other administrative details. Howell considered  these steps important to ensure the eventual successful adoption of the system as a job aid for the service centre personnel.  Howell reported in a telephone conversation with the researcher on November 22 that Brophy's proxy was  unavailable for the project. Howell planned to continue  the cross-referencing with Brophy the  course  for the  other  and shift the focus from the expert system to  expert  was  information in the expert system. The next year. Howell  to  be  primarily  prototype was  responsible  for the  delayed until January of  also discussed with staff from the Management Information  Systems (MIS) division their involvement in the project. *  In  *  *  the week of November 30, Howell conducted group  KA  sessions with four  experts brought in from various areas in British Columbia. They included Steve Lewis  from  Dawson  Abbotsford, and thej' met  Creek,  Brand  from  Victoria,  Herb  Young  from  Grant Coplitt from downtown Vancouver. Together with Brophy,  in Room 211 of the Learning Centre from 8 to 4 everydaj' from 30  November to 4 December. The the  Terry  researcher attended the meetings during two of  five days. See figure 16 for "Agenda for group meeting" for a schedule of  the events at the sessions.  Case on the Test Desk Decision Support Tool / 64 The purpose of the meeting was  primarily to gather and verify information to be  included in the course and the expert system  for training the testdesk persons.  Again, they based their discussion on the two existing manuals compiled in 1981 and  1986.  Howell  and  Brophy had  begun this work and  now  the four local  experts were brought in to continue the verification process. This process involved a verification and update of the two manuals. One read from  a copy  of the manual and  as he  person from the group would  was  doing so, the others made  suggestions on appropriate changes and clarifications.  In this process, substantial discussion arose due  to diversity in equipment and  procedures among the various centres. The downtown Vancouver office (Mutual) is much busier than that at Abbotsford. On  a typical Monday morning, a maximum  of 65 trouble report calls would come in at the Abbotsford centre in comparison to several times that for the downtown office. H e n c e spent  when  differences.  the  Even  disagreements, he  participants  though often was  exchanged  Brophy  acted  as  at a loss on  various  tools  invented  new  and ones  equipment, which  local  were  the  arbitrator  on  time  was  procedural  in the  event  of  standards were  example to illustrate this is the  a standard set of acronyms for the  personnel  totally  information  what the company  since procedures vary from region to region. An use of acronyms. While the company has  considerable  at  the  incomprehensible  different to  centres often  outsiders.  Howell  suggested including a glossary in the course and expert system to overcome this problem.  Despite these differences, the basic duty of the test desk person was  to "test  Case on the Test Desk Decision Support Tool / 65 reported  trouble" and the six major dispatches involved to achieve this were  generic and common across all sites regardless of equipment and tools. Similarly, there existed only fourteen trouble types possible. Focusing on the generic, they updated the materials in the manuals.  After verifying the information on the documents to be accurate and current, they proceeded to integrate the three documents by listing each duty subtasks  on the board. Discussion arose  over  the proper  decomposition  and  its  of the  tasks, whether a specific task should be done elsewhere, or some tasks deleted. The integration that emerged represented what the 5 experts consented to be the best way of conducting the duties. The new arrangement was noted down on the master manual and sent to the typist so that a new version of the manual could be produced.  The  sessions were  also used  as an information  dissemination  forum  so the  testdesk personnel from different districts had a chance to exchange insights and identify problem areas. Brophy noted down these and subsequently  wrote letters  to the staff personnel to request further investigation into identified problem areas and  also to make recommendations that certain procedures  could be passed onto  the other districts' testdesk personnel. They also pondered implementation  issues  such as whether the PC was the best environment for the expert system and if testers were familiar with it.  The  next stage of the KA  would handle  process had the experts together discuss how they  different trouble types. The crucial duty was when the test desk  Case on the Test Desk Decision Support Tool / 66 person had obtained the trouble reports and other relevant data and proceeded to analyze them to identify the trouble type. There were the trouble reports, trouble types and dispatches and the trick was to correctly interprets the trouble reports, accessing  the necessary  databases  and anyalyzing the data  diagnosis, which would point to a type of trouble, and then appropriate dispatch. Again figure that  working  from  to arrive to send  was  indeed  caused  out the  the flowcharts in the manuals (see  17 for sample flowcharts), they described the steps necessary the problem  at a  by a particular  type  to verify  of trouble. They  worked from the existing flowcharts and the experts would comment on them. At one  point,  one  expert  suggested  starting  from  scratch  again  because the  flowcharts were not good. Howell went over to the board and proceeded  to draw  out the flowchart as they indicated.  One expert commented that the flowchart did not capture all necessary details. For example, when a tester called the customer and there was no answer, s/he had  to decide if the customer was out or was the line not working. Brophy  defended  their use among experts and referred to them as "working documents"  to "figure out where everything is, and how it is ending up, and how it came in...and where you have problem." Howell final document to them  on the hand, promised  for verification. (See figure  to send the  18 for a sample redrawn  flowchart).  On the last day of the group meetings, Howell explained the top level design of the expert system to the group. Working out the procedures  to be encoded in  the knowledge base and clarifying part of the static information had been the  Case on the Test Desk Decision Support Tool / 67 objective for the last five days. Howell explained they aimed to build a prototype based on the no-dial-tone type of trouble by January and they  or February  of next year  then expand upon the knowledge base. By the time the prototype is done would  meet  again  to evaluate  it. Meanwhile  the MIS department was  responsible for finding a way to retrieve the necessary data from the databases without having to enter them anew into the PC.  Howell also presented the options of structuring the knowledge base so that a mini expert system together  or have  is developed one expert  for each trouble type system  handle  and have them linked  all trouble  types  within one  knowledge base. The experts preferred the latter for they thought a menu-driven interface that directed the user to the appropriate rule base was best for the "new  guy."  Observations  1.  and  Interviews  with  the KE.  Howell explained to the researcher that the difficulty in the group sessions was  reconciling the differences across the sites : what worked at Victoria  might not be the best for Vancouver or Abbotsford. For the expert system, the objective was to cover the generic problem and local variations would be  added  later  to this  basic  framework.  He  envisioned  a  series of  mini-expert knowledge bases on the different trouble types and then linkages made to connect  them  together. Such  an approach  would  facilitate later  modifications to the knowledge base. 2.  Interaction among the four experts during flowcharting was noticeably more  Case on the Test Desk Decision Support Tool / 68 intense than before. The  method to handle  different trouble types varied  from person to person, and the discussion at times became heated as they could not agree on one  best way  to handle  a trouble type. In general,  however, the experts co-operated in an open, serious yet jovial manner. The atmosphere was old  relaxed and the researcher felt they were like a group of  friends meeting  interesting.  Some  flowchart, but  they  to discuss an heated  issue they all felt was  arguments  were either  erupted  resolved  by  over  important  fine  the group  points or by  in  and the  Brophy's  writing memos to check with some other persons in the company. 3.  Howell  was  skillful as the mediator  of the meeting  and  in making sure  everyone had a chance to speak. * * *  On  16 December  group sessions two  1987,  Howell  and  Brophy met  for the first time  weeks ago. Their current objective was  to concentrate on the  flowchart for the trouble type of no dial tone, which covered 40% reports and whose diagnosis was  Howell Tom  planned  who  was  since the  of all trouble  considered relatively straight forward.  to meet with the software developer the next day a former analyst with the MIS  department  with Jack  and newly appointed  to be in charge of Artificial Intelligence (Al) applications at B.C. Tel. Tom familiar with the databases (e.g. TAS, FMS)  5.5  Development of the Rule Base.  necessary for the expert system.  was  Case on the Test Desk Decision Support Tool / 69 On with  17 December Marie  development  1987, Brophy, Howell and  Burlinson, firm  owner  responsible  of RJM for the  the researcher met  Computer simulation  Systems,  program  for five hours  Ltd., a  on  the  4tel  software testing  equipment. Howell had some ideas on the design of the expert system and spent a good portion of the day explaining to Burlinson his ideas.  Detail implementation issues like number of data files accessible from the shell (PC L E V E L like  work  5), structure of knowledge bases, as well as administrative issues schedule, and  tentative  time  commitments  explained the mechanical details of the domain and considerations to Burlinson, who  were  discussed. Brophy  Howell the software design  would redesign the system after she had gained  sufficient familarity with the shell. Burlinson  anticipated problems  with writing  the rules as she was unfamiliar with the test desk problem and Howell promised to help her in writing the rule base.  They agreed that the first phase was  to write the design specifications, and then  they would develop a prototype to correctly diagnose no-dial-tone type of troubles. Burlinson  asked  for  one  week  to  learn  the  shell.  approaching, they agreed to meet again after the New  Since  22 January prototype I done 5 February revised prototype I done 15 - 18 Februarj' group revision of prototype I  was  Year. By then she would  have the design specifications ready. A tentative schedule was  8 January specs done  Christmas  set up as follows:  Case on the Test Desk Decision Support Tool / 70  On  8 January  1988,  Howell, Burlinson, Tom and the researcher met to outline  the  function and structure of the system. Burlinson explained  what she knew  about the workings of the shell to Howell. They proceeded to write the rules based on the flowchart on "no-dial-tone" produced at the group sessions but soon agreed it would be more productive for Howell to write the rules on his own and  then meet again to edit them. A tentative work schedule  was set up as  follows:  12 Jan 8:30 - 11:30 a.m. write rules 13 Jan 8:30 - 2:30 p.m. program rules 19 Jan 8:30 - 12:30 p.m. program rules 20 Jan 8:30 - 12:30 p.m. program rules 21 Jan 8:30 - 12:30 p.m. program rules 22 Jan whole day finish prototype I  Observations  1.  Howell and Burlinson had co-operated and  previously on the simulation program  had a close working relationship. Burlinson has Masters' degrees in  Computer Science and Statistics and supplied the technical know-how needed in the expert system development team. 2.  Constrained by funding approved for the project, the participants had to set  Case on the Test Desk Decision Support Tool / 71 timetables for themselves by  estimating the time needed to do  the work  and the number of man-days for which the budget allowed. *  On  * *  13 January, Howell reported they concentrated on writing the rules for the  no-dial-tone flowchart up  to the test/no test decision point. He  familarizing himself with the shell and 14 and  expected  spent  an  hour  to continue doing the same on  15. *  *  *  On  18 January, Burlinson tried out the shell and  She  tested out how  to initialize variables and  entered  some dummy rules.  access dbase files. (See figure 19  for some sample rules). *  On  19  January,  the  researcher  * *  attended  the  development  session.  recounted he spent five to six hours learning the software on 14. On Brophy tapped into CRIS, FMS,  and  TAS  and  Howell  18 he and  transferred all cases and  screens  related to the trouble types into files which he passed onto Burlinson. He showed Burlinson the rules he wrote. The  also  rules were syntactically incorrect for  the shell and Burlinson edited them.  Again they based the rule generation on the flowchart. For every decision point on the flowchart, they tried to consider all information relevant to the decision. For  example,  on  dial  tone  versus  no-dial-tone, both  conditions external  and  Case on the Test Desk Decision Support Tool / 72 internal to the plant had to be considered. The databases, memos, or on  line obtained by  information might be from the  the test desk persons. For a first  attempt, they categorized the rules into out-of-company and in-company rules.  As  Howell explained the syntactically-incorrect rules to Burlinson and  the latter  edited and entered them into the computer, they were simultaneous^ discussing the structure of the rule base. They worked through iterative refinement of the rule-base. After putting down a few rules, they backtracked and ensured all the antecedants are covered and added a rule to explain them if necessary.  They planned on expanding the rule base in the next few days. Burlinson would enter them from  and  work on the randomization program to pull out random cases  the simulation program and  and 21. On  Howell would continue to enter rules on  20  Friday, he would bring an expert to review the rules. * * *  Instead of producing the first demonstration prototype on 22 January as planned, both 21  and  conversation Brophjr  was  22 Januarj' were spent expanding with  the  researcher, Howell  the rule base. In a telephone  reported  a  shortage  unavailable for the project this week. Another  Hans Enns, who  used to work for Brophy and was  years test desk and  of experts  as  tesk desk person,  a senior technician with 7  2 years 4tel maintenance experience, would be assigned to  the project instead. Enns would be responsible for entering and refining the rules with Burlinson, a process which was week. Meanwhile  Burlinson was  expected  working  on  to take up the  random  the whole of next number  generator  Case on the Test Desk Decision Support Tool / 73 program. * * *  On  26 January, Enns, Howell, and Burlinson worked together on the rule base.  Enns  and  Howell  focused  on  refining  and  adding  rules  and  debugging them. Enns also collected cases from CRIS, TAS, up  the file structure to transfer them over to dbase  Burlinson  and FMS  on  and set  files. Meanwhile  Howell  reworked the budget and work plan. * * *  On  27 January, they again worked on pulling cases from the simulation program  and screens from CRIS over to the expert system. Howell's boss Gordon met with the MIS  Meanwhile  they  Baker  division to discuss future field installation plans.  continued to add,  refine,  and  verify  the  rule  base. Howell  demonstrated the system to Enns and showed him the hardcopy of rules. Enns then modified the rules on the hardcopy. *  On  28 January, the researcher was  process. Aside  from  entering  *  *  present when they continued the revision  rules, Enns  explanatory text with Howell. Burlinson was  also  the  wording  of the  correcting the syntax of the rules  and ensuring the control structure worked. She something was  modified  probed for clarifications on  how  done and Howell and Enns would check the hardcopy rule base  in response to her queries.  Case on the Test Desk Decision Support Tool / 74 Enns tested the system. As he tried different options, he compared the system's responses with his experience to verify that it reacted as he would at the test desk. In his judgement, the system  contained all the rules necessary to handle  no-dial-tone type troubles up to the test/no test decision point : "the system does not take short cut up to that part."  Since the company demonstration of the system  was scheduled for 9 February,  they spent substantial amount of time discussing how to prepare for it. Burlinson pointed out the interface was "terrible." Presentation of the line of reasoning was difficult  for users  to understand  and she planned  to "pretty  it up." The  prototype in general was not user friendly and did not have good displays.  A schedule was set up to have the system thoroughly debugged by 29 January, and devote  1 February to the interface. On 2 February Brophy would test the  system; Burlinson would spend the rest of 2 and 4 making the changes Brophy deemed necessary. Then on 5 February, Enns would do a last check on  the  system. By 8 February, the demonstration-level prototype would be ready for the presentations scheduled on 9 and 10 of February.  They also discussed starting to write the rules for prototype II and the pros and cons of having a Microtel programmer on the team instead of hiring Burlinson's assistant. Together  with  Enns, they  decided  to call  the system  Decision Support Tool."  Observations  & Interviews  with the  Software  Developer.  "Test Desk  Case on the Test Desk Decision Support Tool / 75 Burlinson was was  enthusiastic about the shell. The most attractive feature she found  the ease with which it allowed users to exit to the environment. a=  2 February,  On  reactions. He  *  *  Brophy tested the prototype and  assumed the user's role and  the researcher observed his  selected different options to test the  system's logic. "Neat", "pretty good" were some of his comments. Specifically he detected a logic error in the program, some non-standard  acronyms and typos,  and that two fields were missing in the database. Recognizing the prototype only dealt  with  simple  problems  within  one  trouble  type,  Brophy  suggested  incorporating a more involved trouble type and providing window access to more than  one  TAS  areas to be  report screen  important  whenever  necessary.  Howell  acknowledged  these  but decided to postpone incorporation of more rules to  handle other trouble types. Presently, thej' would concentrate on providing access to the TAS  screens during the run, more user friendly interface and reporting  features in preparation for the demonstration.  5.6 Epilogue.  On  9 February, the researcher called Brophy for a report on the demonstration.  He  reported that on 8 Februar3% a presentation was  B.C.  Tel, some  MIS  representatives. The technology  MIS  personnel,  some  service  made to the Staff group at  centre  managers,  and  union  people were most receptive and interested in using the  in other applications. They tentatively agreed  to commit people  and  Case on the Test Desk Decision Support Tool / 76 money to support the project. On  9 February, another demonstration was  made  to the service office managers from all areas in the province. Despite materials on A l and expert systems distributed to them several days before the meeting, basic knowledge about the new  The consensus so far was the  courses. But  district  technology was  lacking.  to use the expert S3'stem for internal training and in managers  showed  reservation  in  using  it at  the  operational level. They were unsure how  it would fit into current and future test  desk  to  operations, and  Support  Services,  deferred decision  Steve  Brandner,  who  Brophy's opinion, the expert system  had  the  Staff  Manager  championed  the  of Operational FMS  before. In  had been developed using a "grass-root" or  "bottom-up" approach, and Howell had been the champion behind the project. The time had come for top management to accept the technology :" now to sit back and  let it sink in, and  make sure people can  (we need)  accept it, and let  everybody have time to digest it."  * * * On  10  February, Howell  reported more presentations to senior management of  the company and the reaction had been "favorable." The MIS interest in expanding was  the expert system  group showed keen  technology to other applications and  he  making plans to bring the S3'stem to the other areas in B.C. for feedback.  C H A P T E R 6. CASE ON T H E  WASTEWATER T R E A T M E N T E X P E R T SYSTEM  6.1.  Introduction.  The  Waste  Chilibeck  Treatment  who  Department  was  Expert _System  a  of Civil  ( WASTES  ) was  developed  Master's  student  in environmental  Engineering  of the  University of British  expert system project was  by  engineering  Barry at  the  Columbia.  The  his Master's thesis, and the objective was  "to produce  a prototypical demonstration  tool from which further research could be advanced  from  Department  the  Civil  Engineering  and  in the  Environmental  Engineering  Group" [Chilibeck, 1988:100].  Chilibeck took  a number of courses on wastewater treatment  with the domain. The experts who  and  was  familiar  contributed knowledge for the system included  his thesis supervisor, Dr. William Oldham, and three plant operators. Dr. Oldham is  a  recognized  chairman  authority on  of the  Civil  wastewater treatment  Engineering  Department  plant design  at  the  engineer  and  University of British  Columbia. The three operators were chosen based on their extensive experience in the field. Another Chilibeck  on  professor  software  at  issues. The  the  department, Dr.  system  was  developed  developed by a fellow graduate student Thomas Froess.  6.2  Data Collection Methods.  77  Alan  Russell, counselled  using  a  shell  FRO,  Case on the Wastewater Treatment Expert System / 78 Data  on  the  development  process  interviews with Chilibeck and  was  obtained  primarily  from  post  facto  the chief domain expert, Dr. Oldham. Concurrent  observations on the process was  impossible because the researcher did not learn  of the project until six months after its initiation. Chilibeck began the project in September of 1987  and by February  heard about it, he had only two draft of his thesis was description has  which was  when the researcher  days of coding left for the prototype. A  completed  been compiled  of 1988,  by the end of June. The  from  the interviews and  first  following narrative  from  the draft of his  thesis.  6.3  The  Wastewater Treatment Operations  system aims to aid the operation of conventional secondary  wastewater treatment facilities and diagnose problems that may facilities are called preliminary, primary, secondary  activated sludge arise. Treatment  and tertiary depending on the  level of waste water treatment provided by the combination of unit operations. A treatment plant consists of an interconnected string of unit operations, which are physical, chemical, and biological processes that remove waste materials from the water. A and  secondary  primary  facility includes all lower degrees of treatments, preliminary,  [Chilibeck, 1988:17]. A  plant operator  controls conditions such  as  temperature, air, nutrients, and hydraulic flows in the pipes in order to maintain the system to facilitate or retard bacteria growth. Other  variables that can  be  controlled include age and type of bacteria.  The  secondary  treatment facility is chosen as the model for the system because  Case on the Wastewater Treatment Expert System / 79 the  largest proportion  category,  of plants  and developing  a system  in use in North  America  belongs  to this  for this level of treatment would describe  preliminary, primary, and secondary treatment operations. The treatment of the municipal which  wastewater is usually carried out by the activated sludge process in  microorganisms  metabolize  the waste  material  that  would  otherwise  consume oxygen following discharge into the receiving river, lake estuary, or sea. Too  low an oxygen level will kill fish and other living things, and cause the  body of water to "die." The discharge of inadequately treated wastewater by the city of Philadelphia, for example, has resulted in oxygen levels in the Delaware River estuary that are inadequate to sustain migrating fish during the season of low flow [Denn, 1986, p.28]. The activated sludge process was suitable for expert system technology  because it requires substantial operator  knowledge and input  for process control as compared to other secondary processes [Chilibeck, 1988:23].  The  layout  of  a  secondar3'  sewage  plant  is  shown  in  figure  20  [Chilibeck, 1988:25]. The municipal wastewater or influent first passes through a lift pump  to the racks, shredders,  and grit chambers where large objects are  removed. The sewage then enters the primary clarifiers where 60% of suspended solids settle out. Some of the suspended solids are carbonaceous and have a biological oxygen demand (BOD). The water then passes into the aeration tanks, where microorganisms attack the dissolved and suspended carbonaceous material (as well as nitrogen and phosphorous compounds); the microorganisms grow on this substrate, and produce carbon dioxide and water. Air and oxygen must be added  continuously  to keep  the dissolved oxygen  level  above  a critical  level  [Denn, 1986:28]. The treated sewage finally enters the secondary clarifiers and the  Case on the Wastewater Treatment  Expert System / 80  clarified effluent can be chlorinated and discharged. The primary clarifier sludge and part of the secondary  clarifier sludge is pumped into the sludge treatment  and disposal works. The sludge could undergo a series of unit operations before disposal,  but these  chlorination system  operations  are not covered  in WASTES.  is omitted [Chilibeck, 1988:24]. WASTES  Similarly the  covers the following  unit operations: 1.  raw sewage pumping,  2.  racks and screening,  3.  shredding,  4.  grit removal,  5.  primary clarification,  6.  microscreening,  7.  biological treatment,  8.  a.  activated sludge,  b.  rotating biological contactors,  secondary  Among  these,  clarification. the activated  sludge  basins  and operations  of the secondary  clarifiers are the most important. For detailed explanation of the components of the treatment system, see sections 4,5, and 6 of [Chilibeck, 1988].  Traditional  process  control  of  waste  treatment  plants  has  algorithmically by manual or automatic control systems. WASTES heuristic rules based mixed  liquor  and  on observational data on the color aeration  tanks,  and  [Chilibeck, 1988:50]. The literature has isolated  its  been  done  however, uses  and condition of the  microscopic  inhabitants  a number of visually observable  Case on the Wastewater Treatment Expert System / 81 variables to identify problems in the plant, which include: 1.  activated sludge floe type and settleability,  2.  aeration tank forming and foaming type,  3.  secondary clarifier observations,  4.  microbiological examination.  In conjunction with other numeric variables such as effluent BOD suspension (TSS) levels, and BOD these are used to determine  and food-to-micro-organism  and total solid  (F/M)  loading rates,  the conditions of the activated sludge process. These  variables serve as indices to the knowledge. [Chilibeck, 1988:50].  In a discussion with the researcher, Chilibeck suggested expert system technology improved upon numerical or algorithmic process control for the former was  more  adaptable to unforeseen conditions and often cheaper. Numerical control programs required installation of digital meters at critical sites at the plant so that data on  the  conditions could  investment  and  may  be  fed into  the  program. This  involved considerable  not be suitable for plant operators who  are generally not  technically sophisticated. Their knowledge of the plant is mostly experience and  their method of control is "an  art more than  gained  through  a science." An  expert system allows heuristic rather than numeric control and is more flexible.  In an  interview with the researcher, Dr. Oldham likewise suggested  equations were inadequate  numerical  for process control. Equations for individual processes  existed but there were none that related all the processes together for the entire plant and by  interactions between unit operations were too complex to be described  equations. Moreover, an  equation valid  under one  set of external conditions  Case on the Wastewater Treatment Expert System / 82 became invalid when the conditions changed. For example, domestic waste water in  Vancouver  differed  from  that in Edmonton  due to variances  in housing  development, and the treatment processes for the two sites could not be identical. Hence the experience of operators at the two locations served as a more reliable basis for control than equations.  For Chilibeck, motivation for developing the system also came from the need to fill the gap between research theory and practical treatment The  plant application.  experience that operators possess fill this gap and encoding it in heuristic  rules is one means to capture this valuable asset. A new operator learns his trade by example  and from  the knowledge of the experienced  operators. The  system is therefore useful as an operational as well as teaching aid. Chilibeck indicated  the heavy  university  emphasis  on design  in the engineering faculties  in  the  meant that students often were unfamiliar with operational realities.  Hands-on experience with the expert system  would enable  students to have a  "feel" of what it was like to operate a plant. Anticipated users of the system include  experts  non-experts  such  as waste  treatment  like municipal engineers  plant design  engineers  who are unfamiliar with  as well as  the domain and  students. Chilibeck described his potential users as follows:  One of the goals of the system was to make it simple enough that a person unfamiliar with wastewater treatement facilities could operate the expert system, understand what the system is doing, and learn about the treatment process. At the same time the system should be designed to be useful to someone familiar with wastewater treatment. That person should be able to quickly find an area that he or she is interested in and examine it thoroughly with the system" [Chilibeck, 1988:70].  Case on the Wastewater Treatment Expert System / 83 6.4.  Knowledge Acquisition.  There  were  three  sources  of information  for the system  and their  relative  importance were as follows : 1.  50% from  published works on process  control of waste treatment plants,  these supply primarily diagnostic information. 2.  25% from the chief domain expert, Dr. Oldham, and  3.  25% from the plant operators with whom he had short conversations over the  telephone,  primarily  when  he  needed  to clarify  process  control  information.  The  written sources of information consisted of texts specifically written for plant  operations and process control, articles from magazines, and research publications from journals. The Environmental Protection Agency has published troubleshooting guides and process control manuals for municipal treatment such as Field Manual for Performance Evaluation Facilities[197 8]  and  and  Process  Troubleshooting at Municipal  Control  Manual  for  Aerobic  Wastewater Treatment Biological  Treatment  Facilities [1977] published by the United States Environmental Protection Agency. Additional troubleshooting information was gained from the Water Pollution Control Federation publications on design and operation of treatment facilities as well as their monthly series, Operations Forum [Chilibeck, 1988:68].  From the literature, Chilibeck obtained information which he represented trees.  Then  presented  he asked  the expert-operators  in the logic trees. Since every  for verification  in logic  of the information  plant was different and no model-plant  Case on the Wastewater Treatment Expert System / 84 existed, the analysis presented in the written works often did not coincide with reality. For example, he presented to the expert-operators three different ways of controlling a process derived ask  from three  different schools  of thought. He  them which method they adopted at their plants, and  "the easiest method." Chilibeck then modified  would  the operators chose  the logic tree accordingly  and  the  corresponding rules were incorporated into the knowledge base.  6.5. Development of Rule Base.  After  the  initial phase of acquiring  literature, Chilibeck consolidated  knowledge by  the information  extensive  research  into the  he needed for the expert system  by identifying the following : 1.  unit processes and  operations  to be  included  (see Section  6.3  above for a  list); and 2.  common plant problems.  For each plant problem, key information or parameters both inside and  outside of  the unit operations were identified and the corrective actions obtained.  The  result of this consolidation process was  each  unit operation  relating all possible  a logic structure or flowchart for  problems with  the  examined which determine what precise problems existed, and  parameters to  solutions for them.  Chilibeck then proceeded to convert the flowcharts, starting with the smallest simple ones, into rules, which were subsequently coded and and of  bases  were  developed  in  and  debugged for logic  syntax errors (for sample trees, see figure 21) [Chilibeck, 1988:79]. A mini-knowledge  be  this manner, one  for  each  series unit  Case on the Wastewater Treatment Expert System / 85 process.  Then  they  were  knowledge base. Rule  linked  to the menu  structure and to the main  bases on problems of floating materials and odor, which  affected the entire plant, were also included. For a diagram of the file structure of the system, see figure 22 [Chilibeck,1988:61].  As  the knowledge  base  was  being  developed,  the expert's  contribution of  knowledge played a significant role. Some processes were not well documented in the literature, and Dr. Oldham supplied the necessary clarifications.  According to Chilibeck, knowledge acquisition sessions with the expert were highly informal  and  usually  described  the scenario  lasted  half  an  to Dr. Oldham  hour.  For simple  and through  problems, Chilibeck  discussion arrived  at a  conclusion. For more complex problems, Chilibeck demonstrated the system to Dr. Oldham, who  assumed the user's role and suggested  changes when  necessary.  Chilibeck reported he consciously maintained the expert's interests by showing him the  system. Their interaction  was  continual feedback was obtained and  letting  him see how  facilitated  by a shared  vocabulary. Hence,  from the expert primarily by posing scenarios  the system  functioned. On  5 to 6 occasions, Dr.  Oldham also took the system home over the weekend when he reviewed it as a user to check if all relevant variables have been considered.  In situations when the three operators did not agree among themselves, Chilibeck relied on Dr. Oldham's judgement. With years of experience engineer,  Dr. Oldham  was  able  as a plant design  to resolve differences or attribute  them to  circumstantial variances. If operators disagreed on the optimal procedures because  Case on the Wastewater Treatment Expert System / 86 of differences in plant size and conditions, they might be all correct for their own plants. In such cases, the system had to accomodate all situations.  While had well  the operators were knowledgeable about the plant operations, Dr. Oldham an overall appreciation for the entire operation of the treatment facilities as as how  impacted importance  the environment, such  as the quality  of the incoming  sewage,  the plant. At an interview with the researcher, the expert stressed the of the operators' input for "a good operator can rectify  a lot  of  design errors in a treatment plant; a good designer still needs a good operator at the other end or you wouldn't get results."  The  expert added that while he was helping Chilibeck with his system, he was  simultaneously working closely with a plant operator to improve the waste water treatment  operations in Kelowna. Discussions with  Chilibeck  often clarified his  own understanding of the problem at Kelowna.  In the expert's opinion, WASTES is usable but not foolproof. It is complete in that it has addressed all the important variables that should be considered in a conventional secondary  treatment plant. But for an esoteric facilit}' that involves  processes not currently understood, the system, would have problems. More basic research is needed before a "complete" sj'stem can be developed.  6.6. Epilogue.  On  April  7 1988, Chilibeck  gave a demonstration  of WASTES  to his  fellow  Case on the Wastewater Treatment Expert System / 87 students at the Civil Engineering Department. At the time of writing, he planned to send the system out for evaluation by the field operators.  C H A P T E R 7. CONCLUSIONS  The  study  has reported on an empirical  investigation of three expert  system  development projects. This chapter presents a number of observations drawn from the inquiry and discusses some limitations of the study as well as suggestions for future research.  7.1 General Observations.  1.  The "rapid-prototyping" approach  is a well-established technique in building  expert systems [Gaines, 1987b; Buchanan et al.,1983; Young et al., 1987]. The  notion of "rapidness" however, is much  harder  to precisely  define.  Buchanan et al., for example, suggested implementation should start once "a few  key concepts and relations" are noted down. Similarly, Naumann and  Jenkins  emphasized  the importance  of constructing  a  working  prototype  within a short time [Naumann et al.,1982; Jenkins, 1983]. Observations of the law and tesk desk cases suggest that while prototyping is a valid approach  in building expert systems, implementation  should not begin until  the domain is reasonably understood.  Referring to information systems, Jenkins opined that prototyping rests on the  premise  requirements behaviourally  that  development  of  complete  specifications prior to implementation infeasible  [Jenkins, 1980].  88  and  correct  is both  For expert  information  technically and  sj stems, instead of r  Conclusions / 89 specifying the information requirements, expertise is to be represented. The mapping between expertise and the conceptual model requires a series of modelling  processes,  and prototyping  is appropriate  for ES construction  precisely because it is a tool for modelling which facilitates communication between the K E and the expert [Sol; Jorgensen, 1984]. It is useful in both the latter part of the modelling as well as the knowledge elicitation phases (see figure 2) [Benbasat & Dhaliwal, 1988b].  A  clear definition of prototyping is often lacking in the literature, in the  present context, it refers to a system that is to be expanded, supplemented, or supplanted [Jenkins, 1980], and not a system that is "created as quickly and  as cheaply  as possible to test the feasibility  of a new  concept or  approach [Asner et al., 1987]. In fact, empirical observation from  all three  cases suggests that the latter notion of prototyping is to be avoided and implementation  should begin only when the domain addressed is substantially  clarified.  In the case of the Remoteness Advisor, the expert and knowledge engineer spent  seven  sessions formalizing  system) before the K E reasonabfy  developed  the structure of the domain  (and the  a prototype. At that point, both were  assured that the structure would remain stable. Speaking  experience, Smith and Deedman stressed the importance  from  of coding only a f t e r  the structure of the domain had been formulated. Prototyping early resulted in changing the code in the prototype system, which caused conflict between the expert and the knowledge engineer for the latter found  it difficult to  89a Fig.  2.  A  Validation-Based Life  Cycle  Model  of  the Knowledge-Based  System Development Process (Benbasat A DhaliwsJ, 1988bI  IDEALIZED* KNOWLEDGE BASH <aooeast<at • (cafity>  OTHER ALTERNATIVE KNOWLEDGE-BASES W REALITY < t | , ocbei cxpe/t»>  "SOURCE" KNOWLEDGE BASE IN REALITY <t j., louxce expert >  IMPLEMENTED KNOWLEDGE- BASED SYSTEM  CONCEPTUAL KNOWLEDGE-BASE  ELICITED KNOWLEDGE-BASE  N O M 'Concafflu* and * M « .•trtrton mwtkM Kmmlm\gt l*rfj*%m  M M n  Snpiwnantadan « M s o o n can b« Mrmd V M a f a  Bo4t KjrxrSonX vol rapnanaSonol nSdclon conoftut* K*imtnifl-9fwm  1  VMMjfew,  Conclusions / 90 give up what he had spent time working on. Similarly in the case of the Test  Desk  Decision  Support  Tool, the project team  spent  two months  updating and verifying the knowledge in the manuals before thej' took the design of the expert system to the software developer. In addition, the cost devoted  to clarifying the knowledge was substantial for the four test desk  persons  were released from  their jobs for a week  and brought to the  Microtel Learning Services Centre for the group sessions.  2.  The KA  techniques  that were observed  in the three  cases  include the  following : a.  Interview in the form of question-answer between the expert and KE; typically,  structure  unstructured  ones  interviews at the later  predominate KA  at  the  sessions. This  early  and  technique  was  observed in all three cases. b.  Scenario-elicitation  where the K E  posed  a scenario and the expert  commented on it; this method was observed  in the two engineering  cases. c.  Task analysis by decomposing the domain into goals and subgoals; this was used in test desk case.  d.  Group discussions among the experts, also observed  in the test desk  case. e.  Prototype  or system  refinements,  observed  in the two engineering  cases. Among these, scenario-elicitation was recommended in two out of the three cases. This finding supports  the literature suggestion  that experts find it  Conclusions / 91 easier to demonstrate their expertise in a particular situation rather than formulate it explicitly [Witten et al.,1987]. It is noted that the techniques are by no means mutually exclusive : prototype or system refinement might be a means to elicit scenarios.  3.  Diagramatic representations of the three KA involved  in  each  case  and  are  processes summarize the steps  presented  in  figures  3,  4  and  5.  Characteristics of the three cases are summarized in figure 6.  4.  The  problem domain emerges as a critical factor in the KA  determines  process, which  the number of experts to be consulted. In the two engineering  cases, since the tasks involved diverse locations and could not be adequately modelled  as  a  [Henderson, 1987].  single In  the  process, Test  multiple  Desk  experts  Decision  Support  were  required  Tool  case, for  example, five experts were involved in the knowledge acquisition process and another  one  Treatment  during  case,  four  development experts  of  the  were  rule-base. In  involved  in  the  clarifying  Wastewater the  domain.  Experts from different areas in British Columbia were consulted to ensure local knowledge was  incorporated into  the  system. Only  Advisor case falls within the traditional mode of eliciting In and  the Remoteness from  one expert.  this case, the domain addressed is more subjective than the other two Smith  was  one  of  the  few  (if not  only) proponent  of  a  "deep  structure" interpretation of the law of negligence.  5.  The  domain is also important in another respect. Both the test desk and  read Appendix of [Smth. 84]  c l a r i f y phy. categories  c l a r i f y philo categories  ~T~ combine in struct, chart  develop kb  Fig. 3 KA Process for the Remoteness Advisor  91b  read Manuals  group update of Manuals  flowchart strategies  i  develop  4  kb  systen refinement  Fig. 4 K A Process for the Test Desk Decision Support Tool  read  zrz  EPA Manuals  develop logic trees  T"  verify with operators develop kb  *  1  x systen refinement  Fig.  5 K A Process for WASTES  Fig. 6 Summary Table of Characteristics of 3 Cases  Sample 1  Sample 2  Remoteneas Advleor  Teat Desk D e c i s i o n Support  1  1  1  Number of B x p e r t ( s )  1  5  4  previous experience of KB  yea, 1 ayetem  took a course In B8  no  no  no  Bxpert  System  Nuabil of  Kllll  previous experience of Expert yea, 1 system  Sample i Tool Wastewater Treatment  Expert  8ye  Bxpert-KB r e l e t l o n e h l p  personal,shared vocab  p r o f e s s i o n a l , shared some vocab a d v l a o r - s t u d e n t , shared vocab  Problem D o M l n  remoteness In law  t e a t deak d i a g n o s t i c s  waatawater  Problem Category  c l a s s i f i c a t i o n , bottom up  d i a g n o s t i c , top down  d i a g n o s t i c , top down  Objectlve/Subjectlve  subject 1ve  objective  objective  Olobal va L o c a l Domain  qlobal  global  qlobal  Closed/Open Domain  r e l a t i v e l y c l o s e d - cases  1 of t r o u b l e types s e t  • of u n i t proceasea  Q u a l i t a t i v e va Q u a n t i t a t i v e  qualitative  quant I t l v e * q u a l i t a t i v e  qualitative • quantitative  KA Procesa: aouicaa of Info  caae book, (Smith,1964)  manuals f o r equipment  BPA  IS C o n a t r u c t I o n : t o o l  shell  PC L e v e l 5  PRO  Orq.  Law  H l c r o t e l Learning S e r v i c e s  Dept. of C i v i l  job a i d . I n s t r u c t i o n a l  Job A i d , I n s t r u c t i o n a l  Usee  Setting  :H1  F a c u l t y , UBC  Instructional, advisory  tool  treatment  aet  manuals, j o u r n a l s  BnqlnearIng, UBC tool  Conclusions / 92 wastewater treatment cases involved diagnostic tasks while the law system addresses a classification problem [Hayes-Roth, 1983]. In the former instance, the number of parameters to be examined varies with the location while in the latter, the number of relevant facts are dictated by the set of court cases for this domain.  6.  It has been observed that the stages in the expert system lifecycle in fact overlapped KE's the  [Buchanan, 1983]. The  inquiry supports this finding. Despite the  efforts to keep the discussion to the conceptual level, the expert in Remoteness  during the KA Support  Tool  simulation  Advisor  sessions. The recalled  program. A  implementation  case  frequently considered expert and  screens  or  data  segregation of the  rests on the assumption  KE that  implementation  issues  of the Test Desk Decision could  be  used  from  the  stages of conceptualization and  that the domain expert is ignorant  about technology. In reality, however, experts who  are willing to participate  in expert system projects are individuals highly interested in technology and eager  to  be  Wastewater  involved in developing Treatment  non-computer-oriented implementation 7.  Expert  the  System,  systems. In the  expert  the  case  claimed  of the to  be  and the scheme of separating the conceptualization and  stages prevailed.  Close rapport between the expert and  KE  directly and  positively influences  the quality of the conceptual model produced. The law case is distinctive in that personal friendship existed between the expert and conceptual structure that resulted from  the KA  the KE  sessions was  and  the  a product of  the intellectual as well as personal dynamics that took place [Smith,1988].  Conclusions / 93 8.  The  Remoteness  were developed was  produced  Advisor  and  the  Wastewater  Treatment  Expert  System  in an academic while the Test Desk Decision Support Tool in a  prototype to win  commercial  managerial  environment. The  pressure  of building  a  support applied in the latter but not in the  former two cases.  9.  The  law  experts  and  civil  were  training and  engineering  knowledgeable  cases  in the  are  similar  in that both  domains. Deedman  is a  Chilibeck a Master's student in Environmental  KE  and  lawyer  by  Studies. Hence  they shared the experts' language as well as knowledge. In the law for example, the KE expert's  was  often able to cite relevant cases  categorization schemes; a  non-lawyer  would  not  case  to test the  have  had  such  facility.  7.2 O b s e r v a t i o n s o n t h e M e t h o d o l o g y  Conducting  longitudinal observation of multiple cases  requires a  research team  [Yin, 1984]. This theoretical necessitj' however, is impossible to practice in the present study because the subjects involved did not agree to having alternative observers. expended  Having the  familiarized  time  and  were understandably the  benefits  of  themselves  with  effort to acquaint her  the with  researcher  and  having  the basic jargon, subjects  reluctant to re-invest their energy. While cases do provide  current  data  and  observation  facilitates  understanding,  the  Conclusions / 9 4 demands of keeping track of multiple cases over a period of almost a year were onerous for one person.  Observation  of  the  KA  sessions  was  a  valuable  means  to  obtain  data.  Maintaining rapport and developing trust with the subjects were essential so that they  did not  feel uncomfortable  about being  watched. Taped  sessions were useful if the quality of the tapes could be difficult  especially  subjects who  since it was  impossible  assured. This proved  every  single session  and  took over the responsibility might have been less careful. Group  sessions were difficult to tape and  the recordings were hard  multiple voices might overlap. In the law two  to attend  protocols of the  participants in a reasonably  to transcribe as  case however, since there were only  small room, the tapes  were invaluable when  legal jargon became hard to follow during the session.  Retrospective interviews were useful for clarification purposes and  less so as a  primary data gathering tool. In the case on the Wastewater Treatment System for example, the researcher  had  no  Expert  means of validating the responses  obtained. The interviewees might have succumbed to retrospection biases.  Not  all subjects responded favorably to keeping a diary and  described, it was  not adopted. In the two  in the three cases  cases where it was  proved extremely informative in tracking the process of how  used, the diary  the KE  approached  the task.  Deliverables were in general useful. For example, the structure chart in the law  Conclusions / 95 case and  the Job Profile document in the Test Desk case were indispensable in  facilitating understanding  of the domains. The  third case on treatment plant could  not have been completed without information from Chilibeck's thesis.  7.3 Limitations of the  1.  Study.  This study depended heavily on observational data. Despite the use of taped protocol and  post facto interviews with subjects to triangulate the collected  data, reactive bias due not  avoided.  Negligence reactive  Since  and bias  to presence of the researcher was  the  researcher  observed  the Test Desk cases introduced  likely  the  KA  minimized  sessions  over a period of over  decreased  as  the  subjects  but  for  the  six months, became  more  accustomed to being watched. Another threat to internal validity came from the researcher's record and  interpretation of data, which was  controlled by  showing the reports on each case to the subjects involved for verification. 2.  External validity  is constrained  by  the  fact that only  three  cases  were  studied. Attempts to generalize the lessons to other development efforts can be  made only  with  due  consideration to the  domain  addressed  and  the  human actors involved. 3.  The  emphasis of this study  cases,  only  observation  a  demo-level  terminated.  has  been on  prototype  Hence  the  had entire  the KA been  process. In all three  developed  development  by  lifecycle  the  time  was  not  documented. 4.  The  sample  producing  is biased  in  that  the  reported  projects  all succeeded  in  sj'stems. However, since these systems have not been evaluated,  Conclusions / 96 quality of the resulting systems remain unknown.  7.4 S u g g e s t i o n s f o r F u t u r e  Research  Empirical research in expert system implementation [Benbasat  et al., 1988a], much  is only in its early stages  work in this area needs to be  done in the  future. Some suggestions for further research are presented as follows: 1.  Validation of the findings made in this study can be conducted  by either  investigating other cases of expert system development [Bonoma, 1985] or through laboratory experiments. The  following are some hypotheses  that can  be generated from the inquiry: a.  What are the domain characteristics that necessitate the  involvement  of multiple vs single expert(s)? b.  Does a detailed study of the domain which result in a well-formulated problem  space  provide a better quality  knowledge base than  "rapid  prototyping"? c.  Is a  person  familiar with  the domain who  later acquires computer  knowledge a better candidate for the knowledge elicitation task than one who  has computer knowledge and then learns about the domain?  d.  Does user involvement improve the quality of knowledge base?  e.  Can  expert-user  replace  a  real  user  in  providing  input  in  the  development process? 2.  A  longer  observational period  could  be  undertaken  to  cover  the  entire  development lifecycle including the maintenance phase to better understand  Conclusions / 97 the expert system development lifecycle. The  issue of how  to manage an expert system development project has  seldom been studied and is a logical next step in the research agenda.  BIBLIOGRAPHY [Abrett  et al., 1986] Abrett  Glenn, M.H. Burslein,  "The KREME  Knowledge  Editing Environment," in Proceedings of the First Knowledge Acquisition for Knowledge-Based Systems Workshop, Banff, Canada, November 1986. [Addis,  1987] Addis,  T.R.,  "A  Framework  for Knowledge  Proceedings of the First European Workshop  on Knowledge  Elicitation", in Acquisition for  Knowledge-Based Systems, Reading University, September, 1987. [Alexander et al., 1986] Alexander J.H., M.J. Freiling, S.J. Shulman, S. Rehfuss and  S.L. Messich,  "Ontological  Analysis,  An  Ongoing  Experiment," in  Proceedings of the First Knowledge Acquisition for Kno wledge-B ased Systems Workshop, Banff, Canada, November 1986. [Anjewierden,1987] Anjewierden, A., "The KADS System", in Proceedings of the First  European  Workshop  on Knowledge  Acquisition  for Knowledge-Based  Systems, Reading University, September, 1987. [Asner et al.,1987] Asner, M., King, A., "Prototyping : A Low-Risk Approach to Developing Complex Systems", unpublished paper from Gellman Hayward  &  Partners, Ltd., 1987. [Bell,  1987] Bell,  Jill,  "Teaching  Knowledge  Engineers, the Human  Side of  Knowledge Engineering", in Proceedings of the First European Workshop on Knowledge  Acquisition  for Knowledge-Based  Systems, Reading  University,  September, 1987. [Belkin  et al.,1987] Belkin, N.J., H.M.  Brooks and P.J. Daniels,  "Knowledge  Elicitation Using Discourse Analysis," in Proceedings of the First Knowledge Acquisition for Knowledge-Based Systems Workshop, Banff, 1986.  98  / 99 [Benbasat et al.,1987] Benbasat, Izak, David K. Goldstein, Melissa Mead, "The Case Research Strategy in Studies of Information Systems", in Management Information Systems Quarterly, Vol.11, No.3 (September 1987), 369-386. [Benbasat et al., 1988a] Benbasat, Izak, Barrie  R. Nault, "An  Evaluation of  Empirical Research in Managerial Support Technologies : Decision Support Systems, Group Decision Support Systems, and Expert Systems, Faculty of Commerce  &  Business  Administration,  University  of  British  Columbia,  February 1988.  [Benbasat et al., 1988b] Benbasat, Izak, Jasbir S. Dhaliwal, "A Framework For the Validation of Knowledge  Acquisition", Working Paper No.88-Management  Information Systems-006, Faculty of Commerce & University of British Columbia, May  Business Administration,  1988.  [Bonoma, 1985] Bonoma, Thomas V., "Case Research in Marketing : Opportunities, Porblems, and a Process," in Journal of Marketing Research, Vol.XXII (May 1985), 199-208. [Boose, 1986] Boose, John, Expertise Transfer for Expert System  Design, New  York: Elservier, 1986. [Brueker  et al., 1987]  Brueker, J., B.  Wielinga,  "Knowledge  Acquisition  as  Modelling Expertise : the KADS Methodology", in Proceedings of the First European Workshop on Knowledge  Acquisition for Knowledge-Based  Systems,  Reading University, September, 1987.  [Buchanan et al.,1983] Buchanan, B.G., et al., "Constructing an Expert System", in  Hayes-Roth,  R.,  Waterman,  D.A.,  and  Lenat, D.B.,  Expert Systems, London : Addison-Wesldy, 1983.  (eds.), Building  / 100 [Burton et al., 1987] Burton, A.M.,  N.R.  Shadbolt, A.P. Hedgecock, G. Rugg, "A  Formal Evaluation of Knowledge Elicitation Techniques for Expert Systems : Domain  1", in Proceedings of the First European Workshop on Knowledge  Acquisition  for Knowledge-Based  Systems,  Reading  University,  September,  1987. [Chandrasekaran,1984] Techniques  to  Chandrasekaran, Tasks",  in  B.,  Reitman,  Applications in Business, New  "Expert W.  Systems  (Ed.)  :  Matching  Artificial  Intelligence  Jersey : Ablex Publications Corp., 1984.  [Chandrasekaran et al.,1985] Chandrasekaran, B., Bylander, T., "Al Research at the Ohio State University", A l Magazine, Summer 1985, p.74-79. [Chilibeck, 1988] Chilibeck, Barry M.,  Operation and Diagnostics of Wastewater  Treatment  Facilities Using An  Expert System, unpublished Master's thesis  completed  at the Department  of Civil  Engineering, University  of British  Columbia, 1988. [Deedman, 1986] Deedman, Cal, Building Rule-based Expert System Law,  unpublished Masters' thesis, Faculty  of Law,  in Case-Based  University  of British  Columbia, 1986. [Denn,  1986]  Denn, Morton  M.,  Process Modelling, New  York, Longman Inc.,  1986. [Dhaliwal  et  Empirical  al., 1988]  Dhaliwal,  Investigation  of  Jasbir  S.,  Knowledge  88-Management Information Systems-008,  Izak  Benbasat,  Acquistion,"  Working  Model for Paper  No.  Faculty of Commerce & Business  Administration, University of British Columbia, July  [Fellers, 1987] Fellers, Jack W.,  "A  1988.  Key Factors in Knowledge Acquisition, Computer  Personnel, Vol. 11, No.l, May  1987.  / 101 [Gaines, 1987a] Gaines, Brian R., Integrated Knowledge Support Environments for Rapid Prototyping of Expert Systems, Invited Paper. [Gaines, 1987b] Gaines, Brian Oliff,  M.  Systems  (ed.), and  R.,  "Rapid  Proceedings  the  Prototyping for Expert Systems," in  of the  Leading  Edge  International  in  Conference  Production  Planning  on and  Expert Control,  University of Southern California, 1987. [Gaines,  1987c] Gaines, Brian  R.,  "How  Proceedings of the First European  Do  Experts  Acquire Expertise?," in  Workshop on Knowledge Acquisition for  Knowledge-Based Systems, Reading University, September, 1987. [Gaines, 1987d] Gaines, Brian R., "Knowledge Acquisition for Expert Systems," in Proceedings  of  the  Second  Knowledge  Acquisition  for  Knowledge-Based  Systems Workshop, AAAI, Banff, November, 1987. [Gaines,  1987e]  Environments,"  Gaines,  Brian  R.,  in Proceedings  "Advanced  of the  Second  Expert  System  Knowledge  Support  Acquisition for  Knowledge-Based Systems Workshop, AAAI, Banff, November, 1987. [Grover, 1983]  Grover, M.D.,  "A  Pragmatic  Knowledge Acquisition Methodology,"  Proceedings from the International Joint Conference On  Al, Karlsruhe, West  Germany, 1983, P.436-438. [Hayes-Roth  et al. 1983]  Hayes-Roth, R.,  Building Expert Systems, Reading, MA: [Henderson,  1987]  Systems and  Henderson, H.C.,  Waterman, D.A., Addison-Wesley,  "Finding Synergy  and  Lenat,  1983.  between Decision Support  Expert Systems Research, Decision Sciences, Vol. 18,  Summer 1987, p.333-349.  D.B.,  No.3,  / 102 [Jacobson  et  a l , 1987]  Multi-Paradigm Knowledge,"  Jacobson,  Knowledge in  Chris,  Michael  Acquisition  Proceedings  of  the  J. Freiling,  Tool Second  for  "ASTEK  Complex  Knowledge  : A  Structured  Acquisition  for  Knowledge-Based Systems Workshop, AAAI, Banff, November, 1987. [Jenkins, 1980]  Jenkins,  A.M.,  "The  Prototype  Model  As  A  Management  Information Systems Design Technique", Discussion Paper #1663, School of Business, Indiana University, [Jenkins, 1983]  Jenkins, A.M.,  Development  1980.  "Prototyping : A  of Application  Systems,"  Business, Indiana University, April [Johnson, 1984]  John, P.E.,  Journal of Medicine Jorgensen,  A.H.,  Kuhlenkamp,  K.,  Methodology for the Design  Discussion Paper  "On and  Philosophy, Vol.8, 1983, the  of  1983.  "What Kind Of Expert Should and  #227, School  and  Psychology  Mathiassen,  L.  of  A  System Be?"  The  p.77-97. [Jorgensen,1984]  Prototyping",  in  Budde,  R.,  (Eds.), Approaches to Prototyping,  Berlin : Springer-Verlag, 1984. [Kidd,1987] Kidd, Allison L. (Ed.), Knowledge Acquisition for Expert Systems : A Practical Handbook, New [Kraushaar  et al., 1985]  Method  for  York : Plenum Press, 1987.  Kraushaar,  Applications  J.K.,  Development  and by  Shirland, L.E., End  Systems Specialists", Management Information  Users  "A and  Prototyping Information  Systems Quarterty, Vol. 9,  No. 3, September 1985, p. 189-197. [Naumann et al.,1982] Naumann, J.D., New  and  Jenkins, A.M.,  "Prototyping : The  Paradign for Systems Development," Management Information Systems  Quarterly, Vol. 6, No. 3, September 1982, p.29-44.  / 103 [Pau,1987]  Pau, L.F., "Prototyping, Validation  Based  Systems  Software",  and Maintenance  in the Third  Annual  of Knowledge  Expert  Systems in  Government Conference Proceedings, Washington, D.C.: The Computer Society of IEEE, 1987. [Smith, 1984]  Smith,  J.C., Liability  and Negligence, Toronto, Carswell  Legal  Publications, 1984. [Smith, 1988]  from a discussion the author had with Professor J.C. Smith and  Cal Deedman on August 19, 1988. [Sol]  Sol, H.G., "Prototyping : A Methodological Assessment", unpublished paper from Faculty of Economics, University of Groningen.  [Sviokla, 1986]  Sviokla, John J., Planpower, Xcon, and Mudman  Analysis  Into  Three  Commercial  Expert  Systems  In  : An In-Depth Use, Harvard  University, unpublished doctoral thesis, 1986. [Watkins et al., 1987] Watkins, P., D.E. O'Leary, "Knowledge Acquisition for Small  Scale  Expert Systems  from  Consulting  Experts : A  Approach," in Proceedings of the First European Acquisition  for Knowledge-Based  Systems,  Reading  Field  Study  Workshop on Knowledge University,  September,  1987. [Webb et al., 1966] Webb, Eugene J., Donald T. Campbell, Richard D. Schwartz, Lee  Sechrest, Unobtrusive Measure j _ Nonreactive research  in the Social  Sciences Chicago, Rand McNally, 1966. [Weick, 1968]  Weick, K.E., "Systemactic Observational Methods" in G. Lindzen and  E. Aronson (Eds.), The Handbook of Social Psychology (Vol. 2), Reading, Mass: Addison Wesley, 1968.  / 104 [Weiss  et al.,198?] Weiss, S.M. and CA. Kulikowski, A  Designing  Expert  Systems, Totowa,  New  Jersey,  Practical Guide to  Rowman  &  Allanheld  Publications, 1984. [Winston  et al., 1986] Winston, Philip  H., Karen  A. Prendergast,  The A l  Business The Commercial Uses of Al, Cambridge: The MIT Press, 1986. [Witten et al., 1987] Witten, Ian H., Bruce MacDonald, "Using Concept Learning for  Knowledge  Acquisition,"  in Proceedings  of the Second  Knowledge  Acquisition for Knowledge-Based Systems Workshop, AAAI, Banff, November, 1987. [Yin, 1984] Yin, R.K., Case Study Research, Design, and Methods, California : Sage Publications, 1984. [Young  et al., 1987] Young,  Techniques and Intermediate  R.M.,  J. Gammack,  "Role  Representations in Knowledge  Proceedings of the First European Workshop  on Knowledge  of Psychological Elicitation," in Acquisition for  Knowledge-Based Systems, Reading University, September, 1987.  APPENDIX A Questionnaire  105  1.  General  Characteristics  of the Expert  System  1  What i s t h e p r o b l e m domain o f t h e s y s t e m ?  2  What a r e t h e m a j o ^ c a p a b i l i t i e s explain  reasoning  modify  database  access  database  give  of t h e system?  advice  others 3  Who a r e t h e u s e r s o f t h e s y s t e m ? general  (non-expert)  users  experts 4  Does t h e  ES  access  an e x i s t i n g  database  or  i t s own  ES a n d  existing  database? how d o e s t h e database are 5  Could  there  interface  between t h e  work? just  one o r more t h a n  one d a t a b a s e ?  y o u g i v e me some b a c k g r o u n d on y o u r s e l f ?  why  i s there  the i n t e r e s t  or n e c e s s i t y t o b u i l d  the  ES? I s t h e r e any p r i o r  2.  relevant experience/training?  6  Does t h e s y s t e m use f o r w a r d  7  What m a c h i n e d o e s i t r u n on?  KA 1  o r b a c k w a r d c h a i n i n g a n d why?  Process What a r e t h e s o u r c e s  of information  : your  own e x p e r t i s e ,  textbooks, 2  I f y o u draw whether did  3  or other  I f you  from  experts?  your  own  something should  e x p e r t i s e , how  be  textbooks,  decide  included?  you c o n s u l t o t h e r p e o p l e used  d i d you  were  or r e s o u r c e s  they  the only  source  of  information? 4  I f you i n t e r a c t e d w i t h o t h e r e x p e r t s , c o u l d y o u t h e KA  describe  process?  how  l o n g was a s e s s i o n ?  what happened? did  y o u use automated  did  y o u use  during  interviewing aids?  "common s e n s e "  to  guide  your  questions  the i n t e r v i e w ?  how  d i d you r e c o r d t h e i n f o r m a t i o n ?  how  d i d you o r g a n i z e  you  use?  did  you t r y t o i l l i c i t  organization  information to f i t into  scheme o r v i c e  5  Can t h e KA p r o c e s s  6  What a r e  the  t h e i n f o r m a t i o n , what method d i d  be d i v i d e d  items  of  your  versa?  into  knowledge  stages  or phases?  extracted  from  the  expert? ES C o n t r a c t i o n 1  Could  you d e s c r i b e the p r o c e s s  how  d i d you s t a r t ?  did  t h e development  refinement  of b u i l d i n g  i n v o l v e KA a n d t h e n  t h e ES?  p r o t o t y p e and  o r d i d i t f o l l o w the" t r a d i t i o n a l  cycle  of  design-code-test-deliver? 2  How d i d y o u s t r u c t u r e t h e knowledge?  3  How  did  you  translate  knowledge  into  machine  representation? what  kind  script, 4  of r e p r e s e n t a t i o n  frame,  if-then  D i d y o u u s e any ES s h e l l  d i d you  r u l e s or  use :  schema,  others?  or t o o l s t o b u i l d  the system?  Outcomes 1  How do y o u j u d g e whether criteria  a system i s complete?  f o r judgement  :  speed accuracy quality user  of advice  a c c e p t a n c e of a d v i c e  others 2  Have m o d i f i c a t i o n s  been  made o r new  r u l e s added t o  the  system? how e a s y i s i t t o a d d , d e l e t e , long  or modify the r u l e s ?  3  How  d i d the e n t i r e development  4  How much d i d i t c o s t ?  5  How w o u l d y o u j u d g e p e r f o r m a n c e o f a system? observed does  improvement  i t replace  process  take?  on p r o d u c t i v i t y / a c c u r a c y  or support  how do y o u know t h e a d v i c e  human  effort  i t gives  i s sound  would you Do  you If  say t h e system  anticipate that  problems  outperforms  with user  h a p p e n s , what do you  plan  a  person  acceptance? to  do?  APPENDIX B Sample Tape Protocol  106  I'll  be  going  to M i c r o t e l  30th  o f November.  In  the morning around  started  fastest  --  i t will  be  They have having a group s e s s i o n where they are  o u t s i d e experts and they are meeting  Then he  10:30  testing.  Part  bringing  the  problems  What you're  that  he  had  I t ' s the  s a y i n g i s i t doesn't  ... occur  i f you won't abound  and  to use  might end up on the i s l a n d because they haven't  those.  (Well we  quite s e l f end  up  permanent  but  i s on the i s l a n d i n  screen  e v e r y t h i n g w i l l be s e n s i t i v e sorting  concept e n t i r e l y , i t ' s t h a t you are not going  with what the workload  having  and  the  now  i t ' s not  areas.  things  will  around here and h e ' l l do i t and  t h i s s t u f f out t o where i t ' s supposed to go.  right  being  done,  i t ' s being  We  talked  think island  want t o  i n the  end  we  will  February  o f the  there j u s t  to f i l t e r  these t h i n g s pass  then w i l l put  sensitive,  might end up doing i t about  and  there  is a  haven't  We're going to schedule  I  fact  we're  trying  to work a whole  any  Probably and y o u ' l l  s i l e n t screen.  through; w e ' l l send  For the purpose o f what we  W e ' l l change our  flow  , doesn't  have to go  .) are doing  t h i s , i s t h a t enough o r s h o u l d we  t h i n k a t t h i s p o i n t i t ' s going to have t o be going to work a t i t  see i t i n  i t through a s c r e e n e r , and  i t In a v a t u n l e s s i t ' s you know  through a screener  still  might  But they  i t a l l out.  and March, t h e r e won't be  c h a r t on how  I  because  as y e t .  We  -- there's a v a s t amount t h i n g s t h a t c o u l d be e l i m i n a t e d by h a v i n g the  screen r i g h t  we  ... I t ' s unscheduled  be  been  i t ' s h i s j o b that  p o s i t i o n b e i n g made a v a i l a b l e i f we want to use i t f o r t h a t . s a i d we  the  at M i c r o t e l L e a r n i n g Centre Room 211.  of  p a r t o f that course now.  on  left.  You  ....  know I t h i n k we're  Let us go out and have a l o o k a t i t j u s t f o r  a minute and p l a y with what we have.  With Steve he's  everybody.  He's  already  everybody. talking  I think  to John,  that  some r e a d i n g  too, i s that  c o r r e c t l y t o what was g o i n g on, o r  the enemies would have g o t i t .  working i n the o l d w e l l was that depth takes a l i t t l e depth o f the s c r e e n for  you're  the soup t o go through  get that over t h e r e It  depends l i k e ,  a little  foretell  b i t o f the pressure  off,  b i t o f the p r e s s u r e o f f ,  the data base areas,  t o check  t h e r e t a i n i n g , and  and c l e a n up t h a t .  their  system i s a l i t t l e  I'm i t , I'm t h e guy t h a t to keep e v e r y t h i n g  taking  We'll  d i f f e r e n t than ours.  I n where I am  s i g n s on and runs the queues and t h a t  together.  Well you don't r e a l l y  sort of tries  have a h e c k o f a l o t o f  time to t e s t -- y o u might p i c k up the phone a few times and r i p i t o u t s i d e maybe  do  i t that  way.  Your  main  function  e v e r y f i e l d s g e t s c l e a r , and you know t h a t t h i n g s are  there  pull,  for.  then  t h a t we  .)  Well,  I t says no q u e s t i o n  can view i t as s e r i o u s  sure  that  g e t done a n d t h a t ' s what you  (So t h a t ' s what you guys a r e here f o r . )  each o f us s i g n on as a  question  i s t o make  -- some k i n d  Doug and I, we  purging  out.  o f screening  No that  is You  would  have  whatever. And  still  t o take  courses  g e t t i n g on i n your there  i s an a n a l y z i n g  or b e i n g  preferred  trouble w i t h  i n someone e l s e ' s  j o b or  actual  area. , something t h a t ' s g i v i n g us the tough depends on t h e s i z e o f t h e o p e r a t i o n  and mutuals t h a t s t i l l  and  what's g o i n g under.  What's t h a t . The  process  that....  They have a -- guys t h a t j u s t It's  completely separate  among o t h e r s .  Where the i n i t i a l  view o f the o u t s i d e  guys t o o .  you're i n a d i f f e r e n t e n d o f the b u i l d i n g t e s t i n g i s done and we c a n s c r a t c h  ina  /  d i f f e r e n t area and all  they do;  They've  there's  three guys t h e r e J u s t answer the o u t s i d e guys.  i t ' s their fault  got  specific  an i n t e r a c t i v e r o l e  tasks,  specific  people  to  o p e r a t i n g c e n t r e t h e r e ' s t h r e e guys i n mutual, two North Vancouver.  A l l they do  Assuming. I t s o r t phone them, w i t h We  of  to  somebody who  more  i t on.  Now --  doesn't  to t r y and  size  on  In  guys i n r e a l . . . . two  an  guys i n  o f the  operation,  the  functions  do  calls.  on.  look at i t t h a t way  for  the  dispatch  That's a l l  s c r e e n i n g those r u n n i n g  don't have a s c r e e n  Okay, then s c r e e n  i s work o u t s i d e .  depends on  ....  It  i t ' s just  s a y i n g pay  i t ' s important  understand  can use  mentioning  i t on  the a r e a .  yes.  That  the system by a l l means.  You  have  i s a function  B r i n g s them a l o t  t h i s type o f f u n c t i o n i n A u s t r a l i a .  I can  say i t ' s You  can't be w i t h  that  f u n c t i o n s as near as  yeh  they're  organized.  Yeh,  i t ' s a late  functions.  The  split  up....split  i t screws up  That's You  go  one?  can  separate  Steve gets  except  the j o b  depends on  the  to  them.  really  Okay, does t h a t  to  on  phone y o u r s e l f and  them?  they  f u n c t i o n s t h a t are t h e r e are j u s t p e o p l e ,  s i z e of t h e i r o p e r a t i o n . And  up  We  can do  ahead and  give yourself h e l l .  g i v e us  Yeh.  a f r o n t - e n d p o r t i o n o r something to hang a n y t h i n g  i t t h a t way? work through  Is i t s t i l l the  valid  o l d system  or i s i t s t i l l  is trying  to  cross  on  a good t h i n g out  the  APPENDIX C Diary Form  107  1 Date  Time  Place  Initiator (person initiating  What  triggered  clarification,  new  the event)  (caused)  the  information  new idea, wish to improve  eventi obtained,  code etc.)  Person(s) Involved  Machine(s) Involved  Description  of what took place  (i.e.  what  type  of  information,  somebody's comments,  a  bug  in  e.g. the  wish  for  system, a  Duration of Activity  Objective  you wished to achieve before start of activity  Outcome  : what was accomplished, what was not  What do you plan  to do next?  Thoughts or Comments  APPENDIX D Deliverables  108  0  CjJL  108a  jcrr  VICTIM  L  108b  JtGuA£  jar-  Z  )  3 a j r v A ^ e .  108c  108d  . 11 '  V.  108e  1  I  (So  c-  V^Ov^  108f  A'* » )  Case:  IT  a.  loteationally Induced  1.  Physical  b.  Wrongful Act  2.  Mental  c.  False Report  3.  Sleep  d.  Insufficiently Serious  4.  , No Proof  e.  Plaintiff Only Harm  f.  Plaintiff Only Risk  5.  Held for  6.  Held for Defendant  PlaintiffXLI  Plaintiff Harm, Third Party Harm A Plaintiff Risk, Third Party Risk  1. Check t, II, and  III.  Plaintiff Harm, Third Party Risk 2. If check In IA then check C  Plaintiff Risk, Third Party Harm  3. If check In IB then check D. Third Party Only Death Third Parry Only Injury Third Party Only Risk a.  Child  o.  Spouse  P-  Parent  q.  Sibling  r.  Living Together  a.  Engaged  L  Rescuer  a.  Not Close Relationship  v.  .Witness  w.  , Nearby and Attended  X.  Imminent Accident  y.  Traces of Incident  yy.  Hospital Visit  _ _1  D.  Disfigurement  zz.  Insufficient Exposure  108 g  108h  108 j  Cat*: A.  JurlsdWtloa 1. Australia 2.  Canada  3.  Great Britain  4.  New Zealand  VlctU 1. Rescuer 2.  4. Holding 1. Plaintiff 2.  Defendant  Incidents 1. Deliberate Harmful Act 2.  Thin skull physical  3. Thin skull physical & psychological Aggravated a.  medical natural deterioration  b.  medical negligent treatment  c.  medical non-negligent treatment  d. criminal secondary acts t.  Negligent Harmful Act  non-criminal secondary acts P's own secondary act  D.  Activity 1. Weapons 2.  S. Nothing unusual  Machinery  3. Explosives 4.  Fire  5.  Inflammable substances  F.  Nature of Remoteness  6. Toxic substances 7.  Electricity  8.  Radioactivity  9.  Animals  -o «  c « u  X M  10. Transportation: a. car b. motorcycle c. truck d. van  a.  Victim  b.  Method  c.  Damages  e. bus f.  tractor unit  g. trailer h. tractor trailer L  snowmobile  j.  railway engine  k. railway car I.  train  m. vessel n. airplane o. animal power 11. Not Dangerous  108  1  JOB  PROFILE  TOOLS t EQUIPMENT The tools and equipment available to the TEST DESK PERSON include:  1.0  Touch Tone Telephone Set ( A l l exchanges)  2.0  CRT TERMINAL ( a l l exchanges) Locally based (IBM PC workstation) Tool for addressing 4 TEL, TAS, CRIS, MAIS, FMS, SOUL and other MIS information or other Telco Depts. Other Software DOS, Crosstalk, Desk View, Pascal subrouting for accessing 4 TEL SAC UNITS through MUX  3.0  4 TEL - 4 WAY TELEPHONE TESTING SYSTEM ( A l l exchanges) Locally (service centre) Based Testing System Business and Residence  4.0  TAS - Trouble Administration System - ( A l l exchanges) MIS based" Holds and routes - Trouble reports - business and residence  5.0  CRIS - Customer Records Information System ( A l l exchanges) MIS based Holds Residential - Service Orders - Customer Data (Name, Address Etc.) - F a c i l i t y Data (Cable pair, Exchange Hardware I.D. Etc.) - B i l l i n g Data - Comments Holds Busines - Customer Data - B i l l i n g Data - Comments 108m  JOB PROFILE  6.0  SOUL - SERVICE ORDER UPDATE AND LOCATE ( A l l HIS based " Holds - Business Orders - Leasewire Orders  7.0  MAIS - MECHANIZED ASSIGNMENT INFORMATION SYSTEM (Not a l l Exchanges) Holds f a c i l i t y information (New York Based) Residence and Business  8.0  FMS - FACILITY MANAGEMENT SYSTEM (Not a l l Exchanges-Yet) Holds Detailed Facilty Data (MIS Based) Residence and Business  108m  Exchanges)  fig.  13  ,  TROUBLE ANALYSIS CENTER CURRENT  (Not lor OvwniQM S—«•»)  fig.  13  TROUBLE ANALYSIS FUTURE  CENTER  fig.  15  108p  f  l  ig  a y  AGENDA TOR TEST DESK COURSE TASK ANALYSIS SESSION 30 NOV 87 4 DEC 87  '  30 NOV 1.  Introductions  2.  Purpoit  3.  Information about  4.  Q u e s t i o n s & comments  5.  S h o r t Break  -  6.  Update  Test  LUNCH  of  Dan Brophy  session -  1981  11:45  -  -  Gord Bacon  the  p r o c e s s and outcomes -  9:15 Desk Task A n a l y s i s  Document  12:45  7.  C o n t i n u e w i t h 1981  8.  S h o r t Break  9.  C o n t i n u e w i t h 1981  10.  Summary 4 c l e a n u p -  11.  End S e s s i o n f o r  -  TA Update  2:00 TA Update 3:45  day -  4:00  1 DEC 1.  Review of  2.  C o n t i n u e w i t h 1981  3.  S h o r t Break  4.  C o n t i n u e w i t h 1981  LUNCH  11:45  last  -  session t  -  questions  TA Update  9:15 TA Update  12:45  5.  C o n t i n u e w i t h 1981  6.  S h o r t Break  7.  Complete  8.  Summary t  9.  End S e s s i o n f o r  -  1981  TA Update  2:00 TA Update  cleanup  108q  -  day -  3:45 4:00  -  8:00  Jeff  Howell  2 DEC 1.  Review o f  2.  Update  3.  Short  4.  C o n t i n u e w i t h 1986  LUNCH  last  1986 Break  11:45  -  session t  questions -  8:00  4TEL Task A n a l y s i s -  9:15 TA Update  12:45  5.  C o n t i n u e w i t h 1986  6.  Short  7.  Complete  8.  Summary t  9.  End S e s s i o n f o r  Break  -  TA Update  2:00  1986  TA Update  cleanup -  3:45  day -  4:00  3 DEC 1.  Review o f  last  session 4 questions -  2.  Integrate  1981  and 1986  3.  Short Break  rA.  -  8:00  Task A n a l y s i s  9:15  C o n t i n u e TA  integration  5.  Complete  integration  6.  Short  7.  Review o n g o i n g c o u r s e and ES d e v e l o p m e n t and d i s c u s s any o t h e r o u t s t a n d i n g i t e m s ,  8.  Summary « c l e a n u p -  9.  End S e s s i o n f o r  TA  Break  -  2:00  day -  3:45 4:00  108q  p r o c e s s f o r upcoming i s s u e s or q u e s t i o n s  year,  DEC  -  1.  Review of week and l a s t  2.  Validation  3.  Short  4.  Continue v a l i d a t i o n  LUNCH  Break  11:35  -  4 sign-off -  of  session 4 questions revised Test  -  8:00  Desk Task  9:15 4  sign-off  4  sign-off  12:55  5.  Complete v a l i d a t i o n  6.  Summary of weeks  7.  End Task A n a l y s i s S e s s i o n -  activities,  108q  questions, 2:30  comments  Analys  fig.  17  Fault Analysis Flow Diagram  Part Five  ^  CWTO  ^  OM.TCM TUT  f^CU»TOtl C A U ^  COUPAflC CUSTOMS".  Ae<x*=D» WTTH DATABASE  ROUTE TO CO.  fOUTt TO MSPATCM  CAU.CUSTCAS-" OOlCOPAfCG* TESTwrm CUSTOMER  AOUTi TO? OAT FOUiOWU*  108r  Q  DATA  J  PERFORM MOMTCM TEST-WAT C O U PUUEO  fig.  18  108s  108s  108s  OR OP OP' THEN  there IS a n a v a l a n c h e there are h i g h winds ! 0 0 im fir t h e r e ARE o t h e r A c t s o f G o d c u t of c o m p a n y - f a c t o r s e j e c t i n g  plant  cond11 i ons  RULE IF OR OR OR THEM  -for  d e t e r m i n i n g A c t s o f Man e f f e c t i n g p l a n t conditions there IS h e a v y c o n s t r u c t i o n i n t h e area t h e r e IS a l a r g e a u t o a c c i d e n t there IS s a b o t a g e t h e r e ARE o t h e r A c t s o-f Man o u t o-f c o m p a n y f a c t o r s e - f - f e c t i n g p l a n t conditions  RULE IF THEN  -for  d e t e r m i n i n g h o l i d a y e-f-fecting p l a n t conditions it IS a h o l i d a y o u t o-f c o m p a n y f a c t o r s e - f - f e c t i n g p l a n t conditions  RULE IF THEN ELSE  -for  determining i-f d i s p l a y s c r e e n TAS s c r e e n i s l a b e l e d c u s t o m e r trouble i s customer reported t r o u b l e i s company reported  RULE IF OR OR OP CR OR OF OR CR OR OR OR THEN  for  trouble determining if t r o u b l e t y p e i i s t e d IS t r o u b l e t y p e 1 i s t s d IS t r o u b l e t y p e 1i s t e d IS t r o u b l e t y p e 1 i s t e d IS t r o u b l e t y p e 1 i s t e d IS IS troub1e type l i s t e d t r pi ib 2 e t - B ? 1 i s t e d IS t r o u b l e t y p e 1 i s t e d IS t r o i t l e t v p e 1 i s t e d 13 t r OL b 1 s t y p e i i s t e d IS t r nu t y p e 1 i s t e d IStroubi e type 1isted IS t.-oubl e t y p e r e p o r t e d l  RULE I~ AND AND AND AND AND AND AND AND AND AND AND AND AND THEN ELSE  ior  p r o o f r e a d i n g t h e TAS t r o u b l e report trou-le type i s correct OS NGS f i e l d IS complete partv line f i e l d IS complete time reported IS complete name f i e l d IS complete address f i e l d IS complete city field IS complete who r e p o r t e d t h e t r o u b l e f i e l d IS c o m p l e t e CPE f i e l d IS complete a l l phones f i e l d IS complete critical r e a s o n IS complete all calls field IS complete cable field IS complete pair field IS complete all needed f i e l d s a r e completed f->ere i s a n e r r o r i n the TAS r e p o r t  RULE IF AND  for  proof reading b u s i n e s s customer all needed f i e l d s a r e c o m p l e t e d l i - . e number f i e l d IS complete  108t  i s customer reported  or  company  type is correct NDT CCO CBC RWN 00L N8E CH C&H CUTC FHY 3 CUSTOM C A L L Oata s valId  TAS t r o u b l e  report  reported  THEN  line  ELSE  line  station or  information  station  info  END  108t  ma>  be  is  present  needed  from  c . stonier  •ecorc'  WASTES SYSTEM DOMAIN PRIMARY SlUOCC  RAW 3EWACC UFT  SCREENING  RAW SEWAGE  T  CH.ORWE CONTACT TANK  WASTE SLUOCC  EmUENT  • FILTRATE SLUOCC DISPOSAL  FIGURE ZO-  Conventional Treatment WASTES Syatem Domain  108u  Plant  L a y o u t and  FIGURE 11  - RACKS.KB L o g i c  108v  Structure  FILES O N DISK LI FT. K B • (arm  iiiiiiiiin  f  M  I  RACKS.KB  7^—  LOADED IN M E M O R Y  GRIT.KB  MICRO.KB WASTE. K B I L.  FLOAT. K B RBC.KB ODOR.KB  FICURE 2Z -  WASTES F i l e  108x  Structure  FIGURE 2-1.-  108w  GRIT.KB L o g i c  Structure  


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items