Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Developing and testing an effective interactive voice response (IVR) system for the Workers’ Compensation… Mehra, Gaurav 1999

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-ubc_1999-0310.pdf [ 13.72MB ]
Metadata
JSON: 831-1.0089021.json
JSON-LD: 831-1.0089021-ld.json
RDF/XML (Pretty): 831-1.0089021-rdf.xml
RDF/JSON: 831-1.0089021-rdf.json
Turtle: 831-1.0089021-turtle.txt
N-Triples: 831-1.0089021-rdf-ntriples.txt
Original Record: 831-1.0089021-source.json
Full Text
831-1.0089021-fulltext.txt
Citation
831-1.0089021.ris

Full Text

DEVELOPING AND TESTING AN EFFECTIVE INTERACTIVE VOICE RESPONSE (IVR) SYSTEM FOR THE WORKERS' COMPENSATION BOARD OF BRITISH COLUMBIA By G A U R A V M E H R A M B A (Marketing), Saint Mary's University, Halifax, Nova Scotia, 1996 A THESIS SUBMITTED IN PARTIAL F U L F I L L M E N T OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE (Business Administration) In THE F A C U L T Y OF G R A D U A T E STUDIES (Department of Commerce and Business Administration) We accept this thesis as conforming to the required standard THE UNIVERSITY OF BRITISH C O L U M B I A April 1999 © Gaurav Mehra, 1999 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission. The University of British Columbia Vancouver, Canada Date DE-6 (2/88) Abstract This thesis was the result of a study conducted for the call-centre at the Workers' Compensation Board of British Columbia (WCB). The management at WCB wanted to understand the nature and pattern of calls at their newly opened call-centre. The purpose of this was to provide an efficient customer service while streamlining the flow of calls coming to the call-centre. An extensive data collection exercise was undertaken at the call-centre and two other units of WCB with which the call-centre interacts. The data analysis revealed that a high proportion of calls were related to transfers to these departments. There were also calls related to routine inquiries on claim payment cheques and forms that could potentially be handled by a well designed IVR system. Based on this understanding the development of an effective IVR system was proposed to address the problems that were discovered through documenting the nature and pattern of calls. An extensive review of literature was undertaken to design a new system according to the standard industry guidelines suggested by the best practices and customized to WCB's business needs. Two alternate scripts were developed after analysing the source and purpose of calls to WCB. One was 'person specific' and the other was 'task specific'. The two scripts were tested on students at WCB through a computer-based IVR simulation. The results of the student survey provided evidence that introducing additional options and use of simple and clear instructions in the new scripts could potentially in fact address the problems discovered in the study and they were preferred over the existing WCB script. The IVR simulation is reconfigurable and can be used in future studies to gather further evidence in support of the results obtained in this thesis as well as refine scripts before putting them in a production mode. ii Table of Contents Abstract i i Table of Contents i i i List of Tables v List of Figures vi Acknowledgements viii CHAPTER 1. INTRODUCTION AND OVERVIEW OF RESEARCH 1 1.1 Background and Motivation 1 1.2 Call Centre Technology 1 1.3 Importance of IVR research 2 1.4 Background and Motivation for Current Research 3 CHAPTER 2. LITERATURE REVIEW 7 2.1 Previous Research on IVR 7 2.1.1 Huguenard(1993) [2] 7 2.1.2 Bradley et al. (1996) [3] 10 2.1.3 DePalma & Goodman (1995) [4] 12 2.1.4 Theory of Reasoned Action and its Variants 15 2.2 Implications for Current Research 16 CHAPTER 3. UNDERSTANDING THE NATURE AND TYPE OF CALLS AT THE WORKERS' COMPENSATION BOARD 17 3.1 Introduction 17 3.2 Pre-Test 17 3.3 Data Collection at the Call-Centre 21 3.4 Data Collection at Entitlement and Case Management 21 3.5 Problems encountered during Research 24 3.6 Results of data analysis at the Call Centre, Case Management, and Entitlement.. 25 CHAPTER 4. SCRIPT DEVELOPMENT 27 4.1 Evaluation of WCB Script against the Best Practices 27 4.2 Existing and Proposed Call Centre Architecture 29 4.3 Alternate IVR Scripts 31 4.3.1 Addressing the problems through the new scripts 32 4.4 Laboratory v/s Field 34 4.5 Computer-based simulation for the new scripts 35 4.6 Student Questionnaire 37 4.6.1 External Validity Problems 38 4.6.2 Testing the IVR Simulation 39 4.6.3 Internal Validity Problems 40 CHAPTER 5: STATISTICAL ANALYSIS OF THE QUESTIONNAIRE 44 5.1 Summary Statistics and Statistical tests 44 5.2 The Experimental Model and Hypothesis 46 5.3 Conclusions from the Student Questionnaire 61 CHAPTER 6. CONCLUSIONS AND RECOMMENDATIONS 64 6.1 Concluding comments 64 6.2 Limitations of the study 65 6.3 Suggestions for future research 66 in Bibliography Appendix I ^ Appendix II ^ Appendix III • ^3 Appendix IV ^ Appendix V " Appendix V I iv List of Tables Table 1. Breakdown of different types of calls 18 Table 2. Evaluation of the current IVR system against the Best Industry Practices 28 Table 3. Objectives and Goals for an IVR System for WCB 30 Table 4. Laboratory vs. Field Method 36 Table 5. Design of experiment for Student Survey 42 Table 6. Result of student responses on IVR vs. Operator 62 Table I L L Calls suitable for an IVR 84 v List of Figures Figure 1. Screen-shot of user interface for the data collection tool at the call-centre 20 Figure 2. Screen-shot of user interface for the data collection tool at Entitlement and Case Management 23 Figure 3. Existing Call Centre Architecture 31 Figure 4. Proposed Call Centre Architecture 32 Figure 5. Screenshot of main user interface for the computer-based IVR simulation 37 Figure 6. Screenshot of configuration interface for the computer-based IVR simulation 38 Figure 7. Plot of mean intervals using the Fisher's Least Significant Difference 60 Figure II. 1 Percentage of time spent by CSRs on various activities during phone-times 72 Figure II.2 Calls by people by time of the day 73 Figure II.3 Types of calls by time of the day 74 Figure II.4 'Transfers To' by time of the day 75 Figure II.5 Calls by people by day of the week 76 Figure II.6 Type of calls by day of the week 77 Figure II.7 'Transfers To' by day of the week 78 Figure II. 8 Breakdown of Transfers 79 Figure II.9 Average length of Transfers 80 Figure 11.10 Analysis of'Wrap-up' 81 Figure II. 11 Service and Wrap-up times 82 Figure 11.12 Average Wrap-up time .83 Figure III. 1 Incoming calls by categories of people 86 Figure III.2 Breakdown of incoming calls by source 87 vi Figure III.3 Average time spent on calls with various people 88 Figure III.4 Average time spent on different types of calls with different people 88 . Figure III.5 Incoming calls by categories of people 89 Figure III.6 Incoming calls by source 90 Figure III.7 Average time spent on calls with various categories of people 91 Figure III.8 Average time spent on various types of calls with different people 92 Figure V . 1 IVR Simulator user interface .' 100 Figure V.2 File Menu 101 Figure V.3 Simulation 101 Figure V.4 Configuration Dialogue 102 Figure V.5 Output Configuration 103 Figure V.6 Input Configuration 104 Figure V.7 Error Handling 105 vii ACKNOWLEDGEMENTS I would like to acknowledge the assistance of Dr. Marty Puterman, Stephen Jones, Paul Hiom, and Lauren Gray at the Centre for Operations Excellence (COE) at the University of British Columbia for the financial assistance they provided for this study. I would forever remain indebted to my advisor Dr. Carson Woo who provided every possible resource in order for me to complete this work. His suggestions during weekly meetings at COE greatly improved the quality of not only the WCB project but also several others. My co-advisor Dr. Izak Benbasat who guided the design of the student survey and provided the research instrument developed by him for that purpose. His insightful comments during the 'Research Methods' course helped put this study on a firm ground. I would like to thank research assistants Katy, Jeff, Anoop, Art, and Leo who spent countless hours at the Workers Compensation Board of British Columbia gathering data on the phone calls at the Call Centre, Entitlement, and Case Management units. Marie-Ann Anderson, Jason, and Dorit for helping me analyze the enormous data on calls. Arash who due to his interest in my work agreed to develop the computer-based IVR simulation despite his last semester at UBC. Ednoi whose voice added a touch of professionalism to the sound files that she recorded for the IVR simulation. Colleen who frantically gathered students to participate in the IVR simulation, a day before my departure to Montreal. Lana Burnstad and Carole Michelucci at C N for facilitating my Vancouver trips in order for me to finish my study. My Manager in IT, Doug Wilson, who let me take off for a week to Vancouver to run the laboratory simulation, and also promised to take me out to dinner in one of the finest restaurants of Montreal should I make it to the May graduation deadline. My Director in Marketing, Keith Reardon, who sent me back from Toronto a day before the deadline in order to ensure that I have the thesis in the library by Friday, April 30 t h, 5:00 pm. Special thanks to my best friend Rafael who while being in Spain, was in constant touch with me via phone and email guiding the statistical analysis of the student survey. Numerous other friends, both in Vancouver and Montreal, who provided me encouragement and moral support throughout this study and at a time when I was going through perhaps one of the most turbulent phases in my personal life. Finally, I would like to thank my Mom and Dad. Especially my Dad who made me dread my phone calls home as his first question always used to be not "How my job at C N is going?" but "How much progress I have made on my thesis?" viii CHAPTER 1. INTRODUCTION AND OVERVIEW OF RESEARCH 1.1 Background and Motivation Call-centres are increasingly becoming common in many large and even medium and small-scale organizations and businesses. The growing importance of the call-centres for organizations is due to the fact that it assists in conducting the business of the organization in a smooth and efficient manner and also provides a high level of customer service. The call centre agent acts as the primary interface between the customer and organization and is thus an important element in the value chain. Establishing a call-centre in an organization is an important decision and can have a wide-ranging impact on the information flows and business processes within an organization. The call-centre also interacts with other departments in an organization and it is crucial to establish a clear role for the call-centre and its inter-relationship with other departments. 1.2 Call Centre Technology The technology behind the call-centre is fairly mature and there are various tools available to establish a leading-edge call centre. In order to provide the highest level of customer service all the components of a call centre are required, however, an organization may decide to choose only a few components due to resource constraints. The basic component of a call centre is an Automatic Call Distributor (ACD) which routes incoming calls to a free agent and is similar to a Public Branch Exchange (PBX). The number of incoming trunks into the A C D varies from call-centre to call-centre and depends on the estimated demand and size of the business. Most call-centres usually incorporate an Interactive Voice response (IVR) system which is an automated inquiry line that interacts with corporate databases to provide up-to-date information to callers. Another component of a call-centre is Computer-Telephony Integration (CTI) that co-1 ordinates information entered in the telephone by the caller with the information on the agent's computer. However, CTI technology is expensive and not all call-centres possess that, though because of its many advantages in providing a high level of customer service it is becoming increasingly important in call-centres. The call centre agents are equipped with a telephone and a computer terminal, which is connected with the company's databases. Information is retrieved from and entered into the databases by the agents. 1.3 Importance of IVR research There are many successful examples where businesses have improved their operations and increased customer satisfaction through the implementation of an IVR system. An IVR system provides users with round-the-clock access to variable rather than static information [1]. The use of a well-designed IVR system can help in reducing the number of calls that are to be answered by the Customer Service Representatives (CSR). At the same time, it increases customer satisfaction, as the customers are not restricted to the hours of operation of the business if they require routine information. They can call in 24 hours a day, seven days a week. A number of organizations today depend heavily on the IVR system for the successful operation of their business. Some of these are airline companies, banks and insurance companies, courier agencies, educational institutions, etc. Some of the traditional applications of the IVR are providing account balances, checking status of a package or claim, requesting or providing information, or allowing users to register for various activities. New and innovative applications are constantly being developed to streamline business processes. However, the selection of the right IVR system is important. It is essential to have caller-friendly and easy-to-understand voice scripts, prompts, and logical paths between 2 responses [1]. Only after a careful selection and planning of an IVR system the expected benefits will be realised. It is estimated that 80% of the costs associated with a call-centre are related to employees. Given the high personnel costs it is essential that an optimal number of call centre staff be maintained and their time optimally utilised. However, even with a good IVR system, it is usually assumed that the caller will listen to all the choices before making a selection. This, however, is not always the case. Many callers do not wait to listen to all the choices. They simply press '0 ' to talk to a live person. This is based on the assumption that pressing '0' will in fact provide them this opportunity. Due to this tendency in the user behaviour many IVR scripts are modified to give an option other than pressing '0' to users to speak to a call-centre agent. Some others require users to press '0 ' twice before connecting them to a live operator. Research conducted by some businesses has also indicated that just switching the order of choices can make a big difference in encouraging or discouraging users from choosing certain options. There is a need for research to be conducted on the effectiveness of various IVR scripts in encouraging its use by the callers. There are standard industry guidelines that assist in designing a good IVR script and avoiding some of the pitfalls. However, these standard guidelines have to be customized to suit the particular needs of the business. 1.4 Background and Motivation for Current Research This research was conducted on a real-life call-centre at the Workers Compensation Board (WCB) of British Columbia. The Workers Compensation Board of British Columbia deals with the assessment, entitlement, and adjudication of claims related to 3 injuries incurred by workers with respect to job-related duties. WCB opened a centralized call centre in April 1997 for answering all calls related to claims from the lower mainland area. The decision to implement a call-centre was facilitated by a breakthrough technological innovation implemented earlier by the WCB called the E-File (Electronic-File). For years WCB had been handling paper-based claims which required extensive manual tasks. Starting early 1997, WCB has been gradually moving towards an electronic system called the E-File. Any document related to a particular claim (doctor's report, employer report, employee form, etc.) is scanned by a central scanning and indexing department and is added to the claim file which becomes electronically available to authorized claim officers. The customer service representatives (CSR) in the call-centre access any E-file claim on their computers and are thus able to provide information to the callers. The introduction of the E-File and the opening of the Call Centre have had a profound impact on the employees. The implementation of these new information technologies has significantly affected their roles and responsibilities. This research was prompted by management's desire to understand the nature and pattern of calls received by the call-centre, and streamline call-flows through an effective IVR system based on the findings of the study. The research study was conducted in two phases. During the first phase extensive data was collected in the call centre to analyse the nature and pattern of calls coming into the call-centre. The first phase was also devoted to an understanding of the call flows in the call-centre. The findings of the first phase revealed that 20% of the calls were related to transfers to other departments within WCB, specifically Entitlement and Case Management, and approximately 7% were related to cheque inquiries. Based on this analysis it was suggested that an efficiently designed IVR system could potentially deal with some of these calls and reduce the volume of incoming calls to the 4 call-centre. An efficient IVR system is also necessary to provide a high level of customer service as it can provide information 24 hours a day, 7 days a week. The second phase of the study involved gathering data in two other units of WCB: Entitlement and Case Management. These units handle claims that are more complex or claims are passed on to these units from the CSR after a certain time is elapsed. Gathering data in these two units was also essential to complete the call-flow picture for the call-centre. After a thorough understanding of all the call flows and business processes and extensive review of best practices, two alternate IVR scripts were designed. The important research question that was tested in this study was whether the suggested guidelines for designing effective IVR systems, that were incorporated in the design of alternate scripts, would actually address some of the problems. The new IVR scripts were designed to be effective from the perspective of the organization, i.e. WCB. However, they were tested on potential users as the scripts would have been considered to be effective only if it could be demonstrated that they would actually be used by the callers. An effective IVR system would result in cost savings, and labour savings through improved productivity to the WCB. This would happen as more calls would get closed in an effective IVR without the need for a live agent. However these expected savings will not be realised unless the potential callers actually use the system. Hence, the criteria that will be measured in this thesis is the extent of usability of the new IVR scripts as it has an impact on the effectiveness of the system. The two alternate scripts consisted of a 'role specific' and 'task-specific' script that were designed based on an understanding of the nature and pattern of calls at WCB. This led to the exploration of a second research question that, whether one of the two types of scripts would be preferred over the other or there would be no difference between them. However, this question was tested in a situation where there was no clear hierarchy to 5 order the scripts. This means that the result from this test cannot be generalized to IVR system of companies where the two types (role or task specific) have a vastly different and distinct order of choices. These two alternate scripts along with the original WCB script were tested on a group of students through a computer-based simulation. The students were required to respond to a questionnaire during their interaction with the three different IVR simulations. The data that was gathered was analysed using a statistical software package called SPSS. The results of the statistical tests provided evidence that the two new alternate scripts were generally preferred by the group of students over the existing WCB script. However, between the two new scripts there was no statistical evidence to show that one was preferred over the other except for one task. The next chapter reviews relevant previous research conducted on voice response systems. Chapter 3 describes the process of understanding the nature and pattern of calls at WCB. It also summarizes the key findings from the data collection exercise undertaken at WCB. Chapter 4 describes the process of designing two alternate scripts based on the results of data collection and the best practices in the industry. It also contains the methodology for evaluating the three different IVR systems using computer-based simulation and student subjects. Chapter 5 summarizes and discusses the results of the student questionnaire. The final chapter of this thesis contains concluding remarks, limitations of the study and provides suggestions for future research. \ 6 CHAPTER 2. LITERATURE REVIEW 2.1 Previous Research on IVR There has not been much academic research done on the use of Interactive Voice Response technology. The most notable study done in this field was Brian Huguenard Ph.D. thesis at Carnegie Mellon University titled "Working memory failure in human -computer interaction: Modeling and testing simultaneous demands for information storage and processing." Various consulting firms and independent IVR vendors have carried out a number of studies on the effective design and implementation of IVR systems. However, these studies have mostly been done in the U.S. The current thesis was conducted in a Canadian context by utilising a Canadian workers' insurance organization and the findings are not expected to be any different. However, there might be a chance that it might be different. This chapter reviews some of these studies and their main findings. In addition, section 2.1.4 reviews some of the literature based on the Theory of Reasoned Action and its variants. This would provide guidelines for testing the effectiveness of scripts developed in this study by predicting their likeliness of being used by callers. 2.1.1 Huguenard (1993) [2] Brian Huguenard's study was related to the design of better human-computer interfaces through the use of a computational model, which would reduce the need for experimentation to test the impact of alternative designs on user performance. He developed and tested a computational model (PBI USER) for predicting Working Memory (WM) error rates in a particular type of human-computer interaction: phone based interaction. A working version of PBI USER was developed and used to generate predictions about the impact of three factors on W M failure: structure of the interface menu hierarchy. This refers to the various steps users would have to navigate through in order to accomplish a particular task. 7 individual differences in cognitive abilities, which would affect the W M capacity. People with superior cognitive abilities would be expected to have a higher W M capacity and hence lower W M errors rate. - and task characteristics, which refers to task format and number of tasks a user is expected to perform. A complex task for which a user would have to input information or if a user wishes to perform a series of task would affect the W M errors rate. The characteristic that is most relevant for the current research is the structure of the interface menu hierarchy. The research was based on the theory of W M (Just and Carpenter, 1992) that proposes that the storage and processing of information generate demands for W M resources. The predictions of the theory were tested on human subjects by requiring them to interact with a phone-based registration system of a University (Tele-registration). The empirical results provided strong evidence for the importance of storage demands, and moderate evidence for the importance of processing demands as predictor of W M failure in PBI. The results also provided evidence for the importance of individual memory differences in W M capacity as a predictor of W M failure in PBI. Based on the results of this study, Brian suggested several guidelines for the effective design of phone based interfaces. The results of Brian's study have an important bearing on the current research and his design guidelines as well as the best practices in the industry reviewed in the following sections were utilized in the development of an effective IVR script for the WCB. The research conducted by Huguenard was based entirely on interaction with an automated phone-based system without live agent support. The nature of the current research is slightly different as the option to speak to a CSR at WCB is available during regular business hours, i.e., from 7 am to 7 pm Monday through Friday. However, 8 the study is still helpful for the part where a client interacts with an IVR before speaking to a live agent or exclusively interacts with an IVR system outside of business hours. 2.1.1.1 Structure of the Menu Hierarchy Huguenard's thesis reviewed the literature on the structure of the menu hierarchy in terms of the depth v/s breadth of the menu tree. Menu structures that emphasize depth have lesser number of choices at each menu but the number of menus the user must traverse increases. On the other hand, in broad menus the number of traversed menus decrease but there is an increase in the number of choices at each menu. Earlier research in the design of auditory menus suggest no more than three menus at any given level because of working memory limitations, thus forcing a deep and narrow menu structure. However, current theories of the role of W M favour broad menu structures as they postulate that as depth increases (and breadth decreases) the option labels especially at the top-level menus will tend to become more ambiguous. This will degrade subject's understanding of other option labels, which will produce W M errors and inhibit performance. 2.1.1.2 Results and Discussion Huguenard's study measured two types of errors generated by PBI USER: 1) rehearsal errors in which an item of task information was forgotten, and 2) navigational errors, in which an incorrect option choice was made while traversing the menu structure. Only the results of navigational errors are discussed here as they are of interest to the current study. Navigational errors were generated when the user chose an incorrect option. This happens when the option presents only a partial match to the desired goal and the user chooses that option based on the partial match even though the correct option (which has a superior match to the goal) has not yet been presented. This points towards the need for simple and easy to understand instructions in menu options. 9 The results of the study suggested that the impact of the menu structure is mainly on navigational errors. PBI-DEEP had higher navigational error rate than PBI-BROAD. This effect was because of the insufficient long-term memory knowledge for disambiguating the option labels in PBI-DEEP. Based on these findings, the current research uses the design guidelines of a broad menu structure and avoids the use of ambiguous instructions at each level. 2.1.2 Bradley et al. (1996) [3] A major study on IVR systems in American companies was done by the Enterprise Integration Group (EIG), a consulting company in the U.S. The principal author of the report, that is available in the market for approximately U.S. $3,000, was Gary Bradley who was assisted by a team of consultants from the EIG. The team developed a methodology to rate the IVR systems by compiling a list of over 3000 telephone numbers in the Directory selected at random for businesses they thought would likely have an IVR system. Researchers were trained in what features to look for and how to evaluate the IVR systems. They were then asked to call these numbers and evaluate the systems based on the pre-defined criteria. Results for each survey were closely monitored and reviewed by an IVR specialist. Each surveyor evaluated a variety of IVRs across multiple industries. During the actual evaluation process each surveyor called the telephone numbers assigned to them and recorded the total seconds required for the initial greeting and the first menu, navigated all open menus in the application recording each individual option and the key(s) required to select it. They also accessed restricted menus where they could get hold of the required account numbers and codes through the help of friends, relatives, and team members. They also made written notes of their observations and posted their evaluations to a summary sheet. 10 This study utilized a standard scoring mechanism developed by an expert at EIG and divided the observations in four categories with a maximum attainable of 25 points adding up to a total of 100 points for an IVR system. These four categories were Voice Quality, Information Delivery, User Friendliness, and Ease of Operation. A final count of 409 completed surveys was obtained as about 60% of the 3000 numbers enlisted turned out to be live answer, not IVR. Another 25% were automated attendants which just performs basic operations like directory assistance and were not targeted by the study. These figures reveal that the IVR technology is untapped by a lot of businesses and there are tremendous opportunities for cost savings and process improvements by utilizing this technology. This also points towards the need for more research into this area to help businesses realize benefits from implementing a call centre. The study came up with five general findings: 1. Most IVR applications lack a serious customer-oriented design. They are not easy to use and lack desirable features (such as letting the callers know the anticipated wait-time). 2. A large number of IVR applications offer an excessive number of menu options. This might actually inhibit the use of the IVR system as customers get frustrated with too many options. 3. State-of-the-art IVR technology (that includes features like letting the users know the wait time, automatically faxing information, etc.) is not being aggressively deployed. 4. A large number of businesses are not utilizing the CTI technology. WCB is a case in point. 5. And as a result of the preceding two points, enterprises are missing opportunities for proactive relationship management. Based on these findings the research team developed a methodology for designing and implementing an IVR system in organizations. This process is a disciplined multi-step 11 approach for either re-engineering existing applications or creating new applications. The EIG team developed this methodology iteratively, seeking input from a wide variety of subject matter experts. The suggested guidelines for a good IVR system developed by the team consist of 12 steps. These steps are mostly related to simplified design that restricts menu choices to no more than three, and use of simple and straightforward language. These suggestions were tested in this study by requiring student subjects to respond to questions on 'ease of use' and 'ease of understanding' on the student questionnaire, during their interaction with the IVR simulation developed for WCB. In their steps, the EIG team also suggests testing and refining IVR prototypes, before putting it in a production mode. Due to copyright restrictions, details on the 12 steps cannot be disclosed in this thesis. In addition to the methodology the team also suggested specific application design and menu design criteria. This included recommended introduction in seconds, commands for return and cancel, the choice of delimiter, among many others. These suggestions were reviewed and incorporated in the best practices which was utilized for the design of the IVR script for WCB in this study. 2.1.3 D e P a l m a & G o o d m a n (1995) [4] A major research on the use of Automated Response Systems (ARS) was undertaken by David DePalma and John Goodman of the Technical Assistance Research Programs (TARP), Inc. in 1995. The objective of the study were to : 1) define success measures for the use of ARS; 2) discover which ARS applications were most successful and which were not; and 3) identify any credible empirical research measuring the impact on customer satisfaction, call centre productivity, and cost. 12 The researchers sent out mail-in surveys to companies across a wide variety of industries, which included utilities, hotel chains, telecommunication, and software vendors. Some of the large corporations that participated in the study were American Express, A T & T Universal Card, Federal Express, Marriott, Microsoft, and First Union National Bank. The surveys were complemented by follow-up interviews, on-site visits to some of these companies, and analysis of data from their IVR and A C D reports. 2.1.3.1 Results and Key Findings The key measures of success were found to be Opt Out Rate (OOR) and Customer Satisfaction with the use of ARS. OOR is the percentage of callers that choose a menu option to escape the ARS in order to speak with a Customer Service Representative (CSR). The rationale for using these two factors for success was because they found that a large number of companies had installed ARS to reduce the volume of calls coming to the CSRs. Therefore, it was important that a high percentage of the calls that enter the ARS be closed successfully by the ARS without the need for human intervention. Also, customer satisfaction with the ARS system was an important measure for continued use of the ARS system. The most successful uses of the ARS based on the OOR and Customer Satisfaction were found to be: 1) Routing: where callers are directed to the proper group for assistance; 2) Provision of Basic Information such as balance inquiries; and, 3) Simple Transactions such as credit card payments and funds transfer. They also discovered that more complex transactions like bank-by-phone and collections applications used by utilities for automatic payment arrangements were highly successful given proper attention to customer education. 13 The least successful uses of ARS were: 1) Inquiries requiring timely data; 2) Provision of information that may often be an unpleasant surprise for the customer; 3) Technical assistance for complex issues that will vary by customers. The results of data analysis revealed that the simple implementation of Giving Information had the lowest OOR while for more complex applications like Transactions, over half of the callers ultimately ended with a CSR. The number of menu options also had an affect on the OOR. The results revealed that as the number of options increase, the percent of callers that opt out to a CSR goes up. The results also suggested that if the option to speak to a CSR was placed at the end of the first menu it results in more customers using the ARS system. This is because they are forced to listen to all the choices and hence more calls get closed in the ARS system. Finally, customer education was found to be a critical factor in the successful use of the ARS system. The data analysis revealed that for companies that placed usage instructions on billing statements and wallet cards had a lower OOR and a higher level of customer satisfaction than companies that do not provide such information. The study found that the use of Computer Telephony Integration (CTI) technology could provide tremendous gains to the companies in terms of higher customer satisfaction and reduced costs. Sophisticated CTI technology identifies the caller, anticipates the caller's most likely menu choices based on historical calling behaviour, and then moves the anticipated menu choices to the front of a dynamic menu. Even if the caller opts out to a CSR, it can reduce the talk time as the CSR would not have to ask the customer to repeat the information he has already entered into the ARS. 14 The study also found that companies which were involved in live ARS monitoring achieved a high degree of success with the ARS by analyzing the parts of the script where customers were opting out and then refining the script to make it more pointed. The study discovered that companies that had low Opt Out Rates and high Customer Satisfaction achieved this through: 1) Clear definition of market and customer needs; 2) Clear and pointed IVR scripting; 3) Logical structure of menus and options; and 4) Educating customers on the use of the IVR system. 2.1.4 Theory of Reasoned Action and its Variants There have been numerous studies conducted in Information Systems (IS) research literature on predicting and explaining the utilization of information technology by end-users [24]. Most of these studies have been based on the Theory of Reasoned Action (TRA) [25] and its variants, such as the "technology acceptance model" [26] and the Theory of Planned Behaviour [27]. The Theory of Reasoned Action is a general theory applicable to an array of behaviours, including the forces which influence the use of IT. It specifies the relationships between beliefs, attitudes, and behaviours. TRA is based on the proposition that an individual's overt behaviour is determined by the individual's intention to perform that behaviour, which is a function of his attitude towards performing the behaviour itself and his subjective norm. It further postulates that an individual's attitude towards the behaviour is determined by his belief that performing the behaviour will lead to certain outcomes. In a variant of TRA, called the Technology Acceptance Model (TAM), Davis [26] identified two general constructs, perceived ease of use and perceived usefulness of the 15 system as the beliefs hypothesized to determine attitude. Ease of use was defined as the degree to which a system is perceived as easy to use by an individual. The Theory of Reasoned Action is important to this study as based on this theory the constructs 'ease of use' and 'ease of understanding' were selected to design and test the new IVR scripts. As has been stated before, the criteria for developing an effective IVR system for WCB was to test its 'usability' by the actual users of the system. If it is perceived as 'usable' by the individual(s) it would affect his attitude towards its use and hence the actual usage as postulated by the TRA. Expressed in an equation form Usability = Intent to use (Attitude) + Actual Usage Hence, usability was tested on the student questionnaire as Usable -> Use -> Expected Benefits Arrows above indicate "leads to". 2.2 Implications for Current Research The research on the use of phone-based user interfaces provides valuable input for the current study. The literature review points towards the need for a thorough understanding of customers and business processes and identification of the nature and pattern of calls received by the call centre of a company. Several months were spent at W C B in order to gain an understanding of the nature and pattern of call types and the business processes. The design guidelines for IVR scripting compiled from the literature review and the best practices in the industry was utilized in formulating the most appropriate script for WCB. 16 CHAPTER 3. UNDERSTANDING THE NATURE AND TYPE OF CALLS AT THE WORKERS' COMPENSATION BOARD 3.1 Introduction The first step in designing a good IVR system is to identify the nature and pattern of calls coming into the call centre. The literature review suggested that only after gaining a deeper understanding of the call flow process the calls that are suitable for an IVR system can be pinpointed. It is essential to determine the type of calls received by the call-centre, the time spent in answering the different types of calls, the length of wrap-up associated with certain calls, and the length of pauses between answering the different kinds of calls. 3.2 Pre-Test In order to identify the type of calls coming into the call centre at WCB a representative sample of calls was tracked (approximately 60 in number) by listening in to the phone conversations and an initial classification scheme for categorizing the various types of calls was then developed. It was ascertained that the calls received by the call centre come from the claimant, doctor or employer. The nature of calls received by the call centre could be categorized as either: 1. claim or cheque inquiry, 2. inquiry about a fax number, or 3. claimants requesting a transfer. The transfer category was further sub-divided into three categories: a. clients requesting a transfer and the call ending in a transfer, b. clients requesting a transfer and the call not ending in a transfer, and finally c. the clients not requesting a transfer and the calls ending in a transfer. 17 Calls, which could not be categorized into any of the above categories, were classified as general. A preliminary survey was distributed to five CSRs to confirm these categories. Based on the feedback received from the CSRs the survey was refined to include one additional category, the doctor's billing. The survey was then rolled out to all the CSRs. Data was collected over a week by all the CSRs on approximately 5,000 calls to get a preliminary sense of the proportion of calls that fall under each category. The survey confirmed these categories. Aggregated results from the survey are presented below: Type of call Percentage Claim Inquiry 32% Cheque Inquiry 16% Wage Loss 3% Doctor's Billing 3% Not End in Transfer 7% End in Transfer 18% Not Want Transfer End in Transfer 6% General 15% Table 1. Breakdown of different types of calls Based on the results of the pre-test the WCB project team established that a suitable research instrument had to be designed in order to track the CSR's activity and record the various types of calls. This was necessary because extensive data on calls would be required in order to develop an effective IVR system for WCB, and it was unreasonable to expect CSRs to collect detailed information on calls for this purpose. Therefore, the WCB project team developed a real-time electronic tool in MS Access to track the calls coming into the call centre. The tool captures information pertaining to: 1. who is calling the call centre, 2. what type of calls are received, 3. how many of those calls get transferred, 18 4. the time taken to answer various types of calls, 5. the time spent on wrap-up, 6. the length of pauses between calls, and 7. the time spent on various activities by the CSRs during their phone-times, including making outgoing calls and coffee breaks. The tool was pretested for 3 hours by two research analysts by tapping into the phone calls at the call-centre. Based on the pretest and the feedback received from the CSRs the tool was further refined. A fourth category 'other' was added to the "Who is Calling?" section. To the "Transfer To" section two new categories were added: 'Media (including forms and posters)' and 'Disability'. 'Other' was added to each of "Calling Who?" and "What type?" in the Outgoing calls section. 'Log entry postponed' was dropped from the wrap-up category as it was learnt from the CSRs that they almost never postpone their log entry, but complete it right after the call. A screen-shot of the main user interface along with definitions of important terms used is presented below: 19 * N COE Data Collector [frm_Calls] S Fie g t t Vjew Insert Format Records Tools Window Help -1*1*1 a t i Si ^ 1 M i B -si - 51 Incoming Calls f l Claimant JJj Doctor El Employer J3 Other L~j Claim Inquiry Cheque Inquiry Doctor's Billing 3 Fax Inquiry E"J General H Transfer To Record: '< 1 I Form View Figure 1. Screen-shot of user interface for the data collection tool at the call-centre Important definitions a. Pause: this includes the time when CSRs are plugged in and waiting for an incoming call. During this time they are working on claims or other work related tasks. b. OtherPause: this includes non-work related activities. It is further subdivided into coffee breaks and other activities such as washroom or other breaks. c. Transfer To: i f the primary purpose of the call was a request for a transfer. Notice however, the call may not be transferred. In that case the option 'nowhere' would be checked under "Transfer to" category. Also, there might be instances when the primary purpose of the call was a claim inquiry but it ended in a transfer to some other department. In that case appropriate options would be ticked under the respective categories. 20 3.3 Data Collection at the Call-Centre Data was collected over 3 weeks for approximately 65 hours by three research analysts at the Richmond office. Each research analyst was equipped with a laptop and filled the electronic form while listening to the phone calls with the CSRs. At two of the area offices: Kelowna and Victoria, approximately 8 hours of data was collected. The purpose was to compare and contrast the activities of the head office with the area offices. At the end of the data-gathering period the WCB project team spent one week analyzing the data. As would be discussed in the results section a striking observation was that a high proportion of calls was related to transfers and routine calls like cheque inquiries. This led to the presentation of a proposal to WCB to explore the use of a well-designed IVR system to handle some of these calls. At this time, it was also ascertained that data collection was required in other areas of the organization like Entitlement, Case Management with which the call centre interacts in order to fully understand the call centre operations. Such an analysis will shed light on how some of those operations could be streamlined through the use of a well-designed IVR system. 3.4 Data Collection at Entitlement and Case Management There are two other units of WCB that receive calls related to claims. These are the Entitlement and Case Management units. They receive the calls either directly or as transfers from the call-centre. Several interviews with Entitlement officers and Case Managers were undertaken in order to get a sense of the type and nature of calls that these units get. Based on the interviews and the understanding that had been gained so far through data collection at the call centre, it was ascertained that for the incoming calls to these units it was important to know the call source, and who was calling. The call 21 source was particularly important as that would reveal how many calls get transferred from the call centre. For outgoing calls, it was enough to know who was being called. With this knowledge, a preliminary paper based form was developed which requested information on call source, the type of caller and the nature of incoming calls. For the outgoing calls, the form measured whom the call was going to and the purpose of the call. The paper-based form was refined based on feedback received from the entitlement officers and the purpose of the call was dropped from the Outgoing calls category as it was mostly related to claims. A copy of the paper-based form is presented in Appendix I. A computerized version of the paper-based form was developed by the WCB project team similar to the one developed earlier for call centre, which permits real-time data collection. The computerized tool measured time spent on various types of calls and other activities. On the electronic form the call source was divided into: 1) Direct-in-Dial (DID) 2) Transfer from the call centre, and 3) Other The various categories of people that could call these units were categorized as Claimant, Employer, Health Care Professional, Internal, Other. Same categories were used for the outgoing calls. A screenshot of the main interface is presented below: 22 Outgoing Calls J| Claimant • Health Care Prof. .ll Employer J| Internal M Other ^ COE D a t a Co l l ec to r [fim_Calls] | i El Fie £<St View jTsert Format Records Tools Window Help a Figure 2. Screen-shot of user interface for the data collection tool at Entitlement and Case Management Two research assistants were thoroughly trained in the use of the tool and were also given an overview of the claims process at WCB. Such an extensive training was required in order for them to fully understand the business processes and write intelligent comments in the Remarks field of the electronic form. The research assistants sat with the entitlement officers for the entire course of their working day and monitored every incoming and outgoing call during the shift. However, unlike the data collection at the call centre, the research assistants could not get plugged into the phones of the entitlement officers as their telephone instruments were not designed to have two outlets for headsets. This was not a major hindrance as the research assistants were able to comprehend the nature of the call for the purpose of data collection just by listening to one side of the conversation. Data was collected for every alternate day starting Monday for a week totaling about 60 hours. Simultaneously, paper based forms were distributed to some of the entitlement officers who filled out the forms for their incoming calls and outgoing calls. The Entitlement officers had volunteered to 23 either fill-out the paper-based forms or let a research assistant sit with them to collect the data. The WCB project team met with entitlement officers in small groups to explain the necessity and purpose of the data collection exercise. This was done to ensure that they record the information correctly. Emails were sent out to entitlement officers by their managers explaining the study and only those entitlement officers who were seriously interested in the study responded to the email. A similar exercise was conducted at the case management units. The same computerized and paper-based form was used in this unit as the nature of the work and calls is the same for both entitlement and case managers, it only differs in the level of complexity. Two research assistants collected data for an entire week at the case management unit for a total of approximately 45 hours. As the number of case managers at WCB is smaller than number of entitlement officers, data was collected for a lesser number of hours with them. The intent was to get a sense of the nature and type of calls that they get, particularly the calls that are transferred from the call centre. A week of data gathering was considered sufficient for this purpose. Simultaneously, paper-based forms were distributed to two case managers. Again, the participation in the research was voluntary for the case managers. At the end of each day, the main researcher went over each of the files that contained the data in order to ensure the correctness and quality of the data. It was not unusual to see the call source or some other field blank in the data file. The research assistants were immediately approached in such a case and in most cases they were able to provide the missing information by reading their own comments in the Remarks field. However, in cases where they could not recall the missing information, the particular calls were dropped from the analysis. The researcher also made adequate back-up copies of every file. 3.5 Problems encountered during Research 24 The people at the Call Centre, Entitlement, and Case Management were generally very helpful and co-operative during the study. However, it should be mentioned that different people have different personality styles and some of the staff at the call centre behaved in the most natural way while a research assistant was plugged into the phones with them. Some of the CSRs, however, were very enthusiastic and attempted to give a lot of information to the research assistant during their phone shifts. Although the information was very useful, the sample of calls collected during those shifts would not reflect the true sample that could be obtained had the CSR performed their duties in the most natural way. This can slightly deflate the number of calls on which information could be recorded. This is because the CSRs had put their phones 'on hold' while they were in discussions with the research assistants. Accordingly, the results on data analysis could have deflated figures on calls in each category. 3.6 Results of data analysis at the Call Centre, Case Management, and Entitlement This section provides the conclusions reached after a comprehensive analysis of the data gathered at the three units at WCB. For an in-depth analysis on the nature and pattern of calls to the call centre the reader is referred to Appendix II. For results on calls to the Case Management and Entitlement units please refer to Appendix III. Among the many observations, the one that was the most striking was that on an average 29% of the calls received by the call centre concluded in a transfer to other departments. This figure includes the 14% of the callers who ask for a transfer and get a transfer and 15% of the callers who don't ask for a transfer but the call ends in a transfer (refer to the pie chart in 1.4.1 in Appendix II on transferred calls). Approximately 11% of the calls were related to cheque inquiries and another 4% were fax inquiries. These results led to the identification of the need for the development of an optimal IVR script that could potentially handle some of these routine types of calls. This represents a substantial times savings and therefore provides the motivation to develop a well-designed IVR system. 25 It was discovered that on an average CSRs spend 2 minutes on a call before it gets transferred. Observing the data from the A C D for a representative week revealed that the call centre gets on average 6000 calls per week. The calculation of cost savings resulting from eliminating the transferred calls is demonstrated below (the figure of 20% shown below is an approximation and is based on the assumption that not all the calls that require a transfer would be assumed by an efficient IVR): 20% of 6000 calls = 1200 transferred calls. 1200 transferred calls * 2 minutes = 2400 minutes or 40 hours. 40 hours* $25.00 (average CSR salary/hr.) = $1000/wk or $52,000/yr. This would imply that by reducing the time spent on answering calls, the CSRs could devote more time to handling calls. The example of transferred calls was given for demonstration purposes only. Additional savings would result from eliminating some of the routine type of calls like cheque and fax inquiries. In addition to financial savings, there are intangible benefits that would accrue from the use of a well-designed IVR. For example, an efficient IVR by successfully closing a higher number of calls will lead to a lower level of stress among CSRs and hence increase employee productivity. As well, a caller-friendly fully automated system will provide better customer service as inquiries will be satisfied more efficiently and the callers are not restricted to the hours of operation of WCB. Having identified the types of calls that could potentially be handled by a well-designed IVR system and the potential financial savings that can accrue from it, the next step is the development of an effective IVR system. The next chapter compares the existing WCB script against the best practices in the industry (described earlier in the literature review section and summarized in the next chapter) and describes the process of developing such an effective script. 26 CHAPTER 4. SCRIPT DEVELOPMENT This chapter evaluates the current WCB system against the best practices in the industry. The objectives and goals for an IVR system for WCB are then described based on the understanding gained from the analysis of calls received by the call centre at WCB. This led to the development of two new scripts which can potentially reduce the number of calls that get through to the CSRs from the IVR system. A methodology for evaluating the two scripts on student subjects using a computer- based simulation is also described. 4.1 Evaluation of WCB Script against the Best Practices The following table summarizes the comparison of WCB against the best practices. The criteria listed in the table were gathered from an extensive review of literature on call centres [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16], [17], [18]. The suggestions under each criterion are the opinions of subject matter experts and are based on the results of the studies on voice response systems undertaken by them. C r i t e r i a Best Industry Prac t ice W C B C a l l Cen t re App l i ca t i on designed towards target audience? Thoroughly research your target audience and then design the menus and prompts to best meet their needs. Yes. Tailored towards the needs of injured workers, employers, and health care professionals. Purpose of the appl icat ion Clearly state the purpose and objective function for the IVR. State all your constraints and maximize your objectives while meeting the constraints Not clearly defined. M e n u structure 3*3 design. 3 * 1 design. The menu structure is simple and easy to navigate. Presentation of choices Prompts for menu choices should always end with instructions for the required caller action. Meets the best industry practice. Prompts followed by required action..."If you are an injured worker press ' 1' now." " D u a l Tone Mul t i - f r equency" ( D T M F ) o r Type ahead capabi l i ty Provide the capability to 'cut-through' the menu options and 'type-ahead' the choice without listening to all the prompts. Yes. Data capture Log features that are most commonly used, log where callers error out of the application. Log transaction Plans to capture data extensively once the in-house IVR system is in place. 27 query times. Make every effort to streamline the application. Hang-ups Understand where in the application callers hang-up and review prompt-wording to make improvements to caller interface. Not comprehensive. E r r o r - h a n d l i n g Use descriptive explanations for any errors or faults on the part of the caller instead of hanging-up on caller. Does not hang-up on callers but does not give sufficient explanation for errors, for instance "Your entry must contain exactly seven digits." Sufficient t ime for use r -response. This would be conditioned by the purpose outlined at the beginning for an IVR application. Recommended is 6 seconds for first digit time-out and 4 seconds for inter-digit time-out. Does not give sufficient time to the caller to enter the claim or personal access number (only 3 seconds given). The application is too quick in transferring call to the CSR. Time-outs Transaction time-outs should be set according to user - tolerance. Maximum recommended is 30 seconds. Based on an efficient algorithm (LogN). User wait-experience To reduce caller anxiety provide music on hold or even a promotional message. Mask the query time by playing a message in the background such as "Please wait while we retrieve your account balance." Yes A l a r m s Use threshold alarms for events like database query time-outs, critical function call-errors. The alarms could be as simple as an email message to the system administrator. No alarms. Input attempts Anywhere from three to four depending on the complexity of the information to be input. Only two. C o n f i r m cal ler input For lengthy and complex input like credit card numbers confirm caller-input and give an opportunity to correct mistakes. Confirms the user input of the claim number. Table 2. Evaluation of the current IVR system against the Best Industry Practices The suggested practices under each criterion, in addition to the theoretical framework provided by the Theory of Reasoned Action and its variants, were taken into consideration when developing the alternate IVR scripts for WCB. The next table summarizes the audience for the new script, the purpose it is expected to accomplish, the 28 goals and constraints that have to be kept in mind while developing the new script, and the expected outcome of undertaking this exercise. Audience Injured workers, Employers, and Health Care Professionals Purpose Provide the highest possible level of customer service by providing access to routine claim and cheque inquiries 24 hours a day, 7 days a week. Make live agent support available from 7 am to 7 pm Monday through Friday. Goals 1. Encourage the use of IVR for all types of calls and filter out calls that get through to the CSRs. 2. Reduce the cost of handling incoming phone-calls. 3. Utilize IVR for direct transfers to Case Management and Entitlement officers i f the claimant knows either the last name or extension number of the officer handling their claim. Constraints 1. IVR can be utilized for only the most basic type of claim inquiries (for instance to provide information on whether the claim has been accepted, disallowed, or pending.) 2. IVR can be utilized for providing information only on the last cheque mailed provided the cheque issue date is less than 30 days. 3. If the IVR design becomes too complex it could actually inhibit the use of the IVR system by callers Outcome A simple to use IVR system that reduces the current volume of calls handled by the CSRs by successfully closing the calls and streamlines the call flow process by directing calls to the final destination. Table 3. Objectives and Goals for an IVR System for WCB 4.2 Existing and Proposed Call Centre Architecture This section provides a picture of the existing call centre architecture at WCB and proposes a new architecture based on the results of the data collection and goals and objectives of the new system that were described in the previous section. The following diagram depicts the existing call centre architecture: 29 Figure 3. Existing Call Centre Architecture As can be seen in the above diagram the only option for the calls after they have hit the IVR system is to go to the CSRs. However, i f the purpose is to reduce the number of calls to the CSRs it is necessary to provide other alternatives in the IVR than just reaching the CSRs. As was revealed through the data collection, a large number of calls ended in transfers to Entitlement and Case Management. Hence, providing automatic transfer option to these units in the new IVR system could potentially reduce the number of such calls to the CSR. The proposed architecture, based on this understanding, is depicted in the next diagram. 30 To Case Management 1 • To Entitlement Figure 4. Proposed Call Centre Architecture The proposed architecture directs calls to Entitlement and Case Management units by providing such an option in the IVR system. 4.3 Alternate IVR Scripts This section describes the actual scripts that were developed for WCB based on the understanding gained through the data analysis and the review of best practices. It was ascertained that mainly three groups of people call the WCB call centre: Injured Workers, 31 Doctors, and Employers. The two main reasons they call the call centre are to inquire about the status of a claim or a claim payment cheque. This suggests that a particular category of people call the WCB call centre to accomplish a particular task. Based on this understanding, the WCB project team conceived the idea of testing two alternate designs for the main menu. In one script, the opening menu would ask the caller to identify himself as a doctor, employer, or injured worker after playing the greeting. In the other script, the opening menu would ask the caller what task he wishes to accomplish after the initial greeting. Hence the first script is 'role specific' and the other script is 'task specific'. The idea of such alternate scripts was thought to be an interesting research question to explore. The rationale for developing the two new scripts was to not only compare them against the original WCB script but also to compare them with one another to see i f callers preferred the 'role specific' to 'task specific' script or vice-versa. 4 .3 .1 Addressing the problems through the new scripts As discussed in the previous chapter it was discovered that 29% of the calls on which data was collected were related to transfers to other departments mainly, to case management and entitlement units. Another 11% of the calls were related to cheque inquiries and 4% asked for faxing a form. This led to the motivation to suggest a new IVR system that would address these problems. The goals for such an IVR system were stated in section 4.1 above. These goals were to utilize the IVR for automatic transfers to these units and reduce routine calls; for instance cheque inquires, by designing a simple and easy-to-use IVR system that could successfully close such calls. 32 Keeping this in mind, the option for automatic transfers to other departments, i f the extension number is known, was presented as one of the options in the new scripts. To address routine type of calls an additional option was added that provided automatic faxing capability and general information like information on hours of operation of WCB. However, there is no guarantee that the callers will in fact use the additional options provided in the new scripts. It was therefore important to design the new scripts in such a way that the callers are actually encouraged to use it and not bypass the system to speak to a live agent. By analysing the existing WCB script it was discovered that the opening greeting was 20 seconds long. Even though there are no standard guidelines for the length of the initial greeting most customer oriented announcements are approximately 10 seconds long. [9] Although the initial greeting for the 'task specific' script was 18 seconds long as it provided the choice of four tasks (claim inquiry, cheque inquiry, transfer, and general information and forms) the 'role specific' script was 12 seconds long. The initial W C B script gave lengthy instruction on how to use the system and what was needed. It also gave the callers the option to either use or not use the system. As opposed to this the 'role specific' script by asking the callers to identify themselves could lead to providing a sense of customized choices to users.. And the 'task specific' led directly to what the users wanted to accomplish right from the opening menu. Although the option to speak to a live agent by pressing '0' was present in the new scripts it was not explicitly stated in the opening menu. Although these best practices were adhered to, it was critical to incorporate the theoretical framework suggested by TRA in the design of the new scripts. This was especially important as among the many variables (attitude, subjective norm, and voluntariness) in TRA, attitude was found to have the greatest direct effect on usage. Attitude, as stated in section 2.1.4 of chapter 2, was conditioned by the perceived usefulness and perceived 33 ease of use of the system. The new scripts avoided the use of excessive number of options, and provided instructions that were simple and clear. These factors would affect the perceived usefulness and ease of use of the system, which are the beliefs hypothesized to determine attitude. Affecting attitude would have an affect on usage, and that was the factor tested in the student questionnaire. Other design criteria that were reviewed in Chapter 2 and Section 4.1 above on best industry practices were kept in mind while designing the new scripts. For instance no more than three choices at each menu (except in the 'task specific' menu which gave the choice of four tasks), use of simple language, three input attempts (as opposed to two in case of the existing script) before transferring to live support, etc. were incorporated in the new design. The actual scripts are depicted in Appendix IV. 4.4 Laboratory v/s Field After the two alternate scripts had been developed it was necessary to test them to ascertain whether they were a better alternate to the original WCB script. There were two feasible approaches to test the two different scripts. It could be either be tested in the laboratory or the field. There are pros and cons to each of the suggested methodology. The following table presents the advantages and disadvantages of each approach: 34 Laboratory Method Advantages It is a cheaper alternative to design a simulation than a full-scale IVR system. Disadvantages Results gathered from such an experiment may not have the suggested effect when applied to the call centre at W C B The time for setting the apparatus for conducting the experiment could become long-Motivating subjects to participate in the study would be difficult. Field Method Advantages As the field represents the actual conditions, results gathered from such an experiment will reflect actual change in the caller behaviour in response to the changed script(s). A decision to implement one of the new scripts based on the field experiment can be expected to lead to positive changes in the nature and pattern of calls by reducing the routine calls. Disadvantages There is a risk of annoying callers used to the original script by presenting them with alternate scripts. The cost of placing the new scripts on the system could be high. Table 4. Laboratory vs. Field Method UBC telecommunications department was approached to get an idea for setting up an IVR system. The costs for setting up a full-scale IVR system were found to be prohibitive. It was ultimately decided to develop a computer based IVR simulation. The computer-based simulation was developed by an undergraduate computer science student at U B C and the sound files for the simulation were recorded by another student. The cost of developing the IVR simulation came to approximately $500.00 4.5 Computer-based simulation for the new scripts A computer-based simulation was developed in C++ which imitates a real IVR system. The 'windows' based interface shows a telephone keypad on the screen and plays sound files that were recorded based on the scripts in Appendix IV. 35 In order to ensure that the simulation is as close to reality as possible, the user could enter responses (such as claim number) by using the numeric keys on the keyboard in addition to clicking the numbers on the screen with a mouse. Their interaction with the computer in such a way is just as if they were using a touch-tone telephone. In fact, all subjects who participated in the survey chose the numeric keypad to enter information. A screenshot of the interface is presented below: I IVR Simulator File Simulation Help i 2 3 5 6 7 8 9 * 0 # Figure 5. Screenshot of main user interface for the computer-based IVR simulation The simulation is reconfigurable, that is, new simulations can be added by recording new sound files and specifying the logical path for the script. This is possible because the simulation is state-based, which means that the simulation starts off with a default state and makes appropriate transitions from one state to the next until the end state is reached. Each state consists of output that plays the sound files (that provide information to user or ask for information from user) and input that accepts touch-tone entries from the user and then makes a transition to the next stage based on the user input. When the system enters a new state, it plays any specified output sounds and then accepts input as specified by the current state. If no input (or invalid input) is provided, then the system enters the timeout states as specified by the error handling parameters of the corresponding state. 36 The following picture shows the configuration window of the simulation. The left pane shows the simulation's different states, and the right pane allows one to configure the output, input, and error handling modes of each state. Simulation Configuration OPERATOR Name Description [START J Default start state Output I Input I Error Handling I Output 2 ~ <? NONE r BUFFER C SOUND Output 3 <i NONE r BUFFER C SOUND EJ OK Output 1 d NONE C BUFFER C SOUND Figure 6. Screenshot of configuration interface for the computer-based IVR simulation For detailed information readers are referred to the system documentation in Appendix V . 4.6 Student Questionnaire Various alternatives were considered for choosing subjects for the purpose of testing the IVR scripts. The most representative sample would have been the real clients of WCB on 37 whom the simulation could be tested. However, in spite of this obvious advantage there were numerous difficulties associated with choosing real WCB clients. It would have been necessary to approach not only injured workers but also doctors and employers to participate in the study, as these are the three major groups of people who call WCB. Most organizations are sensitive to causing any inconvenience to their clients and hence getting their participation would have been difficult. Besides, it would have been difficult to find an exact number of subjects that would be enough from a statistical standpoint. Ultimately, it was decided to use student-subjects for the purpose of testing the IVR simulation. The advantage of using student subjects was that they were more easily accessible at UBC. Anybody can be a potential claimant of W C B and hence testing it on students of U B C was considered representative enough of the client base of WCB. 4.6.1 External Validity Problems Utilising student subjects could lead to external validity problems. External validity is defined as the degree to which the causal relationships observed can be generalized to other populations, settings, and time [19]. Student subjects are educated and may not be representative of the typical caller at the WCB call centre (i.e., an injured worker) who has a relatively low level of education compared to university students. They are more likely to be closer to doctors and employers in terms of their academic background. Results gathered from such a survey would not be valid for the typical client base of WCB. There has been extensive debate in the literature on the question of using student subjects as surrogates for real world people. The results have been varied and inconclusive. Ashton and Kramer [28], however, state that the available evidence suggests that real-world decision-makers possess information-processing characteristics and biases that are extremely similar to their student counterparts. A study by Hawkins, Albaum, and Best 38 [29] explicitly investigated both the attitude and decision-making aspects of the surrogation issue. They concluded that despite the fact that the students and the housewives held different attitudes about the five stores in their study, their attitudes predicted their behaviour with equivalent accuracy. In order to counteract the external validity problems one of the suggestions found in the literature is to include several classes of persons, settings, etc. in the sample [19]. Therefore, the computer-based IVR simulation was tested on a varied mix of undergraduate and graduate students. Their academic level ranged anywhere from first year of university to the final year of graduate school. Also, i f it could be proven that there were no differences in responses of graduate and undergraduate students the results of the simulation could be strengthened in terms of its applicability to the population. This is because one would expect graduate and undergraduate students to form two different groups within the student population. One can make generalizations on the two groups based on demographics (undergraduate students are usually younger than the graduate students), the characteristic that would be most relevant for this study is their level of intelligence. Graduate students would be expected to possess a higher level of intelligence and knowledge than their undergraduate counterparts. This is substantiated by the stringent admission requirements to graduate schools in most universities. However, external validity will always be a concern in laboratory experiments [20]. Therefore, the results of this study would only be applicable for a particular category of potential or actual W C B clients, those who are university students. 39 4.6.2 Testing the IVR Simulation The computer-based simulation was tested on a sample of undergraduate and graduate students in the Faculty of Commerce and Business Administration at the University of British Columbia. The simulation was tested on 18 students. 6 students were asked to act as Injured Workers, 6 as Doctors, and 6 students asked to pose as Employers. Due to the limited amount of time and budgetary constraints a larger number of students could not be gathered for the study. Each student was given five tasks to perform. They were: 1. Inquire about the status of a claim 2. Inquire about a claim cheque payment 3. Talk to an adjudicator whose extension number was known 4. Get a form faxed 5. Inquire about the hours of operation of WCB The above tasks were representative of the most typical calls received by the call centre at WCB. The student performed the first two tasks on all the three systems and the last three on only the two new systems as the original WCB system does not provide the functionality to perform those tasks. That was also a good way to ascertain if providing the last three options would address some of the problems described in section 4.3.1 of this chapter. 4.6.3 Internal Validity Problems It was necessary to choose an appropriate research instrument and design to address the internal validity concerns. The various threats to internal validity can be reduced or eliminated by the use of experimental design [21]. Benbasat [20] suggests random assignment of subjects and the use of a statistical design that takes into account learning effects. These suggestions were incorporated into the design of the current study. As 40 mentioned earlier, subjects were randomly drawn from the undergraduate and graduate student population at U B C from different faculties. They were randomly assigned to pose as Doctors, Employers, or Injured Workers. The order in which the scripts were presented to the subjects was altered to minimize or eliminate the learning effect. If all subjects were presented the scripts in the same order there were chances that the first script would have rated lower than the other two as the subjects are trying to learn how to interact with the system. Once they are past the learning curve they would have found the other two scripts easier to use. The design of the study is presented in the following table: Injured Worker Doctor Employer Task 1 (1,2, 3) (1,3, 2) (2, 1,3) (2, 3, 1) (3,2, 1)(3, 1,2) (1,2, 3) (1,3, 2) (2,1,3) (2,3,1) (3, 2,1) (3, 1,2) (1,2, 3) (1,3, 2) (2, 1,3) (2, 3, 1) (3, 2,1) (3, 1,2) Task 2 (2,1,3) (2,3,1) (1,2, 3) (1,3, 2) (3,2, 1)(3, 1,2) (2, 1,3) (2, 3,1) (1,2, 3) (1,3, 2) (3, 2,1) (3, 1,2) (2,1,3) (2,3, 1) (1,2,3)(1,3,2) (3, 2,1) (3, 1,2) Task 3 (2, 3) (2, 3) (2, 3) (3, 2) (3, 2) (3, 2) (2, 3) (2, 3) (2, 3) (3,2) (3,2) (3,2) (2, 3) (2, 3) (2, 3) (3,2) (3,2) (3,2) Task 4 (3,2) (2, 3) (3,2) (2, 3) (3, 2) (2, 3) (2, 3) (3, 2) (2,3) (3, 2) (2,3) (3, 2) (3, 2) (2, 3) (3, 2) (2,3).(3,2) (2,3) Task 5 (2, 3) (3, 2) (2, 3) (3, 2) (2, 3) (3,2) (3,2) (2, 3) (3,2) (2, 3) (3, 2) (2, 3) (2, 3) (3, 2) (2, 3) (3, 2) (2, 3) (3, 2) Table 5. Design of experiment for Student Questionnaire In the above table 1 denotes the original WCB script, 2 denotes the 'task specific' script and 3 stands for the 'role specific' script. Numbers in brackets denote the possible combinations of the three or two different systems/scripts. The students responded to a questionnaire at the end of performing each task on each system. This was necessary because if they had responded to the questionnaire after completing all the tasks on different systems they would not have remembered how their 41 interaction was with different systems. The students took between 15-20 minutes to perform the five tasks on the different systems and they were remunerated $5.00 for their time. In order to secure honest responses from students they were made aware of the monetary incentive before they began the study. The researcher was present throughout the simulation to run and pause the different scripts while the student responded to the questionnaire. However, no words were exchanged between the subject and the researcher during the course of the simulation. After the simulation was completed the subjects were encouraged to provide any written or verbal comments or feedback on the study. The design of the questionnaire was important as it would provide the answers to the research questions stated in Chapter 1 of this thesis. The suggested guidelines in literature review had emphasized a logical structure of menus and use of unambiguous instructions to design a good IVR system. The purpose of a good IVR system is to encourage its use by the callers and hence it should be easy to use and the instructions in the system should be easy to follow. Based on this understanding, the questionnaire was designed to measure two constructs 'ease of use' and 'ease of understanding'. The first construct had four questions and the second construct had three questions. The questions were on a 5 point Likert Scale and asked the respondents to rate their extent of agreement (or disagreement) with the statement made in the question. An additional question asked i f subjects would have likely used an IVR or live operator for the purpose of accomplishing the particular task. For a copy of the questionnaire please refer to Appendix VI. In order to ensure construct validity, a previously developed instrument in MIS research by Moore and Benbasat [22] was utilised for the construct 'ease of use'. The same format and logic was used to derive the questions for the second construct 'ease of understanding'. 42 The results of the questionnaire are described in the next chapt< 43 CHAPTER 5: STATISTICAL ANALYSIS OF THE QUESTIONNAIRE This chapter describes the results of the questionnaire that the students responded to during their interaction with the computer-based simulation for the different IVR systems. The summary of results for the student questionnaire is presented in section 5.1 and 5.2. They also describes the statistical tests that were performed which provided evidence that students preferred the new IVR script over the current WCB script for one of the two tasks and did not make a distinction between the two new scripts. 5.1 Summary Statistics and Statistical tests This section summarizes the results of the questionnaire by calculating the percentage of responses falling under each category. The information is presented in a tabular format. The standard five point Likert scale was utilised on the questionnaire. Since there is not much difference between 'Strongly Agree' and 'Agree' (and likewise 'Strongly Disagree' and 'Disagree'), the two were combined for the purpose of the analysis. The results are presented by grouping questions (a) to (d) that measured the construct 'ease of use' and grouping questions (e) to (g) that measure the construct 'ease of understanding'. The results revealed by aggregating questionnaire responses are then tested by using statistical tests to ascertain if there was a statistical difference between the three different scripts. For the purpose of the tests all the questions (a) to (g) were combined in the same variable. Even though questions (a) to (d) measure a different construct than questions (e) to (g), it was decided that both of them are very much related, and therefore they were combined and their responses were aggregated with even weights assigned to all the questions. The two constructs were chosen because they would affect the 'usability' of the system. That means that i f a system was easy to use and the instruction were easy to understand it would be more likely to be used by the users than the system which does not 44 exhibit these characteristics. An average score for this 'usability' was calculated which became the dependent variable in the statistical analysis. This was also necessary because of the small size of the sample and combining them was more likely to provide statistically valid results. If there were reasons to believe that questions (a) to (d) were more relevant than the second group of questions, then they could have been given higher weights and the analysis would have proceeded accordingly. Same would have been the case had the two constructs were not related and then they would have been grouped separately. For the purpose of statistical analysis the test that was found to be most suitable was a non-parametric test called the Friedman's test. It is suited to a repeated measures design of k related samples, which is the case with the samples in this study, and makes no assumption about the normality of the distribution. However the Friedman's test does not show the variance 'between' and 'within' subjects. It was necessary to test for that variance as each of the samel 8 students performed all the five tasks on the different systems. The statistical test that was chosen for this purpose was A N O V A as it compares the 'effectiveness' of different methods of treatments (in this case the different scripts) and is superior to multiple t-tests. The test statistic for A N O V A is the F ratio, which is the ratio of 'between-groups' and 'within-groups' variance. In the case of the first two tasks N was 54 as 18 students performed the same task on 3 different systems (18*3). And in the case of the remaining three tasks N was 36 as the 18 students performed the tasks on the two new systems (18*2). A third statistical test was used in an attempt to alleviate some of the concerns regarding external validity in laboratory experiments, by testing for differences between the undergraduate respondents and graduate respondents. If it can be statistically proven that there was no difference between them, then one can state that there should not be an 45 observable difference between these student surrogates and the typical WCB caller on the dimension of knowledge or personal sophistication. The test that was found to be best suited for this purpose is the General Linear Model (GLM) available in the Advanced Statistics Option of SPSS. It is a univariate repeated measures analysis using a split-plot design approach. Split-plot means that there are two sizes of experimental units in the data: the subjects, which are the larger experimental units, and the blocks of trial within each subject, which are the smaller experimental units. In this study the student sample is the larger experimental unit which is divided into two subplots: undergraduate and graduate students. They were coded as 1 and 2 respectively when entering the data in SPSS. 5.2 The Experimental Model and Hypothesis The model consisted of the dependant variable 'usability' which, as stated before, was calculated by combining the construct 'ease of use' and 'ease of understanding'. An average score was arrived at for this usability. Expressed in the form of an equation this was: Usability = BO + Bl(Ease of use) + B2(Ease of understanding) Since there were three different scripts, Usability = BO + Bl(Script 1) + B2(Script 2) + B3(Script 3) Also, since differences in responses of undergraduate and graduate students was tested, Usability = BO + Bl(UG) + B2(G) Where U G and G stands for undergraduate and graduate students respectively. 46 The three hypothesis that were formulated were, HI : Students would find the new scripts better than the existing WCB script for the first two tasks. This is because they were formulated based on best practices and perceived ease of use and usefulness in the Theory of Reasoned Action which would positively affect attitude and hence, usage (usability). However, for the next three tasks, one cannot hypothesize that between the two new scripts one would be better than the other. This is because there is no proof in the literature for this hypothesis and the statistical results of the student questionnaire is expected to provide this answer. Hence the null hypothesis is, HO: There would not be any difference in subject's preferences between the two new scripts for the remaining three tasks The final hypothesis is for testing the two groups of students, HI : There would be a difference in responses of graduate and undergraduate students as they form two different groups in the student population. The results from the three tests are presented in the following paragraphs. It should be noted that throughout the statistical analysis a significance level of 0.05 was used. Task Number 1. Inquire about the status of a claim. Questions (a) to (d). Using an IVR is easy WCB Script 2 Labels Strongly Agree Agree Neutral Disagree Strongly Disagree Percentage 5 2 % 2 9 % 1 1 % 7 % 1 % Labels Strongly Agree Agree Neutral Disagree Strongly Disagree Percentage 6 8 % 2 8 % 3 % 1 % 47 Script 3 Labels Percentage Strongly Agree 69% Agree 31% Neutral Disagree Strongly Disagree As can be seen from the table above, for the first task, students found their interaction with the new systems easier than with the original WCB script. Between Script 2 and Script 3 there is less variability in student's responses in Script 3 as 100% of the student sample either strongly agreed or agreed that Script 3 was easy to use. Questions (e) to (g). Instructions in the script were easy to understand WCB Labels Percentage Strongly Agree 44% Agree 26% Neutral 19% Disagree 2% Strongly Disagree 9% Script 2 Labels Percentage Strongly Agree 64% Agree 28% Neutral 4% Disagree 4% Strongly Disagree Script 3 Labels Percentage Strongly Agree 72% Agree 24% Neutral 4% Disagree Strongly Disagree A large proportion, 92% in the case of Script 2 and 96% in the case of Script 3, either strongly agreed or agreed that the instructions were easy to understand as opposed to only 70% in the case of W C B script. 48 The results of Friedman test are shown first: FRIEDMAN TEST Mean Rank WCB 2.50 Script 2 1.72 Script 3 1.78 N 18 Chi-Square 8.561 DF 2 Significance 0.014 As can be seen, the significance is 0.014, which is lower than 0.05; therefore the conclusion is that there is statistical evidence that these sample come from different populations. These results are also corroborated by the A N O V A results shown below. Source Sum of Squares Df Mean Square F-Ratio P-Value Between groups 3.53794 2 1.76897 4.16 0.0213 Within groups 21.7103 51 0.425691 Total (corr.) 25.2482 53 As can be seen in the above table, the F-ratio for groups, which in this case equals 4.15552, is a ratio of the between-group estimate to the within-group estimate. Since the P-value of the F-test is less than 0.05, there is a statistically significant difference between the means of the various groups at the 5.0% significance level. The output from the multiple range tests is demonstrated below to determine which means are significantly different from which others. 49 Only, the multiple range tests for the first task is shown in this chapter as on other tasks the test did not find a statistically significant difference between the means of the different groups. Multiple Range Tests Response variable: Usability Method: 95.0 percent LSD Script Count Mean Homogeneous Groups 3.0 18 1.30944 X 2.0 18 1.41278 X 1.0 18 1.89667 X Contrast Difference +/- Limits 1.0-2.0 *0.483889 0.436617 1.0-3.0 *0.587222 0.436617 2.0-3.0 0.103333 0.436617 * denotes a statistically significant difference. The bottom half of the output shows the estimated difference between each pair of means. An asterisk has been placed next to 2 pairs, indicating that these pairs show statistically significant differences at the 95.0% confidence level. The top half identifies homogenous groups using columns of X's. Within each column, the levels containing X's form a group of means within which there are no statistically significant differences. As can be seen, there was a statistical difference between 1 (WCB script) and 2 and 1 and 3 but not between 2 and 3, the two new scripts. Hence the alternate hypothesis HI , that the new scripts would be better than the existing WCB script, is accepted. The final statistical test tested for differences between undergraduate and graduate student responses. The results of the G L M are shown below: 50 Source Type III SS DF Mean Square F Sig. Corrected Model 3.916 .3 1.305 3.059 0.037 Intercept 118.7 1 118.7 278.19 0.000 GROUP 0.378 1 0.378 0.886 0.351 SCRIPT 3.538 2 1.769 4.146 0.022 Error 21.33 50 0.427 Total 153.25 54 Corrected Total 25.25 53 The column that is of interest in the above table is the Sig. As can be seen above in the row labeled 'GROUP' , the significance level is 0.351 which is higher than the 0.05 level and hence there was no difference in responses of the 2 groups, graduate and undergraduate students. The alternate hypothesis HI is therefore, rejected. Notice, however, the significance level of 0.02 in the row labeled 'SCRIPT', which corroborates the results from the previous two tests. Task Number 2. Inquire about a claim cheque payment. Questions (a) to (d). Using an IVR is easy W C B Labels Percentage S t r o n g l y A g r e e 5 3 % A g r e e 3 2 % N e u t r a l 1 5 % D i s a g r e e S t r o n g l y D i s a g r e e Script 2 Labels Percentage S t r o n g l y A g r e e 5 9 % A g r e e 3 5 % N e u t r a l 6 % D i s a g r e e S t r o n g l y D i s a g r e e Script 3 Labels Percentage S t r o n g l y A g r e e 6 5 % A g r e e 3 1 % N e u t r a l 4 % D i s a g r e e S t r o n g l y D i s a g r e e 51 Again, for the above task, 94% of the student sample in the case of Script 2 and 96% in the case of Script 3, either strongly agreed or agreed that the IVR was easy to use as opposed to a corresponding figure of 85% in the case of WCB script. Questions (e) to (g). Instructions in the script were easy to understand WCB Labels Percentage S t r o n g l y A g r e e 5 9 % A g r e e 2 6 % N e u t r a l 1 5 % D i s a g r e e S t r o n g l y D i s a g r e e Script 2 Labels Percentage S t r o n g l y A g r e e 4 8 % A g r e e 3 9 % N e u t r a l 1 3 % D i s a g r e e S t r o n g l y D i s a g r e e Script 3 Labels Percentage S t r o n g l y A g r e e 6 2 % A g r e e 3 1 % N e u t r a l 7 % D i s a g r e e S t r o n g l y D i s a g r e e However, for the clarity of instructions in the three scripts, there were only marginal differences in student's preferences. As can be seen in the above table, the proportion of responses falling under the first two category was 85% in the case of Script 1, 87% in case of Script 2, and 93% in the case of Script 3. The results of Friedman's test and A N O V A follow. FRIEDMAN TEST Mean Rank WCB 2.28 Script 2 2.06 Script 3 1.67 52 N 18 Chi-Square 4.769 DF 2 Significance 0.092 A N O V A Table Source Sum of Squares Df Mean Square F-Ratio P-Value Between groups 0.283959 2 0.14198 0.49 0.6140 Within groups 14.7051 51 0.288335 Total (coir.) 14.9891 53 As can be seen in the above two tables the P value and the Significance are both higher than the alpha value of 0.05 leading us to reject the alternate hypothesis that students would find the two new scripts better than the old script for performing the second task. The explanation for this can be found in the fact that though the opening greeting of WCB was long, it provided instructions on what was needed (for instance the claim and PIN number) to inquire about a claim payment cheque (the task in question). The opening greeting of the other two scripts were shorter but it did not provide such instructions. In the informal feedback after completing the questionnaire, some students actually suggested that they preferred being provided the instructions beforehand so that they can be well prepared to accomplish that task. The result of the G L M test suggested that there was no significant differences between the graduate and undergraduate student responses leading to the rejection of HI . This is demonstrated in the following table with a significance of 0.870 (higher than 0.05) in the GROUP row. 53 Source Type HISS DF Mean Square F Sig. Corrected 0.292 3 9.732 0.331 0.803 Model Intercept 118.03 1 118.03 401.52 0.000 GROUP 8.007 1 8.007 0.027 0.870 SCRIPT 0.284 2 0.142 0.483 0.620 Error 14.7 50 0.294 Total 139.60 54 Corrected 14.99 53 Total Task Number 3. Ask about an adjudicator whose number is known. Questions (a) to (d). Using an IVR is easy Script 2 Labels Percentage Strongly Agree 60% Agree 29% Neutral 10% Disagree 1 % Strongly Disagree Script 3 Labels Percentage Strongly Agree 64% Agree 26% Neutral 6% Disagree 4% Strongly Disagree For task number 3 that could only be performed on the two new systems, the students did not differentiate between the two systems. The proportion of responses in the first two categories was 89% and 90% for Script 2 and Script 3 respectively. Questions (e) to (g). Instructions in the script were easy to understand Script 2 Script 3 Labels Percentage Labels Percentage Strongly Agree 56% Strongly Agree 64% Agree 24% Agree 13% Neutral 13% Neutral 19% Disagree 7% Disagree 2% Strongly Disagree Strongly Disagree 2% 54 Same is true for the construct 'instructions were easy to understand'. The corresponding proportions are 77% and 80% for Script 2 and 3 respectively. The results of the Friedman test and A N O V A follow: FRIEDMAN TEST Mean Rank Script 2 1.53 Script 3 1.47 N 18 Chi-Square 0.91 DF 1 Significance 0.763 Looking at the significance level in the table above, it can be concluded that there is no difference between Script 2 and Script 3. ANOVA Table Source Sum of Squares Df Mean Square F-Ratio P-Value Between groups 0.027225 1 0.027225 0.05 0.8311 Within groups 20.048 34 0.589647 Total (corr.) 20.0752 35 Since the P-value of the F-test is greater than 0.05, there is not a statistically significant difference between the means of the two scripts at the 5.0% significance level. HO is therefore accepted. 55 That there was no difference between graduate and undergraduate student responses is suggested by the results of the G L M test with a significance level of 0.907 for the row labeled GROUP. HI is therefore rejected. Source Type III SS DF Mean Square F Sig. Corrected 3.571 2 1.786 • 0.029 0.971 Model Intercept 86.203 1 86.203 141.954 0.000 GROUP 8.486 1 8.486 0.014 0.907 SCRIPT 2.723 1 2.723 0.045 0.834 Error 20.04 33 0.607 Total 110.36 36 Corrected 20.08 35 Total Task Number 4. Ask for a form to be faxed. Questions (a) to (d). Using an IVR is easy Script 2 Labels Percentage S t r o n g l y A g r e e 6 8 % A g r e e 1 5 % N e u t r a l 6 % D i s a g r e e 8 % S t r o n g l y D i s a g r e e 3 % Script 3 Labels Percentage S t r o n g l y A g r e e 5 7 % A g r e e 2 6 % N e u t r a l 1 3 % D i s a g r e e 4% S t r o n g l y D i s a g r e e For task number 4 the proportion of responses falling in the first two categories is exactly the same for the two new scripts implying that the students did not prefer one over the other for the construct 'easy to use'. However this has to be confirmed by using a statistical test. 56 Questions (e) to (g). Instructions in the script were easy to understand Script 2 Script 3 Labels Percentage Labels Percentage S t r o n g l y A g r e e 68% S t r o n g l y A g r e e 55% A g r e e 19% A g r e e 30% N e u t r a l 7% N e u t r a l 11% D i s a g r e e 4% D i s a g r e e 4% S t r o n g l y D i s a g r e e 2% S t r o n g l y D i s a g r e e For the construct 'instructions were easy to understand' the proportions for the two scripts in the first two categories are close, being 87% and 85% respectively. This again suggests that the students did not prefer one system over the other for this task. The Friedman test and A N O V A results that follow confirm the above results. FRIEDMAN TEST Mean Rank Script 2 1.42 Script 3 1.58 N 18 Chi-Square 0.818 DF 1 Significance 0.366 There is no difference between Script 2 and Script 3 as revealed by the significance level of 0.366. ANOVA Table Source Sum of Squares Df Mean Square F-Ratio P-Value Between groups 0.0266778 Within groups 25.2914 1 34 0.0266778 0.743866 0.04 0.8509 Total (coir.) 25.3181 35 57 Since the P-value of the F-test is greater than 0.05, there is not a statistically significant difference between the means of the various groups at the 5.0% significance level. HO is therefore true. The results of the G L M test are demonstrated below: Source Type III SS DF Mean Square F Sig. Corrected 9.942 2 4.971 0.065 0.937 Model Intercept 89.532 1 89.532 117.158 0.000 GROUP 7.274 1 7.274 0.095 0.760 SCRIPT 2.668 1 2.668 0.035 .0.853 Error 25.22 33 0.764 Total 118.31 36 Corrected 25.32 35 Total Since the significance level is higher than 0.05 in the row labeled GROUP, we can conclude that there was no difference between graduate and undergraduate student responses leading us to reject the alternate hypothesis. Task Number 5. Inquire about WCB hours of operation. Questions (a) to (d). Using an IVR is easy Script 2 Labels Percentage Strongly Agree 69% Agree 29% Neutral 1% Disagree 1% Strongly Disagree Script 3 Labels Percentage Strongly Agree 58% Agree 32% Neutral 4% Disagree 6% Strongly Disagree 58 However, for the last task students preferred Script 2 slightly over Script 3 as revealed by the corresponding proportions of 98% and 90% for Script 2 and 3 respectively for the first two categories. This was also confirmed by the results of one of the statistical tests demonstrated further. Questions (e) to (g). The Script was easy and clear Script 2 Labels Percentage S t r o n g l y A g r e e 8 1 % A g r e e 1 9 % N e u t r a l D i s a g r e e S t r o n g l y D i s a g r e e Script 3 Labels Percentage S t r o n g l y A g r e e 7 0 % A g r e e 1 9 % N e u t r a l 1 1 % D i s a g r e e S t r o n g l y D i s a g r e e The preference becomes even more apparent in the case of the construct 'instructions were easy to understand'. The corresponding figures are 81% and 70% for Script 2 and Script 3 respectively. The results of Friedman's test and A N O V A follow. Mean Rank Script 2 1.25 Script 3 1.75 N 18 Chi-Square 7.364 DF 1 Significance 0.007 ANOVA Table 59 Source Sum of Squares Df Mean Square F-Ratio P-Value Between groups 0.413878 1 0.413878 1.43 0.2401 Within groups 9.84464 34 0.289548 Total (corr.) 10.2585 35 The P Value of 0.241 in the above table suggests that there was no difference in the two new scripts. However, the results of the Friedman's test reveal a significance level of 0.007, which indicates that there was a statistically significant difference leading one to reject the null hypothesis. Notice, however, that the P value was not as high as in the previous cases. It would be helpful to plot the mean intervals using the Fisher's least significant difference (LSD) procedure to discriminate among the means of the two scripts. This graph is generated as part of calculating the A N O V A table. M e a n U s a b 1.7 1.B 1.5 1.4 1.3 1.2 1.1 Means Plot with 95.0% LSD intervals i • < i • • i i * * ' ' i I I II I I r I I I I 2.0 3.0 Script Axes j Options Interpret Row: X: Y: Figure 7. Plot of mean intervals using the Fisher's least significant difference 60 As can be seen above even though the mean intervals overlap in the case of two scripts, the range of overlap is much smaller compared to the other graphs that were obtained (not shown in the chapter) for the previous two tasks. The chi-square results show that there is a difference in the two systems. Script 2 is preferred over Script 3. This might be because Script 3 was 'role specific' and Script 2 was 'task specific'. The students preferred Script 2 over Script 3, which was more direct and specific and got them immediately to the task that they wanted to accomplish. This is also supported by the informal conversations the researcher had with the students after they had completed the questionnaire. Most students stated that they felt annoyed by the script which asked them to identify themselves as injured workers, employers, or doctors which was an unnecessary step when all they wanted to know was WCB's hours of operation. As in the previous cases, the results of the G L M test showed that there was no difference in the graduate and undergraduate student responses as the significance level was found out to be 0.892. The alternate hypothesis is therefore rejected. 5.3 Conclusions from the Student Questionnaire This section discusses the conclusions reached from designing the new IVR system and the results of the student questionnaire. It was clear from the statistical results of the student questionnaire that a) Students preferred the two new scripts over the WCB script for one task. b) There was no difference between Script 2 and Script 3. However, for the purpose of inquiring about the hours of operation of WCB, students preferred Script 2 which was 'task specific' and hence immediately led them to their specific activity from the opening menu. 61 It was also clear that most students would have likely used an IVR instead of an operator for performing the required tasks. This is presented in the table on the next page that contains the results of the last question on the survey. In the table, T denotes Task and S stands for Script. The numerical suffix after each letter is the number of the task or the number of the system it was performed on. For instance T2S3 means task 2 on system 3. Combination IVR Operator T1S1 13 5 T1S2 15 3 T1S3 15 3 T2S1 17 1 T2S2 16 2 T2S3 15 3 T3S2 17 1 T3S3 17 1 T4S2 15 3 T4S3 15 3 T5S2 15 3 T5S3 14 4 Table 6. Result of student responses on IVR vs. Operator These figures further substantiate the necessity of designing a good IVR system so that it can provide superior service to the users of the system. It is clear that a good IVR design is essential to encourage its use by the callers. That even university students who are more conversant with using technology had trouble interacting with WCB's IVR (or found easier interacting with alternate scripts) suggests a need to make it simpler for WCB's clients to use. It will not only encourage customers to use the IVR both within and outside business hours but also enable WCB to serve customers better by reducing the wait time for customers who need the help of a CSR during peak business hours. Designing an effective IVR system is not an easy task. A standard solution cannot be provided to organizations. The solution has to be customized to each individual 62 organization and a thorough research needs to be conducted to understand the nature of the business and the type of calls that a business gets. However there are good design guidelines that have been developed through previous research on IVR systems and those were incorporated in this study. Through the results of the student questionnaire evidence was provided that the new scripts developed in this study were better than the existing W C B script. 63 C H A P T E R 6. C O N C L U S I O N S A N D R E C O M M E N D A T I O N S This chapter provides concluding comments, discusses the limitations of this empirical study, and provides suggestions for future research in this area. 6.1 Concluding comments This thesis was the result of a study conducted for the call-centre at the Workers' Compensation Board of British Columbia (WCB). WCB had recently opened a call-centre to centralize all calls related to claim inquiries from the lower mainland area. The management at WCB wanted to provide an efficient customer service while streamlining the flow of calls coming to the call-centre. The motivation for this research was to suggest an effective IVR system to W C B that could address some of the problems on routine and transferred calls and help management meet their objectives of efficient customer service. The applied thesis was taken as an opportunity to explore two research questions: whether the suggested guidelines found in literature for a good IVR system would be effective when implemented in practice, and whether there are differences in people's preferences over hearing a person or a task-specific opening menu. The reason 'usability' was used as a dependant variable to demonstrate an effective IVR system was because usability would affect usage and a higher usage would result in the expected benefits for WCB and address some of the problems. As was discovered in the statistical analysis of the student questionnaire the new scripts were found to be usable by the student populations which would increase the likelihood of they being actually used. Although research has been conducted in the use of voice response systems in the past resulting in recommended guidelines for a good IVR system, they were not sufficient for WCB. The literature provided standard industry guidelines but there was no guarantee that they will work in the case of WCB. The nature of WCB's business is different from 64 purely commercial organizations and there were little guidelines for designing good IVR systems for not-for-profit organizations. The customers of WCB are mostly injured workers and it was necessary to be sensitive to their particular needs while designing an IVR system for them. It was essential to identify the nature and pattern of calls to W C B and then suggest a solution that will work for the organization. The literature did not provide evidence of whether introducing additional options in the IVR would address the problems for which they were developed. Such conclusive evidence was provided in this study by testing the new scripts on a sample of students. It was demonstrated in this study that the suggested guidelines, after taking into account the company's unique requirements, could potentially help to streamline the call flow for the company. The use of simple, brief, and clear instructions could potentially lead to an encouragement in the use of IVR. It was also discovered that the student sample did not make a distinction between the 'role specific' and 'task specific' script. Therefore, it can be concluded that whether callers would prefer a 'role specific' or a 'task specific' script would depend on the purpose of the call. A n additional contribution of this study was the development of a fully configurable IVR simulation that can be utilised in future studies. This is especially important as the experiment can be replicated at different times with a different sample to gather further evidence for the results discovered in this study. 6.2 Limitations of the study This study would have the same concerns raised by Benbasat [20] on the degree of realism as any study in information systems area that is conducted in a laboratory setting using student subjects. The participants in this study were aware that they were interacting with a computer-based IVR simulation. Even though the option to speak to a live agent was mentioned in the computer-based simulation, they were aware that it was 65 not possible to speak to a person and they would complete the tasks using the IVR simulation. The reasons for selecting the laboratory method over the field and using students as opposed to real clients of WCB have been discussed in chapter 4. Stone [23] suggests that the range of criterion to which a set of laboratory findings is applicable is limited. This is also true of this study. The fact that student subjects were employed makes the results of the study applicable to only those actual and potential WCB clients that are students. The results of this study cannot be generalized to the entire WCB clientele. The threats to construct validity were reduced by utilising a previously developed research instrument by Moore and Benbasat. However that instrument was used only for the 'ease of use' construct on the student questionnaire. The construct 'ease of understanding' was developed along similar lines but due to the limitations of time it could not be empirically tested. 6.3 Suggestions for future research The limitations stated in the previous sections however can be addressed and that can be the subject for future research. Cook and Campbell [19] suggest that external validity can be enhanced by both replications within a study and across study. An important contribution of this thesis was the development of a computer-based IVR simulation that is fully reconfigurable and can be used in future studies. As was mentioned in chapter 4, due to the limitations of time and money a larger sample of students could not be gathered. This study can be replicated in future with a larger sample size to confirm the results that were obtained from this study. Such a replication is possible not only for this study but also across studies by recording new sound files and configuring new simulations. 66 The reconfigurable IVR simulation can act as a pilot for implementing a full-scale IVR system. The prototype IVR can provide important insights into the parts of the script where subjects encounter problems, which can then be further refined. 67 Bibliography [I] Long, G. G. (1993) "How your telephone system can help your sales staff, Office Systems, January 1993, pp. 19. [2] Huguenard, B. R. (1993) "Working Memory Failure in Human - Computer Interaction: Modeling and Testing Simultaneous Demands for Information Storage and Processing (User Errors, Interface Design)", Ph.D. Thesis, Carnegie-Mellon University 204 pp. [3] Bradley, G., (1996) "Improving Interactive Voice Response", Enterprise Integration Group. 250 pp. [4] DePalma, D. E., Goodman, J. A . (1995) "Key Factors to Successful Implementation and Acceptance of Automated Response Systems: 1995 Benchmarking Results", Technical Assistance Research Programs (TARP), Inc. [5] Mehrotra, V . (1997) "Ringing up Big Business", OR/MS Today, August 1997, pp. 18-24. [6] Nainis, S. W., Agresti, W. W. (1996) "Planning Guidance Document for Remote Initial Claims", Appendix K: Script Development for Remote Initial Claims, Information Technology Support Center, State of Maryland. [7] IVR Tips and Strategies, http://www.alpines.eom/VrTips.htm [8] Avoiding Common IVR Pitfalls, http://www.alpines.com/VrMistakes.htm [9] Interactive Voice Response Forum, http://www.christell.com/ivrboard.htm [10] IBM Call Center Homepage, http://ncc.hursley.ibm.com/callpath/ [II] Benchmarking Example - Call Center Voice Response http://www.olccinc.com/OnLineCustomerCare/biceval.html [12] Binford, G. (1996), "IVR Delivers for Fed Ex", Communications News, October 1996, pp. 16. [13] Scanlon, C. (1996), "Pluses and Pitfalls in voice mail", Nation's Business, May 1996, pp. 57. 68 [14] Cox, B. (1995), "The sky's the limit with Interactive Voice Response", National Underwriter, Cincinnati, January 16, 1995. [15] Hatlevig, T. (1995), "Let your fingers do the talking", Personnel Journal, August 1995, pp. 105- 108. [16] Anonymous, "IVR Tips", Byte, May 1996. [17] Linthicum, D. S. (1996), "Make Voice Response Sing", Byte, May 1996. [18] "Top Ten Tips", http://www.telecomlibrary.com/db area/archives rd/CallCenter/1996/January/3 247 348 .htm [19] Cook, T. D., and Campbell, D.T. (1979) "Quasi Experimentation: Design and Analysis Issues for Field Settings", Boston: Houghton Mifflin. [20] Benbasat, I. (1998) "Laboratory Experiments in Information Systems Studies with a Focus on Individuals: A Critical Appraisal", Harvard Business School Research Colloquium. [21] Campbell, D.T., and Stanley, J.C. (1966) "Experimental and Quasi-Experimental Designs for Research" Chicago: Rand McNally. [22] Moore, G.C, and Benbasat, I. (1991) "Development of an Instrument to Measure the Perceptions of Adopting an Information Technology Innovation", Information Systems Research, September 1991, pp. 192-222. [23] Stone, E. (1978) "Research Methods in Organizational Behaviour", Glenview, Illinois, Scott Foresman. [24] Hartwick, J., and Barki, H. (1994) "Explaining the Role of User Participation in Information Systems Use", Management Science, 40, 440-465. [25] Fishbein, M . , and Ajzen, I. (1975) "Belief, Attitude, Intention, and Behaviour: A n Introduction to Theory and Research", Addison-Wesley, Reading. [26] Davis, F.D. (1989) "Perceived Usefulness, Perceived Use and User Acceptance of Information Technology", MIS Quarterly, 13, 318-339. 69 [27] Ajzen, I. (1991) "The Theory of Planned Behaviour", Organizational Performance and Human Decision Processes, 50, 79-211. [28] Ashton, R.H., and Kramer, S.S. (1980) "Students as Surrogates in Behavioural Accounting Research: Some Evidence", Journal of Accounting Research, Vol . 18, No. 1, Spring 1980, 1-15. [29] Hawkins, H.I., Albaum, G., and Best, R. (1977) "An Investigation of Two Issues in the Use of Students as Surrogates for Housewives in Consumer Behaviour Studies". Journal of Business, April, 1977, 216-22. 70 Appendix 1 * I M P O R T A N T : D A T E & T I M E This data is being collected by the Centre for Operations Excellence to gather information on the nature and pattern of calls for the case management units. We are particularly interested in calls transferred from the call centre and incoming calls on DIDs. This information will be used to design an efficient IVR system that could potentially handle most of the routine type of calls that the case management and the call centre currently get. In the following table please place a tick in the corresponding row and column for every call you get. For example, you get a call from a claimant and that comes on your direct line please put a tick in the row claimant and column DID. If you get a call from an employer which was transferred from the call centre please put a tick in the row Employer and column Transfer from Call Centre. Thank you very much for your co-operation. DID Call Source Transfer from Call Centre Other 1 Claimant 1 Health Care Prof. Employer Internal | Other 71 Appendix II R E S U L T S OF D A T A C O L L E C T I O N A T T H E C A L L C E N T R E This Appendix describes the results of data collection at the call centre II . l Division of time spent on various activities during phone shifts • I n c o m i n g c a l l s • O u t g o i n g C a l l s • W r a p U p • P a u s e * • O t h e r P a u s e * • Co f fee b r e a k s Figure II. 1 Percentage of time spent by CSRs on various activities during phone-times * Pause and Other Pause explained in previous chapter. 1. the CSRs spend about 60% of their 3.5 hour shift on phones, 2. 9% of the time is spent on wrap-up 3. a very small proportion of the phone times is spent on making outgoing calls. * Note that the time spent on outgoing calls made during the time an incoming caller is on hold is still considered a part of that incoming call. 72 II.2 Call Analysis for Richmond by Time of Day This section presents information on the different types of people calling the Richmond call centre by the time of the day. These times are divided into four categories: morning -8 to 12, lunch - 12 to 13, afternoon - 12 to 17 and evening - 17 to 19. II.2.1 Who is calling the Call Centre? Figure II. 2 Calls by people by time of the day From the above graph it can be deduced that: 1. the CSR's spend on average 70% of their approximately 2.5 hours of actual phone time on calls with claimants, 2. 20% on average with Employers, 3. 10% on average with Doctors, 4. minimal time on other calls, 5. the pattern of who is calling remains constant throughout the day, and 6. during the evening most calls come from the claimants as most Employers and Doctors' offices generally close by 5 pm. 73 II.2.2 What are the incoming calls about? Morning Lunch Afternoon Time of the day Evening Claim Inquiry Cheque Inquiry • Doctor's Billing • Fax Inquiry • General • Transfer To Figure II. 3 Types of calls by time of the day 1. 40% of the CSR's actual phone time (approximately 2.5 hours) is spent on calls related to claim inquiry, 2. 20% on calls general in nature, 3. 20% on transferring calls to various departments within WCB, 4. 12% on calls related to a cheque inquiry, and 5. 8% on combined fax and doctor's billings inquiry. Note that general inquiry calls increase during the evening. Since other WCB departments are closed by this time, CSR's service these additional calls. 74 II.2.3 Where do the transfers go? Figure II. 4 'Transfers To' by time of the day 1. 60% of the calls are transferred to Adjudication (Case Management), on average 2. 20% of the calls are transferred to Entitlement on average, and 3. transfers to adjudicators decrease over the day 4. transfers to entitlement increase over the day. 75 II.3 Call Analysis for Richmond by Day of the Week The following graphs now present the same information for the call centre in Richmond by day of the week. H.3.1 Who is calling the Call Centre? • Claimant • Doctor • Employer O Other Monday Tuesday Wednesday Thursday Friday Day of the week Figure II. 5 Calls by people by day of the week 1. the "Who is calling?" pattern remains stable over the week, 2. there is insufficient data for Thursday which may account for the slight deviation from the trend. 76 II.3.2 What are the incoming calls about? Figure II. 6 Type of calls by day of the week 1. claim inquiries rise steadily over the week. There is insufficient data for Thursday, which may account for the slight deviation from the trend, 2. 10% of the calls are related to cheque inquiry, and 3. 15 to 20% of the calls are related to transfers. 77 II.3.3 Where do the transfers go? • Accounts • Adjudication • Appeals • Assessment • Dissability • Entlitement • Media Monday Tuesday Wednesday Thursday Friday Day of the week Figure II. 7 'Transfers To' by day of the week 1. there is a trend for transferred calls to Adjudication (Case Management) and Entitlement during the week as well as during the day. 2. calls transferred to Entitlement increase from 14% to 65% during the week. 3. calls transferred to Adjudication decrease from 50% to 10% during the week. As only 2% of the phone-times is spent on making outgoing calls, graphs on the nature and type of the outgoing calls were not constructed. 78 II.4 Detailed Analysis of transfers This section presents a comprehensive analysis of calls which either request or end in transfer. II.4.1 How many calls get transferred? pi. • Caller asks for a transfer and gets the transfer • Caller asks for a transfer and doesn't get the transfer • Caller doesn't ask for a transfer and gets transferred • Caller doesn't ask for a transfer and doesn't get transferred Figure II. 8 Breakdown of Transfers 1. a client may ask for a transfer and the call may not end in a transfer. 2. a client may not ask for a transfer; however, ends in a transfer. 3. calls ending in a transfer are approximately 30% of the total calls monitored. 7'> II.4.2 Average length of transfers 0:03:36 0:02:53 E E 0:02:10 0:01:26 0:00:43 0:00:00 c ZJ o o o < c o CO o < JO CO CD CL < c CD E CO CO CD CO CO < CO CO CO b c CD E a> LU co •o CD CD Type of transfer Figure II. 9 Average length of Transfers Note that the above graph is only for calls that specifically asked for a transfer. It shows that the average length of time spent on calls before they are transferred. 1. time spent on transferring calls to Adjudication or Entitlement is higher than the time spent on calls transferred to the other departments at WCB 2. time spent on category "nowhere" is the highest as it denotes that the call which initially asked for a transfer was closed successfully by the CSR. 3. minimum time spent on a transfer call is 1 minute and 24 seconds. 80 II.5 Wrap-up Analysis II.5.1 Average Service Time (Call Handle + Wrap-up Time) 0:03:36 Claim Cheque Doctor's Fax General Transfer Inquiry Inquiry Billing Inquiry To C a l l Type Figure 11.10 Analysis of 'Wrap-up' 1. four call types: Claim Inquiry, Cheque Inquiry, Doctor's Billing, and General have average service times in excess of two and a half minutes. 2. the two claim types with the longest wrap-up times are Claim Inquiry and General 3. wrap-up times are disproportionate for each call type. 81 11.5.2 Distribution of time spent on each of the two components of service time Claim Cheque Doctor's Fax General Transfer Inquiry Inquiry Billing Inquiry To Call Type Figure 11.11 Service and Wrap-up times 1. the largest proportion of wrap-up time is for claim inquiry and general, an average of 15% of total service time. 2. the smallest proportion of wrap-up time is for fax inquiries, an average of 5% of total service time. 82 II.5.3 Average Wrap-up time. 0.00:35 000:30 + 000:26 + fi 0.0022 -s £ 000:17 -00009 -000:04 -000:00 -I Claim Inquiry Cheque Inquiry Doctor's Fax Inquiry General Transfer To Billing Call Type Figure II. 12 Average Wrap-up time 1. the average time spent wrapping up claim inquiry is greater than 30 seconds, the highest of all the call types 2. Wrap-up time for calls that request transfer occurs when the CSR responds to the request without transferring the call II.6 Calls suitable for IVR A detailed analysis was then undertaken to categorize calls that would be suitable for handling by an IVR system. Calls where the research assistants had taken extensive comments were identified out of the sample of calls on which data was collected. These calls were then categorized based on the comments that were recorded by the research assistants. It was then ascertained whether these types of calls could potentially be successfully handled by an IVR system. Such an analysis was helpful in the development of the actual wording of the script which is described in the next chapter. The following table depicts this categorization: 83 TOTAL 383 %of Total IVR? COMMENTS CLAIMENT CLAIMS Status 48 12.5% Y / N Queries about status of claim. No remarks regarding actual status. A claim might be accepted and then IVR could replace CSR, it might have been pending and then CSR is needed. Information Missing 7 1.8% Y Queries about status of claim. The problem was missing information. Other 44 11.5% N Queries about claim, for instance, reopening, etc. Must talk to CSR. Claim # 16 4.2% Y Called to find out what their claim number is. Pan 1 0.3% N Called to find out P A N . PAYMENTS Inquiry 13 3.4% Y / N Payment queries. No remarks regarding actual status of payment. A cheque might be in the mail so IVR could replace CSR, i f not, CSR is needed Out 6 1.6% Y Cheque is in the mail. Information Missing 2 0.5% N Problem with payment. Missing information. Must talk to a CSR. Other 15 3.9% N Queries about payment. Must talk to CSR for various reasons. EMPLOYER 26 6.8% N A l l calls that needed to be answered by CSR.* DOCTORS PAYMENTS 4 1.0% Y Was the cheque mailed to me? When? OTHER 11 2.9% N A l l calls that needed to be answered by CSR. TRANSFERS Asked for 61 15.9% N "May 1 speak with Nancy/My case management..." Don't have Ext. # Didn't ask for 20 5.2% N Cases that needed to be transferred to case manager/Adjudicator Voice Mail 6 1.6% Y When the call was only to give information that could have been left on a voice mail. Actual transfers to voice mail are included in transfers (they only happened because no one answered the phone) OTHER Fax# 20 5.2% Y Also includes information about phone # for other departments. Forms 15 3.9% Y Fax-on-Demand Follow-up 12 3.1% Y "Did you receive the fax that I just sent?" General Info.-IVR 19 5.0% Y Address, Hours, How to open a new claim.... General Info.-Other 37 9.7% N "Who is my case manager?", "Is my adjudicator on vacation?", "How to fill a certain form",... • There were few calls from Employers that could be suitable for automation and those have been incorporated under category Claimant above. • At least 21.5% of calls could have been dealt with by the proposed IVR. • Maximum Number of calls that could have been dealt with by IVR is approximately 44%. • 9% of calls require a Fax-on Demand system. A n additional 3% (Follow-up) might also be handled by the system. ' Table II. 1 Calls suitable for an IVR 84 The above table groups calls coming from Claimants, Doctors, and Employers based on the purpose of the calls. Transferred calls are identified separately depending on whether the caller asked for a transfer or the call ending in a transfer even if the caller had not specifically asked for it. Most of the 'other' type of calls can potentially be handled an IVR as they are requests for information. The second column in the table depicts the total number of calls in each category and the next column shows the proportion of such calls out of the sample of calls that were identified for such an analysis. The fourth column shows whether such category of calls will be suitable for automation or not and the last column contains brief comments on each type. 85 Appendix III Discussion of the results of data collection at Entitlement Incoming calls A total of 78 calls were monitored while sitting with the entitlement officers. From the graph below it can be seen that: Incoming calls by categories of people Number of incoming calls Claimant Health Employer Internal Other Care Categories of people Figure III. 1 Incoming calls by categories of people 1. 53% (41) of the calls received by the Entitlement officers are from claimants 2. Calls from Employers, Health Care Professionals, internal departments, and others are equally divided and range anywhere from 10 to 15% each. 86 Breakdown of incoming calls by source: Breakdown of incoming calls by source 14% 3% • DID • Transfers from CC • Other 83% Figure III. 2 Breakdown of incoming calls by source The above graph shows that: 1. 83% of the calls come on the DIDs 2. 14% of the calls received by the Entitlement officers are transfers from the call centre 3. 3% of the calls come from other sources such as internal, etc. 87 Average time spent on calls with various people Average time spent on calls with various people Time 0:00:00 Claimant • Health Care • Employer • Internal • Other Categories of people Figure III. 3 Average time spent on calls with various people The graph above shows that: 1. The longest time is spent with the claimants on the phone on an average followed by Employers and Health Care Professionals Average time spent on different type of calls with different people: A v e r a g e t i m e s p e n t o n d i f f e r e n t t y p e o f c a l l s w i t h d i f f e r e n t p e o p l e 0 :07 :1 2 0 : 0 5 : 4 6 0 :04 :1 9 T i m e 0 : 0 2 : 5 3 0:01 :26 0 : 0 0 : 0 0 iSf 0 the r C la im Inq uiry N a tu re o f c a l is IC la im Inq uiry l O t h e r C a t e g o r ie s o f p e o p le Figure III.4 Average time spent on different types of calls with different people 88 The above graph reveals that: 1. The longest time is spent with claimants on claim inquiry followed by 'other' calls from claimants 2. There were no calls related to cheque inquiries in the sample of calls that were monitored. I n c o m i n g calls Incoming calls by categories of people N u m b e r o f ca l ls Claimant Health Employer Internal Other Care C a t e g o r i e s o f p e o p l e Figure III. 5 Incoming calls by categories of people 1. 24 (52%) of the incoming calls that were monitored are from claimants 2. 9 (20%) and 6 (13%) of the incoming calls that were monitored are from Health Care Professionals and Employers respectively 3. 5(11%) are internal calls 89 Breakdown of incoming calls by source Breakdown of incoming calls by Source • DID • T s f f r o m C C • O t h e r 86% Figure III. 6 Incoming calls by source The above graph shows that 1. 86% of the calls come in the DIDs 2. 7% each are transfers from call centres and other sources respectively. 90 Average time spent on calls with various people Average time spent on calls with various people Figure III. 7 Average time spent on calls with various categories of people 1. The average time spent on calls with Employers is 4 minutes and 42 seconds 2. The average time spent on calls with Health Care Professionals approximately 4 minutes and with claimants is approximately 3 minutes. 91 Average time spent on different type of calls with different people: Average time spent on various type of calls with different people Time 0 : 1 4 : 2 4 0 : 1 1 : 3 1 0 : 0 8 : 3 8 0 : 0 5 : 4 6 0 : 0 2 : 5 3 0 : 0 0 : 0 0 0> .—' O t h e r r C h e q u e Inqu i r y C l a i m Inqu i r y • Claim Inquiry • Cheque Inquiry • Other E LU -t—' O Type of call Categories of people Figure III. 8 Average time spent on various types of calls with different people It can be seen in the above graph that 1. The average time spent with an Employer on a claim inquiry is the highest, 4 minutes and 42 seconds, followed by an average of approximately 4 minutes on a claim inquiry from a claimant. 2. The data from 'other' type of call from 'other' is skewed as there were only 2 observations and one observation was for over 20 minutes. 92 SCRIPT 1 Welcome to the Workers' Compensation Board. Please choose from one of the following options. If you have an inquiry regarding a claim press 1 now If you are calling about payment information press 2 now If you know the four digit extension number of the person you wish to speak to press 3 now If you need general information or forms press 4 now 93 1 If you are inquiring about the status of your claim press 1 now If you require more information on your claim you may press 0 at any time to speak to a Client Service Representative 1st Timeout I Please enter your 8 digit claim number now You did not enter a response. Please try again. Enter the 8 digits from your claim number now 2 n d Timeout Less or more digits or invalid characters This entry is invalid. Your claim number must have exactly eight digits You did not enter a response. Please try again. Enter the 8 digits from your claim number now Claim number xxxx. If this is correct press 1 now. To enter the number again press the star key Please standby while we retrieve your claim information The claim number that you entered does not match our records Not found Please stay on the line and your call will be answered by a Client Service Representative Found Claim information is current to Y Y : M M : D D We require more information to process this claim P,S,I,B Info only The claim has been registered but we need the following documents to process this claim: 1 Doctor's Report 2 Employer's report of injury 3 Worker's form 4 Any other A decision will be made as soon as these documents are received Disallowed/Rejected Health Care only Accepte Interim Adjud The claim has been disallowed for the following reasons: 1 2 The claim has been accepted for health care purposes only as there has been no time-loss on the claim To go back to the main menu press the star key To speak to a CSR press o or simply stay on the line If you have no further questions you may hang-up now 94 2 If you are an injured worker press 1 now If you are a Doctor press 2 now Please enter your Personal Access Number Please enter the claim number P A N xxxx. If this is correct press 1 now. To enter the number again press the star key Claim number xxxx. If this is correct press 1 now. To enter the number again press the star key To CSR The personal access number that you entered does not match our records. Please stay on the line and your call will be answered by a client service representative Not found Not found Found The claim number that you entered does not match our records. Please stay on the line and your call will be answered by a client service representative To CSR To CSR Cheque issue date is older than 30 days. Please wait and your call will be answered by a client service representative Payee1 The most recent cheque was processed on Y Y : M M : D D in the amount of $x. It was mailed on the following business day No payment has yet been issued on this claim For Worker Yes Yes The most recent cheque was processed for your employer on Y Y : M M : D D in the amount of $x. It was mailed out on the following business day 95 For Doctor Payee 3 The most recent cheque was processed on Y Y : M M : D D in the amount of $x. It was mailed on the following business day To go back to the main menu press the star key; To speak to a CSR press o or simply stay on the line; If you have no further questions you may hang-up now Please enter the four digit extension number now If you wish to speak to the person at this extension press 1 now If you wish to leave a voice mail message for this person press 2 now Please leave a voice mail message after the beep. When you are finished you may hang-up Yes Direct Connection Play voice mail message 96 If you require a form to be faxed to you press 1 now If you are a first time caller press 2 now If you require general information and information on hours of operation press 3 now If you are an employer press 1 now If you are a doctor press 2 now If you are an injured worker press 3 now r Please enter your fax number beginning with the area code and then hang-up r Fax either form 6, 7, or 8 based on the input choice If you are an injured worker you need to send us form 6 If you are a doctor you need to send us form 7 If you are an employer please mail or fax form 8 The Call Centre is open from 7 am to 7 pm Monday through Friday. You may use our automated inquiry system for claims and payment information at any time. To go back to the main menu press the star key To speak to a CSR press 0 or simply stay on the line If you have no further questions you may hang-up now 97 S C R I P T 2 Welcome to the Workers' Compensation Board. Please choose from one of the following options. If you are an injured worker press 1 now If you are a doctor press 2 now If you are an employer press 3 now If you have an inquiry regarding a claim press 1 now If you are calling about payment information press 2 now If you know the four-digit extension number of the person you wish to speak to press 3 now If you need general information or forms press 4 now If you have an inquiry regarding a claim press 1 now If you are calling about payment information press 2 now If you know the four-digit extension number of the person you wish to speak to press 3 now If you need general information or forms press 4 now Proceeds in the same way as the previous Script If you have an inquiry regarding a claim press 1 now If you know the four-digit extension number of the person you wish to speak to press 3 now If you need general information or forms press 4 now 98 Appendix V IVR Simulator By Arash Ramin IVR SIMULATOR 99 OVERVIEW 100 I N T R O D U C T I O N : 1 0 0 M E N U I T E M S 101 CONFIGURATION 102 O U T P U T 1 0 3 I N P U T 1 0 4 E R R O R H A N D L I N G 1 0 5 CODE STRUCTURE 106 T O O L S 1 0 6 C L A S S E S 1 0 6 KNOWN ISSUES 107 L I M I T A T I O N S 1 0 7 B U G S 1 0 7 CONTACT INFORMATION 107 99 Overview Introduction The IVR Simulator is capable of supporting multiple simulations, each with completely different configurations. Each simulation's parameters and states are stored in a database. The simulator is a state-based system, meaning that while running the system only is only aware of the current state. Each simulation starts off in the default 'START' state, and will make the appropriate transitions from one state to the next until the ' E N D ' state is reached. Each state consists of output (sound files), input (user touch-tone entries), and error-handling (timeouts or invalid responses) modes. When the system enters a new state, it plays any specified output sounds and then accepts input as specified by the current state. Based on that input, it can then make a transition to the next state, until finally the ' E N D ' state is reached, which marks the end of the simulation. If no input (or invalid input) is made, then the system enters the timeout states as specified by the error handling parameters of the corresponding state. Figure 1 shows the end-user interface for the IVR Simulator. The status bar at the bottom shows the current status of the simulation (the left indicates which simulation is currently open, and the right shows the state of the simulation itself - STOPPED, PAUSED, or RUNNING) When a simulation is in the running state, user input may be made by either clicking on the numeric buttons with the mouse, or by using the keyboard's keypad. I IVR S imu la to r File Simulation Help 'WCB Script 81.DAT r r r a 1 J8,' 3 4 5 6 7 8 9 * J I ' 'STOPPED Figure V. 1 IVR Simulator user interface 100 Menu items A l l menu items are context-sensitive, meaning that they appear enabled when they are valid options or disabled (greyed out) when invalid. For example, the SIMULATION -> STOP menu item is only valid when a simulation is currently running. Figure 2 shows the File Menu, which allows you to open/close existing simulations, or create new simulations altogether. ItS, IVR Simulator | File Simulation Help New Simulation Open Simulation Close Simulation Exit Figure V.2 File Menu Figure 3 shows the Simulation Menu. This allows the operator to start, stop or pause a configured simulation, or to change the configuration of the simulation. mm Help i Simulation Start Ctrl+S Configure Drl+C Figure V.3 Simulation Menu 1 0 1 Configuration Figure 4 shows the simulation's configuration window. The left pane shows the simulation's different states, and the right pane allows you to configure the output, input and error handling modes of each state. Each simulation has two default states - a START state, and an OPERATOR state. Simulation Configuration { O P E R A T O R Name Description [START j Default start state Output | Input ] Error Handling | Output 1 j NONE C BUFFER C SOUND Output 2 C NONE r BUFFER C SOUND Output 3 <? NONE r BUFFER C SOUND OK Figure V. 4 Configuration dialog 102 Output Each state starts in the output mode, which is generally just a sound (wave) file. Sometimes it might be necessary to have a combination of a multiple sound files, or maybe a sound file, followed by the input buffer (explained below), followed by another sound file. The example below is such a case (sound file + input buffer + sound file): "The claim number entered was " + 35654 + "if this is correct, press 1" Up to three different output properties can be defined. [0^ P>5| Input | Error Handling | - Output 1 1 C NONE r BUFFER SOUND : l 1 w a v " J L J r Output 2 | | <? NONE r BUFFER C SOUND Output 3 1 I a NONE r BUFFER C SOUND - i J Figure V. 5 Output Configuration 103 Input Following the output mode, the state enters an input mode. There are 4 different types of input: • NONE - The system does not expect any input, and makes an immediate transition to a specified state. • SINGLE - A response to a query. The system will process the input as soon as a single entry is made. Each number (0 - 9, *, #) can be configured with a resulting state (numbers without resulting states defined become invalid entries). This is how the program switches from one states to the next. • MULTIPLE - Recording to the input buffer. The system will process multiple entries (anywhere from 1 - 20), after which it will enter a resulting state. Presumably the resulting state will playback the input buffer at some point in the output mode. This might be useful for asking the user to enter their account number. • END - The system does not expect any input, and the simulation ends. D^Q| Error Handling | r Input Type i | C NONE d1 SINGLE C MULTIPLE C END No input (transition to newt state) Single (query response} 0 OPERATOR 6 f 1 A01 ' 1 8 1 3 9 4 * 1 5 | # ! f Multiple (record to input buffer) Figure V. 6 Input Configuration 104 Error Handling If the user does not make any input after a set number of seconds (specified by the timeout length) or an invalid entry, then the system will make a transition to the timeout state (by default this is the same state, which in effect repeats the same instructions). After two timouts with either no response or an invalid entry, then the system makes a transition to the timeout end state (by default the operator state). Both the timeout and timeout end states can be changed for each state. Output) Input ! Error Handing il Timeout length 15 J^j Timeout state START T imeout end state ) 0 PE RAT 0 R Timeout states are also used for invalid responses. The 3 r d timeout/invalid response will result in the 'timeout end state' Figure V. 7 Error Handling 105 Code Structure The system is written in C++, with object oriented principles in mind. It is dependent on the V C L (Visual Component Library) framework of Borland C++Builder and the native Win32 API. The underlying database engine (DBISAM) is a custom Delphi/C++Builder solution which compiles directly into the executable (300k overhead). D B I S A M integrates directly into the C++Builder data-aware components (as TDataSet descendants) offering functionality similar to TTable and TQuery. Tools Compiler : Borland C++Builder 3.01 - http://www.borland.com/bcppbuilder/ Database Engine : D B I S A M 1.13 - http://www.elevatesoft.com Classes The code is documented throughout. There are 4 core classes to the IVR Simulator: • TSimulationForm - the user interface (Figure 1). • TSimulationConfigForm - the configuration dialog (Figure 4). • TSimulationDB - the simulation database class, to create, modify and delete simulations. • CSimulation - the simulation engine. These classes are also dependent on several third party V C L classes (specifically the formsC unit, also authored by myself) and can be found on my web site (see below). 106 Known Issues Limitations Due to time constraints, the following limitations were imposed: • Only 3 output properties can be specified in the output mode (figure 5). Multiple states can be created for output (with the first state not accepting any input) i f more output properties are necessary. • The multiple input mode always makes a transition to the same state. There is no cross-referencing of the input buffer against the database to change to different states. • There is no cut through support. Bugs Deleting a state which has other states dependent upon it will result in an unstable simulation. The IVR Simulator does not check i f a state is required by the simulation before deleting it. This is the responsibility of the user. There is however state-based error checking before a state is created or changed (to ensure that all output, intput and error handling parameters have been properly defined). Contact Information My e-mail address may change over the years, however my web site will always list my current contact information. http://www.arash.bc.ca 107 Appendix VI Laboratory study on Interactive Voice Response (IVR) scripts for the Workers' Compensation Board The Workers' Compensation Board of British Columbia deals with the settlement of claims for work-related injuries on behalf of the injured workers. The injured workers typically call the WCB to inquire about the status of their claim or any claim-payments. The other two group of people who call the WCB are the doctors who provide information on the nature of injuries and inquire about payments for doctor's billings, and the employers who provide details of incident leading to the injury at work. This study is designed to test the utility of different scripts for an Interactive Voice Response (IVR) system for the call centre at WCB. Your participation in this study is purely voluntary and you may withdraw your participation at any time without penalty. The responses provided by you will be used for statistical analysis and only the aggregate results will be published. The results are not attributable to any one individual. Your confidentially is assured and we will appreciate your honest responses as you navigate through the alternative scripts provided to you. Please confirm your agreement to participate in this study by signing below: Name Signature Date Thank you very much for your co-operation. 108 Instructions You will be navigating through three computer- based simulations of an IVR system. The objective is to accomplish the following tasks: 1. You want to inquire about the status of a claim 2. You want to inquire about a claim cheque payment 3. You want to talk to an adjudicator whose extension number you know 4. You want to get a form faxed to you 5. You want to inquire about the hours of operation of WCB Imagine yourself to be a claimant who wants to get these five tasks accomplished. You will complete the first two tasks on three different IVR systems and the next three tasks on only two of the three different IVR systems. After completing each task on each system you will be required to respond to a question on a questionnaire. The researcher will be present during the whole simulation to run and pause the three systems and to assist you with filling out the questionnaire. Your responses should be based simply on your experience with the three different scripts. 109 Questionnaire For questions 1 through 5 please circle one of the available choices: 1. For inquiry about the status of a claim a) My interaction with the Interactive Voice Response System was clear and understandable 1 2 3 4 5 Strongly agree Strongly Disagree b) I believe that it is easy to get an IVR System to accomplish that task 1 2 3 4 5 Strongly agree Strongly Disagree c) Learning to operate an IVR System was easy for me 1 2 3 4 5 Strongly agree Strongly Disagree d) I believe that an IVR system was easy for me 1 2 3 4 5 Strongly agree Strongly Disagree e) I found the instructions easy to understand 1 2 3 4 5 Strongly agree Strongly Disagree f) It was easy for me to follow the instructions 1 2 3 4 5 Strongly agree Strongly Disagree g) The instructions were clear tome 1 2 3 4 5 Strongly agree Strongly Disagree h) In general, for inquiring about the status of a claim I would likely use Interactive Voice Response System Operator 110 2. For inquiry about a claim payment cheque a) My interaction with the Interactive Voice Response System was clear and understandable 1 2 3 4 5 Strongly agree Strongly Disagree b) I believe that it is easy to get an IVR System to accomplish that task 1 2 3 4 5 Strongly agree Strongly Disagree c) Learning to operate an IVR System was easy for me 1 2 3 4 5 Strongly agree Strongly Disagree d) I believe that an IVR system was easy for me 1 2 3 4 5 Strongly agree Strongly Disagree e) I found the instructions easy to understand 1 2 3 4 5 Strongly agree Strongly Disagree f) It was easy for me to follow the instructions 1 2 3 4 5 Strongly agree Strongly Disagree g) The instructions were clear to me 1 2 3 4 5 Strongly agree Strongly Disagree h) In general, for inquiring about a claim payment cheque I would likely use Interactive Voice Response System Operator i l l 3. For getting a form faxed a) My interaction with the Interactive Voice Response System was clear and understandable 1 2, 3 4 5 Strongly agree Strongly Disagree b) I believe that it is easy to get an IVR System to accomplish that task 1 2 3 4 5 Strongly agree Strongly Disagree c) Learning to operate an IVR System was easy for me 1 2 3 4 5 Strongly agree Strongly Disagree d) I believe that an IVR system was easy for me 1 2 3 4 5 Strongly agree Strongly Disagree e) I found the instructions easy to understand 1 2 3 4 5 Strongly agree Strongly Disagree f) It was easy for me to follow the instructions 1 2 3 4 5 Strongly agree Strongly Disagree g) The instructions were clear to me 1 2 3 4 5 Strongly agree Strongly Disagree h) In general, for getting a form faxed I would likely use Interactive Voice Response System Operator 112 4. . For inquiry about WCB's hours of operation a) My interaction with the Interactive Voice Response System was clear and understandable 1 2 3 4 5 Strongly agree Strongly Disagree b) I believe that it is easy to get an IVR System to accomplish that task 1 2 3 4 5 Strongly agree Strongly Disagree c) Learning to operate an IVR System was easy for me 1 2 3 4 5 Strongly agree Strongly Disagree d) I believe that an IVR system was easy for me 1 2 3 . 4 . 5 Strongly agree Strongly Disagree e) I found the instructions easy to understand 1 2 3 4 5 Strongly agree Strongly Disagree f) It was easy for me to follow the instructions 1 2 3 4 5 Strongly agree Strongly Disagree g) The instructions were clear to me 1 2 3 4 5 Strongly agree Strongly Disagree h) In general, for inquiring about WCB's hours of operation I would likely use Interactive Voice Response System , Operator 1 1 3 5. To speak to a person whose extension number I know a) My interaction with the Interactive Voice Response System was clear and understandable 1 2 3 4 5 Strongly agree Strongly Disagree b) I believe that it is easy to get an IVR System to accomplish that task 1 2 3 4 5 Strongly agree Strongly Disagree c) Learning to operate an IVR System was easy for me 1 2 3 4 5 Strongly agree Strongly Disagree d) I believe that an IVR system was easy for me 1 2 3 4 5 Strongly agree Strongly Disagree e) I found the instructions easy to understand 1 2 3 4 5 Strongly agree Strongly Disagree f) It was easy for me to follow the instructions 1 2 3 4 5 Strongly agree Strongly Disagree g) The instructions were clear to me 1 2 3 4 5 Strongly agree Strongly Disagree h) In general, to speak to a person whose extension number I know I would likely use Interactive Voice Response System Operator 114 6. Were there any parts of the Interactive Voice Response system where the instructions were not clear? Script 1 Script 2 Script 3 7. Are there any other features you would like to have in an Interactive Voice Response system? 115 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0089021/manifest

Comment

Related Items