Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Evaluation of the electronic document and record management program in a Canadian municipality Xie, Li 2006-12-31

You don't seem to have a PDF reader installed, try download the pdf

Item Metadata

Download

Media
ubc_2006-0722.pdf [ 8.15MB ]
Metadata
JSON: 1.0092752.json
JSON-LD: 1.0092752+ld.json
RDF/XML (Pretty): 1.0092752.xml
RDF/JSON: 1.0092752+rdf.json
Turtle: 1.0092752+rdf-turtle.txt
N-Triples: 1.0092752+rdf-ntriples.txt
Original Record: 1.0092752 +original-record.json
Full Text
1.0092752.txt
Citation
1.0092752.ris

Full Text

EVALUATION OF THE ELECTRONIC DOCUMENT AND RECORD MANAGEMENT PORGRAM IN A CANADIAN MUNICIPALITY by Li Xie ATHESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF ARCHIVAL STUDIES in The Faculty of Graduate Studies THE UNIVERSITY OF BRITISH COLUMBIA October 2006 © Li Xie, 2006 Abstract An electronic record management system (ERMS) is an application designed to improve the management of current electronic records in organizations. It is frequently integrated with an electronic document management system (EDMS), and thus becomes an electronic document/record management system (EDRMS). Document and record are different concepts in archival science and require different handling. Managing electronic documents focuses on bringing in organizations business-related benefits, such as enhanced information sharing and increased office productivity. Managing electronic records focuses on ensuring records reliability and authenticity. The development of EDMS is an industry response to the proliferation of electronic documents brought by digital technologies, while the development of ERMS involves research on the nature of electronic records, how to distinguish them from other electronic information, and how to protect their evidentiary value. Standards such as the DoD5015.2 specify functional requirements for designing an ERMS. Authoritative organizations test functionalities of commercial electronic records management systems, verifying their compliance with standards. There are, however, few systematic investigations done to evaluate the effectiveness of standard-compliant systems with respect to their operation in organizations. This thesis attempts to determine whether an EDRMS implemented in a Canadian municipality has achieved the goals set for its implementation. Currently, there are no standardized methods or services assessing the performance of either document or record management functions in an EDRMS. This thesis employs program evaluation, in particular, theory-driven program evaluation, as methodological framework, to assess the components of the city's EDRMS program. The research is designed as a survey and data are collected using a questionnaire. The evaluation questions were addressed to a particularly defined group of respondents, the users of the system who are charged with records management responsibilities in their offices or departments. Findings are reported in the form of summarized statistics. The evaluation concludes that the EDRMS program is an overall success in regards of both managing electronic documents and electronic records for this group of users. Further analyses and discussions of the findings identify issues and areas requiring improvements as well as suggest recommendations. in Table of Contents Abstract ii Table of Contents v List of Tables ,-vList of Figures viAcknowledgements viii Dedication x Chapter 1 Introduction 1 1.1 Document and Record1.2 Electronic Document and Electronic Document Management System (EDMS) 4 1.3 Electronic Record and Electronic Record Management System (ERMS) 7 1.4 Integration of EDMS and ERMS '. 15 1.5 Organization of the Thesis 17 Chapter 2 Literature Review 9 2.1 The UBC Project 12.2 The DoD 5015.2 Standard 25 2.3 Other Efforts to Define ERMS Requirements.: 30 2.4 EDRMS Implementation Case Studies 33 2.5 Evaluation of EDRMS 46 2.6 Problem Statement and Research Question 50 Chapter 3 Research Methodological Design 53 3. 1 Terminology 53. 2 Research Methodology - Program Evaluation 55 3.2.1 Program and Program Evaluation 53.2.2 Theory-Driven Program Evaluation and Program Theory 58 3.2.2.1 Change Model in Program Theory 59 3.2.2.2 Action Model in Program Theory 61 3.2.2.3. Program Theory in the System View 65 3.2.3 Program Outcome Evaluation 67 3.2.4 Application of Theory-Driven Outcome (Effectiveness) Evaluation to the EDRMS Program 8 3.2.4.1 The EDRMS Program 69 3.2.4.2 Rationales for the Theory-Driven Outcome (Effectiveness) Evaluation Approach 70 3.2.4.3 Program Theory of the EDRMS Program 72 3.3 Research Design - Survey 7 3.4 Research Method - Questionnaire 79 Chapter 4 Data Analysis & Findings 81 4.1 Response Rate 82 4.2 Assessing the Change Model4.2.1 Measurements of the Intervention 83 4.2.1.1 Overall Design 8iv 4.2.1.2 Creation of Electronic Documents 88 4.2.1.3 Documents and Records Locating 95 4.2.1.4 Use of E-mail 94.2.2 Measurements of Outcomes 100 4.2.2.1 Outcome Measurements from All Respondents 101 4.2.2.2 Outcome Measurements from the Comparison Group.... 102 4.3 Assessing the Action Model 104 4.3.1 Measurements of Intervention and Service Delivery Protocols . 105 4.3.1.1 Understandings of RM Concepts and Principles 105 4.3.1.2 Service Delivering Activities 109 4.3.2 Measurements of the Target Population 112 Chapter 5 Discussions and Implications 116 5.1 Document Classification 115.1.1 User Background 7 5.1.2 Design of the File Plan 122 5.1.3 Technical Assistance for Classification 140 5.1.4 Time of Classification 141 5.2 Document Locating 145 5.2.1 Document Locating and Time Period Working with the System 147 5.2.2 Document Locating and Computer Skills 148 5.3 "Mark as a Record" and the Concept of Authenticity 151 5.4 RM Department Assistance 156 5.5 Users'Awareness of Features 160 5.6 Most Welcomed Feature 165 5.7 Titling Guidelines 165.8 Department Staff with RM Responsibilities.: 166 5.9 Learning Experience 9 5.10 Program Outcomes 171 5.10.1 Office Productivity5.10.2 Information Sharing 174 5.10.3 Paper Reduction 6 5.11 Response Rate and Generalizability 178 5.12 Further Research 179 Chapter 6 Conclusions 183 Selected Bibliography 190 Appendices 19Appendix A Questionnaire 198 Appendix B Certificate of Approval 20V List of Tables Table 4.1 Overall Design 87 Table 4.2 Creation of Electronic Documents - Document Profiling 91 Table 4.3 Classification Difficulties 93 Table 4.4 Documents and Records Locating 96 Table 4.5 Documents and Records Locating Methods 98 Table 4.6 Use of E-mail 100 Table 4.7 Outcome Measurements from All Respondents 102 Table 4.8 Outcome Measurements from Comparison Group 103 Table 4.9 Understanding of RM Requirements 108 Table4.10 Service Delivering Activities - General 110 Table 4.11 Service Delivering Activities - Current Assistance 111 Table 4.12 Service Delivering Activities -Types of Assistance 112 Table 4.13 Measurements of the Target Group - Backgrounds 113 Table4.14 Measurements of the Target Group - Learning Experience 114 Table 5.1 Classification and Time Period Working with the EDRMS 117 Table 5.2 Classification and Computer Skills 120 Table 5.3 Document Locating and Time Period Working with the EDRMS 147 Table 5.4 Document Locating and Computer Skills 149 Table 5.5 "Mark as a Record" (Q32) and "History" (Q23) 152 Table 5.6 "Mark as a Record" (Q32) and "Modify Security" (Q10) 152 Table 5.7 RM Department Assistance and User Factors 157 Table 5.8 Currently Provided Assistance and User Factors 159 Table 5.9 Assistance Types 160 Table 5.10 Users' Awareness of Features 162 Table 5.11 Titling Guidelines and Document Locating 166 Table 5.12 Department RM Personnel -1 167 Table 5.13 Department RM Personnel-2 8 Table 5.14 System Learning and Computer Skills 170 Table 5.15 Analysis of Neutral Response to Q24 2 Table 5.16 Analysis of Neutral and Disagreement Response to Q37 173 Table 5.17 Analysis of Information Sharing and Documents Locating 175 vi List of Figures Figure 3.1 A System View of a Program. 57 Figure 3.2 Change Model in Program Theory 60 Figure 3.3 Action Model in Program Theory 2 Figure 3.4 A System View of Program Theory 6 Figure 3.5 A System View of the EDRMS Program 70 Figure 3.6 Change Model of the EDRMS Program Theory 74 Figure 3.7 Action Model of the EDRMS Program Theory 5 Figure 5.1 Conceptual Framework for Enhancing Practical Relevancy.... 179 Acknowledgements First and foremost, I offer my sincerest gratitude to my supervisor, Prof. Terry Eastwood, who has supported me throughout my research with his professional knowledge and long-accumulated research experience whilst allowing me the room to work in my own way. I appreciate deeply his friendly criticism and effective communications with regard to the confusions in my work. Words are inadequate to express my thanks to him - without his encouragement and efforts, this thesis would not have been written. It is also true that this thesis would not have been completed without the support from many other people. I am grateful to all of those who served in the thesis committee and the thesis examination committee: Dr. Heather MacNeil, Dr. Rick Kopak, Dr. Francesca Marini, and Dr. Edie Rasmussen, for their guidance and feedback on my conducting of the research. I also thank Dr. Ann Curry and Debra Lem in the School of Library, Archival and Information Studies, Ann for her advice on the design and layout of my questionnaire and Debra for her assistance in sending the questionnaires to the respondents. My deep appreciation goes to the records manager and records analyst in the city selected as the research site, for their full cooperation and support in allowing me to know the system, providing documentation and encouraging respondents to participate. Their enthusiasm towards good records management greatly inspired me. Last but not least, I give my special thanks to my friends, who endured with me the process of completing the thesis. I would especially like to thank Ian Song for his support and patience, and Heather Dean for her editorial support. IX Dedication tfo city (^oXentb Ping Deng and Bing-Zhong Xie Chapter 1 Introduction This thesis reports on a program evaluation of the implementation and operation of an EDRMS (Electronic Document/Record Management System) program within the setting of a Canadian municipality. An EDRMS is an integrated system designed for the management of electronic documents and electronic records generated in organizations. The city's EDRMS program is defined in accordance with program evaluation, which serves as the methodological framework for conducting the project. The Implementation of the program refers to the rationale underlying the decision of purchasing the EDRMS, while the operation refers to the conditions that impact the use of the EDRMS, and they will be assessed within the methodological framework of program evaluation. The methodological framework, theoretical foundation, data collection, findings, and analysis will constitute their respective chapters. This chapter focuses on the concepts and background information that are critical for the understanding of the system. This chapter will introduce the development of EDRMS in general as well as the particular system in the chosen city. 1.1 Document and Record In archival science, a document means "an indivisible unit of information constituted by a message affixed to a medium (recorded) in a stable syntactic manner," and a record means "a document made or received in the course of a practical activity as an instrument or a by-product of such activity, and set aside I for action or reference".1 The definitions suggest that document and record are related concepts but with fundamental differences. A document is characterized by affixed medium and stabilized information, and is thus capable of conveying meanings and being consulted at any given time. A record inherits these characteristics as it is first a document, but possesses other unique ones as it is closely related to activities. The activity that gives rise to a record endows it with its archival nature, that is, its characteristics of naturalness, uniqueness, impartiality, interrelatedness, and authenticity.2 In other words, since a record is generated or used by its creator in the course of conducting a specific activity, its creation is natural and the message it contains is unique. The record is also impartial because it is created or used for the need of conducting the activity and meant to fulfill practical purposes. In modern society, it is always true that more than one record is needed to complete an activity, and therefore records accumulated in the process of accomplishing the same activity are interrelated with each other, and together, they document the activity in which they are a part.3 This interrelated relationship gives rise to the core concept of archival science: the archival bond. The archival bond, manifested in record identifiers such as classification code, groups records of the same activity within the archival fonds. An archival fonds is the whole body of records accumulated by the same creator and preserved. The archival bond not only determines the structure of the archival 1 Both definitions are from the "InterPARES 2 terminology database"; available from http://www.interpares.org/ip2/ip2_terminology_db.cfm ; Internet; accessed 11 August 2006. 2 Hilary Jenkinson, A Manual of Archives Administration (London: Percy Lund, Humphries & Co. Ltd., 1965), 4; 11. 3 By qualifying this statement with "in modern society", I refer to the fact that, as described by Luciana Duranti, each medieval record contained the whole transaction that generated it since the record was made after the activity was completed. Luciana Duranti, "The Procedure of Creation of Documents," In Diplomatics: New Use for an Old Science (Lanham, Md. : Scarecrow Press, 1998), 114 2 fonds, but also is itself an essential element composing a record. A document becomes a record when it acquires its archival bond.4 In order to reach effectiveness and to be relied on for subsequent actions or reference, a record must be reliable and authentic.5 A reliable record is a record capable of standing for the facts to which it attests, while an authentic record means it is what it purports to be and is free from tampering or corruption. Records reliability depends on the record-making authority, the degree of completeness, and the degree of control exercised in the record making-procedures. The protection of authenticity after records creation ensures their reliability over time.6 Although to serve as evidence for the activity that brought them into being is never the purpose or reason for their creation,7 records are capable of evidencing the activity because of these characteristics. Both documents and records have informational value; however, records' possession of evidentiary or probative value distinguishes them from documents fundamentally. This distinction not only manifests itself at the conceptual level, it also has practical implications on the handling of documents and records. For example, with the exception of copyright, no legal requirements are imposed on the management of documents created to disseminate information. By contrast, records must be managed with the thought in mind that they may serve as evidence in all manner of proceedings. Records management programs have long been established in organizations with 4 Luciana Durant, "The Archival Bond," Archives and Museum Informatics 11(1997): 213-218. 5 This is not to say records are born with reliability and authenticity. Those records whose reliability and authenticity cannot be established can still be records if their creator treats them as such. However, they may not being trusted as evidence in litigation. To ensure records' reliability and to protect records authenticity is an important aim records management. 6 Luciana Duranti, "Reliability and Authenticity: The Concepts and Their Implications," Archivaria 39 (Spring 1995): 5-10. 7 Luciana Durant, "The Archival Bond," 215. 3 responsibilities of assisting the creation, use, and maintenance of current records.8 Records management tools (such as classification and retention schedules) are employed to enable management of records throughout their lifecycle, and various controls are exercised on records to ensure their authenticity and to prevent unauthorized access to them. 1.2 Electronic Document and Electronic Document Management System (EDMS) Electronic documents have been the subject of discussion since the introduction of personal computers into organizational environments. For the purpose of this thesis, electronic documents are defined as documents generated in electronic format; or to be specific, they are documents generated utilizing computer hardware and software, recorded on digital storage media, and transmitted through network infrastructure, including documents generated in the first instance using digital technology and documents converted to digital format from analogue sources9 The purpose of defining them this way is to place the emphasis on the unchanged nature of the concept: electronic documents are still documents and they have the same features of stable content and a fixed medium. B Maygene F. Daniels and Timothy Walch, A Modern Archives Reader. Basic Readings on Archival Theory and Practice (Washington, D.C. : National Archives and Records Service, U.S. General Services Administration, 1984), 24. 9 The use of the term "electronic" to describe this type of documents (and records in the next section) is less precise than the term "digital," because documents and records generated by personal computers or imaged through scanners (and sometimes OCRed using Optical Character Recognition technology) are indeed coded by "digits", and hence the term "digital" is more accurate. The term "digital" also makes sense when we consider the above mentioned technologies are commonly collectively called "digital technologies". Moreover, electronic means can be used to produce many analogue products such as analogue audios and videos, which are different from the documents generated by computers. Nevertheless, the terms "electronic", "electronic document", and "electronic record" are still used in this thesis for the purpose of being consistent with the widely used term "electronic," as in Electronic Document Management System (EDMS) and Electronic Records Management System (ERMS). 4 What has changed is how they are created and recorded. Electronic records are different from paper records in format because they are created and recorded using technologies that are different from writing on paper. This different format consequently causes changes in their use and maintenance, as it allows and, in most cases, requires that they are managed in the electronic environment; that is, by particularly designed computer applications. An electronic document management system (EDMS) is such an application. When supported by an operating system, storage media, and network infrastructure, an EDMS offers management functions addressing the challenges and issues associated with managing the vast volume of electronic documents that exist in organizations today. Definitions of EDMS can be found in the archival community (such as the National Archives of Australia (NAA)), in the records management field (such as the Association for Information and Image Management (AIIM) and the Association of Records Managers and Administrators (ARMA)), and in technical standards (such as the Design Criteria Standard for Electronic Records Management Software Applications (DoD 5015.2-STD)). In its Digital Recordkeeping Guidelines, NAA refers to an EDMS as "an automated system used to support the creation, use and maintenance of electronically created documents for the purposes of improving an organization's workflow."10 The AIIM/ARMA TR48-2004 technical report describes an EDMS as "a set of 10 National Archives of Australia, "Digital Recordkeeping Guidelines"; available from http://www.naa.qov.au/recordkeepinq/er/quidelines.html; Internet; accessed 20 September 2006. 5 software/hardware applications that electronically manages electronic documents contained in an information technology system, using computer equipment and software to manage, control, locate, and retrieve information in the electronic system."11 The DoD 5015.2-STD considers an EDMS "a system used for managing documents that allows users to store, retrieve, and share them with security and version control."12 These definitions vary in some aspects and overlap in others; nevertheless, they collectively denote the typical functions of an EDMS, which a) support document creation through integration with document-creation software applications (such as MS office); b) provide storage management coordinating hardware and software applications (typically a single user interface view of multiple databases on designated document servers with check-in/check-out control of documents in the electronic repository); c) provide document retrieval and information sharing through integration with search software applications; d) provide document viewing and editing, access and version control, 11 AIIM/ARMA Standards Committee on Integration of Electronic Document Management Systems (EDMS) and Electronic Records Management Systems (ERMS) Functional Requirements, Technical Report: Framework for Integration of Electronic Document Management Systems and Electronic Record Management Systems (ANSI/AIIM/ARMATR48-2004), 6. The report can be ordered at the ARMA's website, http://www.arma. orq/bookstore/productdetail.cfm?ProductlD=1479. It was published simultaneously by AllM International and ARMA International In July 2004, and registered with the American National Standards Institute (ANSI). The committee was comprised of a group of individuals and organizations who worked with AIIM International, representatives from ARMA International, the United States federal agencies including the National Archives and Records Administration, representatives from software vendors and systems integration companies, and other interested parties. The technical report defines, describes, and differentiates EDMS and ERMS, and provides a framework for their integration. Its Foreword contains the following statement: "This report is a living document that will evolve as others comment on its contents, as the industry continues to change, as integration experience is gained, and as business needs are assessed." Hereafter cited as "The Technical Report." 12 United States Department of Defense, "Design Criteria Standard for Electronic Records Management Applications"; available from http://www.dtic.mil/whs/directives/corres/pdf/50152std 061902/P50152s.pdf; Internet; accessed 15 September 2006. 6 and history/audit trails through design of the EDMS software; and e) manage a variety of document types including both born digital documents and digital reproductions of hard copies (typically through scanning) These functions, referred as document management functions in this thesis, point out the focus of the EDRM: it is by design a management tool for controlling work-in-progress documents, promoting effective use and re-use of information, and facilitating the creation of records - which are eventually printed out on paper. 1.3 Electronic Record and Electronic Record Management System (ERMS) The practice of records management remained unchanged at the beginning phase of the use of computer technologies in organizations. Most organizations continued to rely on managing records in their traditional systems into which paper copies of documents generated using computers were inserted. Records management requirements such as classification and retention only applied to these printouts as they were treated as official records, and the disposition of what remained in the computer was often at the employees' discretion. The justifications for treating printouts as records and freely destroying information generated in electronic systems were seriously challenged in the case of electronic mail systems, most notably in Armstrong v. Executive Office of the President (also known as the PROFS case). In this decade-long litigation over the creation, use, management, and preservation of White House electronic mail 7 messages, the government argued that the IBM PROFS (Professional Office System) - the e-mail system used by the U.S. National Security Council - was purely a communication substitute for the telephone and not a recordkeeping system, and therefore contained no records. E-mail messages that were determined by officers to be records were printed out and managed as such. Therefore, it was argued, the erasure of e-mail messages in the system fully complied with the records management polices and procedures established by the National Archives and Records Administration (NARA). On the other side, the plaintiff, a non-profit organization called the National Security Archives, which collects and disseminates copies of declassified United States government records, claimed the erasure of the e-mail messages in the system was a violation of both the Federal Records Act and the Presidential Records Act, because the e-mail messages in question qualified as records under these acts. The court eventually ruled against all the government's claims. It ruled that printouts of electronic messages were not capable of capturing the information accompanying electronic messages, and therefore were not the equivalents of them. Instead, they should be managed in the agency's recordkeeping system, preserved as evidence of administrative activities, and could only be destroyed with approval from NARA.13 This high profile case raised questions about records in electronic format, 13 David Wallace, "Preserving the U.S. Government's White House Electronic Mail: Archival Challenges and Policy Implications," available from http://\WAAW.ercim.orq/publication/ws-proceedinqs/DELOS6/wallace.pdf#search-%22%22Preservinq%20t he%20U.S.%20Government's%20White%20House%20Electronic%20Mail%22%22: Internet, accessed 17 August 2006. 8 their nature and the measures for managing them. The concern about electronic records in the archival and records management community was in fact expressed. An indication is the ACCIS (Advisory Committee for the Coordination of Information Systems) study undertaken in 1987 by the United Nations (UN). This study aimed at identifying issues relating to the management of electronic records in UN organizations and developing recordkeeping guidelines to address them. The UN published the report of the study, entitled Management of Electronic Record: Issues and Guidelines, in 1990, with twenty-seven issues identified and a chapter of electronic records management guidelines, subtitled: A manual for policy development and implementation. The report covers such issues as the definition of electronic record, assurance of their authenticity, integration of record-making procedures with business procedures, and security measures, which all needed to be resolved by UN organizations. The UN subsequently extended the scope of its Archives and Records Management Section to include electronic records in 1991 by establishing an electronic records program.14 While the guidelines and recommendations were not implemented immediately by organizations due to various factors, the issues identified by the study have proven to be common for all organizations that had started office automation initiatives. The questions raised by the PROFS case, the very issues identified in the ACCIS study, and the widely recognized necessity of distinguishing records from 14 Liisa Fagerlund, "Management of Electronic Records in the United Nations"; available from http://www.archimuse.com/publishinq/elec prqmstr/elec prqmstr Faqerlund.pdf#search=%22accis%20u nited%20nations%22; Internet; accessed 17 August 2006. 9 other types of information in electronic systems, inspired the formulation of a research project, entitled Preservation of the Integrity of Electronic Records, commonly referred to as "the UBC Project". This project centered itself on the identification of electronic records as they are actively used in electronic systems and identified conceptual requirements for guaranteeing their reliability and authenticity.15 The findings of the project will be discussed in detail in the next chapter because of its close relationship with the DoD5015.2 standard, which, in turn, is highly relevant to the current research. There are others research projects focusing on electronic records, such as the Pittsburgh Project and the Indiana University Electronic Records Project. However, these projects did not have a noticeable impact on electronic records management. With the legally established status of electronic records as records and the continued advance of office automation, electronic records have gradually entered the mainstream of records management in organizations in both the public and private sectors. A survey conducted in 2005 by Cohasset Associates reports that 65% of responding organizations state that electronic records have been included in their organizations' current records programs, a 15% increase over a similar survey conducted in 2003.16 The Cohasset report cites five factors as contributing to this significant improvement, one of which is the "increasingly robust and 15 Luciana Duranti, Terry Eastwood and Heather MacNei, Preservation of the Integrity of Electronic Records (Dordrecht: Kluwer Academic, 2002), 1-8. 16 Robert F. Williams and Lori J. Ashley, "2005 Cohasset Electronic Records Management Survey: a Renewed Call for Action"; available from http://www.merresource.com/downloadWhitepaper.htm?fileld=1; Internet; accessed 1 September 2006. 10 integrated technology solutions offered in the marketplace."17 An EDMS that only has document management functions and only supports records creation in printed format obviously does not fall into the category of "robust" and "integrated" technology. The EDMS solution is no longer sufficient for record creation, because many born electronic records, such as e-mail messages and dynamic/interactive Web content do not have paper equivalents. The dramatically increased number of online government and commercial services requires organizations to mange records in their electronic formats. For the purpose of this thesis, electronic records are defined as records in electronic format and they form a subset of electronic documents. The concept of record in this definition remains unchanged from the sense it has had in the traditional environment. An electronic record is an electronic document made or received in the course of a practical activity as an instrument or a by-product of such activity, and kept for action or reference. That they are only a subset of electronic documents means not all documents satisfy the qualifiers in the definition. EDMS applications do not possess functions capable of maintaining the probative value of electronic records. In other words, EDMS applications manage documents as discrete units and solely focus on information sharing, cost saving, and productivity. They lack the function that links records to their generating activity, and therefore fail to capture the contexts in which the evidentiary quality of records can effectively be assessed. 17 Williams and Ashley, 5. The other four factors are: new regulations - particularly Sarbanes-Oxley and HIPAA; significant court decisions - such as Zubulake v. UBS Warburg2 and Perelman v. Morgan Stanley; the 2003 Cohasset ARMA AIIM Survey - A Call to Action White Paper which authoritatively detailed the severity of current problems regarding the management of electronic records; and the growing realization that good records management is the keystone to achieving compliance which, in turn, is essential to senior management's goal of effective governance. 11 Electronic records management systems are designed to accommodate the specific needs of managing electronic records. The definitions of ERMS can also be found from the above same sources that define EDMS, but none of them is able to clearly distinguish ERMS from EDMS. Why this is so is wrapped up in the evolution of document and records management software. According to the Technical Report, there are five identifiable methods companies have used to develop electronic records management capacity: a) developing paper based records management systems to which ERMS functionality was added; b) developing EDMS systems to which ERMS functionality was added; c) providing ERMS software and toolkits for integrating ERMS with other applications; d) (more recently) developing fully integrated EDMS/ERMS by migrating EDMS or ERMS code into their original application; and e) offering an ERMS model in which ERMS functionality is partially embedded in enterprise applications, which reserves some ERMS functionality for a metadata server.18 To avoid confusion, an ERMS in this thesis is defined as a computer system with specialized functions targeting the management of electronic records designed in accordance with established records management principles and practices while taking into consideration their electronic format. Regardless of the 18 The Technical Report, iv. The "enterprise application" in the last bullet refers to the technological solution addressing the full range of information systems in organizations currently under strong avocation in the content management field. In this vision, EDMS and ERMS are two components in an Enterprise Content Management (ECM) application. 12 differing methods of building them, electronic records management systems typically allow organizations to: a) mark electronic documents as read-only electronic records; b) protect records against modification or tampering; c) file records according to an organizational file plan or taxonomy for categorization; d) mark records as vital records; e) assign retention rules to records; f) freeze and unfreeze retention rules; g) apply access and security controls to records (security rules for electronic records may differ from the source electronic document in the EDMS); and h) have history/audit trails of actions taken on records. It is easy to observe that as stand-alone systems, many functions of EDMS and ERMS overlap. Nevertheless, the most fundamental functional differences and the most typical relationship between the two systems can be briefly summarized as follows: a) Documents can be created, edited, altered, deleted, or saved in an EDMS. Saved documents may be declared as records and copies of such documents may be exported to the control of an ERMS. b) Records can be exported from an ERMS to an EDMS for content re-use, but no editing, altering, or deleting of records in the ERMS environment is permitted. The re-used content of records in an EDMS may be combined and aggregated to create new 13 documents, which may be declared as records.19 Standards or specifications have been developed spelling out the desired functional requirements for the design of an ERMS. The DoD 5015.2-STD is one of them. The standard is issued under the authority of DoD Directive 5015.2, Department of Defense Records Management Program, which provides implementation and procedural guidance for the management of records in the Department of Defense (DoD). The functionalities mandated by the standard were based in considerable measure on the findings of the UBC project. The DoD standard was developed to ensure that only ERMS products that are compliant with its terms can be acquired and installed by agencies or offices of the Department, which are encouraged to identify and satisfy their additional needs in any given implementation. In 2003, NARA, which participated in the revision of the standard, endorsed its use by all United States federal agencies.20 Many state agencies and commercial enterprises have followed suit, making it the de facto ERMS design standard in the United States. The testing and certification program for the standard allows ERMS software vendors to receive a multi-year certification, and the certification of a product acts as the best reference for organizations when making purchasing decisions. The functional requirements relevant to the current research will be discussed in conjunction with the UBC project in greater detail in the next chapter. 19 See a functionality comparison between an EDMS and an ERMS provided by the "Model Requirements for the Management of Electronic Record (MoReq)", available from http://www.cornwell.co.uk/moreq.html; Internet; accessed 15 August 2006: 63-64 on John W. Carlin, "NARA bulletin 2003-03"; available from http://www.archives.qov/records-mqmt/bulletins/2003/2003-03.html; Internet; accessed 15 August 2006. 14 1.4 Integration of EDMS and ERMS The need to integrate EDMS and ERMS comes from the organization's desire for both documents and records management functions. Before the latest development, EDMS and ERMS were designed separately, and this is because EDMS was developed first and its development did not take into consideration records management requirements. The separately designed stand-alone EDMS and ERMS have many overlapping functions, and they need to be streamlined by integration. Because the manner in which the two systems can be integrated is case-specific, depending on the product the organization purchases and the time when it is purchased, the following discussions is put in the context of the subject of this evaluation research, that is, the EDRMS in the chosen city, a system with integrated EDMS and ERMS functions. The city selected as the research site has a well- and long-established records management program equipped with trained records management professionals.21 It first implemented its electronic document management (DM) system, a Hummingbird product, in the mid-1990s for the purpose of bringing better management of electronic documents, fostering the sharing of organizational information, and increasing office productivity. The city's Records Management (RM) department realized that the DM was insufficient for managing records, however, an ERMS with document creation function or an EDMS with 21 The records management program was formally established in 1986 and it has a corporate-wide universal file plan established around the same time - which is always a good indicator of systematical and consistent management of records. The city issued its records management bylaw in 2001, including electronic record in its records management scope. 15 records management capacities was not available at that time. When Hummingbird subsequently developed its records management module, the RM department decided to integrate the DM module with the RM module. According to the Technical Report on the integration of EDMS and ERMS, integration means "the combination of several software applications such that data can be transferred from one application to others through a consistent interface so as to better coordinate tasks and merge information."22 This report also identifies three approaches, Integration of Stand-Alone EDMS and ERMS, Integrated EDMS/ERMS, and Integrating ERMS into an EDMS Repository/Server to bring in the two systems together.23 The add-on of the Hummingbird RM module to its DM product generally falls into the third approach. By this approach, the DM and RM modules are integrated into one end-user interface with the DM repository/server architecture serving as one single centralized documents and records repository. The RM module identifies and categorizes records within the DM module in which the records are originally created. After this capture, information is then sent to a metadata server that tracks the retention, litigation hold, and other lifecycle management aspects of the records while they continue to reside in the DM repository. The RM module manages the classification system, retention schedules, and the disposition process. This approach avoids the redundancy of storage, retrieval, and backup 22 The Technical Report, 4 23 J. Timothy Sprehe, "A Framework for EDMS/ERMS Integration," The Information Management Journal 38 (November/December 2004): 54-62. A general observation for the EDMS/ERMS industry is the development for such system is moving toward the unified EDMS/ERMS approach as a single-product solution. 16 technologies, because documents and records are actually stored in one location, in contrast to other integration approaches that require transferring of identified records from the EDMS server to the ERMS server. It also eliminates the heavy network traffic caused by the transferring of records between EDMS and ERMS environments. The Hummingbird RM module is a DoD 5015.2-STD compliant product, first certified in 1995 and subsequently recertified periodically. Compliance is tested by DoD's Joint Interoperability Test Command (JITC), which issues certification based on the evaluation of the capability of the software to meet defined records management requirements.24 The Hummingbird product currently used in the city is the Hummingbird DM/RM 5.1.05, which will be referred as the EDRMS or the system in the rest of this text. Because the system has operated for more than 10 years, it is a good time to evaluate its performance. 1.5 Organization of the Thesis The thesis is organized into six chapters. In addition to this introduction chapter, they are, Chapter 2, Literature Review, Chapter 3, Research Methodological Design, Chapter 4, Data Analysis and Findings, Chapter 5, Discussions and Implications, and Chapter 6, Conclusions. Chapter 2 examines relevant literature, which in a broader sense include standards related to the system and EDRMS implementation case studies from both system developers 24 The list of certified products can be accessed at the Joint Interoperability Test Command website, "DoD 5015.2-STD Compliant Product Registers"; available from http://iitc.fhu.disa.mil/recmqt/reqister.html; Internet; accessed 29 August 2006. 17 and records management practitioners - in addition to academic writings. Chapter 2 also identifies the research question this study aims to answer. Chapter 3 introduces in detail the research methodology, research design, and research method employed by this study and its application on the evaluation of the EDRMS program. Chapter 4 presents data and reports summarized findings based on data analysis. Chapter 5 further analyzes data in light of the summarized findings and discusses identified issues and their implications. The final chapter presents conclusions drawn from data analysis and offers suggestions for identified issues. 18 Chapter 2 Literature Review As introduced in the previous chapter, the findings of the UBC project and the requirements in DoD5015.2-STD have direct relevance to the current study. This chapter will begin with a close analysis of that relevance. It will then examine other similar standards, and consider reports on and evaluations of EDRMS performance. Finally, it will define the scope of this research and the areas it intends to investigate. 2.1 The UBC Project The UBC project was conceived by Luciana Duranti and Terry Eastwood in the School of Library, Archival and Information Studies at the University of British Columbia (UBC), Canada, and aimed to "identify and define the requirements for creating, handling and preserving reliable and authentic electronic records."25 It employed a deductive research design, that is, it started with theories and tested the theories by applying them to concrete instances. The theories guiding the project came from diplomatics and archival science, with diplomatics studying records as individual units and archival science studing records as aggregates. Diplomatics is a body of concepts and methods, originally developed in the seventeenth and eighteenth centuries, "for the purpose of proving the reliability and authenticity of documents,"26 while archival science "analyses [records'] UBC Project, "Preservation of the Integrity of Electronic Records"; available from http://www.interpares.org/UBCProiect/intro.htm ; Internet; accessed 11 August 2006. Luciana Duranti and Terry Eastwood, "Protecting Electronic Evidence: A Progress Report on a Research Study and Its Methodology," Archivi & Computer 5(3) (1995): 214-215. 19 documentary and functional interrelationships, and studies the ways in which the records with all their relations can be controlled and communicated."27 The concepts taken from the two disciplines, when harmonized as a cohesive whole, formed the conceptual basis for the research team to study the characteristics of electronic records and how they could be recognized. The project constructed templates for identifying the elements that an ideal record should possess, first in the traditional recordmaking environment, and then in the electronic environment. The project concluded that the essential elements qualifying a traditional record, namely medium, form, persons, action, context, archival bond, and content, were applicable to electronic records, although the manifestations of some elements would be different due to the different recordmaking technologies. This means, to qualify as an electronic record, the entity in an electronic system must possess these elements, or, in other words, these elements serve as identifiers for records generated in electronic systems. As discussed in the introduction, a document in paper format is an indivisible unit of information with stable content and fixed medium, and this still holds true for a document generated by digital technologies. For an electronic record to exist, it must be fixed to a medium and its content must be stabilized. It must also possess identifiable persons, contexts, and be connected to the action in which it participates. The persons concurring in the process of records generation, who may either be physical or juridical persons, are the author, writer, and addressee. The author is the person competent to issue the record or in whose name or by whose command the record has been 27 Ibid., 215. 20 issued, the writer is the person competent for the articulation of the content of the record, and the addressee is the person to whom the record is directed or for whom the record is intended. These persons sometimes may be the same person depending on the type and purposes of the record generated. As context can basically relate to anything around the creation of the record, the project identifies five contexts necessary for the creation and understanding of electronic records. From general to specific, they are juridical-administrative context, provenancial context (that is, information about the record-creating body), procedural context, documentary context (that is, the fonds and its internal structure), and technological context.28 As in paper record systems, the archival bond, which connects the record to the action that brought it into being, acts as the most critical differentiator that separates an electronic record from an electronic document, which may contain the other elements but does not have its relationships with other records of the same activity manifested in some way, such as by classification. The form of a record refers to "the rules of representation that allow for the communication of its content."29 It has two types: physical form, which contains the attributes that determine the external make-up of the record, and intellectual form, which comprise the attributes that represent and communicate the action and its immediate contexts.30 The analysis of the forms in diplomatics is the primary means of assessing records' reliability and authenticity after their creation. The concepts of reliability and authenticity apply to 28 Duranti, Eastwood, and MacNeil, Preservation of the Integrity of Electronic Records, 9-20 29 Ibid., 13 30 Ibid., 14 21 electronic records; however, the project realized that to assess the reliability and authenticity of electronic records after creation is much more difficult because records can be easily manipulated or forged in an electronic system without leaving discernible trace on the face of the record. As to the two conditions of establishing a record's reliability, the completeness of its documentary form and the amount of control exercised on the process of its creation, the latter becomes more critical for electronic records. It is the same with record authenticity, the control needed for protecting authenticity should be established at the record making and record keeping stages. With this recognition and based on the analysis of the nature of electronic records, the project developed business rules for ensuring the reliability and authenticity of electronic records when they are created and maintained in the organization's recordmaking and recordkeeping systems. For creating and maintaining electronic records as records, the project identified rules, such as "creating a record profile for every record in the records system".31 A record profile is conceived as an electronic form containing descriptive information created for an electronic record at the time of it being saved into the electronic system, the purpose of which "is to identify a record in a unique manner and to place it in relation to other records belonging in the same aggregation."32 This descriptive information, or metadata, included protocol number, dates, names and addresses of the three persons (author, writer, and addressee), class code, action 31 Duranti, Eastwood, and MacNeil, Preservation of the Integrity of Electronic Records, 31. 32 Ibid., 31. 22 or matter, number of attachments, handling office, and so on. A protocol number, or registration number, is a consecutive number assigned to each incoming or outgoing record. The dates are identified in relation to the lifecycle of the record. The most important ones are the date of receipt, the date of transmission, the date of the record (that is, the date assigned to it by the author), and the archival date (that is, the date assigned to it by the records office). In addition to identifying records, this descriptive information can also be utilized to facilitate records retrieval. In line with the conceptual foundation, the project promotes "retrieving records in context", meaning to retrieve electronic records along with their profiles, attachments and annotations (if any), and profiles of other related records, for the purpose of allowing records to be viewed in the contexts surrounding them. As with traditional management of paper records, intellectual control over records, such as index and thesaurus, were also identified as facilitating tools that provided access points for retrieving records effectively.33 For ensuring and protecting the reliability and authenticity of electronic records, the project formulated a set of procedural controls to be applied during their creation, use, and maintenance, including "establishing agency-wide control over all the records in the records system" and "establishing and implementing access privileges".34 The specific methods for ensuring record reliability in its creation phase include "integrating business and documentary procedures" and "classifying records". These methods provide the means of protecting records 33 Ibid., 35-36. 34 Ibid., 39-56. 23 authenticity. In addition, a set of additional methods were identified such as "controlling the handling and use of records", "controlling the transmission of records", and "managing the scheduling and disposition of records". These methods involved implementing procedural rules such as controlling annotations, protecting confidentiality, and controlling drafts and copies. Another critical method involved creating, managing, and executing a retention schedule. A retention schedule "is a timetable associated with each class of record that determines its period of active, and semi-active retention, establishes the office of primary responsibility (usually the office issuing the record and responsible for maintaining it in terms of the records schedule) and indicates the final disposition of the records".35 The project also recommended integrating retention schedules with classification, because managing records in classes facilitates their disposition. To illustrate all the activities of electronic records management, the project created a model of lifecycle management of the whole body of records generated in an organization or agency. Rules were written, accompanying the model, in order to provide guidance for records management personnel to establish procedures capable of ensuring and guaranteeing the creation and keeping of reliable and authentic records.36 The findings of the project establish the grounds for effectively managing electronic records in organizations.37 They suggest that electronic records must 35 Ibid., 52 36 Ibid., 92-143 37 The project team in fact conducted the research and communicated its findings with a hybrid records management system - a system managing both paper and electronic records - in mind. Consideration of paper records in a hybrid environment is intentionally omitted here because the discussion in this thesis 24 be identified first from other types of electronic information and then particular and specific management controls need to be developed, and to achieve both, the solution should be a combination of procedures and technologies. The most noticeable solution built upon such a combination is the DoD5015.2 standard, which is a standard specifying functional requirements for software applications designed for managing electronic records. Many of the functional requirements are translated from the findings of the UBC project, as this will be demonstrated in the section that follows. 2.2 The DoD 5015.2 Standard The United States Department of Defense developed a standard, commonly labeled DoD 5015.2-STD, that sets forth mandatory baseline functional requirements and identifies non-mandatory features deemed desirable for records management application (RMA) software. The standard was formulated to give software developers criteria that their products would have to meet in order for offices of the Department of Defense to purchase them. A RMA in this standard is defined as the "software used by an organization to manage its records."38 Its primary functions are "categorizing and locating records and identifying records that are due for disposition."39 It also "stores, retrieves, and disposes of the electronic records that are stored in its repository."40 A RMA is considered as an electronic records management system (ERMS) in this thesis because its solely focuses on electronic records. United States Department of Defense, 17. (full citation in chapter 1) Ibid. 25 definition and functions fit into the ERMS defined in the previous chapter for the purpose of this thesis. The term "RMA" will only be used in the following introduction to, and discussion on, the DoD standard. The standard relied heavily on the work that members of DoD's Records Management Task Force of the mid-1990s did with researchers in the UBC Project.41 Chapter 2 of the standard covers the main facets of electronic records management: implementing file plans, scheduling records, declaring and filing records, storing records, access control, and system audits. It also provides sets of detailed requirements within each facet. The following statements from the standard are linked to concepts articulated and utilized in determining procedural guidelines by the UBC Project, as indicated in italics in the bracket after each requirement. Familiarity with these requirements will also be useful for the understanding of the ERMS selected as the subject of this research, which is a DoD5015-STD certified product. Each statement is preceded by reference to the section from which it comes in Chapter 2. C2.2.1.4 "RMAs shall ensure that identifiers are unique so that ambiguous assignments, links, or associations cannot occur." (record component: procedural context) C2.2.3.8 "RMAs shall prevent subsequent changes to electronic records stored in its supported repositories. The content of the record, once filed, shall be preserved." (record components: content, archival bond; procedural rule: authenticity protection) 41 For details on the close collaboration between the Records Management Task Force and the UBC project, see Duranti, Eastwood, and MacNeil, Preservation of the Integrity of Electronic Records, 6-7. 26 C2.2.3.9 "RMAs shall not permit modification of the metadata fields indicated by this Standard as not editable." [procedural rule: reliability assurance and authenticity protection) C2.2.5.1. "RMAs shall provide at least one portal that provides access to all associated repositories and databases storing electronic records and their metadata." (Record component: medium; procedural rule: agency-wide control over all records) C2.2.5.2. "The RMAs shall prevent unauthorized access to the repository(ies)." (procedural rule: access privilege) The audience for the standard is primarily software developers and organizations or agencies that intend to purchase such software.42 However, parts of it speak directly to the roles persons play in implementing, administering, and using a RMA. The standard defines Application Administrators as "individuals who are responsible for setting up the RMA infrastructure (DL1.1.4 in the definition section.)," Privileged Users as "individuals who are given special permission to perform functions beyond those of typical users (DLL 1.61)," and "Authorized Individual" as "a Records Manager or other person specifically designated by the Records Manager as responsible for managing various aspects of an organization's records (DLL 1.8)". General users of the RMA are referred as "users" or "typical users" by the standard. The following requirements are related to the general users of a RMA, in such a way that they either indicate user responsibilities, such as profiling and 42 Stating that the agencies and organizations are also audiences of the standard is because the Standard • allows customization in many aspects, and knowing these requirements should benefit customer-initiated configurations. 27 classifying records, or regulate the assistance a RMA should provide for users to discharge these responsibilities, such as automatic data population. These responsibilities and assistance will be indicated in italics in the bracket after each requirement. C2.2.3.11. "For records that are being filed via the user interface, RMAs shall provide the user with the capability to edit the record metadata prior to filing the record, except for data specifically identified in this Standard as not editable." (responsibility) C2.2.4.3. "RMAs shall provide the user the option of filing e-mail and all its attachment(s) as a single record, or filing selected e-mail item(s) as individual record(s), or to do both." (responsibility) C2.2.3.10 "RMAs shall (for all records) capture, populate, and/or provide the user with the capacity to populate the metadata elements before filing the record." (assistance) C2.2.3.25. "RMAs shall provide the capability for users to create and maintain shortened "quick-pick" lists from the authorized lists." (assistance) C2.2.3.26. "RMAs shall provide the capability for users to create and maintain templates that automatically populate commonly used data into record metadata fields." (assistance) The following requirements address a RMA's record-locating functions,43 which can be categorized as locating method and post-search management. They will also be indicated in italics in the bracket after each requirement. 43 The term "locate", instead of "search", is used here because these functions include "browse" as well as "search". In common usage, browse means to find a record through navigating file directories, and search refers to find a record through executing certain search criteria. For the same reason, the term "find" will be used interchangeably with "locate" in the rest of the thesis. 28 C2.2.6.8.1. "RMAs shall allow users to browse the records stored in the file plan based on their user access permissions." (locating method: browse) C2.2.6.8.2. "RMAs shall allow searches using any combination of the record and/or folder metadata elements." (locating method: simple search) C2.2.6.8.3. "RMAs shall allow the user to specify partial matches and shall allow designation of "wild card" fields or characters." (locating method: simple search) C2.2.6.8.4. "RMAs shall allow searches using Boolean and relational operators: "and," "and not," "or," "greater than" (>), "less than" (<), "equal to" (=), and "not equal to" (<>), and provide a mechanism to override the default (standard) order of precedence." (locating method: advanced search) C2.2.6.8.5. "RMAs shall present the user a list of records and/or folders meeting the retrieval criteria, or notify the user if there are no records and/or folders meeting the retrieval criteria. RMAs shall allow the user to select and order the columns presented in the search results list for viewing, transmitting, printing, etc." (post-search management: displaying and sorting search results) C2.2.6.8.7. "RMAs shall provide to the user's workspace (filename, location, or path name specified by the user) copies of electronic records, selected from the list of records meeting the retrieval criteria, in the format in which they were provided to the RMA for filing." (post-search management: setting up workspace based on search results) These requirements together shape a well-conceived system, at least in theory, that controls the creation, use, and maintenance of electronic records. A system designed as such should be capable of ensuring records reliability and protecting records authenticity and at the same time providing technological assistance to general users of the system, who begin to share records 29 management responsibility in a RMA environment. 2.3 Other Efforts to Define ERMS Requirements Three other attempts to define requirements for electronic records management are noteworthy: the MoReq (Model Requirements for the Management of Electronic Record) specification, the TNA (United Kingdom, The National Archives) requirements, and the NAA (National Archives of Australia, 2006) specifications. The MoReq specification was prepared for the IDA (Data between Administrations) Program of the European Commission by Cornwell, a consultancy firm based in the United Kingdom, under the guidance of a group of experts drawn from a number of countries. While the specification was primarily developed by consultants, the majority of the review experts came from archival schools or archival institutions.44 The methodology of constructing the specification was to analyze existing standards, guidelines, research findings, and other references, which include, among others, the findings of the UBC project and the DoD5015.2-STD.45 In addition to expert review, the development project went through a validation process, which, through a questionnaire, collected feedback from software suppliers and records managers in both the public and private organizations. There is, however, no information provided regarding what feedback was collected and how it was considered by the project team 46 The goal of the project is to identify generic, model requirements covering different countries, different industries and different types of records. It states that its development has taken traditional archival science and records 44 For the list of the names of the project team, see MoReq, 121 45 MoReq, 117 46 For the process of developing the specification, see MoReq, 118. 30 management disciplines into account (the expert review), and heavily emphasizes its practical usability. While primarily focusing on specifying requirements for an ERMS, the specification also addresses the incorporation of records management requirements with requirements for document management, workflow, metadata, and other related technologies. It believes that the requirements embodied in the specification should result in a system capable of managing electronic records with the "desired levels of confidence and integrity."47 Its coverage is certainly broader than that of the DoD standard since it means to be generic and not platform-specific, but the mandatory functions it specifies are not different from those in the DoD. They are therefore not discussed further in this thesis. The TNA requirements were developed by the United Kingdom National Archives as one among a series of documents collectively titled as Requirements for Electronic Records Management Systems.48 The other three are a metadata standard, a reference document, and an implementation guide, and they are required to be consulted in conjunction with each other when implementing an ERMS. These requirements were originally formulated in 1999 and revised in 2002 for the purpose of assisting United Kingdom government agencies to manage electronic records through deploying an ERMS. Its 2002 revision took into account feedback from government departments, developments in the e-government context, experience in testing software applications, and the requirements from new legislation on data protection and freedom of information. It also incorporates 47 MoReq, 4 48 United Kingdom National Archives, "Functional Requirements," available from http://www.nationalarchives.qov.uk/electronicrecords/function.htm; Internet; accessed 15 August 2006. 31 requirements specified in MoReq. 9 As in the DoD5015.2 standard and the MoReq specification, the TNA requirements include both mandatory (core) and optional elements, and the core requirements are organized into categories of records management activities (such as records organization, retention and disposal, search, display and presentation) and system management activities (such as access control and audit). The NAA specification was posted on the NAA's website in 2006 as an exposure draft.50 It is one development in a suite of standards, policies and guidelines developed by the NAA to assist Australian government agencies to manage electronic records. The centerpiece of the suite is the DIRKS Manual,51 a methodology recommended by the NAA for agencies to design and implement recordkeeping systems. The functional specification for ERMS was developed to accompany this methodology through specifying requirements for agencies who intend either to purchase or design an ERMS. The reference publications used in developing these specifications include the MoReq specification and the TNA requirements.52 The requirements in the specification are also generic in nature, and they are set out in a familiar pattern for identifying and presenting requirements. National Archives of Australia, "Functional Specifications for Electronic Records Management Systems Software,"available from www.naa.qov.au/recordkeepinq/er/erms/specifications.html; Internet; accessed 15 August 2006. The comments colleting process has completed and the specifications are currently under revision. National Archives of Australia, "The DIRKS Manual. A Strategic Approach to Managing Business Information," available from http://www.naa.qov.au/recordkeepinq/dirks/dirksman/dirks.html.; Internet; accessed 15 August 2006. National Archives of Australia, "Functional Specifications for Electronic Records Management Systems Software," 13 32 2.4 EDRMS Implementation Case Studies While the most influential research projects on managing electronic records were led by archival scholars and carried out under the guidance of archival/records management principles, scholarly studies from the archival community remain relatively silent on the topic of EDRMS. This is probably due to the fact that, although some organizations, in both the public and private sectors, have been experimenting with technological solutions to manage electronic information, the practice is not yet widespread or of longstanding. As a result, most archival institutions, with the exception of a few national archives, such as NARA, have not had the opportunity to accession electronic records generated in an EDRMS. No matter what the reasons are, one thing is certain: Inquiry into the implementation, operation and outcomes of employing such systems would benefit both organizations who intend to implement them and the archival/records management community who is charged with responsibilities of guiding the management of electronic records. Such inquiries could help determine the degree to which such systems have achieved their intended outcomes, where they fall short, and how their performance might be improved. With these objectives in mind, the following section attempts first to gather information about the implementation of EDRMS in general and then examines in detail some cases implemented in government agencies, as they are considered relevant to the current research interest. Two sources of EDRMS implementation case studies were found: 1) reports 33 of "success stories" by software developers and 2) articles/presentations by project managers or consultants who report both experiences and outcomes. These success stories combined with customer lists on the developers' websites provide useful information for a broad understanding of the current developments in the field of electronic information management. The term "electronic information" (other than "electronic records") is used here to reflect the fact that these EDRMS developers are now providing highly integrated technological solutions targeting the management of all kinds of information and documents, such as records, e-mail messages, and the contents of Web sites. The term "enterprise content management (ECM)" is often used by software developers and consulting companies to refer to their integrated products. For the purpose of the current research, only cases reporting the implementation of EDRMS are reviewed. The representative software developers, among other companies that produce DoD 5015.2-STD certified products, are Hummingbird Ltd. and Tower Software, both leading ECM technology providers.53 The organizations procuring and implementing such products include governments (at all levels), legal services, financial services, health, education, telecommunication, energy, automotive, and many other regulated and/or highly competitive industries.54 These success stories all report positive results brought about by the EDRMS implementation such as increased productivity, informed decision-making, Hummingbird, "[Home Page]"; available from http://www.humminqbird.com; Internet; accessed 15 August 2006; Tower Software, "[Home Page]"; available from http://www.towersoft.com/qlobal; Internet; accessed 15 August 2006. See details at http://connectivitv.humminqbird.com/industrv/customers/index.html and http://www.towersoft.com/ap/Customers/Customer+List. 34 enhanced service delivery, and ensured legal compliance. Examples from Hummingbird include Baker & McKenzie, one of the world's largest global law firms with 69 offices in 38 countries, the Government of Canada, and the City of Red Deer, Alberta, Canada.56 Examples from the Tower Software include the United Kingdom Department for Constitutional Affairs, the City of Charles Sturt, a suburb of Adelaide, South Australia, and the University of Western Sydney.57 These stories are presented in a focused and logical manner and normally consist of sections of introduction to the organizational settings, challenges/problems they were facing, technological solutions the companies provided, implementation processes, and conclusions with achieved benefits. Most stories are featured with comments and quotations provided by information managers or senior management of the implementing organizations. Conclusions are drawn from system statistics such as how many documents and records are saved in the system and how effective the locating of electronic documents and records has become. The success is exclusively credited to the software application. Results achieved using document management functions, such as increased productivity, are presented as the most attractive benefits. Even the goal of "legal compliance" is linked to the search function the system provides. Records management performance, for example, how the system ensures records reliability and 55 See details at http://connectivitv.humminqbird.com/industrv/customers/index.html and http://www.towersoft.com/ap/Knowledqe+Centre/Case+Studies. 56 See details at http://www.humminqbird.com/press/2005/bakermckenzie.html: http://mimaqe.humminqbird.com/alt content/binary/pdf/collateral/ss/qovtofcanada.pdf: and http://mimaqe.humminqbird.com/alt content/binary/pdf/collateral/ss/municipalqov.pdf. 57 See details at http://www.towersoft.com/ap/Knowledqe+Centre/Case+Studies/Department+for+Constitutional+Affairs: http://www.towersoft.com/ap/Knowledqe+Centre/Case+Studies/TRIIvl+empowers+Citv+of+Charles+Sturt: http://www.towersoft.com/ap/Knowledqe+Centre/Case+Studies/Universitv+of+Western+Svdnev. 35 authenticity, is usually ignored, and so are users' opinions on the implemented system. Four articles on EDRMS implementation case studies appeared in two issues of the Records Management Journal in 2005: the implementations at the National Weights and Measures Laboratory (NWML), the National Health Service Purchasing and Supply Agency, the Estates Department of the British Library, and the Public Record Office of Northern Ireland (PRONI). The implementing organizations are all United Kingdom government agencies with staff complements ranging from 52 to 330. The impetus for adopting EDRMS were similar across all these agencies, which had all encountered electronic document management issues such as retrieval difficulties, lack of information sharing and reuse, and lack of version and duplication control. At the same time, they all were required to comply with the Freedom of Information Act passed in 2005. All these agencies chose products from the Tower Software that satisfied TNA requirements. These cases identify some factors critical for implementing an EDRMS, including support from senior management, collaboration with information technology departments, and skills for project and change management. All of the project teams also provided intensive training for end users, either done by in-house records managers or consultants. In the NWML, for example, the training was designed to provide each end user with a half day one-on-one training session offered by a records manager, and the session length could be prolonged based on user needs. The training was conducted with the live system and, during the training process, the user's favorite menus were set 36 up, sample records were created using integrated applications such as Word, Excel, and Outlook, and retrieval was practiced. Users started using the system as soon as the training was completed. Follow-up training could be arranged by users and re-training was also initiated by records managers upon checking system use. In addition to the availability of records managers, thirteen super-users were trained by the software company with the intention of equipping them as resource persons capable of assisting general users. A training manual was eventually produced based on experience gained through the training sessions and made electronically accessible.58 The differences among these cases are manifested in areas relating to records management functions, user input and acceptance, and the final outcomes of the EDRMS project. As in the NWML case, the "excellent take-up by users" as a result of the intensive training was considered one major positive outcome. The NWML case was reported as an overall successful implementation that, in addition to the full system usage, achieved all expected outcomes such as enhanced productivity, information sharing, and cost saving. For instance, the agencies human resources department showed productivity gains in a staff assessment it conducted. Statistics were cited to support claims of cost saving, such as the reduction of physical files from 276 to 108. The author also credited the designing of a "Business Classification Scheme" based on functional analysis as being one major reason for the success of the project. Although the 58 David J. Williams, "EDRM Implementation at the National Weights and Measures Laboratory," Records Management Journal 15, no.3 (2005): 158-166. 37 construction of the function-based classification system was a "painstaking work", the classification was considered a major success since it enabled users to find "a familiar place to file their records", and after the initial construction, "very few additional business classifications [were] requested".59 However, except for the brief introduction to the construction of the function-based classification system, records management functions of the EDRMS are not discussed. While introducing the NWML had established recordkeeping procedures before the EDRMS, there is no indication of whether the existing recordkeeping procedures had an impact on the EDRMS project. It is observed that the existence of the records management procedures was not included in the author-summarized reasons for success. Another area not discussed in the case is e-mail management. The article only stated that saving e-mail messages into the system was mostly welcomed by users, who no longer needed to print and file them, how e-mail messages are saved and filed in the EDRMS is not addressed. The article also does not discuss user inputs in the implementation process; somewhat on the contrary, when stating the reasons for selecting the EDRMS, the author mentioned that the product was chosen because it has "less flexibility, which would reduce the demands from users for a large amount of configuration prior to installation."60 In the case of the National Health Service Purchasing and Supply Agency, the user of the system is the center of discussion. This case was reported as a 59 Ibid., 161 60 Ibid., 162. 38 success, but the implementation encountered many difficulties.61 User resistance is the major one. The majority of the agency's employees, who became users of the system after implementation, were professional experts with highly specialized job duties, and everyone believed that "their way of working was the best way of doing things".62 Although these experts had experienced difficulties locating needed documents, and therefore had to create many convenience copies, they resisted using the EDRMS because the controls imposed by the system would change their way of doing things. Because there was no records management program in the Agency before the implementation, users of the system had little sense of records management principles and practices. Throughout the article, the author stressed the importance of culture change and believed the use of "a mixture of persuasion and compulsion", or, "both carrots and sticks", was necessary for pushing the use of the system. In the author's opinion, "it is the sticks that work best".63 The "stick" used in this case was the termination of "the old ways of working completely."64 One tactic used was to reduce users' ability to store e-mails using their own e-mail accounts, thus forcing them to save them in the system. With strong management support for these "sticks", the author reports, the use of the system dramatically increased. Three records management functions are briefly discussed in this case: classification system, records retention schedules, and access rights. The 61 Keith Gregory, "Implementing an Electronic Records Management System: A Public Sector Case Study," Records Management Journal 15, no.2 (2005): 80-85 62 Ibid., 82 63 Ibid., 84 64 Ibid., 84 39 classification system was built based on agency functions and incorporated into the system along with retention schedules. The author reports that setting access rights was a time-consuming process, but considers it was vital to information sharing and sensitive information protection. In the opinion of the author, who was the records manager of the agency, these three records management functions constitute a significant part of the overall success of the implementation. The author also offers thoughts on the abilities that implementers should possess. He stresses that there was a steep learning curve in the process of implementing the system and the process was laborious. He comments that "it is probably not necessary for IT specialists to become records managers but it is absolutely vital that records managers learn some IT skills".65 The Public Record Office of Northern Ireland (PRONI), which has 90 staff members, initiated an EDRMS project aiming to improve its the records management procedures.66 The project started with the establishment of a recordkeeping system following the methodology recommended by ISO 15489, the international records management standard. This was considered necessary since the project recognized that a simple introduction of an EDRMS would not be sufficient to solve records management problems in the agency. Interestingly, PRONI is an agency charged with responsibilities of advising other government units of Northern Ireland on records management practices; however, records management policies and procedures had long been neglected by the agency 65 Ibid., 83. 66 Zoe A. Smyth, "Implementing EDRM: Has It Provided the Benefits Expected?" Records Management Journal 15, no. 3 (2005): 141-149 40 itself. Before implementation, the project team educated users on general records management principles and promoted the awareness of records management issues. It then engaged users from all business units in the process of constructing a function-based classification system. The details of constructing the classification system were introduced by the author in another dedicated article, which will be discussed in chapter 5 of this thesis. The author emphasizes the users' familiarity with the structure of the classification, which he considers a key factor for users' acceptance and system use. Methods such as focus groups were employed to gather user opinions and to promote the notion of information sharing. The author acknowledges that the construction of a function-based classification was both time- and resource-consuming, and, despite systematic user engagement in the process, the lack of thorough analysis of business processes still created gaps between the classification system and user needs. In addition, the need to develop document naming conventions was reported as a "significant issue", since departments all asked for guidelines that accommodate their local needs. As in other cases, the project provided intensive training when the system was rolled out to all employees from "the Chief Executive to the entry-level civil servant." The tactic of giving more intensive training to "power users" or "super users", who then could help others, was also employed. The project intentionally selected as super users staff members who would most likely remain in PRONI for the purpose of knowledge continuity. The success of the EDRMS project was largely credited to the support from the senior management. The implementation was also evaluated through a 41 "post-implementation survey". That approximately 60% of staff members used the system on a regular basis was considered a success, because about 40% of them did not have filing requirements as part of their daily job, and therefore only used the system to access information. The post-implementation survey revealed that the "handy hints" reflecting staff queries developed by the project team were very much welcomed, for 94% of PRONI staff appreciated their usefulness, whereas only 77% of staff considered the user manuals useful.67 The author considered the revamped records management policies and procedures in the agency to be the most significant benefit of the EDRMS project. As PRONI is an "agency with executive responsibilities to advise on records management best practice," this project gave it the "opportunity to get its own house in order first."68 According to the author, PRONI now has a policy regarding EDRMS, an up-to-date classification scheme reflecting the agency's business processes, less duplication of information, and a culture of sharing information. The case of the Estates Department of the British Library was the only one among the four reporting a failure.69 The project was commenced in early 2001 and the system was intended for about 80 users. Although the department was required to comply with the Freedom of Information Act, the major impetus for implementing an EDRMS came from the chaotic status of electronic documents generated in the department and the urgent needs of sharing information among 67 Ibid., 148. 68 Ibid., 149. 69 Rachael Maguire, "Lessons Learned from Implementing an Electronic Records Management System," Records Management Journal 15, no. 3 (2005): 150 -157 42 employees who were geographically distributed. An early version of the Tower Software that satisfied TNA requirements was purchased along with a thesaurus that was intended to control variations of vocabulary for metadata and retrieval. A function-based classification was constructed, and file types and security levels were decided. The classification was designed as a hierarchical structure having two function levels and one subject level for the purpose of simplicity. And for the same reason, only three out of eleven fields in the metadata sheet require user attention. E-mail messages were set as one document type and e-mail attachments as a subdocument type. When filing a chain of e-mail messages, the first was filed in as a document and the rest as subdocuments. Training was designed based on the records manager's learning in Australia and were provided in a very intensive manner - in the sense of both the various training methods and the time allocated. The records manager observed a general take-up by users at about nine months after the implementation: "most staff appeared to be using the system regularly and some used it very well".70 However, the usage dropped sharply two years later as only 20 out of 60 users were using the system on a regular basis. A survey was conducted to discover the reasons. The major problems of using the system were identified as: 1) classification (especially classifying e-mail), 2) identification of records, 3) the mandated use of the thesaurus for users to fill out the metadata sheet, and 4) the counter-intuitive quality of EDRMS functionalities, including the interface, folder structure, searching, and results display. The survey concluded that although there was a 70 Ibid., 154. 43 general appreciation of the benefits brought by the system such as security, version control, and information sharing, the difficulties in learning the functional features and creating metadata deterred the majority of users from using the system. In 2004, senior management decided to abandon the system. To cope with the legal requirements of the Freedom of Information Act, the Library reformed its records management program and established a new records management policy. Under the new policy, existing information technologies, such as shared hard drives and e-mail message repositories, were structured, and the Library felt confident that this would be sufficient for managing its electronic documents and records. The author summarized lessons learned. They all pointed to the importance of user acceptance of the system. For example, the author suggested that projects must "focus on good records management behavior first" before implementing any EDRMS, since technologies would not change the poor records management practice in organization. Also, it is critical to select "a user-friendly system that is as simple as possible to use," since to add metadata to records "will always be resisted".71 The importance of user acceptance was recognized by other EDRMS implementations. Using four EDRMS cases in the United Stated federal agencies as examples, Timothy Sprehe and Charles McClure advocate "non-intrusive ERMS implementation" to minimize "records management decision-making by 71 Ibid., 156-157. 44 end users". u Records management decisions in their article refer to the identification and classification of records and other records management responsibilities shared by end users in an ERMS environment. One example is the so-called "three-buckets-two-click" approach for records classification employed in the United States Government Accounting Office (GAO). The GAO recently added the Hummingbird records management module to its Hummingbird document management system that had been used for more than 10 years. The three-buckets-two-click approach refers to the simplified classification performed by end users. The project categorized three areas for the major functions carried out by GAO, called three buckets, and listed sub-functions within each bucket. User classification requires only two clicks, one for the bucket and another for the function. After the classification, the records profile is completed by adding a title to the record. The system design allows the user to continue working in a given classification area, such that subsequently created records automatically inherit the previous classification and the user only need to add titles to the records and save them into the system. The GAO's ERMS project reported many same positive results as other successful cases, but it emphasized that, when implementing an EDRMS, measures should be taken to avoid adding too much extra work to end users. Ideally, of course, the goal is to achieve good records management results without imposing impediments to effective J. Timothy Sprehe and Charles R. McClure, "Lifting the Burden", The Information Management Journal 39. no.4 (July/August 2005): 47-59 45 completion of job duties. 2.5 Evaluation of EDRMS Generally speaking, evaluation of EDRMS can be categorized into two types: the tests of its functionalities and the assessment of its effectiveness (or in extreme terms, success or failure) after it was implemented in the organization. Within the first type, two organizations currently act as authorities in testing the functionalities of commercial software against established requirements: the United States Joint Interoperability Test Command (JITC) and the United Kingdom National Archives. Both organizations list the function-approved and then certified products on their websites and prepare results reports for each tested product for potential consumers to consult. The TNA results reports are not available from its website, but the JITC provides summary reports of the detailed results. The summary report for the Hummingbird product implemented in the city verifies its compliance with Chapter 2 of the DoD5015.2 standard, and provides information on how the functions of the EDRMS satisfy the requirements. Further discussions on the test of the EDRMS' functional requirements are beyond the scope of this thesis. The intention of introducing the test programs is because both JITC and the United Kingdom National Archives have established standardized procedures for testing the application, for example, the JITC's RMA Compliance Test Procedures (version 7.5). With respect to the second type of evaluation, however, there are no standardized methods that are publicly 73 Carol Brock and Peter Espada, "GAO's Electronic Records Management System (ERMS) - Presentation to the ARMA Northern Virginia Chapter"; available from www.labat.com/presentation/erms.ppt; Internet; accessed 17 August 2006. 46 communicated for evaluating the effectiveness of implemented EDRMS. In August 2004, the eGovernment Shared Interest Group of the Industry Advisory Council (IAC) published a white paper titled The Use of Metrics in Electronic Records Management (ERM) Systems, which reports the findings and conclusions resulting from the investigation the Study Group conducted on ERMS metrics used in government agencies and private companies.74 The Study Group was formed to respond to the request of the Office of Management and Budget (OMB) and NARA to examine industry best practices and lessons learned in the area of ERMS metrics. The Study Group did not define what an ERMS is; rather, it conducted the investigation based on the working definition of "electronic records" provided by NARA. Electronic records in this study basically encompass all kinds of objects in the electronic system. The goal of the Study Group is to determine appropriate metrics that are meaningful and not too difficult to capture. The methodology the Study Group employed was to identify and then examine case studies or exemplars of candidate metrics deployed in organizations, in both government and industry, that have implemented electronic records management systems. The Study Group identified metrics in eleven categories, namely access to services, accuracy, capacity, efficiency, participation, productivity, search and retrieval, system, user satisfaction, utilization, and legal. As the OMB and NARA request focuses heavily on 74 American Council for Technology/Industry Advisory Council (ACT/IAC), "The Use of Metrics In Electronic Records Management (ERM) Systems"; available from www.actqov.ora/actiac/documents/siqs/eqov/08032004ERMMetricsFinal.pdf; Internet; accessed 16 August 2006 47 measuring the ERMS in relation to the mission of government agencies, these metrics tend to cover every aspect of the system's performance. For relevance purposes, the following section only discusses the findings regarding the categories of efficiency, participation, productivity, search and retrieval, and user satisfaction. The efficiency metrics attempt to "measure how well an organizational entity or individual can perform an ERM-related day-to-day task."75 The Study Group found that it is difficult to measure since many factors such as work environment, policies and procedures, user training, computing resources, and information dissemination would determine the degree of efficiency, and the measurements of these factors are largely subjective. In contrast to efficiency metrics, productivity metrics attempt to "quantify the value of combined technical and organizational efficiencies realized by organizational entities or individuals in performing ERM related tasks."76 These efficiencies were considered related to the business processes supported by the ERM system. One example the business process studied was the number of invoices per hour that a clerk can process in comparison with the same measure before the implementation of ERM. Participation metrics address "the use that is being made of the system by the Ibid., 26 Ibid., 28 48 owner of the system". The owner of the system was defined in this metric as "anyone who declares, classifies, and manages records within the ERM system, and not those who simply refer to, or make use of, those records". The Study Group considered the level of participation was a critical measure of the system's success, but found it was less often and less easily measured in organizations. It is easy to measure participation in absolute numbers, such as how many people are using the system, as was done in the cases already discussed here, all of which used number of users using the system as a major measurement. It is easy to measure because the number of users using the system is captured in system audit trails. It is more difficult, to take one example, to measure how many people are declaring records versus how many should be declaring them. The Study Group considered the number of successful records searches to be one metric for measuring search and retrieval, but discovered that the success of search, which, in turn, determined the success of the system, could be influenced by many factors such as system search time, system retrieval time, quality of the metadata, and the popularity of the search domain, and therefore it is very difficult to measure.78 The Study Group found that user satisfaction is one of the most commonly measured aspects of the system as it was deployed by almost all studied systems. Users of the system were basically divided into internal and external users, and their satisfaction was measured periodically through surveys. The level of 77 Ibid., 27 78 Ibid., 29 49 satisfaction of internal users is in theory connected to productivity since when employees "buy into" the system, productivity goes up. Internal users' satisfaction is also considered as likely being transfer to customer/client satisfaction.79 From the analysis of these metrics, the Study Group drew a number of conclusions. The most important one appears as that "there is no one metric that seems to capture the success of an ERM system and relate unambiguously to the mission of an agency; nor is there a single, universal metrics capture and reporting tool (or product) that can be adopted for widespread use in ERM systems."80 The eleven categories of metrics can be assessed in different ways. Some can be automatically generated by the system, but others require gathering data related to many other factors. The Study Group concludes that the measurement of ERM performance is currently immature, and that most of the critical factors are IT-related rather than RM-related. The Study Group finally suggests that NARA and OMB should encourage the development of standardized metrics in effectively capturing and reporting ERM performance.81 2.6 Problem Statement and Research Question As the literature reveals, there has been to date no systematic research conducted with respect to the effectiveness in managing electronic records in the implementing organizations. Only a small number of organizations report on their EDRMS projects and,'not surprisingly, these reports focus on the system 79 Ibid., 32 80 Ibid., 1 81 Ibid., 1 50 implementing process and only report in very general terms the results of the projects. The literature also reveals that there are no standardized metrics for evaluating the performance of systems. All case studies report outcomes of the project based on system statistics, such as how many documents have been generated in the system and how many employees are using the system, measurements that can be automatically captured by the system. How to capture "soft metrics" to assess performance are not discussed. Two projects employed user surveys to evaluate the EDRMS, but details on how the surveys were conducted and what areas were investigated are not provided. Nevertheless, these case studies and the IAC study on ERMS metrics denote that user acceptance of the system is a critical factor for the success of the project. The case studies also demonstrate that compliance of the ERMS' functions with standards does not necessarily guarantee its success after implementation, because, it may be hypothesized, how the system performs can only be assessed by understanding all the factors, many of them outside the system in the work environment, that impinge on success. In fact, these tests are normally carried out in the system's records management mode, meaning functions are tested from the perspective of records managers, or the "authorized individuals" as the DoD5015.2 standard called them.82 That is to say, while these functions are designed and certified with records management in mind, the effectiveness of these functions in managing electronic records in real settings are not assessed 82 See, for example, United Kingdom, National Archive, "TNA ERMS Test Evaluation Report for IBM DB2Records Manager 4.1.1 Suite," available from downloads.nationalarchives.gov.uk/.../Dexmar%20Ltd%20-%20Dexmar%20KnowPro%20EDRM.pdf; Internet; accessed 11 September 2006. 51 by any dedicated endeavors. This research is the attempt to evaluate such effectiveness of an implemented EDRMS, both its document and record management functions, through collecting information about users' experience of and opinions on the system as a means of answering the question of whether the EDRMS program has reached the goals set for it by the implementation organization. The term "program" is used here in accordance with the research methodology, that is, program evaluation, chosen for this study. The research methodology for the current study will be discussed in detail in the next chapter. 52 Chapter 3 Research Methodological Design 3.1 Terminology As observed by Zina O'Leary, the initial challenge for a first-time researcher conducting social research may be the haziness of terminology about social science research methods.83 The confusion arising from the different versions of "research design" and "methodology", as just two examples demonstrate, could easily increase with the number of research method books one reads. On the bright side, however, the numerous discussions on research methods, although confusing, also provide an opportunity for researchers to select the most suitable methodology for communicating their research - as long as definitions are provided for readers' understanding. The following terminology is thus selected from sources the researcher considers appropriate and has been modified to accommodate the needs of this particular research project. 84 Research design means the framework for the collection and analysis of data; there are five different types: experimental design, survey design, longitudinal design, case study design, and comparative design. Experiment refers to the research design that rules out alternative explanations of findings deriving from it by having at least (a) an experimental group, which is exposed to a treatment, and a control group, which is not, and (b) random assignment to the two groups. Zina O'Leary, The Essential Guide to Doing Research (London ; Thoundand Oaks : SAGE, 2004), 85. Please note, these terms are not organized in alphabetic order; rather, they are listed according the relationships between the terms, for purposes of clarity. 53 Survey is a cross-sectional design and in relation to such design, a body of quantitative or quantifiable data in connection with two or more variables are collected (predominantly by questionnaire) on more than one case at a single point in time, which are then examined to detect patterns of association. Research method refers to the technique for collecting data such as a self-completion questionnaire, a structured interview schedule, or participant observation. Self-completion questionnaire means a questionnaire that the respondent answers without the aid of the researcher. Research strategy refers to the general orientation towards the conduct of social research; there are two types of strategies identified here: quantitative research and qualitative research. Qualitative research usually emphasizes words rather than quantification in the collection and analysis of data. Quantitative research usually emphasizes quantification in the collection and analysis of data.85 Purposive sampling means the researcher decides the sample formulation criteria based on some particular purposes.86 Research Methodology provides the framework associated with a particular set of paradigmatic assumptions used to conduct research, that is, scientific method, ethnography, action research, or program evaluation. Research Methodological design refers to the methodological plan for These first eight definitions are from Alan Bryman, Social Research Methods. 2nd ed. (New York : Oxford, 2004), 625. H. Russell Bernard, Social Research Methods: Qualitative and Quantitative Approaches (Thousand Oaks: Sage : 2000), 176. 54 conducting research that includes research design, methodology, strategy and method.87 In applying these interpretations to the current research, the research methodology is program evaluation and the research design is survey. The research method is self-completion questionnaire. This chapter, research methodological design, is thus about evaluation research, survey, and questionnaire, and their use in this study. 3. 2 Research Methodology - Program Evaluation 3.2.1 Program and Program Evaluation Program evaluation, also called evaluation research by some evaluators and evaluation theorists,88 is generally considered as a type of applied social research. Applied social research, as opposed to pure theoretical research, is more action-oriented because it focuses on generating practical implications. Program evaluations are usually conducted under the request of program stakeholders or policy makers to assess the program for the purposes of justifying its existence or facilitating improvements. In its broadest use, evaluations or evaluative activities can be conducted on almost every aspect of society and in relation to almost every type of human endeavor: book reviewing, commercial product testing, job performance assessing - to name a few. As social science activity, program 87 These two are from Zina O'Leary, 80 88 Although some writers, such as Moira J. Kelly, distinguish program evaluation from evaluation research, more writers, such as, Earl Babbie and Peter H. Rossi, equates evaluation research with program evaluation. Moira J. Kelly, "Qualitative evaluation research" in Qualitative Research Practice, ed. Clive Seale et al. (London ; Thousand Oaks, Calif.: SAGE, 2004), 332; Earl Babbie, "The Practice of Social Research," 9th ed., Wadsworth, Thomson Learning Inc. 2001), 333; and Peter H. Rossi and others, Evaluations Systematic Approach, 7th ed. (Thousand Oaks, CA: Sage, 2004), 6. Evaluation, evaluation research and program evaluation will be used interchangeably in this thesis. 55 evaluation has a more restrictive meaning in that it is qualified by the utilization of social research methods. While it can be dated back to the seventeenth century, systematic evaluation research employing social science methods is a relatively modern twentieth-century development. It was first conducted in education and public health fields in the 1930s and became commonplace for many other social service programs in the 1950s. The growth and refinement of social research methods in addition to societal changes have greatly facilitated the development of evaluation research. It emerged as a distinct specialty field in the social sciences during the early 1970s and gained its mature status in the 1980s.89 Among its abundant and varying definitions, Rossi's version of program evaluation is selected for present purposes. Rossi defines program evaluation as "the use of social research methods to systematically investigate the effectiveness of social intervention programs in ways that are adapted to their political and organizational environments and are designed to inform social action in ways that improve social conditions."90 The social intervention program in this definition means "an organized, planned, and usually ongoing effort designed to ameliorate a social problem or improve social conditions."91 The application of program evaluation should not be limited to social service programs. Program evaluation can be conducted in many fields other than social 89 For the history of evaluation, see Rossi, Chapter 1, "An Overview of Program Evaluation" in Evaluation: a systematic approach. 90 Peter H. Rossi and others, 16 91 Ibid., 434 56 service programs, as demonstrated by the various evaluations conducted by the United States General Accounting Office (GAO) that include the procurement and testing of military hardware, quality control for drinking water, and the maintenance of major highways.92 In this regard, the term "program", rather than "social program", will be used in the following discussions to refer to any general program or the program that is the subject of this thesis. Huey-Tsyh Chen illustrates the nature and characteristics of a program using the terminology of system theory. A program viewed this way consists of five components: inputs, transformation, outputs, environment, and feedback. These components and the relationships among them can be graphically presented as follows: 93 Environment Input Transformation Output w w A < Feedback Figure 3.1 A System View of a Program (Source: Chen, 2005) Inputs in this system refer to resources taken in from the environment that may 93 lb'd' 6-Huey-Tsyh Chen, Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness (Thousand Oaks, Calif. : Sage Publications, c2005), 4. 57 include finances, technology, equipment, facilities, and personnel.,Inputs normally require systematic organization or implementation in order to be effective. Transformation is the process of converting inputs into outputs, and may include a series of events necessary to achieve desirable outputs. Outputs are the results of transformation, among which the attainment of the program's goals is the one deemed most crucial. Environment here refers to any factors that can foster or constrain the program's implementation, such as social norms, political structures, and the economy. Feedback in this view refers to information about the aforementioned components and their responses to each other. Feedback is vital in terms of improving or justifying a program, and it is what program evaluation is all about.94 3.2.2 Theory-Driven Program Evaluation and Program Theory Theory-driven program evaluation is a development of evaluation research advanced in the late 1980s. It developed as a response to the realization that the traditional perspective, characterized as method-oriented evaluation, lacks a theoretical framework guiding the assessment. While the method-oriented approach, through the utilization of rigorous social science research methods (for example, randomized experiment design), is capable of generating highly convincing evidence on the relationship between the input and output, it neglects the transformation process between inputs and outputs that actually makes things happen. In other words, the method-oriented evaluation reports on the gross effects of the program under assessment, but does not tell why and how the 94 Ibid., 4-6 58 program has reached these effects. By contrast, theory-driven evaluation, believes that there are always a set of assumptions, explicit or implicit, associated with a given program. They are manifested in the design and operation of the program and dictate how it should conduct its business and attain its goals. The set of assumptions, which can be derived from scientific theories, stakeholder expectations, or a combination of both, is termed as program theory in theory-driven evaluation. Theory-driven evaluation promotes the idea of conducting program evaluation under the guidance of the identified program theory.96 According to Chen, program theory is "a specification of what must be done to achieve the desirable goals, what other important impacts may also be anticipated, and how these goals and impacts would be generated".97 Two types of assumptions are suggested by this definition: descriptive assumptions or a change model and prescriptive assumptions or an action model (they will be discussed in detail in the following sections). The terms change model and action model will be used in this thesis following Chen. 3.2.2.1 Change Model in Program Theory 98 The change model (as shown in Figure 3.2) consists of descriptive assumptions concerning the causal processes underlying the program and its 95 Huey-Tsyh Chen, Theory-driven Evaluations (Newbury Park, Calif. : Sage Publications, CI990), 23. 96 Ibid., 39 97 Chen, Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness, 16 98 Ibid., 20-23 59 outcomes. Since these assumptions dictate the strategies of building a program, their validity determines the effectiveness of the program. In other words, a program based on invalid assumptions is unlikely to succeed. Change Model Intervention Determinants Outcomes • w p Figure 3.2 Change Model in Program Theory (Source: Adopted from Chen, 2005) The intervention, determinants, and outcomes are the three components of the change model. Intervention refers to the activities or efforts introduced by a program to meet a need or improve a situation. The assumption for the intervention in the change model is that, by implementing the activities or efforts, the program changes the identified determinant(s). Determinant, also called mediating variable or intervening variable, is "a leverage mechanism or cause of a problem" upon which the intervention can be developed. The identification of program determinant(s) provides the focus of a given program, which, in turn, guides the design of the program as well as the design of evaluation. The assumption for the determinant in this model is that the program goals will be achieved with the activation of determinants. Chen further explains that "the determinant often relates to the program designers' understanding of what 60 actually causes the problem they want to alleviate and on which exact cause or causes they want a program to focus." Program outcomes are the concrete, measurable aspects of program goals, which, in contrast, are usually articulated in very general language expressing abstract ideas. The solid arrows in the model indicate causal relationships between these components. A causal relationship here refers to the assumption that changing one component creates change(s) in the other(s). In this model, this means when the intervention changes the determinants, outcomes occur. 3.2.2.2 Action Model in Program Theory 99 The action model (as shown in Figure 3.3) consists of assumptions prescribing components and activities necessary for a program to achieve its goals. It directs the design of a program when it is at the designing stage and it guides the assessment for a program when it is under evaluation. Like the change model, a program is unlikely to succeed if the action model is based on invalid prescriptive assumptions, that is, if it is poorly implemented or simply unrealistic. Ibid., 23-28 61 Implementing organizations Associate organizations and community partners Implemented • i I Ecological context Intervention and service delivery protocols Target populations Figure 3.3 Action Model in Program Theory (Source: Adopted from Chen, 2005) As the above illustration shows, there are five implementing components and activities in the action model: the implementing organization, program implementers, associate organizations/community partners, ecological context, intervention and service delivery protocols, and target populations. The intervention and service delivery protocols are the two requirements necessary for translating the general and abstract ideas about intervention in the change model into concrete and organized activities that the action model can implement. The intervention protocol is a prospectus stating the exact nature, content, and activities of an intervention and the service delivery protocol details the particular steps to be taken in order to deliver the intervention to the target group. 62 To implement a program, a program implementation organization is needed to provide both political and material supports for such as activity coordination, resource allocation, and staff recruitment. The structure of the implementation organization and its capacity of implementing a program have direct impact on the program implementation. The program implementer, the people responsible for delivering the intervention to the target group, is another required element. The qualifications and competency of the implementers, and their enthusiasm about and commitment to the program, all directly affect the quality of delivering the intervention. It is vital for the implementing organization to have policies in place to build, maintain, and foster implementers' competency and commitment. The element of associate organizations/community partners is related to the cooperation or collaboration between the implementation organization and other organizations. Properly established cooperation or partnership benefits program implementation in the sense that, in today's world, rarely can social activities occur in isolation. If the program requires such cooperation or partnership and is not established under the action model, the implementation of the program may be hindered. The ecological context refers to the portion of the environment that directly interacts with the program. For most programs, a supportive environment or context vitally facilitates the program's success. The ecological context can be supportive at both micro- and macro-levels. The micro-level contextual support 63 refers to the support aiming at the intervention receivers, and the macro-level contextual support refers to the immediate environment that influences the program, such as community norms, cultures, and political and economic processes. Both contexts contribute to a program's success. The last element of the action model is the target populations, the people that the program is planned to serve, or to whom the intervention is delivered. The qualifications and other attributes of the target population a particular program requires also play an important role in achieving the goals of the program. The one-way double-banded arrows in the model represent a sequential order between the components. This means the one component is the other's basis or prerequisite. For example, the implementing organization must exist first before the implementers come into play, and the implementing organization together with the implementers are the prerequisites for developing the intervention and service protocols tailored to the target population. The only two-way double-banded arrow between the boxes of implementing organizations and associate organizations and community partners means the two collaborates in planning program activities. How program theory and the two models influence this research will be revealed in the section that addresses the application of the methodology of program evaluation. 64 3.2.2.3. Program Theory in the System View 100 As the above discussion suggests, the change model for a program reflects general and abstract ideas about the intervention and determinants. These ideas must be translated into concrete and organized elements and activities that are implementable. The action model fulfills this need. Employing an action model that specifies and systematically arranges elements and activities is building a road map that guides the delivery of the intervention to the target population. In other words, the change model leads the program to attain its goals, and the action model puts the program in motion by activating the change model. By putting the program theory in a system view (see Figure 3.2), the action model is the input, the change model is the output, and the program implementation is the transformation process. Figure 3.4 depicts the relationship between the two models and the action that link them together. Ibid., 29-32 65 Resources Environment Action Model Implementing organizations Associate organizations and community partners Ecological 1 context Intervention and service delivery protocols Implementers Target populations 1 Change Model -I Program implementation Intervention >j Determinants Outcomes Figure 3.4 A System View of Program Theory (Source: Chen, 2005) The environment in which the program operates provides resources for the program to start and requires feedback about the program's operation. Feedback in the diagram is represented as dotted arrows and there are two types of them. The different types of feedback are indicated by the different manners in which the dotted arrows move. The ones that are constrained within the solid square are internal feedback and the ones that pass to the environment then back to the program are external feedback. Internal feedback and external feedback are collected for different purposes and the collection of them requires distinct 66 approaches. These different types of feedback constitute the foundation of the taxonomy of program evaluation proposed by Chen.101 The program evaluation taxonomy categorizes evaluation in accordance with the stage the program is at and the purposes the evaluation is expected to serve. These stages include program planning, implementation, and outcome. Evaluation purposes can be either improvement, or assessment, or both. Outcome evaluation is represented in the diagram as the dotted arrow that moves from the outcome box in the change model, passing the environment, then back to the action model. The current research considers outcome evaluation best suits the evaluation need, and therefore provides further explanation in the next section. 3.2.3 Program Outcome Evaluation One advantage provided by program theory, as mentioned in the previous section, is that it helps select the type of evaluation suitable for intended evaluation. Outcome evaluation takes place when a program reaches its maturity and it serves the interest of understanding the ultimate effect of the program. In other words, if the aim of the evaluation is to investigate what happened thanks to a program or to answer the question, "Is this program achieving its goals?", outcome evaluation is the choice. There are two ramifications within the realm of outcome evaluation: efficacy and effectiveness evaluation. Efficacy evaluation assesses the effects that a program generated within an ideal environment and assumes that only under tight 67 research controls can the actual effect be assessed validly and convincing evidence be generated. Effectiveness evaluation, by contrast, attempts to scope out the effects of a program in real-world conditions and aims at providing practical information useful for program improvement. The different assumptions underlying efficacy and effectiveness evaluation require distinct research design. Efficacy evaluation usually employs randomized and controlled experimental design for the purpose of producing credible evaluation results. However, it is normally difficult (if it is not impossible) for an effectiveness evaluation to employ such rigorous designs since it is conducted in a real-world setting. It is also undesirable in the sense that effectiveness evaluation aims at being more practice relevant.102 A theory-driven program outcome effectiveness evaluation identifies program theory and conducts evaluation under its guidance. Its purpose is to provide information about not only whether a program has reached its goals but also the hows and whys behind its success or failure.103 3.2.4 Application of Theory-Driven Outcome (Effectiveness) Evaluation to the EDRMS Program This section introduces the EDRMS program, justifies the selected evaluation approach, and identifies the program's components in accordance with its program theory. Ibid., 195-227 Ibid., 231 68 3.2.4.1 The EDRMS Program The program this thesis project will assess is the implementation and operation of an electronic document and record management system (EDRMS) in a Canadian municipality. The EDRMS is a commercial off-the-shelf application, which, at the time of purchase, was designed to manage electronic documents. It was subsequently extended to include records management (RM) functions through integrating a RM module from the same company, whose product was certified as compliant with the United States DoD 5015.2 standard. The electronic document management functions of the system include document creation, storage, retrieval and use, and the electronic records management functions include mark-as-records (that is, making a document read-only), classification, and retention. The city implementing the EDRMS operates within the jurisdiction of one Canadian province and is subject to both federal and provincial legislation relating to document and records management, such as the Freedom of Information and Protection of Privacy Act and Evidence Act. The city's RM program was established in 1986 and was given formal status in 2000 through the passing of the city's records management bylaw. Its records management department, charged with responsibilities of managing both paper and electronic records, is staffed with records management professionals and has been practicing records management principles and following standards such as ISO 15489. Applying Chen's system view and Rossi et al.'s definition of program, this EDRMS program can be represented as the following: 69 The City, IM & IT Disciplines Professional Development * Implementation Feedback Figure 3.5 A System View of the EDRMS Program In this diagram, the environment, that is, the city and the information management (IM) and information technology (IT) disciplines, provides resources and the professional developments in IM and IT fields, as inputs to the system. The term "information" is used here to cover both documents and records. The implementation process translates these inputs into outputs, that is, the results of the program. The action of collecting and analyzing feedback about the inputs, implementation, and outputs (as indicated by the dotted arrows) is the conduct of program evaluation. 3.2.4.2 Rationales for the Theory-Driven Outcome (Effectiveness) Evaluation Approach Unlike most program evaluations, the evaluation of the EDRMS program is not initiated by its stakeholders. Instead, it was first conceived as a thesis project as part of the completion of the researcher's master's degree. The major stakeholder of the EDRMS program, the RM department in the chosen city, was then contacted and a mutual understanding about the nature of the evaluation 70 project was reached. It was agreed that the researcher would be responsible for designing and conducting the evaluation and the city would support the project through providing necessary assistance. A working relationship had been maintained during the conduct of the project, which enabled the design of the evaluation and the execution of the whole evaluation process. Both the stakeholder and the researcher expressed the desire to assess the program in a way that it not only assessed the gross effects of the program but also analyzed the how and why behind the gross effects. Theory-driven evaluation fits this expectation perfectly. It was also decided to conduct this evaluation as an outcome evaluation since the EDRMS program has been running for ten years and is considered by the stakeholders as having reached its maturity. To decide program maturity in program evaluation requires conducting evaluability assessment. According to Chen, an evaluability assessment is a pre-assessment of the program's maturity in order to determine if the program is truly ready for rigorous outcome evaluation.104 There are criteria established for assessing the evaluability, and a program needs to satisfy the followings in order to be considered as ready: a) The goals, objectives, important side effects, and priority information needs of the program are well defined; b) Goals and objectives of the program are plausible; c) Relevant performance data can be obtained; and d) Intended users of evaluation results agree on how they will use 104 Chen, Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness, 197-198. 71 them.105 Through utilizing recommended techniques, such as site visit, records review, and meeting with stakeholders, to gather information against the above criteria, the researcher concluded that the EDRMS program satisfied all the above criteria and therefore an outcome evaluation was an appropriate choice. i Since the EDRMS program is an existing program operating in the real world, effectiveness evaluation is the obvious choice. In addition, the EDRMS program satisfies two other conditions, as Chen identifies, which favor the use of effectiveness evaluation: a) Stakeholders are curious about the effects of the ongoing program, and b) Stakeholders require the evaluation to be relevant and of practical benefit to their practices related to a program.106 The evaluation approach for the EDRMS program was therefore decided as a theory-driven outcome (effectiveness) evaluation. Its program theory will be identified in the following section. 3.2.4.3 Program Theory of the EDRMS Program As with most established programs in reality, the program theory of the EDRMS program is not explicitly articulated in one statement. The process of 105 Joseph S. Wholey, Harry P. Hatry, and Kathryn E. Newcomer, ed., Handbook of Practical Program Evaluation (San Francisco : Jossey-Bass, 1994 ), 15-39; cited in Chen, Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness, 198. 106 Chen, Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness, 201 72 identifying the program theory (that is, the change model and the action model) started when the researcher was doing a practical project in the city's RM department as part of her academic studies. This project, concerning the establishment of classification codes for a project then taking place in the city and the promotion of their use, provided opportunities for the researcher to understand both the records management program in the city and the EDRMS. More importantly, the experience gained from the project became the source of the motivation to conduct a theory-driven evaluation. The researcher learned, through the project that classifying documents in accordance with the city's universal classification plan had remained problematical for users ever since the classification plan was integrated into the EDRMS. Both the RM department and the researcher wanted to know why. During the period of the thesis project, the researcher employed techniques recommended by Chen, such as reviewing existing documents and communicating with stakeholders, to articulate and finalize the EDRMS program theory. The change model in the EDRMS program theory is identified as the following: 73 Determinant EDM Determinant ERM Outcomes Increased Productivity Enhanced Information Sharing Paper Records Reduction ER Reliability & Authenticity Figure 3.6 Change Model of the EDRMS Program Theory In this model, the EDRMS is the intervention and the electronic document management (EDM) and electronic record management (ERM) are two determinants. The EDRMS is identified as the intervention because it was purchased and implemented under the assumption that it is capable of bringing changes to the management of electronic documents and records in the city. The EDM and ERM are identified as two determinants in the sense that their changes determine the attainment of the program's goals. The goals of the program are explicitly listed in the program manual as 1) to aid organizational efficiency, 2) to facilitate management decision-making, 3) to allow compliance with legislative/regulatory requirements, and 4) to reduce cost through reducing paper records volume. 107 The fact that these goals are set for an EDRMS program denotes that more and more electronic documents and records are being generated in the city and that a paper-based system is no longer sufficient in managing information in electronic form. From these goals, four outcomes for the 107 The EDRMS program manual, unpublished internal document, version 2003. 74 purpose of this evaluation were identified: increased office productivity, enhanced information sharing, reliable and authentic electronic records, and reduced paper volume. The components of the action model of the EDRMS program theory are identified in the following diagram: The City • The RM Department Figure 3.7 Action Model of the EDRMS Program Theory The implementing organization in this action model is the city, which establishes the RM program and formalized it through passing a RM bylaw. It allocates resources for purchasing the EDRMS and hiring consultants, and supervises implementers. The program implementer is the RM department, who is charged by the Bylaw with responsibilities of managing records generated in the city, including records in electronic form. The records manager and records analyst who fully participated in the design and implementation of the EDRMS 75 Professional Developments 1 RM program 1 Program Manual RM Activities & Procedures City Employees program are trained professionals with many year's working experience in the field of records management. The most important associate partner in this program is the city's IT department, who is responsible for the technological infrastructures necessary for the program's implementation. The macro-level ecological context is identified as the professional and technological developments in the field of managing electronic documents and records, which interacts with the program through providing consulting services and standard-compliant applications. The micro-level ecological context is identified as the accumulated records management practices in the city over 20 years, which had been successful in managing paper records. The intervention protocol is the program manual, which spells out the goals of and the rationales for the program and explains the functional features of the EDRMS. The provision of trainings to the users of the EDRMS, which is embedded in the RM department's day-to-day working procedures, is identified as the program's service delivery protocol. The target population of the EDRMS program is all of the employees who work with the city and their office computers are connected to the city's information communication network. The above identified program theory can be viewed as scientific theory, as opposed to stakeholder theory that normally constitutes the program designer's own understanding and/or experience about the intervention and determinant(s). The program theory of the EDRMS program is fully based on records management principles and concepts articulated in the literature on managing electronic records. 76 3.3 Research Design - Survey According to the contingency view of program evaluation discussed by Chen, there is no single best research design that suits all programs.108 As opposed to the claim that evaluation should always employ rigorous experiment design, the research design for evaluation should be situational, meaning the factors relate to the program's nature, the evaluation purposes, and the contextual circumstances should be taken into consideration.109 Applying this view to the current project, which is decided as a theory-driven outcome effectiveness evaluation, every aspect of the decided evaluation plays a role in the design of the research. Outcome evaluation requires scientific research designs to generate credible evidence. Effectiveness evaluation assesses programs that operate in a real-world setting, and this makes the scientific research design impossible to be a randomized and controlled experiment. Effectiveness evaluation is also required by stakeholders to be practically useful for the purpose of justifying and/or improving the program. A theory-driven evaluation aims at answering how and why a program has or has not achieved its goals, and this requires qualitative research designs since qualitative data are able to provide more detailed information for a deeper understanding. This evaluation decides to employ a survey design because it meets both the needs of the decided evaluation and the realities the evaluation faces. As identified by the literature review in the previous chapter, there is a lack of research done with respect to the assessment of the EDRMS implemented in organizations through colleting and analyzing user 108 Ibid., 11-12 109 Ibid. 77 opinions. A survey design collects data about many variables from a large group of respondents at a single point in time, it thus suits the need of gathering user opinions in this evaluation. While it usually collets quantitative or quantifiable data for pattern analysis, it can also be used, with caution, to collect qualitative data. The collection of qualitative data needs to be cautious is because too many questions asking for textual answers could lower the response rate since respondents may not be willing to spend too much time on answering research questions. Qualitative data collected through focus groups or interviews satisfy the need of conducting theory-driven evaluation; given the time constraints, however, they cannot be used in this evaluation. The decision was made to employ survey design to collects primarily quantifiable data with a small amount of qualitative data. A questionnaire is predominantly used in survey design as a data collection instrument. In a typical survey design, researchers select a sample of respondents and administer a questionnaire to them. The sample group selected for the current evaluation is a purposive sample, and the selection criterion is based on the city employees' job descriptions. At the time of selection, all employees whose job descriptions specify the management of records in their departments or offices as part of their job duties were selected. The purpose for setting this sampling criterion was to get the "super users" of the EDRMS to answer the questions. They are considered as "super users" because the responsibility of managing departmental or office records makes their use of the EDRMS a daily activity. It must be pointed out that their opinions about the 78 responsibility of managing departmental or office records makes their use of the EDRMS a daily activity. It must be pointed out that their opinions about the EDRMS may or may not be the same as those of general users of the system. Time constraints also require the investigation into the opinions of general users to be left for future research. 3.4 Research Method - Questionnaire The questionnaire contains three parts with 37 questions in total. Part I collects general information about the respondents for the purposes of understanding their computer skills and their length of employment with the city and the EDRMS. The construction of Part II is primarily based on the understanding of the above identified program theory, with questions asked about components in the two models. Part III of the questionnaire is an additional section, constructed for the users who have worked with the city for more than 10 years (as will be revealed in Part I). The intention for this section is to build a comparison group and to collect "before and after" experience from the respondents in it. The majority of the statements in the questionnaire employ Likert scale. To accommodate the need of collecting qualitative data as justified in the survey design section, partially closed-ended questions are also included. The questionnaire was tested by the RM department and revisions were made accordingly. The testing of the questionnaire by the RM department serves a dual purpose: an approval for the program theory and a pilot project for the 79 questionnaire. In conformity with University policy, approval was obtained for this research in terms of protection of research subjects. For details, see Appendix B. This chapter introduces the research methodology, research design, and research method employed by this evaluation project. It also introduces the program under evaluation, the EDRMS program, and its components in relation to the research methodological design. The next chapter explains in detail the variables that constitute these components and report data analysis and findings. 80 Chapter 4 Data Analysis & Findings This chapter presents quantitative data collected from the first two questions in the questionnaire and quantifies qualitative data collected from the rest of the questionnaire that consists of Likert scale statements and partially close-ended questions. The data coding method for quantification will be introduced in individual data analysis sections in connection with the statements or questions analyzed. Data analysis and findings in this chapter refer to the presentation of descriptive statistics such as frequencies and percentages of measured variables. MS excel worksheets serve as the raw data matrix and tables are used to summarize data and report the calculated frequencies and percentages. The data analysis process does not strictly follow the order of the three parts in the questionnaire or the order of individual questions - which were primarily designed for the respondents' convenience to answer questions. The data analysis process and corresponding findings are organized in accordance with the assessed program components that consist of a number of variables. The units of analysis in this chapter are individual variables. The analysis of associations between variables and the implications that can be drawn from such analysis will be discussed in the next chapter. Since the questionnaire contains both questions and Likert scale statements, the terms "question" and "statement" will be used interchangeably in the data analysis process in this chapter and the discussions of findings in the next chapter. But only "question" (abbreviated as "Q") will be used in tables reporting findings, 81 for the purpose of letting all questions and statements in the questionnaire bear consecutive numbers. 4.1 Response Rate According to the sampling criterion, 60 respondents were selected from the city's employees who use the EDRMS to carry out their job duties on a daily basis. The three-part questionnaire was sent to them with one coyer letter from the researchers and another letter from the city's RM department. The researcher's cover letter stated the purpose of the evaluation and invited respondents' participation, while the letter written by the RM department explained the relationship between the city and the evaluation project, and encouraged participation. Thirty-two questionnaires had been returned in the first week after the questionnaire packages were sent out, and another 18 arrived after the second letter followed up restating the purpose of the project and reminding the deadline of completing the questionnaire. Originally, the participating time was set for three weeks; however, with the knowledge that some of the respondents would be on vacation during that time period, the deadline was extended to include another week in order for those vacationers to have time to participate. In the end, 50 completed questionnaires were returned to the researcher, making the response rate 50 out of 60, that is, 83%. 4.2 Assessing the Change Model The change model in the program theory, as introduced in the chapter of research methodological design, consists of three components: intervention, 82 determinants, and outcomes. This section reports the assessments of the components of intervention and outcomes. The relationships among these three components, that is, whether the intervention is a proper choice for the determinants and whether the intervention causes the outcomes to happen, will be assessed in chapter 6. 4.2.1 Measurements of the Intervention The intervention component in the change model is the EDRMS. It is deemed capable of bringing changes to the management of electronic documents and records generated in the city (the component of determinant in the change model). Since the EDRMS is a complicated application with many documents and records management functions, it is impossible for this evaluation to ask questions about every function offered by the system. In light of the evaluation question, which focuses on the effectiveness of the EDRMS from a user's perspective, four aspects, each consisting of a set of functional features, were selected for inquiry. These four aspects, namely, the overall design of the system, creation of electronic documents, documents and records locating, and use of e-mail, are considered as the most basic and fundamental ones and are most influential areas to consider when evaluating the effectiveness of the system. 4.2.1.1 Overall Design Overall design here refers to the general considerations of the layout and features of the EDRMS that influence the use of other functional features in the system. Five statements were employed to collect user opinions: 83 Q4 The [EDRMS] 110 interface is easy to use; Q5 The applications (e.g., Microsoft Office) accessed through [the EDRMS] are sufficient for doing my job; Q6 Saving all city documents and records in [the EDRMS] facilitates information sharing among departments; Q19 "Project Folders" are useful since they allow me to group my documents and/or records together as I wish; and Q30 Learning how to use [the EDRMS] has been easy. The term "interface" in Q4 refers to the window view the user will first see after he or she launches the system. There is no definition provided for it in the questionnaire since its use was approved by the pilot project informants when the questionnaire was tested by them. The word is commonly used and is also employed in training materials designed for users. The EDRMS is fully integrated with the Windows operating system, thus many Windows Explore features, such as drop-down menus and various tool bars, are also available in the system. Apart from this, the interface is split into three panes, and each of them has distinctive functions. Through these panes, users can launch integrated applications, browse the universal file plan, create file folders/workspace, and search, view, and/or edit documents including their metadata. 111 Q4 is asked under the assumption that its user-friendliness has direct impact on users' experience of working with the system. 1 The square brackets here denote the fact that the actual name of the system was used in the questionnaire under the assumption that the respondents are more familiar with it than with a general expression of EDRMS, which, in turn, facilitates the understanding of questions asked. EDRMS is used in this text for the purpose of keeping the participating city anonymous. 111 Documents' metadata are primarily contained in their profiles, which will be discussed in the section on document creation. 84 Q5 is concerned with the fact that only MS Office suite applications are integrated in the system. Under the assumption that a centralized documents and records repository for the entire city facilitates information sharing across administration units, all documents and records generated by city staff are saved into the centralized database, of which the file plan is the manifestation. This means through searching the database or browsing the universal file plan, users can access all or any city documents and records that they have a right to access, without leaving their office. Q6 asks users' opinion about this design idea. The centralized database design, however, changes the way users manage their own documents. When saving a document into the system, users do not have choices in terms of where to place the document - it automatically goes to the centralized database. In the environment of Windows Explore without document management functions, users have the autonomy to decide where to save their documents. When saving, they are prompted to create the file path (for example, on the hard drive or other storage media, either local or shared - folder - subfolder - document). Following the created file path to browse folder directories to find their documents after creation is the primary way of locating documents for most users.112 In the EDRMS, locating documents and records, including those users have themselves created, entails searching the database or 112 Users can also do a keyword search of file folders and/or documents through the search functions provided by Windows, but they are normally used when users cannot remember the file path for the document or folder name. 85 browsing the file plan. To complement this design and to accommodate users' habit of locating documents developed through using personal computers, the EDRMS provides a number of other means for documents locating.113 The Project Folder (referred to in Q19) is one of them. By creating project folders, users can group documents and/or records already existing in the database based on their needs. It is expected to facilitate access to frequently used documents and records for users' convenience. Users learn how to operate the EDRMS in two ways: through the system's help file and with assistance from the RM department. Q30 asks how users learn about the EDRMS in a general way, and thus covers both activities. It is grouped here with other questions about the overall design of the system since assistance from the RM department is separately addressed in the section of measurements of operation. The findings for overall design are reported in Table 4.1. The degree of agreement regarding these statements are coded using numbers 1 to 5, with 1 for "Strongly Agree", 2 for "Agree", 3 for "Neutral", 4 for "Disagree", and 5 for "Strongly Disagree". This manner of coding is consistently employed in the data analysis process for both Likert scale statements and partially close-ended questions. 113 They will be discussed in the section on documents locating. 86 Table 4.1 Overall Design NR 1 2 3 4 5 TNR (1+2)NR/TNR% Q4 10 36 3 1 0 50 92% Q5 15 29 3 3 0 50 88% Q6 26 23 0 1 0 50 98% Q19 9 19 19 2 0 49 56% Q30 11 23 11 4 1 50 68% NR and TNR in the table refer to "number of responses" and "total number of responses", and they will be used in other tables where appropriate. The last column in the table contains percentages of the combined responses for "Strongly Agree" (code 1) and for "Agree" (code 2) in the total number of responses, each representing the overall degree of agreement for one statement, which is considered to indicate a positive attitude towards the feature under evaluation. Without selecting any one of the options provided by the scale statement, respondent ID13 answered Q19 with "don't use". This answer is decided as not in favor of the statement, and therefore the overall positive attitude towards this statement is 56%, 28 out of 50. The percentages in the table show a general satisfaction with the system's overall design, especially with the centralized documents repository (98%), the interface (92%), and the applications provided (88%). Only 56% of respondents, however, think project folders are useful for grouping their documents, including 87 one respondent who does not use this feature. Q19 will also be analyzed with other search functions in the section on documents locating (4.2.3). Respondents' learning experience with the EDRMS, which has a diverse response, will be discussed in light of respondents' background information in the next chapter. 4.2.1.2 Creation of Electronic Documents The creation of electronic documents refers to the three methods of adding documents or records into the EDRMS: a) to generate MS office documents using integrated applications b) to import documents with file formats supported by the EDRMS from network shared drives, local drives, or from external media such as CD ROMs or floppy disks, and c) to save e-mail messages (with attachment(s)) into the system from MS Outlook. The imported documents and saved e-mail messages, if policies and procedures are strictly observed, should be records rather than documents,114 because only business-related documents need to be saved into the system as the city's official records. The documents created in the system using integrated applications can remain as documents until the moment when the author either marks them as records or throws them into the trash can. Despite this difference, all three types of document generation require the completion of document profiles - a compulsory condition for documents to be saved into the system. To profile documents means to supply values for various fields in the pop-up profile 114 See discussion on records and documents in Chapter 1. 88 window, one of which is the classification code field. As mentioned in the methodology chapter, incorrectly classified documents have remained an unsolved problem for the city's RM department ever since the universal file plan was integrated with the system, and classification backlogs have been accumulating after the "classify later" option was introduced into the file plan. 115 Profiling and classifying documents, therefore, forms the center of inquiry in this section. The questionnaire does not ask any questions about the actual use of MS applications out of the consideration that the MS Office suite has long been an integral element of office work, and familiarity with it is required by the job qualifications of these selected respondents. The questionnaire also does not ask questions about how to import records. Importing records into the EDRMS is infrequent because the system has been running for more than 10 years. However, the use of e-mail constitutes an independent aspect of investigation and will be discussed later in this chapter (section 4.2.4). There are two reasons for this decision: 1) e-mail is used in the city as an important communication channel, and 2) it is more complex to save e-mail messages into the system than it is to import documents, especially when e-mail messages have attachments. Four statements in the questionnaire collect information on individual elements required by a document profile. They are: 115 The option "classify later" in the file plan was introduced to cope with the problem of incorrectly classified documents, but it also gives users the chance to avoid classification, and at the same time, save the documents into the system. 89 Q8 Supplying a descriptive title when profiling documents is easy; Q9 Selecting security options when profiling documents is easy; Q12 Classifying documents is easy; and Q14 Setting fields in the document profile with default values reduces the time of profiling documents. There are more elements than the above four in the document profile form, but some of them, such as "Author" and "Entered by," were not included because of their clarity and simplicity.116 These four are considered as applicable for all respondents, and each has a particularly important meaning for assessing the profiling function. The program theory posits that a descriptive title, as referred to in Q8, facilitates locating documents and records. The RM department uses the expression "descriptive title" in its training materials, and therefore this term is used in the statement without definition. Selecting security options (Q9) is done through the feature "Modify Security". It is important since selecting the proper security option protects personal and/or confidential information, assists the protection of record authenticity, and, at the same time, facilitates information sharing since the less restrictive access rights are, the more accessible information will be. Classifying documents is of critical importance for records management purposes. The ease of supplying values for these elements decides the time needed for completing the profile. Q14, setting default values, is asked out of the same concern. Setting default values is designed to help users who mostly deal with the same types of documents to reduce profiling time. 116 From the perspective of creating metadata necessary for documents and records management, these elements are crucial without any doubt. The purpose here is to investigate how effective the system is in terms of facilitating job duties, so the profiling elements considered as difficult or time-consuming require more attention. 90 The findings are presented in Table 4.2, and coding number 6 in the table indicates the user's unawareness of the feature of setting default value. Table 4.2 Creation of Electronic Documents - Document Profiling NR 1 2 3 4 5 6 TNR (1+2)NR/TNR% Q8 25 23 1 1 0 50 96% Q9 23 19 4 4 0 50 84% Q12 10 13 10 17 0 50 46% Q14 17 21 7 1 0 4 50 76% The high positive percentage for Q8 (96%) denotes that users generally find it easy to supply descriptive titles to documents. The degree of agreement is also high (84%) for the ease of selecting security options. The answers to Q12 confirm the RM department's experience and the researcher's observation, because 17 respondents (34%) think classifying documents is difficult and 10 respondents (20%) hold a neutral attitude - making the overall positive attitude towards this statement only 46%. This is considered as a low percentage not only because of the absolute number, but also in the sense that it is low in comparison with other measurements in this evaluation, the majority of which are higher than 50%. Although 76% of respondents think setting default values for document profile reduces profiling time (Q14), 4 respondents answered that they are not aware of that feature. In addition to collecting an indication of general experience regarding classifying, Q12 acts as a contingency question that filters out respondents who 91 are not qualified to answer Q13. Respondents who selected "Neutral", "Disagree" and "Strongly Disagree" (which was not selected by any of the respondents) for Q12, "Classifying documents is easy", were invited to answer Q13, a partially close-ended question. Q13 asks respondents to select items that all apply from a list of classification difficulties, which is exhausted with "others (please specify)". The difficulties are coded using numbers from 1 to 4 as shown below: 1- There are too many levels and too many choices in the file classification system; 2 - Not all of the primaries (categories) in the file classification system are self-explanatory to me, and the explanations (scope notes) of the categories are not linked to these categories; 3 - "Recently Used Files" is not helpful; and 4 - The file classification system does not accommodate my needs. "Recently Used Files" is a feature attaching to the field of classification in the profile form, and it is designed to help find classification codes through recently used files under the assumption that, for a particular user, some classification codes are repeatedly used if he or she only deals with documents of the same nature. The data from Q13 are grouped in Table 4.3: 92 Table 4.3 Classification Difficulties Q13 NR117 TNSD118 27 49 1 2 3 4 NSD119 19 16 7 7 NSD/TNSD% 39% 33% 14% 14% NSD/NR% 70% 59% 26% 26% The above table shows that Difficulty 1, too many levels and too many choices, was selected 19 times, giving it the highest percentage response rate among the selected difficulties (19 out of 49, 39%), and making it a difficulty for the majority of respondents (70%). The next most selected difficulty is Difficulty 2, the one about the construction of primaries in the file plan and the scope notes that explain them, with 59% of respondents considering it problematical. It is understood that it would be very difficult, if it is not impossible, to construct primaries that are completely self-explanatory, on the one hand, and try to make them as short as subject heading entries, on the other. The reason for providing this difficulty in Q13 is that it is the researcher's observation that users do not have access to the scope notes explaining primaries, to which only the RM department has access. The data collected here confirms this observation, as the difficulty was selected 16 times, 33% of the number of selected difficulties. 117 Number of respondents who answered Q13. 118 Total number of selected difficulties. 119 Number of selected difficulty. 93 Difficulties 3 and 4 both were selected 7 times, indicating they are minor difficulties in the users' consideration. This implies, if thinking from the opposite side, that "Recently Used Files" (in Difficulty 3) is helpful for most users and the file plan is comprehensive enough to cover user needs-Five respondents provided additional comments in the space left for "Other (please specify)". Respondent ID12 comments that, "Classifying info is still somewhat subjective. Different people may file the same document under different classifications." This comment does not point to any specific classification difficulty (as the question asked for); instead, it offers the user's observation about the use of a subject-based file plan. The basic structure of the city's universal file plan is hierarchical in nature, constituting four levels: sections, primaries, secondaries, and tertiaries (not for all secondaries), from general to specific. The establishment of sections is based on the functions the city performs such as Administration, Finance, and Legislative Services. However, the primaries broken down from sections, the secondaries broken down from primaries, and the tertiaries broken down from secondaries are all subject areas or subject categories. The observation provides some points for the argument that it is difficult to decide subjects when classifying documents and different people may classify the same document under different subjects. Respondent ID27, who did not select any other difficulties provided, offered his or her experience of classifying documents: "It is only easy to use if you know where to file the info already." It is probable that this seemingly simple sentence 94 indeed implies a two-fold meaning: one refers to the difficulty of classifying documents - you really need to know where to place the document; another refers to the technological convenience provided by the system - it is easy to use once you know where to place it. The same respondent also comments that, "There are also a lot of steps to get to the correct file." This is considered as the same problem as Difficulty 1 in the pre-conceived classification difficulties. Respondent ID34 complains that, "Too much time is spent trying to find the right spot." This respondent also selected Difficulties 1, 2, and 3, indicating that he or she has encountered all these classification difficulties. The suggestion from respondent ID38, "A master list with more detailed description is needed", is considered as the same problem as Difficulty 2, the lack of scope notes for users. Respondent ID48 points out a specific difficulty in relation to the nature of his or her job duties: "Often I am working on a document that has come from someone else, so I'm not always aware of exactly what the document is about and how it should be classified." This explains the fact that this respondent reported that the file plan did not meet his or her needs (Difficulty 4). 4.2.1.3 Documents and Records Locating Measurements of documents and records locating focus on EDRMS search and browse functions, including other features facilitating the location of documents and/or records. The statements used to collect the information are listed below, and the findings are reported in Table 4.4. Coding number 6 indicates the respondent's 95 unawareness of the retrieval feature asked. Q15 Finding documents or records in [EDRMS] is easy. Q17 "Quick Searches" are convenient because the queries I formulate can be saved for later use or edited for new use. Q18 Sorting search results (e.g., sorting by columns like document title) helps me find desired documents and/or records. Q19 "Project Folders" are useful since it allows me to group my documents and/or records together as I wish. The "Quick Searches" feature allows the user to save search criteria frequently performed for quick access and the saved "quick searches" enables other users with assigned access rights (such as project members) to perform the same search without re-formulating search criteria. Sorting search results is designed to help users in situations where a large number of hits are returned and the desired responses are not readily found. Sorting is done by selecting columns such as title, date, or author, depending on user needs. Table 4.4 Documents and Records Locating NR 1 2 3 4 5 6 TNR (1+2)NR7TNR% Q15 10 23 12 4 1 50 66% Q17 16 18 11 1 0 4 50 68% Q18 17 15 14 0 0 4 50 64% Q19 9 19 19 2 0 49 56% 96 Table 4.4 shows that users' responses to the general statement on finding documents in the system (Q15) is consistent with their opinions on other specific document-locating features ("Quick Searches", "Shorting", and "Project Folders") - they are almost at the same level of agreement (56%-68%). The comparatively large numbers of "Neutral" for all four statements indicate the users' uncertainty when using these features. Also worth noting is that, for Q17 and Q18, four respondents express that they are not aware of "Quick Searches" and "Shorting". To further the above inquiry, a partially close-ended question, Q16, was asked to identify respondents' preferred methods of locating documents. The findings are provided in Table 4.5 with coding numbers 1 to 5 representing the following search methods: 1 - Single field search in the profile (e.g., document number or author or title); 2 - Multi-fields search in the profile (e.g., document number and author and title); 3 - Recently Edited Documents; 4 - Browsing the file plan; and 5 - Advanced Search (easy search, content search, custom search) The feature "Recently Edited Documents" lists the last 30 documents the user edited. The advanced search allows the user to form more sophisticated search formula through the use of Boolean and/or proximity operators. 97 Table 4.5 Documents and Records Locating Methods Q16 1 2 3 4 5 NSLM120 35 30 36 5 20 TNSLM121 126 NSLM/TNSLM% 28% 24% 29% 3% 16% TNR 50 NSLM/TNR% 70% 60% 72% 10% 40% In the table, locating methods 1, 2, and 3 have very close selection percentages (28%, 24%, and 29%), with 3, "Recently Edited Documents", slightly higher (29%). Locating method 5, advanced search, is considered by only 20 respondents as one of their preferred methods. Consistent with the identification of classification difficulties in the previous section, locating method 4, Browsing the file plan, ranks as the least preferred method (3%), and only 10% respondents (5 out of 50) selected it. Among these five respondents, two selected "Strongly Agree" and three selected "Agree" for Q12, "Classifying documents is easy", confirming that they are good at using the file plan. 4.2.1.4 Use of E-mail The city's e-mail application, MS Outlook, is not integrated with the EDRMS. Saving e-mail messages into the system is not mandatory due to the huge volume of messages created every day and the transitory nature of most messages. The 120 Number of selected locating method. 121 Total number of selected locating methods. 98 RM department advises employees to save business-related e-mail messages into the system as the city's official records. Saving e-mail messages in electronic format is promoted as a more reliable method than printing them out in the sense that electronic versions capture many other technological data in addition to textual content, and it also makes the messages electronically searchable. The use of e-mail in this section refers to attaching documents or records to e-mail messages and saving e-mail messages (with attachment(s)) into the system. Three statements used for opinion collection are listed below: Q20 Attaching documents or records in [the EDRMS] to an e-mail message is easy. Q21 Saving e-mail messages in [the EDRMS] is easy. Q22 Saving e-mail messages that have attachment(s) in [the EDRMS] is easy. Saving e-mail messages into the EDRMS means to select and profile the messages; the "Application" field in the profile automatically captures their format as MS Outlook documents. Saving e-mail messages that have attachment(s) into the EDRMS can be done in two ways: to save the message and the attachment(s) together and therefore profile them as one record, or to save the message and the attachment(s) separately and profile them separately. Since profiling documents lays the foundation for documents locating, the attachment(s) saved with the messages cannot be independently searched. Data are reported in Table 4.6. 99 Table 4.6 Use of E-mail NR 1 2 3 4 5 TNR (1+2)NR/TNR% Q20 20 21 3 5 1 50 82% Q21 10 20 16 1 1 48 62.5% Q22 9 16 21 2 1 48 52% Respondent ID13 did not make any selections for Q21 and Q22 since he or she "have not done this (saving e-mail message)". While similarly answered Q21 with "never done", respondent ID33 selected "3", which means "Neutral", for Q22. This answer is considered conflicting with the "never done" answer for Q21 since if one has saved e-mail messages with attachment(s), one must have also saved e-mail messages into the system. The "Neutral" answer therefore is not included into the total number of responses for Q22, which makes the total numbers of responses for both questions 48. This number is consequently used for calculating the percentages. The percentages of agreement descend from attaching e-mail message, saving e-mail message, to saving e-mail message with attachment(s), confirming the assumption that the difficulty for, or time needed to, perform these functions increases along the same order. 4.2.2 Measurements of Outcomes The expected outcomes from the EDRMS program are, as identified in the program theory, increased productivity, enhanced information sharing, reliable 100 and authentic electronic records, and reduced paper records volume. Two statements collect information about office productivity and information sharing from all identified respondents, and five statements collect information about office productivity, information sharing, and paper records volume from the comparison group, who answer the five additional questions based on their working experience in the city before and after the implementation of the EDRMS. The questions about whether the records in the system are reliable or authentic are difficult to ask from users' perspectives, and it would be more logical to address them to the RM department who are charged with the responsibilities of ensuring records reliability and authenticity. In this thesis, therefore, the assessment of records reliability and authenticity is addressed in the form of assessing users' understanding of certain features that are designed for reliability and authenticity as required by the DoD5015.2 standard. The assessment of understanding will be discussed in section 4.3.1.1 in connection with the trainings provided by the RM department. 4.2.2.1 Outcome Measurements from All Respondents Two questions, Q6 regarding information sharing and Q24 regarding office productivity, are listed below. The findings are reported in Table 4.7. Q6 Saving all city documents and records in [the EDRMS] facilitates information sharing among departments/divisions/sections. Q24 In general, the functions provided by [the EDRMS] help me with my job tasks. 101 Table 4.7 Outcome Measurements from All Respondents NR 1 2 3 4 5 TNR (1+2)NR/TNR Q6 26 23 0 1 0 50 98% Q24 20 25 5 0 0 50 90% Q6 was also presented in section 4.2.1 in connection with the system's overall design of having a centralized documents and records repository. It is also presented here under the assumption that users' agreement with this design will motivate their active participation in sharing information. However, this statement by itself - even it has a very high percentage of agreement (98%) - does not indicate an achievement of the outcome. The outcome of information sharing in this thesis is assessed in conjunction with the measurements of document/record locating and cooperation among working units, which are assessed in the comparison group. It will be impossible to share information if it cannot be located. Cooperation entails information sharing. The findings of these three variables will be interpreted and analyzed together in the next chapter. Office productivity is another major goal of the EDRMS program, and the high percentage from Q24 (90%, without any selection of "Disagree" or "Strongly Disagree") demonstrates a great degree of achievement of this goal. Both outcomes will be further assessed with the comparison group. 4.2.2.2 Outcome Measurements from the Comparison Group The five statements used for the comparison group are formulated as follows and findings are presented in Table 4.8. 102 In comparison with the situation before [EDRMS] was implemented, I find that Q33 less time is needed to retrieve documents or records; Q34 the volume of paper records is reduced; Q35 using the EDRMS speeds up my completion of work, because I now can access documents and records (that I have the right to see) from any computers that are connected to the City's computer network, regardless of time or location; Q36 co-operating with other departments becomes easier, because I now can access documents and records created by other departments, and vice versa; and Q37 my office productivity has increased. As stated in the previous section, information sharing and office productivity are two major goals of the EDRMS program. Q36, from another angle, collects opinion on information sharing in a specific situation. Q33, Q35, and Q37 are all about office productivity, with the former two concerning specific aspects and the latter summarizing the overall experience. To reduce the volume of paper records is also a major goal of the program since the assumption is, by reducing the volume of paper records, costs will also be reduced. Table 4.8 Outcome Measurements from Comparison Group NR 1 2 3 4 5 TNR (1+2)NR/TNR% Q33 12 16 1 1 0 30 93% Q34 8 5 6 10 1 30 43% Q35 10 17 3 0 0 30 90% Q36 10 17 2 1 0 30 90% Q37 9 13 7 1 0 30 73% 103 The high level of agreement of Q36 (90%) indicates a great degree of achievement of the outcome of information sharing in the case of co-operation. The two aspects of office productivity, reduced time for finding documents (Q33) and unlimited time and location for information access (Q35), also have very high levels of affirmation (93% and 90%). The overall experience on increased office productivity, however, has a comparatively lower percentage (73%), with 7 uncertain respondents and 1 respondent disagrees. It is interesting to notice that Q34, the volume of paper records is reduced, has the lowest percentage of agreement in the entire evaluation. 4.3 Assessing the Action Model Only two components in the action model, the intervention and service delivery protocols and the target populations, were evaluated in this thesis project because they are the only components reasonable for users of the system to provide opinions. The other components such as implementation organizations, implementers, and associate organizations and community partners were articulated in the previous chapter for the purpose of understanding the program and the program theory, but were decided not for evaluation. The RM department in the city and the partners of the program such as the IT department were considered as more suitable informants for evaluations of these components than the users of the system. It is necessary to point out that all these components have impacts on the outcomes of the program, and they too require evaluation. 4.3.1 Measurements of Intervention and Service Delivery Protocols The intervention and service delivery protocols are identified in the action model as the program manual and the EDRMS-related services the RM department provides. The program manual spells out the goals of and the rationales for the program and explains the functional features of the EDRMS. Together with trainings provided by the records management personnel, the program manual is considered the primary source for users to understand records management concepts and principles in relation to electronic records in an EDRMS environment. The EDRMS-related services provided by the RM department include trainings and internal guidelines, which are embedded in the RM department's day-to-day working procedures. The measurements reported below are organized as: 1) users' understandings of records management concepts and principles and 2) users' opinions about the service delivering activities currently performed by the RM department. 4.3.1.1 Understandings of RM Concepts and Principles The statements used to measure the understanding of RM concepts and principles are listed below: Q7 Creating profiles for documents is necessary; Q10 "Modify Security" is useful for sharing information (when full access is assigned to the document) and protecting confidentiality (when selected access rights are assigned to the document); Q11 Classifying documents when profiling documents is necessary; Q23 "History" is useful because it includes information about a document and 105 what has happened to it since it was created (e.g., information about who accessed it and when); Q31 When using documents in [the EDRMS], I trust them as reliable information sources; and Q32 "Mark as a Record" is a necessary feature of [the EDRMS]. The above questions about "necessary" and "useful" features of the system (Q7, Q10, Q11, and Q23) test users' understandings of why they are required to perform some activities (Q7, profiling and Q11, classifying) and why certain features are provided by the system (Q10, "Modify Security" and Q23 "History"). These activities and features are new to the EDRMS users since they did not appear in their past working environment employing a paper records system.122 They are critical for managing electronic documents and records. Profiling documents is indeed the procedure of creating metadata, and a profile containing sufficient information can serve many purposes including enhancing documents locating, ensuring records reliability and authenticity, and facilitating long-term preservation. 123 Such importance may not be that obvious to users. For general users, who are not trained information professionals, a keyword search is the most common means for locating electronic resources. In addition, general users are not concerned with the authenticity and long-term preservation of electronic 122 The feature "Modify Security", however, may be an exception to this statement. Depending on the legal environment, different jurisdictions may have different requirements in terms of setting classification levels for documents. In China, for example, its Keeping Secrets Law stipulates that the author of the document has the responsibility of setting the classification level. 123 Some examples of the different purposes of metadata are provided here. The descriptive information recorded in the fields such as "Title", "Author", can be used for documents locating; the identification of author, use of "Modify Security", and checking on "History" help to decide the degree of records reliability and authenticity. The file formats recorded in "Application" could be helpful when developing a long-term preservation strategy. These examples are not meant to be mutually exclusive. Metadata types in fact overlap with each other in many cases. The types of metadata for records management purposes are discussed in the ISO 23081-1:2006 metadata standard. 106 resources as part of their day-to-day work. The rationales for creating mandatory profiles need to be explicitly articulated for their understanding. "Modify Security" and "History" are another two features useful for realizing both documents and records management functions, but they may not be understood by users in the same way as they are by records management professionals. Classifying documents in the past served as the primary method for users to locate documents. Therefore it was easier for them to understand why classification was needed. In today's electronic environment, keyword searching greatly reduces users' reliance on file classification to find desired documents or records. Classification remains vital for records management functions, since records need to be managed by classes, that is, they need to be scheduled and disposed by class. It is also true, of course, that classification is a means to assemble records in aggregations (of an office, series, file, etc.) that exhibit their relationships. Statements 31 and 32 assess users' understanding of reliability and authenticity more directly. It is assumed that information reliability is every user's concern since it has direct impact on fulfilling their job duties. The feature "Mark as a Record" is considered as the most effective method of ensuring records authenticity in the electronic environment since the execution of it makes the documents read-only. The program manual explains it as the equivalent of "sending to file" in the paper system. When a document is marked as a record, the system flags this change of status using a small red dot on its document format 107 icon.124 From that point onward, the record is unalterable and no one will be able to change or edit it, including the author. The record can be accessed for viewing and its content can be saved as another copy for editing or other uses. Considering it a critical feature of the system, the RM department produced a brochure offering guidelines on when documents should be marked as records.125 The findings for this section are reported in Table 4.9. Coding number 6 for Q23 and Q32 indicates the respondents' lack of awareness of the feature; however, it indicates "I have never thought about this" for Q31. Table 4.9 Understanding of RM Requirements NR 1 2 3 4 5 6 TNR (1+2)NR/TNR % Q7 31 15 2 2 0 50 92% Q10 29 19 2 0 0 50 96% Q11 19 25 2 4 0 50 88% Q23 22 25 1 0 0 2 50 94% Q31 12 29 5 1 0 3 50 82% Q32 18 15 11 3 1 2 50 66% By a quick glance at the percentages in the table, a general conclusion can be reached that the respondents have a very good understanding of records management requirements. Despite the fact that many of them report difficulties of classifying documents, 34 respondents (88%) agree that classifying documents is necessary and not one respondent strongly disagrees with this statement. The 124 The icon that denotes it is a Word document or a Power Point presentation. 125 See discussion on "Mark as a Record" in chapter 5. 108 necessity of profiling documents and the usefulness of "History" also have very high levels of agreement (92% and 94%). There are, however, two respondents who report a lack of awareness of the feature "History". The best understood feature, or the most popular one, is "Modify Security". It is selected by 48 respondents (96%) for its usefulness in allowing information sharing on the one hand, and protecting sensitive information on the other. Compared to the percentage for Q31 (82%), which concerns information reliability, the statement on the necessity of marking documents as records (Q32) has a lower level of agreement (66%). Three respondents report that they have never thought about whether the information they use in the system is reliable or not, and 2 respondents express that they are not aware of the feature "Mark as a Record". 4.3.1.2 Service Delivering Activities The service delivering activities currently carried out in the EDRMS program are measured through three statements and two partially close-ended questions, which are respectively listed before the tables reporting findings. Three statements: Q25 It is important to have a staff person in each department with assigned responsibility to manage documents and records; and Q26 Using the titling guidelines facilitates document and records retrieval; Q27 Assistance from the RM Department (such as training sessions, manuals and brochures, and help provided through phone calls) has been necessary for 109 my understanding and use of [EDRMS]. Q25 is a policy-related statement concerning the fact that in an electronic working environment, records management responsibilities are shared between records management professionals and individual documents creators. It intends to collect opinions from these identified key users who have the responsibility to manage documents and records with regard to this shared responsibility. Q26 assesses users' understanding of the benefits of using titling guidelines, which the RM department develops for documents creators to supply descriptive titles for the documents they create. Assistance from the RM department is considered in the action model one major component impacting the success of the program. Q27 gathers comments about the services the RM department provides. Table 4.10 Service Delivering Activities - General NR 1 2 3 4 5 TNR (1+2)NR/TNR% Q25 13 13 13 9 2 50 52% Q26 18 22 10 0 0 50 80% Q27 25 21 1 3 0 50 92% There are 46 respondents who think the assistance from the RM department is necessary for their understanding and use of the EDRMS (25 "Strongly Agree" and 21 "Agree"), making the positive attitude a high percentage of 92%. Respondents also demonstrate a good understanding about the use of titling guidelines (80%), with no one respondent selecting "Disagree" or "Strongly Disagree". About half of the respondents, however, feel uncertain or disagree no (including two who selected "Strongly Disagree") with the importance of having a staff person in each department responsible for managing documents and records. Data from the following two partially close-ended questions are summarized into Table 4.11 and Table 4.12, respectively: Q29 The assistance currently provided by the RM Department for using the EDRMS is 1 - Extremely helpful 2 - Helpful 3 - Neutral (neither helpful nor not helpful) 4 - Not helpful 5 - Totally useless Table 4.11 Service Delivering Activities - Current Assistance NR 1 2 3 4 5 TNR (1+2)NR/TNR% Q29 25 16 8 0 0 49 84% The level of agreement regarding the usefulness of assistance from the RM department presents also a high percentage, and no one respondent selected "Disagree" or "Strongly Disagree". Since respondent ID38 did not select any answer for this question, the total number of response was calculated as 49 and the positive attitude percentage is 84%, or 41 out of 49. Q28 further explores the types of assistance users prefer. in Q28 The following types of assistance have been most valuable to me (select all that apply) 1 - Classroom computer training 2 - One-on-one instruction (including telephone and on-site) 3 - Training videos (TV snacks) on the Intranet 4 - Manuals and brochures 5 -Other (please specify) Table 4.12 Service Delivering Activities -Types of Assistance Q28 1 2 3 4 5 TNSTA126 NSTA127 44 34 4 28 0 110 NSTA /TNSTA% 40% 31% 3.6% 25.4% Among the four types of assistance currently available in the city, Type 1, Classroom Computer Training, ranks as the most preferred training method (40%). Type 2, One-on-One Instruction, and Type 4, Manuals and Brochures, follows as the second and third (31% and 25.4%). The training videos (Type 3) mounted on the city's intranet recording consultant's instructions about the system and its features, was selected the fewest times (4 out of 110, 3.6%). 4.3.2 Measurements of the Target Population The target population identified in the action model constitutes all employees in the city for whom the EDRMS is a tool of managing job-related documents and records. The current evaluation only collects information from a portion of this group, as stated in the methodology chapter, these being the key users of the 126 Total Number of Selected Types of Assistance. 127 Number of Selected Types of Assistance. 112 system whose job duties include the management of documents and records. The term "target population" is used for the portion of the group in discussions in this evaluation and its findings. Two aspects of the target population were assessed: one is about their work-related backgrounds and another is about their experience of learning how to use the EDRMS. Three close-ended questions and one statement were used in the measurements. Q1, Q2, and Q3 asked, respectively, how long they had worked with the city, how long they have worked with the EDRMS, and how they view their computer skills. The findings are reported in Table 4.13. Table 4.13 Measurements of the Target Group - Backgrounds Q1 <10 >10 NR 20 30 Q2 1 2-4 5-7 8-10 NR 3 6 18 23 Q3 V. Strong Strong Sufficient Weak V. Weak NR 21 22 7 0 0 There are 20 respondents who have worked with the city for fewer than 10 years (inclusive) and 30 for more than 10 years. Most of them have worked with the EDRMS for a not-too-short time period as the numbers of respondents who select 5-7 years (NR=18) and 8-10 years (NR=23) indicate. It is impressive that 21 respondents rate their computer skills as "Very Strong" and 22 rate as "Strong". With the rest of the respondents (7) rating their computer skills as "sufficient for 113 job requirements", no one rates their skills as "weak" or "very weak". Q30, Learning how to use [the EDRMS] has been easy, is reported in Table 4.14 as the following; which results in a not high percentage. Table 4.14 Measurements of the Target Group - Learning Experience Q30 1 2 3 4 5 TNR (1+2)NR7TNR% NR 11 23 11 4 1 50 68% The data collected from Q1 are themselves not intended to be analyzed for the purpose of assessing the program theory; instead, Q1 is a contingency question asked for the purpose of constructing a comparison group of respondents. Respondents who have worked with the city for more than 10 years were invited to answer questions in the additional section, Part III, of the questionnaire. The intention of constructing a comparison group and designing an additional section in the questionnaire is to collect opinions about the EDRMS from users who have experience working with the city before and after the implementation of the system. This chapter explains the rationales behind the construction of questions in relation to the program theory, summarizes data in accordance with evaluated program components, and reports descriptive statistics of individual variables. The (1+2)NR/TNR% column in the tables indicate users' positive attitudes to the features offered by the EDRMS, the program implementation, and the outcomes. 114 ( Classification difficulties, user preferred document-locating methods and types of RM assistance are presented in more detail. Further discussions based on the above findings will be the focus of the next chapter, Discussions and Implications. 115 Chapter 5 Discussions and Implications The previous chapter analyzes data collected by the questionnaire and summarizes them in tables. Each table reports findings in the form of individual variables, and the fourteen tables together generate the overall assessment about the EDRMS program under evaluation. This chapter will discuss the issues emerging in the findings, analyze possible relationships between two or more variables, and identify areas requiring further attention. 5.1 Document Classification As Table 4.2 displays, supplying classification codes for documents in accordance with the city's universal file plan (Q12) is considered by users as the least easy one to do among the three profiling fields evaluated (Q8, Q9, and Q12). Among 50 respondents, 10 express uncertainty about their experience with classifying documents (code 3, "Neutral"), which may indicate that sometimes classifying is easy or difficult. While no respondents select "Strongly Disagree", 17 respondents explicitly express that classifying documents is difficult (code 4, "Disagree"). These 27 respondents who do not think classifying documents is easy further identified classification difficulties through answering a partially close-ended question. This section discusses respondents' backgrounds, the design of the file plan, and the time for classification with respect to these classification difficulties. 116 5.1.1 User Background Users' backgrounds are considered as one factor influencing their experience of classifying documents. Three questions in the questionnaire collect users' background information: Q1, the time period working with the city, Q2, the time period working with the EDRMS, and Q3, the self-rated computer skills. The data Q1 collects appear less relevant to the analysis of classification, and are therefore not discussed here. The background information collected by Q2 and Q3 are analyzed with Q12, Classifying documents is easy, in the following tables. Table 5.1 Classification and Time Period Working with the EDRMS Q2 1 year 2-4 years 5-7 years 8-10 years NR 3 6 18 23 Q12 Classifying documents is easy128 N(SA+A) 2 0 9 12 NDA 1 4 4 8 NN 0 2 5 3 N(SA+A)/NR% 66.7% 0% 50% 52.2% NDA/NR% 33.3% 66.7% 22.2% 34.8% NN/NR% 0% 33.3% 27.8% 13% The analysis of classification experience with time period of working with the system (which includes the use of the file plan) is based on the assumption that the more time the users spend on using the file plan, the more familiar they are with the classification, and therefore the easier classifying documents will be. In 128 The statements or questions in the tables of this chapter may be paraphrased for the purpose of fitting into the table. 117 the table, numbers of selections of the combined "Strongly Agree" and "Agree" (N(SA+A)), "Disagree" (NDA), and "Neutral" (NN)129 are listed in relation to the four time periods, each of which has a number of respondents, namely, 3, 6, 18, and 23, respectively. The percentages of selections are calculated for each time period and then compared across the four time periods. These percentages apparently indicate that no correlation can be established between these two variables, that is, there is no direct proportional relationship between classifying documents and the time factor as premised by the assumption. Among the respondents who have worked with the system for only one year, 66.7% strongly agree or agree that classifying documents is easy, and among the respondents who have worked with the system for 2-4 years, no respondents strongly agree or agree with the statement. While the percentage of the combined "Strongly Agree" and "Agree" for the group that has worked for "8-10 years" is slightly higher than that for the group that has worked "5-7 years" (52.2% vs. 50%), both percentages are lower than that for the group of one year (66.7%), which contradicts the assumption that longer experience eases the difficulty of classifying documents. The contradiction is also evident in the percentages disagreeing with the statement. Respondents with 2-4 years working time with the system disagree the most (66.7%) among the four user groups, and the user group with a longer working time with the system (5-7 years) disagrees the least (22.2%). The "Neutral" percentages generally descend when the length of time period increases, except for the first group. This may imply that time is not a definitively influential 129 No selection of "Strongly Disagree" in Q12. 118 factor for the ease of classifying documents, but it may enhance users' understanding of the file plan, since there is a tendency that the longer they have worked with the system, the less uncertain they are about classifying records. Another assumption employed to analyze classification is that users' computer skills may have an influence on their classification activities, that is, that stronger computer skills make classification easier. Computer skills in this research refer to the general understanding of the Windows operating system and commonly used applications for personal computers. This assumption is based on the fact that the file plan is integrated with the EDRMS as a structured, expanding-collapsing categories tree with some classification-help features, such as Recently Used Documents, which requires users to understand both the operation system and the application. The respondents generally regard themselves as having good computer skills. In answer to Q3, all said that their skills were at least sufficient. Following the same analyzing pattern, numbers of selections for Q12 are listed in relation to the three groups of respondents who possess different levels of computer skills. Percentages are calculated for each group and compared across the three groups. 119 Table 5.2 Classification and Computer Skills Q3 Self-rated computer skills V. Strong Strong Sufficient NR 21 22 7 Q12 Classifying documents is easy N(SA+A) 10 10 3 NDA 6 7 4 NN 5 5 0 N(SA+A)/NR% 47.6% 45.5% 43% NDA/NR% 29% 32% 57% NN/NR% 23.8% 22.7% 0% The above table indicates that respondents with stronger computer skills tend to agree or agree more strongly with the statement: 47.6% in the "Very Strong" group, 45.5% in the "Strong" group, and 43% in the "Sufficient" group. The percentages of disagreement are consistent with this tendency as they increase from 29%, to 32%, and to 57% when the levels of rated computer skills go down. However, even in the "Very Strong" and "Strong" groups, there are large numbers of "Neutral percentages" (23.8% and 22.7%), indicating there are other factors than computer skills that affect users' experience of classifying documents. Much more goes into determining users' capability than the two factors analyzed here indicate. Other factors, such as adequacy of training, undoubtedly contribute to the experience of using the file plan. Moreover, the number of years respondents have worked with the system may not reveal how experienced they are with using the file plan. While respondents all have responsibilities of 120 managing records, the degree to which they use the file plan obviously varies depending on the work they do. Jobs focusing on records keeping require more time for classification than jobs focusing on other administrative tasks.130 This means that users who have worked with the EDRMS for a shorter period of time may have spent more time on classification than some who have worked with the EDRMS for a longer period of time, and this could affect the analysis results reported in Table 5.1. In addition, the analysis results could also be affected by the type of documents or records the user classifies. If the user's job requires he or she to deal with documents of the same nature, he or she would know the classification categories for those types of documents very well after using the file plan for a while, which then would make the subsequent classifications easier and faster. However, in the situation depicted by respondent ID48, where he or she often works with documents from someone else and the subject(s) of these documents require content scrutiny every time for the purpose of classification, the time period of working with the system and of using the file plan would appear to be less relevant to his or her classification activities. As reported in the answer to Q13, the problem is that he or she does not know "exactly what [the] document is about and how it should be classified." Given these considerations, further questions about users' job tasks and the nature of documents they tend to classify may generate a more accurate understanding on the relationship between the time factor and classification. Given the time and research method constraints, however, questions at a more general level or questions serving multiple 130 This is quite apparent in their job descriptions. 121 purposes take priority in this research,131 and areas requiring more explorations are left to further research. The identification of such areas, as they emerge from other discussions in this chapter, is one of the goals of the current research. 5.1.2 Design of the File Plan The above section addresses classification through analyzing it in connection with users' computer skills and experience working with the system. As the discussion demonstrates, experience using the system cannot alone explain classification problems. It is also true that users with strong computer skills also encounter classification difficulties. This leads to the thinking about the design of the city's universal classification system. As introduced in the previous chapter, the city's universal file plan considers itself a system with mixed functions and subjects, as the city's major functions form sections at the highest level and subject areas constitute primaries, secondaries, and tertiaries at lower levels. 132 Sections are organized based on the division of administrative and operational functions, and subject areas are arranged in alphabetical and/or chronological orders. The following points are derived from users' indication of the reasons why they find classifying records difficult. a) The most selected classification difficulty is that there are too many levels and choices in the file classification system (Difficulty 1 in Q13); b) Respondent ID34 complains that "too much time is spent trying to find the right spot," 131 For example, Q2, the time period of working with the EDRMS, is also used to analyze user search experience with the system. 132 In addition to subject files, secondaries in the file plan may also include case files. Case files are left out of the discussion, because classifying them appears to be less problematical. 122 c) Respondent ID12 comments that "classifying info is still somewhat subjective. Different people may file the same document under different classifications," d) Respondent ID27 comments that "it is only easy to use if you know where to file the info already." e) Seven respondents report that the file classification system does not accommodate their needs (Difficulty 4 in Q13); and f) As indicated by the above discussion on comments provided by respondent ID48, the natures of the job duties users perform impact their classifying activities. These points can then be summarized into three areas: a) the design of the file plan as a universal file plan, b) the design of the file plan as a subject-based file plan, and c) the design of the file plan based on user needs analysis. Design of the File Plan as a Universal File Plan A universal file plan is a records classification system or classification scheme designed and used for the entire organization. As a critical RM tool, it offers many advantages from a management perspective. A universal file plan identifies all records generated in the organization in accordance with established logic and categories, and therefore enables systematic and consistent records management. Identifying classes of records forms the basis for establishing and executing retention schedules, facilitates managing vital records and avoiding risks, and protects and preserves corporate intellectual assets. A universal file plan also helps overcome inconsistencies in classifying records typically caused by different classification practices employed by different departments or offices in 123 the organization, which, in turn, facilitates information locating and sharing. All these advantages, however, depend on users' assigning the correct classification codes to each and every document in the classification system. An EDRMS environment requires all users of the system to classify their own documents as opposed to the traditional centralized paper records environment in which file clerks classify records for records creators. As the Unite Kingdom National Archives puts it, "with the advent of ERMS, we are all filing clerks now."133 For general users who lack classification and indexing skills that, in the past, were possessed by specialists, a universal file plan for all documents generated in the organization can become overwhelming. Respondents often report that they had to browse the entire file plan in order to classify one document, that "there are too many levels and too many choices" in the file plan, and that "too much time" is spen[t] on finding the right classification code. Although a corporate wide classification system is no doubt desirable, when individual employees have to use the file plan, more considerations should be given to the assistance users need to classify records in an appropriate and effective manner. Design of the File Plan as a Subject-Based File Plan Classification, as a means of sorting and categorizing objects, has been employed by many disciplines.134 While the development of classification systems in the records management field has been less standardized than it has 133 The United Kingdom National Archives, "Business classification scheme design"; available from http://www.nationalarchives.qov.uk/electronicrecords/advice/pdf/bcs toolkit.pdf; Internet; accessed 20 August 2006. 134 Vanda Broughton, Essential Classification (New York : Neal-Schuman, 2004), 4-5. 124 in libraries, and records classification systems are mostly locally constructed, their construction has also often been based on subject classification principles. Classification by subject requires understanding of the objects being classified and familiarity with the logic and structure of the classification system. For discrete or stand-alone items such as books and published maps, the subjects of which can be identified through examining them at hand, the subject-based classification system serves both management and retrieval purposes very well. It has proved to be less advantageous for classifying records, which usually require additional information to be understood. Records are either instruments created to carry out practical activities or by-products resulted from the process of carrying out those activities. To understand records requires, in addition to contents of the records, contextual information on their generation.135 Classification solely based on the subjects of individual records could obscure their relationships with other records generated by the same activities and thus hinder records management and archival activities. Function-based classification systems emphasizing the context of records have been receiving increased attention in recent years. The idea of building classification systems according to administrative functions rather than subjects is nothing new. T.R. Schellenberg described the "function-activity-transaction" structure for developing a classification system in 1956,136 but it was not adopted widely by organizations in any rigorous way. To distinguish these two types of classification systems in a general manner, 135 See more discussion on the concept of record in chapter 1. 136 Theodore. R. Schellenberg, Modern Archives: Principles and Techniques (Chicago : University of Chicago Press, 1956), 56. 125 a subject-based classification system can be viewed as using a bottom-up approach, since it starts with individual records. By contrast, a function-based classification system employs a top-down approach, since it starts with the structure of the organization. In subject-based classification systems, subject terms abstracted from content analysis constitute the classification categories and the relationships among these terms are semantic in nature, that is, in relation to the same subject, narrower terms are grouped under a broader term. Function-based classification systems are constructed through analyzing the organization in relation to its external and internal environments and decomposing the identified functions to records-creation level with intention of charactering records in their originating contexts. It is advocated as the best practice of managing records, since it conforms to the processes of conducting business or fulfilling tasks. For the purpose of this thesis, two examples of types of function-based classification system are identified: the classification system that is built upon the Step B: Analysis of Business Activity in the Australian DIRKS manual (Designing and Implementing Recordkeeping Systems: Manual for Commonwealth Agencies)™7 and the classification system that is built upon the Canadian BASCS methodology (Business Activity Structure Classification)™8. National Achieves of Australia, "DIRKS Manual"; available from http://www.naa.qov.au/recordkeepinq/dirks/dirksman/contents.html; Internet; accessed 17 August. 2006. Library and Archives Canada, "BASCS Guidance"; available from http://www.collectionscanada.ca/information-manaqement/002/007002-2089-e.html; Internet; accessed August 17 2006. 126 The eight-step Australian DIRKS manual is developed by the National Archives of Australia (NAA) in support of commonwealth agencies' compliance with the national records management standard, AS 15489-2000. It was the first and most comprehensive methodology made publicly available that addresses the construction of function-based classification system. Step B of the manual, Analysis of Business Activity, aims to develop "a conceptual model of what [the] organisation does and how it does it".139 This method utilizes two types of analysis: analysis of business activities in light of the organization's mission and goals, which identifies functions and activities, and analysis of business processes at the operational level, which analyzes business transactions. Following the AS 15489-2000 standard, the manual defines function, activity, and transaction as follows.140 Functions are the largest unit of business activity in an organization. They represent the major responsibilities that are managed by the organization to fulfill its goals. Functions are high-level aggregates of the organization's activities. Activities are the major tasks performed by the organization to accomplish each of its functions. Several activities may be associated with each function. Transactions are the smallest unit of business activity. They should be tasks, not subjects or record types. Transactions will help define the scope or boundaries of activities and provide the basis for identifying the records that are required to meet the business needs of the organization. 139 National Achieves of Australia, "DIRKS: Step B - Analysis of business activity"; available from http://www.naa.qov.au/recordkeepinq/dirks/dirksman/step B.html; Internet; accessed 18 August 2006. 140 The explanatory information about "function", "activity", and "transaction" was most directly taken from, National Achieves of Australia, "DIRKS: Step B - Analysis of business activity"; available from http://www.naa.qov.au/recordkeepinq/dirks/dirksman/step B.html; Internet; accessed 18 August 2006, but re-organized and sometimes paraphrased for discussion purpose. 127 The immediate product resulting from this analysis is the Business Classification Scheme (BCS), a hierarchical model of the relationship between the organization's functions, activities and transactions. A BCS representing the identified functions, activities, and transactions, including scope notes and date ranges, acts as the foundation for the development of recordkeeping tools, from which the records classification scheme and another important classification tool, the function thesaurus, can be developed.141 A records classification scheme built on a BCS has a hierarchical structure with three levels, namely, function, activity, and transaction, in parallel with the levels in the BCS, and basically employs the same terms used to describe the functions and activities in the BCS for its first and second levels. The third level, called the transaction level or topic level is formed by analyzing the flow or steps in transactions, and it is left to users to describe.142 The other important classification tool, the functions thesaurus, is a list of terms purposefully selected as preferred ones in depicting the functions, activities, and transactions identified by the BCS. These terms are accompanied by scope notes explaining their meanings and usages within the broad business contexts. The functions thesaurus displays these preferred terms (also called authorized terms) in an alphabetical structure; each term in the list, however, is placed in the hierarchy of "function-activity-topic-subtopic". For user convenience, a functions 141 The terms "records classification scheme" and "functions thesaurus" are derived from the Australian Standard for Records Management, AS ISO 15489, Part 2, Clause 4.2.2.1. Australian Standard for Records Management AS ISO 15489 - 2002. Part2, Clause 4.2.2. 142 National Archives of Australia, "Overview of Classification Tools for Records Management"; available from http://www.naa.aov.au/recordkeepinq/control/tools.pdf; Internet; accessed 17 August 2006. 128 thesaurus also includes terms that are similar to identified authorized terms but are not preferred for classifying or titling records. These non-preferred terms act as links that direct users to authorized/preferred terms. Not different from other thesauri, the major advantage of a functions thesaurus is the control it provides over the use of natural language, and thereby promoting greatly enhanced information retrieval. 143 A records classification scheme (function-based classification system) aided by a functions thesaurus standardizes records classification in describing records categories and avoids misplacing records due to similar meanings of various terms. The Keyword AAA: A Thesaurus of General Terms developed by the State Records Authority of New South Wales is a typical example of a functions thesaurus.144 The BASCS methodology is developed and promoted by the Library and Archives of Canada for the purpose of constructing function-based classification system to replace its Subject Block Numeric Classification System, which has been in use for decades. The development of BASCS is influenced by the macro-appraisal methodology and structural-functional analysis adopted in the 1990s by the then National Archives of Canada (NAC) in assessing records values.145 The BASCS function-based classification method also has a three-tire hierarchy to characterize business context, but they are termed function, sub-function, and activities. The use of these terms is explained in the BASCS National Archives of Australia, "Keyword AAA"; available from . http://www.naa.qov.au/recordkeepinq/control/KevAAA/summarv.html; Internet; accessed 18 August 2006. 3 Paul Sabourin, "Constructing a Function-Based Records Classification System: Business Activity Structure Classification System," Archivaria 51 (Spring 2001):153. 129 design principles as follows.1 A function is any high level purpose, responsibility, task or activity assigned to the accountability agenda of an institution by legislation, policy or mandate. It comprises a set or series of sub-functions. Sub-functions are the major and unique steps of the business process an institution puts in place to fulfill a function. A business process will entail a linear or cyclical progression of activities designed to support an institution in producing the expected results in terms of the goods or services it is mandated or delegated to provide. Activities are derived from the major tasks or actions performed by the institution to accomplish each step (i.e., sub-function) of the business process. Activities are the unique components of a sub-function which (may) occur in a linear or cyclical sequence that results in fulfilling the sub-function. Activities encompass transactions. Transactions are defined as the smallest measurable unit of work carried out as part of a business process in support of a BASCS activity. Compared to the explanations of the three tiers in DIRKS, the BASCS explanations lack clarity. The term "activity" is used to describe all of the three levels of function, sub-function, and activity (underlined in the explanations by the researcher). Saying a function is any "activity" is less clear than saying it is a "unit" or "aggregate" of activities. It is especially misleading when the term "activities" is used to explain sub-functions, since, to what the "activities" really refer are the "steps" of the business process, which are in sequential order. In other words, 146 The explanatory information about "function", "business process", and "activity" was most directly taken from, Library and Archives Canada, "BASCS Guidance"; available from http://www.collectionscanada.ca/information-manaqement/002/007002-2091-e.html#five; Internet; accessed 19 August 2006, but re-organized and sometimes paraphrased for discussion purpose. 130 they are the sub-functions. The use of "activities" here confuses the "activities" used at the third level, which are also considered in sequential order (if identifiable). It is critical to have clear explanations for these terms, especially function, since as categories in the function-based classification system, users' understanding of them directly influences how they associate records with these categories. As observed by the NAA in promoting the use of records classification scheme developed using DIRKS methodology, users are sometimes confused with the functions defined in the classification system, typically due to similar functions carried out in the same agency. 147 In other words, how to clearly set the boundaries of each function and make them sensible to the users of the classification system determines the effectiveness and successful use of the classification system. The above introduction to the two methodologies makes it clear that there is a major difference between them. The DIRKS records classification scheme, under each function, lists activities associated with the function in alphabetical order, which is viewed by BASCS as less meaningful. BASCS emphasizes, at its sub-function level, the sequential order of carrying out the business process and rationalizes it with the fact that many government functions and activities are regulated by legislation or polices to be carried out in a clearly identifiable sequential order. Therefore, listing sub-functions in sequential rather than alphabetical order in function-based classification systems is both theoretically National Archives of Australia, "Overview of Classification Tools for Records Management". 131 sound and practically feasible as the sequential order naturally exists.148 Under sub-function at the activity level, BASCS also promotes identifying the possible linear or cyclical order of fulfilling the tasks of the activity. BASCS allows subjects or other sorting schemes to be used at the activity level - if no logical sequence can be identified - and at transaction (topic) level. As pointed out by Paul Sabourin, however, "the primary structural design of a BASCS is to first map out the natural sequence of activities within a business process model composing the function and sub-functions before listing the subjects, [ ], and case files in alphabetical order or using some other scheme."149 Although it is not included in the DIRKS manual, the analysis of work process at the transaction level to identify sequential order is recommended by the 2003 Australian standard AS 5090, a standard respecting work process analysis for recordkeeping and a complement to AS ISO 15489.150 The emphasis on decomposing transactions through sequence analysis to reflect the business model implemented in the organization for classification purposes is not different from what BASCS recommends; BASCS, however, focuses more on decomposing sub-functions (which are the activity level in the DIRKS records classification scheme) than activities (which are the transaction level in the DIRKS records classification scheme). The merits and limitations of these two 148 Library and Archives Canada, "BASCS Guidance"; available from http://www.collectionscanada.ca/information-manaqement/002/007002-2091-e.html#one; Internet; accessed 20 August 2006. 149 Sabourin, 138-139. 150 Anne Liddell, "The NAA Experience of Using AS 5090 - Australian Standard for Work Process Analysis for Recordkeeping to Support its DIRKS Project"; available from http://www.naa.qov.au/recordkeepinq/rkpubs/fora/03Nov/AS 5090 paper.pdf#search=%22%22Anne%20L iddell%22%20NAA%22: Internet; accessed 22 August 2006. 132 methodologies remain unclear at this stage since the BASCS methodology is still under development and not much experience of applying it has been reported. It is clear that there is a lack of consensus regarding functional analysis. Two classification systems in Canada illustrate the difference between the DIRKS-kind function-based classification systems and subject-based classification systems: the Nova Scotia's STAR (Standard for Administrative Records) and STOR (Standard for Operational Records) system151 and the British Columbia's ARCS (Administrative Records Classification System) and ORCS (Operational Records Classification System) system.152 Both are developed by records management divisions in the provincial archives and are designed to manage records generated by all government departments. STAR and ARCS address records resulting from administrative functions common to all government agencies, and STOR and ORCS address records resulted from operational functions unique to each individual agency. While both claim to be based on the analysis of government functions, the selection of terms to describe the categories in the classification systems reflects their different approaches to constructing the classification systems. Take the first and the highest level as an example, the common administrative functions in ARCS are described as Administration, Buildings and Properties, Equipment and Supplies, Personnel, Finance, and Nova Scotia Archives and Record Management, "The Standard for Administrative Records;" available from http://www.gov.ns.ca/nsarm/organization/rm/star5/index.htm ; Internet; accessed 12 August 2006. 12 Government of British Columbia, "ARCS Online,: available form http://www.bcarchives.gov.bc.ca/arcs/index.htm; Internet; accessed 12 August 2006.; Government of British Columbia, Corporate Information Management Branch, "The Standard ORCS Kit - 2001 Edition available from t http://www.mser.gov.bc.ca/CIMB/policy/default.htm#std_orcs_kit; Internet; accessed 12 August 2006. 133 Information Technology; 153 in STAR, however, they are described as Administration Main Group, Facility Management Main Group, Financial Management Main Group, Human Resources Management Main Group, Information Management Main Group, and Material Management Main Group. STAR categories are obviously more activity-indicating. The construction of primaries under functions displays the same difference. The example provided here are primaries under the function Equipment and Supplies in ARCS and Material Management in STAR, which are suggested by their scope notes as similar functions. Under Equipment and Supplies, primaries are listed as Clothing, Computers, Foods, Vehicles, and so on; under Material Management, however, primaries are listed as Material Inventory, Material Maintenance, and Motor Vehicle Management - the latter, again, reads as more activity-depicting. These categories denote that ARCS is a subject-based system and STAR is a function-based system, like a DIRKS function-based system, as there is no sequential order identified among these categories. In a BASCS function-based system, the sub-functions (primaries) for the same function would be listed as the following: Function Materiel Management Sub-functions Assessing Material Requirements Planning Materiel Requirements Acquiring Materiel Assets & Related Services 153 This is the actual order in the online version of ARCS, which is not alphabetical. 134 Operating Materiel Assets Using Materiel Assets Maintaining Materiel Assets Replacing Materiel Assets Disposing of Materiel Assets 154 Clearly not organized in alphabetical order, these categories proceed following the business sequence, which logically starts with the "Assessing Material requirements" and ends with "Disposing of Material Assets". Since the city's universal file plan is modeled on ARCS, it is also a subject-based classification system and inherits classification issues caused by subject-based classification systems. As the respondents observed, determining the subject content of records without adequate tools to guide the process is perplexing and time-consuming since there is no standardized or straightforward rules for identifying subjects, and it is very common that today's office documents have more than one subject. The difficulties of deciding subject is further compounded by the design of the file plan utilizing the division between administrative and operational functions, which is what the term "universal" means in this type of classification. By design as such, administrative functions, which are normally defined as functions common to all organizations or common to all departments in the organization such as finance and Information Technology Management. These administrative functions group altogether records created by common activities identified under them irrespective of where the activities 154 Sabourin,153. 135 organically reside. Take the administrative function Finance and its activity Budget/Budgeting as an example. Records generated by the activity Budgeting are possible from many functions in the organization, including the Finance function that budgets for the entire organization and other functions such as the Public Communication or any ad hoc projects, which budgets for its own operation. Grouping all these records under the administrative function Finance in fact is an action that takes these records out of their originating contexts. This application of "universal" does not conform to the essence of a universal file plan. "Universal" in the method of constructing file plan should be understood as a consistent approach of applying established rules or analytical tools to the management of records. A universal file plan is a file plan taking into account the entire organization and that is used by all departments and offices in the organization. The division of administrative and operational functions is indeed more meaningful and practical for archival institutions charged with responsibility for assisting more than one organization's or agency's records management activities by developing model categories for administrative functions. It is not necessary in a particular organization to group all common administrative activities under the common administrative function; the commonality of these activities can still be reflected by describing them in the same way but placing them under their originating functions or projects. While the major advantages of constructing a function-based classification system are usually offered from the perspective of records and archival management, it can be argued that function-based classification systems are 136 more user-friendly for EDRMS users. Users of the classification system in an EDRMS are participants in business processes and naturally understand the functions, sub-functions, and activities that shape their job duties. It is easier for them to classify records in terms of the transaction that generates them, which saves time for finding the higher level categories of functions and activities. While subjects may still be needed to identify individual records at the lowest level, a function-based classification system eases the subject-identification problems by limiting subjects in an intellectual framework bounded by the higher levels of functions and activities, with which the users are familiar. At the same time, the standardized use of terms to describe subjects supported by a functions thesaurus developed in the same functional analysis process assists users' selection of descriptive terms and reduces frustration. Design of the File Plan Incorporating User Needs In an EDRMS environment that requires every records creator to classify records, the analysis of the needs of these records creators, who are also the users of the classification system, is of unprecedented importance. As indicated by the responses listed in the beginning of this section, the nature of the respondents' job duties plays a critical role in their classification activities. Probably because such systems have not yet been widely implemented, there is little discussion in the field of records management and archival community specifically addressing user needs with respect to classification of electronic records. When discussing taxonomies in relation to records management, writers 137 either report their experience of developing taxonomies for organizations or introduce different types of taxonomies in a general manner.155 These articles, which loosely use the term "taxonomy" as an equivalent of classification system,156 focus very much on the advantages - primarily the facilitation of information retrieval - that taxonomies can bring to the management of organizational information. This indicates that the development of a corporation-wide taxonomy or classification system is still a challenge for many organizations and their records management programs. Writers also suggest that taxonomies for users' benefit should be simple and straightforward, but no techniques are offered so far for achieving this purpose. While all recommend testing established taxonomies with users, they do not define users of the taxonomies in the articles, nor do they specify the size of the testing user group. The discussion of function-based classification system in archival professional literature also focuses very much on advantages the function-based classification systems can offer for managing records. However, analyzing users' needs at the individual level, as it is necessary in an EDRMS environment, are not addressed. The findings of this research suggest individual users' needs should be incorporated into the design of a classification system. The expression of See, for example, Trish O'Kane, "United by a Common Language: Developing a Corporate: Taxonomy Case Study Emphasizes the Need for Consensus, Understanding of Technical Features of Metadata," Information Management Journal 40, no. 4 (July/August 2006): 58-63; Susan L. Cisco and Wanda K.Jackson, "Creating Order out of Chaos with Taxonomies," Information Management Journal 39, no.3 (May/June 2005):44-50; Denise Bruno and Heather Richmond, "The Truth About Taxonomies," Information Management Journal 37, no.2 (March/April 2003): 44-53. Taxonomy is the science of classification and taxonomies constructed in accordance with its classification principles are stricter in structure and logic than classifications, which can include any sort of organization of objects with or without a well-reasoned hierarchical structure. Taxonomies nowadays, however, are loosely used by various information management professionals to refer to any vaguely structured set of terms in a subject area. For more information about taxonomy, see Vanda Broughton, Essential Classification (New York : Neal-Schuman, 2004),12-13. 138 "individual users' needs" is used here to differentiate user general inputs that are gathered during brainstorming sessions in a traditional classification development process, which do not adequately address individual user's classification needs. The analysis of individual user's needs is different from setting user profiles in an EDRMS, the purpose of which is to gather information for control and management (for example, assigning access rights). The incorporation of user needs is at the transaction level of the classification system and based on an analysis of users' job tasks, as recommended by the AS 5090 Work Process Analysis for Recordkeeping standard. The Northern Ireland Civil Service classification development project offers a close example. This project basically follows the Canadian BASCS methodology. For the transaction level, the project team asked the government departments "to identify the transactions and records which make up the lower levels [...] as a result of analyzing their own business processes".157 The project rationalizes this "would provide greater ownership of files at the local level by increasing the relevance of filing in relation to work/business processes."158 It is not clear in the article, however, whether or not these department-developed categories are approved by the records management department and who has responsibility of maintaining them. Admittedly, identifying and incorporating individual user needs into the design of a corporate classification system will demand more time and resources, which complicate gaining management support. There are tactics for dealing with this. 157 Zoe" A. Smyth, "Adopting a Functional Classification of Business Processes in Northern Ireland," Journal of the Society of Archivists 26, no. 2 (October 2005): 234. 158 Ibid. 139 When organizations are implementing BPM (Business Process Management) or workflow systems, or undertaking major business re-engineering projects, the analysis of user needs for classification purposes could seamlessly fit into these projects and could easily be done. To incorporate user needs at the individual level into the classification system is only one more step. 5.1.3 Technical Assistance for Classification Classification problems can also be addressed through providing technical assistance to users. It is possible to do a number of things - with particular reference to the EDRMS under evaluation - to assist users: a) providing access to explanatory information (scope notes) for each category at each level; b) allow users to group the established classification codes in the profiling form for their own purpose (for example, in the Look-up button), which is not now possible using the customized subset of classification codes.159 c) incorporating with the universal file plan a keyword searchable index for terms used; and e) establishing a subject term thesaurus regulating the selection of subjects for classification. The need for explanatory information of categories is suggested by the data collected through Q13. Fully 59% of respondents consider the lack of scope notes in the current file plan to be a deficiency. One respondent (ID38) noted that "a master list with more detailed description is needed." Scope notes of categories, 159 The RM department in fact has a good practice of developing brochures of classification codes. Given resource constraints, however, these brochures are mainly developed for large-scale projects and are not integrated in the EDRMS for electronic access. 140 in both subject- and function-based classification systems, assist users' understanding of the categories and guide their classification activities. The recommendation for a customized subset of classification codes for individual users is partly based on the evidence that 26% of respondents consider the "Recently Used Files" feature is not helpful. The "Recently Used Files"feature lists the last 30 file classifications being used by a particular user, which does not necessarily represent the user's real classification needs.160 The index and search function are proposed to accommodate users' different information seeking habits, some of whom prefer browsing while others use the search option frequently. In order to make effective use of the index and the search function, a thesaurus needs to be constructed facilitating the use of controlled vocabulary. With the current subject-based design, a searchable file plan aided by controlled terms can be great helpful on determining subject terms. 5.1.4 Time of Classification As introduced in the section on documents creation in the previous chapter, supplying classification codes in the EDRMS takes place in three instances: a) when new documents created by the user are being saved into the system, b) when documents created outside the system are being imported into the system, and c) when e-mail messages are being saved into the system. Documents in these three instances have different statuses. When documents are decided to be The system has another feature intending to assist classification, which allows classification codes to be grouped in one place for a department or office. It is not currently in use due to the difficulty of setting proper user profiles for multiple purposes. Although also a customizing tool for classification, it is different from what is proposed here: to allow individual users to customize their classification environment. 141 imported and e-mail messages be saved, the decision of declaring them as records has already been made. The RM department advises authors to mark them as records right after they are imported or saved. The documents created in the system through the integrated applications retain their status as documents, and they may or may not become records at a later stage. The city's records management bylaw defines records as "recorded information in any form or medium, created or received in the transaction or conduct of business, and kept as evidence of such activity."161 Based on this definition, the RM department instructs records creators in their Mark as a Record brochure that a document needs to be marked as a record when it: a) is no longer in draft stage; b) is no longer requiring any editing; c) meant to be unalterable and view only; and d) is decided as an official record of the City.162 Only documents satisfying these conditions are managed as records. Since the system is an integrated document and record management system, both documents and records are stored in one centralized database and accessed through one interface. The document management functions and record management functions accordingly are mixed and appear to be offered by one application. For end users of the system, these mixed functions have no negative influence on their work except on the requirement of classifying every 161 For the purpose of keeping the city anonymous, citation is not provided here. 162 See the above note. 142 document that needs to be saved.163 Declaring and filing164 records in an electronic record management system is a mandatory requirement in the DoD 5015.2 standard,165 yet its application to documents is open to debate. In the situation of the city's EDRMS environment, the imported documents and saved e-mail messages should be classified since they are already decided by creators as records; for documents generated in the system, however, not all of them should be classified since not all of them will later become records. It is sensible to argue that documents generated in the system should be classified only at the moment when declaring/marking them as records. Classification at this moment should be compulsory, and this can be achieved easily through technological means of bounding marking documents as records and classifying them at the same time. Arguments for this suggestion are both simple and complex. The simple one is that if the classified documents are deleted later, the time spent on classifying them is wasted. Classifying fewer documents lessons the burden on users and increases office productivity. The complex one involves discussion of advantages classification can offer. As a long-standing best practice in managing records in a paper records environment, classification serves as the most effective locating method. However, in an EDRMS environment and for electronic documents and records, classification is no longer the primary method of locating This may relate to the integration approach for the DM and RM functions as introduced in the introduction chapter. However, the product is indeed powerful in the sense that it provides many options for the installation of the RM extension to the DM module. The discussions here refer more to the customization in the integration process. 1 The DoD5015.2 standard defines "file (verb)" in DL1.1.44.2 as "the act of assigning and storing records in accordance with the file plan.", 14. 5 United States. Department of Defense, c 2.2.3, 28 143 documents and records;166 instead, searching metadata and/or content serves most EDRMS users' information needs, provided all documents are profiled. As evidenced by the responses to Q16 (preferred document-locating methods), only 5 out of 126 respondents selected "browsing the file plan" as a preferred document-locating method, making it the least selected locating method (see Table 4.5 for details). This is especially true if the file plan can only be navigated but is not searchable. Profiling documents should still be mandated for all documents in the system because it enables access and search in metadata and/or content. This suggestion should not be taken to dismiss the advantages of searching by classification in order to "see" records together under the same activities that give rise to them. There is an accompanying consideration with this suggestion. A procedure that requires all documents to be profiled and classified before being saved is one measure of discouraging users from creating too many business-irrelevant documents and making the EDRMS a more official space. The out of control office freedom in creating and saving electronic documents that occurred widely in the 1980s and early 1990s, it has been observed, made of the modern office a kind of "wild frontier."167 Placing control over the wild frontier is one of the ultimate goals 166 The statement that classification has become less useful in retrieving electronic information is conditioned by this particular case, that is, the EDRMS environment in which search functions are offered. Electronic information in organizations is comparatively limited in quantity in comparison with that on the Internet or in commercial databases. The information searchers in organizations are also information creators and they are, at least to some degree, familiar with the overall setting of the organization and therefore familiar with documents generated by others. This normally makes their keyword search result in higher precision. Classification, especially faceted classification, is heavily used by many commercial websites and database publishers to enhance searching within a large quantity of electronic information for which general users always face challenges of locating desired information. 167 John McDonald," Managing Records in the Modern Office: Taming the Wild Frontier," Archivaria 39 (Spring 144 of implementing a management system such as the EDRMS. With this only-classifying-records suggestion, new procedures, such as, to delete documents that have not been declared as records for a certain time period, need to be put in place to prevent business-irrelevant documents from occupying the repository. 5.2 Document Locating168 Since the EDRMS employs the approach of managing all documents in a centralized repository, the effectiveness of locating documents is vital to the success of the program. As introduced in the previous chapter, the EDRMS offers a number of search methods (profile-field search and content search, for example) and browse tools ("Recently Edited Documents" and "Project Folders", for example) for locating documents. It also provides post-search management functions such as "Quick Searches" and "Sorting". A system that offers these features should be able to generate desired results of locating documents; However, this study indicates that users do not fully appreciate these search features. As indicated in Table 4.4, the positive percentage towards Q15 ("Finding documents and records in [the EDRMS] is easy") is only 66%, and there are large numbers of "Neutral" responses, and even indications of a lack of awareness of some locating-related features. A comparative examination on these findings reveals that respondents who think finding documents is easy at a general level 1995): 70-79.; and his following-up article, "The Wild Frontier Ten Year on," in Managing Electronic Records, ed. Julie McLeod and Catherine Hare (London, [UK]: Facet Pub., 2005), 1-17. 168 Since there are no fundamental differences between the locating of documents and the locating of records, "locating documents" or "document locating" will be used to cover both. 145 (Q15) also generally agree other features are useful or helpful (Q17, Q18, and Q19). Among the 10 "Strongly Agree" in Q15, 9 strongly agree with Q17, 8 strongly agree with Q18 and 5 strongly agree with Q19. This indicates that users' overall satisfaction with documents locating derives from their satisfaction with each individual feature. The reasons for their dissatisfaction, however, cannot be discerned from the current data except that they have different opinions regarding different locating features. This requires more data gathering from users who report locating difficulties. The unexpected textual response from respondent ID11 confirms this need. In addition to selecting "Neutral" for Q15, this respondent additionally reports that it is hard to find documents saved by members in the same project team, and comments, "we need everybody to fully understand where documents are being stored." Many questions may arise from this particular data. Why does this respondent not know how to use "Project Folders" to solve this problem, which, by design, is meant for meeting such needs? Are there any functional insufficiencies in this feature, since his or her answer to Q19 ("Project Folders" are useful) is "Neutral"? Is the reason that he or she lacks the necessary training on how to make effective use of this feature? Has he or she ever asked for assistance with the problem in the 5 to 7 years he or she has worked with the system (see the answer to Q2)?. All in all, does this indicate that training should be more carefully tailored and more outreach is needed for delivering the RM assistance? To answer these questions and to gain clear understanding on this reported problem requires information, which could only be obtained in individual 146 interviews, from not only the respondent but also the RM department.169 5.2.1 Document Locating and Time Period Working with the System The assumption that time period of working with the system is instructive for document locating responses is based on the understanding that performing effective and successful document locating activities needs knowledge and practice, and a longer time working with the system permits more chances of acquiring needed knowledge to execute good practice. The following table identifies Q15 responses in relation to the four groups of respondents who have different time periods of working with the system: Table 5.3 Document Locating and Time Period Working with the EDRMS Q2 Time period working with the system (year) 1 2-4 5-7 8-10 NR 3 6 18 23 Q15 Finding documents is easy N(SA+A) 0 5 11 17 N(DA+SDA) 1 0 3 1 NN 2 1 4 5 N(SA+A)/NR% 0% 83% 61% 74% N(DA+SDA)/NR% 33% 0% 17% 4% NN/NR% 67% 17% 22% 22% While this table cannot fully establish a direct proportional relationship During the time period of conducting this evaluation research, the researcher observed that the RM department was operating under tight budgetary constraints. Providing user-centered services is undoubtedly not currently practicable. 147 between the time factor and the ease of finding documents, it does show the impact of greater familiarity with the system on users' document locating experience. Respondents having longer time working with the system generally agree that finding documents is easy (0%, 61%, and 74%) and respondents having shorter time tend to think finding documents is not easy (33%, 17%, and 4%), with the exception of the group with "2-4 years" experience (83% in agreement percentages and 0% in disagreement percentages). It is not clear why a significant percentage of experienced users (22%) declared themselves "neutral" in answers to this statement. 5.2.2 Document Locating and Computer Skills As document locating in an EDRMS environment is performed through manipulating locating features provided by the system, computer skills are considered relevant to effective performance of retrieving documents. Table 5.4 compares respondents' document locating experience with their computer skills: 148 Table 5.4 Document Locating and Computer Skills Q3 Self-rated computer skills V. Strong Strong Sufficient NR 21 22 7 Q15 Finding documents is easy N(SA+A) 16 11 6 N(SD+D) 1 4 0 NN 4 7 1 N(SA+A)/NR 76% 50% 86% N(SD+D)/NR 5% 18% 0% NN/NR 19% 32% 14% In this table, no relationships can be established between these two compared variables. The absolute majority of respondents with "Sufficient" computer skills (which is the lowest level among the self-rated computer skills) strongly agree or agree that finding documents is easy (6 out of 7, 86%), and none of them selected "Strongly Disagree" or "Disagree". Correspondingly, respondents with very strong or strong computer skills have selected "Disagree" and even "Strongly Disagree" (5% and 18%), and the largest number of "Neutral" response appears in the "Strong" group. This implies that even though computer skills are necessary and helpful for working in an electronic environment, they do not translate directly into strong document search skills. This finding reveals, together with the fact that there are comparatively fewer selections of "advanced 149 search" in Q16 (40%), that users need more training on document search regardless of their self-assessed level of computer skills. Users' document search experience in an EDRMS may be influenced by other factors than the above discussed ones. To fully understand the data collected by this research requires further investigation into areas such as users' job duties. In addition to individual's information seeking habits, users' job duties dictate how they search for desired documents and records. For example, if the user's job mostly deals with documents created by themselves, the Recently Edited Documents feature and profile search would be most convenient and useful for them. If, on the contrary, they need to search frequently for documents created by others, content search (keyword search) would be the most powerful tool for achieving their purposes, unless the author of the document sends them metadata (for example, title or document number). A respondent with such job duties would tend to think finding documents is not easy if he or she is not comfortable with the advanced search features. Job duties also decide whether or not certain features such as Project Folders are useful. A response expressing that project folders are not useful may either refer to the poorly designed functionality or to the fact that the user has not been involved in any project activities and therefore does not think it is useful. Moreover, the amount of training the respondents have received could be another factor influencing users' document locating experience. 150 5.3 "Mark as a Record" and the Concept of Authenticity The requirement C2.2.3.8 in the DoD 5015.2 standard mandates that electronic records management applications should "prevent subsequent changes to electronic records stored in its supported repositories".170 The "Mark as a record" feature in the EDRMS is designed to satisfy this requirement, and it is considered as the most effective means of ensuring the authenticity of records generated, used, and kept in the electronic environment. Despite the fact that the program manual stresses it as a critical feature and the RM department developed a brochure stressing how important it is, the necessity of marking as records appears not to be full appreciated by all users. Among the evaluated features relating to records management requirements, Q32 ("Mark as a record is a necessary feature") receives the lowest agreement percentage (see Table 4.7 for details). There are 11 "Neutral", 3 "Disagree", and 1 "Strongly Disagree" responses for this statement, and 2 respondents answered that they were not aware of this feature. It is interesting to note that neither of the two respondents who indicated that they were unaware of the feature is new to the system: one has worked with it for 2 to 4 years and other for 5 to 7 years. While the reasons for the large number of "Neutral" responses are not revealed by this study, comparing "Mark as a Record" with "History" and "Modify Security" may explain these "Strongly Disagree" and "Disagree" selections. As displayed in Table 5.5, almost all respondents who think "Mark as a Record" is not necessary (including the two who do not know this feature) strongly agree or 170 United States. Department of Defense, 30. 151 agree that the feature "History" is useful. The two respondents who do not know the feature "History", on the other hand, strongly agree or agree that the feature "Mark as a Record" is necessary. Table 5.5 "Mark as a Record" (Q32) and "History" (Q23) Q32 Disagree (3) + S. Disagree (1) + Unaware (2) 6 Q23 S. Agree Agree Neutral 3 2 1 Q23 Unaware 2 Q32 S. Agree Agree 1 1 Similarly, in Table 5.6, all respondents who think "Mark as a Record" is not necessary (including the two who do not know this feature) strongly agree or agree that the feature "Modify Security" is useful (see Table 5.6). Table 5.6 "Mark as a Record" (Q32) and "Modify Security" (Q10) Q32 Disagree (3) + S. Disagree (1) + Unaware (2) 6 Q10 S. Agree Agree 3 3 The above analysis signals that there are confusions existing among 152 respondents regarding the purposes of the three features: respondents who think "Modify Security" and "History" are useful tend to consider that "Mark as a Record" is unnecessary. "Mark as a record" is designed for authenticity purposes and is solely a records management function. "History" and "Modify Security" can be used for both document management and record management purposes.171 The city's records management bylaw defines authenticity through defining an authentic record, which a) is what it purports to be; b) is unalterable, or is made unalterable; and c) has not been manipulated, substituted, falsified, or tampered with, either intentionally or unintentionally, in any way. The core of this definition, in particular within the EDRMS environment, is the unalterable status confirmed when a document is declared as a record. The desire for records authenticity, from an organization's perspective, originates fundamentally from the needs of maintaining business evidence and complying legislative requirements and effectively managing information as intellectual capital. Ensuring that records are stable and unalterable satisfies all these needs. "Mark as a record" is an authenticity feature in the sense that it technically 171 The Technical Report (full citation in chapter 1) identifies that "History" and "Modify Security" can be employed by both documents management system and records management system when they are stand-alone systems (not integrated as one system). This report does not give out examples or explanations for this conclusion. The following example and explanations are provided by the researcher. One incidence of using "History" in a documents management system is multi-editing. When a document is under multi-editing, the tracked transactions in "History" inform these editors about when and by whom the document was accessed and what has been edited. In a records management system, while "Mark as a record" protects the record by making it read-only, the transactions tracked by "History" could still be useful in situations where doubts as to the record's authenticity are raised due to IT system problems. The "History" information can then be used to demonstrate whether corruptions did or did not happen to the record during the problem period. "Modify Security" is useful for both documents and records in the sense that it protects personal and/or confidential information that may occur in documents or records from unauthorized access. 153 makes a document read-only. A document marked as a record becomes an official business record and no one, including the author of the document/record, can make changes to its content or profile after the marking. Both evidentiary and informational values of the record are protected from that point onward. Neither "Modify Security" nor "History" fully has this capability. "Modify Security" protects records but to a lesser degree, and the protection could probably be realized only at the expense of information sharing. Moreover, even "Modify Security" assigns strict access rights to a document - which prevents potential manipulations from unauthorized access - it is still open to the possibility that changes could be made, either accidentally or purposefully, by the users with access rights (including the author of the document). This lesser degree of protection makes records less evidentially strong and informatively useful. If, for the purpose of improving the protection, no access rights are assigned to other users except the author (who naturally has the right), information sharing would be hindered. The feature "Mark as a record", by contrast, easily prevents any changes that may happen to the document and at the same time allows access rights to be assigned to as many users as possible. The "History" feature cannot replace the "Mark as a Record" feature either. While it tracks every action that has happened to a document after its creation, this accumulated information is only useful when record authenticity comes into question.172 That is to say, by the "History" feature itself authenticity cannot be 172 According to the benchmark authenticity requirements established by the InterPARES (International Research on Permanent Authentic Records in Electronic Systems) Project I, certain information needs to be 154 protected; it can simply allow one after a document was created to establish that something was changed, which is a far cry from the assurance that it could not be changed without great effort. Without the chance of interviewing the RM department, it remains unclear how the concept of authenticity and the differences between the above three features are communicated to the system's users. The RM department's brochure on the "Mark as a Record" feature clearly explains when a document should be marked as a record and how to make it happen. When communicating the importance of marking a document as a record, the term "authenticity" is avoided and emphasis has been given to the advantages an unalterable record can offer in the scenario of information re-use. It is understandable that the concept of authenticity is difficult to communicate, and it is always advocated that effective communication requires easiness and straightforwardness. It can be argued that authenticity, the most vital concept in managing electronic records, should be understood not only by records management professionals but also by all users who work with documents and records in an electronic environment. As the Policy on the Management of Government Information (MGI) states, all employees are responsible for the management of information under their control and custody; and one of their responsibilities in the management process is "[to apply] known in order to assume record authenticity. According to its Authenticity Task Force Report, "a presumption of authenticity is an inference that is drawn from known facts about the manner in which a record has been created and maintained" (p3). By saying the information accumulated in "History" is useful, I mean some of the authenticity requirements, such as, the dates and the handling offices, are satisfied by the information recorded in "History". See the report at InterPARES, "Requirements for Accessing and Maintaining the Authenticity for Electronic Records"; available from http://www.interpares.org/book/interpares book k app02.pdf; Internet; accessed 13 August 2006. 155 information management principles, standards, and practices in the performance of their duties."173 ISO 15489, the world's first records management standard, specifies the same requirements. To build a RM program compliant to this standard, organizations are required to establish RM policy and assign RM responsibilities. The RM policy/which must be communicated and implemented at all levels in the organization, sets its goal as "the creation and management of authentic, reliable and usable records, capable of supporting business functions and activities for as long as they are required."174 In a RM program established as such, "all employees are responsible and accountable for keeping accurate and complete records of their activities."175 Records are organizations' valuable assets. In an electronic environment, it is more evident than ever that every employee has the responsibility of documenting their business activities through creating records and ensuring their authenticity, that is, by establishing the unalterable status of a record through technical means such as marking documents as records. 5.4 RM Department Assistance It is reasonable to credit the overall good understanding of records management requirements found in the city to the education and assistance 173 Treasury Board Canada Secretariat, "Policy on the Management of Government Information"; available from http://www.tbs-sct.gc.ca/pubs pol/ciopubs/TB GIH/mqih-qrdq1 e.asp#pol; Internet; accessed 13 August 2006. 174 International Organization for Standards, 15489-1:2001 Records management - Part 1: General: 5 175 Ibid., 6 156 provided by the RM department, as is evidenced by the high agreement percentages for Q27 (assistance has been necessary, 92%) and Q29 (how useful the assistance has been, 84% for "Extremely useful" and "Useful"). Even with their less than satisfactory experience of classifying documents, the majority of users understand very well that classification is necessary (88% in Q11). For exploring purpose, the "Neutral" and "Disagree" responses to Q27 and Q29 are examined in relation to user background and other experience of using the system. The results are reported in table 5.7 and 5.8. Table 5.7 RM Department Assistance and User Factors Q27 Assistance has been necessary Neutral (1) + Disagree (3) 4 Q3 Self-rated computer skills V. Strong Strong Sufficient 2 1 1 Q12 Classifying documents is easy S. Agree (2) + Agree (0) Neutral Disagree 2 1 1 Q15 Finding document or records is easy S. Agree (0) + Agree (3) Neutral Disagree 3 1 0 Q30 Learning how to use the system has been easy S. Agree (1) + Agree (1) Neutral Disagree 2 2 The above table indicates that those respondents considering assistance 157 provided by the RM department unnecessary (including 1 "Neutral") generally have strong computer skills, consider classifying documents and finding documents is not difficult, and learning how to use the system has been easy. In other words, respondents who have positive experience of working with the system feel less in need of assistance. People of this sort could be identified by RM departments as EDRMS experts, and added as RM advisors to the organization's knowledge management network.176 While these respondents do not necessarily understand the EDRMS as much as the RM personnel do, they are more familiar with the work their co-workers do and could be very helpful in solving particular classifying and/or locating problems, and in improving fellow workers' learning of the system. By promoting their success of using the EDRMS, the RM department would enhance the EDRMS culture in the organization. Table 5.8 presents the analysis for the "neutral" answers in Q29. This could be easily done if the organization has an established knowledge management program. 158 Table 5.8 Currently Provided Assistance and User Factors Assistance is neither helpful nor unhelpful Q29 Neutral 8 Classifying documents is easy Q12 S. Agree (1) + Agree (0) Neutral Disagree 1 4 3 Finding document or records is easy Q15 S. Agree (0) + Agree (5) Neutral Disagree 5 1 2 Learning how to use the system has been easy Q30 S. Agree (0) + Agree (5) Neutral Disagree 5 3 0 Among the eight respondents who are uncertain about whether or not the assistance provided by the RM department is helpful for their use of the EDRMS, seven of them have negative experience with documents classification, but only two of them think finding documents is not easy, and no one thinks learning the system is difficult. This analysis may indicate that these respondents think the provided assistance is "neither helpful nor unhelpful" because they feel finding documents and learning the system is easy, and at the same time, even training cannot help their classification difficulties. No matter what experience they have had regarding assistance, almost all of 159 these respondents (except one) selected assistance types that are valuable to them (see Table 5.9). Table 5.9 Assistance Types Q27 Assistance has been necessary Neutral Disagree 1 1 2 3 Q28 Most valuable type(s) of assistance 0 1;4 1;4 1 Q29 Assistance is neither helpful nor unhelpful 1 2 3 4 5 6 7 8 Q28 Most valuable type(s) of assistance 0 1 1;4 1;2;4 1;2 1;4 1 4 Assistance type 1, Classroom computer training, appears as the most preferred type of assistance; it is followed by assistance type 4, Manuals and Brochures, and 2, One-on-one instruction. These respondents' preference for Type 1 is consistent with other respondents' preference as indicated in Table 4.10. It is interesting to note that their preference for Type 4 does not go with that of other respondents, who mostly select Type 2, One-on-One instruction. This preference of manuals and brochures may to some extent explain why they think finding documents and learning the system is easy. 5.5 Users' Awareness of Features Compared to users of the system who do not have responsibility to manage 160 departmental or office records, respondents selected for this research are more frequent system users. Nevertheless, while small in number, some of them do not know certain features the system offers. Table 5.10 examines these unaware features (Q14, Q17, Q18, Q23, and Q32) in relation to Q2, the time period of working with the system, and Q28, user-referred assistance types. 161 Table 5.10 Users' Awareness of Features Q14 Setting default values in profile form NUA177 ID22 ID32 ID33 ID44 Q2 8-10 8-10 5-7 5-7 Q28 1;4 1;2 1;2;4 4 Q17 Quick Searches NUA ID3 ID31 ID38 ID39 Q2 2-4 8-10 8-10 8-10 Q28 1 1;2 1;2 1;2 Q18 Sorting search results NUA ID11 ID12 ID33 ID39 Q2 5-7 5-7 5-7 8-10 Q28 1;2 1;2 1;2;4 1;2 Q23 History NUA ID9 ID39 Q2 1 8-10 Q28 1;2;4 1;2 Q32 Mark as a record NUA ID3 ID17 Q2 2-4 5-7 Q28 1 1;2 Among all these respondents who are not aware of one or more features, only one respondent has used the system for a short time, that is, one year. The short time period of using the system may explain why he or she is not aware of Number of selections for unaware feature. some features. It is understandable that learning the system is neither a goal nor a priority for new employees who have other job duties to deal with first. Nevertheless, it is more difficult to understand why respondents with (much) longer time periods of using the system - there are six "5-7" and seven "8-10" years - still do not know some of the features. Their selections on assistance types may suggest some explanations. The selected assistance types are, by a descending order, "Classroom Computer Training" (15 out of 16 responses), "One-on-one Instruction" (12 out of 16), and "Manuals and Brochures" (5 out of 16). Noticeably, the program manual is least appreciated by these respondents, which, indeed, is the place where all information about the system can be found. In-house classroom computer trainings, although welcomed by almost all respondents, normally have a short and fixed time, within which not all of the features can be introduced. Neither is one-on-one instruction meant to cover all features. They are either delivered to users who cannot attend classroom computer training sessions with the same coverage or are initiated by users who have a specific problem, which is unlikely to be about an unknown feature. Another factor contributing to this fact could be, again, the job duties the users perform. Some features may not be that useful for their day-to-day job. For example, the features "Quick Searches" would not be a desired function if the respondents do not frequently perform complex searches or do not search for documents created by others on a regular basis. Similarly, "Sorting" may not be needed if the returned search hits are not in large numbers. In scenarios like these, the "Recently Edited Documents" feature and/or profile-field search 163 function could already meet their needs very well. Also, if the respondents do not create new documents very often, they may not feel the need of setting default values in the profiling form. However, unlike "Quick Searches", "Sorting", and setting default values, which are more document management oriented, the unawareness of "Mark as a record" may denote an issue. As MGI also regulates, in an electronic working environment, documenting decisions and decision-making processes is every employee's responsibility,178 and there are always risks for a document to be changed or manipulated before they become read-only.179 The feature "History" relates to the reliability of documents, including those that later will be declared as records.180 One important function "History" serves is to assist the establishment of reliability. Reliability is traditionally established by examining the completeness of the record's documentary form and the amount of control exercised on the process of its creation.181 In an electronic environment, the establishment of reliability relies on more the degree of control than documentary form, which is now much easier to be imitated than in the past. This means even though the document has an incomplete form, its reliability can be established by examining metadata accumulated in its creation process. "History" is the feature that accumulates such metadata as who created the document, who edited it and/or its profile, and how many users have access to it, and so on, 178 Treasury Board Canada Secretariat," Policy on the Management of Government Information". 179 See also discussion in the section on "Mark as a Record". 180 After a document is marked as a record, its reliability is translated into the form of authenticity and protected by other technological means. The discussion of reliability applies to the record's document status. 181 See discussion on reliability in the chapter 2. 164 assisting the user of the document to judge how reliable the document is. As the data collected by Q31 {"When using documents in [the EDRMS], I trust them as reliable information sources.") suggest, the majority of users trust the documents in the system as reliable information (82%). There may be other reasons for the users to trust the documents in the system (such as the fact that the system is password-protected), nevertheless, the very high agreement percentage for Q23 ("History" is useful for tracking information, 94% with one respondent selecting "Neutral") could be the major one. As proof, one of the two respondents who is not aware of the "History" feature (ID19) did select "Neutral" for Q31. 5.6 Most Welcomed Feature The best understood feature, or the most welcomed one, is Modify Security (Q10). It is selected by 48 respondents (96%) for its usefulness in allowing information sharing on the one hand and protecting sensitive information on the other. Considering this in connection with the high degree of agreement with the statement (Q9), "Selecting security options when profiling documents is easy" (84%), this finding implies that users welcome features that are of practical relevance to them and, at the same time, are easy to use. 5.7 Titling Guidelines Q26 asks respondents' opinions on how they think that titling guidelines facilitate document locating. Ten respondents (20%) chose "Neutral", but none selected "Disagree" or "Strongly Disagree". Eight of the ten consider finding documents is easy (see Table 5.11). This may suggest that their satisfactory 165 experience of locating documents (which likely comes from their sophisticated search skills), makes these respondents consider titling guidelines are not necessary. Nevertheless, the fact that most users appreciate the assistance offered by these titling guidelines justifies their importance in effectively locating documents and records. Table 5.11 Titling Guidelines and Document Locating Using the titling guidelines facilitates document locating Q26 Neutral 10 Finding documents and records in is easy Q15 S. Agree Agree Neutral Disagree 1 7 1 1 5.8 Department Staff with RM Responsibilities In a paper records environment, setting a staff member in a department or office with records management responsibilities constitutes one important component in a well-established corporate records management program, and is considered as the best practice for managing records. In an EDRMS environment where everyone has the responsibility of managing the records under their control, the necessity of setting such a position provokes discussion. While these respondents in this research are traditionally charged with responsibilities of managing records in their departments or offices, almost half of them do not agree with the statement, It is important to have a staff person in each department with assigned responsibility to manage documents and records" (Q25, 48%). For the 166 purpose of understanding, these respondents' experience with classification (Q12), documents locating (15), and saving email with attachment(s) (Q22) is examined in Table 5.12. Table 5.12 Department RM Personnel -1 Q25 RM staff person in departments Neutral (13) Disagree (9) S.Disagree (2) 24 S. Agree Agree Neutral Disagree Q12 Classifying documents is easy 5 6 2 11 Q15 Finding documents is easy 5 13 4 2 Q22 Saving emails with attachment(s) is easy 7 6 10 0 These respondents who are not in favor of setting a RM staff person, although most agree that finding documents is easy, many have issues with classification and saving e-mail messages with attachment(s). In addition to selecting provided options, some respondents volunteered textual opinions to Q25. Respondent ID18, while selecting "Disagree", comments that managing documents and records should be done by "all employees".182 Respondent ID35, who selected "Neutral", believes "each staff member may easily learn to use [the EDRMS]". Table 5.13 groups their selections to Q12, classification is easy, Q15, finding documents is easy, and Q22, saving e-mail 182 Emphasis in original. 167 messages with attachments is easy. Table 5.13 Department RM Personnel -2 Q12 Q15 Q22 ID18 S. Agree Agree Agree ID35 Neutral S. Agree S. Agree Compared to Table 5.12, the two respondents in Table 5.13 indicate a more positive attitude towards the three activities. It seems two messages can be inferred from this analysis. Firstly, respondents with positive experience with these difficult activities (agreement percentage for Q12, Q15, and Q22 are 46%, 66%, and 52%) generally think having a staff person managing local records is unnecessary, because everybody can learn how to use the system and manage records on their own. Secondly, with particular reference to classification and saving e-mail messages with attachments (which is indeed closely related to classification), managing records created by others could be a daunting task, and these records should be easier for the creators themselves to classify. Although these findings imply, to a large degree, that there is a high acceptance of the notion that it is everybody's responsibility to manage records in an EDRMS environment, this alone should not be a reason to discard the practice of designating records management staff persons in departments. The decision of having or not having such a person should be justified with respect to the local working environment and the business activities performed by the department or office in question. In a department, for example, where the work of its professional 168 experts involves a large number of e-mails messages with attachments and the majority of them are business in nature, saving and classifying these documents by the experts themselves could take too much time from their professional related job tasks. The RM assistance provided by a staff person in the department frees the professional power, reduces frustrations, and consequently benefits the department and the organization as a whole. Another advantage of setting a records management staff person is that such a person naturally becomes the liaising or contact person for the organization's RM department. Even though he or she does not classify records for others, they could help with routine records management queries from their colleagues and assist the RM department to develop training tools addressing specific needs identified at the local level. 5.9 Learning Experience As introduced before, users' background information encompasses many educational and professional aspects, and all of them could contribute to their experience of learning and using the system, more or less. This research only collects limited information of users background, one of which, users' computer skills, is used here to explore its potential impact on users' experience of learning the system. 169 Table 5.14 System Learning and Computer Skills Q3 Self-rated computer skills V. Strong Strong Sufficient NR 21 22 7 Q30 Learning how to use the System has been easy N(SA+A) 17 13 4 N(SA+A)/NR 81% 59% 57% NN 3 6 2 NN/NR 14% 27% 29% N(DA+SDA) 1 3 2 N(DA+SDA)/NR 5% 14% 29% There is a clear indication in the table that computer skills have a positive impact on the respondents' learning experience. The agreement percentages descend from 81% for those responding "Very Strong" as to their computer skills, to 59% for the "Strong" group, and to 57% for the "Sufficient" group. The disagreement percentages confirms this in reverse, as percentages increase from 5% for the "Very Strong", to 14% for the "Strong", to 29% for the "Sufficient" groups . Even the "Neutral" percentages, which could be affected by many indeterminate factors, follow the same pattern: they increase when the levels of computer skills go down. Another factor relevant to users' system learning experience should be the assistance provided by the RM department. As discussed in the section on RM 170 department assistance, the majority of users consider that the assistance from the RM department contributes greatly to their learning of the system. 5.10 Program Outcomes The discussion of program outcomes focuses on three aspects: office productivity, information sharing, and paper records reduction. 5.10.1 Office Productivity The statements relevant to the discussion of office productivity come from both general and comparison groups: in the general group, Q24, "In general, the functions provided by [the EDRMS] help me with my job tasks"; in the comparison group, Q33, "less time is needed to retrieve documents or records", Q35, "using the EDRMS speeds up my completion of work, because I now can access documents and records (that I have the right to see) from any computers that are connected to the City's computer network, regardless of time or location", Q36, "co-operating with other departments becomes easier, because I now can access documents and records created by other departments, and vice versa", and Q37, and "My office productivity has increased'. The analysis results for Q24 are reported in Table 5.15. While there is a high agreement percentage for Q24 (90%), five respondents selected "Neutral". Considering that Q4, Q5, Q12, Q15, Q19, and Q22 could be factors affecting the respondents' experience of using the system, Table 5.15 examines the responses for these factors in relation to the five neutral selections. 171 Table 5.15 Analysis of Neutral Response to Q24 Q24 EDRMS functions help with job tasks Neutral 5 ID1 ID3 ID5 ID34 ID48 Q4 EDRMS interface is easy to use 2 4 2 2 2 Q5 Applications are sufficient for doing job 2 2 2 4 4 Q12 Classifying documents is easy 4 4 1 4 4 Q15 Finding documents is easy 2 2 3 4 3 Q19 Project Folders are useful 3 3 3 2 2 Q22 Saving emails with attachment(s) is easy 3 3 3 3 3 The above table exhibits that respondents feel uncertain over the assistance the EDRMS can provide for the completion of their job tasks normally have negative experience with one or more identified influential factors. For example, for the 6 factors, ID3 has 2 "Disagree" and 2 "Neutral" and ID34 has 3 "Disagree" and 1 "Neutral". Among the six factors, Q12, again, presents itself as the most problematic one, while Q4 works with most of the respondents. Also notably, Q22 has a consistent "Neutral" response from all respondents, indicating saving e-mails messages with attachment(s) could hinder job completion or office 172 productivity. The negative responses for Q33, Q35, and Q36 in the comparison group require additional information for further analysis, they are therefore not analyzed here. The analysis results for Q37 are reported in Table 5.16. There are 7 neutral responses and one "Disagree" for Q37. The analysis for Q37 identifies Q12, Q15, and Q22 as potentially influential factors. Table 5.16 Analysis of Neutral and Disagreement Response to Q37 Office productivity has increased Q37 Neutral (7) Disagree (1) 8 ID27 ID29 ID34 ID38 ID40 ID41 ID47 ID48 Q12 Classifying documents is easy Q15 Finding documents is easy Q22 Saving emails with attachment(s) is easy Similar to the results of the analysis of Q24, respondents who hold "Neutral" or "Disagree" opinions over the statement of increased office productivity all have negative experiences with one or more identified influential factors, with ID34, ID 38, and ID48 as representatives. This analysis may explain the fact that, while Q33 (locating time has been reduced), Q35 (completion of work has been speeded up), and Q36 (co-operation has become easier) - all of them contribute to office productivity - have high percentages, the overall office productivity is hindered by the difficulties caused by classifying and finding documents, and 173 saving email messages with attachments. These above analyzed "Neutral" or "Disagree" responses are all small numbers among the total number of responses. Nevertheless, these responses suggest issues for consideration. How to address the balance, for example, between the controls the system exercises, which are essential for managing electronic records, and employees' working habits, which greatly influence office productivity, could be one. In the above analysis, the indication that classification, saving e-mail messages with attachments, and difficulties of locating documents could reduce realizing the advantages brought by the system is clear and strong. 5.10.2 Information Sharing Information sharing has been promoted as one of the most attractive benefits of implementing an EDRMS in organizations, as the system permits widespread electronic access to business information. The idea of electronic access is basically built upon a combination of database(s) holding information and search tools that allow locating of information. The promise of sharing information as widely as possible, however, relies on not only the functionalities the system possesses but also on how users can make effective use of these functionalities. As indicated by the agreement percentage of Q6, the design of a central database encompassing all city records for the purpose of sharing information is fully appreciated by respondents of this research. When it comes to document locating, however, users consider browsing the universal file plan that contains all these documents and records the least preferred locating method (Q16, only 10% of 174 respondents select it). Moreover, as an overall assessment, finding documents is considered by 34% of respondents as difficult, and only 40% of them are comfortable with advanced search functions, which is the most powerful and effective means of searching documents or records created by others. Less effective or even unsuccessful documents locating causes user frustration and hinders timely information sharing. By contrast, successful and effective access to information generates higher user satisfaction and aids completion of job tasks. The analysis on the relationship between Q36 and Q15 confirms this (see Table 5.18). Table 5.17 Analysis of Information Sharing and Documents Locating Q36 Co-operation among departments becomes easier S. Agree (10) + Agree (17) 27 Q15 Finding documents is easy S. Agree (8)+Agree(12) Neutral Disagree 20 5 2 Q15 Neutral + Disagree 7 Q6 Centralized database facilitates information sharing S. Agree Agree 3 4 Among the respondents in the comparison group who agree that co-operation between departments become easier thanks to the system, 20 out of 27 agree that finding documents is easy. Information sharing between business or project 175 partners is to some degree different from information sharing in general. The usually tighter relationship and more constant communication between business partners or project members allow more information sharing channels. For example, the 7 respondents, while selecting "Neutral" (5) and "Disagree" (2) to Q15, all believe the design of central database facilitates information sharing (Q6). For these respondents, the central database could enhance information sharing in such a way that allows documents and records to be searched by "Document Number", a unique identifier for each and every document or record in the system, which can be easily sent among business or project partners through e-mail or phone calls. Nevertheless, searching documents and records in the system still remains the primary and most frequently used method. 5.10.3 Paper Reduction Reducing paper records volume is not a function provided by the system; however, it is expected as a positive outcome of implementing an EDRMS in organizations. For example, the RM department, through its program manual, advises users not to print out e-mail messages since saving e-mail records in the system captures not only content of messages and attachments but also technical information necessary for understanding them, and saves the storage costs from keeping their paper counterparts. Another strong argument for reducing paper records volume is the wide acceptance of electronic records as documentary evidence in legal proceedings. The Canadian Evidence Act, for example, applies the best evidence rule to electronic documents and it is satisfied "on proof of the 176 integrity of the electronic documents system by or in which the electronic document was recorded or stored." 183 The combination of the city's well-established records management program and the procedurally and technologically ensured integrity of the EDRMS, in the researcher's understanding, allows full confidence of the acceptance of the city's records as evidence and being given weight in any possible court disputes. There are possibly some factors contributing to the case of un-reduced paper volume. The first could reflect the current governmental business practice, which requires, for certain business procedures such as the handling of public requests for government information, use of paper records as official records. The second may relate to the lack of a sound strategy for the long-term preservation of electronic records. Since many records' retention schedules are longer than the lifespan of many information systems, some records may still need to be preserved on paper in order to remain accessible for the required time periods if there are no established effective preservation methods to maintain their electronic formats. The third may be the reason of convenience. For respondents who report finding documents in the system is not easy (34%), hard copies of records could be convenient for their day-to-day use. In any case, the fact that only 13 among 30 respondents consider paper records volume has been reduced requires more affirmative information and further analysis.184 Department of Justice, "Canada Evidence Act, R.S., c. E-10, s. 1, 31.2 (1) a)"; available from http://laws.iustice.qc.ca/en/C-5/232082.html: Internet; accessed 15 August 2006. 4 It is interesting to note that the centralized database of the EDRMS is design to reduce electronic copies of documents and records. The copying of any document or record in the system is indeed a creation of link to 177 5.11 Response Rate and Generalizability While this research has a more than satisfactory response rate (50 out of 60) in the sense of statistic analysis, its findings and analysis are not meant to be generalized to all or any other EDRMS program or general users of EDRMS. Every program has its unique macro- and micro-environments, and the respondents for the current research are selected for a particular purpose. Evaluations of EDRMS implemented in different environments and based on opinions from different user groups may generate different results. The purpose of presenting the findings and analysis here, in addition to reporting the evaluation results to the stakeholders of the target EDRMS program, is to communicate with the community of electronic records management at large and instruct EDRMS programs implemented in similar environments. To achieve this end, information on the program, the environment in which it operates, and the evaluation methodology are presented as detailed as possible. In the theory of program evaluation, generalization of evaluation results refers to practical relevancy of the results.185 In order to make practical use or adopt the evaluation results, potential users and adopters need to know what the program has done to reach the expected outcomes. In other words, with understanding of both the change model and the action model, users or adopters will be more confident about the use or adoption of the evaluation results. Figure 5.1 illustrates the idea of adopting a research system to a generalized system. the document or record in the central database, without any actual writing on any other storage medium. Chen, Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness, 219. 178 Research System Generalizing System Implementation environment Intervention Determinant Outcome Implementation environment Intervention Determinant Outcome Generalization Figure 5.1 Conceptual Framework for Enhancing Practical Relevancy (Source: Chen, 2005) 5.12 Further Research Consideration of areas for further research requires some discussion of research design and the program components evaluated in this research. Employing Likert scale statements to collect data from a comparatively large group of respondents allows the generation of an overall understanding of the EDRMS program; however, such data rarely explains the reasons behind respondents' choices. Two examples are provided here for illustration purpose. The first one comes from Q5, which asked about the applications integrated in the system (namely, the Microsoft Office suite). The majority of respondents considered the applications accessed through the EDRMS were sufficient for doing their jobs, but three respondents disagreed. The reasons for their 179 disagreement cannot be revealed in this research; neither is it possible to know whether or not this perceived insufficiency affects their completion of job tasks. Another example could be the response that the user had "never done" saving e-mail messages into the system, but had used the system for 5-7 years. It is unclear from this research, why, for such a long period of time, this respondent had never saved e-mail messages as business records. This response raised even more questions when considering the respondent has indeed responsibilities to manage departmental or office records. In this regard, all variables that resulted in uncertain and/or negative responses need to be further investigated. Research techniques such as focus groups or interviews can be employed to collect qualitative data for further analysis. Using only the "super users" of the system to be respondents also creates the need for further research. As the action model suggests, components other than the two evaluated in this research, such as the RM department, the general users of the system, and the IT department, should also be analyzed in order to produce a thorough assessment of the EDRMS program. The RM personnel have more experience with the system's RM administration function such as auditing, cutting off records, and profiling user groups and records management functions such as scheduling and disposing records. General users of the system, who manage their own documents and records, may experience more or fewer difficulties when using the system and classifying records. Their opinions on the system should be able to both deepen and broaden the understanding about the program and its operation. Cooperation with the IT department is critical since the EDRMS manages 180 electronic documents and records and is implemented in an electronic and networked environment. Information collected from the IT department regarding how they view the EDRMS system and what experience and lessons learned in the process of implementing, operating, and maintaining the system from an IT perspective would greatly assist the evaluation of the EDRMS program. The areas for further research suggested by program components (namely, the intervention and outcomes) include function-based classification, user needs analysis, and system customization as they presented themselves in the process of data analysis and discussions. Function-based records classification has been recently promoted as best practice in managing records and all the cases discussed in the chapter of literature review claimed their projects incorporated function-based classifications with the implemented EDRMS, but research on the construction, use and impact of the EDRMS on user experience has been little. User needs analysis is a relatively new topic in the records management professional. The importance of analyzing user needs has dramatically increased with the introduction of the EDRMS into organizations, which requires users to share some records responsibilities. More research on the balance between office productivity and management control needs to be carried out. System customization is a consideration relating to workflow analysis and also to user needs analysis. Most EDRMS are designed to be flexible to accommodate customization requirements and many of them have integrated workflow functions, but how and to what degree these functions can be integrated with records management requirements and customized to facilitate users' activities of 181 managing records has not been investigated in any comprehensive or systematic manner. This chapter discusses findings reported in Chapter 4 and extracts implications from analysis between and among variables. These discussions and implications, together with the statistic summaries, form the foundation that allows conclusions to be drawn upon and recommendations to be made. 182 Chapter 6 Conclusions This study has gathered some data that indicates that the implementation and operation of the EDRMS successfully meets the needs of the user group studied. For this group, the program has achieved, to a great degree, the goals set for it, that is, it has in broad measure achieved the desired outcomes. The high response rate of the questionnaire (83%) gives confidence to this conclusion, for it is unlikely that data about the experience of the few users in this group who did not respond would significantly change the picture. However, this general conclusion cannot be extended to all user groups in the city administration. Their experience of the system would have to be examined to draw such a conclusion. This is one methodological limitation of the current study. The expected outcomes were to enhance information sharing, to increase productivity, to allow for the production and maintenance of reliable and authentic records, and to reduce the volume of paper records. Fully 98% of respondents agree that the centralized documents/records repository facilitates information sharing, and 90% of them think the functions provided by the EDRMS help with their job tasks. Among the respondents who had worked in the city before the implementation of the system, 93% of them report that time has been reduced for searching and retrieving documents/records, 90% agree that the EDRMS has speeded up the completion of job tasks, and another 90% consider cooperation with other working units has become easier. While there are no standardized criteria for EDRMS success, it is fair to claim that these high agreement percentages demonstrate a very positive result of the implementation of the EDRMS. 183 However, not all of the measured outcomes have the same level of achievement. Within the comparison group, the agreement percentage regarding increased office productivity is a comparatively low one (73%) and the agreement percentage regarding reduced paper records volume is only 43%. While the evaluation was conducted under the guidance of the program theory, which allows the researcher to gain an understanding of why the outcomes occur, the reasons why long-time users think that the volume of paper records has not been reduced are not revealed. The volume of paper records is not controlled by the EDRMS, which simply supports the printing-out of electronic documents and records onto paper as users wish. The limitations caused by research methodological design and areas revealed during data analysis are identified for further research. In contrast, the evaluation of program components, as they are identified by the program theory, gives some understanding of why the goal of office productivity is not fully achieved. The findings and analyses of these evaluated program components indicate the following factors that have a negative influence on productivity: a) system functions: only 68% of the respondents, the majority of whom rate their computer skills very strong or strong, consider learning how to use the system is easy; b) classification: only 46% of users agree classifying documents is easy c) document locating: only 66% of users agree finding documents is easy; d) e-mail management: only 52% of users agree that saving e-mail messages and their attachments is easy 184 Productivity obviously depends on both the system and the knowledge and ability of users of the system. Records reliability and authenticity were measured based on users' understanding of records management concepts and practices. The assumption is that users' understanding of records management concepts and practices influences their attitudes towards their records management responsibilities. The study shows that, thanks to the training provided by the city's RM department, users generally clearly understand that profiling, classifying, and assigning access rights to records are necessary. However, there are a significant number of users who do not think declaring documents as records, which is a vital function for records management purposes, is necessary. This finding reveals that technology alone does not necessarily guarantee the assurance of records reliability and authenticity. The action of marking documents as records is not technically difficult, but, if users are not willing to perform it, records' reliability and authenticity cannot be fully protected. Moreover, since the decision of marking documents as records is decided by the action officers, there is a possibility that the city's business records are not adequately captured. The analysis concludes that every evaluated program component, namely, the EDRMS, the trainings provided by the RM department, and the users of the system, plays an important and indispensable role in the implementation and operation of the EDRMS program. User acceptance of the system appears to be the most critical factor cultivating success, and only training addressing user 185 needs will lead to desired outcomes. This overarching conclusion confirms the experience reported in the implementation case studies. User acceptance of the system is relevant to both documents and records management functions. The study strongly indicates that the best way to ensure user acceptance is to conduct careful user needs analysis. User needs analysis should include the analysis of user backgrounds, both their educational and professional experience, and the analysis of their specific job duties. The former can be accomplished through conducting user interviews, while the latter should be conducted taking into account the business units the user works or cooperates with. This suggests that the analysis of user job duties is logically linked to business process analysis. Business process analysis can be carried out focusing on outlining and defining steps in the current processes or on analyzing current processes with business re-engineering in mind, depending on organizational needs. This analysis should be carried out at an individual user level, as opposed to the unit level as the current practice has done.186 The analysis of user background and business processes forms the foundation for effective solutions. The system can be customized and technological assistance can be designed with specific user needs in mind. The analysis also enables user-friendly records management tools, such as function-based classification and business activity thesauri, and user-centered training materials and methods to be designed. Such 186 To solicit user inputs at the unit level and to include user representatives in the process of implementing an EDRMS, especially for constructing a file plan, are communicated and advocated by records management practitioners as best practice. See, for example, the articles about taxonomy in chapter 5, and also LynSne Downing, "Implementing EDMS: Putting People First," Information Management Journal 40, no. 3 (July/August 2006): 44-50. 186 user-centric solutions offer the best guarantee of avoiding the kinds of user resistance reported in the literature. Conducting business process analysis generates other benefits. Contributing to knowledge management is one of them. Knowledge management divides knowledge as explicit knowledge and tacit knowledge. Explicit knowledge can be articulated and communicated, and thus can be learned and disseminated easily. By contrast, tacit knowledge refers to personal experience, or "how-to" knowledge, which is normally difficult to capture and document.187 For example, as observed by records professionals, "business processes often reside in people's heads rather than in formalized writing."188 Tacit knowledge is believed to constitute the majority amount of knowledge, and it is the most valuable intellectual assets of the organization. The classic illustrative example for this point is the loss to organizations when employees leave. To capture tacit knowledge require first to identify the "knowers" of the knowledge, and conducting business process analysis can be a suitable time to identify them. Their expertise can then be documented in the process of identifying their needs for performing records management functions. It is also an action for fostering a culture of sharing. Business process analysis can be initiated by the desire to implement an EDRMS or for other reasons, such as the need to develop business process/workflow applications or the more comprehensive business process 187 Fred Nickols, "The Knowledge in Knowledge Management (KM)," available from http://home.att.net/~nickols/Knowledae in KM.htm: Internet; accessed 20 August 2006. 188 Ibid., 49 187 management applications. In fact, these technologies, if they are designed properly as modules, can be integrated with applications such as EDRMS. The more effective technological infrastructure should put the management of documents and records behind the scene of business so that, whenever and wherever documents and records management tasks come up, both technological and records management assistance are available for users. Conducting user needs analysis is of course not an easy task since careful user needs analysis places intensive demands on both human and financial resources. In this respect, intelligent user needs analysis presents a way for records managers to present a sound business case to gain management support. Management controls over documents and records in an electronic environment is beyond a doubt necessary, and users' share of records management responsibilities in an EDRMS is inevitable. However, balance should be achieved between control and productivity, and conscious efforts to reduce extra work should be made based on user needs analysis. The need for this balance is apparent in the DoD5015.2 standard, as it stipulates controlling functions and at the same time requires the system to provide assistance to users. In the end, technological advances and innovative approaches cannot replace sound records management concepts and principles, which should be embedded firmly in organizations' records management policies, and be endorsed by top management. Technologies and standards by themselves do not fix poor records 188 management practices, nor does it automatically increase office productivity. Accommodating user needs also does not mean giving users whatever they want. Their needs should be meshed with the business and records management needs of the entire organization, which inevitably means making choices and compromises. It is also true that it is management's responsibility to ensure that users know how to use the system and follow requirements, as one respondent reported: "In my opinion, the real problem with [the EDRMS] is the lack of enforcement by management. The system is brilliant. The people who use it (or not use it as they should) are causing the problem." (Respondent ID50) 189 Selected Bibliography AIIM/ARMA Standards Committee on Integration of Electronic Document Management Systems (EDMS) and Electronic Records Management Systems (ERMS) Functional Requirements. "Technical Report: Framework for Integration of Electronic Document Management Systems and Electronic Record Management Systems (ANSI/AIIM/ARMATR48-2004). Available from http://www.arma.org/bookstore/productdetail.cfm?ProductlD=1479; Internet; accessed 18 August 2006. American Council for Technology/Industry Advisory Council (ACT/IAC). "The Use of Metrics In Electronic Records Management (ERM) Systems." Available from www.actqov.org/actiac/documents/sigs/egov/08032004ERMMetricsFinal.pdf; Internet; accessed 16 August 2006. Babbie, Earl. The Practice of Social Research. 9th ed.. Wadsworth, Thomson Learning Inc., 2001. Bernard, H. Russell. Social Research Methods: Qualitative and Quantitative Approaches. Thousand Oaks: Sage, 2000. Brock, Carol and Espada, Peter. "GAO's Electronic Records Management System (ERMS) -Presentation to the ARMA Northern Virginia Chapter." Available from www.labat.com/presentation/erms.ppt; Internet; accessed 17 August 2006. Broughton, Vanda. Essential Classification. New York: Neal-Schuman, 2004. Bruno, Denise and Richmond, Heather. "The Truth About Taxonomies." Information Management Journal 37, no.2 (March/April 2003): 44-53. Bryman, Alan. Social Research Methods. 2nd ed. New York : Oxford, 2004. 190 Carlin, John W.. "NARA bulletin 2003-03." Available from http://www.archives.gov/records-mgmt/bulletins/2003/2003-03.html; Internet; accessed 15 August 2006. Chen, Huey-Tsyh. Practical Program Evaluation: Assessing and Improving Planning, Implementation, and Effectiveness. Thousand Oaks, Calif.: Sage Publications, c2005. Chen, Huey-Tsyh. Theory-driven Evaluations. Newbury Park, Calif.: Sage Publications, 1990. Cisco, Susan L, and Jackson, Wanda K.. "Creating Order Out of Chaos with Taxonomies." Information Management Journal 39, no.3 (May/June 2005):44-50. Cornwell Management Consultants. "Model Requirements for the Management of Electronic Record (MoReq)." Available from http://www.cornwell.co.uk/moreq.html; Internet; accessed 15 August 2006. Daniels, Maygene F. and Walch, Timothy.ed.. A Modern Archives Reader: Basic Readings on Archival Theory and Practice. Washington, D.C.: National Archives and Records Service, U.S. General Services Administration, 1984. Department of Justice. "Canada Evidence Act, R.S., c. E-10, s. 1, 31.2 (1) a)." Available from http://laws.iustice.gc.ca/en/C-5/232082.html; Internet; accessed 15 August 2006. Downing, LynSne. "Implementing EDMS: Putting People First." Information Management Journal 40, no.3 (July/August 2006): 44-50. Duranti, Luciana. "Reliability and Authenticity: The Concepts and Their Implications." Archivaria 39 (Spring 1995): 5-10. Durant, Luciana. "The Archival Bond." Archives and Museum Informatics 11(1997): 191 213-218. Duranti, Luciana. Diplomatics: New Use for an Old Science. Lanham, Md. : Scarecrow Press, 1998. Duranti, Luciana and Eastwood, Terry. "Protecting Electronic Evidence: A Progress Report on a Research Study and its Methodology." Archivi & Computer 5, no.3 (1995): 214-215. Duranti, Luciana and Eastwood, Terry and MacNeil, Heather. Preservation of the Integrity of Electronic Records. Dordrecht: Kluwer Academic, 2002. Fagerlund, Liisa. "Management of Electronic Records in the United Nations." Available from http://www.archimuse.com/publishing/elec prgmstr/elec prgmstr Fagerlund.pdf# search=%22accis%20united%20nations%22: Internet: accessed 17 August 2006. Government of British Columbia. "ARCS Online." Available form http://www.bcarchives.gov.bc.ca/arcs/index.htm: Internet; accessed 12 August 2006. Government of British Columbia, Corporate Information Management Branch. "The Standard ORCS Kit - 2001 Edition." Available from http://www.mser.gov.be.ca/CIMB/polocy/default.htm#std_orcs_kit; Internet; accessed 12 August 2006. Gregory, Keith. "Implementing an Electronic Records Management System: A Public Sector Case Study." Records Management Journal 15, no.2 (2005): 80-85. Hummingbird. "[Home Page]." Available from http://www.hummingbird.com; Internet; accessed 15 August 2006; Tower Software. "[Home Page]." Available from http://www.towersoft.com/global; Internet; accessed 15 August 2006. 192 International Organization for Standards, 15489-1:2001 Records management -Part 1: General: 5 InterPARES. "Requirements for Accessing and Maintaining the Authenticity for Electronic Records." Available from http://www.interpares.org/book/interpares book k app02.pdf: Internet; accessed 13 August 2006. InterPARES. "InterPARES 2 terminology database." Available from http://www.interpares.org/ip2/ip2 terminology db.cfm; Internet; accessed 11 August 2006. International Organization for Standardization, ISO 23081-1:2006 Information and documentation - Records management processes - Metadata for records - Part 1: Principles. Kelly, Moira J.. "Qualitative evaluation research" In Qualitative Research Practice, ed. Clive Seale and others, 332. London; Thousand Oaks, Calif.: SAGE, 2004. Library and Archives Canada. "BASCS Guidance." Available from http://www.collectionscanada.ca/information-management/002/007002-2091-e.ht ml#one; Internet; accessed 20 August 2006. Liddell, Anne. "The NAA Experience of Using AS 5090 - Australian Standard for Work Process Analysis for Recordkeeping to Support its DIRKS Project." Available from http://www.naa.gov.au/recordkeeping/rkpubs/fora/03Nov/AS 5090 paper.pdf#se arch=%22%22Anne%20Liddell%22%20NAA%22; Internet; accessed 22 August 2006. Maguire, Rachael. "Lessons Learned from Implementing an Electronic Records Management System." Records Management Journal 15, no. 3 (2005): 150 -157. 193 McDonald,John. "Managing Records in the Modern Office: Taming the Wild Frontier." Archivaria 39 (Spring 1995): 70-79. McDonald,John. "The Wild Frontier Ten Year on." In Managing Electronic Records, ed. Julie McLeod and Catherine Hare, 1-17.London, [UK]: Facet Pub., 2005. National Archives of Australia. "Digital Recordkeeping Guidelines." Available from http://www.naa.gov.au/recordkeeping/er/guidelines.html; Internet; accessed 20 September 2006. National Achieves of Australia. "DIRKS Manual." Available from http://www.naa.gov.au/recordkeeping/dirks/dirksman/contents.html; Internet; accessed 17 August. 2006. National Archives of Australia. "The DIRKS Manual: A Strategic Approach to Managing Business Information." Available from http://www.naa.gov.au/recordkeeping/dirks/dirksman/dirks.html; Internet; accessed 15 August 2006. National Achieves of Australia, "DIRKS: Step B - Analysis of Business Activity." Available from http://www.naa.gov.au/recordkeeping/dirks/dirksman/step B.html; Internet; accessed 18 August 2006 National Archives of Australia. "Functional Specifications for Electronic Records Management Systems Software." Available from http://www.naa.gov.au/recordkeeping/er/erms/specifications.html; Internet; accessed 15 August 2006. National Archives of Australia. "Keyword AAA." Available from http://www.naa.gov.au/recordkeeping/control/KevAAA/summary.html; Internet; accessed 18 August 2006. National Archives of Australia. "Overview of Classification Tools for Records 194 Management." Available from http://www.naa.gov.au/recordkeeping/control/tools.pdf; Internet; accessed 17 August 2006. Nickols,Fred. "The Knowledge in Knowledge Management (KM)." Available from http://home.att.net/~nickols/Knowledge in KM.htm; Internet; accessed 20 August 2006. Nova Scotia Archives and Record Management. "The Standard for Administrative Records." Available from http://www.gov.ns.ca/nsarm/organization/rm/star5/index.htm ; Internet; accessed 12 August 2006. O'Kane, Trish. "United by a Common Language: Developing a Corporate: Taxonomy Case Study Emphasizes the Need for Consensus, Understanding of Technical Features of Metadata." Information Management Journal 40, no. 4 (July/August 2006): 58-63. O'Leary, Zina. The Essential Guide to Doing Research. London; Thoundand Oaks: SAGE, 2004. Rossi, Peter H., Lipsey, Mark W., and Freeman Howard E. Evaluation: A Systematic Approach. 7th ed. Thousand Oaks, CA : Sage, 2004. Sabourin, Paul. "Constructing a Function-Based Records Classification System: Business Activity Structure Classification System." Archivaria 51 (Spring 2001):153. Schellenberg, Theodore.R.. Modern Archives: Principles and Techniques. Chicago : University of Chicago Press, 1956. Smyth, Zoe" A.. "Adopting a Functional Classification of Business Processes in Northern Ireland." Journal of the Society of Archivists 26, no. 2 (October 2005): 195 234. Smyth, Zoe A. "Implementing EDRM: Has It Provided the Benefits Expected?" Records Management Journal 15, no. 3 (2005): 141-149. Sprehe, J. Timothy. "A Framework for EDMS/ERMS Integration." The Information Management Journal 38 (November/December 20 04): 54-62. Sprehe, J. Timothy and McClure, Charles R. "Lifting the Burden." The Information Management Journal 39. no.4 (July/August 2005): 47-59. Treasury Board Canada Secretariat. "Policy on the Management of Government Information." Available from http://www.tbs-sct.gc.ca/pubs pol/ciopubs/TB GIH/mgih-grdq1 e.asp#pol; Internet; accessed 13 August 2006. United Kingdom National Archives. "Business Classification Scheme Design." Available from http://www.nationalarchives.gov.uk/electronicrecords/advice/pdf/bcs toolkit.pdf; Internet; accessed 20 August 2006. United Kingdom National Archives. "TNA ERMS Test Evaluation Report for IBM DB2Records Manager 4.1.1 Suite." Available from http://downloads. nationalarchives.gov. uk/.../Dexmar%20Ltd%20-%20Dexmar%2 0KnowPro%20EDRM.pdf; Internet; accessed 18 August 2006. United Kingdom National Archives. "Functional Requirements." Available from http://www.nationalarchives.gov.uk/electronicrecords/function.htm; Internet; accessed 15 August 2006. United States, Department of Defense. "Design Criteria Standard for Electronic Records Management Applications." Available from http://www.dtic.mil/whs/directives/corres/pdf/50152std 061902/p50152s.pdf; 196 Internet; accessed 15 September 2006. UBC Project. "Preservation of the Integrity of Electronic Records." Available from http://www.interpares.org/UBCProiect/intro.htm ; Internet; accessed 11 August 2006. Wholey, Joseph S., Hatry, Harry P. and Newcomer, Kathryn E. ed. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass, 1994. Wallace, David. "Preserving the U.S. Government's White House Electronic Mail: Archival Challenges and Policy Implications." Available from http://www.ercim.org/publication/ws-proceedings/DELOS6/wallace.pdf#search=% 22%22Preserving%20the%20U.S.%20Government's%20White%20House%20EI ectronic%20Mail%22%22; Internet; accessed 17 August 2006. Williams, David J. "EDRM Implementation at the National Weights and Measures Laboratory." Records Management Journal 15, no.3 (2005): 158-166. Williams, Robert F. and Ashley, Lori J. "2005 Cohasset Electronic Records Management Survey: a Renewed Call for Action." Available from http://www.merresource.com/downloadWhitepaper.htm?fileld=1; Internet; accessed 1 September 2006. 197 Appendices Appendix A Questionnaire INSTRUCTIONS • The questionnaire consists of three parts. • Please complete Part I and Part II (questions 1 - 30). • Please complete Part III (questions 31-35) only if you have worked for the City for more than 10 years. • Please place a tick mark in the space next to the answer you select. • Please return the questionnaire in the self-addressed, stamped envelope provided with this questionnaire. Part I: General Information 1. How long have you worked for the City? Fewer than 10 years (inclusive) More than 10 years 2. How long have you used [the EDRMS]? _ 1 _2-4 _5-7 _8-10 3. How would you rate your computer skills? Very Strong Strong Sufficient for job requirements _Weak Very Weak End of Part I. Please go to Part II on the next page. 198 Part II: Your Experience of Using Tthe EDRMS1 Please select the answer that most closely matches your level of agreement with each statement. 4. The [EDRMS] interface is easy to use. Strongly Agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 5. The applications (e.g., Microsoft Office) accessed through [the EDRMS] are sufficient for doing my job. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 6. Saving all city documents and records in [the EDRMS] facilitates information sharing among departments/divisions/sections. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 7. Creating profiles for documents is necessary. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 199 8. Supplying a descriptive title when profiling documents is easy. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 9. Selecting security options when profiling documents is easy. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 10. "Modify Security" is useful for sharing information (when full access is assigned to the document) and protecting confidentiality (when selected access rights are assigned to the document) Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree I'm not aware of "Modify Security" 11. Classifying documents when profiling documents is necessary. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 12. Classifying documents is easy. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 200 If you selected "Neutral", "Disagree" or "Strongly Disagree" when answering question #12, please answer question #13 and then continue with question #14. If you selected "Strongly Agree" or "Agree" to question #12, please go to question #14. 13. It is difficult to classify documents because (select all that apply) there are too many levels and too many choices in the file classification system not all of the primaries (categories) in the file classification system are self-explanatory to me, and the explanations (scope notes) of the categories are not linked to these categories "Recently Used Files" is not helpful the file classification system does not accommodate my needs other (please specify) 14. Setting fields in the document profile with default values reduces the time of profiling documents. Strongly Agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree I am not aware of default values 15. Finding documents or records in [the EDRMS] is easy. Strongly Agree Agree Neutral Disagree Strongly Disagree 201 16. My preferred methods of retrieving documents or records in [the EDRMS] are (select all that apply) Single field search in the profile (e.g., document number or author or title) Multi-fields search in the profile (e.g., document number and author and title) Recently Edited Documents Browsing the file classification system Advanced Search (easy search, content search, custom search) 17. "Quick Searches" are convenient because the queries I formulate can be saved for later use or edited for new use. Strongly Agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree I am not aware of "Quick Searches" 18. Sorting search results (e.g., sorting by columns like document title) helps me find desired documents and/or records. Strongly Agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree I'm not aware of sorting search results 19. "Project Folders" are useful since it allows me to group my documents and/or records together as I wish. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 202 20. Attaching documents or records in [the EDRMS] to an e-mail message is easy. Strongly Agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 21. Saving e-mail messages in [the EDRMS] is easy. Strongly Agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 22.Saving email messages that have attachments in [the EDRMS] is easy. Strongly Agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 23. "History" is useful because it includes information about a document and what has happened to it since it was created (e.g., information about who accessed it and when) Strongly Agree Agree Neutral Disagree Strongly Disagree I'm not aware of "History" 24. In general, the functions provided by [the EDRMS] help me with my job tasks. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 203 25. It is important to have a staff person in each department with assigned responsibility to manage documents and records. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 26. Using the titling guidelines facilitates document and records retrieval. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 27. Assistance from the City's [RM Department] (such as training sessions, manuals and brochures, and help provided through phone calls) has been necessary for my understanding and use of [the EDRMS]. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 28. The following types of assistance have been most valuable to me (select all that apply) Classroom computer training One-on-one instruction (including telephone and on-site) Training videos (TV snacks) on the Intranet Manuals and brochures Other (please specify) 204 29. The assistance currently provided by the [RM Department] for using [the EDRMS] is Extremely helpful _ Helpful Neutral (neither agree nor disagree) _ Not helpful Totally useless 30. Learning how to use [the EDRMS] has been easy. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 31. When using documents in [the EDRMS], I trust them as reliable information sources. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree I have never thought about this. 32. "Mark as a Record" is a necessary feature of [the EDRMS]. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree I'm not aware of "Mark as Record" End of Part II. Please go to Part III on the next page. 205 Part III: Section for Employees of More Than 10 Years If you selected "More than 10 years" when answering question #1, please answer the following additional questions: In comparison with the situation before [the EDRMS] was implemented, I find that 33. less time is needed to retrieve documents or records. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 34. the volume of paper records is reduced. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 35. using [the EDRMS] speeds up my completion of work, because I now can access documents and records (that I have the right to see) from any computers that are connected to the City's computer network, regardless of time or location. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 36. co-operating with other departments becomes easier, because I now can access documents and records created by other departments/division/sections, and vice versa. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree 206 37. my office productivity has increased. Strongly agree Agree Neutral (neither agree nor disagree) Disagree Strongly Disagree End of the Questionnaire. THANK YOU VERY MUCH FOR YOUR TIME AND CO-OPERATION. 207 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Country Views Downloads
United States 86 17
Canada 60 30
Unknown 53 10
Philippines 49 71
Germany 26 75
United Kingdom 19 16
Nigeria 18 10
Uganda 16 4
Kenya 15 7
Botswana 11 13
Poland 10 0
Sweden 8 0
France 8 0
City Views Downloads
Unknown 261 223
Ottawa 18 2
Halifax 12 0
Ashburn 11 0
Gaborone 11 13
Mountain View 11 2
Toronto 8 9
Sofia 8 0
Stockholm 8 0
New York 7 2
Tarlac City 6 3
Tokyo 6 0
London 5 2

{[{ mDataHeader[type] }]} {[{ month[type] }]} {[{ tData[type] }]}
Download Stats

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0092752/manifest

Comment

Related Items