UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Developing effective knowledge management systems Nevo, Dorit 2003

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
831-ubc_2003-859398.pdf [ 9MB ]
Metadata
JSON: 831-1.0091216.json
JSON-LD: 831-1.0091216-ld.json
RDF/XML (Pretty): 831-1.0091216-rdf.xml
RDF/JSON: 831-1.0091216-rdf.json
Turtle: 831-1.0091216-turtle.txt
N-Triples: 831-1.0091216-rdf-ntriples.txt
Original Record: 831-1.0091216-source.json
Full Text
831-1.0091216-fulltext.txt
Citation
831-1.0091216.ris

Full Text

D E V E L O P I N G E F F E C T I V E K N O W L E D G E M A N A G E M E N T S Y S T E M S By DORIT NEVO B.A Economics, University of Haifa, 1995 M.Sc Economics, Technion, 1998 A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE STUDIES Saucier School of Business We accept this thesis as conforming to the required standard THE UNIVERSITY OF BRITISH ( J O L U M B I A June 2003 © Dorit Nevo, 2003 In presenting this thesis in partial fulfilment of the requirements for an advanced degree at the University of British Columbia, I agree that the Library shall make it freely available for reference and study. I further agree that permission for extensive copying of this thesis for scholarly purposes may be granted by the head of my department or by his or her representatives. It is understood that copying or publication of this thesis for financial gain shall not be allowed without my written permission. Department of Srtudef Slhcxy) C f^ IcOsJrygsS, The University of British Columbia Vancouver, Canada Date DE-6 (2/88) ABSTRACT Knowledge Management Systems (KMS) are computerized systems intended to support the management and application of organizational knowledge. Despite the many potential benefits from KMS> industry reports show that companies are having difficulties in realizing these benefits. This dissertation sets out to identify the reasons for those difficulties - from a requirements analysis point of view - and to propose a way to improve the design of KMS in order to enhance their benefits to companies. We develop a theory-based approach to the evaluation of Knowledge Management Systems and identify the main shortfalls of existing systems. The findings show two inter-related problems. First, the lack of a conceptual model such as an organizational ontology or of additional knowledge about the knowledge might inhibit KMS ability to support specific organizational KM processes. Second - the lack of an integrated KM product reduces organization's ability to attain an overall solution for the management of organizational knowledge. In addition, the analysis pinpoints the inability of current technology to support effective management of tacit knowledge - an important aspect of KMS design. In the second part of the dissertation we focus on one of the problems identified - namely, the lack of a unified meta-knowledge set - and identify the specific meta-knowledge that should be incorporated into the design of KMS. The expected benefits from the inclusion of meta-knowledge are better management of organizational memory and enhanced adoption of knowledge by KMS users. We conduct an empirical investigation - using conjoint analysis - to identify specific attributes of knowledge and of knowledge sources that individuals take into account when making their knowledge use decision, and propose that these attributes should be included as meta-knowledge in the KMS design. The results indicate that the 'accuracy' and 'relevance' of the knowledge are the top two important attributes in the knowledge selection decision and 'extent of knowledge' and 'trustworthiness' are the top two important attributes in the selection of a knowledge source. In addition the results show that knowledge plays a slightly more important role than knowledge source in the overall decision. Finally, several demographic and contextual variables - such as the knowledge search task and organization size - are shown to affect the importance of different attributes in the overall evaluation of knowledge and of knowledge source by individuals. Table of Content ABSTRACT ii List of Tables v List of Figures vi Acknowledgement vii Chapter 1: Introduction 1 Research objective 4 Part I: Identifying the shortfalls of existing Knowledge Management Systems 6 Chapter 2: A Framework to Evaluate Knowledge Management Systems 8 The KM Processes 10 Knowledge generation 10 Knowledge codification 11 Knowledge transfer. 12 Deriving KMS functionalities 14 Individual learning 14 Socialization 15 Externalization / Retrieval 16 Internalization / Storage 16 Implied design features of KMS 18 Mapping design features to KM processes 21 Chapter 3: An evaluation of existing KM tools 24 Analysis and Evaluation 27 Discussion 32 Part II: Exploring meta-knowledge requirements for Knowledge Management Systems. 34 Chapter 4: The role of meta-knowledge in KMS design 35 Managing organizational memory 38 Transactive memory theory. 40 Enhancing the adoption of knowledge 41 Elaboration Likelihood Model 43 Research objective 44 Chapter 5: Methodology • 46 Theoretical foundations 47 Selection of attributes - a Delphi study 52 Delphi Study Design 54 The Delphi process 57 Results 60 Discussion 61 Conjoint study design 63 Overview 63 Selection of data collection method 64 Identifying attribute levels 66 Stimulus set construction 69 Selection of preference model 69 Estimation method 70 Reliability and Validity 70 Participants 71 Chapter 6: Findings 73 Interpreting the conjoint analysis results 73 Knowledge attributes 75 Secondary data analysis - cluster analysis 76 Characterizing clusters' members 80 Source attributes 82 Secondary data analysis - cluster analysis 85 Characterizing clusters' members 88 Discussion 89 Expanding The Information Adoption Model 92 Practical Implications 93 Chapter 7: Contributions, Limitations, and Future Research 96 Contributions 96 Limitations 97 Future work 98 Appendix A: Evaluation of knowledge management systems 99 Tools reviewed 99 Tools classification examples 99 Example of deriving functionalities 101 Appendix B: Delphi questionnaires 102 Letters and emails 102 Approach letter to companies 102 First round invitation 104 Second round invitation 104 Third round invitation 105 Final round invitation 105 Questionnaires: 106 Delphi study - consent form and demographics questionnaire 106 Delphi study - Round 1 questionnaire 108 Delphi study - Round 2 questionnaire 112 Delphi study - Round 3 questionnaire 114 Delphi study - round 4 questionnaire 7 75 Appendix C: Conjoint questionnaire 117 Interview questions 117 First interview 117 Second interview 778 Second interview results - by example 121 Letters and emails 123 Approach letter to companies 123 Email to KMC list 125 Conjoint questionnaire 126 Appendix D: Conjoint outputs 139 Importance of knowledge attributes 139 Overall summary 139 Professionals subgroup 140 MBA subgroup 141 Undergraduates subgroup 142 Importance of knowledge source attributes 143 Overall summary 143 Professionals subgroup 144 MBA subgroup „ 145 Undergraduates subgroup 146 Importance of knowledge and source attributes - by question 147 Reference 148 iv List of Tables Table 1: Definitions of knowledge management 9 Table 2: Summary of functionality support in KM processes 18 Table 3: How design features of KMS support KM processes 22 Table 4: Classification of commercial KMS 25 Table 5: Evaluation of KMS 28 Table 5a: Functionalities and design features required from KMS' categories evaluated.... 28 Table 5b: Percent of tools supporting a given design feature or a functionality 28 Table 6: Meta-knowledge solution for organizational memory problems 39 Table 7: Examples of Delphi research in IS 53 Table 8: Demographics of panel members 56 Table 9: Results of round 1 and 2 of the Delphi study 59 Table 10: Ranking of meta-knowledge attributes 60 Table 11: First interview results 66 Table 12: Second interview results 67 Table 13: Levels of attributes used in the survey 68 Table 14: Demographics of conjoint study participants 72 Table 15: Knowledge clusters' centers 78 Table 16: Multinomial logit regression on knowledge clusters 81 Table 17: Knowledge source clusters' centers 86 Table 18: Knowledge source clusters vs. knowledge/source importance weights 87 Table 19: Multinomial logit regression on knowledge source clusters 88 Table 20: Conjoint analysis results (averaged across all respondents) 91 Table 21: Proposed vs. existing knowledge and source attributes 94 Table 22: Interview with Delphi panel members 117 Table 23: Data from second interview. 122 v List of Figures Figure 1: Structure of the evaluation framework 7 Figure 2: Channels for knowledge transfer in organizations 13 Figure 3: KM processes and supporting functionalities 15 Figure 4: Required design features of KMS 21 Figure 5: Shortfall of existing KMS to support knowledge transfer 30 Figure 6: Percent of tools that effectively support KMS functionalities 31 Figure 7: The information adoption model (Source: Sussman and Siegal, 2003) 43 Figure 8: Information Integration Theory (source: Anderson, 1981) 48 Figure 9: Factorial design for functional measurement (source: Anderson, 1981) 49 Figure 10: Examples of conjoint profiles (Profile 1a - knowledge source; Profile 1b -knowledge) 68 Figure 11: Types of relations between attribute levels 70 Figure 12: Example of conjoint analysis output 74 Figure 13: Knowledge attributes importance (by group) 75 Figure 14: Dendrogram for knowledge clusters 77 Figure 15: Bi-plots for knowledge clusters solution (15a: comp.1-2; 15b: comp.2-3) 79 Figure 16: Knowledge source attributes importance (by group) 83 Figure 17: Dendrogram for knowledge source clusters 85 Figure 18: Bi-plot for knowledge source clusters solution 86 Figure 19: Screen shot of questionnaire - knowledge/source importance question 87 Figure 20: Knowledge/source importance weights 90 Figure 21: The revised information adoption model 92 Figure 22: Example of tools classification 100 Figure 23: Example of deriving functionalities 101 vi Acknowledgement I would not be writing these words without some very important people, to whom I would like to thank. To my advisor, Yair Wand, I owe you many thanks for your invaluable guidance and support throughout the years. I want to especially thank you for letting me find my own way but at the same time keeping me focused and making sure I didn't wander off too much. To my committee members, Izak Benbasat from whom I learned a great amount, and Martin Schulz who opened my eyes to the world outside of MIS - thank you for sharing your knowledge with me. Finally, I would like to thank my thesis examiners - Al Dexter, Rick Kopak, and Maryam Alavi, who took the time and effort to read and evaluate my work. I also could not have survived the process without the friendship and support of Paul Chwelos, Dan Gardiner, and Dan Putler. I would like to thank you for your tremendous help and knowledge. And to my 'comrades' - Nanda Kumar and Joerg Evermann, thanks for reading the countless versions of this work and providing comments, but more importantly - for always taking the time to have the much-needed coffee break and conversation. To Saggi, I know it was difficult at times but you never stopped supporting, helping, and loving. Now it's your turn and I promise to be there. Finally - to my parents, for their endless love. vii Chapter 1: Introduction This dissertation aims to extend our understanding of user requirements from Knowledge Management Systems and to provide guidelines for systems' designers in building such computerized systems intended to support effective knowledge management in organizations. In recent years, practitioners as well as researchers have come to recognize the importance of knowledge as an asset to the organization. Knowledge that resides in various places in the organization represents experiences, education, and other valuable lessons for the management and operation of the organization. Such knowledge is crucial to success in an environment of increased competition and globalization (Alavi, 2000; Zack, 1999). To enhance organizations' abilities to handle knowledge, various practices for Knowledge Management (KM) have evolved. Some of these are managerial in nature, such as documenting "best practices" or encouraging learning (Davenport and Prusak, 1998) while others are more technology focused and are termed "Knowledge Management Systems" (KMS). Regardless of the specific practices or technologies used, the aim of such KM approach, and subsequently - the aim of knowledge management systems, can be defined as: "Knowledge management involves efficiently connecting those who know with those who need to know, and converting personal knowledge into organizational knowledge" (the Yankee group, cited in Cairncross, 2000:s20)1 Knowledge Management (KM) in organizations is now widely recognized and expected to be an important part of organizational practices in the future (GartnerGroup, 2002a). In a 1998 study on knowledge management (KPMG, 1998) 53% of the respondents indicated that lack of attention to knowledge management had negative effects such as damaged relationships with clients and suppliers or loss of income. In a more recent industry study (KPMG, 2000) findings show that the majority of participating companies (81% of 423 companies) already engaged in some KM practices or initiatives, usually with the help of technology. Expected benefits included better decision-making, better customer handling, improved employee skills, improved productivity, increased profits, and faster response to key business issues. Unfortunately - the findings of the KPMG report also show that in many cases - companies were not realizing the 1 While there are other approaches to knowledge management - such as the knowledge based view of the firm (summarized in Conner and Prahalad, 1996; Grant, 1996) - the focus in this work is bounded by the context of organizational knowledge management systems and therefore I focus on the approach represented by this definition. 1 expected benefits from their knowledge management systems. These companies reported problems such as information overload, reinventing the wheel, difficulties in sharing tacit knowledge, lack of time to share knowledge, and inability to use the technology effectively. To understand why IT fails to effectively support KM it is important to examine first the nature of what is being managed by technology - in other words, the nature of organizational knowledge. As Alavi and Leidner (2001) note in their review of KMS: "the design of information systems should be rooted in and guided by an understanding of the nature and types of organizational knowledge" (p.115). In epistemology - the theory of knowledge - knowledge is defined as a "Justified True Belief" (JTB), which means that in order for a person to know some predicate S, S has to be true, the person must believe that S is true (and be aware of the existence of S), and this belief must be justified in some way - for example, empirically (Lehrer, 1990). While this is the basic definition of knowledge note that various theories exist as to the nature of truth and justification (Moser and VanderNat, 1987) as well as some criticism on the strength of the definition itself (e.g. Gettier, 1963; Harman, 1968). Keeping within the scope of this work, adopting the JTB definition to organizational knowledge requires some adaptation. Specifically - researchers have diminished the importance of 'truth' and added a forth dimension stressing the result of the knowledge. A popular definition demonstrating these changes is "knowledge is a justified belief that increases an entity's potential for effective action" (Alavi and Leidner, 2001:109). Other definitions of organizational knowledge also follow the requirement for effective action, for example, knowledge is defined as "a capacity to act" (Sveiby, 1997:37); "information that changes something or somebody" (Drucker, 1989:251); or simply as "information in action" (O'Dell and Grayson, 1998:5). These latter definitions also embed the hierarchy of data, information, and knowledge that is common in the KM literature (Alavi and Leidner, 2001; Davenport and Prusak, 1998). Nonaka (1994) - building on a previous work by Polanyi (1967) - applied Polanyi's distinction between tacit and explicit knowledge to the context of organizational knowledge - where knowledge is defined as a "justified true belief. Explicit knowledge can be expressed using language or other formal representation and communicated easily. On the other hand, tacit knowledge is highly personal and hard to formalize and is rooted in action, commitment, and involvement in a specific context. Tacit knowledge can be technical - representing skills and crafts - or cognitive, referring to our beliefs, ideas and mental models. While Nonaka used this distinction to develop a theory of organizational knowledge creation, the tacit/explicit distinction 2 is material in other aspects of the KM literature and is especially relevant in the development of KMS, determining a basic requirement that a KMS be able to manage, or at least support the management of, both tacit and explicit knowledge (Alavi, 2000; Ruggles, 1997; Stenmark, 2000). This discussion of knowledge highlights one potential explanation for why KMS are not as effective as expected. As McDermott (1999) posits "the great trap in knowledge management is using information management tools and concepts to design knowledge management systems" (p. 104). McDermott focuses on the fact that knowledge is inherently different from information and thus technology that is fit to support the management of information may not be as effective in managing knowledge. Similarly, Walsham (2001) argues against the application of technology to capture and transfer knowledge by focusing on Polanyi's original discussion of tacit and explicit knowledge. According to this view, the distinction made by many researchers between tacit knowledge and explicit knowledge is inaccurate. In fact - every knowledge has a tacit dimension that affects the individual's understanding of this knowledge. Therefore - technology that attempts to support a process such as 'knowledge transfer' will not be effective since it cannot capture the tacit dimension of the transferred knowledge. Both Walsham and McDermott support the application of technology in one specific aspect of KM - communications within small working groups with shared understanding, such as communities-of-practice. The above argument against a general KMS however is not widely accepted and many researchers support the use of IT to manage both tacit and explicit knowledge in various organizational knowledge management processes - such as knowledge creation, knowledge codification and storage, knowledge transfer, and knowledge application (Alavi and Leidner, 2001, Davenport and Prusak, 1998, Ruggles, 1997). Zack (1999), for example, classifies knowledge into declarative ('know what'); procedural ('know how') and causal ('know why') and proposes that' most of this knowledge can be explicated and subsequently managed by knowledge repositories or similar systems. Blackler (1995) identifies five different types of organizational knowledge - some of which can be managed by KMS. Embrained knowledge is knowledge that depends on conceptual skills and cognitive abilities; Embodied knowledge is action oriented (e.g. the knowledge of a craftsman) and mostly tacit; Encultured knowledge represents shared understanding; Embedded knowledge resides in routines; and finally, Encoded knowledge is knowledge that is conveyed by signs and symbols (documents and books for example). Technologies that can manage these types of knowledge are best practices databases, discussion forums, and knowledge exchanges, to mention only a few. Finally, the literature on organizational memory - which holds organizational knowledge - identifies 3 numerous benefits for the use of technology in managing organizational memory (Stein and Zwass, 1995). Returning therefore to the question of why KMS fail to deliver effective KM, a second explanation is attributed to insufficient understanding of the organizational and individual requirements from knowledge management systems (Alavi, 2000). This explanation is not new to the IS field considering the 1994 Chaos report2 attributing IT projects' failures to incomplete and unclear requirements. It is also a plausible explanation considering the numerous definitions for KM and for KMS that exist in the literature (Table 1 in the next chapter summarizes these definitions). In this dissertation we therefore adopt this explanation for KMS failure and investigate it further. Research objective The discussion above indicates the lack of understanding of both organizational and individual requirements in the design of existing knowledge management systems. This might be a potential explanation as to why these systems are not as effective as expected. The main objective of this dissertation is to analyze the requirements from KMS and to propose some potential improvements to their design. Note that this dissertation focuses on knowledge management systems - systems intended to support the management of organizational knowledge, and not knowledge based systems (KBS) that embed organizational knowledge in their design and automatically apply it to various tasks in the organization. A main difference between these two types of systems is the focus on human actors. KMS are designed to support people engaged in various knowledge management processes, while KBS provide a more automated approach to support knowledge management and especially knowledge application in organizations. Two levels of analysis are discussed in this dissertation - organizational and individual - in order to answer two specific research questions: (1) what are the main shortfalls of existing KMS? (2) How can we improve the design of these systems? Following this, the dissertation is structured in two main parts - in the first part (chapters 2 and 3) we derive the technical requirements for KMS by examining some organizational theories related to KM. In addition, we identify the main shortfalls in the design of existing systems based on these requirements. In the second part (Chapters 4-6) we focus on one design aspect - namely, the inclusion of meta-2 http://www.standishgroup.com/sample_research/chaos_1994_1 .php 4 knowledge - and identify empirically the individual user's requirements for the specific meta-knowledge that should be included in a KMS. A review of the relevant literature will be included in each part in order to facilitate the discussion of each of the research questions. Chapter 7 concludes this dissertation, and discusses contributions, limitations, and suggestions for future research. Finally, all instruments used in the empirical studies are attached in the appendices. 5 Part I: Identifying the shortfalls of existing Knowledge Management Systems In this part we focus on identifying the organizational-level requirements for KMS and develop an evaluation framework for knowledge management systems that is based on organizational knowledge management requirements. Our main research questions are: (1) What functionalities should be included in a KMS in order to effectively support organizational KM, and how should the systems be designed to best support these functionalities? (2) What is the 'state of the practice' in commercial KMS in terms of providing the functionalities and design that would best support KM? The framework developed to identify KMS functionalities and design is theory-based and is derived by examining organizational processes related to KM. For example, if one goal of organizational KM is to enable the sharing of knowledge between its members then a derived requirement from the KMS is to provide the technical functionalities that would enable this knowledge sharing, such as discussion forums or emails. After developing the framework we apply it to the evaluation of commercial tools purporting to support KM. Before describing the method of developing the framework we define the following terms: • A KM process refers to a specific component, or part, of knowledge management (Alavi and Leidner, 2001). For example - knowledge sharing, knowledge codification, or knowledge application, can all be defined as KM processes. • A functionality refers to a specific capability of a computerized system. For example, word processing systems have functionalities such as 'typing letters', 'formatting documents', 'saving documents', and so on. 'Enabling communications through discussion forums' is an example of a potential functionality of a KMS. • Design features refer to the components (or aspects) of the conceptual design of a computerized system. For example, a 'knowledge repository' is an underlying design feature in some knowledge management systems - providing the ability to store and retrieve knowledge The goal of the framework - developed in chapter 2 - is to create a path leading from the KM process (or processes) that a KMS is intended to support to the functionalities and design 6 features that should be included in the system in order to effectively support this process. For example, if a KMS is designed to support the process of 'knowledge sharing' the framework would identify the functionalities required to support knowledge sharing in the organization, and the design feature(s) that would best provide these functionalities. This path - from process to design - is depicted on the left hand side of Figure 1. The right hand side of Figure 1 shows the method of evaluating commercial KMS, described in chapter 3. In this analysis, we examine whether existing KMS provide good support to the KM processes that they purport to support. To this end, we identify for each KMS the KM process, or processes, it is intended to support. We then examine the functionalities provided by the KMS and its underlying design features and compare them to the required design features identified by our framework in chapter 2. Stage I: Evaluation Framework r o a r Functionality A o o CQ Design feature I KM Processes Functionality B Legend: Implies Functionality C Design feature III Stage II: Evaluating Commercial KMS Functionality A Functionality B Functionality C Design feature (s) Design feature (s) Compare — • Purports to support • Includes Design feature (s) Figure 1: Structure of the evaluation framework 7 Chapter 2: A Framework to Evaluate Knowledge Management Systems We begin this chapter with a review of the various definitions for knowledge management in order to define the scope of the evaluation framework. Table 1 summarizes some of the definitions of KM and demonstrates that generally KM is defined in one of two ways: (1) by its main processes (e.g. knowledge creation or knowledge sharing) or (2) by its goal or expected result (e.g. 'bringing the right knowledge to the right people at the right time'). Based on Table 1, some of the major processes included under knowledge management are knowledge identification or creation; knowledge maintenance; knowledge organization, access and sharing; knowledge use; knowledge reuse; and knowledge adaptation. When focusing on knowledge management systems, three processes remain which are knowledge generation, codification and storage, and sharing (Davenport and Prusak, 1998; Ruggles, 1997)3, and some also add a fourth process - knowledge application (Alavi and Leidner, 2001). In this framework we group knowledge sharing and application into one process termed knowledge transfer. Knowledge transfer is a "process through which one unit (e.g., individual, group, department, division) is affected by the experience of another" (Argote and Ingram, 2000:3). Knowledge transfer differs from knowledge sharing since it holds the requirements that the knowledge be used (cause an effect on the recipient). In this sense - knowledge transfer can be used as a proxy for knowledge use and we therefore group knowledge sharing and use into this one process. The scope of our evaluation framework is thus limited to the three KM processes of generation, codification, and transfer. In addition, we focus on each of these processes separately, identifying the functionalities and design required to support each process regardless of other KM processes. This approach is in line with Argote and Ophir (2002) who note that even though these processes are related they describe different aspects of organizational learning and are predicted by different factors, and therefore should be analyzed separately. In addition, the accepted classification of KMS into knowledge generation systems, knowledge codification systems, and knowledge transfer systems (Ruggles, 1997) also supports the separate analysis since it is important to identify specific functionalities and design features that can individually 3 Argote and Ophir (2002) review existing literature on intra-organizational learning and identify similar processes. They use the terms 'creation', 'retention', and 'transfer' to describe these processes. 8 support each of the three groups of KMS. We turn next to describing the three KM processes in more detail. Author Definition Main KM processes/goals Chait, 1999:24 "Managing knowledge is a multidimensional process. It requires the effective concurrent management of four domains: content, culture, process, and infrastructure." Processes: Manage content; culture; process; and infrastructure Grey (cited on the KM Forum site) "Knowledge management is an audit of "intellectual assets" that highlights unique sources, critical functions and potential bottlenecks which hinder knowledge flows to the point of use. It protects intellectual assets from decay, seeks opportunities to enhance decisions, services and products through adding intelligence, increasing value and providing flexibility." Processes: Auditing knowledge; maintaining relevance and flow Goal: Enhancing decisions; increasing flexibility KPMG, 1998:7 "[Knowledge management is]...a systematic and organized attempt to use knowledge within an organization to transform its ability to store and use knowledge to improve performance." Goal: Use knowledae to improve performance Macintosh, Filby, and Kingston, 1999 "Knowledge assets are the knowledge regarding markets, products, technologies and organizations, that a business owns or needs to own and which enable its business processes to generate profits, add value, etc. ... Knowledge management involves the identification and analysis of available and required knowledge assets and knowledge asset related processes, and the subsequent planning and control of actions to develop both the assets and the processes so as to fulfill organizational objectives." Processes: Identification and analysis of knowledge; planning and control of actions. Malhotra, 1998:59 "KM caters to the critical issues of organizational adaption, survival, and competence in face of increasingly discontinuous environmental change. Essentially, it embodies organizational processes that seek synergistic combination of data and information processing capacity of information technologies, and the creative and innovative capacity of human beings." Goal: Achieving organizational adaption; survival; and competence. Newman, 1991 (KM Forum site) "Knowledge Management is the collection of processes that govern the creation, dissemination, and utilization of knowledge." Processes: Creation; dissemination; and utilization of knowledge O'Dell and Grayson, 1998:6 "Knowledge management is ... a conscious strategy of getting the right knowledge to the right people at the right time and helping people share and put information into action in ways that strive to improve organizational performance." Goal: The right knowledge to the right people at the right time in order to improve organizational performance. The authors also identify these Processes: create; identify; collect; organize; share; adapt; and use. O'Leary, 1999 "Knowledge management is the formal management of knowledge for facilitating creation, access, and reuse of knowledge, typically using advanced technology." Processes: Creation; access: reuse. Wiig, 1994 "In its broadest sense, knowledge management (KM) is a conceptual framework that encompasses all activities and perspectives required to making the organization intelligent-acting on a sustained basis. KM includes activities to gaining overview of, dealing with, and benefiting from the areas that require management attention by identifying salient alternatives, suggesting methods for dealing with them, and conducting activities to achieve desired results." Processes: Create, maintain and develop knowledge and a knowledge environment. Use knowledge efficiently. Goal: Make the organization "intelligent-acting on a sustained basis." Table 1: Definitions of knowledge management 9 The KM Processes Knowledge generation. Knowledge generation is the process of developing new content or replacing existing content in the organization's knowledge (Alavi and Leidner, 2001). We distinguish the process of knowledge generation discussed in this framework from the macro level 'organizational learning' construct (Schulz, 2002), and focus on the individual and group processes that lead to the creation of new knowledge (an approach similar to the one described by Argote and Ophir, 2002). Organizational knowledge can be created or acquired through various organizational learning processes (Stein, 1995; Walsh and Ungson, 1991). Nonaka (1994) presents a theory of organizational knowledge creation that is initiated by individual learning, which then spreads across the organization through various communication mechanisms. The theory builds on interactions between tacit and explicit knowledge. Tacit knowledge is highly personal and hard to formalize, while explicit knowledge is expressed using formal representation and can be communicated easily. Nonaka describes a model of organizational knowledge creation that draws on four patterns of interactions between tacit and explicit knowledge, namely -Socialization (from tacit to tacit), Combination (from explicit to explicit), Externalization (from tacit to explicit), and Internalization (from explicit to tacit). The process begins with a generation of new individual tacit knowledge through hands-on experience. Socialization then follows, involving the construction of a 'field of interaction' whose members share experiences and perspectives. Dialogues between members allow the conceptualization of the tacit knowledge and trigger externalization. Next follows combination of the new knowledge with existing explicit knowledge and finally, the new concepts are articulated through experimentation and internalized. Once this process is completed the new knowledge is evaluated - i.e. tested whether it is worthwhile for the organization - and if proven useful, stored. According to Nonaka's model, individual learning and socialization are two processes that play a key role in the generation of new knowledge. The other processes are mainly channels through which this generated knowledge is communicated and stored across the organization. Delving deeper into the learning process, Argyris and Schon (1978) describe two types of learning in organizations. Single-loop learning occurs when members of the organization respond to internal and external events and modify their actions ('theories-in-use') in order to 10 keep performance in accordance with organizational norms. Double-loop learning involves changing these organizational norms to fit the new knowledge acquired. Here, again, individual learning is followed by an adaptation of existing organizational knowledge (e.g. existing norms) and the creation of new knowledge (new norms). Focusing on the socialization aspect, Argote and Ophir (2002) provide support for the importance of teams in the process of knowledge creation. Knowledge creation can be enhanced by the heterogeneity of group members, by the existence of social networks (Rulke, Zaheer, and Anderson, 2000), or by group brainstorming processes (Paulus and Yang, 2000). The above discussion provides evidence that knowledge generation, or the creation of new content, mainly involves individual learning and socialization that enhances learning and generates new collective knowledge. Knowledge codification. Knowledge codification includes the capture, representation, and storage of knowledge in knowledge bases and the representation of this knowledge in a communicable way (Ruggles, 1997). We distinguish between organizational knowledge and organizational memory, which store knowledge from the past to support present activities (Stein, 1995). Organizational knowledge is often codified and stored in the various retainers of organizational memory. Walsh and Ungson (1991) analyze organizational memory and describe five retainers of memory: Individuals, who retain knowledge in their memory stores or in their belief structures, values, or assumptions; Culture that stores knowledge in language, shared framework, symbols, and stories; Transformations, procedures, and rules which include embedded knowledge such as the logic behind them; Structure and roles that represent the organization's perception of the environment, and social expectations; and finally, the physical settings of the workplace represent knowledge about status hierarchy and behaviour perceptions. Organizational knowledge can also be stored in retainers external to the organization, such as government agencies, market reports, and others. The acquisition of knowledge into the retainers of organizational memory involves the process of learning. This process was described earlier as knowledge generation. However, Stein and Zwass (1995) discuss a more specific acquisition process of knowledge into an organizational memory information system (OMIS), which is more representative of the process of knowledge codification used in this framewrok. According to this process, knowledge codification requires that knowledge be explicated first (externalized from tacit to explicit) and then encoded into one 11 of the individual memory retainers. Moreover, an additional role of the OMIS involves the management of the knowledge stored in it. Knowledge transfer. Knowledge transfer is a "process through which one unit (e.g., individual, group, department, division) is affected by the experience of another" (Argote and Ingram, 2000:3). Knowledge transfer is distinguished from the traditional 'knowledge sharing' concept by the requirement for evidence of results of the transfer (i.e. of the use of the knowledge). Dixon (2000) identifies five types of knowledge transfer in organizational teams: serial transfer occurs when a team applies past knowledge to new tasks; near transfer involves applying a team's knowledge in other (similar) teams; far transfer is similar to near transfer only it involves non routine tasks and tacit knowledge; strategic transfer occurs when a team takes on an infrequent task and seeks to gain from the experiences of other teams that have engaged in a similar task; finally, expert transfer occurs when a team faces a technical problem beyond its knowledge and seeks expert help from others in the organization. The above classification focuses on knowledge transfer at the team level and focuses on identifying the various types of knowledge transfer. To understand the knowledge transfer mechanism at the individual level, we propose the use of Shannon's classic model of a communication system (Shannon and Weaver, 1949) to analyze the process of knowledge transfer. According to this model, a communication system consists of five parts: the source -that produces a message, the transmitter which transforms the message into the signal that can be transferred, the communication channel that serves a the medium for the transfer, the receiver that inverts the operation of the transmitter, and the destination - to whom the message is intended. Based on this view - in the context of organizational knowledge transfer - two specific communication channels exist that bring knowledge to knowledge seekers: (1) directly communicating knowledge through socialization, or more generally through communications between individuals or groups; (2) indirect retrieval of captured knowledge from codified organizational memory. This is depicted in Figure 2. 12 Source Extemalization Encoding Channel Channel 1: Socialization Channel 2: Organizational memory Internalization Destination People Retrieval Figure 2: Channels for knowledge transfer in organizations We explain the two channels in more detail below: Channel 1 - socialization. The socialization channel draws upon the construct of socialization described by Nonaka (1994) but also includes other acts of communicating knowledge between individuals - not limited to the transfer of tacit knowledge as in Nonaka's model. The socialization channel can be described in terms of Brown and Duguid's idea of communities of practice (Brown and Duguid, 1991). Such communities involve members of a close workgroup that share knowledge and experiences in order to overcome the inadequacy of canonical organizational practices in handling practical problems. Socialization is a personal informal communication channel in which knowledge is transmitted in its original form rather than being encoded and captured before transmission. At the initiation point, knowledge should be transferred from tacit to explicit - or externalized (this concept is adopted from Nonaka, 1994). The knowledge can then be communicated to the receiver of knowledge who internalizes it. Channel 2 - codified organizational memory. The second communication channel in the organization is the codified organizational memory (Stein, 1995) through which knowledge can be transferred from the source to the receiver. The organizational memory channel is more formal than the socialization channel and requires some additional transmitting mechanisms for the knowledge. At the in itiation point of this communication channel knowledge is transmitted from the source and encoded into a formal knowledge base - i.e. retainers of organizational memory. Note that if the knowledge originates from a person then it may first have to be externalized and only then encoded into organizational memory. The knowledge is stored in the retainers of organizational memory (that is - in the organizational memory channel) until it is requested. Once requested, knowledge is retrieved and provided to the receiver who internalizes it. 13 Figure 2 indicates that knowledge transfer involves the externalization of tacit knowledge held by individuals and similarly - the retrieval of codified knowledge held in organizational memory. It also involves the encoding of knowledge into organizational memory or - similarly - the internalization of knowledge by individuals. Finally, knowledge transfer also involves the socialization channel, which enables the direct transfer of knowledge between people. To summarize the discussion so far, the three processes examined by this framework are knowledge generation, codification, and transfer. After examining some underlying theories related to these processes we identify specific requirements for the execution of each process such as the externalization of tacit knowledge, the encoding of knowledge into organizational memory, or the transfer of knowledge between individuals through the socialization channel. Next we examine what functionalities would be required from a KMS intended to support these KM processes. Deriving KMS functionalities As mentioned in the introduction to this chapter, functionalities refer to specific capabilities of a KMS, for example - 'enabling communications through discussion forums'. In this section we derive required KMS functionalities based on the three processes that the KMS are intended to support. Our analysis of these processes indicates the need for four general groups of functionalities shown at the bottom row in Figure 3. These are 'individual learning functionalities', 'socialization functionalities', 'externalization and retrieval functionalities', and 'internalization and storage functionalities'. Each of these groups can include one or more functionalities that would enable the KMS to support some of the KM processes as indicated by the arrows in Figure 3. Individual learning 'Individual learning' functionalities enable people to develop their knowledge by providing a learning environment. Specific information systems that facilitate individual learning can be e-learning systems or business intelligence systems that enable data mining and online processing of data. Since the current scope of KMS is not intended to directly support individual learning (learning systems and business intelligence systems are generally not included in the group of KMS (e.g. Ruggles, 1997)) we exclude this group of functionalities from the following analysis. This is the reason why 'individual learning' is depicted in Figure 3 using the dashed line. 14 Generation Codification Transfer Individual learning Socialization Externalization / retrieval Internalization and storage - • 'Supports' KM process KMS functionality Figure 3: KM processes and supporting functionalities Socialization Socialization functionalities are functionalities that enable people to exchange knowledge with their peers, brainstorm on specific topics, or similar activities. Socialization functionalities are important for knowledge generation by enabling the brainstorming and discussions of problems and new knowledge between individuals, leading to the generation of organizational knowledge. They also support the transfer of knowledge through conversations and other communications. We identify two specific socialization-related KMS functionalities, annotated S01 and S02, that support knowledge generation and knowledge transfer respectively: 501 - provide the abilities for individuals to interact, brainstorm, and create new knowledge. 502 - provide a medium for sharing and disseminating knowledge between individuals (for example, by email) Differences between the two functionalities can be viewed in terms of the synchronicity of the medium - the knowledge generation functionality depending more on synchronous medium while transfer can be well support by asynchronous medium - or 15 in terms of the media richness (Daft and Lengel, 1986; Dennis and Kinney, 1998), the first functionality requires richer media to support knowledge generation while the second impose (but is not restricted to) lower demands on the type of medium used -limiting the requirements from the media to providing the ability to transfer knowledge. Externalization / Retrieval The externalization and retrieval functionalities are grouped together since they both support the retrieval of knowledge, either tacit knowledge that resides only in people's mind, or explicit knowledge from organizational memory. Externalization involves the transformation of knowledge from tacit to explicit. This transformation is important for knowledge transfer and for knowledge codification, which requires transferring tacit knowledge into explicit forms. The KMS should provide a common language for communications thus enhancing the exchange of knowledge between groups - such as different communities of practice - and enabling the externalization of the knowledge. We therefore identify the following externalization-related KMS functionality that supports both the codification and transfer of knowledge: ER1 - support the externalization of knowledge: transforming knowledge from tacit to explicit (e.g. explaining best practices) The retrieval of knowledge is a wider concept than externalization. While externalization focuses on retrieving tacit knowledge from individuals, the general retrieval process is not limited to a specific knowledge retainer. Retrieval of knowledge is important for the transfer of knowledge encoded in organizational memory. We identify the following retrieval-related KMS functionality that supports knowledge transfer: ER2 -facilitate search and retrieval of knowledge from organizational memory (e.g. search engines) Internalization / Storage Similar to the previous group of functionalities, internalization and storage are grouped together since both support the encoding of knowledge, either encoding tacit knowledge into people's minds, or explicit knowledge into some formal retainers of organizational memory. An important goal of KM is the retention, or preservation, of knowledge in the organization and thus the KMS should be able to maintain and update knowledge in the organizational memory by providing 16 functionalities that support the knowledge codification process. We identify the following two storage-related KMS functionalities intended to support knowledge codification: ST1 - enable the management (location, organization, classification, etc.) of stored knowledge in organizational memory retainers (e.g. content management systems) ST2 - provide the ability to capture and encode knowledge into retainers of organizational memory. Internalization is the transformation of knowledge from explicit to tacit that is generally a personal process occurring within peoples' minds. To some extent we can identify KMS functionalities that might support internalization by enhancing individual's absorptive capacity, the ability of individuals to recognize the value of new information, assimilate it, and apply it. This ability largely depends on the individual's prior knowledge (Cohen and Levinthal, 1990; Szulanski, 1996). An internalization functionality can therefore be the ability of the KMS to provide some additional information about the knowledge in memory in order to enhance the receiver's absorptive capacity. This functionality can be tied to the storage functionality ST1 since the management of organizational memory would imply capturing and representing information about the knowledge stored in memory4. In support of this last argument, Argote and Ophir (2002) show that meta-memories (memories about memories) that provide the ability to manage collective knowledge stored in groups' memories play an important role in supporting the ability of individuals to retain organizational knowledge. This process of embedding knowledge in organization members is similar to the process of internalization discussed here and thus we conclude that the storage functionality ST1 can also play a role in supporting the internalization of knowledge and through it - the organizational memory channel of knowledge transfer. By now we have identified the main functionalities that are required from knowledge management systems in order to support the three KM processes. These functionalities are defined in terms of the process they support and therefore, KMS that are only intended to support one of the processes need only to provide the functionalities required for this specific process. For example, a system intended to support knowledge codification should include functionality ER1 - the ability to externalize tacit knowledge - and functionality ST2 - the ability 4 This functionality is similar to a database management functionality that involves the collection and representation of information (meta-data) about the data stored in the database. 17 to capture explicit knowledge and encode it into the system. The support of functionalities in the KM processes is summarized in Table 2 (the names in brackets are just mnemonics for the various functionalities). Generation Codification Transfer Socialization S01 (interactions) S02 (knowledge sharing) Externalization / Retrieval ER1 (externalization) ER1 (externalization); ER2 (retrieval) Internalization / Storage ST1 (storage management) ST2 (encoding) ST1 (storage management)1 _ 1 1 1 1 This support is limited to the above discussion of the ability of a KMS to support internalization Table 2: Summary of functionality support in KM processes We now turn to discuss specific design features that can enable KMS to provide these functionalities. Implied design features of KMS In the previous section we have identified some functionalities required by a KMS to support the three KM processes. The question remains as to what are the best design features to provide these functionalities. In this section we propose three specific design features (architectures), namely - a network model, an organizational ontology, and meta-knowledge, to support the six functionalities described above (S01, S02, ER1, ER2, ST1, ST2). The network model of KMS. Alavi (2000) identifies two underlying designs for KMS: repositories and networks. Knowledge repositories are bases of explicit knowledge that exist in various forms such as the company's Intranet or "best practices" documentation. KMS that are based on the network model are intended to create links and interactions between individuals to facilitate the sharing of knowledge. Examples for such networks are knowledge maps that point to the source of the knowledge (Alavi, 2000). While the repository model is essential for the creation and maintenance of organizational memory, and is thus useful for the design of specific organizational memory retainers, we argue that the network design is more suitable to serve as the basis for a general KMS intended to support the main KM processes of generation, codification, and transfer. The main reason lies in the ability of the network design to provide the socialization functionalities that are important to both the generation and transfer of knowledge. In addition, this design provides for managing 18 both tacit and explicit knowledge. The repository model, while supporting simplicity and efficiency of attaining information, does not support socialization, and hence might inhibit knowledge creation. Organizational Ontology. A growing trend in the design of knowledge systems is the use of ontologies. An ontology can be defined as a specification of objects, concepts, and other entities that exist in a domain, and the relationships among them (Gruber, 1995). The use of ontology for KM is not widely acknowledged in the organizational KM literature. However, O'Leary (1999) discusses the importance of organizational ontology in KM and its ability to define the scope of discussion groups, provide strong search capabilities, support filtering of information (by defining the concepts to filter), facilitate efficient archiving of documents, and provide a common language for communications. An ontologically based KMS can provide necessary tools for representing knowledge and for externalizing tacit knowledge. In addition, using ontology can support socialization by providing a language for communications across the organization. For example, the ability of the socialization channel of communications to convey knowledge in a clear way depends on language. Within small enough groups - such as communities of practice - a shared language exists (Brown and Duguid, 1998). However, as the groups grow larger the language may differ between different sub-groups. An organizational ontology thus mainly supports the externalization and retrieval of knowledge by providing the linguistic tools needed to support externalization and by maintaining the context of knowledge stored in memory. Ontology also supports - to a lesser extent - the socialization functionalities by facilitating communications between members of different parts of the organization. Meta-knowledge. Meta-knowledge is knowledge about the knowledge. In this work we distinguish between ontology, which specifies what the knowledge is about, and meta-knowledge, which provides information about specific knowledge "chunks". Examples for meta-knowledge can be the context of the knowledge, the characteristics of the source knowledge, or the currency of the knowledge to mention only a few. The motivation for using meta-knowledge in a KMS is derived from the traditional design of IS. One can view organizational memory as an inventory of 19 organizational knowledge. Thus, a KMS should have aspects similar to any information system intended to manage other assets - e.g. physical inventory or regular business data. Such systems provide relevant information about the items managed (for example quantities, date of purchase, and cost for inventory items). The information needs for these systems are derived from the nature of the managed domain and from users requirements. Similarly, a KMS should maintain and provide information about the knowledge, namely, meta-knowledge. The fact that knowledge is often tacit (and hence not even "storable"), and might be dispersed in a multitude of forms, makes the availability of meta-knowledge particularly critical. For example, meta-knowledge can support knowledge maps pointing to retainers of knowledge. It is not completely clear from the literature what meta-knowledge should be provided about organizational knowledge. Some work on meta-knowledge defines it as "structure that describes other structures" (Plant and Gamble, 1997). Kalfoglou et al. (2000) include in meta-knowledge ontologies, problem-solving methods, and other models. Meta-knowledge is also related to meta-cognition (thinking about thinking), more specifically to meta-cognitive knowledge. This aspect of meta-cognition includes knowledge of person-related variables, task variables, and strategy variables (Flavell, 1979). In addition, meta-knowledge can also be tied to the concept of meta-memory (Wegner, 1987). Meta-memory supports the development of collective memory within small groups and provides group members with information about the source and location of knowledge. Finally, meta-knowledge can be tied to the concept of meta-data, which is used in many systems to describe the data stored in them. Similar to the role of meta-data in organizational systems or meta-memory in small groups, meta-knowledge mainly supports the management of organizational memory, which is part of the internalization and storage functionalities. It also enhances the network model in supporting socialization by providing some information about the knowledge and the knowledge sources that exist in the organization. Finally, meta-knowledge can also provide additional contextual information that would support the role of ontology in retrieving knowledge from memory. For example, meta-knowledge can provided background information about the source that would help focus the search results on desired sources. These three design features - network model, organizational ontology, and meta-knowledge -are described at the bottom of Figure 4. Figure 4 also ties the design features to the specific functionalities that they support, and through them, to the KM process supported by each design feature. 20 Processes Generation Codification Transfer Functionalities Socialization Externalization (ER1) Internalization and (S01; S02) / Retrieval (ER2) Storage (ST1; ST2) l • i A Design feature^/ N etwork Design Ontology Meta-knowledge 'Main support' ~~) KM Process 'Secondary support' KMS Functionality y / KMS Design feature Figure 4: Required design features of KMS The next section summarizes our evaluation framework by tying the proposed design features to the three KM processes to demonstrate how a KMS that is based on these design features can support the desired KM processes. Mapping design features to KM processes Above we proposed three design features for KMS by analyzing the functionalities derived from three organizational KM processes, namely knowledge generation, codification, and transfer. In Table 3 we directly tie design features to specific organizational KM processes using the KMS functionalities identified. The table describes the specific functionalities by which each design feature supports each of the KM processes. For example, if a system is intended to support knowledge transfer (Column 4) it should provide a socialization functionality that facilitates interactions in which knowledge is shared (S02). The best design feature to support these interactions is the network design that links knowledge sources to knowledge seekers. Therefore, the network design provides support for the knowledge transfer process. The specific algebra of Table 3 includes direct effects, e.g. the Network model supports the generation of knowledge through its support of socialization, in particular through interactions 21 (501) . Indirect effects are also possible. For example, the Ontology aspect also supports knowledge generation though interactions (S01) but it does so indirectly through its support of externalization of tacit knowledge (ER1). Such indirect effects are represented using brackets (S01[ER1]). More complex algebra is also enabled using this framework. For example, the meta-knowledge model supports transfer directly by supporting the internalization of knowledge by individuals (ST1). However, the meta-knowledge aspect also supports the transfer of knowledge between individuals by supporting the location of this knowledge in organizational memory (ST1), its retrieval from memory (ER2), and then its sharing through conversations (502) . This is represented in the following way: S02[ST1*ER2] (the '*' is equivalent to the logical 'and'). Finally, knowledge processes can be supported by two different functionalities. For example, the effect of ontology on transfer is indirect through its support of the transformation of tacit to explicit knowledge (ER1), which then supports the communication of this knowledge (S02[ER1]). The ontology feature also supports the retrieval of knowledge from organizational memory (ER2), which also supports the transfer of knowledge between memory retainers. Therefore, we represent the support of ontology in knowledge transfer as S02[ER1], ER2 (the ',' is equivalent to the logical 'or'). KM Process Design feature^~~~~~~~^ Generation Codification Transfer Network Model S01 S 0 2 Ontology Based S01[ER1] ER1 S02[ER1], ER2 Meta-knowledge model S01[ST1] ST2 ST1, S02[ST1*ER2] S01 - direct effect; [S01] - indirect effect; S01*S02 - combined effect; S01, S02 - two separate effects SO (socialization). Specific functionalities are: 501 - provide the abilities for individuals to interact, brainstorm, and create new knowledge. 502 - provide a medium for sharing and disseminating knowledge between individuals. ER (Externalization / retrieval). Specific functionalities are: ER1 - support the externalization of knowledge: transforming knowledge from tacit to explicit. ER2 -facilitate search and retrieval of knowledge from organizational memory. ST (storage). Specific functionalities are: ST1 - enable the management (e.g. location, organization, classification, etc.) of stored knowledge in organizational memory retainers ST2 - provide the ability to capture and encode knowledge into retainers of organizational memory. Table 3: How design features of KMS support KM processes Using the links described in Table 3 we can fully analyze how each of the KM processes is supported - directly or indirectly - by the proposed design features. For example, knowledge 22 transfer is supported by four specific functionalities at various levels of support: the main functionality that supports knowledge transfer is a socialization functionality that enables the discussions in which knowledge is shared (S02). Knowledge transfer is also indirectly supported by an externalization functionality that enables the transfer of tacit knowledge (ER1) and a retrieval functionality that extracts knowledge from organizational memory retainers (ER2). This latter functionality is only required if the transfer is not between two people. Finally, a storage functionality - the management of organizational memory retainers (ST1) - that enables the internalization of knowledge is also important for the success of knowledge transfer. All the three design features - namely the network model, the organizational ontology, and meta-knowledge - are necessary to provide these functionalities. Each design feature provides different functionalities: interactions (S02) are supported by the network design that links knowledge sources to knowledge seekers. Externalization (ER1) and retrieval (ER2) are supported by the shared language provided by the ontology. Memory management (ST1) is supported by the existence of meta-knowledge that directs users to the right knowledge in the organizational memory and provides them with the information needed in order to decide whether to use or not use the knowledge. Therefore - the design of a knowledge transfer system depends on the number of functionalities it provides. At the minimum, such system should provide the knowledge sharing functionality (S02) and therefore be based on the network model of KMS. So far we have developed a theoretical approach to evaluate KMS. To provide a more prescriptive approach to the design of KMS, based on the theoretical framework, we applied the information in Table 3 - that is the identification of the best design features to support each KM process - to evaluate KM tools based on the match between their actual design features and the design implied by the KM processes they are intended to support. For example, we expect that tools intended to support knowledge codification will include an organizational ontology as well as meta-knowledge (see column 2 in Table 3). The next chapter describes our analysis of commercial tools that support KM. 23 Chapter 3: An evaluation of existing KM tools The goal of the evaluation of commercial KMS is twofold - first, it is a test of the applicability of the proposed framework to the evaluation of KM tools (KMS). Second, we aim to evaluate the awareness of KMS designers to the design requirements identified in our analysis. Specifically, we would like to evaluate whether commercial KM products whose description indicates they are intended to support certain KM processes, indeed possess the required functionalities and design features to be effective. For example - if a tool were intended to support knowledge generation, we would expect this tool to primarily possess functionalities providing for socialization (e.g. the interactions, or brainstorming, functionality S01). This is based on our framework (especially Table 2) that indicates that knowledge generation is best supported by the socialization functionalities. Moreover, according to Table 3, such tool would have to be based on the network model to provide this functionality directly, and can include ontology and meta-knowledge to enhance its support in knowledge generation. We applied our framework to the evaluation of 40 tools from a list of the top 100 KMS providers available on the 'KM World' Web site; we excluded the business intelligence tools from the scope of this analysis5. The list of tools analyzed is presented in Appendix A. To apply the framework to the analysis of specific tools we must first classify the 40 tools into knowledge generation tools, knowledge codification tools, and knowledge transfer tools. However, a review of commercial classifications of KMS reveals that tools are not usually categorized according to these three categories identified by Ruggles (1997). For example, a GartnerGroup report (2000) identified seven types of KM products: repositories for storage and management of explicit knowledge; information access products providing search and retrieval capabilities; user interface tools; skill recording and expertise location systems; collaboration products; decision support tools; and application development platforms and tools. Similarly, a KPMG report on KM (2000) that examined common KMS used in organizations supports these types and identifies groups of tools such as Intranets; document management systems; decision support systems; groupware; artificial intelligence; and data warehousing / mining. A more recent classification of commercial tools from GartnerGroup (2002b) is presented in Table 4. The general functionalities that can be expected from tools in each group, based on the 5 This is based on the fact that we excluded individual learning from our analysis at an earlier stage. Business intelligence tools are significantly different than KMS and would require a separate evaluation framework. 24 GartnerGroup report, are presented in the second column and the last column represents a mapping of these KMS categories to the KM processes examined in our framework. For example, content management tools are regarded as supporting the knowledge codification process since they provide the functionalities to store and manage content in organizational memory. Similarly, collaboration tools are considered as supporting the knowledge transfer process. As the table shows the mapping from this market classification to the KM processes is not a one to one mapping. For example, community technologies are tools that provide the ability to create and manage communities of individuals. While this can be matched to the knowledge transfer process these tools might also provide more advanced functionalities - such as capturing and managing the community's knowledge - which can be perceived as supporting the knowledge codification process, or some brainstorming capabilities that would support knowledge generation. Category Functionalities KM Process supported Content Management Manage unstructured data such as documents, e-and mail messages. Some Web content management functionalities may also be included. Knowledge codification (organizational memory management) * Most tools also include a knowledge retrieval functionality Collaboration Asynchronous collaboration (e.g. threaded discussions, e-mail); Synchronous collaboration (e.g. chat and instant messaging) Knowledge transfer / knowledge generation Information Organization and Retrieval Techniques and technologies that include automated or manual taxonomy creation, document categorization and indexing, and various levels of search capabilities. Knowledge codification / knowledge transfer Expertise Location and Management Captures content that experts produce, dynamically profiles users' skills and connects them to each other. Knowledge transfer Community Technology Technologies that create communities, enable interactions, manage community content, archive knowledge, etc. Knowledge transfer. / knowledge generation / knowledge codification (depending on the extent of each of the functionalities offered by the specific tool) Ad Hoc Process Support Support the management and sharing of knowledge encapsulated in business processes. Not included in this analysis (automated processes, embedded knowledge) Portal Framework User interface tools that allow a single view of information sources and applications. Cross-repository search is a key enabler. Knowledge transfer (from repositories to individuals) / Knowledge codification (enable the management of knowledge from a single interface) Table 4: Classification of commercial KMS Based on the information provided in Table 4 we categorized the tools analyzed into four groups. The groups of tools studied are: 25 • Content management tools - systems offering abilities to integrate, sort, classify, and codify knowledge from various sources (for example: systems that enable the integration of knowledge from different sources and some advanced document management systems). Based on the information in Table 2, to effectively support the knowledge codification process a tool must provide the externalization functionality as well as the two storage functionalities. We therefore expect tools in this category to provide these functionalities. • Knowledge sharing systems - this group includes three categories described in the GartnerGroup classification - collaboration tools, expertise locators, and community technologies. These systems are intended to support the sharing of knowledge between people or other agents by supporting the socialization and the externalization functionalities and through them - knowledge transfer, and to some extent - knowledge generation. Since these systems are focused around human actors we do not expect them to have the retrieval functionality identified as necessary for the knowledge transfer process in Table 2 (recall that this functionality was identified as important for knowledge transfer through the organizational memory transfer). In addition, based on the definitions of these tools (column 2 in Table 4) they are focused on knowledge sharing rather than transfer and therefore we also do not expect to find the internalization functionality. • Knowledge search and retrieval systems - this group included the information retrieval tools as well as the portals described in Table 4. Such systems mainly enable knowledge search and retrieval from organizational memory (for example search engines) thus supporting the retrieval functionality and through it - the knowledge transfer process. • General KMS - as mentioned earlier, community technologies might support more than one KM process, the best example of this can be Lotus Discovery server that developed from the Notes groupware into a general KM solution for organizations. Such systems are therefore intended to provide an overall solution for a company's KM needs and purport to support all three KM processes. The assignment of a specific tool into a specific group is based on claims made by vendors on the tool's website (see examples for this in Appendix A). Two judges classified the tools into the different categories and agreement was good (Kappa = 0.83). 26 Analysis and Evaluation To evaluate commercial tools based on. the framework developed we used two evaluation criteria. First, we examined the prevalent design features and functionalities provided within each category of tools. That is, the number of tools within each category that included a specific design feature or that purported to provide a specific functionality (e.g., how many tools within the knowledge-sharing category included the network design; how many tools within this category purported to support the externalization functionality, ER1). Second, we examined the effective support of the system in the intended functionalities. We define effective support as supporting a specific functionality with the most suitable design. For example, the network model effectively supports socialization, to support internalization and storage effectively the tool would have to provide meta-knowledge. The identification of the best design features for each functionality was explained in chapter 2 (Figure 4) and is described again below: • Socialization (S01, S02) is best supported by the network design • Externalization / Retrieval (ER1, ER2) are best supported by the use of an ontology • Internalization / Storage (ST1, ST2) are best supported by meta-knowledge A summary of the design features and the functionalities that we expect to find in tools from each category of tools - based on the KM processes they intend to support - is presented in Table 5a. The evaluation of the tools was conducted in three steps: first we identified the category that each tool belonged to. At the second stage we identified the specific KM functionalities the tool aims to support based on what was reported on the vendor's website. For example, if the vendor promises to improve users' ability to locate and retrieve knowledge then we note that the tool purports to support retrieval from organizational memory (functionality ER2). An example for this is presented in Appendix A. In addition, in cases where a demonstration of the tool was available we used it to evaluate the tool. Finally we identified the specific design features - from the three design features we derived in Figure 4 - that are included in each tool. Our findings are described in Table 5b. Table 5b shows the main design features we identified for the tools in each category (column 2) and the main functionalities the tools in this category aim to provide (column 3). The numbers represent the percent of tools in the group that provided each functionality and each design feature. For example, nine of the 10 content management tools examined included some meta-27 knowledge; 10 out of 10 content management tools examined aimed to support the storage management functionality (ST1), however - only four of the 10 tools aimed to support the second storage functionality - the capturing and encoding of knowledge (ST2). Table 5: Evaluation of KMS Category (# of tools) Design Features Functionalities Network Ontology Meta-Knowledge S01 S02 ER1 ER2 ST1 ST2 Content Management (10) X X X X Knowledge Sharing (13) X X X X Knowledge Search & Retrieval (12) X X General KMS (5) X X X X X X X X X 501 - provide the abilities for individuals to interact, brainstorm, and create new knowledge. 502 - provide a medium for sharing and disseminating knowledge between individuals. ER1 - support the externalization of knowledge: transforming knowledge from tacit to explicit. ER2 -facilitate search and retrieval of knowledge from organizational memory. ST1 - enable the management (e.g. location, organization, classification, etc.) of stored knowledge in organizational memory retainers ST2 - provide the ability to capture and encode knowledge into retainers of organizational memory. Table 5a: Functionalities and design features required from KMS' categories evaluated Category (# of tools) Design Feature Functionalities Network Ontology Meta-Knowledge SOI S02 ER1 ER2 ST1 ST2 Content Management (10) 0 40% 90% 0 20% 30% 100% 100% 40% Knowledge Sharing (13) 85% 15% 54% 54% 92% 15% 31% 38% 62% Knowledge Search & Retrieval (12) 33% 67% 50% 0 33% 42% 100% 33% 25% General KMS (5) 60% 40% 100% 80% 100% 0 100% 80% 20% Table 5b: Percent of tools supporting a given design feature or a functionality As noted, the evaluation of the functionalities promised by the tools in each category is intended to examine the awareness of KMS designers to functionalities required to support the KM processes that the tools aim to support. Based on Table 5a (and our evaluation framework described in chapter 2), content management tools are required to provide the two internalization and storage functionalities (ST1, ST2) as well as the externalization functionality (ER1), in order to fully support the knowledge codification process that they intend to support. Table 5b shows that within the content management category, all the tools aimed to provide the functionality to manage organizational memory (ST1) as well as the retrieval of knowledge from 28 memory (ER2). However, only 40% of the tools aimed to provide functionality ST2 - encoding knowledge into memory, and only 30% aimed to provide the externalization functionality (ER1). Knowledge sharing systems were expected to provide the two socialization functionalities (the interactions functionality S01, and the knowledge sharing functionality S02). These tools mainly provided the ability to share knowledge between individuals (S02) and only to some extent the generation of new knowledge (S01). We also expected knowledge-sharing tools to provide the functionality supporting the transformation of tacit knowledge to explicit (ER1). This was only supported by 15% of the tools examined. Finally, 62% of the knowledge sharing tools also purported to enable the capturing and encoding of knowledge into organizational memory (ST2), mostly by capturing discussion messages and FAQ. Knowledge search and retrieval tools were expected to provide only the retrieval functionality ER2. The tools examined showed good support in this functionality (100%). Finally, general KMS were expected to provide all six functionalities but in reality focused on four of the six. General KMS purported to provide the two socialization functionalities (S01, S02), and the management and retrieval of stored knowledge storage (ST1, ER2). Only 20% of the tools claimed to provide the capturing and encoding functionality (ST2) and no tool intended to provide the externalization of knowledge from tacit to explicit (ER1). The main shortfall of commercial KMS apparent from the analysis above is that there is very little support to the externalization of tacit knowledge (ER1) and to the capturing and encoding of knowledge in memory retainers (ST2). Figure 5 presents the model of knowledge transfer channels discussed earlier (Figure 2 on page 13). In Figure 5 functionalities ER1 and ST2 are reflected in the left hand side (externalization and encoding), which describes how knowledge is transferred from the source into the communication channel. Our analysis shows that there is a gap in technology support for this part of the communication system and most existing technologies focus on supporting the right hand side of Figure 5 - the retrieval and delivery of knowledge from the communication channel to the destination of the knowledge. We will discuss the implications of this in the next section. The second part of this analysis (Column 2 in Table 5b) examines the design of the KMS. As can be seen from Table 5b, the most prominent design feature of content management tools is the meta-knowledge support. The Network design was mostly available in the knowledge sharing tools and linguistic and semantic tools were used in the design of knowledge search and retrieval systems, but not in all of them (67%). All five general KMS evaluated included 29 some meta-knowledge, 60% of them supported a knowledge network, but only 40% had some linguistic features (such as ontology). Source r • Channel • Channel 1: Socialization Externalization 1 ^ Internalization • w | i i • r Channel 2: Organizational Encoding 1 w Retrieval ) w W w J memory Destination - - Lack of support of existing KMS Figure 5: Shortfall of existing KMS to support knowledge transfer Figure 6 presents a different summary of the data. Each cell in the data table of Figure 6 represents the number of tools in a category of tools that effectively support its intended functionalities, according to our evaluation framework. As described earlier, effective support refers to providing the most suitable design to support a specific functionality or functionalities (e.g. meta-knowledge effectively supports internalization and management of stored knowledge). We constructed the data in Figure 6 in two steps: first we identified the number of tools in each category of tools that purports to support a specific group of functionalities (e.g. the socialization functionality). This number is the denominator in each cell. For example, if two content management tools purported to provide the socialization functionalities, namely S01 or S02, then the denominator for the cell that crosses content management and socialization is two. The numerator in each cell is the number of tools that effectively supported this group of functionalities, that is - the number of tools that were based on the most suitable design for these functionalities. For example, the most suitable design feature for socialization is network design6. No content management tools possessed this design and hence, no content management tools effectively support socialization. The numbers shown represent the percent of effective support of each category for each KM functionality. The links between design features and KMS functionalities were described in Figure 4. 30 100% 50% Content \f\fanagem ent (10) t4jon4edge Sharing (1$ tdonledge Search & Retrieval \General KMS (5) • Socialization 0% 85% 50% 60% H Externalization 40% 50% 67% 10% I internalization 90% 70% 75% 100% Figure 6: Percent of tools that effectively support KMS functionalities Figure 6 shows that some tools are designed in a way that effectively supports KMS functionalities and therefore the KM processes (the high numbers in the data table). To examine each category in more detail we summarize below - based on the empirical findings summarized in Table 5b - the functionalities that most tools intended to support (in reality not as expected based on the framework) within each category of tools: • Content management tools aimed to support storage (ST1) and retrieval (ER2) of knowledge. • Knowledge sharing tools aimed to support socialization (S02) and to some extent - the storage functionality (ST2). • Knowledge search and retrieval aimed to support retrieval (ER2). • General KMS aimed to support the socialization, retrieval, and storage (S01, S02, ER2, ST1). Figure 6 shows that strong scores were attained for content management tools and knowledge sharing tools for the categories' main supported functionalities. 90% of the content management tools effectively supported the internalization and storage functionalities. 85% of the knowledge sharing tools effectively supported socialization. However, only 67% of the search and retrieval tools effectively supported externalization and retrieval, and general KMS only performed well in 31 effectively supporting the socialization and internalization functionalities. We discuss the implications of these findings in more depth in the next section. Discussion The goal of the evaluation framework was to identify the main shortfalls in the design of knowledge management systems based on the organizational requirements from these systems. Specifically this framework was motivated by observations made in the literature about KMS failures to effectively support KM in organizations. We derived evaluation criteria for KMS based on theoretical considerations and applied them to existing KM tools. The findings show that many KM tools are geared towards supporting organizational needs and designed in a way that will best support these needs. However, some problems remain: Knowledge externalization and encoding. A main finding of our analysis is that there is a gap in technology support for the organizational communication channels. Specifically we find that there are very few technologies that effectively support or purport to support the externalization of tacit knowledge and the encoding of this knowledge into organizational memory retainers. This gap may be one reason why KMS are not effective and may provide some explanation to the findings stated in the KPMG report (KPMG, 2000) indicating that 45% of the companies that implemented KM programs were still experiencing the problem of reinventing the wheel, and 50% of these companies were having difficulties in sharing tacit knowledge. Socialization. Of the tools evaluated, most are intended to support socialization in the sense of the exchange of knowledge between individuals (functionality S02) rather than to enhance conversation and creation of knowledge (functionality S01). To this end, many of the tools direct people to knowledge held by other people but provide limited communication tools (such as email) that may not support direct interactions and discussions on a deeper level. This results in the transfer of knowledge between people and the preservation of that knowledge in organizational memory retainers, but only partially supports externalization of tacit knowledge and might not support the creation of new knowledge. Based on the media richness theory discussed earlier (on page 15, Daft and Lengel, 1986) we posit that a more effective socialization tool should provide richer media to support knowledge generation. Organizational Ontology. Many knowledge search and retrieval systems are based on some linguistic tools that enable the classification of new knowledge and the retrieval of relevant knowledge from organizational memory. However, it was surprising to see that most of the tools did not support an organizational ontology, namely, a list of concepts (and their relationships) 32 relevant to the organization. It appears that while the benefits of using an organizational ontology are recognized in the literature (O'Leary, 1999) its use in practical tools is not as common. Perhaps this is due to the difficulties involved in constructing and maintaining organization-specific ontologies. Nevertheless, the use of an organizational ontology is important for knowledge management since it provides a relevant shared set of concepts that can facilitate externalization and transfer of knowledge. Meta-knowledge. Our last observation refers to meta-knowledge. Most of the tools (especially the content management tools) include some meta-knowledge but the actual content of meta-knowledge varies substantially between these tools. Some examples for components of meta-knowledge used by different tools are the identification of the knowledge source; the structure of the data; and classifications based on taxonomies and related terms. This lack of a unified approach to meta-knowledge may underlie the inability of existing systems to prevent information overload and to deliver 'the right knowledge to the right person and in the right format'. To alleviate this problem we propose that further investigation into the nature of meta-knowledge is necessary to identify the most important pieces of meta-knowledge to KMS users. Finally, our findings show that while KMS embed some understanding of the organizational KM requirements, organizations may need to adopt several different tools in order to attain a complete KM solution. However, since these tools have different objectives, and might come from different vendors, it might not be easy to create an integrated KM environment. Hence, a more comprehensive KM solution is still needed to provide all the required functionalities of a KMS. Our discussion in Part I has outlined the main shortfalls of existing KMS, thus answering our first research question. As the goal of this work is designing more effective knowledge management systems our second research question concerned potential improvements to the design of existing KMS. The second part of this dissertation will focus on proposing ways to improve one of the problems identified by our analysis. Specifically - we focus on the meta-knowledge aspect, identifying which meta-knowledge should be included in KMS to encourage more effective management and use of knowledge. Part 2 of the dissertation therefore discusses meta-knowledge and the motivation for studying it in the context of KMS, as well as our empirical studies focused at identifying the meta-knowledge requirements of potential users of KMS. 33 Part II: Exploring meta-knowledge requirements for Knowledge Management Systems In this part we focus on the second research question identified in Chapter 1: How can we improve the design of knowledge management systems? Specifically, we focus on one design feature - namely meta-knowledge - and examine empirically what meta-knowledge should be included in a KMS. Mefa-knowledge literally means knowledge about the knowledge, for example knowing the context of the knowledge (e.g. the underlying ontology or taxonomy). However, in our context a more general interpretation also includes information about the knowledge, for instance - the subject of the knowledge, the identity of the author, or the date in which the knowledge was created. Meta-knowledge is thus a broad name for a group of specific pieces of information about knowledge or about the knowledge source, which we collectively term attributes7 (either knowledge attributes or source attributes). As our previous analysis (in Chapter 2) indicates, meta-knowledge has two main roles in the design of KMS: (1) to enhance the management of organizational memory in order to facilitate knowledge storage, location, and retrieval by users of the KMS, and (2) to support the internalization of knowledge by these users. Following this, our objective is to understand what specific meta-knowledge - or knowledge and source attributes - should be included in the design of KMS in order to facilitate these two roles. Thus, the next chapters focus on the individual level of analysis and empirically identify which knowledge and source attributes should be included in a KMS in order to increase its effectiveness in organizational memory management and knowledge adoption (internalization) by individuals. The analysis begins (Chapter 4) with a review of the literature on meta-knowledge and a more detailed discussion of the two roles of meta-knowledge in KMS design. Chapter 5 discusses the methods used in our empirical studies, namely the Delphi method and conjoint analysis, and Chapter 6 brings the findings of these studies and a discussion of both the theoretical and practical contributions made. 7 The use of the term 'attribute' here follows the conventional meaning used in ontology, conceptual modeling, and information systems modeling. For example in Bunge's ontology (Bunge, 1977) attributes are used to model properties of things (elementary units in the ontology). Similarly, Hoffer et al. (2002) define attributes used in database design as "a named property or characteristic of an entity that is of interest to the organization" (p. 314). Based on these definitions we use the term 'attribute' to represent properties of knowledge - such as the date or the accuracy of the knowledge, and properties of the knowledge source - such as their job title, experience, or level of trustworthiness. We use the general term 'meta-knowledge' or the more specific 'knowledge and source attributes' interchangeably. 34 Chapter 4: The role of meta-knowledge in KMS design A review of the literature related to meta-knowledge shows that metaknowledge appears in different forms depending on the context of the study and the application of the knowledge itself. One approach - that can be termed the conceptual approach - views it as a "structure that defines other structures" (Plant and Gamble, 1997). Based on this approach, meta-knowledge consists of a combination of models that describe the knowledge domain - such as ontologies, problem solving methods, or guidance patterns (Kalfoglou et al., 2000). The conceptual approach identifies numerous benefits to the use of meta-knowledge in the construction and maintenance of knowledge-based systems. Meta-knowledge in such applications facilitates better communications through the use of the ontology as a shared language, enables inter-operability of systems with different modeling methods and paradigms, assists in intelligent knowledge search, and can be used as a starting point in exploring new domains (Menzies et al., 2000). The latter benefit of meta-knowledge was also noted in an empirical study by Cross, Rice, and Parker (2001) who reported the benefits of meta-knowledge as a pointer to potential knowledge sources. While the conceptual approach to meta-knowledge is mostly used in projects focusing on the design of knowledge-based systems, the notion that meta-knowledge plays an important role in problem-solving links us to the role of meta-knowledge in the cognitive science literature. This cognitive approach defines meta-cognitive knowledge, or "knowing about knowing" as an important dimension of professional expertise (Van . Der Heijden, 2000). Meta-cognitive knowledge refers to acquired knowledge about cognitive processes and can be further divided into three categories. Knowledge of person variables refers to beliefs about the nature of one's, and others', learning processes. Knowledge of task variables refers to the knowledge about the requirements of the task and the suitability of the information available for completing the task. Finally, knowledge of strategy variables refers to beliefs on what cognitive strategies should be employed for achieving a specific goal (Flavell, 1979). A common use of another type of meta-knowledge is as metadata in databases and - more recently - on the Web. According to this more descriptive approach - metadata is used to index databases and to facilitate the location and use of data and information that resides in these databases, as well as the combination of data from different databases. Metadata can be classified into five types: 'administrative' metadata is used in administering information resources; 'descriptive' metadata is used to describe information resources; 'preservation' 35 metadata relates to the preservation of information resources; 'technical' metadata relates to how a system functions; and 'use' metadata relates to the level and type of use of information resources (Lazinger, 2001). Some benefits associated with the use of metadata include the increased accessibility of the data, the retention of context, the expansion of the use of the data, and the ability to create many versions of the data (Gilliland-Swetland, 1998). On the Web, there are several initiatives to create interoperable metadata standards in order to facilitate information exchange and discovery. For example, CanCore (http://www.cancore.ca) is one initiative intended to facilitate the exchange of data and records related to educational resources both in and out of Canada. The Dublin Core (http://dublincore.org) is another example for such initiative that spans the whole Web. The Dublin Core's mission is to facilitate finding resources using the Internet, through developing metadata standards for discovery across domains; defining frameworks for the interoperation of metadata sets; and facilitating the development of community (domain) specific metadata sets8. Examples for the type of metadata included in the Dublin Core are Title, Creator, Subject, Description, Publisher, Contibutor, Date, Type, Format, and more. The attributes included as metadata are mostly informative in nature, for example, the author's information, the scope of the knowledge, the intended audience, and the format and currency of the knowledge (Basch, 1990; Katz, 1992; Stoker and Cooke, 1995). However, meta-knowledge can also be used to influence attitudes of potential users of the knowledge, where attitudes are defined as "psychological tendencies expressed by evaluating a particular entity with some degree of favor or disfavor" (Eagly and Chaiken, 1993). The persuasive approach to meta-knowledge identifies characteristics such as source credibility, expertise, and trustworthiness as affecting attitudes towards a specific message (Hovland, Janis, and Kelly, 1953; Higgins, 1999). In addition, message specific characteristics such as the number of arguments or the type of appeal presented in the message were also identified in the persuasive literature (Petty and Cacioppo, 1986; Stiff, 1994). As previous discussion indicates meta-knowledge also plays a role in the design of KMS, however not much work has been conducted on meta-knowledge in this context and the knowledge and source attributes that are included in commercial KMS vary widely. For example, Lotus Discovery server - a leading general purpose KMS - provides some basic attributes of the knowledge stored in it, such as document summaries, author identity, origin of the document (where it was retrieved from), and a calculated relevance rank for each document. http://dublincore.org/about/overview/ 36 In addition, Lotus Discovery provides some source attributes such as the source's name, job title, location and contact information, affinity to the topic sought, background information, and availability to" answer questions. On top of these basic attributes defined by Lotus Discovery, users can define additional attributes based on their preferences. A similar approach to meta-knowledge is built into Microsoft's SharePoint - a content management server. As in Lotus Discovery, SharePoint provides basic document profiling attributes including title, author, keywords, document description, and a 'best bet' score -indicating the relevance of a document to the specific search. In addition, SharePoint enables users to define their own profiling attributes. A different kind of KMS, Knexa - a knowledge exchange system - focuses on source attributes and provides customizable, information about the knowledge source including their identity, area of expertise, short bio, and links to previous answers provided by the knowledge source. A more automated approach is designed into Interwoven's content intelligence system. Interwoven's MetaTagger™ automatically reads through various documents and creates a taxonomy of terms as well as generates document summaries and derived meta-data. The above examples of the numerous types of knowledge and source attributes embedded in the design of various KMS illustrate that effective knowledge management combines together attributes from the different types of meta-knowledge discussed earlier. This observation builds on the fact that the task of searching and adopting knowledge provided by a KMS includes attributes from the above four types of meta-knowledge. For example, meta-knowledge is important in facilitating the location of knowledge in the knowledge base (the role of metadata), linking to other relevant knowledge (the role of ontologies and conceptual meta-knowledge), providing some evaluation of the knowledge (the role of persuasive meta-knowledge), and providing some indication of the best source and the best knowledge to use given the knowledge user's background, education, and experiences (the role of meta-cognitive knowledge). This need for numerous knowledge and source attributes - however - raises two problems for KMS designers and administrators. The first problem concerns information overload, in this case - having too much meta-knowledge. The second problem is economic in nature and involves the cost of capturing, operationalizing, and maintaining knowledge and source attributes in order to represent them in the KMS. This cost increases with the number of attributes operationalized. Moreover - it is not necessarily the case that the same set of attributes should be included in all KMS regardless of their intended use, or users. It is therefore necessary to identify a subset of effective knowledge and source attributes that should be included in the design of knowledge management systems. 37 In order to identify this subset of attributes and to ensure their effectiveness in the context of KMS we return to examine - in more depth - the two roles of meta-knowledge in KMS design identified in Chapter 2. The first role involved the enhancement and support for the management of organizational memory, and the second - increasing the likelihood of knowledge adoption (or internalization) by KMS users. We explain each of these roles of meta-knowledge in more detail below. Managing organizational memory "Organizational memory is the means by which knowledge from the past is brought to bear on present activities, thus resulting in higher or lower levels of organizational effectiveness" (Stein, 1995:22) Effective sharing and use of knowledge available to an organization depends - to a large extent - on the organization's ability to create and manage its collective memory. This collective memory is often referred to as Organizational Memory (OM). According to Stein and Zwass (1995) organizational memory can increase organizational effectiveness by supporting the coordination of work, management of information, the organization's responsiveness to changes, and the definition and pursuit of organizational goals. Such memory generally resides in different retainers in the organization and organization members retrieve its content based on their work needs (Walsh and Ungson, 1991). The characteristics of organizational memory and its similarity to databases make it a good candidate for Information Technology (IT) support (Stein and Zwass, 1995), and an effective KMS - or more specifically an Organizational Memory Information System - would have to facilitate the acquisition, retention, maintenance, and search and retrieval of knowledge in the various memory retainers. However, there are several major problems that might inhibit the management of OM with the use of IT. We identify five specific problems: First, Much of the knowledge in the OM is contextualized. When knowledge is transferred the receiving end of the communication system often does not know the original context of the knowledge and therefore cannot interpret it correctly (Anand, Manz, and Glick, 1998; Stein, 1995). Related to this, Ackerman and Halverson (2000) state that to cross boundaries - either departmental or organizational - the knowledge needs to be stripped of its context for the receiving end to be able to understand it. Second, as Walsh and Ungson (1991) suggest organizational knowledge resides in five different memory retainers, namely - individuals, culture, structure and roles, transformations and rules, and physical settings. These retainers 38 may be in different locations and their memories might be difficult to combine into one system (Anand, Manz, and Glick, 1998; Wijnhoven, 1999). Third, organizational knowledge is often tacit and therefore difficult to track and maintain in large organizational memories (Stein and Zwass, 1995). Fourth, since knowledge is quite volatile, organizational memory might change frequently (Wijnhoven, 1999). In addition, combined with the problem of context dependence, the volatility of knowledge further complicates the search and retrieval of knowledge included in the OM. Finally, since some knowledge is retained outside the organization (Walsh and Ungson, 1991) or in unfamiliar sources within the organization, a measure of the retainer's legitimacy and reliability is required (Anand, Manz, and Glick, 1998). According to Stein (1995) an inquirer is motivated to retrieve knowledge if the inquirer is aware of the knowledge and sees potential value in the knowledge. This information should be included in the OM management system to facilitate the retrieval and use of knowledge. The five problems discussed above are summarized in the first column of Table 6. We propose that the first role of meta-knowledge in the design of KMS is therefore to alleviate these problems, as described in the second column of Table 6. Problem with OM Proposed Solution Knowledge in the OM is contextualized. Conceptual meta-knowledge can help reconstruct the original context ofthe knowledge. Retainers of OM may be in different locations and their memories might be difficult to combine. Descriptive meta-knowledge locates different knowledge retainers as well as provides information on the most up-to-date version of the knowledge. Knowledge may be tacit. Tacit knowledge can be located and evaluated using meta-knowledge to point to the people it. The meta-knowledge itself is not tacit. The content of OM changes often. The responsibility to 'keep track of knowledge' is now at the hand of the relevant expert. The system maintains only meta-knowledge, which is less volatile than the knowledge itself. Knowledge seekers require a measure of the retainer's legitimacy and reliability when retrieving knowledge. Descriptive and persuasive meta-knowledge is included in the KMS and can be used as an indicator of the source's reliability and legitimacy. Table 6: Meta-knowledge solution for organizational memory problems The specific mechanism by which meta-knowledge can successfully support the management of organizational memory can be adapted from a theory concerning a related group memory -namely, transactive memory theory. 39 Transactive memory theory Wegner (1987) defines a transactive memory system as "...a set of individual memory systems in combination with the communication that takes place between individuals" (p. 186). Such a system is built on the distinction between internal and external memory encoding. In many cases individuals encode new knowledge internally. That is, they learn something new and catalogue it in memory for future retrieval and use. However, even more often individuals encode knowledge externally either in diaries, in books, or even in other people's memory. In these cases, the individual internally encodes the label (subject) of the knowledge as well as its location but not the knowledge itself. Transactive memory systems are built on this view of individuals playing the role of external memory for other individuals who - in turn - encode meta-memories (i.e. memories about the memories of others). Wegner (1995) proposes, that two types of meta-memories are maintained in people's minds - information about the subjects of knowledge of each member (i.e. areas of expertise) and information about the locations of the knowledge. Knowledge is encoded, stored, and retrieved from the collective memory through various transactions between individuals, based on their meta-memories. Initial research on transactive memory focused on dating couples (Wegner, 1987) but in a series of later experiments, Moreland, Argote, and Krishnan (1996) examined the existence and benefits of transactive memory developed from shared experiences in small groups. They assigned groups the simple task of building the AM portion of an AM/FM radio and provided shared training for some of the groups while members of other groups were trained individually and assigned to groups after the training session. The findings of these experiments indicated that training together led to the development of transactive memory in the group and also resulted in improved group performance. Moreland and Myaskovsky (2000) also suggest that another benefit of transactive memory is the creation of a more efficient problem-solving mechanism, since group members know more about each other and are able to match problems with the people who are most likely to solve them. While the benefits of transactive memory are appealing to organizations, not much research has been done on the application of transactive memory theory to organizational memory9. A potential reason might be the difficulty in developing an organization wide transactive memory Anand, Manz, and Glick (1998) examined the existence and relations between various transactive memory systems within organizational memory. However, they did not extend a single transactive memory system to the whole organization but rather remained within the framework of small groups. 40 system due to physical constraints such as the size and geographical dispersion of organizations, the memory capacity of individual members, the lack of relations between group members (e.g. perceptions of trust and expertise), and the existence of knowledge that is not 'formally' assigned to any member or group in the organization. For example - if one person in the accounting department' is also interested in technology, members of the department will allocate technology knowledge to this person and consult with her when such knowledge is sought. However, formally - this person is an accountant and therefore other members of the organization are not likely to consult with her on technology matters. This will result in some organizational knowledge being unavailable to all members of the organization. The role of meta-knowledge in supporting organizational memory is therefore to provide the equivalent of meta-memories that can be stored in the KMS and support an organization-wide transactive memory system. In addition to illustrating the role of meta-knowledge in organizational memory management, transactive memory theory also provides some insights regarding the specific knowledge and source attributes that should be collected and maintained by the KMS. First, the two types of meta-memory identified by Wegner (1995), namely - the subject and location of knowledge -should be included in the KMS. In addition, Rulke and Rau (2000) note that members of small groups also maintain meta-memories about their perceptions or evaluation of group members, which they employ when selecting the best knowledge for their needs. This latter observation leads us to the second role of meta-knowledge identified earlier - enhancing knowledge adoption by individuals. In Chapter 2 we discussed this role of meta-knowledge using the concept of absorptive capacity of individuals. In the next section we return to examine the concept of knowledge adoption relying on research on knowledge transfer as well as on a recent theory of information adoption in computer-based communications. We use this theory as the basis for our research question concerning the specific knowledge and source attributes that a KMS should include. Enhancing the adoption of knowledge Knowledge adoption refers to the internalization and subsequent use of knowledge - once discovered - by individuals. Understanding the adoption of knowledge by knowledge seekers is an important question in KMS research and specifically with relation to knowledge transfer and application (Alavi and Leidner, 2001; Sussman and Siegal, 2003). Knowledge adoption provides 41 support for knowledge application, which is "the phase [of KM] in which existing knowledge is brought to bear on the problem at hand" (Alavi and Tiwana, 2002)10. The importance of knowledge application is evident from a recently published KM report by KPMG (KPMG, 2003) in which 78% of the respondents believed that they are currently missing out on business opportunities by failing to exploit available knowledge. Furthermore, these missed opportunities are estimated at a loss of nearly 6% of revenues. It is therefore important to identify the specific role of meta-knowledge in knowledge adoption and to pinpoint the specific knowledge and source attributes that should be included in a KMS in order to support knowledge adoption. Knowledge adoption can also be tied to the concept of knowledge transfer, which indicates how one unit in the organization is affected by the experiences of another (Argote and Ingram, 2000; Sussman and Siegal, 2003). Some success factors of knowledge transfer have been studied in recent years and Argote and Ophir (2002) note that successful knowledge transfer would depend on the previous experience of the parties in transferring knowledge. In addition, trust also plays an important role in successful knowledge transfer (Kale, Singh, and Perlmutter, 2000). From the opposite view, studying the transfer of best practices in organizations Szulanski (1996; 2000) identified several factors that might inhibit successful knowledge transfer. Among these are: low absorptive capacity of the recipient of the knowledge, the recipient does not perceive the utility of the transferred knowledge, the recipient perceives the source as unreliable, or the source and recipient have no preexisting relationships. Based on this, meta-knowledge can play a role in supporting successful knowledge transfer by providing the receivers of the knowledge with information such as indicators of the identity and reliability of the source or of the utility of the knowledge. Focusing on the concept of adoption, Sussman and Siegal (2003) develop a model of information adoption in computer-mediated communications that draws on two theoretical foundations - the Technology Acceptance Model (TAM, Davis, 1989) and the Elaboration Likelihood Model (ELM, Petty and Cacioppo, 1986). Based on TAM, which identifies the role of perceived usefulness and ease of use on computer usage, the information adoption model (described in Figure 7) links adoption to the perceived usefulness of the information -hypothesizing that higher perceived usefulness will lead to higher levels of adoption. Based on ELM, the model hypothesizes that perceived usefulness of information is affected by two 1 0 The term 'knowledge application' as used here is human oriented - that is, we are interested in how individuals apply the knowledge they gain to perform their job. We do not include in the scope of this study the automatic application of knowledge embedded in process or similar. 42 variables, namely argument quality and source credibility. Since this dissertation focuses on identifying attributes that might increase knowledge adoption, we elaborate on this second set of hypotheses below, beginning with a brief overview of ELM. Recipient's Expertise Argument Quality Source Credibility Recipient's Involvement Information Information Usefulness w Adoption Figure 7: The information adoption model (Source: Sussman and Siegal, 2003) Elaboration Likelihood Model The Elaboration Likelihood Model identifies two routes to persuasion - the central route involves attitude change due to message-based thinking, and the peripheral route involves attitude change based on simple cues about the attitude-object (what the attitude is about). The basic postulates of the model are that people want to hold correct attitudes but the extent of their willingness and ability to process the persuasive argument in order to reach these attitudes varies widely. 'Elaboration likelihood' is the likelihood of people to think about issue-relevant arguments and is said to be high when motivation and ability are high. When elaboration likelihood is high, the central route to persuasion is more likely to occur. When elaboration likelihood is low, people will follow the peripheral route to persuasion, relying on peripheral cues and mechanisms such as heuristics, affective mechanisms, or social roles mechanisms (Eagly and Chaiken, 1993). The model also postulates that two variables can affect the amount and direction of attitude change, namely argument quality and peripheral cues. Argument quality refers to the persuasiveness of the information included in the message and Petty and Cacioppo (1986) define either strong or weak arguments. Strong arguments are those that instruct subjects to think about the message, generating predominantly favorable thoughts, and weak arguments generate predominantly unfavorable thoughts. For example, "Prestigious universities have 43 comprehensive exams to maintain academic excellence" is a strong argument while "adopting the exams would allow the university to be at the forefront of a national trend" is a weak argument (Eagly and Chaiken, 1993). Peripheral cues are stimuli that can affect attitudes without necessitating processing of arguments. Petty and Cacioppo (1986) - based on their work as well as previous persuasion studies - discuss several peripheral cues related either to the source of the message or the message itself. Source cues include things as the source's expertise, source likeability, and source credibility. Message cues include the number of message arguments, or the validity of the arguments. In addition to their potential effect on attitude, peripheral cues also affect the route to persuasion by affecting the motivation or ability of people to elaborate on the message's arguments. The empirical results of the information adoption model developed by Sussman and Siegal (2003) show that both argument quality and source credibility (as a representative of peripheral cues) have a significant affect on the perceived usefulness of information and through it - on adoption. In addition, the model tested for two moderators - receiver's expertise and involvement in the task - the two variables were shown to moderate the central route (argument quality) but only involvement moderated the effect of source credibility on perceived usefulness. Although only source credibility was examined in their paper, the authors note that "it seems likely that some additional cues played a role in the influence of peripheral processes in this study" (p. 59). Since peripheral cues can be interpreted as a type of meta-knowledge (i.e. persuasive meta-knowledge) these results support the important role of meta-knowledge in the design of KMS. Nevertheless, apart from source credibility, the model does not identify the specific knowledge and source attributes that can effectively improve knowledge adoption. Thus, since none of the theories and approaches to meta-knowledge discussed so far identified in detail the attributes that should be included in a KMS to support both management and adoption of knowledge, we frame this question as our research objective in the next section. Research objective Our research question presented in Chapter 1 was to identify ways to improve the design of KMS. Specifically, in this part of the thesis we chose to focus on identifying the meta-knowledge that should be included in a KMS to effectively support KM. We have so far discussed the motivation, and potential benefits, for using meta-knowledge to facilitate the management of 44 knowledge by the KMS and its use by organizational members. In addition, we identified several approaches to meta-knowledge that exist in the literature. However, it is not clear which of these meta-knowledge approaches should govern the design of KMS and this problem is evidenced by the multiplicity of meta-knowledge types included in existing KMS. In order to answer our main research question we therefore need to identify first the list of important knowledge and source attributes (i.e. meta-knowledge) as well as their relative importance, or ranking. In addition, it is useful to identify whether the same set of attributes can be used across users and applications or whether there are contingencies to their use. Specifically we are interested in identifying potential groups of users that might share preferences towards these attributes in order to enable better customization of the KMS. To identify the best set of knowledge and source attributes we take a marketing oriented approach to knowledge and identify the set of attributes of knowledge and of knowledge sources that make them attractive to people. Our main research question is: What are the most important knowledge attributes and the most important knowledge source attributes that people consider when making a knowledge adoption decision? An additional question is: are there groups of people who share preferences with regards to knowledge or to knowledge sources? How can these groups be characterized? 45 Chapter 5: Methodology The selected method to identify the knowledge and source attributes that the KMS should provide to its users is conjoint analysis. Conjoint analysis is a multivariate technique used to understand respondents' preferences towards products or services. As the name suggests, it is used to understand the joint effect of two or more independent variables on the ordering of a dependent variable (Green and Rao, 1971). Conjoint analysis was first developed in the 60s as a psychometric method (Luce and Tukey, 1964) and the method has since grown in popularity in numerous fields. It is widely used to evaluate acceptance of new products, identify new market shares, or predict consumer behavior (Cattin and Wittink, 1982; Green, Krieger, and Wind, 2001; Wittink and Cattin, 1989). Some examples for applications of conjoint analysis are the design and evaluation of a new product/technology (a study of EZ-Pass in the US by Vavra, Green, and Krieger, 1999), evaluation of services (a study of the Mariott Courtyard Hotels by Wind et al., 1989), and evaluation of consumers' perceptions. (For example, Gordon and Lima-Turner (1997) evaluate consumers' attitudes towards Internet advertising. Soo (1999) examines risk perceptions of online shoppers). Conjoint analysis is a de-compositional approach and it relies on a basic assumption that respondents' preferences toward a product can be decomposed to derive the values they place on specific attributes of the product (Hair et al., 1992). In this sense conjoint is similar to the widely accepted hedonic price model (Rosen, 1974) used in economics to identify individuals' preferences towards specific attributes of a product, as revealed by their willingness to pay more for products that include desirable attributes. There are several main benefits for using methods such as conjoint analysis to study individuals' preferences towards specific attributes over the compositional, self-explicating methods that ask people to state the desirability and importance of a each product's attribute separately. First, decision makers in the conjoint analysis are faced with options that vary across two or more attributes and therefore are forced to make trade-offs that more closely imitate real life decisions. Second, in the self-explicating process when responding to questions such as 'how much is attribute A important to you?' respondents might provide the socially desired answer rather than their true valuation of the attribute. Third, if some attributes are correlated, for example gas mileage and economy of a car, double counting is more likely to occur in the self explicating method than in conjoint analysis. Finally, in self-explicating methods relative desirability of specific levels of attributes (e.g. three different price levels) is more likely to be linear than in conjoint analysis. Conjoint 46 analysis enables fitting the best model for the relative desirability of levels of each of the attributes (Green and Srinivasan, 1990; Hairet al., 1992; McCullough, 2002). In addition to enabling the accurate evaluation of individuals' preferences towards specific knowledge and source attributes conjoint analysis is commonly used to identify and analyze differences in the attribute values placed by specific groups or segments (Cattin and Wittink, 1982; Green and Rao, 1971; Hair et al, 1992). To this end cluster analysis methods are used to identify market segments within respondents. These clusters can then be analyzed using various methods to identify the characteristics of their members (Green and Srinivasan, 1978; SPSS, 2001). The main result of such analysis is the identification of groups within respondents based on the overall evaluation of the product examined. In the context of this study, the cluster analysis enables us to group people based on their overall evaluation of a specific piece of knowledge or a specific source. Studying these results can provide insights as to what makes a specific knowledge or source more attractive to specific groups of people. The choice of conjoint analysis for this study is derived from the study's objective: to evaluate the relative importance of knowledge and knowledge source attributes to knowledge seekers. In this sense the task that respondents face is similar to the product choice decision in which conjoint analysis is often used (Cattin and Wittink, 1982). Moreover, the ability of the conjoint method to support segmentation is especially relevant to the context of IS design since it enables us to distinguish between potential groups of KMS users. Identifying such groups would enable the systems' designers to customize the system (in this study - the KMS) to the needs of the specific group of users, an important factor in the success of the design process. We begin this chapter with a discussion of the theoretical foundation of conjoint analysis followed by the specific methodology used in this study. Theoretical foundations The theoretical foundations of conjoint analysis can be traced back to multiattribute attitude models such as Fishbeins's expectancy-value model (Green and Srinivasan, 1978). The expectancy-value model explains how attitudes are formed by combining beliefs about specific attributes of an attitude object together with the evaluation of these attributes, as shown in equation 1 (Eagly and Chaiken, 1993): <=i 47 Where Ao is the attitude towards an object o, bj is the subjective probability that o includes attribute i, and ei is the evaluation of attribute i. Finally, n is the number of attributes of object o. However, conjoint measurements differ from the expectancy-value model in two main aspects: first, expectancy-value models are compositional - the total utility of an object is composed of the individual utilities of its components, judged separately by individuals. Conjoint, on the other hand, is a decompositional method in which individuals evaluate the total worth of an object and the individual utilities are derived from this total by using this method. Second, the main objective of conjoint models is more predictive than explanatory while expectancy-value models are generally more focused on explanations than predictions (Green and Srinivasan, 1978). Therefore, a closer theoretical basis for conjoint measurement is seen in Anderson's Information Integration Theory (Green and Srinivasan, 1978; Louviere, 1988). The basic idea of Information Integration Theory (IIT) is that individuals make judgments based on evaluation and integration of information stimuli. According to the process described by IIT, an individual is faced with a set of observed stimuli S that are translated into their psychological values - s - through a valuation function V(S). The psychological values are combined through an integration function I into an implicit response - r. This response is then externalized by a response function M to become an observable response R (Anderson, 1981). This is illustrated in Figure 8. 51 + s r — - ~ s. \ \ \ 5 2 • S 2 -}-_____> r • R / / / s 5 3 • S3-—''' Figure 8: Information Integration Theory (source: Anderson, 1981) The main constructs of the theory are valuation and integration (Eagly and Chaiken, 1993). Valuation describes the attachment of a subjective psychological meaning to the observed stimulus. Each stimulus is assigned a scale value representing the location of the stimulus along the judgment dimension, and a weight value, which represents the relative salience of the stimulus in the overall response. Integration represents the combination rule of the stimuli using algebraic models such as an adding model or an averaging model, to arrive at an overall response. 48 A main implication of NT is functional measurement - the ability to derive subjective valuations of the stimuli from the observed response R. To show this, consider the following two-variable factorial design: S B I S B 2 S B J S A I R n R-12 ... ... S A 2 1^ 22 ... ... : ; : S A I R i i R i 2 ... R i j ... ; : R.2 R.j Figure 9: Factorial design for functional measurement (source: Anderson, 1981) The levels of the row factor are the observed stimuli of factor A and the columns are the observed stimuli of factor B. Ry are the observed responses to the two stimuli. In an adding model, an individual's implicit response is the sum of the subjective values of the stimuli: rtj = sAi + S B J - Since all three factors in this equation are subjective it seems difficult to derive the true valuations of individuals. However, in the functional measurement approach the row and column means of the factorial design will be estimates of the subjective values of the row and column stimuli, respectively. To show this, we first assume that the observed response is a linear function of the implicit response: (3) Rv=C0+Cirg (C 0 and C-[ are constants.) For I rows, the mean of column j entries is given by: (3) R..=(l/I)£iRii 1=1 which can be written as: 49 R.J=(l/I)YJC0+Cl(sAi+sBj) = (3) (1 / / )EC 0 +C,(1//)X^ +C, ( ! / / ) £ > * 1=1 " i=i i=i where sA is the mean of sAi. Since C 0 is a constant it can be represented as C0 to rewrite (3) as: (4) R.j=C'Q+ClSv The result of equation (4) is that subjective valuations of column stimuli are a linear function of column means, and the same can be shown for row means. Metric conjoint models support the same idea as functional measurement (Louviere, 1988). Consider a brand j with three attributes: 1, 2, and 3. The individual's utility U from this brand can be described as: (5) tf,=C + F ( S v ) + K(S 2 > ) + K ( S v ) where V(S..) is the valuation of each of the observed attributes. We can also specify the levels of these attributes as p, q, and r respectively. The individual's utility from brand j is therefore: (6) Upqr=C + V(Slp) + V(S2q) + V(S3r) since we cannot observe U (the true utility of the individual) we observe R instead, a score on some category-rating scale. R is assumed to be a linear function of U (as was assumed by IIT) and therefore we can write equation (6) as: (7) Rpqr = C + V(Sip) + V(S2q) + V(SJr) + epqr Similar to equation (3) we can derive the part worth utilities of a specific attribute from averaging the other two attributes: (8) Rp.. =[C + V(S2.) + V(S3.)] + V(Slp) + ep.. 50 The algebraic formulations above demonstrate that conjoint measurement can be used to evaluate the judgment strategy of individuals and enable us to identify the part worth utilities of specific attributes based on observed responses on a rating scale. In the context of this thesis, we can observe the part worth utilities of specific knowledge and source attributes based on the observed rating of the knowledge itself. We now turn to describe the specific methodology used to conduct conjoint analysis and the construction of our study. There are several steps/decisions involved in the design of a conjoint study (Green and Srinivasan, 1990). The first step involves identifying the attributes of the product evaluated and the potential levels of these attributes. The next steps are the selection of the preference model - representing the preference structure of respondents and the selection of the data collection method - or the conjoint model. Following this, the stimulus set is constructed, setting the number of stimuli to be used in the study; a presentation method is selected; and the measurement scales (ranking or rating) are determined. The final step is selecting the estimation method. This can be a metric method (multiple regression), non-metric methods (MONANOVA), or choice-based methods (logit, probit). The rest of the chapter is built around these steps. The first choice is the selection of attributes used to evaluate the product - in our study, the product evaluated is knowledge (and knowledge sources) and therefore we need to identify the set of relevant knowledge and source attributes. The selection of the right set of attributes in a conjoint study is important and all relevant attributes should be included in order to correctly represent the value of the product evaluated (Hair et al., 1992). As discussed in previous chapters, there is no clear identification of a full set of important knowledge and source attributes in the literature and therefore an empirical method is required. Common methods for deriving the list of relevant attributes include personal interviews, expert judgment, group interviews, or computerized methods (Wittink and Cattin, 1989; Green and Srinivasan, 1990). In this study we use an exploratory Delphi study described in the next section. 51 Selection of attributes - a Delphi study Delphi is a method for exploring ideas or producing information from a group of experts. It aims to obtain a consensus of opinions from a group of experts with repeated use of questionnaires and controlled feedback. The method was first developed in the 1950s at the RAND Corporation and its initial goal was to improve methods of forecasting. Today the method is also used in management studies to achieve a group consensus about the relative importance (ranking) of issues (Schmidt, 1997). Linstone (1978) defines Delphi as "a method for structuring a group communication process so that the process is effective in allowing a group of individuals, as a whole, to deal with a complex problem". The basic process of Delphi involves three elements: structuring of information flow, feedback to participants, and anonymity of participants. The process begins with the selection of a panel (one or more) usually consisting of experts in the topic. The next stage is drafting and validating the first questionnaire and administering it to panel members. This is followed by an analysis of responses and the creation of the second questionnaire. Again, the questionnaire is sent to panel members and responses are analyzed. The last stage is repeated until a consensus is reached (Linstone, 1978). The Delphi method is most useful when one cannot use precise analytical techniques; when the problem is new and unexplored; or when the problem requires the exploration and assessment of numerous issues (Adler and Ziglio, 1996). The results of the Delphi study can serve to ensure that all major possible options concerning a particular issue have been investigated; to estimate the impact and consequences of particular options; or to examine the acceptability of a particular option. Delphi has not been used much in the IS literature, however some studies exist that use Delphi for ranking the importance of issues as described in Table 7. 52 Authors Research objective Number of panelists Description of questionnaires, process and rounds(R) Brancheau and Wetherbe, 1987 Identify the ten most critical issues facing IS executives Order issues based on importance How much agreement is there among IS executives? R1 - 90 (out of 180 sent). R2 - 54 IS executives; 21 general managers R3 - 68 IS executives; 12 general managers R1 - to IS executives: select 10 most critical issues out of a list, update and add new issues. R2 - to IS executives: feedback on previous round's results and an updated list. Select again the 10 most critical issues. In addition, similar questionnaire was given to general managers in the participating organizations R3 - to IS executives and general managers (all participants from previous rounds): feedback on previous round's response. Rank the top 10 issues one last time. Schmidt et al., 2001 Identify typical project risk factors Rank risk factors Identify effective countermeasures Three groups (G): G1 - 9 G 2 - 1 9 G 3 - 2 1 R1 - brainstorming round. Each panel member requested to submit at least six factors including a short description. Authors constructed two separate lists and then compared and combined them. The final list was submitted to panel members for validation. R2 - panelists in each group requested to choose at least 10 factors they considered important. Responses were used by authors to reduce the list. R3 - Multiple ranking rounds. Panelists requested to rank items on the reduced list until a consensus was reached. Holsapple and Joshi, 2000 Test a framework characterizing organizational knowledge resources. R1 - 31 (out of 122 requests sent) R 2 - 1 7 R1 - panelists requested to evaluate framework based on provided criteria. R2 - the revised framework; response analysis documents; and the round 1 questionnaire were sent to panelists. R3 was deemed unnecessary. Couger, 1988 Identify the 10 most important issues in HR management in the 90s Determines relative importance of the issues R1 - 57 IS executives; 21 HR executives. R3 - 43 IS executives; 21 HR executives. R1 - panelists requested to identify six most important HRM issues for the 90s, the rational for the choice, and a ranking of these issues. Responses were consolidated according to mean ranking. R2 - panelists received the consolidated list and requested to evaluate their original ranking against this list. Panelists were also requested to rank their top 20 issues. R3 - panelists received the list of top 20 issues and asked to provide a final ranking. Also, panelists were asked to rank their top 10 issues and give the remaining issues a rank of zero. Table 7: Examples of Delphi research in IS 53 Delphi Study Design 1 1 The underlying task of the Delphi study was to identify the most important knowledge and source attributes that are considered when searching for a specific answer or solution to a problem. We referred to 'knowledge' as the answer required to complete a given task successfully, 'Knowledge source' was initially defined as a person, book, document, web page, database, or any other entity that may provide the knowledge required to answer the query. To facilitate the understanding of the task we employed specific knowledge search examples to provide a context for participants. However, to maintain external validity and to ensure that we elicit a wide set of responses, we used three different knowledge search examples. In the first example, we asked participants to find the correct answer to a multiple-choice question from the psychology domain (similar to a standardized test question). In the second example, we asked participants to assist the MBA office in the admission decision of new MBA students. Finally, for the third example we asked participants to provide a knowledge search example from their everyday work. The choice of these three examples was based on evidence from the literature indicating that required meta-knowledge might be affected by a person's level of expertise in the domain and by the equivocality of the task. Expertise is defined as the possession of a substantial body of knowledge and procedural skills (Nelson et al., 2000). According to Bedard et al. (1993) experts own more meta-knowledge than novices and have a stronger sense of what is relevant. We therefore expected that expertise might affect the ability of participants to identify the meta-knowledge they require. Following this we asked participants to provide their own example from their domain of expertise in order to solicit a more complete set of meta-knowledge. The second factor, equivocality, is the multiplicity of meaning conveyed by information and is associated with confusion and lack of understanding (Lim and Benbasat, 2000). Daft and Macintosh (1981) found a negative relationship between task equivocality and task analyzability as well as between equivocality and the amount of task information processed. As mentioned earlier, one type of meta-knowledge - namely meta-cognitive knowledge - includes knowledge of task variables, which refers to the knowledge about the requirements of the task and the suitability of the information available for completing the task. The negative impact of equivocality on task analyzability might therefore reduce the knowledge of task variables that 1 1 Delphi questionnaires and approach letters are attached in appendix B 54 individuals have, thus potentially affecting the amount meta-knowledge that individuals have. Similarly, the negative relationship between task equivolcality and the amount of task information processed might also reduce the amount of meta-knowledge processed. To ensure a more complete set of responses in round 1 we employed two examples with different levels of equivocality. The multiple-choice question represented a low equivocality task and the MBA admission question represented the high equivocality task (following Dennis and Kinney, 1998)12. The Delphi study consisted of a panel of 28 members from seven different Canadian organizations and was conducted over the Web. The demographics of panel members are described in Table 8. Table 8 also includes two measures of the organizational characteristics of the participants, namely - knowledge intensity and formalization. Knowledge Intensity is the extent to which a firm depends on its knowledge as a source of competitive advantage (Autio, Sapienza, and Almeida, 2000). Formalization is the extent to which rules, procedures, instructions, and communications are written in the organization (Pugh et al., 1968). We included these two items in the demographics of the panel to gain some indication of the level of familiarity of the respondents with knowledge searches and - potentially - with accepted meta-knowledge used in current organizational systems. For example, in organizations with high levels of formalization there might be some guidelines as to what is the accepted meta-knowledge. This is used to ensure external validity in terms of the panel selected. We conducted two pilot tests to evaluate the suitability of the questionnaire and technology to the task. The first pilot included a panel of five PhD students and tested the suitability of the website for the study as well as the clarity of the instructions. The second pilot included a panel of 12 participants from the target population of the study itself. Following this pilot we made two main changes to the study's design: (1) we reduced the number of examples presented to each panel member in the first round from three to two to reduce load on participants; and (2) we requested panel members to rank meta-knowledge characteristics instead of rating them. Originally we requested panel members to rate characteristics on an importance scale of one to ten. Responses from the pilot showed that this was ineffective since panel members rated all factors as important (7 or 8 and above). Therefore, to avoid this in the actual study, we asked panel members to rank the characteristics and to avoid ties. 1 2 Note that both 'equivocality of the task' and 'expertise of the respondent in the task's domain' are used to support external validity, that is to ensure that a more complete set of knowledge and source attributes are attained from the Delphi study. We do not formalize hypotheses about the potential effect of these variables on knowledge and source attributes. 55 N=28 Age (average) 30-40 Level of education High School 0 Some college 3 Bachelor 9 Masters of higher 14 Professional certificate 2 Other 0 Years working 13 (Range: 3 to 35) Years with company 6 (Range: 0.5 to 29) Years in current position 3 (Range: 0.75 to 25) Knowledge Intensity1 Mean: 2.3 (Range: 1.0 to 5.33) Formalization2 Mean: 4.3 (Range: 2.5 to 5.83 Knowledge use Frequency of using external knowledge in your work Often 22 Sometimes 5 Rarely 1 Never 0 Average number of knowledge sources used. 3 Percent of panel members that used each source: People 86% Books 54% DB 39% Web 96% Other (professional publications; media; research facilities; 18% documentation) Familiarity with knowledge source Very familiar 10 Somewhat familiar 18 Not familiar 0 Familiarity with the concept of KMS Yes 17 No 11 If yes, what KMS are you currently using? Intranet 57% Lotus Notes 21% Best Practices database 11% E R P 4% Other (discussion groups, newsgroups and forums; in house 29% developed systems; data warehouse; decision support system) a=0.85 2 Formalization was measured using Ferrel and Skinner (1988) 6 items scale of 1 (low formalization) to 7 (high formalization). oc=0.75 Table 8: Demographics of panel members 56 The Delphi process The Delphi process employed in this study included four rounds, following the procedure proposed by Schmidt et al. (2001). Round one was a brainstorming round in which we asked participants to identify the most important knowledge and source attributes for the different knowledge search examples described in the previous section. To reduce the load on panel members, we first randomly assigned each panel member to one of two examples - either the multiple-choice question or the MBA admissions question. We presented participants with the following situation - 'you are asked to find an answer to this question (i.e. the correct answer to the multiple choice question, or a decision on which students to admit to the MBA program). To help you in your task you may consult with any one of 20 people who indicated that they have some knowledge on the topic. What information about these people and their knowledge would you require in order to select the best person to turn to?' Note that the question refers to people only as knowledge sources. However, since we initially defined 'knowledge source' as a person, book, document, web page, database, or any other entity, we asked participants to also consider four other sources - books, documents, web, and internal databases. For each of these sources participants noted how their answer to the above question would differ. Specifically - they noted which attributes they would add and which they would remove if the source were not a person but one of the other four sources. We found that the attributes requested about these four types of sources did not differ much from those available in the metadata literature. This result is encouraging since it validates the procedures and the other results in our study. Nevertheless, to be able to attain new and meaningful results - without overloading respondents - in the rest of the study we focused only on people as the knowledge source. After participants considered their required meta-knowledge for one of the two examples provided, we also asked them to provide their own knowledge search example and to identify the meta-knowledge they would require for selecting the knowledge and knowledge source for that example. Some examples provided were: • "/ need to understand the manufacturing technology currently used by a potential customer" • "I need to know how Cardiac Centres are evaluated and ranked" 57 • "I need knowledge of comparability of the world's educational systems to that of Canada's." • "/ need to know how to perform a protein purification procedure" • "/ need to know the preferable technology path for electronic procurement given COMPANY'S architectural considerations and operating requirements." At the end of the round, two judges (the author and another PhD candidate) each analyzed the results and created a combined list of knowledge and source attributes that were identified by panel members, as well as definitions for each attribute on the list. Agreement between the two judges was good (Cohen's Kappa = 0.75). This initial list included 30 attributes overall, described in Table 9 on the next page. To ensure the reliability of the list two more judges (PhD candidates in two different areas) were asked to review both the responses from round one and the combined list created by the first two judges and to assign the responses from round one to the corresponding item on the combined list. Agreement between these two judges was also good (Cohen's Kappa = 0.74) and disagreements were resolved through discussions. In order to reduce the size of the list and facilitate the ranking of attributes we requested panel members - in round two - to review the list created and to pare down the list to a manageable size. Participants were asked to select at least six attributes that they felt were most important to them. All 28 participants completed this round of the study as well as the next two ranking rounds. Based on the number of votes for each attribute we ranked the list of thirty attributes to identify a drop in the number of votes. The overall list did not show a significant drop in votes but after dividing it into two separate lists - for the knowledge and for the source - a small drop from 12 votes to 9 votes exists after the sixth knowledge attribute (accuracy). Building on this we then also excluded source attributes that received less that nine votes, leaving the top eight source attributes. The final list carried on to round three included therefore the top fourteen attribute of both knowledge and source. This list is displayed in Table 10 with an initial ranking based on the number of panel members that selected each attribute as important in each round. Round three and round four were similar in the task provided for participants. In each of these rounds panel members were asked to review the list of fourteen attributes and to rank it in order of importance to them (avoiding ties). Initial consensus after round three was medium (Kendall's W - used to measure consensus - was 0.37) and we therefore initiated another ranking round. After round four, consensus has increased to 0.76. According to Schmidt (1997) good 58 consensus exists when Kendall's W > 0.7 and therefore this was the last round of our study. The final results are described in the last column of Table 10. Knowledge Attribute Definition Votes (round 2) Relevance The knowledge is relevant for the current problem 20 Currency The knowledge is up-to-date 16 Credibility The knowledge can be trusted 16 Extent of knowledge The depth (level of detail) and breadth (coverage) of the knowledge 14 Validity The knowledge can be verified and supported by other sources or by empirical results 13 Accuracy The knowledge contains no errors 12 Logical The knowledge is constructed in a logical manner and does not include internal contradictions 9 Usefulness The knowledge is constructive 9 Communicable The knowledge is clear and easy to understand. The knowledge can be easily communicated and presented to others. 8 Newness The knowledge provides new and unique insights that were not available before 7 Completeness The knowledge covers the whole domain and does not omit any relevant details 7 Source credentials The identity and authority of the source of the knowledge 6 Objectivity The knowledge is not biased 6 Origin of knowledge The method in which the knowledge was obtained (e.g. theoretical knowledge, knowledge from experience, etc.) 6 Minimality (specificity) The knowledge is succinct and specifically answers the problem 3 Related knowledge The knowledge refers to other knowledge that may be related 3 Source Attribute Definition Votes (round 2) Experience Extent of past experience that the source has dealing with this domain or with similar problems 23 Extent of domain knowledge The extent of knowledge about the domain that the source has 17 Accessibility Availability ofthe source to answer questions in a timely manner 16 Communications skills The source can clearly communicate ideas and explain the topic in a simple way 12 Awareness of other resources The ability of the source to locate relevant resources that may provide additional knowledge on the topic 11 Willingness to help The source is approachable and willing to help or provide feedback 11 Up to date The source is current in the knowledge domain 11 Trustworthiness The source is honest and provides credible responses 10 Reputation The source's reputation as knowledgeable in the specific domain 9 Reasoning capabilities The source is able to draw logical conclusions and infer from their knowledge on related problems 8 Objective The source is unbiased 7 Education (background) The formal level of education ofthe source (e.g. past degrees and certificates) 6 Past Performance The source has been a useful knowledge source in the past 6 Occupation The profession and/or job definition of the source 1 Table 9: Results of round 1 and 2 of the Delphi study 59 Results Table 10 shows the fourteen attributes that panel members felt were the most/important to them when searching for knowledge. Attributes Rank1 - round 2 (with ties) Rank - round 3 (no ties, W=0.37) Rank - round 4 (no ties, W=0.76) Relevance of the knowledge 2 2 1 Experience of source 1 1 2 Credibility of the knowledge 4 3 3 Accuracy of the knowledge 9 4 4 Validity of the knowledge 8 5 5 Currency of the knowledge 4 6 6 Extent of knowledge 7 7 7 Accessibility 4 9 8 Extent of domain knowledge of the source 3 8 9 Source is up-to-date 11 10 10 Trustworthiness of the source 14 11 11 Source is willing to help 11 12 12 Communications skills of the source 9 13 13 Source is aware of other resources 11 14 14 Rank in round 2 is computed by summing the number of people that selected an attribute as one of the most important attribute to them. Table 10: Ranking of meta-knowledge attributes With the exclusion of the experience of the source, all the knowledge attributes ranked higher than source attributes. At the top of the list are the relevance of the knowledge to the current problem and the experience of the knowledge source. This is in line with the literature on metadata that emphasizes the importance of ranking the relevance of query results - especially in web searches - and the literature on knowledge management (e.g. Davenport and Prusak, 1998) and persuasion (Hovland, Janis, and Kelly, 1953), which emphasize the important ofthe source's experience and expertise in the problem domain. The next group of attributes refers to elements of the knowledge - specifically the credibility, accuracy, validity, currency, and extent (bredth and depth). Finally, the second half of the list concerns attributes of the source of knowledge, namely - the availability of the source, the extent of domain knowledge that the source has, the currency of the source in the domain, the trustworthiness of the source, the source's willingness to help, the communication skills of the source, and the source's awareness of other resources. 60 Table 10 also shows the changes in responses between study rounds. It is interesting to examine the responses to round 2 (the paring round) that sort attributes according to the number of people who selected them as most important, and round 4 - the final ranking round. We can see a large shift in the ranking of two specific attributes - the accuracy of the knowledge and the extent of domain knowledge of the source. Smaller changes are also apparent in the ranking of the validity of the knowledge and the accessibility of the knowledge. This can be attributed to the 'no ties' constraint imposed in the ranking rounds, which forced participants into making trade-offs between attributes in rounds three and four. We intend to examine this trade-off further in the conjoint study as it represents more closely the real-life situation in which knowledge seekers are forced into weighing the importance of each attribute when making their knowledge selection decision. Discussion The Delphi study focused on the identification of meta-knowledge required by users of a knowledge management system in order to facilitate the design of the conjoint study. Nevertheless, the results are interesting in and of themselves. The results of the study indicate that knowledge seekers require a combination of knowledge and source attributes that provides both objective (e.g. currency) and subjective (e.g. willingness to help) information about the knowledge and the source of the knowledge. These results provide some meaningful insights on the nature of meta-knowledge required for the design of more effective KMS. Comparing the results in Table 10 to the literature reviewed throughout this dissertation indicates that some of the results are in line with previous literature on knowledge transfer and adoption while some new attributes have also been identified in our study. Most of the knowledge attributes were already identified in the meta-data literature (e.g. Basch, 1990; Dublin Core; Stoker and Cooke, 1995). However - our study reveals that there is a strong emphasis on using several different measures for the quality of the knowledge - accuracy, validity, and credibility - an emphasis that does not exist in previous literature. With regards to the knowledge source, some of the source attributes have been previously discussed in the literature, for example the trustworthiness and experience of the source are two attributes that have been identified in the knowledge transfer and related literature (Hovland, Janis, and Kelly, 1953; Kale, Singh, and Perlmutter, 2000; Petty and Cacioppo, 1986). An interesting comparison can also be made between the trustworthiness of a knowledge source and the trustworthiness of a seller in consumer-to-consumer (C2C) electronic commerce, if we 61 refer to the source as a seller of knowledge and the receiver as the buyer (as some commercial KMS, such as Knexa, do). In such settings, Strader and Ramaswami (2002) showed that the trustworthiness of the seller was second only to price in the purchase decisions of buyers in a C2C marketplace. In addition to trustworthiness, Borgatti and Cross (2003) note that the extent to which a person will seek information from another person depends on the seeker's evaluation of the source's knowledge and skills in the domain. This observation is in line with the identification of the importance of the extent of the source's knowledge as important in our results. The Delphi study reveals two new attributes that were not strongly emphasized in past literature. The first - communication skills of the source - indicates the need to identify whether the source is capable of communicating the knowledge. This result somewhat ties to Szulanski's (1996) work on predictors of knowledge stickiness (inability to transfer knowledge) that identified the lack of existing relationship between the source and recipient as an inhibitor of knowledge transfer. Szulanski identified that part of this relationship involves the ease of communications between the source and recipient. A similar discussion of importance of pre-existing relationship that facilitates better communications and thus more successful knowledge transfer appears in Hansen (1999). An additional contribution of the Delphi study is that it identifies the importance of the awareness of the source to other resources. We are not aware of studies that identified the requirement to know whether the source is aware of other resources in case he or she does not have the answer sought, in the context of meta-knowledge. This result is especially interesting since Cross, Rice, and Parker (2001) identified pointers to other knowledge as one of five benefits that people derive from a knowledge search. Our study shows that people actually use this criterion as a prerequisite to the search. This result also emphasizes the importance of building an underlying network model in the design of KMS. As Alavi (2000) defines, KMS that are based on the network model are intended to create links and interactions between individuals to facilitate the sharing of knowledge. Such systems can therefore provide information on the amount of resources (network links) available to each of their members. Identifying the correct attributes of a specific product is a crucial step in the design of conjoint surveys (McCullough, 2002). Based on the results attained from the Delphi study we are confident in the importance of the attributes identified by panel members as well as in the completeness of the set of attributes. Therefore, these results strengthen the validity of the 62 conjoint study. We continue this chapter with a description of the specific design of the conjoint study. Conjoint study design Overview The conjoint study was conducted as a web based survey and participants were solicited from three sources: organizations, MBA students, and Commerce undergraduate students. Prior to posting the survey we conducted three pilot studies to test both the questionnaire as well as the technology. Each pilot study included four respondents and some changes were made after the first pilot. These changes are incorporated in the detailed description of the design process below. The second and third pilots included respondents from the target populations of the study - professionals, MBA students, and undergraduate students. The general task in the conjoint study was similar to that of the Delphi study - identify the most important knowledge and source attributes that you consider when seeking knowledge. To ease the task we provided respondents with a specific knowledge search example (task) to refer to when answering the questionnaire. To maintain external validity we used four different examples13: • "UBC has decided to consult with students and industry partners in the admission of new MBA students. You are asked to review applications and recommend the most suitable applicants in your opinion. In your decision you are requested to take into account UBC's admissions policies and standards. To assist you in this decision you can ask the opinion of any one of 20 people who indicated that they have some knowledge on the topic." • "Due to recent security problems in your company, you are asked to devise a detailed policy for company Internet usage. To assist you in your task you can ask the opinion of any one of 20 people who indicated that they have some knowledge on the topic." Unlike in the Delphi study, the choice of these examples is not intended to manipulate levels of equivocality and expertise. The main goal of the conjoint study is to identify the relative importance of the various knowledge and source attributes and thus the use of different example is intended to ensure external validity of these findings by presenting individuals with different scenarios. 63 • "You have an idea for a new company. In order to contact potential investors you need to prepare a business plan describing your idea. To assist you in your task you can ask the opinion of any one of 20 people who indicated that they have some knowledge on the topic." • "You decided to pursue an MBA degree to continue your education. You now have to select what universities you should apply to (where would be the best place to study, who has the best reputation, etc.). To assist you in this decision you can ask the opinion of any one of 20 people who indicated that they have some knowledge on the topic." The specific instructions given at the beginning of the questionnaire were: "In this study we ask for your help in evaluating knowledge and knowledge sources. The context of the study is a knowledge search task, that is - you will be given a specific question and will be asked to choose the knowledge source that you wish to consult in answering the question. There are two parts for this study -one part deals with evaluating the trade-offs between the characteristics of different knowledge sources - people that you will consult with, and the other part deals with evaluating the trade-offs in the characteristics of the knowledge underlying the answer to the question provided. Before beginning the evaluation of the sources and knowledge, we would like you to familiarize yourself with the knowledge search task described below. '"4 Note that the instructions distinguish between knowledge and source attributes. The study was conducted as two separate conjoint surveys — one for the knowledge and one for the knowledge source - due to the limitation of the chosen data collection method to deal with a large number of attributes, as explained below. Selection of data collection method Data collection methods in conjoint studies refer to the method of presenting the stimuli to respondents and the elicitation of preferences. There are three common data collection methods: the trade-off method, the full profile method, and hybrid conjoint models. The trade-off method presents respondents with two attributes at a time and asks to rank the two. It is simple to administer but has many limitations (Green, Tull, and Albaum, 1988). A much more popular 1 4 The full questionnaire is attached in appendix C 64 method is the full profile method (Green and Srinivasan, 1990; Hair et al., 1992). In the full profile method respondents are presented with combinations of all the attributes and are asked to rank or to rate these combinations. However, the full profile method is limited when more than eight or ten attributes are involved. In this case respondents experience information overload and are likely to make their selection based only on a partial combination of attributes. Hybrid conjoint models (Green, 1984) were developed to facilitate conjoint analysis with a large number of attributes. Hybrid conjoint models are so called since they combine a self-explicated compositional model followed by a de-compositional conjoint model. In the first stage respondents are asked to state - for each attribute - the desirability of the various attribute's levels to them, as well as the importance they assign to this attribute. Multiplying the two indicators yields the value of each combination of attributes and their levels. At the second stage respondents evaluate a small set of three to nine stimuli using full profile conjoint analysis to further refine the derived values. Hybrid conjoint models are not as powerful as the full profile method mainly because of the self-explicated component they entail. In a study by Green (1984) the full profile method proved to have better cross validity than the hybrid method in five out of the seven cases examined. The results of the Delphi study indicate that fourteen attributes are important to knowledge seekers in searching for knowledge. This implies that in order to use the full profile method we must reduce the number of attributes presented to respondents. The Delphi results also indicate a separation between the ranking of knowledge attributes and source attributes. This result enables us to adopt the full profile method as well as use all the attributes identified by performing two separate conjoint surveys - one for the knowledge (six attributes originally identified) and one for the knowledge source (eight attributes identified). To ensure that no correlation exists between some attributes that might cause unrealistic profiles (attribute multicollinearity) we conducted interviews with six Delphi panel members. We asked respondents to identify attributes that they thought were correlated. Most responses indicated combinations of knowledge and source attributes such as 'currency of the knowledge' and 'the source is up-to-date'. Since we used two different surveys - one for knowledge and one for source - no attribute was eliminated. Within the knowledge attributes however, two judges combined credibility and validity and one other judge combined accuracy and validity. Since this was a small number of judges we did not group any of the attributes at first. Nevertheless - since attribute multicollinearity may harm conjoint results by creating unrealistic stimuli - we tested this further in a first pilot study. Four Commerce graduate students 65 completed a paper version of the conjoint survey and responded to some questions after completing the survey. The respondents indicated some unrealistic combinations of accuracy, credibility, and validity and therefore - based on the feedback from respondents - the attribute 'credibility' was eliminated from the questionnaire. In addition - the term 'validity' was changed to 'supported' to better represent the definition 'the knowledge can be verified and supported by other sources or by empirical results' (see Table 9). Apart from 'credibility', the attribute 'accessibility of the knowledge' was also eliminated after the interviews. Respondents felt that 'accessibility' was more of a constraint than an evaluation factor. If the knowledge were not accessible then they would not consider anything about it and therefore 'not accessible' was not perceived as a realistic case. The final set of attributes included five knowledge attributes and seven source attributes, shown in Table 11. We describe the selected levels of these attributes next. Identifying attribute levels 'Attribute levels' represent specific values that an attribute can have. For example - the attribute 'price' may have levels such as '$5', '$7', or '$15'. We initially explored the levels of attributes in the interviews with Delphi panel members. The results are summarized in Table 11 1 5. Attributes # of levels selected Measures Relevance of the knowledge 3 Percent relevance o Accuracy of the knowledge 3 Percent accurate s> CD Support for the knowledge ('supported') 3 How many sources support or refute it o. CQ CD Currency of the knowledge 3 Time since created (in years) Extent of knowledge 2 Comprehensiveness Experience of source 3 Years of experience 7\ Extent of domain knowledge of the source 3 High/Medium/Low awledge soi Source is up-to-date 2 Yes/No awledge soi Trustworthiness of the source 2 High/Low awledge soi Source is willing to help 2 High/Low - i o CD Communications skills of the source 3 Good/Reasonable/Poor Source is aware of other resources 2 Yes/No Table 11: First interview results 1 5 The full interview results are described in appendix C 66 For five of the attributes - relevance, accuracy, experience, currency of the knowledge, and support for the knowledge - we asked respondents to provide specific values (numbers) for the levels. Strong agreement was attained for the three levels of 'years of experience' and they were set at 10 years/5 years/ and 2 years. Similarly the three levels of support were set at 'the knowledge is supported by 2 or more sources'; 'the knowledge is neither supported nor refuted by any sources'; and 'the knowledge is refuted by one or more sources'. However, the selected levels of relevance, accuracy, and currency of the knowledge differed between judges and therefore we conducted another set of interviews to determine those levels. In the second interviews - conducted with nine Gommerce PhD students - we elicited the values for the three levels of currency, accuracy, and relevance for each of the examples used. The results of these interviews are summarized in Table 12 1 6. Example Currency (years) Accuracy(%) Relevance(%) MBA admission <1 3 >10 95 70 30 95 70 30 MBA selection <1 3 >10 95 70 30 95 70 30 Internet policy 6 months 2 >6 95 70 30 95 70 30 Business plan 6 months 3 >10 95 70 30 95 70 30 Table 12: Second interview results Table 13 summarizes the full set of attributes and levels used in the study, and an example of the actual profiles that were presented to respondents is shown in Figure 10. The full interview questionnaire and results are described in Appendix C 67 Source attributes Knowledgeable Very knowledgeable Somewhat knowledgeable Not knowledgeable Trustworthiness High Low Awareness of other resources Yes No Experience 10 years or more 5 or 6 years 2 years or less Willingness to help High Low Communications skills Good Reasonable Poor Up to date Yes No Knowledge attributes Supported 2 or more source support this knowledge No sources support or refute this knowledge 1 or more sources refute the knowledge Extent Comprehensive Not Comprehensive Accuracy 95% 70% 30% Relevance 95% 70% 30% Currency (*Depending on the example used) Less than 6 months old / or Less than 1 year old 2 years old / or 3 years old Over 6 years old / or Over 10 years old Table 13: Levels of attributes used in the survey Profile 1a Knowledgeability Trustworthiness Awareness of other resources Experience Willingness to help Communications skills Up to date Somewhat knowledgeable Low No 5 or 6 years Low Good Yes ° 1 ° 2 O Ci 3 4 Ci c 5 O 6 7 o o 8 c n 9 10 " 11 I am not at all likely to use this source I may or may not use this source I would certainly use this source Profile 1 b ' " • " • „ ' [' : • Supported Extent Accuracy Relevance ( Refuted by 1 or more sources Comprehensive 70% 30% Over 6 years old O O CJ 1 2 3 Ci o 4 5 n n c 6 7 o o 8 9 Ci 10 11 I am not at all likely to use this knowledge I may or may not use this knowledge I would certainly use this knowledge Figure 10: Examples of conjoint profiles (Profile 1a - knowledge source; Profile 1b - knowledge) 68 After setting the number of attributes and levels, the other decisions in designing the conjoint study concern the construction of the stimuli set and preference model, and the selection of the statistical model. We describe these steps next. Stimulus set construction The number of stimuli (or profiles) used in the conjoint analysis is derived from the number of attributes and levels within each attribute. For example, in a study with three attributes with two levels each the total number of stimuli is eight (2x2x2). A design that includes all possible combinations ofattribut.es and levels is called a full factorial design. As the number of attributes and levels grows the number of stimuli in the full factorial design also grows and increases the burden on respondents. A fractional factorial design enables the use of an orthogonal subset of all possible stimuli. We created a fractional factorial design using SPSS and the result was 20 profiles for each of the two parts of the survey (knowledge and source). The evaluation of the stimuli by respondents can be carried out by ranking or rating. Ranking is more reliable but much more difficult to administer, especially with a large number of stimuli (Green, Tull, and Albaum, 1988). We therefore used a metric rating scale of respondents' evaluation of the knowledge and knowledge sources. Selection of preference model The preference model represents the preference structure of the respondents and consists of a composition rule and a part-worth relationship. The composition rule describes how respondents combine the separate attribute utilities to obtain the overall value of the product. The part worth relationship represents the relationship between different levels within the same attribute. The most common composition rule used is the additive (main-effect) model in which respondents simply add the separate utilities to attain an overall value of the product (Hair et al., 1992). A different composition rule is the interactive effects model in which interactions between two or more attributes are possible. Empirical evidence shows that models with interaction terms lead to lower predictive validity (Green and Srinivasan, 1990). In addition, these models require a much larger set of stimuli (profiles to be evaluated) and this is not effective with a large set of attributes as is the case for this study. Therefore, although there might be some interactions between attributes in our study we are constrained by the number of attributes used to employing the additive model. Nevertheless, we did not find any theoretical argument for the existence, or the lack of such interactions. 69 There are three possible part-worth relationships in conjoint analysis, shown in Figure 11. Figure 11: Types of relations between attribute levels The most restrictive - the linear form - assumes a single part worth that is simply multiplied by the level's value. The least restrictive - the separate part worth (or discrete) - allows every level to have its own estimate. A third relationship, the quadratic, or ideal-point, model falls in between the two. The most commonly used relationship is the separate part worth (Green and Srinivasan, 1990) and since it is also the least restrictive one we will use it in this study. The linear and quadratic models can be special cases of this relationship. Estimation method Our selection of the metric rating method for the measurement of the dependent variable implies that we will use a least squares regression analysis as our estimation method. This type of method is common in many conjoint studies and was used in over 50% of conjoint analysis studies evaluated by Wittink and Cattin (1989). In addition, in order to identify potential segments within respondents that have similar preference structure towards knowledge and knowledge sources we will use cluster analysis methods on the conjoint results (Green and Srinivasan, 1978; SPSS, 2001). Finally, we will use multinomial logit regression to analyze the characteristics of members of these clusters (Demaris, 1992; Powers and Xie, 2000). Reliability and Validity Empirical evaluations of conjoint analysis studies provide evidence for high reliability and validity ofthe method in general (Gegaxand Stanley, 1997; Green and Srinivasan, 1990). Specific tests to evaluate the reliability and validity of a conjoint study include test-retest reliability and holdout samples (Wittink and Cattin, 1989; Green, Tull, and Albaum, 1988). Test-retest reliability is conducted by including several repeated stimuli for each respondent. These stimuli are evaluated using a correlation coefficient or by using a special F test, known as the Chow test, to identify differences between the two replicated stimuli. Internal validity is evaluated using a 70 holdout sample. This sample includes specific stimuli - drawn from the full range of stimuli - that are evaluated by respondents and compared to the prediction of the model developed in the analysis. We include four holdout profiles in our survey17 to ensure validity. In addition - we include one profile that is constructed from the highest levels of all attributes. We expect this profile to be highly ranked by respondents. Participants Three main groups of participants were included in the study - professionals (42), MBA students (36), and fourth year Commerce undergraduate students (72). Table 14 describes the demographics of participants in the study. Undergraduate students represented the novice population in companies such as new employees. We limited the study to fourth year students who will soon join the job market and will be looking for knowledge to assist them with their jobs. Conjoint analysis enables identifying different market segments and therefore we will be able to isolate the opinions of the undergraduate students from those of the more experienced MBA students and professionals. Since the study was web-based and required a userlD and password, we recruited participants in advance. To recruit professionals to participate in the study we posted an email on a mailing list of the Knowledge Management Consortium, a KM interest group18. Of about 60 or 70 members on the mailing list 20 (-30%) responded to the email. In addition, we contacted nine organizations of which five agreed to participate and between one to five people from each of these companies completed the study. Of 46 professionals that agreed to participate in the study, 42 completed the survey (91%). Of 45 MBA students that signed for the study, 36 completed it (80%). And finally - of 89 undergraduate students that signed for the study, 72 completed it (81%). Of a total of 150 responses 148 were usable. The final questionnaire is presented in Appendix C. The questionnaire includes three parts - an introduction and demographics questions, a conjoint survey for knowledge sources, and a conjoint survey for knowledge. The presentation order of the two conjoint surveys was randomized across participants. Finally - in responding to the questionnaire participants were asked to review the knowledge search task to provide a context for their responses. They were randomly assigned to one of the four tasks described earlier. We present our findings next. 1 7 The four holdout questions for the knowledge attributes are profiles number 3,4,7, and 13 in Appendix C . The holdout questions for the source attributes are profiles number 2,8,14, and 15. 1 8 See Appendix C for the email posting 71 Professional & MBA Undergraduate N=148 students (N=78) students (N=72) Age (average) 30-40 20-30 Level of education Some college 2 20 Bachelor 25 47 Masters of higher 47 1 Professional certificate 4 1 Other 0 2 Years working 14 (Range: 1 to 35) 1 (Range: 0 to 9) Years with company 7 (Range: 0.3 to 33) 2* (Range: 0.3 to 4) Years in current position 4 (Range: 0.25 to 31) 2* (Range: 0.3 to 4) Organization size* Large (Over 1000) 26 2 Medium (500 to 1000) 8 4 Small (100,-to499) 16 6 Very Small (Less than 100) 19 10 Industry Banking and Financial Services 9 4 IT 14 0 Consulting 5 0 Transportation 8 2 Retail 1 6 Education 14 50 Other (Legal; Biotech; Chemical; Environmental; Forestry; 27 10 Government; Non-profit; Health care; hospitality; media) Knowledge use Frequency of using external knowledge in your work Often 60 36 Sometimes 17 29 Rarely 0 4 Never 0 3 Average number of knowledge sources used 3 2.5 Percent of panel members that used each source: People 73% 79% Books 63% 56% DB 50% 29% Web 95% 82% Other (professional publications; internal manuals and 15% 1% documentation; seminars and workshops; help pages) Familiarity with knowledge source Very familiar 40 28 Somewhat familiar 37 47 Not familiar 1 2 Familiarity with the concept of KMS Yes 22 4 Somewhat familiar 33 32 No 23 36 If yes, what KMS are you currently using? Intranet 74% 21% Lotus Notes 30% 8% Best Practices database 17% 4% ERP 13% 3% Other (in house developed systems; commercial KMS such 20% 6% as Groove, Crystal knowledge base, and others) * Only for those students who are working in addition to school Table 14: Demographics of conjoint study participants 72 Chapter 6: Findings This chapter describes the results from the conjoint study. Specifically we examine the part-worth utilities as well as the importance weights that respondents placed on each of the attributes of knowledge and of knowledge sources. In addition to the conjoint results, we employ cluster analysis methods to further explore the data and to identify factors that potentially affect respondents' preferences towards a specific piece of knowledge or towards a specific knowledge source. Interpreting the conjoint analysis results The results of a conjoint survey provide detailed information on the preference structure of each of the participants in the study. Figure 12 presents an example of the SPSS conjoint output for one of the respondents. Two main measures are provided for each respondent: the utility estimates of the different levels of each attribute with their standard errors, and the importance weights of the attribute. The utility estimates indicate the relation between each of the attributes' levels and the preferences of the individual. Positive utility means that the attribute's level is positively related to preference (e.g. to the individual represented in Figure 12 - the more sources that support the knowledge the more preferable it becomes). The importance value for each attribute is calculated by dividing the range of the attribute's utilities by the sum of ranges of all the attributes. For example, to calculate the importance of the attribute 'support' in figure 12 we first calculate the utilities range for this attribute, which is 3.5, and then divide it by the sum of utility ranges of all the attributes, which is 10.75 (3.5+0.625+2.125+1.75+2.75). This indicates that the importance of the attribute 'support' is 3.5/10.75=0.3256, or 32.56%. Thus, importance values represent a measurement of the relative contribution of each attribute to the overall evaluation of a specific piece of knowledge or a specific knowledge source. For example, in Figure 12 the 'support' has the highest importance value and therefore it is the most dominant attribute in determining the evaluation of knowledge by the individual represented in Figure 12. The reliability of the conjoint model is evaluated using Pearson's correlation coefficient (R) and Kendall's tau (coefficient of rank correlation), both measures are calculated between the actual profile evaluations (ratings) provided by respondents and the profile evaluations provided by the conjoint model based on the calculated attributes' utilities. These measures represent the strength of association between observed profile ratings and their predicted scores from the conjoint model. Finally, validity is measured by the inclusion of holdout questions - questions that the respondents rate but are not included in the conjoint estimation. Kendall's tau is 73 calculated for these holdout questions to test agreement between observed profile ratings not used to calculate the conjoint model and estimated ratings from the model. ISUBJECT NAME: 19 Importance U t i l i t y ( s . e . ) Factor SUPPORT 32 . 56 5 .81 19 . 77 16.28 25 . St -2.0833( .5917) .6667( .5917) 1.4167( .5046) .3125( .3125( -1 ,2083 ( .2917 ( , 9167 ( ,9167( , 8333 ( . 0833 ( -1.0000 ( - .7500 ( 1 . 7500 ( Pearson's R Kendall's tau iKendall's tau .3785) .3785) .5917) .5917) .5046) .5917) . 5917) .5046) . 5917) .5917) . 5046) EXTENT ACCURACY RELEVANCE CURRENCY 6.1458( .4548) CONSTANT . 924 .802 1.000 f o r 4 holdouts 1 or more sources refute the knowledge No sources support or refute the knowledge 2 or more sources support the knowledge Not Comprehensive Comprehensive Low Medium High Low Medium High Low Medium High S i g n i f i c a n c e = .0000 Si g n i f i c a n c e = .0000 Si g n i f i c a n c e = .0208 Figure 12: Example of conjoint analysis output 74 Knowledge attributes Although the conjoint procedure provides detailed analysis for each respondent, it is often useful to examine the aggregates of these results. Figure 13 provides a summary (calculated as an average of the individual responses) of the importance ratings for the three participating groups of subjects: professionals (PRO), MBA students (MBA), and undergraduate students (UG). As can be seen both Pearson's R and Kendall's tau are high and significant indicating good agreement between the average profile rating and the profile score predicted by the conjoint model (SPSS, 2001). In addition, Kendall's tau for the holdout questions is also high and significant. The text summaries of the results as well as a graph showing the importance of knowledge attributes by question (knowledge search task) are presented in Appendix D. 3 5 % 3 0 % oj 2 5 % o c ro o 20%-CL E OJ 15%-ra i_ > < 10% 5% 0% HPRO UMBA DUG S U P P O R T EXTENT A C C U R A C Y R E L E V A N C E C U R R E N C Y P R O 17.58% 9.32% 27.00% 23.55% 22.55% MBA 19.08% 9.60% 30.18% 22.59% 18.54% UG 21.87% 9.75% 27.66% 22.76% 17.96% Column average (weighted) 20.00% 9.60% 28.08% 22.94% 19.38% Pearson's R = .992 S i g n i f i c a n c e = .0000 Kendall's tau = .917 S i g n i f i c a n c e = .0000 Kendall's tau = 1.000 f o r 4 holdouts S i g n i f i c a n c e = .0208 Figure 13: Knowledge attributes importance (by group) The data table shows that all groups rated accuracy as the most important attribute and relevance as the second most important attribute. The professionals' subgroup also assigned 75 high importance to currency but the MBAs and undergraduate students only rated currency fourth, after support. However, the differences in the importance weights between these two attributes are quite small. Finally, all groups rated the extent of the knowledge as the least important attribute. These results are summarized in result 1 below: Result 1: the group level analysis for the knowledge attributes shows that accuracy and relevance are the most important attributes in the selection of knowledge across all groups and extent is the least important attribute. However, there are subgroup differences in how the groups view the other two attributes. In order to learn more from the data gathered in the conjoint study and to identify potential segments in the group of respondents we conduct a secondary data analysis using cluster analysis techniques. Secondary data analysis - cluster analysis As mentioned earlier, cluster analysis is often used to identify market segments within the conjoint results. Specifically, we are looking for groups of people who assign similar importance values for similar attributes. Cluster analysis is a group of multivariate techniques aimed at identifying groups within the data and classifying them according to the characteristics of interest. These groups should have high homogeneity within groups and high heterogeneity between groups (Alenderfer and Blashfield, 1984; Hair et al., 1992). Cluster analysis can be used as a secondary data analysis for conjoint studies in order to identify subject groupings or segments that can the be tied to demographics and other characteristics of the respondents (SPSS, 2001; Wolfe and Putler, 2002). There are numerous ways to conduct cluster analysis. Here we will briefly discuss the specific methods selected for this analysis. There are two common clustering algorithms - hierarchical and non-hierarchical. Of the hierarchical clustering procedures the agglomerative methods are often used. These methods begin by putting each observation into a separate cluster and gradually joining together similar clusters, usually by examining the Euclidean distance between any two points (Hair et al., 1992). Of the various agglomerative methods, Ward's method is widely used in the social sciences (Alenderfer and Blashfield, 1984; Hair et al., 1992). This method joins clusters based on minimizing the within-groups sum of squares (squared distance between each point and the cluster's centroid). Non-hierarchical clustering procedures - also known as k-means methods -76 are iterative methods that require a predefined number of clusters. At the first stage seed points are determined as cluster centroids - based on the predefined number of clusters - and each observation is assigned to the cluster with the nearest centroid. Once all observations have been assigned, a new centroid is calculated for each cluster and the observations are distributed again between the clusters. These iterations continue until a good cluster solution is attained (based on within-groups sum of squares). To analyze the clusters within the conjoint data we use a combination of these two clustering methods (Hair et al., 1992). First, Ward's method is used to identify the optimal number of clusters. This number is then specified in the k-means method for the final cluster solution. We cluster the data based on the importance values of the five knowledge attributes, attained from the conjoint analysis. The distance calculated is therefore the distance between different individuals' importance scores assigned to various attributes. The dendrogram in Figure 14 shows the results of the Ward's procedure for the knowledge attributes. The Figure shows four potential clusters in the data - marked with the four arrows. 77 The corresponding centers for the four clusters (from the k-means cluster procedure) are presented in Table 15. The table shows that members of cluster 1 are especially concerned with the currency of the knowledge while members of cluster 2 focus more on the relevance of the knowledge. Similarly, members of cluster 3 are concerned with the accuracy of the knowledge and cluster 4 with the support for the knowledge. Comparing these cluster centers to the data presented in Figure 13 we can see that the four clusters are formed around the top four attributes identified in the conjoint analysis, meaning that for the knowledge attributes differences in how individuals view the top four attributes match the dominant attributes in the preferences towards a specific piece of knowledge for the groups (or segments) within respondents. Cluster Size Support Extent Accuracy Relevance Currency 1 37 0.14 0.12 0.22 0.18 0 .34 2 31 0.14 0.10 0.25 0 .35 0.17 3 32 0.18 0.07 0.41 0.23 0.11 4 48 0.30 0.09 0.26 0.19 0.15 Table 15: Knowledge clusters' centers In order to facilitate the visualization of the cluster assignments we draw a bi-plot, presented in Figure 15, based on the principal component analysis (Jolliffe, 1986; Wolfe and Putler, 2002)19. In Figures 15a and 15b cluster assignments (1, 2, 3, or 4) are plotted along the first three principal components of the data, which account for almost 90% of the variance in the data. The data used for the analysis are the importance values of the five attributes for the 148 respondents in the study. From Figure 15a we can see that the main difference along the first principal component is between currency and the other three main attributes (support, accuracy and relevance). We can characterize this first partition of the data as between people that are 'timely' - people who are concerned with the currency of the knowledge - and people that are 'cautious' - people who care more about the correctness of the knowledge. The next contrast is along the second principal component between 'support' and 'accuracy' and 'relevance'. A possible interpretation here focuses on the difference between the need for external validation of the knowledge 1 9 The principal component analysis is only used here as a diagnostic tool - to enable the better visualization of clusters and to examine the main difference between the attributes in the four clusters. 78 (support) as opposed to more 'self-contained' measures (accuracy and relevance). Finally, the third principal component (Figure 15b) shows the contrast between 'accuracy' and 'relevance'. ,0:5' AGC 3 •0.0 0.5 1.0 4; ;REL -0:2 .-'-;o;i -0.0; 0 fl' 0':2 'CbmpS: Each point in the figures above represents one person who responded to the questionnaire. The numbers indicate the cluster membership of that person. The horizontal and vertical axes indicate the principal components in the data and the dispersion of the points on the graph is based on the loadings matrix shown below. Comp. 1 Comp. 2 Comp. 3 Comp. 4 Comp. 5 SUPP -0.3148370 0.80146728 0.07545669 0.22986940 0.4472072 EXT 0.1496409 0.04935185 0.11074354 -0.87346080 0.4471846 ACC -0.4473517 -0.37459478 -0.67639682 0.04538585 0.4471942 REL -0.1891084 -0.46338918 0.69473997 0.25848166 0.4472495 CURR 0.8016232 -0.01279891 -0.20461150 0.33963751 0.4472325 Figure 15: Bi-plots for knowledge clusters solution (15a: comp. 1-2; 15b: comp.2-3) Result 2 summarizes the above discussion: Result 2: the cluster analysis shows that four clusters can be formed from the importance weights of the knowledge attributes. In addition, these clusters are formed around the four most important attributes (from Figure 13) and can be therefore termed: currency, relevance, accuracy, and support. The analysis also shows that the main contrast in the data is between 'currency' and the other three attributes, indicating that respondents can be characterized as either caring about the timeliness or the correctness of the data. The latter group can be 79 further divided into those who focus on measures within the knowledge itself and those who require external validation. Characterizing clusters' members Our final analysis, before moving to discuss the knowledge source, is an attempt to characterize the members in each cluster based on the demographics measures collected earlier (table 14). To better understand what factors affect cluster membership we use multinomial logit regression analysis (Demaris, 1992; Powers and Xie, 2000) since the dependent variable - cluster membership - is categorical. The dependent variable is therefore the cluster membership of respondents and the independent variables are their demographic characteristics (e.g. years of experience, organization size, etc.) and the example they were assigned (three dummy variables were created to represent the four examples), as described below: Pro A dummy variable indicating whether the respondent is a professional or an MBA student Q1, Q2, Q3 Dummy variables for the specific example the respondent referred to Y_Exp The years of experience of the respondent no_sou Number of knowledge sources that respondents usually consult with sou_fam The level of familiarity of the respondents with the knowledge sources no_KMS The number of KMS that the respondent is familiar with k_freq The frequency in which respondents use external knowledge org_size The size of respondents' organizations Expertise The expertise of the respondent in the domain of the example used K_weight The relative weight the respondent placed on knowledge as opposed to knowledge source For this analysis we focus on the professional and MBA subgroups since the undergraduate subgroup is highly homogenous in the demographics characteristics (age, education, etc.) as well as missing some characteristics of interest such as 'years of experience' or 'organization size' 2 0. The results of the regression are presented in Table 16. The clusters were recreated for the subgroup analyzed. 80 Full Model Chi 2 (36df) 69.3738 Sig. 0.005 Baseline cluster - Currency Relevance Cluster (S.E.) Support Cluster (S.E.) Accuracy Cluster (S.E.) Chi 2 (3df) Sig. (Intercept) -2.45 (3.479) 3.17(3.924) 5.20 (4.005) Professional 0.32(1.637) 3.38 (1.514) 0.03 (1.546) 9.084 0.05 Question 1 - MBA admissions 1.46 (1.169) 0.57 (1.334) 1.56 (1.437) 2.1819 Question 2 - Internet usage policy 0.22(1.036) 1.45(1.128) 1.87 (1.357) 3.2468 Question 3 - Business plan 2.58 (1,333) 2.88 (1.405) 3.66 (1.570) 8.8982 0.05 Experience 0.01 (0.068) -0.22 (0.087) -0.06 (0.085) 12.1633 0.01 # of sources 0.29 (0.332) -0.10 (0.370) -0.04 (0.370) 1.6209 Source fam. -0.12(0.791) -0.79 (0.856) -0.80 (0.893) 1.451 # of KMS -0.35 (0.415) -0.10(0.433) -0.89 (0.512) 4.7667 K. Frequency 1.03 (1.139) 1.45 (1.312) -0.02 (1.348) 2.2644 Org. size -0.14 (0.333) -0.61 (0.363) -0.60 (0.379) 19.0676 0.005 Expertise -0.02 (0.413) -0.79 (0.488) -0.43 (0.473) 3.8241 Knowledge weight 0.01 (0.024) 0.01 (0.025) -0.01 (0.026) 3.7661 Table 16: Multinomial logit regression on knowledge clusters Two very interesting results observed from Table 16 are the significance of question 3 and of organization size. Question 3 is the business plan task - respondents were asked to search for knowledge on how to write a business plan to attract potential investors. The results show that respondents to question 3 were least likely to be in the 'currency' cluster and most likely to be in the 'accuracy' cluster. The three dominant attributes in the 'accuracy' cluster are accuracy, relevance, and support, which indicate that respondents felt that for the purpose of writing a business plan to attract potential investors the correctness and relevance of, and support for, the knowledge are more important than its currency. More generally this indicates that the relative importance of the various knowledge attributes in the knowledge selection decision depends on the context of the knowledge search. The other significant variable - organization size - indicates that people from larger organizations will tend to belong to the 'currency' cluster more than any of the other three. The result also shows that organization size increases the odds of being in the 'relevance' cluster than in the 'accuracy' or 'support' clusters. One potential explanation is the information overload problem in many large organizations (as KPMG (2000) reported - 65% of the organizations surveyed reported information overload as a main problem in the organization). Information 81 overload might enhance the importance of relevance and currency as opposed to the other attributes. 'Years of experience' also had a significant effect on cluster memberships albeit a small one. Specifically people with more years of experience are least likely to be in the 'support' cluster. A comparison of the clusters shows that the largest difference between the 'support' cluster and the other three is in the support attribute, thus if we consider years of experience as some proxy for expertise, this result indicates that experts are more likely to judge the knowledge themselves rather than rely on the opinions of others. Finally, the results of the regression also show that there is a significant difference between the professional and MBA subgroups in the cluster assignment. The professionals are most likely to be in the 'support' cluster and least likely to be in the 'currency' cluster (and the opposite for the MBA students). This result might be attributed to the sample used. For example, there are two more demographic variables that were collected in the study - 'years in position' and 'years with the company' but were not included in the regression since MBA students mostly did not respond to these questions since they are currently in school. The significant differences between the groups might be a result of these omitted variables. Result 3 summarizes the conclusions from this analysis: Result 3: the results of the multinomial logit regression indicate the context of the knowledge search has a significant effect on the relative importance of knowledge attributes. In addition the results provide some indication to information overload problems in large organizations. Finally, the results indicate that longer experience significantly reduce the likelihood of being in the 'support' cluster. We have so far focused on ranking the attributes of the knowledge and assessing how they affect the likelihood of selecting a specific piece of knowledge. We now turn to analyze the results of the knowledge source attributes. The analysis will follow the same steps explained previously. Source attributes Figure 16 provides a summary of the importance rates of knowledge source attributes for the three participating groups of subjects: professionals (PRO), MBA students (MBA), and 82 undergraduate students (UG). The text summaries of the results as well as a graph showing the importance of knowledge attributes by question (knowledge search task) are presented in Appendix D. CD O c CD •c o CL E CD O) CO CD > < 30% 25% 20% 15% 10% 5% 0% m p R O El M B A OUG I t EXTENT TRUST A W A R E OF RES . EXP. WILL. TO HELP C O M M . SKILLS UP-TO-DATE PRO 25.51% 20.01% 8.18% 9.92% 12.18% 13.51% 10.69% MBA 25.92% 19.41% 9.06% 14.26% 11.28% 14.05% 6.01% UG 25.27% 18.68% 8.50% 11.86% 12.35% 14.15% 9.19% Column average (weighted) 25.49% 19.23% 8.55% 11.90% 12.04% 13.95% 8.84% Pearson 1 s R Ke n d a l l 1 s t a u Ke n d a l l 1 s t a u . 974 . 900 .333 f o r 4 holdouts S i g n i f i c a n c e = .0000 S i g n i f i c a n c e = .0000 S i g n i f i c a n c e = .2485 Figure 16: Knowledge source attributes importance (by group) As before, both Pearson R and Kendall's tau are high and significant indicating good agreement between the conjoint model estimates and the observed ratings. However, for the source attributes, Kendall's tau for the holdout questions is low and insignificant. With only four holdout questions Kendall's tau is sensitive to reversals in the data (SPSS, 2001). A reversal means that a specific level that was expected to have a positive (or negative) affect on preference had - in fact - a negative (or positive) affect. An examination of reversals in the source attributes data indicates that most reversals occurred within the attribute ' y e a r s 0 T experience'. Specifically, respondents seemed to divide into three groups: one group perceived, less years of experience as better than more, giving high utilities for the lower levels and low utilities for the higher21. A 2 1 These were mostly the undergraduate students 83 second (smaller) group assigned the lowest utilities to the mid level ('5 or 6 years'). Finally - the third (largest) group preferred more to less years of experience. This reversal in the attribute 'years of experience' might explain the low Kendall value for the holdout questions. Despite this low tau for the holdout questions, the overall Pearson's R and Kendall's tau values are high and significant and we therefore proceed with our analysis of source attributes. The data table in Figure 16 shows that all groups rated the extent of source's knowledge as the most important attribute of a knowledge source, followed by the source's trustworthiness. The professionals and undergraduate students rated communication skills and willingness to help as the next important attributes, while the MBA subgroup rated experience before these two attributes. Finally, the source is up-to-date and awareness of other resources were rated as less important by all groups. Comparing this last finding to the knowledge attributes we can see that the 'currency' of knowledge was ranked higher than 'the source is up-to-date'. 'Currency' refers to the timeliness of the specific knowledge while 'source is up-to-date' refers to the general awareness of the source for new developments and innovations in the domain. The results indicate that this was not as important as the actual date of the knowledge sought. Our results are summarized in result 4 below: Result 4: the group level analysis for the knowledge attributes shows that extent of knowledge and trustworthiness are the most important attributes in the selection of a knowledge source across all groups. Awareness of other resources and up-to-date are the least important attributes for the MBA and undergraduate groups, experience and awareness of other resources were the least important attribute for the professional subgroup. There are a few differences in how the groups view the other attributes - the main difference is in the rating of the attribute 'experience'. The MBA group perceived this attribute to be more important than did the other two groups. In order to learn more from the data gathered in the conjoint study and to identify potential segments in the group of respondents we conduct a secondary data analysis using cluster analysis techniques. 84 Secondary data analysis - cluster analysis The dendrogram in Figure 17 shows the results of the Ward's procedure for the knowledge source attributes. The Figure shows four potential clusters in the data - marked with the four arrows. Source Dendrogram (full sample) Figure 17: Dendrogram for knowledge source clusters The corresponding centers for the four clusters (from the k-means cluster procedure) are presented in Table 17. The table shows that the cluster solution for the source attributes is not as tied to the conjoint result as the knowledge clusters. Based on the clusters' center we identify that members of cluster 1 are especially concerned with the extent of the source's knowledge, members of cluster 2 focus on the source's willingness to help and trustworthiness, members of cluster 3 are concerned with the trustworthiness and extent of knowledge of the source, and members of cluster 4 with the communication skills of the source as the well as the extent of the source's knowledge. 85 Cluster Size Extent of knowledge Trust-worthiness Aware of resources Experience Willingness to help Communica-tion skills Up-to-date 1 28 0.41 0.13 0.06 0.11 0.09 0.13 0.07 2 31 0.15 0.20 0.10 0.09 0.22 0.14 0.11 3 46 0.26 0.31 0.07 0.11 0.09 0.11 0.05 4 43 0.22 0.11 0.10 0.16 0.10 0.18 0.13 Table 17: Knowledge source clusters' centers As in the previous analysis of knowledge attributes, Figure 18 provides a more qualitative interpretation of the cluster solution. As in the previous section, cluster assignments (1, 2, 3, or 4) are plotted along the first two principal components of the data, which account for almost -1-.5 2 , A. AAA- 1 ' V -A:ff- 1 % '4 -Qi? '-P;;-l, Q:o •Cbmpvl 60% of the variance in the data. 0:1 0:2 Figure 18: Bi-plot for knowledge source clusters solution. We can see that the main difference along the first principal component is between the extent of the source's knowledge and the source's willingness to help. The contrast along the second principal component is between the source's trustworthiness and all the other attributes. Based on these results we can name cluster 1 as 'knowledge' - people who are concerned with the knowledge of the source, cluster 2 can be named 'personality' people who care more about the personal attributes of the source than about their professional abilities, cluster 3 can be named 'competence' - focusing on trust and knowledgeability, and cluster 4 'communications'. 86 In order to gain some insights as to the different clusters in the source attributes, it is interesting to examine - at this point - one other piece of information that respondents provided at the end of the questionnaire. We asked respondents to divide 100 points between knowledge and knowledge source based on their importance to them. The specific question was presented to respondents as depicted in Figure 19: Before you logout of the study please answer this last question: In the first page we introduced the knowledge search task and said that to assist you in your task you may turn to any one of 20 people that indicated that they have some| knowledge on the topic. In selecting a person to turn to - would you place more importance on the characteristics of the person (the knowledge source) or on the characteristics of the knowledge itself? To answer this question please divide 100 points between the two options below based on their importance to you. For example, if you only care about the characteristics ofthe knowledge then write '100' next to "The knowledge" and '0' next to "The knowledge source". If the two are equally important to you then write '50' next to "The knowledge" and '50' next to "The knowledge source", and so on. The knowledge; I JThe knowledge source Figure 19: Screen shot of questionnaire - knowledge/source importance question Table 18 compares the source cluster assignments and the knowledge/source importance data from the questionnaire. This data shows that 68% of the people in cluster 1 - the 'knowledge' cluster - generally place more importance on the knowledge than on the knowledge source. This indicates that the contrast along the first principal component in Figure 16 can be perceived as knowledge vs. source, since 'extent of knowledge' is the source's attribute that is most closely related to the knowledge itself. That is, it depicts the knowledge of the source rather than personal attributes such as trustworthiness or willingness to help. Importance of knowledge and source Source is more Knowledge is more Cluster important Equal importance important Grand Total 1 - 'knowledge' 7 2 19 (68% of row total) 28 2 - 'personality' 10 5 16 31 3 - 'competence' 20 6 20 46 4 - 'communication' 10 10 23 43 Grand Total 47 23 78 148 Table 18: Knowledge source clusters vs. knowledge/source importance weights 87 Result 5 summarizes the above discussion: Result 5: the cluster analysis shows that four clusters can be formed from the importance weights of the knowledge source attributes. These clusters can be named: knowledge, personality, competence, and communications. The analysis also shows that the majority of the people in the 'knowledge' cluster are those who generally place greater importance on the knowledge itself rather than on the knowledge source. Characterizing clusters' members As before, our final analysis is an attempt to characterize the members in each cluster based on demographics variables. To better understand what factors affect cluster membership we use multinomial logit regression analysis, regressing the demographic characteristics on the cluster assignments. Again, for this analysis we focus on the professional and MBA subgroups since the undergraduate subgroup is highly homogenous in the demographics characteristics (age, education, etc.) as well as is missing some characteristics of interest such as 'years of experience' or 'organization size'. The results of the regression are presented in Table 19. Full Model Chi 2 (36df) 67.7653 Sig. 0.005 Baseline Cluster - Competence Communications Cluster (S.E.) Personality Cluster (S.E.) Knowledge Cluster (S.E.) Chi2(3df) Sig. (Intercept) -4.13 (0.3204) -2.55 (3.199) -1.11 (3.928) Professional -0.04 (1.129) 0.11 (1.249) 1.70 (1.492) 1.7662 Question 1 - MBA admissions -0.71 (1.174) -0.87(1.315) -0.22(1.427) 0.6202 Question 2 - Internet usage policy -0.92 (1.100) -1.80(1.265) -0.97(1.349) 2.2564 Question 3 - Business plan -1.13 (1.159) 0.40 (1.090) -0.06(1.354) 2.0742 Experience 0.01 (0.067) 0.05 (0.074) 0.00 (0.077) 2.0467 # of sources 0.24 (0.340) 0.40 (0.349) -0.54 (0.462) 5.8238 Source fam. -1.04 (0.779) -0.02 (0.774) -0.74 (0.936) 2.4215 # of KMS 0.58 (0.420) 0.48 (0.447) 0.79 (0.470) 3.4718 K. Frequency 0.31 (1.009) . 0.63 (1.059) 0.17(1.439) 0.3742 Org. size 0.37 (0.314) 0.00 (0.342) 0.01 (0.378) 10.4249 0.025 Expertise 0.16(0.396) 0.02 (0.439) 0.14 (0.468) 0.2147 Knowledge weight 0.05 (0.023) -0.02 (0.024) 0.01 (0.027) 13.3809 0.01 Table 19: Multinomial logit regression on knowledge source clusters Only two variables have a significant effect on cluster membership with regards to knowledge source attributes - organization size and knowledge weight. The 'knowledge weight' variable 88 indicates whether respondents generally place more importance on the knowledge or on the knowledge source. The differences in cluster assignments are small but indicate that when knowledge is perceived to be more important than source the odds of being in the 'communication' or 'knowledge' cluster somewhat increase. This makes sense if we consider that trustworthiness and willingness to help - two attributes that are less dominant in the communications and knowledge clusters and more dominant in the other two clusters - are more person oriented than knowledge oriented, as opposed to 'knowledgeable' - which focuses completely on the person's knowledge, and 'communications skills' - which indicates the person's ability to pass the knowledge to the receiver. This result is in line with the information presented in table 18. The main effect of organization size is in increasing the odds of being in the 'communication' cluster as opposed to the other three. This can again be a result of the amount of knowledge and sources available in large organizations as opposed to a smaller one. Perhaps the ability to 'choose' the source with the best communication skills exists less in smaller organizations. The effect of organization size on the desirability of a source with good communications skills can also be tied to Hansen (2002) who examined the knowledge sharing of teams in a knowledge network. Hansen found that the path length (the number of intermediaries along the path of communications) for a specific team affects the amount of knowledge obtained from other teams. This can be explained by the fact that the larger number of intermediaries might corrupt the knowledge between the source and the receiving team thus resulting in lower communication quality (similar to a game of Chinese whispers). This might affect the awareness of members of larger organizations (networks) to the importance of communications skills. Result 6 summarizes the conclusions from this analysis: Result 6: the results of the multinomial logit regression indicate that the more importance respondents place on knowledge rather than source the more likely they are to be in either the 'knowledge' or the 'communication' clusters. In addition, people from larger organizations are more likely to be in the 'communication' cluster. Discussion The goal of the conjoint study was to identify the main knowledge and source attributes that can potentially affect the knowledge adoption decision of KMS users. To achieve this goal we conducted a preliminary Delphi study to identify potential knowledge and knowledge source 89 attributes and followed with a conjoint survey to identify the relative importance of these attributes in the knowledge use decisions of individuals. The significance of the conjoint models supports our claim that individuals consider various knowledge and source attributes (various types of meta-knowledge) when making knowledge use decisions. An additional result of our analysis is that knowledge seekers can be assigned into groups based on their relative preferences towards a specific piece of knowledge or a specific knowledge source. These groups can be characterized by various demographic or by contextual differences. The limitations of the methodology used led us to conduct two separate conjoint studies - one for knowledge attributes and one for knowledge source attributes. To learn whether people generally place more importance on knowledge or on knowledge sources we asked respondents to divide 100 points between knowledge and knowledge sources based on their relative importance to them. For example, if they generally perceive knowledge as more important than source respondents were asked to assign more points to the knowledge than to the source, and so on. Figure 20 shows the division of points between knowledge and source for all the respondents in the survey. The respondents shown on the left hand side of the graph gave about 90 points to knowledge and 10 to source while the respondents on the right hand side gave about 20 points to knowledge and 80 to source. As the figure shows, knowledge is perceived as slightly more important than the knowledge source. This result is in line with our previous Delphi results as well as with some of the literature reviewed in Chapter 4 - for example ELM - that identifies both source and message characteristics as important types of meta-knowledge with slightly more focus on the knowledge attributes. Respondent Figure 20: Knowledge/source importance weights 90 As summarized in Table 20, our findings identify the most important knowledge attributes and the most important source attributes that affect people's decision to use a specific piece of knowledge and a specific knowledge source. Tying this result back to our research objective, these attributes should be represented as generic meta-knowledge in the KMS in order to increase the likelihood of knowledge adoption. We use the term 'generic' since our results show that some groups of users may differ in preferences from others and a certain level of customization might be required when designing or implementing the KMS. In chapter 5 - following the Delphi results - we discussed the contribution of the attributes identified compared to the relevant literature. As noted, while many of the attributes have been previously identified in various areas some new attributes, such as the communication skills of the source or the awareness of the source to other resources, were identified in our study. Following the conjoint study, an additional contribution made in this dissertation is identifying the ranking of knowledge and source attributes and focusing attention on the set of attributes that are perceived as important by potential KMS users. This contribution is important in order to eliminate the information overload problems as well as to reduce the costs of operationalizing, collecting and updating knowledge attributes. Knowledge Attributes Conjoint results Knowledge Source Attributes Conjoint results Accuracy of the knowledge 1 Extent of domain knowledge 1 Relevance of the knowledge 2 Trustworthiness of the source 2 Support for the knowledge 3 Communications skills of the source 3 Currency of the knowledge 4 Source is willing to help 4 Extent of knowledge 5 Experience of source 5 Source is up-to-date 6 Source is aware of other resources 7 Table 20: Conjoint analysis results (averaged across all respondents) Comparing the results in Table 20 with our previous Delphi results (Table 10) show some changes in the ranking of several attributes. For example, the attribute 'accuracy' which ranked first in the conjoint only ranked fourth in the Delphi study. Similarly, the experience of the knowledge source, an attribute which ranked second in the Delphi results was only ranked fifth among the source attributes in the conjoint study. It is difficult to directly compare the Delphi and conjoint results due to the fact that in the Delphi study respondents ranked both knowledge and source attributes together while the conjoint was conducted as two separate studies. Nevertheless, a potential explanation for the different results stems from the self-explicating nature of the Delphi method. The differences between conjoint analysis and self-explicating 91 methods were discussed in chapter 5 and can be generally regarded as the level of realism that is embedded in each method. Conjoint analysis presents a more realistic decision scenario to respondents and thus expected to better reveal their real preferences. Expanding The Information Adoption Model In Figure 7 we presented the information adoption model developed by Sussman and Siegal (2003). Based on our analysis we propose the following extensions to the model, depicted in Figure 21 using the dashed line: R e c i p i e n t ' s I n v o l v e m e n t A r g u m e n t Q u a l i t y W i l l i n g n e s s to h e l p Up-to-date R e c i p i e n t ' s E x p e r t i s e I n f o r m a t i o n — ^ I n f o r m a t i o n U s e f u l n e s s A d o p t i o n S i z e o f R e c i p i e n t ' s O r g a n i z a t i o n I m p o r t a n c e o f K n o w l e d g e V s . S o u r c e K n o w l e d g e S e a r c h C o n t e x t R e c i p i e n t ' s E x p e r i e n c e S i z e o f R e c i p i e n t ' s O r g a n i z a t i o n Extension to the information adoption model Figure 21: The revised information adoption model (1) The first proposed addition expands the set of peripheral cues from 'source credibility' only to the full set of knowledge and source attributes identified in our study. We propose the full set of attributes since they all significantly impacted individuals' decisions, however - a smaller set might be used based on the importance ranking of the various attributes. (2) We propose that in addition to affecting perceived information usefulness, knowledge attributes also play a role in increasing users' motivation and ability to elaborate on the 92 message arguments (thus the link to argument quality). We base this link on the Elaboration Likelihood Model, which states that peripheral cues can play a role in augmenting the central route to persuasion. For example, consider a situation where a KMS provides many results to a specific query. A KMS user with limited time on her hands has therefore low ability to process all these results in full and needs to rely on peripheral cues. Based on ELM - in addition to leading to the peripheral route to persuasion - some of these cues might enhance the user's ability (or motivation) to process the message arguments. Thus, a relevance ranking of the results may lead the user to examine the top ranking results in full, taking the central route to persuasion. (3) In addition to identifying knowledge and source attributes, our analysis indicates the existence of specific groups within the data with regards to the relative importance of these attributes, and identifies the demographic and contextual characteristics of these groups. Specifically, for the knowledge attributes we find that the context of the knowledge search, the recipient's experience, and the organization size of the recipient can moderate the relative importance of attributes in the knowledge use decision. For the source attributes we find such moderating effect for the recipient's organization's size and for the general importance placed on knowledge as opposed to source. We therefore propose that these characteristics can be included in the information adoption model as moderators of the effect of peripheral cues on the perceived information usefulness. Finally, Sussman and Siegal also tested moderators of the effect of source credibility and argument quality on the perceived usefulness of knowledge. Specifically, they examined expertise and task involvement as two potential moderators. Their results indicated that argument quality was moderated by the two variables but source credibility was only slightly moderated by involvement and not by expertise. In this study we find similar results for expertise since it was not a significant determinant of cluster memberships for source attributes. Practical Implications Examining our findings from a more practice oriented point of view we compare the important attributes identified in the study to accepted meta-knowledge attributes that are included in KMS. In chapter 4 we provided some examples for meta-knowledge contained in existing KMS (summarized in Table 21 under 'existing KMS'). Knowledge attributes include mainly meta-data such as document summaries, the title of the document, or the author's identity, but also a calculated relevance rank based on the query term and some conceptual meta-knowledge such 93 as taxonomies or meta-models. Similarly, source attributes include basic source information such as name, job title, and background information, but also more qualitative attributes such as the source's affinity for the topic, area of expertise, and links to previous answers provided. These attributes can indicate the source's expertise and extent of knowledge on the topic. Since many KMS also provide the systems' administrators with the ability to define additional meta-knowledge based on their needs, a first contribution of our study is by focusing attention on a subset of important meta-knowledge attributes that should be operationalized and included in the KMS to support knowledge use (summarized in Table 21 under 'proposed attributes'). Attributes in Existing KMS Proposed Attributes Knowledge attributes Meta-data: document summaries, the title of the document, author identity, and keywords. Calculated relevance rank Conceptual meta-knowledge such as subject terms, taxonomies or meta-models. Accuracy; Relevance; Support for the knowledge; Currency; Extent (breadth and depth) Source attributes Source information: name, job title, location and contact information, availability to answer questions, and additional background information. Qualitative attributes: the source's affinity for the topic sought, general areas of expertise, and links to previous answers provided by the knowledge source. Extent of domain knowledge; Trustworthiness; Communications skills; Willingness to help; Experience; Source is up-to-date; Awareness of other resources Table 21: Proposed vs. existing knowledge and source attributes A comparison of existing and proposed attributes reveals some surprising findings concerning specific knowledge and source attributes. One example is the importance placed on the communication skills of the source. Since only people were considered as knowledge sources in this study, 'communication skills' can be viewed as a proxy to the availability of the knowledge. Knowledge that resides with a source who cannot communicate it well might be perceived as unavailable. The operationalization and inclusion of the communication skills of the source is therefore important for knowledge management systems that focus on the exchange of knowledge between human sources. Another new attribute identified in our study is the awareness of the source to other resources. Although low in its relative importance this attribute was still identified as important and indicates 94 that users would like to know the extent of the knowledge networks available to the source. This result indicates the importance of an underlying network structure in designing knowledge management systems. Finally, 'trustworthiness' - although extensively discussed in the organizational and persuasive literature (Hovland, Janis, and Kelley, 1953; Kale, Singh, and Perlmutter, 2000; Mayer, Davis, and Schoorman, 1995; Zaheer, McEvily, and Perrone, 1998)-is currently not fully represented by existing KMS. Our findings show that measurements of the source's trustworthiness are important to knowledge seekers and should be included in the KMS as part of the source's attributes. Also interesting is the relatively lower importance placed on the source's years of experience compared to some other source attributes. Our findings show that respondents might have traded-off 'years of experience' and 'extent of knowledge', placing greater importance on the overall level of the source's knowledge regardless of the how the knowledge was attained (e.g. from experience, books, courses, etc). The practical implication of this is that - although 'years of experience' is an overall important source attribute - including only one general indicator of the source's knowledge in the KMS can be sufficient for the KMS users. This is especially important if constraints exist on the number of attributes that can be included in the KMS. In addition to identifying important knowledge and source attributes our study also identifies that attributes' importance is contingent on several demographic and contextual variables, such as organization size, years of experience, or the context of the knowledge search. The main implication of this finding is that systems' designers need to take into account the potential users and uses of the system before deciding on which attributes to include and in what order of importance. This contingency can be demonstrated using the 'currency' attribute. Our findings show that the attribute 'currency of the knowledge' ranked fairly low among the knowledge attributes. The low rank assigned to currency is surprising since it was an important factor distinguishing between different clusters within respondents (as was shown by the principal component analysis in Figure 15). Based on the results of the multinomial logit regression, however, we can see that this low rank can be due to the specific knowledge search examples used in the study. If the system is intended to provide knowledge concerning time sensitive activities, 'currency' might be perceived as more important and therefore the order of representing attributes might vary. Moreover, if the cost of operationalizing attributes limits the number of attributes included in the system then the decision on which specific attributes to include should take into account these demographic and contextual variables identified in our study. 95 Chapter 7: Contributions, Limitations, and Future Research Contributions This dissertation set out to understand the knowledge management requirements of both organizations and individuals in order to propose an explanation to the problems that companies experience with KMS and to identify potential ways to improve these systems. Within this scope this work makes four main contributions. First, the evaluation framework developed in Chapter 2 and applied in chapter 3 led us to identify the main problem areas of KMS and to suggest where efforts should be focused both in research and practice. In addition, by employing organizational theories to derive KMS design requirements we introduce a more founded approach to the construction of KMS that ensures agreement between organizational KM requirements and KMS functionalities. Second, this paper introduces a meta-knowledge approach to the design of KMS that enables both the effective management of knowledge in organizational memory as well as the retrieval and use of knowledge by knowledge-seekers. This approach can alleviate the integration problem identified by our evaluation framework and improve the applications of KMS in organizations. Moreover, the meta-knowledge approach can also solve the difficulties of KMS to support externalization since it links knowledge seekers to knowledge sources who can own either tacit or explicit knowledge. Third, this paper identifies a set of knowledge and source attributes used by individuals in their knowledge adoption decision. Our results support previous work on knowledge adoption by demonstrating that the availability of meta-knowledge changes the likelihood of selecting a specific piece of knowledge or a knowledge source to consult with. The results expand on this body of literature by identifying the specific knowledge and source attributes that potentially affect knowledge adoption and should be included as meta-knowledge in the design of KMS. Finally, the research identifies the relative importance of specific knowledge and source attributes as well as the relative preferences of respondents towards a specific piece of knowledge or towards a specific knowledge source and the demographic and contextual factors that affect these preferences. This is the first time that meta-knowledge is being investigated in the context of KMS and the findings are important to our understanding of people's knowledge use decisions. This contribution is especially important to practitioners as it provides guidelines as to what meta-knowledge should be included in the design of KMS in order to facilitate the 96 use of the system and the knowledge it holds, based on the needs of the potential users of the system. This paper also makes several more minor contributions that include: (1) the development of a framework that can serve as a guideline to companies on how to select a KMS that best fits their KM requirements; (2) the adaptation of Shannon's communication model to knowledge communications and the identification of the main hurdles to KMS ability to manage tacit knowledge; (3) the application of transactive memory theory to the design of organizational memory systems; and (4) the use of Delphi and Conjoint methods to identify the attributes of an intangible product, namely - knowledge. Limitations The main limitation of our study is the relatively small sample size in the conjoint study. This limitation is apparent in our reduced ability to identify factors that potentially affect the relative meta-knowledge ranking since we were limited in this analysis to the small subgroup of professionals who participated in the study. In addition, the low significance of Kendall's tau for the four holdout questions of source attributes indicates low predictive ability of the conjoint model for the knowledge source. Nevertheless, the models developed still explain much of the individuals' knowledge use decision criteria. Finally, the use of four specific knowledge search examples - while indicating an interesting effect of the knowledge search context on attributes importance - limits the results to the specific examples used. Another limitation of the conjoint study is the separation of source and knowledge attributes. The limitations of the methodology used limit our ability to create a joint ranked list of knowledge and source attributes. There are two implications for this limitation. First, from a research perspective we are unable to compare the self-explicated Delphi results to the conjoint results and to identify whether individuals can accurately report the importance of knowledge and source attributes in their knowledge use decision. Second - from a practice point of view additional research is needed to identify the most important overall knowledge and source attributes to be included in a KMS, especially if KMS designers are interested in including only a limited number of attributes rather than the whole fourteen. A final limitation concerns the evaluation framework discussed in part I. The evaluation framework is limited in scope - covering only specific categories of KMS. For example, we have not included data mining and business intelligence tools. In addition - the evaluation of the tools is based on information provided by the vendors and not on actual testing of the tools 97 themselves. While this approach has interesting implications - indicating whether KMS designers are aware of KM requirements of organizations - it may be useful to examine the actual design of KMS and identify the problems in this design. Future work This dissertation has highlighted the importance of meta-knowledge in the knowledge use decisions of individuals. Following this, the main focus of our future work is on the application of meta-knowledge in KMS and the identification of the benefits that arise. The main question of interest is whether the inclusion of meta-knowledge in the design of KMS will improve the benefits that companies are realizing from these systems. In chapter 4 we proposed two potential benefits for the use of meta-knowledge in KMS - the first was by facilitating the management of organizational memory and the second by enhancing the adoption of organizational knowledge. These two motivations for meta-knowledge open two future research channels: (1) testing the benefits from organizational memory systems that are based on meta-knowledge; and (2) refining the specific effect of various knowledge and source attributes on knowledge adoption in different contexts and within different populations. In addition, it is interesting to study the interactions of the two previous questions - the exact relation between knowledge adoption and KMS benefits. A different line of research coming out of the first part of our dissertation is studying the scope and definitions of KMS. In our framework we investigated the requirements from KMS to support three specific processes - namely, knowledge generation, codification, and transfer. However, organizational definitions for KM are wider in scope and include various other processes - not fully supported by current KMS, such as knowledge protection, knowledge maintenance, and others. This raises the need to examine the role of technology in supporting organizational KM and to define the scope of KMS in order to avoid failures of these systems, as the ones reported in the KPMG report. 98 Appendix A: Evaluation of knowledge management systems Tools reviewed 1. 80-20 Retriever 15. Hyperwave eKnowledge 29. Ptech 2. 80-20-DME 6.9 SPI Portal 30. Quantum 1.3 3. Ask me enterprise 16. ICQ Groupware 31. Quiqconnect 4. Autonomy 17. Infinos 32. RetrievalWare 5. BASIS 18. Knexa 33. Searchserver 6. ClearResearch 19. KnowledgeLEAD (KRM) 34. Semio map 7. DecisionStream 20. LexiQuest Guide 35. SERbrainware™ 8. Divine Athena 21. Lotus Discovery 36. Smartlogik Insight™ 9. Divine opinionware 22. Metadata Creator 37. Tacit Knwoledge 10. Domain Knowledge 23. MetaTagger 38. Verity K2 Enterprise 11. eGain Knowledge 24. Muscat Discovery™ 39. Viador E-Portal Gateway 25. Netegrity PortalMinder Framework 12. Enfish Enterprise 26. Notes R5 40. Web based groupwar 13. Enterprise Expertise 27. Organik Management 28. Plumtree Collaboration 14. eRoom Server Tools classification examples The screen shots on the next pages provide an example for how tools were classified. The first tool was classified as a knowledge-sharing tool. The Highlighted section in the product description indicates the tool is intended to support the sharing of knowledge by organizational members. K n o w l e d g e L f A O 7 W ( view Demo » KnowledgeLEAD™ incorporates a suite of specific features designed to enable, empower and encourage individuals to share their knowledge and to collaborate with others by automating knowledge sharin The system fosters collaboration by locating professionals with specific expertise and skills. The following are three main modules of KnowledgeLEAD™. I. PeopleSpace PeopleSpace is a place where individuals declare their experiences, education and interest to create their professional profiles. Here, individuals can quickly search for experts by skills, department, name etc. This is the linchpin of the process of identifying the corporate experts, and the first step in the process of turning intellectual assets into competitive advantage. II. ExpertSpace ExpertSpace is the place where experts disseminate their knowledge to their colleagues. Individuals who are deemed to be an expert in certain subject matter or business knowledge can quickly and conveniently host an ExpertSpace site to share their knowledge. Here experts can post articles, websites and news updates related to their interest.and expertise. By doing so, these experts share their knowledge as well as the sources of their knowledge, thus becoming KnowledgeGuides. In addition, individuals can post questions directly on the 99 The second screen shot provides an example for a tool that was classified as a content management tool. As the product description indicates this tools is defined as a 'content processing software' enabling organizations to manage various types of content. APPLIED SEMANTICS Concept Server News Series Taxonomies Domain Name Solutions , AtiSej CO U J OC LU CO CO <: UJ Appl ied Semantics News Series The Applied Semantics News Series is comprehensive content processing software developed specifically for today's electronic and print publishers. Organizations struggling with the convergence of disparate media, digitization of content, and an increased need for collaboration can tap into Applied Semantics' powerful content categorization, summarization, event identification and concept tagging software. The Applied Semantics News Series is bundled with industry standard taxonomies such as IPTC Subject Codes and SIC Codes, and works with XML standards such as NITF, NewsML and PRISM. The result is a series of applications that automatically enrich the news flowing through your existing editorial, archive, wire, library, and syndication systems. The Applied Semantics News Series automates the costly, inaccurate and time-consuming process of populating news content with valuable metadata. By adhering to content standards, the Applied Semantics News Series enables publishers to reduce costs, increase productivity and improve content quality to help organizations be responsive and competitive in today's rapidly changing environment. annliprt Rpmantirc Nawfi Spring fan' Another example for a content management tool is shown in the screen shot below: etaTagger intelligence •Tag it. Fmdfc. Use St. Is finding content in your business like trying to find a needle in a haystack? Customers and employees need to find business information quickly-- and easily. MetaTagger makes this possible by automatically enriching content with metadata, ensuring that your business-critical information is readily available to those who need it most. MetaTagger is based on third-generation classification and categorization technology developed by leading library scientists and engineers.With MetaTaggeryou can maximize your investment in portal, search, and syndication infrastructure by organizing and describing your enterprise content for precise delivery. You'll also boost your employees' productivity and efficiency by minimizing the time they spend searching for content, and by avoiding the need to reauthor existing content. Figure 22: Example of tools classification 100 Example of deriving functionalities Below is a screen shot from the website of one of the content management tools. The highlighted section implies that the tool supports the retrieval of knowledge from memory and therefore functionality ER2 (retrieval of knowledge). The analysis of tools was made by identifying as many sentences as this one on the company's website and classifying them according to the functionalities identified in this paper. A knowledge delivery application Smartlogik Insight™ is an Intranet based enterprise knowledge delivery solution. ItHBSWIBBIBBlBBIIBfflSIIiBISBHiB^BBI intuitive, enabling individual users to turn information into knowledge. It comprises an integrated suite of applications that automate the discovery, storage, indexing and delivery of both internal and external sources of corporate knowledge. The underlying architecture is scalable and can access any search engine technology, enabling organisations to leverage their investment in existing information retrieval technologies. The Office Tools module also provides full integration with Microsoft's Office products, providing knowledge delivery and publishing directly within the Microsoft Office™ and Internet Fxnlnrer™ environments Figure 23: Example of deriving functionalities 101 Method of study: The study will be conducted using the Delphi method. This method aims to obtain a consensus of opinions among a panel of participants. Panel members will answer two short questionnaires and evaluate other members' suggestions. The main question we ask in the study is: "In your opinion, what are the most impodant characteristics of a knowledge source and the most impodant characteristics of knowledge?" Time required: We estimate that the study will involve between three to five interactions of 10-30 minutes each, over the course of several weeks, for a total of one and a half hours. We use web technology to conduct the study and therefore participants can complete the questionnaire from any location at any time, at their convenience. As a token of our appreciation for participating in the study we will provide participants with a $50 gift certificate to Chapters store or Website. Number of participants: We hope to recruit approximately 40 panel members overall (from several organizations). We would appreciate having five to ten participants from «Company». Who can participate: We are looking for people who in their work often or occasionally seek knowledge from others people, documents, or other sources within or outside the organization. An example is someone who is working on a project and may wish to consult with co-workers who have worked on similar projects before. An example of the questionnaires is available at: http://mis.commerce.ubc.ca/kms 103 First round invitation Dear I would like to thank you again for agreeing to participate in our study of knowledge management systems. We are now ready to begin the first round of the study. The link below will take you to the study's main website. From there you can login to the study at any time. Link: http://knowledgestudy.commerce.ubc.ca/loqin.htm Your login ID is: «Email» And your password is: «Password» (lower case) Please complete the questionnaire by August 31 s t. When you log into the site for the first time you will see detailed instructions for the study's procedures as well as a short pre-study questionnaire intended to provide us with some information about your knowledge search habits. Please make sure to sign your name in the consent box to indicate your willingness to participate in the study. After you submit this page you will see the first study's questionnaire. We estimate that it will require between 20 to 30 minutes of your time. You may logout at any time and return to complete the questionnaire at a later time - your responses will be saved in our database. If you have any questions or concerns please don't hesitate to contact me to this email address of by phone at (604) - 8224772. Sincerely, Second round invitation Dear I would like to thank you for participating in round one of the Knowledge Management Study. Based on the responses from round one, we have created a summarized list of characteristics of knowledge and knowledge sources. In this second round of the study we ask you to review the list of characteristics and select those that you feel are the most important. We estimate that this questionnaire will require between 10 to 15 minutes of your time. The link below will take you to the study's main website. After you login to the study you will see the questionnaire as well as more detailed instructions for this round. Link: http://knowledqestudy.commerce.ubc.ca Your login ID is: «Email» And your password is: «Password» Please complete the questionnaire by September 17. Note that once you submit your answers in this round you will not be able to review them. However, you may overwrite you answers by submitting the questionnaire again. If you have any questions or concerns please don't hesitate to contact me to this email address, or by phone at (604) 822-4772. Sincerely, 104 Third round invitation Dear Thank you very much for providing us with your assessment of the key characteristics of knowledge and knowledge sources. We very much appreciate the time you invested in your response. In this third round, your suggestions have been integrated with those of your peers in a list summarizing the most important characteristics of knowledge and knowledge sources. We now ask that you rank this list in order of importance to you. A crucial goal of this research is to achieve a high level of consensus about the importance of each characteristic. After we receive your ranking, we will calculate the average rank for each characteristic and then determine the level of consensus among panel members. If it is necessary, we may ask you to reconsider your ranking in a fourth (and final) round. Thank you very much for your participation in the study! To see the questionnaire follow the link below: Link: http://knowledqestudy.commerce.ubc.ca Your login ID is: «Email» And your password is: «Password» Please complete the questionnaire by October 4. Note that once you submit your answers in this round you will not be able to review them. However, you may overwrite you answers by submitting the questionnaire again. If you have any questions or concerns please don't hesitate to contact me to this email address, or by phone at (604) 822-4772. Sincerely, Final round invitation Dear. . . , I would like to first thank you for your valuable inputs in the previous rounds. As the objective of this study is to seek consensus about the importance of each characteristic of knowledge and knowledge sources, we need your assistance one more time. Your help will enable us to complete the study and develop the final list of key characteristics. Because of the small sample size, your timely input is critical to the success of our study. In this round we ask that you - again - rank the characteristics of knowledge and knowledge sources. However, we ask that you also consider the group's average rank in your ranking of the most important characteristics. As a token of our appreciation for your participation in the study, a copy of the final results will be made available to you. As well - we will send you a Chapters gift certificate for the amount of $50. After you submit your answers to this final questionnaire, you will see a request for your mailing address. This is to enable us to mail the gift certificate to you. To see the questionnaire follow the link below: Link: http://knowledqestudv.commerce.ubc.ca Your login ID is: «Email» And your password is: «Password» Please complete the questionnaire by October 25, 2002. Note that once you submit your answers in this round you will not be able to review them. However, you may overwrite you answers by submitting the questionnaire again. If you have any questions or concerns please don't hesitate to contact me to this email address, or by phone at (604) 822-4772. Sincerely, 105 Questionnaires: Delphi study - consent form and demographics questionnaire The University of .British Columbia - .. Facult^ '-or.Com-m'erce'&'Busin'ess.-Administxation: To change your password click here (Not*: Jfyouclidt on this UrJi wilhout submJUJrgthe questioniudr* yvi will lost your due) Investigator: Dr. Yarr Wand, Management Information Systems, Faculty of Commerce, UBC. Tel: (604) 822-8395 Co-investigator: Dorit Ngvo, Management Information Systems, Faculty of Commerce, UBC. Tel: (604) 822-4772 Thank you for accepting the invitation to participate in our survey on users' requirements from Knowledge Management Systems. A Knowledge Management System is a system that supports various knowledge management activities in the organization, such as knowledge generation, knowledge sharing, and the storage of existing knowledge. The goal of the study is to better understand users' requirements from knowledge search and the characteristics of knowledge sources that are most important to users. The results of the study will be used to enhance our understanding of users' requirements and for designing better Knowledge Management Systems. We have placed you in a group of 40 experts who are asked to generate and rank a set of the most important characteristics of knowledge and knowledge sources. The responses of all panel members will be consolidated into a single list and sent for your feedback. This feedback process maybe repeated two or three times. We estimate that the whole study will take approximately one and a half hours over the course of the next six weeks. In return for your participation in the study you will receive a gift certificate to Chapters bookstore or Amazon.com. Below is a diagram of the complete study procedure. The rectangles on the right represent your part in the study and the circles are the processing done by the researchers. We may also request that you participate in a short interview at the end of the study. The purpose of this interview is to understand the reasons leading to your selections. The interview is not a mandatory part of the research and you may chose not to participate in it. F n t QueeLwiuetiie (Wh»t ue the importurt ch&xictenjnc* of knowledge aid knowledge sources?) Reducing the Kst (Select Hie most important items from the. lis tattained ui round 1)-: Second' Questionnaire (Rank the ltems^on tie lutbased on their importance to you) Feedback - Revise the rarjted list created by Hie gjcup (This step may be repeated to achieve oonsssisui among panel members) Participation in this study is entirely voluntary. You may withdraw from the study at any time at your own discretion. However, because of your expertise and the small size of the group, your participation is critical to the successful completion of this study. From this website you will access all the rounds of the survey. We will inform you via email when the next round of the survey is ready to be completed. In this round you will complete the first questionnaire (highlighted in the diagram above) which should take about 30 minutes. You may log out of the site before completing the questionnaire and log in again at a later time. Your responses will be saved in the database. Upon receiving your completed questionnaire we will integrate and summarize your responses with those from the other group members and then we will send the results back to you for rating. Your responses to all parts of the study will be summarized anonymously and be kept strictly confidential - no individuals or organizations will be identified in any of the final reports. If you have any questions regarding this study, please feel free to call Dorit Nevo at (604) 822-4772 or send an email. If you have any questions about your treatment or rights as a research subject you may contact the Director of Research Services at the University of British Columbia at 822-8598. Consent By signing your name below you agree that you have read and understood the contents of the Information Sheet provided and have decided to participate in the study. You may print a copy of this consent form for your own record. NAME: | ~~ j DATE: 3/31/2003 106 Before beginning the study please take a few minutes to complete the information below. This information is important for our anarysis and will not be used for any other purposes. A. General Information 1-Age: |- S 2. Level of Education: |; HI 3. Profession: f ' 4. Job Title: | | 5. Years worki ng: | i 6. Years with thi s company: | ! 7. Years in present position: | I B. Knowledge search 8. How Frequently do you use external knowledge (knowledge other than your own) in your work? I: Ml 9. What are the most common sources of knowledge that you use? Ci People G Books r Databases n Web sources D Other (specify): I ~ I 10. How familiar are you with these sources? I: s | (Note: Please r«f«r to yout average l«v«l of familiarity with a}l sources. For example, it you are very familiar with lomt but not it j!l familiar with others select'somewhat familiaf) 11. Are you familiar with the concept of knowledge management systems? |;_ Ml 12. Are you currently using any of the following? G Lotus Notes or si mi lar products C Intranet r Best practices database l j ERP systems r Other systems that support your knowledge search (please provide details) C. Organization characteristics 13. How much do you agree with the statements below? Please rate your agreement on the following scale: 1 (strongly agree) - 7 (strongly disagree) When consideringjsJM procedures in your orgiar^izaiior£ 1. If a written rule does not cover some situation, we make up informal rules for doing things as we go along Strongly 1 4 7 Strongly ,„.. C O C O O O C disa,,.* 2. There are many activities in my company that are not covered by some formal procedures. Strongly 1 4 7 Strongly agree O 0 0 O O O 0 disagree 3. Usually, my contact with my company and its representatives involves doing things "by the rule book". Strongly 1 4 7 Strongly agree G f< O O O O disagree 4. Contacts with my company and its representatives are on formal preplanned basis. Strongly 1 4 7 Strongly agree O C C C C C C d isa g i • e 5. I ignore the rules and reach informal agreements to handle some situations. Strongly 1 4 7 Strongly a g re e O C 0 C 0 C G disagree 6. When rules and procedures exist in my company, they are usually written agreements. Strongly 1 4 7 Strongly „„. n o r, n n c c d « a a , . . 7, We have a strong reputation of technological excellence Strongly 1 4 . 7 Strongly a g re e 0 C C. C C C C disa g re e 8. Knowledge intensity ('the extent to which a firm depends on the knowledge inherent in its activities and outputs as a source of competitive advantage") is a characteristic of our business Strongly 1 4 7 Strongly agree C C, C, C C C C disagree 9. There is a strong knowledge component in our products and services. Strongly 1 4 7 Strongly agree f< f} G C- Cj O 0 disagree Please submit this form and continue to the questionnaire ES i i i f l tfeiiiil Contact Information: Dorit Nevo Management Information Systems, Faculty of Commerce, UBC. Tel: (604) 822-4772 Delphi study - Round 1 questionnaire The University of British (Columbia Panel member name: new If you are not new, please click here To change your password click here [Not*: It you click on this link without submitting the questionnaire you will lose your data) Consent Form QUESTIONNAIRE #1 - INSTRUCTIONS The purpose of this questionnaire is to draw on your experience in searching for knowledge to evaluate the most important characteristics of knowledge sources. The questionnaire consists of 2 questions and your task will be to evaluate possible knowledge sources that may provide an answer to these questions. You are not required to know the answer to the questions but to evaluate possible sources that may provide you with the answer. Specifically, we would like you to answer the following question: In your opinion, what are the most important characteristics of a knowledge source and the most important characteristics of knowledge? In answering this question, please take account of the following definitions: ! ! We refer to "knowledge" as the answer that you would require to complete the given tasks successfully. ! ! . By a "knowledge source" we mean a person, book, document, web page, database, or any other entity that may provide you with the knowledge required to answer your query. You are asked to prepare an analysis using a new statistical analysis software package. You need help to perform this analysis using this specific package. You may consult with any of these sources of knowledge to answer this question: John (programmer), Jack (statistical analysis division), 'Introduction to matlab' (book), wm.stals.com(web). What are the characteristics that will be important to you in selecting one of these sources? Your response: Characteristics that are important to me about the selected source are: Years of education (if it's a person); The date of the knowledge (for all sources) You may list as many characteristics of knowledge and sources as you like, but 5-10 characteristics would suffice. If you would like to make any comments you can do so at the end of thi s page. Please complete this questionnaire and submit the form in time for our analysis on August 31st You may log out at any time by clicking the 'Submit' icon at the bottom of this page. Your responses will be saved and you will be able to complete the questionnaire at a later time. In this first part we would like you to consider the following knowledge search task UBC has decided to consult with industry partners in the admission of new MBA students. You are asked to review applications and recommend the most suitable applicants in your opinion. In your decision you are requested to take into account UBC admissions policies and standards. To assist you in this desicionyou can ask the opinion of any one of 20 people that indicated that they have some knowledge on the topic. Please rate the following with regards to this example: Mv expertise on this topic (MBA admission process) is: Low O C c C C High Knowledge Source Characteristics You are now asked to select the source (among the 20 people) that you will turn to for help. Note that in the first stage we would like you to limit the source to persons only. *********************** **************** ****************** For Example: r * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * t * t T**********«* i PART A 108 Please list characteristics of the knowledge source (for example: 'Years of education'), or characteristics of the knowledge itself (for example 'the currency of the knowledge') that you feel are important to you in making your selection. Please define what you mean by each term (For example, if you add the characteristic: 'currency' please explain what currency refers to. In this case: 'the time in which the knowledge was attained or last updated') Characteristics of the Knowledge Source Characteristic Definition (if different than before) Dl II "I 2)1 | | I s i r z Z Z Z Z J l Z Z Z *)l i l Z Z i 5)1 I . ~ ..I | More Characteristics "| Characteristics ofthe Knowledge Characteristic Definition (if different than before) Dl Jl ] 2) I 11 J 3) |_ j I Z Z 4) Z Z Z Z Z Z i I _ _ _ I 5) 1 ZJl I I More Characteristics I Other knowledge sources Will your list be different if the source of knowledge was a book? OYES ONO If yes what would you add: What would you Remove: h 15 fesl a Will your list be different if the source of knowledge was a web page? O YES C NO If yes what would you add: What would you Remove: 1 1 lj Will your list be different if the source of knowledge was a database? o YES o NO If yes what would you add: What would you Remove: B Will your list be different if the source of knowledge was a document? o YES c NO If yes what would you add: What would you Remove: m n 141 111 PART B At the second stage of this questionnaire we would like you to provide your own knowledge search example. In the space below please describe a task or a question - from your everyday work - In which you may need assistance from external knowledge sources jExftijiplpc for "possible rfttounsHs: ••<. "I need to lano-w how to perform a Delphi study" "I need to learn about the approval procsss of a loan applicant" "I need know led ae about previous projects with a specific client" i 109 Before discussing the knowledge source characteristics, please rate the following with regards to this example: Mv expertise on this topic is: Low O O C C C High Do you agree with the statements below (with regards to your example)? 1. Different people could have different opinions about the best solution for this task/question 2. Most people would clearly agree on what information is important and unimportant for this task/question 3. The information needed to solve this task/question can be interpreted differently by different people. 4. More than one reasonable solution exists for the problems faced in this task/question. 5. The information needed to complete this task/question can be found in books. 6. The rules and criteria for solving this task/question are clear and can be found in books. Knowledge Source Characteristics You are now asked to select the source you will turn to. Note that in the first stage we would like you to limit the source to persons only. Please Ii st characteri sti cs of the knowledge source, or characteri sti cs of the knowledge that you feel are important to you i n maki ng your selection. If you repeat characteristics that you mentioned in the previous example then you do not have to redefine them. However, if you add new characteristics please define what you mean by each term. Characteristics of the Knowledge Source Characteristic Definition (if dEfferent than before) 2)C 5)C j- More Charafieristics V ' j Characteristics ofthe Knowledge Characteristic Definition (if different than before) Dl ~ ~ i l ; 2) 1 ~ ~ ~ ] | 3) 1 I 4) 1 ~ | 5) i .r : f- More Characteristics } Other knowledge sources Will your list be different if the source of knowledge was a book? C YES O NO If yes what would you add: What would you Remove: 15 y Will your list be different if the source of knowledge was a web page? O YES 0 NO If yes what would you add: What would you Remove: a m H El Will your list be different if the source of knowledge was a database? c YES c NO If yes what would you add: What would you Remove: El S i 1 Will your list be different if the source of knowledge was a document? c YES 0 NO If yes what would you add: What would you Remove: s a k: I 110 Strongly 1 7 Strongly agree 0 G O O O O O disagree O O O O O C C r. r o o o o c O C O r. r r. c o o o o o o c agree C C O O 0 C C If you would like to make any comments you may do so In the space below: Thank You for completing questionnaire #1. Please submit your answer by clicking on the button below. | 'Submit | Contact Information: Dorit Nevo Management Information Systems, Faculty of Commerce, UBC. Tel: (604) 822-4772 Delphi study - Round 2 questionnaire ET. _~H The University of British Columbia •aa.vroa Faculte of Commerce & Business Administration Panel member name: new If you are not new, please click here To change your password click here (Note: it you click on this link without submitting the questionnaire you will lose your data] H Consent Form QUESTIONNAIRE #2 - INSTRUCTIONS The table below summarizes the characteristics of knowledge and knowledge sources (persons only) suggested by all panel members in round 1. We would now like to know what characteristics are most important to you. From the list of characteristics below please select at least 6 characteristics in total that you feel are the most important. You may select any combination of knowledge and knowledge source characteristics. For example, if you think that only source characteristics are important you can select only characteristics of knowledge source. If you feel some source characteristics are important as well as some knowledge characteristics you may select characteristics from both lists. In your selection of characteristics please consider characteristics that apply across all knowledge search situations (i.e. different types of questions that you might search knowledge for). Please select characteristics that you feel are most important in any situation. Important Notes: 1. We refer to "knowledge" as the answerthat you would require to complete a given task successfully. 2. In selecting characteristics please referto people only (not books, web pages, etc.) A'knowledge source' is therefore defined asany Eerson that may provide you with the knowledge required to answer your query. 3. Please review the full list of characteristics - both of knowledge and of knowledge sources - before you make your selection. 4. You may select more than six characteristics if you feel other characteristics are also important. While we do not restrict the number of characteristics you may select, please try to select only those that are most important to you. 5. If you would like to make additional comments or suggestions for revising the description ol a characteristic, you may do so in the space provided below the table. Select Knowledge Source (person) Characteristics: > Definition - " I l-al .I'L-lavfefe n Extent of domain knowledge The extent of knowledge about the domai n that the source has j r Education (background) The formal level of educati on of the source (e .g. past degrees and I certificates) \ r Reputation The source's reputation as knowledgeable in the specific domain ' r Occupation The profession and/or job definition of the source r Experience Extent of past experience that the source has dealing with this domain or i with similar problems i r Awareness of other resources The ability of the source to locate relevant resources that may provide :• additional knowledge on the topic n Accessibility Availability of the source to answer questions in a timely manner Willingness to help The source is approachable and willing to help or provide feedback : • Up to date The source i s current i n the knowledge domai n j r Past Performance The source has been a useful knowledge source in the past j r Communications skills The source can clearly communicate ideas and explain the topic in a i simple way | r Trustworthiness The source is honest and provides credible responses j r Reasoning capabilities The source is able to draw logical conclusions and infer from their | knowledge on related problems j c Objective The source is unbiased j Select"" Knowledge i Characteristics | Definition ' • Relevance The knowledge is relevant for the current problem r Currency The knowledge is up-to-date o Newness The knowledge provides new and unique insights that were not available before n Extent of knowledge The depth (level of detail) and breadth (coverage) of the knowledge o Minimality (specificity) The knowledge is succinct and specifically answers the problem r, Accuracy The knowledge contains no errors o Validity The knowledge can oe verified ana supported by other sources or by empirical results n Source credentials The identity and authority of the source of the knowledge • Completeness The knowledge covers the whole domain and does not omit any relevant details • Objectivity The knowledge is not biased o Logical The knowledge is constructed in a logical manner and does not include internal contradictions o Communicable The knowledge is clear and easy to understand. The knowledge can be easily communicated and presented to others. • Usefulness The knowledge is constructive [J Related knowledge The knowledge refers to other knowledge that may be related n Credibility The knowledge can be trusted n Origin of knowledge The method in which the knowledge was obtained (e.g. theoretical knowledge, knowledge from experience, etc.) If you have any questions or comments please write them below: m 83 Si Thank You for completing questionnaire #2. Please submit your answer by clicking on the button below. Contact Information: Dorit Nevo Management Information Systems, Faculty of Commerce, UBC. Tel: (604) 822-4772 Delphi study - Round 3 questionnaire E 7 _ _*1 The University of British Columbia .**""****** Faculty of Commerce & Business Administration* Panel member name: TestID If you are not TestID, please click here To change your password click here (Note: it you olick on this link without submitting tha questionnaire you will lose your data) || Consent Form QUESTIONNAIRE #3 - INSTRUCTIONS Thank you very much for providing us with your assessment ofthe key characteristics of knowledge and knowledge sources. We very much appreciate the time you invested in our study so far. Based on Round 2, we have developed an initial list ofthe most important characteristics of knowledge and knowledge sources. The list is ordered according to the number of panel members that selected each characteristic in round 2. In addition, only characteristics that were selected by at least one third ofthe panel members are included in this round. We now ask that you review the list of characteristics below and rank the characteristics again based on their importance to you. Please assign a rank of 1 (one) to the most important characteristics. 2 (two) to the second most important characteristic, and so on. Please avoid ties, do not assign the same rank to two characteristics Important Notes: 1. To review the definition of a characteristic click on the"?" image next to it. 2. The column "votes from round 2" refers to the total number of people who selected .this characteristic as important in the previous round. 3. We refer to "knowledge" as the answer that you would require to complete a given task successfully. 4. In selecting characteristics please refer to people only (not books, web pages, etc.| A "knowledge source" is therefore defined as any person that may provide you with the knowledge required to answer your query. 5. If you would like to make additional comments or suggestions for revising the description of a characteristic, you may do so in the space provided below the table. Please rank the list of characteristics below from 1 (one) to 14 (fourteen), 1 being the most important and 14 being the least imprtant. Please avoid ties in ranking the characteristics. iYour jRank•* i I . Votes from round l 2, Characteristic?: 1 [Your IRank Votes from round 2 ' Characteristic i n 23 «jj Experience of source i n 13 fij Validity of the knowledge i n 20 ^ | Relevance ofthe knowledge n 12 JU Accuracy of the knowledge I n " 17 fl] Extent of domain knowledge ofthe source n 12 E] Communications skills ofthe' source i n 16 £J Credibility of the knowledge n 11 1] Source is up-to-date i f — 16 Hi Currency of the knowledge n 11 (|J Source is willing to help i n 16 8 Accessibility of the knowledge n 11 l|) Source is aware of other resources i n 14 1] Extent of knowledge n 10 HI Trustworthiness ofthe source If you have any questions or comments please write them below: 3 V i zJ Thank You for completing questionnaire #3. Please submit your answer by clicking on the button below. ," SgbmS "I Contact Information: Dorit Nevo Management Information Systems, Faculty of Commerce, UBC. Tel: (604) 822-4772 114 Delphi study - round 4 questionnaire £J _ J 3 The University of British Columbia Bjanii^aja p a c d [ l -y 0 f Commerce & Business Administration "X Panel member name: TestID If you are not TestID, please click here To change your password click here (Note: if you click on this link without submitting th« questionnaire you will lose your data) Consent Form QUESTIONNAIRE #4 - Instructions Thank you very much for providing us with your assessment of the key characteristics of knowledge and knowledge sources. We very much appreciate the time you invested in our study so far. Based on the Round 3 rankings from you and your peers, we have developed an initial ranked list of the most important characteristics of knowledge and knowledge sources Before you proceed, please review the ranked list and definitions of characteristics in the table below. Reviewing the definitions is important for distinguishing between characteristics that seem similar. For example, note that "extent of knowledge" refers to the depth and breadth of the specific knowledge (e.g. a specific answer to your question), while "extent of domain knowledge of the source" refers to the extent that the source is knowledgeable in the domain in general. Similarly, 'the source is up-to-date' refers to the general currency of the source's knowledge in the domain while the 'currency of the knowledge' refers to the date stamp on the specific knowledge. Rank Characteristics » = Definition 1 Experience of source Extent of past experience that the source has dealing with this domain or with similar problems — Relevance of the knowledge The knowledge is relevant for the current problem Credibility of the knowledge The knowledge can be trusted (is believable) MM Accuracy of the knowledge The knowledge contains no errors Validity of the knowledge The knowledge can be verified and supported by other sources or by empirical results mm Currency of the knowledge The knowledge is up-to-date 7 Extent of knowledge The depth (level of detail) and breadth (coverage) of the knowledge 8 Extent of domain knowledge of the source The extent of knowledge about the domain that the source has mm Accessibility of the knowledge Availability of the source to answer questions in a timely manner H I Source is up-to-date The source is current in the knowledge domain 11 Trustworthiness of the source The source is honest and provides credible responses mm Source is willing to help The source is approachable and willing to help or provide feedback 13 Communications skills of the source The source can clearly communicate ideas and explain the topic in a simple way 14 Source is aware of other resources The ability of the source to locate relevant resources that may provide additional knowledge on the topic We now ask that you rank the list of characteristics one more time based on their importance to you and on the group's average.rankfrom the previous round. Please review each characteristic, the group's average rank and your initial ranking, and make a new ranking decision. If your new ranking is different from the group's average by more than three points (or you would like to comment on a recommendation), please explain your decision for your ranking in the space provided at the bottom of this page. Please rank the list of characteristics below from 1 (one) to 14 (fourteen), 1 being the most important and 14 being the least imprtant Please avoid ties in ranking the characteristics. 115 Important Notes: 1. To review Ihe definition of a characteristic cl ick on the"?" i m a g e next to it. 2. We refer to "knowledge" as the answer that you would require to complete a given task successfully. 3. In selecting characteristics please refer to people only (not books, web pages, etc.) A "knowledge source" is therefore defined as any person that may provide you with the knowledge required to gnswer your query. New, raKl<:| Previous rank Group's average rar * Characteristic 5 1 2 j|| Experience of source [ J 6 if| Relevance of the knowledge l_l 1 3 0 Credibility of the knowledge • 4 4 0 Accuracy of the knowledge I~J 3 5 H Validity of the knowledge CJ 12 6 FJ Currency of the knowledge • . . 8 7 0 Extent of knowledge New rank^ Previous r'ai «. Group's average rank Characteristic • 13 8 §] Extent of domain knowledge of the source • 2 9 {?] Accessibility of the knowledge • 10 10 Fj Source is up-to-date 1 I 7 11 0 Trustworthiness of the source 1 11 FJ Source is willing to help n 14 13 1^1 Communications skills of the source L.J 9 14 Source is aware of other resources ] If you have any questions or comments please write them below: Thank You for completing Round #4. Please submit your answers by clicking on the button below. .Submit Contact Information: Dorit Neva Management Information Systems, Faculty of Commerce, UBC. Tel: (604) 822-4772 116 Appendix C: Conjoint questionnaire Interview questions First interview The interview was conducted with six of the panel members from the Delphi study. We asked respondents to identify the number of levels for each attribute as well as what measures they would use to represent it. The responses are displayed in Table 22 below. Characteristics 1 2 3 4 5 6 # of levels selected Measures Relevance of the knowledge 3 5 3 3 3 3 3 Percent relevance Experience of source 3 5 3 3 3 3 3 Years of experience Accuracy of the knowledge 3 2 3 3 3 3 3 Percent accurate Validity of the knowledge 3 2 3 3 2 3 3 ' How many sources support or refute it Currency of the knowledge 3 5 3 3 3 3 3 Time since created (in years) Extent of knowledge 2 5 3 3 3 3 2* Level of detail* Extent of domain knowledge of the source 3 5 2 or 3 3 3 3 3 High/Medium/Low Source is up-to-date 2 5 3 3 2 2 2 Yes/No Trustworthiness of the source 3 2 2 or 3 2 3 2 2 High/Low Source is willing to help 2 3 2 2 2 2 2 High/Low Communications skills of the source 3 5 3 3 2 3 3 Good/Reasonable/Po or Source is aware of other resources 2 5 2 or 3 2 3 2 2 Yes/No Table 22: Interview with Delphi panel members The attribute 'extent of knowledge' was initially represented as three levels of detail. However, we did not feel that 'level of detail' is a full representation of the concept 'extent of knowledge'. In the Delphi, 'extent' was defined as the breadth and depth of the knowledge, which corresponds more with the term 'comprehensiveness' than with 'level of detail'. For example, a document may be very detailed but not embed knowledge that is very broad or deep. In addition to changing the term used we reduced the number of levels to two - 'comprehensive' and 'not comprehensive' to ensure that the difference between the levels is significant. 117 Second interview These are the questions used to elicit the value of the attribute levels for the different examples. Seven examples were examined of which four were selected based on the similarity of their attribute levels and the feedback from interviewees. Example 1 You have an idea for a new company. In order to contact potential investors you need to prepare a business plan describing your idea. To assist you in your task you can ask the opinion of any one of 20 people who indicated that they have some knowledge on the topic. 1. Currency is defined as the number of years since the knowledge was created. If you are asked to evaluate the currency of the knowledge required to answer this example in terms of months or years, what would you consider very current or 'high currency' (how many months/years); what is 'medium currency'; what is 'low currency'? High Currency Medium Currency Low Currency 2. Accuracy is defined as a percentile ranking of the lack of errors in the knowledge. If you are asked to evaluate the accuracy of the knowledge in terms of percent (i.e. what percent of the document is accurate) - what percent would you consider 'high accuracy'; what would be 'medium accuracy'; what would be 'low accuracy'? High Accuracy Medium Accuracy Low Accuracy 3. If you were asked to evaluate the relevance of the knowledge in terms of percent (e.g. like a search engine would return results and say 'this result is X% relevant to your query') - what percent would you consider 'high relevance'; what would be 'medium relevance'; what would be 'low relevance'? High Relevance Medium Relevance Low Relevance Example 2 You are looking to buy a used car. You now have to test and select the car you would like to buy. To assist you in your task you can ask the opinion of any one of 20 people who indicated that they have some knowledge on the topic. High Currency Medium Currency Low Currency High Accuracy Medium Accuracy Low Accuracy High Relevance Medium Relevance Low Relevance 118 Example 3 Due to recent personnel problems in your company you are asked to devise an ethical code of behaviour for your company. To assist you in your task you can ask the opinion of any one of 20 people who indicated that they have some knowledge on the topic. High Currency Medium Currency Low Currency High Accuracy Medium Accuracy Low Accuracy High Relevance Medium Relevance Low Relevance Example 4 UBC has decided to consult with students and industry partners in the admission of new MBA students. You are asked to review applications and recommend the most suitable applicants in your opinion. In your decision you are requested to take into account UBC's admissions policies and standards. To assist you in this decision you can ask the opinion of any one of 20 people who indicated that they have some knowledge on the topic. High Currency Medium Currency Low Currency High Accuracy Medium Accuracy Low Accuracy High Relevance Medium Relevance Low Relevance Example 5 Due to recent security problems in your company, you are asked to devise a detailed policy for company Internet usage. To assist you in your task you can ask the opinion of any one of 20 people who indicated that they have some knowledge on the topic. High Currency Medium Currency Low Currency High Accuracy Medium Accuracy Low Accuracy High Relevance Medium Relevance Low Relevance 119 Example 6 You received a small budget for updating your computer hardware. You now need to decide what computer and peripherals to purchase and from where. To assist you in this decision you can ask the opinion of any one of 20 people who indicated that they have some knowledge on the topic. High Currency Medium Currency Low Currency High Accuracy Medium Accuracy Low Accuracy High Relevance Medium Relevance Low Relevance Example 7 You decided to pursue an MBA degree to continue your education. You now have to select what universities you should apply to (where would be the best place to study, who has the best reputation, etc.). To assist you in this decision you can ask the opinion of any one of 20 people who indicated that they have some knowledge on the topic. High Currency Medium Currency Low Currency High Accuracy Medium Accuracy Low Accuracy High Relevance Medium Relevance Low Relevance 120 Second interview results - by example Person Currency (years) Accuracy (%) Relevance(%) MBA Admission 1 <2 2-5 >5 >80 50-80 <50 >80 30-80 <30 2 <2 2-4 >4 >70 50-70 <50 >80 60-80 <60 3 3 5 7 80 60 40 80 60 40 4 <1 1-3 >3 >95 85-95 <85 >95 75-95 <75 5 <3 3-5 >5 >80 50-80 <50 >80 50-80 <50 6 3 5 >10 80 70 <50 80 60 <50 7 0.25 1 2 90 75 50 90 75 50 8 <2 2-10 >10 >70 50-70 <50 >70 30-70 <30 9 <2 2-5 >5 >75 50-75 <50 >75 50-75 <50 Selected <1 3 >10 95 70 30 95 70 30 MBA Selection 1 <1 1-3 >3 >90 70-90 <70 >80 30-80 <30 2 <1 1-3 >3 >90 80-90 <80 >85 70-85 <70 3 2 5 10 90 70 50 90 70 50 4 <1 1-2 >2 >90 80-90 <80 >90 70-90 <70 5 <3 3-5 >5 >80 50-80 <50 >90 60-90 <60 6 2 3 >5 75 60 50 85 70 50 7 0.5 2 5 80 60 50 80 60 50 8 <1 1-5 >5 >80 50-80 <50 >70 30-70 <30 9 <1 1-5 >5 >85 45-85 <45 >80 50-80 <50 Selected <1 3 >10 95 70 30 95 70 30 Internet policy 1 <2 2-4 >4 >80 60-80 <60 >80 30-80 <30 2 <.5 0.5-2 >2 >90 80-90 <80 >85 70-85 <70 3 1 2 3 90 75 60 90 70 50 4 <1 1-2 >2 >95 85-95 <85 >90 80-90 <80 5 <2 2-3 >3 >80 50-80 <50 >70 40-70 <40 6 0.25 0.5 >1 95 85 <75 95 85 <70 7 1 3 5 80 60 40 80 60 40 8 <0.5 0.5-2 >2 >90 60-90 <60 >80 60-80 <60 9 <0.5 1.5-2 >2 >85 50-85 <50 >90 50-90 <50 Selected 6 months 2 >6 95 70 30 95 70 30 Business plan 1 <2 2-10 >10 >80 50-80 <50 >70 30-70 <30 2 <.5 0.5-1 >1 >80 60-80 <60 >90 70-90 <70 3 2 4 5 85 70 60 85 70 50 4 <.5 0.5-1.5 >1.5 >95 70-95 <70 >95 85-95 <85 5 <2 2-5 >5 >90 70-90 <70 >80 50-80 <50 6 1 3 >3 90 80 <50 85 60 30 7 2 4 15 90 75 50 90 75 50 8 <0.5 0.5-5 >5 >80 60-80 <60 >80 40-80 <40 9 <1 1-5 >5 >90 60-90 <60 >90 60-90 <60 Selected 6 months 3 >10 95 70 30 95 70 30 PC Purchase (not used) 1 <.5 0.5-1 >1 >90 70-90 <70 >90 60-90 <60 2 <.25 0.25-1 >1 >90 70-90 <70 >80 60-80 <60 3 1 2 3 90 70 50 90 70 50 4 <1 1-1.5 >1.5 >95 80-95 <80 >90 80-90 <80 5 <3 3-5 >5 >70 40-70 <40 >80 60-80 <60 6 0.25 0.75 >1.5 95 90 <80 95 90 <85 7 0.25 0.5 1 90 75 50 90 75 50 8 <0.5 0.5-2 >2 >70 50-70 <50 >80 50-80 <50 9 <0.5 1.5-2 >2 >80 50-80 <50 >75 50-75 <50 Ethics polic) / (not used) 1 <5 5-10 >10 >80 50-80 <50 >80 30-80 <30 2 <2 2-4 >4 >80 60-80 • <60 >75 50-75 <50 3 4 6 8 85 70 60 80 60 50 4 <1 1-2 >2 >90 80-90 <80 >90 80-90 <80 5 <1 1-3 >3 >90 60-90 <60 >90 60-90 <60 6 1.5 2 >5 75 65 45 75 60 40 7 0.5 1 2 90 75 50 90 75 50 8 <2 2-10 >10 >70 50-70 <50 >70 30-70 <30 9 <3 3-10 >10 >80 50-80 <50 >80 50-80 <50 Used car (not used) 1 <3 3-10 >10 >80 50-80 <50 >80 30-80 <30 2 <2 2-4 >4 >85 70-85 <70 >90 80-90 <80 3 6 10 15 80 60 40 85 60 50 4 <0.25 0.25-1 >1 >95 70-95 <70 >95 80-95 <80 5 <0.5 0.5-2 >2 >80 40-80 <40 >80 50-80 <50 6 0.5 0.75 >1 95 85 <80 >95 85 <80 7 2 5 10 80 60 40 80 60 40 8 <1 1-5 >5 >80 50-80 >50 80 40-80 40 9 1-5 5-15 >15 >75 50-75 <50 >80 50-80 <50 Table 23: Data from second interview 122 Method of study: The study will be conducted using a method aimed at identifying the trade-offs that people make between properties of a specific product in the selection process of that product. In our study the product examined is knowledge and we are interested in identifying the most important characteristics of the knowledge and the source of the knowledge that people take into account when deciding whether to use it. The information gained from the study will assist us in proposing a more effective design for knowledge management systems and in avoiding problems of information overload and trust in the knowledge management system, that exist today. Participants in the study will be presented with a knowledge search task - i.e. finding an answer to a question presented to them - and will be asked to evaluate potential knowledge sources that can assist them in answering the question. Each time, the characteristics of the source presented to them will be varied. A more detailed example of the process is available on our website at http://knowledqestudy.commerce.ubc.ca/conioint.html Time required: We estimate that participants will spend around 30 minutes completing the questionnaire. As a token of our appreciation for participating in the study we will provide participants with a small gift. Number of participants: We would appreciate having 15 to 20 participants from «Company». Who can participate: We are looking for people who in their work often or occasionally seek knowledge from others people, documents, or other sources within or outside the organization. An example is someone who is working on a project and may wish to consult with co-workers who have worked on similar projects before. 124 Conjoint questionnaire |UBG| The'Univ.ers^ vjSjal Facultj'.of..C6mmefcecVlB Investigator: Dr. Yair Wand. Management Information Systems, Faculty of Commerce, UBC. Tel: (604) 822-8395 Co-investigator: Dorit Nevo. Management Information Systems, Faculty of Commerce, UBC. Tel: (604) 822-4772 Thank you for accepting the invitation to participate in our survey on users' requirements from Knowledge Management Systems. A Knowledge Management System is a system that supports various knowledge management activities in the organization, such as knowledge generation, knowledge sharing, and the storage of existing knowledge. The goal of the study is to better understand users' requirements from knowledge search and the characteristics of knowledge sources that are most important to users. The results ofthe study will be used to enhance our understanding of users' requirements and for designing better Knowledge Management Systems. In the followi ng pages you wi II be asked to evaluate how Ii kery you are to use different knowledge sources and thei r knowledge based on information that we will provide you about these sources and their knowledge. We estimate that the questionnaire will take approximately 30 to 35 minutes to complete In return for your participation in the study you will receive a small gift Participation in this study is entirely voluntary. You may withdraw from the study at anytime at your own discretion. Your responses to all parts of the study will be summarized anonymously and be kept strictly confidential - no individuals or organizations will be identified in any of the final reports. If you have any questions regarding this study, please feel free to call Dorit Nevo at (604) 822-4772 or send an email. If you have any questions about your treatment or rights as a research participant you may contact the Director of Research Services at the University of British Columbia at (604) 822-8598. By completing and submitting this questionnaire you agree that you have read and understood the above information and have decided to participate in the study Before beginning the study please take a few minutes to complete the information below. This information is important for our analysis and will not be used for any other purposes. A. General Information Please answer the questions below to the best of your ability. If you are currently in school you'may leave some of the fields empty or answer based on the last job you had. LAge: |- El 2. Level of Education: I: !EI 3. Profession: I i 4. Job Title: I i 5. Years working: | '• 6. Years with this company: | ! 7. Years i n present positi on: | I B. Knowledge search 8. How Frequently do you use external knowledge (knowledge other than your own) in your work? |- [£) 9. What are the most common sources Of knowledge that you use? You may select more than one answer O People G Books (j Databases O Web sources n Other (specify): | | 10. How familiar are you with these sources? |- IEI (Note: Please refer to yourseerage level of familiarity with ill sources. For example, if you are very familiar with soma but not at all familiar with others select 'somewhat familial*) 11. Are you familiar with the concept of knowledge management systems? | ; El 12 Are you currently using any of the following? • Lotus Notes or similar products D Intranet O Best practices database D ERP systems • Other systems that support your knowledge search (please provi de detai Is) C. Organizational Information 13. What is the size of your organization?!: 14. In what industry does your organization operate? [ Please submit this form and continue to the questionnaire fSubrriF| Contact Information: Dorit Nevo Management Information Systems, Faculty of Commerce. UBC. Tel: (604) 822-4772 UBcy The'Uriivergit'y.'ofBritisri Columbia'-&r%M Faculty ..of Comm«ce'&'B,usihess : Adrninisiration In this study we ask for your help in evaluating knowledge and knowledge sources. The context of the study is a knowledge search task, that is -you will be given a specific question and will be asked to choose the knowledge source that you wish to consult in answering the question. There are two parts for this study - one part deals with evaluating the trade-offs between the characteristics of different knowledge sources - people that you will consult with, and the other part deals with evaluating the trade-offs in the characteristics of the knowledge underlying the answer to the question provided. Before beginning the evaluation of the sources and knowledge, we would like you to familiarize yourself with the knowledge search task described below. For the purpose of this study please consider the following knowledge search task: Due to recent security problems in your company, you are asked to devise a detailed policy for company Internet usage. To assist you in your task you can ask the opinion of any one of 20 people who indicated that they have some knowledge on the topic. Please rate your expertise on this topic (Internet usage policy): 1 o 2 0 3 G 4 c 5 c Low High After you have read and understood the task above, please continue to Part A below. If - at any point-you have difficulties understanding the questionnaire please email vour question to Dorit Nevo. Part A: Characteristics of Knowledge Sources A knowledge source is a person that you may consult with to find the knowledge required for completing the Internet usage policy task described above. Knowledge sources are evaluated based on the seven criteria shown in the Table below. One criterion, for example, is the experience of the source in the knowledge domain - in this case, the Internet usage policy task. To describe the source's experience we use the number of years that the source has working in this domain. There are three levels of experience - the source has been working in the domain for 10 years or more, working for 5 or 6 years, or working for 2 years or less. Similarly, to describe the source's willingness to help you with your task we use two levels - 'high', indicating that the source is very willing to assist you, or 'low1- indicating that the source is not too willing to assist. Please take a few minutes to carefully review the seven criteria and their levels. Factor Description Possible levels Knowledgeable Extent of domain knowledge the source has - the levels representing how knowledgeable the source is in the domain Very knowledgeable Somewhat knowledgeable Not knowledgeable Trustworthiness The level in which the source can be trusted to provide credible knowledge High Low Awareness of other resources If the source does not know that answer he or she would be able to refer you to other potential sources Yes No Experience Years of experience the source has in the domain 10 years or more 5 or 8 years 2 years or less Willingness to help The extent to which the source is friendly, approachable, and is willing to provide the knowledge High Low Communications skills The source can clearly communicate the knowledge and explain ideas in a simple way Good Reasonable Poor Up to date The source is updated in the domain and is aware of the most recent knowledge developed Yes No Next you will be asked to evaluate the 20 potential knowledge sources according to these criteria. Before we begin, please make sure the task is clear to you by reviewing the example below. 128 Example Based on trie knowledge search task described above (the Internet usage policy task) and your personal preferences, please indicate - for the person profiled below - how likely you are to use this person as your knowledge source: Example #1 -Knowledgeability I Trustworthiness Awareness of other resources Experience Willingness to help Communications skills Up to date Very knowledgeable ; Yes 5 or 6 years High Reasonable No : 0 1 0 2 G 3 <~>A 0 5 G 6 0 7 O B O g 010 O n 1 am not at all I may or may I would iikeiy to not certainly use this source use this source use this source Based on the information provided in the table we can derive the following about the knowledge source: 1. The source is very knowledgeable in the domain 2. The trustworthiness of the source is high 3. The source is aware of other resources 4. The source has 5 or 6 years of experience in the domain 5. The source is willing to help 6. The source's communications skills are reasonable 7. The source in not up to date You are asked to indicate - on the scale of 1 to 11 presented below the table - how likely you are to use this person as your knowledge source. Once you feel comfortable that the above is clear, please continue with the questionnaire below. If you have any questions please email Dorit Nevo. Based on the Internet usage policy task and your personal preferences, please indicate - for each profile below- how likely you are to use this person as your knowledge source: Profile #1 i i ij Awareness of Knowledgeability n Trustworthiness ij other ij ]| resources Experience Willingness to help Communications skills Up to date! Somewhat . . - i II Low Yes knowedgeab e 10 years or more High Poor Yes | 0 1 G 2 C 3 C 4 I am not at all likely to use this source 0 5 G 6 0 7 0 8 I may or may not use this source 0 9 o 10 o n I would certainly use this source Profile #: Knowledgeability Trustworthiness Awareness of other resources Experience Willingness to help Communications skills Up to date Somewhat knowledgeable Low Yes 5 or 6 years High Good No * O l 0>2 0 3 O 4 I am not at all likely to use this source 0 5 0 6 0 7 0 8 I may or may not use this source O g O 1 0 0 1 1 I would certainly use this source Profile #3 • Knowledgeability Trustworthiness Awareness of othBr resources Experience Willingness to j help i Communications skills Up ta date Not knowledgeable High No 10 years or more Low Reasonable Yes 0 1 0 2 0 3 0 4 I am not at all likely to use this source 0 5 0 6 0 7 ' 0 8 I may or may not use this source 0 9 0 1 0 0 1 1 I would certainly use this source 129 Profile #4 " ' Knowledgeablity Trustworthiness Awareness of other resources Experience Willingness to help Communications skills Up to date Very knowledgeable High Yes 10 years or more Low Good No G1 0 2 O3 O4 I am not at all likely to use this source 05 C6 07 O B I may or may not use this source C9 O10 C11 I would certainly use this source Profile #5 ' - - - | Knowledgeablity Trustworthiness \ Awareness of other resources Experience Willingness to help Communications skills Up to date) Very knowledgeable H.gh No _ 5 or 6 years High Poor No j O i C 2 O3 0>4 O5 C 6 <"-7 C 8 O9 G l O C11 I am not at all I may or may I would likely to not certainly use this source use this source use this source Profile #6 Knowledgeability it Awareness of :i i Trustworthiness ;i other -I Experience M resources i| J Willingness to help Communications skills Up to date Not knowledgeable High i[ Yes ;| 5 or6 years High Good Yes j C 1 0 2 0 3 O4 0 5 C B 0 7 0 8 0 9 0 1 0 0-11 I am not at all ' I may or may I would likely to not certainly use this source use this source use this source Profile #7 Knowledgeablity Trustworthiness Awareness of\ other resources Experience Willingness to help Communications skills Up to date Somewhat knowledgeable Low No 5 or 6 years Low Good Yes C 1 0 2 C 3 0 4 I am not at all likely to use this source 0 5 C 6 0 7 0 8 I may or may not use this source 0 9 0 1 0 011 I would certainly use this source Profile #8 Knowledgeability \ Trustworthiness Awareness of 1 other 1 resources :i Experience jl Willingnes j! help to 1 Communications skills [ ] lUp to date; I | Very knowledgeable | Low | No"" \ 2 years or less ]| High Good r n° i O l 0 2 0 3 " O4 0 5 0 6 C 7 0 8 09 C1Q o n I am not at all I may or may I would likely to not certainly use this source use this source use this source Profile #9 Knowledgeability Trustworthiness Awareness of other resources Experience Willingness to help Communications skills Up to date Very knowledgeable Low Yes 5 or 6 years Low Reasonable No 0 1 0-2 0 3 04 I am not at all likely to use this source O5 0-6 0 7 0 8 I may or may not use this source 0 9 C 1 0 011 I would certainly use this source Profile #10 Knowledgeability Trustworthiness ; Awareness of other resources Experience Willingnes help s to i Communications i skills Up to datej Somewhat knowledgeable High No 1D years or more High Reasonable No 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 O10 011 I am not at all likely to use this source I may or may not use this source I would certainly use this source Profile #11 Knowledgeability Trustworthiness Awareness of other resources Experience Willingnes help s to Communications skills Up to date Very knowledgeable Low Yes 2 years or less High Reasonable Yes | 0 1 0 2 0 3 0 4 0 5 0 6 0-7 0 8 0 9 O10 011 I am not at all likely to use this source I may or may not use this source I would certainly use this source Profile #12 Knowledgeability Trustworthiness Awareness of other resources Experience Willingness to help Communications skills Up to date Very knowledgeable High No 2 years or less Low Poor Yes O 1 O 2 0 3 OA 0 5 0 6 0 7 0 8 0 9 C 1 0 011 I am not at all likely to use this source I may or may not use this source I would certainly use this source Profile #13 Knowledgeability Trustworthiness Awareness of other resources Experience Willingnes help s to Communications skills Up to date Very knowledgeable High Yes 10 years or more High Good Yes 0 1 0 2 C 3 OA OS 0 6 0 7 0 8 0 9 Oio 011 1 am not at all likely to use this source I may or may not use this source I would certainly use this source (Profile #14; I I Knowledgeability I || Awareness of Trustworthiness u other ]i resources Experience Willingnes help s to Communications skills Up to date | Very knowledgeable Low Yes 2 years or less Low Reasonable Yes j 0 1 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 011 I am not at all likely to use this source I may or may not use this source I would certainly use this source Profile #15 ; j j, i. Knowledgeability Trustworthiness Awareness of other resources Experience Willingness to help Communications skills Up to date Somewhat knowledgeable Low Yes 5 or 6 years High Reasonable Yes 0 1 0 2 0 3 OA 0 5 0 6 0 7 0 8 0 9 O l O 011 I am not at all likely to use this source I may or may not use this source I would certainly use this source Profile #16 K n o w l e d g e a b i l i t y Trustworthiness A w a r e n e s s of other resources E x p e r i e n c e W i l l i n g n e s s to help C o m m u n i c a t i o n s skill s Up to date Not knowledgeable Low No 2 years or less High Good No C 1 C 2 G 3 • O 4 0 5 0 6 0 7 0 8 O 9 0 1 0 O H I am not at all likely to use this source I may or may not use this source I would certainly use this source Profile #17 -K n o w l e d g e a b i l i t y Trustworthiness. A w a r e n e s s other resources of E x p e r i e n c e Willingness help 5 to C o m m u n i c a t i o n s skill s Up to date Very knowledgeable Low No 10 years or more Low Good Yes O i C 2 0 3 O 4 0 5 0 - 6 0 7 0 8 0 9 0 1 0 G 1 1 I am not at all likely to use this source I may or may not use this source I would certainly use this source Profile #18 - i K n o w l e d g e a b i l i t y Trustworthiness A w a r e n e s s of other r e s o u r c e s E x p e r i e n c e W i l l i n g n e s s to help C o m m u n i c a t i o n s skill s Up to date Not knowledgeable Low Yes 10 years or more Low Poor No C 1 0 2 0 3 0-4 0 5 0 6 0 7 0 8 O 9 0 1 0 0 1 1 1 am not at all likely to use this source I may or may not use this source I would certainly use this source Profile #19 • ' "* \ K n o w l e d g e a b i l i t y Trustworthiness A w a r e n e s s other resources of E x p e r i e n c e W i l l i n g n e s help s to C o m m u n i c a t i o n s skill s Up to date Very knowledgeable Low No 10 years or more High Good No 0 1 0 2 C 3 0 4 0 5 0 6 0 7 0 8 0 9 O 1 0 011 I am not at all likely to use this source I may or may not use this source I would certainly use this source Profile #20 K n o w l e d g e a b i l i t y Trustworthiness A w a r e n e s s of other r e s o u r c e s E x p e r i e n c e W i l l i n g n e s help s to C o m m u n i c a t i o n s skill s Up to date Somewhat knowledgeable High Yes 2 years or less Low Good No o i ' 0 2 0 3 0 4 0 5 0 6 0 7 0 8 O g O 1 0 011 I am not at all likely to use this source 1 may or may not use this source I would certainly use this source When you have completed evaluating all profiles please submit your answers and continue to Part B. | Submit | Contact Information: Dorit Nevo Management Information Systems. Faculty of Commerce. UBC. Tel: (604) 822-4772 |yj|Q The'Uritversi^ A Faculty;iofCommerce,'& Busiiiess:A.dmirii£ftrahon-, Part B: Characteristics ofthe Knowledge Instructions In this part you will be presented with similar profiles as in part A of the questionnaire. However, the profiles will now describe the knowledge itself rather than the knowledge source. 'Knowledge' is often defined as "information in action", stating that an important requirement for defining knowledge is our ability to act based on it. For the purpose of this study please consider the desired characteristics ofthe knowledge conveyed in the answer to your question. For the task described above, this knowledge would be - for example - how to write an internet usage policy. Knowledge is evaluated based on the five criteria shown in the Table below. For example, to indicate the relevance ofthe knowledge to the Internet usage policy task we use a percentile relevance ranking similar to the one used by many search engines. There are three potential relevance levels - 95%, 70%, and 30% - indicating the level of relevance out of 100%. which means that the knowledge is fully relevant to the task. Similarly, the extent of the knowledge can be evaluated using two levels - the knowledge can be either 'Comprehensive' or 'Not Comprehensive'. Please take a few minutes to carefully review the criteria and levels below. Factor Description Possible levels Supported The number of sources that support or refute this knowledge 2 or more source support this knowledge No sources support or refute this knowledge 1 or more sources refute ! the knowledge ! Extent The level of detail ofthe knowledge Comprehensive Not Comprehensive Accuracy A percentile ranking of the level of accuracy - lack of errors - of the knowledge (100% means that the knowledge contains no errors) 95% 70% 30% Relevance A percentile ranking ofthe relevance ofthe answer to the Internet usage policy task (100% means that the knowledge is fully relevant to the question) 95% 70% 30% Currency The number of years since the knowledge was created Less than 6 months old 2 years old Over 6 years old I Next you will be asked to evaluate the knowledge conveyed in 20 potential answers to your question (regardless ofthe source ofthe knowledge) according to the above criteria. Based on the Internet usage policy task and your personal preferences, please indicate - for each profile below - how likely you are to use this knowledge: i-Profile #1 it ^ Supported n Extent I Accuracy ;i Relevance j!Supported by 2 or more sources]! Not Comprehensive I 30% I 30% Currency Less than 6 months old C 1 C 2 I am not at all likely to use this knowledge 0 3 0 4 0 5 O B 0 7 I may or may not use this knowledge c a 0 9 d o on I would certainly use this knowledge [Profile #2 i Supported i H Supported by 2 or more sources i| Extent Comprehensive Accuracy 95% Relevance 95% Currency Less than 6 months old 0 1 0 2 0 3 0 4 C 5 C 6 0 7 I am not at all I may or may likely to not use this knowledge use this knowledge O g C I O C11 I would certainly use this knowledge 133 |Profile#3 1 i Supported < Extent ii Accuracy ji Relevance i| Currency 1 | Refuted by 1 or more sources ; Comprehensive 70% || 30% ij Over 6 years old j C 1 C 2 C 3 G 4 Cs C 6 C-7 0 8 0 9 O-10 o n 1 am not at all 1 may or may I would likely to not certainly use this knowledge use this knowledge use this knowledge (Profile #4 _ J | Supported Extent Accuracy Relevance [ Currency j Refuted by 1 or more sources Comprehensive 30% 95% j Less than 6 months [ old 0 1 0 2 0 3 0 4 CS O 6 O 7 0 8 0 9 0 1 0 0/11 I am not at all likely to use this knowledge - us I may or may not B this knowledge I would certainly use this knowledge (Profilers. . . . . | | -Supported Extent j | Accuracy Relevance !j Currency j | Refuted-by '^1 ,'bri more 'sources' ; Comprehensive || ^30% 95%^ J Over'6'years'6ld | °1 02j 0 3 0 4 0 5: 0 6 0 7 O S . 0 9 OTO- O i l 1 am.nottat.all. likely.to- _ use tfiis:knowledge. 1 mayor.may* ;not: use this.knowledge I would _ certainly usethis knowledge profile #6 I ! Suppmtud Lxtunt Arcuracy Relevance ii Currency | jSupported:by.2or. more-sources || Not Comprehensive i| 95% 95%- :| Over 6-years old j p-i: C]2. G 3 G'4. 0 5 / 0 6 0 7 os c ;9 •.010; -O.l-I .lianvnot at all 1 may or may 1 would likely to ' noi certainly. use thisikhowledge' use this knowledge use^this knowledge Profile Ul - ""T~~" "7 "^"""1 Supported i| Extent i Accuracy |i Relevance I Currency t Supported by-2-ormore;sources|| Comprehensive i| '70% || 70%' :| .Ov'er6:,years:old j O l 0,2; -G3 0 4 : 0 5 0 6 0;7 .0/8 0 9 0 1 0 0-11 j.am not;at all' likely to usethis knowledge I may or may ... >noV use.this knowledge I would ... . certainly. usf:t his; knowledge |Profile #8 . - . | | Supported Extent || Accuracy Relevance Curr miry Supported by 2 or more sources Not Comprehensive jj 95% 95% Less than 6 months old 0 1 0 2 0 3 0-4 I am not at all likely to use this knowledge 0 5 0 6 0 7 0 8 I may or may not use this knowledge 0 9 0 1 0 O n I would certainly use this knowledge 134 (Profile #9 * • ! | Supported Extent Accuracy Relevance Curency Refuted by 1 or more sources Comprehensive 70% 95% Less than 6 months old C1 C2 C3 G4 C5 C6 C7 OS 09 C10 o n 1 am not at all likely to 1 may or may not I would certainly use this knowledge use this knowledge use this knowledge j[Profile#10 Supported No sources support or refute the knowledge Extent Comprehensive ALLUiacy 95% || Relevance 70% II Curency jlLess than 6 months old; G1 0 2 C3 0 4 C5 0 6 C7 I am not at all I may or may likely to not use this knowledge use this knowledge 08 0 9 0 1 0 on I would certainly use this knowledge •Profile #11 . .j | Supported j Extent i| Accuracy Ji Relevance :[ Curency | [ Supported by 2 or more sources j Not Comprehensive j| 70% ll 7 0 % lj Over 6 years old j 01 0 2 C3 G4 G5 06 G7 08 09 010 C11 I am not at all I may or may I would likely to not certainly use this knowledge use this knowledge use this knowledge Profileffl2 _ _____: ______ • - • . ^ | Supported I Extent i Accuracy |i Relevance f Curency | Supported by 2 or more sources |[ Comprehensive |J 30% | 70% I 2 years old Oi G2 G3 OA 05 06 C7 06 09 C10 G11 I am not at all I may or may I would likely to not certainly use this knowledge use this knowledge use this knowledge iProfile #13 . - ' I | Supported j[ Extent 1 Accuracy |[ Relevance i| Curency j | Supported by 2 or more sources jj Comprehensive • J| 95% }[ Over 6 years old j 01 0 2 O 3 C4 05 06 07 08 G9 G10 011 I am not at all 1 may or may I would likely to not certainly use this knowledge use this knowledge use this knowledge [Profile #14 I xlent : : : ' CuiHi-y | Supported Accuracy RuluvcinLc Refuted by 1 or more sources Not Comprehensive 95% J 7 Q % Less than 6 months old 01 0 2 O 3 G4 05 OB 07 OB 09 O10 o 11 I am not at all I may or may I would likely to not certainly use this knowledge use this knowledge use this knowledge 135 |Profile#15 - • • . j I Supported Txu nt Accuracy , Relevance Cuiiiiniy No sources support or refute* the knowledge , Comprehensive 95% [ 30% | Over 6 years old C 1 02 03 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 011 I am not at all I may or may I would likely to not certainly use this knowledge use this knowledge use this knowledge i|Profile#16_ , - • . j II Supported Extent Accuracy Relevance Currency jj Supported by 2 or more sources Comprehensive i 70% 30% Less than 6 months old 0 1 G 2 0 3 0-4 0 5 0 6 0 7 0 8 0 9 0 1 0 O n 1 am not at all I may or may I would likely to not certainly use this knowledge use this knowledge use this knowledge Profile #17' Supported Extent Accuracy Relevance Currency | No sources support or refute, the knowledge Not Comprehensive 30% ! 95% Less than 6 months old C 1 0 2 0 3 0 4 0 5 0 6 - 0 7 0 8 0 9 0 1 0 011 I am not at all likely to use this knowledge I may or may not use this knowledge I would certainly use this knowledge (Profile #18 | I Supported Extent i| Accuracy il Relevance i| Currency j | Refuted by 1 or more sources Not Comprehensive J | 95% J | 30% ij 2 years old j G 1 0 2 O3 I am not at all likely to use this knowledge 04 0 5 0 6 0 7 I may or may not use this knowledge O B 0 9 0 1 0 C-11 I would certainly use this knowledge |Profile #19 : . j [ Supported E X I I M I I Accuracy Relevance Currency j No sources support or refute the knowledge \ Not Comprehensive 70% 95% 2 years old O l 0 2 0 3 0 4 0 5 0 6 0 7 0 8 0 9 0 1 0 011 I am not at all likely to use this knowledge 1 may or may not use this knowledge I would certainly use this knowledge (Profile #20 • I [ Supported ![ Extent 1  Accuracy || Relevance i[ Currency ] | Supported by 2 or more sources j| Comprehensive || 95% | 95% ij 2 years old j G 1 0 2 0 3 0 4 0 5 0 6 0 7 O B 0 9 0 1 0 G11 I am not at all I may or may I would likely to not certainly use this knowledge use this knowledge use this knowledge Before you logout of the study please answer this last question: In the first page we introduced the knowledge search task and said that-to assist you in your task-you may turn to anyone of 20 people that indicated that they have some knowledge on the topic. In selecting a person to turn to - would you place more importance on the charateristjcs of the person (the knowledge source) or on the charateristies of the knowledge itself? To answer this question please divide 100 points between the two options below based on their importance to you. For example, if you only care about the characteristics of the knowledge then write '100' next to "The knowledge" and '0' next to "The knowledge source". If the two are equally important to you then write '50' next to "The knowledge" and '50' next to "The knowledge source", and so on. I j The knowledge; | | The knowledge source When you have completed answering all the questions please submit your answers to logout. | Submit | Contact Information: Dwit_Nevo Management Information Systems, Faculty of Commerce, UBC. Tel: (604) 822-4772 Ug£ji The'University.of^Bntish^Qlurrib'ia-Faculty'of-Commerce.A^Busmess'Administration-•uauyauaiBMisiiiias Thank you for participating in this study. If you would like to receive a copy of the results please email Dorit Nevo. or call (604) 822-4772. As a token of our appreciation for participating in the study we would like to provide you with your seletion of one of two small gifts - a UBC travel mug or umbrella (blue with gold UBC crest) (Picture). In the text box below please indicate what gift you would like to receive as well a location for delivering the gift (e.g. your company and office location), or a mailing address to which the gift can be mailed. 1 m so . I | Submit | Contact Information: Dorit Nevo Management Information Systems, Faculty of Commerce. UBC. Tel: (604) 822-4772 Logout page for undergraduate students: l U g g| The'Uhrvefsityof^British-Columbia' wK^ yf Facialty,:qfComm'ercej&.B usiness;Administratioti,; Thank you for participating in this study. As a token of our appreciation for your help in the study we would like to give you $10. Please see Dorit Nevo at room 673D or send an email to receive the money (click here for the times you can pick up the money). Contact Information: Dorit Nevo Management Information Systems, Faculty of Commerce, U B C . Tel: (604) 822-4772 Appendix D: Conjoint outputs Importance of knowledge attributes These results are the output of SPSS conjoint method. Both Pearson's R and Kendall's tau have high and significant values, indicating strong relation between the actual and predicted results. Kendall's tau for the holdout cards is also high and significant. Overall summary Factor Model Levels Label SUPP d 3 EXT d 2 ACC d 3 REL d 3 CURR d 3 (Models: d=discrete, l = l i n e a r , i = i d e a l . A l l the fa c t o r s are orthogonal. SUBFILE SUMMARY ai= a n t i i d e a l , <=less, >=more) Averaged Importance 19 . 99 9.60 28 .11 22 . 94 19.38 U t i l i t y -.5932 - .2486 . 8418 - .4016 .4016 -1.4448 . 1650 1.2798 -1.1914 .1954 .9961 - . 8733 .2534 .6199 5.4148 Factor SUPP EXT ACC REL CURR CONSTANT Pearson's R = .992 Kendall's tau = .917 Kendall's tau = 1.000 f o r 4 holdouts 1 or more sources refute the knowledge No sources support or refute the knowledge 2 or more sources support the knowledge Not Comprehensive Comprehensive <%=lowACC%> <%=mediumACC%> <%=hignACC%> <%=lowREL%> <%=mediumREL%> <%=highREL%> <%=lowCURR%> <%=mediumCURR%> <%=highCURR%> Si g n i f i c a n c e S i g n i f i c a n c e S i g n i f i c a n c e .0000 .0000 .0208 139 Professionals subgroup Factor Model Levels Label SUPP d 3 EXT d 2 ACC d 3 REL d 3 CURR d 3 (Models: d=discrete, l = l i n e a r , i=ideal, a i = a n t i i d e a l , <=less, >=more) A l l the fac t o r s are orthogonal. SUBFILE SUMMARY Averaged Importance 17.58 9.32 U t i l i t y -.6016 - . 1748 . 7764 .4695 .4695 Factor SUPP EXT 1 or more sources refute the knowledge No sources support or refute the knowledge 2 or more sources support the knowledge Not Comprehensive Comprehensive 27.00 23 . 55 22.55 ACC Pearson's R -1.5407 .1789 1.3618 •1.2358 . 1179 1.1179 -1.3852 .3526 1. 0325 5 . 0772 = . 991 Kendall's tau = Kendall's tau = REL CURR CONSTANT . 929 .667 f o r 4 holdouts <%=lowACC%> <%=mediumACC%> <%=highACC%> <%=lowREL%> <%=mediumREL%> <%=highREL%> <%=lowCURR%> <%=mediumCURR%> <%=highCURR%> Si g n i f i c a n c e = .0000 Si g n i f i c a n c e = .0000 Si g n i f i c a n c e = .0871 SUBFILE SUMMARY No rever s a l s occured i n t h i s s p l i t f i l e group. 140 MBA subgroup Factor Model Levels Label SUPP d 3 EXT d 2 ACC d 3 REL d 3 CURR d 3 (Models: d=discrete, l = l i n e a r , i=ideal, a i = a n t i i d e a l , <=less, >=more) A l l the factors are orthogonal. SUBFILE SUMMARY Averaged Importance U t i l i t y 19 . 08 9.60 .4640 . 2748 . 7387 -.2804 .2804 Factor SUPP EXT 1 or more sources refute the knowledge No sources support or refute the knowledge 2 or more sources support the knowledge Not Comprehensive Comprehens ive 30 .18 ACC 22.59 18 . 54 REL •1.5034 . 1926 1.3108 •1.1588 .3074 . 8514 - .6363 .2083 .4279 5 . 7151 Pearson's R = .986 Kendall's tau = .917 Kendall's tau = 1.000 f o r 4 holdouts CURR CONSTANT <%=lowACC%> <%=mediumACC%> <%=highACC%> <%=lowREL%> <%=mediumREL%> <%=highREL%> <%=lowCURR%> <%=mediumCURR%; <%=highCURR%> Si g n i f i c a n c e S i g n i f i c a n c e S i g n i f i c a n c e .0000 .0000 .0208 SUBFILE SUMMARY No rever s a l s occured i n t h i s s p l i t f i l e group. 141 Undergraduates subgroup Factor Model Levels Label SUPP d 3 EXT d 2 ACC d 3 REL d 3 CURR d 3 (Models: d=discrete, l = l i n e a r , i=ideal, a i = a n t i i d e a l , <=less, >=more) A l l the fac t o r s are orthogonal. SUBFILE SUMMARY Averaged Importance U t i l i t y 21. 87 9 . 75 -.6565 -.2780 . 9345 .4259 .4259 Factor SUPP EXT 1 or more sources refute the knowledge No sources support or refute the knowledge 2 or more sources support the knowledge Not Comprehensive Comprehens ive 27 . 66 22 . 76 ACC 17 . 96 .3577 . 1423 .2155 -1.1827 . 1815 1. 0012 -.6988 .2190 .4798 5 .4539 Pearson 1s R Kendall 1s tau Kendall's tau . 992 REL CURR CONSTANT . 933 1.000 f o r 4 holdouts <%=lowACC%> <%=mediumACC%> <%=highACC%> <%=lowREL%> <%=mediumREL%: <%=highREL%> <%=lowCURR%> <%=mediumCURR%> <%=highCURR%> Si g n i f i c a n c e = .0000 Si g n i f i c a n c e = .0000 Si g n i f i c a n c e = .0208 SUBFILE SUMMARY No rever s a l s occured i n t h i s s p l i t f i l e group. 142 Importance of knowledge source attributes These results are the output of SPSS conjoint method. Both Pearson's R and Kendall's tau have high and significant values, indicating strong relation between the actual and predicted results. However - Kendall's tau for the holdout cards is low and not significant. This indicates low predictive ability of the model constructed and is probably due to a reversal in the data. Overall summary Factor Model Levels Label EXT d 3 TRUST d 2 RES d 2 EXP d 3 HELP d 2 COMM d 3 DAT d 2 (Models: d=discrete, l = l i n e a r , i=ideal, a i = a n t i i d e a l , <=less, >=more) A l l the fac t o r s are orthogonal. SUBFILE SUMMARY Averaged Importance 25 .50 19 . 23 8 . 55 11. 92 12 . 03 13 . 95 8 . 81 U t i l i t y -1.3967 . 1135 1.2832 -1.0866 1.0866 - .4295 .4295 - .2052 - .0397 . 2449 - .6381 .6381 - .4468 - . 1292 .5760 - .3501 .3501 Factor EXT TRUST RES EXP HELP COMM DAT Not knowledgeable Somewhat knowledgeable Very knowledgeable Low High No Yes 2 years or le s s 5 or 6 years 10 years or more Low High Poor Reasonable Good No Yes 5.0816 CONSTANT Pearson's R = .974 Kendall's tau = .900 Kendall's tau = -.333 for 4 holdouts S i g n i f i c a n c e S i g n i f i c a n c e S i g n i f i c a n c e .0000 .0000 .2485 143 Professionals subgroup Factor Model Levels Label EXT d 3 TRUST d 2 RES d 2 EXP d 3 HELP d 2 COMM d 3 DAT d 2 (Models: d=discrete, 1 =linear A l l the factors are orthogonal. SUBFILE SUMMARY Averaged Importance U t i l i t y 25 . 51 20 . 01 i . l c 9 . 92 12 . IE 13 .51 10 . 69 -1.4898 . 0467 1.4431 -1.1646 1.1646 -.4512 .4512 - .2541 . 0386 .2154 - .6982 .6982 -.3455 -.2358 . 5813 - .4451 .4451 Factor EXT TRUST RES EXP HELP COMM DAT Not knowledgeable Somewhat knowledgeable Very knowledgeable Low High No Yes 2 years or le s s 5 or 6 years 10 years or more Low High Poor Reasonable Good No Yes 4.8577 Pearson's R = .969 Kendall's tau = .850 Kendall's tau = -.333 CONSTANT for 4 holdouts Si g n i f i c a n c e S i g n i f i c a n c e S i g n i f i c a n c e . 0000 . 0000 .2485 SUBFILE SUMMARY No rever s a l s occured i n t h i s s p l i t f i l e group. 144 MBA subgroup Factor Model Levels Label EXT d 3 TRUST d 2 RES d 2 EXP d 3 HELP d 2 COMM d 3 DAT d 2 (Models: d=discrete, 1 =linear A l l the factors are orthogonal. SUBFILE SUMMARY Averaged Importance U t i l i t y 25 . 92 1 9 . 4 1 9 . 06 14 .26 11.2c 14 .05 6 . 01 -1 .3311 . 1014 1 .2297 -1 .0743 1.0743 - . 4 2 5 7 .4257 - .2140 .0293 . 1847 - .5034 . 5034 - .3829 - . 1 8 0 2 .5631 - .1689 . 1689 5 .1239 Pearson's R Kendall's tau Kendall's tau Factor EXT TRUST RES EXP HELP COMM DAT CONSTANT . 969 . 917 .333 f o r 4 holdouts Not knowledgeable Somewhat knowledgeable Very knowledgeable Low High No Yes 2 years or le s s 5 or 6 years 10 years or more Low High Poor Reasonable Good No Yes Si g n i f i c a n c e S i g n i f i c a n c e S i g n i f i c a n c e .0000 . 0000 .2485 SUBFILE SUMMARY No rever s a l s occured i n t h i s s p l i t f i l e group. 145 Undergraduates subgroup Factor Model Levels Label EXT d 3 TRUST d 2 RES d 2 EXP d 3 HELP d 2 COMM d 3 DAT d 2 (Models d=discrete l= l i n e a r , A l l the fact o r s are orthogonal SUBFILE SUMMARY Averaged Importance U t i l i t y 25.27 18 .68 8 . 50 11. 86 12 .35 14 .15 9 .19 -1.3768 . 1589 1.2179 -1.0473 1. 0473 - .4188 .4188 -.1720 - . 1220 .2940 - .6741 .6741 -.5399 -.0399 .5798 - .3902 .3902 Factor EXT TRUST RES EXP HELP COMM DAT Not knowledgeable Somewhat knowledgeable Very knowledgeable Low High No Yes 2 years or le s s 5 or 6 years 10 years or more Low High Poor Reasonable Good No Yes 5.1905 CONSTANT Pearson's R = .975 Kendall's tau = .867 Kendall's tau = .000 f o r 4 holdouts S i g n i f i c a n c e S i g n i f i c a n c e S i g n i f i c a n c e .0000 . 0000 .5000 SUBFILE SUMMARY No rever s a l s occurred i n t h i s s p l i t f i l e group. Importance of knowledge and source attributes - by question Importance of knowledge attributes (by question) 35% 30% 25% 20% 15% 10% 5% 0% mm Support Extent Accuracy Relevance Currency BQ4(36) 19.53% 9.78% 26.03% 20.42% 24.24% H Q 3 (38) 20.85% 9.40% 30.78% 21.43% 17.53% • Q2 (40) 19.23% 9.64% 28.83% 24.07% 18.23% FJQ1 (34) 20.42% 9.52% 26.49% 25.93% 17.65% * In brackets - number of people assigned to this question Importance of source attributes (by question) 30% 25% 20% H 15% A 10% 5% A 0% nu Extent Trust Aware of Resources Experience Willing to help Comm. Skills Up to date 0 Q 4 (36) 23.37% 18.64% 8.45% 12.60% 12.96% 14.77% 9.21% H Q 3 (38) 26.90% 22.42% 7.84% 11.56% 11.90% 10.88% 8.51% • Q2 (40) 27.09% 16.79% 9.15% 10.87% 11.22% 15.19% 9.69% • Q1 (34) 24.30% 19.05% 8.83% 12.86% 12.18% 15.02% 7.76% 147 Reference Ackerman, M.S. and C A . Halverson. 2000. Reexamining organizational memory. Communications ofthe A C M , Vol .43 no. 1. Adler, M. and E. Ziglio. 1996. Gazing into the oracle: The delphi method and its application to social policy and public health: Jessica Kingsley Publishers. Alavi, M. 2000. Managing organizational knowledge. In Framing the domains of it management: Projecting the future from the past, ed. R.W. Zmud. Cincinnati: Pinnaflex Educational Resources. Alavi, M. and D.E. Leidner. 2001. Review: Knowledge management and knowledge management systems: Conceptual foundations and research issues. MIS Quarterly, Vol. 25, no. 1: 107-136. Alavi M. and A. Tiwana. 2002. Knowledge integration in virtual teams: The potential role of KMS. Journal of the American Society for Information Science and Technology, Vol. 53 no. 12:1029-1037. Alenderfer, M.S. and R.K. Blashfield. 1984. Cluster analysis. Quantitative applications in the social sciences: Sage Publications. Anand, V.;, C C ; Manz, and W.H. Glick. 1998. An organizational memory approach to information management. Academy of Management Review, Vol. 23, no. 4: 796-809. Anderson, N.H. 1981. Foundations of information integration theory. New York: Academic Press. Argote, L ; and P. Ingram. 2000. Knowledge transfer: A basis for competitive advantage in firms. Organizational Behavior and Human Decision Processes, Vol. 82, no. 1: 150-169. Argote, L , and Ophir, R. 2002. Intraorganizational learning. In J . A. C . Baum (Ed.), The Blackwell Companion to Organizations (pp. 181-207). Oxford, England: Blackwell Business. Argyris, C . and D.A. Schon. 1978. Organizational learning: A theory of action perspective (addison-wesley, reading, ma, 1978). Autio, E., H.J. Sapienza, and J . G . Almeida. 2000. Effects of age at entry, knowledge intensity, and imitability on international growth. Academy of Management Journal, Vol. 43 no. 5. Basch, R. 1990. Measuring the quality of the data: Report on the fourth annual scoug retreat. Database Searcher, Vol. 6, no. 8. Bedard, J . , M. Chi, L. Graham, and J . Shanteau. 1993. Expertise in auditing; discussion. Auditing: A Journal of Practice & Theory, Vol. 12, no. 21. Blackler, Frank. 1995. Knowledge, knowledge work and organizations: An overview and interpretation. Organization Studies, Vol. 16, no. 6. Borgatti, S .P. and Cross, R. 2003. A Relational View of Information Seeking and Learning in Social Networks. Management Science, Vol. 49 no. 4:432-445 Brancheau, J . C and J . C Wetherbe. 1987. Key issues in information systems management. MIS Quarterly, Vol. 11 no. 1. Brown, J .S. ; and P. Duguid. 1991. Organizational learning and communities of practice: Toward a unified view of working, learning, and innovation. Organization Science, Vol. 2, no. 1: 40-57. 148 . 1998. Organizing knowledge. California Management Review, Vol. 40, no. 3: 90-111. Bunge Mario. 1977. Treatise on basic philosophy. Vol 3 Ontology I: The Furniture ofthe World. Reidel, Massachusetts. Cairncross, Frances. 2000. Survey: E-management--a little knowledge... The Economist, November 2000. Cattin, P. and D. R. Wittink. 1982. Commercial use of conjoint analysis: a Survey. Journal of Marketing, Vol. 46: 44-53. Chait, Laurence P. 1999. Creating a successful knowledge management system. The Journal of Business Strategy, March 1999 / April 1999. Cohen, W. and D. Levinthal. 1990. Absorptive capacity: A new perspective on learning and innovation. Administrative Science Quarterly, vol. 35 no. 1 Conner, Kathleen R and C K . Prahalad. 1996. Resource-based theory ofthe firm: Knowledge versus opportunism. Organization Science, Sep/Oct 1996. Couger, D.J. 1988. Key human resource issues in is in the 1990s: View of is executives versus human resource executives. Information & Management, Vol. 14: 161-174. Cross, R., R.E. Rice, and A Parker. 2001. Information seeking in social context: Structural influence and receipt of information benefits. IEEE Transactions on Systems, Man, and Cybernetics- Part C: Applications and Reviews, Vol. 31, no. 4. Daft, Richard L., and Robert H. Lengel. 1986. Organizational Information Requirements, Media Richness and Structural Design. Management Science, May 1986. Daft, R. L. and N. B. Macintosh. 1981. A tentative exploration into the amount and equivocality of information processing in organizational work units. Administrative Science Quarterly, Vol. 26, no. 2. Davenport, T. and L Prusak. 1998. Working knowledge: How organizations manage what they know. Boston, MA: Harvard Business School Press. Davis, D.F. 1989. Perceived usefulness, perceived ease of use, and user acceptance of information technology. Management Information Systems Quarterly, Vol. 13 no. 3: 318-339. Demaris, A. 1992. Logit modeling: Practical applications. Edited by M.S. Lewis-Beck. Quantitative applications in the social sciences: Sage Publications. Dennis, A.R. and S.T. Kinney. 1998. Testing media richness theory in the new media: The effects of cues, feedback, and task equivocality. Information Systems Research, Vol. 9, no. 3. Dixon, Nancy M. 2000. Common Knowledge. Harvard Business School Press (Boston, Massachusetts). Drucker, Peter F. 1989. The new realities. New York: Harper & Row. Eagly, A .H. and S. Chaiken. 1993. The psychology of attitudes: Harcourt Brace Jovanovich College Publishers. Ferrell, O . C and S .J . Skinner. 1988. Ethical behavior and bureaucratic structure in marketing research organizations. Journal of Marketing Research, Vol. 25. 149 Flavell, John.H. 1979. Metacognition and cognitive monitoring, a new area of cognitive - developmental inquiry. American Psychologist Vol. 34, no. 10. GartnerGroup. 2002a. The 2002 knowledge management hype cycle. F. Caldwell, K. Harris, InSide Gartner IGG-01232002-01. GartnerGroup. 2002b. The first wave of smart enterprise suits. Research note number M-16-2729 GartnerGroup. 2000. Can we Size the Knowledge Management Market? Research Note, April, 2000. Gegax, D. and L.R Stanley. 1997. Validating conjoint hedonic preference measures: Evidence from valuing reductions in risk,. Quarterly Journal of Business and Economics: 31-54. Gettier, Edmund. 1963. Is justified true belief knowledge? Analysis 23. Gilliland-Swetland, A . J . 1998. Defining metadata. In Introduction to metadata: Pathways to digital information, ed. M. Baca: Getty Information Institute. Gordon, M.E. and K.D. Lima-Turner. 1997. Consumer attitudes towards internet advertising a social contract perspective. International Marketing Review Vol. 14, no. 5. Grant Robert M. 1996. Toward and Knowledge-Based Theory ofthe Firm. Strategic Management Journal, vol 17:109-122. Green, P. E. and V.R Rao. 1971. Conjoint measurement for quantifying judgmental data. Journal of Marketing Research, Vol. 8. Green, P. E. and V. Srinivasan. 1990. Conjoint analysis in marketing: New developments with implications for research and practice. Journal of Marketing Vol. 54, no. 4. Green, P., D.S. Tull, and G . Albaum. 1988. Research for marketing decisions: Prentice-Hall. Green, P.E. 1984. Hybrid models for conjoint analysis: An expository review. Journal of Marketing Research, Vol. 19. Green, P.E., A.M Krieger, and Y. Wind. 2001. Thirty years of conjoint analysis: Reflections and prospects. Interfaces Vol. 31, no. 3. Green, P.E. and V. Srinivasan. 1978. Conjoint analysis in consumer research: Issues and outlook. The Journal of Consumer Research Vol. 5, no. 2. Gruber, T.R. 1995. Towards principles for the design of ontologies used for knowledge sharing. International Journal of Human-Computer Studies Vol 43, no. 5/6: 907-928. Hair, J .F . , R.E. Anderson, R.L. Tatham, and W . C Black. 1992. Multivariate data analysis. New York: Macmillan Publishing Company. Hansen M. T. 2002. Knowledge networks: Explaining effective knowledge sharing in multiunit companies. Organization Science, Vol. 13 no. 3. Hansen M. T. 1999. The search-transfer problem: The role of weak ties in sharing knowledge across organization subunits. Administrative Science Quarterly, Vol. 44 no. 1. Harman, G . 1968. Knowledge, inference, and explanation. American Philosophical Quarterly'Vol. 5, no. 3. Higgins, M. 1999. Meta-information, and time: Factors in human decision making. Journal of the American Society for Information Science Vol. 50, no. 2. 150 Hoffer Jeffrey A., George Joey F., and Joseph S. Valacich. 2002. Modern Systems Analysis and Design. 3 r d Edition, Prentice Hall. Hollingshead, A .B. 2000. Perceptions of expertise and transactive memory in work relationships. Group Processes and Intergroup Relations Vol. 3: 257-267. Holsapple, C.W.; and K.D. Joshi. 2000. An investigation of factors that influence the management of knowledge in organizations. Journal of Strategic Information Systems Vol. 9: 235-261. Hovland, C.I., I.L. Janis, and H.H. Kelly. 1953. Communication and persuasion; psychological studies of opinion change. New Haven: Yale University Press. Jolliffe, IT. 1986. Principal component analysis. New York: Springer-Verlag. Kale Prashant, Singh Harbir, and Howard Perlmutter. 2000. Learning and protection of proprietary assets in strategic alliances: Building relational capital. Strategic Management Journal, March 2000. Kalfoglou, Y., T. Menzies, K.D. Althoff, and E. Motta. 2000. Meta-knowledge in systems design: Panacea ...Or undelivered promise? The Knowledge Engineering Review Vol. 15, no. 4. Katz, W.A. 1992. Introduction to reference work. New York: McGraw-Hill. K P M G . 1998. Knowledge management research report 1998. . 2000. Knowledge management research report 2000. Accessed. Available from http://kpmg.interact.nl/home/images/kmreport.pdf. . 2003. Insights from KPMG's European Knowledge Management Survey 2002/2003. http://www.kpmg.nl/Docs/Knowledge_Advisory_Services/Documents/KPMG%20KMSURVEY%20 RESULTS%20JAN%202003_877K.pdf Lazinger, S. 2001. Digital preservation and metadata: History, theory, practice. Englewood, Colo.: Libraries Unlimited. Lehrer, Keith. 1990. Theory of knowledge. San Francisco: Westview Press. Lim, K. and I. Benbasat. 2000. The effect of multimedia on perceived equivocality and perceived usefulness of information systems. MIS Quarterly Vol. 24, no. 3. Linstone, H.A. 1978. The delphi technique. In Handbook of futures research, ed. J . Fowles: Greenwood press. Louviere, J . J . 1988. Analyzing decision making: Metric conjoint analysis. Edited by J.L. Sullivan and R.G. Niemi. Quantitative applications in the social sciences: Sage Publications. Luce R. D and Tukey J . W. 1964. Simultaneous conjoint measurement. Journal of Mathematical Psychology, vol. 1: 1-27. Macintosh, Ann, Ian Filby, and John Kingston. 1999. Knowledge management techniques: Teaching and dissemination concepts. International Journal for Human-Computer Studies, vol. 51. Malhotra, Yogesh. 1998. Tools@work: Deciphering the knowledge management hype. Journal for Quality and Participation, special issue on learning and information management, vol. 21, no. 4. Mayer R. C , Davis J . H., and F. D. Schoorman. 1995. An Integrated Model of Organizational Trust. Academy of Management, Vol. 20 no. 3. 151 McCullough, D. 2002. A user's guide to conjoint analysis. Marketing Research, vol. 14, no. 2. McDermott, R. 1999. Why information technology inspired but cannot deliver knowledge management. California Management Review Vol. 41, no. 4:103-117. Menzies, T., K.D. Althoff, Y. Kalfoglou, and E. Motta. 2000. Issues with meta-knowledge. International Journal of Software Engineering and Knowledge Engineering Vol. 10, no. 4. Moreland, R.L, L. Argote, and R. Krishnan. 1996. Socially shared cognition at work: Transactive memory and group performance. In What's social about social cognition?, ed. Nye and Brower:57-86: S A G E publications. Moreland, R.L. and L. Myaskovsky. 2000. Exploring the performance benefits of groups training: Transactive memory or improved communication? Organizational Behavior and Human Decision Processes Vol. 82, no. 1. Moser, Paul K. and Arnold VanderNat. 1987. Human knowledge: Classical and contemporary approaches: Oxford University Press. Nelson, K., S. Nadkarni, V. K. Narayanan, and M. Ghods. 2000. Understanding software operations support expertise: A revealed causal mapping approach. MIS QuarterlyVol. 24, no. 3. Nonaka, I. 1994. A dynamic theory of organizational knowledge creation. Organization Science, Vol. 5, no. 1:14-37. O'Dell, C . and C . J . Grayson. 1998. If only we knew what we know: Identification and transfer of internal best practices. California Management Review Vol. 40, no. 3: 154-174. O'Leary, D. 1999. Using Al in knowledge management: Knowledge bases and ontologies. In The knowledge management yearbook 1999-2000, ed. Cortada and Woods: Butterworth-Heinemann. Paulus P. B. and H. Yang. 2000. Idea generation in groups: A basis for creativity in organizations. Organizational Behavior and Human Decision Process Vol. 82, no. 1. Petty, R.E. and J.T. Cacioppo. 1986. Communication and persuasion: Central and peripheral routes to attitude change. New York: Springer-Verlag. Plant, R. and R.F. Gamble. 1997. Using meta-knowledge within a multi-level framework for kbs development. International Journal of Human-Computer Studies, April, 1997. Polanyi, Michael , 1967, Tacit dimension. London: Routledge & K.Paul Powers, D.A. and Y. Xie. 2000. Statistical methods for categorical data analysis: Academic Press. Pugh, D.S., D.J. Hickson, C.R. Hinings, and C. Turner. 1968. Dimensions of organization structure. Administrative Science Quarterly. Vol. 13. Rosen, Sherwin. 1974. Hedonic Prices and Implicit Markets: Product Differentiation in Pure Competition. Journal of Political Economy, Vol. 82:34-55. Ruggles, R.L. 1997. Knowledge management tools. Boston, MA: Butterworth Heinemann. Rulke, D. and D. Rau. 2000. Investigating the encoding process of transactive memory development in group training. Group & Organization Managementso\. 25: 373-396. Rulke D., Zaheer S., and M. H. Anderson. 2000. Source of managers' knowledge of organizational capabilities. Organizational Behavior and Human Decision Process Vol. 82, no. 1. 152 Schmidt, R., K. Lyytinen, M. Keil, and P. Cule. 2001. Identifying software project risks: An international delphi study. Journal of Management Information Systems Vol. 17, no. 4. Schmidt, R.C. 1997. Managing delphi surveys using nonparametric statistical techniques. Decision Sciences Vol. 28, no. 3. Schulz, Martin, 2001. Organizational Learning. In J . A. C . Baum (Ed.), The Blackwell Companion to Organizations (pp. 181-207). Oxford, England: Blackwell Business. Shannon, C. and W. Weaver. 1949. The mathematical theory of communication: Urbana: University of Illinois Press. Soo, J.T. 1999. Strategies for reducing consumers' risk aversion in internet shopping. The Journal of Consumer Marketing Vol. 16, no. 2. S P S S . 2001. Advanced market research: S P S S . Stein, E. 1995. Organizational memory: Review of concepts and recommendations for management. International Journal of Information Management Vol. 15, no. 1. Stein, Eric W. and Vladimir Zwass. 1995. Actualizing organizational memory with information systems. Information Systems Research Vol. 6, no. 1. Stenmark Dick. 2000. Leveraging tacit organizational knowledge. Journal of Management Information Systems, Vol. 17, no. 3. Stiff, J .B. 1994. Persuasive communication. New York: The Guilford Press. Stoker, D.S. and A.L. Cooke. 1995. Evaluation of networked information sources. Proceedings of the 17th International Essen Symposium: 287-312. Strader T. J . and S. N. Ramaswami. 2002. The Value of Seller Trustworthiness in C 2 C Online Markets. Communications of the ACM, Vol. 45 no. 12. Sussman, S.W. and W.S Siegal. 2003. Information influencein organizations: An integrated approach to knowledge adoption. Information Systems Reseaerch Vol. 14, no. 1. Sveiby, Karl E. 1997. The new organizational wealth, managing and measuring knowledge-based assets. San-Francisco: Berret-Koehler Publishers Inc. Szulanski Gabriel. 2000. The process of knowledge transfer: A diachronic analysis of stickiness. Organizational Behavior and Human Decision Processes, May 2000. Szulanski Gabriel. 1996. Exploring internal stickiness: Impediments to the transfer of best practice within the firm. Strategic Management Journal, Winter 1996. Van Der Heijden, B.I.J.M. 2000. The development of psychometric evaluation of a multidimensional measurement instrument of professional expertise. Hgh Ability Studies Vol. 11, no. 1. Vavra, T., P.E. Green, and A.M Krieger. 1999. Evaluating ez-pass. Marketing Research Vol. 11, no. 3. Walsh, J.P.; and G.R. Ungson. 1991. Organizational memory. Academy of management review Vol. 16, no. 1: 57-91. Walsham, G . 2001. Knowledge management: The benefits and limitations of computer systems. European Management Journal Vo\. 19, no. 6: 599-608. 153 Wegner, D. M. 1987. Transactive memory: A contemporary analysis of the group mind. In Theories of group behavior, ed. B. Mullen and G.R Goethals. New York: Springer-Verlag. Wegner, D.M. 1995. A computer network model of human transactive memory. Social Cognition Vol. 13, no. 3. Wiig, Karl. 1994. Knowledge management: Schema Press, Arlington, Texas. Wijnhoven, F. 1999. Development scenarios for organizational memory information systems. Journal of Management Information Systems Vol. 16, no. 1: 121-146. Wind, J . , P.E. Green, D. Shifflet, and M. Scarbrough. 1989. Courtyard by marriott: Designing a hotel facility with consumer-based marketing models,. Interfaces Vol. 19, no. 1. Wittink, Dick R., and Philippe Cattin. 1989. "Commercial Use of Conjoint Analysis: An Update." Journal of Marketing, vol. 53: 91 -96. Wolfe, R.A. and D.S. Putler. 2002. How tight are the ties that bind stakeholder groups? Organization Science, vol. 13, no. 1. Zack, Michael H. 1999. Managing codified knowledge. Sloan Management Review, Vol. 40, no. 4: 45-58. Zaheer A., McEvily B., and V. Perrone. 1998. Does Trust Matter? Exploring the Effects of Interorganizational and Interpersonal Trust on Performance. Organization Science, vol. 9 no. 2: 141-159 154 

Cite

Citation Scheme:

        

Citations by CSL (citeproc-js)

Usage Statistics

Share

Embed

Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                        
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            src="{[{embed.src}]}"
                            data-item="{[{embed.item}]}"
                            data-collection="{[{embed.collection}]}"
                            data-metadata="{[{embed.showMetadata}]}"
                            data-width="{[{embed.width}]}"
                            async >
                            </script>
                            </div>
                        
                    
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:
http://iiif.library.ubc.ca/presentation/dsp.831.1-0091216/manifest

Comment

Related Items