Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

User-centered design of identity and access management systems Jaferian, Pooya 2014

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


24-ubc_2015_february_jaferian_pooya.pdf [ 5.21MB ]
JSON: 24-1.0167054.json
JSON-LD: 24-1.0167054-ld.json
RDF/XML (Pretty): 24-1.0167054-rdf.xml
RDF/JSON: 24-1.0167054-rdf.json
Turtle: 24-1.0167054-turtle.txt
N-Triples: 24-1.0167054-rdf-ntriples.txt
Original Record: 24-1.0167054-source.json
Full Text

Full Text

User-centered Design of Identity and Access ManagementSystemsbyPooya JaferianMSc, Amirkabir University of Technology, 2006BSc, Amirkabir University of Technology, 2004A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFDoctor of PhilosophyinTHE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES(Electrical and Computer Engineering)The University of British Columbia(Vancouver)November 2014c© Pooya Jaferian, 2014AbstractIT security management (ITSM) technologies are important components of IT security in orga-nizations. But there has been little research on how ITSM technologies should incorporate hu-man and social issues into their design. Identity and Access Management (IAM) systems, as animportant category of ITSM, share such a gap with other ITSM technologies. The overreachinggoal of this research is to narrow the gap between IAM technologies and social context.In the first phase, we developed a set of usability guidelines, and heuristics for design and us-ability evaluation of ITSM tools. We gathered recommendations related to ITSM tools from theliterature, and categorized them into a set of 19 high-level guidelines that can be used by ITSMtool designers. We then used a methodical approach to create seven heuristics for usabilityevaluation of ITSM tools and named them ITSM heuristics. With a between-subjects study, wecompared the usage of the ITSM and Nielsen’s heuristics for evaluation of a commercial IAMsystem. The results confirmed the effectiveness of ITSM heuristics, as participants who usedthe ITSM heuristics found more problems categorized as severe than those who used Nielsen’s.In the second phase, we conducted a field-study of 19 security practitioners to understand howthey do IAM and identify the challenges they face. We used a grounded theory approach tocollect and analyze data and developed a model of IAM activities and challenges. Built on themodel, we proposed a list of recommendations for improving technology or practice.In the third phase, we narrowed down our focus to a specific IAM related activity, access re-iiview. We expanded our understanding of access review by further analysis of the interviews,and by conducting a survey of 49 security practitioners. Then, we used a usability engineeringprocess to design AuthzMap, a novel user-interface for reviewing access policies in organiza-tions. We conducted a user study with 430 participants to compare the use of AuthzMap withtwo existing access review systems. The results show AuthzMap improved the efficiency infive of the seven tested tasks, and improved accuracy in one of them.iiiPrefaceThe materials in chapters 3 to 6 of this dissertation have each been either published or acceptedfor publication. The author of this dissertation conceived of the research idea, performed allthe design and evaluation, except in Chapter 6 where the design and execution of the study#3 were shared by other co-authors. He also wrote all the papers resulting from this research,under the supervision of the co-authors who provided feedback and guidance throughout theresearch process. Below are the ethics information, and publication details for each chapter.• Chapter 3: The materials of this chapter has been published in the CHIMIT (ComputerHuman Interaction for Management of Information Technology) conference. The inter-views in this chapter were approved by the UBC’s Behavioral Research Ethics Board(Certification number: H06-80413, Project title: HOT Admin: Human, Organization,and Technology Centred Improvement of IT Security Administration).Pooya Jaferian, David Botta, Fahimeh Raja, Kirstie Hawkey, and Konstantin Beznosov.2008. Guidelines for designing IT security management tools. In Proceedings of the2nd ACM Symposium on Computer Human Interaction for Management of InformationTechnology (CHiMiT ’08). ACM, San Diego, CA, USA, Article 7 , 10 pages. (accep-tance rate: 29%)• Chapter 4: The materials of this chapter has been published in CHI Extended Abstracts,Symposium On Usable Privacy and Security (SOUPS), and Human-Computer Interac-tion Journal. The user study conducted in this chapter was approved by the UBC’s Be-ivhavioral Research Ethics Board (Certification number: H09-03371, Project title: ITSMHeuristics).Pooya Jaferian, Kirstie Hawkey, Andreas Sotirakopoulos, Maria Velez-Rojas, and Kon-stantin Beznosov. Heuristics for evaluating IT security management tools. Human-Computer Interaction, Vol. 29, Iss. 4, 2014. (impact factor: 3.039)Pooya Jaferian, Kirstie Hawkey, Andreas Sotirakopoulos, Maria Velez-Rojas, and Kon-stantin Beznosov. 2011. Heuristics for evaluating IT security management tools. InProceedings of the Seventh Symposium on Usable Privacy and Security (SOUPS ’11).Pittsburgh, PA, USA, Article 7 , 20 pages. (acceptance rate: 33%, best paper award)Pooya Jaferian, Kirstie Hawkey, Andreas Sotirakopoulos, and Konstantin Beznosov.2011. Heuristics for evaluating IT security management tools. In CHI’11 ExtendedAbstracts on Human Factors in Computing Systems (CHI EA ’11). ACM, Vancouver,BC, Canada, 1633-1638. (acceptance rate: 45%)• Chapter 5: A subset of materials of this chapter has been published in the the CHIMIT(Computer Human Interaction for Management of Information Technology) conference.The interviews in this chapter was approved by the UBC’s Behavioral Research EthicsBoard (Certification number: H08-02527, Project title: IdM CA).Pooya Jaferian, David Botta, Kirstie Hawkey, and Konstantin Beznosov. 2009. A casestudy of enterprise identity management system adoption in an insurance organization.In Proceedings of the Symposium on Computer Human Interaction for the Managementof Information Technology (CHiMiT ’09). Baltimore, MD, USA, Article 7 , 10 pages.(acceptance rate: 33%)• Chapter 6: The initial versions of this chapter has been published as a CHI extendedabstract. Most of the material in the chapter has been accepted for publication in SOUPSconference. The studies presented in this chapter has been approved by the UBC’sBehavioral Research Ethics Board: (1) Access Review Survey ( Certification number:H08-02527, Project title: IdM CA) (2) Comparative Heuristic Evaluation (Certificationnumber: H12-01411, Project title: Access certification interfaces ) (3) User study ofAuthzMap (Certification number: H12-03717 , Project title: AuthzMap Study)vPooya Jaferian, Hootan Rashtian, and Konstantin Beznosov. 2014. To authorize or notauthorize: helping users review access policies in organizations. In Proceedings of theSymposium on Usable Privacy and Security (SOUPS 2014). Pages 301-320. MenloPark, CA. (acceptance rate: 26%)Pooya Jaferian, Hootan Rashtian, and Konstantin Beznosov. 2014. Helping users reviewand make sense of access policies in organizations. In CHI’14 Extended Abstracts onHuman Factors in Computing Systems (CHI EA ’14). ACM, Toronto, ON, Canada,2017-2022. (acceptance rate: 49%)Note that without my supervisor’s guidance and support, this dissertation would not have beenpossible. I therefore have opted to use the term we throughout this dissertation.viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xviGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiiAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxiv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51.2 Organization of the Thesis . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 Background and Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.1 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92.2 Socio-technical Aspects of ITSM . . . . . . . . . . . . . . . . . . . . . . . . . 112.2.1 The HOT Admin Project . . . . . . . . . . . . . . . . . . . . . . . . . 112.3 Identity and Access Management . . . . . . . . . . . . . . . . . . . . . . . . . 12vii2.4 Activity Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142.5 Grounded Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 Guidelines for Designing ITSM Tools . . . . . . . . . . . . . . . . . . . . . . . . 183.1 Background and Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 193.1.1 User Interface Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . 193.1.2 Challenges in IT Security Management . . . . . . . . . . . . . . . . . 213.1.3 Guidelines for IT Security Tools . . . . . . . . . . . . . . . . . . . . . 223.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.3 ITSM Design Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.3.1 General Usability Guidelines . . . . . . . . . . . . . . . . . . . . . . . 263.3.2 Technological Complexity Guidelines . . . . . . . . . . . . . . . . . . 273.3.3 Organizational Complexity Guidelines . . . . . . . . . . . . . . . . . . 313.3.4 Task Specific Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . 363.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 403.4.1 Applying the Guidelines . . . . . . . . . . . . . . . . . . . . . . . . . 403.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 424 Heuristics for Usability Evaluation of ITSM Tools . . . . . . . . . . . . . . . . . 434.1 Background and Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 454.2 Proposed ITSM Heuristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464.2.1 Methods for Creating Usability Heuristics . . . . . . . . . . . . . . . . 464.2.2 Our Methodology for Creating ITSM Heuristics . . . . . . . . . . . . . 484.2.3 Proposed ITSM Heuristics . . . . . . . . . . . . . . . . . . . . . . . . 544.3 Evaluation Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 624.3.1 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 684.4 Evaluation Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 704.4.1 Performance of Individual Heuristics . . . . . . . . . . . . . . . . . . . 75viii4.4.2 Impact of Participants’ Background on Their Performance . . . . . . . 784.4.3 Participants’ Feedback in Post-evaluation Questionnaire . . . . . . . . 794.4.4 Qualitative Feedback During Focus Group/interview Session . . . . . . 804.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 834.6 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 874.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 885 Field Study of Identity and Access Management . . . . . . . . . . . . . . . . . . 905.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 925.1.1 Recruitment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 925.1.2 Interview Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 945.1.3 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 955.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 965.2.1 Definition of Identity and Access Management (IAM) . . . . . . . . . . 965.2.2 Stakeholders . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1005.2.3 The Basic IAM Life Cycle . . . . . . . . . . . . . . . . . . . . . . . . 1015.2.4 ID Creation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1025.2.5 Automatic Provisioning . . . . . . . . . . . . . . . . . . . . . . . . . . 1075.2.6 Manual Provisioning . . . . . . . . . . . . . . . . . . . . . . . . . . . 1125.2.7 Change Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1185.2.8 Identity Removal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1215.2.9 Audit and Accountability . . . . . . . . . . . . . . . . . . . . . . . . . 1235.3 From Manual to Automatic Provisioning . . . . . . . . . . . . . . . . . . . . . 1255.3.1 Coping with Manual Provisioning Challenges . . . . . . . . . . . . . . 1275.3.2 Integration With and Managing the End-points . . . . . . . . . . . . . 1285.3.3 Engineering and Mining Roles . . . . . . . . . . . . . . . . . . . . . . 1295.3.4 Challenges in Automatic Provisioning . . . . . . . . . . . . . . . . . . 1325.4 An Integrated Model of IAM and its Challenges . . . . . . . . . . . . . . . . . 135ix5.4.1 Identity Management Challenges Model . . . . . . . . . . . . . . . . . 1385.5 Grounding the Work in Access Control Literature . . . . . . . . . . . . . . . . 1405.5.1 Theoretical Aspects of Access Control . . . . . . . . . . . . . . . . . . 1405.5.2 Practical Aspects of Access Control . . . . . . . . . . . . . . . . . . . 1445.6 Implications For Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1525.7 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1555.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1566 Designing Usable Access Review Interfaces . . . . . . . . . . . . . . . . . . . . . 1576.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1576.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1596.3 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1606.4 Study 1: Understanding the Activity . . . . . . . . . . . . . . . . . . . . . . . 1616.4.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1616.4.2 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1656.4.3 Access Review Challenges . . . . . . . . . . . . . . . . . . . . . . . . 1706.5 Study 2: Evaluating an Existing Technology . . . . . . . . . . . . . . . . . . . 1756.6 AuthzMap Design Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1786.7 Study 3: Comparative Heuristic Evaluation . . . . . . . . . . . . . . . . . . . . 1886.7.1 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1886.7.2 Recruitment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1896.7.3 Evaluated Interfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . 1906.7.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1916.7.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1976.8 Study 4: Evaluation of AuthzMap . . . . . . . . . . . . . . . . . . . . . . . . 1986.8.1 Evaluation Methodology . . . . . . . . . . . . . . . . . . . . . . . . . 1996.8.2 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2086.8.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209x6.9 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2196.9.1 Efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2206.9.2 Accuracy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2236.9.3 Subjective Satisfaction and Learnability . . . . . . . . . . . . . . . . . 2256.9.4 User Study Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . 2256.9.5 Moving from Prototype to an Actual System . . . . . . . . . . . . . . . 2276.10 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2277 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2297.1 Revisiting Social-technical Gap in IAM . . . . . . . . . . . . . . . . . . . . . 2297.2 Implications Beyond IAM or Access Review . . . . . . . . . . . . . . . . . . . 2317.3 Validity of the Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2338 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2378.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2408.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2428.2.1 Improvements on AuthzMap . . . . . . . . . . . . . . . . . . . . . . . 2428.2.2 Field-study of IAM . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2448.2.3 Heuristic Evaluation of Other IT Security Tools . . . . . . . . . . . . . 2458.2.4 Future Work on Guidelines for ITSM Tools . . . . . . . . . . . . . . . 245Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247Appendices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263A Heuristic Evaluation Study Material . . . . . . . . . . . . . . . . . . . . . . . . . 264A.1 Evaluation Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264A.2 Usability Problem Specification Form (ITSM Condition) . . . . . . . . . . . . 270A.3 Background Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271A.4 Post-Evaluation Questionnaire (ITSM Condition) . . . . . . . . . . . . . . . . 272xiB IAM Field Study Material . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274B.1 Interview Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274B.1.1 Organizational Context . . . . . . . . . . . . . . . . . . . . . . . . . . 274B.1.2 Questions About IAM Process . . . . . . . . . . . . . . . . . . . . . . 276B.1.3 Probing Specific Activities (Depends on Participant’s Role) . . . . . . . 277B.1.4 Questions About IAM Technologies . . . . . . . . . . . . . . . . . . . 281B.1.5 Working/Dealing with Other Stakeholders . . . . . . . . . . . . . . . . 283C Access Certification Survey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286D Detailed Description of AuthzMap, List, and Search . . . . . . . . . . . . . . . . 295E Authzmap Initial Prototypes at Different Levels of Fidelity . . . . . . . . . . . . 299F AuthzMap User Study Material . . . . . . . . . . . . . . . . . . . . . . . . . . . 303xiiList of TablesTable 2.1 Market presence of the leading IdM vendors (Cser, 2009) . . . . . . . . . . 11Table 4.1 Comparison of the major heuristic creation literature. The “T”, “B”, and “?”indicate top-down, bottom-up, and unknown method of heuristic creation. . . 49Table 4.2 Participants’ demographics for each condition. . . . . . . . . . . . . . . . . 65Table 4.3 Details of the four scenarios used during the comparative study. . . . . . . . 68Table 4.4 Examples of the problems identified by the participants. “Context” de-scribes the context in which the problem was identified. “Problem” de-scribes the problem. “Freq.” shows the number of times the problem isreported in the ITSM(I), and Nielsen(N) conditions. “Avg. Sev.” showsthe average severity of the problem. “Heuristics” shows the heuristics withwhich the problems were identified (e.g., I4 means ITSM heuristic #4). “IC”indicates that the problem could not be associated to a heuristic by an ITSMparticipant. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71Table 4.5 Overview of the number and classification of identified problems in eachcondition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72Table 4.6 Individual differences in participants’ ability to find problems. . . . . . . . . 72xiiiTable 4.7 Similarity between individual ITSM and Nielsen’s heuristics. Each cellshows the value of similarity metric for the heuristics denoted by row andcolumn indexes. For each ITSM heuristic, the cell with the highest number(i.e., the most similar Nielsen heuristic) is highlighted. . . . . . . . . . . . . 77Table 4.8 Ability of each of Nielsen’s and the ITSM heuristics to find problems uniqueto their condition. The “Proportion of unique” row shows the proportion ofproblems uniquely found in the Nielsen or ITSM conditions using the corre-sponding heuristic. The “Average severity” row shows the average severityof those unique problems. . . . . . . . . . . . . . . . . . . . . . . . . . . . 78Table 5.1 Interview Participants’ Demographics . . . . . . . . . . . . . . . . . . . . . 92Table 5.2 The use of different access control mechanisms in IT systems (figure from (Con-nor and Loomis, 2010)) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145Table 6.1 Reported problems with the Search interface during heuristic evaluation.The alphanumeric codes show evaluators who reported the problems. Eval-uators with code starting with N used Nielsen’s heuristics, and evaluatorswith codes starting with I used ITSM heuristics to find problems. . . . . . . 177Table 6.2 Overview of the identified usability problems for each of the three evaluatedinterfaces . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192Table 6.3 Classification of participants according to their progress in the study. “Con-sented” indicates those participants who consented to the study. “Started”indicates those participants who at least started the background question-naire. “Finished” indicates those participants who completed all of the studysteps. “Valid” are those participants that we used their data for analysis. . . . 210Table 6.4 Participants Demographics . . . . . . . . . . . . . . . . . . . . . . . . . . . 212xivTable 6.5 Median time to completion (TTC) for each of the tasks (in seconds), andpairwise comparison of TTCs. “A” stands for AuthzMap, “L” stands forList, and “S” stands for Search. The last three columns indicate null hy-potheses (e.g., “A=L” indicates the following null hypothesis: there is nodifference between TTC in AuthzMap (A) condition and List (L) condi-tion). The highlighted cells show the cases where the null hypothesis wasrejected and median TTC for AuthzMap condition was lower than the othercondition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212Table 6.6 Comparing the correctness of participants’ responses to the four compo-nents of the training task. The highlighted cells show the cases where theaccuracy in AuthzMap condition was higher than the accuracy in the othercondition, and the difference was statistically significant. . . . . . . . . . . . 213Table 6.7 Comparing the correctness of participants’ choices in common review task. . 214Table 6.8 Comparing the correctness of participants’ choices in user comparison task. . 215Table 6.9 Comparing the correctness of participants’ choices in privilege accumula-tion task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215Table 6.10 Comparing the correctness of participants’ choices in SoD violation detec-tion task. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216Table 6.11 Comparing the correctness of participants’ choices in application review task. 218Table 6.12 Pairwise comparison of participants’ responses to post-evaluation question-naire. The highlighted cells show the cases where the null hypothesis wasrejected and the AuthzMap ratings were higher than the other interface. . . . 219xvList of FiguresFigure 1.1 Overview of thesis components and the relationship between them . . . . . 8Figure 2.1 Components and internal relations of an activity system . . . . . . . . . . . 15Figure 3.1 Framework of design guidelines for IT security management tools. Thereferences listed under each guideline point to the supporting literature for it. 25Figure 4.1 Overview of the process of developing ITSM heuristics . . . . . . . . . . . 50Figure 4.2 An example of heuristic synthesis process: we used a bottom-up approachby analyzing literature on ITSM tools (a) to create guidelines (b), and a top-down approach (c) using activity theory to extract preliminary heuristics (d)which later were reworded to final heuristics (e). . . . . . . . . . . . . . . . 53Figure 4.3 Study protocol overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 66Figure 4.4 Average proportion of problems found by aggregate of participants in ITSMand Nielsen conditions. We also overlaid the results from Nielsen’s Mantelexperiment (Nielsen and Molich, 1990), and Baker’s Groove and Group-Draw (Baker et al., 2002) experiments to allow comparisons. . . . . . . . . 74Figure 4.5 Problems identified by each participant in each condition. Each row corre-sponds to a participant and each column corresponds to a problem. Partic-ipants in each condition are sorted from top (weak) to bottom (strong) andproblems are sorted from right (easy) to left (hard). . . . . . . . . . . . . . 76xviFigure 4.6 The number and mean severity of problems identified by each heuristic.The ITSM heuristics are shown using black diamonds and Nielsen’s heuris-tics are shown using gray circles. Each heuristic is labeled with its number. 77Figure 4.7 Mean scores of participants’ reported usefulness, learnability, and ease ofapplication for the different heuristics (5=strongly agree, 1=strongly dis-agree). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80Figure 5.1 The overall IAM life-cycle . . . . . . . . . . . . . . . . . . . . . . . . . . 102Figure 5.2 The interactions between stakeholders involved in the manual provisioningactivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113Figure 5.3 The integrated model of IAM and its challenges . . . . . . . . . . . . . . . 137Figure 5.4 Tensions in identity management activity . . . . . . . . . . . . . . . . . . 140Figure 5.5 Access matrix proposed by Lampson . . . . . . . . . . . . . . . . . . . . . 141Figure 5.6 Adoption of role based access control in the organizations (figure from (Con-nor and Loomis, 2010)) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145Figure 6.1 The job title of the survey participants . . . . . . . . . . . . . . . . . . . . 164Figure 6.2 The participants years of IT security and IAM experience. Each dot indi-cates a participant. The blue dashed line is the identity line. The points arejittered to avoid over-plotting. . . . . . . . . . . . . . . . . . . . . . . . . 165Figure 6.3 Overview of access review activity presented in the triangular model ofactivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166Figure 6.4 Survey participants’ responses on who currently performs and who ideallyshould perform access review in their company . . . . . . . . . . . . . . . 168Figure 6.5 Communication channels that should be used during access review fromsurvey participants’ perspective . . . . . . . . . . . . . . . . . . . . . . . . 169xviiFigure 6.6 Descriptive statistics on the scale of access review in survey participants’organization. Participants were allowed not to answer the questions, andtherefore, the number of data points in each graph may not add up to 49. . . 172Figure 6.7 Frequency of performing access review in survey participants’ organization 173Figure 6.8 The list of events that can trigger access review in survey participants’ or-ganization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174Figure 6.9 A screenshot of the Search interface. (1) Reviewer searches for a user. (2)Selects the users. (3) Clicks on the Select button and certifies or revokesaccess privileges in Level 2. A more detailed description of the Searchinterface is available in Appendix D. . . . . . . . . . . . . . . . . . . . . . 176Figure 6.10 Usefulness of different pieces of contextual information during access re-view. Survey participants were asked to rate the usefulness on a 5-pointlikert scale. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183Figure 6.11 Risk indicators during access review. Survey participants were asked torate the risk associated with each item on a 5-point likert scale. . . . . . . . 184Figure 6.12 Mapping between ITSM guidelines, AuthzMap design goals, and the in-terface mechanisms used in Authzmap. Each non-empty cell indicates thatthe guideline in the corresponding row is used to achieve the design goal inthe corresponding column. Letters refer to the actual interface mechanism(in Figure 6.13) used to realize the guideline. . . . . . . . . . . . . . . . . 185Figure 6.13 The three levels of the AuthzMap interface. The reviewer is presented withLevel 1 of the interface. He can go into Levels 2 and 3 for making furthersense of the accesses of the users. The access privileges are shown as filesin this version of the interface for the purpose of a user study presentedin Section 6.8. A more detailed description of the AuthzMap interface isavailable in Appendix D. . . . . . . . . . . . . . . . . . . . . . . . . . . . 187xviiiFigure 6.14 A screenshot of the List interface. Reviewer identifies the user and clickson the View button. Reviewer is presented with the second level of theinterface that includes the list of user’s access privileges. The icon markedas (a) allows batch actions on privileges, and the four small icons (markedas b) do the following (from left to right): sets the access expiry time, writesnotes for each privilege, shows history of actions on each privilege, andshows history of rejections for each privilege. A more detailed descriptionof the List interface is available in Appendix D. . . . . . . . . . . . . . . . 192Figure 6.15 Problems identified in (a) AuthzMap, (b) List, and (c) Search. Each rowin the grids indicates a known problem, and each column represents anevaluator (there are no overlapping problems between grids). Each cellshows a problem token, color coded by severity. The known problems aresorted from top to bottom based on the number of evaluators who foundthem. The evaluators are sorted from left to right based on the number ofproblems they found. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193Figure 6.16 Number of problems per heuristics. The horizontal axis shows lists ofITSM heuristics in graph (a) and Nielsen’s heuristics in graph (b). Thevertical axis shows the number of raw problems that were associated toeach heuristic by evaluators. . . . . . . . . . . . . . . . . . . . . . . . . . 195Figure 6.17 An overview of the comparative user-study protocol. Participants had tocomplete each step in the provided order, except the five study tasks thatwere randomized using a Latin square technique. . . . . . . . . . . . . . . 199Figure 6.18 Total time needed to complete the study for participants in each condition.The numbers shown on the box plots are the median TTCs. Notches on thebox plots indicate 95% confidence interval for the medians. . . . . . . . . . 211Figure 6.19 Number of attempts in completion of training test. . . . . . . . . . . . . . 211Figure 6.20 Time needed to complete the training task for participants in each condition. 213xixFigure 6.21 Time needed to complete the common review task for participants in eachcondition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214Figure 6.22 Time needed to complete the user comparison task for participants in eachcondition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215Figure 6.23 Time needed to complete the privilege accumulation task for participantsin each condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216Figure 6.24 Time needed to complete the SoD violation detection task for participantsin each condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217Figure 6.25 Time needed to complete the application review task for participants ineach condition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217Figure 6.26 Time needed to complete the comprehension task for participants in eachcondition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219Figure 6.27 Summary of participants responses to comprehension questions. Partici-pants were asked to rate the risk associated with each access on a five-pointlikert scale. The areas with a dashed border indicate correct responses. . . . 220Figure 6.28 An overview of post-evaluation questionnaire responses. Participants wereasked to rate their agreement with each statement on a five-point likert scale. 221Figure A.1 Usability Problem Specification Form (ITSM Condition) . . . . . . . . . . 270Figure A.2 Background Questionnaire . . . . . . . . . . . . . . . . . . . . . . . . . . 271Figure A.3 Post-Evaluation Questionnaire - Part I . . . . . . . . . . . . . . . . . . . . 272Figure A.4 Post-Evaluation Questionnaire - Part II . . . . . . . . . . . . . . . . . . . . 273Figure D.1 Level one of the AuthzMap interface. We used the notion of files in theuser study, but eventually columns in the grid indicate roles, permissions,files, or any other type of entitlements. . . . . . . . . . . . . . . . . . . . . 295Figure D.2 Level two of the AuthzMap interface. Reviewer can access this level byclicking on the magnifier icon in the level 1 of the interface. . . . . . . . . 296xxFigure D.3 Level one of the List interface. The original interface used the notion of“entitlements”, but we changed it to files for the purpose of the user study. . 296Figure D.4 Level two of the List interface. . . . . . . . . . . . . . . . . . . . . . . . . 297Figure D.5 Level one of the Search interface. The original interface used the notion of“Roles”, but we changed it to files for the purpose of the user study. . . . . 297Figure D.6 Level two of the Search interface. . . . . . . . . . . . . . . . . . . . . . . 298Figure E.1 Low fidelity prototype of AuthzMap (developed in MS Visio) showing theinitial state of the interface. . . . . . . . . . . . . . . . . . . . . . . . . . . 299Figure E.2 Low fidelity prototype of AuthzMap (developed in MS Visio) showing thedetail view. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300Figure E.3 Low fidelity prototype of AuthzMap (developed in MS Visio) showing theSoD violations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300Figure E.4 Medium fidelity prototype of AuthzMap (developed in Flash) showing thefish-eye view. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301Figure E.5 Medium fidelity prototype of AuthzMap (developed in Flash) showing thehistory view. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302xxiGlossaryACL Access Control ListAmazon MTurk Amazon Mechanical Turk, a crowd-sourcing platform that can be used forusability studies.ANOVA Analysis of VarianceBetween-subjects Study A study design in which each participant is only assigned to one ofthe conditions.Discount Usability Evaluation A usability evaluation method that offers a fast, cheap, andearly focus on usability. Examples would be heuristic evaluation, and cognitive walkthrough.Grounded Theory A data collection and analysis methodology that is used to develop abstracttheories from qualitative data.HCI Human Computer InteractionHOT Human, Organizational, and TechnologicalHR Human ResourcesHeuristic Evaluation A discount usability evaluation technique. Multiple evaluators inspectxxiithe usability of an interface using a set of usability heuristics, and identify usabilityproblems.IAM Identity and Access ManagementIdM Identity ManagementIT Information TechnologyITSM IT Security ManagementRBAC Role Based Access ControlSeparation of Duties A security control that require different people be responsible for dif-ferent parts of a task.SoD Separation of DutiesSOX Sarbanes-Oxley Act, A United States act that requires organizations to adopt certainsecurity controls.SP Security PractitionersWithin-subjects Study A study design in which all participants are exposed to all conditions.xxiiiAcknowledgmentsI would like to sincerely thank many people who have supported and helped me during my PhDstudies at the University of British Columbia. This dissertation would not have been possiblewithout them.My deepest gratitude is to my research supervisor, Professor Konstantin (Kosta) Beznosov,for his support during my study. He has contributed intellectually to almost every part of thisdissertation. He has been very supportive, and patient during my learning process as a PhDstudent, and cared about my growth both as a student and as a human being.I would like to acknowledge Professor Philippe Kruchten, Sathish Gopalakrishnan, and PurangAbolmaesumi, who served on my supervisory committee and have provided insightful feed-back and constructive criticism to improve this dissertation. I would be grateful to ProfessorSidney Fels and Jennifer Shapka for being my university examination committee, and Profes-sor Heather Richter Lipford, who served as the external examiner of my doctoral examination.They provided many helpful constructive suggestions to strengthen this dissertation.Thanks to all my co-authors, including Dr. Kirstie Hawkey, Dr. David Botta, Fahimeh Raja,Andreas Sotirakopoulos, and Hootan Rashtian, who helped me in different parts of this thesiswith their effort, feedback, and guidance.My thanks go to all my colleagues from the Laboratory for Education and Research in SecurexxivSystems Engineering (LERSSE). I am thankful for their valuable feedback and insightful dis-cussions on many parts of my research. It was an honour working beside such an intelligentand insightful people.Special thanks to my wife, Dr. Azadeh Goudarzi, for her unconditional love and supportthroughout my years as a PhD student. She accompanied me on this journey, sharing myhappiness and stress, and providing positive energy.Last, and not the least, I want to thank my parents (Yahya Jaferian and Soheyla Atighi) whooffered me love and support. I hope that this dissertation makes them proud. Thanks to mybrother, Koosha Jaferian, for his encouragement during these years. I am lucky to have him asmy brother. Thank you all.xxvThis dissertation is dedicated toMy parentsfor their endless encouragement and supportMy wifefor her unconditional lovexxviChapter 1IntroductionSecurity of Information Technology (IT) is an important aspect of any organization’s IT thathelps protecting organization’s information assets. The 2009 IT security market overview byForrester (Penn, 2009) shows a steady increase in IT security spending over the past years(e.g., 10% increase from 2008 to 2009). Data theft issues, compliance requirements, businesscontinuity planning, and a constantly moving security baseline drive the increasing need for ITsecurity. Prior studies show that IT Security Management (ITSM) is challenging (Werlingeret al., 2009a); that is, there are technological, organizational and human challenges in managingIT security.Computer security research has contributed to addressing technological challenges in ITSM byimproving existing security mechanisms, proposing new ones, and identifying and eliminatingsecurity vulnerabilities. However, according to Botta et al. (2007), there has been little atten-tion by the research community to the use of these technologies in a social context. Werlingeret al. (2009b) show that ITSM involves collaboration between different stakeholders, and theinteraction between stakeholders is particularly complex as they have different levels of se-curity knowledge and awareness, the information is spread across the organization, and the1stakeholders need to pay attention to the core business requirements and organizational con-straints such as tight schedules and budgets. Furthermore, Werlinger et al. (2009a) show that ITsecurity involves technological complexity, and Gagne´ et al. (2008) argue that such complexityis even more than regular IT.Designing usable interfaces for IT security tools is an important aspect in addressing the social,and technological challenges in ITSM. Chiasson et al. (2007) argue that the usable interfacesis a matter of security since humans are more likely to make mistakes in cases where theydeal with complexity, and when interfaces are too cumbersome, present too little or mislead-ing information, or overwhelm the administrator with too much information. However, fieldstudies of security practitioners by Botta et al. (2007) show that the existing ITSM tools do notaddress the needs of security practitioners, and there are numerous opportunities to improveITSM technologies.The overreaching problem that this research seeks to address is the gap between ITSM tech-nologies and the social and organizational context in which they are deployed. Ackerman(2000) defines such problems as a social-technical gap, a fundamental mismatch between whatis required socially and what we can do technically. More usable tools can narrow the gap, andbetter support the ITSM activities. While this problem is considered a grand challenge, in thisthesis we address three smaller problems:Problem 1: A lack of comprehensive guiding principles for designing usable ITSM tools:A sub-problem that prevents bridging the gap in ITSM domain is the difficulty in designand evaluation of ITSM tools. First, there is the absence of a comprehensive set ofusability guidelines for building ITSM tools. While there are guidelines in the literature,they target specific ITSM tools such as intrusion detection systems (Werlinger et al.,2008c), focus on specific aspect of ITSM activities such as collaboration (Werlingeret al., 2009b), or apply to a domain similar to, but different from ITSM such as ITmanagement (Haber and Bailey, 2007).2Besides lack of guidelines for design, the usability evaluation of ITSM tools is diffi-cult. Laboratory experiments may have little validity due to the complexity of real-worldsecurity problems, difficulties in recruiting security practitioners, and the spontaneous(e.g., security incident response), longitudinal (e.g., deploying an identity managementsystem), and collaborative (e.g., diagnostic work) nature of ITSM activities. Discount us-ability evaluation techniques (Nielsen, 1995) such as heuristic evaluation could be viablesubstitutes for lab studies. While generic usability heuristics such as Nielsen’s (Nielsenand Molich, 1990) could be useful for evaluation, prior research (Baker et al., 2002;Mankoff et al., 2003; Somervell and McCrickard, 2005) shows that domain-specificheuristics are more effective for evaluation of specific classes of systems than genericheuristics. However, we could not find a validated set of usability heuristics for ITSMdomain.To address this problem, our goal is to identify the relationship between existing designguidelines for ITSM tools, and develop a comprehensive framework of ITSM usabil-ity guidelines. Furthermore, we aim to develop and evaluate a set of domain-specificusability heuristics for evaluation of ITSM tools.Problem 2 A lack of understanding how organizations manage users’ identities and ac-cess: Bridging the social-technical gap in ITSM requires a fundamental understandingof the dynamics of a specific ITSM activity. Therefore, we narrow down our focus toone of ITSM activities, Identity and Access Management (IAM), due to the challeng-ing nature of the activity, and the wide reach of it across the organization. There hasbeen very few empirical studies that provide a real understanding of the IAM activitiesand challenges in actual organizations. Prior research touched upon certain key areas inIAM, such as challenges of managing file system access (Bauer et al., 2009), authoringaccess policies (Beckerle and Martucci, 2013; Brodie et al., 2005; Inglesant et al., 2008),understanding and resolving conflicts in file system access policies (Reeder et al., 2008),3managing access in document sharing systems (Smetters and Good, 2009), and casestudies of implementing access control in organizations (Kern et al., 2002; Schaad et al.,2001; Stevens and Wulf, 2009). Nonetheless, there is no integrated model that showshow various IAM related activities such as identity management, access management,and audit relate to each other, and why they are challenging. Such a model can helporganizations better understand the limitations of their existing approach to IAM, andidentify ways to improve their practice. Furthermore, it offers new ways of improvingtechnology to overcome IAM challenges.To address this problem, our goal is to study how different organizations manage users’identities and accesses, what approach do they use, and how the organizational contextimpacts IAM activities. Furthermore, we will identify challenges in conducting IAM,and propose an abstract model of the IAM and its challenges. Finally, our goal is tosuggest improvements to the technology or practice.Problem 3: A lack of usable interfaces for understanding and reviewing access policiesin organizations: After our initial analysis of interviews, we chose access review asone of the IAM related activities we found particularly challenging. We found that ex-isting access review systems only show the immediate access policy (i.e., users, accessprivileges, and user-to-access privilege assignments) and hide other relevant contextualinformation such as other users’ access, job information, the history of the policy, andpolicy violations. For example, during access review comparing one user’s access toother similar users would help detecting and removing unnecessary access, or seeingpolicy violations in the interface can help resolving those violations. Nevertheless, thesepieces information were not immediately available in the existing systems, and the userhad to go through multiple steps or use multiple information sources to find them. As aresult, we found that the process of access review is time-consuming, error-prone, anduncertain.4To address this problem, our goal is to obtain a realistic understanding of access reviewactivity, review existing technologies, and design and evaluate a new user interface thatcan support access review activity, and address the shortcomings of the existing systems.1.1 ContributionsThe main contribution of this research is addressing the three mentioned problems. In thefollowing, we give an overview of the three parts of our research and contributions of each:• Guidelines and heuristics for design and evaluation of ITSM tools: In Phase One,we collected different usability guidelines, good design practices, and useful features ofITSM tools from the literature on social aspects of ITSM. We categorized and combineddata from the literature into a set of 19 high level usability guidelines and identified therelationships between the guidelines and challenges in IT security management. We alsoillustrated the need for the guidelines, where possible, with quotes from five interviewswe performed with security practitioners.Then we analyzed the collected guidelines through a theoretical lens of activity theory,and generated an abstract, and generalizable set of principles. We refined these principlesinto a set of seven usability heuristics, and named the set ITSM heuristics. We thenconducted a between-subjects user study with 28 participants to compare the use of ITSMheuristics to Nielsen’s heuristics for usability evaluation of an IAM system. Our resultsshow those evaluators who used ITSM heuristics reported problems with higher severity,and fewer false positives compared to those who used Nielsen’s heuristics. We alsostudied different aspects of conducting heuristic evaluation using ITSM heuristics suchas the number of evaluators needed for evaluation, and their required background.The unique contributions of this phase to the overall goal of the thesis are three-fold:First, developing a set of usability guidelines for ITSM tools that can guide the design5of such tools, and eventually help in bridging the social-technical gap. Second, devel-oping a set of validated usability heuristics for identifying usability problems in ITSMtools. Third, a detailed, step by step process for the creation of domain-specific usabilityheuristics that can be used by other researchers to develop heuristics for their domain ofchoice. The overall outcome of this phase can be invoked as a vehicle to facilitate bridg-ing social-technical gap between IAM activities and technologies. We use the guidelinesand ITSM heuristics in the phase 3 of this thesis to design and evaluate our proposedsolutions.• Understanding the socio-technical context of IAM: In Phase Two, we narrow downour focus to one specific ITSM activity, identity and access management. To understandIAM and its challenges, we conducted 19 interviews with security practitioners engagedin IAM in large organizations. We followed a grounded theory approach to collect andanalyze qualitative interview data. We provided a thick description of how the IAMis performed in organizations, and what are the challenges faced by stakeholders. Wefurther identified the relationship between the challenges and activities, and proposed anabstract explanatory model of IAM. The unique contribution of this phase is a profoundunderstanding of the socio-technical context of IAM. Such an understanding is essentialbefore attempting to bridge the social-technical gap.• Develop and validate solutions for improving IAM technologies and practices: InPhase Three, we narrow down our focus again to one specific IAM activity, access re-view. We extended our understanding of access review activity by further analyzinginterview data and conducting a survey of 49 security practitioners. Furthermore, westudied the usability problems with existing technologies for access review using heuris-tic evaluation. Based on the interview, survey, and heuristic evaluation data, we identifieda set of design goals for access review interfaces. To show that our goals are feasible andeffective, we developed a novel access review tool named AuthzMap. We improved the6tool through multiple rounds of informal feedback and one round of heuristic evaluationby 12 usability experts. We then compared the use of AuthzMap to two of the existingaccess review systems. We measured the effectiveness of each system in a lab study with430 participants. The unique contribution of this phase is a profound understanding ofaccess review activity, and its challenges. We also highlighted the importance of contextin understanding and interpreting access policies. Furthermore, we proposed a novel userinterface for access review, and showed its effectiveness through a lab study.1.2 Organization of the ThesisThis thesis include three phases and six different studies. The relationship of these studies areshown in Figure 1.1. In Chapter 2, we provide a necessary background on the study topic,and review some of the related work. In Chapter 3, we present our method for creation ofguidelines for ITSM tools, and our guidelines framework. Chapter 4 includes a review ofprior heuristic creation literature, our method for heuristic creation, the set of ITSM heuristics,and the user study for validation of ITSM heuristics. Chapter 5 presents the methodology,and results of our interview study. In Chapter 6, we present various studies towards bridgingthe social technical gap for access review including: (1) interview and survey study of accessreview, (2) heuristic evaluation of an existing access review tool, (3) design of a novel userinterface, AuthzMap, for access review, (4) comparative heuristic evaluation of AuthzMap andtwo of the existing access review systems, (5) a comparative user study of AuthzMap and twoof the existing access review systems. In Chapter 7, we discuss how we addressed social-technical gap, present implications of our findings beyond IAM, and discuss the validity ofthe research. Chapter 8 summarizes the results of individual studies, discusses the relationshipbetween them, lists the contributions of the dissertation, and suggests areas for future work.7Phase II Phase IIIPhase IHeuristics for evaluation of ITSM toolsValidation of ITSM heuristicsField study of access and identity managementField study of access reviewHeuristic evaluation of an existing Access review toolDesigning AuthzMapIterative Improvement of AuthzMapEvaluation of AuthzMapA survey of access reviewComparative heuristic evaluation of AuthzMap and two existing systemsGuidelines for designing ITSM toolsChapter 3Chapter 4Chapter 4Chapter 5 Chapter 6Chapter 6Chapter 6Chapter 6Chapter 6Chapter 6Chapter 6Figure 1.1: Overview of thesis components and the relationship between them8Chapter 2Background and Related WorkIn this section, we provide a brief overview on the definition and scope of ITSM technologiesand Identity and Access Management (IAM) systems. Then, we review the prior research onsocio-technical aspects of ITSM and IAM. Finally, as we extensively use activity theory, andgrounded theory throughout this thesis, we provide background on these topics. We includethe chapter specific related work inside their the approprate chapters.2.1 BackgroundNational Institute of Standards and Technology (NIST) special publication 800-36 definesITSM products as “components in the design, development, and maintenance of the secureinformation technology infrastructure” (Grance et al., 2003). Forrester Research Inc. (Penn,2009) provides a taxonomy that classifies ITSM products into eight categories: (1) contentsecurity (e.g., email or web security), (2) endpoint security (e.g., personal firewalls or an-tiviruses), (3) identity and access management, (4) application security (e.g., code testing orweb application firewalls), (5) network security (e.g., firewalls), (6) data security (e.g., file en-cryption), (7) security operations (e.g., log management, forensics, or security configuration9management), and (8) risk and compliance management (e.g., security assessment, training).These technologies are used directly by multiple stakeholders. For example, the primary usersof content security, network security, data security, and security operations tools are securitypractitioners (SPs); application security tools are used by developers; risk and compliance man-agement tools are used by auditors; and end-point security and identity and access managementtools are used by variety of end-users in their day-to-day activities. A tool can affect stakehold-ers indirectly as well. For example, a firewall impacts how employees access network, but theydo not directly use the firewall.Identity and Access Management (IAM) comprises the processes and infrastructure for the cre-ation and maintenance of user’s digital identities and the designation of who has access to re-sources, who grants that access, and how accountability and compliance are maintained (Blum,2005). International Organization for Standardization (ISO) proposes a framework for identitymanagement (ISO, 2009a) and a framework for access management (ISO, 2009b). In the ISOframework, the access management framework includes identity management, which providesguarantee of the authenticity and legitimacy of the users. The access management frameworkalso has other components including privilege management; user authentication management;and control, traceability, monitoring, and review of user access.In the scope of this thesis, we refer to a system that provides both identity management and ac-cess management as an identity and access management system or IAM system. Other sourcesmight call such a system identity management system (IdM), access management system, oraccess and identity management system (AIM). Forester’s review of IAM vendors publishedin 4th quarter 2009 (Cser, 2009) shows that Oracle, CA, and IBM are the current leaders of themarket followed by Sun Microsystems, and Novell as strong performers. Courion, Hitachi IDsystem, SAP, and Microsoft are contenders. We show an overview on the market presence ofthe three market leaders in Table 2.1.10Table 2.1: Market presence of the leading IdM vendors (Cser, 2009)Vendor Production customers New customers IdM revenue Overall revenueOracle 2200-2600 400-600 $160-$180 million $22,430 millionCA 2000-2500 400-500 $180-$200 million $4,271 millionIBM 2700 200 $220-$240 million $103.6 billion2.2 Socio-technical Aspects of ITSMSocial issues in IT management and subsequently ITSM have been studied before. Accordingto Botta et al. (2007), ITSM is one aspect of IT management. Therefore, the result of the priorstudies on socio-technical aspects of IT is also applicable to ITSM. Researchers from IBM havedone extensive research on the nature of IT administration work and its challenges (Barrettet al., 2004) and identified recommendations for tool design (Barrett et al., 2005; Haber andBailey, 2007). As a part of their research, Kandogan and Haber (2005) focused on ITSMtools and practices. Their findings show that security practitioners work in a collaborativeenvironment, need to communicate with people with different backgrounds, work with largedata sets, and interact with complex systems. In a different study, Siegel et al. (2006) showsthe importance of organizational and human factors in ITSM. Kraemer and Carayon (2007)study the factors that result in human errors in IT security. Goodall et al. (2004) study intrusiondetection task in ITSM and show its complex and collaborative nature.2.2.1 The HOT Admin ProjectThe Human, Organization, and Technology Centred Improvement of IT Security Administra-tion (HOT Admin) research project aims to investigate human, organizational, and technolog-ical factors of IT security from the perspective of SPs. According to Hawkey et al. (2008a),the project goals are: (1) to devise a methodology for evaluating the usability of ITSM tools;and (2) to design effective technological solutions and guidelines to aid SPs. The HOT Ad-min researchers have performed a participatory observation in one academic workplace and11conducted interviews with 36 SPs. Data from the interviews and observation were analyzedaccording to several themes including: tools and practices (Botta et al., 2007), security man-agement model (Hawkey et al., 2008b), differences between ITSM and IT (Gagne´ et al., 2008),collaboration in ITSM (Werlinger et al., 2009b), challenges in ITSM (Werlinger et al., 2009a),and suboptimal situations in ITSM (Botta et al., 2011). The results of HOT admin project showthat IT security practitioners (SPs) work in a complex and fast paced environment (Botta et al.,2007). They also show that ITSM is challenging (Werlinger et al., 2009a) and it involves com-munication and collaboration between different stakeholders (Werlinger et al., 2009b). Wer-linger et al. (2009c) studied the complexity and collaboration aspects of ITSM in detail bynarrowing down their focus to diagnostic work in ITSM. Additionally, Gagne´ et al. (2008)identified differences between IT management and ITSM. Finally, Werlinger et al. (2008c) ob-served the process of deployment and on-going usage of an intrusion detection system (IDS)and identified the challenges in this process.2.3 Identity and Access ManagementIdentity and Access management in enterprise context has been studies by both academic andnon-academic entities.The socio-technical issues in IAM have been studied before by non-academic organizations.The Burton Group (2005)1 commissioned several studies about various aspects of IdM. TheBurton Group sources include their client organizations, non-client organizations who use iden-tity management, presentations by vendors of identity management products, and discussionwith consultants from other advisory organizations. Their findings can be classified into threemain categories: (1) Business drivers for IdM systems (2) Prerequisites for success (3) Re-quirements for IdM systems. Burton Group reports are targeted towards the business sector1The Burton Group is a firm that provides in-depth, IT research and advisory services to executives andtechnologists at Global 2000 organizations. The company has been recently aquired by Gartner Inc.12and there is no focus on solutions for improving technology. In addition, they are more fo-cused on the functionality of IdM systems rather than the usability.The academic studies of Identity and Access Management has been usually focused on tech-nical and theoretical aspects. Particularly, access control which is a subset of IAM has beena mainstream area in Computer Security research since nineties, but the research communityhas not paid enough attention to non-technological aspects of it. For example, a large scalesurvey by Fuchs et al. (2011) on the state of role-based access control research confirms thisfact. They show that 52% of the recent publications in the area of role-based access controlhad a theoretical focus. 41% of the publications focused on integrating RBAC into varioustechnologies (e.g., cloud, web, middleware). Only about 7% of the publications focused onpractical aspects of role-based access control in organizations. Our review of literature has ledto few studies in this area that we particularly review in Chapter 5.The Identity Project (Wright, 2007) is a study of IdM practices in UK higher education in-stitutions. The results of Identity Project are based on a broad survey (Brown and Smith,2007), which were validated and refined by 161 semi-structured interviews in the participat-ing institutions (McLeish, 2007). The Identity Project’s results are broad and significant, andare tuned to the improvement of business processes in the UK higher education sector. Thefindings of their research can be classified into challenges and best practices in IdM. Beingbroad, the Identity Project findings can be complemented by more detailed study of the IdMwith the focus on user centered design of the tools. Additionally, the findings of the IdentityProject are limited to academic institutions. As academic freedom has been found to lead tocertain challenges (Werlinger et al., 2009a), it remains to be seen whether their findings aregeneralizable.Most recently, Bauer et al. (2009) describe real life challenges in access control management asgleaned through interviews with policy professionals. The main focus of their study is on theongoing management of accesses to file systems and physical environments, and they did not13address the full process of IdM including management of identities, auditing, etc. In addition,they only focus on file system and physical security which involve simple policies.Although not explicitly about IdM, Heckle et al. (2008) discuss organizational challenges inimplementing a single sign-on system without previously assisting end-users to develop anaccurate mental model. Also, Post and Kagan (2007) identify security controls as an interferingfactor to end-users’ work and propose recommendations for alleviating this problem.2.4 Activity TheoryAccording to Halverson (2002), HCI theories can be useful for four different purposes: first,they have descriptive power and can help in making sense of and describing the world. Second,they have rhetorical power and can “help us talk about the world by naming important aspectsof the conceptual structure and how it maps to the real world.” Third, they have inferentialpower and they can help in making inferences about phenomena that have not yet sufficientlyunderstood. Fourth, they can be applied to inform system design.According to Rogers (2012), several theories have been developed or imported from other fieldssuch as cognitive psychology, sociology, and cultural studies to be used as a theory in HCI toserve any of the four purposes mentioned above. According to Halverson (2002), “theoriesare more like a pair of dark glasses. We put them on and the world is tinted. The changebrings some objects into sharper contrast, while others fade into obscurity.” Therefore, onecan choose the theory that best fits the problem at hand. In this thesis, we use Activity Theoryextensively for its descriptive power to develop ITSM heuristics (in Chapter 4), inferentialpower to interpret the results of the field study (in Chapter 5), and it pragmatic power to informsystem design (in Chapter 6). Therefore, here we give a brief summary of activity theory.Activity theory was developed by Leontiev (1974) as a general psychological theory, and waslater proposed as a potential framework for HCI research by Kuutti (1995). Activity theory14SubjectRules CommunityDivision of LabourObjectArtifactsOutcomeFigure 2.1: Components and internal relations of an activity systemmoves the unit of analysis beyond user actions, with “Human Activity” as the unit of analysis.Kaptelinin and Nardi (2006) suggested five basic principles of activity theory: (1) Conscious-ness and object-orientedness: all human activities are performed by a conscious actor towardsan object. (2) Hierarchical structure: activities have three levels: activity, actions, and oper-ations. (3) Activities involve internalization and externalization. (4) Mediation: activities areperformed by using and transforming artifacts. (5) Development: activities evolve and developover time. Engestro¨m (1999) proposed a formulation of activity theory to explicate the compo-nents and internal relations of an activity system (Figure 2.1). According to Engestro¨m (2001),the components of activity can be classified into subject, object, mediating artifacts, rules,community, and division of labour. He also suggested five activity theory principles includ-ing: (1) activity as a unit of analysis, (2) multi-voicedness, (3) historicity, (4) contradictions,and (5) transformation. The sets of principles suggested by Kaptelinin and Nardi (2006) andEngestro¨m (2001) are not mutually exclusive or contradictory; but they do provide differentperspectives on activity theory.152.5 Grounded TheoryGrounded theory is a methodology that can be used to develop theories and models from qual-itative data. It has been widely used in the HCI domain to facilitate insight into peoples values,understanding and experience with technology (Furniss et al., 2011).Grounded theory is considered a non-intrusive approach, and it usually starts with iterativecollection of qualitative data including interviews, videos, images, etc. Then the data is brokendown to pieces by performing coding, and then the codes will be related to each other to identifypatterns and themes in the data. Then a theory or model is built based on the themes and theirinterrelationship. Grounded theory was first proposed by Glaser and Strauss (1967), and laterdeveloped in two main directions: Glaser and Holton (2004) advocated the classic groundedtheory approach, and insisted that researcher should perform the data collection and analysiswithout any bias, preconceived theory, or use of literature. Furthermore, Glaser and Holton(2004) suggested a more open approach to coding data, and a positivist perspective. On theother hand, Corbin and Strauss (1990) argued for more structured approach to grounded theoryby providing a set of tools and methods that should be used during data analysis. Additionally,researcher can use literature, or prior theories to design the study, and guide the analysis of thedata. Furthermore, Corbin and Strauss (1990) highlighted the importance of paying attentionto the contextual factors that may be impacting the study phenomena, and offered a version ofgrounded theory that fits a constructivist paradigm. A survey by Matavire and Brown (2008) of126 grounded theory studies from Information Systems (IS) literature suggests that groundedtheory was mainly used in four different forms: (1) Glaser’s version of grounded theory, (2)Strauss’s version of grounded theory, (3) Mixed-methodology that combines grounded theorywith other approaches such as case studies, (4) application of grounded theory techniques,without adhering to strict grounded theory process suggested by either of Glaser or Strauss. Inthis dissertation, we use Strauss’s version of grounded theory as our approach for the followingreasons. First, our goal was to provide an understanding of IAM rather than just suggesting16a theory, and Strauss’s version, with its reliance on thick description was better suited for ourpurpose. Second, Strauss’s version acknowledges that researcher’s interpretation is a part of theresearch results. We find this perspective more realistic. Third, we found explicit guidelinesand methods suggested by Corbin and Strauss (1990) helpful rather than strict during dataanalysis.17Chapter 3Guidelines for Designing ITSM Tools1IT security is an important issue for organizations that want to protect their information assetsfrom threats inside or outside the organization. Previous research such as an interview studyof security practitioners by Botta et al. (2007) and observation study of three security prac-titioners by Rayford B. Vaughn Jr. and Fox (2001) show that, beside technological factors,human and organizational factors also impact IT security management. Security practitioners(SPs) face challenges (discussed in detail in Section 3.1.2) related to each of those factors ac-cording to Werlinger et al. (2008a). In order to improve the effectiveness of IT security in anorganization, these challenges must be addressed.One way to address the challenges in ITSM is to develop effective technological solutions andtools to aid IT practitioners in managing security. A key factor that impacts the effectivenessof ITSM tools is their usability according to Chiasson et al. (2007). In this chapter, we presenta set of guidelines for ITSM tools based on the available literature and results of the HOTAdmin project (see Chapter 2 for an overview of the project). Developing a set of guidelines1This chapter is based on the following publication:P. Jaferian, D. Botta, F. Raja, K. Hawkey, and K. Beznosov. 2008. Guidelines for designing IT security management tools. In Proceedingsof the 2nd ACM Symposium on Computer Human Interaction for Management of Information Technology (CHiMiT ’08). ACM, San Diego,CA, USA, Article 7 , 10 pages. (acceptance rate: 29%)18specific to such tools is necessary, due to the importance of IT security in organizations andthe evolving and competitive market of tools for managing it (Beal, 2005). For each guideline,we identify the challenges that it can alleviate. We support the need for each guideline throughliterature and illustrate it through quotes from five SPs interviewed in the HOT Admin project.In addition, we propose a framework for classification of the guidelines. This framework canbe used by tool developers to select appropriate guidelines when developing ITSM tools, aswell as by SPs and their managers for evaluating such tools.The rest of this chapter is organized as follows. Next section presents background and relatedwork. Section 3.2 describes the methodology we used to obtain and classify guidelines. Sec-tion 3.3 presents our framework of guidelines for ITSM tools, discussing each guideline inturn. Section 3.4 describes how to apply the guidelines and discusses limitations of our workand our plans for future research. Section 3.5 concludes.3.1 Background and Related Work3.1.1 User Interface GuidelinesAccording to Smith and Mosier (1986), user interface design entails a considerable investmentby various stakeholders and design guidelines can help the stakeholders in the design process.For example, from the guidelines, a system analyst can derive design requirements, a softwaredesigner can derive application-specific design rules, and a manager can make the interfacedesign process more efficient. According to Smith and Mosier (1986), there are challengesand considerations when guidelines are applied and used. First, it should be noted that not allguidelines are applicable to all tools. Therefore, developers should select a subset of guidelinesthat are applicable to the specific tool they are developing. Second, guidelines must be gener-ally worded so that they might apply to many tools. Therefore, specific design rules should bederived from more general guidelines.19Smith and Mosier (1986) suggest that every guideline development effort should begin and endby acknowledging the significant contributions of other people. Therefore, reviewing availableliterature on the subject under study is an essential part of developing guidelines. Also, guide-lines can be based on experience—either practical or derived through research. For example,a literature review has led to the development of guidelines for designing multi-media learn-ing tools by Grunwald and Corsbie-Massay (2006) or designing systems to support co-locatedcollaborative work on a tabletop display by Scott et al. (2003). In contrast, Theng et al. (1999)propose guidelines based on case studies of three digital libraries, while Baldonado et al. (2000)propose design guidelines for systems that use multiple views based on their experience. Onelarge set of guidelines is the Research-Based Web Design & Usability Guidelines by Koyaniet al. (2006). A survey of literature and other sources resulted in an initial set of guidelines.These were reviewed to eliminate duplicate and conflicting guidelines. Consequently, the rel-ative importance and strength of evidence for each guideline were determined by external re-viewers. Finally, they were grouped and organized through a card sorting exercise with a groupof web designers. Guidelines related to IT and ITSM tools have been discussed in the litera-ture. Haber and Bailey (2007) developed a set of design guidelines for IT administration tools.They compiled their list of guidelines based on field studies. While their extensive data collec-tion and rigorous analysis support the validity of their results, their guidelines do not addressspecifics of ITSM which might be different from IT as shown by Gagne´ et al. (2008). Chias-son et al. (2007) discuss the possibility of using four approaches (fostering an effective mentalmodel of the system, ecological interface design (see Vicente and Rasmussen, 1992), socialnavigation (see DiGioia and Dourish, 2005), and persuasive technology (see Fogg, 2002)) indesigning tools for security administrators. Based on these approaches, they compiled a list often guidelines for ITSM tools. While they note that their list is preliminary, the use of persua-sive technology might not be suitable for ITSM context. The goal of persuasive technology isto address the problem of security being the secondary task which is not the case in ITSM.The guidelines for ITSM tools that we present in this chapter were derived through a combina-20tion of our own research results from the HOT Admin project and a thorough survey of relatedwork.3.1.2 Challenges in IT Security ManagementIn Chapter 2, we summarized the results of the HOT-Admin project. The HOT-Admin findingsby Werlinger et al. (2008a) concerning challenges to ITSM are important to our developmentof guidelines, as our goal is to address the challenges by proposing improvements for securitytools.One set of challenges in IT security arises from fairly ubiquitous human and cultural traits thatbecome an issue, in particular when SPs need to interact with other stakeholders (Werlingeret al., 2008a). To begin with, a lack of security culture can challenge the modification of ex-isting practices (e.g., multiple employees using the same account to access a system). Lackof training makes implementation of security controls difficult, as people are not well edu-cated about best IT security practices. Further, communication of security issues can sufferfrom communication break-downs, usually because of different stakeholders having differentperceptions of risk.A second set of challenges in IT security are related to the characteristics of organizations (Wer-linger et al., 2008a). Besides people having different perceptions of risk, establishing the orga-nizational process of risk estimation is a challenge. The trade-off between security and businessprocesses often results in low priority of security, which, combined with the costliness of IT se-curity, leads to insufficient budgets. SPs are typically over-worked and tight schedules can leadto human errors or suboptimal security controls. Mergers, acquisitions of other organizations,and business partnerships all involve the challenge of interaction with other organizations thathave different IT security needs, cultures, and practices. Distribution of IT security across theorganization is commonplace; large organizations may have different IT departments, each of21which is responsible for its own security. Controlling access to data is challenging as sensitivedata is often distributed in organizations and accessed by many stakeholders. Finally, withan open academic environment, solutions need to allow for academic freedom in educationalorganizations.A third group of challenges is related to technological issues (Werlinger et al., 2008a). Thecomplex structure of computer networks (e.g., many nodes and users), and the need for dif-ferent solutions (e.g., firewall, intrusion detection system, anti-virus) for managing IT securitycreate a challenge of technological complexity. The frequent revelation of new vulnerabilitiesis a challenge, because SPs must deal with them or risk their security being compromised. Themobility of access to organizations’ IT poses yet another security challenge.3.1.3 Guidelines for IT Security ToolsNext, we briefly overview some of the main research efforts in developing design guidelinesfor ITSM tools. Sources in support of specific guidelines are given in Section 3.3.As discussed above, the results of the HOT Admin project (see Chapter 2) comprise one sourcefor guidelines. While some of the research themes offer guidelines (e.g., Botta et al. (2007);Gagne´ et al. (2008); Werlinger et al. (2008a,b)), the provision of guidelines is not the main goalof these papers and they do not provide an integration of all guidelines.Based on data collected from ethnographic field studies of system administrators, IBM re-searchers propose guidelines (Barrett et al., 2004, 2005; Haber and Bailey, 2007). As thereare many similarities between general IT practitioners and security practitioners (Gagne´ et al.,2008), guidelines for general IT tools are often applicable to ITSM tools as well. Kandoganand Haber (2005) also propose a small set of guidelines specifically for ITSM tools.Chiasson et al. (2007) combine results from usable security, ecological interface design, social22navigation, and persuasive technology to propose an initial set of design principles for securitymanagement systems. How their principles might address the breadth of challenges has yet tobe articulated.3.2 MethodologyOur main research questions in this work are: (1) What are the characteristics of a good ITSMtool? (2) How can these characteristics be implemented in ITSM tools? (3) Which challengesin IT security are addressed by these characteristics? (4) How can we express these character-istics in the form of guidelines and organize them in a way that can be useful for developers?To answer these questions, we collected data from two different sources: related literature andthe HOT Admin corpus of semi-structured interviews with SPs who use IT security tools.To develop the core guidelines, we first selected a set of primary publications to analyze. Thisset contained publications from the HOT Admin project (4 papers) and publications aboutITSM tools that we found important (14 papers, including those mentioned in Section 3.1.3).Using these sources, we started compiling the guidelines for ITSM tools. We included explicitguidelines as well as recommendations for improving security tools, good practices followedin a specific ITSM tool, and wish lists about tools. In this process we identified 164 guidelines.After identifying the guidelines, we categorized them using a Grounded Theory approach (Char-maz, 2006). First, we performed open coding using codes that emerged from the data itself.Then we employed axial coding to combine those open codes that are conceptually the same.As we had a large number of guidelines, we wanted to combine the guidelines in the waythat would be more useful for tool developers. We therefore performed a card sorting exerciseand grouped the guidelines according to the challenges that they address. This resulted in anearly version of the framework, similar to that shown in Figure 3.1. To validate and refine theguidelines, we both broadened our survey and analyzed additional interviews.23To broaden the survey, we performed a more comprehensive literature search. We reviewedthe papers published in top conferences related to the topic, performed keyword searches, andmined the references from our original set of 18 papers. The result of this search was a list of56 papers. We then reviewed the papers and found another 22 papers that could contribute toour guidelines.We analyzed five semi-structured interviews with SPs to find support for our guidelines andillustrative examples. The interviews are part of the HOT Admin corpus, but had not been ana-lyzed when the HOT Admin papers cited in our survey were written. In this chapter, we use P1to P5 for referring to participants. Participants included two security managers at a technologycompany (P1, P2), a security analyst at a telecommunications company (P3), a security con-sultant (P4), and a security analyst/manager at a second telecommunications company (P5).Each interview was 1-2 hours long, audio-recorded, and transcribed. In the interviews, SPswere asked about their tasks, their organizational model, the tools they used, and the ITSM-related challenges. It is worth mentioning that the interviews were not performed solely to gainknowledge about design guidelines for security tools; however, they did contain considerableinformation about ITSM tools. To analyze the interviews, we used the guidelines initially iden-tified as codes (i.e., pre-defined codes constructed from prior materials as suggested by Coffeyand Atkinson (1996)).3.3 ITSM Design GuidelinesWe have developed a framework (Figure 3.1) for classifying the design guidelines for ITSMtools. Its main purpose is to aid developers in selecting the guidelines. Each layer of theframework addresses a different set of challenges. The lower layers contain the guidelines thatare applicable to a larger set of tools, while the upper layers show guidelines that are morespecific to a certain set of tools. For example, the lowest layer in the framework comprisesgeneral usability guidelines for ITSM tools. These guidelines are applicable to all ITSM tools,24General Usability GuidelinesTechnological Complexity GuidelinesOrganizational Complexity GuidelinesTask Specific GuidelinesDiverse Stakeholders GuidelinesConfiguration and Deployment Guidelines Intensive Analysis GuidelinesMake tools combinable Beal (2005); Botta et al. (2007); Haber and Bailey (2007); Kandogan and Haber (2005)Use multiple levels of information abstraction Abdullah et al. (2005); Baldonado et al. (2000); Ball et al. (2004); Burns et al. (2003); Chiasson et al. (2007); Hornbaek and Frokjaer (2001); Thompson et al. (2007); Vicente and Rasmussen (1992); Werlinger et al. (2008b)Provide customizability Botta et al. (2007); McGann and Sicker (2005)Use different presentation / interaction methods Abdullah et al. (2005); Baldonado et al. (2000); Ball et al. (2004); Komlod et al. (2005); Thompson et al. (2007); Yurcik et al. (2003, 2007)Support knowledge sharing Botta et al. (2007); Chiasson et al. (2007); DiGioia and Dourish (2005); Kesh and Ratnasingam (2007); Lee and Copeland (2006); Rogers (1992); White and Lutters (2007)Help task prioritization Dijker (2006); Werlinger et al. (2008a)Provide flexible reporting Botta et al (2007); Garigue and Stefaniu (2003); McGann and Sicker (2005); Nohlberg and Backstrom (2007)Provide an appropriate UI for stakeholders Botta et al (2007); Nohlberg and Backstrom(2007)Make configuration manageable Andrew (2005); Haber and Bailey (2007)Support rehearsal and planning Andrew (2005); Barrett et al. (2004, 2005); Haber and Bailey (2007); Werlinger et al. (2008a)Make configuration easy to change Haber and Bailey (2007); Werlinger et al. (2008c)Provide meaningful errors Haber and Bailey(2007); Nielsen(1995); Werlinger et al. (2008c)Provide customizable alertingHaber and Bailey (2007)Provide automatic detection Kandogan and Haber (2005); Thompson et al. (2007)Provide data correlation and filtering Abdullah et al. (2005); Kandogan and Haber (2005)Distributed ITSM GuidelinesCommunication GuidelinesSupport collaboration  Barrett et al. (2004,2005); Haber and Bailey (2007)Work in a large workflow Beal (2005); Botta et al. (2007); Haber and Bailey (2007)Provide communication integrationBarrett et al. (2004,2005); Killcrece et al. (2003); Werlinger et al. (2008b)Facilitate archiving Gagne et al. (2008); Halverson (2004)SpecificityFigure 3.1: Framework of design guidelines for IT security management tools. The ref-erences listed under each guideline point to the supporting literature for it.25as well as other tools. The next two layers contain guidelines that are necessary due to the workenvironment of SPs, which is characterized by technological and organizational complexity. Asmost of the ITSM tools should work in complex technological environments, the guidelines inthe technological complexity layer are applicable to most ITSM tools (but not security toolsfor end-users). The guidelines in the next layer deal with the organizational complexity ofITSM. These are subdivided into three groups: guidelines to address general communicationchallenges, guidelines applicable to tools used in a process that involves other stakeholders,and guidelines applicable to tools used by distributed SPs. The upper layer of the frameworkcontains guidelines that are grouped based on task properties of the tool: guidelines for toolsthat require intensive configuration and deployment, and guidelines for tools used in a processthat requires intensive analysis.We next discuss the guidelines contained within each layer. For each guideline, we discuss theITSM challenges addressed and cite the related work that supports its inclusion in our frame-work. When possible, we provide illustrative examples from participants and give alternativesof the guideline.3.3.1 General Usability GuidelinesThe first layer includes general usability guidelines and recommendations that are applicableto tools for SPs. When performing the card sorting exercise, we realized that many of theguidelines for security tools were based on general usability principles such as those proposedby Nielsen (1995) and Smith and Mosier (1986). Because these guidelines were originallydeveloped for more general tools and interfaces and are available in many different sources, wedo not list all of them here. However, we give an example of a general usability guideline thatis particularly important for ITSM tools: providing help and documentation to users.In the literature, there are sets of guidelines about help and documentation features for of IT26and ITSM tools. For example, tool documentation should be available on the Internet andsearchable using search engines (Haber and Bailey, 2007). Several help features have beensuggested for security tools (Herzog and Shahmehri, 2007); although directed at tools for end-users, most of them are applicable to ITSM tools as well. These include providing contextsensitive help, online help, wizards, light-weight help features, and social navigation. Onetechnique, safe staging, may not be as useful if the tool will only be used by expert users.3.3.2 Technological Complexity GuidelinesAccording to Werlinger et al. (2008a), there are multiple challenges related to technical com-plexity, including mobile access and vulnerabilities. We next present guidelines that can ad-dress these challenges.Make Tools CombinableBotta et al. (2007) discovered SPs must often use multiple tools to perform a single task, butthe process of combining tools to perform a task is not well supported by available tools (Beal,2005; Kandogan and Haber, 2005). As one of our participants illustrated: “So the vendorsthemselves are looking at things in isolation instead of looking at it as a whole thing thatneeds to be addressed” (P4). Another described challenges when using multiple tools: “Weare really, really having a problem at correlating output from all these tools. At the beginningthey were using three or four, it was easy to manually correlate, but when they started hittingsix, seven, eight, plus, it was very difficult to correlate because the outputs are all different”(P5). This participant also mentioned that development of a console to configure and execute17 vulnerability analysis tools resulted a significant decrease in the time needed to perform ananalysis task (from 10-15 days, to two days).Combining tools in an ad hoc fashion is a kind of bricolage; it is recommended that tools27should survive in an arena of bricolage (Botta et al., 2007). Vendors should standardize eventformats to permit integration of tools (Kandogan and Haber, 2005). Standardized configura-tion and logging formats will allow files from different tools to be searched and correlatedtogether (Haber and Bailey, 2007). Another option is to provide APIs/plug-ins to facilitate in-tegration of tools into system-wide monitoring or management meta-tools (Haber and Bailey,2007).Support knowledge sharingAs SPs perform their tasks within complex technological environments, a great deal of knowl-edge is created. This knowledge is either kept in the mind of the security practitioner, orwritten in notes or documents, or kept in the form of executable scripts (Botta et al., 2007).This knowledge is a valuable asset and can be used in the future by the same or other SPs; ittherefore should be kept and managed (Kesh and Ratnasingam, 2007). This knowledge canbe managed at two levels: among SPs in the organization or among all the users. There-fore, security tools should facilitate knowledge management at different levels. To supportknowledge sharing, SPs can use databases (Lee and Copeland, 2006), Microsoft SharePointsites, document management systems, or Wikis (White and Lutters, 2007), or synchronouscommunication channels (Rogers, 1992). This practice is illustrated by one participant whomentioned: “We have an IT manual which is kept up to date electronically and hard copy. Wehave our SharePoint site where they can go and everything is at their fingertips. It links themto every single place they need to know how to go to” (P4). Another form of sharing is socialnavigation, which is suggested as a model for usable security by DiGioia and Dourish (2005),and shown to be important in the context of ITSM by Chiasson et al. (2007). Although aris-ing from technological complexity, the need to support knowledge sharing is closely related toorganizational complexity as described in Section different presentation/interaction methodsAccording to Baldonado et al. (2000), presenting information in multiple views or presentationformats can facilitate investigation of a single conceptual entity. Textual and speech data issequentially processed through auditory cognitive functions, while graphical data has the ad-vantage of using parallel visual and spatial functions (Ball et al., 2004). Therefore, graphicaldata results in faster situational awareness and effective identification of patterns and vulnera-bilities in network. Furthermore, using different presentations of the same data can help situ-ation awareness through a reduction in the high cognitive load characteristic of ITSM (Yurciket al., 2003). In the related literature, this guideline often accompanies a proposal for differentvisualization methods for a large set of data. For example, one proposed visualization methodfor intrusion detection system (IDS) alarms is to show alarms in a two-dimensional space (y-axis: IP address, x-axis: time) (Abdullah et al., 2005). Three-dimensional space has also beenproposed for IDS data (color, opacity, shape) (Komlod et al., 2005). Different levels of detailcan allow SPs to zoom in for more details (Komlod et al., 2005). Visualization of the net-work data can reduce the time and training required for network traffic analysis (Ball et al.,2004). The combination of concurrent textual and visual interfaces has been advocated forsecurity tools by Yurcik et al. (2007) and for IDSs by Thompson et al. (2006) as each interfaceis shown by Thompson et al. (2007) to has its own strengths and weaknesses. This guidelinemay be particularly important for those tools that involve intensive analysis, as described insection 3.3.4.Use multiple levels of information abstractionVicente and Rasmussen (1992) suggest using an abstraction hierarchy in order to support op-erators of complex systems during unanticipated events. As SPs need to deal with vulnerabili-ties and unanticipated scenarios in complex network environments, tools can follow Vicente’smethod of showing the system at different levels of abstractions, with the Ecological Interface29Design (EID) framework. Other researchers suggest using EID in the design of ITSM toolsin general (Chiasson et al., 2007) and for network monitoring tools in particular (Burns et al.,2003). According to Werlinger et al. (2008b), presenting information at different levels of ab-straction to different stakeholders can help prevent disclosure of confidential information bypresenting it to each stakeholder at an appropriate level. It can also prevent miscommunicationby providing information appropriate to the stakeholder’s level of security knowledge.In many tools, presenting information at different levels of abstraction is realized by providingan interface with two views: overview and details. A study about reading electrical documentsby Hornbaek and Frokjaer (2001) suggests that presenting information at different levels ofdetail can reduce user errors. Similarly, Ball et al. (2004) show that using multiple levels ofabstraction, as well as visualization, reduce the time and training required for network trafficanalysis.According to Baldonado et al. (2000), when there is diversity in levels of abstraction, differentpresentation formats (Section 3.3.2) can be used. For example, both textual and graphicalinterfaces for IDSs have been used by Thompson et al. (2006) to present information at differentlevels of abstraction. Similarly, a visualization technique for IDS by Abdullah et al. (2005)provides an overview of its alarms with details on demand.Provide CustomizabilityAs SPs frequently deal with unpredictable situations (e.g., new vulnerabilities), an essentialfeature for ITSM tools is to be customizable (Botta et al., 2007). This need is illustrated by oneof our participants (P1): “For the reasons why I have built some stuff from hand, we’ll say thatno, they don’t do everything that I need them to do. So sometimes I do need to custom craftsomething or I need to automate something. Or I need to do something maybe that the tooldoesn’t do.”30In a comparison of security analysis tools for SIP-based VoIP systems by McGann and Sicker(2005), one important criterion was the ability to define new and customized test-cases.Help Task PrioritizationSPs frequently must deal with many competing priorities. A survey by Dijker (2006) found thatone of the main factors that frustrates SPs is wasting time and that they need better planningand organization. Therefore, it is important for tools to facilitate the process of planning andprioritization. Planning facilities can be implemented in different ways. At the most basic, atool could afford note-keeping functionality so that SPs can write down their priorities withregards to the tool. With some intelligence, a tool could help to prioritize vulnerabilities basedon their criticality (Werlinger et al., 2008a).3.3.3 Organizational Complexity GuidelinesSeveral aspects of organizational complexity must be addressed by the guidelines. SPs needto communicate with many stakeholders, including both other SPs and diverse stakeholderswithin the organization (Werlinger et al., 2008b). We first present guidelines that address gen-eral communication challenges and then present the guidelines that specifically address thechallenge of dealing with diverse stakeholders. Finally, we present guidelines that address thechallenges arising from the distribution of security tasks across multiple SPs.Communication GuidelinesKraemer and Carayon (2007) show that ineffective communication is a contributing factor tohuman errors. According to Werlinger et al. (2008b), SPs need to communicate with otherstakeholders during many activities, and the current tools do not provide sufficient communi-cation support for SPs. Tools that facilitate communication can address challenges of commu-31nication of security issues, distribution of IT management, interaction with other organizations,and different perceptions of risk.Provide Communication Integration One way for tools to facilitate communication is to al-low for integration with communication media. Tools should have communication facilitiesto allow collaboration between different users (Barrett et al., 2004, 2005). Tools can reducecommunication overhead between different stakeholders by showing relevant security config-uration information to different stakeholders (Werlinger et al., 2008b). One important feature,whether communication is between a tool and a user or between users, is to support a securemethod of communication (Killcrece et al., 2003).The need to support communication between tools and users is illustrated by one of our par-ticipants when discussing a network monitoring tool (P4): “You hate to find out you have aproblem when you are actually working; rather get paged at night and be able to fix it beforethey show up.” Tools should be integrated with different channels (e.g., email, text messages,web site) (Werlinger et al., 2008b). Mobile communication modalities (e.g., pagers, Blackberryemail) should be integrated into the solutions (Barrett et al., 2005). Furthermore, tools shouldbe configurable as to the destination and stakeholder to which these messages (e.g., alarms,logs) should be sent (Werlinger et al., 2008b).Facilitate Archiving Tools should facilitate keeping track of communication and informationrelated to tools. Practicing this guideline has two benefits. First, keeping a record of communi-cation between different stakeholders is already practiced by SPs; this may be due to the needfor SPs to adhere to legislation (Gagne´ et al., 2008). It is also illustrated by one participant(P4): “So we have archives with backup tapes—we have the Cadillac of backup tapes for ourkind of organization because we have a thing you can walk into practically—so we keep every-thing.” If tools provide support for this need, they can help remove the burden of archiving and32managing communication. A second benefit of archiving is to keep the information and knowl-edge that is generated during one project or incident (Halverson, 2004). This information canbe used in future incidents to analyze the trends in network, or it can be used as a knowledgebase (previously discussed in Section 3.3.2).Diverse Stakeholders GuidelinesAccording to Werlinger et al. (2008a), one important organizational challenge of IT security isthe involvement of various stakeholders within the organization, and some aspects of securitymanagement may involve non-experts (Botta et al., 2007). Furthermore, effective communi-cation of security issues to different stakeholders is an important factor to be considered. Ac-cording to von Solms and von Solms (2004), “Not realizing the core importance of informationsecurity awareness amongst users” is considered one of the “deadly sins” of ITSM. We nextpresent guidelines that are mainly aimed at addressing communication challenges (communi-cation of security issues, different perceptions of risk) that result from diverse stakeholders.Provide an Appropriate UI for Stakeholders According to Botta et al. (2007), ITSM toolsoften have many types of end-users including experts such as SPs and less technical adminis-trators and managers. Each category of users may have its own preferences and needs in termsof the user interfaces. As illustrated by one of our participants (P5): “We actually use the com-mand line interface route and we try to keep it as simple as possible because we were puttinganother layer on top of it we couldn’t go into the graphical one. But sometimes clients wantgraphical stuff. Especially if they are not 100% techie, it’s easier.” Therefore tools shouldprovide appropriate user interfaces based on the user’s expertise and needs. One suggestionby Nohlberg and Backstrom (2007), when developing a UI for ITSM tools that will be usedby managers, is to provide an overview early with as little information as possible and pro-vide further details on demand. This guideline relates to Sections 3.3.2 and 3.3.2, as different33stakeholders require different presentations of data or different levels of detail in their userinterface.Provide Flexible Reporting One aspect of flexible reporting is generating reports that are cus-tomized to contain information for a specific stakeholder. For example, reports that are aimedat managers should be concise and mainly focus on business objectives and the effectivenessof the organization in reaching them (Garigue and Stefaniu, 2003; Nohlberg and Backstrom,2007). This is illustrated by one of our participants (P5): “And the CEO is comfortable talkingto me because I am talking his language. I am talking your return on investment, . . . and thoseare the terms that I use quite often. I do it deliberately. It’s a technique that I’ve learned andI’ve used shamelessly.” Also, the report may be packaged differently depending on the type ofstakeholder. For example, one participant talked about packaging a report for managers (P4):“It’s got to have color and it’s got to be flashy so that they’ll pay attention. They don’t want toread a ten page document on anything. They want a quick learn.” From a different angle, twoof our participants (P5, P1) discussed that reports should contain constructive recommenda-tions that are simply represented (e.g., in a table). According to Garigue and Stefaniu (2003),reports can be packaged based on predefined templates or standard frameworks like SarbanesOxley bill or IS 17799. Furthermore, they classify reports into four categories (governance andpolicies, application and systems development and deployment, active security posture, infosecoperational services), and provide examples of the important reports that can be generated ineach category.Another aspect of the flexible reporting is making reports accessible by different stakeholders.To realize this, reports should be easily distributable and accessible across the organization.According to Botta et al. (2007), reports can be generated in standard formats like HTML,PDF, and spreadsheets; generating reports in the web format is considered an important featurefor security analysis tools (McGann and Sicker, 2005).34The flexible reporting guideline relates to Section 3.3.2 because a well sorted report can helpprioritization. Also the guideline relates to Section 3.3.2, as providing different presentations ofdata will make reports more understandable. Finally, flexible reporting relates to Section 3.3.2,as the reports generated by a tool should provide an appropriate level of detail based on theintended audience.Distributed ITSM GuidelinesThe guidelines presented next address the challenge of distribution of IT management. In manyorganizations ITSM is distributed across multiple SPs (Botta et al., 2007), either informally orthrough an official distributed organizational model for ITSM (Hawkey et al., 2008b). In theseorganizations, SPs need to collaborate with each other, as well as other stakeholders, to performtasks (Werlinger et al., 2008b). Tools should function as part of a larger workflow and providesupport for collaborating and sharing.Work in a Large Workflow One of the important needs of SPs while working under distributedITSM is to be able to automatically distribute tasks. According to Beal (2005), security toolsshould follow the way corporate networks have evolved and become integrated together. To al-low collaboration among stakeholders, Botta et al. (2007) point to a need for workflow supportfor the varying roles of different individuals. One of our participants (P3) desired an accesscontrol platform that supported the workflow of granting access to a user: from the end userrequest, to the person in charge of authorization, to the administrator making changes to thesecurity controls. According to Haber and Bailey (2007), shifts in responsibilities could beencoded in scripts with a new sysadmin automatically notified when it is their turn and thepertinent interface displayed.35Support Collaboration One important feature of tools that can help collaboration is to providea shared view of the system state (Barrett et al., 2004, 2005; Haber and Bailey, 2007). Accord-ing to Haber and Bailey (2007), tools should formally support sharing by showing which usersare currently working with system and what they are doing. In addition, Barrett et al. (2004,2005) suggest sharing can be supported with proper approval and authentication. Another im-portant aspect of collaboration is to provide support for grounding new participants as quicklyas possible when they join the activity (Haber and Bailey, 2007). This guideline is an extensionof the knowledge sharing guideline (Section 3.3.2).3.3.4 Task Specific GuidelinesThis layer contains guidelines that may or may not be applicable, depending on the natureof the application. The first set of guidelines is specific to applications that require intensiveconfiguration, particularly during deployment. The second set is specific to applications thatrequire SPs to perform intensive analysis.Configuration and Deployment GuidelinesSPs must often perform complex configuration of tools, particularly during deployment. Dueto the technological complexity of ITSM and its tools, the task of configuration can require agreat deal of effort (Werlinger et al., 2008a). A second challenge that impacts configurationand deployment is vulnerabilities (Werlinger et al., 2008a). To deal with frequent vulnerabil-ities, SPs need to patch systems often; however, patching a network of thousands of nodes istedious work that can be very costly. For example, manually deploying a patch on a 1000-nodenetwork can cost as much as $1M (Andrew, 2005). Because security has a low priority withinmany organizations (Werlinger et al., 2008a), SPs may be urged to complete configuration anddeployment as quickly as possible, and without compromising availability and performance.We next present several guidelines aimed at dealing with these challenges.36Make Configuration Manageable As described above, SPs frequently need to apply config-uration changes to hundreds of nodes in a network or deploy nodes at a similar scale. Thiscomplex process should be done very quickly and accurately. Therefore, tools should en-able SPs to automate and manage this process, as well as control its details. To realize this,Haber and Bailey (2007) suggest that tools should provide progress indicators, forecast the de-ployment process, perform operations in an asynchronous non-blocking manner, and providehistory and detailed steps of the executed operation. According to Andrew (2005), tools shouldalso support change roll-back.Support Rehearsal and Planning As SPs work with complex and critical systems, changes inconfiguration may have unanticipated outcomes that cannot be tolerated by other stakeholders.For example, security patches are not usually tested for all environments (Andrew, 2005). Also,Werlinger et al. (2008a) illustrate how a security patch that decreased the performance of anapplication triggered conflict between SPs and internal users. Therefore, SPs should be verycareful in deploying new solutions, changing configuration, or applying patches. Accordingto Haber and Bailey (2007), system administrators practice rehearsal and planning to avoidunanticipated events on production systems. They first rehearse the operation on a test systemand then apply it to the production system. Security tools that require extensive configurationshould support rehearsal and planning practices. According to Barrett et al. (2005), it shouldbe easy to build a test system with various degrees of fidelity to the production system, and itshould be easy to validate the results of the test system.Also, tools should support migration of scripts/operations from test to production environ-ments (Haber and Bailey, 2007). Logging each step of the procedure and providing facilitiesto compare the outputs from test and production environments would facilitate rehearsal andplanning (Barrett et al., 2004). According to Andrew (2005), virtual environments (e.g., usingtools like VMware) can assist with testing.37The rehearsal and planning guideline is related to making the configuration process manageable(Section 3.3.4). The rehearsal process can be completed more easily and with less overhead iftools support features like undo. Furthermore, providing forecasting of the deployment can bea useful indicator for comparing the rehearsal with actual execution.Make Configuration Easy to Change One of the tasks of SPs is to change the configuration ofthe system. Configuration frequently requires dealing with many parameters, some of whichare unknown to the security practitioner. Therefore, tools should provide facilities that helpSPs change configuration of the system easily. To realize this, Haber and Bailey (2007) sug-gest that tools should provide commented configuration files and/or group related parameterstogether in high-level profiles, so that a change in the profile would change all related parame-ters automatically. Also, Werlinger et al. (2008c) recommend that tools should provide a quicktuning option that allows batch configuration of parameters.Provide Meaningful Errors Although providing meaningful errors is a standard usabilitypractice (Nielsen, 1995), we re-iterate the guideline here as configuration and deployment ofcomplex systems is particularly error prone. To ease the process, particular care should betaken for any error messages generated by tools during configuration and deployment so thatit is presented to the user in a meaningful way. For example, (Werlinger et al., 2008c) showthat insufficiently meaningful error messages caused delays during installation of an IDS inone academic organization. Haber and Bailey (2007) suggest that tools should provide help incase of errors or alerts, instead of presenting cryptic messages.Intensive Analysis GuidelinesAccording to Werlinger et al. (2008a), investigation of attacks and vulnerabilities is one of themost important and challenging tasks for SPs and requires periods of intensive analysis. To deal38with the challenge of vulnerabilities, tools should support SPs in the investigation tasks. AsSPs must conduct analysis within the constraints of tight schedules (Werlinger et al., 2008a),tools should provide mechanisms to reduce the number of false positives (FPs) because FPshave to be investigated. The next three guidelines are applicable to tools which require SPs toperform intensive analysis.Provide Customizable Alerting Many security tools that monitor systems, generate alarmscommunicated to SPs. Haber and Bailey (2007) recommend that tools should provide cus-tomizable thresholds for generating alarms and selectable destinations for sending alarms (e.g.pager, email, console). Furthermore, SPs should be able to suppress alarms with lower priority.As mentioned by one of our participants (P3): “Given that I had knowledge of the perimetersecurity systems that were protecting these systems internally in the organization, I modifiedthat critical level [of the tool].”Provide Automatic Detection According to Kandogan and Haber (2005), SPs need to findattacks or unusual behavior patterns in large amount of logs and data that are linked together.To help SPs perform their tasks more effectively, Kandogan and Haber (2005) recommendproviding automation in detecting problems. Application of data mining and other analyticmethods in activity classification, analysis, and noise reduction can help; automatic detectioncould be implemented as software agents or bots that handle obvious cases and notify SPs aboutcritical ones (Kandogan and Haber, 2005). Thompson et al. (2006) suggest using intelligentpattern recognition techniques to find salient patterns.Provide Data Correlation and Filtering According to Kandogan and Haber (2005), duringanalysis, SPs frequently need to collect data from several sources and then correlate it. Forexample, Abdullah et al. (2005) show that correlating alarms of different IDSs can reduce39the number of false alarms. As discussed by one participant (P3): “These tools generategeneral or global reports based on what they are analyzing, right? And, I took those reports,with other tools I complemented like NMAP to do the same analysis, and I was checking andcorroborating that they effectively correspond.” Security tools can improve the process of datacorrelation by providing required filtering to reduce the large quantity of data, providing outputin standard formats that can be shared between different tools, providing facilities to deal withthe problem of out-of-sync clocks in correlating time-stamped data from different sources, andproviding facilities to automate the process of data correlation (Kandogan and Haber, 2005).3.4 Discussion3.4.1 Applying the GuidelinesThese guidelines can be used for multiple purposes. They can be used by developers as theycompile requirements for applications with SPs as end-users. Consideration of the guidelinesmay also be helpful when developing rich use case scenarios to ensure that the scenarios ad-dress the technological and organizational challenges of the intended operational environment.The guidelines could also be used by SPs and managers as they evaluate tools in the context ofthe challenges inherent within their organization. In this case, the guidelines should be treatedas high-level criteria. If a criterion is not met in some way, then the application may have roomfor improvement.Whether designing an application, deciding which one to acquire, or evaluating an application,the sets of guidelines that are relevant to the application and to the situation should be consid-ered. The guidelines are grouped by whether they are (1) generally applicable, (2) relevant totechnological or organizational characteristics of the operational environment, or (3) relevantto task-specific challenges. General usability guidelines apply to all ITSM-relevant applica-tions. We argue that the guidelines to address technological complexity are also applicable to40all ITSM-relevant applications. Not only is the technological environment of ITSM changingwith the advent of new technologies, but the rate of change in ITSM is faster than in generalIT (Gagne´ et al., 2008). That is, technological complexity is characteristic of ITSM. As dis-cussed next, the applicability of the remaining guidelines will depend on the organizationalenvironment and the specifics of the tasks the application will perform. Not all the guidelineswill be applicable in all situations.The organizational environment in which the application is deployed may be such that theguidelines to help organizational complexity are relevant. An organization may be large withmany cooperating SPs, or small with an IT department consisting of a “one-man shop.” De-pending on the intended use of the tool in that environment, guidelines to help multiple stake-holders, or guidelines to help distributed ITSM, or both may apply. A developer of a toolthat will be used in a “one-man shop” may not need to consider guidelines that address orga-nizational complexity, particularly those guidelines that address distributed ITSM. However,a tool—such as an IDS—installed in a large organization would likely require ongoing dis-tributed cooperation between SPs (Werlinger et al., 2008c) and therefore benefit from the guide-lines on distributed ITSM. In contrast, if an access control system were to be implemented inthe same large organization, it would require a great deal of initial cooperative consultation toestablish job roles and their corresponding privileges. For such an application, the organiza-tional complexity would likely focus more on multiple stakeholders than on distributed ITSM,so it would benefit from guidelines to help multiple stakeholders.Task specific challenges may be addressed by guidelines to help intensive analysis or guidelinesto help configuration and deployment. For example, an IDS is typically difficult to configureand also requires intensive ongoing analysis (Werlinger et al., 2008c), while a network scan-ning application (e.g., Nessus) requires intensive analysis, but typically does not require muchconfiguration. The network scanning application would therefore only need the guidelines tohelp configuration and deployment while the IDS could benefit from consideration of all the41guidelines in this layer.3.5 ConclusionIn this section, we provided the result of our survey on design guidelines for ITSM tools. Theprimary sources for the guidelines are recommendations about ITSM tools available in theliterature; we have augmented these from our own experiences interviewing SPs in the HOTAdmin project. We have gathered the different recommendations and combined them into aframework of high-level design guidelines for ITSM tools. The guidelines in our frameworkare high level and blur the boundaries between usability, organizational usability (Elliott andKling, 1997), and utility. This framework can be used by tool developers, as well as by SPs andmanagers evaluating security tools. To justify the guidelines, we provided empirical evidenceof their need. In addition, we identified relationships between the guidelines and known chal-lenges in ITSM. These relationships can help users of the framework determine the importanceof each guideline for their tools.42Chapter 4Heuristics for Usability Evaluation ofITSM Tools1Information technology security management (ITSM) tools serve several purposes includingprotection of network and data, detection of threats and vulnerabilities, and management ofusers and their accesses (Beal, 2005). Previous research (see Chapter 2) has highlighted theimportance of collaboration and information sharing support between various stakeholders inIT security tools. Werlinger et al. (2009b) identifies nine security activities that require collab-orative interactions and developed a model of the complexity of their interactions. This com-plexity arises from organizational attributes (e.g., distribution of IT management); the need forsecurity practitioners (SPs) to interact with multiple stakeholders; and their need to engage inmultiple security related activities. Each of these activities may require different explicit ortacit knowledge and kinds of information to be conveyed.1This chapter is based on the following publications:P. Jaferian, K. Hawkey, A. Sotirakopoulos, M. Velez-Rojas, and K. Beznosov. Heuristics for evaluating IT security management tools. Human-Computer Interaction, Vol. 29, Iss. 4, 2014. (impact factor: 3.039)P. Jaferian, K. Hawkey, A. Sotirakopoulos, M. Velez-Rojas, and K. Beznosov. 2011. Heuristics for evaluating IT security managementtools. In Proceedings of the Seventh Symposium on Usable Privacy and Security (SOUPS ’11). Pittsburgh, PA, USA, Article 7 , 20 pages.(acceptance rate: 33%, best paper award)P. Jaferian, K. Hawkey, A. Sotirakopoulos, and K. Beznosov. 2011. Heuristics for evaluating IT security management tools. In CHI’11Extended Abstracts on Human Factors in Computing Systems (CHI EA ’11). ACM, Vancouver, BC, Canada, 1633-1638. (acceptance rate:45%)43According to Chiasson et al. (2007), usability is an important quality for ITSM tools, butevaluating the usability of specific ITSM tools is challenging. Laboratory experiments mayhave little validity due to the complexity of real-world security problems and the need to situatea specific tool within a larger context (Neale et al., 2004). However, it is difficult to recruit SPsfor simple interviews, let alone field observations (Botta et al., 2007; Kotulic and Clark, 2004).Direct observation of tool use can be time consuming as much security work is spontaneous(e.g., security incident response), or occurs over many months (e.g., deploying an identitymanagement system). As ITSM tool use is intrinsically cooperative, its study inherits thedifficulties of studying cooperation listed by Neale et al. (2004). Therefore, heuristic evaluationof ITSM tools could be a viable and low cost component of tool usability evaluation.The existing sets of heuristics did not capture many of the challenges specific to the ITSMdomain. As we report in this chapter, we needed to explore how domain specific heuristicshave been created in the past, develop a methodological approach for creating them, and applythe method to the creation and evaluation of a new set of heuristics for usability evaluation ofITSM tools. Our results suggest that using a combination of a bottom-up approach (by derivingguidelines from literature and interview data), and a top-down approach (by abstracting theguidelines using activity theory (see Kaptelinin and Nardi, 2006)) can lead to a set of heuristicsthat can find problems in IT security tools. In this chapter, after presenting the set of heuristicswe created, we report on the empirical evaluation of our heuristics in which we comparedtheir usage to Nielsen’s (Nielsen and Molich, 1990). We conducted a between-subjects studywith 28 participants and examined different aspects of evaluation when deploying the two setsof heuristics. Our results suggest that the number of major problems that are found usingthe ITSM heuristics is higher than the number of problems that are found using Nielsen’s.Furthermore, while Nielsen showed that about five evaluators are able to find about two thirdsof the problems, in our evaluation of the IdM system, five evaluators only found about half ofthe problems found by 14 evaluators; we observed few overlaps between problems identifiedby individual participants using either Nielsen’s or the ITSM heuristics. Based on the result of44evaluation and participants’ feedback, we discuss how ITSM and Nielsen’s heuristics can beemployed for usability evaluation of ITSM tools.4.1 Background and Related WorkHeuristic evaluation (HE) is a usability evaluation method based on a set of usability principlescalled heuristics. According to Nielsen (2005a), they are called “heuristics” because they aremore in the nature of rules of thumb than specific usability guidelines. Usability heuristicsare more general than guidelines, and require more expertise of the evaluator. An evaluatorinspects a user interface and identifies usability problems and their severity based on heuristicsand his judgment of the interface. According to a survey by Vredenburg et al. (2002), heuris-tic evaluation is the most popular informal usability evaluation technique among user-centereddesign (UCD) practitioners. Furthermore, Jeffries et al. (1991) show that heuristic evaluationcan lead to finding more serious usability problems compared to usability testing, guidelinesand cognitive walkthrough (see (Polson et al., 1992) for details of cognitive walkthrough).Nielsen’s theoretically grounded and extensively tested heuristics (Nielsen and Molich, 1990)are the most widely accepted heuristics. They were developed based on existing HCI guide-lines, are consistent with Norman’s theory of action (Norman and Draper, 1986), and focus onthe dialogue between a single user and the physical world. Nielsen’s heuristics have been mod-ified or extended to create domain specific heuristics (e.g., ambient displays (Mankoff et al.,2003), virtual reality (Sutcliffe and Gault, 2004), medical devices (Zhang et al., 2003), intel-ligent tutoring systems (Muller and McClard, 1995), and intrusion detection systems (Zhouet al., 2004)). Furthermore, heuristics have been created without using Nielsen’s heuristics(e.g. computer games (Pinelle et al., 2008), groupware (Greenberg et al., 2000), shared visualwork surfaces for distance-separated groups (Baker et al., 2002), Large Screen Information Ex-hibits (LSIE) (Somervell, 2004), Ubiquitous systems (Scholtz and Consolvo, 2004)). We onlyhave found one instance of applying HE on an ITSM tool. Zhou et al. (2004) developed six45heuristics based on Nielsen’s heuristics for intrusion detection systems. The authors describedthat they developed the heuristics based on surveys and interviews, but they did not provideany detail of their methodology. Four of their six heuristics are identical to Nielsen’s heuristicsand the rest are extensions to Nielsen’s.4.2 Proposed ITSM HeuristicsIn this section, we describe how usability heuristics can be created, classify prior heuristiccreation literature according to its methodology, and discuss the benefits and drawback of eachapproach. Then we describe the method of heuristic creation we employed, followed by thelist of proposed heuristics.4.2.1 Methods for Creating Usability HeuristicsAccording to Shneiderman and Plaisant (2010, chap. 2), guidance to designers can emerge inthree forms: “(1) high level theories and models, (2) middle-level principles, and (3) specificand practical guidelines”. Te’eni et al. (2007, chap. 8) suggest that principles represent thetheory with an eye to what should be practiced, and the guidelines take the principles onestep further toward their application. Nielsen (2005a) defines heuristics as “general rules thatseem to describe common properties of usable interfaces.” Considering these definitions, weclassify heuristics as middle-level principles. In addition, Nielsen (1994) suggests that a listof heuristics should be short (about seven to ten) and easy to teach. Heuristics should also beopen to interpretation, so that multiple evaluators can use them to find diverse problems.Two approaches can be used to develop domain specific principles: (1) Bottom-up: qualitativedata is collected and analyzed to understand the characteristics of the domain, and principlesare created using real-world data. (2) Top-down: high-level expert knowledge, and/or theory isused to derive specific recommendations for the target domain. Table 4.1 provides comparison46of the major literature on heuristic creation.In a bottom-up approach, two types of qualitative data were used in the literature to synthesizeheuristics. First, researchers studied positive and negative aspects of specific systems in the tar-get domain. For example, Somervell (2004) employed claims analysis2 or Pinelle et al. (2008)used content of product reviews for analyzing positives and negatives aspects of systems. Sec-ond, guidelines from prior literature were used to synthesize heuristics (Nielsen, 1994). Theadvantage of the bottom-up approach is that the heuristics are grounded in real-world data, andreflect real problems with the tools in the target domain. The disadvantage is that the producedheuristics are limited by the scope and richness of the qualitative data and the interpretation ofthat data by the researchers.In a top-down approach, expert knowledge is used to derive heuristics from high-level theoriesor principles. Heuristics can be derived from a substantive theory,3 a formal theory,4, or exist-ing heuristics. For example, Baker et al. (2002) used the mechanics of collaboration framework(a substantive groupware theory), or Scholtz and Consolvo (2004) used the general HCI lit-erature (formal theories) to derive heuristics. Also expert knowledge can be used to modifyNielsen’s heuristics for the target domain (Mankoff et al., 2003). The top-down approach re-lies on expert knowledge to modify a theory or an existing heuristics set, and customize itfor usability evaluation of the domain specific systems. Therefore, the process of heuristicderivation is not systematic, and is prone to researcher bias.To address the above limitations, a more rigorous process can be used by combining bothbottom-up and top-down heuristic creation. The process can be started bottom-up from em-pirical data by using a qualitative data analysis method such as Grounded Theory (Glaser andStrauss, 1967). This process will result in design guidance grounded in empirical data. Then a2See (Carroll and Rosson, 1992) for details of the claims analysis method3“a theoretical interpretation or explanation of a delimited problem in a particular area.” (Charmaz, 2006, pg.189)4“a theoretical rendering of a generic issue or process that cuts across several substantive areas of study.” (Char-maz, 2006, pg. 187)47top-down approach can be used to justify, support, and combine the identified design guidanceinto heuristics. We advocate that the complementary top-down approach be rooted in a theoryrather than expert knowledge, to leverage a more formal and less ad-hoc process. The use oftheory reduces researcher bias in interpreting qualitative data, and abstracting and refining thefindings into heuristics. In addition, a link between theory and heuristics will provide insightinto the theory behind the heuristics, and help in communicating them to others. In the lit-erature, Somervell (2004) adopted this approach by combining both bottom-up and top-downapproaches systematically. We used a similar approach to create ITSM heuristics that suits bestto the ITSM context. Unlike large screen information exhibits systems (the domain of interestin (Somervell, 2004)), which are limited in number, there are a vast number of ITSM tools.Because analyzing those tools was not a viable approach, we used literature and interviewsas a data source, and grounded theory as the analysis method. For our top-down approach,we used formal theory as there was no substantive theory for ITSM. We further discuss ourcreation process in the next section.4.2.2 Our Methodology for Creating ITSM HeuristicsWe used a combination of top-down and bottom-up approaches to develop ITSM heuristics.These approaches consisted creating guidelines from the literature5, and interpreting and ex-plaining guidelines using the theoretical lens of activity theory.Guideline creation: We started with a bottom-up approach by understanding the characteristicsof IT security management (ITSM) tools that help SPs perform activities more effectively andefficiently. We collected data from two sources: related work and interviews performed inthe HOT-Admin project (see Section 2.2.1). We first analyzed a set of primary publicationsthat included HOT-Admin publications (4 papers) and other publications about ITSM tools5We elaborated the guideline creation in Chapter 3. In this chapter, we dedicate one paragraph to reiterate andsummarize the guideline creation process, as it is an important part of the entire heuristic creation methodology.48Table 4.1: Comparison of the major heuristic creation literature. The “T”, “B”, and “?”indicate top-down, bottom-up, and unknown method of heuristic creation.Author DomainCreationmethodCreation method detailsNielsen (1994)General BT First, a bottom-up approach was used to gather101 usability guidelines and principles from differ-ent sources. Then a top-down approach was usedby performing factor analysis and expert review tocombine and narrow the guidelines to heuristics.Mankoff et al.(2003)AmbientdisplaysT The heuristics were developed by changingNielsen’s heuristics based on prior experience ofauthors and by getting feedback from experts indesigning ambient displays.Baker et al.(2002)SharedvisualworkspacesT Mechanics of collaboration theoretical frameworkwas used to derive heuristics. The exact process ofderivation was not described.Greenberg et al.(2000)Groupware T Locales framework concepts were re-cast asheuristics.Pinelle et al.(2008)ComputergamesB 108 game reviews from were ana-lyzed using qualitative analysis techniques, prob-lems were extracted from reviews, categorized,and then heuristics were derived from the cate-gories.Scholtz and Con-solvo (2004)Ubiquitouscomput-ingT Expert knowledge and general HCI literature wereused to derive a framework for evaluation of ubiq-uitous systems.Somervell (2004)LargeScreenInfor-mationExhibits(LSIE)TB A bottom-up approach was used by performingclaims analysis Carroll and Rosson (1992) to an-alyze design decisions of five major LSIE systems.A top-down approach was used to combine similarclaims and derive high-level heuristics. Each claimwas classified according to its impact on three crit-ical parameters for design of notification systems.In addition, scenario-based design categories Car-roll and Rosson (1992) were used to further classi-fying the claims. The heuristics were created usingthe categories.49(14 papers). We identified 164 explicit guidelines for building ITSM tools, recommendationsfor improvement, design decisions in a particular tool that have positive impact on usability,and pros and cons of various tools. We categorized these using Grounded Theory. First, weperformed open coding using codes that emerged from the data, followed by axial codingto combine conceptually similar open codes. Meanwhile, following the theoretical samplingtechnique, we broadened our sources of data by reviewing the papers published in well-knownconferences related to the topic, performing keyword searches, and mining the references fromour original set of 18 papers. Our goal in this stage was to saturate the identified themes in thefirst round of analysis, refine the identified guidelines and find a better relationship betweenthem. The result of this search was a list of 56 papers. We then reviewed the papers and foundanother 22 papers that could contribute to our guidelines. We also analyzed five semi-structuredinterviews with SPs to find support for our guidelines and illustrative examples. This processresulted in 19 guidelines for ITSM tools (see Chapter 3 for details of the guidelines).!Finding guidelines instances (164 Guidelines)Open!coding!Axial coding Card sorting22!papers! 56!papers!5 interviews 7!heuristics!1- Results of the HOT-Admin project 2- Other papers on human and social aspects of ITSM published in major conferences Interpretation and abstraction based on activity theory 18 papers 19 guidelines Figure 4.1: Overview of the process of developing ITSM heuristicsThe identified guidelines were based on collected data and were specific and limited to thedata we analyzed. Therefore, we used a top-down approach to look at the guidelines througha theoretical lens provided by a theory that can describe the characteristics of ITSM domain.Using theory leveraged our interpretation of data and added another level of validation to ourfindings.Choosing a theory: To choose a theory that can be applied to the analyzed data, we searched50for specific IT security management theories, but to our knowledge, no such theory existed.We then sought general HCI theories that have been applied to contexts with technological,social, and organizational complexity. Information processing psychology, which has been ex-tensively used as the dominant theoretical foundations for HCI according to Kaptelinin et al.(2003), was the first candidate; but it was rejected quickly as it doesn’t take into account thecontext in which users’ actions are situated. Consequently, we reviewed three widely usedpost-cognitivist theories: Activity Theory (Kaptelinin and Nardi, 2006), Distributed Cognition(DCog) (Hollan et al., 2000), and Phenomenology (Dourish, 2001). According to Kaptelininet al. (2003), all of these theories can be used as foundations for understanding the use oftechnology in a social and organizational context. As we describe next, we found the ac-tivity theory perspective to be the best fit for the ITSM domain. From the phenomenologyperspective, each instance of human activity is unique and different from other instances. Phe-nomenology argues against abstracting human activities and finding commonalities betweenvarious instances. Such a perspective is advantageous in describing a specific human activity,but according to Nardi (1995), it has limited ability in “higher-order scientific tasks where someabstraction is necessary” (e.g., developing abstract heuristics). From the DCog perspective, asocial system can be modeled as a network of people, and artifacts, all of which are capableof cognition and transformation of information. Two main assumptions of DCog are the sym-metry of human and tools, and smooth functioning of the system. While such viewpoints canbe advantageous in contexts where smooth functioning and limited creativity is expected (e.g.,the call center in an organization), it is limited in the ITSM domain, which involves unknownsituations, breakdowns, creative use of artifacts, judgment and reflection, contradictory goals,and learning. Activity theory principles fit well with ITSM characteristics. For example, prin-ciples such as contradictions can describe breakdowns, mediation can describe creative use ofartifacts, and internalization and externalization can describe judgment and reflection. Addi-tionally, the prior use of activity theory by Zager (2002) for modeling certain aspects of ITsecurity shows the fit between the theory and the domain.51Applying the theory: To use activity theory to abstract and combine the guidelines, we analyzedthe guidelines using the theoretical lens provided by activity theory. We used ten activitytheory principles from two well-known sources by Engestro¨m (2001), and Kaptelinin and Nardi(2006) (see Section 2.4 for the list of principles), and cross-tabulated them with the guidelinesin a matrix. The matrix allowed us to summarize how theory explains each guideline. We thenchose one of the principles as the main explaining principle and the rest as supporting principlesbefore combining guidelines explained by the same main principle. This led to 13 guidelinescombined under six categories. The remaining guidelines could not be classified under a singlecategory as the guidelines had different components explained by different main principles.These guidelines were broken down and classified under four of the previous categories anda new category. We then tried to convert each category into a heuristic. When categories arecrafted as heuristics, they should be concise, easy to understand, and open to interpretation.We used an iterative approach of multiple piloting sessions and getting feedback from peers.We illustrate an example of our heuristic synthesis process in Figure 4.2. In this example, wegenerated the guideline “Make tools combinable” (Fig. 4.2b) using six sources (Fig. 4.2a), and“make tools customizable” using four sources. Then activity theory could explain “Make toolscombinable” by the creation (externalization) of mediating artifacts (mediation) to addressunexpected conditions (contradictions), and “make tools customizable” by the customization(externalization) of the mediating artifacts (mediation) as users’ knowledge or activity evolvesover time (development) and the tool is no longer best suited to the activity (contradictions).We then chose “mediation” as the main principle. We then combined these two guidelines into“flexible mediation” (Fig. 4.2d). We later reworded the heuristic to “flexible representation ofinformation” based on the feedback from our pilot testing participants (Fig. 4.2e).52“These tools should be customizable, for example, enabling the construction of different types of reports depending on the recipient.” (Botta et al., 2007)“The framework allows for scripting of new interoperability and vulnerability detection test cases. The included tests are specifically developed from common known implementation errors that cause problems.”(McGann and Sicker, 2005)“The above discussion shows that no matter how advanced a security tool is, diagnostic work in the context of ITSM still requires customization of the tool to the specific reality of a given organization.”(Werlinger et al., 2009c)“Many study participants reported the use of custom tools to monitor system state, configure components, and perform many other maintenance tasks, suggesting the need for scriptability.”(Velasquez and Weisband, 2008)“In summary, the various security tools are not well integrated. When one tool wouldproduce a certain piece of information, we observed admins using manual tools to derive other information.”(Kandogan and Haber, 2005)“[Access control tools] should automatically update compound policies […] Integrating these different systems so that the implemented access-control policy is automatically kept up to date has many potential benefits. Several groups we interviewed used systems that had this functionality.”(Bauer et al., 2009)“Provide APIs/plug-ins so the tool can be integrated into system-wide monitoring and management meta-tools (e.g. dashboards).”(Haber and Bailey, 2007)“Many organizations implement security products in a piecemeal approach, i.e. with little technical and operational integration between deployed IT security products such as anti-virus software, firewalls and access management. The gaps pose a security risk and are an impediment to streamlined business activity.”(Beal, 2005)“Depending on the diagnostic work performed, our practitioners used scripts either as standalone tools or in combination with other IT tools via bricolage, i.e., the re-use of existing tools in new and unanticipated ways.”(Werlinger et al., 2009c)“Our participants had limited toolkits of tools that they knew well. The handful of trusted tools were, together, versatile. That is, they could be used together in creative ways to accomplish different tasks in various scenarios.”(Botta et al., 2007)MediationExternalization ContradictionsMediationExternalization ContradictionsDevelopmentFlexible MediationFlexible Representation of InformationPilot TestingMake tools combinableMake tools customizableabd ecFigure 4.2: An example of heuristic synthesis process: we used a bottom-up approachby analyzing literature on ITSM tools (a) to create guidelines (b), and a top-downapproach (c) using activity theory to extract preliminary heuristics (d) which laterwere reworded to final heuristics (e).534.2.3 Proposed ITSM HeuristicsIn this section, we present seven heuristics for the usability evaluation of ITSM tools. Weprovide the title and the description of each heuristic and then empirical support for it from theliterature. To illustrate the importance of the heuristic with real-world examples, we includeinterview snippets from seven interviews with SPs conducted as a part of our ongoing researchprojects (participants are identified by codes from SP1 to SP7). We then provide theoreticalsupport for the heuristics.Heuristic #1 - Visibility of activity status: “Provide users with awareness of the status ofthe activity distributed over time and space. The status may include the other users involvedin the activity, their actions, and distribution of work between them; rules that govern theactivity; tools, information, and materials used in the activity; and progress toward the activityobjective. Provide communication channels for transferring the status of the activity. Whileproviding awareness is crucial, limit the awareness to only what the user needs to know tocomplete his actions.”Discussion: In ITSM, the actions that form an activity are distributed across time and space.These actions are performed in an organizational context with certain norms and rules. Plansare created and modified by different stakeholders, and roles are assigned dynamically to ad-dress unknown conditions. Prior ITSM research shows the importance of providing awarenessof organizational constraints (Zager, 2002), communication channels (Werlinger et al., 2009b),methods for sending cues to different stakeholders to inform them about when and how toact (Botta et al., 2011), awareness of what other stakeholders perform in the system, sharing thesystem state between different SPs and grounding new participants in ITSM activities (Haberand Bailey, 2007). To illustrate this, SP2 said his team receives daily reports on employees’malicious actions and described the importance of awareness in preventing insiders’ maliciousbehavior: “We can lock down - we use policies and things like this to keep people from doing54what they shouldn’t be doing. We lock down firewalls so that they cannot do what they shouldn’tbe doing. Because we are running reports and you know who is doing stuff.”Looking at the problem through the lens of activity theory, tools can provide awareness aboutthe components of activity including artifacts, community, and rules. Carroll et al. (2003)identify three types of awareness: (1) social awareness, the understanding of current socialcontext in an activity (e.g., rules, artifacts); (2) action awareness, the understanding of actionsof collaborators on shared resources; and (3) activity awareness, the understanding of howshared plans are created and modified, how things are evaluated, and how roles are assigned.As ITSM tools deal with sensitive information, they should follow the recommendation byErickson and Kellogg (2000) and provide visibility in the form of social translucence ratherthan social transparency.This heuristic is different from Nielsen’s “Visibility of System Status”, which focuses on im-mediate status of the system to help users select appropriate actions and evaluate the outcomeof their actions. The ITSM heuristic includes aspects of the system status that might not beavailable locally and immediately.Heuristic #2 - History of actions and changes on artifacts: “Allow capturing of the historyof actions and changes on tools or other artifacts such as policies, logs, and communicationsbetween users. Provide a means for searching and analyzing historical information.”Discussion: Accountability and reflecting on work are important aspects of ITSM (Gagne´et al., 2008; Velasquez and Durcikova, 2008). As ITSM involves creative work to addressunknown conditions, providing usage histories supports creativity, learning, and quality im-provement (Shneiderman, 2000). Audits, which aid in reflecting on work, are mandated inIT security as a part of regulatory legislations such as the Sarbanes-Oxley Act (see Sarbanes,2002). Prior ITSM research by Gagne´ et al. (2008) shows SPs archive logs and keep a historyof communications for audit and accountability. To illustrate, SP7 described that health care55organizations allow physicians to openly access patient data but they archive and audit everyaccess attempt: “I let you do it, but I audit a crap out of the system. So if somebody complainsor someone reports that I saw somebody accessed something and I don’t think it is appropriatethen you’ve got a really robust audit records.” Histories can also be used to understand otherstakeholders’ actions. For example, Bauer et al. (2009) found that sometimes access controlpolicies are changed by multiple SPs; keeping track of changes will help other SPs maintaina working knowledge of the implemented policy. Finally, Velasquez and Durcikova (2008)show that historical information can be used for trend analysis, learning about the network,and evaluating the outcome of actions that span time and space.From the theoretical perspective, artifacts in an activity carry a history with them. Awarenessof this history influences the way those artifacts are used. Hollan et al. (2000) studied expertsworking in complex environments and found that usage histories are incorporated in cogni-tively important processes. Historical information could be in the form of the usage historiesof the user himself or of other users of the system. According to Shneiderman (2000), usagehistories can be employed to reflect on work, and to get feedback from peers.Heuristic #3 - Flexible representation of information: “Allow changing the representationof information to suit the target audience and their current task. Support flexible reports.Allow tools to change the representation of their input/output for flexible combination withother tools.”Discussion: Botta et al. (2007) show that SPs often use inferential analysis and pattern recog-nition to develop policies, audit security, or troubleshoot security incidents. For example, theyneed to look for certain patterns in network traffic to detect an anomaly; or they need to analyzeusers’ access to different resources in order to build an effective set of role-based access control(RBAC) roles. To perform these activities, SPs often use their tools in creative ways that werenot anticipated by tool developers; or alternatively, they combine their tools. For example,SP2 described their reason for building custom tools: “Sometimes, I do need to custom craft56something or I need to automate something. Or I need to do something maybe that the tooldoesn’t do.” Botta et al. (2007) identify SP’s practice of bricolage (i.e., combining differenttools in new ways) to address complex problems and argue that ITSM tools should survive inthe arena of bricolage. Haber and Bailey (2007) and Beal (2005) also highlight the need forbetter integration between ITSM tools. SP3 described the problem with correlating data from17 vulnerability analysis tools: “We are really, really having a problem at correlating outputfrom all these tools. At the beginning they were using three or four, it was easy to manuallycorrelate, but when they started hitting six, seven, eight, plus, it was very difficult to correlatebecause the outputs are all different”. Therefore, they wrote a homegrown solution to convertand import all the data into a database, and cross-reference the findings of different tools.Tools should also be flexible in representing information. This allows users to use a represen-tation that best suites the task, and the background and expertise of the user. SP3 describedwhy they preferred vulnerability analysis tools with command line interface when they builtthe homegrown solution: “We actually use the command line interface route and we try to keepit as simple as possible because we were putting another layer on top of it we couldn’t go intothe graphical one.” SP3 also clarified why some users prefer GUI based tools: “... sometimesclients want graphical stuff. Especially if they are not 100% techie, it’s easier.” In prior ITSMresearch, the need for flexible interaction methods (e.g., Command Line Interface and Graphi-cal User Interface) (Botta et al., 2007; Thompson et al., 2007), flexible reporting (Botta et al.,2007; Velasquez and Weisband, 2008; Werlinger et al., 2009b), visualization techniques (Dour-ish and Redmiles, 2002), and multiple views (Haber and Bailey, 2007) are highlighted.From the activity theory perspective, ITSM tools are mediating artifacts. Their mediation rolecan be between users (e.g., wiki, or other communication channels), between users and othertools (e.g., visualization of network traffic), or between two other tools (e.g., a script) (Bottaet al., 2011; Maglio et al., 2003). Therefore, a tool should be able to process an input from auser or another tool, and to provide an output that is understandable to the user or the tool that57receives the output. This concept was further explained by Norman (1991). He described thatartifacts have two types of representation: the internal representation that is not accessible bythe outside world, and the surface representation that is their interface to the world. Provid-ing a flexible surface representation is particularly important in ITSM as Botta et al. (2011)show that tools are creatively combined by SPs and their output is used by users with differ-ent knowledge and background. Previous activity theory research by Rabardel and Bourmaud(2003) also argued in favor of highly customizable and open tools, when users combine andadapt different tools to build instruments for unexpected and unknown conditions.Heuristic #4 - Rules and constraints: “Promote rules and constraints on ITSM activities,but provide freedom to choose different paths that respect the constraints. Constraints can beenforced in multiple layers. For example, a tool could constrain the possible actions based onthe task, the chosen strategy for performing the task (e.g., the order of performing actions), thesocial and organizational structure (e.g., number of stakeholders involved in the task, policies,standards), and the competency of the user.”Discussion: ITSM tools are used in organizational context with rules, norms, and constraints.Violating these constraints will result in sub-optimal situations; therefore, tools can help en-force such constraints. Botta et al. (2011) show that enforcing norms by ITSM tools in the formof procedures for notification and support for particular templates and standards can preventcommunication and collaboration breakdowns. Werlinger et al. (2009a) argue that ITSM toolscan promote security culture in organizations and address the lack of training by enforcingpolicies. SP4 further clarified the importance of leveraging policies using tools: “So really IDS[Intrusion Detection System] should really be something that supports you in your ability toleverage policy so really security I think is 90% policy and the rest of it is tools. [...] You’vegot to have the policy structure behind you and the tools to find out if your policies are beingrespected.”From the activity theory perspective, there are rules and norms that govern every activity.58According to Kaptelinin and Nardi (2006), promoting rules and norms by tools can lead toawareness and internalization of those norms by stakeholders. Vicente (2000) points out theimportance of enforcing rules and constraints by tools, while allowing users to flexibly explorethe possible action space. This helps users be aware of constraints, and gives them flexibil-ity to adapt to unexpected situations. Vicente argues that constraints can be expressed at fivedifferent levels: work domain, control tasks, strategies, social-organizational, and worker com-petencies. Rules in ITSM can include security and privacy policies or standards, organizationalconstraints, and organizational culture.Heuristic #5 - Planning and dividing work between users: “Facilitate dividing work be-tween the users involved in an activity. For routine and pre-determined tasks, allow incor-poration of a workflow. For unknown conditions, allow generation of new work plans andincorporation of new users.”Discussion: Botta et al. (2007) show that the ITSM context requires quick responses to un-known conditions by stakeholders, who work with tight schedules in which ITSM has a lowpriority. Therefore, Werlinger et al. (2009a) show that planning and dividing work betweenstakeholders is important. SPs often need to coordinate activities with multiple stakeholdersinvolving other SPs, IT admins, managers, end-users, and external stakeholders. For example,to address a security incident, SPs often need to collect data from end-users; analyze the inci-dent; coordinate and collaborate with IT specialists, who own the impacted sub-systems (e.g.,database admins); communicate with managers to warn them about the risks associated withthe incident and possible disruptions in service; and even collaborate with external SPs to solvethe problem. In all of these cases, proper planning tools should be available to quickly involvestakeholders and divide work between them. SP1 described the importance of dividing workon the efficiency of the security group: “First and foremost, explicit definition of what you doand do not do. Everyone is capable of doing more than they can do on a day-to-day basis.However, if you haven’t established clear lines of demarcation from what is your responsibility59and what is not, particularly security people have the need to save the world and so they tendto do everything and therefore they burn out.”Activity theory points to the division of labour as an important aspect of activity. Furthermore,division of labour should take into account constraints at the social organizational level, as wellas possible methods for generating plans and collaborating considering those constraints.Heuristic #6 - Capturing, sharing, and discovery of knowledge: “Allow users to captureand store their knowledge. This could be explicit by means of generating documents, web-pages, scripts, and notes, or implicit by providing access to a history of their previous ac-tions. Tools should then facilitate sharing such knowledge with other users. Furthermore,tools should facilitate discovery of the required knowledge source. The knowledge source canbe an artifact (e.g., document, web-page, script) or a person who possesses the knowledge.Provide means of communicating with the person who possesses the knowledge.”Discussion: According to Botta et al. (2011), SPs rely heavily on knowledge to perform theirtasks. For example, to implement security access controls, a SP needs to know about the ac-tivities that a stakeholder performs, and the resources required for performing those activities.When asked how one can know the people that should be contacted in order to grant accessto any of the 600 available applications in the organization, SP6 responded: “So our accessprocedures state that every application that has any level of criticality is supposed to have apublished knowledge-base document in our service desk. [It] defines what the application is,who owns it, who is the technical owner.” Therefore, SPs need to discover and use the knowl-edge of other stakeholders whether they are inside or outside of the organization. Prior researchshows the importance of managing knowledge (Botta et al., 2011; Kesh and Ratnasingam,2007) and suggests policy specification as a method to transfer knowledge (Werlinger et al.,2009a). Rogers (1992) shows the importance of transmitting knowledge at the “window of op-portunity” during troubleshooting in a network environment that involves multiple stakeholdersand describes it as a challenging task.60From the theoretical perspective, the relationship between different actors in the activity ismediated by artifacts. According to Engestro¨m (1999), in order to transfer knowledge, usersshould be able to externalize their knowledge as artifacts. Facilities for identification and ac-cess to the required knowledge must then be provided. If externalization of knowledge is notfeasible, a method for finding and starting collaboration with the person who possesses theknowledge should be provided. In this case, the communication channel is considered themediating artifact.Heuristic #7 - Verification of knowledge: “For critical ITSM activities, tools should help SPsvalidate their knowledge about the actions required for performing the activity. Allow users toperform actions on a test environment and validate the results of these actions before applyingthem to the real system. Allow users to document the required actions in the form of a note ora script. This helps the users or their colleagues to review the required actions before applyingthem to the system.”Discussion: Many actions in ITSM are responses to new, unseen, and complex situations (Bottaet al., 2011, 2007), and they are performed on artifacts critical to the organization. Moreover,the actions are distributed in time and space and the result of an action cannot be evaluatedin real time. Therefore, errors in ITSM activities could lead to a security breach or disruptservices, which might impose high costs on the organization. For example, Botta et al. (2011)show that an error during deployment of a security patch might disrupt service and conflictwith an organization’s business activities. It can be hard to predict, or instantly determine,the outcome of the patching process, as other stakeholders need to confirm that the serviceis not impacted. To mitigate this, Haber and Bailey (2007) show that SPs employ “rehearsaland planning”, by rehearsing the actions on a test system before performing it on a productionsystem. SP5 described this activity during installation of an IdM system update: “[We have]multiple environments where we can rehearse different [changes to the system]. Because thecustomer releases are so complex to do, you definitely want to try it a couple of times before61you do it in production.”This practice can be clarified from a theoretical perspective. To find a solution to a problem, aSP consults several information sources and combines them into a single artifact (e.g., a plan,a guide document, a check list). This artifact acts as an external memory to the SP. The SP alsointernalizes knowledge from different sources, which might not be completely correct or ap-plicable to the situation at hand. Therefore, this knowledge should be verified before applyingit to the system. Engestro¨m (1999) explains that the process of revising knowledge involvesexternalization, revision, and internalization of the revisions. In the context of ITSM, SPs per-form externalization when they employ rehearsal. If something goes wrong in the rehearsal,SPs re-examine their interpretation of the external knowledge sources and go through the re-hearsal and revision cycle again. After successful rehearsal, SPs can perform the rehearsedactions on the critical artifact.4.3 Evaluation MethodologyBackground: While the ITSM heuristics are grounded in empirical data and supported by the-ory, the effectiveness of them must be evaluated. Heuristic creation literature has tackled theproblem of evaluation in four ways: (1) no evaluation or informal evaluation (Greenberg et al.,2000; Scholtz and Consolvo, 2004), where the effectiveness of heuristics have not been for-mally evaluated; (2) long-term evaluation by using and refining the heuristics in real-worldprojects (Nielsen, 1994); (3) controlled study of the effectiveness without using a controlgroup (Baker et al., 2002; Pinelle et al., 2008); and (4) controlled comparative evaluation,where the effectiveness of heuristics is compared to existing heuristics (Mankoff et al., 2003;Somervell, 2004).We chose the last method to evaluate the effectiveness of the ITSM heuristics. Similar toother domain specific heuristics, we did not use a long term evaluation approach as it requires62longitudinal studies, and access to real-world usability projects. The controlled study withouta control group does not allow recommending the use of the new heuristics over Nielsen’s. Acontrolled comparative evaluation can show us if the new heuristics are more effective thanNielsen’s for the ITSM domain, and if using them adds value to the heuristic evaluation.According to Hartson et al. (2001), the ultimate criteria for effectiveness of a set of heuristics(or a usability evaluation method in general) is finding real problems that user will encounter inreal work contexts, which will have an impact on the usability (e.g., user performance, produc-tivity, and/or satisfaction). However, Olson and Moran (1998) argue that it is almost impossibleto determine if each usability problem is real or not. The best we can do is to estimate the im-pact of the potential problem on the users who will use the system. Therefore, we evaluatedour approach based on the following criteria proposed by Hartson et al. (2001): (1) thorough-ness, the ability of the method to find most of the known problems (see Section 4.3.1 for thedefinition of known problems); (2) reliability, the ability of the method to find severe problems;and (3) validity, the ability of the method to find valid problems (4) effectiveness, ability of themethod to find most of the known problems while it leads to few invalid problems, (5) cost-effectiveness, the cost of using method. While Hartson proposed six criteria for evaluation, weexcluded downstream utility, which refers to the quality of the reported problems and how wellthey lead to effective redesign of the technology. According to Hartson et al. (2001), whileevaluating the downstream utility of usability evaluation methods is commendable, it requireslong-term studies of the impact of identified problems and it is out of the scope of this thesis.We also investigated the characteristics of heuristics evaluation using the ITSM heuristics in-cluding (1) the impact of the number of evaluators on the results; (2) performance of individualheuristics; (3) similarity between ITSM and Nielsen’s heuristics; (4) the impact of participants’background on the reported problems; and (5) the usefulness, learnability, and applicability ofheuristics.To achieve the aforementioned goals, we performed a comparative study of the ITSM heuristics63with Nielsen’s heuristics. This between-subjects study divided participants into two groups:those that used Nielsen’s heuristics (Nielsen condition, 14 participants) and those that used theITSM heuristics (ITSM condition, 14 participants). For the Nielsen condition, we performedfour in person (three, two, two, and one participants per session) and six remote evaluationsessions (one participant per session). For the ITSM condition, we performed three in person(three, three, and one participants per session), and seven remote evaluation sessions (oneparticipant per session).Recruitment: The main inclusion criteria were a HCI background, and familiarity with heuristicevaluation. We sent emails to all graduate students in the CS and ECE departments of UBC. Wealso sent emails to the user experience mailing lists in Vancouver, to online HCI communities,and the CHI-Announcements mailing list, in order to reach participants with HCI experience;and to the HCI-Sec mailing list6 to reach participants with a background in both security andusability. All participants were given a $50 honorarium for their participation.Participants: In an attempt to balance the expertise of participants in each group, we screenedthem to assess their HCI and computer security background. In Table 4.2, we present theparticipants’ demographics. All but one participant had received formal HCI training, withthe majority (17) receiving formal training on heuristic evaluation. The majority (19) hadperformed at least one heuristic evaluation in the past. The participants’ average years ofprofessional computer security experience in ITSM condition was about 3 times more thanthat of Nielsen condition. This difference was mainly due to the high variance in computersecurity background7. We further examine if the difference in computer security backgroundcould have an impact on the outcome of the evaluation in Section 4.4.2.As we described, the majority of participants had performed heuristic evaluation before. Ac-6HCI-Sec is a mailing list for those who do research on usability of security technologies.7There was one outlier with 8 years of professional computer security experience in the ITSM condition.Removing the outlier changes the average years of professional computer security experience as follows: x¯ = 0.46,x˜ = 0, σ2 = 0.44.64Table 4.2: Participants’ demographics for each condition.Condition ITSM Nielsen TotalGroup Size (N) 14 14 28Age 19-24 2 2 425-30 6 7 1331-35 4 1 536-45 2 4 6Gender Female 6 6 12Male 8 8 16Education Diploma 1 0 1Undergraduate 6 8 14Graduate 7 6 13Years of ex-perienceHCI research and profes-sional3.57, 2.5, 13.03 3.29, 2.0, 8.49 3.43, 2.0, 10.70(x¯, x˜, σ2) Computer security research 0.64, 0, 1.93 0.50, 0, 2.57 0.57, 0, 2.18Computer security profes-sional1.0, 0, 4.46 0.32, 0, 0.52 0.66, 0, 2.52cording to Nielsen (1994), it would be impossible to wipe the mind of evaluators of the addi-tional usability knowledge they have, and in reality each evaluator would apply certain heuris-tics from the sets he or she was not suppose to use. Therefore, familiarity with Nielsen’sheuristics would be an advantage for participants in ITSM group. We deliberately recruitedparticipants with a heuristic evaluation background, and made the trade-off between controllingdifferences in the heuristic evaluation background and ecological validity of the study. Ratherthan controlling the knowledge of evaluators in the ITSM group by recruiting participants with-out prior exposure to Nielsen’s heuristics, we recruited participants who were representative ofthose who will use ITSM heuristics in the real world.Target System: We chose an Identity Management (IdM) system for performing the heuristicevaluation. An IdM system is used to manage the digital identities of users in an enterprise andcontrol the accesses of those identities to resources. Furthermore, the system allows request,review, approval, certification, and removal of access. An IdM system is used by various stake-holders in an organization. End-users use the system for creating accounts, requesting access,or changing their passwords. Managers use the system for approving employees’ requests foraccess, reviewing and verifying the validity of their employees’ access, and checking who have65access to the resources they own. Security administrators use the system to implement the re-quests for access, perform large scale provisioning and de-provisioning of access, and createroles.We chose IdM system because of its significance. We showed (Section 4.1) that IdM systemshave wider reach across the organization, and are used in day-to-day activities by various stake-holders. This increases the importance of usability in such tools. Additionally, IdM systemsimpact the functioning of other applications, because they integrate with and manage the accessto those applications.We installed CA Identity Manager 12.0 CR3 in a laboratory environment on a virtual machineusing VMWare Server. Access to the system was through its web interface.Study protocol: An overview of the study protocol is provided in Figure 4.3; we now describethe details of each step.Consent and Background Questionnaire(10 minutes)Training (50 minutes)Evaluation(2 hours)Focus Group / Interviews(1 hour)Figure 4.3: Study protocol overviewWe began by obtaining the participants’ consent, and then asked them to complete a back-ground questionnaire to obtain demographic information and data to assess their HCI andcomputer security background.We then provided training on heuristic evaluation, and described the specific heuristic set to beused. We demonstrated the application of the heuristics in a running example of evaluating anetwork firewall system. The examples were designed to reflect problems with real networkfirewalls. For example, we described the application of ITSM heuristic #2 using a problemwhere two security administrators make changes in the firewall rules, but there was no historyof who made the changes. Or we described Nielsen’s heuristic #4 using a problem where the66firewall rules file contained the following rule: “eth0 inbound block”, but in the UI, the samerule is shown as “block all incoming connections on eth0” (i.e., there is a lack of consistencybetween inbound, and incoming). We concluded the training session with an introduction tothe IdM system. In all cases, training material was presented through online slides with vocalnarratives. That allowed us to provide exactly the same training to all participants regardlessof their location.After the training, participants inspected the interface individually. They had access to thelist of ITSM (first paragraph about each heuristic in Section 4.2.3 including the bold title anditalicized description) or Nielsen’s heuristics (the version available in (Nielsen, 2005a)), aninstance of the IdM system, and an evaluation guide. In the evaluation guide, we limited theevaluation to a few typical usage scenarios (see (Rosson and Carroll, 2002) for description ofscenarios in the context of HCI) to manage the scope of the evaluation and guide evaluatorsduring the evaluation. The participants could then login to the system as the various stakehold-ers while they performed the steps of the scenarios. An overview of the four scenarios used inthe study is presented in Table 4.3. We asked participants (1) to identify usability problems;and (2) for each problem, to specify the scenario and the heuristic (using an online form). Par-ticipants had two hours to perform the evaluation. We limited the evaluation time to control thetime variable, avoid participant fatigue, and emphasize the discount and time-limited nature ofheuristic evaluation.After the evaluation, participants were provided with a post-evaluation questionnaire to ratetheir experience in using heuristics. We then conducted either a focus group (for sessions withmultiple participants) or an interview (for sessions with one participant) to collect qualitativedata on participants’ experience.We piloted and refined our study protocol and materials through several iterations. We per-formed two complete pilot study sessions (six and two participants); and we held several pilottests as we iterated upon the individual study components, including the background question-67Table 4.3: Details of the four scenarios used during the comparative study.Scenario DescriptionSelf-serve user creation A contractor arrives at a company and wants to create a useraccount. He uses the self-service feature in the IdM system tocreate an account. Then a SP reviews and approves his request.Bulk user creation A SP receives a file containing all the users’ job status changesin HR system, uploads the file to the IdM system, and trou-bleshoots errors.Request privileges An employee initiates an access request. The request is ap-proved by a manager, and then reviewed, and implemented bya SP.Certification process The security team initiates a certification campaign and sets adeadline. Managers receive requests to review and certify theprivileges of employees. At the deadline, the security teamcloses the certification process by revoking all of the non-certified privileges.naire (six participants), the description of the heuristics (six), the training materials (two) andthe evaluation guide (seven).4.3.1 Data AnalysisThe following steps were performed by two researchers to aggregate the identified problemsand determine their severity (any inconsistencies were resolved by consensus):Aggregating problems: To aggregate problems found in each condition and generate a list ofknown problems we performed two steps:1. Problem Synthesis: We first decomposed problems into their finest level of granularity.Compound problems should be decomposed as each component of the problem might have acertain severity, and therefore a priority for fixing. Compound problems include those that re-fer to different actions, different artifacts, or different mechanisms in the interface. Then, if anevaluator reported duplicate problems, we removed the duplicate. We then eliminated unknown68problems, which we could not reproduce (e.g., it happened due to a sudden breakdown or crashin the system during the study, or the description of the problem was not understandable). Weremoved false positives, which had any of the following characteristics: (1) the problem wascaused by the constraints or requirements of the underlying operating system or hardware/soft-ware infrastructure, (2) the problem was caused by the business constraints or requirements ofthe program, or (3) the reasoning of the participants in describing the problem was fallacious.2. Combining problems: We began with an empty list of aggregated problems. Each identifiedproblem was compared with the problems in the aggregated list and if it was not present, it wasadded to the list. Otherwise, the description of the problem and its associated heuristics in theaggregated list were updated.Assigning severity ratings: Similar to the study conducted by Nielsen (1992), we asked fourresearchers with knowledge of usability and security, who had training in heuristic evaluation,to independently judge the severity of the problems. The participants used the following pro-tocol to determine the severity of the problems: First, we asked them to answer the followingquestions: (1) Will the problem happen frequently to the users when they perform the activ-ity? (2) Will it be easy for users to achieve their goal when they face the problem? (3) Is it aone-time problem that users can overcome once they know about it or will users repeatedly bebothered by the problem? Then, we asked them to use subjective judgment to categorize theproblem into one of the five levels of severity proposed by Nielsen (2005a): 0-not a usabilityproblem (I don’t agree that this is a usability problem at all), 1-cosmetic (need not be fixedunless extra time is available on project), 2-minor (fixing this should be given low priority),3-major (important to fix, so should be given high priority), and 4-catastrophe (imperative tofix this before product can be released). We gave the list of all problems to each expert withoutany information about the evaluators or heuristics with which the problems were found. Basedon the mean rating, we categorized problems into major (mean severity > 2) and minor (meanseverity≤ 2). We demonstrate examples of problems and their severity in Table 4.4. From 13169identified problems, we chose two high severity (3.00 and 2.75), and two low severity (1.25and 1.25) problems that were found mainly by ITSM and Nielsen participants.4.4 Evaluation ResultsOverview: Table 6.2 shows the classification of the problems in each condition. “ProblemReports” shows the initial number of problems reported by the participants. “Valid” showsthe number of valid reported problems after the synthesis stage. “Known” shows the numberof problems after the combining step in which we combined similar valid problems into oneknown problem. Table 6.2 also shows the classification of known problems as either major orminor severity. We provide a summary of participants’ individual performance in Table 4.6. Wecalculated the performance of the strongest and weakest participants, the proportion of prob-lems found by the first and third quartile, and the ratio between these values. These proportionsare calculated based on the total number of problems (131).Performance of heuristics: We compared the performance of the heuristics used in each condi-tion according to their thoroughness, reliability, validity, and effectiveness. We will discuss thecost-effectiveness in Section 4.4.3. We compared the results from two different perspectives:(1) a per condition basis: we compare the output of evaluation as a whole. (2) a per evaluatorbasis: we compare the performance of individual participants.Thoroughness: We calculate thoroughness as the proportion of the problems identified in eachcondition. Our results show that the evaluation with the ITSM heuristics resulted in finding71% of total known problems (93 out of 131) while the evaluation with Nielsen’s heuristicsresulted in finding 66% of them (86 out of 131). In some cases, finding fewer, but more severe,problems might be more important than finding many minor problems. To examine this, assuggested by Hartson et al. (2001), we used the notion of Weighted Thoroughness (WT) byincreasing the weight of the problems based on their severity. Using Equation 4.1 (we used an70Table 4.4: Examples of the problems identified by the participants. “Context” describesthe context in which the problem was identified. “Problem” describes the problem.“Freq.” shows the number of times the problem is reported in the ITSM(I), andNielsen(N) conditions. “Avg. Sev.” shows the average severity of the problem.“Heuristics” shows the heuristics with which the problems were identified (e.g., I4means ITSM heuristic #4). “IC” indicates that the problem could not be associatedto a heuristic by an ITSM participant.Context ProblemFreq. Avg. HeuristicsI N SevAs a part of Scenario #2, partici-pants should upload a file that per-formed a bulk create, update, andremove. Out of 8 actions scriptedin the file, 1 always failed, and thesystem showed a message that 7records have been updated and 1has failed.There is no way for theuser to know if the filecauses an error. If the fileis large, there is no wayfor the user to determinewhich record caused an er-ror.9 2 3.00 I1, I7,N1, N5As a part of scenario #3, the em-ployees could write their accessrequest in a free text submissionform, and then submit the requestfor processing.During writing the request,if the user pressed enter,the request was submitted,instead of creating a newline. There was no way toedit the request again, orrevert the action.1 4 2.75 IC, N5As a part of scenario #1, theemployee could perform self-registration from the login pageby filling a form. After self-registration, the user was presentedwith a screen saying you aresuccessfully logged out of thesystem.There is no link back to thelogin, or any other pagesfrom the login page.5 8 1.25 N7, ICAs a part of scenario #3, the secu-rity admin should review the em-ployee’s access request and grantthe required access.There is no knowledgebase for the security adminto see the consequences ofsuch access or a way to geta second opinion on givinguser the access.3 0 1.25 I671Table 4.5: Overview of the number and classification of identified problems in each con-dition.Condition Reports Valid Known Major Minor False Positive UnknownITSM 239 201 93 38 55 18 16Nielsen 233 187 86 20 66 45 17All 472 388 131 43 88 62 33Table 4.6: Individual differences in participants’ ability to find problems.Condition Max(%) Min(%) Q1(%) Q3(%) Max/Min Q3/Q1ITSM 23.7 3.82 7.1 13.9 6.2 2.0Nielsen 18.3 3.1 5.9 11.5 6.0 1.9equivalent equation for Nielsen condition), the weighted thoroughness of ITSM and Nielsen’sheuristics are 77% and 60% respectively.WT =∑p∈KnownIT SMSeverity(p)∑p∈KnownSeverity(p)×100 (4.1)To compare two conditions on a per evaluator basis, we tested the following hypothesis: (1)H1: Participants will report more problems if they use ITSM heuristics than Nielsen’s. H0:There is no difference in the number of reported problems. The result of a Mann-Whitney Utest did not reject H0.Reliability: It is important for a set of heuristics to be able to identify major usability issues asthey may seriously hinder the ability of the user to operate the system effectively and efficiently.The results (Table 6.2) show that participants using the set of ITSM heuristics found almosttwice as many major usability problems than the participants using Nielsen’s set.We tested the following hypothesis to show the difference in severity on a per-evaluator ba-sis: H1: The average severity of the problems reported by participants will be higher if thoseindividuals use ITSM heuristics than if they use Nielsen’s. H0: There is no difference in theaverage severity. The result of a Mann-Whitney U test rejected H0 in favor of H1 (U=26,72Z=-3.309, p=0.001).Validity: We examined whether the evaluation with the ITSM heuristics generated fewer falsepositives than Nielsen’s. Participants using the ITSM heuristics reported 201 valid problemsand 18 false positives, whereas participants using Nielsen’s heuristics reported 187 valid prob-lems and 45 false positives. The ITSM heuristics yielded fewer false positives (Table 6.2) thanNielsen’s heuristics. Comparing the number of unknown problems identified in each conditionrevealed a very small difference between conditions.We tested the following hypothesis about difference in the number of false positives on a per-evaluator basis: H1: participants will report fewer false positives if they use ITSM heuristicsthan if they use Nielsen’s. H0: There is no difference in the number of false positives. Theresult of a Mann-Whitney U test rejected H0 in favor of H1 (U=38, Z=-2.823, p=0.005).Effectiveness: We calculated the effectiveness using Equation 4.2 suggested by Hartson et al.(2001). We used the same weight (α) for validity and thoroughness. Our results showed thatthe effectiveness of ITSM heuristics was 0.80 and the effectiveness of Nielsen’s heuristics was0.72.E f f ectiveness =1α(1validity)+(1−α)(1thoroughness) (4.2)The number of evaluators required to perform the evaluation: To replicate Nielsen’s origi-nal analysis (Nielsen and Molich, 1990), we formed aggregates of participants and found theproportion of usability problems identified by each size of aggregate. Following Nielsen’smethodology, we calculated the proportion of found problems based on the total number ofproblems found in each condition. The result is depicted in Figure 4.4. The graph shows thatincreasing the number of evaluators will increase the proportion of the identified problems, butthe rate of the increase diminishes as we increase the number of evaluators. The two plots fromour experiment are very similar and they show a similar trend as compared to the results from73the Mantel, Groove, and GroupDraw experiments.8 Yet, Nielsen’s experiment shows fasterdiminishment compared to our results.lllll ll l l ll l l l2550751001 2 3 4 5 6 7 8 9 10 11 12 13 14Aggregate SizeProportion of problems foundl MantelGrooveGroupDrawNielsenITSMLinearFigure 4.4: Average proportion of problems found by aggregate of participants in ITSMand Nielsen conditions. We also overlaid the results from Nielsen’s Mantel ex-periment (Nielsen and Molich, 1990), and Baker’s Groove and GroupDraw (Bakeret al., 2002) experiments to allow comparisons.We illustrate the distribution of the known problems that are found by participants in the ITSMor Nielsen condition in Figure 4.5. To generate the diagram, we grouped participants basedon their condition and then sorted them from weak to strong (participant A is stronger thanparticipant B, if A found more problems than B). We also sorted the problems from easy to hard(problem A is easier to find than problem B, if A was found by more participants than B). Wehighlighted the severity of the problems by color. The diagram shows that, similar to Nielsen’soriginal experiment (Nielsen and Molich, 1990), there are easy problems that are overlookedby strong participants while there are hard problems that are only found by weak participants.Also, there were major problems that were only found by weak participants and there wereminor problems that were only found by strong participants. This confirms Nielsen’s argumentthat heuristic evaluation is a method that should be done collectively (i.e., no strong evaluatorcan uncover all of the major problems). Figure 4.5 also shows that there was relatively littleduplication between participants in each condition. We further discuss the lack duplication in8To allow comparison, and since the mentioned experiments employed more evaluators, we assumed that thetotal number of problems in each experiment was equal to the problems found by aggregate size of 14.74Section Performance of Individual HeuristicsTo see which heuristics contributed the most in finding usability problems, we visualized thenumber and mean severity of known problems associated with each heuristic in Figure 4.6.The graph shows that the severity of the problems found with ITSM heuristics is higher thanthe ones found with Nielsen’s heuristics. The graph also indicates that ITSM heuristics #6 and#7 were associated with the fewest problems and ITSM heuristic #1 was associated with themost problems. The average severity of the problems associated with ITSM heuristic #2 wasthe highest and Nielsen’s heuristic #8 was the lowest.As our results indicated a large overlap between the problems found using ITSM and Nielsen’sheuristics (i.e., there were known problems that were found in both conditions), we furtherinvestigated the similarity between the two sets. For this we calculated a similarity metricbetween the two heuristics (A and B) as follows:WT =|A∩B||A∪B|×100 (4.3)The result of the similarity analysis is presented in Table 4.7. For each ITSM heuristic, wehighlighted the most similar Nielsen heuristic. These similarities are not surprising. For in-stance, the ITSM #1 and Nielsen’s #1 can both lead to finding a subset of visibility problems;both ITSM #3 and Nielsen’s #7 can lead to problems related to interacting with users withdifferent levels of expertise, ITSM #4 and Nielsen’s #5 will both lead to preventing errors byapplying constraints. ITSM #6 and Nielsen’s #9 are also shown to be similar, as providing userswith the required knowledge will help them understand errors and recover from them. We alsoshow how each of the Nielsen’s heuristics performed in finding problems unique to their con-75PPPPPPPPPPPPPPPPPPPPPPPPPPPPA10 < Severity < 1ITSMNielsen1 ≤ Severity < 2 2 ≤ Severity < 3 3 ≤ Severity < 4 Easy Hard Strong Weak Strong Weak Figure 4.5: Problems identified by each participant in each condition. Each row corre-sponds to a participant and each column corresponds to a problem. Participantsin each condition are sorted from top (weak) to bottom (strong) and problems aresorted from right (easy) to left (hard).7640 22 22 3626 18 12 18 235N7 N8 N9 N1017.9 11.5 10.6 11.58.7 0 12.5 7.727.1 12.8 9.3 10.413.5 6.3 3.7 17.210.5 3 0 02.8 3.6 15 3.69.1 3.7 10 023.5 17 6.1666667 18.75 13.416667 11 12.75 13 4.2560.25641 68 38.541667 53.571429 47.916667 61.111111 49.038462 72.222222 35.41666716 13 3 17 9 7 9 11 341 52 19 49 32 39 35 61 251.47 1.31 2.06 1.10 1.49 1.57 1.42 1.18 1.42N1 N2 N3 N4 N5 N6 N7 N8 N90 0 0 1 0 0 0 0 01 0 0 1 0 0 0 1 01 0 0 0 0 0 0 0 00 0 0 0 0 0 0 0 01 0 0 0 1 0 0 0 10 0 0 0 0 0 0 0 00 0 0 1 1 0 0 0 00 0 0 0 1 1 0 0 01 1 0 1 0 0 1 1 00 0 0 1 0 0 0 1 00 0 0 1 1 0 0 0 01 0 0 1 1 1 0 1 01 0 0 0 1 0 0 0 01 0 1 1 0 0 0 0 00 1 0 0 0 0 0 0 01 0 1 1 1 1 1 0 01 1 0 1 1 1 0 1 00 0 0 0 0 0 0 1 00 1 0 1 0 0 0 0 11 1 0 1 1 0 0 0 01 1 0 1 0 0 0 0 01 0 1 1 1 1 0 0 01 1 0 1 0 0 0 1 01 0 0 0 0 0 1 1 01.2	  1.4	  1.6	  1.8	  2	  2.2	  2.4	  0	   5	   10	   15	   20	   25	   30	   35	   40	   45	  Average SeverityNumber of problemsITSMNielsen's3	  1	  4	  6	  7	   5	  2	  9	  3	  8	  10	  6	  2	  5	  7	  4	  1	  Figure 4.6: The number and mean severity of problems identified by each heuristic.The ITSM heuristics are shown using black diamonds and Nielsen’s heuristics areshown using gray circles. Each heuristic is labeled with its number.Table 4.7: Similarity between individual ITSM and Nielsen’s heuristics. Each cell showsthe value of similarity metric for the heuristics denoted by row and column indexes.For each ITSM heuristic, the cell with the highest number (i.e., the most similarNielsen heuristic) is highlighted.Nielsen Heuristics1 2 3 4 5 6 7 8 9 10ITSMHeuristics 1 29.5 16.1 14.3 17.2 15.3 16 17.9 11.5 10.6 11.52 14.5 6.5 8.1 7.3 15.6 2.4 8.7 0.0 12.5 7.73 25.4 11.1 13.3 18.6 18.9 15.2 27.1 12.8 9.3 10.44 14.6 7.9 6.7 8.5 18.9 9.7 13.5 6.3 3.7 17.25 5.8 7.9 0.0 4.1 4.8 6.3 10.5 3.0 0.0 0.06 6.4 2.9 3.8 0.0 5.4 3.6 2.8 3.6 15.0 3.67 6.5 2.9 4. 4.7 8.6 3.7 9.1 3.7 10.0 .0dition (i.e., problems that were not reported in ITSM condition), and the average severity ofthose unique problems in Table 4.8. We will corroborate this data with participants’ feedbackto determine which of Nielsen’s heuristics complement the ITSM heuristics in Section 4.5.77Table 4.8: Ability of each of Nielsen’s and the ITSM heuristics to find problems uniqueto their condition. The “Proportion of unique” row shows the proportion of problemsuniquely found in the Nielsen or ITSM conditions using the corresponding heuristic.The “Average severity” row shows the average severity of those unique problems.Nielsen Heuristics1 2 3 4 5 6 7 8 9 10Proportion of unique 0.41 0.52 0.19 0.49 0.32 0.39 0.35 0.61 0.25 0.56Average severity 1.47 1.31 2.06 1.10 1.49 1.57 1.42 1.18 1.42 1.33ITSM Heuristics1 2 3 4 5 6 7Proportion of unique 0.43 0.50 0.37 0.25 0.63 0.55 0.50Average severity 2.12 2.31 1.87 2.38 2.35 2.29 Impact of Participants’ Background on Their PerformanceNielsen (1992) suggests that the evaluator’s HCI and domain expertise are two factors that in-fluence the quality of heuristic evaluation. We analyzed the HCI and computer security back-ground of the participants to find the correlation between expertise (years of HCI and computersecurity experience, number of previously performed heuristic evaluations), and performance(number of raw, known, false positive problems, and average severity). We first used a factorreduction technique to find the possible medium or strong correlations and then investigatedcorrelations and their statistical significance with either Pearson’s product-moment coefficient(for normally distributed data) or Kendall tau rank correlation coefficient (for non-normaldata). In the Nielsen condition, we found a strong negative correlation between the numberof previously performed heuristic evaluations and the average severity of reported problems(r =−0.70, p < 0.05,N = 14). In the ITSM condition, we found a strong positive correlationbetween the number of previously performed heuristic evaluations and the number of falsepositives (τ = 0.55, p < 0.05,N = 14). For the overall study data, we identified a medium tostrong correlation between the years of HCI experience and the number of reported problems(r = 0.47, p < 0.05,N = 28). We did not find any correlation between the severity of the re-ported problems and the background of the participants (i.e., between severity of the reported78problems and years of HCI (τ = −0.17, p = 0.21,N = 28), professional computer security(τ = 0.18, p = 0.28,N = 28), or academic computer security (τ = 0.19, p = 0.22,N = 28) ex-perience.) In Section 4.3, we reported that the average length of computer security experiencein ITSM condition was more than three times higher than in Nielsen condition before removingthe outlier. Yet, as we showed above, there is no correlation between the amount of computersecurity experience and the severity of the reported problems. This suggest the differences aredue to the condition rather than participants’ security experience.4.4.3 Participants’ Feedback in Post-evaluation QuestionnaireWe asked our participants to evaluate with a 5-point Likert scale (5=strongly agree, 1-stronglydisagree) how useful the set of heuristics was in identifying usability problems (usefulness),how easy it was to understand and learn the heuristics (learnability), and how easy it was toapply the heuristics to the IdM system (applicability). The mean usefulness, learnability, andapplicability ratings for ITSM condition was 3.14, 3.36, and 2.86 respectively, and for Nielsencondition was 3.36, 3.57, and 3.5. We conducted a Mann-Whitney U test to evaluate whetherthe set of heuristics used impacted the usefulness, learnability, and applicability, as reportedby our participants. Although, the ITSM heuristics were new to our participants there wasno significant difference between the ratings for the two sets of heuristics. As we highlightedbefore, one measure of cost-effectiveness of a usability evaluation method is the effort requiredto learn it. Therefore, our results suggest that there is no statistically significant differencebetween the cost-effectiveness of the two sets of heuristics.We also asked participants to evaluate with a 5-point Likert scale the usefulness, learnability,and applicability of each individual heuristics. The mean scores of the ITSM and Nielsen’sheuristics are shown in Figure 4.7. Repeated measures ANOVA calculations between the meanscores of individual Nielsen’s heuristics revealed only a significant difference in terms of use-fulness (F(9,117) = 2.40, p < 0.05). Post hoc tests using Bonferroni correction showed a79 Usefulness Std. Deviation Learnability Applicability Usefulness Learnability ApplicabilityI1 1.5000 .65044 1.2857 .61125 1.3571 .63332 4.5000 4.71428571428571 4.64285714285714I2 1.9286 .82874 1.4286 .75593 1.7143 .91387 4.0714 4.57142857142857 4.28571428571429I3 2.3571 .92878 1.4286 .64621 2.3571 1.08182 3.6429 4.57142857142857 3.64285714285714I4 2.7143 1.13873 2.5000 1.09193 2.3571 .92878 3.2857 3.50000000000000 3.64285714285714I5 2.4286 .93761 1.8571 .94926 2.2857 .82542 3.5714 4.14285714285714 3.71428571428571I6 3.3571 1.00821 2.2857 .82542 3.4286 1.22250 2.6429 3.71428571428571 2.57142857142857I7 3.5714 .93761 2.2143 .89258 3.5714 .75593 2.4286 3.78571428571429 2.42857142857143N1 1.7143 1.5000 1.7857 4.2857 4.50000000000000 4.21428571428571N2 2.3571 1.8571 2.3571 3.6429 4.14285714285714 3.64285714285714N3 3.0000 1.7857 2.6429 3.0000 4.21428571428571 3.35714285714286N4 2.2143 1.7143 2.2143 3.7857 4.28571428571429 3.78571428571429N5 1.7143 1.3571 2.1429 4.2857 4.64285714285714 3.85714285714286N6 2.2857 1.5714 2.0000 3.7143 4.42857142857143 4.00000000000000N7 2.6429 2.0714 2.6429 3.3571 3.92857142857143 3.35714285714286N8 2.5714 1.6429 2.4286 3.4286 4.35714285714286 3.57142857142857N9 1.7143 1.5000 2.2857 4.2857 4.50000000000000 3.71428571428571N10 2.2857 1.2143 2.1429 3.7143 4.78571428571429 3.85714285714286VAR00018 14VAR00019 14VAR00020 14VAR00021 14Descriptive Statistics1.0$1.5$2.0$2.5$3.0$3.5$4.0$1$ 2$ 3$ 4$ 5$ 6$ 7$ 8$ 9$ 10$Average'ra(ng'on'a'5,point'likert'scale'Nielsen's'heuris(cs'Usefulness$Leanability$Applicability$1.0$1.5$2.0$2.5$3.0$3.5$4.0$4.5$5.0$I1$ I2$ I3$ I4$ I5$ I6$ I7$ N1$ N2$ N3$ N4$ N5$ N6$ N7$ N8$ N9$ N10$Average rating on a 5-point likert scale!The ITSM heuristics!Usefulness! Learnability! Applicability!The Nielsen heuristics!Figure 4.7: Mean scores of participants’ reported usefulness, learnability, and ease ofapplication for the different heuristics (5=strongly agree, 1=strongly disagree).marginally significant difference9 between heuristics N#1 and N#3 (p=0.138), N#5 and N#3(p=0.138). Repeated measures ANOVA tests for the ITSM heuristics determined a statisti-cally significant difference in heuristics usefulness (F(6,78) = 10.18, p < 0.005), learnability(F(6,78) = 6.92, p < 0.005) and applicability (F(6,78) = 12.45, p < 0.005). Post hoc tests us-ing Bonferroni correction shows that the significant difference in usefulness was mainly causedby heuristics I#6 and I#7, the significant difference in learnability was caused by the differencebetween heuristic I#4 and heuristics I#1 and I#3, and the difference in applicability was mainlycaused by heuristic I# Qualitative Feedback During Focus Group/interview SessionIn this section, we provide a summary of participants’ feedback during interviews and focusgroups. We identify participants in the ITSM condition by PI1 to PI14, and the Nielsen condi-tion by PN1 to PN14.We asked open-ended questions about the usefulness, ease of understanding, and applicabilityof the heuristics. Furthermore, we asked participants if they noticed problems that could notbe found with or associated to the heuristics, and to improve the current heuristics set or add anew heuristic to it.9The use of Bonferroni with large number of pair-wise comparisons leads to a very conservative result; there-fore we considered the difference significant.80In the Nielsen condition, all participants confirmed the heuristics’ usefulness, e.g., PN4 ex-plained:“They give me a standard way to review each of the screens. [...] At least be able toevaluate based on a common set of methods or processes”. But it was challenging for some ofthe participants to apply heuristics to the system, without understanding the background of thereal users of the system, e.g., PN8 explained: “I found them useful for some of the actors [in thescenarios]. When it gets to [a manager], it becomes harder to get into the user’s mindset. Andwhen we get to the [security admin], it is not useful at all because he is an expert”. This pointwas also confirmed by PN10, and PN13 who indicated that the heuristics such as “Flexibilityand efficiency of use”, and “Match between system and real world” require understanding offlexibility for a security admin and his mental model of the real world. PN8 also indicatedthat many of the problems that might be important for end-users, might not be as important forsecurity admins who will be trained to use the tool.Similarly, the ITSM participants found the heuristics useful, but not without problems. Many ofthe participants (PI7, PI5, PI6, PI1, PI14, PI10) indicated that while they understood heuristics#6 or #7, they were not applicable to the four scenarios in the study. Yet, PI13 indicated thatit took some time to grasp the last two heuristics: “at the beginning, I was very focused onthe first heuristics. But it was towards the end that I was starting to think about the problemsrelated to [ITSM #6, #7]. But when you start thinking about them, it becomes intuitive to seethe problems related to those.” On the contrary, PI8 describe ITSM heuristic #4 as hard toapply: “I had a hard time to apply [ITSM #4] because it can be applied at different levels. Youcan say the system should limit what user can enter in his request as much as possible, or usercan enter whatever he wants and then it is up to manager to review and decide whether userentered a valid request.” PI8 then expressed disagreement that tools always “should” constrainpossible actions, and suggested that tools “could” constrain possible actions depending on thesituation.We asked participants in the Nielsen condition about the aspects of the system not covered by81the heuristics. Four participants indicated that problems with the workflow cannot be classifiedin Nielsen’s heuristic and that they classified them as lack of showing different steps of work-flow (PN4, PN10), lack of ability to revert back to one of the previous steps of the workflow(PN6), and lack of a coherent workflow (PN1). PN12 believed that N#1 should be changedto “Visibility” because visibility can go beyond the system status. PN7 described that the in-terface offered too many options for performing tasks, and no heuristic covered that. Then,PN7 suggested a heuristic for changing the presentation based on the role of the user in theorganization. Similarly, PN8 suggested dividing N#8 to two heuristics: (1) aesthetics and, (2)the level of detail for expert and non-expert users.In the ITSM condition, PI6, PI14, and PI12 asked for Nielsen’s heuristics set to use in additionto the ITSM heuristics, and other participants suggested individual Nielsen’s heuristics. Forexample, PI7 indicated the need for an error prevention heuristics, as well as better error mes-sages. PI1 suggested an error recovery like undo or redo. The need for a consistency heuristicwas indicated by PI2, PI14, and PI6. A heuristic for organization of the screen was suggestedby PI2, PI12, and PI6. Understandable language was indicated by PI10, PI12, PI14.We mapped the heuristics that participants in each condition indicated as missing to one ofthe heuristics in the other condition. As a result, we saw participants in the Nielsen conditionfound ITSM heuristics #1, #3, and #4 necessary. Participants in the ITSM condition indicatedthe need for Nielsen’s heuristics #2, #3, #4, #5, #8, and #9.Almost all participants explained that they first identified problems and then tried to “assign”each problem to one of the heuristics. Yet, they still found the heuristics helpful in findingproblems: e.g., PN2 explained: “[The heuristics] remind you of existence of possible problems.I might forget to look at help and documentation if I don’t have the heuristics.” PN9 hadheuristics in mind when looking at the interface: “For me, I found the problem and then Imatched it. But I had heuristics in my mind, and when I looked at the interface I was thinkingif it breaks anything.” PN8 explained the role of heuristics in disambiguating problems: “I82look for a submit button, I don’t see it. I think, it might be a problem. But then heuristics helpme find exactly what the problem might be.” PI12 described the role of heuristics in predictingproblems and designing test cases to uncover them: “As soon as I read the description forscenario one, I thought ‘oh I bet that is gonna break heuristic number five’. And then I figuredout a little test case and tested it.” After this point was brought up by PI12, it was confirmedby the two other participants in the same focus group.When commenting on the study procedures, participants indicated that if they had more time,they would have found more usability problems.4.5 DiscussionThe evaluation results suggest that our heuristics performed well overall in finding usabilityproblems in ITSM tools. In this section, we interpret the results and discuss their implications.Few overlaps between individual evaluators: We observed fewer overlaps between problemsidentified by evaluators in our experiment than problems identified by evaluators in Nielsen’soriginal experiment (Nielsen and Molich, 1990). In both conditions of our experiment, onlythree problems were identified by the majority of participants, and more than half of the prob-lems were identified by only one. In contrast, in Nielsen’s evaluations of the Mantel andSavings systems (Nielsen and Molich, 1990), only one and two problems respectively wereidentified by just one participant. In Baker et al. (2002) evaluation of GroupDraw and Groove,14 out of 64 and 5 out of 43 problems were found only by one participant. Our results showfewer overlaps between problems identified by different participants, compared to Nielsen’sand Baker’s results. Four factors might have contributed to this outcome. First, the evaluatedIdM system was fairly large; the participants had to visit 20 different web pages in order tosuccessfully complete all scenarios. This multitude of web interfaces provided an opportunityfor finding more diverse problems than in the cases of systems used in Nielsen’s or Baker’s83evaluations (e.g., Mantel only had a single screen and a few system messages, GroupDraw hadtwo screens). Second, we used fewer participants (14 per condition) compared to 77, 34, 25,and 27 participants in Mantel, Savings, GroupDraw, and Groove systems, respectively. Third,the evaluated system was a commercial product rather than a prototype and, it did not containmany obvious usability problems. Fourth, participants thought they could find more problemsif they had more time. For reference, Nielsen and Molich (1990) do not mention any timeconstraints during their study; and Baker et al. (2002) allowed participants to use as much timeas they needed to evaluate the interfaces. Our results illustrate how hard and time consumingthe heuristic evaluation of ITSM tools in the size and complexity of the IdM systems is; wediscuss why we limited the evaluation time to two hours in Section 4.3.Large overlaps between known problems reported in the ITSM and Nielsen conditions: TheITSM heuristics were consistent with activity theory and Nielsen’s heuristics were consistentwith action theory. As a result, we expected to have very few problems that overlap the twoconditions. But our results show that 48 problems (37%) were found in both conditions. Threefactors might have resulted in this observation. First, participants in the ITSM condition couldremember Nielsen’s heuristics, which helped them see problems at the action level. Second,most of the participants found a problem first, and then fit it into one of the heuristics. Third,the similarity between heuristics (Table 4.7) could also result in overlaps.Usefulness of ITSM heuristics: It is encouraging that despite the novelty of the ITSM heuris-tics, participants found them to be no less effective, easy to use, or easy to learn than Nielsen’sheuristics. Yet, when we looked at the individual heuristics, we saw that ITSM heuristics #6and #7 were not as useful and easy to apply. Looking at the number of problems reportedusing those heuristics confirms this observation (Figure 4.6). We can provide several explana-tions for this observation. First, the study scenarios did not include extensive deployment orconfiguration tasks that involve verification to a great extent, or tasks that deal with unfore-seen conditions or troubleshooting, which require extensive knowledge sharing. Second, one84participant indicated that the last two heuristics were ignored because of focusing on the firstheuristics in the list. Therefore, the order of the heuristics might have influenced their use inour time-limited evaluation sessions. Our judgment here is based on the participants’ feedbackand it needs further study. Third, we believe that the ITSM heuristics #6 and #7 were less opento interpretation than heuristics #1 or #3, which are applicable over a broad range of tasks.Specificity of heuristics to the ITSM domain: While our goal was to develop specific ITSMheuristics, some of the heuristics seem to be rather general and applicable to other domainsas well. This generality was the result of finding general guidelines that eventually led tocreation of heuristics. Looking at the data that supports those general guidelines shows thatITSM shares characteristics with other domains. For example, it shares complexity with IT,creativity with software development, and uncertainty with military. As a result, some of therecommendations for designing better ITSM systems might be similar to the recommendationsfor designing other systems with similar characteristics.An ideal set of heuristics for evaluation: Our results suggest that using the ITSM heuristicsleads to finding more severe problems. However, using Nielsen’s heuristics led to findinga unique set of problems that couldn’t be found using ITSM; while those problems mightnot be as severe, addressing them can improve the interaction between user and the system.Therefore, we believe that the ITSM and Nielsen’s heuristics can offer different perspectivesfor evaluation and they can be combined and used in three different ways all of which havetrade-offs that should be considered according to the ITSM system being evaluated and theresources available for the evaluation: First, both sets can be used together in one evaluationsession. This approach gives participants a holistic view of possible problems at both the actionand activity levels. On the contrary, Nielsen argued that the use of more than 10 heuristics isnot effective, and evaluators cannot remember all of the heuristics. Furthermore, evaluatorswho have previous experience with Nielsen’s heuristics might tend to focus more on thoseheuristics and might ignore the ITSM heuristics. The second approach would be to use a85subset of Nielsen’s heuristics (at most three to be consistent with Nielsen’s recommendation ofusing at most 10 heuristics) in addition to the ITSM heuristics. Our participants suggested theneed for six of Nielsen’s heuristics. Based on the data from Table 4.8, we can suggest the useof Nielsen’s heuristics #2, #4, and #5. The benefit of this approach is reducing the evaluators’mental overload, and allowing them to find action level problems that are critical to the targetapplication. The drawback is that participants might focus on those three Nielsen’s heuristicsthat they know, rather than the ITSM ones. The suggestion of specific Nielsen’s heuristicsis solely based on the data of this study, and the evaluated IdM system. The choice of theheuristics should be changed depending on the goal of the evaluation. For example, if the maindesign goal of a project is aesthetics, one can replace one of Nielsen’s other heuristics with#8. The third approach would be to use Nielsen’s and ITSM heuristics in separate evaluationsessions by the same or different evaluators. We expect this approach to have the highestthoroughness and yet the highest cost.The impact of participants’ background on their performance: Our results suggest that theaverage years of HCI experience is positively correlated with the number of reported problems,but not with their severity. This result supports Nielsen’s finding that regular specialists willfind more usability problems than novice evaluators. On the other hand, the results suggestthat the number of previously performed heuristic evaluations negatively impacted the sever-ity of the problems in the Nielsen condition and false positives in the ITSM condition. Thisobservation was surprising. We hypothesize that participants with prior heuristic evaluationexperience tend to evaluate the systems for end-users, with a focus on aesthetics of the inter-face. This resulted in minor problems in Nielsen condition, and false positives in the ITSMcondition. Further study is needed to validate the reasons behind this observation.Generalizability of evaluation results: If we were to replicate the comparative evaluation studyfor a different ITSM tool (e.g., one of those listed in Section 4.1), we would expect the ITSMheuristics to be still applicable and to find more severe problems than with Nielsen’s. The scope86of the empirical data which the heuristics were created based on was ITSM tools in general,rather than a specific IdM system. Also, the heuristics were supported by a general HCI theory.This leads us to believe the ITSM heuristics are general enough for evaluating most ITSMtools.At the same time, the performance of individual heuristics may vary for different categories ofITSM tools. As we discussed in Section 4.1, IdM systems have a wide reach across the organi-zation, and are used by many users. Therefore, the study participants reported a large numberof usability problems for the visibility of activity status. In contrast, a security operations toolsuch as network traffic analyzer, has a narrow reach across the organization and is mainly usedby SPs. Such a tool should help SPs deal with complex and large scale network traffic logs, anddetecting malicious, unknown content in network traffic. Therefore, the evaluators of the toolmay focus on flexible representation, and knowledge sharing rather than visibility of activity.The evaluation results could vary between individual systems of the same type. For example,the evaluated IdM system offered rather meaningful error messages to users. Therefore, heuris-tic N#9 (error recovery) was the least used Nielsen heuristic. Another IdM system may presenterrors as, say alphanumeric codes, or without indicating exact problems. In evaluating such asystem, we might see more use of N#9.4.6 Limitations and Future WorkWe used participants with HCI and heuristic evaluation background, and as we discussed inSection 4.3, we made a trade-off to compare the two sets of heuristics in an ecologically validsetting. As a result, we cannot make arguments about the performance of the two sets ofheuristics when they are used by participants who were not previously exposed to heuristicevaluation.While increasing the number of participants in either of the conditions did not saturate the87list of identified problems, we observed that the rate of finding problems decreased. Con-tinuing the experiment with more participants would certainly allow us to find the point ofdiminishing returns in the number of problems. But comparing our results with the Group-Draw evaluation (Baker et al., 2002) suggests that even doubling the number of participantswould not allow us to observe saturation. Furthermore, our study required a four-hour timecommitment from participants with an HCI background, which made recruitment challenging.Because determining such a saturation point was not the main goal of our study, we leave suchinvestigation for future work.There are several opportunities for improvement and future work. First, during the problemsynthesis stage, the severity of problems was estimated by four severity raters with a back-ground in both usability and security. While this is a standard approach for determining theseverity of problems in heuristic evaluation, it is only an approximation of severity. Askingthe opinion of real system users to determine the severity of the problems is another method.Neither of these approximations might be precise, but combining the ratings would increaseconfidence.4.7 ConclusionIn this chapter, we reviewed the prior research on heuristic creation. We then described ourmethodical way of creating domain specific usability heuristics, which we applied to createa set of usability heuristics for evaluation of ITSM tools. To examine the applicability ofthe heuristics, we compared their use with Nielsen’s heuristics for the evaluation of an IdMsystem. We tried to maximize the ecological validity of the study by using a real ITSM tool,and recruiting participants with an HCI background and familiarity with heuristic evaluation.Our results show that a combination of a top-down and bottom-up approach resulted in a set ofheuristics that were applicable to the target domain (as they were based on the domain-specific88data), and yet were general enough to help evaluators find diverse set of problems. Comparingthe new heuristics to Nielsen’s heuristics revealed that the severity of the problems found byparticipants in the ITSM condition was higher than those found in Nielsen condition. Further-more, our participants found the ITSM heuristics to be as relevant, easy to apply, and easyto learn as Nielsen’s. The results of our evaluation also shed light on the use of the heuristicevaluation for evaluating a complex domain-specific system. While Nielsen found that fiveevaluators are able to find about two thirds of the problems, in our evaluation of the IdM sys-tem, five evaluators only found about half of the problems found by 14 evaluators. Additionally,the complexity and scale of the system can result in a lack of overlapping problems betweenevaluators. Finally, our results show that Nielsen’s heuristics can also be effective in finding aclass of problems in ITSM tools that cannot be found by the ITSM heuristics. Therefore, wediscussed three approaches for using a combination of the two sets of heuristics.The ITSM heuristics are a component of tool usability evaluation and can be used as a discountmethod to find usability problems in prototypes or actual tools. These problems can be furtherinvestigated by a user study or a contextual inquiry session. Design guidelines (e.g., Chapter 3)can then be used to address the problems.89Chapter 5Field Study of Identity and AccessManagement1Identity and Access Management (IAM) comprises the processes and infrastructure for thecreation and maintenance of user’s digital identities and the designation of who has accessto resources, who grants that access, and how accountability and compliance are maintained(Blum, 2005). While IAM can be studied in different contexts (e.g., the Internet), the scope ofIAM in this dissertation is enterprise identity and access management.IAM has become an important aspect of IT security infrastructure in organizations, and someconsider it to be the most important solution for enabling compliance with legislative require-ments (Wright, 2007). According to Microsoft Corporation (2006), further drivers of IAMadoption include cost reduction, better security, better access to information, and better agilityduring mergers and acquisitions. However, the practice of IAM is challenging, both organi-zationally and technologically (Microsoft Corporation, 2006; Wright, 2007). Identifying these1The initial version of the field study (analysis of 4 interviews) was published as a conference paper:P. Jaferian, D. Botta, K. Hawkey, and K. Beznosov. 2009. A case study of enterprise identity management system adoption in an insuranceorganization. In Proceedings of the Symposium on Computer Human Interaction for the Management of Information Technology (CHiMiT’09). Baltimore, MD, USA, Article 7 , 10 pages. (acceptance rate: 33%)90challenges and studying how they can be addressed are important steps toward improving IAMsystems and practices in organizations.Despite the widespread and increasing adoption of IAM solutions, there are few available em-pirical studies that examine the practice of IAM in organizations. In order to improve theusability of IAM systems, or change the IAM practices, more empirical studies using real-world data are needed to illuminate nuances of the issues that are already indicated by priorresearch, and to reveal topics for further research.In this chapter, we present the results of a field study of IAM in organizations. We started ourresearch with three broad questions:1. How do organizations manage identities and accesses of employees ?2. Why is IAM challenging, and what are the challenges in managing users’ identities andaccesses ?3. How can technology or practice be improved to address the challenges ?Answering the above research questions allows a deep understanding of the IAM and its chal-lenges, and could guide improvement of technology or practice to address the challenges. Toanswer the above questions, we chose grounded theory to collect and analyze real-world dataabout IAM. Grounded theory is an appropriate methodology to perform an exploratory studyof a socio-technical phenomena, and to provide a rich understanding of the studied phenomenausing qualitative data. As the outcome of our data analysis, we provide a thick descriptionof IAM activities, and their challenges. We further develop two models that explain the rela-tionship between IAM activities, and challenges. We further compare our findings with priortheoretical and empirical access control literature, and discuss similarities and differences.915.1 MethodologyTo answer the research questions listed in the beginning of this chapter, we used groundedtheory methodology. We conducted 19 semi-structured interviews with security practitionersinvolved in identity and access management in large organizations. The list of interview par-ticipants, their role, and their organization sector are shown in 5.1. The scope of the interviewswas various activities related to IAM in the participants’ organizations. The interviews wereaudio-recorded, transcribed, and analyzed.Table 5.1: Interview Participants’ DemographicsCode Job Title Organization GenderP1 Security Manager Insurance A MaleP2 Security Analyst Insurance A MaleP3, P7 Security Manager Software A MaleP4 Security Administrator Software A FemaleP5 Security Administrator Software A MaleP6 Compliance Manager Software A MaleP8 Security Analyst Insurance A FemaleP9 Consultant Health care MaleP10 Consultant Financial FemaleP11 Consultant Financial MaleP12 VP Sales Software B MaleP13 Business Executive Software C MaleP14 IT Manager Insurance B MaleP15 Chief Technology Officer Software D MaleP16 Consultant Consulting MaleP17 Consultant Consulting MaleP18 Security Administrator Insurance A MaleP19 Security Analyst Insurance A Male5.1.1 RecruitmentAccording to Charmaz (2006), grounded theory requires two types of sampling: purposive,and theoretical. We began recruitment in December 2010 with purposive sampling. Our in-clusion criteria for participants was experience with deployment, or ongoing management ofidentity and access management systems. While our initial impression was that any security92administrators could be a potential participant, we soon realized that few administrators haveexperience in this area, and finding target participants would be extremely challenging. Weused multiple techniques to reach to potential participants:Professional contacts: We previously had experience of recruiting security professionals as apart of the HOT-Admin project (see Chapter 2 for an overview of the project). We iden-tified some of those contacts that might fall into our inclusion criteria, and invited themfor interviews. The following participants were recruited through professional contacts:P1, P2, P10, P11Industry partner: We asked our industry partner to identify potential participants and intro-duce them to us. We recruited the following participants through our industry partner:P3, P4, P5, P6, P7Local IT security communities : We participated in local IT security networking events suchas Vancouver Security Special Interest Group (Infosec BC) meetings, and VancouverInformation Systems Security Association (ISSA) meetings to identify potential partic-ipants, and connect with them. We also presented the results of our studies on twooccasions to promote our research, create awareness about our project, and get feedbackon the findings. We recruited the following participants through local communities: P8,P9Online: At the later stages of the project, and as a part of our theoretical sampling, we recruitedparticipants through a survey (see Chapter 6 for survey results, and Appendix C forsurvey questions). We designed a survey with the focus on access review activity, andpublicized the survey with the help of Forrester Research company. At the end of thesurvey, we invited participants to have an interview with us. We recruited the followingparticipants through our online survey: P12, P13, P14, P15, P16, P1793We offered the participants who were recruited online a $50 Amazon gift card as an honorar-ium. We did not offer the rest of the participants any incentives.We recruited participants P1 to P11 as a part of our purposive sampling. Then we turnedour focus on a particular set of IAM activities, such as provisioning and access review. Werecruited P12 to P17 as a part of our theoretical sampling, by focusing on participants withexperience of ongoing IAM management rather than IAM deployment and configuration. Wealso used data from two of the previous interviews that were done by other researchers in theHOT-Admin project. Those two interviews (P18 and P19) were not performed as a part of thisproject (i.e., done using a different interview guide, and by researchers other than the author).But the participants in those interviews talked extensively about their experience with identityand access management.5.1.2 Interview ProcessThe interviews were conducted by one or two interviewers in-person (9 interviews) or over thephone (11 interviews). Each interview was started by explaining the research project to theparticipants and then asking them to sign a consent form (participants P1 to P9, and P18 toP19) or verbally consent to participate in the study (participants P10 to P17). Then we startedaudio-recording the interview. During the in-person interviews, we collected pictures of theartifacts participants showed to us. We also gathered and analyzed the documents provided bythe participants. The length of the interviews was between one and three hours.We designed a comprehensive interview guide (Appendix B) that we used during the inter-views. The interview guide gave us the list of all possible topics to cover, but in almost all ofthe interviews we chose to use a subset of questions based on our analysis of previous inter-views, and the participant’s experience and expertise. We also modified and re-phrased someof the questions based on the participant’s responses to previous questions.945.1.3 Data AnalysisWe used grounded theory coding methods suggested by Charmaz (2006) for data analysis.We imported the transcripts of the interviews to a qualitative data analysis software (initiallyQualrus v2.1 and then MAXQDA v11) for coding and memoing purpose. The four initialinterviews were analyzed by two researchers, and the rest of the interviews were analyzed byone researcher.Data analysis was started with an initial listening to the interview to make sense of the entireinterview before getting into the analysis. Then we started the formal analysis by performingopen-coding. According to Glaser and Strauss (1967), open coding uses codes emerged fromthe data itself, and they could be constructed codes, or be in-vivo codes (participants’ ownwords in the interviews). We used a line-by-line coding approach recommended by Glaser(1978) in analyzing the initial set of interviews (P1 to P11, and P18-P19). We performed con-stant comparison suggested by Glaser and Strauss (1967) by constantly comparing the similarincidents from the same or different interviews with each other, and created categories for sim-ilar incidents. The constant comparison technique led to identifying certain emergent themesin the data, which we chose as a set of focused codes. Interviews (P12 to P17) were codedusing focused coding technique, and were done on incident-to-incident basis (i.e., the codescould span over multiple paragraphs).After the initial open coding stage, we performed axial coding suggested by Corbin and Strauss(1990). We identified the relationship between the emerged focused codes, and categorized thesimilar codes as sub-categories of more abstract codes.Towards the end of the analysis, we performed selective coding. We chose one category asthe core category around which the theory is constructed. We chose access provisioning asour core category for multiple reasons. First, we saw that the category appeared frequently inall of the interviews, and it was one of the main concerns for our participants. Second, the95category was very well connected to other categories. Third, we saw that the ultimate goal ofIAM is achieved through the access provisioning activity, and other observed activities play asupporting role in the entire IAM big picture.We used two types of memos extensively during the analysis. First, during the coding practice,we wrote our interpretation of the coded excerpt in the qualitative analysis software. Second,during the initial listening (for the last six interviews), we wrote memos while listening to theinterview using an audio bookmarking mobile application.We stopped data collection and analysis, as we reached theoretical saturation. While analyzingthe last six interviews, we observed that the identified categories were saturated, and the newdata fitted in those categories.5.2 ResultsIn this section, we present the findings from interview data. We first define identity and accessmanagement to provide a foundation for the rest of the results. Then we organize the findingsaccording to the following themes emerged from the data: (1) motives for IAM, (2) stakeholderdiversity, (3) IAM life-cycle and its sub-themes, (4) audit in IAM, (5) transition from manual toautomatic provisioning, (6) coping with manual provisioning challenges. Each of the emergedthemes provide an answer to one or multiple research questions presented in the beginning ofthis chapter. Particularly, we list and discuss the challenges related to each theme.5.2.1 Definition of Identity and Access Management (IAM)Our participants provided a consistent definition for identity and access management. Theyusually divided IAM into three parts: (1) Identity management, which involves creation of anidentifier through an approval process, and: “making sure the identifier is created for the right96person, sponsored correctly” (— P1). Furthermore, it involves updating that identifier as theuser changes job in the company, and removing the identifier as the user leaves the company.(2) Access management, which “is building on the top of identity management and basicallygranting access to necessary systems and limiting the access to only what that person needs andnot giving anything more.” (— P11). Furthermore, participants mentioned that IAM should beauditable, and allow answering the question of who has access to what ?: “[IAM] provides fullauditing of any kind of events associated with this identity lifecycle.” (— P10).Participants explained that the IAM is an enterprise-wide system: “I define it as an enterprisewide system. It could cater to both internal or external users” (— P8). Most of our participantspreferred to exclude customers from the external users of IAM. When we asked (P1) if theyinclude customers in IAM he replied:“We keep them completely separate, and the reason is that we don’t want externalidentifiers and internal identifiers to mix. and we try to keep a very clear separationbetween externally facing systems and the access to them and the internally facingsystems and the access to them.”— P1A common theme among participants was the three motives for identity and access manage-ment. Participants considered cost reduction, efficiency, and security as the main motives:97“Broadly theres three motives. But what somebody will ask – whats their prioritydepends on where they are at. Big one is cost. They want to squeeze cost out of theirhelp desk, out of their security admin team. Another one is customer service or SLA,right. They want to turn the requests around more quickly. They want the request youwant to be easier. They want approvals to happen faster. They want fulfillment to be,you know, same day, same hour, not next week. And the last one and this is the one thatpeople market the most but its not necessarily the most prevalent is security, which issometimes labelled regulatory compliance or risk management or controls but they arejust synonyms for security. And thats usually driven by an audit failure or regulatorypressure. People dont usually care about security unless there is some external pressureto do so. And when there is external pressure they care a lot.”— P15Next, we elaborate the definitions of “identity”, “access privilege”, and “enterprise context”,which are the basis for understanding IAM.IdentityOur participants defined identity as an identifier. An identifier is created based on differentattributes of a person, such as name, employee number, location, and the relationship withthe company (i.e., internal or external). For example, (P1) described their policy for identitycreation: “identity is basically a user ID and a user ID is based upon the employee numberplus their initials.” Participants also used the terms identity and user interchangeably.While identity is usually associated with a person, some of our participants provided exam-ples of identities associated with an organization, or a group of people. For example, (P10)described that the external users of their IT system are not specific persons: “We have financialorganizations represented as users. These are users within the organizations who have theknowledge of this particular user account and act on behalf of this organization.” ( — P10).When we asked (P10) why they do not create identities for individual users in the external or-98ganization, she described that it is for the simplicity of implementation, and they plan to createindividual identities for the users at next phases of their IAM implementation.Access PrivilegesAccess privileges “are assignable security rights” (P15) and they guarantee a user’s accessto certain IT related artifacts. The artifacts can be an application, part of an application, anaction (read/write/delete) on a certain object in an application, ability of execute transactions,etc. (P12) explained that access privileges can also include access to assets: “mobile deviceis a resource. A pass card is a resource. Your office is a resource that is assigned to you.”.(P15) also explained that groups and roles can be considered access privileges. During theinterviews, we saw that participants used the terms access privilege, entitlement, permission,or resource interchangeably.Enterprise ContextAlmost all of our participants explained that identity and access management is particularlyimportant in large enterprises with a complex IT infrastructure. For example, (P1, P2, P8)’scompany had about 2,500 employees at the time of the study. Approximately 2,000 of themworked in the head office, and the rest in branch offices. Their processing environment includeda single IBM mainframe (z/OS), more than 200 Intel-based Microsoft Windows 2003 servers,and several UNIX (AIX) servers. The personal computers that were in use at both the headoffice and branch offices numbered around 3,000. Participants in other organizations also dealtwith such a complex IT infrastructure. For example, (P3) described: “we have about 600applications. So each of those counts as a system and each has its own owner.”Participants described the organizational context as extremely dynamic, in which people changejob, change location, and go through re-organizations, mergers, and acquisitions. Even duringthe course of our interviews, we saw that the lead of IAM changed three time (from P18 to P299to P8) in one of our visited sites. We saw similar dynamics in other sites as well. For example,(P6) explained:“I can tell you I’ve been in this role for 2-1/2 years and I’ve seen five departmentconsolidations in finance alone. And that’s not the count - finance is a differentdepartment for us.”— P65.2.2 StakeholdersA common theme in the data was stakeholder diversity in identity and access management.Participants identified the following stakeholders:User: Users were those for whom an identity was created, and access was provisioned. Userswere usually performed actions such as requesting access, and changing (or recovering)passwords.Manager: Managers were accountable for requesting, approving, reviewing, and revokingaccess from employees under their authority. For example, (P6) mentioned:“So the process for getting access at [Company Name] is you can request it, yourmanager must approve it.”— P6Application owner: Application owners were responsible for deciding who should access theapplications they owned. (P2) clarified the role of application owners in the IAM activi-ties:“ Any business related data is managed by a single person in that area. So for exampleall our financial information; an executive in the finance is responsible for the usage ofthat data and what appropriate usage of that data should be.”— P2100He also added that application owners can delegate their responsibility to another personwith a technical knowledge of the application:“Obviously that’s not a job one single person at an executive level would be able tohandle, so they have the ability to delegate some of that responsibility to what we call,data stewards; and those usually are the system analysts in those areas that have a betterunderstanding of that data, what it should be used for, what it shouldn’t be used for.”— P2Security Team: For most organization, security team was responsible for setting the accesspolicy, and overseeing other stakeholders:“So anything policy related actually lives right now with information security [...]Anything to do with access administration in the context of a, I guess centrally managedinformation system be it RACF or active directory goes through our securityadministration group; any access to LAN resources - files, folders on various serversthroughout the organization - is actually decentralized to the various departments.”— P25.2.3 The Basic IAM Life CycleWhile the details of IAM life cycle was different from organization to organization, we ob-served the following theme for IAM life-cycle (Figure 5.1):1. An identity is created for a new employee2. A basic set of access privileges, usually common to a large number of employees is givento the user (e.g., email, Internet, LAN) automatically3. User is given a set of job specific access privileges manually4. The employee changes job in the organization101Employee Arrival ID CreationAutomatic ProvisioningInitial Manual ProvisioningEmployment or Organizational ChangeChange of AccessEmployee leavesID RemovalManual ProvisioningFigure 5.1: The overall IAM life-cycle5. User’s access privileges changes as a result of the job change6. When employee leaves, the access is revoked, and the identity is discardedThis theme suggests that IAM involves four major phases: ID creation, automatic provisioning,manual provisioning, and ID deletion. ID creation and deletion fall under the identity manage-ment definition. Automatic and manual provisioning are part of the access management. Inthe subsequent sections, we provide a detailed description of each phase, the challenges ourparticipants faced, and the practices they adopted to deal with the challenges.5.2.4 ID CreationID creation was one of the themes emerged from the data. Participants identified ID creation asan essential but challenging IAM activity. Most of our participants described that the identitycreation started from the Human Resources (HR) system. When a new employee was hired bythe company, its information was populated in the HR database. The IAM system waited untilthe HR information was complete, and then it received a feed from the HR system containingthe employee’s information:102“So human resources will get a new hire - whether it’s an employee or a non-employeecontract. They put all that information into the HR system. It goes through that HRprocess. The end result - and whatever they do, it’s sort of black boxed to me. Right?Because the person may never start, they may, you know - they may not - thebackground checks may not come out - or it’s just a black box. The end result is theywill come up with a finalized start date. There is a feed from that HR system nightlythat comes over and says here are the people that we have hired and they are confirmedstarts. That’s feed to our system”— P1(P7) explained that it is important to use only one system as the source for creation of identities.In his company such a source was the HR system:“Unsuccessful shops will flounder, and the reason why they are unsuccessful is thatunless you are getting your human resources from one source - in our case it’s ourhuman resources ERP, you will have people going around the process all the time.”— P7But using a single source of identity was not always possible. For example, in health-care do-main, users such as clinician were not considered an employee, and therefore, their informationwas not kept in an HR system:“ There is a whole class of individuals like clinicians who are regulated providers comeinto the hospital and they have a different process you know they are not hospitalemployees typically they have their own practice somewhere. But they have hospitalprivileges in terms of coming in and using operating rooms and everything else.”— P9ID Creation GoalsAs we show in Section 5.2.1, the interview data suggests three main goals for identity creationactivity: the validity of the created identities (i.e., security), efficiency, and cost reduction.103Participants explained that efficiency is particularly important for two main reasons:Cost reduction: Those new employees who are not productive due to lack of access are con-sidered a cost for the business. Therefore, timely access provisioning would be finan-cially beneficial for the company:“So in this case we do hear feedback. One of the costs that we are trying to optimize iswe might have consultants who start right away. We don’t want to pay them to sitaround and wait for them to be provisioned.”— P7Employee retention: Timely provisioning also shows employees that the company valuestheir work, and wants them to be productive as soon as possible:“How many times do you show up the first day of work they don’t have a phone for you,they don’t have a computer for you; they don’t have an identity for you. What are yousupposed to do? Are you supposed to sit around, be the new guy, gill in some formswith pen and paper and take a coffee mug and what else? Take your pads and your officesupplies and wait for everything else to happen. That’s just ridiculous and embarrassing- it doesn’t say hey, I’m a valued employee, I’m expected to achieve results here.”— P7ID Creation ChallengesUsing an HR system removed the burden of identity creation from the security team, andprovided a single source of authority for employee data. But our interviews suggest that usingHR introduces a set of challenges:Incomplete Meta-data: The data from the HR system was not specifically collected to createidentities, and provide those identities with access. This led to lack of required meta-datafor making access related decisions:104“At the same while we are sending out SAP and other technologies that are role based,we understood that the HR systems could not have all the meta data necessary to makeautomated decisions about who should be an extension in what roles.”— P7Data sensitivity: HR database contained sensitive employee information that should not beexposed to other systems (e.g., social security number, salary, benefits). For example,(P3) described HR data as “purely gold master”, and explained that they built a systemto extract non-sensitive data from HR for IAM purpose:“It’s an extract from the HR data base which is all the non-sensitized employee data.Anything that is not really sensitive; like the employee name, their manager maybe;their phone number. But nothing about the person themselves - no social securitynumber, no home address, no salary.”— P3Efficiency: Participants highlighted that the employee information was entered in the HR sys-tem after it was fully collected, all the legal requirements (such as background checks)were satisfied, and all the necessary paper-work was completed. This process was timeconsuming, and usually went beyond the employee’s first day at work. Consequently, theIAM system did not receive employee information in a timely fashion, and the employeecould not be provisioned until his HR data is complete:“we move forward and there’s so much shift now in decentralized work force and noteverybody shows up day one at this nice tidy little office where everything is donetogether, right? It’s – we’ve got an oil and gas company where they are talking aboutpeople in the field that they never even meet ! They never even meet these people ! Soimagine if you could accomplish this through this process in that regard because thesepeople are mailing papers in or they are forced to come somewhere just to be in personto accomplish some of this set-up.”—P12(P12) then explained one of their customers started identity creation from an applicant105tracking and recruiting software, and the applicant itself provided the initial identitydata. (P12) further clarified that the process of ID creation should happen before HRdata entry:“you step back and ask the question, in an organization, where does your identity getcreated? And most identity implementations today would say, oh, it’s after – afteryou’re created in HR. But that’s a gross fallacy because how did all your data get there?The business process happens way in advance of that.”— P12(P2) reported a similar issue in his organization. The unionized environment of the com-pany led to delays in processing employee information in the HR system, and eventuallyidentity creation:“we’re a unionized environment, when they actually get assigned an employee numberis a critical time because it feeds into their union seniority. So it ties into when the job’sactually accepted by the individual and there’s some rules and regulations around thatand it sometimes actually delay process for that person. So they can actually be hereand have accepted the job, but they haven’t actually been processed per-se by HR”— P2External employees: (P1) described that the company’s workforce was not only limited tointernal employees, but also people from external organizations who need access tocompany’s resources. When the HR system was used for identity creation, the exter-nal employee’s information was entered in the HR system. But those external employeeswere not paid by the company, and were not belonged to the company’s HR system:106“they’re going to have to use our HR system, at least for registering some superficialinformation about their employees. so we don’t want to know about the pay scale andall that kind of stuff but they’ll have to be using our HR system because our HR systemprovides the basic identifier so they’re going to have to go through our HR system tohave that identifier generated. [...] they don’t follow our HR policies and they don’tfollow our security policies. it’s a weird relationship.”— P1This issue became more complicated when an identity should be created for a group ofpeople that represent a whole organization (as we discussed in Section 5.2.1). In thiscase, an entry should be created in the HR system that does not represent a real person.Data quality: The HR data directly impacted how the identities were created, and what accessprivileges were assigned to the identities. Therefore, the quality of HR data was criticalfor the success of the IAM. But the HR employees did not possess computer securityknowledge, and they were not aware of the consequences of their errors in data entry. Forexample, (P7) talked about “a real pivotal step in maturity”, as they generated awarenessin their HR department about the importance of HR data quality:“Now they understand the ramifications of setting certain key pieces of data into thehuman resources system is tied directly to types of access people have in the IT systems.So that, building that level of syncopation between us and that organization.”— P75.2.5 Automatic ProvisioningAfter the identity was created for the user, a set of access privileges should be linked to theidentity. Our participants named this activity provisioning. To create the association, accountsare created on the end-points (applications). One participant used the key ring metaphor todescribe the relationship between identity and accounts: “[Identity would be] sort of like the107key ring and all of the little keys upon it would be the accounts.” (P4). After account creation,the fine-grained permissions could be added to the account. Therefore, access provisioning isthe process of associating a user’s identity with accounts, and accounts with permissions. Amajor theme in the data was that provisioning can be done automatically or manually.We define automatic provisioning as the automated provisioning of access privileges based ona user’s attributes. When we asked participants what constitute user attributes, (P4) described:“It’s very simple, it’s based on the type of employee they are. Their are regions and in acouple of cases there’s more to it, it might be based on a department code, or a title.Right now it’s really pretty simple.”— P4We observed that the type of access users got as a result of automatic provisioning was usuallyhigh-level, and common to many employees:“But we have an idea of a primary type of account so the whole infrastructure of[company name] - you have an Active Directory domain and it is one domain so there isone account. That gives you what they call basic access. That’s access to the network,access to remote in to the network, remote access and access to the intranet. Soessentially you could log in to your workstation on day one that you are hired and youwill have this basic level of access. Internet - remote access - VPN, you get an exchangeaccount mailbox so you could transfer mail. Pretty much, that’s about it.”— P1To achieve automatic provisioning, two types of mechanisms were used: identity policies thatdefined how users were assigned to roles, and roles that determined the access of users whowere assigned to them.108Identity PoliciesSome organizations used the notion of identity policies to describe the rules using which auto-matic provisioning was done. Identity policies were usually in the form of conditional state-ments. If a user’s attributes matches the antecedent of the conditional statement, the user willbe assigned to one or a set of roles:“So when a user is created by [the IAM software], [the IAM software] has certaincustom logic built in - well built in - I put it there. But it will look as it creates the userat the attributes and if it matches a certain policy such as [employee type] A, C, or S andregion equals North America, then give them this role.”— P4One of the participants (P4) showed us the set of identity policies used in their company. Thepolicies were designed to assign users to a subset of 20 to 25 roles, based on two types ofattributes: the employee type (e.g., full-time, part-time, contractor) and the employee work lo-cation (e.g., North America, Europe). While they used very few attributes to assign roles, theiridentity policies were quite complex, and hard to understand. For example, even the partici-pant, who wrote the identity policy herself, could not interpret the policy fully, as it containedcryptic codes. We asked the participant to describe the meaning of different attribute values inthe policy description, but she could only recall one out of five values in the policy. Further-more, the policy contained two different attribute values for the same purpose. For example,when we asked the participant what the employee status “AC” means, she said: “Active - or A- A also stands for Active.” (— P4) Then we asked about the difference between “A” and “AC”,she replied: “Nothing it’s just that we used to use AC and now we use A but I’ve left it in therejust on the odd chance that there’s somebody still around with an AC.” (— P4)109RolesWe asked our participants to define role-based access control. The high level definition wascommon between participants. For example, (P4) defined role-based access control as:“Based upon certain user attributes and this is in general for [IAM software], you wouldassign users roles and then these roles are used to grant them some kind of access.”— P4According to this definition, users are assigned to roles based on a set of user attributes. Inother words, “You do not pick a role. A role picks you.” (P12). Consequently, these roles wereused to assign one or a set of access privileges to users. We usually observed that roles have aconceptual meaning, i.e., they represent a concept outside of the IAM system. This conceptualmeaning usually matched certain user attributes such as a location, a job, or an application. Forexample, (P8) described that one role was used to create accounts on one large application (i.e.,abstracted an application concept). She also added that they use roles that abstract the conceptof departments in the company:“we have about 200 roles and that consists of base roles, and one application role, wellour one application is like a huge application that I would say most of our users in thisorganization will touch so it covers a lot of user base and also three or four divisionroles or department roles.”— P8(P12) further clarified that roles can have various conceptual meanings, and they can map to auser type, a job function, or a location:110“We use different role types as well. So for example, we use something called a ‘usertype role’ which maps to more of the entity type that I am. So for example, I am a fulltime employee. I’m a part time employee. I’m a contractor. I’m a consultant. Asignificant amount of access will map to those correlated attributes. Okay? And then wehave what we call ‘BU roles’ which align to your job function. So whether it be – againa cost center job code, whatever the case may be, it aligns to some job function. So ifI’m an accounts payable clerk with job code 1234567 or 8 I get this access. And thenwe have what we call ‘IT roles’ that are more siloed based on some of their attributeslike a location. Everybody at the [City Name] location gets in the [City Name]distribution group. Everybody – you know, whatever those attributes may be.”— P12(P6) talked about a set of simple roles that are not necessarily have any conceptual meaning,but they group a set of access privileges. He called these simple roles ingredients of buildingaccess for users based on their job:“So what we do is we have an ingredient list of roles, what I mentioned before - let’s sayabout a 115 - we call these simple roles. These are the task based roles. They are broken- some of them have as little as one T-code. Some have as much as let’s say 15 volume.”— P6In contrast to simple roles, (P12) talked about the concept of enterprise roles, which are acollection of simple roles, and assets:“But we also use the term enterprise role or profile which is a broader aggregation ofaccess and assets that users with common requirements share. So, for example, I am abranch teller and I may get the teller role in application A and the teller admin role inapplication B and a pass card and a branch key and a whole set of things that are grantedto me because of my enterprise role.”— P12As a summary, we saw that roles can have different meanings. They can be as narrow as a111single access privilege, or as large as multiple roles and assets. We saw that users were mainlyassigned to roles automatically using identity policies, but we also saw that administrators canmanually assign users to roles. In the context of this thesis, we define roles as a group of accessprivileges that are used for automatic provisioning. If a role is used for manual provisioning,then we call it an access privilege.5.2.6 Manual ProvisioningAfter the initial automatic provisioning, users will be provisioned with job specific access usingthe manual provisioning process. While the automatic provisioning assigned users to roleswithout human involvement, manual provisioning was a labor intensive activity that involvedmultiple stakeholders:“Anything that’s non-basic access, which is anything else - that’s what we are focusingon now; because that is terribly manual today - still. It involves somebody figuring outwhat they need access to, creating a ticket in Service Desk, and waiting for someoneelse to figure out which systems they are referring to and then going in and provisioningthat access. And there is some document ultimately about what - who is the owner ofthe system, who is the access approver of the system. Here is the method that you giveaccess to.”— P3Figure 5.2 shows the most common interaction pattern we saw between different stakehold-ers involved in the manual provisioning activity. The activity was initiated by a manager whoidentified a set of systems and assets that his new employee needed. In some cases, the em-ployee itself requested access, and the request was reviewed and approved by the manager.We saw that in different organizations, different communication channels were used to sendaccess requests, such as an electronic form, email, a ticketing system, or an IAM system. Therequest was usually made to either the service desk, or the security group. After receiving therequest, a member of the security group (or service desk) determined the desired systems and112Manager /Employee Security / Service DeskApplication OwnerApplication OwnerApplication OwnerSecurity / Service DeskFigure 5.2: The interactions between stakeholders involved in the manual provisioningactivitydeployed the request to the application owners of those systems. The data might be distributed,and thereby owned by several application owners. In this case, the security administrator (orservice desk) made multiple requests. After receiving the request, the application owner (ora delegate with technical knowledge of the application) decided whether the requested accessis appropriate or not, and notified the security administrator (or service desk) about the deci-sion. The security administrator (or service desk) implemented the request upon applicationowner’s approval. If the security administrator (or service desk) did not receive a response, heperformed a follow-up cycle, to handle non-response or lag from application owners.When we discussed the details of manual provisioning with participants, we identified thatthe activity was inefficient, and error prone, and it involved communication and collaborationcomplexity. We found the following reasons for the observed challenges:Managers’ lack of knowledge of what to request: As we discussed before, managers usedgeneralized communication channels to specify their request, but they usually did notknow what to ask. In some companies, the security group provided a detailed multi-page form for managers to identify resources and request access. The managers wereoverwhelmed by the amount of information that they were expected to fill in (P18). This113led to inefficiencies, and errors in the process.Mirroring: Frequently, managers would ask the security group to make a new employee’saccess the same as some another current employee’s access:“A manager who hired a new employee who knew that you had the access that youneeded to do the job for him or her would say, ‘Oh, make this new employee’s accessjust like yours.’ And so then an employee would then inherit very high levels ofprivileges and access based on the success of a previous employee in terms of doing thatjob.”— P1One of our participants (P6) called this practice mirroring, and explained it generallyleads to errors. Some of our participants such as (P1) and (P6) disagreed with mirroringpractice, and even (P1) suggested that they disallowed this kind of generalized request asit could provide more access than required for the employee. Conversely, (P14) explainedthat they use this practice in their organization frequently. But (P14) also suggested thatthey do audits regularly, and make sure the mirrored user does not have unnecessaryaccess privileges.In cases where mirroring was disallowed, those managers who did not know what to askfor then tried to work around the security requirement by asking for the list of accessespossessed by a current employee, and requesting access for every item on that list (P1).Lack of common terminology: In some cases, there was no common terminology throughoutthe organizations for requesting access, resulting in communication and collaborationcomplexity. When an access request was received, it was difficult for the security group(or service desk) to identify which systems in which divisions were requested:114“[...] but it’s terminology we don’t know. We say what area is this in, is this in theassessment part, is this prevention, is this the claims? - sometimes the user has no idea,they just say I don’t know I just have to get access to it. So we end up going to thesecurity coordinators for all those divisions saying are you familiar with this app? Orwhatever they are talking about. Oh yeah, this is actually this. Aha. Then we know, andthen we can tell if the request has come to us in a form or an e-mail and it’s approved,we can set up the access or send it to the area responsible for setting up the access.”— P18Also when managers were requesting access using the access profile of a similar user,they did not necessarily understand the esoteric access rules and system names – thesehad to be translated into language that the managers could easily understand:“[...] and we get this huge profile - here’s all the access the user has. We then have totranslate that into more of an English format for the individual, saying this means this,this means this, and this means this. And then, they have to put that into a form of whatthey want.”— P18Communication with application owners: When the security team (or service desk) pro-cessed the request and deployed it to various application owners, it was often difficultto identify the correct owner. Also, as mentioned, some owners would not respond in atimely fashion, or even not respond at all, making provisioning impossible. The securityadministrator could end up implementing the requested access in a piecemeal fashion,thereby decreasing employee productivity as the employee would then have to wait foraccess. This led to communication and collaboration complexity, and inefficiency:115“[...] and so the security administrators would then send notes, e-mail, to the dataguardians saying, ‘Are you OK if we grant Bob or Jill access to the system?’ And thenthere would be that sort of follow-up cycle where the data guardian would ignore themor not respond or say I’m not the data guardian or that kind of crap. So there wouldoften be a delay in terms of getting approval... so systems access to the employee wouldthen build over time as individual data guardians would respond”— P1Lack of context in requests: The communication between stakeholders happened using gen-eralized request management tools, such as email or a ticketing system. Therefore, it wasimpossible to enforce rules such as separation of duties during request process, whicheventually led to inefficiencies and errors:“A ticketing system like, you know, BMC Remedy or HPO Service Desk, they’ll dorequest management but they are not compliance or identity tools. They are just dumbrequest forms for the most part where it doesn’t know what you already have access toor what any of the compliance policies are”— P12Additionally, generalized request management tools were providing too many choicesfor users without really thinking about what the user can theoretically have access to,further contributing to the inefficiency, and errors in the process:“Traditionally [companies] just put together a big pick list of everything under the sunthat you can ask for and then hope that it’s going through some approval process andthen relaying on some periodic attestation to catch things that are inappropriate.”— P12Understanding Access PrivilegesOur participants adopted a set of practices to alleviate some of the aforementioned challenges.(P3) suggested that they documented the process of requesting access to each application, to116create a common terminology, and make the identification of application owners easier:“So our access procedures state that every application that has any level of criticality issupposed to have a published knowledge-base document in our service desk that defineswhat the application is, who owns it, who is the technical owner of the words — abusiness person may own it but someone in a tech area, you know IT has to actually dostuff for the business. There is a tech owner - right — who is the approver or reviewer insome cases they call them BAR, business access reviewer. Sometimes the BAR is theapprover, sometimes there is a separate approver like the BAR’s VP or something. Andthen it contains something like what service desk group you should open a ticket in andother types of identification information.”— P3Similarly, (P12) talked about a resource catalog that includes all the policies related to theaccess privileges of an application:“Whether they be SoD policies that say you can’t have A if you have B or what we call‘restricted access’ policies that say you can’t have entitlement X if you are not costcenter Y or division X or whatever the rule is, the ability to define that rule it lives withthe entitlement in the resource catalog. So as the resource owner we have the resourcecatalog of all requestable resources and entitlements. And the rules about them belongwith those in the resource catalog.”— P3(P7) talked about their plans to eliminate the generalized access request forms, and to provideusers with an “access catalog”. Such a catalog will show all the requestable access privilegesfor an employee, and then automatically triggers the approval process, when the user requestsan access privilege:117“[The new technology we are planning to deploy, will be] allowing people to say hey, Ineed special limited access for twitter - I am in marketing and I need to twit, and I needto consume twits. I am going to go ahead - here it is in the catalog - here’s where Irequest it - the request goes in, the work flow now says oh, for this type of access I needSUP this person reports to approve it. I need to check their - I need someone inoperations to check their educational requirements to make sure they have done theirsecurity training so they know how to use these tools effectively and safely. Then wecan go ahead and wrap this because people actually grant entitlements. Or, route it tothe system that automatically grants the entitlement once those conditions are satisfied.What we are doing here is we are combining human work flow with automated workflow.”— P7Similarly, (P12) suggested that a set of proactive control should be put in place to prevent usersfrom initiating requests that violate existing policies or rules:“we also applied what we call proactive controls. So our whole kind of philosophy isthat before I even request something we’re going to apply compliance policies like SoDor other restrictive policies like if I’m not in Division X, I can’t even request access toSystem Y. So we take all of those policy controls and put them right in part of therequest process”— P125.2.7 Change ProcessAs we discussed in Section 5.2.1, the enterprise context is extremely dynamic. When userschange responsibilities, jobs, or departments, their access needs to change as well. A commontheme was that the change process involved both automatic and manual provisioning. Whenan employee changed job, its user id was assigned to appropriate roles through automatic pro-visioning, and revoked from the roles related to the previous job. Additionally, the employee’s118new manager requested additional access through manual provisioning. Therefore, the changeprocess faced similar challenges to manual provisioning. Despite this similarity, the removalof access from the last job made change more difficult than the initial provisioning:“So the critical difference and you know we always talk about change is the hardestidentity process because on-boarding, you’re just adding everything you can think ofand at off-boarding you are taking away everything you can figure out. Change is thehardest because you’ve not only got to figure out delta – so in any given job changebecause of the enterprise roles, there’s lots I’m probably keeping. Like I’m probably notgoing to whack my LAN ID because I’m transferring departments but here is someaccess that will change. So there’s the delta but there’s also the timing of this changebecause job changes take effect on effective dates and there’s overlaps required withpre-existing access so change is by far the hardest business process.”— P12(P1) defined privilege accumulation as keeping the access privileges from the past job duringchange process. In P1’s company, people frequently accumulated access privileges as theychanged jobs:“Historically here if you were an individual who started at [The Company Name] in1930 (I’m exaggerating) by the time you retired and had 40 years you would haveaccess to every single system that you had ever used in your entire lifetime with [TheCompany Name].”— P1Participants explained why privilege accumulation is a hard problem to address. First, usersusually went through the manual provisioning during their previous job. Revoking those priv-ileges later was a task similar to manual provisioning, and shared similar challenges. (P2)discussed that the automated de-provisioning during change process is their ultimate goal, asthey currently use human process, and it is error-prone:119“So having a system in place that provided that compliance and that checkpoint thatwhen somebody moved jobs it would remove the old entitlements and add the new oneswith a guarantee. Any time there’s human involvement there’s ... it’s prone to error.”— P2Second, there was a lack of accountability from the managers to ask for removal of accessfrom the employees: “if a manager fails to advise sec admin to cancel access, there were noconsequences to the manager.” (— P1)Third, (P19) talked about the sense of entitlement that some employees had about their pastaccess privileges. He explained that lack of security awareness led to employees demanding tokeep access from their previous jobs:“So some people view that as an infringement upon their union rights. You can’t takethings away from me. I have seniority. You can add to me, but you cannot take awayfrom me. They don’t understand like the security concept of you’re doing this job now,you’re not doing this job, you don’t need that access anymore”— P19To cope with these challenges, some of the companies had an access governance team in placethat monitored the changes, and made sure users have the correct access:“Now what happens is that we have a report that runs every single day and it tells mepeople transport or change. So if she - what happens is the trigger would be one of twothings: she gets a promotion. She went from warehouse manager to public relationsmanager. She will request something. I need a public relations manager role. My teamgoes automatically - why? That’s not what you are. You are warehouse. No, I got apromotion, I’m this. Okay, we’ll give you these three but you are losing those three.”— P6While using automated provisioning could address the challenges with errors, and accountabil-ity, it made handling exceptional cases difficult. For example, (P6) explained that in certaincases, employees requested a transitioning period, and asked for keeping access from their120previous job for a period of time, making automated change process ineffective:“We send a request to the manager that says [Employee Name] has changed fromposition A to position B. They are requesting position B roles. We are going to removehis position A roles. Do you agree with that? And if the manager says no, he is trainingthis person as replacement for three months. Okay fine, we will allow them to have it;we send a notice to security and security will send - will set those three roles to expire inthree months.”— P65.2.8 Identity RemovalAnother identified theme in our data was ID removal. ID removal or leaver process happenedwhen an employee left the organization:“As soon as your employment gets inactive in your company’s IT system, your identityshould be automatically transferred in a locked state. All your user accounts andentitlement should be locked that you would not be able to use it.”— P17Similar to identity creation, identity removal was usually triggered by the HR system. TheIAM system received a notification from HR about the termination of the employee, and itsuspended all of the user’s accounts:“On the flip side, if somebody leaves - is terminated - a reverse process happens. HRenters the term and they send us the action saying this is a confirmed term - we backprocess and it goes and it will then remove those accounts. Well it removes the activedirectory account and it will disable the actual mailbox.”— P3(P1) described that they used to do the identity removal manually, and based on notificationsfrom managers. But they found this process to be ineffective, as managers were not accountable121for reporting the terminated employees:“It used to be that managers are responsible for filling out a form and advisingsec-admin that employees were terminated. and this was almost universally never doneand the reason it was never done was that although it was the manager’s responsibility,back into that operational thing again right? which says, ‘I know that that’s the rules butI’m not going to do it because I don’t need to.’ there’s no accountability, if a managerfails to advise sec admin to cancel access, there were no consequences to the we were finding that many employees would depart [The Company Name] and yettheir IDs would still exist. and the access those IDs still was active.”— P1(P1) explained that they changed the identity removal procedure by getting notifications fromthe HR system, and removing identities automatically. But he also found the automated methodineffective in exceptional cases:“[Status changes from HR are] not universally 100% effective because what can happenis employees can not quite depart. Meaning that you can go on severance, which meansthat you have a year, 2 years worth of severance and you haven’t departed. [...] Youaren’t in the building. It fails sometimes. Or you go on a leave of absence and then,again, you haven’t quit departed. So you could have somebody who, say, goes on aleave of absence for a year, two years, or whatever. Sick leave, LTD; that could go oneforever and then you never know officially whether that employee’s still an employee of[the name of the organization]. So status changes weren’t 100% reliable”—P1To address the issue with those exceptional cases, (P1) explained that they“periodically run one query job and say, ‘identify the IDs that have not been logged intofor the last six months, some period of time.’ and then we follow up saying, ‘what’sgoing on?”’—P1122As we described above, the ID removal faced challenges similar to ID creation phase, such asefficiency, data quality, etc. But the lack of efficiency was particularly a critical issue for ID re-moval. (P17) explained that the lack of timely ID creation causes inefficiencies in productivity,but lack of timely ID removal has security consequences:“From the efficiency perspective it’s the joiner process which is critical. From thecompliance or from the audit stance, from the control standpoint it’s typically the leaverprocess which is critical.”— P17(P7) further explained that terminating an employee in the HR system might not seem to beurgent, but revoking access from the terminated employees should be done as soon as possible.This led to some companies do ID removal manually:“It’s a 24/7 business and right now there is emergency - if we have a situation where anemployee is terminated for whatever reason and we want to revoke their access rightaway, there are some manual processes that we use today to make sure we get that stuffout of the system in a timely fashion. And then the automate system goes in and does thecontrol level function of removing their identities and unhooking their access and such.”— P75.2.9 Audit and AccountabilityIn the preceding sections, we show the basic IAM process and elaborated its activities. Further-more, we show that the process involves human, and it is prone to human errors. As a commontheme, IAM stakeholders perform audit activities on the access policies to detect errors. Wesaw three types of audit during our interviews:Policy Review: In Section 5.2.5, we talked about the use of identity policies, and roles inautomatic provisioning. The policies and roles were usually created by the security teamin collaboration with the business. But those policies could become outdated, or contain123errors. In some cases, companies reviewed the policies and roles, and checked theirvalidity:“So again in our world we look at it and say – we use this terminology called amembership criteria that governs the pre-approved access that you get. So again if I’min cost centre A, B or C and my job code is 1, 2 or 3, I get the teller enterprise role. Andthe teller enterprise role gives me a LAN ID, an SAP account, a mainframe account anda pass card, right? So I’m just as interested from a compliance and auditing perspectiveto have an auditor sign off on the membership criteria by which that pre-approvedaccess is granted. So that’s another piece of the certification puzzle that’s, I think, quiteimportant.”— P12Since the scale of roles and identity policies was not large, we did not see participants’emphasis on any challenges in policy review.Access Review: To check the validity of the access given through the manual provisioning,companies reviewed the access privileges of employees. They called this activity accessreview, access certification, access recertification, or access certification and attesta-tion. In the context of this thesis, we call the activity access review. The access reviewwas usually done by a manager reviewing employees under his authority. We found thataccess review involves challenges such as the large scale of review, lack of technicalknowledge by managers, high frequency of reviews, human errors, and difficulty in han-dling exceptions in users’ access. We discuss the details of access review in Chapter 6.Access Log Review: One of our participants, who worked in health care domain, explainedthat the access policy in health care is not as strict as enterprise domain. They allowedusers to have access to a wide range of resources, but they collected and reviewed exten-sive audit logs of who accessed the data:124“So the whole access model in health care tends to be, you let people do what they needto do to get the job done. They may not be like why did you look at so and so’s recordit’s not because I’m not trying to predict whether or not you have a relationship andshould have access to that . Because that is a really tough problem. I let you do it but Iaudit a crap out of the system so if somebody complains or someone reports that I sawsomebody accessed something and I don’t think it is appropriate then you’ve got areally robust audit records, and you can call them to task and you involve professionalbodies and all the sudden you know you’re disciplined or suspended or whatever.”— P9Such a model was later confirmed by (P15) and (P16) who had experience in implement-ing IAM in health care domain. But in an enterprise context, keeping audit logs wouldbe a performance overhead, and was not done regularly in organizations:“the reality is that the audit log creation can impact system performance so thereforeaudit logs are not reliably created. Application developers do not capture therequirement from a security point of view to create audit logs. So not all systems createaudit logs.”— P15.3 From Manual to Automatic ProvisioningAmong all of the activities we discussed before, we identified the manual provisioning asthe most challenging. It was a manual, labour intensive process, required communicationand collaboration between multiple stakeholders, was prone to human errors, and it requiredintensive audits.As a common theme, almost all of the participants expressed their desire to automate the pro-visioning process and eliminate manual provisioning challenges. Automated provisioning wasone of the main drivers for those companies adopting an identity and access management sys-125tem, as it led to efficiency and compliance. For example, (P2) who was managing the adoptionof an IAM system explained that they are planning to automate the provisioning by adopting arole-based access control approach:“And so part of the identity management project is to go through a role-based accesscontrol and role mining exercise to define roles for the different business areas whereyou’ve got a defined, concrete, set of privileges somebody needs to do that specific job.So at that point when somebody moves into a new job you know exactly which accessthey require and what access their old job had and could be removed and that’s all donethrough an automated system rather than a person actually going in there having to doall that stuff.”— P2But surprisingly, none of our participants had success with automating the entire provisioningprocess. For example, a year after we talked to (P2), we visited (P8) who replaced (P2) inmanaging IAM in the company. She explained to us that they still have not used role-basedaccess control throughout their organization.This led us to choose access provisioning as the core category during selective coding process.It was a high level category (theme) that included three sub-categories (sub-themes): automaticprovisioning, manual provisioning, and transition from manual to automatic provisioning. Webuilt our theory around the access provisioning concept, and identified the relationship betweenother categories and our core category. This helped us to answer the following questions:• Why is provisioning challenging ?• How do companies deal with the challenges ?• How can organizations automate the provisioning ?• How do other IAM activities contribute to provisioning ?126Further data collection and analysis revealed a set of challenges in both automatic and manualprovisioning. In the next sections we discuss these challenges.5.3.1 Coping with Manual Provisioning ChallengesIn Section 5.2.6, we identified three main challenges in manual provisioning: (1) inefficiencies,(2) errors, (3) and communication and collaboration complexity. A common theme was to copewith the manual provisioning challenges in three different ways:• We demonstrated in Section 5.2.6 that companies try to understand, and document accessprivileges, and improve the way the manual provisioning process operates. This practicepositively impacted all of the three challenges in access review, but did not eliminate anyof them.• We shown in Section 5.2.9 that companies perform access reviews to prevent, identify,and fix errors during the manual provisioning. On the other hand, access review did notaddress inefficiencies or communication and collaboration complexity.• Companies can transition from manual to automatic provisioning to eliminate all of thementioned challenges. This approach seems to be ideal, but transitioning required aset of activities including integration of IAM with end-points, role-mining, and role-engineering. These activities acted as barriers to the transition. Furthermore, when com-panies moved to automatic provisioning, a set of challenges forced them to abandon theapproach and go back to the manual provisioning. In the subsequent sections, we willexplain these issues in detail.1275.3.2 Integration With and Managing the End-pointsTransitioning from manual to automatic provisioning required an integration between an IAMsystem, and the end-points so that the IAM can manage the assignment of users and accessprivileges. But the integration was proved to be difficult due to a mix of technological andorganizational challenges:Unmanageable access privileges: In many cases, organizations owned applications that didnot expose external APIs for managing their internal access privileges. The problemwas not only a technological limitation, but in some cases, was policy related. Theorganization’s policy did not allow managing certain critical applications centrally, asthe application should be managed by its own dedicated administrators:“But again you know we find new applications, it doesn’t mean that I can get to theentitlement system. It still could be black boxed. For security reasons - so a treasuryapplication - maybe they don’t want people understanding how the internal securityworks or the entitlements work. Or, for example the systems that are hosted - for us it’sjust a web interface to a bank, treasury or whatever. There’s no way to get in there andthey are not going to let us in.”— P3Autonomy of applications: Some applications followed a set of universally standardized bestpractices, and therefore, the company preferred to manage them separately:“SAP is a little different - SAP was something that was added on top of all the existingprocesses and SAP comes to its own best practices and thing.”— P3Technological complexity: It was difficult to integrate some of the applications with the IAMsystem. After doing a cost-benefit analysis, organizations decided to manage those ap-plications manually:128“We have actually quite a number of applications that would allow us to actually createthe permissions and set the permissions for most of our applications and we haven’t hadany luck with them. As a matter of fact we’ve taken out of the creation simply becausethey have actually created more of a burden to us on the applications that we have triedthat would allow us to actually have sort of like the consolidated account creation andpermission settings.”— P145.3.3 Engineering and Mining RolesTo enable role based automated provisioning, a complete and correct set of roles needs tobe created. The creation of the roles was one of the difficult aspects of setting up the IAMsystem for most of our participants. Few of the companies that we talked to considered en-gineering roles. For example, in (P1, P2, and P8, P18, P19) organization, the security groupdecided to build roles by following both a top-down and a bottom-up role engineering ap-proach (see (Vaidya et al., 2007) for a discussion of role engineering). They started the roleengineering process by developing a set of roles from existing user-permission assignments,which was called role mining. This was a bottom-up approach that required mining exist-ing user-permission assignments in different access control repositories. (P1) highlighted theimportance of role-mining:“if you don’t know how to do discovery, if your tool can’t do discovery you’recommitting the staff to 2-3 years work of heavy lifting to do discovery. So a tool thatdid discovery and managed roles potentially can save you years of effort.”— P1The role mining engine in their IAM system could analyze different repositories in each ap-plication and find users with similar accesses (P18). Consequently, the security group collabo-rated with individuals from each business area to check the differences between those similaraccesses and created a single role that corresponded to a single job description (a top-down129approach). Additionally, the security group collaborated with application owners to determinewhich roles should be authorized to access each resource in the organization. About a yearfrom the time we talked to P1 and P2, we spoke to the new IAM leader in the company. Shedescribed that the role engineering project is still incomplete, and they only implemented rolesin few departments. She further explained that their plan is to extend role engineering processto other departments.We further explored why role engineering is so challenging to perform. The participants pro-vided three main challenges: business complexity, uniqueness of access, and lack of accuratedata.Business complexity: (P3) explained that they are reluctant to start a role engineering processas it should start from business, and business involves complexity, and constant changes:“Well it starts - it doesn’t at IT - if it starts at IT it’s going to fail. It starts in the businessand the business is a mess - not only our business, any business is a mess, right? Youhave to figure out what people are doing - whether it is right or wrong - or the best wayor not the best way. And then you have to manage the change. Ten years ago in [thecompany name] I could tell you that we used to reorganize about twice a year and thatcould mean your office numbers, back changes, your cost centre changes, you know allthese different things changed. Your titles - titles used to change very frequently.”— P3Uniqueness of access: (P12), who had years of consulting experience with various compa-nies, explained that the uniqueness of people’s access makes it hard to generate rolesautomatically, and therefore, part of the provisioning should still be done manually:130“We do identity consulting which led us down these paths so we’ve used the tools fromVIOU, Birdstream, Eurekify, which have all of course since been acquired by CA,Oracle, Sun, whatever, and they are interesting tools in the sense that you can collect thesource data and run it through some algorithms to try to detect patterns, but the catch 22that seems to get lost on people is if people’s access is even in the slightest way able tobe correlated, you wouldn’t have this mess. So our experience has been that those rolemining tools produce fairly predictable outcomes which are useful. Like, for example,everybody gets a LAN ID. Okay. Thanks a lot ! That’s really good ! I could have toldyou that !”— P12Lack of accurate data: (P2) Explained that the access control data for role-mining exercisemay not be accurate, as users may have more or fewer access privileges than what theirjob requires. Therefore, the output of role-mining should be evaluated and refined byconsulting to people who understand the business:“[The output of the role mining might be] appropriate or not appropriate. But a lot oftimes it’s interesting because you’ll identify those because if you’ve got a group of, youknow, 15 case managers for example and you bring them into the system it’ll say, ‘Ok,12 out the 15 have 80% of access in common and these two people only have 20%.’And at that point we would go with the business analysts and go through thoseentitlements and say, ‘oh this person has access they should not have that has beencarried over from somewhere else’.”— P2We further explored what is the proper way to tackle the role engineering problem. The partic-ipants suggested that the right way to engineer roles is to do it gradually, and over-time. (P12)explained that: “when we engage with a customer, we will embark on some role modeling butit’s not the be all, end all. We don’t try to bring it to even a – state of ultimate completion.”He further explained that the access management process is composed of role-based automaticprovisioning, and ad-hoc manual provisioning. The company should start with a small scale131role-based provisioning, and as they continue to perform manual provisioning, their under-standing of the meaning of access privileges, policies, and exceptions will grow. Then usingthe manual provisioning data, roles can be engineered for some of the areas in the company.5.3.4 Challenges in Automatic ProvisioningIn previous sections, we showed that transitioning from manual to automatic provisioning waschallenging. Even if those challenges were overcame, performing and maintaining automatedprovisioning was difficult as well.In Section 5.2.5, we explained that the roles are used to automatically provision users. (P15)explained that roles give organizations a leverage in provisioning:Defining a role lets you assign a bunch of entitlements at once and then lets you givethose entitlements a label that business users will understand. And it lets you in manycases automatically assign a bunch of entitlements based on identity attributes. All ofthat together is leverage.— P15We found that in two situations, the challenges of using roles outweigh their leverage. First,people’s access tend to be unique, and it is difficult to assign a large number of users to a role.Second, organizational change causes changes in roles and makes their maintenance difficult.Uniqueness of AccessWhile we show in Section 5.3.3 that uniqueness of access makes role-mining difficult, it alsonegatively impacts the leverage provided by roles. Roles would be effective only if there are alarge number of users with the same access requirements:132That leverage only works if the number of users that have the exact same requirementsis large. Otherwise you have this level of a direction between the user and theentitlements and you apply it to one guy or two guys, right. Useless. So RBAC is anefficient model where you have substantial populations of users with identical securityaccess requirements.— P15(P12) suggested it is impossible to implement role-based access control fully, as every users’access is unique, and hard to determine a priori:“It’s not realistic to think that 100% of your access is going to be automated andpre-approved because you’ll end up in a scenario where for every user you have anindividual role, which completely defeats the whole purpose. We prescribe a model of,if you can even get to 60 — 70% of your access automated, pre-approved through rolesthat would be tremendous and phenomenal.”— P12Some of our participants elaborated why people’s access is unique. (P6) explained that the jobtitle does not necessarily determine the access of a user (e.g., a business analyst in one depart-ment gets a completely different set of accesses than a business analyst in another department):“If you have uniform job titles throughout a company you can do [role based accessmanagement]. You can build all the roles and you can do this assignment automatically.If you don’t, then you wind up having a request - so you get a set of generic roles, youget ESS, MSS, SAP utilities - which are some basic simple things just to get in - like anetwork password. Then after that your job roles are done by request.”— P6Similarly, (P11) explained that roles usually define a set of functions for users to perform.In addition to that, the scope of objects that the user can apply those functions on should belimited and be different from one user to another:133“That is correct. So defining those roles, it can get quite complex. Because, you are notjust defining what tasks they are capable of doing, you are defining the scope of whatthey are capable of doing that on.”(P6) gave an illustrative example of why limiting the scope of the access is important. Theircompany designed a set of roles for provisioning users in North America. But they soon re-alized that those roles would not fit in Europe, Middle East, and Africa. When we asked theparticipant why you cannot use the same roles when you are expanding your provisioning toother parts of the world, he explained:“So the difference is at a very high level you can take a role, give it to a user and youcould use this logic: if you could make journal entry in New York, you could make ajournal entry in Tokyo and you could make one in Berlin. It’s the same principle.However: How do I restrict you from making that same journal entry when you aresitting in Berlin to do it in New York’s journal?”— P6Constant Changes in the BusinessAnother barrier to use roles is constant changes in the business, which leads to changes inthe responsibilities, departments, and applications. (P6) explained that the main concern withfunctional roles (roles that are mapped to job functions) is the constant changes in the respon-sibilities of employees, which would lead to constant changes to the structure of the roles:134“They have something called composite roles. Which is the amalgam of these alreadybuilt-in. But based on our principle where we do not have this job based level, havingthese becomes a problem because you start winding up having multiple versions ofthem. So if you have 115 roles that translates to lets say 50 functional roles. [...] Nowone person and this is assigned to five people each per composite role. One of thesepeople wants a change to this composite role. Well that change now affects five people.And if I can’t do that and I say well there’s only one person wants it, I have to createanother composite role. [...] So when you see all those changes happening, thosecomposite roles hurt you. Because then you have to keep generating them over and overagain. And there are maintenance problems because they affect large numbers ofpeople. ”— P6(P6) also informed us they are planning to re-evaluate the use of functional roles in the future,as the rate of change in the company may reduce, and the organization may become morestable. His motive for that was the drawbacks of simple roles, which were assigned manually:“Because you have a large number of people doing it. You could have 500 of [simpleroles] and every day you are changing them. It’s a maintenance nightmare. You have alot of people that are changing and moving them and looking at them.”— P65.4 An Integrated Model of IAM and its ChallengesIn this section, we present the abstract model we developed using grounded theory (Figure 5.3).The model explains the relationship between different IAM activities, and the challenges weobserved.The ultimate goal of identity and access management is granting users access to necessarysystems and limiting the access to only what that user needs. Manual and advance provisioning135play a pivotal role to achieve this goal. Therefore, the centerpiece of our IAM model is thosetwo activities.Our results indicate that companies used a combination of manual and advance provisioning.Manual provisioning involved creating accounts on the individual systems, and managing theaccesses of users manually. This process involved challenges such as inefficiency, errors, andcommunication and collaboration complexity (Section 5.2.6 and Section 5.2.7). These chal-lenges motivated companies to move away from manual provisioning to automatic provision-ing. On the other hand, transitioning to and staying in the automatic provisioning was difficult,which led to companies chose to perform part of their provisioning manually. We observedthat companies frequently cycle between the manual and automatic provisioning, and duringthe cycle, their knowledge of business and access privileges increases.Besides the two centerpiece activities, there was a set of supporting activities that contributedto the provisioning.First, identity management activity orchestrated the automatic provisioning, and partly themanual provisioning. When the identity was created for a user, automatic provisioning as-signed the user to roles based on the identity attributes. Furthermore, during manual provision-ing, accounts were created by security administrators (or service desk) on those end-points thatwere not managed automatically, and then those accounts were associated with access privi-leges. The success of automatic provisioning relied heavily on the accurate and efficient iden-tity management. Therefore, an additional challenge that prevented companies from adoptingautomatic provisioning was ineffective identity management. Due to the importance of identitymanagement, we focus on modeling its challenges in the next section.Second, access review was a supporting activity for manual provisioning. We showed that themanual provisioning was error prone, and the goal of access review was to detect and eliminatethose errors. In Chapter 6, we will show that access review is challenging, and those challenges136Automatic Provisioning Manual ProvisioningRole EngineeringRole MiningUnderstanding Access PrivilegesIntegration1- Inefficiency2- Errors3- Communication and Collaboration Complexity4- Extensive Audits1- Uniqueness of Access2- Constant Changes in Business3- Identity Management ChallengesActivity ChallengeIdentity ManagementAccess ReviewSupportPartially SupportTransition from manual to automatic provisioningTransition from automatic to manual provisioningResponse to challengesFigure 5.3: The integrated model of IAM and its challengescontribute to the challenges of manual provisioning.Towards transition to automatic provisioning, companies perform four activities: understand-ing and documenting access privileges, role mining, role engineering, and integration with dif-ferent end-points. Understanding access privileges is a supporting activity that contributes toboth manual provisioning, and to the transition from manual to automatic. A well understoodset of privileges will reduce all three challenges we found in manual provisioning. Further-more, such an understanding would help engineering of roles. We also show that the otherthree activities that support transition from manual to automatic provisioning are challenging,and those challenges are a barrier to the transition.137In summary, the integrated model of the IAM and its challenges explains that why companiesuse both automatic and manual provisioning. They constantly analyze their manual provision-ing, and identify opportunities for transitioning to automatic. Meanwhile, they face exceptionalcases, and business changes during their automatic provisioning, and address them by manualprovisioning. Cycles between manual and automatic provisioning usually lead to better under-standing of business processes and access privileges.5.4.1 Identity Management Challenges ModelIn the previous section, we discussed how identity management (IdM) orchestrated the provi-sioning, and the challenges in IdM led to sub-optimal operation of access management. Theeffective functioning of IdM relies on the identity data source. Our interviews showed thatthe biggest tensions in IdM operation was caused by the relationship between IdM and thesource. To study the interaction of these two activities, we modeled both in activity theoryframework suggested by Engestro¨m (1999) (see Chapter 2 for an overview of activity theory)and presented the model in Figure 5.4.The model shows two activities that should work together in order to create, and manage iden-tities effectively. The IdM activity needs to rely on an identity source that is usually managedin a separate activity to keep track of people. Our interviews showed that an HR system wasmainly used as the identity source to detect new hires and create identities, to track changes inemployee statuses and update their identity, and to remove identities for those employees wholeft the company. Comparing the interrelationship of the two activities suggests four tensionsbetween various components of them. These tensions are labeled by numbers in Figure 5.4 andlisted below:Tension #1: Mismatch between the notion of employee and identity: The main object of HRis an employee, and the main object of IdM is an identity. Similarly, the HR keeps em-138ployee records as artifacts, and IdM keeps identity records as artifacts. Our interviewsshowed that there was a mismatch between the notion of identity and employee. Whilean employee was usually mapped to a unique identity in the IdM system, there werecases where multiple employees were mapped to one identity (i.e., multiple employeesshared an account). Furthermore, identities may go beyond the employees, and includecontractors, visitors, or even external organizations. This led to companies changing thenotion of employee in their HR system to include non-employees.Tension #2 and #3: IdM and HR motives: The two main motives of identity managementactivity were security and efficiency. But we observed tensions between IdM motives,and the HR activity. First, the security motive of IdM was not the main concern of HRactivity subjects. Furthermore, HR activity subjects did not necessarily possess securityawareness, and in many cases were not aware of the consequences of incorrect or in-complete HR data. Second, the efficiency motive contradicted the rules governing HRactivity, many of which were irrelevant in the context of IdM. Therefore, we observedthat the employee information entry in the HR system was delayed in order to complywith HR related rules and regulations.Tension #4: HR and IdM collaboration We also observed a tension in collaboration betweenHR and Security. HR could not provide all the HR data to security, as the subset of thedata was considered sensitive (e.g., social security number, salary). Therefore, organiza-tions could not directly have access to the HR database, and many of them had to designan intermediate system between HR and IdM.139HR EmployeesRulesCommunitiesDOLEmployeeHRISEmployee RecordsSecurity TeamRules CommunitiesDOLIdentityIdentity Management SystemIdentity RecordsSecurityEfficiencyTensionIdentity Management Activity Human Resources Management Activity1234Figure 5.4: Tensions in identity management activity5.5 Grounding the Work in Access Control Literature5.5.1 Theoretical Aspects of Access ControlAccording to Samarati and Vimercati (2001), there are three different abstractions to an accesscontrol system:1. access control policy, which defines the rules that are used to grant or deny access.2. access control model, which provides a formal representation of the policy and how itworks.3. access control mechanism, which defines the low level functions that implement theaccess control.Sandhu and Samarati (1994) classify access control policies into three main categories: DAC,MAC, and RBAC. In the following we provide an overview on each policy.140 File 1 File 2 File 3 User 1 Read Write Read  User 2 Read  Read Write  User 3 Read Write  Own Read Write  Figure 5.5: Access matrix proposed by LampsonDiscretionary access control model (DAC)The idea behind DAC is to determine the access of a user to an object based on the identity ofthe user and the authorizations that the user has on the object.Access control matrix suggested by Lampson (1971) is the simplest DAC model. It consists ofthree entities: Users, Objects, and the Access Rights that users have on objects. According toSandhu and Samarati (1994), the access matrix can be implemented using various mechanismsincluding access control lists (ACLs) or capabilities. In the ACL implementation, each objecthas an ACL containing the list of users and their access rights on that object. In the capabilityimplementation, each user has a list of objects he has access to. In more advanced forms ofACLs, groups can be defined to assign access rights to users in bulk, or rules can be definedto automatically assign access rights to users based on certain attributes of users (see ABAC inSection 5.5.1). According to Sandhu and Samarati (1994), ACL is known as a flexible methodto define the access rights of a user. On the other hand, there are certain disadvantages to thismechanism. First, there is no control over how information is passed between different users,which is important in a military context. Second, if the context changes rapidly, it is hard tomanage and change access control lists and capabilities and keep them up to date. Third, un-derstanding the effective policy in this model, which can also include negative authorizations,can be challenging as shown by Reeder et al. (2011).In our field-study, we saw that companies use ACLs during manual provisioning. As we dis-cussed in Section 5.2.6, account are created for users in an end-point, and then those accounts141are associated with the specific resources in the end-point. Furthermore, our results show thatACLs are hard to maintain in case of rapid changes as they require manual changing of thepolicy. Conversely, we did not see the challenges with information flow, as our results werelimited to enterprise context rather than military. Furthermore, we did not see the problem withnegative authorizations, as we only saw instances of negative authorizations in windows filesystem as discussed by Reeder et al. (2011).Mandatory access control model (MAC)MAC model addresses the issue with passing of information between different users by as-signing security labels to users and objects. Each object has a security label which determinesits sensitivity, and each user has a security label that determines its trustworthiness. There arevarious MAC policies, but two of the most famous policies are write up and read down (see(Sandhu and Samarati, 1994) for details of each). MAC model is primarily used in militarycontext, in which the flow of the information is important. As expected, during our field studywe did not see the use of MAC model in the enterprise context.Role based access control (RBAC)Role based access control addresses some of the issues with previous models by introducingthe notion of roles. Osborn et al. (2000) show that RBAC is policy neutral and it is possible toimplement MAC and DAC policies using RBAC.The core RBAC model by Ferraiolo et al. (2001) consists of three entities: users, roles, andpermissions (also called privileges or entitlements). Users are assigned to roles, and permis-sions are assigned to roles as well. Roles should resemble the job functions in the organization.A user gets permissions by becoming a member of roles according to his job function.The core RBAC model also requires the ability to review (or query) user-role assignments as a142mandatory feature, and role-permission assignments as an optional feature.There are two additions to the core RBAC model that organizations can implement dependingon their needs: role hierarchies, and static and dynamic separation of duties.Kern et al. (2002) suggest that the assignment of users to roles can be done manually, or auto-matically based on certain attributes of the user (e.g., location).The RBAC model can be used in conjunction with Attribute Based Access Control (ABAC)(Kuhn et al., 2010) model or variations of it (Thomas and Sandhu, 1997; Thomas, 1997). InABAC, the assignment of users to roles is determined by certain attributes of the user, theobject, and the context. Using ABAC allows for creating dynamic active policies.As we show in Section 5.2.5, our participants did not solely use the classic RBAC model withroles mapped to organizational structure. They rather used roles with different meanings, suchas location, department, and job functions. But there was not a direct mapping between the or-ganizational structure and the roles structure. Furthermore, none of our participants mentionedthe use of hierarchical RBAC. Role hierarchies is particularly useful when RBAC model re-sembles the organizational structure, and therefore, lack of using hierarchical RBAC can beexplained by lack of mapping roles to job functions. The access control model used in most ofthe companies for automatic provisioning was similar to ABAC, were users were provisionedwith roles based on their user attributes. On the other hand, manual provisioning followed aDAC policy, implemented with access control list mechanism.Our findings (Section 5.2.9 and with more details in Chapter 4) explain the need for queryinguser-to-role assignments. The organizations we talked to, usually required managers to reviewthe access privileges of the employees. In contrast, if the automated role based provisioningwas used, the query of user to role assignments was not mandatory for reviews, but prove tobe helpful as a contextual information. Also, few companies required the application ownersreview the roles, and check if they group a correct set of permissions. To satisfy this need, their143IAM system should provided a report of who has access to what.5.5.2 Practical Aspects of Access ControlIn the previous section, we reviewed the three main access control policies, and their associ-ated models, and mechanisms. We also compared our findings to the theoretical and practicalaspects of access control. In this section, we discuss the issues with practicing such mech-anisms in the organizations and go beyond the theoretical issues by focusing on human andorganizational aspects of access control.Empirical studies on RBACPrevious empirical studies reported certain benefits and challenges for RBAC. According toGallaher et al. (2002), RBAC could simplify the administration of access in a dynamic, fastchanging organizational context, enhance organizational productivity, reduce new employeedowntime, enhance the security, and simplify compliance. On the flip side, Gallaher et al.(2002) report that implementation of RBAC is challenging. Role definition is a time-consumingactivity. After defining roles, integrating all of the IT systems in the organization with theRBAC system requires a huge effort. Connor and Loomis (2010) conducted a survey of accessand identity management in 2010. More than 150 organizations in various sectors (exceptmilitary) responded to the survey. The result of the survey shows that RBAC adoption isincreasing every year in the IT-intensive industries (Figure 5.6).Additionally, Connor and Loomis (2010) determined the primary access control mechanismsused to manage different end-points in organizations. Their results (Figure 5.2) show that bothroles and ACLs are widely used as the access control mechanism. Many of the NIST’s surveyrespondents reported the use of hybrid approaches. In a hybrid approach, RBAC and ACLs areused at the same time as the primary and secondary mechanisms.144Economic Analysis of Role-Based Access Control ES-6 Figure ES-2. RBAC Adoption, 1992–2010  Note: Industries were defined by 2-digit NAICS code and included utilities; manufacturing; wholesale trade; retail trade; information; finance and insurance; professional, scientific, and technical services; educational services; health care and social assistance; arts, entertainment, and recreation; other services; and public administration. Table ES-1. Adopters’ Experiences with RBAC Question Yes No Has the use of roles improved the efficiency of maintaining your organization’s access control policy? 84% 16% Do you use roles that are native within applications? 78% 22% Do you use enterprise roles via an identity management solution that manages permissions for users across multiple applications and/or systems? 54% 46% Does your organization run an enterprise resource planning (ERP) system (i.e., Oracle, SAP)? 54% 46% Has your organization encountered any challenges with routine provisioning because of a lack of standardization in roles or specifications across different applications or systems?  55% 45%  ƒ use enterprise roles via an identity management solution that manages permissions for users across multiple applications and/or systems (54% vs. 46%), and ƒ encounter challenges because of a lack of standardization in roles or specifications across different applications and systems (55% vs. 45%). Figure 5.6: Adoption of role based access control in the organizations (figure from (Con-nor and Loomis, 2010))Table 5.2: The use of different access control mechanisms in IT systems (figurefrom (Connor and Loomis, 2010))Executive Summary ES-7 As shown in Table ES-2, roles were most likely to be used as the primary access control mechanism for  ƒ human resource information systems (56%); ƒ sales and customer relationship management systems (52%);  ƒ purchasing, order management, and logistics systems (50%); ƒ accounting and financial management systems (50%); and ƒ business process management systems (44%). Almost all organizations reporting hybrid approaches—such as roles and ACLs—reported using either roles as the primary mechanism and ACLs as the secondary one, or ACLs as primary followed by roles as secondary. As one respondent noted, “While we attempt to build RBAC controls, they tend to be implemented by using ‘groups’; hence the separation of ACL and RBAC is difficult, as they tend to overlap.”  In their remarks, respondents often expressed a preference for RBAC but were faced with the reality of business operations, applications, and systems that were inhospitable to it or for which RBAC would be c unterproductive. The three most common barriers were th  ollowing: ƒ Certain combinations of user types, systems, and workflows do not lend themselves to effective management via roles. ƒ Legacy systems were not designed with sufficiently granular levels of authorization to be compatible with roles. Table ES-2. Primary Access Control Mechanism Used, by Information System Category System ACLs Roles Rules Attributes Other Human resource information systems 37% 56% 6% 2% 0% Sales and customer relationship management systems 41% 52% 2% 4% 0% Accounting and financial management systems 41% 50% 6% 2% 2% Purchasing, order management, and logistics systems 41% 50% 7% 2% 0% Business proces  ma agement systems  42% 44% 7% 4% 2% Enterprise database systems 43% 41% 10% 6% 0% Electronic health record and health information systems 48% 34% 10% 7% 0% Identity management systems 39% 34% 15% 7% 5% Physical security services 50% 28% 9% 9% 4% Directory services 49% 27% 10% 6% 8% Network identity services  53% 22% 14% 6% 4% Web services 51% 20% 14% 6% 8%  145Connor and Loomis (2010) compare the time requirements of four provisioning tasks usingRBAC and ACLs. They show that the time to perform four tasks: (1) assigning existing privi-leges to new users, (2) changing privileges of existing users, (3) establishing new privileges toexisting users, and (4) terminating privileges will be 12.4, 7.8, 9.8, and 7.6 minutes respectivelyusing ACLs and will be 6.9, 6.6, 8.0, and 4.7 respectively using roles. On the other hand, theauthors do not provide their methodology for calculating the provisioning times.Our results support most of the NIST findings. First, our integrated model 5.4 shows why thecompanies use a hybrid approaches in managing access, as completely automating the processis impossible, and using manual provisioning is costly. In our interviews, we saw all of thebenefits reported by NIST about using roles. We also show that organizations need to engagein role-engineering and integration activities to achieve automated role-based provisioning,and those two activities are challenging. Conversely, NIST did not report the challenges withmaintaining an RBAC system, including the constant changes in the company, uniqueness ofaccess, and identity management challenges. We find these three set of challenges as barriersto adoption of RBAC, and an important decision factor for companies not to use RBAC forautomatic provisioning.Prior research shows that the access control policy changes rapidly in organizations. Kern(2002), in a report on implementation of RBAC, shows that in a bank with 40,000 users, thereare about 12,000 changes of user-role assignments per week and 600 changes of permission-role assignments per week.Managing Exceptions in Access PoliciesManaging exceptional cases is one of the main issues with using any of the access controlmodels discussed in section 5.5.1. For example, sometimes an employee changes job and heneeds access to his previous job’s applications for a short period of time. In this scenario,146an exception should be made to the policy, and the employee should keep the access for thelast job. The issue with exceptions has been previously discussed in the literature. In a studyby Bauer et al. (2009), security administrators responsible for file system access control andphysical security mentioned that managing exceptions are administrative nightmares. Stevensand Wulf (2009) performed a series of field studies in various organizations to understand theiraccess control practices. Their findings show that while organizations use mostly access matrixmodel, access control decisions can be made at three different times:Ex-ante control, in which a policy has been implemented before access attempt happens, andthe policy is enforced automatically by the system.Uno-tempore control, in which the permissions are decided at the time that access happens.Ex-Post control, in which the permission is audited after the access attempt happens.The authors discuss that most of the technological solutions only support Ex-ante control, butpeople use the existing technologies to practice the two other types of control manually. Forexample, they make exceptions to the security policy to grant temporary access to a user thaturgently needs access to a resource, or they keep audit logs in order to prevent people fromaccessing resources that they are not supposed to access. The authors argue that the lack oftechnological support can lead to human errors and inefficiencies, when people try to align theexisting technologies to their access control practices.While Stevens and Wulf (2009) show that Ex-ante, and Uno-tempore is practiced in organi-zations, they found limited evidences of Ex-Post practice. The Ex-Post practice has been alsodiscussed in the literature as a new access control paradigm. Povey (2000), in his paper ti-tled “optimistic security” points to the gap between what the organizations’ need and whatmechanisms can implement. He mentions that it is impossible for an access control systemto be aware of dynamically changing context. He proposes the concept of optimistic security,which can be summarized by the following principle suggested by Blakley (1996): “Make the147users ask forgiveness not permissions”. In optimistic security, the system allows the access butprovides mechanisms for auditing the access attempts and consequently reversing the illegalactions, removing privileges, or punishing users. According to Povey (2000), such systemsmight not be suitable for contexts with high risk of fraud, but could be useful particularly inhealth care domain or any context that requires timely responses to the access needs of users.Our results support findings of Stevens and Wulf (2009) and Bauer et al. (2009). We showthat exceptions lead to uniqueness of access and eventually forces companies to adopt manualprovisioning. The automatic provisioning in our integrated model (Section 5.4) is similar toEx-ante control where policies are set before hand and applied automatically, before accessattempts are made. The Uno-tempore is similar to manual provisioning, where a user hasto manually request access to resources. We also elaborated the Uno-tempore control, andshow that such requests can be made before actually attempting access by a manager whoexpects that the user will attempt access in the future. Furthermore, we identified that in mostcompanies they adopt generalized tools such as email and ticketing systems for practicing Uno-tempore controls. But we also saw some of the companies started adopting new techniques forsupporting Uno-tempore controls by offering access catalogs, and access control knowledgebases.We also saw the instances of Ex-Post control (or optimistic security) in our interviews 5.2.9,but only in medical context, where urgent and immediate access is an important requirements.Similar to the suggestion by Povey (2000), we have not seen the use of optimistic securityin organizational context. Our participants suggested that implementing optimistic security inorganizations and recording a log of all access attempts is considered a performance overhead,and it is hard to maintain.Additionally, our results further clarify the previous findings, and show the relationship be-tween Ex-ante, Uno-tempore, and Ex-post controls, and explains the benefits and drawbacksof each approach.148Ferreira et al. (2006) also point to the need for a method to provide users with access to unau-thorized resources at the time they attempt the access. The authors call this approach break theglass, as the user access the unauthorized object, but there will be cues that the user exercisedthe access. Like the optimistic security, in this approach users’ actions on the unauthorizedobjects should be monitored. In their proposal, Ferreira et al. (2006) use role based accesscontrol model to implement break the glass approach. Whenever users try to access an object,their permissions are determined using the roles they were assigned to. If they do not havepermissions, they are presented with a warning message that reminds them all of their actionswill be logged. Upon users’ consent, the access is granted. This provided non-repudiation to abreak the glass method.In our interviews, we saw a modification of the approach suggested by Ferreira et al. (2006).One of our participants (P15) suggested that users should be able to request access, when theytry to attempt access to a resource. But such a request should still go through the regularapproval workflow before user can access the resource.Schaad et al. (2001) describe a case study of adopting RBAC in a financial organization. Theauthors explain that during implementation of RBAC in a European bank, they designated awell-established method to assign users to roles. But they soon realized that administrationof freelancers and consultants broke their control principles. For such users, the access wasdirectly granted by including users’ identity in the ACL of applications.Our model explains why the organization in the case study by Schaad et al. (2001) had to man-age freelancers and consultants manually. First, managing them involved identity managementchallenges of external employees. Second, their access could be unique and not similar toactual employees of the company, which makes the already built roles impractical.149Communication and Collaboration ChallengesWhile access control mechanisms are capable of implementing various policies, communica-tion and collaboration complexity can result in ineffective access control.Bauer et al. (2009) performed 11 interviews with security administrators who were responsiblefor managing access to file system or physical resources in organizations. Their goal is tounderstand the challenges faced by administrators in their daily activities. The authors do notexplicitly mention which access control model was used in the interviewees’ organizations.But based on the information provided in their publication, and the fact that their participantsadministered file systems and physical security, we expect the use of DAC policies with ACLimplementation. Using the data collected through the interviews, the authors describe twoactors involved in the access control policy generation and management: (1) Policy Makerswho define the access policy, and (2) Policy Implementers who implement the policy. Theyshow that policies are managed by multiple people and miscommunication and lack of sharedview of the policy can cause errors in access configuration. Therefore, it is hard for each actorto maintain an understanding of the implemented policy. Usually, the policy maker does nothave a direct access to the implemented policy, and cannot verify if it is implemented correctly.Also, the policy implementer is usually blamed for incorrect policy implementation, while themistake could be from the policy maker.Our study results support Bauer et al. (2009), and further expand their findings. We saw thatthe policies were usually set by three different stakeholders: managers, application owners,and security administrators (or service desk). Managers and application owners played a roleof policy maker, and security administrators (or service desk) played a role of policy imple-menter. We also identified similar miscommunication problems, and lack of shared views inSection 5.2.6. Our results also suggest to deal with the errors, companies made the policymakers accountable, and asked them to frequently review the access. In addition, they tried to150understanding and document access privileges to deal with the communication and collabora-tion complexities.Role EngineeringIn a case study of a European bank by Schaad et al. (2001), the authors provide details oftheir implementation of RBAC for the financial institute. They mention that the access shouldbe determined based on a combination of official job position and job function of employees.For example, positions can be Clerk, Group Manager, or Regional Manager, and job functionscan be financial analyst, share technician or internal software engineer. Therefore, the bankdecided to use the concatenation of all combinations of job positions and functions as roles.This approach resulted in 1300 roles in an organization with 40,000 users. The authors arguethat this is a normal ratio of roles to users in any organization (3% to 4%). Most of the users inthe authors’ organization were assigned to one role and at most four roles, and the distributionof users across the roles was not uniform.Our results suggest that maintaining such a large number of roles would be a huge challengefor organizations. Furthermore, our participants suggested that companies should focus onautomating access provisioning for the accesses common between a large number of users, andavoid using roles for provisioning users in very specialized jobs.Kern et al. (2002) also report on the experience of engineering roles in different companies.The authors argue against using large number of roles and suggested using only one dimensionin roles (e.g., job function). Applying a role engineering methodology, Kern et al. (2002) coulddesign 120, 400, and 450 roles to manage 17,000, 30,000, and 31,000 users in three differentorganizations. In a different publication, Kern (2002) mentions that they parameterized rolesby assigning attributes to users, roles, and permissions, and made automatic decisions based onthe attributes. The second challenge with implementing RBAC is the number and complexity of151applications in the organizations. Every application has its own method of controlling access.To implement RBAC throughout the organization, an IAM System should be used as a layeron top of all applications. Also Kern et al. (2002) show that implementing all RBAC conceptsis not possible with the use of an AIM system. For example, the concept of sessions cannot beimplemented in an AIM system, as it is not capable of managing of users’ access when accessattempt takes place.We observed similar challenges in our field study as well. Our integrated model (Figure 5.3)showed that integration of IAM with end-points was a necessary step for role based automatedprovisioning, and it was a big technical and organizational challenge, and it was impossible insome cases. Furthermore, as we discussed before, the tracking of people’s access (sessions)was difficult, as it involved an overhead on the system.5.6 Implications For DesignIn this section, we provide a set of recommendations for improving IAM technologies andpractices based on the field-study findings.Use a dedicated identity source: IdM challenges model (Section 5.4.1) showed that the rela-tionship between IdM and HR activities causes a number of tensions. Furthermore, ourmain model (Section 5.4) showed that such tensions will impact the effectiveness of au-tomated provisioning. To address these tensions, we suggest having a dedicated identitysource for creating and managing identities (as suggested by P12). We saw the use ofa similar proxy system between HR and IdM in (P1, and P3’s organization). But thosesystems did not allow initiating identity creation. Such identity source should allow on-boarding, and off-boarding of users separately from the HR system. Furthermore, thesystem can be managed by the security team (or their delegates), who are aware of theconsequences of errors in identity related data entry. As our participants insisted on the152importance of having a single source of authority for identity management, we suggestthe dedicated ID source should be able to synchronize with the HR system, and allowsecurity administrators to review and resolve the differences between the two system.Educate the HR team about security: The IdM challenges model (Section 5.4.1) showed thelack of security awareness of HR employees will impact the performance of IdM activ-ity. Our participants emphasized on the impact of mutual understanding between securityand HR team. In cases where companies use HR data to create identities, the HR staffshould be made aware of the use of HR data for identity creation purpose, and be edu-cated about the consequences of their errors.Design safeguards against HR errors: Our integrated model of IAM shows that simple er-rors in HR data entry may lead to significant issues in the IAM system. Therefore, propersafeguards should be put in place against such errors. Rather than automatically mod-ifying identities in high risk situations, the IAM system should ask administrators tomanually confirm the sanity of data received from the HR system. This recommendationis also supported by the general error prevention usability principle suggested by Nielsen(2005a).Allow conversion of ad-hoc policies to roles and identity policies: Our proposed model (Sec-tion 5.4) suggest that companies continuously try to analyze their existing ad-hoc poli-cies, and convert them to roles and identity policies. To help the transition, tools shouldallow security administrators to analyze existing ad-hoc policies, finding patterns, andconverting that pattern to a combination of identity policies, and roles. Our interviewdata suggests this process cannot be accomplished by security team alone, therefore,tools should allow seeking help from business people to interpret the meaning of iden-tified patterns in ad-hoc policies, creating roles, and writing identity policies that mapusers to roles.153Verification of the impact of new roles: To provide any leverage, roles should be designedin a way that a large number of users can be assigned to them. Furthermore, such anassignment should not break any of the existing policies. Therefore, role engineeringtools should allow evaluating the impact of the created roles, so that security team candetermine if they provide any leverage. For example, tools can show users who will beassigned to a newly created role, and highlight the changes of the access policy due to thechanges to the role structure. Therefore, security administrator can verify if the createdrole will lead to automatic provisioning of a large number of users, and can validate ifassignment of the users to roles will violate any of the existing policies such as SoD.Document the meaning of access privileges and roles: Our integrated model of IAM showedthat understanding access privileges can contribute to both automatic and manual provi-sioning. Therefore, we suggest the information about the access privileges including thesystem they belong to, their technical information (e.g., their system code), their busi-ness meaning, the risk associated with having the access privilege, and the informationof a person who can answer questions about the access privilege should be documented.Such information can act as a common vocabulary between managers, application own-ers, and security team, and will help them in creation, interpretation, and implementationof the requests. Furthermore, the information can be helpful during the role-engineeringexercise, and access review.Show context during access request work-flow: Our participants suggested that they use gen-eralized ticketing systems (or even emails), for making, and tracking the access requests.IAM tools should allow incorporation of context into access request tools. For example,tools should show the employee what he can request, and show the approver the infor-mation about the requester, his other access privileges, and the history of his requests.Allow understanding and making sense of access policy: Our model (Section 5.4) highlightedthat the access review activity cannot be completely automated, as companies cannot154eliminate advance provisioning. Therefore, tools should allow reviewers make sense ofaccess policy, and make a judgment about the validity of access. We further expand thisrecommendation in Chapter 6, and show how this recommendation can be implemented.5.7 DiscussionWe showed that efficiency, cost reduction, and security were three motives behind identityand access management. Ideally, an IAM system should provide users with the least set ofprivileges required for their job, and not more. But we showed that such a model leads toinefficiencies, as users need to ask for privileges for any ad-hoc task that requires additionalprivileges, and then the excessive privileges should be revoked from the users as soon as theyare no longer needed. While this model aims to ensure the principle of least privilege, it can bea road block for employees doing their job, and overload managers and security administratorswith access requests, reviews, and removals. We also show that this model can eventuallynegatively impact the security of the system as managers or security administrators tend tocommit errors when facing large number of requests, or access privileges to review. We alsoshowed that complete automation of this process is impossible. We observed companies tryto find a balance between security and business continuity by perform risk assessment, and incases where business continuity outweighs security risk, they relax the security requirements.Therefore, we suggest a model where security administrators act as facilitators, rather thangetting involved in every access decision. We believe this model can benefit both security andbusiness continuity. In this model, employees can be responsible for requesting access to theresources they need through an access catalog, and access can be automatically granted in theabsence or tolerable level of risk. Depending on the risk level, security team can be notifiedabout the request, and monitor the users’ access attempts using an optimistic security paradigm(see Section 5.5.2). If the access request involves high degree of risk, it can go through properapproval workflow, and then the policy can be reviewed and updated, so that future cases can155be handled automatically. In this model, users can perform most of their access requests ina self-serve fashion, and efficiency will improve. The managers can focus only on approvingand reviewing high-risk access privileges, which eventually reduces human errors, and securityteam can focus on improving and evolving the high level access policy, rather than dealing withad-hoc access requests.5.8 ConclusionIn this chapter, we used grounded theory to collect and analyze qualitative data from 19 securitypractitioners about their experience with IAM adoption and use. Our results provided a thickdescription of different IAM activities and their challenges. We then provided an explanatorymodel to describe how different activities and challenges are linked to each other. We thenreviewed related work in this area, and tried to analyze them through the lens of our findings.Finally, we provided suggestions for improving technology or practice, and suggested a user-centric approach for access management.156Chapter 6Designing Usable Access ReviewInterfaces16.1 IntroductionUnderstanding and authoring access control policies has been known as a challenging problemaccording to Reeder et al. (2008), and Smetters and Good (2009). But the focus of previousstudies were on personal access control, where the data owner, policy maker, and policy imple-menter are the same person. This problem has not been extensively studied in organizationalcontext. Bauer et al. (2009) found that managing access control policies in organizations facesa unique set of challenges. In large organizations, those who make policies are different fromthose who implement these policies. Therefore, developing a shared understanding of policybetween different stakeholders is challenging. In this chapter, we explore and address thisproblem by proposing and evaluating AuthzMap, a new user interface for sense making and1A subset of material presented in this chapter has been published or accepted as the following conferencepublications:P. Jaferian, H. Rashtian, and K. Beznosov. 2014. To authorize or not authorize: helping users review access policies in organizations. InSymposium On Usable Privacy and Security (SOUPS 2014). Pages 301-320. Menlo Park, CA. (acceptance rate: 26%)P. Jaferian, H. Rashtian, and K. Beznosov. 2014. Helping users review and make sense of access policies in organizations. In CHI’14 ExtendedAbstracts on Human Factors in Computing Systems (CHI EA ’14). ACM, Toronto, ON, Canada, 2017-2022. (acceptance rate: 49%)157reviewing implemented access policies or, in short access review.In Chapter 5, we showed that access review is an important IT security activity in organi-zations, and is used to detect and eliminate errors that happen in manual provisioning. Themanagers are mandated by many security regulations (e.g., SOX (United States Code, 2002),HIPAA (Centers for Medicare & Medicaid Services, 1996)) to regularly review and validate theaccess privileges of users. However, Cser (2011) suggests that access review for every 2,000to 3,000 users consumes approximately one full-time-employee equivalent per year, and manyorganizations cannot even finish one access review process before a new campaign begins.Recent security incidents that cost governments and organizations billions of dollars show theimportance but yet lack the ability in reviewing users’ access rights. For example, the USarmy soldier, Chelsea Manning, who leaked the US embassy cables was cleared to accessclassified resources when she was on training as an intelligence analyst. She then changedher job and location multiple times before going to Iraq. According to Swensen (2011), if asuperior reviewed Mannings’ access and requested the revocation of unnecessary privileges,she would not have been able to leak the data.The overarching goal of this chapter is to investigate improvements to technology supportfor access review. Towards this goal, we performed four studies. In the first study, we usedinterviews from Chapter 5 as well as a survey of 49 security practitioners to understand howpeople make sense, and review access of users, and to identify the challenges in access review.In the second study, we used the results of heuristic evaluation from 24 evaluators (Chapter 4) tounderstand usability problems with one of the existing access review tools. We then designeda new user interface, guided by activity theory and ITSM guidelines (Chapter 3), to addressthe identified challenges. We named the proposed interface AuthzMap. In the third study,we improved AuthzMap through multiple rounds of informal evaluation, expert reviews, anda heuristic evaluation with 12 usability experts. In the fourth study, we conducted an onlineexploratory study with 430 participants to test if AuthzMap improves the usability over two of158the existing interfaces.6.2 BackgroundOrganizations use many IT applications to run their business. Employees who use an applica-tion for their job are provided with a set of access privileges, and other employees should beprohibited from accessing the application. Therefore, applications provide a set of permissionsthat can be assigned to a user to control what the user is authorized to do. Sometimes, permis-sions are grouped into roles to simplify and automate the provisioning process. In Chapter 5,we show that organizations use a combination of automated and manual provisioning. Further-more, we show that the manual provisioning is prone to errors. Therefore, organizations aremandated by many security regulations (e.g., SOX (United States Code, 2002)) to frequentlyperform access reviews to make sure that users have the least set of privileges required for theirjob.Next, we describe how access review is performed through an example. In an organization,security administrator, John, sends a request to manager Bob to review the access privileges offifty employees who work in Bob’s department. Bob is provided with the list of employees andtheir access privileges. He reviews the list of users one user at a time, looks at their privileges,and verifies if the user-to-privilege assignments are valid. For example, Bob sees that Alice isassigned to 20 privileges (R1, R2, . . . R20). Bob needs to understand the meaning of the priv-ileges, what they authorize Alice to do, and if the authorizations are required for Alice to doher job. If an authorization is required, Bob certifies the assignment of Alice to the privilege.Otherwise, he revokes the assignment. If Bob cannot understand the meaning of an accessprivilege, he may communicate with John or other managers to ask for a clarification. This ex-ample shows that access review requires intensive analysis, communication, and collaborationwith other stakeholders.1596.3 Related WorkThere have been few studies related to access review in organizational context. Bauer et al.(2009) performed a field study of access control practices in organizations. They suggest thatthe implemented access policy and the record of changes should be understandable and visible.Our findings confirm this, and our proposed interface improves understandability and visibilityof access policy.As opposed to access review, the problem of policy authoring has been previously studied.Brodie et al. (2005) designed a privacy policy management workbench called SPARCLE tocreate policies in natural language. Although SPARCLE was successful in facilitating policydefinition and management, it was not used or evaluated for the access review. Inglesant et al.(2008) studied personal access control in Grid computing context. They showed that resourceowners have difficulty expressing policies in RBAC and they prefer the use of natural language.Reeder et al. (2008) proposed a new UI named “expandable grid” for understanding effectiveaccess policy in case of conflicting access rules. Expandable grid improves the understandingof access policy by end-users of commodity OSs, and their main goal is to address the issuewith conflicting access rules that happen regularly in the Windows file system. The data fromour interviews show that in enterprise environments, standard role-based access control with-out negative authorization rules is used. We also adopt the idea of expandable grid for use in anorganizational context and use it in the design of AuthzMap. Smetters and Good (2009) stud-ied the use of policy authoring for personal documents. They found that users rarely changeaccess policies, and tend to specify complex and error-prone policies. Our findings suggestthat unlike access control for documents, the users’ accesses change frequently in organiza-tions. Vaniea et al. (2012a,b) examined the effect of proximity of access management interfaceand the resources. They show that users detect errors better if controls are positioned nearresources. Their proposed method was implemented and evaluated in the context of managingphoto album privacy policy. In an organizational context, this proposal might not be possible,160as resources do not have direct graphical representation, and the number of resources and per-missions could be large. Beckerle and Martucci (2013) identified six guidelines for designingusable access control rule sets, and showed that implementing those guidelines will help under-standability of access policies. Their proposed solution can be used before presenting accesscontrol rule set in AuthzMap to reduce the complexity of policy.6.4 Study 1: Understanding the ActivityAs we discussed in Chapter 5, the primary goal of the interview study was to understand howorganizations perform identity and access management, identify the challenges they face, andunderstand how technology can be improved to address the identified challenges. After wedeveloped our integrated model of IAM and its challenge, we decided to focus on one specificactivity in the model that could be improved by better technological support. Therefore, wefocused on access review. Particularly, we tried to answering the following research questions:(1) Why organizations perform access review? (2) Who are the involved stakeholders? (3) Whyaccess review is challenging? (4) How better decisions can be made during access review?6.4.1 MethodologyWe used a combination of qualitative and quantitative methods to understand the access re-view activity. Initially, we performed 13 semi-structured interviews with security practitionersresponsible for access management in large organizations. After the initial analysis of the in-terviews, we performed a survey to collect descriptive statistics about various aspects of accessreview found during the interviews. We also used the survey to recruit 6 more interview par-ticipants as a part of our theoretical sampling. Our results are mainly based on the qualitativedata analysis. We used the survey data to triangulate, complement, and elaborate the qualitativefindings. We discussed the methodology of our interview study in Chapter 5. In this section,161we provide the details of our survey methodology.Survey GoalsWe designed a survey with three goals in mind: (1) Clarify certain findings in the interviewstudy, in which we observed contrasting results. (2) Collect quantitative data that can help us indesigning a new interface for access review. (3) Collect data that help us design an ecologicallyvalid study to test the AuthzMap interface.In particular we tried to answer the following questions:1. How are users provided with access to resources in organizations ?2. Who is responsible for performing access review in organizations?3. Which stakeholders are involved in access review, and how do they collaborate with eachother?4. What pieces of information can be useful during access review?5. What would be an indication of risk when evaluating access privileges of users?Answering questions 1-3 extends our understanding of access review, and questions 4 and 5will help us designing better technologies for access review. To answer the aforementionedresearch questions, we designed a questionnaire with 20 questions. The questions were acombination of closed and open-ended, and participants were allowed to skip answering anyof them. Based on our field study experience, the target audience for the survey were securitypractitioners who are busy, and not willing to spend long period of time on a study. Therefore,we minimized the time required for completing the survey (between 10 and 15 minutes). Foreach question, we specified a set of possible answers based on the field study results. But wealso added an “Other” option, so that participants can enter their answer (i.e., in case their162answer is not in the provided options). The survey questions are presented in Appendix C.RecruitmentWe distributed the survey through Linked-In communities. The link to and the description ofthe survey was posted twice to three different Linked-In groups related to identity and accessmanagement (the three groups had 12,125, 8,792, and 4,598 members at the time of conduct-ing the survey). Also the link to the survey were posted to IAM related discussion forums,including support forums for CA, Oracle, and Novel IAM systems. We also posted the surveyto the Forrester Research2 discussion forum. Unfortunately, these recruitment efforts did notlead to recruitment of any participants. But we received a collaboration request from ForresterResearch company. They showed interest in collaborating with us in publicizing the survey, inexchange for survey data. As a result, we sought help from Forrester Research. We used theirsocial media channels (including their weblog and twitter feed) to distribute the survey notice.As a token of appreciation, we promised a raffle for a 128Gb iPad, and a complementary reportof the survey results from Forrester Research. We should note that our sampling method wasnot completely random, as participants may self-select themselves for the study. Furthermore,our participants were among those who read Forrester research Weblog or twitter feed, or thosewho have heard about the survey through the word of mouth. We also do not expect the raf-fle to had an impact on the participants’ decision of participating in the study. As we showin the next section, most participants were highly experienced security professionals, who arebusy, and well paid. Therefore, we believe the raffle did not introduce further self-selectionbias. We received 57 responses to the survey, out of which 49 were valid responses (e.g., twoparticipants declined the consent form, and 5 just browsed through the survey).2Forrester Research is an independent technology and market research company that provides advice on exist-ing and potential impact of technology, to its clients and the public.163010203040FrequencyJob Title Security consultantOtherIT security managerIT managerSecurity analystAuditorSecurity administrator181476211Figure 6.1: The job title of the survey participantsSurvey Participants DemographicsThe job title of the survey participants is shown in Figure 6.1. Our results show that the partici-pants were mostly security consultants and managers. Also we had very few security adminis-trators responding to the survey. Those participants who chose the “Other” option in the survey,had the following job titles: Analyst, Chief Technology Officer (CTO), Business Executive, Di-rector of Financial Controls, Lead IDM Consultant, I&AM Engineering Manager, Applicationsupport including identity systems, Technical architect IdM & Access management, Software,Developer, Human Factors/Design Research, Solution Architect, System Administration, SalesRep, Business development IAG, Business Systems Analyst, VP Enterprise Architecture.We then asked participants about their experience in IT security and experience with iden-tity and access management. The results of these two questions are presented in Figure 6.2.There was a strong correlation between the years of IT security experience and IAM experi-164051015200 10 20 30Years of IT security experienceYears of IAM ExperienceFigure 6.2: The participants years of IT security and IAM experience. Each dot indicatesa participant. The blue dashed line is the identity line. The points are jittered toavoid over-plotting.ence (Pearson′s r(47) = 0.70, p < 0.01). In other words, participants were dealing with IAMproblems during their career as a security professionals. For most cases, participants’ yearsof IT security experience was higher or equal to years of IAM experience. There were twocases were participants reported no experience in ITSM, but some experience with IAM. Inone case, participant was a director of financial controls and noted that “I’m in Finance, notIT. I am responsible for IT controls as part of the overall system of controls. I have experienceof an indirect nature with several IAM applications.” The other case was a participant whoreported his/her job as “Human Factors/Design Research”.6.4.2 ResultsIn this section, we combine the results of the interview study with the survey to provide adetailed description of access review activity. We provide our description in the activity theoryframework, and then discuss the identified challenges. As we discuss each component of the165activity or challenge, we first present our findings from qualitative study, and then elaboratethe findings with quantitative data (only in cases where we collected quantitative data throughour survey). Artifacts Subject Rules Community Division of work Object Users information Access privilege information User-to-privilege assignment  Managers Security admins Application  owners SoD rules Certification deadline Security policy  Security admin Application owner User Manager  Application owners provide information on privileges / Security admins provide information on user-to-privilege assignments / Managers provide information on users   Goal (User-to-privilege assignment)  (Reduce the risk of unauthorized access)  Figure 6.3: Overview of access review activity presented in the triangular model of activ-ityWe use the triangle model of activity proposed by Engestro¨m (1999) to lay out our descriptionof access review (Figure 6.3). We will later refer to this formulation when we justify our designdecisions.The goal of the activity: Access review is an activity with the goal of verifying users’ accessrights to minimize the risk of unauthorized and unmanaged access and comply with regulatorylegislations.166Subject: “Reviewer” is the main actor in the activity who performs access review. Our partici-pants indicated that the following stakeholders act as reviewers:Managers: Most of the participants indicated that managers review employees under theirauthority. P1 further described the role of the manager in access review: “[Manager asks:]what access does Jim have? I’d like to review Jim’s access because he’s changing roles withinmy department, there’s no official job posting but I’m doing a realignment and I would like toreview Jim’s access. So you need to do a specific report on Jim, which is to say here is theaccess profile that Jim has.”Application owners: Two of our participants indicated that an application owner reviews theusers who have access to the application, and certifies or revokes the users access privileges:“Our team wrote some [scripts]. It goes out and it collects from these 80 or so applications,what the access lists are, what the rights are, it creates a report, we put it in a service deskticket. Then it goes out to the [application owners] and they review it.” (P3)Security administrators: P6 explained that his team is responsible for security compliance ofa large enterprise application, and therefore he performs access reviews: “We send a requestto the manager that says Bob has changed from position A to position B. They are requestingposition B roles. We are going to remove his position A roles. Do you agree with that?”We asked survey participants who currently performs access review in their company, and whoshould perform it. We included different options based on the interview data, and also added“Other” option to check if the survey participants perform review differently or can think ofnew approaches. The summary of the responses is shown in Figure 6.4. Two participantschose the “Other” option as their current method of performing reviews: one mentioned thatthey used a mixture of the methods, without providing further details, and one participantmentioned that they review the role structure. For the desired state of access review, four par-ticipants chose the “other” option. One mentioned that the governance team in the company167303716319 10 91114240102030ManagersApp OwnersSecurity TeamAuditorsAutomaticOtherReviewerFrequency ConditionCurrentDesiredFigure 6.4: Survey participants’ responses on who currently performs and who ideallyshould perform access review in their companyshould review those access privileges that are undecided by the managers after the review. An-other participant wrote that the type of access privilege should determine who reviews it. Oneparticipant described that “role owners” should also review the access, beside the managers.Furthermore, those people who approved the assignment of user to role should also reviewthe access. One other participant mentioned that while managers and application owners bothshould review the access, the composition of the roles should be reviewed as well.Object: The object towards which the activity is performed is a user-to-access privilege assign-ment. When managers or security administrators perform access reviews, they review a set ofaccess privileges assigned to a user (user access review). When application owners performreviews, they review a set of users assigned to an access privilege (application access review).We limit the scope of the AuthzMap to user access review as our survey suggested that this isthe dominant approach. The same design techniques can be applied for building an interface168141312107610510Communication ChannelsFrequencyCommunication ChannelsWhen users' managers do the certification,  they need to contact application owners for helpWhen an application owners do the certification,  they need to contact users  managers for helpWhen either of users  manager or application owners do the certification,  they need to ask security administrators for helpWhen security administrators do the certification,  they need to contact application owners and users  managers for helpI don t knowOtherCertifiers need to contact previous certifiers for helpFigure 6.5: Communication channels that should be used during access review from sur-vey participants’ perspectivefor application access review.Community and division of work: Access review involves security team members, employees,managers, and application owners. Involved stakeholders divide the work as follows: A mem-ber of the security team requests review of users’ access rights. The reviewer (a manager inmost cases) receives the request. He goes through the list of users, selects a user, and identifiesthe user’s access privileges. For each user-to-access privilege assignment, he chooses to certifyor revoke the assignment. The reviewer might contact the application owners, the user, or thesecurity team when he is unable to determine the correct action. To confirm and extend thelist of communication channels used during the review, we asked participants about the pos-sible communications between various stakeholders involved in the access review activity. Asummary of the participants’ responses is shown in Figure 6.5:169Those participants who chose the “Other” option offered the following suggestions: (1) Oneparticipant mentioned that there is no need for any communication if the list of users and theiraccess privileges is available to the manager. Another participant noted that a recertificationteam should be in place to monitor and help if needed. One participant explained that managerneeds to discuss with users if there is a business need for the access.Rules and constraints: Different rules and constraints impact access review. (1) The securitypolicy of the organization determines the validity of a user-to-access privilege assignment. Forexample, P9 explained that in health care, they follow an optimistic security paradigm andallow more access than usual so the physicians can access patients’ files in emergency cases:“So the whole access model in health care tends to be, you let people do what they need to doto get the job done.” (2) Static separation of duties rules determine if a user can be assignedto two or more specific roles at the same time. (3) The review deadline set by security teamconstrains the time window of the review.Artifacts: Reviewers use three artifacts during access review: (1) User’s information, whichinclude the identity related information, the job title, and other attributes like the phone number,email, department, etc. (2) User-to-access privilege assignment information, which includewho requested, who approved, and who implemented the assignment, when and why the userwas assigned to the access privilege, and who previously reviewed the assignment. (3) Accessprivilege information, which include the privilege’s name, description, the owner, and the typeof access it provides.6.4.3 Access Review ChallengesOur interviewees indicated that access review is a challenging activity. We classified thesechallenges into five categories:Scale: Access review can involve large number of users, roles, and permissions. P6 explained170that just one of the large applications in his organization has 16,000 users, up to 115 roles peruser, and up to 407 permissions per role. He also indicated that reviewers have to review up to200 users in a review activity. While these numbers vary from application to application, andfrom organization to organization, they show the magnitude of data that a reviewer needs todeal with.Our survey results also shed light on the scale of access review. We asked survey participantsto estimate the number of users, number of applications, number of access privileges per user,and the number of users reviewed by one manager in each review session. The summary of theresponses is shown in Figure 6.6.Lack of knowledge: When managers act as reviewers, they do not have the expertise to under-stand the meaning of the access privileges. P2 illustrated this problem in detail: “we send thesegod-awful long reports to the new manager hiring the employee is going into, saying ‘let usknow which access this person needs to keep and what they need to remove.’ And a lot of it’s,you know, cryptic RACF information and stuff they just have no idea what they’re even readingso they either take their best guess and say, ok, then maybe this sounds kind of like somethingthey might need. Or they just say they need it all.”Frequency: While reviewing access is not the main job of managers, they are frequently askedto perform this activity. For example, P3 explained why they perform quarterly access reviews:“... Once a quarter! We do quarterly access reviews. [...] Once a year is never good for anycontrol because if you fail, you fail; at least twice a year you have a chance to re-mediate.”Additionally, P3 talked about ad-hoc access reviews: “Every day, [access management soft-ware] looks at [every] person who has access and says has the person changed in any way.Did they move departments, did they move to geographical locations - if so it triggers an eventwhich puts a ticket into the service desk system, sends a note to the Access Reviewers and saysyou need to review this ...”171010203040FrequencyRange 100−50010000−50000 More than 10000050000−100000 1000−50005000−10000 Less than 100500−1000107666421(a) Number of users in the organization10200/30FrequencyRange 10−50100−500 5000 or more1000−5000 500−1000Less than 10 50−10010643331(b) Number of applications in the organization05101520FrequencyRange 10−50100−500 Less than 105000 or more 1000−5000107211(c) Number of access privileges per user051015FrequencyRange 10−2020−30 5−1030−40 40 or more76321(d) Number of users to be certified by one managerFigure 6.6: Descriptive statistics on the scale of access review in survey participants’ or-ganization. Participants were allowed not to answer the questions, and therefore,the number of data points in each graph may not add up to 49.17212 12109760510On an ad hoc basis Quarterly Twice each year Annually I do not know OtherFrequencyFrequency On an ad hoc basisQuarterly Twice each yearAnnually I do not knowOtherFigure 6.7: Frequency of performing access review in survey participants’ organizationOur survey data provided more details on the frequency of access reviews. We asked partic-ipants how frequently they do reviews, and they could choose multiple answers ranged fromAd-hoc to once a year. The answers to this question are summarized in Figure 6.7. Partici-pants also could choose the “Other” option, and provide comments. One participant wrote thatreview should be done “On every employee job transfer, both by previous and current man-agers.” Another participant also suggested reviews on job transfers, and added that a reviewshould be done during deployment of new applications as well. Another participant wrote thatthey determine the review schedule at the beginning of each year, and one other participant saidthat they do not do it consistently. Finally, one participant stated that they do it automatically,and in real-time. The survey results show that the ad-hoc and quarterly reviews are the mostcommon, followed by twice a year, and annual reviews.We also asked participants what events in the company can trigger access review. Participants1732618151087 7 7 6401020TriggersFrequencyTriggers Scheduled reviews Changes in users'  job function Requests by managers or application owners A security incident Changes in role entitlement assignments  if roles used  I don t know Reorganization of a business unit Additional access request by a user Mergers and acquisitions OtherFigure 6.8: The list of events that can trigger access review in survey participants’ orga-nization.could choose multiple options including the “Other” option. The summary of responses ispresented in Figure 6.8.Those participants who provided the other option, suggested the following events can triggeraccess review: (1) Change of a manager (2) Employee separations (3) Audit findings (4) Newprojects (5) New applications (6) Hiring new employees.Human Errors: P3 described why human errors are common during reviews: “So the policiesof the company states that the business is responsible for the access. So the ultimate decisionmaker is the business. However they failed because it’s a human process right? It’s eyeballing[and] sometimes the lists are large.” Such errors would be costly for organizations, both in174terms of leading to data breaches, and failing compliance reviews.Exceptional Cases: In organizations, the validity of user-to-privilege assignments cannot bedetermined accurately only by knowing the user’s job function. Users might need to fill inanother employee’s role for a period of time, or they might need temporarily access certainresources when they are on training. P6 explained a case where they thought they shouldremove existing access from a user because he asked for new access. They later realized theuser is on training and still has his old job: “The manager says no, he is training this person,as replacement, for three months.”To design a tool that addresses the identified challenges, we will first analyze the usability ofone of the existing tools, and then propose a set of design goals to address the challenges andusability problems we found.6.5 Study 2: Evaluating an Existing TechnologyBesides the field study and the survey, we analyzed the set of problems reported by the partic-ipants in the heuristic evaluation study presented in Chapter 4. The goals of this analysis wereto triangulate field study findings with heuristic evaluation findings, and to find concrete set ofproblems in one of the existing access review interfaces.Methodology: We provided the full details of the methodology in Chapter 4. Here, we providea summary of our method. We used two sets of heuristic for the evaluation: Nielsen’s heuristicsby Nielsen and Molich (1990), and IT security management tools (ITSM) heuristics (see Chap-ter 4 for the list of ITSM heuristics). We recruited 28 participants with background in HCI andheuristic evaluation and asked them to evaluate the usability of an IAM system for performing4 tasks, one of which was access review. We divided the participants into two groups, Nielsenand ITSM. Participants in Nielsen group used Nielsen’s and participants in ITSM group usedITSM heuristics for evaluation. Each group was given a short recap on heuristic evaluation,175and an introduction to the heuristics that were assigned to them. We also gave them a five min-utes introduction to the target IAM system. Then we asked participants to evaluate the systemduring a two hour period based on four provided scenarios. One of the scenarios was accessreview, which involved a security administrator requesting and setting a deadline for review,and a manager going through the list of users that need to be reviewed, reviewing their roles,and making certification or revocation decisions. In this section, we only focus on the prob-lems reported in the fourth scenario of the heuristic evaluation study about the access reviewinterface.Results: The main access review interface evaluated by the participants is shown in Figure 6.9.We name this user interface “Search” throughout this chapter. After the study, the problemsreported by individual evaluators were aggregated to form a complete list of problems. For thepurpose of this study, we then chose those problems that were related to access review.123Level 1Level 2AllenBFigure 6.9: A screenshot of the Search interface. (1) Reviewer searches for a user. (2)Selects the users. (3) Clicks on the Select button and certifies or revokes accessprivileges in Level 2. A more detailed description of the Search interface is avail-able in Appendix D.176Table 6.1: Reported problems with the Search interface during heuristic evaluation. Thealphanumeric codes show evaluators who reported the problems. Evaluators withcode starting with N used Niel