Applied Science, Faculty of
Nursing, School of
DSpace
UBCV
Abdulai, Abdul-Fatawu
Doctor of Philosophy - PhD
University of British Columbia
Background: Digital health technologies are increasingly used to support people living with sexual health-related conditions. Despite stigma being a major issue in sexual health, it is rarely considered in the design of health technologies. How stigma is understood and addressed in the context of digital health technologies remains under-explored. The objective of this dissertation was to develop design principles that would serve as a guide in designing destigmatising sexual health-related websites, and to determine how such principles might be applied. Methods and Results: This research consisted of four studies. 1) A scoping review focusing on studies that explored stigma in relation to the design and impact of sexual health-related web apps on stigma. This review revealed that digital health technologies for addressing stigma could inadvertently perpetuate stigma. 2) A usability and stigma analysis to understand patient perspectives on the usability of a newly developed website and the design of destigmatising websites. This study revealed that destigmatising design principles can be reflected in a trauma-informed care framework. 3) A Delphi study to develop a set of destigmatising design principles with experts in stigma and sexual health-related research, producing 19 items that were mostly emotional and content-related principles. And, 4) a qualitative descriptive study to examine software designers’ perceptions of the applicability of the design principles, which were considered to be important, but largely perceived to be beyond the scope of practice of software designers. Conclusion: The possibility of perpetuating stigma via digital health technologies indicates a need to develop technologies that could alleviate sexual health-related stigma, prevent re-traumatization, and promote emotionally safe user interactions. This research suggests that a trauma-informed care framework, complemented by user-centered design approaches, may help in designing destigmatising digital health technologies. The principles developed in this research offered a useful reference guide for healthcare software designers to help expand the scope of health technology design from a predominant focus on usability and functionality to emotional and non-functional design elements. Further studies are needed to examine the applicability of the principles in practical design situations and to specific conditions.
DEVELOPMENT AND APPLICATION OF DESTIGMATISING DESIGN PRINCIPLES IN SEXUAL HEALTH-RELATED TECHNOLOGIES by Abdul-Fatawu Abdulai B.S.N., University of Ghana, 2012 M.Sc., UiT, The Arctic University of Norway, 2017 A DISSERTATION SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Nursing) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) June 2022 © Abdul-Fatawu Abdulai, 2022 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the dissertation entitled: Development and Application of Destigmatising Design Principles in Sexual Health-Related Technologies submitted by Abdul-Fatawu Abdulai in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Nursing Examining Committee: Leanne M. Currie, Associate Professor, School of Nursing, UBC Supervisor A. Fuchsia Howard, Associate Professor, School of Nursing, UBC Supervisory Committee Member Paul J. Yong, Associate Professor, Department of Obstetrics and Gynaecology, UBC Supervisory Committee Member Helen Brown, Associate Professor, School of Nursing, UBC University Examiner Lisa Nathan, Associate Professor, School of Information, UBC University Examiner iii Abstract Background: Digital health technologies are increasingly used to support people living with sexual health-related conditions. Despite stigma being a major issue in sexual health, it is rarely considered in the design of health technologies. How stigma is understood and addressed in the context of digital health technologies remains under-explored. The objective of this dissertation was to develop design principles that would serve as a guide in designing destigmatising sexual health-related websites, and to determine how such principles might be applied. Methods and Results: This research consisted of four studies. 1) A scoping review focusing on studies that explored stigma in relation to the design and impact of sexual health-related web apps on stigma. This review revealed that digital health technologies for addressing stigma could inadvertently perpetuate stigma. 2) A usability and stigma analysis to understand patient perspectives on the usability of a newly developed website and the design of destigmatising websites. This study revealed that destigmatising design principles can be reflected in a trauma-informed care framework. 3) A Delphi study to develop a set of destigmatising design principles with experts in stigma and sexual health-related research, producing 19 items that were mostly emotional and content-related principles. And, 4) a qualitative descriptive study to examine software designers’ perceptions of the applicability of the design principles, which were considered to be important, but largely perceived to be beyond the scope of practice of software designers. Conclusion: The possibility of perpetuating stigma via digital health technologies indicates a need to develop technologies that could alleviate sexual health-related stigma, prevent re-traumatization, and promote emotionally safe user interactions. This research suggests that a iv trauma-informed care framework, complemented by user-centered design approaches, may help in designing destigmatising digital health technologies. The principles developed in this research offered a useful reference guide for healthcare software designers to help expand the scope of health technology design from a predominant focus on usability and functionality to emotional and non-functional design elements. Further studies are needed to examine the applicability of the principles in practical design situations and to specific conditions. v Lay Summary Digital health technologies (e.g., patient-facing technology like a website) are common ways for healthcare providers to communicate with people living with sexual health-related conditions. This dissertation consisted of four studies that examined how stigma can be addressed in the context of digital health technologies for people with sexual health-related conditions. The first study found that some websites designed to reduce stigma instead increased stigma. The second study found that a trauma-informed care framework could be used to describe ways to reduce stigma for people with sexual-health related conditions. In the third study, a set of 19 destigmatising design principles were developed to guide design of sexual health websites. The fourth study found that software designers considered the design principles to be useful in addressing stigma but were not in their normal scope of practice. Further studies are needed to examine the applicability of the principles in practical design situations and to specific conditions. vi Preface The dissertation consisted of four studies that examined how sexual health-related stigma can be addressed in the context of digital health technologies. The study design for the four studies were conceptualised by the researcher with guidance from Dr. Currie, Dr. Howard and Dr. Yong. Study 1 (i.e., Chapter 3) is a scoping review and was conceived of, carried out and written by the researcher with guidance from Drs. Currie and Howard. This study was published by Taylor and Francis as “Abdulai, A. F., Howard, A. F., & Currie, L. M. (2022). Stigmatizing and de-stigmatizing properties of Web apps for sexual health-related conditions: a scoping review. International Journal of Sexual Health, 34(1), 144-159.” Study 2 is a usability study on patients’ perspectives on designing destigmatising websites that was conducted at the British Columbia Women’s Center for Pelvic Pain and Endometriosis (REB #H19-03556). The researcher (AFA) was responsible for data collection, analysis, and manuscript writing. Heather Noga and Gurskiran Parmer helped in participant recruitment and manuscript writing while Drs. Currie, Howard and Yong provided guidance on data collection, analysis and manuscript writing. Study 2, which constituted Chapter 4 was published in JMIR, Human Factors as “Abdulai, A. F., Howard, A. F., Yong, P. J., Noga, H., Parmar, G., & Currie, L. M. (2022). Developing an Educational Website for Women With Endometriosis-Associated Dyspareunia: Usability and Stigma Analysis. JMIR Human Factors, 9(1), e31317.” In Study 3, a Delphi study (REB #H21-00760) and Study 4, a qualitative descriptive study (REB #H21-02553), the researcher (AFA) was responsible for recruitment, data collection, data analysis, and the write up. Drs. Currie, Howard, and Yong provided guidance on the development of the Delphi study survey items, the qualitative interview questions, offered vii strategies for data collection, data analysis and provided feedback throughout the dissertation writing process. viii Table of Contents Abstract ......................................................................................................................................... iii Lay Summary .................................................................................................................................v Preface ........................................................................................................................................... vi Table of Contents ....................................................................................................................... viii List of Tables ............................................................................................................................. xvii List of Figures ............................................................................................................................. xix Glossary ........................................................................................................................................xx Acknowledgements ................................................................................................................... xxii Dedication ................................................................................................................................. xxiv Chapter 1: Introduction ................................................................................................................1 1.1 Digital Health .................................................................................................................. 1 1.2 Design of Digital Health Technologies ........................................................................... 2 1.3 Sexual Health-Related Stigma ........................................................................................ 3 1.4 The Use of Digital Health Technologies in Addressing Sexual Health-Related Stigma 5 1.5 Problem Statement and Objective ................................................................................... 6 1.6 Research Questions ......................................................................................................... 6 1.7 Significance of the Study ................................................................................................ 7 1.8 Relevance to Nursing ...................................................................................................... 8 1.9 Structure of Dissertation ................................................................................................. 9 Chapter 2: Literature Review: User-Centered Design, Design Principles & Emotional Design ............................................................................................................................................10 2.1 Introduction ................................................................................................................... 10 ix 2.2 What is Design? ............................................................................................................ 10 2.3 User-Centered Design ................................................................................................... 11 2.4 Design Principles .......................................................................................................... 13 2.5 Emotional Design and Healthcare Technologies .......................................................... 16 2.6 How Design Principles are Developed ......................................................................... 18 2.7 Summary and Conclusion ............................................................................................. 19 Chapter 3: Literature Review: Stigma, Sexual Health-Related Stigma & Trauma-Informed Care ...............................................................................................................................................21 3.1 Introduction ................................................................................................................... 21 3.2 Stigma ........................................................................................................................... 21 3.2.1 Types or Forms of Stigma ..................................................................................... 23 3.2.2 Stigma as a Communication Attribute .................................................................. 27 3.3 Sexual Health-Related Stigma ...................................................................................... 28 3.3.1 Forms of Sexual Health-Related Stigma ............................................................... 30 3.4 Stigma Reduction Strategies in Sexual Health ............................................................. 32 3.4.1 Interventions at the Individual or Intrapersonal Level .......................................... 32 3.4.2 Interventions at the Community and Societal Level ............................................. 35 3.4.3 Institutional or Governmental Level Strategies .................................................... 36 3.5 Conceptual Framework: Trauma-Informed Care .......................................................... 37 3.6 The Relationship Between Trauma and Stigma ............................................................ 38 3.6.1 Fallot and Harris' Principles of Trauma-Informed Care ....................................... 39 3.7 Digital Health Technologies and Sexual Health-Related Stigma ................................. 41 x Chapter 4: Stigmatising and Destigmatising Properties of Web Apps for Sexual Health-related Conditions: A Scoping Review .......................................................................................42 4.1 Introduction to Chapter 4 .............................................................................................. 42 4.2 Background ................................................................................................................... 42 4.3 Materials and Method ................................................................................................... 46 4.3.1 Search Strategy (identifying the studies) .............................................................. 46 4.3.2 Inclusion and Exclusion Criteria ........................................................................... 48 4.3.3 Data Extraction and Categorization ...................................................................... 49 4.4 Results ........................................................................................................................... 49 4.4.1 Summary of Study Characteristics ........................................................................ 51 4.4.2 Research Question 1: What is the state of the literature on stigma and the design of web apps? ......................................................................................................................... 56 4.4.3 Research Question 2: What is the impact of web apps on stigma? ...................... 57 4.4.3.1 Addressing/Alleviating Stigma ......................................................................... 57 4.4.3.2 Research Question 3: How do web apps alleviate or aggravate stigma? .......... 59 4.4.3.3 Sharing Stigma-related Experiences via Web Apps ......................................... 59 4.4.3.4 Display of Appropriate or Inappropriate Sexual Health-Related Content ........ 60 4.4.3.5 Implied Disclosure and Stigma-by-Association ............................................... 61 4.5 Discussion ..................................................................................................................... 62 4.5.1 Digital Health Strategies and Stigma .................................................................... 63 4.5.2 The Impact of Digital Health on Sexual Health-Related Stigma .......................... 64 4.5.3 Health Providers’ Moderation of Web Apps ........................................................ 66 4.5.4 Recommended Design Considerations to Address Stigma ................................... 66 xi 4.6 Limitations of the Scoping Review ............................................................................... 68 4.7 Quality of the Studies .................................................................................................... 68 4.8 Conclusion .................................................................................................................... 69 4.9 Summary of Chapter 4 .................................................................................................. 69 Chapter 5: Developing an Educational Website for Women with Endometriosis-associated Dyspareunia: A Usability and Stigma Analysis ........................................................................70 5.1 Introduction ................................................................................................................... 70 5.2 Background ................................................................................................................... 70 5.2.1 Sex, Pain, and Endometriosis Website Development ........................................... 72 5.3 Methods......................................................................................................................... 75 5.3.1 Recruitment ........................................................................................................... 75 5.4 Data Collection ............................................................................................................. 76 5.4.1 Demographics ....................................................................................................... 76 5.4.2 Usability Analysis ................................................................................................. 76 5.4.3 Stigma Analysis .................................................................................................... 77 5.5 Data Analysis ................................................................................................................ 77 5.5.1 Usability Analysis ................................................................................................. 77 5.5.2 Stigma Analysis .................................................................................................... 78 5.6 Results ........................................................................................................................... 79 5.6.1 Demographic Information ..................................................................................... 79 5.6.2 Task Completion ................................................................................................... 80 5.6.3 Overall Usability ................................................................................................... 80 5.6.4 Usability Problems ................................................................................................ 81 xii 5.7 Destigmatising Properties of Sexual Health-related Websites ..................................... 84 5.7.1 Theme 1. Emotional safety ................................................................................... 85 5.7.2 Theme 2. Collaboration ........................................................................................ 87 5.7.3 Theme 3. Empowerment ....................................................................................... 88 5.7.4 Theme 4. Trustworthiness ..................................................................................... 90 5.7.5 Theme 5. Choice ................................................................................................... 91 5.8 Discussion ..................................................................................................................... 92 5.9 Limitations .................................................................................................................... 97 5.10 Conclusion .................................................................................................................... 97 5.11 Summary of Chapter 5 .................................................................................................. 98 Chapter 6: Developing Destigmatising Design Principles: A Delphi Study ...........................99 6.1 Introduction ................................................................................................................... 99 6.2 Methods......................................................................................................................... 99 6.2.1 Overview of the Delphi Methodology ................................................................ 100 6.2.2 Philosophical Underpinnings of Delphi Study ................................................... 101 6.2.3 Pre-Delphi Stage ................................................................................................. 102 6.3 Participant Recruitment, Data Collection, and Analysis ............................................ 105 6.3.1 Sample and Sample Size ..................................................................................... 105 6.3.2 Participant Recruitment ...................................................................................... 107 6.3.3 Data Collection ................................................................................................... 109 6.3.4 Data Collection in Round 1 ................................................................................. 110 6.3.5 Modifying and Generating Items for Round 2 .................................................... 111 6.3.6 Data Collection in Round 2 ................................................................................. 112 xiii 6.4 Modifying and Generating Items for Round 3 ............................................................ 112 6.4.1 Data Collection in Round 3 ................................................................................. 113 6.4.2 Data Analysis ...................................................................................................... 113 6.4.3 Quantitative Data Analysis ................................................................................. 113 6.4.4 Level of Consensus ............................................................................................. 113 6.4.5 Content Validity Index ........................................................................................ 114 6.4.6 The Inter-Quartile Range .................................................................................... 115 6.4.7 Stability of Items ................................................................................................. 116 6.4.8 Qualitative Data Analysis ................................................................................... 117 6.4.9 Validity, Reliability, and Methodological Rigour of Delphi Studies ................. 118 6.5 Results ......................................................................................................................... 118 6.5.1 Demographic Information ................................................................................... 119 6.6 Round 1 Survey........................................................................................................... 120 6.6.1.1 Non-consensus Items in Round 1 ................................................................... 122 6.6.2 Revision of Design Principles in Round 1 .......................................................... 123 6.6.3 Newly Generated Principles in Round 1 ............................................................. 128 6.7 Round 2 survey ........................................................................................................... 130 6.7.1 Consensus Items in Round 2 ............................................................................... 131 6.7.2 Design Principles that did not Achieve Consensus in Round 2 .......................... 132 6.7.3 Stability of Design Principles That Did Not Achieve Consensus in Round 2 .... 134 6.7.4 Revision of Design Principles That Did Not Meet Consensus in Round 2 ........ 136 6.7.5 Newly Generated Design Principles in Round 2 ................................................ 139 6.8 Round 3 Survey........................................................................................................... 139 xiv 6.9 Summary of Findings .................................................................................................. 141 6.10 Discussion ................................................................................................................... 144 6.10.1 Trauma-Informed Care Framework and Stigma ................................................. 145 6.10.2 Functional and Non-functional Design Requirements ........................................ 146 6.10.3 Stigma Management Strategies ........................................................................... 148 6.10.4 Hiding/Avoiding/Passing for Normal/ Covering Stigmatising Attributes .......... 148 6.10.5 Education Strategies ............................................................................................ 150 6.10.6 Contacting ........................................................................................................... 151 6.10.7 Inclusivity ........................................................................................................... 153 6.10.8 Validity and Reliability of the Design Principles ............................................... 154 6.11 Limitations of the Delphi Study .................................................................................. 155 6.12 Conclusion .................................................................................................................. 155 Chapter 7: Applicability of Destigmatising Design Principles: A Qualitative Descriptive Study ...........................................................................................................................................157 7.1 Methods....................................................................................................................... 157 7.1.1 Sample and Sample Size ..................................................................................... 158 7.1.2 Participant Recruitment ...................................................................................... 159 7.1.3 Materials ............................................................................................................. 160 7.1.4 Data Collection ................................................................................................... 161 7.1.5 Data Collection Instrument ................................................................................. 162 7.1.6 Data Analysis ...................................................................................................... 163 7.1.7 Rigour ................................................................................................................. 164 7.2 Results ......................................................................................................................... 165 xv 7.3 Themes ........................................................................................................................ 166 7.3.1 Understanding of Stigma and the Design Principles .......................................... 167 7.3.2 Responsibility for Addressing Design Principles ............................................... 168 7.3.3 Emotional design ................................................................................................ 170 7.3.3.1 Addressing Emotional Safety in Software Design .......................................... 171 7.3.4 Collaborative design ........................................................................................... 174 7.3.5 Availability of Resources .................................................................................... 177 7.4 Discussion ................................................................................................................... 178 7.4.1 Addressing Internalized and Public Stigma ........................................................ 179 7.4.2 Responsibility for Addressing the Design Principles ......................................... 180 7.4.2.1 Collaborative Design ...................................................................................... 180 7.4.2.2 Education for Software Designers .................................................................. 182 7.4.2.3 Resource Allocation ........................................................................................ 183 7.4.2.4 Interprofessional Approaches ......................................................................... 184 7.4.3 Emotional Design ................................................................................................ 185 7.4.4 Functional Design Strategies to Address Stigma ................................................ 186 7.4.5 Participatory Design Strategies to Address Stigma ............................................ 188 7.4.6 Limitations .......................................................................................................... 190 7.4.7 Directions for Future Research ........................................................................... 190 7.4.8 Conclusion .......................................................................................................... 191 Chapter 8: Overall Discussion & Conclusion ..........................................................................192 8.1 Introduction ................................................................................................................. 192 8.2 Themes ........................................................................................................................ 192 xvi 8.2.1 The Focus of Stigma Reduction .......................................................................... 193 8.2.2 Perpetuating Stigma via Digital Health Technologies ........................................ 195 8.2.3 Addressing Stigma via Digital Health Technologies .......................................... 196 8.2.4 Trauma-Informed User-Centered Design ........................................................... 199 8.2.5 Interprofessional Approaches to Digital Health Technology Design ................. 200 8.3 Recommendations ....................................................................................................... 201 8.4 Recommendations for Nursing Informatics ................................................................ 202 8.4.1 Recommendations for Nursing Informatics Practice .......................................... 202 8.4.2 Recommendations for Nursing Informatics Policy ............................................. 203 8.4.3 Recommendations for Nursing Informatics Research ........................................ 204 8.5 Recommendation for Human-Computer Interaction Practice and Research .............. 204 8.5.1 Recommendations for Human-Computer Interaction Practice ........................... 204 8.5.2 Recommendations for Human-Computer Interaction Research ......................... 205 8.6 Overall Limitations ..................................................................................................... 206 8.7 Future Research Directions ......................................................................................... 207 8.8 Conclusion .................................................................................................................. 208 References ...................................................................................................................................210 Appendices ..................................................................................................................................254 Appendix A - Study 2 Materials ............................................................................................. 254 Appendix B - Study 3 Recruitment Materials ......................................................................... 256 Appendix C - Study 3 Original Survey Items Deployed in Round 1 ...................................... 259 Appendix D - Study 4 Recruitment Materials ........................................................................ 269 Appendix E - Study 4 Vignette with Interview Guide ............................................................ 270 xvii List of Tables Table 2.1 Nielsen’s Heuristics ...................................................................................................... 14 Table 2.2 Shneiderman’s 8 Golden Rules ..................................................................................... 15 Table 2.3 Universal Design Principles .......................................................................................... 15 Table 3.1 Original and adapted definition of trauma-informed care categories ........................... 40 Table 4.1 Study characteristics, impact on stigma, and corresponding themes ............................ 52 Table 5.1 Task completion rate ..................................................................................................... 80 Table 5.2 Interface problems from usability analysis ................................................................... 82 Table 5.3 Suggested design solutions ........................................................................................... 83 Table 5.4 Post-study systems usability questionnaire ................................................................... 84 Table 6.1 Preliminary destigmatising design principles and source(s) ....................................... 104 Table 6.2 Demographic characteristics ....................................................................................... 119 Table 6.3 Design principles that achieved consensus in Round 1 .............................................. 121 Table 6.4 Design principles that did not achieve consensus in Round 1 .................................... 122 Table 6.5 Original and revised principles for items meeting consensus in Round 1 .................. 127 Table 6.6 Original and revised principles for items not meeting consensus in Round 1 ............ 128 Table 6.7 Newly generated design principles in Round 1 .......................................................... 130 Table 6.8 Design principles that achieved consensus in Round 2 .............................................. 132 Table 6.9 Design principles that did not achieve consensus in Round 2 .................................... 133 Table 6.10 Stability of items that did not achieve consensus in Round 2 .................................. 135 Table 6.11 Revision of design principles not achieving consensus in Round 2 ......................... 138 Table 6.12 Design principles meeting consensus in Round 3 .................................................... 140 Table 6.13 Design principles not meeting consensus in Round 3 .............................................. 141 xviii Table 6.14 Final consensus design principles mapped to the trauma-informed care framework 142 Table 6.15 Principles not meeting consensus mapped to the trauma-informed care framework 143 Table 7.1 Demographic characteristics of participants ............................................................... 166 xix List of Figures Figure 4.1 Prisma flow diagram for scoping review .................................................................... 50 Figure 5.1 Sample of the website interface in Study 2 ................................................................ 74 Figure 5.2 Global themes, sub-themes, and concepts based on trauma ........................................ 85 xx Glossary Design guidelines: Universally applicable rules of thumb, laws, or considerations that help people to find reasonable solutions to a design problem. Delphi study: A methodology that is used to build expert consensus by soliciting opinions from experts through a series of questionnaires. Digital health technology(ies): For this dissertation project, digital health technology refers to patient-facing technology such as websites, mobile applications, and patient forums that are designed for patient use. Functionality: The ability of a software to serve the purpose for which it has been designed. Functional requirements: Are simply the statements or principles that specify what the system should provide, how it should act in specific situations, and how it should respond to specific commands. Heuristic evaluation: A user interface inspection method that designers employ to find design problems based on a set of design guidelines. Human-computer interaction: A multidisciplinary field that focuses on the design of computerized systems and the study of peoples’ interaction with computers. Non-functional requirements: Are simply the quality constraints the system must meet to promote good user experience Sexual health-related conditions: A group of conditions or clinical syndromes that are epidemiologically heterogeneous but generally affect an individual’s sexuality or sexual functioning and include sexually transmitted infections (STIs), sexual dysfunctions, and infertility (WHO, 2010). xxi Software designer: For the purpose of this dissertation, a software designer is defined as an individual involved in the design of websites including individuals who focus on how the technology is presented to the user (e.g., User Interface (UI) and User Experience (UX) designer, project manager, and other human-computer interaction experts) and individuals who focus on how the technology functions (e.g., software developers or software engineers). Stigma: A socially constructed (perceived or experienced) state that is characterized by blame, rejection segregation, and devaluation (See Chapter 3 for a full definition). System designers: System designers are typically designers, software engineers or computer programmers who design and translate functional requirements into computer software applications and websites. Usability: “The extent in which a product can be used by specified users to achieve specific goals with effectiveness, efficiency, and satisfaction in a specified context of use” (ISO, 1998). User interface: The space where humans interact with a computer or machine, e.g., computer interface, mobile phone interface. User interface (UI) design: User interface design is the process of building interfaces with a focus on the look and feel of computerized devices, while User experience (UX) design: User experience design focusses on how computerized devices can be created to provide meaningful and relevant experience to users. xxii Acknowledgements Many people have contributed to this work, and I am grateful to them all. First and foremost, I would like to thank the research participants who were the source of valuable information that made this work a success. To my academic advisor and Mentor, Dr. Leanne M. Currie, who inspired this academic journey: My sincere thanks for the generous opportunities and for the skillful guidance during the past four years. Your patience and commitment to my learning has been exceptional, and I could not have hoped for a better guide on my side. Thank you for the way you believed in me, beyond what I could have ever seen in myself. I would not tell this story without you in every chapter, encouraging and guiding me every step of the way. It has been a great ride! My heartfelt appreciation also goes to my committee members Dr. Fuchsia Howard and Dr. Paul J. Yong, whose expertise, critique, and advice have broadened my horizons. I am ever grateful to have worked with you both. I am also grateful to Dr. Yong’s lab, The BC Women’s Center for Pelvic Pain and Endometriosis, for providing the space and resources for Study 2. I am also grateful to staff at this lab, especially Heather Noga who assisted in data collection for Study 2. I also wish to express my heartfelt gratitude to Dr. Rachel Ralph at the Center for Digital Media and Dr. Caley Raber at the Emily Carr University Health Design Lab for assisting with participant recruitment for Study 4. xxiii I am extremely grateful to the NSERC-CREATE funding provided to me through Dr. Currie’s participation in the Designing for People Program at the University of British Columbia. I was also fortunate to have benefitted from the Janet Gormick Memorial Graduate Scholarship and the Lyle Creelman endowment fund, both from the University of British Columbia School of Nursing. I would also like to acknowledge my family for their ongoing support and encouragement. Special thanks go to my wife, Rasheedatu Yakubu, who took care of my two kids, Hierra Yasna Naaba and Junoor Muiz Naaba and kept my family running while I concentrated on this work. Your patience and unwavering support throughout this process has been invaluable. I also extent my sincere gratitude to my siblings including Mumuni Abdulai, Abdul-Rahim Abdulai, Abdul-Karim Abdulai and Naaba Rukaya who kept my family at home running while I was away from home. Special thanks to Daniela Richter for supporting me in the early part of my PhD. Finally, a big thank you to my friends and colleagues at the University of British Columbia who supported and encouraged me throughout this work. Sharing the journey with fellow PhD students made the process fun and I learned so much from each of you. xxiv Dedication This dissertation is dedicated to my parents, Mr. Naaba Abdulai and Mrs. Naaba Fanmara, who never had any formal education, yet taught me the importance of education and supported my pursuits from an early age. 1 Chapter 1: Introduction 1.1 Digital Health Technologies that connect, facilitate, and support collaborative work between healthcare providers and their patients have become increasingly common as healthcare has shifted towards patient-centered approaches that emphasise patient-provider relationships (Reynolds, 2020). The shift towards patient-centered care has given rise to the growth of new digital health technologies, including websites, mobile apps, and social networking sites (Guse et al., 2012). For instance, following the advent of smartphones, there has been a sudden increase in the use of mobile health applications for clinical care, public health, consumer health applications, and for biomedical research and administrative purposes (Torous et al., 2020). The increase in use of digital technologies for healthcare purposes is partly attributed to convenient and easy access to health information via such technologies, the ability to engage in online communication with healthcare providers, and the anonymity and confidentiality associated with online information-seeking (Millikan, 2001). Indeed, recent estimates indicate that approximately 70% of Canadians, 81% of adults in the United States, and 53% of Europeans rely on digital technologies for health-related information before visiting a health professional (Borcsa et al., 2021; Johnson, 2016; Wang et al., 2020). Many terms have been used to describe the use of technology in healthcare including health informatics, eHealth, mobile health (mHealth), precision health, and digital health. Digital health is a relatively recent term that is broadly defined as the use of information technologies in healthcare including mhealth, wearable devices, telehealth, health-related websites, and precision medicine (via electronic health records) (Fatehi et al., 2020). In this dissertation the terms digital 2 health technologies, digital health, and digital technologies will be used interchangeably to describe the use of web-based health technologies. A large amount of research suggests that digital health technologies are widely perceived to be useful (Reynolds, 2020). Despite the perceived usefulness of digital health, there are often patient-reported barriers that limit patient engagement with the healthcare system or in worse cases, result in adverse patient events (Hesketh et al., 2017). Such patient reported barriers are largely attributed to the absence of critical interface elements that can facilitate users’ engagement with digital health technologies (Alqahtani & Orji, 2019). Critical interface elements refers to the features and functions of a technology that must be present to achieve the targeted goal. For instance, a technology may be designed to enable patient-provider communication but the absence of critical interface elements that facilitate engagement may limit the ability of people to use such a technology. Therefore, the ability of digital health technologies to have sustained user-engagement depends on whether the relevant user-interface design requirements are met (Apted et al., 2009). 1.2 Design of Digital Health Technologies Digital health technology design that facilitates effective and seamless user-engagement has been identified as a major factor required for the widespread adoption of technology-based interventions in healthcare (Shidende & Mörtberg, 2014). Effective design becomes an imperative as patients and patient rights advocates are increasingly demanding technologies that create less frustration, reduce error rates, and improve the overall user-experience (Robinson et al., 2020). In response to this demand, user-centered design and design guidelines have arisen as 3 one of the best techniques to develop efficient, effective, and usable digital health technologies that facilitate patient engagement (Prince et al., 2019; Schnall et al., 2015). Unfortunately, current design principles and methods have been largely focused on effectiveness, efficiency, and functionality of digital health with less attention to emotional design and user-experience aspects. Therefore, even though available design principles may facilitate the development of functional and usable digital health technologies, such a technology may be counter-productive to its goals if it does not consider other fundamental issues necessary for promoting sustained user-engagement. An important area of consideration is the issue of stigma, which has been identified as a major barrier in the management and prevention of many health-related challenges, yet not often considered in health technology design (Vaes, 2019). For instance, patient anonymity, equity, and diversity of patient representation have been identified as some of the factors that can address sexual health-related stigma, but these factors are not usually considered in the design of sexual health-related technologies (Bauermeister et al., 2019). 1.3 Sexual Health-Related Stigma For this dissertation, the term sexual health-related conditions will be used to describe a group of conditions or clinical syndromes that are epidemiologically heterogeneous but generally affect an individual’s sexuality or sexual functioning and include sexually transmitted infections (STIs), sexual dysfunction, and infertility (WHO, 2010). Stigma has been identified as a major barrier in the management and prevention of sexual health-related conditions. Even though many biomedical and structural interventions have been developed for clinical and self-management of sexual health-related conditions and sexual disorders, the stigma associated with utilizing such 4 interventions often poses a barrier to their uptake (Lunny et al., 2014). Managing sexual health-related conditions may be more problematic for those who face a multiplicity of issues that a society generally considers unusual. For instance, people with a sexual health-related condition, who at the same time also identify as lesbian, gay, bisexual, transgender, queer, questioning, intersex, asexual, or Two Spirit (LGBTQIA/2S) may have intersecting stigmas that could intensify stigma compared to people who only live with a sexual health-related condition (Flanders et al., 2019; Magee et al., 2012). Available evidence suggests that sexual health-related stigma can lead to poor health and poor wellbeing, including mental health problems, social withdrawal, disclosure-associated anxiety (i.e., fear of inadvertently disclosing a condition), and reduced sexual wellbeing (Mill et al., 2009; Rueda et al., 2016). While many forms of stigma have been identified, prior research demonstrates that people affected by or living with sexual health-related conditions could experience two main types of stigma – public stigma and internalized stigma. Public stigma refers to the act of labeling, stereotyping, and discriminatory practices by others that view someone as a tainted individual, while internalized stigma refers to the acceptance of negative attributes and a reduced sense of self-worth because of the possession of an attribute that is considered abnormal or subnormal by a society (De Visser & O’Neill, 2013; Knight et al., 2013). Because most sexual health issues are not immediately visible to the public, people who have a sexual health issue often experience more internalized stigma than public stigma (Bauermeister et al., 2019). Therefore, because of the potential for anonymity and other features, digital technologies are seen as one of several non-traditional stigma-reduction strategies that may aid in addressing sexual health-related stigma (Hood & Friedman, 2011). 5 1.4 The Use of Digital Health Technologies in Addressing Sexual Health-Related Stigma The growth of new digital health technologies, such as websites, mobile apps, and social networking sites, has fundamentally changed the way health information is accessed, analyzed, and utilized (Guse et al., 2012). Redston et al. (2018) suggest that people are more likely to use technology-based interventions for sexual health problems that are perceived as embarrassing, stigmatising, and difficult to discuss face-to-face. The anonymity, accessibility, and confidentiality associated with the use of digital health technologies may reduce stigma and possibly mitigate the barriers to accessing sexual health interventions (Barman-Adhikari & Rice, 2011). Indeed, various technology-based interventions have been developed to address stigma by promoting virtual connections, improving peoples’ knowledge and coping skills, and eventually reducing or removing the need for people to attend sexual health clinics in-person (Chaudoir et al., 2012; Kerr et al., 2015). While digital technologies might be able to alleviate stigma by minimizing in-persons visits to sexual health clinics, current technologies are limited in their abilities to address the internalized feelings of stigma associated with sexual health-related conditions (Dudina & Tsareva, 2018). In some cases, digital health not only fails to address the stigma of sexual health issues but may also contain interface components that could inadvertently aggravate stigma among end-users (Shearer et al., 2012). The inability of digital health technologies to alleviate stigma or the possibility of provoking stigma points to a need to explore destigmatising design principles that might help address stigma during technology design. 6 1.5 Problem Statement and Objective As the influence of digital health technologies in everyday health practices continues to grow, design principles, methods, and frameworks that guide technology design also continue to evolve. However, despite stigma being a major issue in sexual health, current design principles that guide the creation of sexual health-related technologies do not adequately address the distinct element of stigma commonly experienced by people with sexual health-related conditions. Available design principles that focus on the functional aspects of a website fall short of critical interface elements that can alleviate stigma (Apted et al., 2009). Therefore, it is not known how sexual health-related technologies could be designed to address stigma. Thus, the objective of this dissertation was to develop design principles that would serve as a guide in designing destigmatising sexual health-related websites, and to determine how such principles might be applied. 1.6 Research Questions To realise the study objective, this dissertation is divided into four interrelated studies including: 1) A scoping review to identify the stigmatising and destigmatising properties of sexual health-related technologies; 2) A website evaluation study to examine patients’ perspectives of how destigmatising websites could be designed; 3) A Delphi study to develop a set of destigmatising design principles; and 4) A qualitative descriptive study to understand how software designers might apply the principles developed in the Delphi study. The following research questions were addressed in the respective studies: 7 • Study 1: What is the state of the literature that has examined stigma in relation to the design, impact, and ways digital health technologies alleviate or aggravate sexual health-related stigma? • Study 2: What are patient perspectives towards the usability of a sexual health-related website? and What are patients’ perspectives towards the design of destigmatising sexual health-related websites? • Study 3: What are key design principles to support the design of destigmatising sexual health-related web technologies? • Study 4: What are software designers’ perceptions of the applicability of the destigmatising design principles developed in Study 3? 1.7 Significance of the Study This research produced destigmatising design principles that might help inform the design and development of destigmatising sexual health-related technologies. Without explicit design principles that can address stigma via digital health technologies, healthcare teams, developers, researchers, and policymakers might create technologies that fail to reduce stigma, or worse, inadvertently provoke stigma (Glasgow et al., 2014). And digital health technologies that fail to challenge public perceptions of stigma will, at the very best, remain underutilized and, at worst, be abandoned by end-users (Granja et al., 2018; Kaplan & Harris-Salamone, 2009). 8 1.8 Relevance to Nursing Historically, the development of digital health technologies has been championed by software designers, human factors specialists, and computer scientists, for whom functionality, usability, and accessibility has been the dominant consideration (Hassenzahl, 2005; Kientz et al., 2010). Furthermore, most of the methods and techniques for designing digital technologies originated from large IT firms such as Apple Inc. and IBM Corp. And yet, while nurses and their patients are most often the ultimate beneficiaries of digital health technologies, nurses’ perspectives are not always considered in the design process (Dykes & Chu, 2021). For this reason, nursing researchers have increasingly expressed the need and advocated for nurses’ perspectives in the design, development, implementation, and use of healthcare technologies (Dykes & Chu, 2021). This is important because the perspectives of nurses are considered essential to developing digital health technologies that incorporate emotional, relational, and ethical values, as well as social justice as aspects of healthcare. This study moves away from the predominantly technology-centered approach to developing design principles by involving experts in the domain area of sexual health and stigma. By situating this study in the discipline of nursing, it can be argued that the final design principles reflect nursing values including human dignity, trauma-informed care, privacy, anonymity, autonomy and choice, sympathy, human relationships, and strength-based approaches, which are considered cardinal hallmarks of nursing care (Schmidt & McArthur, 2018). Furthermore, the principles that were developed have implications for the use of digital health technologies among nurses and patients. Finally, by having an awareness of the fundamental nursing values as contained in the final principles, software designers may learn a 9 crucial additional skill that can help in the design of digital health technologies that address health concerns on a broader level. 1.9 Structure of Dissertation This dissertation contains two literature reviews: Chapter 2 presents an overview of the literature on user-centered design, design principles, emotional design, and how design principles are developed. Chapter 3 then presents an overview of the literature on sexual health-related stigma and a trauma-informed care framework. The literature review chapters are followed by four chapters that present the four studies: Chapter 4, (Study 1) is a scoping review on the stigmatising and destigmatising properties of sexual health-related websites; Chapter 5, (Study 2) presents a usability evaluation of a sexual pain website using a trauma-informed care framework to examine usability and patient perspectives of how destigmatising websites might be designed; Chapter 6, (Study 3) is a Delphi study in which destigmatising design principles were developed in consultation with 15 experts in sexual health-related stigma and; Chapter 7, (Study 4) is a qualitative descriptive study involving 14 software designers to examine the applicability of the destigmatising design principles developed in Study 3. Each of the four studies includes a background, a description of the methods, the results, as well as discussion. Chapter 8, which is the final chapter, includes an overall discussion of the findings, implications and recommendations for nursing practice, nursing policy, nursing research and human computer interaction research, as well as overall study limitations, and directions for future research. 10 Chapter 2: Literature Review: User-Centered Design, Design Principles & Emotional Design 2.1 Introduction In this chapter, an overview of the various aspects of user-interface design that underpins this research is presented. The chapter includes a brief introduction to user-interface design, an overview of user-centered design and design principles, as well as an introduction to emotional design and digital health technologies. The chapter concludes by providing an overview of how user-interface design principles are developed. 2.2 What is Design? The concept of design can have different meanings depending on the context and the discipline in which it is being used. For instance, the concept of design has been used differently in fields of engineering, fashion, architecture, and most recently, in human-computer interaction (Dam & Siang, 2020). While design can have a different meaning in different contexts, the concept generally refers to the act of finding creative solutions to problems in the target domain. In this dissertation research, design is used in the context of human-computer interaction, which refers to the act of finding creative and reasonable solutions in computerized technologies in a way that facilitates a seamless interaction between humans and computers (Shamat et al., 2017). The need to find reasonable solutions to digital health technologies has gained increasing acceptance as healthcare moves towards patient-centered approaches that emphasize patient-provider partnerships (Reynolds, 2020). While design may sound simple, how to get a reasonable 11 solution in computerized interfaces that satisfies the needs of the user is often problematic in the healthcare technology arena. Sometimes, in finding solutions to complex design problems, other issues arise, especially when patient needs are not being recognized or cannot be accommodated into the design of digital health technologies. This is common in healthcare technologies as human needs in such technologies move beyond the technical or functional requirements to include user-experience and non-functional aspects of computer use (Mburu et al., 2013). In an effort to accommodate user needs, user-centered design and design principles arose as popular frameworks to help design teams find reasonable solutions that satisfy the needs of the user. In the context of this dissertation, the user is the client or patient who is the recipient of care. 2.3 User-Centered Design According to Donald Norman, if anything is sacred to people in the field of Human-Computer Interaction and user-interface design, then it is the idea of “know the user” (Norman, 2005). After all, it would be difficult to develop a meaningful product without a detailed knowledge and understanding of the user population. The term user-centered design originated from the Donald Norman Laboratory at the University of California, San Diego in the 1980s (Anderson et al., 1988). Since then, the concept of user-centered design has become an internationally endorsed best practice (defined in ISO13407) that places the user at the forefront of the development process. User-centered design is a multidisciplinary design approach that emphasizes the active involvement of the user at all stages of the design process, from research, idea generation, task requirements, and to final product development and evaluation. The role of the designer is typically to facilitate the user’s task and ensure that the user can use the product 12 with minimum effort. Since its inception, user-centered design has, and continues to be adopted as a popular method for designing digital health technologies. As healthcare moves closer to the ideal of patient-centered care, the design of healthcare technologies is also moving from system-centered to user-centered design approaches (Reynolds, 2020). However, the application of a user-centered design framework mostly emphasizes product usability and usefulness – two concepts that are considered the key to overcoming the problems inherent in the hitherto systems-centered design. Despite the popularity of user-centered design approaches, the literature shows that the method has not been optimally applied in product design (Mao et al., 2008; Norman, 2005). Overcoming designers’ assumptions, coordinating the timing of user-involvement, and managing heterogeneity among stakeholders are some of the commonly reported problems in user-centered design (Cornet et al., 2020). These challenges may be more pronounced when the technology is being developed for people with health-related challenges as they are often perceived to be vulnerable, or their ill-health makes it difficult for them to be involved. In such cases, user engagement in user-centered design may just be limited to one or a few phases. Indeed, a systematic review of 325 patient decision aids and other personal health technologies revealed that patient involvement in designing these technologies were largely limited to one stage of the development process (i.e., either understanding users, developing the prototype, refining the prototype, or user testing) (Vaisson et al., 2021). Based on these and other related challenges, some authors argue that the practice of user-centered design needs fine-tuning to provide a more structured approach that guides user-interface design (Mao et al., 2008). 13 2.4 Design Principles Given the limitations of user-centered design, structured metrics popularly known as design principles (or design heuristics) have been developed to complement user-centered design approaches. A design principle is generally described as a “rule of thumb” or “universally applicable law, rule, or consideration” that is used as a foundation for a pleasant design (Interaction Design Foundation, 2020). For instance, Norman argued that simply asking designers to apply user-centered design is insufficient in realizing a meaningful product. He thus advocated for and suggested seven design principles that are essential for facilitating the work of the designer (Robinson & Norman, 1994) including: 1) Use both knowledge in world and knowledge in the head; 2) Simplify task structures; 3) Make things visible; 4) Get the mapping right (User mental model = Conceptual model = Designed model); 5) Convert constraints into advantages (Physical constraints, Cultural constraints, Technological constraints); 6) Design for Error; 7) When all else fails − Standardize. In 1987, Ben Shneiderman also articulated a list of design principles, which later became known as the 8 golden rules (Shneiderman, 1987). Jakob Nielsen also developed a set of 10 design principles, which became popularly known as the Nielsen’s heuristics (Nielsen & Molich, 1990). Other design principles, such as the seven inclusive design principles, were later developed to cater mostly to the needs of people with disabilities in using everyday products (Jordan, 2000). Among all the design principles, Shneiderman’s 8 golden rules, Nielsen’s 10 usability heuristics, and universal design principles are considered the most common and predominantly applied set of principles. These three sets of principles are considered traditional design principles, and they are mostly used as a reference guide in the design and evaluation of digital 14 technologies (Masip et al., 2011). However, traditional design principles have been criticized as being too general, technically focused, and may therefore not meet the needs of modern-day health technologies (Shaw et al., 2018). As shown in Tables 2.1 and 2.2, most of the design principles are quite technical in nature and are focused on improving the functions of technologies. However, as technologies continue to evolve, more specialised design principles continue to be developed to complement/supplement traditional design principles (Blank & Kosuru, 2020; Quiñones & Rusu, 2017; Sanz et al., 2016). Even though additional sets of design principles continue to be developed including Universal Design Principles displayed in Table 2.3 (Story et al., 1998), to the best of my knowledge, no design principles exist to address the emotional and experiential aspects of people living with sensitive health topics such as sexual health issues. Meanwhile, people with such conditions are constantly faced with emotional and personal issues such as stigma. Table 2.1 Nielsen’s Heuristics # Principle Definition 1 Visibility of system status “..keep users informed feedback within reasonable time.” 2 Match between system and the real world “The system should speak the user’s language with words and concepts familiar to the user, rather than system-oriented terms.” 3 User control and freedom “Users need a clearly marked exit. Support undo and redo” 4 Consistency and standards “… words, situations, or actions [should] mean the same thing.” 5 Error prevention “… careful design can prevent a problem from occurring” 6 Recognition rather than recall “Make objects, actions, and options visible.” 7 Flexibility and efficiency of use “Allow users to tailor frequent actions” 8 Aesthetic and minimalist design “Every extra unit of information in a dialogue competes with the relevant units of information and diminishes its relative visibility” 9 Help users recognize, diagnose and recover from errors. “Error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution” 10 Help and documentation “…documentation may be necessary to provide help...” 15 Table 2.2 Shneiderman’s 8 Golden Rules # Principles Definition 1 Strive for consistency “The interface should have consistent sequence of actions in similar situations” 2 Enable frequent users to use shortcuts A system/ user interface should contain some abbreviations, hidden commands, short-cuts for users who want to reduce the number of interactions 3 Offer informative feedback There should be some system feedback for every operation performed on an interface 4 Design dialogue to yield closure Sequence of actions should be organized with a beginning, middle, and an end 5 Offer simple error handling Design the system so serious errors cannot be made. And if an error is made, the system to offer mechanisms for managing the error. 6 Permit easy reversal of actions The system should support a user to undo an action 7 Support internal locus of control Users should be initiators of actions, rather than responders. This makes people feel they are in control of the system. 8 Reduce short term memory load. Every extra information on an interface competes with the relevant unit of information. Use simple displays, multiple pages should be consolidated Table 2.3 Universal Design Principles # Principles Definition 1 Equitable use The design is useful and marketable to people with diverse abilities. 2 Flexibility in use The design accommodates a wide range of individual preferences and abilities. 3 Simple and intuitive use Use of the design is easy to understand, regardless of the user's experience, knowledge, language skills, or current concentration level. 4 Perceptible information The design communicates necessary information effectively to the user, regardless of ambient conditions or the user's sensory abilities. 5 Tolerance for error The design minimizes hazards and the adverse consequences of accidental or unintended actions. 6 Low physical effort The design can be used efficiently and comfortably and with a minimum of fatigue. 7 Size and space for approach Appropriate size and space is provided for approach, reach, manipulation, and use regardless of user's body size, posture, or mobility. 16 2.5 Emotional Design and Healthcare Technologies For some time now, human-computer interaction research has followed the functionalist paradigm, which defines design activities and information systems as objective and, thus, are unrelated to subjective feelings and emotions (Batchelor & Bobrowicz, 2014). Following the advent of user-centered design, Jordan proposed the Hierarchy of Customer Needs that deals with design beyond the functional and utilitarian aspect of products (Jordan, 2002). According to Jordan, functionality is a pre-requisite but not enough to promote overall user-experience and the success of a product. In addition to usability and functionality, people need products that deliver emotional benefits while helping them overcome various obstacles to care. Norman also proposed emotional design as the best strategy to complement the usability and functionality aspects of products (Norman, 2004). Indeed, Norman argues that the emotional aspects of products may be more critical to a product’s success than its practical elements. This is especially important in healthcare where experiential and emotional factors such as stigma often serve as a major obstacle to the management and prevention of health-related challenges (Quinn & Chaudoir, 2009). Despite the recognition of emotional design as being important, this aspect of design has not been adequately taken up in the development of digital health technologies. Instead, digital health technology design continues to follow the functionalist paradigm, for which technical and functional considerations are paramount. For instance, a systematic review of digital mental health technologies used in the Chinse context was largely focused on usability, effectiveness, acceptability, and safety considerations (Zhang et al., 2021). Other equally important components, such as stigma that is often associated with mental health challenges, were not 17 addressed in the reviewed studies. This reinforces that a functionalist approach is taken to designing health technologies. It is also interesting to note that of the 29 studies reviewed in that manuscript, only one study adopted a user-centered design approach involving both patients with mental health issues and healthcare professionals. Similarly, another scoping review on designing, implementing, and evaluating mHealth technologies for chronic disease management identified usability, and functionality as the recommended design features for successful implementation of mobile health technologies (Matthew-Maich et al., 2016). As in the previous review, this review did not consider user-experience or emotional aspects of product design. Another systematic review of 35 studies on mobile health technologies designed to remediate unhealthy lifestyles and support chronic disease prevention programs identified seven metrics to promote user engagement including aesthetics, navigation, personalization, message presentations, reinforcement, communication, and credibility (Wei et al., 2020). While these seven design features could, to some extent, promote user engagement, they were largely technology focused. Emotional aspects such as stigma were left unaddressed in the studies reviewed despite their influence in managing health-related challenges. As can be observed, these reviews have one thing in common. They all seem to address user-needs of digital technologies from a functionality perspective. While the functionality and usability of technologies are, without a doubt, essential attributes in software design, they are insufficient in promoting the overall user-experience and addressing emotional needs of patients in digital technologies. The issue of how to address intangible emotional design components such as stigma in digital health technologies remains underexplored in health technology design. 18 Meanwhile, for some products such as sexual and mental health-related technologies, the emotional elements may be just as critical to a product’s success as its functional components. As human-computer interaction research and user-interface design gradually moves towards more user-experience design, a better understanding is needed in investigating the emotional and user experience aspects of technologies, rather than only the functional and utilitarian aspects of digital health tools (Vermeeren et al., 2016). That is, further to the various design principles, additional design principles that address emotional aspects such as stigma are needed in technologies designed to support sexual health. 2.6 How Design Principles are Developed To date, there is no specific agreed upon method for developing design principles. Existing design principles were developed from methods ranging from experts’ personal experiences in user-interface design, adaptation of previous principles, and through other newly developed methods (Quiñones et al., 2018; Quiñones & Rusu, 2017). For instance, Shneiderman’s 8 golden rules and Nielsen heuristics were developed largely based on personal experiences of Ben Shneiderman and Jakob Nielsen respectively (Nielsen, 1994a; Shneiderman, 1987). However, Nielsen later improved his heuristics through a study that used factor analysis of 249 common usability problems resulting in a list of 10 heuristics that best explained actual usability problems (see the 10 heuristics in Table 2.1) (Nielsen, 1994a). Following the development of the two sets of traditional design principles, Quiñones & Rusu (2017) suggested other processes for developing design principles. These processes include: 1) A combination of experience and adaptation of previous principles; 2) A review of 19 the literature on the domain field as well as existing heuristics; 3) Identifying and analyzing domain-specific usability problems through usability evaluation; 4) Interviews; 5) Design standards and recommendations from previous studies; and 6) A combination of literature reviews, interviews, usability evaluations and recommendations from previous studies. Quiñones, Rusu, & Rusu, (2018) later extended their methodology to include validation of design principles via a number of methods including expert review and user testing. Other design principles have been developed by adapting existing ones. For instance, Nielsen’s heuristics have been adapted for the design of domain-specific technologies including, robotics, video games, instructional design, persuasive technologies, and humorous products (Baker et al., 2002; Ko et al., 2013; York & Ertmer, 2011; Yu & Nam, 2014). One thing common to these design principles is that their development was championed by technology and human-computer interaction experts. People with domain knowledge are rarely involved in the development and application of design principles. The technology-centered nature of design principles calls for different approaches that involve people with domain expertise in developing any new set of design principles. 2.7 Summary and Conclusion This chapter provided a definition of design and how design is applied in digital health technologies. The literature revealed that although several design principles already exist to guide the design and application of digital health technologies, such principles are considered too general, technically focussed, and do not consider the emotional and user-experience aspects of 20 end-users. Furthermore, the literature shows that design principles are largely developed from a technology-centered perspective – with little focus on domain expertise. 21 Chapter 3: Literature Review: Stigma, Sexual Health-Related Stigma & Trauma-Informed Care 3.1 Introduction In this chapter, an overview of stigma, the various forms of stigma, and sexual health-related stigma is presented. The various stigma management strategies used in the context of sexual health are also outlined. The chapter then concludes by providing an overview of a trauma-informed care framework, the relationship between trauma-informed care and stigma, and why this framework was adopted for this dissertation research. 3.2 Stigma The term stigma dates to the Greeks, where the skin of criminals, slaves, and people considered as traitors were branded or tattooed with inscriptions for easy identification as tainted and immoral people (Goffman, 1963). As the term evolved1 over the years, stigma has come to mean more than a small physical mark but is now considered an attribute or characteristic and a dynamic social process characterized by widespread social disapproval, blame, rejection, devaluation, and segregation (Scambler, 2009). Goffman, the first person to publish a conceptualization of stigma in the modern context, described stigma as “an attribute that is deeply discrediting and reduces a person from a whole and a usual person to a tainted and 1 From https://www.etymonline.com/word/stigma: “stigma (n.), 1) 1590s (earlier stigme, c. 1400), "mark made on skin by burning with a hot iron," from Latin stigma (plural stigmata), rom Greek stigma (genitive stigmatos) "mark of a pointed instrument, puncture, tattoo-mark, brand," from root of stizein "to mark, tattoo," from Proto-Indo-European language (PIE) root *steig- "to stick; pointed" (see stick (v.)). 2) Figurative meaning "a mark of disgrace" in English is from 1610s. Stigmas "marks resembling the wounds on the body of Christ, appearing supernaturally on the bodies of the devout" is from 1630s; earlier stigmate (late 14c.), from Latin stigmata.” 22 discredited one” (Goffman, 1963). Link and Phelan (2001) extended Goffman’s conceptualization of stigma by proposing a sociological definition of stigma as the act of labeling, stereotyping, discrimination, and status loss within a context where power is exercised. According to Link and Phelan (2001), stigma occurs when the following elements converge: First, people differentiate and label human differences. Second, dominant cultural beliefs and societal norms link the labeled person to undesirable social characteristics. Third, the labeled person is placed in a category that is distinct from people with supposedly “normal” social characteristics. And finally, being in a distinct category from the “normal” may lead to feelings of shame, discrimination, status loss, and insecurity. The sociological model of stigma stipulates that stigma is not merely an act of possessing a discredited identity but is also a socially constructed category of people who have an attribute that society considers unusual. Put another way, people are not just stigmatised because of the possession of an attribute but because they are categorized into a specific social group with similar “unusual” characteristics. As researchers tend to apply stigma to broadly related concepts, it is essential to distinguish between stigma and other closely related concepts including prejudice and discrimination. Although stigma, prejudice, and discrimination are often used interchangeably, the three concepts have different focus areas, and the differences between stigma, prejudice, and discrimination has more to do with the subject of interest rather than real conceptual differences (Stuber et al., 2008). Stigma describes a negative stereotype about a person with an attribute, while discrimination is unfair treatment that is attributed to people with the negative attribute (Canadian Mental Health Association, 2020). Prejudice, on the other hand, is a negative stereotype about a person based on perceived group membership (Center for Addiction and 23 Mental Health, 2020). Stigma research has traditionally focused on people with non-normative or unusual conditions, including physical disfigurements, sexual health-related conditions, and mental illness. On the other hand, research on prejudice and discrimination has often focused on the ordinary but often powerful implications of gender, race, and class divisions (Stuber et al., 2008). Although stigma, discrimination, and prejudice may concurrently affect people living with sexual health-related conditions, it is beyond the focus of this dissertation to determine how discrimination and prejudice impact the lives of people living with sexual health-related stigma. Rather, this research focuses on stigma and de-stigmatization in the context of digital health technologies. 3.2.1 Types or Forms of Stigma There have been several attempts to provide a comprehensive taxonomy of stigma dating back to Goffman’s 1963 classification. Goffman classified stigma as deformities or abnormal physical marks of the body, blemishes of the character, or an association with tribal or ethnic identity. Goffman, (1974) later classified stigma as either having a discreditable identity (the possession of a stigmatising quality that has not yet been revealed) or a discredited identity (the possession of outward attributes that have already been socially judged). Towler & Schneider, (2005) also established a set of stigma categories within which various stigma identities can be classified. These categories include physical disabilities, mental disorders, sexual conditions or sexual disorders, and gender/racial identities. More recently, Pryor and Reeder (2015) developed a classification of stigma as four dynamically interrelated manifestations of “untoward” or abnormal physical and social 24 characteristics that occur (covertly or overtly) at the individual, interpersonal, institutional, and societal levels. The four forms of stigma in the Pryor and Reeder model include: public stigma, self-stigma, stigma-by-association, and institutional stigma. Public stigma represents the cognitive, affective, and behavioural reaction of people to someone who has or is perceived to have a discrediting mark or attribute (Pryor & Reeder, 2015). Public stigma, also called enacted stigma, is often triggered by a visible or a known label such as a physical deformity, or a known sexually transmitted infection (STI) status, or a sexual disorder. Such labels can manifest as an implicit or explicit negative reaction towards people who possess those attributes. Pryor and colleagues stipulated that public stigma is at the core of all types of stigmas, while Bos, Pryor, Reeder, & Stutterheim, (2013) noted that public stigma informs most stigma-related research. The second form of stigma in Pryor and Reeder’s model is self-stigma, which is thought to result from public stigma and usually occurs due to the social and psychological impact of negative reactions from the public or the possessing of a stigmatising attribute. According Herek (2007), self-stigma can either manifest as felt stigma (the experience or anticipation of stigma due to the possession of a stigmatising attribute) or internalized stigma (shame, depersonalization, the reduction of self-worth, and psychological distress associated with the fears of being exposed). Self-stigma largely depends on the extent to which the stigmatising attribute is exposed to or concealed from the public. People who live with stigmatising conditions related to sexual health may be conscious of an impending negative reaction from the public. In their article, Pinel & Bosson, (2013) suggest that internalized stigma in particular, could impose on people a sense of impending overt or subtle negative comments or treatment by others. They termed it “stigma consciousness” and indicated that people with high levels of 25 stigma consciousness tend to have higher perceived levels of stigma than those with low levels of stigma consciousness. Other stigma researchers describe self-stigma, felt stigma, and internalized stigma in ways that differ from Pryor and Reeder’s classification. These different classifications are provided in subsequent paragraphs in this section. The third form of stigma in Pryor and Reeder’s classification is stigma-by-association. This occurs when people are stigmatised by their meaningful or arbitrary association with someone with a stigmatising condition or attribute. The fourth form of stigma in the model – structural stigma – occurs when societal ideologies and institutional norms tend to create, perpetuate, or exacerbate stigma (Corrigan & Lam, 2013). While public stigma, self-stigma, and stigma-by-association can result from individual perceptions about what is abnormal, structural stigma is entirely contingent upon the social, economic, and political power imbalances and pre-existing inequities among people in society. In addition to Pryor and Reeder’s (2015) classification, several researchers and organizations have categorized stigma in ways that reflect or differ from Pryor and Reeder’s classification. For instance, stigma is widely categorized as felt/self-stigma, perceived stigma, anticipated stigma, public/enacted stigma, stigma-by-association, and structural stigma (Canadian Mental Health Association, 2016; Earnshaw et al., 2012; Fuster-Ruizdeapodaca et al., 2014; Sheehan et al., 2016). From these perspectives, the first and foremost form of stigma is felt/self-stigma. Felt stigma or self-stigma occurs when people feel embarrassed or ashamed of themselves or when people personally accept stigma as part of their self-concept. Perceived stigma also occurs when people think they will be stigmatised by possessing a discredited attribute even though they may not necessarily be stigmatised. On the other hand, anticipated 26 stigma occurs when people believe that others will eventually direct stigma at them in the future (Quinn & Chaudoir, 2009). These forms of stigma together constitute what is termed internalized stigma and mainly occur at the personal and interpersonal level (Earnshaw et al., 2018). The experiences of felt, perceived, and anticipated stigma suggest that people are not just passive recipients of stigma but active perceivers who interpret, cope with, and respond to stigma in various ways. This classification differs from Pryor and Reeder’s classification in that Pryor and Reeder differentiated self-stigma from felt stigma and considered anticipated stigma as part of self-stigma. Other forms of stigma that may occur at the community, institutional, and societal level include enacted stigma, stigma-by-association, and structural stigma. These three are collectively classified as public stigma, which some refer to as enacted stigma or external stigma (Earnshaw et al., 2018). Although there are varying classifications of stigma, public stigma and internalized stigma represent the most common forms of stigma reported by people with sexual health-related conditions (Masuch et al., 2019). The varying classifications of stigma suggest that the use of the concept is not specific to one condition but varies across time, place, and discipline. The different classifications also suggest that there may be more nuanced conceptualizations of stigma than previously thought. Indeed, classifying stigma in various forms is undoubtedly valuable for understanding how this broad concept is applied across disciplines. More importantly, the different forms of stigma have given room for multi-disciplinary collaboration in stigma research – which has recently witnessed significant contributions from psychology, sociology, and healthcare. 27 3.2.2 Stigma as a Communication Attribute Aside from the various classifications of stigma, stigma could also result from societal discourse and communication strategies that are used to convey issues of sexual health, such as language or commonly used terminologies that dictate the narratives surrounding a particular condition (Meisenbach, 2010). These communication cues are mostly used by society to distinguish people with a different identity, stigmatise that identity, categorize stigmatised people into a separate group, implicate the stigmatised individual as being responsible for the stigmatised identity, and finally, subject the stigmatised individual to social and emotional anguish (Smith, 2007). These attributes emphasize the social discourse of stigma and the important role that communication cues play in classifying people as either normal or abnormal. However, despite the role that societal discourse and communication strategies play in forming stigma, stigma research had not adopted a communication approach until Smith, (2007) conceptualized stigma as also including four distinct communication attributes: These include: 1) Distinguishing people; 2) Categorizing the distinguished individuals as a separate group; 3) Taking responsibility for the distinguished group; and 4) Linking the distinguished group to physical and social peril. Meisenbach, (2010) later built on Smith’s attributes to propose a model of stigma management communication by offering three ways of how stigma operates from a communication perspective. According to Meisenbach, (2010), stigma is first, discursively constructed from the perspective of the stigmatised and non-stigmatised groups. This discursive construction involves the creation of perceptions through talk and seeks to address how and who should be designated with a stigmatising attribute (Meisenbach, 2010). Second, discourses and 28 material conditions determine what constitutes stigma at any point in time – suggesting that stigma is not permanent but changes depending on the discourse in a specified context. Finally, stigma varies in depth and degree depending on the nature of the stigmatising attribute and how it is perceived by the affected person or people. The different classifications of stigma also suggest that there may be varying definitions of stigma and different foci of stigma research within specific disciplines. For instance, stigma definitions and stigma research in healthcare tend to have (slightly) different foci depending on the type of health condition and the degree of its concealment. Therefore, the definition, research focus, and stigma management strategies in persons with sexual health-related conditions may differ from that of those with stigma related to mental illness or disabilities. A domain-specific stigma category will not only provide a clear research focus but also offer stigma management strategies specific to that field. Having clarified the different forms of stigma in healthcare, this review is focused on sexual health-related stigma. 3.3 Sexual Health-Related Stigma Sexual health-related stigma is defined as “the negative regard, inferior status, and relative powerlessness that society collectively accords to people with any sexual health-related condition including STIs, (e.g., HIV/AIDS, genital herpes, gonorrhea, and chlamydia) as well as disorders that affect sexual health functioning (e.g., endometriosis, infertility, sexual dysfunctions) (Herek, 2007). Sexual health-related stigma presents a fundamental challenge for individuals and their families (WHO, 2015). While stigma affects all groups of people with a sexual health-related condition, Logie and colleagues suggest that stigma is worse among people 29 who simultaneously identify as LGBTQIA/2S and live with sexual health-related conditions compared to those who have a sexual health-related condition and identify as cis-gendered (Logie et al., 2020). The degree to which a condition is stigmatised may be related to several factors including its cultural origin, the ability to hide or conceal it, and the extent to which the condition might affect others (Breitkopf, 2004). In that sense, sexual health-related stigma has over the years been shaped by prevailing cultural norms, religious beliefs, and in recent times, the media. Such beliefs tend to influence public perceptions regarding how such conditions are acquired or transmitted (Hood & Friedman, 2011). Whereas cultural perceptions of transmission may not apply to conditions such as endometriosis or painful sex, dominant cultural norms tend to link certain reproductive health issues to promiscuity, infidelity, and homosexuality (Bonnington et al., 2017). These traits, which are historically considered immoral behaviours, are perceived as the most common – and in some settings the only – way of contracting STIs (Ross et al., 2014). The historical trend of associating promiscuity, homosexuality, and infidelity with sexual health-related conditions tends to influence stigma because these behaviours are considered culturally taboo ways of engaging in acts of sexual intimacy (Shearer et al., 2012). In this sense, STIs or other sexual health-related conditions may be regarded as punishment for immoral sexual behaviour in which stigma may be used as a tool to enforce prudent behaviours. The perception that sexual health-related conditions are primarily acquired during voluntary sexual activities strengthens the belief that people affected by conditions related to sexual health are largely personally responsible for their condition – a sentiment that can be deeply discrediting and stigmatising (Herek, 2002). The tendency to blame people for sexual 30 health-related conditions makes such conditions an individual problem, thereby absolving society and the community from taking responsibility for the creation and perpetuation of stigma (Fuster-Ruizdeapodaca et al., 2014). 3.3.1 Forms of Sexual Health-Related Stigma The attributes of sexual health-related stigma correspond to the various classifications of stigma as outlined previously Turan et al. (2017) categorized sexual health-related stigma as experiencing stigma (overt discriminatory practices), perceived stigma (how individuals think others think about them), anticipated stigma (expectations of future discriminatory practices), and self-stigma (the acceptance of negative societal values leading to reduced self-worth, and feeling of guilt and shame). While people with sexual health-related conditions may be exposed to all forms of stigma, internalized stigma remains the most common, and in some cases, the most difficult form of stigma to be addressed (Turan et al., 2017). As most sexual health-related conditions are primarily hidden from the public, people with such conditions may harbour feelings of anxiety, depression, isolation, and hopelessness in anticipation of a possible (accidental) disclosure of status. This is particularly possible in people with conditions that will eventually manifest with physical symptoms that may be difficult to conceal from sexual partners, healthcare providers, and in some cases, the public. In such cases, the only possible way to obviate a possible exposure may be to abstain from having sexual partners or refrain from seeking testing and treatment. While the concealment of symptoms may – in the meantime – prevent disclosure of the disorder, internalized feelings of stigma may still 31 affect the life and wellbeing of the person. This may not only aggravate internalized stigma but also impede recovery, curtail prevention efforts, and increase feelings of guilt. In their article, Herek & McLemore, (2013) examined self-stigma and its impact on the psychological wellbeing of 197 people living with HIV/AIDS. They found that higher levels of self-stigma and felt stigma were significantly correlated with poor mental wellbeing. In another study, Chaudoir, Earnshaw, & Andel, (2013) explored the impact of internalized or concealed stigma versus visible or public stigma. They demonstrated how internalized stigma could “get into the skin” and create health disparities among labeled categories and “normal” people. These studies suggest that even though sexual health-related conditions may be concealed from the public, the internalized stigma associated with hidden conditions could worsen the health of people compared to those with visible conditions. Contrary to this assertion, it is suggested that people with internalized stigma would likely have a comparative advantage in stigma-related negative reactions than those with visibly stigmatising characteristics (i.e., public stigma) (Meisenbach, 2010). Even though this may be true for people with some conditions in certain situations, Goffman argued that this may not always be the case because people with internalized stigma “must necessarily pay a high psychological price, [and have] a very high level of anxiety, in living a life that can be collapsed at any moment” (Goffman, 1963, p87). With this in mind, it is important to recognize that internalized stigma deserves at least the same amount of attention as that of public stigma (e.g., physical disability) among intervention developers. Yet, many stigma-related studies are solely focused on public stigma. 32 3.4 Stigma Reduction Strategies in Sexual Health Different stigma management strategies have been proposed by researchers in the different domains who work in stigma-related research. However, these strategies are not mutually exclusive to each field but could be applied to other areas. A substantial amount of research has been conducted by researchers in behavioural and social science, social psychology, health communications, and healthcare research into providing recommended stigma reduction strategies for various stigmatised conditions including sexual health-related issues (Corrigan & Kosyluk, 2013; Friedman et al., 2014; Hood & Friedman, 2011; Rocha-Jiménez et al., 2018). In addition to the stigma reduction strategies in sexual health, stigma reduction strategies in other disciplines including mental health and disability stigma could reasonably be applied to address sexual health-related stigma (Chang & Bazarova, 2016; Kondrat & Teater, 2009; National Academies of Sciences, 2016; Nayar et al., 2014). The kind of stigma reduction strategy largely depends on the extent of concealment of the stigmatising attribute but typically ranges from interventions at the self/intrapersonal level, interpersonal level, community level, institutional, and the societal/governmental level (Stangl et al., 2013). The strategies outlined in this review represent the literature on the current state of the science in addressing sexual health-related stigma. However, to the best of my knowledge, the effectiveness of these strategies in alleviating stigma has not been empirically tested at the time of writing this dissertation. 3.4.1 Interventions at the Individual or Intrapersonal Level Early stigma theorists, including Goffman, tend to conceptualize stigma from an individual perspective (Goffman, 1963). Consequently, many stigma researchers have proposed 33 several stigma reduction strategies that focus on individual behaviour change (Heijnders & van der Meij, 2006). Therefore, even though stigma management interventions at the community and societal level have also seen a recent increase, available evidence suggests that individual-level stigma reduction strategies remain the most commonly adopted (Stangl et al., 2013). Even though Goffman did not claim to have articulated a comprehensive list of stigma management strategies, his strategies encompassed recommendations from previous studies and include socio-psychological strategies that seek to accept, reduce, deny, and even avoid the stigmatising attribute (Goffman, 1963). Some of the strategies recommended by Goffman include: 1) Directly protesting the stigmatising attribute; 2) Seeing stigma as a blessing rather than as a curse; 3) Keeping stigma from “looming large” by minimizing the display of the stigmatising attribute; 4) Leveraging stigma for secondary gains; 5) Engaging in a “sympathetic re-education of the normal”; 7) Intentionally disclosing one’s stigmatising attribute; and 8) Redirecting the burden of stigma on the non-stigmatised. Historically, people with a stigmatising attribute tend to use some individual strategies in addressing stigma. These strategies may include isolating oneself, passing for normal (the attempt to hide stigmatising quality completely), discussing marginalized social statuses among persons with shared stigma attributes, and oftentimes, challenging or protesting stigmatising behaviour, or ignoring the stigma (Heijnders & Van Der Meij, 2006). However, the application of these stigma management strategies largely depends on the type of stigma and the level of concealment. For instance, a protest may not be an appropriate stigma management strategy for someone with a genital herpes infection or sexual pain because the protest could expose such people to public derision and place them as a separate group. In such an instance, passing for 34 normal, or hiding the stigmatised attribute would likely be the best possible stigma management strategy, even though these strategies would likely intensify the anxiety and pressure of internalized stigma (Goffman, 1963). In addition to Goffman’s strategies, other researchers have recommended strategies that focus largely on the individual. Some of these strategies are focused on changing characteristics of individuals, including attitudes, knowledge, empowerment, boosting self-esteem, and enhancing personal coping skills (Turan et al., 2017). Indeed, many stigma researchers have stipulated that improving the attitudes, education, and providing people with accurate information have the potential to destigmatise and normalize sexual health conditions before, during, and after diagnosis (Hood & Friedman, 2011) Despite the usefulness of these stigma management strategies, they are mostly unorganized, partial, and typically seek to manage stigma by solely relying on individual decision-making and personal responsibility (Pachankis et al., 2015). These strategies seem to suggest that people living with sexual health-related conditions are personally responsible for their problems (Pachankis et al., 2015). Interestingly, some stigma reduction strategies at the individual level have ironically employed stigmatising messages or images, and in some cases, have used negative campaign tactics to coerce people to adopt less risky sexual behaviours (Hood & Friedman, 2011). Rather than alleviating stigma, such destigmatising strategies have inadvertently perpetuated and reinforced both stigma and the culture of blaming individuals for their sexual health problems. Furthermore, while individual-level stigma management strategies could provide insight on how to improve the wellbeing of people living with sexual health-related conditions, they do not adequately address community-level factors and societal norms or ideologies that produce and/or perpetuate stigma and shape individual attitudes and behaviours. 35 As Heijnders (2006) argued, the inability of individual-level interventions to effectively alleviate stigma could be attributed to the fact that stigma is produced and reinforced by society, and as such, cannot be easily overcome by interventions at the individual level alone. This calls into question the efficacy of the current individual-level strategies to wholly reduce and sustain a long-term reduction of stigma at individual, community, and societal levels. Moreover, as technology-based modalities are increasingly being used for sexual health management, how destigmatising management strategies could be integrated into such platforms remains unknown. 3.4.2 Interventions at the Community and Societal Level While individual-level interventions are needed to minimize the impact of internalized stigma, the very source that produces and perpetuates stigma must equally be tackled to break the stigma cycle completely. Stigma management strategies at the community level typically focus on increasing knowledge related to the condition of interest, as well as addressing stigma within a specific community setting (Heijnders & van der Meij, 2006). The most commonly adopted stigma reduction strategy at the community level involves education – usually through conventional onsite campaign programs and mass media campaigns (Corrigan & Kosyluk, 2013). The conveyance of sexual health information via mass media and onsite campaign programs directly influences how the public views sexual health (Corrigan & Kosyluk, 2013). While education is instrumental in dispelling the myths and misconceptions that fuel sexual health-related stigma, the way such educational messages are crafted and conveyed to the public is oftentimes stigmatising (Smith, 2007). In many cases, the messages tend to reinforce the perception that those who are affected by sexual health conditions are not only responsible for 36 their state but might be treated with disdain and isolation (Smith, 2007). It is also important to note that the way sexual health-related messages are conveyed is in significant contrast to how chronic disease conditions such as asthma and heart disease are communicated (Smith, 2007). This contrast suggests that sexual health-related stigma could be shaped by the attitude of those charged with educating the public as well as the overall environment in which prevention efforts are supposed to take place. This could partly account for the reason why many people with sexual health-related conditions fail to utilize conventional sexual health-related services. It is also necessary to note that the way onsite educational strategies are delivered could differ from how they are delivered on web-based platforms. Therefore, to ensure web-based platforms achieve the same or better stigma reduction as that of the conventional activities, special considerations are needed in designing web-based interventions. 3.4.3 Institutional or Governmental Level Strategies Another group of stigma management strategies that differ from interventions at the individual level are interventions at institutional, structural, and governmental levels. Legal, policy, and structural interventions constitute the strategies often taken to protect or enforce the rights of people with stigmatising conditions in workplaces or society at large. While stigma can be manifested at the community, societal, and institutional level, it is beyond the focus of this dissertation to explicate the stigma management strategies at the institutional or governmental level. Yet, some of the findings of this research may help to inform destigmatising strategies at institutional and governmental levels. 37 3.5 Conceptual Framework: Trauma-Informed Care Trauma-informed care is a holistic approach to healthcare that seeks to offer an understanding of and a thoughtful response to individuals who have experienced a traumatic event, thereby enhancing their resilience and self-efficacy (Hopper et al., 2009). According to Hopper et al. (2009), trauma-informed care provides: “…a strength-based framework that is grounded in an understanding of and responsiveness to the impact of trauma, that emphasizes physical, psychological and emotional safety for both providers and survivors, and that creates opportunities for survivors to rebuild a sense of control and empowerment.” Trauma-informed care has seven main characteristics including: 1) An understanding that trauma does not only influence the individual but also influences their environment, social networks, and treatment experiences; 2) A realization of traumatic histories; 3) Ensuring privacy and emotional safety; 4) Avoiding triggers from prior trauma; 5) Patient-centered and patient-controlled care; 6) Emphasizing individual strengths; and 7) Avoiding re-traumatization (Hopper et al., 2009). Harris & Fallot, (2001) conceptualized trauma-informed care into five main principles. These principles, which capture the essential elements of trauma-informed care, include safety, collaboration, trustworthiness, choice, and empowerment. For a system or an approach to care to be considered trauma-informed, it must thoroughly incorporate an understanding of trauma, its consequences, and the conditions that promote healing (Hopper et al., 2009). Although the Harris & Fallot (2001) conceptual framework of trauma-informed care was originally developed for reducing traumatic experiences in health and social services, the framework is flexible enough to allow for adaptation to various fields and domains. 38 3.6 The Relationship Between Trauma and Stigma In the United States, the Substance Abuse and Mental Health Service Administration (SAMSHA-USA) defined trauma as “…an event or series of events or set of circumstances that are experienced by an individual as physically or emotionally harmful, or life-threatening and that has lasting adverse effects on the individual functioning including the mental, physical, social, emotional, or spiritual well-being (SAMSHA-USA, 2014).” Traumatic experiences could lead to signs and symptoms that reflect feelings of internalized stigma including anxiety, distress, and panic attacks. Indeed, studies suggest that trauma and stigma co-exist and tend to reinforce each other through a simultaneous, albeit unclear feedback loop (Frieh, 2020). On one hand, traumatic experiences can create the potential for stigma. On the other hand, stigma can also create potential vulnerabilities to traumatic experiences. According to the modified labeling theory of stigma, the vulnerable and devalued feelings associated with stigmatised conditions could fuel the potential for traumatic experiences (Collins et al., 2008; Frieh, 2020; Frieh, 2020). The interplay between stigma and traumatic events is commonly experienced among people affected by or living with conditions related to sexual health or sexual disorders. For example, individuals living with HIV/AIDS have a disproportionally higher rate of traumatic experiences (Whetten et al., 2012). This mediated role between stigma and trauma can produce an ongoing process of victimization, devaluation, re-stigmatization, and re-traumatization. The simultaneous effects of stigma and trauma on people with sexual health-related conditions have been demonstrated in a study that tested the relationship between stigma, trauma, social support, and risky sexual behaviours among LGBTQIA people who engage in risky sexual behaviours (Burnham et al., 2016). The findings from this study revealed that 39 internalized stigma mediated the relationship between trauma-related symptoms and sexual risk behaviours related to HIV/AIDS. That is, gay men who sleep with men and who are traumatised over their HIV diagnosis were more sensitive to HIV stigma, yet engaged more in HIV transmission risk behaviours. The study also revealed that people with traumatic distress related to the diagnosis of a sexual health problem may be more sensitive to stigmatization than people who have never experienced trauma. Frieh (2020) also indicated that women with a history of traumatic events related to their sexuality tend to be more worried about the potential for stigmatization than women who have not experienced trauma. Additionally, internalized stigma was found to be significantly correlated with the severity of trauma symptoms in a study by Deitz, Williams, Rife, & Cantrell (2015). These studies demonstrated the simultaneous effects of stigma and traumatic experiences among people affected by, or living with, conditions related to sexual health. 3.6.1 Fallot and Harris' Principles of Trauma-Informed Care As trauma and stigma are inherently intertwined and work to reinforce each other (Frieh, 2020), adopting a trauma-informed approach to developing destigmatising design principles may help to address stigma concerns among people who use digital health technologies. Therefore, to develop design principles that go beyond the usability and functionality of digital health technologies, this study adopted Fallot and Harris’ Trauma-Informed Care framework to investigate design principles that incorporate human values including human dignity, privacy, anonymity, autonomy and choice, sympathy, and human relationships in health technology design. This may not only help in addressing stigma and trauma, but also improve the overall 40 user experience of digital technologies. The decision to use a trauma-informed approach in generating the destigmatising design principles stemmed from the following: 1) The intertwined relationship between stigma and trauma; 2) The high prevalence of traumatic experiences in persons affected by, or living with, conditions related to sexual health; 3) The potentially amenable nature of the framework to different domains; and most importantly 4) The obligation to provide digital health technologies that could alleviate sexual health-related stigma, prevent re-traumatization, and promote emotionally safe user interactions. Since the Harris & Fallot (2001) framework was originally developed for use in organizational and service contexts, the meaning of the principles as contained in the original framework may not be directly applicable in the context of technology design. To ensure that the framework is applicable in the context of user-interface design, the core principles of the framework were refined to focus on anti-stigma approaches. Table 3.1 contains the original definitions as well as the adapted definitions of the trauma-informed care framework. This was the starting point of the adapted definitions used throughout the dissertation. Table 3.1 Original and adapted definition of trauma-informed care categories Trauma-Informed Care Principle Original Definition Adapted Definition Safety Ensuring physical and emotional safety. Services should be welcoming, and privacy respected. Ensure that the emotional safety of people with sexual health-related conditions is addressed in digital health technologies. Choice Ensuring individuals have choice and control over their treatment/life. Providing appropriate messages about individual rights and responsibilities. Ensure consumers have choice and control over the content and features of technology such as a sexual health website. Provide flexibility with regards to the use of web technologies. Ensure the technology adapts to users’ visual preferences. 41 Table 3.1 Original and adapted definition of trauma-informed care categories: (continued) Collaboration Sharing power and making decisions with the individual. Create avenues on web-based platforms for collaboration between clients and among people with a similar condition. Trustworthiness Clarity, consistency, and interpersonal boundaries. Maintain respectful boundaries. Provide clear, credible, and consistent information about the condition on web-based platforms Empowerment Prioritizing authority and skills building. Providing an atmosphere that allowed individual to feel validated and affirmed about themselves and with personal contacts. Facilitate avenues that enable a patient to gain control of stigmatising situations and encourage participation in destigmatization activities via digital technologies. Integrate additional features that would empower users to combat stigma. 3.7 Digital Health Technologies and Sexual Health-Related Stigma Given the complex nature and different meaning assigned to the concept of stigma, different approaches have been developed to address this issue depending on the nature of the condition. In recent times, digital health technologies have been frequently adopted as a popular approach to help people avoid, reduce, recover from, and cope with sexual health-related conditions. However, whether such technologies address stigma remains unknown. The next chapter of this dissertation presents a scoping review of the literature that examined stigma in relation to the design, use, and effects of digital health technologies for sexual health-related stigma. 42 Chapter 4: Stigmatising and Destigmatising Properties of Web Apps for Sexual Health-related Conditions: A Scoping Review 4.1 Introduction to Chapter 4 In order to understand whether current sexual health-related technologies address or aggravate stigma, a scoping review was conducted to understand the state of the literature that has examined stigma in relation to the design, impact, and ways web-based healthcare applications (web apps) might alleviate or aggravate sexual health-related stigma. In this chapter, the method, results, and discussion of the scoping review are presented. In this chapter, the term web apps is used to refer to digital health technologies. The scoping review provided a background and helped to inform subsequent chapters in this dissertation. A version of this chapter was published in the International Journal of Sexual Health (Abdulai et al., 2021). For this study, the study team included Abdul-Fatawu Abdulai (AF), A. Fuchsia Howard (FH), and Leanne M. Currie (LC). Co-authors’ initials are used to indicate participation in specific activities. 4.2 Background The growth of web-based applications, henceforth called web apps, such as websites, mobile apps, and social networking sites, has fundamentally changed the way health information is accessed, analyzed, and utilized (Guse et al., 2012). Recent estimates indicate that half of all adults across Europe and the United States, and approximately 70% of all Canadians rely on digital technologies for health-related information before visiting a health professional (Beck et 43 al., 2014; Hesse et al., 2010; Statistics Canada, 2013; Tonsaker et al., 2014). The number of people utilizing technology-based interventions may be greater for sexual health problems that are perceived as embarrassing, stigmatising, and difficult to discuss (Berger et al., 2005 Redston et al., 2018). The accessibility, anonymity, and confidentiality of web apps present an incredible opportunity for developing health interventions for people who are otherwise disconnected from conventional health services (Broaddus et al., 2015). More particularly, the privacy and anonymity of web apps may be appealing to people who may experience stigma deriving from a sexual health-related condition (Magee et al., 2012). According to Goffman, stigma dates back to the Greeks and is defined as “an attribute that is deeply discrediting and reduces a person from a whole and a usual person to a tainted and discredited one” (Goffman, 1963). More recently, stigma researchers have identified two main types of stigma: public/enacted stigma and internalized stigma. Public or enacted stigma is the implicit and/or explicit negative reaction towards people with a condition that is considered by some members of society to be abnormal, while internalized stigma generally consists of individuals’ personalized acceptance of stigma as part of their value system and self-concept (Bos et al., 2013; Fuster-Ruizdeapodaca et al., 2014). Modified labeling theory offers a useful framework for explaining the relationship between public/enacted stigma and internalized stigma. According to this theory, when a person is publicly labeled with a stigmatised attribute, such attributes become salient to the person and he/she could internalize the stigmatising ideas (Link, 1987). The individual may then withdraw socially because they expect others to reject them based on their attributes (Rosenfield, 1997). According to modified labeling theory, public stigma perpetuates internalized stigma. 44 Although both types of stigma affect people with sexual health-related conditions, internalized stigma is the most challenging form of stigma to be addressed in the context of sexual health (Hood & Friedman, 2011). This can be attributed to the fact that, unlike public stigma, the hidden feelings of shame, blame, and guilt that accompany people with sexual health-related conditions are not immediately apparent, and therefore, may be difficult to directly address. Indeed, people with sexual health-related conditions may be harbouring feelings of anxiety, depression, isolation, and hopelessness in anticipation of a possible (accidental) disclosure of status. As Goffman, (1963) argued, people who live with invisible marks of stigma may ultimately “pay a high psychological price, a very high level of anxiety, in living a life that can be collapsed at any moment” if their status is revealed. Due to the effect of stigma on peoples’ lives, web apps may be used as plausible alternatives to the predominantly conventional-based onsite services such as sexual health or sexually transmissible infection (STI) clinics, school health programs on STI, and sexual health campaign programs at designated locations. As web apps are increasingly designed to complement, and in some cases replace, face-to-face services (Jones & Lacroix, 2012), many posit that web apps may have an inherent benefit of assuaging public stigma and possibly reducing internalized stigma among people living with sexual health-related conditions. Indeed, various technology-based interventions have demonstrated effectiveness in reducing sexual health-related stigma by promoting virtual connections, improving peoples’ knowledge and coping skills, and eventually reducing or completely removing the need for people to attend sexual health clinics in-person (Chaudoir et al., 2012; Kerr et al., 2015). 45 Although web apps are recognized as important tools for promoting engagement in care and potentially alleviating the stigma of sexual health-related conditions (Wadham et al., 2019), limited published research exists on the relationship between stigma and the design and impact of web apps on sexual health-related conditions. Therefore, it is not known whether current sexual health-related web apps would indeed alleviate stigma or may inadvertently exacerbate stigma. The design of sexual health-related apps is important because the issues of stigma, shame, and embarrassment are major concerns among people with sexual health-related conditions, yet very little consideration is given to addressing these concerns during technology design (Baker et al., 2002). Almost all sexual health web apps are designed by software designers and human-factor specialists for whom technical considerations such as usability and functionality have been the most paramount considerations (Vaes, 2019). However, it has been demonstrated that emotional design considerations including shame and stigma may be more critical to a product's success than its utilitarian features (Norman, 2004). The purpose of this review was to explore, identify, and summarize the current state of the literature that has examined stigma in relation to the design and impact of web apps on sexual health conditions and the ways web apps can alleviate or aggravate stigma. The research questions that guided this review were: 1) What is the state of the literature on stigma in relation to the design of sexual health-related web apps?; 2) What is the impact of web apps on sexual health-related stigma?; and 3) How do web apps alleviate or aggravate sexual health-related stigma?. By “design” we mean the main features or characteristics of web apps that are employed to address stigma, and “impact” indicates the effect web apps have on alleviating or perpetuating stigma (public/enacted or internalized). We focused on web apps because they: 1) 46 Have a standalone user interface that facilitates user interaction; 2) Could enable people to access and share information both publicly and anonymously; and 3) Can be used to obtain information anytime and anywhere. Text messaging was excluded because it does not have a user interface with a specific focus, is not anonymous, and typically does not provide users with specific information/feedback in a given time. 4.3 Materials and Method Given that the use of web apps for alleviating sexual health-related stigma is a relatively new area with scarce research evidence, a scoping review methodology was considered suitable for identifying and mapping the key concepts and issues. The review followed the methodology of Arksey & O’Malley (2005) by: 1) Identifying the research question; 2) Identifying the relevant studies; 3) Selecting the relevant studies; 4) Charting the data; and 5) Collating, summarizing, and reporting results. While Arksey and O’Malley suggested a possible phase of consultation with a core group to whom the topic is of interest, we omitted that phase in this chapter because the consultation phase was integrated into Study 2, which is presented as Chapter 5 in this dissertation. The research questions provided the roadmap for the subsequent stages of the review. 4.3.1 Search Strategy (identifying the studies) Due to the multidisciplinary nature of the topic, we examined a variety of information sources in medicine, nursing, allied sciences, computer science/engineering, and psychology. Six electronic databases including CINAHL, PubMed, Embase, Compendex Engineering Village, 47 Medline, and PsychINFO were searched for literature from database inception up to April 25, 2021. We identified grey literature by searching up to ten pages of Google and Google Scholar each, however, only 7 pages were retrieved for Google (See search terms below). An additional manual search was conducted by reviewing reference lists of included articles to identify relevant publications that were not returned through the database searches. The search strategy was developed by two authors (AA, LC) and the related search terms were divided into three categories comprising digital health/eHealth, stigma, and sexual health. In line with the Joanna Briggs Institute (JBI) guidelines for scoping reviews, we outlined the population, concepts, and context to guide our search strategy. The population for our study included people of all age groups and sexual identities who use sexual health technologies. The concept includes stigma, sexual health-related conditions, and web apps. Context includes original research articles using surveys, experimental, observational, and qualitative designs as well as review articles such as scoping reviews, systematic reviews, and meta-analyses conducted in all settings across the globe. With the JBI framework as a guide, we used a different combination of search terms for each database to ensure the search results reflected the population, concepts, and context of our topic. Given that web apps and sexual health-related conditions are sometimes inconsistently defined, we used broader search terms that would maximize a full scope of articles returned in each domain. The search strategy included (sex* edu*cation/ OR sexuality/ OR sex* health OR sexually transmitted disease/ Infect* OR sexual dysfunction/ OR sexual behavior/behavior) AND (Internet OR web* OR app* OR internet searching OR web development OR worldwide web OR eHealth OR mHealth OR telehealth OR telenursing OR ubiquitous health OR digital health OR Website OR web technology OR 48 informatics) AND (Stigma* OR Social Stigma* OR, de-stigma* OR stereotype). Where possible, database-specific terms including CINAHL headings and MeSH headings in PubMed were used to supplement the search terms. 4.3.2 Inclusion and Exclusion Criteria For the purpose of this review, web apps included digital technologies such as websites, mobile apps, and patient portals that people use to access and share information. Sexual health-related conditions include sexually transmitted infections (STIs) (e.g., HIV/AIDS, genital herpes, gonorrhea, and chlamydia) as well as conditions/disorders that affect sexual health functioning (e.g., endometriosis, infertility, sexual dysfunction). Articles were included if they: 1) Were focused on patient-facing web apps; 2) Were web interventions designed for alleviating stigma OR described stigma management strategies OR described how stigma impacted the use of web apps; 3) Were original studies with empirical data; and 4) Were web apps that have a user-interface specifically designed for sexual health-related conditions. Articles were excluded if they: 1) Were web apps that involved only text messaging; 2) Were web apps that focused on changing the behaviour of health professionals or healthcare institutions; 3) Were not available in English; and 4) Were theory papers, review papers, posters, and abstracts. All retrieved articles were screened based on the title, followed by abstract screening, and finally, full-text review. 49 4.3.3 Data Extraction and Categorization General and specific information from the included articles was extracted and summarized using a spreadsheet to categorize and compare studies and to identify related themes. Information extracted included title of the article, year of publication, authors’ names, geographical location, research methodology, type of web app, type of sexual health-related condition, sample size, study purpose, and main findings/conclusion. To answer the research questions, two reviewers (AA, LC) independently screened the papers against the inclusion and exclusion criteria and highlighted patterns across each paper. Following JBI methods of qualitative synthesis and qualitative content analysis described by Elo and colleagues (Elo et al., 2014; Elo & Kyngäs, 2008), codes were inductively generated and assigned to the highlighted text, and initial categories related to our research questions began to emerge. Related categories were clustered into initial themes and then again into major themes that represented a slightly higher level of abstraction of the narrative in each paper. The categories and the grouping into themes were reviewed by all authors and revised through ongoing team discussions. 4.4 Results The search strategy yielded 1,017 articles, and all were exported to Endnote X9 (Clarivate™, 2020) software. Google and Google Scholar results were manually added. Endnote was used to remove 251 duplicates. Five hundred and forty-eight (548) articles were removed based on title screening, and 179 were removed based on an abstract review. This resulted in 39 articles that received a full-text review. A manual search of the reference lists of the articles that received full-text review added three articles that were not returned by the search strategy. Thus, 50 a full-text review was conducted on 42 articles. Twenty-five articles were excluded because they did not fully meet the inclusion criteria. Seventeen articles were included in the review. (See Figure 4.1) Figure 4.1 Prisma flow diagram for scoping review 51 4.4.1 Summary of Study Characteristics The 17 studies were original peer-reviewed research articles, published from 2012 to 2019. Ten studies used qualitative data collection methods including focus group discussions (3 studies), interviews (2 studies), needs assessment interviews (1 study), and analysis of participants’ posts in online forums (4 studies). Three studies used quantitative methods, including randomized control designs (2 studies) and a cross-sectional design (1 study), and four studies used mixed-method designs. Measurement instruments used in the quantitative studies were largely researcher-developed survey tools. The validity and reliability of these instruments were not reported. The participants in the studies were mostly < 30 years old, with sample sizes ranging from 31 to 935. Most studies were from the United States (10 studies) followed by Canada (2 studies), Australia (2 studies), United Kingdom (1 study), China (1 study), and Russia (1 study). The findings revealed that the use of web apps for sexual health-related stigma was solely focused on the stigma of STIs. No study was found that used web apps in addressing the stigma of disorders that affects sexual functioning (e.g., endometriosis). Six studies were explicitly focused on HIV/AIDS, one on genital herpes, and ten were related to general sexual health and combined STIs (e.g., HIV/AIDS, genital herpes). Among those ten studies, five were focused on STIs among people who also identified as LGBTQ. Only two studies used a theoretical model of stigma to guide the research: Kang & An, (2013) used a theoretical model of stigma to guide their study design, while Noltensmeyer, Peters, Meisenbach, & Eastman-Mueller, (2016) used a theoretical model of stigma as an approach to technology design. Only two studies reported on the design and development of web apps (Klein & Lomonaco, 2016; Noltensmeyer et al., 2016). Table 4.1 shows the details of each study. 52 Table 4.1 Study characteristics, impact on stigma, and corresponding themes Author/ Year/ Country Objective/interventions (if relevant) Method Sample & Sample size Findings/Conclusion Impact of web apps on stigma Theme Sharing Stigma Post Content Display Disclosure 1 Bauermeister et al., (2019), USA To determine if engagement in stigma-related discussions in online forums changes stigma over time. Baseline, Follow-up surveys & analysis of forum post 238 YBMSM aged 18-30 years YBMSM who discussed experiencing HIV stigma in the forums reported decreased stigma over time. Those whose posts indicate anticipated stigma reported an increase in stigma over time Public; Reduced X 2 Chen et al. (2019) China To examine the influence of perceived threats on seeking social support among gay men living with HIV. Content analysis of “Weibo” post 133 gay men with HIV Perceived threat was a motivation to seek online support. However, people who experienced high perceived stigma and perceived severity of HIV reported posting more emotional support seeking messages General; No Change X X 3 Israel et al. (2019), USA To develop and test the efficacy of an online intervention to reduce internalized stigma among bisexual adults with STIs. Randomized control design 702 bisexual adults with STI Internalized binegativity and internalize stigma were reduced in the intervention group compared to the control group. The intervention reduced anticipated stigma in the intervention group Internal; Reduced 4 Dudina & Tsareva, (2018), Russia How people living with HIV/AIDS discuss status disclosure in online platforms in the context of stigmatization. Analysis of forum post and follow-up interview Unspecified Online communities were related to disclosure and HIV stigmatization. differences in how newly diagnosed and those who live with HIV for a long time react to and manage stigma in online spaces Public; Worsened X X NOTE: YBMSM = Young Black Men who Sleep with Men; STI = sexually transmitted infection 53 Table 4.1 Study characteristics, impact on stigma, and corresponding themes (continued) Author/ Year/ Country Objective/interventions (if relevant) Method Sample & Sample size Findings/Conclusion Impact of web apps on stigma Theme Sharing Stigma Post Content Display Disclosure 5 Karamouzian, et al., (2018), Canada Compared how stigma differs between online vs. face-to-face and how each mode reduced/exacerbated stigma. Semi-structured exploratory interviews 71 youth aged 15-24 years Online testing can potentially ameliorate experiences of external stigma. Internalized stigma, exacerbated by internal feelings of shame around STIs continues to persist. Improved Public, not Internal X 6 Flickinger et al., (2018), USA To determine whether online discussions of stigma change the stigma levels of people living with HIV at baseline and after 12-months. Analysis of messaging boards & stigma scores. 77 &1834 stigma-related posts People who posted stigma-related content were more likely to show reduced stigma levels after 12 months of follow-up. Public; Reduced X 7 Ho, et al., (2017), USA To examine how stigma is perpetuated on HIV testing websites and how it differs based on the levels of online monitoring Analysis of online post 210 threads and 319 post Users largely displayed self-stigmatising attitudes towards HIV testing in online forums. Public; Worsened X 8 Kleine et al., (2016), USA To develop an HIV digital health program for managing stigma in MSM. Surveys and focus group discussions 61 HIV care providers and black MSM Stigma prevention/management and a safe space/forums for people with STI were the main factors that inform the design of the technology. General; No Change X 9 Noltensmeyer, et al., (2016), USA To use a stigma management theory to develop a web-based sexuality education and stigma management program. Questionnaire and interviews 171 University student Missouri People who publicly challenge stigma were less likely to accept that they have a stigmatising attribute. Individuals who challenge public stigma have greater self-efficacy in practicing safe sex. Public; Reduced X 54 Table 4.1 Study characteristics, impact on stigma, and corresponding themes (continued) Author/ Year/ Country Objective/interventions (if relevant) Method Sample & Sample size Findings/Conclusion Impact of web apps on stigma Theme Sharing Stigma Post Content Display Disclosure 10 Payton, & Kvasny, (2016), USA To understand the technology affordance associated with the HIV prevention and stigma awareness website Focus group discussions 60 female black students Students largely expressed concerns about stigma-by- association, their personal reputations, online social capital resulting in accidental disclosure of status General; Worsened X X 11 Witzel, et al., (2016), UK Evaluate how users’ privacy concerns influence their engagement with sexual health programs on Facebook Semi-structured phone interview 40 black African gay or bisexual men Safety concerns, implied disclosure, and stigma-by-association were identified as barriers to using sexual health-related websites. General; Worsened X X 12 Christensen et al., (2013). USA Test the effectiveness of shame and stigma-reducing video game intervention Randomized control design 935 MSM MSM in the “SOLVE” intervention reported more reduction in shame and stigma over three months than the control group. Internal; Reduced 13 Kang, & An, (2013), USA To evaluate whether drug prescription websites for stigmatised medical conditions contained stigma-reducing components Content Analysis of websites 88 drug websites ( e.g HIV, herpes, chlamydia) Websites for drug prescription on stigmatised conditions are underutilized in disseminating stigma-reducing information, as half of the websites did not contain any stigma alleviating element. General; No Change X 14 Byron et al (2013), Australia To examine the potential for sexual health information on social media Focus group discussions 22 people aged 16-22 Fear of stigma and privacy concerns were raised. The use of humorous videos were noted as the best way to avoid stigma No reported impact X X NOTE: MSM = men who have sex with men 55 Table 4.1 Study characteristics, impact on stigma, and corresponding themes (continued) Author/ Year/ Country Objective/interventions (if relevant) Method Sample & Sample size Findings/Conclusion Impact of web apps on stigma Theme Sharing Stigma Post Content Display Disclosure 15 Davis, et al., (2012) Canada Understand young people's views on how sexual health websites can address stigma Interviews and focus group discussions 31 young people in Vancouver More explicit content elicited negative responses from participants and many reacted negatively seeing sexual health information presented callously or crudely. General; Worsened X X 16 Magee, J., et al., (2012), USA Explore aspects of the internet for sexual health information, how the internet may reduce/ exacerbate stigma Interviews and surveys 32 LGBTQ people aged 16-24 years The main reason for not using online health information includes the stigma associated with being caught accessing sexual health information/ stigma-by-association General; Worsened X X X 17 Shearer et al., (2012), Australia Apply destigmatising strategies in online video content and examine success in creating destigmatising messages. Thematic content analysis of 103 videos 103 videos Five destigmatising strategies were identified in the videos. Humorous messages were found to aggravate stigma Public; Worsened X X NOTE: LGBTQ = Lesbian, Gay, Bisexual, Transgender, and Queer56 4.4.2 Research Question 1: What is the state of the literature on stigma and the design of web apps? The studies involved varying web apps including websites (10 studies), websites integrated with social media (4 studies), mobile app (1 study), YouTube (1 study), and a gaming website (1 study). Digital platforms such as “Get Yourself Tested” (https://www.cdc.gov/std/sam/gyt/default.htm) and “It Starts With Me” (https://itstartswithme.com/) are specific examples of web apps that were analyzed by researchers to understand how their design can address sexual health-related stigma (Karamouzian et al., 2018; Witzel et al., 2016). The main strategies to address stigma in eight of the studies involved mostly interactive features and educational content. For instance, the studies reported that interactive features including messaging boards and patient forums enable users to access information, express stigma-related concerns, and engage with their peers for social support as ways of addressing sexual health-related stigma (Bauermeister et al., 2019; Dudina & Tsareva, 2018; Flickinger et al., 2018; Ho et al., 2017; Israel et al., 2019; Karamouzian et al., 2018; Noltensmeyer et al., 2016; Payton & Kvasny, 2016; Witzel et al., 2016). Two studies explored healthcare providers' interactions with web app users (Ho et al., 2017; Noltensmeyer et al., 2016), while one study provided participants an opportunity for engagement through customized avatars deployed via a gaming technology (Christensen et al., 2013). Other studies were based on social media platforms integrated with humorous videos and messaging boards/forums as a way of addressing stigma among young people (Byron et al., 2013; Witzel et al., 2016). However, in web apps that used social media platforms, the messaging boards and forums were viewed by participants as non-secure, non-anonymous, and could potentially result in accidental disclosure of status (Payton & Kvasny, 2016; Witzel et al., 2016). 57 Five of the studies were based on educational websites that enabled users to access information on various stigma management strategies without the ability for interpersonal engagement or interaction (Davis et al., 2012; Israel et al., 2019; Kang & An, 2013; Magee et al., 2012; Shearer et al., 2012). While educational messages may provide credible and reliable information on web apps, how those messages were crafted to specifically address stigma was not explicitly described or examined in any of the studies. The majority of the studies were centered on web apps that were used in various ways to connect and engage with others through online discussion threads, messaging boards, online forums, and for forming online community support groups. Although the main purpose of the web apps analyzed in the studies was meant to manage sexual health-related conditions and address stigma, only four studies explicitly mentioned stigma management strategies for addressing sexual health-related stigma (Davis et al., 2012; Kang & An, 2013; Noltensmeyer et al., 2016; Shearer et al., 2012). Other studies recommended strategies such as testimonials and conveying sexual health-related information in a humorous manner as ways of controlling the public narratives that surround sexual health-related conditions (Bauermeister et al., 2019; Dudina & Tsareva, 2018; Noltensmeyer et al., 2016; Shearer et al., 2012). 4.4.3 Research Question 2: What is the impact of web apps on stigma? 4.4.3.1 Addressing/Alleviating Stigma Nine studies reported on how web apps either have a positive or a negative impact on sexual health-related stigma. Among the nine studies, six were focused on addressing public stigma (Bauermeister et al., 2019; Dudina & Tsareva, 2018; Flickinger et al., 2018; Ho et al., 2017; Noltensmeyer et al., 2016; Shearer et al., 2012), two focused on addressing internalized 58 stigma (Christensen et al., 2013; Israel et al., 2019), while one study was meant for addressing both internalized and public stigma (Karamouzian et al., 2018). Three of the studies that focused on public stigma were found to have a positive impact by showing a reduced level of stigma (Bauermeister et al., 2019; Flickinger et al., 2018; Karamouzian et al., 2018; Noltensmeyer et al., 2016), while three studies found that the web apps inadvertently worsened public stigma (Dudina & Tsareva, 2018; Ho et al., 2017; Shearer et al., 2012). Two studies that had a focus on internalized stigma also had a positive impact by showing some marginal reduction in internalized feelings of shame and stigma (Christensen et al., 2013; Israel et al., 2019). One study (Karamouzian et al., 2018) found a reduction in public stigma, but not internalized stigma. The other eight studies were focused on stigma in general, and four of these studies reported a negative impact on stigma (Davis et al., 2012; Magee et al., 2012; Payton & Kvasny, 2016; Witzel et al., 2016), while four reported no impact (Byron et al., 2013; Chen et al., 2019; Kang & An, 2013; Klein & Lomonaco, 2016). Among the studies that were focused on public stigma, two sought participants’ views on how web apps could alleviate or might aggravate stigma (Karamouzian et al., 2018; Shearer et al., 2012), while the other four investigated only how web apps could alleviate stigma but did not evaluate how web apps might aggravate stigma. Interestingly, many of the studies that focused on public stigma also happened to be web apps that enabled people to connect online through forums and messaging boards. In summary, six studies had a positive impact on stigma, four reported no change, while seven studies however reported an increase in stigma attributed to web design. Table 4.1 lists the impact of web apps on stigma. 59 4.4.3.2 Research Question 3: How do web apps alleviate or aggravate stigma? Three overarching themes were identified that represented ways in which web apps may alleviate or aggravate stigma. These include sharing stigma-related experiences via web apps (9 studies), the display of appropriate/inappropriate sexual health-related content (7 studies), and implied disclosure/stigma-by-association (8 studies). The studies were not mutually exclusive to the themes as most studies addressed more than one thematic area. The themes are listed in Table 4.1. 4.4.3.3 Sharing Stigma-related Experiences via Web Apps Nine studies (53%) reported on how the use of web app functionalities such as online forums, message boards, and the expression of stigma-related experiences or stigma management strategies on web apps could address or aggravate stigma (Bauermeister et al., 2019; Byron et al., 2013; Chen et al., 2019; Dudina & Tsareva, 2018; Flickinger et al., 2018; Ho et al., 2017; Magee et al., 2012; Noltensmeyer et al., 2016; Payton & Kvasny, 2016). This theme is directly related to how the various web apps were used to connect people online. Three studies reported that engaging in online forums and messaging boards could address stigma by promoting virtual connections and boosting emotional support while allowing people affected by similar conditions to engage in anonymous discussions on stigma-related concerns (Bauermeister et al., 2019; Magee et al., 2012; Noltensmeyer et al., 2016). Chen et al. (2019) specifically indicated how people who have high perceived threats and high perceived stigma were likely to post emotional-support-seeking messages on web apps as a way of addressing the threats and stigma inherent in HIV. According to Noltensmeyer et al. (2016), emotional support and anonymous virtual connections are essential in managing sexual health-related stigma for people who may be 60 worried about stigma in face-to-face settings. Aside from connecting people through forums and messaging boards, three studies also reported that sharing testimonials, documentaries, and biographies of well-known people via web apps could reduce internalized feelings of stigma and promote a sense of shared experiences (Bauermeister et al., 2019; Dudina & Tsareva, 2018; Noltensmeyer et al., 2016). Although testimonials have been recommended, no studies were located that empirically evaluated how and if the integration of these stories reduced stigma. While sharing supportive messages via online forums and messaging boards can promote engagement in care, four studies argued that stigmatising behaviours can still occur in online forums despite these forums tending to be relatively anonymous (Bauermeister et al., 2019; Flickinger et al., 2018; Ho et al., 2017; Magee et al., 2012). These studies reported on how sharing experiences, testimonials, or stories via web apps may give people the chance to share negative perspectives and promote online bullying that could eventually (re)produce, perpetuate, and exacerbate stigma. Magee et al. (2012) for instance argued that the face-to-face stigma people experience could be transferred to online spaces. The potential for re-stigmatization was noted to be particularly common in web apps with social media platforms because the mere act of “liking” or “sharing” web content could expose someone to ridicule or online bullying (Payton & Kvasny, 2016; Witzel et al., 2016). 4.4.3.4 Display of Appropriate or Inappropriate Sexual Health-Related Content Seven articles highlighted how the display of positive or negative sexual health-related content could alleviate or aggravate stigma, respectively (Davis et al., 2012; Karamouzian et al., 2018; Klein & Lomonaco, 2016; Magee et al., 2012; Payton & Kvasny, 2016; Shearer et al., 2012; Witzel et al., 2016). The studies specifically indicated how inappropriate or vulgar images 61 such as explicit medicalized images, sexualized images of people kissing, or messages such as having “having a fuck buddy” or other sexualized body diagrams on web apps could potentially aggravate stigma. While the intent of displaying explicit sexual images may be to communicate in a way that resonates with the target users, it is important to note that such efforts could inadvertently (re)produce and perpetuate stigma among users who may find such content undesirable (Davis et al., 2012). Some studies further reported on how some web interventions employ stigmatising or embarrassing images and messages in a satirical or ironic format (Witzel et al., 2016), and how some interventions use humour in a stigmatising manner to convey sexual health information (Shearer et al., 2012). According to these studies, such strategies may not only fail to resonate with the intended users but may (re)introduce stigma and stifle general care and prevention efforts. Three studies reported that the use of sexually explicit images or satirical messages could aggravate internalized feelings of stigma and could further widen the barriers to the use of web apps (Davis et al., 2012; Magee et al., 2012; Shearer et al., 2012). Despite the possibility that explicit images might exacerbate stigma, Kang & An (2013) suggested that web apps that display biological organs and symbolic medical diagrams of disease conditions in a positive manner may have the potential to address stigma. However, the impact of the display of such images was not explicitly measured in any of the included studies. 4.4.3.5 Implied Disclosure and Stigma-by-Association A theme identified in eight of the studies related to how web apps could lead to accidental or implied disclosure of status and the likelihood of stigma-by-association (Byron et al., 2013; Chen et al., 2019; Davis et al., 2012; Dudina & Tsareva, 2018; Magee et al., 2012; 62 Payton & Kvasny, 2016; Shearer et al., 2012; Witzel et al., 2016). Participants were generally concerned about the potential for, and experiences of stigma based on their engagement with web apps. For instance, Witzel et al. (2016) and Byron et al. (2013) indicated that using a sexual health-related web app could be interpreted by others as having the condition (i.e., stigma-by-association). The act of stigmatising people by mere usage of a web app could further increase privacy concerns among users of web apps. Stigma-by-association may not only worsen stigma but could further limit the use of web apps by people both with and without the condition (Witzel et al., 2016). While accidental disclosure and stigma-by-association could occur in all apps that enable people to connect online, it was more of a concern to participants using web apps that did not permit anonymization, including social media platforms (Payton & Kvasny, 2016; Witzel et al., 2016). Stigma via accidental disclosure may be higher on social media platforms because social media sites may be more “porous” or non-secure than web apps designed solely for a specific condition (Byron et al., 2013). 4.5 Discussion This review focused on studies that explored stigma in relation to the design and impact of sexual health-related web apps on stigma. The results suggest that although many web apps are designed to manage and address sexual health-related stigma, such platforms may also perpetuate stigma through sharing stigma-related experiences, displaying inappropriate sexual health content, accidental disclosure of status, and stigma-by-association. 63 4.5.1 Digital Health Strategies and Stigma Although the main purpose of the studies was to address sexual health-related stigma, many of the studies did not report on any specific strategies that could have been taken to address stigma via web interfaces. Instead, the main web strategies for addressing stigma tended to be largely technical considerations such as web forums, messaging boards, and tailored educational content. Few studies reported on web apps that specifically adopted an anti-stigma approach in designing the user interfaces (Kang & An, 2013; Noltensmeyer et al., 2016). This suggests that many web interventions do not adopt destigmatising design strategies in their approach despite stigma-reduction being a major goal of their web intervention. According to Bauermeister et al. (2019), web forums and messaging boards, when properly integrated with appropriate educational content in web apps, could be very helpful in managing the stigma of conditions such as HIV/AIDS. The integration of patient forums and messaging boards in the various web apps could also be beneficial to people who may otherwise be disconnected from or unwilling to use conventional health services (Broaddus et al., 2015). However, other studies suggest that such design features could inadvertently foment stigma among its users, especially when the people’s identity or status are accidentally disclosed via such forums (Bauermeister et al., 2019; Dudina & Tsareva, 2018; Flickinger et al., 2018; Ho et al., 2017; Magee et al., 2012; Noltensmeyer et al., 2016; Payton & Kvasny, 2016). Furthermore, other studies relied on or evaluated web apps that used educational information, integrated with online engagement as the main strategies of addressing stigma. While this could promote an understanding of the disease condition while ensuring emotional support, the content could also aggravate internalized stigma if not thoughtfully designed or displayed on the web app. In other 64 words, educational content, if not appropriately conveyed on web platforms may rather foment instead of address stigma. One important strategy for addressing stigma was through the use of online community of support groups. For instance, web apps that support online collaboration offer an opportunity for like-minded people who might experience stigma in conventional health contexts (e.g., attending an HIV clinic) to form online community support groups and offer emotional support to members (Kang & An, 2013; Payton & Kvasny, 2016; Witzel et al., 2016). Although the details of the online community support groups were beyond the focus of this review, the use of patient forums and messaging boards as a form of stigma management strategy was applied in eight of the studies (Bauermeister et al., 2019; Dudina & Tsareva, 2018; Flickinger et al., 2018; Ho et al., 2017; Israel et al., 2019; Karamouzian et al., 2018; Noltensmeyer et al., 2016; Payton & Kvasny, 2016; Witzel et al., 2016). Despite the importance of support groups in web apps, the sensitive nature of sexual health issues may also raise privacy and security concerns among people who engage in online activities. According to Kim et al. (2018), this form of engagement may further inhibit what people are willing to discuss or disclose on online platforms. 4.5.2 The Impact of Digital Health on Sexual Health-Related Stigma Some of the studies suggest that web apps with combined educational and engagement/online connectivity functions were successful to some extent in addressing public stigma (Kang & An, 2013; Magee et al., 2012; Noltensmeyer et al., 2016; Shearer et al., 2012). Only three studies evaluated web apps that were focused on internalized stigma. Two of the three studies found a marginal reduction in internalized feelings of shame and stigma (Christensen et al., 2013; Israel et al., 2019). Although few web apps might have been developed for addressing 65 internalized stigma, the results demonstrate the potential for the positive impact of web apps on internalized stigma. This suggests that web apps may be a favourable platform for addressing internalized feelings of stigma that may accompany people with sexual health-related conditions. Overall, the findings show that studies that demonstrated a reduction in some levels of stigma were largely limited to web apps that promote two-way communication through online collaboration functions such as discussion boards and patient forums (Bauermeister et al., 2019; Christensen et al., 2013; Flickinger et al., 2018; Karamouzian et al., 2018). Standalone educational web apps (i.e., apps without forums, messaging boards, or patient portals) may have a limited capacity to engage people because of their general unidirectional nature. Findings from the studies in this review suggest that apps without two-way communication could perpetuate internalized stigma through an inappropriate choice of words and images in their content design (Davis et al., 2012; Shearer et al., 2012). For instance, Davis et al. (2012) specifically reported how explicit or vulgar sexual images and words such as sexualized images and derogatory statements on web apps could aggravate sexual health-related stigma and deter potential users. In effect, the way words, images, and videos are used to convey a sexual health message in a web app could in itself amplify sexual health-related stigma. This is not to suggest that sexual images or videos would automatically aggravate stigma. On the contrary, sexual images and videos that conform to socially acceptable norms may assuage internalized stigma, as people are more likely to view images that are socially acceptable to them (Hood & Friedman, 2011). The use of socially acceptable images may address stigma because according to Rüsch, Angermeyer, & Corrigan, (2005), understanding the causes of disease as biological processes depicted via images may help reduce the stigma associated with such conditions. 66 4.5.3 Health Providers’ Moderation of Web Apps While online collaboration functions may enable people to connect with others virtually, professional moderation and links to health professionals in forums and messaging boards remained largely missing in the majority of the apps analyzed by the studies. Only Noltensmeyer et al. (2016) recommended some professional moderation of messaging boards. The involvement of healthcare professionals as web moderators may mitigate bullying and online stigmatization by peers, regulate the posting of misleading information, and offer additional stigma management strategies to people most vulnerable to online stigmatization (Bauermeister et al., 2019; Flickinger et al., 2018). More specifically, healthcare professional moderation may improve online discourse by allowing users to discuss negative emotions more candidly and mitigate issues in web apps that participants viewed as insecure and non-anonymous (Payton & Kvasny, 2016; Wadden et al., 2020; Witzel et al., 2016). However, many of the studies analyzed in this review did not involve or consider health provider moderators on the web apps. Despite the benefits of provider moderation, the “presence” of providers may also inhibit participants’ expressions in forums. In a worst-case scenario, provider moderation may prevent the use of web technologies because of the “discomfort” with relaying information to unknown health providers and the limitations of computer-mediated communication channels including misrepresentation by unscrupulous people (Van Uden-Kraan et al., 2008; Witzel et al., 2016). 4.5.4 Recommended Design Considerations to Address Stigma The limitations of web apps in alleviating stigma suggests that web developers need to adopt design strategies that can guarantee the safety of the user in online spaces. Web designers may consider integrating online support groups and social networking functionalities that 67 facilitate anonymous identification with the stigmatised group, promote social identification and social support. Social identification and social support are particularly important because they are hypothesized to increase resistance to stigma and the rejection of negative stereotypes (Crabtree et al., 2010). It has also been demonstrated that online support groups provide opportunities that facilitate social support from a network of people living with sexual health-related conditions as well as empowering them to reject and overcome stigma (Heijnders & Van der Meij, 2006; Lee & Craft, 2002). Furthermore, strategies including testimonials provided by well-known people on web apps could help change the public narrative and people’s attitudes towards sexual health-related stigma (Heijnders & Van der Meij, 2006). Also, as many people may resort to web apps for fear of being stigmatised in healthcare settings (Arkell et al., 2006; Nyblade et al., 2019), providing credible, reliable, and accurate health information may also dispel the myths and misconceptions that often fuel sexual health-related stigma, promote trust, normalize, and destigmatise such conditions (Hirsch et al., 2017; Kwakernaak et al., 2019; Meisenbach, 2010; Sbaffi & Rowley, 2017). These strategies were particularly emphasized in studies that analyzed web apps that relied predominantly on educational content. More importantly, strategies that ensure appealing and comfortable web content are needed to alleviate internalized stigma. This means the framing of sexual health-related language including the use of non-sensational language and images, and non-embarrassing content should be considered when designing web-based interventions (Davis et al., 2012). These strategies may prevent depicting sexual health-related conditions in a negative or unflattering manner on digital platforms (Davidson & Wallack, 2004; Smith, 2007). While these strategies might have already been adopted or recommended by some of the studies reviewed, there is little empirical evidence to support or refute their use. Therefore, 68 further studies are needed to examine how the integration and use of these design considerations on web apps could address sexual health-related stigma. 4.6 Limitations of the Scoping Review Studies were limited to English language publications. Given that stigma is context-specific (Link & Phelan, 2001), studies that were not in English, yet could be quite relevant to this review, might have been excluded. Also, conference abstracts that might be relevant to the topic may have been excluded. Finally, as web interventions for sexual health-related stigma are still in their infancy, many interventions may not yet have been published or evaluated. 4.7 Quality of the Studies While it is beyond the expectations of a scoping review to examine study quality, the wide variety of study designs demonstrates that this is an emerging topic. Despite showing results for improved or worsened levels of stigma in six studies, only two of the studies were based on randomized control trials (Christensen et al., 2013; Israel et al., 2019). The rest were based on either self-report or an analysis of stigma-related posts on web platforms – methods that are at the bottom of the evidence hierarchy. Therefore, although participants of the web-related forum posts might have revealed some level of stigma reduction or worsening of stigma, such findings may be subject to study biases. Secondly, most of the claims made with regards to stigma-reducing components on various web platforms were subjected to little empirical evaluation. Therefore, it is difficult to ascertain whether such strategies will indeed address stigma when implemented on web apps. Due to the weaknesses of the studies, further studies are 69 needed to empirically test the effectiveness of the stigma-reduction claims made in the studies before a conclusion can be drawn on their effectiveness in addressing stigma. 4.8 Conclusion The review revealed that web apps with online collaboration functions such as patient forums and messaging boards provide an opportunity to alleviate public stigma by promoting engagement in care and offering emotional support to people living with sexual health-related conditions. Two web apps demonstrated a modest reduction in internalized stigma. Educational web apps without online collaboration functions may inadvertently provoke stigma by exposing users to inappropriate sexual health-related content or by accidentally disclosing their (supposed) sexual health conditions. These findings have considerable implications for how sexual health-related technologies could be designed and deployed to reduce stigma. Web content that inadvertently exacerbates stigma should be minimized (Davis et al., 2012). The design of web apps should include appropriate content that is thoughtfully displayed to be less stigmatising to users. The use of appropriate sexual content and how that content is displayed could have a de-stigmatising effect on people with sexual health-related conditions. 4.9 Summary of Chapter 4 In this chapter, the literature on stigma in relation to the design, use, and effects of digital health technologies for sexual health-related stigma was reviewed. The literature revealed that while digital interventions may have been developed to alleviate stigma at the societal level, such interventions may not be effective in alleviating internalized stigma among users of sexual health-related technologies. 70 Chapter 5: Developing an Educational Website for Women with Endometriosis-associated Dyspareunia: A Usability and Stigma Analysis 5.1 Introduction Chapter 4 revealed that even though digital technologies are increasingly designed to address stigma, such technologies could inadvertently foment stigma among its users. This study sought to understand the problems patients face when using a sexual health-related website and examine patients’ perspective on how destigmatising sexual health-related websites could be designed. The study was based on a sexual pain website developed as part of an end of grant knowledge translation project. The first part of this study reports on the usability evaluation study while the second part reports on patients’ perspectives of how destigmatising web technologies could be designed. A version of this study was published in JMIR- Human Factors (Abdulai et al., 2022). For this study, the research team included Abdul-Fatawu Abdulai (AFA), A. Fuchsia Howard (FH), Paul J. Yong (PY), Heather Noga (HN), Gurskiran Parmer (GP) and Leanne M. Currie (LC). Co-authors’ initials are used to indicate participation in specific activities. 5.2 Background Endometriosis is a chronic condition in which endometrial-like tissue is present outside the uterus, typically in the pelvic cavity (Leyland et al., 2010). While the actual prevalence of endometriosis may be difficult to quantify, it is estimated that the disease affects approximately one out of every ten women globally (Caruso et al., 2015; Giudice, 2010). Women with this disorder tend to suffer from a variety of symptoms including sexual pain, menstrual pain, chronic 71 pelvic pain, and infertility (Wahl et al., 2020). Although pelvic pain is the most common symptom of endometriosis, 50% of women also suffer from endometriosis-associated dyspareunia – pain experienced during, or after penetrative vaginal intercourse (Ferrero et al., 2005; Yong, 2017). Endometriosis-associated dyspareunia affects multiple aspects of life, leading to absenteeism from work, poor interpersonal relationships, and impaired social functioning, and has a profoundly negative impact on quality of life (Di Donato et al., 2014). Despite the negative effect of sexual pain on the life of people with endometriosis, many patients and health providers do not take this symptom seriously, while others neglect pain symptoms or are unaware of the link between painful sex and endometriosis (Denny, 2004). Even if the healthcare provider and patient are both knowledgeable about dyspareunia, the private nature of sexual pain may inhibit patient-provider discussions on the topic (Weijts et al., 1993). Due to fears of social stigma, some patients with dyspareunia may also feel reluctant to visit or disclose their sexual pain experiences to healthcare providers (Weijts et al., 1993). Indeed, stigma has been identified as a significant barrier to the uptake of sexual health-related interventions including management of endometriosis-associated dyspareunia (Di Donato et al., 2014). Given the potentially stigmatising nature of sexual pain, websites have been identified as complementary tools for disseminating patient information concerning endometriosis and its related symptoms (Hirsch et al., 2017). With the internet becoming a major source of health information, people can explore sensitive and intimate health topics, such as sexual pain, in a private setting. Available evidence suggests that the use of online platforms for information seeking is particularly common among people with conditions or symptoms that are perceived as stigmatising and difficult to discuss (Berger et al., 2005; Redston et al., 2018). Online 72 information is particularly important because it may help patients to understand the complex relationship between endometriosis-associated pelvic pain and painful sex and improve patients’ access to and adherence to recommended treatments while maintaining anonymity (Denny, 2004). The advantages associated with access to online information on sensitive health topics presents an opportunity for developing an online educational resource for people with endometriosis-associated dyspareunia. In response to this need, we developed a patient-centered educational website to provide evidence-based information on endometriosis-associated dyspareunia, which became the Sex, Pain, and Endometriosis Website (https://endopain.endometriosis.org). In this study a usability analysis and a stigma analysis were performed on the website as part of the website development process. These will be described in detail below. 5.2.1 Sex, Pain, and Endometriosis Website Development The website was developed by a multidisciplinary team of scientists, healthcare professionals, patient partners, and community organizations in close collaboration with a web design company as part of an end-of-grant knowledge translation project. We adopted a patient-oriented research approach throughout the website development process, wherein patient partners were equal team members, in recognition that patients provide critical experience-based perspectives that are essential for creating a meaningful website. The research group drew on the Knowledge to Action Framework and Technology Enabled Knowledge Translation in developing the website (CIHR, 2018). We first conducted a needs assessment with our patient partners to determine the scope of the website, content, aesthetics, and key messages. Key findings from the needs assessment included prioritizing necessary information on the causes of 73 endometriosis-associated sexual pain and options for treatment. Our patient partners also expressed their desire to develop a website that would promote inclusiveness (e.g., of diverse gender-identities, sexual orientations, ethnocultural backgrounds, and ages) and help address the stigma of endometriosis-associated sexual pain. While the idea of addressing stigma was not in the forefront at the time of developing the website, the development team was aware of the need for the website to, in one way or the other, address stigma. Therefore, the images and language used on the website were designed so as not to offend or stereotype website users. We also conducted a landscape analysis of pre-existing endometriosis/sexual pain websites to determine their content and features. This was followed by iterative website development. Following the release of version 1 of the website, we conducted the usability and stigma analysis reported here to determine its usability, functionality, and whether the website met users’ needs. The website was modified based on the findings from the usability and stigma analysis before a final launch in February 2021. The main purpose of this website is to help people understand endometriosis-associated dyspareunia, feel empowered, and dispel the myths and misconceptions surrounding endometriosis and sexual pain (Weijts et al., 1993). The website has six main sections – providing information on endometriosis, painful sex, causes of painful sex, treatment for painful sex, resources, and frequently asked questions. The process of website development is described in a forthcoming publication. The purpose of this study was two-fold: 1) To evaluate the usability of the website; and 2) To assess for the destigmatising properties of sexual health-related online resources. The usability evaluation was expected to help improve the web design to make it easy to use, satisfying, and acceptable to end-users. We assessed for destigmatising properties because websites on sensitive health topics are not typically assessed for their ability to address or 74 exacerbate stigma, despite stigma being an outcome of interest for web developers (Bauermeister et al., 2019; Klein & Lomonaco, 2016; Magee et al., 2012). That is, it is unclear how the design of digital platforms for general sensitive health problems could help address stigma or could inadvertently (re)produce and perpetuate stigma among users (Abdulai et al., 2021; Davis et al., 2012). Figure 5.1 Sample of the website interface in Study 2 75 5.3 Methods We conducted usability analysis by employing think-aloud observations, followed by post-think-aloud interviews and a post-interview questionnaire with 12 study participants. Using our website as a reference point, we also conducted a stigma analysis by asking participants how sexual health-related websites may be designed to address stigma. Due to the COVID-19 pandemic, all activities were carried out online. 5.3.1 Recruitment There have been no studies that we are aware of that provide population-based information on women with endometriosis and dyspareunia in western Canada. However, at the time of conducting this study, there were approximately 300 patients diagnosed with endometriosis who were registered in the Endometriosis Pelvic Pain Interdisciplinary Cohort Data Registry of a large urban healthcare center in western Canada, and who consented to be contacted for future research. Using a systematic sampling approach, we selected every 11th person from a list of approximately 300 patients in the data registry. Forty-five participants were subsequently contacted. Participants who expressed interest but did not meet the inclusion criteria were excluded. Inclusion criteria were: 1) 18 years of age or older; 2) New patients or re-referred to the center between May 1st, 2019, and December 31st, 2019; 3) Consented to be contacted for future research; 4) Known to have clinically suspected or diagnosed endometriosis; and 5) Experienced self-reported deep sexual pain (alone or partnered). Exclusion criteria were: 1) Not sexually active (alone or partnered); 2) Never experienced sexual pain (alone or partnered); and 3) Not fluent in English. Ethics approval was obtained from the Children and 76 Women’s Research Ethics Board at the University of British Columbia (REB # H19-03556). Of the 45 participants contacted, twelve (12) participants consented to take part in the study. 5.4 Data Collection 5.4.1 Demographics Prior to data collection and after obtaining consent, participants were first asked to complete a demographic questionnaire that included age, gender identity, ethnicity, computer skills, and frequency of computer use. 5.4.2 Usability Analysis We conducted usability analysis by employing think-aloud observations, a Post-Study Systems Usability Questionnaire (PSSUQ), and a follow-up interview. In the think-aloud procedure, we asked participants to carry out five task scenarios on our website while sharing their screen. The task scenarios represented different ways of searching for information about endometriosis-associated dyspareunia. These included: 1) The meaning of cul-de-sac in relation to endometriosis; 2) Different ways of treating sexual pain; 3) Information on fertility; 4) Anticipation of the pain cycle; and 5) The role of nervous system sensitization in painful sex. (See Appendix A for the detailed task scenarios). Participants were allowed to begin with any of the tasks. The think-aloud observations were conducted remotely via Zoom (Zoom Video Communications, Inc.) and all sessions were audio-recorded and the shared screen was video-recorded. To protect participants’ privacy, the recording did not include their faces. The PSSUQ was conducted to evaluate the overall systems usefulness, information quality, and interface 77 quality (Lewis, 1995). Adapted from (Stinson et al., 2010), we conducted a follow-up interview to understand aspects of usability that were not made obvious during the think-aloud procedure. 5.4.3 Stigma Analysis Given that some sexual health-related technologies may inadvertently exacerbate stigma (Abdulai et al., 2021; Davis et al., 2012), the follow-up interviews also included questions to understand ways in which such websites can be designed to address stigma. In other words, the focus of the interviews was to use our website as a reference point to identify generalities about how destigmatising sexual health-related websites could be designed. The interviews were audio-recorded and transcribed verbatim by the first author (AFA). (See Appendix A for the interview guide). 5.5 Data Analysis 5.5.1 Usability Analysis The think-aloud video data were analyzed by the first author based on Kushniruk & Patel’s (2004) coding scheme for analyzing think-aloud data in patient information systems (Kushniruk & Patel, 2004). Using the coding scheme as a guide, the first author watched and annotated the videotape and audio recordings with usability problems. Usability problems in the video/audio data were categorized into Kushiniruk & Patel’s five main thematic areas including: 1) Navigation problems (related to the user finding desired information, icons, and labels); 2) Comprehensiveness of graphics and text (problems related to participants’ understanding of labels, icons, and content); 3) Systems’ response (how the website responded to users’ actions); 4) Information content (which aspects of the system contained too much or too little 78 information); and 5) Mismatch between the website and users’ expectations. As the analysis proceeded, additional labels that were not captured in the initial coding scheme but emerged during the analysis were added. A second person (GP) independently reviewed all the video recordings, marked the usability problem, and categorized them based on the five thematic areas. The two reviewers’ results were compared, and differences were resolved by discussion. Content analysis procedures were used to identify additional usability problems in the follow-up interview, while demographics and the PSSUQ were analyzed with descriptive statistics (Minimum = 1, maximum = 7, where lower scores mean better satisfaction and higher scores indicate poor satisfaction). Problems identified from the follow-up interviews were also categorized under the five themes in Kushniruk & Patel’s framework. 5.5.2 Stigma Analysis Data relating to potentially (de)stigmatising properties of sexual health-related websites were analyzed using a deductive approach to qualitative content analysis (Braun & Clarke, 2006). The deductive analysis followed the trauma-informed care framework introduced earlier (Harris & Fallot, 2001) to identify the destigmatising properties of websites. Trauma and stigma are inherently intertwined (Frieh, 2020). As such, we suggest that a trauma-informed care framework is relevant for analyzing data elicited from participants about stigma because it may inform recommendations to address stigma concerns among people who use sexual health-related websites (Frieh, 2020; Harris & Fallot, 2001). The Harris & Fallot, (2001) five principles of trauma-informed care were used to guide the analytic approach. These principles include: 1) Emotional safety (ensuring services are welcoming); 2) Choice (ensuring individuals have options over their treatment/life); 3) Collaboration (sharing power and making decisions with 79 individuals); 4) Trustworthiness (provide clear, credible, and consistent information about the condition); and 5) Empowerment (providing an atmosphere that allowed individuals to feel validated and affirmed). All transcripts were uploaded into NVivo version 11 (QSR International Pty Ltd.). Using Braun and Clark’s (2006) steps for conducting deductive thematic content analysis, the first author (AFA) initially familiarized himself with the data. Second, codes and concepts related to each of the five principles of the trauma-informed care framework were assigned to the text. Third, patterns and themes were searched across the different interviews. And finally, the codes and concepts together with the sub-themes were matched with their respective global themes, (i.e., the principles of trauma-informed care). This deductive approach allowed for systematic identification of participants’ perspectives across three levels including, global themes, sub-themes, and concepts. Data saturation was achieved when no new concepts were identified in the data. The coding was discussed with co-authors FH and LC until consensus was achieved. 5.6 Results 5.6.1 Demographic Information Participant ages ranged from 30 to 63 years with a mean age of 38.75 ±8.55. All participants self-identified as heterosexual and all were from a large metropolitan area in western Canada. Eleven (11) participants self-identified as a woman and one (1) did not disclose. Six participants identified as White, two as Hispanic, one as Indigenous, and three did not disclose their ethnic identity. All participants reported using the internet daily, and five described their computer skills as very good, six indicated quite good, and one indicated their skills were neither good nor bad. 80 5.6.2 Task Completion Tasks that required finding information in the Mechanisms section of the website were the most difficult for participants to complete, followed by finding the meaning of cul-de-sac in the Endometriosis section (see Table 5.1). All participants were able to locate information in the Treatment section, and 11 out of the 12 participants located the information on fertility. Tasks were considered incomplete if participants were assisted by the researcher or if they made several mistakes before locating the item. Table 5.1 shows the task completion rate, and the average time (in seconds) it took participants to complete each task. The entire study took approximately 1.2 hours for each participant to complete. Table 5.1 Task completion rate # Task Website Section Number of participants completing task Completion rate (%) Mean time (Seconds) 1 Find the meaning of cul-de-sac in relation to endometriosis Endometriosis 9/12 75.0 165 2 Find the different ways of treating sexual pain Treatment 12/12 100 50 3 Find the information on fertility FAQs 11/12 91.6 80 4 Locate the anticipation of pain cycle Mechanisms 8/12 66.6 206 5 Find the role of nervous system sensitization in painful sex Mechanisms 7/12 58.3 260 FAQs – Frequently asked questions 5.6.3 Overall Usability Most participants expressed minimal difficulty with the think-aloud procedure, although a few had to be occasionally reminded to speak out loud. All participants were able to complete 81 the tasks, although three needed some hints such as “please click on this link” or “the tab is located up there.” Generally, participants were happy with the layout of the website. Participant 3, a 63-year-old woman said “this is a very simple website…. I am not bombarded with too much information; the writing is good for my age……. I don’t have to strain my eyes to read this.” 5.6.4 Usability Problems The think-aloud observation and the post-session interview produced 30 usability problems. Twenty-three problems were identified via the think-aloud process, and seven were identified in the post-session interview. Table 5.2 contains the problems identified from the usability analysis organized according to Kushniruk & Patel’s five thematic areas, the location of each problem on the website, the number of times the problem occurred, and the number of users encountering the problem. The problems were largely content-related issues, particularly related to a preponderance of provider perspectives on the website. For instance, it was generally agreed that the content of the website was medically oriented and currently lacked patients’ perspectives on endometriosis and sexual pain. Participants also suggested solutions to some of the usability problems they identified (See Table 5.3). The mean overall PSSUQ score was 2.41 (SD = 0.85) on a range of 1-7, where lower scores equate with higher satisfaction, thus indicating relatively high satisfaction. While participants’ scores across the three PSSUQ metrics were all below the average PSSUQ scores, systems usefulness had the lowest score indicating a superior score of usability on that metric. This was followed by interface quality and then information quality. Table 5.4 shows the mean score for each PSSUQ item as well as the overall mean score for systems’ usefulness, information quality, and interface quality, respectively. 82 Table 5.2 Interface problems from usability analysis Cate-gory (n) # Usability Problem Page location No. of times No. of users Navigation problems (n= 8) 1 There is no search bar Home page 15 12/12 2 The information underneath the homepage not apparent to users Home page 8 5/12 3 Link to other websites open on the same page. Difficult to navigate back to main page Home page 6 3/12 4 The ‘go-to top’ icons at the bottom of pages not immediately visible Several 5 2/12 5 The sexual response cycle diagram does not fit in the screen for a whole view Mechanism 6 3/12 6 In-text references are not directly linked to the reference list. Users have to scroll up and down in search of references. Several pages 5 5/12 7 Treatment pop-ups too small. Not convenient to users Treatment 9 6/12 8 Not enough hyperlinks and hypertext to redirect users to different but related pages. Several pages 7 5/12 Comprehension of graphics and text ( n=5) 9 Mechanism section not clearly understood Home page 14 12/12 10 The ‘slider’ affordance on the diagram showing signs of endometriosis portrays a click function rather than a slider Endometriosis page 6 8/12 11 Users did not understand the term dyspareunia Pain types 5 8/12 12 Not enough affordances to prompt users to click on diagrams and text Several 8 6/12 13 Too much content in treatment options and ‘mechanisms* Treatment 4 3/12 System Response ( n=3) 14 The link to ‘entry pain’ does not respond Pain types 9 7/12 15 Pages load quite slowly. Takes an average of 8-10 seconds Several 8 7/12 16 Meaning on labels not immediately apparent to users Several 5 4/12 Information content (n=5) 17 Too much text in treatment pop-up Treatment 6 3/12 18 Endometriosis not explained on the home page Home page 3 3/12 19 Not enough content on symptoms except the description of sexual pains Symptoms 2 2/12 20 Content on treatment pop-ups is too cluttered Treatment 5 4/12 21 Patient perspectives or voices are lacking on the website. It is medically oriented Several pages 3 4/12 83 Table 5.2 Interface problems from usability analysis (continued) Cate-gory (n) # Usability Problem Page location No. of times No. of users Mismatch between system and users’ expectations (n= 9) 22 Images too cheerful to portray feelings of pain Symptoms 4 5/12 23 Some bolded text looked like hypertexts but not responsive when users clicked on it Mechanisms 4 4/12 24 Clicking on ‘psychological aspects of sexual pain’ takes a user to the ‘symptoms’ page Treatment 3 3/12 25 Clicking on ‘learn more about how the nervous system and low arousal contribute to painful sex’ takes the user to the symptoms page Treatment 12 11/12 26 Clicking on pain types takes the user to the symptoms page Pain types 5 4/12 27 Users think ‘anticipation of pain cycle’ was located in ‘pain types’ Home 2 2/12 28 The information underneath each section is not apparent until the section is opened Home 6 4/12 29 Links to treatment options are currently limited to only the image and not the entire box where the image is located Treatment 11 9/12 30 The website is one-sided in favour of females Several 5 5/12 Table 5.3 Suggested design solutions # Problem Suggested solution 6 In-text references are not directly linked to the reference list. Users have to scroll up and down in search of references. Clicking on a reference should take a user directly to that reference 7 Treatment pop-ups too small. Not convenient to users. Pop-ups should open on a new page 13 Too much content in treatment options and mechanisms. Bullet points preferred 16 Meaning on labels not immediately apparent to users. Automatically display label meanings when hovering around the image 20 Content on treatment pop-ups are too cluttered. Bullet points preferred 21 Patient perspectives or voices are lacking on the website. It is medically oriented. Include psychosocial aspect of sexual pain 28 The information underneath each section is not apparent until the section is opened. A drop-down menu under each section preferred 29 Links to treatment options are currently limited to only the image and not the entire box where the image is located. Extend the link to the entire box 30 The website is one-sided in favour of female partners. Include male images 84 Table 5.4 Post-study systems usability questionnaire Item Mean System Usefulness Overall, I am satisfied with how easy it is to use this website 2.44 It was simple to use this website 2.00 I was able to complete the task and scenarios quickly using this website 2.38 I felt comfortable using this website 1.88 It was easy to learn to use this website 1.88 I believe I can know about sexual pain quickly using this website 2.25 Mean system usefulness 2.13 Information Quality The website gave me error messages that tell me that something went wrong 4.63 Whenever I made a problem using the system, I could recover easily and quickly 3.00 The information provided on the website is clear 2.13 It is easy to find the information I need 2.25 The information is effective in helping me complete the tasks and scenarios 2.38 The organization of the information on the screen is clear 2.13 Mean information quality 2.36 Interface Quality The user interface of this website was pleasant 1.75 I would like to use this website 2.25 This website has all the functions and capabilities I expect it to have 2.75 Overall, I am satisfied with this website 2.38 Mean interface quality 2.28 NOTE: Lower scores indicates better metrics of usability 5.7 Destigmatising Properties of Sexual Health-related Websites Although the focus of the stigma analysis was to identify generalities about designing destigmatising sexual health-related websites, participants largely referred to our website to illustrate their point. We identified participants’ responses that fit within the five main principles of Fallot and Harris’s trauma-informed care framework (Harris & Fallot, 2001). The five principles of trauma-informed care represent the global themes under which various sub-themes and concepts emerged. These themes represent participants’ perspectives on how to design destigmatising sexual health-related websites. 85 Figure 5.2 Global themes, sub-themes, and concepts based on trauma 5.7.1 Theme 1. Emotional safety According to Fallot and Harris (Harris & Fallot, 2001), the principle of emotional safety denotes that both the setting and interaction within services are psychologically harmless, 86 comfortable and easy to use. In other words, emotional safety means having an awareness of individuals’ discomfort or unease in using services. Based on this conceptualisation of emotional safety, most participants in our study viewed websites as platforms that can promote emotional safety by ensuring that users are not (re)stigmatised or (re)traumatised when using sexual health-related websites. Two subthemes emerged describing how stigma may be addressed through emotional safety. Appropriate use of web content: One area of concern among participants related to how images and content are displayed on web platforms. Many of the participants indicated that the display of sexual health-related content, particularly images, determines whether they will use a web platform or not. Others argued that the use of websites with explicit images will likely draw peoples’ attention to what they are watching, a phenomenon that many participants disliked. Participant 5, a 45-year-old woman stated that “When I heard this was a sexual pain website, I thought I was going to see some horrible stuff…..you know what I mean…… but that’s not the case. You have chosen your words carefully and I wouldn’t shy away from browsing this website, even on a bus.” Because of the potential for some web images to foment stigma, one participant suggested using images or words that would not easily draw people’s attention, especially in public places. Privacy and anonymity: Another strategy the participants identified as a means to ensure emotional safety via sexual health-related websites was to safeguard the privacy and anonymity of people who use web-based platforms. Several participants desired privacy and anonymity, indicating that they might feel uncomfortable disclosing their information to health providers or peers. Participant 1, a 37-year-old woman reported that 87 “I would not want to discuss with my doctor or family members because of the stigma. I mean… you know how people behave when it comes to sexual health issues….. So I don’t think I will feel comfortable discussing some of the issues you have on this website with my care provider, but I think this website can provide me with what I want without exposing myself.” Participants generally agreed that educational websites, such as ours, could enable people to access needed information without necessarily disclosing their private information or their identity. Participant 10, a 51-year-old woman specifically described that “websites should be able to protect peoples’ identity by using anonymous identities during an online interaction with others or health providers.” 5.7.2 Theme 2. Collaboration Collaboration denotes partnerships, and the recognition that healing happens in relationships and in a meaningful sharing of power, problems, and experiences (Harris & Fallot, 2001). Using our website as a reference, several participants indicated that web platforms can provide a novel opportunity for collaboration among people without revealing their identity. This form of collaboration can be either with people who have a similar condition or with healthcare professionals. Collaboration was regarded as important because many participants indicated that people with sexual pain tend to suffer alone, in isolation, and with limited or no opportunities to discuss or share their experiences with others. One subtheme, Interactive Engagement, emerged under collaboration. Interactive engagement: The desire to interact with peers via websites was a dominant theme described by majority of the participants. For instance, Participants 2 and 8 suggested the use of online chat rooms to promote active engagement among people with similar problems or to engage with healthcare providers for additional support. In reference to our website, 88 Participant 2 specifically stated that “Maybe find a way of connecting people, like, have a place where people would want to hear from each other.” More than half of the participants suggested that contact information for health professionals could be integrated into sexual health-related websites so that people could always establish an anonymous connection with healthcare providers for additional information that may not be contained on web platforms. This was specifically echoed by Participant 11, a 39-year-old woman – “if you have the contacts of people or those behind the website where I can contact them in case I need personalized information would be useful… don’t you think this will enhance the credibility and relevance of the information?” Although several participants generally agreed to the potential benefit of some form of online collaboration, a few others were worried about the potential privacy risk of online interactive engagement. One participant contended that a person’s identity could inadvertently be revealed through such engagements. 5.7.3 Theme 3. Empowerment In the context of trauma-informed care, empowerment is the recognition that an individual’s strengths are acknowledged, built on and validated (Harris & Fallot, 2001). In other words, empowerment is a strengths-based approach aimed at conveying a sense of optimism and hope, or expanding resources or an individual’s capabilities. When asked how websites may help to address stigma, several participants reported strategies that reflected Fallot and Harris’s principle of empowerment. These participants described empowering actions that could promote a sense of optimism and hope among people who use sexual health-related websites. These empowering strategies are categorized under the two main subthemes of Inclusiveness and Validating personal experiences. 89 Inclusiveness: Almost all participants noted that inclusiveness of people from diverse ethnicities, age groups, and sexes/genders on sexual health-related web platforms are important strategies to address sexual health-related stigma. For instance, there was consensus across several of the participants that the presence of partners in this type of web platform was crucial in promoting emotional wellbeing and addressing stigma-related concerns. About half of the participants suggested including images of partners as well as having a resource section on what partners could do to support and empower their partners who suffer from sexual pain. Citing our website as an example, Participant 4, a 33-year-old woman voiced that “I definitely think you need to get … partners involved in this as well because I think that’s quite important … you [are] definitely not gonna fix this problem or make things easier if one party is trying to do this in isolation without some perspectives from the other party or why they can’t understand what the other party is experiencing … so I think you definitely need to get that involvement in there.” Validating personal experiences: Several participants noted that validating each other’s experiences was an empowering strategy that may help address stigma. Participant 6, for instance, expressed the desire for a section on websites where participants could narrate their experiences as a way of empowering other people who have similar conditions. With specific reference to our website, she noted that “I think having a section where I can read about other people’s stories or hear someone narrate their experiences may be helpful to me.” Similarly, Participant 4 also indicated that “…. having personal experiences on websites can help people who are living in isolation because such experiences can help them overcome stigmatising problems that they otherwise would not want other people to know about.” Validating other people’s experiences were considered by many as a way to make people feel understood when using web-based platforms. A few participants also felt that having their experiences validated by others would mean they are not suffering in isolation. 90 5.7.4 Theme 4. Trustworthiness Trustworthiness denotes clarity in tasks or information, accuracy, consistency, transparency, and respectful boundaries (Harris & Fallot, 2001). Applying this principle in the context of health technologies, many participants reported strategies that they thought could help dispel the myths and misconceptions regarding the causes and possible effects of endometriosis and sexual pain. For instance, Participant 3 indicated that the “mere existence” of a website that accurately explains endometriosis and sexual pain could help dispel the myths and misconceptions that often fuel stigma. Participant 6, a 39-year-old woman also indicated that “a website such as this will clear a lot of doubts that many people are having regarding endometriosis and how sexual pain comes about.” The facts and evidence on our website were also seen as ways of promoting trust of web-based contents. The use of Factual content and Non-judgmental information were the two sub-themes that emerged under the global theme of Trustworthiness. Factual content: Several participants viewed factual content as a way of addressing stigma via websites. They specifically noted that the information on our website was very factual and was based on available evidence as seen in the references. One participant suggested that other sexual health-related websites should emulate the way our website provided factual content. Participant 6 emphasized, “Just as I have said before, the information on all websites should be backed by evidence like you have here. I can see you have references to some of your claims so I can know that there is some credibility to what I am reading.” While several participants were positive about the facts on the websites, a few others viewed the website as too medically oriented. In other words, the website did not portray patients’ perspectives of 91 endometriosis-associated sexual pain. For instance, Participant 2 noted that “you only presented the facts” without highlighting the psychosocial aspects or patient experiences of sexual pain. Non-judgmental: Several participants commented on and appreciated the non-judgmental nature of the information on the sexual health-related website. With reference to our website, Participant 2 (a 32-year-old woman) reported that “your message does not seem to cast doubts or pass judgment on people who suffer from endometriosis so it can be trusted by people.” The non-judgmental nature of the information was seen by participants as key to addressing the stigma that results from othering and stereotyping of people who live with an illness. Participant 4 indicated that “the information is not personal in nature so I don’t think it may be stigmatising. I don’t think it also stereotypes anyone.” 5.7.5 Theme 5. Choice Choice is the recognition for an individualized approach while strengthening peoples’ experiences of options in services (Harris & Fallot, 2001). In this study, about half of the participants indicated ways in which a website might provide diverse information that maximizes an individual’s choices or the diversity of options from which they could choose. A subtheme that emerged under choice relates to diversity of treatment information and opinions in web-based platforms. Diversity of web-based information: About half of the participants indicated that for a website to address stigma, it should facilitate peoples’ choices by including diverse information on the treatments, supports and resources for endometriosis used by a range of people. Awareness of diverse approaches to treating and managing endometriosis and sexual pain was considered by many to be essential to understanding the various approaches and options they 92 might seek further information about or even consider. Considering the private nature of endometriosis and painful sex, there were currently few opportunities to learn from others about what they had tried, both successfully and unsuccessfully. Participant 7 (a 42-year-old woman) specifically saw benefit in asking people who have endometriosis about how they manage stigma related to sexual pain. She said that “asking these people about stigma will help you develop something that can address the problem.” Participant 5 indicated that having different perspectives on the website will maximize a person’s choice of sexual pain and stigma management strategies. “Maybe my final suggestion will be to also get different views of people suffering from sexual pain into the website. I think it’s good to hear from different perspectives of how people manage the problem.” 5.8 Discussion This study aimed to assess the usability of the ‘Sex, Pain and Endometriosis’ website and assess for destigmatising properties of sexual health-related websites in general. The usability findings revealed that except for the absence of a search bar, the possible confusion with the “mechanisms tab”, and the little pop-up windows, participants generally found the website to be simple, uncluttered, quite easy to use, and satisfying. The system’s usefulness, information quality, and interface quality sub-scales of the PSSUQ all fell below 3 on average, indicating good usability and satisfaction with the website. It is possible that the system’s usefulness, information quality and the interface quality were generally perceived as positive because the information was presented in plain language and the website was quite basic so as not to pose usability challenges. Aside from the good PSSUQ ratings, participants also perceived the content of the website to be credible, evidence-based, non-judgmental, and appropriate to the age group 93 most affected by endometriosis-associated dyspareunia during the post-test interviews. The use of non-judgmental and age-appropriate content on the website is similar to the findings from previous studies which found websites to be age appropriate and non-judgemental (Parasar et al., 2017). However, the positive findings in this study are in sharp contrast to a review that found the content of 54 endometriosis websites to be fairly inaccurate, of poor quality, non-credible, and fairly difficult to read (Hirsch et al., 2017). The participants also provided suggestions and recommendations including improving the visibility of homepage information, explanations of endometriosis and dyspareunia, and providing a drop-down menu under each section heading to make it easier to find content. Despite the positive findings, the largely minor usability problems encountered suggested the need for some redesign before the website was launched. Participants were unsatisfied with the use of text-heavy information on the website. These findings also remind us that although more text may be needed to explain certain concepts, the frequent use of dense text on educational websites may not be favourable. Instead, images and bullet points could be more engaging to participants. The recommendations from the research participants informed the revision of the website in ways that better met the needs of potential users. This evaluation study underscores the importance of user feedback in designing patient-centered educational websites on sensitive and intimate health topics (Orenstein et al., 2019). To the best of our knowledge, this work is the first usability evaluation to assess for (de)stigmatising properties of sexual health-related online resources. Even though participants found the website to be generally non-stigmatising, the findings from the stigma analysis suggest that the website did not fully utilize the necessary stigma-alleviating strategies that can empower people to address stigma. For instance, the absence of male partners and scarce information on patients’ personal experiences were seen as non-empowering. The limitations of our website in 94 addressing stigma could be attributed to the fact that we did not systematically incorporate stigma prevention into our design process, even though it emerged during our needs assessment and landscape analysis. Despite these hindrances, this study provides preliminary evidence that suggests a trauma-informed approach may inform strategies that can help web designers in developing destigmatising websites. This is important because web platforms are increasingly used to disseminate information on sensitive health topics (Redston et al., 2018). Yet, no strategies or guiding principles exist to help designers in addressing stigma on websites despite stigma being a major issue for sexual health-related issues. As trauma and stigma are inherently intertwined and work to reinforce each other (Frieh, 2020; Whetten et al., 2012), the findings suggest that adopting a trauma-informed approach to developing digital platforms may help address stigma concerns among users of online resources. For instance, the anonymous interactive channels identified under the principle of collaboration may motivate people to reach out when in need of support, provide avenues for online consultation or engagement with health professionals and provide a channel for sharing coping mechanisms and supporting each other to overcome the stigma of pain symptoms. The findings from this study are in alignment with other studies that have adopted a trauma-informed approach to successfully design interventions that have helped address stigma concerns among people living with HIV/AIDS in other settings (Whetten et al., 2012). The study also shows how inclusiveness can be applied in web design to address sexual health-related stigma. Inclusive design approaches suggest all-encompassing ways websites could be optimized to be usable and acceptable to diverse populations with respect to race, ethnicity, gender, age, and other forms of human differences. While inclusiveness was a concern among the patient partners during our needs assessment, we cannot say our website fully 95 maximized the principles of inclusiveness as participants noted the absence of male partners during the stigma analysis. Future websites on sensitive health topics should start with the principles of inclusive design from the outset. With the principle of inclusiveness, this study extends the emphasis of inclusive design principles from a predominant focus on older adults and people with disabilities to include diverse human variability such as ethnicity, gender, and diversity in perspectives (Newell et al., 2011). Although the various strategies identified by participants could help in creating destigmatising websites, some of the strategies may have an inherent privacy risk. In other words, some of the strategies may potentially reveal rather than conceal a user’s identity to others. For instance, some participants were worried about the privacy risk of online communication features such as chatrooms. This worry was not surprising because the use of chatrooms and other online web features have been associated with privacy breaches in other studies (Ralph et al., 2011). Participants were not particularly worried about our website as it does not offer any interactive communications features. But they expressed privacy concerns for other sexual health-related websites that collect personal information. The findings reflected studies that suggest that anonymous websites or websites that do not collect or store any personal information including ours, may be useful for obtaining information on stigmatised conditions or symptoms such as sexual pain (Gannon et al., 2020; Muessig et al., 2015). However, participants’ suggestion for anonymous chatrooms in websites may be an avenue for abusive and online stigmatization by peers (Bauermeister et al., 2019). This could negate the intent to create a safe space. The potential privacy and security risk inherent in chatrooms suggests the need for special design considerations to make anonymous chatrooms or live chat safer. 96 Given that people may be exposed to all kinds of information from varying sources, the facts and evidence-based information on our website were seen as credible and reliable information that could ultimately address the misconceptions of endometriosis and sexual pain. Although there is still room for improvement, our website also ensures a full range of choices by showing different treatment options for endometriosis-associated dyspareunia. What was missing from fully maximizing the principle of choice was having diverse patient perspectives captured on the platform. Our team grappled with providing information about self-management or treatment strategies for which there is only anecdotal evidence, deciding in the end to highlight the information that has greater theoretical or empirical evidence to support its use. Furthermore, the desire for diverse information and that of factual information (under the principle of trustworthiness) were very interesting but contradictory to each other, but consistent with our team’s internal debate while creating the website. While the diversity of web-based information was considered essential to understanding the various options and approaches that people with endometriosis-associated sexual pain might take, evidence may not be available to support all of the different options people may desire. The need for websites to convey real-life experiences and facilitate interaction between people as a way of addressing sexual health-related stigma are consistent with previous studies (Lagu et al., 2019; Magee et al., 2012). These prior studies demonstrated how the use of positive language and patients’ personal experiences can promote the uptake of web-based resources. Also, the findings show that our website may be particularly useful for people who need evidence-based information on endometriosis-associated dyspareunia but are fearful of disclosing or discussing their symptoms with providers. 97 5.9 Limitations Instead of in-person usability testing, we opted for remote usability testing to comply with public health orders during the Covid-19 pandemic. To protect users’ privacy, we turned off the video and therefore, we could not observe body gestures and facial expressions that typically convey participants’ reactions during a think-aloud procedure (Alhadreti & Mayhew, 2018). However, given the sensitive nature of the topic, the remote usability testing might have turned out to be a blessing in disguise as participants had the opportunity to explore the website from their home, and in relative anonymity. Although participants emphasized the important role of partners in managing sexual health-related stigma, our sample included only people with diagnosed endometriosis and not their partners. This limitation is consistent with the broad sexual health literature where partner engagement in women’s sexual health research is particularly challenging, and in some cases disregarded (Davis et al., 2018, 2019; Richters et al., 2014). Future usability studies on websites that address sexual health issues should include partners to ensure an inclusive end-product. For an educational website on endometriosis-associated dyspareunia such as this, partner perspectives are particularly important because the psycho-social impact of sexual pain can be experienced by partners as well (Facchin et al., 2015). 5.10 Conclusion Websites on sensitive health topics are increasingly designed to provide anonymous platforms for people to obtain evidence-based information towards empowering users, dispelling myths, and alleviating stigma. However, no guiding principles exist to guide website designers in 98 ensuring that such websites address stigma associated with the conditions for which they have been designed. 5.11 Summary of Chapter 5 Chapter 5 provided patient perspectives on the usability of a sexual pain-website. Participants’ perspectives of designing destigmatising sexual health-related websites were also explored using a trauma-informed care framework. The study in Chapter 5 provided preliminary evidence on the usefulness of the trauma-informed care framework in designing sexual health-related digital technologies. Patients’ perspective on developing destigmatising websites in this chapter helped inform Study 3, which is presented next in Chapter 6. 99 Chapter 6: Developing Destigmatising Design Principles: A Delphi Study 6.1 Introduction As observed from Chapters 4 and 5, current sexual health-related technologies have not fully considered non-functional or emotional design components to address stigma. This suggests that the design of digital health technologies could benefit from guiding principles that would help website designers to address stigma during system design. Chapter 5 (Study 2) also provides preliminary evidence that suggests that a trauma-informed care framework could prove useful in designing destigmatising health-related technologies. Based on this preliminary evidence, this chapter (Study 3) sought to develop a set of destigmatising design principles. In this chapter, the methods and results for developing the destigmatising design principles are presented. The chapter concludes with a discussion of the findings and limitations. 6.2 Methods An online Delphi methodology was employed to develop a set of destigmatising design principles. In employing this methodology, participants appraised and critiqued a set of principles identified via a literature review as well as from Study 2 (Chapter 5). Given that it would typically be difficult and costly to gather subject matter experts in one location, the online Delphi study was considered the best possible methodology for engaging experts across different countries and continents. Stigma and sexual health experts appraised and critiqued each destigmatising design principle through an online survey. This online appraisal allowed participants from seven countries to participate at their own time and locations and perform the survey at their own pace (Meshkat et al., 2014). 100 6.2.1 Overview of the Delphi Methodology The Delphi methodology was first developed at RAND Corporation in 1954 and was mainly used by the American Military as a technique for technology forecasting studies (Murphy et al., 1998). Since then, it has become a popular research methodology for structuring communication and engaging the opinions of experts in a given field. The Delphi methodology is a highly structured and iterative consensus-building process that is repeated until the best possible level of consensus or a predetermined number of stages (usually 2-3) has been reached (Murphy et al., 1998). The Delphi technique has four main features including: anonymity, iteration, feedback, and the use of descriptive statistics to decide and report on the findings. Anonymity means that group members should remain concealed from each other to allow for free communication while minimizing biases and dominance by other group members (Skulmoski et al., 2007). Secondly, iteration describes the number of Rounds in which participants appraise the items (usually 2-3 rounds). Thirdly, feedback describes the process of providing participants with the aggregate results from previous rounds. This gives participants the chance to retain or revise their ratings (Skulmoski et al., 2007). The fourth and final feature involves the use of descriptive statistics to determine consensus among participants, measure stability, and interpret results. Depending on the nature and characteristics of the study participants, these activities may not apply to all Delphi studies, yet they all apply to the study presented here. The decision to employ a Delphi methodology for this dissertation research was based on the following reasons: 1) The study population was geographically dispersed and a Delphi study was a reasonable consensus-building methodology that allowed geographically dispersed participants to engage in the same consensus-building process (Hanafin, 2004); 2) A Delphi 101 methodology supports the collection and analysis of expert opinions on research topics that are ill-understood or minimally researched, including sexual health-related technologies (Hejblum et al., 2008); and 3) The iterative process of a Delphi study tends to specifically benefit the development of design principles by allowing experts to suggest items and critique others’ ideas (Douglass & Moustakas, 1985; McPherson et al., 2018). Furthermore, recent studies suggest that a Delphi methodology is considered one of the best approaches for reaching consensus on developing design heuristics and clinical guidelines (Chalmers & Armour, 2019; McMillan et al., 2016). 6.2.2 Philosophical Underpinnings of Delphi Study Currently, there is no consensus on a suitable philosophical approach for Delphi studies. Many authors, however, agree that the Delphi technique is mostly used as mixed-methods research that draws on both post-positivist and constructivist paradigms (Choi et al., 2020; Humphrey-Murto et al., 2017). In the post-positivist paradigm, the researcher assumes the role of an objective and uninvolved observer in the research process. The aim of the post-positivist research paradigm is to depict the objective single reality of a phenomenon. The post-positivist paradigm in the Delphi methodology is supported through the use of a survey instrument for data collection and statistical measures to determine consensus. Also, the inclusion of experts and the reductionist approach to the identification of survey items in a Delphi study all assume an ontological position of a single reality (Hanafin, 2004). Some aspects of the Delphi methodology also conform to constructivist perspectives. For instance, the process of generating ideas, participants’ comments and appraisal of items, the iterative consensus-building process, and individual subjective views regarding the topic under study in the Delphi process can be 102 understood as adhering to a constructivist paradigm. Despite straddling both post-positivist and constructivist paradigms, the epistemological basis for a Delphi technique could favour either a post-positivist paradigm or a constructivist paradigm depending on the purpose of the research. Just like previous Delphi studies, this study also straddles both post-positivist and constructivist approaches, and both paradigms were applied in generating and revising the items as well as determining consensus. On one hand, the revision of the items as well as the process of generating new items were largely based on the qualitative data. On the other hand, the decision on determining consensus was based on quantitative metrics. 6.2.3 Pre-Delphi Stage Before Round 1 of a Delphi study, it is usually recommended to generate a candidate list of items that will be deployed to research participants. The items can either be based on a predetermined list generated through a literature review or can be iteratively generated by experts in each round of the Delphi process through a process called ideas generation (Humphrey-Murto et al., 2017). In this study, a combination of literature review and expert ideas in each round were used to obtain candidate destigmatising design principles for the initial and subsequent rounds, respectively. Table 6.1 shows a candidate list of destigmatising design principles that were generated from a review of the literature on stigma as well as the design literature. The generation of these principles were guided by the Harris & Fallot (2001) framework of trauma-informed care described previously. In other words, design principles that aligned with and could advance the core principles of the Harris and Fallot framework were selected from the available literature. As a result, items were included as destigmatising design principles if they were considered to: 1) 103 Ensure the emotional safety of people living with sexual health conditions; 2) Enhance the trustworthiness of technology-based sexual health-related information; 3) Promote choice and ensure flexibility of web features; 4) Enhance collaboration between people with the same condition; and 5) Facilitate avenues that enable a patient to gain control of stigmatising situations. The items generated from the review also covered various stigma management strategies employed by earlier stigma researchers (e.g., Goffman, 1963; Heijnders & Van der Meij, 2006; Link & Phelan, 2001). To enhance clarity and ensure the appropriateness of the principles to the subject matter experts, design principles obtained from the review process were carefully worded into destigmatising principles. These lists were reviewed by the dissertation research supervisors and a university faculty member with expertise in health equity and trauma-informed care before they were deployed to study participants. All three rounds of the Delphi study were conducted in the form of a web-based survey. At Round 1, participants were asked to suggest additional design principles that were not identified during the literature review. The results from Round 1 (i.e., ratings, suggested design principles, and participants’ comments) were used to construct the survey for the Round 2. The results from Round 2 subsequently informed the Round 3 survey. To ensure narrative flow in this dissertation, the terms ‘item(s)’ and ‘design principle(s)’ are used interchangeably to refer to destigmatising design principle(s). Table 6.1 contains the preliminary destigmatising design principles that were identified from the literature for each of the five categories of trauma-informed care. An alphanumeric unique identifier (ID) was used to distinguish unique design principles from each other (e.g., DP1 = Design Principle 1). 104 Table 6.1 Preliminary destigmatising design principles and source(s) Trauma-informed Care Category ID Pre-Delphi Destigmatising Design Principles from Literature Review = 28 Emotional Safety (ensuring that the emotional safety of people with stigma is addressed in digital health technologies) DP1 Provide participants with the ability for anonymous engagement (Kientz et al., 2010; Marent et al., 2018) DP2 Encrypt websites that collect personal information to prevent unauthorized access to personal data (Gkatzidou et al., 2015) DP3 Prevent errors by having features that warn users from taking actions that can lead to accidental disclosure of status (Nielsen, 1994) DP4 Use non-sensational language and images on web platforms (Hood & Friedman, 2011) DP5 Logos, icons, and terminologies should be made subtle so that they do not immediately draw the users’ or bystanders’ attention to sexuality DP6 Reduce offensiveness of images by minimizing the display of explicit or profane sexual images on websites (Meisenbach, 2010) DP7 Avoid conveying negative experiences associated with a condition and rather emphasize positive messaging (Shearer et al., 2012) DP8 Use humour in a very positive and sensitive manner to lessen the fear and seriousness associated with a condition (Makoae et al., 2008; Penson et al., 2005) DP9 Avoid using language that seeks to blame, or cast moral judgment on people living with or infected by a condition (Pulerwitz et al., 2010) Choice (Ensure consumers have choice and control over the content and features of the website. Let the technology adapts to users need) DP10 Provide opportunities for selective and voluntary disclosure of status to provide emotional connection and support DP11 Provide interpersonal contacts for people to seek direct online counseling and other psychological support (Corrigan & Kosyluk, 2013; Hamill & Goldmeier, 2005; Warren & Ebel, 2005) DP12 Include links to other credible sites and contact information for users to seek further information DP13 Ensure appropriate flexibility of web features to allow a user to hide certain content/images that they find offensive (Chanco et al., 2019) DP14 Ensure customization features to enable users to adapt aspects of the interface to suit their visual preference (Kientz et al., 2010) DP15 Expose people to a range of messaging that addresses different aspects of stigma Note: DP = Design Principle 105 Table 6.1 Preliminary destigmatising design principles and source(s) (continued) Trauma-informed Care Category ID Pre-Delphi Destigmatising Design Principles from Literature Review = 28 Trust (clear, credible, and consistent information about the sexual health condition) DP16 Include information that seeks to correct wrong perceptions and myths surrounding a particular condition to enable users to reject negative and inaccurate beliefs DP17 Educate users about stigma by providing factual and plain language information that normalizes and de-stigmatises sexual health-related conditions (Hannigan & Jones-Devitt, 2004; Hood & Friedman, 2011; Shearer et al., 2012) DP18 Use positive, credible, and diverse language and images DP19 Use images and pictures of people from a variety of backgrounds who have experienced the condition (Hood & Friedman, 2011) DP20 Ensure a match between the system and the users’ culture (Paz et al., 2014) Empowerment (Gain control of stigma situations, under patients’ role, and encourages participation in anti-stigma activities) DP21 Use personal stories of renowned and relatable personalities who have experienced the condition DP22 Provide information on the fundamental rights of people affected by or living with the condition. E.g., Rights of persons living with HIV/AIDS DP23 Use inclusive language such as “we, our, or us” and “you are not alone” DP24 Use symbols of optimism, encouragement, and hope (Hood & Friedman, 2011) DP25 Provide mechanisms for creating online advocacy groups DP26 Include a video or written testimonials of real people to talk about stigma and their experiences with the condition DP27 Avoid othering and stereotyping people with a sexual health-related condition (National AIDS Trust, 2016) Collaboration (partnership, among patients and between patients and providers) DP28 Interventions for addressing men’s/women health or sexual health-related conditions should be developed and delivered in partnership with living with the condition Note: DP = Design Principle 6.3 Participant Recruitment, Data Collection, and Analysis 6.3.1 Sample and Sample Size Participant selection is of utmost importance in a Delphi study because participants must have an adequate understanding of the field to be able to provide a reasonable judgement (Chalmers & Armour, 2019). Instead of randomly sampling participants in the study population, 106 a Delphi methodology recommends a purposive selection of participants based on a predetermined set of criteria. Therefore, the sample for this study was selected through a purposive and snowballing sampling technique. This ensured the selection of information-rich participants with expertise that was relevant to the topic under investigation (Palinkas et al., 2015). According to Ericsson (2009), a person could be considered an expert in a field when they are engaged in specialized domain-related work continuously for 5,000-10,000 hours or 5-10 years, respectively. Based on this definition, potential study participants were expected to have met the following criteria to be considered an expert for inclusion in this study. Thus, a participants had to: 1) Be engaged in sexual health or stigma-related research or be an informatician with specialization in sexual health for a minimum of 5 years; 2) Have a minimum of an undergraduate degree; and 3) Be able to communicate in English. Although there is no standard sample size for a Delphi study, most Delphi studies tend to have 15-20 participants (Hirschhorn, 2019), with some occasionally having more participants depending on the purpose and available resources (Humphrey-Murto et al., 2017). Other studies suggest that 10-15 experts could be considered large enough, especially when the participants have homogeneous backgrounds (Hall et al., 2018). After considering the research objectives in this study and because research participants might have had varied experiences in stigma and sexual health-related research, it was decided that a purposive sample of approximately 15 to 20 participants would likely be large enough to extract, appraise, and reach a reasonable consensus on the destigmatising design principles. However, due to the high attrition rate often associated with Delphi studies, a number larger than 20 was recruited to start with. This was to ensure that the final Round included at least 10-15 participants. 107 6.3.2 Participant Recruitment Prior to recruitment, the study was approved through the local research ethics board (H21-00760). Participants were purposefully recruited through convenience and snowball sampling techniques. That is, participants who met the inclusion criteria were asked to suggest other potentially qualified participants. In Delphi studies, the required number of participants must be recruited before the start of the first round. No additional participants can be recruited once the first-round ends. To recruit the required number of participants before Round 1, a list of potential participants was created mainly through publications from sexual health and stigma-related technology research. A few other participants were recruited through conference speaker lists (i.e., the Together against Stigma, the Global Alliance Against Stigma, and “SX TECH” conference), and via web pages of journals on stigma and sexual health. The researcher also used professional networks to recruit some participants across North America. Since the objective of the research was to develop design principles for website designers without stipulating a specific geographical location, participants were recruited from mainly English-speaking countries. Participants with expertise in stigma-related research who were proficient in English yet were not resident in English speaking countries were also recruited. Recruitment from diverse settings may ensure that the final design principles can be applied to user interface design across different settings. A list of 158 names were compiled through the various recruitment sources outlined above. Each participant was subsequently contacted directly with an email containing a Qualtrics (Qualtrics International Inc.) link to a pre-qualifying checklist and a demographic survey (See Appendix B for the recruitment message and demographic survey). Interested participants were directed to click on a link to the Qualtrics survey. The link contained the following qualifying 108 questions: 1) Can you communicate in English Yes/No?; 2) Do you have at least a university degree Yes/No?; 3) Do you have 5 or more years of research or work experience Yes/No? The survey automatically closed for participants who answered No to any of these questions. Those who responded Yes to the 3 questions could continue to complete the demographic questionnaire including name, email address, age, country of work, gender, level of education, organization, and number of years of work experience, position, and consent to be contacted for the Delphi study. Participant recruitment occurred over approximately one month – from May 11th, 2021, to June 15th, 2021. For participants who had not responded to the first email, reminders were sent two weeks after the first email. The recruitment period ended when potential participants were no longer responding to the email reminders. Twenty-two participants responded to the survey. All 22 participants met the eligibility criteria and consented to be contacted for the Delphi study. The list of principles shown in Table 6.1 above were deployed to participants through the web-based survey. The consent form was included as a preamble to the main survey (See Appendix B). Participants who clicked on the survey link had the chance to review the consent form and understand more about the study and participant confidentiality. The survey automatically closed for participants who did not consent to the study. For participants who consented to the study, the main survey (with further instructions) then opened up. To maintain confidentiality, unique IDs were assigned to each participant. The IDs were used in place of participants’ names throughout the three Rounds of Delphi. The data were protected by enabling password protection in the Qualtrics Online Survey Software. Due to the amount of time usually associated with Delphi studies (Keeney et al., 2001), a CAD$15 amazon gift card (or its equivalent in the denomination of the country where the individual was located) was provided to 109 participants for completing each Round of the Delphi study. To minimize the attrition rate in this study, reminders were sent out each week and participants were also thanked and, at the end of each round, informed of the planned timing for the next round. 6.3.3 Data Collection There is no standard number of Rounds for collecting data in Delphi studies. Even though some Delphi studies can occasionally involve more than three rounds of data collection, two to three rounds of data collection are commonly used (Aronson et al., 2012). All things being equal, a Delphi study could be repeated until the best possible level of consensus, or a predetermined number of rounds has been achieved (Murphy et al., 1998). Theoretically, a Delphi study should not be terminated until a consensus has been reached on all items. However, to reduce participants’ fatigue, minimize the higher attrition rate often associated with lengthy Delphi studies (Hsu & Sandford, 2007), and ensure timely completion of the successive rounds, the number of rounds for collecting data in this study was limited to three. All 22 participants were sent an introductory email to take part in Round 1 survey (See Appendix B). Quiñones et al. (2018) proposed three main dimensions in which experts can appraise a set of design principles. These includes utility (how useful is the principle), clarity (how clear is the design principle), and ease of use (how easy can the principle be used). This study adopted the first two dimensions in developing the set of destigmatising design principles. That is, participants were required to rate each design principle across two dimensions of clarity and usefulness (Quiñones et al., 2018). While the third dimension ‘ease of use’ is involved in design decisions, it was excluded from this study because the study population (researchers in sexual health-related stigma) were not applying the design principles in practice to be able to 110 assess ease of use. Further, there was a plan to assess ease of use and applicability in Study 4 (see Chapter 7). At each stage of the Delphi process, participants were asked to rate the extent to which they agreed or disagreed with the clarity and usefulness of each of the design principles in addressing stigma. That is, participants were asked if a design principle was clear to them, and if it was useful for addressing sexual health-related stigma via websites. The appraisal was based on a 7-point Likert scale (where 1 = strongly disagree, 2 = disagree, 3 = mildly disagree, 4 = neutral, 5 = mildly agree, 6 = agree, and 7 = strongly agree). In addition to the ratings, a section for free-text comments was created under each destigmatising design principle for participants to justify their ratings or offer a qualitative critique on the clarity and usefulness of each item. A section was also created at the end of the survey for participants to suggest additional ideas or to write down their thoughts on what they believed could help in designing destigmatising health-related technologies. The first survey took approximately one hour for each participant to complete, the second survey took approximately 20-30 minutes, while the final survey took about 10-15 minutes. Participants were expected to complete each survey round within two weeks. However, given that participants were mostly research faculty members who might have been on vacation at some point in time during the Delphi study (i.e., June – September 2021), a one-week extension was given for the first and second rounds. Email reminders were sent one week and a day before the deadline for each survey round respectively. (See Appendix B for the introductory and reminder emails). 6.3.4 Data Collection in Round 1 In Round 1, the candidate list obtained from the literature review in the pre-Delphi stage were deployed to the 22 participants via the Qualtrics Online Survey Software (See Appendix C 111 for Round 1 Survey). Each destigmatising design principle was rated on a seven-point Likert scale across the two dimensions of clarity and utility. 6.3.5 Modifying and Generating Items for Round 2 The data from Round 1 were analyzed to determine design principles that achieved consensus. To reduce survey redundancy and alleviate participants’ fatigue during the second round, items meeting positive consensus at Round 1 were removed and placed as part of the final list of destigmatising design principles. However, because this was the first round, participants provided insightful comments to all design principles irrespective of whether an item achieved consensus or not. Therefore, these comments were used to revise eight items even though the items achieved consensus in Round 1. The revisions were done by changing the wording or phrasing of the item based on specific comments from the participants. Since these revisions could have possibly changed the intended meaning of the principles, the revised items were fed back to participants in Round 2. Destigmatising design principles that did not meet consensus were also revised the same way as that of the items that achieved consensus. See Table 6.5 (in the results section) for the original design principles and their corresponding revisions. The revised principles were each assigned an alphanumeric unique identifier with a suffix after a decimal to differentiate them from design principles in the previous rounds. For example, if a design principle that was revised was assigned DP20 in Round 1, it was assigned DP20.1 in Round 2, and if it was revised again, it was assigned DP20.2 in Round 3. An additional set of principles were also generated from participants' comments and suggestions. Similar to the revised principles, the newly generated principles were also assigned unique identifiers to differentiate them from the originally developed items. For instance, a new design principle 112 (NDP) that was created during Round 1 analysis was labeled NDP35 (e.g., if the new principle ‘NDP35’ did not meet consensus in Round 2, it was revised and assigned ‘NDP35.1’). Participants completed the Round 1 survey within three weeks and the results were analyzed to determine items that reached consensus (consensus analysis is described in section 6.4 below). 6.3.6 Data Collection in Round 2 The revised principles in Round 1, as well as the newly generated principles, were fed back to participants in Round 2. Just like Round 1, participants rated and critiqued each principle in Round 2. Participants completed the Round 2 survey within three weeks and the results were analyzed to determine consensus. 6.4 Modifying and Generating Items for Round 3 After completing Round 2, items reaching consensus were added to the final list of destigmatising design principles. Unlike Round 1, the items that achieved consensus in Round 2 did not have enough comments to warrant another revision. Free-text comments and additional suggestions were analyzed to generate additional design principles. Principles that did not achieve consensus in Round 2 were revised based on participants’ comments. In this round, stability was calculated on items that did not achieve consensus. Items not achieving consensus at this stage, but which were judged to be stable (i.e., showing less probability of change in the next round) were dropped from the survey. Stable items were dropped because items that show stability are less likely to change in subsequent rounds and do not need further assessment. Unstable design principles and newly generated design principles were revised and sent to participants in Round 3 for final appraisal. 113 6.4.1 Data Collection in Round 3 In Round 3, the revised principles as well as newly generated principles were sent to participants as the final round. Just like in Rounds 1 and 2, participants rated and critiqued each principle. Since this was the last round, additional ideas were not required from participants but were still requested using an overall comments box (i.e., participants could still suggest ideas by writing down their final thoughts regarding the destigmatising design principles in general). Items meeting consensus in Round 3 were added to the final list of design principles. Because this was the last round, no further revisions were made to the items that did not meet consensus. Instead, items not meeting consensus were not included in the final list. 6.4.2 Data Analysis At the end of each round, two types of data were generated for each destigmatising design principle including: 1) Quantitative data from Likert scale responses across the two dimensions; and 2) Qualitative data from participants’ comments and suggestions. Descriptive statistics (i.e., interquartile range, and a content validity index) were calculated for the quantitative data. Participants’ comments that reflected additional destigmatising design principles were generated from the text (qualitative data). The details of the quantitative and qualitative analytic processes are outlined below. 6.4.3 Quantitative Data Analysis 6.4.4 Level of Consensus Consensus in Delphi studies generally represents the level of unity among participants’ rating of items. There are different methods to establish consensus among experts in Delphi 114 studies. These include the percentage level of agreement, otherwise known as the content validity index (CVI), the inter quartile range (IQR) or standard deviation, and the stability of items. To enhance the credibility of results, some Delphi studies have used a combination of two criteria in determining consensus (Chen, 2019; York & Ertmer, 2011). Based on this prior evidence and the desire to obtain credible results in this study, a combination of the CVI, the IQR, and the stability of items were used to determine consensus of items. 6.4.5 Content Validity Index According to Polit, & Beck (2012), the CVI is an index summarizing the degree to which a panel of experts agree on an instrument’s content. In this study, the CVI was calculated as the number of experts who selected ‘agreed’ (i.e., mildly agree, agree, and strongly agree) for an item, divided by the total number of experts. The CVI was the first criteria for determining if consensus had been achieved among participants about which items to include or discard. Researchers using Delphi studies have used different percentage levels for determining the participants’ consensus. Even though a 51% level of agreement or higher could be considered large enough for an item to achieve consensus (Diamond et al., 2014), recent Delphi studies have generally set the level of agreement at ≥ 70% (Choi et al., 2020; Strudwick et al., 2019). It must be noted that there are trade-offs in setting a very high percentage level of agreement for defining consensus in Delphi studies. On one hand, a higher percentage level of agreement will enhance the credibility and precision of the results. On the other hand, setting a very high level of agreement may decrease the probability of reaching consensus and increases participants’ fatigue and attrition rate. This is because items not reaching consensus would be repeatedly sent back to participants until they achieved high consensus. Despite recent studies that used ≥ 70% for 115 achieving consensus, others indicate that best practices in Delphi studies is ≥ 80% for meeting consensus among experts (Polit, & Beck, 2012). This study adopted Polit, & Beck (2012) recommended percentage levels of ≥ 80% for determining the proportion of people that agreed or disagreed on the clarity and usefulness of each destigmatising design principle. Therefore, design principles with a CVI of ≥ 80% on both clarity and usefulness dimensions were considered as having reached consensus. Design principles with a CVI less than 80% on either of the dimensions were considered as not having reached consensus and would need to be revised (Polit et al., 2007). For example, an item in this study would not be considered as having reached consensus if its CVI was more than 80% in the clarity dimension but less than 80% in the usefulness dimension. 6.4.6 The Inter-Quartile Range To ensure greater credibility of the results, the IQR was used as the second criteria in determining consensus. An IQR generally describes the measure of dispersion, and it is equal to the difference between the 25th and 75th percentiles (Wan et al., 2014). The level of IQR for determining consensus usually depends on the purpose of the study. In most Delphi studies, the IQR is generally related to the level of the scale used. For instance, a 9-point Likert scale is likely to have a larger IQR than a five or seven-point Likert scale. Consistent with prior Delphi studies (Humphrey-Murto et al., 2017), the IQR for this study was set at £ 1. That is, to achieve consensus, an item must have achieved an IQR of £ 1 on both clarity and usefulness dimensions. Items with an IQR above 1 on either or both dimensions were considered as not having reached consensus. The IQR range is chosen as the measure of dispersion for this study because the IQR is not affected much by outliers, and it is also considered a better measure of dispersion for 116 smaller sample sizes than a standard deviation. That is, the lower the IQR, the smaller the distance between the highest and the lower values in the data, and the higher the IQR, the greater the distance between the highest and lowest value in the data. This means a lower IQR indicates a higher level of agreement and vice versa. The IQR helped to establish the level of variation within items that had reached or had not reached consensus. 6.4.7 Stability of Items Stability refers to the consistency of answers between two successive rounds of a Delphi study (Dajani et al., 1979). Stability is calculated on items not meeting consensus to determine which items to drop and which will proceed to the next round. Depending on the number of Rounds, stability can be calculated during the second or third round when participants would have provided responses to at least two Rounds. Stability calculations provide a measure of change in participants’ responses for items between successive rounds. Because this study was limited to three rounds, stability was calculated in Round 2 to determine how many participants changed their ratings for items that did not meet consensus between the first and second rounds. Stability was calculated by following the Scheibe, Skutsh, & Schofer (2002) guidelines for determining the stability of items in Delphi studies. According to Scheibe et al. (2002), stability can be calculated by: 1) Estimating the frequency of participants’ responses for each item in each round; 2) Calculating the absolute difference between the frequencies in the first and second rounds; 3) Adding the unit of change across all items; 4) Dividing the summed units by the number of rounds; and 5) Dividing the results from step 4 by the number of participants. For instance, if 5 participants responded “agree” to an item in the first round and 7 participants responded agreed in the second round, the absolute difference will be 2. 117 The final absolute score could range from 0% to 100%. A higher score suggests that there are wide variations in participants' responses between the two rounds and those items may benefit from another round. A lower score suggests participants' responses for an item between two rounds are likely the same and may not change from another round. There is no universally accepted cut-off limit for determining the stability of items in Delphi studies. However, previous studies have generally placed stability levels between 15% and 20% (Carmody, 2013; Scheibe et al., 2002; York & Ertmer, 2011). Based on this guidance, the stability level for this study was set at 20%. Therefore, for an item to be considered stable, it must have had a stability level ≤ 20% on both the clarity and usefulness dimensions. Items that did not meet consensus but that achieved a stability level of ≤ 20% were discarded, while non-consensus items with stability levels above 20% were revised and sent to participants in Round 3. Non-consensus items that achieved stability were dropped because there is a very low probability of achieving consensus if the items were sent to the subsequent round. 6.4.8 Qualitative Data Analysis The qualitative data were used to revise items by rewording or rephrasing some items based on specific comments from participants. Additional design principles were also developed using qualitative data by taking statements directly from participants' comments or reframing suggestions into relevant principles. The new items generated from participants’ qualitative comments each round as well as the revised items were fed back to participants in subsequent stages. In effect, the qualitative data were converted to items that were rated on a Likert scale. These items were then analysed using the quantitative methods described above. 118 6.4.9 Validity, Reliability, and Methodological Rigour of Delphi Studies Usually, the nature of a Delphi study helps to enhance the validity of the study. According to Hasson & Keeney (2011), instead of enhancing only the validity and reliability of Delphi studies, researchers employing this methodology should generally consider improving methodological rigour by establishing the trustworthiness and credibility of the Delphi methodology. Validity in the context of quantitative research design is “a quality criterion referring to the degree to which inferences made in a study are accurate and well-founded,” while reliability refers to “the extent to which scores for people who have not changed are the same for repeated measurements” (Polit, & Beck, 2012). To further enhance methodological rigour the study also adhered to Humphrey-Murto et al. (2017) recommendations for conducting rigorous Delphi studies including: 1) Defining the study objectives; 2) Describe in detail how items were generated for the Delphi study; 3) Describe in detail how participants were selected; 4) Describe a-priori, the number of rounds, and the criteria for ending the Delphi study; 5) Iteratively conduct anonymous rounds of surveys to pool expert opinions and describe how consensus will be determined and anonymity maintained; and 6) Collect individual and group feedback to identify convergence/divergence of opinions and describe/provide feedback after each round and report the results. 6.5 Results This section begins by providing an overview of the research participants and their demographic characteristics. This is followed by the findings for each round of the Delphi study describing in detail the design principles that achieved or did not achieve consensus. A summary of the findings for the Delphi study is presented at the end of the chapter. 119 6.5.1 Demographic Information Of the 158 participants who were contacted, 22 (13.9%) completed the demographic and qualifying survey. All participants who completed the survey met the qualifying criteria and consented to be contacted for the Delphi study. Although the original plan was to recruit participants from only English-speaking countries, the participants from Germany, the Netherlands, Rwanda, and Switzerland were recruited because they had extensive expertise in the subject matter and were proficient in English. The mean number of years of work experience in sexual health and stigma-related research among the participants was 21.6 years (SD, 10.26) and ranged from 6-46 years. Except for two participants who were working in public health and a consulting agency, the other 20 participants were employed as faculty members in research-intensive universities. Demographic characteristics are shown in Table 6.2. Table 6.2 Demographic characteristics Characteristic N (%) Screened = 22 N (%) Participated = 15 Gender Woman 13 (59.1) 10 (66.6) Man 8 (36.3) 4 (26.6) Not reported 1 (5.54) 1 (6.6) Title/ Position Assistant Professor 6 (27.3) 3 (20) Associate Professor 5 (22.7) 4 (26.6) Professor 6 (27.3) 5 (33.3) Other 5 (22.7) 3 (20) Country Canada 8 (36.3) 5 (33.3) United States 7 (31.8) 4 (26.6) United Kingdom 3 (13.6) 2 (13.3) Germany 1 (5.54) 1 (6.6) Netherlands 1 (5.54) 1 (6.6) Rwanda 1 (5.54) 1 (6.6) Switzerland 1 (5.54) 1 (6.6) 120 6.6 Round 1 Survey Of the 22 participants who started Round 1, fifteen (68%), completed the survey in the first round. The data were downloaded and entered into SPSS (IBM® SPSS Statistics for Macintosh) version 26 and each item was analyzed to determine consensus. However, during data cleaning it was observed that one respondent only partially answered many of the questions and their submission was subsequently removed. Fourteen responses were finally analyzed. Using the recommended CVI and IQR criteria outlined above, twelve design principles met consensus in Round 1 by achieving a CVI of ≥ 80% and an IQR of ≤ 1 on both clarity and usefulness dimensions. The CVI for design principles that reached consensus ranged from 85.7% to 100% while the IQR ranged from 0-1. Only four design principles achieved a CVI of 100% on both clarity and usefulness dimensions. These include DP1 (i.e., Provide participants with the ability for anonymous engagement), DP10 (i.e., Provide selective and voluntary disclosure of status), DP9 (Avoid language that seeks to blame or cast moral judgement), and DP27 (i.e., Avoid othering and stereotyping people with a condition/disorder). Two design principles, DP17 (i.e., Provide factual and plain language information) and DP18 (i.e., Use positive, diverse, and credible language) achieved a 100% consensus on usefulness but not on clarity. One design principle, DP2 (i.e., Encrypt websites that collect personal information), had 100% consensus on clarity but not usefulness. Among the items that reached consensus, DP9 (i.e., Avoid using language that seeks to blame, or cast moral judgement) and DP28 (i.e., Develop interventions in partnership with people with the condition) had the lowest level of variability in participants’ responses by having an IQR of 0 on both clarity and usefulness dimensions. While only 12 of the original 28 design principles met consensus in Round 1, participants provided insightful comments that warranted the revision of eight of the items that met 121 consensus. That is, only four of the original 28 design principles were retained in their original state as part of the final set of destigmatising design principles. Table 6.3 shows the design principles that achieved consensus at the end of Round 1. Design principles with asterisks indicate those that were retained in their original form. Those without asterisks are principles that were revised for Round 2. Table 6.3 Design principles that achieved consensus in Round 1 ID Design Principle Clarity Usefulness IQR CVI IQR CVI DP1* Provide participants with the ability for anonymous engagement 1 100 1 100 DP2* Encrypt websites that collect personal information to prevent unauthorized access to personal data 1 100 1 92.8 DP9 Avoid using language that seeks to blame, or cast moral judgment on people living with or infected by a condition 0 100 0 100 DP10 Provide opportunities for selective and voluntary disclosure of status to provide emotional connection and support 1 100 1 100 DP11 Provide interpersonal contacts for people to seek direct online counseling and other psychological support 1 85.7 1 85.7 DP16 Include information that seeks to correct wrong perceptions and myths surrounding a particular condition to enable users to reject negative and inaccurate beliefs 1 92.8 0 100 DP17 Educate users about stigma by providing factual and plain language information that normalizes and de-stigmatises sexual health-related conditions 0 92.8 0 100 DP18 Use positive, credible, and diverse language and images 1 85.6 0 100 DP19 Use images and pictures of people from a variety of backgrounds who have experienced the condition 1 92.8 0 92.8 DP26 Include a video or written testimonials of real people to talk about stigma and their experiences with the condition 1 92.8 1 85.7 DP27* Avoid othering and stereotyping people with a sexual health-related condition 0 100 0 100 DP28* Interventions for addressing men's/women health or sexual health-related conditions should be developed and delivered in partnership with living with the condition 1 92.8 0 92.8 Note: DP = Design Principle. Asterisks indicate items that were retained in their original state as part of the final lists of principles. 122 6.6.1.1 Non-consensus Items in Round 1 Of the 28 original principles, 16 did not achieve consensus in Round 1. Despite not reaching consensus, four items achieved consensus on at least one criterion. Design principles including, DP4 (i.e., Use non-sensational language and images), DP12 (i.e., Have links to credible websites), DP21 (i.e., Use personal stories of renowned people), and DP22 (i.e., Provide information on the fundamental rights of people) achieved consensus on the first criterion by having a CVI of more than 85% on both clarity and usefulness dimensions. However, these design principles had an IQR of more than 1, suggesting that there was variability in participants’ responses indicating a lack of consensus. Apart from these four, the remaining design principles in this category failed to achieve consensus on either CVI or IQR for either clarity or usefulness. The most disagreed upon design principle was DP3 (i.e., prevent errors by having features that warn users from taking certain actions). This principle achieved a CVI < 36% and an IQR > 3 on the clarity and usefulness dimensions respectively (see Table 6.4 for details of each principle). Table 6.4 Design principles that did not achieve consensus in Round 1 ID Design Principle Clarity Usefulness IQR CVI IQR CVI DP3 Prevent errors by having features that warn users from taking actions that can lead to accidental disclosure of status 3 35.7 4 35.7 DP4 Use non-sensational language and images on web platforms 2 85.7 2 92.8 DP5 Logos, icons, and terminologies should be made subtle so that they do not immediately draw the users’ or bystanders’ attention to sexuality 3 42.8 3 35.7 DP6 Reduce offensiveness of images by minimizing the display of explicit or profane sexual images on websites 4 57.1 4 50.0 DP7 Avoid conveying negative experiences associated with a condition and rather emphasize positive messaging 4 71.4 4 64.2 Note: DP = Design principle 123 Table 6.4 Design principles that did not achieve consensus in Round 1 (continued) ID Design Principle Clarity Usefulness IQR CVI IQR CVI DP8 Use humour in a very positive and sensitive manner to lessen the fear and seriousness associated with a condition 2 57.1 1 50.0 DP12 Include links to other credible sites and contact information for users to seek further information 1 85.7 2 85.7 DP13 Ensure appropriate flexibility of web features to allow a user to hide certain content/images that they find offensive 2 78.5 3 57.1 DP14 Ensure customization features to enable users to adapt aspects of the interface to suit their visual preference 3 78.5 3 71.4 DP15 Expose people to a range of messaging that addresses different aspects of stigma 4 71.4 2 78.5 DP20 Ensure a match between the system and the users’ culture 4 64.2 2 78.5 DP21 Use personal stories of renowned and relatable personalities who have experienced the condition 2 92.8 2 85.7 DP22 Provide information on the fundamental rights of people affected by or living with the condition. e.g. Rights of persons living with HIV/AIDS 1 92.8 2 100 DP23 Use inclusive language such as “we, our, or us” and “you are not alone” 3 71.4 3 64.3 DP24 Use symbols of optimism, encouragement, and hope 3 71.4 2 78.5 DP25 Provide mechanisms for creating online advocacy groups 3 71.4 3 64.3 Note: DP = Design principle 6.6.2 Revision of Design Principles in Round 1 Revision of design principles in Round 1 included the eight design principles that achieved consensus (but had valuable comments) as well as the 16 design principles that did not achieve consensus. Participants’ comments for the eight principles that achieved consensus mainly reflected problems with a lack of clarity of items. Therefore, the revisions were mainly done to improve items’ clarity. For instance, Participant 16 suggested that the use of “variety of backgrounds” in DP19 (i.e., Use images and pictures of people from a variety of backgrounds) was not clear enough. They thus suggested the use of more specific terms that define a variety of 124 backgrounds such as ethnicity, age, and gender identity. In the same way, participants indicated that the use of “selective and voluntary disclosure” in DP10 (i.e., Provide opportunities for selective and voluntary disclosure of status) does not apply to websites that do not have interpersonal contact (e.g., patient education websites). This principle was thus revised to be more specific to websites that provide opportunities for interpersonal contact. Furthermore, the use of the phrase “provide interpersonal contacts” in DP11 (i.e., Provide interpersonal contacts for people to seek direct online counseling) was seen as unclear. This item was thus revised to “provide contact information.” Some design principles were assessed by participants as doubled-barreled. For instance, Participant 16 stated that the use of the phrases – “testimonials to talk about stigma” and “testimonials to talk about experiences” in DP26 (i.e., Include a video or written testimonials of real people to talk about stigma and their experiences with the condition) can mean two different things. To make this item clearer, it was revised to “Include videos/testimonials that center on people’s experiences with the condition, including stigma.” Similarly, Participant 9 also viewed DP17 (i.e., Educate users about stigma by providing factual and plain language information that normalizes and de-stigmatises sexual health-related conditions) as double-barreled. It was thus revised to “provide factual and plain language information that normalizes and de-stigmatises sexual health-related conditions.” The 16 items that did not meet consensus in Round 1 were revised the same way as those that met consensus. Unlike the eight items that met consensus, the revisions of the other 16 design principles were mainly done to improve on their usefulness. Generally, participants' comments suggested that design principles that were meant to hide or prevent people from being exposed to stigmatising content might rather perpetuate or worsen stigma. For instance, design 125 principles including, DP3 (i.e., Prevent errors by having features that warn users from taking actions), DP13 (i.e., Ensure appropriate flexibility of web features), DP5 (i.e., Make logos, icons, and terminologies subtle), and DP6 (i.e., Minimize the display of offensiveness of images on websites) were seen as perpetuating stigma by encouraging suppression of peoples’ feelings and ability for expression. For instance, Participant 9 stated that “making logos and icons subtle may signal to the users to go back into hiding…. They need to develop a sense of pride if they want to live a life that is safe and respected.” Participant 1 also commented on DP13 (i.e., Ensure appropriate flexibility of web features to allow a user to hide certain content/images that they find offensive) by stating that “selecting certain content/images could inadvertently perpetuate stigma…… and also, allowing users to hide content/images they find offensive seems more likely to perpetuate stigma than reduce it.” Furthermore, warning users about disclosure in DP3 was also seen by participants as more likely to aggravate stigma than alleviate it. For instance, Participant 13 stated that “placing extra focus on the precariousness of disclosure may actually lead to increased anticipated stigma” while Participant 19 indicated that “warning users will likely pick up on issues that they might not even think could lead to stigma.” Participant 6 further reiterated that DP7 (i.e., Avoid conveying negative experiences associated with a condition) was seen as amounting to “dishonesty or failure to address relevant issues simply because they are negative.” Consequently, the principles that sought to hide potentially stigmatising content were revised accordingly to reflect participants’ comments and suggestions. Participants also suggested that some design principles were useful but not relevant to stigma reduction. For instance, Participant 13 commented on DP12 (i.e., Include links to other credible sites and contact information) by stating that “information provision is relevant and first step but does not lead to actual stigma reduction.” Similarly, the use of customization in DP14 126 (i.e., Ensure customization features to enable users to adapt aspects of the interface), was seen as useful for enriching participants’ experiences in using websites but was not considered relevant to stigma reduction. Participants also noted that some design principles overlapped with or were covered by other principles. For instance, design principles including DP14 (i.e., Ensure customization of web features), DP4 (i.e., Use non-sensational language and images on web platforms), and DP24 (i.e., Use symbols of optimism, encouragement, and hope), were generally seen by participants as overlapping with DP13 (i.e., Ensure appropriate flexibility of web features to allow a user to hide certain content/images that they find offensive). Similarly, DP6 (i.e., Reduce offensiveness of images) overlapped with DP5 (i.e., make logos, icons, and terminologies subtle). In addition to these, DP3 (i.e., Prevent errors by having features that warn users from taking actions) was covered under a newly generated principle – NDP32 (i.e., Have a function that allows people to leave the website immediately), while DP20 (i.e., Ensure a match between the system and the users’ culture) was covered under the newly generated principle NDP34 (i.e., Involve all those affected by the condition, not just people with the condition. e.g., partners of people with the condition). Design principles that overlapped with or were covered by other items were not revised and were thus removed from Round 2. Tables 6.5 and 6.6 contain the original principles and their corresponding revisions. The eight items in Table 6.5 are the design principles that achieved consensus but were revised, while the items in Table 6.6 are the design principles that did not achieve consensus and were revised. The revisions were reviewed by the supervisory committee members before they were rolled out to participants in Round 2. 127 Table 6.5 Original and revised principles for items meeting consensus in Round 1 ID Original Design Principle Revised ID Revised Design Principle DP5 Logos, icons, and terminologies should be made subtle so that they do not immediately draw the users’ or bystanders’ attention to sexuality DP5.1 Use logos, icons, and terminologies that clearly depict the topic for the website DP7 Avoid conveying negative experiences associated with a condition and rather emphasize positive messaging DP7.1 Ensure a balanced presentation of information by presenting both positive and negative experiences about a sexual health issue DP8 Use humour in a very positive and sensitive manner to lessen the fear and seriousness associated with a condition DP8.1 Use humour only if appropriate and do extensive user testing DP9 Avoid using language that seeks to blame, or cast moral judgment on people living with or infected by a condition DP9.1 Avoid using language that seeks to blame or judge people living with a condition DP10 Provide opportunities for selective and voluntary disclosure of status to provide emotional connection and support DP10.1 For websites that provide interpersonal connection, provide opportunities for selective and voluntary disclosure of status to provide emotional connection and support DP11 Provide interpersonal contacts for people to seek direct online counseling and other psychological support DP11.1 Provide contact information for people to seek online counseling or other psychological support DP12 Include links to other credible sites and contact information for users to seek further information DP12.1 Use hyperlinks to direct users to other websites that provide more information DP13 Ensure appropriate flexibility of web features to allow a user to hide certain content/images that they finds offensive DP13.1 Use images that foster kindness, happiness, connection, support, engagement, respect, and community Note: DP4, DP6, DP14, DP20, and DP24 overlapped with other principles and were not revised. 128 Table 6.6 Original and revised principles for items not meeting consensus in Round 1 ID Original Design Principle Revised ID Revised Design Principle DP15 Expose people to a range of messaging that addresses different aspects of stigma DP15.1 Include a range of information that touches on different aspects of the condition DP16 Include information that seeks to correct wrong perceptions and myths surrounding a particular condition to enable users to reject negative and inaccurate beliefs DP16.1 Include information that corrects myths about a condition, to enable users to get accurate information DP17 Educate users about stigma by providing factual and plain language information that normalizes and de-stigmatises sexual health-related conditions DP17.1 Provide factual and plain language information that normalizes and de-stigmatises sexual health-related conditions DP18 Use positive, credible, and diverse language and images DP18.1 Selection of language/images should be done in consultation with the community to ensure diversity DP19 Use images and pictures of people from a variety of backgrounds who have experienced the condition DP19.1 Use images of people from diverse ethnicity, age, and gender identity who have experienced the condition DP21 Use personal stories of renowned and relatable personalities who have experienced the condition DP21.1 Use personal stories of people who have experienced the condition. e.g. celebrities, or lesser-known personalities DP22 Provide information on the fundamental rights of people affected by or living with the condition. e.g., Rights of persons living with HIV/AIDS DP22.1 Consider including links to information on the fundamental rights of people affected by or living with the condition DP23 Use inclusive language such as “we, our, or us” and “you are not alone” DP23.1 Use inclusive language that is sensitive to contextual issues. e.g. partner instead of husband/wife, person instead of woman/man DP25 Provide mechanisms for creating online advocacy groups DP25.1 Provide a secure link or a section for creating online support groups DP26 Include a video or written testimonials of real people to talk about stigma and their experiences with the condition DP26.1 Include videos/testimonials that center on people’s experiences with the condition, including stigma Note: DP4, DP6, DP14, DP20, and DP24 overlapped with other principles and were not revised. 6.6.3 Newly Generated Principles in Round 1 Eight new design principles (NDP) were generated from participants’ comments and suggestions. For instance, Participants 1 and 7 suggested a functional principle that would enable people to quickly exit websites that contain stigmatising content. Participant 1 specifically related this suggestion to violence prevention sites by stating that…… "have a button that allows 129 participants to leave the website immediately …… you often see this functionality on violence prevention sites.” Another new design principle – NDP30 (i.e., Develop trust by providing options for different gender identities) – was generated from a comment for DP10 (i.e., Provide opportunities for selective and voluntary disclosure of status to provide emotional connection and support). Participant 8 critiqued DP10 by stating that “instead of just saying provide opportunities for selective disclosure, I will flip this to say provide many options that caters for different gender identities like homosexuals, bisexuals and other LGBTQ people.” The participants also suggested additional principles to address gender and sexual identities. While the original principles did not specifically address stigma related to different sexual identities, participants reiterated that it is important to have the design principles address issues of sexual identity. Participant 9 specifically stated that "What is important is that sexual identity and sexual health information match. If I am bi-sexual, then my information should include the risks of sleeping with both genders and I want to see that in your principles in round 2.” This statement was reframed into NDP30 (i.e., Develop trust by providing options for different gender identities). The issue of gender identity was also captured in DP23.1 (i.e., Use inclusive language that is sensitive to contextual issues. e.g., partner instead of husband/wife, person instead of woman/man). Participants also suggested ideas about end-users sharing commonalities about each other’s experiences as a way of alleviating stigma. Participants particularly emphasized the importance of sharing commonalities of what is experienced among people with similar conditions and most importantly, sharing information that is factual. Participant 15 for instance stated that “I think the key ideas here are around normalizing the conditions, which can be accomplished by sharing others' stories and getting them connected to other people with the 130 condition.” Participant 17 further suggested an additional principle by stating that “create collaboration spaces through the website so that those with similar experiences can connect.” Table 6.7 shows the additional eight destigmatising design principles that were developed from participants’ comments in Round 1. Table 6.7 Newly generated design principles in Round 1 ID Newly generated design principle NDP29 Provide factual content by including references to all information NDP30 Develop trust by providing options for different gender identities NDP31 Have a section for website users to narrate their stories NDP32 Have a function that allows people to leave the website immediately NDP33 Create collaboration spaces (forums, chatrooms) on the website so that those with similar experiences can connect NDP34 Involve all those affected by the condition, not just people with the condition. e.g., partners of people with the condition NDP35 Ensure rigorous methods to know the target audience NDP36 Have clear, factual, and neutral information Note: NDP = New Design Principle 6.7 Round 2 survey The Round 2 survey began with a set of 26 design principles. These include the eight principles that met consensus in Round 1 but were revised, ten design principles that did not meet consensus in Round 1 and were revised, and another additional eight design principles that were generated from participants' comments and suggestions. These 26 design principles were sent to the 15 participants who completed the Round 1 survey. However, only 14 participants responded. Upon a cross check of the data, it was found that the participant who provided incomplete responses in Round 1 was the same person that did not respond to the Round 2 survey. 131 6.7.1 Consensus Items in Round 2 Twelve design principles achieved consensus in Round 2. Among the items that achieved consensus, seven had already achieved consensus in Round 1 but were revised, three were additional principles generated from Round 1, while two were revised principles from items that did not achieve consensus in Round 1. Design principles, including DP9.1 (i.e., Avoid language that has a tone of blame or judgment), DP16.1 (i.e., Include information that corrects myths about a condition), DP17.1 (i.e., Provide factual and plain language information) and DP18.1 (Selection of language/images should be done in consultation with the community) achieved the highest level of consensus by having a CVI of 100% and IQR of ≤ 1 across the clarity and usefulness dimensions, respectively. Design principles including, DP26.1 (i.e., Include videos/testimonials that center on people’s experiences), DP11.1 (i.e., Provide contact information of counselors), NDP30 (i.e., Develop trust by providing options for different gender identities), and NDP35 (i.e., Ensure rigorous methods to know the target audience) all achieved consensus via IQR, they achieved the lowest level of consensus by having a CVI of at least 84% in either clarity or usefulness. Unlike Round 1, participants’ comments for the design principles that met consensus were not substantial enough to warrant another revision. Table 6.8 contains the 12 items that achieved consensus in Round 2. 132 Table 6.8 Design principles that achieved consensus in Round 2 ID Design Principle Clarity Usefulness IQR CVI IQR CVI DP9.1 Avoid using language that has a tone of blame or judgment of people living with the condition 1 100 1 100 DP11.1 Provide contact information of counselors or other psychological supports 1 92 1 84 DP16.1 Include information that corrects myths about a condition, to enable users to get accurate information 0 100 1 100 DP17.1 Provide factual and plain language information that normalizes and de-stigmatises sexual health-related conditions 1 100 1 100 DP18.1 Selection of language/images should be done in consultation with the community to ensure diversity 1 100 1 100 DP19.1 Use images of people from diverse ethnicity, age, and gender identity who have experienced the condition 1 100 1 92 DP22.1 Consider including links to information on the fundamental rights of people affected by or living with the condition 1 100 1 92 DP23.1 Use inclusive language that is sensitive to the context of the condition e.g., partner instead of husband/wife, the person instead of woman/man 1 92 1 100 DP26.1 Include videos/testimonials that center on people’s experiences with the condition, including stigma 1 92 1 84 NDP30 Develop trust by providing options for different gender identities 1 92 1 92 NDP35 Ensure rigorous methods to know the target audience 1 84 1 92 NDP36 Have clear, factual and neutral information 1 92 1 100 6.7.2 Design Principles that did not Achieve Consensus in Round 2 Fourteen design principles did not achieve consensus in Round 2. These included one principle that previously achieved consensus in Round 1 but had been revised, eight principles that did not achieve consensus in Round 1, and five of the eight additional principles generated from Round 1. Design principles including, DP25.1 (i.e., Provide a secure link or a section for creating online support groups), DP5.1 (i.e., Use logos, icons, and terminologies that clearly depict the topic for the website), DP21.1 (i.e., Use personal stories of people who have 133 experienced the condition), and NDP33 (i.e., Create collaboration spaces like forums, chatrooms on the website) achieved consensus in the clarity dimension but not on the usefulness dimension. There was greater variability in participants’ ratings for the design principles not achieving consensus. For instance, some design principles had an IQR as high as 3 on either the clarity or usefulness dimension, or on both dimensions. Among the design principles with an IQR of 3, the usefulness dimension had the highest number of principles (9 items) while the clarity dimension had the least number of principles (4 items) indicating that usefulness had higher variability than clarity. Design principles including DP8.1 (i.e., Use humour only if appropriate and do extensive user testing), DP13.1 (i.e., Use images that foster kindness, happiness, connection, support, engagement, respect), and NDP31 (i.e., Have a section for website users to narrate their stories) achieved an IQR of 3 on both the usefulness and clarity dimensions. Table 6.9 contains the design principles that did not achieve consensus in Round 2. Table 6.9 Design principles that did not achieve consensus in Round 2 ID Design Principle Clarity Usefulness IQR CVI IQR CVI DP5.1 Use logos, icons, and terminologies that clearly depict the topic for the website 1 92 3 69 DP7.1 Ensure a balanced presentation of information by presenting both positive and negative experiences about a sexual health issue 2 92 3 76 DP8.1 Use humor only if appropriate and do extensive user testing 3 76 3 76 DP10.1 For websites that provide interpersonal connection, provide opportunities for selective and voluntary disclosure of status to provide emotional connection and support 3 76 1 84 DP12.1 Use hyperlinks to direct users to other websites that provide more information 2 92 2 76 DP13.1 Use images that foster kindness, happiness, connection, support, engagement, respect, and community 3 76 3 69 134 Table 6.9 Design principles that did not achieve consensus in Round 2 (continued) ID Design Principle Clarity Usefulness IQR CVI IQR CVI DP15.1 Include a range of information that touches on different aspects of the condition 2 84 3 69 DP21.1 Use personal stories of people who have experienced the condition. e.g., celebrities and other personalities 1 92 3 76 DP25.1 Provide a secure link or a section for creating online support groups 1 84 2 76 NDP29 Provide factual content by including references to all information 2 84 2 84 NDP31 Have a section for website users to narrate their stories 3 76 3 69 NDP32 Have a function that allows people to leave the website immediately 1 76 3 69 NDP33 Create collaboration spaces (forums, chatrooms) on the website so that those with similar experiences can connect 1 92 3 76 NDP34 Involve all those affected by the condition, not just people with the condition. e.g., partners of people with the condition 2 76 2 92 6.7.3 Stability of Design Principles That Did Not Achieve Consensus in Round 2 Before the start of Round 3, stability was calculated on design principles not achieving consensus to determine if participants' responses would likely change in Round 3. This calculation was done for nine design principles that did not achieve consensus in Round 2. Stability was not calculated on the five additional principles generated from Round 1 because they were being appraised for the first time. Only two of the nine items had stability levels of less than 20% on both the clarity and usefulness dimensions. These include DP10.1 (i.e., Provide opportunities for selective and voluntary disclosure of status for websites that provide interpersonal connection) and DP21.1 (i.e., Use personal stories of people who have experienced the condition). As described above a higher score stability suggests that there are wide variations in participants' responses between the two rounds and items with high variability may benefit 135 from another round, while a lower score suggests participants' responses for an item between two rounds are likely the same and may not change from another round. Thus, these two design principles were removed from Round 3 because there was a low possibility that participants would change their responses (i.e., even though the goal of a Delphi study is to achieve consensus, it is unlikely if consensus will ever be achieved on these items). The other design principles with stability levels above 20% were revised based on participants' comments. Table 6.10 shows the stability for design principles that did not achieve consensus in Round 2. Table 6.10 Stability of items that did not achieve consensus in Round 2 Clarity Usefulness ID Design Principle Unit Δ Stability Unit Δ Stability DP5.1 Use logos, icons, and terminologies that clearly depict the topic for the website 12 42.8 % 14 50.0 % DP7.1 Ensure a balanced presentation of information by presenting both positive and negative experiences about a condition 12 42.8 % 10 35.7 % DP8.1 Use humour only if appropriate and do extensive user testing 13 46.4 % 13 46.4 % DP10.1* Provide opportunities for selective and voluntary disclosure of status to provide emotional connection and support 4 14.3 % 4 14.3 % DP12.1 Use hyperlinks to direct users to other websites 8 28.5 % 5 17.8 % DP13.1 Use images that foster kindness, happiness, connection, support, engagement, respect & community 11 39.3 % 9 32.1 % DP15.1 Include a range of information that touches on different aspects of the condition 10 35.7 % 6 21.4 % DP21.1* Use personal stories of people who have experienced the condition 3 10.7 % 5 17.8 % DP25.1 Provide a secure link or a section for creating online support groups 8 28.5 % 7 25.0 % Note: Unit Δ = Unit change between responses in Round 1 and Round 2; Asterisks indicate items that did not meet consensus but were stable, and thus were dropped for Round 3. 136 6.7.4 Revision of Design Principles That Did Not Meet Consensus in Round 2 Even though 14 design principles did not achieve consensus in Round 2 and were supposed to have been revised, four principles were not revised because they were either judged as being stable and would likely not meet consensus in future rounds (i.e., DP10.1 and DP21.1) or were covered by a newly generated principle in Round 2. For instance, NDP31 (i.e., Have a section for website users to narrate their stories) and NDP33 (i.e., Create collaboration spaces like forums, chatrooms on websites) were dropped because they were covered by a newly generated principle in Round 2, NDP38 (i.e., Ensure that careful consideration has been given when interactive features will be used. e.g., is a moderator required). The remaining 10 items were revised because they contained comments that suggest that some items could indirectly foment stigma. For instance, Participant 16 suggested that DP9.1 (i.e., Using images that foster kindness, happiness, connection, support, engagement, respect, and community) may have a negative impact on stigma and “can lead to perceptions of being emotionally manipulated.” Participant 19 also commented on this same principle by stating that “I am not sure of how images can realistically foster happiness, kindness.” Other design principles that received negative comments included DP12.1 (i.e., Use hyperlinks to direct users to other websites that provide more information) and NDP31 (i.e., Have a section for website users to narrate their stories). Participants raised security and privacy issues and doubted the usefulness of these two principles in addressing stigma. For instance, Participant 9 expressed security and trust issues with the use of hyperlinks while Participant 7 stated that having a section for narrating people’s experiences would be “inappropriate and insecure without the proper operational staff to moderate such platforms.” Participant 16 further reiterated that “facilitation of such groups is a key part of the value here.” 137 Other design principles were seen as useful but considered not feasible to implement. For instance, NDP29 (i.e., Provide factual content by including references to all information) was seen by participants as not feasible because it is not possible to include references to “all content.” Others also thought that content can be credible and factual without references. Participant 16 stated that “content can be factual without including references. Perhaps this is more about trustworthiness? I don't think most people using services care about references, but more about the perception of overall agency credibility.” The issue of culture and context of use for each principle resonated among participants. That is, the context in which a principle is applied would determine whether it would reduce or exacerbate stigma. For instance, Participant 1 stated that “the ability of the destigmatising design principles to offer support and help reduce experiences of stigma would really depend on context.” In addition, Participant 8 commented on DP13.1 (i.e., Use images that foster kindness, happiness, connection, support, engagement, respect, and community) by stating that “although I understand the intent of this principle, I am concerned that this may be a sticky wicket that may be more problematic. I think there may be cultural nuances that make it difficult to ensure the images used are universally understood in a way that would reduce stigma without introducing stereotypes or bias. For example, what may be interpreted as depicting kindness to someone may be threatening or triggering to others depending on the people captured in the different images” Another context-specific design principle was DP8.1 (i.e., Use humour only if appropriate and do extensive user testing). Some participants argued that it may be difficult to find “humour” that is universally acceptable to many people. Participant 19 reiterated that “I still believe humour is culturally relevant and may be hard to find context-specific appropriateness to all – if you are on any social media sites that share humour (even from long ago) you may notice how quickly responses can take on negative/ political/ bullying components within the comments (I just don't bother reading them, but others who are vulnerable may be affected) ... just tread carefully here” 138 Participant 8 also stated that the use of humour was “risky because of cultural components.” Given the valuable comments for the principles in this round, the 10 items were revised accordingly. Table 6.11 contains the principles that had been revised for Round 2 and the corresponding second revisions used in Round 3. Table 6.11 Revision of design principles not achieving consensus in Round 2 ID Design Principle ID Revised Design Principle DP5.1 Use logos, icons, and terminologies that clearly depict the topic for the website DP5.2 Use logos, icons, and terminologies that help to explain the topic for the website as long as privacy issues are considered (e.g., shoulder surfing). DP7.1 Ensure a balanced presentation of information by presenting both positive and negative experiences about the condition DP7.2 Ensure a balanced presentation of information by highlighting more on positive experiences but not lying by omission DP8.1 Use humour only if appropriate and do extensive user testing DP8.2 Use humour only if appropriate and do extensive user testing to confirm context-specific appropriateness DP12.1 Use hyperlinks to direct users to other websites that provide more information DP12.2 Consider using hyperlinks to other websites that provide more information DP13.1 Use images that foster kindness, happiness, connection, support, engagement, respect, and community DP13.2 Use pictures that depict kindness, connection, support, engagement, respect, and community, as long as it does not promote perceptions of being emotionally manipulated DP15.1 Include a range of information that touches on different aspects of the condition DP15.2 Include a range of evidence-based information that touches on different aspects of the condition DP25.1 Provide a secure link or a section for creating online support groups DP25.2 Provide a secure link for provider-moderated online support groups NDP29.1 Provide factual content by including references to all information NDP29.2 Provide trustworthy content by having a reference list to factual information NDP31.1 Have a section for website users to narrate their stories NDP31.2 Covered by a new principle NDP32.1 Have a function that allows people to leave the website immediately NDP32.2 Have a function that allows people to change the content of the website to maintain privacy 139 Table 6.11 Revision of design principles not achieving consensus in Round 2 (continued) ID Design Principle ID Revised Design Principle NDP33.1 Create collaboration spaces (forums, chatrooms) on the website so that those with similar experiences can connect NDP33.2 Covered by a new principle NDP34.1 Involve all those affected by the condition, not just people with the condition. E.g., partners of people with the condition NDP34.2 Consider involving all those affected by the condition, not just people with the condition (e.g., partners of people with the condition). 6.7.5 Newly Generated Design Principles in Round 2 Unlike Round 1, participants' comments in Round 2 were minimal and only two design principles were generated out of these comments. The newly generated principles included: 1) Consider involving all those affected by the condition, not just people with the condition (e.g., partners of people with the condition) (NDP37); and 2) Ensure that careful consideration has been given when interactive features will be used. e.g., is a moderator required (NDP38). 6.8 Round 3 Survey Round 3 began with the set of 10 revised principles and the two new design principles (making 12 items). These design principles were sent to the 14 participants who had responded to the Round 2 survey. Thirteen of the fourteen participants responded to the Round 3 survey. At the end of Round 3, only three items achieved consensus and participants provided fewer comments. Despite revisions in Round 1 and Round 2, DP12.2 (i.e., Consider using hyperlinks to other websites), DP25.2 (i.e., Provide a secure link for provider-moderated online support groups), DP13.2 (i.e., Use pictures that depict kindness, connection, support), DP5.2 (i.e., Use logos, icons, and terminologies that helps to explain the topic of the website), DP7.2 (i.e., Ensure 140 a balanced presentation of information), and DP8.2 (i.e., Use humour and doing extensive user-testing) failed to achieve consensus in the third round. Design principles that had to do with links or hyperlinks including DP12.2 (i.e., Consider using hyperlinks to other websites), and DP25.2 (i.e., Provide a secure link for provider-moderated online support groups) were judged as clear but participants doubted their usefulness in addressing stigma. Despite several revisions to these two principles, participants still thought that links or hyperlinks were neutral with respect to addressing stigma. Others believed that links or hyperlinks may aggravate stigma especially when their reliability and safety cannot be ascertained by the user. For instance, Participant 9 stated that “hyperlinks for many of us are anxiety-provoking as you never know if they are safe. It needs to be clear that the sites you direct users to are appropriate and don't have any offensive materials on them." Similarly, design principles including DP13.2 (i.e., Use pictures that depict kindness, connection, support), and DP5.2 (i.e., Use logos, icons, and terminologies that help to explain the topic of the website) were still seen as unclear and not useful in addressing stigma despite the several revisions. Tables 6.12 and 6.13 show the design principles at the end of Round 3. Table 6.12 Design principles meeting consensus in Round 3 ID Design Principles meeting consensus Clarity Usefulness IQR CVI IQR CVI DP15.2 Include a range of evidence-based information that touches on different aspects of the condition 1 100 1 92.3 DP29.2 Provide trustworthy content by having a reference list to factual information 1 100 1 92.3 DP34.2 Consider involving all those affected by the condition, not just people with condition (e.g., partners of people with the condition) 1 92.3 1 100 141 Table 6.13 Design principles not meeting consensus in Round 3 ID Design Principles not meeting consensus Clarity Usefulness IQR CVI IQR CVI DP5.2 Use logos, icons, and terminologies that help to explain the topic for the website as long as privacy issues are considered (e.g., shoulder surfing). 4 69.2 3 61.5 DP7.1 Ensure a balanced presentation of information by highlighting more on positive experiences but not lying by omission 2 84.6 2 84.6 DP8.2 Use humour only if appropriate and do extensive user testing to confirm context-specific appropriateness 2 92.3 2 84.6 DP12.2 Consider using hyperlinks to other websites that provide more information 1 84.6 2 53.8 DP13.2 Use pictures that depict kindness, connection, support, engagement, respect, and community, as long as it does not promote perceptions of being emotionally manipulated 4 69.2 2 84.6 DP25.1 Provide a secure link for provider-moderated online support groups 1 92.3 3 76.9 DP32.2 Have a function that allows people to change the content of the website to maintain privacy 2 76.9 3 84.6 DP37.2 Ensure that careful consideration has been given when interactive features will be used (e.g., is a moderator required ?) 3 69.2 3 69.2 DP38.2 Consider using artwork/photographs/videos etc. created by members of the affected community 2 69.2 2 84.6 6.9 Summary of Findings Table 6.14 shows the final destigmatising design principles mapped into the five categories of Fallot and Harris trauma-informed care framework. Trustworthiness had the most items while Choice had the least number of items. Twelve items reached consensus in Round 1 (but only four were retained), an additional 12 design principles reached consensus in the Round 2, and only three principles met consensus in Round 3. At the end of the third round, 19 items achieved consensus. Unlike Round 1, participants provided fewer comments in Rounds 2 and 3. Most of the items that did not achieve consensus at the end of Round 3 had not met consensus throughout the Delphi study. In Round 2, six design principles were found to have overlapped 142 with other principles and were either removed or merged with others. Overall, participants did not come to consensus regarding the clarity and usefulness of nine design principles at the end of the third round despite several revisions. In other words, participants generally disagreed on the usefulness of nine items in addressing stigma. Except for four principles that maintained their originality from the pre-Delphi stage, all the other principles met consensus after some revisions. Table 6.14 Final consensus design principles mapped to the trauma-informed care framework Category ID Destigmatising design principle Emotional safety DP1 Provide participants with the ability for anonymous engagement DP2 Encrypt websites that collect personal information to prevent unauthorized access to personal data DP9.1 Avoid using language that has a tone of blame or judgment of people living with the condition Choice DP11 Provide contact information of counselors or other psychological supports DP15.2 Include a range of evidence-based information that touches on different aspects of the condition Trust-worthiness DP16.1 Include information that corrects myths about a condition, to enable users to get accurate information DP17.1 Provide factual and plain language information that normalizes and de-stigmatises sexual health-related conditions DP18.1 Selection of language/images should be done in consultation with the community to ensure diversity DP19.1 Use images of people from diverse ethnicity, age, and gender identity who have experienced the condition NDP30 Develop trust by providing options for different gender identities NDP29.2 Provide trustworthy content by having a reference list to factual information NDP34 Ensure rigorous methods to know the target audience NDP36 Have clear, factual and neutral information Empower-ment DP22.1 Consider including links to information on the fundamental rights of people affected by or living with the condition DP23.1 Use inclusive language that is sensitive to the context of the condition e.g., partner instead of husband/wife, the person instead of woman/man DP26.1 Include videos/testimonials that center on people’s experiences with the condition, including stigma DP27 Avoid othering and stereotyping people with a sexual health-related condition 143 Table 6.14 Final consensus design principles mapped to the trauma-informed care framework (continued) Collab-oration DP28 Interventions for addressing men's/women health or sexual health-related conditions should be developed and delivered in partnership with those living with the condition NDP34.2 Consider involving all those affected by the condition, not just people with the condition (e.g., partners of people with the condition) Table 6.15 shows the destigmatising design principles that did not meet consensus, also mapped to the trauma-informed care framework. Table 6.15 Principles not meeting consensus mapped to the trauma-informed care framework Category ID Destigmatising design principle Emotional safety DP3 Prevent errors by having features that warn users from taking actions that can lead to accidental disclosure of status (Nielsen, 1994b) DP4 Use non-sensational language and images on web platforms DP5 Logos, icons, and terminologies should be made subtle so that they do not immediately draw the users’ or bystanders’ attention to sexuality DP6 Reduce offensiveness of images by minimizing the display of explicit or profane sexual images on websites DP7 Avoid conveying negative experiences associated with a condition and rather emphasize positive messaging DP8 Use humour in a very positive and sensitive manner to lessen the fear and seriousness associated with a condition Choice DP10 Provide opportunities for selective and voluntary disclosure of status to provide emotional connection and support DP12 Include links to other credible sites and contact information for users to seek further information DP13 Ensure appropriate flexibility of web features to allow a user to hide certain content/images that they finds offensive DP14 Ensure customization features to enable users to adapt aspects of the interface to suit their visual preference Trustworthiness DP20 Ensure a match between the system and the users’ culture Empowerment DP24 Use symbols of optimism, encouragement, and hope DP25 Provide mechanisms for creating online advocacy groups NDP31 Have a section for website users to narrate their stories NDP32 Have a function that allows people to change the content of the website to maintain privacy NDP33 Create collaboration spaces (forums, chatrooms) on the website so that those with similar experiences can connect 144 6.10 Discussion The Delphi study was conducted to develop a set of destigmatising design principles that can guide website developers in designing sexual health-related websites. The study started with 28 items generated from the pre-Delphi stage and an additional 10 items generated from participants comments and suggestions (making 38 unique items). The Delphi study ended after Round 3 and consensus was determined during each round. At the end of Round 3, 19 items, representing 50% of the proposed principles, were judged by the panel of experts as clear and useful in addressing sexual health-related stigma. The 22 participants who initially responded to the recruitment survey and the 15 participants that ended up taking part in the Delphi study had similar demographics (see Table 6.2). While only 22 (14%) of the 158 original invitees responded, the study had relatively low attrition. Except for the first round, the participant attrition rate between successive rounds in this study was minimal compared to other Delphi studies (Gargon et al., 2019; McPherson et al., 2018). Of the 22 participants recruited for the first round, seven (31.8%) dropped at Round 1, while one participant dropped at Round 2 (6.6%) and Round 3 (7.1%) respectively. Furthermore, participants were mostly consistent with their ratings and comments between the successive rounds. This consistency was demonstrated by the nearly uniform interquartile range scores across the clarity and usefulness dimensions across most of the items. For instance, the comments for design principles that did not achieve consensus did not change significantly between Round 1 and Round 2. The similarity observed in participants’ scores and comments between successive rounds demonstrates a common understanding regarding the usefulness of the principles in addressing stigma. The additional principles suggested, as well as the rich comments in Round 1 also suggests that the experts were indeed knowledgeable in the field of 145 sexual health and stigma. This was not surprising given that the average number of years of research experience in stigma and sexual health-related research was more than 20 years. Although previous studies have explored how digital technologies could be used to address sexual health-related stigma, (Gkatzidou et al., 2015; Karamouzian et al., 2018; Magee et al., 2012; Noltensmeyer et al., 2016), no guiding framework was used in the actual design of such technologies. To the best of my knowledge, this study is the first to address that gap by producing design principles that may serve as a guide for website designers in designing destigmatising sexual health-related websites. In the next sections, the findings of the Delphi study are discussed in the context of the trauma-informed care framework and stigma management strategies. The subsequent section discusses the limitations of the Delphi study. 6.10.1 Trauma-Informed Care Framework and Stigma Fallot and Harris’ five principles provides a framework on how services and systems can be implemented to reduce the likelihood of re-traumatization (Harris & Fallot, 2001). Unlike traditional design principles that usually stem from human-computer interactions experts (Jacob Nielsen, Don Norman) or companies that specialize in technology design (e.g., Microsoft Corp., Apple Inc.) (Hulshof et al., 2013), this study is the first that attempts to develop a set of design principles from a health-related domain by employing a trauma-informed care framework. Despite dropping half of the principles at the final round, the final principles still reflect Fallot & Harris’ five principles of trauma-informed care (Harris & Fallot, 2001). Therefore, even though the trauma-informed care framework has not been empirically tested in this study, the results align with the findings in Study 2 suggesting that a trauma-informed care framework could be helpful in addressing stigma in health technology design. Furthermore, participants’ agreement 146 on the usefulness of principles across the five trauma-informed care categories also confirms previous studies that adopted a trauma and violence-informed care framework in reducing the stigma of sexually transmitted and blood borne infections among government organizations including the Canadian Public Health Association (MacLean, 2018). Consistent with the previous study by MacLean (2018), it could be argued that adopting this framework helped to produce non-functional design principles that could address the emotional concerns of people who use digital health technologies. 6.10.2 Functional and Non-functional Design Requirements The initial set of design principles were generated to cover non-functional (emotional and content-related) as well as functional principles. Functional requirements are simply the statements or principles that specify what the system should provide, how it should act in specific situations, and how it should respond to specific commands (Shehadeh et al., 2021). Non-functional requirements, on the other hand, are simply the quality constraints the system must meet to promote good user experience (ibid). Even though participants generally agreed to the clarity of the functional design principles, like the use of links and hyperlinks, they believed that the functional design principles were neutral with respect to addressing stigma. Only one functional design principle (i.e., Encrypt websites that collect personal information to prevent unauthorized access to personal data (DP2)), was considered as useful in addressing stigma. Instead, participants largely considered the non-functional and mostly content-related design principles as useful in addressing stigma. The predominant emphasis of participants on non-functional principles (i.e., emotional, and content-related design principles) was probably because participants had domain expertise on the subject matter under study. 147 While traditional design principles could prove useful in improving the usability and functionality of sexual health-related technologies (Zhang et al., 2021), the principles developed in this study may help developers to design digital health technologies that meets patients’ emotional expectations. This contribution is significant because traditional web developers typically tend to focus on the usability and functionality of digital technologies with minimal attention to emotional and content-related requirements (Blank & Kosuru, 2020; Quiñones & Rusu, 2017; Sanz et al., 2016). This study, indeed, extended the predominant focus on usability and functionality of web developers to include other aspects of design that are typically overlooked but may be considered useful to the end-user (i.e., the person with a sexual health-related issue). Participants’ agreement on the usefulness of the non-functional emotional, and content-related design principles also reinforces the need for and importance of incorporating non-functional requirements in designing digital health technologies (Vermeeren et al., 2016). The development of non-functional design principles that address stigma is timely because non-functional requirements are generally less obvious, more difficult to specify, and most often neglected by software designers during systems’ design because of the large investment of time to generate non-functional requirements. Emotional and content-related design principles might help software designers to specify stigma-specific non-functional design requirements before embarking on software design. Furthermore, developing design principles from the perspective of a specific domain in healthcare demonstrates a possible shift from a predominant trend where human factors specialists or technology companies are largely seen as the only experts capable of developing design principles (Hassenzahl, 2005). 148 6.10.3 Stigma Management Strategies As previously described in this chapter, destigmatising design principles in the pre-Delphi stage were generated to cover a broad range of stigma management strategies for various stigmatised conditions. The findings from the Delphi study (i.e., design principles that achieved consensus at the end of Round 3) contrasted as well as supported some of the stigma management strategies proposed by earlier stigma researchers (Goffman, 1963 ; Heijnders & van der Meij, 2006; Link & Phelan, 2001). That is, participants generally disagreed on design principles that aligned with earlier stigma researchers’ strategies of denying or hiding the stigmatising attributes. Rather, they advocated for and supported design principles that sought to challenge, expose, and normalize the stigma attributes and make stigma more of a societal concern than an individual problem. The findings of the Delphi study are now discussed in the context of the following stigma management strategies. 6.10.4 Hiding/Avoiding/Passing for Normal/ Covering Stigmatising Attributes Hiding/avoiding/passing strategies simply means restricting the display of stigmatising attributes or keeping the stigma from “looming large” in the face of others (Goffman, 1963). According to the earlier stigma researchers (Corrigan & Kosyluk, 2013; Goffman, 1963; Heijnders & Van der Meij, 2006; Meisenbach, 2010), strategies including hiding and avoiding strategies were considered appropriate for alleviating stigma when a condition is hidden or concealed from the public. However, the findings of this study contrasted this long held assertion that hiding or disguising strategies are most appropriate in addressing stigma when the stigmatising attribute is hidden or concealed from the public. On the contrary, hiding, evading, or avoiding the stigmatising attributes may only intensify the internalised feelings of stigma and 149 lead to feelings of isolation, self-contempt, anxiety, and feelings of hopelessness (Bos et al., 2013; Fuster-Ruizdeapodaca et al., 2014). The negative effects of hiding strategies in addressing stigma, ironically, confirms Goffman’s (1963) previous claim that hiding stigmatising attributes may ultimately lead to a very high level of anxiety because such people are living a life that can be collapsed at any moment if their status is revealed. The lack of positive consensus on the hiding and avoiding strategies in this study suggests that earlier stigma management strategies such as avoiding stigmatising situations, hiding the stigmatising attributes, distancing oneself from stigma, and making favourable social comparisons, might have outlived their usefulness in addressing stigma in our current context. A reason that was advanced by participants is the fact that hiding stigmatising content may rather lead people to accept public perceptions or accept the status quo regarding the stigmatising attribute, an assertion held by Meisenbach (2010). In other words, hiding or avoiding strategies may perpetuate internalized stigma by forcing people to accept the negative attributes associated with their condition. Instead of encouraging hiding strategies, participants in this Delphi study supported and suggested design principles that would expose and normalize sexual health-related content on web-platforms. This open approach to addressing stigma reflects modern stigma management strategies that seek to shift the focus from the individual to include other people and larger societal factors that simultaneously operate to shape and perpetuate stigma (Friedman et al., 2014; Hood & Friedman, 2011; Rocha-Jiménez et al., 2018). Even though web-based platforms may be used by individual end-users, adopting an open and public approach to displaying sexual health-related information may signal to the user that sexual health-related problems are not necessarily the individual’s problem. This approach is important given the fact that many people 150 with sexual health-related conditions seem to present with a higher degree of internalized stigma than public stigma, for which a personalized and individualized approaches are almost always adopted, yet are ineffective (Fuster-Ruizdeapodaca et al., 2014). 6.10.5 Education Strategies Education strategies involve interventions that aim to inform the public, community groups, and individuals by increasing their knowledge about a specific condition by providing facts that counterbalance the false assumptions upon which stigma is based (Heijnders & Van der Meij, 2006). Participants in this study had a positive consensus on two of the four principles that adopted education strategies to alleviate stigma. These principles include DP16 (i.e., Include information that correct myths about the condition), and NDP35 (i.e., Provide factual and plain language information). Participants’ agreement on the usefulness of education strategies in addressing stigma confirms the predominant use of educational messages in stigma reduction campaigns (Heijnders & van der Meij, 2006). Furthermore, participants not only agreed on the usefulness of these educational principles in addressing stigma but also agreed with principles that would make educational messages accurate, reliable, and non-judgmental. This reinforces earlier studies that suggest that educational messages that are accurate, reliable, credible, and non-judgmental are crucial for addressing stigma (Cisu et al., 2019; Killip et al., 2020). While education strategies were considered as useful in addressing sexual health-related stigma, participants’ comments suggested that the content of educational messages is very important for its effects on stigma and stigma reduction. Therefore, a contextual consideration in presenting or displaying sexual health-related content in web platforms is fundamental in addressing or aggravating stigma. The effect of educational content on stigma reduction seems to 151 be more pronounced in information displayed on sexual health-related websites (Hightow-Weidman et al., 2015; Shearer et al., 2012). In effect, participants thought that the way sexual health messages are displayed on web platforms could inadvertently aggravate stigma. The negative effect of the inappropriate display of sexual health content on a website was identified in previous studies (Davis et al., 2012; Dudina & Tsareva, 2018; Payton & Kvasny, 2016; Shearer et al., 2012). These studies generally revealed that education strategies in digital and print media that were intended to address stigma instead contributed to and perpetuated stigma and the culture of fear among people living with sexual health-related conditions. This assertion suggests that the way educational messages are crafted and displayed on websites would determine their success in alleviating stigma. Therefore, even though participants argued against hiding or avoiding stigmatising content, adopting the opposite approach by displaying content in a superficial and unfavourable manner, without a consideration of contextual factors, could inadvertently perpetuate stigma. Since what is considered to be stigmatising differs from setting to setting, the effect of content display on addressing stigma, to a large extent, depends on context. The effect of context-specific content display on stigma alleviation was not explored in this study. Further studies are needed to examine how, for example, the same web-based content impacts stigma in different cultural and linguistic setting. 6.10.6 Contacting Contacting refers to the interaction between patients and health professionals or the interaction between different patients with the objective of reducing stigma. Two design principles that centered on contacting including, DP1 (i.e., Provide participants with the ability for anonymous engagement) and DP11 (i.e., Provide contact information of counselors or other 152 psychological supports) were among the final set of items that achieved consensus at the end of Round 3. Participants’ agreement to the usefulness of contacting strategies supported Brown et al.’s (2003) findings on the importance of contacting as a useful strategy in addressing stigma. Other prior studies have demonstrated that a personal relationship with people affected by HIV and sharing testimonials from others affected by the condition will likely demystify and dispel misinformation that often fuels stigma (Brown et al., 2003; Stangl et al., 2013; Waqas et al., 2020). The use of contacting strategies in addressing stigma could be more beneficial on web-based platforms where people are more likely to solicit online health information or online counselling (Fasciano et al., 2015). Other studies suggest that contacting strategies supported by health provider moderation may provide stronger linguistic coordination between patients and providers and improve the efficacy and safety of web-based technologies for stigmatised conditions (Wadden et al., 2020). Thus, the findings of this study largely demonstrate that contacting strategies in the context of digital health technologies might be useful in addressing sexual health-related stigma. Despite the usefulness of contacting strategies in addressing stigma, some participants also thought that contacting strategies might rather foment stigma if they are not moderated by health professionals. This assertion is consistent with previous studies that suggest that contacting strategies may have some inherent risks, especially when such strategies are not secure and anonymous (Carter & Wilson, 2015; Macias et al., 2005; Swire-Thompson & Lazer, 2019). That is, contacting strategies on websites such as chatrooms and social networking sites, if not implemented well, may aggravate stigma by way of online bullying and online victimization (Payton & Kvasny, 2016; Witzel et al., 2016). Given the mixed findings on the use of contacting strategies in this study, it is not just a matter of designing contacting platforms on 153 digital technologies but most importantly, how those platforms are designed and secured/encrypted. Thus, further research is needed to determine how web platforms can be designed to facilitate the use of contacting strategies without the possibility for online bullying and online victimization. 6.10.7 Inclusivity Inclusive design is an “approach to design in which the designers ensure that their products and services address the needs of the widest possible audience, irrespective of age or ability” (Design Council, 2020). The study produced destigmatising design principles that reflected principles of inclusive design. Unlike the original inclusive design principles that were largely focused on designing products for older people and people with disabilities (Jordan, 2000), this study extends the notion of inclusive design to include younger people, and people with different sex and gender identities. At a time when the digital divide is negatively impacting people from marginalised backgrounds (Brewer et al., 2020), the inclusive design principles produced in this study may help technology developers understand how to design digital health technologies that include marginalized groups. This is important because such groups tend to be the most affected by stigma. Furthermore, even though the original principles did not contain inclusive design principles, participants recommended design principles that reflected different gender identities. The recommendations from the experts attest to the importance of sex and gender considerations in reducing disparities and stigma among sexual and gender minorities who are frequently faced with stigma and discrimination (Layland et al., 2020). The inclusion of principles that address sex and gender identities is crucial because digital channels are the most common platforms for 154 seeking health-related information among people who identify as LGBTQIA/2S, yet their concerns are often overlooked by health technology designers during systems design (Martinez & Tang, 2020). Therefore, design principles that inform the design of digital technologies that are inclusive of people who identify as sexual minorities may help address the stigma associated with their identities. That is, sex and gender minority groups would not feel excluded and discriminated against when websites on sexual health-related conditions integrate their needs and concerns. 6.10.8 Validity and Reliability of the Design Principles While the psychometric properties of the design principles were not specifically tested in this study, adopting the general recommendations for conducting Delphi studies might have enhanced the validity, reliability, and methodological rigour of the Delphi methodology. For instance, the use of participants with extensive expertise in stigma and sexual health research might have helped to improve the content validity of the design principles. The multiple rounds of surveys, on the other hand, might have enhanced concurrent validity (Goodman, 1987). The iterative process also allowed participants to review, reflect, and critique their responses as the study progressed. It could also be argued that the credibility of the Delphi study might have been enhanced by the iterative feedback of participants while biases might have been reduced by the anonymous and virtual engagement of participants. The personal biases of the researcher, it can be argued, were likely reduced by stating the criteria for participants’ recruitment as well as the criteria for item consensus well in advance of the study. As the validity of Delphi studies could be affected by a low response rate (Hasson et al., 2000), the rigorous steps taken to maintain participants resulted in 13 of the 15 participants staying until the end of the study. 155 6.11 Limitations of the Delphi Study The Delphi study has some limitations. First, the destigmatising design principles have a broad focus on heath conditions related to sexual health including, STIs, men’s/women’s health issues, as well as sexual disorders. Such a broad focus may dilute the applicability of the destigmatising design principles on websites that are specific to a sexual health-related condition. However, such high-level principles are needed as a starting point for further research on how to apply the principles to specific a condition or disorder. A further limitation of Delphi studies is that the data collection procedure did not allow for participants to discuss the topic. While Delphi studies are usually limited to a smaller number of participants, the statistical significance of participants’ agreement and disagreement cannot be ascertained without larger sample sizes. Finally, out of 158 participants contacted, only 22 agreed to participate, and 15 finally participated. It is unknown if the demographics of participants who agreed to participate or those who participated were truly representative of the 158 people who were originally contacted. 6.12 Conclusion In conclusion, the findings of the Delphi study extend software design from a predominant focus on usability and functionality of digital health technologies to include emotional and content-related principles that are often overlooked, yet considered critical for addressing stigma among people who use sexual health-related technologies. Unlike the earlier stigma management strategies, the design principles reflected a modern approach to managing stigma by suggesting that the stigmatising attribute be made public. This public approach has the potential to ultimately make stigma more of a community or societal concern than an individual 156 attribute. Despite the usefulness of these principles, it is not known how software designers will apply these principles in practice. Study 4 was designed to answer this research question. 157 Chapter 7: Applicability of Destigmatising Design Principles: A Qualitative Descriptive Study After obtaining the set of destigmatising design principles from Study 3, it was important to ascertain how software designers might apply the principles in a typical design situation. Therefore, Study 4, described herein, was designed to answer the research question – What are software designers’ perceptions of the applicability of the destigmatising design principles developed in Study 3? For the purpose of this chapter, software designers are defined as individuals involved in the design of websites including individuals who focus on how the technology is presented to the user (e.g., User Interface (UI) and User Experience (UX) designer, project manager, and other human-computer interaction experts) and individuals who focus on how the technology functions (e.g., software developers or software engineers). According to the Interaction Design Foundation (2020), User Interface (UI) design is the process of building interfaces with a focus on the look and feel of computerized devices, while User Experience (UX) design refers to how computerized devices can be created to provide meaningful and relevant experience to users. 7.1 Methods This study adopted a qualitative descriptive design to examine the applicability of the destigmatising design principles from the perspective of software designers. According to Sandelowski (2010), qualitative descriptive approaches are used to provide a straightforward description of participants’ experiences and perceptions regarding a phenomenon of interest, particularly where little is known about the topic under investigation. Focusing on software 158 designers’ accounts of creating websites, this study design was considered appropriate for capturing participants’ subjective opinions and perceptions regarding the applicability of the destigmatising design principles. The use of a qualitative descriptive design provides an excellent methodology for answering research questions where the focus is not on increasing theoretical or conceptual understanding but contributing to change and quality improvement in practice settings (Doyle et al., 2020). Although qualitative descriptive studies have been criticized as lacking “epistemological credibility,” other researchers have argued that a qualitative descriptive method is best aligned with a constructivist approach that seeks to use naturalistic methods in investigating a phenomenon of interest (Lincoln et al., 2018). Consistent with a qualitative descriptive approach, the goal of this study was to focus on producing a rich description of how software designers might apply the design principles. The data sampling strategy, data collection methods, analysis, and interpretation in this study were all informed by a qualitative descriptive approach. Prior to data collection, ethics approval was obtained from the UBC Behavioural Research Ethics Board (H21-02553). 7.1.1 Sample and Sample Size The sample for this study included software designers with at least one year's experience in website development, design and evaluation, design research, heuristic evaluation, or general expertise in health informatics research. There are no specific recommendations regarding sample sizes in studies that involve website evaluation or evaluating items based on a user interface. With a carefully chosen sample, a relatively small number of participants may yield sufficient data to answer a research question in such studies. The small sample size does not always indicate a small data set, especially when the study has to do with interviewing people on 159 several items. This is because even a few items can still yield several data points with a small number of participants (e.g., 15 items X 15 participants = 225 data points) (Johnstone et al., 2006). Previous studies that have applied similar approaches to evaluating user interfaces generally have samples sizes of approximately 15 participants (Nielsen, 1994b; Hwang & Salvendy, 2010; Schmettow, 2012); beyond 15 participants there is a low chance of yielding additional information (Six & Macefield, 2016). Consistent with prior studies, 15 software designers were recruited to participate in this study and 14 finally participated. 7.1.2 Participant Recruitment Consistent with qualitative descriptive design (Doyle et al., 2020), recruitment occurred via a purposive and snowballing sampling technique. Qualitative descriptive studies usually adopt purposive sampling to recruit participants who have the requisite knowledge and experience to be able to provide information on a phenomenon being investigated (Doyle et al., 2020). Based on this, the software designers were recruited from design labs in two universities in western Canada. The two universities were purposively selected because they contained participants that were believed to have the requisite experience in software design. An initial contact letter was sent to the leads of each of the two design labs explaining the purpose of the study and the rationale for recruiting from their current students or alumni. The initial contact email contained a recruitment message targeting potential participants (see Appendix D for the initial contact letter and the recruitment message). The leads of the two design labs subsequently included the recruitment message in their regular newsletter to existing students and alumni of the two institutions. Using snowball sampling, potential participants were also asked to forward the recruitment message to other potential participants whom they believed would meet the 160 eligibility criteria. The recruitment message had an embedded Qualtrics link (Qualtrics International Inc.) for interested participants to review the consent form and complete a demographic survey. The demographic information included age, gender, location (country and city), educational qualification, years of experience in user interface design, current employer, current position and title, name, and email address. Participants who completed the screening survey and met the inclusion criteria were contacted to participate in the interview. Participants who completed the screening survey but did not meet the inclusion criteria were thanked for their time in completing the pre-qualifying survey. The recruitment message was sent out on a biweekly basis until the desired sample size was reached. All participants were recruited before data collection began. 7.1.3 Materials In this study, a vignette was used to elicit information about participants’ perceptions regarding the applicability of the 19 destigmatising design principles described above in typical design situations. A vignette is a short story about a person or a situation that is presented to research participants to elicit information about their beliefs regarding the phenomenon of interest (Gourlay et al., 2014). This vignette was developed by drawing from the researchers’ experiences in evaluating sexual health-related websites and was created to portray a design scenario that the participants could relate to. A vignette is similar to a task scenario in human-computer interaction research which is often used to make concepts and research situations more realistic to the participants. In addition to the vignette, the Options for Sexual Health websites was chosen as a sample website for participants to demonstrate how they might apply each of the design principles. This website was selected because it contains content on many sexual health 161 topics and strives to have a ‘sex positive’ orientation, which seeks to reduce sexual health-related stigma. A link to this sample website (https://www.optionsforsexualhealth.org/), the vignette, and the 19 set of destigmatising design principles were sent to participants at least three days prior to an interview. Since some participants might not have been familiar with the concept of stigma, a short definition was included as a preamble to the list of destigmatising design principles. The preamble on stigma stated the following: “Stigma is defined as disapproval of, and discrimination against someone based on a perceived social characteristic that serves to distinguish them from other members of the society. There are two main types of stigma. Public/enacted stigma and internalized stigma. Public/enacted stigma is the outward disapproval/discrimination against someone by others while internalized stigma is the internal negative feelings that a person has due to possession of an attribute that she/he perceives as negative.” The preamble also contained instructions on how the interview would be conducted. (See Appendix E for the study materials and interview guide). 7.1.4 Data Collection According to Sandelowski (2000), data collection in qualitative descriptive studies are aimed at discovering the who, the what, the how and the where of a phenomenon of interest. In other words, qualitative descriptive studies aim to describe the phenomenon by providing a literal description and an interpretation of the data without moving (too far) away from the literal description (Sandelowski, 2000). Even though face-to-face interview techniques have been commonly used in qualitative descriptive studies (Kim et al., 2017), other interviewing techniques including telephone interviews, focus group discussions, and web-based interviews (i.e., Skype, Zoom) were increasingly being adopted (Doyle et al., 2020) even prior to COVID-162 19. However, due to COVID-19 restrictions at the time of conducting this study, semi-structured, one-on-one interviews were conducted virtually over Zoom (Zoom Video Communications Inc.). To protect participants’ privacy, they were asked to turn off the video function of Zoom before the recording started. Therefore, only the audio and the screen showing participants’ mouse interactions with the website were recorded. No personal information was collected during the interviews and participants were informed not to reveal any identifying information. Data collection occurred from November 1, 2021, to December 4, 2021, and each interview lasted approximately 90 minutes. Due to the labour-intensive nature of the interview, participants were compensated with $100 honorarium for their time spent in the interview session. 7.1.5 Data Collection Instrument An interview guide was developed based on the vignette described above (see Appendix E). The interview guide contained questions on four main aspects of the design principles. These included: 1) Software designers’ understanding of stigma and their general perceptions regarding the design principles; 2) The ease and difficulty of applying each of the principles; 3) Strategies that software designers might adopt to apply each of the design principles in redesigning the sample website described above; and 4) The challenges they anticipated in applying each principle and how they might overcome those challenges. For items that were considered difficult to apply, probing questions were asked about the kind of support participants might need to be able to address those principles. For items that were considered easy to apply, participants were asked to demonstrate how they might apply the principle to redesign the Options for Sexual Health Website in a way that addresses stigma. 163 7.1.6 Data Analysis According to Doyle et al. (2020), data analysis in qualitative descriptive studies is supposed to be data-driven and does not seek to adopt any pre-existing philosophical or epistemological perspective in analyzing data. Therefore, the data analysis approach in this study was informed by the study objectives rather than by any pre-existing framework. After the interview, data were transcribed using Temi online subscription software (Temi, USA Inc.). The data were subsequently exported into NVivo version 12 (QSR International Pty Ltd.). Following transcription and verification of transcript accuracy, the transcripts were read three times and initial ideas were noted. By reading the transcripts, the researcher familiarized himself with the data and understood it as an entire unit of knowledge. A further reading of the transcripts was done by watching the screen recording alongside the transcripts to facilitate annotation of instances where the participants explained how they might apply a particular design principle. Such an annotation was intended to add richness to the data by capturing aspects of participants demonstrations on the website that were not captured during the transcription process, including sections of the website where they would want to display certain information. Even though Sandelowski (2000) suggested qualitative content analysis as the approach of choice in analyzing qualitative descriptive studies, a thematic analysis approach could equally be adopted depending on the objectives of the study (Neergaard et al., 2009). Therefore, Braun and Clarke’s thematic analysis approach was employed in analyzing the data in this study (Braun & Clarke, 2006). A thematic analysis approach in qualitative descriptive studies helps researchers to move beyond a description of the data to interpreting the findings without a significant departure from the literal description (Braun & Clarke, 2006; Guest et al., 2012). 164 After data familiarization, inductive codes were generated using NVivo version 12 (QSR International Pty Ltd.). The coding was guided by Saldana’s (2009) suggestions on the use of coding lenses and filters in coding qualitative data. By adopting the Saldana (2009) approach, the coding process started by using words or expressions in the data that spoke to the research question. A coding framework was developed inductively from the first four transcripts and then applied to the remaining data. This coding framework was revised iteratively as new data emerged from the transcripts. The codes were later grouped together to form emerging subthemes through an iterative and systematic process. Subthemes were further clustered into categories depending on their overall importance to the research question. Clustering of the themes moved the thematic categories from the subthemes to the thematic areas that represented a slightly higher level of abstraction of common participants’ perceptions and experiences regarding the applicability of the design principles. According to Willis et al. (2016), reporting comprehensive thematic findings in qualitative descriptive studies helps to move the data beyond individual participants’ reports to include an interpretation of the findings that reflects the participants’ cumulative experiences, yet stays close to the data. 7.1.7 Rigour Rigour generally refers to ways to establish trust or confidence in a qualitative study (Krefting, 1991). Lincoln & Guba, (1985) argued that rigour in qualitative studies cannot be ascertained by the post-positivist notion of validity but rather on the constructivist notions of credibility, transferability, dependability, and confirmability. Several processes were adopted to enhance the rigour of this study. Credibility is a quality criterion of ensuring that the findings of a study truly reflect the experiences of people with common characteristics (Krefting, 1991). To 165 enhance the credibility of this study, software designers with at least one year of experience in user-interface design were recruited. This was to ensure that participants’ views on the applicability of the principles reflected that of a typical software designer. To ensure confirmability, the video transcripts were watched to ascertain further information and to support the transcribed data. Thus, any doubts or concerns about the transcripts were verified from the video recordings. Transferability refers to the ability to apply research findings to another context. Whether qualitative research findings can be transferred to another context depends on the users of the research and not the researchers (Lusardi, 1996). To aid the reader’s decision on whether to transfer the findings, a detailed description of the research methods and the context in which the study was conducted has been provided. These processes are expected to provide some level of trustworthiness in the findings. 7.2 Results Fourteen software designers took part in the interviews. Even though some of the participants were graduate students at the time of conducting this study, they had all worked in software design at some point in their career. Participants with prior experience in health technology design were also able to provide useful insights on how the design principles might be implemented to address stigma, not only for sexual health-related challenges but across other sensitive health topics. Table 7.1 shows the demographic characteristic of the participants. The nine participants with a bachelor’s degree were also students in a master’s program at the time of conducting the study. To ensure the anonymity of the participants in the reported findings, their names have been replaced with pseudonyms. 166 Table 7.1 Demographic characteristics of participants Characteristic Participants, n = 14 Continuous Variables Mean (SD) Age 29.8 (5.4) Years of work experience 4.30 (1.8) Categorical Variables Frequency (%) Gender Woman 7 (50.0) Man 6 (42.8) Non-binary 1 (7.20) Area of specialisation in software design UI/UX Designer 7 (50.0) Project Manager 2 (14.3) Software Developer 2 (14.3) Other 3 (21.4) Experience with Healthcare Apps Yes 5 (35.7) No 9 (64.3) Education Bachelors 9 (64.3) Masters 5 (35.7) Country of residence Canada 12 (85.8) United Kingdom 1 (7.2) Swaziland 1 (7.2) Note: UI = User Interface and UX = User Experience 7.3 Themes The findings of the study revealed two main themes: 1) Understanding of stigma and the design principles; and 2) Addressing the design principles. Addressing the design principles has four subthemes related to applying the design principles to address stigma including: 1) 167 Responsibility for addressing the design principles; 2) Emotional design; 3) Collaborative design; and 4) Availability of resources. 7.3.1 Understanding of Stigma and the Design Principles It was clear that most participants lacked a comprehensive understanding of stigma. Participants’ understanding of stigma was limited to public stigma or what one participant referred to as “the act of discrimination or prejudice experienced by other people.” Even though participants’ subsequent responses indicated their awareness of the idea of internalized stigma, the majority did not know that internalized stigma existed as a named form of stigma until this study. Indeed, internalized stigma was generally understood as “shame feelings” experienced as a result of the reactions of others and not within the person’s inner self. Esther, a 31-year-old UX designer portrayed her understanding of stigma as quite limited to public/enacted stigma, “Until I saw this interview guide, my understanding of stigma was about how people treat others because of them having an attribute or belonging to one race. I didn’t know that stigma can also emanate from our own selves because of these same attributes. Alright, I think going through the list of principles it makes sense to have two forms of stigma.” Apart from a limited understanding of stigma, participants also considered the design principles to be at a high level, quite abstract and more theoretical than the traditional principles they are familiar with. The destigmatising design principles were considered less relevant when it comes to actual design work but may serve as a good reference source and a checklist to guide website design process. Jane, a 31-year-old UI designer portrayed the abstractness of the principles by stating that, “I wonder whether they were written by health professionals because they are very, very high level. It is very high level and abstract when it comes to actual design work.” The principles were considered abstract because they were examined in relation to 168 traditional design principles. Other participants did not regard them as design principles but rather as “design guidelines that were meant to guide the design process.” When asked about the differences, participants were less clear about the differences but indicated that “[a] design principle is a sub-component of a design guideline.” 7.3.2 Responsibility for Addressing Design Principles Before explaining how the principles might be implemented, participants first defined their responsibility with respect to each design principle. Each design principle was depicted as either the responsibility of the software designer or another member of the team depending on whether the principle is functional or non-functional (i.e., emotional, and content-related principles). While functional design principles were viewed as the responsibility of the software designer, the content-related and emotional design principles were perceived as beyond the scope of work of a software designer. The idea that “this is not the job of a designer” was conveyed by all participants for most of the content-related design principles. Cindy, a 34-year-old UI/UX designer specifically defined her responsibility for the content-related principles by stating that “a designer will not typically do this, but I can recommend who can help in getting this addressed. Maybe a subject matter experts or clients.” Although participants found the content-related principles to be beyond their scope of work, it was agreed that implementing the content-related principles would be the ultimate responsibility of the software designer “if the right content is provided.” Indeed, it was evident that a software designer will ultimately be responsible for implementing the content in a manner that enables end-users to easily grasp the basic and most important information a few seconds after visiting the website. Kerry, a 31-year-old UI/UX teacher described how her “design team” 169 would typically approach content-related principles even though she indicated such principles were “beyond her job description.” She stated, “If we have the information as a design team, then it's our job to determine how we structure the information on the website so that it's easy to navigate and how to give them confidence that this is evidence-based information that they can trust. But someone has to ensure that they provide the content. That’s most important for us.” As the content-related principles will eventually be implemented by software designers, some participants argued for a need for software designers to be educated on stigma and sexual health and to be part of the content generation process. By this education, software designers would not only be abreast of the subject area but it would also make it easier to implement in a manner that reflects the needs of the end-user. The need for education of software designers was typified by the expression of Sasa, a 32-year-old UI designer, “The education we need is about the subject matter of stigma because this is a new subject matter for us. So, education about the community, the community perception of the conditions, education about the diseases themselves, like a medical perspective, sensitivity education, medical knowledge training, what are the facts? What are the truths?” In addition to subject matter education, many participants also identified aspects of education relating to privacy laws, ethics, and empathy in contexts where software designers do design work. Kay, a 25-year-old UX designer conveyed the need for software designers to engage in context-specific ethics and legal education related to their subject areas in order be able to address stigma, “designers do not have to be like lawyers, but they should have [a] general idea about the law in order not to mislead the users. This is a huge, huge issue in designer community right now because of Facebook. We should be careful not to mislead people into agreeing with things that will negatively affect them. So, privacy law and ethics education would be very helpful.” 170 7.3.3 Emotional design Despite participants’ perception that the content-related principles were outside the scope of their work, it was generally agreed that the content-related principles were indeed important and useful for addressing stigma and could serve as a framework for changing the design process. All participants also indicated that the destigmatising design principles were a useful reference point for designing websites that address emotional concerns like stigma. That is, the content-related principles were portrayed as a toolkit that could help website designers think beyond the functional focus of user interface design to thinking about non-functional elements and emotional design components like stigma. James, a 30-year-old UI designer indicated how the emotional design principles could help raise his awareness about stigma-inducing components that he had not previously paid attention to, “If you're working with something that is as sensitive as sexual health, I think you want to have these ideas in mind so that you are not going off and showing happy, hunky-Dory photos of everybody happy, or you are using very vibrant and super fun colors. Like you are designing a music festival. So, in that way, they heightened my emotional sensitivity to what I'm putting in front of the client. So that's quite helpful.” These content-related and emotional principles were considered useful because it was acknowledged that such principles are often less obvious and more difficult to be specified by software designers during the design of a website. In addition to participants’ appreciation of the usefulness of the design principles, it was evident that software designers with prior experience in health technology design were more articulate about the emotional component of design. They particularly welcomed the principles that touched on enhancing emotional safety and signaled how those principles could help raise their emotional sensitivity when designing sexual health-related websites. Esther, a 31-year-old UI designer with prior experience in digital health design 171 explained how she tried to address emotional safety in a mental health app and how the content-related principles could aid her in future work, “I think in my last project on mental health app, we were thinking of ways to see how people can feel emotionally safe when using our app. We kind of glossed over these things but we could not nail it down like you have here. You are talking about sexual health, but I think they are also relevant in mental health. I have been involved in doing health related projects and this is a good starting point.” On the other hand, participants without prior experience in healthcare app design found the principles to be generally useful but were less clear about the emotional components of the principles. That is, people who have designed healthcare websites are more attuned to the relevance of destigmatising design principles in healthcare apps than those with no prior experience. 7.3.3.1 Addressing Emotional Safety in Software Design Even though most of the destigmatising design principles were considered outside the job description of a software designer, some specific strategies were suggested as a way of ensuring attention to emotional aspects and enhancing the security for web users when they visit sexual health-related websites. These include the use of: 1) Pop-up notifications regarding anonymity of websites; 2) Controlling cookies;2 and 3) Infographics and video testimonials for emotional connection. Pop-up notifications: Pop-up notifications on the landing page were considered as a strategy to ensure emotional safety by reassuring patients of the anonymity, safety, and security of sexual health-related websites during their first visits. Sim, a 32-year-old UI designer 2 Cookies are text files with pieces of data built for internet web browsers that allows the web browser to track, personalize, or save information about a user when they open a web page. 172 explained how he might design a pop-up notification that would first “greet website users” with assurances of emotional safety immediately when they log on the website, “I would throw a pop-up message to them at the very beginning that you are safe with us, that we won't be collecting any personal information. This [the website] is very protected and encrypted and guarantees your safety. This may have an assurance like, okay, this is a reliable source.” Cookies: Further to pop-up notifications, participants discouraged the use of website features that might give users the impression that their data could be collected or accessed by “people behind the website project.” The use of cookies was the most commonly expressed website feature that the participants perceived to be a privacy and security risk to website users. Even though cookies may not necessarily transmit peoples’ personal information (Aladeokin et al., 2017), participants argued that the use of cookies could be perceived by patients as invasive and a risk to their anonymity. It was quite apparent that participants preferred limiting, preventing, or giving users control over the use of cookies on websites. Karl, a 26-year-old software developer described the security risk of cookies and how they can be implemented to address stigma, “it's easy to track the data for people. There are many, many ways, like we have cookies, and analytics software where we capture all the data. To address stigma, it should not be easy to capture these kinds of information that is personal to people… just capture their engagement, not the personal details” Infographics and video testimonials: Aside from pop-up notifications and limiting cookies, other features like infographics and video testimonials were equally deemed as appropriate ways of providing emotional safety for peoples’ experiences with websites. Customized infographics and storytelling testimonials were particularly emphasized as a strategy to make website users feel engaged and emotionally connected to what others are recounting. Compared to “photoshopped images or images/videos from photo repositories,” custom images 173 and videos were suggested to be more effective in forming the kind of emotional connections that can help address stigma. Furthermore, participants indicated that story-telling based testimonials would be a preferred form of video to help describe the user journey. Esther, a 31-year-old UI designer vividly described the essence of customized images and videos in this statement, “when a website feels not original, when a website feels like these are just photos that are just taken from everywhere, the target audience can feel less trust, can feel disengaged. They might think the website owners are not serious because they're not putting enough effort into this. So again, custom photos, would be my recommendation.” While there was a consensus on displaying videos on the landing pages of websites as a way of addressing stigma, participants varied on the type of videos to include depending on their background. On the one hand, participants with prior experience in designing healthcare technologies were more interested in using customized videos involving real users. James, a 30-year-old UI designer indicated his preference for customized videos by stating that, “I definitely would have a real video on the main page on the home screen because nowadays the behaviour of people, they like to watch videos more than having to read a whole page of information. I’ve seen that when I evaluated a mental health app. So having like a nice introductory video that shows real faces is nice.” On the other hand, those with other backgrounds were inclined to suggest the use of avatars3 in place of human subjects. Preference for avatars were more common among software designers whose design experience was in gaming technologies. Despite the stated importance of infographics and video-based testimonials in addressing stigma, obtaining images and videos that reflected the diversity of website users was reported as a major potential challenge. While software designers acknowledged a general difficulty in 3 An avatar is a graphical illustration that represent a computer user or another character in a user interface 174 creating custom-based videos for display on websites, the difficulty was heightened for videos that involve people with stigmatised conditions, including sexual health. This challenge was highlighted by participants who had some prior experience in designing mental health apps. Cindy, a 34-year-old software developer with expertise in designing mental health apps expressed her worry in getting customized videos that reflects the diversity of potential website users, “Adding a video is technically easy but the harder part is the stakeholder management in getting the right video. So also, is privacy for people. Sometimes if health care professionals are stakeholders, they may be more worried about the kind of testimonials and videos to include. People may ask if people in the video represent everyone. So that's why that one's hard.” Furthermore, it was evident from the narratives that participants believed that a limited number of individuals would be willing to share and be identified with the stigmatised condition. Therefore, even though story-telling based testimonials were considered the best approach to form emotional connections towards addressing stigma, getting people living with sexual health-related conditions to narrate their stories via videos was acknowledged as a major potential challenge. 7.3.4 Collaborative design Even though there was common agreement amongst participants that applying the content-related destigmatising design principles is “not the job of a designer”, all participants discussed the role of other experts in addressing the design principles. Indeed, all participants noted that designing a website is not the sole work of a software designer but a collaborative process that involves several other stakeholders and team members. The role of other experts was made evident in participants’ perspectives on how they might implement the content-related 175 principles. Several professionals, such as a subject matter expert, a content writer, a software developer, and a project manager were among some of the experts that were identified as necessary in a collaborative design project. Sasa, a 32-year-old UI designer described who needs to get involved and what should be done if he were tasked with implementing the set of principles in a design project, “the best thing to do is to have like this three-step process, I would always encourage the client to send us some reading material at first. So, our team get introduced to the topic, especially if we’re new to it. From then we will have an interview with the client and a subject matter expert to we can ask questions based on the material that we have read, right. The content writer then writes it up and the software engineer puts it up on the website. This process is a bit complex so you may need a project manager to coordinate it.” Participants did not just merely name the job title of the required experts but were also explicit about their roles in getting the design principles addressed. The role of the content writer was described by the software designers as the person who would be responsible for the content-related design principles. Even though it was agreed that a subject matter expert or a client may provide content, it was suggested that a content writer would be required to rewrite the content and present it on the website in a way that is comforting, engaging and blends well with the user journey. Kerry, a 31-year-old UI/UX designer explained the role of a content writer in a website project if stigma reduction is to be achieved. “As a designer, the content can be provided to me, and I can just put it up on the website. But since this is about addressing stigma, I think a content writer can re-write the content in a way that is good for the website, with the right language, the right sensitivities of the topics and at a level everyone can understand. In my other projects I do everything myself but for websites like this, I will engage a content writer.” In addition to the collaborative stakeholder engagement, it was apparent that other collaborative design approaches, including user testing and user research, were deemed as appropriate ways of engaging the end-users towards addressing stigma. Instead of restricting user 176 testing to the beginning or the end of a design project, most participants wanted each destigmatising design principle to be tested with the end-user before, during and after implementation. User testing was particularly emphasized for content-related principles because content is where stigma was likely to be amplified. Sasa, a 32-year-old UI designer emphasized the need for user-testing of each principle by stating that, “…It has to be user tested as well. You cannot just sit down and write it down and then publish it on the website. You have to put it in front of people, do you think it is coming off as judgemental? Is it accessible language? Um, does it help de-stigmatise sexual health related conditions? So, I think that's why there's a lot of work that goes into this before you are sure you hit the mark.” While there was a consensus on user testing, there was variability about the type of user testing to be adopted for sexual health-related technologies. Some participants, particularly those with prior experience in mental health apps, emphasized face-to-face user testing while others were inclined to use analytic tools (i.e., such as UsabilityHub.com or Crazy Egg™ 4). The argument for adopting analytic tools was to engage the user without introducing social desirability bias, where end-users in face-to-face user testing might tell the software designer that they like the website, in order to seek the approval of the designer. While participants confirmed that they frequently use analytic tools for business-related websites, they contended that such tools can equally be applied for usability testing of sexual health-related websites. Analytic usability tools were postulated to help patients express their concerns with sexual health-related websites in a safe way because the end-user would not be face-to-face with the software designer. 4 These are remote user research and user testing platforms that are used to validate designs and reduce researcher biases. UsabilityHub.com, for example allows people to sign up to review websites or prototypes of websites, and the person providing the feedback received about 20 cents per review. 177 7.3.5 Availability of Resources Despite acknowledging the usefulness and importance of the destigmatising design principles, it was evident from participants’ narratives that limited resources could be a barrier to fully implementing the set of principles. Almost all participants raised the issue of limited budgets and time limitations often imposed on website projects. Although intensive user research and user testing was considered the best strategy for addressing the content-related design principles, it was hypothesized that budgetary limitations would likely be encountered during the design process that involves user research and user testing. Financial constraints were reported to be most common at the user-research and content generation stages than the actual software development. Jay, a 28-year-old project manager, described the financial costs associated with each of the various steps involved in addressing the principles, the cost of which would ultimately be passed on to the customer, dramatically increasing the website development budget, and perhaps extending timelines as well, “A major challenge is how to get the resources to address all these principles. To cater for all these principles, a website design company might charge you a lot of money because there are so many things they need to do. And they may charge you by the hour of all the interviews and everything. That costs a lot of money. So again, how much time do we have to develop this? What's your deadline, you know, and how much budget would that be? So, you have to think of all the resources that comes into play.” Although it was acknowledged by participants that additional resources might be needed to implement most of the content-related principles, some of them felt that clients or “project owners” often do not dedicate resources to user research and user testing activities involving content. In contrast, it was evident from participants’ narratives that the content-related principles would be critical in addressing stigma and would therefore benefit from intensive user research and user testing. There was a major concern with limited resources for user research 178 activities for healthcare related projects as evident in the response of a software designer who was asked about his prior experiences in designing a mental health app, “You know website projects are usually smaller projects and clients just allocate some small money to it. But getting the money to pay all the people will be difficult to come by. I mean how do you convince a client that you want to employ a content writer or a UX researcher for a simple website project. So we often tend to do everything on our own and I don’t think I can independently address all these principles without the necessary resources” Financial resources were acknowledged as one of the most critical success factor in the implementation of the principles because participants indicated that resources will be needed to hire the experts involved in the various aspects of the design process. 7.4 Discussion In this study, software designers’ impressions of using destigmatising design principles were explored. In general, software designers lacked a comprehensive understanding of the concept of stigma, yet thought that knowledge of the design principles could enhance their work on designing healthcare websites. While they considered many aspects of the design principles to fall outside the scope of practice of software designers, they indicated that awareness of these principles could perhaps contribute to the destigmatising potential of web design. Given the limited understanding of stigma, many software designers, particularly those with no prior experience in healthcare were less explicit with how they described the application of the design principles to address stigma. In this section the findings are discussed in the context of addressing internalized and public stigma, responsibility for addressing stigma and required resources, emotional design, and specific design approaches that software designers might adopt 179 to address the principles towards reducing stigma. The section concludes with a description of the study limitations and directions for future research. 7.4.1 Addressing Internalized and Public Stigma It was evident from the interviews that participants lacked a comprehensive understanding of the two main forms of stigma inherent in people with sexual health-related conditions. Instead, software designers’ understanding of stigma reflected the commonly held description of stigma as discrimination and prejudice against people with a certain characteristic or against people from a certain group (Pryor & Reeder, 2015; Canadian Mental Health Association, 2020). Although stigma was used interchangeably with prejudice and discrimination by participants in this study, it should be noted that the three concepts have different focus areas. These differences, however, typically center on the subject matter of interest rather than real conceptual differences (Stuber et al., 2008). Thus, despite demonstrating a limited understanding of internalized stigma, it can be argued that some of the emotional safety strategies recommended by software designers in this study reflect management strategies for internalised stigma in sexual health (Stangl et al., 2013; Turan et al., 2017). That is, displaying pop-up notifications regarding the anonymity of websites, the practice of limiting or controlling cookies, and displaying storytelling testimonial videos on landing pages were all meant to alleviate the internalised feeling of stigma associated with sexual health-related conditions. This suggests that while software designers might not have been aware of the term internalised stigma, they may possess skills that they can apply to address internalised stigma in software design. The ability to address internalised stigma via websites is an essential attribute because internalized stigma is considered the most common and most difficult form of stigma to be addressed in the context of 180 sexual health (Turan et al., 2017). Of note, even though participants demonstrated a relatively better understand of public stigma than internalised stigma, strategies that sought to address public stigma were less emphasized in this study. That is, no strategies were suggested for implementing most of the design principles meant to address public stigma. This apparent lack of suggestions to remediate public stigma could be due to the perception that public stigma management strategies cannot be implemented in digital health technologies, or perhaps public stigma was considered too obvious, and therefore overlooked. 7.4.2 Responsibility for Addressing the Design Principles One of the most interesting findings of this study was the indication from all the participants that the application of the destigmatising design principles was outside of the scope of work of software designers. While the software designers in this study acknowledged the importance of the design principles and described strategies that they might employ to apply the destigmatising design principles, it appeared that they expected the project owner to provide non-stigmatising content. The approaches and issues discussed by the software designers about how to address responsibility for addressing the design principles are presented below. 7.4.2.1 Collaborative Design It was revealed that while a software designer might be able to easily implement functional design decisions, implementing non-functional and content-related design principles could be problematic without the active involvement of other stakeholders and team members. Having acknowledged that the content-related design principles were outside the scope of practice of software designers, participants felt it was imperative to describe how additional 181 experts could help to address the design principles. While the process might differ from project to project, participants suggested that stakeholders should have the following roles and expertise: 1) Subject matter experts who produce the content in consultation with patients and the design team; 2) A content writer who would re-write the content into simple language and structure the layout of the content; 3) A UI or UX designer who would design how end-users would interact with the content on the user-interface; 4) A software developer who would encode and create the website; and 5) A project manager who would coordinate all stakeholders and provide direction for the project. Despite the important role of these stakeholders identified in this study, prior studies revealed limited to no involvement of major stakeholders like subject matter experts or a content writer in the design process (Ahmed et al., 2018; Ferron et al., 2017; Geuens et al., 2016). Participants in this study indicated that it would be very difficult to implement the non-functional and content-related principles without the stakeholders. Further, most indicated that a project manager was considered the most important stakeholder in getting all the principles addressed effectively. Even though a project manager might not be directly involved in the technical or content design process, they were tipped as being responsible for mobilizing the necessary resources, and to make ultimate decisions and take ultimate responsibility for the project (Gandomani et al., 2020). Simply put, a project manager sits at the top of the organogram in a design team and would be crucial if stigma reduction is to be achieved in web-based projects. While different definitions exist, the Project Management Institute define a project manager as “the person assigned by the performing organization to lead the team that is responsible for achieving the project objectives” (Rose, 2013). Project managers are increasingly becoming common in various software design projects (Gandomani et al., 2020), which might 182 reflect the increased awareness of the importance of project managers for website design noted in this study. Indeed, project management skills have been included as an essential skills of nursing informaticians in two main nursing informatics competency scales (i.e., the TIGER-based Assessment of Nursing Informatics Competencies (TANIC) and the Nursing Informatics Competency Assessment (NICA)) (Sipes, 2016). The inclusion of project management skills in nursing informatics competencies suggests that nursing informaticians could possibly take on the role of project managers in digital health-related projects for stigmatised conditions. 7.4.2.2 Education for Software Designers Given indications that the content-related design principles were outside the scope of software designers, domain-specific education was advocated for software designers to equip them with adequate domain knowledge, and empathy building skills before embarking on a design project. The need for education on stigma, ethics, empathy, and privacy laws were particularly emphasized by participants in this study. The need for ethics and domain-specific education is consistent with the universal call for ethics education for software designers in the domain areas where they conduct their work (Génova et al., 2007; Lurie & Mark, 2016). While no literature could be found to support the idea of educating software designers on the topic of stigma, there is a stated need that context-specific education on ethics and privacy laws is important for software designers who develop web-based products for people in specified contexts (Génova et al., 2007; Lurie & Mark, 2016). Awareness of ethics, privacy laws and ethical values like empathy might help software designers assess the goodness and consequences of their actions when designing healthcare technologies (Génova et al., 2007). Indeed, ethical decision making and ethical sensitivity to the context of practice is considered a necessary 183 precursor to good nursing care (Milliken & Grace, 2017). Software designers could also be educated to understand that sexual health-related technologies that incorporate ethical values including human dignity, empathy, and emotional connections may prove useful in addressing stigma inherent in sexual health. The participants in this study consistently indicated that education for software designers is necessary considering the challenges they anticipated in adequately addressing unfamiliar concepts or factors that fall outside their scope of practice. 7.4.2.3 Resource Allocation Further to the participants’ identification of the types of stakeholders required to implement the destigmatising design principles, resources including time, personnel and money were acknowledged as central components that can determine the successful implementation of the design principles. Of these, financial resources were considered the most important component. Indeed, limited financial resources has been reported as a major challenge in the design, utilization, and sustainability/maintenance of digital health technologies (Cummins & Schuller, 2020). Despite the importance of financial resources in securing additional experts to address the content-related design principles in this study, limited resources are often dedicated to ensuring that the emotional concerns of patients are properly addressed in digital health technologies (Bright & Coventry, 2013; Hassenzahl, 2005). Instead, many more resources are often dedicated to the platform development, implementation, and maintenance of digital health technologies than to content of said technologies (Vayena et al., 2018; WHO, 2016). While resources are, without a doubt, necessary for the development and utilization of digital health technologies, project leads must be made aware that effective utilization might not take place when the emotional and less tangible concerns of the patient such as stigma are not addressed in 184 the early stages of design. For effective utilization of digital health technologies, a solid foundation must be built by way of addressing stigmatising components before carrying out the actual software development and subsequent utilization. 7.4.2.4 Interprofessional Approaches Finally, while it is important to identify who is ultimately responsible for addressing each of the destigmatising design principles, an interprofessional approach might be needed to fully address the principles in a way that tackles both internalized and enacted stigma. Recent research on interprofessional approaches to support activities that are at the intersection of healthcare and computer science may provide some recommendations. For example, in a recent study by Vaughn et al. (2021), they found that students in an undergraduate course on healthcare information visualization were able to identify that people from different disciplines had differing roles, but all were needed to successfully carry out the project. Further, a large examination of the professional disciplines involved in precision medicine (i.e., using technology to match patients’ genetic information to specific drugs) revealed that computer scientists and engineers were both involved in the health systems aspects of precision medicine (Xu et al., 2021). These recent approaches to collaboration between traditionally siloed fields may help inform approaches that ensures that software designers are aware of the destigmatising design principles and healthcare providers also understand the suitable content to provide that can help in destigmatization efforts. An interprofessional approach is needed because it has been hypothesized that a team with diverse background, facilitated by an integrated and interactive team environment, is anticipated to effectively address complex challenges in modern healthcare systems (Kim, 2019). The need for an interprofessional approach in this study is consistent with 185 prior studies that examined physicians’ education on theoretical and practical knowledge of ultrasound physics and the need for artificial intelligence to integrate pragmatic clinical issues (Celebi et al., 2019; Renn et al., 2021). It can therefore be argued that similar approach involving the destigmatising design principles might be adopted for healthcare software designers to understand the basics of sexual health-related conditions and stigma. 7.4.3 Emotional Design Participants largely described how the design principles in this study would help them extend their focus beyond the defined scope of work to include emotional design skills. The emotional component is important because emotional design principles are generally less obvious, more difficult to specify, and most often neglected by software designers during systems’ design (Shehadeh et al., 2021). While emotional design has long been emphasized in healthcare technologies (Norman, 2004; Vaes, 2019), software designers, including healthcare software developers, still adopt largely technology-centered approaches in developing such technologies. This study extends that focus to include the emotional, affective, and user experience aspects of digital health technologies. Despite the wide recognition of emotional design in healthcare technologies, a recent systematic review of digital mental health technologies used within China suggested that digital health technology design was largely focused on usability, effectiveness, acceptability, and privacy to the neglect of emotional design elements including stigma (Zhang et al., 2021). Even though software designers in this current study recounted the usefulness of traditional design principles in improving the usability and functionality aspects of sexual health-related technologies, Norman, (2004) argued that the emotional aspect of design, including 186 addressing stigma, could be more critical to a product’s success than its practical features. Thus, the content-related design principles in this study (which are largely non-functional principles) might help software designers to create websites that address both public and internalized stigma, or at best, do not provoke internalized stigma in end-users. Furthermore, as software designers showed a limited understanding of stigma, an explicit set of domain-specific principles to guide them through web design may be necessary in producing technologies that that addresses the emotional concerns of patients arising from stigmatising conditions. 7.4.4 Functional Design Strategies to Address Stigma Design strategies including pop-up notifications, infographics, and videos, and controlling cookies were deemed as appropriate functional design strategies to address security and emotional concerns of web users. The emphasis on the use of pop-up notifications and storytelling-based testimonials reflected participants’ position that the content of a website could be crucial for addressing internalized feelings of guilt and stigma that may emanate from a web interface. Studies about pop-up notifications used in other technologies including electronic medical records and clinical decision support systems suggest that notification features could be a valuable warning indicator for clinical staff on critical issues related to patient care (Bright et al., 2012; Jaspers et al., 2011; Zhang et al., 2015). It can therefore be argued that pop-up notifications could be implemented on sexual health-related web technologies to pre-inform users of the anonymity and security guarantees associated with the use of the website. While pop-up notifications were considered a useful strategy by the participants in the current study, other studies have revealed that the use of general notifications on sexual health-related technologies (e.g., test results or partner notification) could lead to an increased level of anxiety 187 and could perpetuate stigma among people with sexual health related conditions (Morris et al., 2014; Wood et al., 2018). However, the forms of notifications in these studies (which centered on displaying test results or partner notification, which could induce anxiety) differed from the pop-up notifications suggested by the software designers in this study. The use of text-based content in addressing stigma via sexual health-related websites in previous studies contrasted with software designers’ preference for the use of custom-based storytelling testimonials in this study (Karamouzian et al., 2018; Witzel et al., 2016). It was argued that the use of custom-based infographics and video testimonials that present patients’ stories in simple and clear language could be capable of supporting emotional connections while dispelling myths and misconceptions that often fuel stigma. The use of storytelling as a myth reduction strategy proved successful in combating anti-vaccine misinformation among parents because participants identified with the storytellers (Shelby & Ernst, 2013). Participants in the current study suggested the use of custom-based videos with real faces because they thought that could help people feel connected to and trust the website. Software designers’ preference for videos in this study is consistent with patients’ preferences for the use of videos and infographics in place of text in Study 2 (i.e., Chapter 5). Despite the usefulness of storytelling-based testimonials, it was noted that it could be challenging to recruit people living with stigmatised conditions who would be willing to share their stories on a website. This finding is consistent with prior research describing the difficulty of obtaining custom-based images and videos from individuals who may experience stigma (Livingstone, 2007; Vojtila et al., 2021). Thus, further research is warranted that investigates alternate and possibly creative ways of making custom-based video testimonials, perhaps through animation, metaphorical images, or the use of actors, to portray sensitive and personal 188 experiences. Further studies could also explore ways of partnering with individuals and communities to find ways of creating customized videos with people living with stigmatised conditions. 7.4.5 Participatory Design Strategies to Address Stigma Participatory design is a design approach that goes beyond user-centered and co-operative design approaches to include end-users as active participants in the design process (Kushniruk & Nøhr, 2016). Software designers’ strategies of user research and user testing reflected the key principles of participatory design approaches. User research: User research is a common practice in designing digital health technologies, although the extensiveness of this research and the role of the user are not made explicit in traditional user-centered design approaches (Wilson, 2013). Therefore, user research tends to vary depending on the philosophical orientation of the researchers and the purpose of the intended technology. Available evidence suggests that patient involvement is often limited to a single stage of the development process even though user-engagement in designing digital health technologies has been strongly recommended (Vaisson et al., 2021). This is in contrast to the current study wherein the software designers recommended active patient engagement and patient decision making, which extends the practice of user research beyond the traditional process of user involvement. Furthermore, it can be argued that the form of user research as suggested in this study sought to extend ethnographic studies typical in human-computer interaction research to include patients, ethnographers, health researchers, and user experience researchers as active partners in the research process. Even though the form of user research proposed by participants in this study could be time consuming and costly, it can be argued that 189 intensive user research might be the only way to get to the pain points of stigma reduction. A collaborative approach to user research is important because it can be difficult to engage with a user group that experiences stigma using conventional user research approaches. User testing: User testing, which is simply the practice of trying out new design ideas or prototypes among users, is a common practice in software design. Unlike traditional usability testing that is typically limited to testing of design prototypes or end-products after the software programming has been completed (Borsci et al., 2015), the participants in this study suggested the need to extend user testing to include every design principle (particularly the content-related principles). That is, instead of waiting to test out a prototype or a final product after the design is finished, the participants recommended testing out each aspect of the design before a prototype or a final product is completed. While this approach could be costly and time consuming, it may be less expensive in the long run because the cost of redesigning a product after it has been developed is thought to be 5-10 times higher than the cost incurred in conducting thorough user research or user testing (Dalaba et al., 2014; Field et al., 2008). Furthermore, unlike traditional user testing that is largely focused on identifying usability and functionality problems (Jaspers, 2009; Tullis et al., 2002), this study suggests that software designers see the need to extend the focus of user testing to include identifying any stigmatising components inherent in content design. Incorporating the destigmatising design principles into user testing could be very important because traditional user testing might not be effective in identifying stigma-related problems that typically fall outside the expertise of software designers. This is to say that to address stigma it is important to ensure that the content of a website does not provoke stigma in the intended users, and this can only be done if all content is tested among or created alongside the end-users. 190 7.4.6 Limitations There were some limitations associated with this study. First, the software designers were not applying the set of design principles in actual design situations. Their perceptions of the design principles were all based on descriptive reports. Therefore, even though they generally thought the design principles were important, it is not clear if they would apply the principles because they did not perceive it to be part of their work. Second, it is possible that some participants might have reported that they should apply the design principles when in fact they actually believed that they would not do it in practice. This kind of social desirability bias could have been mitigated if participants had been asked to use the design principles to design a product. Furthermore, most of the research participants were either graduate students or in the early career of their professions. Even though all had some level of experience in software engineering, their ability to apply the principles might have been limited by their relatively limited software design experience (i.e., 4.3 years on average). Thus, the 4.30 years did not meet Ericson’s (2009) criteria of being considered an expert in a given field. Finally, the research was conducted during a pandemic, and it was not possible to meet participants face-to-face. Thus, a virtual demonstration of how participants might apply the principles may have been inferior to a face-to-face demonstration using paper-based scenarios. 7.4.7 Directions for Future Research Reflecting on the findings from this study, there are several directions for future research. First, this study was based on participants’ perceptions of the applicability of the destigmatising design principles. Future research should develop a prototype web-based application using the principles as a reference guide. This kind of research could determine how software designers 191 would apply the design principles in practice as well as explicate the real challenges inherent in implementing the principles. Second, since this research was conducted without reference to a specific condition related to sexual health, future research should examine the applicability of the design principles using a specific condition related to sexual health. Finally, since stigma is context-specific, future research should consider applying the principles among software designers in other settings, particularly among designers that specialize in design for stigmatised conditions. 7.4.8 Conclusion This chapter presented a qualitative descriptive study that examined software designers’ perceptions of the applicability of a set of destigmatising design principles. The findings of this study indicated that the destigmatising design principles were considered to be important, but the designers did not perceive that they had the ultimate responsibility for applying most of the content-related design principles. 192 Chapter 8: Overall Discussion & Conclusion 8.1 Introduction This dissertation presented four studies that were focused on addressing sexual health-related stigma in the context of digital health technologies. These include: 1) A scoping review on the stigmatising and destigmatising properties of sexual health-related websites; 2) A usability evaluation of a sexual pain website using a trauma-informed care framework to examine patient perspectives of designing destigmatising websites; 3) A Delphi study in which a set of 19 destigmatising design principles (see Table 6.14, page 142) were developed in consultation with experts in stigma and sexual health-related research; and 4) A qualitative descriptive study to examine how the destigmatising design principles might be applied by software designers. The findings of this dissertation produced important contributions that can advance knowledge on designing digital health technologies for health-related conditions that are considered sensitive and uncomfortable to discuss. In this chapter, the impact of this dissertation research, and implications for nursing informatics and human-computer interaction are discussed. The chapter concludes by presenting the overall limitations of the research and direction for future studies. 8.2 Themes The following five overarching themes are present in this dissertation. These include: 1) The focus of stigma reduction; 2) Perpetuating stigma via digital health technologies; 3) Addressing stigma via digital health technologies; 4) Trauma-informed user-centered design; and 5) Interprofessional approaches to digital health design. 193 8.2.1 The Focus of Stigma Reduction The literature shows that stigma can be manifested as either public stigma, internalised stigma (self-stigma), stigma-by-association, or structural stigma (Pryor & Reeder, 2015). The four studies produced varying recommendations to address each of these forms of stigma. Stigma-by-association and structural stigma had the least attention, while public and internalised stigma had the most. In regard to stigma-by-association, it was revealed in the scoping review that participants were generally concerned about the potential for, and experiences of stigma based on their engagement with websites (Witzel et al. 2016; Byron et al. 2013). Similarly, patients in the website evaluation study (Study 2) also expressed concerns of being associated with a health condition for merely looking through a website in a public space. Even though one design principle was proposed in the Delphi study to include content that does not draw people’s attention when using a sexual health-related website in public spaces, this principle did not achieve consensus among the panel of experts. It has been established that societal ideologies and institutional norms tend to create, perpetuate and exacerbate structural stigma (Corrigan & Lam, 2013). However, strategies to address structural factors that exacerbate sexual health-related stigma were not evident in any of the studies. Forming advocacy groups in websites was the only design principle proposed by participants in the Delphi study to address structural stigma, but it failed to achieve consensus at the end of the 3rd round. Furthermore, it was not evident whether strategies recommended by participants in the Delphi study to normalize stigma by displaying sexualized images could address structural stigma. The apparent lack of structural design principles could be attributed to 194 the fact that factors that perpetuate structural stigma (i.e., the social, economic, and political power imbalances and pre-existing inequities) are beyond what could be addressed via a website. Aside from stigma-by-association and structural stigma, most of the destigmatizing design principles and the proposed strategies for implementing them were focused on addressing the two main forms of stigma (i.e., public stigma and internalised stigma). Compared to internalised stigma, most of the studies (i.e., six studies) in the scoping review (Study 1) were specifically focused on web technologies designed to address public stigma (Bauermeister et al., 2019; Dudina & Tsareva, 2018; Flickinger et al., 2018; Ho et al., 2017; Noltensmeyer et al., 2016; Shearer et al., 2012). Similarly, participants in the Delphi study were more inclined to make public stigma the focus of attention when designing sexual health-related technologies. The emphasis of the participants in these studies on public stigma contrast with the largely individual stigma management strategies espoused by earlier stigma researchers (Goffman, 1963; Heijnders & van der Meij, 2006; Link & Phelan, 2001). Despite the focus of the scoping review and the Delphi study on public stigma, software designers’ strategies for addressing stigma via web interfaces were largely centered on internalized stigma. Software designers’ focus on internalized stigma is important because internalized stigma components are usually not obvious to intervention developers (Shehadeh et al., 2021), despite being the the most common and most difficult form of stigma to be addressed in the context of sexual health (Turan et al., 2017). The findings of the various studies on addressing public stigma, internalised stigma or stigma-by-association suggests that all facets of stigma must be given equal attention when designing sexual health-related technologies. Placing equal focus on all forms of stigma may not only enable users of sexual health-related technologies to have reduced feelings of guilt (i.e., internalized stigma) but also offers stigma management strategies needed to tackle stigma from 195 the community and societal perspective (i.e., public stigma), and reduce the chances of stigma-by-association. 8.2.2 Perpetuating Stigma via Digital Health Technologies The findings of Study 1 indicate that even though digital health technologies are increasingly used to address stigma, the way in which the content of such technologies are designed could inadvertently perpetuate both internalized and public stigma. Of the 17 studies that were focused on addressing stigma, seven studies were found to inadvertently worsen either public stigma, or internalized stigma, or both public and internalized stigma. A possible explanation for provoking stigma in web design was provided in Study 4 where software designers were found to not be experts in healthcare and might not have been aware of what constitutes stigmatising or non-stigmatising components when designing such technologies. Prior studies have revealed that software designers have a limited understanding of domain knowledge in health because they are not usually involved in the content generation process (Darvish et al., 2014). Even though the use of infographics and videos on websites were preferred by participants in the website evaluation study (Study 2) and the qualitative descriptive study (Study 4), the findings of the scoping review (Study 1) revealed that the display of images could inadvertently perpetuate internalized feelings of stigma. The findings of the scoping review specifically revealed that images displayed in a vulgar or explicit manner, and sexualized images of the body on websites could potentially aggravate stigma and further widen the barriers to the use of web-based technologies (Davis et al., 2012; Karamouzian et al., 2018; Klein & Lomonaco, 2016; Magee et al., 2012; Payton & Kvasny, 2016; Shearer et al., 2012; Witzel et al., 196 2016). While explicit images could aggravate stigma, participants in the Delphi study were concerned that hiding or making images too subtle may rather perpetuate stigma by signalling people to go into hiding and encouraging suppression of peoples’ feelings. Therefore, while getting appropriate content was considered by software designers an important strategy in addressing stigma in Study 4, it should be noted that is not just a matter of producing appropriate content, but how such content is eventually displayed on the website is crucial in addressing stigma. The Delphi study, for instance, revealed that instead of hiding images, they should rather be displayed on websites to convey the message that it is acceptable for users to express hidden feelings that often fuel internalised stigma. Therefore, the original design principles presented to the Delphi study participants that suggested to hide or conceal sexualised images were subsequently revised to ‘normalise’ such content. Given the contradictory findings on the use of images, the need for user testing of such images is warranted to ensure that all images conform to the socially acceptable norms in the users’ context. 8.2.3 Addressing Stigma via Digital Health Technologies In the website evaluation study (Study 2) the strategies recommended by patients to ensure privacy, anonymity, emotional support, and inclusivity, and to empower web users through validating personal experiences were consistent with the principles that met consensus in the Delphi study (Study 3). Similarly, patients’ desire for sexual health-related websites to convey real-life experiences and facilitate interaction between people as a way of addressing stigma are consistent with software designers’ recommendations for storytelling-based testimonials in the qualitative descriptive study (Study 4). Since many sexual health-related conditions are hidden, it can be argued that ensuring anonymity, privacy, and security of 197 websites and getting non-sensitive and less sexually explicit content might prevent stigma “from looming large” and possibly reduce the internal distress and negative feeling that tend to fuel stigma (Goffman, 1963). The consistency of the various strategies across studies could mean that the various perspectives (i.e., patients, stigma researchers, and software designers) taken in thinking about developing web-based sexual health interventions have an implicit understanding of what is needed in their respective domains to address stigma. It can therefore be argued that this dissertation research has produced the knowledge that brings these different perspectives together in a design project, while providing strategies and identification of supports needed by the design team to address stigma during the design of sexual health-related technologies. Another destigmatising design principle consistent across the four studies was the suggestion to have real-life experiences displayed on websites as a way of addressing stigma. Participants in the website evaluation study (Study 2), the Delphi study (DP26.1, page 142), and the qualitative descriptive study (Study 4) consistently suggested that websites should use videos of real-life experiences. Yet, the findings of the scoping review (Study 1) were divergent about whether text-based experiences or video-based experiences would be most appropriate in addressing stigma. The importance of displaying real life experiences on web platforms were demonstrated in prior studies that suggested a reduced level of stigma among people living HIV/AIDS and promoted the uptake of web-based resources (Lagu et al., 2019; Magee et al., 2012). Despite the usefulness of real-life experiences on web platforms, other researchers have argued that sharing experiences, testimonials, or stories via web apps may open up the opportunity to share negative perspectives and promote online bullying that could eventually (re)produce, perpetuate, and exacerbate stigma (Bauermeister et al., 2019; Flickinger et al., 2018; Ho et al., 2017; Magee et al., 2012). As suggested by the software designers in Study 4, the best 198 approach may be to create web content that includes storytelling-based testimonials which should be vetted by diverse users to ensure such testimonials do not provoke the emotional vulnerabilities of potential users. Another destigmatising strategy from the qualitative descriptive study (Study 4) was for the use of pop-up notifications and controlling the use of cookies. This suggests that software designers could use functional strategies to implement non-functional design principles towards addressing sexual health-related stigma. The success of pop-up notifications in other technologies such as clinical decision support systems and electronic medical records suggests that they can be a valuable feature in alerting web users to the anonymity and security of sexual health-related web technologies (Bright et al., 2012; Jaspers et al., 2011; Zhang et al., 2015). Provider moderation of websites emerged as a strategy of addressing stigma in the scoping review (Noltensmeyer et al. 2016). However, provider moderation of websites failed to achieve consensus in the Delphi study (Study 3), neither was it evident in the qualitative descriptive study (Study 4). A possible reason that could account for this could be the fear of obtrusiveness by unscrupulous persons pretending to be health providers or the possibility of web-based stigmatization, bullying, and victimization of website users (Payton & Kvasny, 2016; Witzel et al., 2016). Another possible reason could be that the resources for website moderation (i.e., time and compensation) might be quite large and project owners may be unwilling to commit resources for such moderation. Even though software designers in the qualitative descriptive study (Study 4) raised legal and ethical challenges related to getting contact of health professionals onto websites, it can be argued that adding contact information of health professionals may be a less costly and less obtrusive alternative than provider moderation. 199 8.2.4 Trauma-Informed User-Centered Design User-centered design is generally described as an iterative design process in which designers focus on the needs of users in each phase of the design process (Reynolds, 2020). While user-centered design approaches are typically applied from a technically focused perspective (Mao et al., 2008), the findings across the four studies demonstrates that a trauma-informed care framework used in the context of user-centered design may be an appropriate method in addressing stigma. Given the limited and often vague application of user-centered design (Cornet et al., 2020), it is important to situate or modify the method to context-specific technologies including sexual health-related technologies. Indeed, Mao et al. (2008) have argued for fine-tuning user-centered design practices in specific fields to provide a more structured approach that guides user-interface designers. It can be argued that by adopting a trauma-informed framework, this study provides a structured approach for designing destigmatising sexual health-related technologies. Other technology interventions for sexual health have successfully incorporated elements of trauma-informed care. For example, Sockolow et al. (2018) developed a smartphone gaming application for reducing sexual risk behaviours among black adolescent girls. Even though addressing stigma was not a primary objective in their study, Sockolow and colleagues’ findings suggest that a trauma-informed care framework could shift the focus of digital health technologies from a predominant focus on usability and functionality to include interface components that can facilitate emotional connections between people engaging in risky sexual behaviours. Patient engagement is a cardinal requirement in traditional user-centered design involving patient-centred digital health technologies (Gagnon et al., 2020). However, such involvement is frequently limited to one stage of the development process (i.e., either understanding users, 200 developing/refining prototypes, or user testing) (Vaisson et al., 2021). The trauma-informed care framework in the website evaluation study (Study 2) provides insight on the kind of trauma-informed design principles that can help address stigma during technology design from the perspective of the patient. Furthermore, the user testing approaches proposed in the qualitative descriptive study (Study 4) extend beyond conventional user testing to include content-related principles. Given that sexual health content displayed in a vulgar manner could perpetuate stigma (Davis et al. 2012), consistent user testing of any planned content is essential to ensure that sexual health technologies do not produce unintended stigma and trauma. As people are more likely to feel connected to images that are socially acceptable to them (Hood & Friedman, 2011), user testing of such content could eventually produce sexual health technologies that conform to socially acceptable norms, and perhaps reduce stigma. Therefore, adopting a trauma-informed user-centered design approach throughout the entire design process might not only place the end-user at the forefront of web development but might also help stakeholders reduce the chances of re-traumatization and re-stigmatization of patients via digital health technologies. 8.2.5 Interprofessional Approaches to Digital Health Technology Design It was suggested in the scoping review (Study 1), the website evaluation study (Study 2), and the qualitative descriptive study (Study 4) that software designers do not have substantive knowledge in sexual health and stigma. Further, given that Delphi study participants were neutral about technology functions to reduce stigma (Study 3), it is possible that many health professionals do not have deep knowledge of health technology design. Since health professionals and software designers lack the necessary knowledge in each other’s field, an interprofessional approach could support the design of technologies that adequately addresses 201 stigma. Even this dissertation research may have functioned as educating the entire design team not only on stigma but also on design practices. An interprofessional approach to health technology design was demonstrated in a recent study on information visualisation where students noted that multiple types of expertise were needed to successfully carry out the project (Vaughn et al. 2021). Thus, healthcare professionals could be educated about human-computer interaction while software designers could be educated about stigma and sexual health respectively. This interprofessional approach may help inform design practices that ensures that both software designers and healthcare providers are aware of the destigmatising design principles and understand the suitable content needed for designing sexual health-related technologies. An interprofessional approach is consistent with prior studies where computer scientists and engineers were both involved in the health systems aspects of precision medicine (Celebi et al., 2019). Further, clinical informatics professionals who have a clinical background and expertise in software development could be integral to brokering the conversation between software designers and clinicians (Sieck et al., 2020). This may possibly reduce the tendency of design teams to leave content generation exclusively in the hands of health professionals who may not know the implications of their content on technical design decisions. 8.3 Recommendations Based on the findings from the studies presented in this dissertation, there are several insights to consider in addressing stigma via digital health technologies. The following recommendations address two key areas: nursing informatics (i.e., nursing informatics practice, nursing informatics policy, and nursing informatics research), and human-computer interaction (i.e., human-computer interaction practice and research). 202 8.4 Recommendations for Nursing Informatics The findings from this dissertation provided insights that can provide direction for nursing informatics practice, nursing policy, and nursing research. The implications of the study on these three nursing areas are discussed below. 8.4.1 Recommendations for Nursing Informatics Practice The findings from the studies show that design teams should all possess at least an awareness of destigmatising design principles as well as knowledge in health technology design to be able to determine if a technology is stigmatising or not. Nurses’ awareness of the design principles produced in this research might help them to create digital health technologies that foster emotional and compassionate connection within patient groups and between patients and health professionals. Even though nurses might not be directly involved in implementing design decisions, an understanding of the principles might help nurses appreciate how sexual health-related content could be designed to address stigma. In addition to forming emotional connections, the studies offer some insights on the importance of project management skills among health professionals. Since many healthcare related technologies are owned or managed by health organizations, project management skills are needed to coordinate design activities in informatics-related projects. While some work has been done to integrate project management skills into nursing informatics competencies (Sipes, 2016), nurse informaticians should consider adding the content of destigmatising principles into existing or future competency frameworks. The integration of these principles in clinical informatics competencies is relevant as there are increasing calls for digital health technologies to incorporate nursing ethical values including compassionate care and emotional care (Wiljer et al., 2019). 203 8.4.2 Recommendations for Nursing Informatics Policy The findings across the four studies could also provide some policy directions for advancing nursing informatics policy. It is not uncommon to see nurse leaders taking part in requirements gathering for designing digital health technologies (Darvish et al., 2014). Nurse leaders should advocate for the adoption of trauma-informed user-centered design practices among design teams for patient facing software. Thus, nursing informatics leaders could be advocating for digital health technologies that thoughtfully incorporate elements of destigmatising design principles and the principles of trauma-informed care. Study 4 revealed the crucial role of a project ‘owner’ in taking important decisions to address stigma. As project management skills are a requirement for some informatics specialist positions (Davies et al., 2020), it is expected that clinical informatics specialists would take leadership positions in digital health technology projects. By this arrangement, the role of nurses in technology design might not just be limited to requirements gathering or simple recommendations but as policy advocates and policy makers in informatics related projects (Lau, 2019; Peltonen et al., 2019). Finally, this dissertation research might also provide some policy direction for nursing informatics education. Nursing education leaders can integrate human-computer interaction, including a trauma-informed user-centered design into undergraduate and graduate nursing education curricula. Incorporating trauma-informed human-computer interaction in nursing education might provide the foundational knowledge and skills that are needed for developing trauma-informed and destigmatising healthcare technologies. This knowledge, in the long run, might keep nurses abreast of the rapid changes in digital health technologies while advancing nursing practice and patient care (Booth et al., 2021). 204 8.4.3 Recommendations for Nursing Informatics Research The research also points to some important indications for nursing research involving stigmatised groups. It can be argued that the idea of user research, user testing, patient and nurse engagement in designing destigmatising digital health technologies all pointed to patient-oriented research approaches (Canadian Institute of Health Research, 2016). To address stigma, nursing researchers should ensure that user research and user testing research activities must not just involve but engage the patient as an active member of the research design and decision-making processes (Gagnon et al., 2020). However, it must be noted that active engagement of patients in user research can be a complex and challenging process because of the potential vulnerabilities of patients, which is often cited as a reason for minimal patient engagements in human-computer interaction research (Blandford, 2019; Lessard et al., 2020; Stephanidis et al., 2019). Research strategies including forming closer ties with people identifying as a member of a stigmatised group or employing patient engagement specialists might be useful to obtain the most important design criteria from the perspectives of stigmatised or vulnerable groups (Domecq et al., 2014). 8.5 Recommendation for Human-Computer Interaction Practice and Research In addition to the recommendations for nursing, the research also produced insights that could provide direction for human-computer interaction in the areas of health technology design and research. 8.5.1 Recommendations for Human-Computer Interaction Practice It was evident in this dissertation research that software designers do not have a deep understanding of stigma and a limited understanding of healthcare in general. This trend might 205 have resulted from the historic predominant focus in human-computer interaction practice on the functionality and usability of digital health technologies (De Souza Van Der Linden & Brendler, 2012) and the practice of outsourcing content generation. As human-computer interaction gradually moves towards user experience, a better understanding is needed to create non-functional, affective, and emotional design principles that addresses critical patient concerns when designing healthcare technologies. Just like Nielsen’s heuristics are used in addressing usability and functionality challenges of digital health technologies (Nielsen & Molich, 1990), the destigmatising design principles developed in this dissertation could equally be used to address stigma during system design. Thus, the principles could be used as a reference and perhaps even an evaluation guide for the design and evaluation of digital health technologies for stigmatised conditions or disorders (Masip et al., 2011). By serving as a checklist, human-computer interaction practitioners would be made aware of the potential implications of their content design approaches, even though they may not be directly involved in content generation. 8.5.2 Recommendations for Human-Computer Interaction Research The study also provided insights that suggests the need for human-computer interaction research to employ a humanist perspective to research. Humanistic human-computer interaction refers to “any HCI [human-computer interaction] research or practice that deploys humanistic epistemologies (e.g., theories and conceptual systems) and methodologies (e.g., critical analysis of design, and implementation; critical analysis) in service of HCI [human-computer interaction] processes, theories, methods, agenda-setting, and practices” (Bardzell & Bardzell, 2015, p29). That is, instead of employing purely functionalist paradigms in human-computer interaction research that emphasizes objective aspects of design (such as via a cognitive walkthrough), 206 human-computer interaction researchers should consider employing humanistic perspectives to research. Such humanistic perspectives could capture the tacit knowledge of patients and the nonobvious concerns of technology users (Herbig et al., 2001). It can therefore be argued that this dissertation research provided preliminary evidence that suggests that a trauma-informed care framework (which could be considered a humanistic approach) might also be a useful strategy for human-computer interaction researchers in addressing stigma. 8.6 Overall Limitations While each study has specific limitations, there were other limitations that cut across more than one study. First, this dissertation research was conducted during a pandemic. While the pandemic might not have influenced the methods for the Study 1 (i.e., scoping review) and Study 3 (i.e., the Delphi study), it definitely influenced the data collection methods for Study 2 and Study 4 which used online methods instead of traditional face-to-face studies. The online data collection methods might have eliminated the observational part inherent in scenario-based studies (Kushniruk & Patel, 2004), which can be a rich source of data. A second overall limitation of the dissertation is related to the use of a descriptive research designs in the three studies (Study 2, Study 3 and Study 4). Thus, patients’ views of designing destigmatising health technologies, the development of the destigmatising design principles, as well as how to apply the principles, were all based on descriptive reports of research participants. No interventional study was conducted to develop the design principles or to ascertain the true applicability of the principles by software designers. Therefore, while the principles were generally perceived as useful and applicable for user-interface design, it cannot 207 be concluded that the principles can be applied in practice situations, or if they indeed address stigma. Finally, another limitation of the dissertation is the general definition of stigma and a general definition of “sexual health-related conditions” upon which the four studies are based. Indeed, given that what is stigmatising is context-specific (Corrigan & Calabrese, 2006; Masuch et al., 2019; Yang et al., 2007), some of the design principles might not be applicable to certain digital health technologies developed for a context different from the study context. Similarly, situating the study in the broader topic of sexual health might make some of the design principles inapplicable in digital health technologies designed for specific issues of sexual health. While this was noted as a limitation, it can be argued that a broader focus is needed as a starting point for future work in designing health technologies for specific stigmatised conditions. 8.7 Future Research Directions The dissertation research shows that other digital health technologies for conditions not related to sexual health could potentially benefit from the set of destigmatising design principles (e.g., in Study 4, software designers mentioned potential relevance to mental health). Further studies are therefore needed to examine the applicability of the principles to other stigmatised conditions including mental health-related conditions. Similarly, since the research was conducted in the context of the broader topic of sexual health, future studies should work on fine-tuning the design principles and exploring their applicability to specific conditions related to other health-related conditions. Furthermore, it is important to test out the principles with patients living with sexual health-related conditions, healthcare providers as well as informatics specialists. Validating the principles with these groups might help the other design team 208 members appreciate the true importance and value of the principles in addressing stigma. Further research might also benefit from a multidisciplinary perspective that emphasizes patients’, nurses’ and other clinicians’ roles for content generation and design practices for digital health technologies. Nurses in particular are ideally positioned to identify stigmatising content and recommend strategies for mitigation because nursing is associated with essential human values including empathy, human dignity, privacy and anonymity, human relationships, and strength-based approaches (Schmidt & McArthur, 2018). These values, integrated with a trauma-informed approach to care could help in addressing stigma via digital platforms. Thus, there is the need for nurses to occupy a more significant role in the advancement of digital health technologies (Dykes & Chu, 2021; Risling & Risling, 2020). Additionally, validating the principles with clinicians and emphasizing their role in technology design might help them develop digital empathy skills when engaging in web-based communication with patients with sexual health-related conditions (Terry & Cain, 2016). 8.8 Conclusion Digital health technologies are increasingly used as complementary strategies in addressing sexual health-related stigma (Bauermeister et al., 2019; Byron et al., 2013; Noltensmeyer et al., 2016; Shearer et al., 2012). Despite the wider presence of technology interventions in the digital health arena, some of these technologies inadvertently foment and perpetuate stigma among its users. This dissertation research examined stigma in the context of sexual health-related technologies and produced a set of 19 design principles to guide software designers in developing destigmatising digital health technologies. The principles provided a useful reference guide for software designers to extend their design activities from a predominant 209 focus on the objective metrics of functional design to emotional and non-functional design. While the destigmatising design principles were largely perceived as useful and applicable to software designers, their applicability was based on descriptive reports from participants. Therefore, further studies are needed to examine the applicability of the principles in practical design situations and examine their applicability to specific conditions related to sexual health and other health conditions. 210 References Abdulai, A. F., Howard, A. F., & Currie, L. M. (2021). Stigmatizing and de-Stigmatizing Properties of Web Apps for Sexual Health-Related Conditions: A Scoping Review. International Journal of Sexual Health. https://doi.org/10.1080/19317611.2021.1949655 Abdulai, A.-F., Howard, A. F., Yong, P. J., Noga, H., Parmar, G., & Currie, L. M. (2022). Developing an Educational Website for Women with Endometriosis-Associated Dyspareunia: Usability and Stigma Analysis. JMIR Human Factors, 9(1). https://doi.org/10.2196/31317 Ahmed, I., Ahmad, N. S., Ali, S., Ali, S., George, A., Danish, H. S., Uppal, E., Soo, J., Mobasheri, M. H., King, D., Cox, B., & Darzi, A. (2018). Medication adherence apps: Review and content analysis. In Journal of Medical Internet Research. https://doi.org/10.2196/mhealth.6432 Aladeokin, A., Zavarsky, P., & Memon, N. (2017). Analysis and compliance evaluation of cookies-setting websites with privacy protection laws. 2017 12th International Conference on Digital Information Management, ICDIM 2017, 2018-January. https://doi.org/10.1109/ICDIM.2017.8244646 Alhadreti, O., & Mayhew, P. (2018). Rethinking thinking aloud: A comparison of three think-aloud protocols. Conference on Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3173574.3173618 Alqahtani, F., & Orji, R. (2019). Usability issues in mental health applications. ACM UMAP 2019 Adjunct - Adjunct Publication of the 27th Conference on User Modeling, Adaptation and Personalization. https://doi.org/10.1145/3314183.3323676 211 Anderson, N. S., Norman, D. A., & Draper, S. W. (1988). User Centered System Design: New Perspectives on Human-Computer Interaction. The American Journal of Psychology, 101(1). https://doi.org/10.2307/1422802 Apted, T., Collins, A., & Kay, J. (2009). Heuristics to Support Design of New Software for Interaction at Tabletops. Proceedings of the Workshop on Multitouch and Surface Computing. Arkell, J., Osborn, D. P. J., Ivens, D., & King, M. B. (2006). Factors associated with anxiety in patients attending a sexually transmitted infection clinic: Qualitative survey. International Journal of STD and AIDS. https://doi.org/10.1258/095646206776790097 Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology: Theory and Practice. https://doi.org/10.1080/1364557032000119616 Aronson, B. D., Janke, K. K., & Traynor, A. P. (2012). Investigating student pharmacist perceptions of professional engagement using a modified Delphi process. American Journal of Pharmaceutical Education. https://doi.org/10.5688/ajpe767125 Baker, K., Greenberg, S., & Gutwin, C. (2002). Empirical development of a heuristic evaluation methodology for shared workspace groupware. Proceedings of the ACM Conference on Computer Supported Cooperative Work. https://doi.org/10.1145/587091.587093 Bardzell, J., & Bardzell, S. (2015). Humanistic HCI. Synthesis Lectures on Human-Centered Informatics, 8(4), 1–185. https://doi.org/10.2200/S00664ED1V01Y201508HCI031 Barman-Adhikari, A., & Rice, E. (2011). Sexual Health Information Seeking Online Among Runaway and Homeless Youth. Journal of the Society for Social Work and Research. https://doi.org/10.5243/jsswr.2011.5 212 Batchelor, R., & Bobrowicz, A. (2014). Empathic and ethical design of technology. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8513 LNCS(PART 1). https://doi.org/10.1007/978-3-319-07437-5_1 Bauermeister, J. A., Muessig, K. E., LeGrand, S., Flores, D. D., Choi, S. K., Dong, W., Sallabank, G., & Hightow-Weidman, L. B. (2019). HIV and Sexuality Stigma Reduction Through Engagement in Online Forums: Results from the HealthMPowerment Intervention. AIDS and Behavior. https://doi.org/10.1007/s10461-018-2256-5 Beck, F., Richard, J. B., Nguyen-Thanh, V., Montagni, I., Parizot, I., & Renahy, E. (2014). Use of the internet as a health information resource among French young adults: Results from a nationally representative survey. Journal of Medical Internet Research. https://doi.org/10.2196/jmir.2934 Berger, M., Wagner, T. H., & Baker, L. C. (2005). Internet use and stigmatized illness. Social Science and Medicine. https://doi.org/10.1016/j.socscimed.2005.03.025 Berger, M., Wagner, T. H., & Baker, L. C. (2005b). Internet use and stigmatized illness. Social Science and Medicine. https://doi.org/10.1016/j.socscimed.2005.03.025 Blandford, A. (2019). HCI for health and wellbeing: Challenges and opportunities. International Journal of Human Computer Studies, 131. https://doi.org/10.1016/j.ijhcs.2019.06.007 Blank, D., & Kosuru, R. K. (2020). Design principles for increasing the comprehensibility of icons in hmis through motion design. Advances in Intelligent Systems and Computing. https://doi.org/10.1007/978-3-030-19135-1_44 Bonnington, O., Wamoyi, J., Ddaaki, W., Bukenya, D., Ondenge, K., Skovdal, M., Renju, J., Moshabela, M., & Wringe, A. (2017). Changing forms of HIV-related stigma along the HIV 213 care and treatment continuum in sub-Saharan Africa: A temporal analysis. Sexually Transmitted Infections. https://doi.org/10.1136/sextrans-2016-052975 Booth, R. G., Strudwick, G., McBride, S., O’Connor, S., & Solano López, A. L. (2021). How the nursing profession should adapt for a digital future. The BMJ, 373. https://doi.org/10.1136/bmj.n1190 Borcsa, M., Pomini, V., & Saint-Mont, U. (2021). Digital systemic practices in Europe: a survey before the Covid-19 pandemic. Journal of Family Therapy, 43(1). https://doi.org/10.1111/1467-6427.12308 Borsci, S., Federici, S., Mele, M. L., & Conti, M. (2015). Short scales of satisfaction assessment: A proxy to involve disabled users in the usability testing of websites. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). https://doi.org/10.1007/978-3-319-21006-3_4 Bos, A. E. R., Pryor, J. B., Reeder, G. D., & Stutterheim, S. E. (2013). Stigma: Advances in Theory and Research. Basic and Applied Social Psychology. https://doi.org/10.1080/01973533.2012.746147 Braun, V., & Clarke, V. (2006). Qualitative Research in Psychology Using thematic analysis in psychology Using thematic analysis in psychology. Qualitative Research in Psychology. Breitkopf, C. R. (2004). The Theoretical Basis of Stigma as Applied to Genital Herpes. In Herpes. Brewer, L. P. C., Fortuna, K. L., Jones, C., Walker, R., Hayes, S. N., Patten, C. A., & Cooper, L. A. (2020). Back to the future: Achieving health equity through health informatics and digital health. JMIR MHealth and UHealth, 8(1). https://doi.org/10.2196/14512 214 Bright, A. K., & Coventry, L. (2013). Assistive technology for older adults: Psychological and socio-emotional design requirements. ACM International Conference Proceeding Series. https://doi.org/10.1145/2504335.2504344 Bright, T. J., Wong, A., Dhurjati, R., Bristow, E., Bastian, L., Coeytaux, R. R., Samsa, G., Hasselblad, V., Williams, J. W., Musty, M. D., Wing, L., Kendrick, A. S., Sanders, G. D., & Lobach, D. (2012). Effect of clinical decision-support systems: A systematic review. In Annals of Internal Medicine. https://doi.org/10.7326/0003-4819-157-1-201207030-00450 Broaddus, M. R., Marsch, L. A., & Fisher, C. B. (2015). Risks and Benefits of Text-Message-Delivered and Small-Group-Delivered Sexual Health Interventions Among African American Women in the Midwestern United States. Ethics and Behavior. https://doi.org/10.1080/10508422.2014.948957 Brown, L., Macintyre, K., & Trujillo, L. (2003). Interventions to reduce HIV/AIDS stigma: What have we learned? AIDS Education and Prevention, 15(1), 49–69. https://doi.org/10.1521/aeap.15.1.49.23844 Burnham, K. E., Cruess, D. G., Kalichman, M. O., Grebler, T., Cherry, C., & Kalichman, S. C. (2016). Trauma symptoms, internalized stigma, social support, and sexual risk behavior among HIV-positive gay and bisexual MSM who have sought sex partners online. AIDS Care - Psychological and Socio-Medical Aspects of AIDS/HIV. https://doi.org/10.1080/09540121.2015.1096894 Byron, P., Albury, K., & Evers, C. (2013). “It would be weird to have that on facebook”: Young people’s use of social media and the risk of sharing sexual health information. Reproductive Health Matters. https://doi.org/10.1016/S0968-8080(13)41686-5 215 Canadian Institute of Health Research (CIHR), (2016). Strategy for Patient-Oriented Research - CIHR. In Canadian Institutes of Health Research. Canadian Mental Health Association. (2016). https://cmha.calgary.ab.ca/mental-health/your-mental-health/what-is-stigma/ Canadian Mental Health Association. (2020). Stigma and Discrimination. https://ontario.cmha.ca/documents/stigma-and-discrimination/ Carmody, K. W. (2013). Exploring serious game design heuristics: A delphi study. Dissertation Abstracts International Section A: Humanities and Social Sciences. Carter, J. M., & Wilson, F. L. (2015). Cyberbullying: A 21st Century Health Care Phenomenon. Pediatric Nursing, 41(3). Caruso, S., Iraci, M., Cianci, S., Casella, E., Fava, V., & Cianci, A. (2015). Quality of life and sexual function of women affected by endometriosis-associated pelvic pain when treated with dienogest. Journal of Endocrinological Investigation. https://doi.org/10.1007/s40618-015-0383-7 Celebi, N., Griewatz, J., Malek, N. P., Krieg, S., Kuehnl, T., Muller, R., Pauluschke-Fröhlich, J., Debove, I., Riessen, R., Zipfel, S., & Fröhlich, E. (2019). Development and implementation of a comprehensive ultrasound curriculum for undergraduate medical students - A feasibility study. BMC Medical Education, 19(1). https://doi.org/10.1186/s12909-019-1611-1 Center for Addiction and Mental Health. (2020). Stigma and prejudice | CAMH. https://www.camh.ca/en/health-info/mental-health-and-covid-19/stigma-and-prejudice Chalmers, J., & Armour, M. (2019). The delphi technique. In Handbook of Research Methods in Health Social Sciences. https://doi.org/10.1007/978-981-10-5251-4_99 216 Chanco, C., Moquillaza, A., & Paz, F. (2019). Development and Validation of Usability Heuristics for Evaluation of Interfaces in ATMs. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). https://doi.org/10.1007/978-3-030-23535-2_1 Chang, P. F., & Bazarova, N. N. (2016). Managing Stigma: Disclosure-Response Communication Patterns in Pro-Anorexic Websites. Health Communication. https://doi.org/10.1080/10410236.2014.946218 Chaudoir, S. R., Earnshaw, V. A., & Andel, S. (2013). “Discredited” Versus “Discreditable”: Understanding How Shared and Unique Stigma Mechanisms Affect Psychological and Physical Health Disparities. Basic and Applied Social Psychology. https://doi.org/10.1080/01973533.2012.746612 Chaudoir, S. R., Norton, W. E., Earnshaw, V. A., Moneyham, L., Mugavero, M. J., & Hiers, K. M. (2012). Coping with HIV stigma: Do proactive coping and spiritual peace buffer the effect of stigma on depression? AIDS and Behavior. https://doi.org/10.1007/s10461-011-0039-3 Chen, L., Guo, Y., & Shi, J. (2019). Social Support Seeking on Social Media among Chinese Gay Men Living with HIV/AIDS: The Role of Perceived Threat. Telemedicine and E-Health. https://doi.org/10.1089/tmj.2018.0136 Chen, Y. (2019). Exploring Design Guidelines of Using User-Centered Design in Gamification Development: A Delphi Study. International Journal of Human-Computer Interaction. https://doi.org/10.1080/10447318.2018.1514823 Choi, M., Kim, M., Kim, J. A., & Chang, H. (2020). Building Consensus on the Priority-Setting for National Policies in Health Information Technology : A Delphi Survey. 26(3), 229–237. 217 Christensen, J. L., Miller, L. C., Appleby, P. R., Corsbie-Massay, C., Godoy, C. G., Marsella, S. C., & Read, S. J. (2013). Reducing shame in a game that predicts HIV risk reduction for young adult MSM: a randomized trial delivered nationally over the Web. Journal of the International AIDS Society. CIHR. (2018). Knowledge translation - Canadian Istitute of Health Research. http://www.cihr-irsc.gc.ca/e/29529.html Cisu, T. I., Mingin, G. C., & Baskin, L. S. (2019). An evaluation of the readability, quality, and accuracy of online health information regarding the treatment of hypospadias. Journal of Pediatric Urology, 15(1). https://doi.org/10.1016/j.jpurol.2018.08.020 Collins, P. Y., von Unger, H., & Armbrister, A. (2008). Church ladies, good girls, and locas: Stigma and the intersection of gender, ethnicity, mental illness, and sexuality in relation to HIV risk. Social Science and Medicine. https://doi.org/10.1016/j.socscimed.2008.03.013 Cornet, V. P., Toscos, T., Bolchini, D., Ghahari, R. R., Ahmed, R., Daley, C., Mirro, M. J., & Holden, R. J. (2020). Untold stories in user-centered design of mobile health: Practical challenges and strategies learned from the design and evaluation of an app for older adults with heart failure. JMIR MHealth and UHealth, 8(7). https://doi.org/10.2196/17703 Corrigan, P. W., & Calabrese, J. D. (2006). Strategies for Assessing and Diminishing Self-Stigma. In On the stigma of mental illness: Practical strategies for research and social change. https://doi.org/10.1037/10887-011 Corrigan, P. W., & Kosyluk, K. A. (2013). Erasing the Stigma: Where Science Meets Advocacy. In Basic and Applied Social Psychology. https://doi.org/10.1080/01973533.2012.746598 218 Corrigan, P. W., & Lam, C. (2013). Challenging the Structural Discrimination of Psychiatric Disabilities: Lessons Learned from the American Disability Community. Rehabilitation Education. https://doi.org/10.1891/088970107805059869 Crabtree, J. W., Haslam, S. A., Postmes, T., & Haslam, C. (2010). Mental Health Support Groups, Stigma, and Self-Esteem: Positive and Negative Implications of Group Identification. Journal of Social Issues. https://doi.org/10.1111/j.1540-4560.2010.01662.x Cummins, N., & Schuller, B. W. (2020). Five Crucial Challenges in Digital Health. Frontiers in Digital Health, 2. https://doi.org/10.3389/fdgth.2020.536203 Dajani, J. S., Sincoff, M. Z., & Talley, W. K. (1979). Stability and agreement criteria for the termination of Delphi studies. Technological Forecasting and Social Change. https://doi.org/10.1016/0040-1625(79)90007-6 Dalaba, M. A., Akweongo, P., Williams, J., Saronga, H. P., Tonchev, P., Sauerborn, R., Mensah, N., Blank, A., Kaltschmidt, J., & Loukanova, S. (2014). Costs associated with implementation of computer-assisted clinical decision support system for antenatal and delivery care: Case study of Kassena-Nankana district of northern Ghana. PLoS ONE. https://doi.org/10.1371/journal.pone.0106416 Darvish, A., Bahramnezhad, F., Keyhanian, S., & Navidhamidi, M. (2014). The role of nursing informatics on promoting quality of health care and the need for appropriate education. In Global journal of health science (Vol. 6, Issue 6). https://doi.org/10.5539/gjhs.v6n6p11 Davidson, A. E., & Wallack, L. (2004). A content analysis of sexually transmitted diseases in the print news media. In Journal of Health Communication. https://doi.org/10.1080/10810730490425268 219 Davies, A., Mueller, J., & Moulton, G. (2020). Core competencies for clinical informaticians: A systematic review. In International Journal of Medical Informatics (Vol. 141). https://doi.org/10.1016/j.ijmedinf.2020.104237 Davis, A. C., Carrotte, E. R., Hellard, M. E., & Lim, M. S. C. (2018). What Behaviors Do Young Heterosexual Australians See in Pornography? A Cross-Sectional Study. Journal of Sex Research. https://doi.org/10.1080/00224499.2017.1417350 Davis, A. C., Wright, C. J. C., Temple-Smith, M. J., Hellard, M. E., & Lim, M. S. C. (2019). Assessing the usability of SCOPE: A health education website developed to meet young people’s information needs about web-based pornography and sharing of sexually explicit imagery. Journal of Medical Internet Research. https://doi.org/10.2196/12824 Davis, W. M., Shoveller, J. A., Oliffe, J. L., & Gilbert, M. (2012). Young people’s perspectives on the use of reverse discourse in web-based sexual-health interventions. Culture, Health and Sexuality. https://doi.org/10.1080/13691058.2012.714800 De Souza Van Der Linden, J. C., & Brendler, C. F. (2012). The hierarchy of needs to inclusive design. Work. https://doi.org/10.3233/WOR-2012-0324-1357 De Visser, R. O., & O’Neill, N. (2013). Identifying and understanding barriers to sexually transmissible infection testing among young people. Sexual Health. https://doi.org/10.1071/SH13034 Deitz, M. F., Williams, S. L., Rife, S. C., & Cantrell, P. (2015). Examining Cultural, Social, and Self-Related Aspects of Stigma in Relation to Sexual Assault and Trauma Symptoms. Violence Against Women. https://doi.org/10.1177/1077801215573330 Denny, E. (2004). Women’s experience of endometriosis. In Journal of Advanced Nursing. https://doi.org/10.1111/j.1365-2648.2004.03055.x 220 Design Council |. (2020). The Design Council. https://www.designcouncil.org.uk/ di Donato, N., Montanari, G., Benfenati, A., Monti, G., Bertoldo, V., Mauloni, M., & Seracchioli, R. (2014). Do women with endometriosis have to worry about sex? European Journal of Obstetrics and Gynecology and Reproductive Biology. https://doi.org/10.1016/j.ejogrb.2014.05.022 Diamond, I. R., Grant, R. C., Feldman, B. M., Pencharz, P. B., Ling, S. C., Moore, A. M., & Wales, P. W. (2014). Defining consensus: A systematic review recommends methodologic criteria for reporting of Delphi studies. Journal of Clinical Epidemiology. https://doi.org/10.1016/j.jclinepi.2013.12.002 Domecq, J. P., Prutsky, G., Elraiyah, T., Wang, Z., Nabhan, M., Shippee, N., Brito, J. P., Boehmer, K., Hasan, R., Firwana, B., Erwin, P., Eton, D., Sloan, J., Montori, V., Asi, N., Dabrh, A. M. A., & Murad, M. H. (2014). Patient engagement in research: A systematic review. In BMC Health Services Research (Vol. 14). https://doi.org/10.1186/1472-6963-14-89 Douglass, B. G., & Moustakas, C. (1985). Heuristic inquiry: The Internal Search to Know. Journal of Humanistic Psychology. https://doi.org/10.1177/0022167885253004 Doyle, L., McCabe, C., Keogh, B., Brady, A., & McCann, M. (2020). An overview of the qualitative descriptive design within nursing research. Journal of Research in Nursing, 25(5). https://doi.org/10.1177/1744987119880234 Dudina, V., & Tsareva, A. (2018). Studying Stigmatization and Status Disclosure Among People Living with HIV/AIDS in Russia Through Online Health Communities | SpringerLink. 221 Dykes, S., & Chu, C. H. (2021). Now more than ever, nurses need to be involved in technology design: lessons from the COVID-19 pandemic. In Journal of Clinical Nursing (Vol. 30, Issues 7–8). https://doi.org/10.1111/jocn.15581 Earnshaw, V. A., Quinn, D. M., & Park, C. L. (2012). Anticipated stigma and quality of life among people living with chronic illnesses. Chronic Illness. https://doi.org/10.1177/1742395311429393 Earnshaw, V. A., Reisner, S. L., Menino, D. D., Poteat, V. P., Bogart, L. M., Barnes, T. N., & Schuster, M. A. (2018). Stigma-based bullying interventions: A systematic review. In Developmental Review. https://doi.org/10.1016/j.dr.2018.02.001 Elo, S., Kääriäinen, M., Kanste, O., Pölkki, T., Utriainen, K., & Kyngäs, H. (2014). Qualitative Content Analysis: A Focus on Trustworthiness. SAGE Open. https://doi.org/10.1177/2158244014522633 Elo, S., & Kyngäs, H. (2008). The qualitative content analysis process. Journal of Advanced Nursing. https://doi.org/10.1111/j.1365-2648.2007.04569.x Ericsson, K. (2009). Development of professional expertise [electronic resource] : toward measurement of expert performance and design of optimal learning environments. In Cambridge books online. Facchin, F., Barbara, G., Saita, E., Mosconi, P., Roberto, A., Fedele, L., & Vercellini, P. (2015). Impact of endometriosis on quality of life and mental health: Pelvic pain makes the difference. Journal of Psychosomatic Obstetrics and Gynecology. https://doi.org/10.3109/0167482X.2015.1074173 Fasciano, K. M., Souza, P. M., Braun, I., & Trevino, K. (2015). An innovative website in the United States for meeting the emotional and supportive care needs of young adults with 222 cancer. Journal of Adolescent and Young Adult Oncology, 4(1). https://doi.org/10.1089/jayao.2014.0035 Fatehi, F., Samadbeik, M., & Kazemi, A. (2020). What is digital health? review of definitions. Studies in Health Technology and Informatics, 275. https://doi.org/10.3233/SHTI200696 Ferrero, S., Esposito, F., Abbamonte, L. H., Anserini, P., Remorgida, V., & Ragni, N. (2005). Quality of sex life in women with endometriosis and deep dyspareunia. Fertility and Sterility. https://doi.org/10.1016/j.fertnstert.2004.07.973 Ferron, J. C., Brunette, M. F., Geiger, P., Marsch, L. A., Adachi-Mejia, A. M., & Bartels, S. J. (2017). Mobile Phone Apps for Smoking Cessation: Quality and Usability Among Smokers With Psychosis. JMIR Human Factors. https://doi.org/10.2196/humanfactors.5933 Field, T. S., Rochon, P., Lee, M., Gavendo, L., Subramanian, S., Hoover, S., Baril, J., & Gurwitz, J. (2008). Costs Associated with Developing and Implementing a Computerized Clinical Decision Support System for Medication Dosing for Patients with Renal Insufficiency in the Long-term Care Setting. Journal of the American Medical Informatics Association. https://doi.org/10.1197/jamia.M2589 Flanders, C. E., Anderson, R. A. E., Tarasoff, L. A., & Robinson, M. (2019). Bisexual Stigma, Sexual Violence, and Sexual Health Among Bisexual and Other Plurisexual Women: A Cross-Sectional Survey Study. Journal of Sex Research. https://doi.org/10.1080/00224499.2018.1563042 Flickinger, T. E., DeBolt, C., Xie, A., Kosmacki, A., Grabowski, M., Waldman, A. L., Reynolds, G., Conaway, M., Cohn, W. F., Ingersoll, K., & Dillingham, R. (2018). Addressing Stigma Through a Virtual Community for People Living with HIV: A Mixed Methods Study of the 223 PositiveLinks Mobile Health Intervention. AIDS and Behavior. https://doi.org/10.1007/s10461-018-2174-6 Friedman, A. L., Bozniak, A., Ford, J., Hill, A., Olson, K., Ledsky, R., Inokuchi, D., & Brookmeyer, K. (2014). Reaching youth with sexually transmitted disease testing: Building on successes, challenges, and lessons learned from local get yourself tested campaigns. Social Marketing Quarterly. https://doi.org/10.1177/1524500414530386 Frieh, E. C. (2020). Stigma, trauma and sexuality: the experiences of women hospitalised with serious mental illness. Sociology of Health and Illness. https://doi.org/10.1111/1467-9566.13034 Frueh, B. C., Knapp, R. G., Cusack, K. J., Grubaugh, A. L., Sauvageot, J. A., Cousins, V. C., Yim, E., Robins, C. S., Monnier, J., & Hiers, T. G. (2005). Special Section on Seclusion and Restraint: Patients’ Reports of Traumatic or Harmful Experiences Within the Psychiatric Setting. Psychiatric Services. https://doi.org/10.1176/appi.ps.56.9.1123 Fuster-Ruizdeapodaca, M. J., Molero, F., Holgado, F. P. ablo, & Mayordomo, S. (2014). Enacted and internalized stigma and quality of life among people with HIV: the role of group identity. Quality of Life Research : An International Journal of Quality of Life Aspects of Treatment, Care and Rehabilitation. https://doi.org/10.1007/s11136-014-0653-4 Gagnon, M. P., Ndiaye, M. A., Chabot, C., Buyl, R., Fortin, J. P., Giguère, A., Leblanc, A., Légaré, F., Motulsky, A., Sicotte, C., Witteman, H. O., Kavanagh, É., Lépinay, F., Roberge, J., Hakim, H., Brunet-Gauthier, M., Délétroz, C., Rahimi, S. A., Tchuente, J., & Sasseville, M. (2020). User-centered design for promoting patient engagement in chronic diseases management: The development of CONCERTO+. Studies in Health Technology and Informatics, 270. https://doi.org/10.3233/SHTI200473 224 Gandomani, T. J., Tavakoli, Z., Zulzalil, H., & Farsani, H. K. (2020). The Role of Project Manager in Agile Software Teams: A Systematic Literature Review. In IEEE Access (Vol. 8). https://doi.org/10.1109/ACCESS.2020.3004450 Gannon, B., Davis, R., Kuhns, L. M., Rodriguez, R. G., Garofalo, R., & Schnall, R. (2020). A mobile sexual health app on empowerment, education, and prevention for young adult men (MyPEEPS mobile): Acceptability and usability evaluation. Journal of Medical Internet Research. https://doi.org/10.2196/17901 Gargon, E., Crew, R., Burnside, G., & Williamson, P. R. (2019). Higher number of items associated with significantly lower response rates in COS Delphi surveys. Journal of Clinical Epidemiology, 108. https://doi.org/10.1016/j.jclinepi.2018.12.010 Génova, G., González, M. R., & Fraga, A. (2007). Ethical education in software engineering: Responsibility in the production of complex systems. Science and Engineering Ethics, 13(4). https://doi.org/10.1007/s11948-007-9017-6 Geuens, J., Swinnen, T. W., Westhovens, R., de Vlam, K., Geurts, L., & vanden Abeele, V. (2016). A Review of Persuasive Principles in Mobile Apps for Chronic Arthritis Patients: Opportunities for Improvement. JMIR MHealth and UHealth. https://doi.org/10.2196/mhealth.6286 Giudice, L. C. (2010). Clinical practice. Endometriosis. The New England Journal of Medicine. https://doi.org/10.1056/NEJMcp1000274 Gkatzidou, V., Hone, K., Sutcliffe, L., Gibbs, J., Sadiq, S. T., Szczepura, A., Sonnenberg, P., & Estcourt, C. (2015). User interface design for mobile-based sexual health interventions for young people: Design recommendations from a qualitative study on an online Chlamydia 225 clinical care pathway. BMC Medical Informatics and Decision Making. https://doi.org/10.1186/s12911-015-0197-8 Glasgow, R. E., Phillips, S. M., & Sanchez, M. A. (2014). Implementation science approaches for integrating eHealth research into practice and policy. International Journal of Medical Informatics. https://doi.org/10.1016/j.ijmedinf.2013.07.002 Goffman, E. (1963). Ch. 1 Stigma and Social Identity. In Stigma: notes on the management of spoiled identity. https://doi.org/10.4319/lo.2013.58.2.0489 Goffman, E. (1974). Stigma; Notes on the management of spoiled identity. JASON ARONSON, NEW YORK, N.Y. https://doi.org/10.2307/2575995 Goodman, C. M. (1987). The Delphi technique: a critique. Journal of Advanced Nursing. https://doi.org/10.1111/j.1365-2648.1987.tb01376.x Gourlay, A., Mshana, G., Birdthistle, I., Bulugu, G., Zaba, B., & Urassa, M. (2014). Using vignettes in qualitative research to explore barriers and facilitating factors to the uptake of prevention of mother-to-child transmission services in rural Tanzania: A critical analysis. BMC Medical Research Methodology. https://doi.org/10.1186/1471-2288-14-21 Granja, C., Janssen, W., & Johansen, M. A. (2018). Factors determining the success and failure of ehealth interventions: Systematic review of the literature. In Journal of Medical Internet Research. https://doi.org/10.2196/10235 Guest, G., MacQueen, K., & Namey, E. (2012). Applied Thematic Analysis. https://doi.org/10.4135/9781483384436 Guse, K., Levine, D., Martins, S., Lira, A., Gaarde, J., Westmorland, W., & Gilliam, M. (2012). Interventions using new digital media to improve adolescent sexual health: A systematic review. In Journal of Adolescent Health. https://doi.org/10.1016/j.jadohealth.2012.03.014 226 Hall, D. A., Smith, H., Heffernan, E., & Fackrell, K. (2018). Recruiting and retaining participants in e-Delphi surveys for core outcome set development: Evaluating the COMiT’ID study. PLoS ONE. https://doi.org/10.1371/journal.pone.0201378 Hamill, M., & Goldmeier, D. (2005). Patients support innovative models of care in recurrent genital herpes clinics: Results from a patient satisfaction study. International Journal of STD and AIDS. https://doi.org/10.1258/0956462054944499 Hanafin, S. (2004). Review of literature on the Delphi Technique. Dublin: National Children’s Office. https://doi.org/10.1177/0013164404266386 Hannigan, S., & Jones-Devitt, S. (2004). Removing the stigma of sexually transmitted infections among women. In Nursing times. Harris, M., & Fallot, R. D. (2001). Envisioning a trauma-informed service system: a vital paradigm shift. New Directions for Mental Health Services. https://doi.org/10.1002/yd.23320018903 Hassenzahl, M. (2005). Hedonic, emotional, and experiential perspectives on product quality. In Encyclopedia of Human Computer Interaction. https://doi.org/10.4018/978-1-59140-562-7.ch042 Hasson, F., & Keeney, S. (2011). Enhancing rigour in the Delphi technique research. Technological Forecasting and Social Change. https://doi.org/10.1016/j.techfore.2011.04.005 Hasson, F., Keeney, S., & McKenna, H. (2000). Research guidelines for the Delphi survey technique. Journal of Advanced Nursing. https://doi.org/10.1046/j.1365-2648.2000.t01-1-01567.x 227 Heijnders, M., & van der Meij, S. (2006). The fight against stigma: an overview of stigma-reduction strategies and interventions. In Psychology, health & medicine. https://doi.org/10.1080/13548500600595327 Hejblum, G., Ioos, V., Vibert, J. F., Böelle, P. Y., Chalumeau-Lemoine, L., Chouaid, C., Valleron, A. J., & Guidet, B. (2008). A web-based delphi study on the indications of chest radiographs for patients in ICUs. Chest. https://doi.org/10.1378/chest.06-3014 Herbig, B., Büssing, A., & Ewert, T. (2001). The role of tacit knowledge in the work context of nursing. Journal of Advanced Nursing. https://doi.org/10.1046/j.1365-2648.2001.01798.x Herek, G. M. (2002). Thinking about AIDS and stigma: A psychologist’s perspective. Journal of Law, Medicine and Ethics. https://doi.org/10.1111/j.1748-720X.2002.tb00428.x Herek, G. M. (2007). Confronting sexual stigma and prejudice: Theory and practice. Journal of Social Issues. https://doi.org/10.1111/j.1540-4560.2007.00544.x Herek, G. M., & McLemore, K. A. (2013). Sexual Prejudice. Annual Review of Psychology. https://doi.org/10.1146/annurev-psych-113011-143826 Hesketh, K. R., Lakshman, R., & van Sluijs, E. M. F. (2017). Barriers and facilitators to young children’s physical activity and sedentary behaviour: a systematic review and synthesis of qualitative literature. In Obesity Reviews. https://doi.org/10.1111/obr.12562 Hesse, B. W., Moser, R. P., & Rutten, L. J. (2010). Surveys of physicians and electronic health information. In New England Journal of Medicine. https://doi.org/10.1056/NEJMc0909595 Hightow-Weidman, L. B., Muessig, K. E., Pike, E. C., LeGrand, S., Baltierra, N., Rucker, A. J., & Wilson, P. (2015). HealthMpowerment.org: Building Community Through a Mobile-Optimized, Online Health Promotion Intervention. Health Education and Behavior. https://doi.org/10.1177/1090198114562043 228 Hirsch, M., Aggarwal, S., Barker, C., Davis, C. J., & Duffy, J. M. N. (2017). Googling endometriosis: a systematic review of information available on the Internet. In American Journal of Obstetrics and Gynecology. https://doi.org/10.1016/j.ajog.2016.11.1007 Hirschhorn, F. (2019). Reflections on the application of the Delphi method: lessons from a case in public transport research. International Journal of Social Research Methodology. https://doi.org/10.1080/13645579.2018.1543841 Ho, C. L. L., Pan, W., & Taylor, L. D. (2017). Stigma of HIV testing on online HIV forums: Self-stigma and the unspoken. Journal of Psychosocial Nursing and Mental Health Services. https://doi.org/10.3928/02793695-20170905-01 Hood, J. E., & Friedman, A. L. (2011). Unveiling the hidden epidemic: A review of stigma associated with sexually transmissible infections. In Sexual Health. https://doi.org/10.1071/SH10070 Hopper, E. K., Bassuk, E. L., & Olivet, J. (2009). Shelter from the Storm: Trauma-Informed Care in Homelessness Services Settings. The Open Health Services and Policy Journal. Hsu, C. C., & Sandford, B. A. (2007). The Delphi technique: Making sense of consensus. Practical Assessment, Research and Evaluation. Hulshof, A., Pemberton, L., & Griffiths, R. (2013). Design principles for preschool children’s interactive TV. Universal Access in the Information Society, 12(1). https://doi.org/10.1007/s10209-011-0263-7 Humphrey-Murto, S., Varpio, L., Gonsalves, C., & Wood, T. J. (2017). Using consensus group methods such as Delphi and Nominal Group in medical education research*. Medical Teacher. https://doi.org/10.1080/0142159X.2017.1245856 229 Hwang, W., & Salvendy, G. (2010). Number of people required for usability evaluation: The 10±2 rule. Communications of the ACM. https://doi.org/10.1145/1735223.1735255 Interaction Design Foundation. (2020). What are Design Principles? | Interaction Design Foundation. https://www.interaction-design.org/literature/topics/design-principles Israel, T., Choi, A. Y., Goodman, J. A., Matsuno, E., Lin, Y. J., Kary, K. G., & Merrill, C. R. S. (2019). Reducing internalized binegativity: Development and efficacy of an online intervention. Psychology of Sexual Orientation and Gender Diversity. https://doi.org/10.1037/sgd0000314 Jaspers, M. W. M. (2009). A comparison of usability methods for testing interactive health technologies: Methodological aspects and empirical evidence. International Journal of Medical Informatics. https://doi.org/10.1016/j.ijmedinf.2008.10.002 Jaspers, M. W. M., Smeulers, M., Vermeulen, H., & Peute, L. W. (2011). Effects of clinical decision-support systems on practitioner performance and patient outcomes: A synthesis of high-quality systematic review findings. Journal of the American Medical Informatics Association. https://doi.org/10.1136/amiajnl-2011-000094 Ji-Ye Mao, Vrendenburg, K., Smith, P. W., & Carey, T. (2008). The state of user-centered design practice. IEEE Engineering Management Review, 33(2). https://doi.org/10.1109/emr.2005.26748 Johnson, E., & Johnson C, S. (2016). Internet Use and Access to Health Information among Canadians: Are the Elderly on the Sidelines? Journal of Gerontology & Geriatric Research, 05(06). https://doi.org/10.4172/2167-7182.1000367 230 Johnstone, C. J., Bottsford-Miller, N. A., & Thompson, S. J. (2006). Using the think aloud method (cognitive labs) to evaluate test design for students with disabilities and english language learners (NCEO Technical Report). National Center on Educational Outcomes. Jones, R., & Lacroix, L. J. (2012). Streaming weekly soap opera video episodes to smartphones in a randomized controlled trial to reduce HIV risk in young urban african american/black women. AIDS and Behavior. https://doi.org/10.1007/s10461-012-0170-9 Jordan, P. W. (2000). Inclusive design: An holistic approach. Proceedings of the XIVth Triennial Congress of the International Ergonomics Association and 44th Annual Meeting of the Human Factors and Ergonomics Association, “Ergonomics for the New Millennium.” https://doi.org/10.1177/154193120004403865 Jordan, P. W. (2002). Designing pleasurable products- An induction to new human factors.pdf. In Design Issues (Vol. 18, Issue 1). Kang, H., & An, S. (2013). How direct-to-consumer drug websites convey disease information: Analysis of stigma-reducing components. Journal of Health Communication. https://doi.org/10.1080/10810730.2013.798378 Kaplan, B., & Harris-Salamone, K. D. (2009). Health IT Success and Failure: Recommendations from Literature and an AMIA Workshop. Journal of the American Medical Informatics Association. https://doi.org/10.1197/jamia.M2997 Karamouzian, M., Knight, R., Davis, W. M., Gilbert, M., & Shoveller, J. (2018). Stigma associated with sexually transmissible infection testing in an online testing environment: Examining the perspectives of youth in Vancouver, Canada. Sexual Health. https://doi.org/10.1071/SH17089 231 Keeney, S., Hasson, F., & McKenna, H. P. (2001). A critical review of the Delphi technique as a research methodology for nursing. International Journal of Nursing Studies. https://doi.org/10.1016/S0020-7489(00)00044-4 Kerr, J. C., Valois, R. F., Diclemente, R. J., Carey, M. P., Stanton, B., Romer, D., Fletcher, F., Farber, N., Brown, L. K., Vanable, P. A., Salazar, L. F., Juzang, I., & Fortune, T. (2015). The effects of a mass media HIV-risk reduction strategy on HIV-related stigma and knowledge among African American adolescents. AIDS Patient Care and STDs. https://doi.org/10.1089/apc.2014.0207 Kientz, J. A., Choe, E. K., Birch, B., Maharaj, R., Fonville, A., Glasson, C., & Mundt, J. (2010). Heuristic evaluation of persuasive health technologies. IHI’10 - Proceedings of the 1st ACM International Health Informatics Symposium. https://doi.org/10.1145/1882992.1883084 Killip, S. C., Kwong, N. K. R., Macdermid, J. C., Fletcher, A. J., & Carleton, N. R. (2020). The quality, readability, completeness, and accuracy of ptsd websites for firefighters. International Journal of Environmental Research and Public Health, 17(20). https://doi.org/10.3390/ijerph17207629 Kim, H. N. (2019). A conceptual framework for interdisciplinary education in engineering and nursing health informatics. In Nurse Education Today (Vol. 74). https://doi.org/10.1016/j.nedt.2018.12.010 Kim, H., Sefcik, J. S., & Bradway, C. (2017). Characteristics of Qualitative Descriptive Studies: A Systematic Review. Research in Nursing and Health, 40(1). https://doi.org/10.1002/nur.21768 Kim, J., Gambino, A., Sundar, S. S., Rosson, M. B., Aritajati, C., Ge, J., & Fanning, C. (2018). Interface cues to promote disclosure and build community: An experimental test of crowd 232 and connectivity cues in an online sexual health forum. Proceedings of the ACM on Human-Computer Interaction. https://doi.org/10.1145/3274359 Klein, C., & Lomonaco, C. (2016). Real talk: Developing a computerdelivered sexual health program for black men who have sex with men. AIDS Education and Prevention. https://doi.org/10.1521/aeap.2016.28.6.455 Knight, R., Shoveller, J. A., Oliffe, J. L., Gilbert, M., & Goldenberg, S. (2013). Heteronormativity hurts everyone: Experiences of young men and clinicians with sexually transmitted infection/HIV testing in British Columbia, Canada. Health (United Kingdom). https://doi.org/10.1177/1363459312464071 Ko, S. M., Chang, W. S., & Ji, Y. G. (2013). Usability Principles for Augmented Reality Applications in a Smartphone Environment. International Journal of Human-Computer Interaction. https://doi.org/10.1080/10447318.2012.722466 Kondrat, D. C., & Teater, B. (2009). An anti-stigma approach to working with persons with severe mental disability: Seeking real change through narrative change. Journal of Social Work Practice. https://doi.org/10.1080/02650530902723308 Krefting, L. (1991). Rigor in qualitative research: the assessment of trustworthiness. In The American journal of occupational therapy. : official publication of the American Occupational Therapy Association. https://doi.org/10.5014/ajot.45.3.214 Kushniruk, A., & Nøhr, C. (2016). Participatory design, user involvement and health IT evaluation. In Evidence-Based Health Informatics: Promoting Safety and Efficiency through Scientific Methods and Ethical Policy (Vol. 222). https://doi.org/10.3233/978-1-61499-635-4-139 233 Kushniruk, A. W., & Patel, V. L. (2004). Cognitive and usability engineering methods for the evaluation of clinical information systems. In Journal of Biomedical Informatics. https://doi.org/10.1016/j.jbi.2004.01.003 Kwakernaak, J., Eekhof, J. A. H., De Waal, M. W. M., Barenbrug, E. A. M., & Chavannes, N. H. (2019). Patients’ use of the internet to find reliable medical information about minor ailments: Vignette-based experimental study. Journal of Medical Internet Research. https://doi.org/10.2196/12278 Lagu, T., Norton, C. M., Russo, L. M., Priya, A., Goff, S. L., & Lindenauer, P. K. (2019). Reporting of patient experience data on health systems’ websites and commercial physician-rating websites: Mixed-methods analysis. Journal of Medical Internet Research. https://doi.org/10.2196/12007 Lau, F. (2019). Improving usability, safety and patient outcomes with health information technology : from research to practice. Layland, E. K., Carter, J. A., Perry, N. S., Cienfuegos-Szalay, J., Nelson, K. M., Bonner, C. P., & Rendina, J. (2020). A systematic review of stigma in sexual and gender minority health interventions. In Translational Behavioral Medicine (Vol. 10, Issue 5). https://doi.org/10.1093/tbm/ibz200 Lee, J. D., & Craft, E. A. (2002). Protecting one’s self from a stigmatized disease... once one has it. Deviant Behavior. https://doi.org/10.1080/016396202753561248 Lessard, D., Engler, K., Vicente, S., Bilodeau, M., & Lebouché, B. (2020). Challenges of Patient Engagement in an HIV Clinical Research Program: A Qualitative Analysis of Stakeholder Accounts. Journal of Patient Experience, 7(6). https://doi.org/10.1177/2374373520975728 234 Lewis, J. R. (1995). IBM Computer Usability Satisfaction Questionnaires: Psychometric Evaluation and Instructions for Use. International Journal of Human-Computer Interaction, 7(1), 57–78. https://doi.org/10.1080/10447319509526110 Leyland, N., Casper, R., Laberge, P., Singh, S. S., Allen, L., Arendas, K., Allaire, C., Awadalla, A., Best, C., Contestabile, E., Dunn, S., Heywood, M., Leroux, N., Potestio, F., Rittenberg, D. A., Soucy, R., Wolfman, W. L., & Senikas, V. (2010). Endometriosis: Diagnosis and Management. Journal of Obstetrics and Gynaecology Canada. https://doi.org/10.1016/S1701-2163(16)34589-3 Lincoln, Y. S., & Guba, E. G. (1985). Establishing Trustworthiness. Naturalistic inquiry. In Naturalistic inquiry. Lincoln, Y. S., Lynham, S. A., & Guba, E. G. (2018). Paradigmatic controversies, contradictions, and emerging confluences. In The SAGE handbook of qualitative research. Link, B. G. (1987). Understanding Labeling Effects in the Area of Mental Disorders: An Assessment of the Effects of Expectations of Rejection. American Sociological Review. https://doi.org/10.2307/2095395 Link, B. G., & Phelan, J. C. (2001). Conceptualizing Stigma. Annual Review of Sociology. https://doi.org/10.1146/annurev.soc.27.1.363 Livingstone, S. (2007). The challenge of engaging youth online: Contrasting producers’ and teenagers’ interpretations of websites. European Journal of Communication. https://doi.org/10.1177/0267323107076768 Logie, C. H., Perez-Brumer, A., Mothopeng, T., Latif, M., Ranotsi, A., & Baral, S. D. (2020). Conceptualizing LGBT Stigma and Associated HIV Vulnerabilities Among LGBT Persons in Lesotho. AIDS and Behavior. https://doi.org/10.1007/s10461-020-02917-y 235 Lunny, C., Taylor, D., Memetovic, J., Wärje, O., Lester, R., Wong, T., Ho, K., Gilbert, M., & Ogilvie, G. (2014). Short message service (SMS) interventions for the prevention and treatment of sexually transmitted infections: A systematic review protocol. Systematic Reviews. https://doi.org/10.1186/2046-4053-3-7 Lurie, Y., & Mark, S. (2016). Professional Ethics of Software Engineers: An Ethical Framework. Science and Engineering Ethics, 22(2). https://doi.org/10.1007/s11948-015-9665-x Lusardi, P. (1996). Qualitative Research in Nursing: Advancing the Humanistic Imperative. Nursing Research, 45(1). https://doi.org/10.1097/00006199-199601000-00014 Macias, W., Lewis, L. S., & Smith, T. L. (2005). Health-related message boards/chat rooms on the web: Discussion content and implications for pharmaceutical sponsorships. Journal of Health Communication, 10(3). https://doi.org/10.1080/10810730590934235 MacLean, R. (2018). Resources to address stigma related to sexuality, substance use and sexually transmitted and blood-borne infections. Canada Communicable Disease Report. https://doi.org/10.14745/ccdr.v44i02a05 Magee, J. C., Bigelow, L., DeHaan, S., & Mustanski, B. S. (2012). Sexual Health Information Seeking Online: A Mixed-Methods Study Among Lesbian, Gay, Bisexual, and Transgender Young People. Health Education and Behavior. https://doi.org/10.1177/1090198111401384 Makoae, L. N., Greeff, M., Phetlhu, R. D., Uys, L. R., Naidoo, J. R., Kohi, T. W., Dlamini, P. S., Chirwa, M. L., & Holzemer, W. L. (2008). Coping with HIV/AIDS Stigma in Five African Countries. Journal of the Association of Nurses in AIDS Care, 19(2), 137–146. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2346777/pdf/nihms43524.pdf Marent, B., Henwood, F., Darking, M., Leon, A., West, B., Lockhart, D., Garcia, D., Fatz, D., Wallitt, E., Beck, E., Gomez, E., Teofilo, E., Gonzalez, E., Garcia, F., Rodrigues, G., 236 Whetham, J., Begovac, J., Block, K., Pereira, L., … Mandalia, S. (2018). Development of an mHealth platform for HIV care: Gathering user perspectives through co-design workshops and interviews. JMIR MHealth and UHealth. https://doi.org/10.2196/mhealth.9856 Martinez, T. S., & Tang, C. (2020). Design Implications for Health Technology to Support LGBTQ+ Community: A literature review. PervasiveHealth: Pervasive Computing Technologies for Healthcare. https://doi.org/10.1145/3421937.3421951 Masip, L., Granollers, T., & Oliva, M. (2011). A heuristic evaluation experiment to validate the new set of usability heuristics. Proceedings - 2011 8th International Conference on Information Technology: New Generations, ITNG 2011. https://doi.org/10.1109/ITNG.2011.82 Masuch, T. V., Bea, M., Alm, B., Deibler, P., & Sobanski, E. (2019). Internalized stigma, anticipated discrimination and perceived public stigma in adults with ADHD. ADHD Attention Deficit and Hyperactivity Disorders. https://doi.org/10.1007/s12402-018-0274-9 Matthew-Maich, N., Harris, L., Ploeg, J., Markle-Reid, M., Valaitis, R., Ibrahim, S., Gafni, A., & Isaacs, S. (2016). Designing, Implementing, and Evaluating Mobile Health Technologies for Managing Chronic Conditions in Older Adults: A Scoping Review. JMIR MHealth and UHealth. https://doi.org/10.2196/mhealth.5127 Mburu, S., Franz, E., & Springer, T. (2013). A Conceptual Framework for Designing mHealth Solutions. MobiHoc Workshop on Pervasive Wireless Healthcare (MobileHealth ’13)., 31–36. https://doi.org/10.1145/2491148.2491154 237 McMillan, S. S., King, M., & Tully, M. P. (2016). How to use the nominal group and Delphi techniques. In International Journal of Clinical Pharmacy. https://doi.org/10.1007/s11096-016-0257-x McPherson, S., Reese, C., & Wendler, M. C. (2018). Methodology update: Delphi studies. In Nursing Research. https://doi.org/10.1097/NNR.0000000000000297 Meisenbach, R. J. (2010). Stigma management communication: A theory and agenda for applied research on how individuals manage moments of stigmatized identity. Journal of Applied Communication Research. https://doi.org/10.1080/00909882.2010.490841 Meshkat, B., Cowman, S., Gethin, G., Ryan, K., Wiley, M., Brick, A., Clarke, E., & Mulligan, E. (2014). Using an e-Delphi technique in achieving consensus across disciplines for developing best practice in day surgery in Ireland. Journal of Hospital Administration. https://doi.org/10.5430/jha.v3n4p1 Mill, J., Edwards, N., Jackson, R., Austin, W., MacLean, L., & Reintjes, F. (2009). Accessing health services while living with HIV: Intersections of stigma. Canadian Journal of Nursing Research. Millikan, E. (2001). Networking Health: Prescriptions for the Internet. American Journal of Health-System Pharmacy, 58(24). https://doi.org/10.1093/ajhp/58.24.2412 Milliken, A., & Grace, P. (2017). Nurse ethical awareness: Understanding the nature of everyday practice. Nursing Ethics, 24(5). https://doi.org/10.1177/0969733015615172 Morris, J. L., Lippman, S. A., Philip, S., Bernstein, K., Neilands, T. B., & Lightfoot, M. (2014). Sexually transmitted infection related stigma and shame among African American Male Youth: Implications for testing practices, partner notification, and treatment. AIDS Patient Care and STDs. https://doi.org/10.1089/apc.2013.0316 238 Muessig, K. E., Nekkanti, M., Bauermeister, J., Bull, S., & Hightow-Weidman, L. B. (2015). A Systematic Review of Recent Smartphone, Internet and Web 2.0 Interventions to Address the HIV Continuum of Care. In Current HIV/AIDS Reports. https://doi.org/10.1007/s11904-014-0239-3 Murphy, Black, Lamping, McKee, Sanderson, Askham, & Marteau. (1998). Consensus development methods, and their use in clinical guideline development. Health Technology Assessment. https://doi.org/10.3310/hta2030 National Academies of Sciences, E. and M. (2016). Approaches to Reducing Stigma. In Ending Discrimination Against People with Mental and Substance Use Disorders: The Evidence for Stigma Change. https://doi.org/10.17226/23442 National AIDS Trust. (2016). Tackling HIV stigma: What works? National AIDS Trust, April, 34. Nayar, U. S., Stangl, A. L., De Zalduondo, B., & Brady, L. M. (2014). Reducing stigma and discrimination to improve child health and survival in low-and middle-income countries: Promising approaches and implications for future research. In Journal of Health Communication. https://doi.org/10.1080/10810730.2014.930213 Neergaard, M. A., Olesen, F., Andersen, R. S., & Sondergaard, J. (2009). Qualitative description-the poor cousin of health research? In BMC Medical Research Methodology (Vol. 9, Issue 1). https://doi.org/10.1186/1471-2288-9-52 Newell, A. F., Gregor, P., Morgan, M., Pullin, G., & Macaulay, C. (2011). User-Sensitive Inclusive Design. Universal Access in the Information Society. https://doi.org/10.1007/s10209-010-0203-y 239 Nielsen, J. (1994a). Enhancing the explanatory power of usability heuristics. Conference on Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/259963.260333 Nielsen, J. (1994b). Usability inspection methods. Conference on Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/259963.260531 Nielsen, J., & Molich, R. (1990). Heuristic evaluation of user interfaces. Conference on Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/97243.97281 Noltensmeyer, C. J., Peters, S., Meisenbach, R. J., & Eastman-Mueller, H. P. (2016). Managing sexual health and related stigma through electronic learning environments. Health Communication and Mass Media: An Integrated Approach to Policy and Practice, Cdc 2009, 65–80. https://doi.org/10.4324/9781315586335 Norman, D. A. (2004). Emotional design. Ubiquity. https://doi.org/10.1145/985600.966013 Norman, D. A. (2005). Human-centered design considered harmful. Interactions, 12(4), 14–19. https://doi.org/10.1145/1070960.1070976 Nyblade, L., Stockton, M. A., Giger, K., Bond, V., Ekstrand, M. L., Lean, R. M., Mitchell, E. M. H., Nelson, L. R. E., Sapag, J. C., Siraprapasiri, T., Turan, J., & Wouters, E. (2019). Stigma in health facilities: Why it matters and how we can change it. BMC Medicine. https://doi.org/10.1186/s12916-019-1256-2 Orenstein, E. W., Boudreaux, J., Josephson, C. D., Orenstein, E. W., Boudreaux, J., Rollins, M., Jones, J., Josephson, C. D., Rollins, M., Williams, H., Bryant, C., Hike, J., Kilgore, T., Karavite, D., Muthu, N., Carter, A. B., & Josephson, C. D. (2019). Formative Usability Testing Reduces Severe Blood Product Ordering Errors. Applied Clinical Informatics. https://doi.org/10.1055/s-0039-3402714 240 Pachankis, J. E., Hatzenbuehler, M. L., Hickson, F., Weatherburn, P., Berg, R. C., Marcus, U., & Schmidt, A. J. (2015). Hidden from health: Structural stigma, sexual orientation concealment, and HIV across 38 countries in the European MSM Internet Survey. In AIDS. https://doi.org/10.1097/QAD.0000000000000724 Palinkas, L. A., Horwitz, S. M., Green, C. A., Wisdom, J. P., Duan, N., & Hoagwood, K. (2015). Purposeful Sampling for Qualitative Data Collection and Analysis in Mixed Method Implementation Research. Administration and Policy in Mental Health and Mental Health Services Research. https://doi.org/10.1007/s10488-013-0528-y Parasar, P., Ozcan, P., & Terry, K. L. (2017). Endometriosis: Epidemiology, Diagnosis and Clinical Management. Current Obstetrics and Gynecology Reports. https://doi.org/10.1007/s13669-017-0187-1 Payton, F. C., & Kvasny, L. (2016). Online HIV awareness and technology affordance benefits for black female collegians - maybe not: The case of stigma. Journal of the American Medical Informatics Association. https://doi.org/10.1093/jamia/ocw017 Paz, F., Paz, F. A., Pow-Sang, J. A., & Collantes, L. (2014). Usability heuristics for transactional web sites. ITNG 2014 - Proceedings of the 11th International Conference on Information Technology: New Generations. https://doi.org/10.1109/ITNG.2014.81 Peltonen, L.-M., Pruinelli, L., Ronquillo, C., Nibber, R., Peresmitre, E. L., Block, L., Deforest, H., Lewis, A., Alhuwail, D., Ali, S., Badger, M. K., Eler, G. J., Georgsson, M., Islam, T., Jeon, E., Jung, H., Kuo, C. H., Sarmiento, R. F. R., Sommer, J. A., … Topaz, M. (2019). The current state of Nursing Informatics – An international cross-sectional survey. Finnish Journal of EHealth and EWelfare, 11(3). https://doi.org/10.23996/fjhw.77584 241 Penson, R. T., Partridge, R. A., Rudd, P., Seiden, M. v, Nelson, J. E., Chabner, B. A., & Jr, T. J. L. (2005). S chwartz C enter R ounds Update : Laughter : The Best Medicine ? Humor International Journal Of Humor Research, 651–660. Pinel, E. C., & Bosson, J. K. (2013). Turning Our Attention to Stigma: An Objective Self-Awareness Analysis of Stigma and Its Consequences. Basic and Applied Social Psychology. https://doi.org/10.1080/01973533.2012.746593 Polit, & Beck, C. T. (2012). Nursing research: generating and assessing evidence for nursing practice., 9th edn: Philadelphia: In Lippincott Williams & Wilkins. https://doi.org/10.1017/CBO9781107415324.004 Polit, D. F., Beck, C. T., & Owen, S. V. (2007). Focus on research methods: Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Research in Nursing and Health. https://doi.org/10.1002/nur.20199 Prince, R. M., Soung Yee, A., Parente, L., Enright, K. A., Grunfeld, E., Powis, M., Husain, A., Gandhi, S., & Krzyzanowska, M. K. (2019). User-centered design of a web-based tool to support management of chemotherapy-related toxicities in cancer patients. Journal of Medical Internet Research. https://doi.org/10.2196/jmir.9958 Pryor, J. B., & Reeder, G. D. (2015). Collective and individual representations of HIV/AIDS stigma. In The Social Psychology of HIV Infection. https://doi.org/10.4324/9781315722924 Pulerwitz, J., Michaelis, A., Weiss, E., Brown, L., & Mahendra, V. (2010). Reducing HIV-related stigma: Lessons learned from horizons research and programs. In Public Health Reports. https://doi.org/10.1177/003335491012500218 Quinn, D. M., & Chaudoir, S. R. (2009). Living With a Concealable Stigmatized Identity: The Impact of Anticipated Stigma, Centrality, Salience, and Cultural Stigma on Psychological 242 Distress and Health. Journal of Personality and Social Psychology. https://doi.org/10.1037/a0015815 Quiñones, D., & Rusu, C. (2017). How to develop usability heuristics: A systematic literature review. Computer Standards and Interfaces. https://doi.org/10.1016/j.csi.2017.03.009 Quiñones, D., Rusu, C., & Rusu, V. (2018). A methodology to develop usability/user experience heuristics. Computer Standards and Interfaces, 59(February), 109–129. https://doi.org/10.1016/j.csi.2018.03.002 Ralph, L. J., Berglas, N. F., Schwartz, S. L., & Brindis, C. D. (2011). Finding teens in TheirSpace: Using social networking sites to connect youth to sexual health services. Sexuality Research and Social Policy. https://doi.org/10.1007/s13178-011-0043-4 Redston, S., de Botte, S., & Smith, C. (2018). Resolving embarrassing medical conditions with online health information. International Journal of Medical Informatics. https://doi.org/10.1016/j.ijmedinf.2018.03.010 Renn, B. N., Schurr, M., Zaslavsky, O., & Pratap, A. (2021). Artificial Intelligence: An Interprofessional Perspective on Implications for Geriatric Mental Health Research and Care. Frontiers in Psychiatry, 12. https://doi.org/10.3389/fpsyt.2021.734909 Reynolds, T. L. (2020). Designing Health Interface Technologies to Support Patients’ and Healthcare Providers’ Individual and Collaborative Work. 2020 IEEE International Conference on Healthcare Informatics, ICHI 2020. https://doi.org/10.1109/ICHI48887.2020.9374368 Richters, J., Badcock, P. B., Simpson, J. M., Shellard, D., Rissel, C., de Visser, R. O., Grulich, A. E., & Smith, A. M. A. (2014). Design and methods of the Second Australian Study of Health and Relationships. Sexual Health. https://doi.org/10.1071/SH14115 243 Risling, T. L., & Risling, D. E. (2020). Advancing nursing participation in user-centred design. Journal of Research in Nursing, 25(3). https://doi.org/10.1177/1744987120913590 Robinson, A., Oksuz, U., Slight, R., Slight, S., & Husband, A. (2020). Digital and mobile technologies to promote physical health behavior change and provide psychological support for patients undergoing elective surgery: Meta-ethnography and systematic review. In JMIR mHealth and uHealth (Vol. 8, Issue 12). https://doi.org/10.2196/19237 Robinson, & Norman, D. (1994). Things That Make Us Smart: Defending Human Attributes in the Age of the Machine. Design Issues. https://doi.org/10.2307/1511661 Rocha-Jiménez, T., Morales-Miranda, S., Fernández-Casanueva, C., Brouwer, K. C., & Goldenberg, S. M. (2018). Stigma and unmet sexual and reproductive health needs among international migrant sex workers at the Mexico–Guatemala border. International Journal of Gynecology and Obstetrics. https://doi.org/10.1002/ijgo.12441 Rose, K. H. (2013). A Guide to the Project Management Body of Knowledge (PMBOK® Guide)-Fifth Edition. Project Management Journal, 44(3). https://doi.org/10.1002/pmj.21345 Rosenfield, S. (1997). Labeling mental illness: The effects of received services and perceived stigma on life satisfaction. American Sociological Review. https://doi.org/10.2307/2657432 Ross, M. W., Nyoni, J., Ahaneku, H. O., Mbwambo, J., McClelland, R. S., & McCurdy, S. A. (2014). High HIV seroprevalence, rectal STIs and riskxy sexual behaviour in men who have sex with men in Dar es Salaam and Tanga, Tanzania. BMJ Open. https://doi.org/10.1136/bmjopen-2014-006175 Rueda, S., Mitra, S., Chen, S., Gogolishvili, D., Globerman, J., Chambers, L., Wilson, M., Logie, C. H., Shi, Q., Morassaei, S., & Rourke, S. B. (2016). Examining the associations between 244 HIV-related stigma and health outcomes in people living with HIV/AIDS: a series of meta-analyses. BMJ Open. https://doi.org/10.1136/bmjopen-2016-011453 Rüsch, N., Angermeyer, M. C., & Corrigan, P. W. (2005). Mental illness stigma: Concepts, consequences, and initiatives to reduce stigma. In European Psychiatry. https://doi.org/10.1016/j.eurpsy.2005.04.004 Saldana, J. (2009). An Introduction to Codes and Coding. The Coding Manual for Qualitative Researchers., 2006. https://doi.org/10.1519/JSC.0b013e3181ddfd0a SAMSHA. (2014). Substance Use Disorder Treatment Providers and CCBHCs | SAMHSA. https://www.samhsa.gov/section-223/care-coordination/substance-use-disorder-treatment-providers Sandelowski, M. (2000). Whatever happened to qualitative description? Research in Nursing & Health, 23(4). https://doi.org/10.1002/1098-240x(200008)23:4<334::aid-nur9>3.0.co;2-g Sandelowski, M. (2010). What’s in a name? Qualitative description revisited. Research in Nursing and Health, 33(1). https://doi.org/10.1002/nur.20362 Sanz, F., Galvez, R., Rusu, C., Roncagliolo, S., Rusu, V., Collazos, C. A., Cofré, J. P., Campos, A., & Quiñones, D. (2016). A set of usability heuristics and design recommendations for u-Learning applications. In Advances in Intelligent Systems and Computing. https://doi.org/10.1007/978-3-319-32467-8_85 Sbaffi, L., & Rowley, J. (2017). Trust and credibility in web-based health information: A review and agenda for future research. In Journal of Medical Internet Research. https://doi.org/10.2196/jmir.7579 Scambler, G. (2009). Health-related stigma. In Sociology of Health and Illness. https://doi.org/10.1111/j.1467-9566.2009.01161.x 245 Scheibe, M., Skutsh, M., & Schofer, J. (2002). Experiments in Delphi Methodology. The Delphi Method: Techniques and Applications. Schmidt, B. J., & McArthur, E. C. (2018). Professional nursing values: A concept analysis. Nursing Forum, 53(1). https://doi.org/10.1111/nuf.12211 Schnall, R., Mosley, J. P., Iribarren, S. J., Bakken, S., Carballo-Diéguez, A., & Brown III, W. (2015). Comparison of a User-Centered Design, Self-Management App to Existing mHealth Apps for Persons Living With HIV. JMIR MHealth and UHealth. https://doi.org/10.2196/mhealth.4882 Shamat, N. A., Sulaiman, S., & Sinpang, J. S. (2017). A Systematic Literature Review on User Interface Design for Web Applications. In Journal of Telecommunication, Electronic and Computer Engineering (Vol. 9, Issues 3-4 Special Issue). Shaw, J., Agarwal, P., Desveaux, L., Palma, D. C., Stamenova, V., Jamieson, T., Yang, R., Bhatia, R. S., & Bhattacharyya, O. (2018). Beyond “implementation”: digital health innovation and service design. Npj Digital Medicine. https://doi.org/10.1038/s41746-018-0059-8 Shearer, L. S., Simmons, L., Mindel, A., Stanberry, L. R., & Rosenthal, S. L. (2012). Reducing the stigma of herpes simplex virus infection: Lessons from an online video contest. Sexual Health, 9(5), 438–444. https://doi.org/10.1071/SH11188 Sheehan, L., Nieweglowski, K., & Corrigan, P. W. (2016). Structures and types of stigma. In The Stigma of Mental Illness - End of the Story? https://doi.org/10.1007/978-3-319-27839-1_3 Shehadeh, K., Arman, N., & Khamayseh, F. (2021). Semi-Automated Classification of Arabic User Requirements into Functional and Non-Functional Requirements using NLP Tools. 246 2021 International Conference on Information Technology, ICIT 2021 - Proceedings. https://doi.org/10.1109/ICIT52682.2021.9491698 Shelby, A., & Ernst, K. (2013). Story and science: How providers and parents can utilize storytelling to combat anti-vaccine misinformation. In Human Vaccines and Immunotherapeutics (Vol. 9, Issue 8). https://doi.org/10.4161/hv.24828 Shidende, N. H., & Mörtberg, C. (2014). Re-visiting design-after-design: reflecting implementation mediators connectedness in distributed participatory design activities. Proceedings of the 13th Participatory Design Conference on Research Papers - PDC ’14, 2005, 61–70. https://doi.org/10.1145/2661435.2661437 Shneiderman, B. (1987). Designing the user interface strategies for effective human-computer interaction. ACM SIGBIO Newsletter. https://doi.org/10.1145/25065.950626 Sieck, C. J., Pearl, N., Bright, T. J., & Yen, P. Y. (2020). A qualitative study of physician perspectives on adaptation to electronic health records. In BMC Medical Informatics and Decision Making (Vol. 20, Issue 1). https://doi.org/10.1186/s12911-020-1030-6 Sipes, C. (2016). Project management: Essential skill of nurse informaticists. Studies in Health Technology and Informatics, 225. https://doi.org/10.3233/978-1-61499-658-3-252 Six, J. M., & Macefield, R. (2016). How to Determine the Right Number of Participants for Usability Studies. UX Matters. Skulmoski, G., T. Hartman, F., & Krahn, J. (2007). The Delphi Method for Graduate Research. Journal of Information Technology Education: Research. https://doi.org/10.28945/199 Smith, R. (2007). Media Depictions of Health Topics: Challenge and Stigma Formats. In Journal of Health Communication. https://doi.org/10.1080/10810730701266273 247 Sockolow, P., Joppa, M., & Zhu, J. (2018). Assessment of evaluation frameworks for design of a sexual risk prevention game for black adolescent girls. Studies in Health Technology and Informatics, 250. https://doi.org/10.3233/978-1-61499-872-3-101 Stangl, A. L., Lloyd, J. K., Brady, L. M., Holland, C. E., & Baral, S. (2013). A systematic review of interventions to reduce HIV-related stigma and discrimination from 2002 to 2013: how far have we come? In Journal of the International AIDS Society. https://doi.org/10.7448/ias.16.3.18734 Statistics Canada. (2013). Canadian Internet Use Survey, 2012. The Daily. Stephanidis, C., Salvendy, G., Antona, M., Chen, J. Y. C., Dong, J., Duffy, V. G., Fang, X., Fidopiastis, C., Fragomeni, G., Fu, L. P., Guo, Y., Harris, D., Ioannou, A., Jeong, K. ah (Kate), Konomi, S., Krömker, H., Kurosu, M., Lewis, J. R., Marcus, A., … Zhou, J. (2019). Seven HCI Grand Challenges. In International Journal of Human-Computer Interaction (Vol. 35, Issue 14). https://doi.org/10.1080/10447318.2019.1619259 Stinson, J., McGrath, P., Hodnett, E., Feldman, B., Duffy, C., Huber, A., Tucker, L., Hetherington, R., Tse, S., Spiegel, L., Campillo, S., Gill, N., & White, M. (2010). Usability testing of an online self-management program for Adolescents with Juvenile Idiopathic Arthritis. Journal of Medical Internet Research. https://doi.org/10.2196/jmir.1349 Story, M. F. , Mueller, J. L. , & Mace, R. L. (1998). The universal design file: Designing for people of all ages & abilities Strudwick, G., Nagle, L. M., Morgan, A., Kennedy, M. A., Currie, L. M., Lo, B., & White, P. (2019). Adapting and validating informatics competencies for senior nurse leaders in the Canadian context: Results of a Delphi study. International Journal of Medical Informatics. https://doi.org/10.1016/j.ijmedinf.2019.06.012 248 Stuber, J., Meyer, I., & Link, B. (2008). Stigma, prejudice, discrimination and health. In Social Science and Medicine. https://doi.org/10.1016/j.socscimed.2008.03.023 Swire-Thompson, B., & Lazer, D. (2019). Public health and online misinformation: Challenges and recommendations. In Annual Review of Public Health (Vol. 41). https://doi.org/10.1146/annurev-publhealth-040119-094127 Terry, C., & Cain, J. (2016). The emerging issue of digital empathy. American Journal of Pharmaceutical Education, 80(4). https://doi.org/10.5688/ajpe80458 Tonsaker, T., Bartlett, G., & Trpkov, C. (2014). Health information on the Internet: gold mine or minefield? Canadian Family Physician Medecin de Famille Canadien. Torous, J., Myrick, K. J., Rauseo-Ricupero, N., & Firth, J. (2020). Digital mental health and COVID-19: Using technology today to accelerate the curve on access and quality tomorrow. In JMIR Mental Health (Vol. 7, Issue 3). https://doi.org/10.2196/18848 Towler, A. J., & Schneider, D. J. (2005). Distinctions among stigmatized groups. Journal of Applied Social Psychology. https://doi.org/10.1111/j.1559-1816.2005.tb02090.x Tullis, T., Fleischman, S., & McNulty, M. (2002). An empirical comparison of lab and remote usability testing of web sites. Usability Professionals …. Turan, B., Hatcher, A. M., Weiser, S. D., Johnson, M. O., Rice, W. S., & Turan, J. M. (2017). Framing mechanisms linking HIV-related stigma, adherence to treatment, and health outcomes. In American Journal of Public Health. https://doi.org/10.2105/AJPH.2017.303744 Vaes, K. (2019). Design for Empowerment, the Stigma-Free Design Toolkit. Advances in Intelligent Systems and Computing. https://doi.org/10.1007/978-3-319-96071-5_104 249 Vaisson, G., Provencher, T., Dugas, M., Trottier, M. È., Chipenda Dansokho, S., Colquhoun, H., Fagerlin, A., Giguere, A. M. C., Hakim, H., Haslett, L., Hoffman, A. S., Ivers, N. M., Julien, A. S., Légaré, F., Renaud, J. S., Stacey, D., Volk, R. J., & Witteman, H. O. (2021). User Involvement in the Design and Development of Patient Decision Aids and Other Personal Health Tools: A Systematic Review. In Medical Decision Making (Vol. 41, Issue 3). https://doi.org/10.1177/0272989X20984134 van Uden-Kraan, C. F., Drossaert, C. H. C., Taal, E., Lebrun, C. E. I., Drossaers-Bakker, K. W., Smit, W. M., Seydel, E. R., & van de Laar, M. A. F. J. (2008). Coping with somatic illnesses in online support groups: Do the feared disadvantages actually occur? Computers in Human Behavior. https://doi.org/10.1016/j.chb.2007.01.014 Vaughn, J., Kamkhoad, D., Ford, S., Subramaniam, A. P., Khairat, S., & Shah, N. (2021). An Innovative Undergraduate Interdisciplinary Research Course: Development, Implementation, and Student Perspectives. Computers, Informatics, Nursing : CIN, 39(10). https://doi.org/10.1097/CIN.0000000000000836 Vayena, E., Haeusermann, T., Adjekum, A., & Blasimme, A. (2018). Digital health: meeting the ethical and policy challenges. In Swiss medical weekly (Vol. 148). https://doi.org/10.4414/smw.2018.14571 Vermeeren, A. P. O. S., Roto, V., & Vï¿½ï¿½nï¿½nen, K. (2016). Design-inclusive UX research: Design as a part of doing user experience research. In Behaviour and Information Technology. https://doi.org/10.1080/0144929X.2015.1081292 Vojtila, L., Ashfaq, I., Ampofo, A., Dawson, D., & Selby, P. (2021). Engaging a person with lived experience of mental illness in a collaborative care model feasibility study. In 250 Research Involvement and Engagement (Vol. 7, Issue 1). https://doi.org/10.1186/s40900-020-00247-w Wadden, D., August, T., Li, Q., & Althoff, T. (2020). The effect of moderation on online mental health conversations. In arXiv. Wadham, E., Green, C., Debattista, J., Somerset, S., & Sav, A. (2019). New digital media interventions for sexual health promotion among young people: A systematic review. In Sexual Health. https://doi.org/10.1071/SH18127 Wahl, K. J., Orr, N. L., Lisonek, M., Noga, H., Bedaiwy, M. A., Williams, C., Allaire, C., Albert, A. Y., Smith, K. B., Cox, S., & Yong, P. J. (2020). Deep Dyspareunia, Superficial Dyspareunia, and Infertility Concerns Among Women With Endometriosis: A Cross-Sectional Study. Sexual Medicine. https://doi.org/10.1016/j.esxm.2020.01.002 Wan, X., Wang, W., Liu, J., & Tong, T. (2014). Estimating the sample mean and standard deviation from the sample size, median, range and/or interquartile range. BMC Medical Research Methodology. https://doi.org/10.1186/1471-2288-14-135 Wang, X., Shi, J., & Kong, H. (2020). Online Health Information Seeking: A Review and Meta-Analysis. Health Communication. https://doi.org/10.1080/10410236.2020.1748829 Waqas, A., Malik, S., Fida, A., Abbas, N., Mian, N., Miryala, S., Amray, A. N., Shah, Z., & Naveed, S. (2020). Interventions to Reduce Stigma Related to Mental Illnesses in Educational Institutes: a Systematic Review. In Psychiatric Quarterly (Vol. 91, Issue 3). https://doi.org/10.1007/s11126-020-09751-4 Warren, T., & Ebel, C. (2005). Counseling the patient who has genital herpes or genital human papillomavirus infection. In Infectious Disease Clinics of North America. https://doi.org/10.1016/j.idc.2005.03.011 251 Wei, Y., Zheng, P., Deng, H., Wang, X., Li, X., & Fu, H. (2020). Design features for improving mobile health intervention user engagement: Systematic review and thematic analysis. In Journal of Medical Internet Research (Vol. 22, Issue 12). https://doi.org/10.2196/21687 Weijts, W., Houtkoop, H., & Mullen, P. (1993). Talking delicacy: speaking about sexuality during gynaecological consultations. Sociology of Health & Illness. https://doi.org/10.1111/1467-9566.ep10490531 Whetten, K., Reif, S., Toth, M., Jain, E., Leserman, J., & Pence, B. W. (2012). Relationship between trauma and high-risk behavior among HIV-positive men who do not have sex with men (MDSM). AIDS Care - Psychological and Socio-Medical Aspects of AIDS/HIV. https://doi.org/10.1080/09540121.2012.712665 WHO. (2010). Developing sexual health programmes. A framework for action. Reproductive Health, 55. WHO. (2015). Sexual health, human rights and the law. Reproductive Health Matters, 23(46), 193–195. https://doi.org/10.1016/j.rhm.2015.11.013 WHO. (2016). Monitoring and evaluating digital health interventions. A practical guide to conducting research and assessment. In Who. Wiljer, D., Charow, R., Costin, H., Sequeira, L., Anderson, M., Strudwick, G., Tripp, T., & Crawford, A. (2019). Defining compassion in the digital health age: Protocol for a scoping review. In BMJ Open (Vol. 9, Issue 2). https://doi.org/10.1136/bmjopen-2018-026338 Willis, D. G., Sullivan-Bolyai, S., Knafl, K., & Cohen, M. Z. (2016). Distinguishing Features and Similarities Between Descriptive Phenomenological and Qualitative Description Research. Western Journal of Nursing Research, 38(9). https://doi.org/10.1177/0193945916645499 252 Wilson, C. (2013). User Interface Inspection Methods: A User-Centered Design Method. In User Interface Inspection Methods: A User-Centered Design Method. https://doi.org/10.1016/C2012-0-06519-2 Witzel, T. C., Guise, A., Nutland, W., & Bourne, A. (2016). It Starts with Me: Privacy concerns and stigma in the evaluation of a Facebook health promotion intervention. Sexual Health. https://doi.org/10.1071/SH15231 Wood, J. M., Harries, J., Kalichman, M., Kalichman, S., Nkoko, K., & Mathews, C. (2018). Exploring motivation to notify and barriers to partner notification of sexually transmitted infections in South Africa: A qualitative study. BMC Public Health. https://doi.org/10.1186/s12889-018-5909-4 Xu, X., Hu, J., Lyu, X., Huang, H., & Cheng, X. (2021). Exploring the interdisciplinary nature of precision medicine：network analysis and visualization. JMIR Medical Informatics, 9(1). https://doi.org/10.2196/23562 Yang, L. H., Kleinman, A., Link, B. G., Phelan, J. C., Lee, S., & Good, B. (2007). Culture and stigma: Adding moral experience to stigma theory. Social Science and Medicine. https://doi.org/10.1016/j.socscimed.2006.11.013 Yong, P. J. (2017). Deep Dyspareunia in Endometriosis: A Proposed Framework Based on Pain Mechanisms and Genito-Pelvic Pain Penetration Disorder. In Sexual Medicine Reviews. https://doi.org/10.1016/j.sxmr.2017.06.005 York, C. S., & Ertmer, P. A. (2011). Towards an understanding of instructional design heuristics: An exploratory Delphi study. Educational Technology Research and Development. https://doi.org/10.1007/s11423-011-9209-2 253 Yu, Y., & Nam, T. J. (2014). Let’s Giggle!: Design principles for humorous products. Proceedings of the Conference on Designing Interactive Systems: Processes, Practices, Methods, and Techniques, DIS. https://doi.org/10.1145/2598510.2598557 Zhang, X., Lewis, S., Firth, J., Chen, X., & Bucci, S. (2021). Digital mental health in China: A systematic review. In Psychological Medicine. https://doi.org/10.1017/S0033291721003731 Zhang, Y., Li, H., Duan, H., & Zhao, Y. (2015). Mobilizing clinical decision support to facilitate knowledge translation: A case study in China. Computers in Biology and Medicine. https://doi.org/10.1016/j.compbiomed.2015.02.013 254 Appendices Appendix A - Study 2 Materials Task Scenarios for Study 2 Usability Testing Task Scenarios As you know, a lot of people with endometriosis use the internet to search for information. The goal of this study is to evaluate the usability of a website for people with endometriosis-associated painful sex, which is currently under development. Participating in this research involves an approximately one-hour study made up of three parts. In the first part, you will be asked to use the website to execute the task scenarios stated below. This will be followed by a semi-structured interview and a final questionnaire that assesses your satisfaction and the functionality of the website. With your permission, a screen recording of your interaction with the website will be taken, and the interview will be audio-recorded and transcribed to keep an accurate record. The screen recording of their interaction with the website will be focused on the screen-shared only and will not capture any identifying images, such as your face and name. About the website In the summer of 2019, the Centre for Pelvic Pain and Endometriosis, in collaboration with patient partners, developed a website for endometriosis and painful sex. The main purpose of this website is to raise awareness of endometriosis-associated painful sex and to improve care by sharing evidence-based information. We would like to understand the usability and functionality of the website from the perspective of the users before we share it with the public. We would like you to use the website to answer complete the following task. Please note that this study is not about your knowledge of endometriosis or your computer skills. We aim to establish if the website is easy and enjoyable to use. 1. Imagine you recently heard from a lecture about ways of treating sexual pains. Can you find the psychological ways of treating painful sex? 2. Imagine you were recently diagnosed with endometriosis and are worried that the condition can affect your ability to get pregnant. Please find the information on fertility 3. Imagine your healthcare professional has told you a bit about how pain works and how it is related to your psychological wellbeing. You want to know more about the cycle of pain. How would you locate the anticipation of the pain cycle on this website? 4. Imagine you have heard about central sensitization and are curious to know what it is and how it causes pain. Using this website, find where it describes the relationship between central sensitization and painful sex. 5. Imagine that you just learned about the term cul-de-sac in relation to endometriosis, and you are curious to know more about it. Can you use the website to find the meaning of the cul-de-sac? 255 Study 2 Post-Test Interview Guide (adapted from Stinson J. et al 2010) Thank you for your time in performing this usability testing. In the next part, you will be asked questions pertaining to the navigation of the website, its looks, the ease in finding the information, your thoughts on the information on the website etc. please remember that your responses are very valuable in re-developing the website to suit the needs of other users 1. What did you like most about the website? Probes animations, layout, information, icons 2. What did you not like about the website? Probes animations, layout, information, 3. How easy it was to navigate the website? Probes what challenges did you experience in navigating the website? How easy can you move between pages, simplicity of navigation, do navigation buttons looks like clickable versions of real world tabs 4. What do you think about the overall look of the website? Probes can you tell me more about the colors, designs, images of the website. Can you comment on the consistency and readability of fonts? Are they visually appealing? What would make the site more appealing? 5. What are your thoughts regarding the information on the website? Probes what are your thoughts about the amount of information on the site, how well are the information presented, how easy can you find the information you need (would you prefer search functions) how easy is it to read and understand, will the information help you understand endometriosis better than what you currently know, are they additional information that you would want to see on the website. What are your comments on the text and picture information? 6. What, if anything caused you frustration with the use of the website? 7. Are there any instances where the website is slow or non-responsive? 8. If you could make some changes to the website, what would they be? Think about the problems you encountered and what you don’t like about the website. 9. Would you prefer additional features to make the website more interactive? Probes Discussion boards, video clips 10. how will this and other websites help address stigma a. How would you want this to appear? How should the website look like? 11. In what ways can websites provoke stigma 12. Do you have other suggestions on how websites can be developed to reduce stigma 256 Appendix B - Study 3 Recruitment Materials Recruitment Message & Consent Form Dear [Name of Participant], My name is Abdul-Fatawu Abdulai, a student co-investigator at the University of British Columbia School of Nursing. We are conducting a Delphi study titled “Development of destigmatising Design Principles for Use in Sexual Health-Related Technologies”. The Principal Investigator for this study is Dr. Leanne Currie, an Associate Professor at UBC school of Nursing. The purpose of the study is to develop a set of design principles that will guide web designers in creating sexual health-related technologies that can address stigma. I got your name from [Conference, Resource name, Or Publication] and I believe you may be a good candidate for this study because of your expertise in [sexual health or stigma-related research]. In this study, you will be asked to rate the extent of your agreement with the clarity and usefulness of a list of design principles in three rounds of surveys. You will also justify your rating for each item. At the end of the study, you will also have the chance to suggest additional ideas. The study may require a total of up to 2 hours of participation time. In appreciation for your time and efforts, you will receive CND $15 (or its equivalent) in a form of an amazon gift card for each round that you have completed. If you are interested in this study, please click on this link to fill out a qualifying checklist and the demographic information [Link to Online Qualtrics Survey]. All necessary steps will be taken to protect you confidentiality. If you have questions concerning the study, contact me directly. Please accept our sincere thanks for taking the time to participate. We truly appreciate your involvement and want you to know that your responses will help us, and web designers understand ways sexual health/men/women health-related websites can be designed to address stigma Sincerely Abdul-Fatawu Abdulai Ph.D.(c) and student co-investigator School of Nursing, University of British Columbia Round 1 Introductory Email Dear Participant Thank you for responding to my invitation to take part in developing a set of destigmatising design principles for use in sexual health-related technologies. You have met the expertise needed for our study and I am contacting you with the link to the first round of survey. Please rate the extent to which you agree/disagree with the clarity and utility of each destigmatising design principle and comment on same. At the end of the survey, you may propose additional or ideas on how sexual health technologies can be designed to address stigma. You will have two weeks to complete this survey. Please note that you do not need to complete the survey in one sitting. You can stop and continue at another time. Your study ID is…... You will need to enter this study ID in the survey. Contact me directly via email if you have further questions Thanks once again and we look forward to receiving your responses 257 Abdul-Fatawu Abdulai Student co-investigator Round 1 Reminder Email Dear We are extending the closing date for the survey from Monday June 21st to Friday June 25th 2020 at 11:59 pm Pacific. We will be very grateful if you can complete the survey by June 25th 2021 at 11:59pm Pacific time. This survey is expected to take approximately 1 hour to complete. Please note that you do not necessarily need to complete the survey at once. You can save your responses and continue at another time. Please let us know if you have any questions Thank you once again for your time and do note that your responses are very valuable in this research We look forward to receiving your response Sincerely Abdul-Fatawu Abdulai RN PhD (c) Round 2 Introductory Email Dear Many thanks for responding to Round 1 of the Delphi study. We truly appreciate your valuable contributions in the last round. We have analyzed the findings and 12 out of the 28 items met Consensus among the panel of experts. However, given the valuable comments from participants, we have revised all items and came out with 26 items. Just like Round 1, please rate the extent of your agreement/disagreement with the clarity and usefulness of each principle. Any additional comments on any of the items will be greatly appreciated. You may also suggest additional ideas if you have any. This survey will take approximately 20-30 minutes and you will have two weeks to submit your responses. Please note that you do not need to complete the survey in one sitting. You can stop and continue at another time. If you have any questions, please contact me or the Principal Investigator. As a reminder, your assigned study ID is…. Please enter this ID when taking the survey. Thanks once again and we look forward to receiving your responses. Most sincerely Abdul-Fatawu Abdulai PhD (c) Student co-investigator School of Nursing, University of British Columbia Round 2 Reminder Email Dear This is a reminder regarding the Round 2 Delphi survey. We will be very grateful if you can complete Round 2 by July 30th, 2021 at 11:59 Pacific time. This survey is expected to take approximately 20 minutes to complete. You do not necessarily need to complete the survey in one sitting. You can save your responses and continue at another time. As a reminder, your study ID is…. Please contact me directly if you have any questions Many thanks for your time and do note that your expertise is very valuable in this research Looking forward to receiving your response Sincerely 258 Abdul-Fatawu Abdulai Ph.D. Candidate, UBC School of Nursing Student co-investigator Round 3 Introductory Email Dear Many thanks for responding to Round 2 of the Delphi study. We have analyzed the findings and 12 out of the 26 items met consensus among the panel of experts, while 4 items overlapped. We have revised the items and came up with 12 items Just like Round 1, please rate the extent of your agreement/disagreement with the clarity and usefulness of each principle. Any additional comments on any of the items will be greatly appreciated. You may also write your general impressions of your concluding thoughts regarding the destigmatising design principles. This survey will take approximately 10-15 minutes and you will have two weeks to submit your responses. Please note that you do not need to complete the survey in one sitting. You can stop and continue at another time. If you have any questions, please contact me or the Principal Investigator As a reminder, your assigned study ID is …... Please enter this ID when taking the survey. Thanks once again and we look forward to receiving your responses. Most sincerely Abdul-Fatawu Abdulai PhD (c) Student co-investigator School of Nursing, University of British Columbia Round 3 Reminder Email Dear… This is a reminder regarding the Round 3 Delphi survey. We will be very grateful if you can complete Round 3 by August 15th, 2021 at 11:59 Pacific time. This survey is expected to take approximately 20 minutes to complete. You do not necessarily need to complete the survey in one sitting. You can save your responses and continue at another time. As a reminder, your study ID is …... Please contact me directly if you have any questions Many thanks for your time and do note that your expertise is very valuable in this research Looking forward to receiving your response Sincerely Abdul-Fatawu Abdulai Ph.D. Candidate, UBC School of Nursing Student co-investigator 259 Appendix C - Study 3 Original Survey Items Deployed in Round 1 Q1 Provide participants with the ability for anonymous engagement Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q2 Encrypt websites that collect personal information to prevent unauthorized access to personal data Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q3 Prevent errors by having features that warn users from taking actions that can lead to accidental disclosure of status Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o 260 Q4 Include a video or written testimonials of real people to talk about stigma and their experiences with the condition Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q5 Provide opportunities for selective and voluntary disclosure of status to provide emotional connection and support Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q6 Provide interpersonal contacts for people to seek direct online counseling and other psychological support Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o 261 Q7 Include links to other credible sites and contact information for users to seek further information Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q8 Provide mechanisms for creating online advocacy groups Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q9 Ensure appropriate flexibility of web features to allow a user to hide certain content/images that they finds offensive Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o 262 Q10 Ensure customization features to enable users to adapt aspects of the interface to suit their visual preference Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q11 Use non-sensational language and images on web platforms Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q12 Logos, icons, and terminologies should be made subtle so that they do not immediately draw the users’ or bystanders’ attention to sexuality Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o 263 Q13 Reduce offensiveness of images by minimizing the display of explicit or profane sexual images on websites Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q14 Avoid conveying negative experiences associated with a condition and rather emphasize positive messaging Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q15 Use humour in a very positive and sensitive manner to lessen the fear and seriousness associated with a condition Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o 264 16 Avoid using language that seeks to blame, or cast moral judgment on people living with or infected by a condition Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q17 Include information that seeks to correct wrong perceptions and myths surrounding a particular condition to enable users to reject negative and inaccurate beliefs Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q18 Educate users about stigma by providing factual and plain language information that normalizes and de-stigmatizes sexual health-related conditions Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o 265 Q19 Use positive, credible, and diverse language and images Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q20 Use images and pictures of people from a variety of backgrounds who have experienced the condition Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q21 Use personal stories of renowned and relatable personalities who have experienced the condition Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o 266 Q22 Provide information on the fundamental rights of people affected by or living with the condition. e.g., Rights of persons living with HIV/AIDS Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q23 Use inclusive language such as “we, our, or us” and “you are not alone” Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q24 Use symbols of optimism, encouragement, and hope Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o 267 Q25 Expose people to a range of messaging that addresses different aspects of stigma Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q26 Avoid othering and stereotyping people with a sexual health-related condition Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Q27 Ensure a match between the system and the users’ culture Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o 268 Q28 Interventions for addressing men's/women health or sexual health-related conditions should be developed and delivered in partnership with living with the condition Strongly disagree Disagree Mildly disagree Neither agree nor disagree Mildly agree Agree Strongly agree Clarity o o o o o o o Usefulness o o o o o o o Additional ideas Please note that the principles you just appraised are only preliminary, obtained from a literature review. In this section, we would like you to suggest additional ideas or ways you think sexual health-related websites can be designed to address stigma. As an expert in the field, we strongly encourage you to provide us with your ideas on how to design destigmatizing sexual health-related websites. There are no right or wrong answers. Your suggestions and other participants’ suggestions will be brought to you for appraisal in round 2. Any suggestion with regards to how sexual health-related websites can help alleviate stigma is particularly welcomed. ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ ________________________________________________________________ 269 Appendix D - Study 4 Recruitment Materials Study 4 Initial Contact Letter & Participant Recruitment Message Initial Contact Letter Dear [Director of institution] My name is Abdul-Fatawu Abdulai, a Ph.D. candidate at UBC School of Nursing. We are conducting a study titled “Application of destigmatising design principles in sexual health-related technologies” and we are hoping to interview faculty, students and alumni from your institution who have some expertise in web design, web development, heuristic evaluation or informatics. The Principal Investigator for this study is Dr. Leanne Currie, an Associate Professor at UBC School of Nursing. The purpose of the study is to ascertain how web designers will apply a set of destigmatising design principles in a typical design situation. Below this email is the recruitment message that you can send to potential participants via your regular newsletter. Interested participants can complete a demographic checklist contained in the recruitment message and the study team will be in contact with them. Participants will receive CND $100 in appreciation for their time at the end of the study. If you have any questions, please contact me directly or the principal investigator Thanks in advance for your help. Kind regards, Abdul-Fatawu Abdulai Participant Recruitment Message: Designing for Reducing Stigma – Study 4 Researchers at UBC School of Nursing are looking for 15 web designers/developers to participate in a study titled: “Application of destigmatising design principles in sexual health-related technologies”. Participants would spend up to 90 minutes with the researcher in an interview (via Zoom) about how to apply a newly created set of design principles. If you are interested in this study, please follow this link https://ubc.ca1.qualtrics.com/jfe/form/SV_3vFQAdMZ7hg4Xhc to review the consent form and fill out a demographic survey and the researchers will contact you for the study. If you have a challenge with the link, please contact the researchers directly. You will receive CND $100 in appreciation for their time at the end of the study. All necessary steps will be taken to protect your confidentiality. If you have questions concerning the study, contact the student co-investigator (Abdul-Fatawu Abdulai) or the Principal Investigator 270 Appendix E - Study 4 Vignette with Interview Guide Study 4 Case scenario for interview Sam is living with HIV/AIDS and does not visit conventional health facilities for fear of stigma. The options for sexual health (https://www.optionsforsexualhealth.org/) were developed to provide evidence-based information and clinical services for people with sexual and reproductive health needs. Sam is now a regular visitor of the website but she still experiences sexual health-related stigma. When questioned further, she does not know what bothers her but the website, in general, bothers her. As a web designer, you have been tasked to redesign this website in a manner that addresses stigma. You have been presented with a set of 19 destigmatising design heuristics that will serve as a guide in your redesign of the website. We are interested in how you understand and apply these design heuristics in a manner that addresses stigma. We understand that some of the principles may not be directly related to your area of expertise. In that case, you should tell me what you will need to consider in order to get the principle applied in the website design. You will be asked the following questions in relation to each of the destigmatising design principles. Rather than specifically fixing something on the website, tell us the approach that you would take to apply these principles. Preamble on stigma: Stigma is defined as disapproval of, and discrimination against someone based on a perceived social characteristic that serves to distinguish them from other members of the society. There are two main types of stigma. Public/enacted stigma and internalized stigma. Public/enacted stigma is the outward disapproval/discrimination against someone by others while internalized stigma is the internal negative feelings that a person has due to possession of an attribute that she/he perceives as negative. Important: To familiarize yourself with the study materials, please explore the website (https://www.optionsforsexualhealth.org/) and go through the list of items on this table ahead of the interview date. The interview will be based on these items in the table. Table: List of destigmatising design principles ID # Destigmatising design principle P1 Provide participants with the ability for anonymous engagement P2 Encrypt websites that collect personal information to prevent unauthorized access to personal data P26 Avoid othering and stereotyping people with a sexual health-related condition P28 Interventions for addressing men's/women health or sexual health-related conditions should be developed and delivered in partnership with those living with the condition P4.1 Include videos/testimonials that center on people’s experiences with the condition, including stigma P6.1 Provide contact information of counsellors or other psychological supports P16.1 Avoid using language that has a tone of blame or judgment of people living with the condition P17.1 Include information that corrects myths about a condition, to enable users to get accurate information 271 Table: List of destigmatising design principles: Cont’d P18.1 Provide factual and plain language information that normalizes and de-stigmatises sexual health-related conditions P19.1 Selection of language/images should be done in consultation with the community to ensure diversity P20.1 Use images of people from diverse ethnicity, age, and gender identity who have experienced the condition P22.1 Consider including links to information on the fundamental rights of people affected by or living with the condition P23.1 Use inclusive language that is sensitive to the context of the condition e.g., partner instead of husband/wife, the person instead of woman/man P25.2 Include a range of evidence-based information that touches on different aspects of the condition NP36 Have clear, factual and neutral information NP30 Develop trust by providing options for different gender identities NP35 Ensure rigorous methods to know the target audience NP29.2 Provide trustworthy content by having a reference list to factual information NP34.2 Consider involving all those affected by the condition, not just people with condition (e.g., partners of people with the condition) Instructions 1. If you have access to a printer, kindly print the list of items ahead of the interview. Don’t worry if you don’t have access to an printer. You can share your screen with me 2. Open the website 3. Display the list of items and the website side by side on your screen 4. As you answer question 4, please reflect on the level of ease and difficulty in applying each principle Interview questions Note: You can decide to answer some of the questions by referring to website. That is, you can decide to show how you will actually apply such principle on the website. But this is not compulsory. 1. Prior to reading this definition of stigma, what was your understanding and experience with the concept? 2. Please describe your general impression regarding these design principles 3. What approach would you take to apply these principles a. Is this approach appropriate? b. What challenges do you envision in using this approach 4. From your perpective, what items would be easy/difficult to apply. Please rate each item on a scale of 1 (very easy to apply), 2 (not easy but also not very difficult to apply), 3 (very difficult to apply). a. For those that you indicated as easy to apply, can you describe the level of ease in applying these principles b. For those that are difficult to apply, you describe the level of difficulty in applying these principles 5. For items you indicated very difficult to apply 272 • Why do you think this is difficult to apply? • What would you consider to be able to apply this principle? o Why do you think it does not apply o Who should play a role in helping to address this principle? o What steps would you take with the team you are consulting for to ensure that this principle is addressed? 6. What type of education do you think you will need to apply these principles 7. Is there anything else you would like to add regarding your role in applying these principles
Thesis/Dissertation
eng
Nursing
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Currie, Leanne M.
Development and application of destigmatising design principles in sexual health-related technologies
Text