Medicine, Faculty of
Population and Public Health (SPPH), School of
DSpace
UBCV
Burt, Heather
Master of Science - MSc
University of British Columbia
Post-secondary students are typically assuming greater life responsibilities and independence, which can become emotionally overwhelming, resulting in a high prevalence of psychosocial issues. At a time when many young adults are moving away from their support networks and transitioning from pediatric to adult health care systems, there is a perceived lack of resources and numerous barriers to mental health treatment. A digital psychosocial self-assessment and guidance tool for post-secondary students (HEARTSMAP-U) was adapted to address this growing concern. HEARTSMAP-U has already undergone comprehensive adaptation and evaluation work among both a broad population of clinicians who support young adults in post-secondary education, as well as among post-secondary students themselves. Through a multiphasic study, this research aimed to 1) evaluate the inter-rater reliability of HEARTSMAP-U among young adults pursuing post-secondary education, as applied to a set of fictional cases, and 2) evaluate, on the same set of fictional cases, the scoring agreement between student and clinician assessors. In Phase 1 (n = 15), an iterative process was used to evaluate the fictional vignettes for comprehensiveness and clarity. Feedback received was reviewed and incorporated into the next version of the vignette and this process was conducted four times until all comments were positive and saturated. In Phase 2 (n = 34), HEARTSMAP-U’s inter-rater reliability was evaluated among post-secondary students. Students displayed substantial to near perfect inter-rater scoring agreement in applying HEARTSMAP-U to the finalized fictional clinical vignettes, with weighted kappas on tool domains ranging from 0.72 (Student Health; 95% CI: 0.71, 0.73) to 0.81 (Psychiatry; 95% CI: 0.80, 0.82). In Phase 3, a clinician applied HEARTSMAP-U to the same vignettes and a large proportion of scoring agreement was found between student and clinician responses (median range 0.82-0.85) on concern severity and on whether the individual described in the vignette had resources in place (97%). Together, these results indicate that HEARTSMAP-U can be consistently interpreted by young adults pursuing post-secondary education. These study results will add to HEARTSMAP-U’s ongoing evaluation in which HEARTSMAP-U’s predictive validity will be assessed.
UNIVERSAL PSYCHOSOCIAL SCREENING FOR POST-SECONDARY STUDENTS WITH HEARTSMAP-U: EVIDENCE FOR INTER-RATER RELIABILITY by Heather Burt B.A., Colgate University, 2019 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE in The Faculty of Graduate and Postdoctoral Studies (Population and Public Health) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) August 2021 © Heather Burt, 2021 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, a thesis entitled: UNIVERSAL PSYCHOSOCIAL SCREENING FOR POST-SECONDARY STUDENTS WITH HEARTSMAP-U: EVIDENCE FOR INTER-RATER RELIABILITY submitted by Heather Burt in partial fulfillment of the requirements for the degree of Master of Science in Population and Public Health Examining Committee: Quynh Doan, Associate Member, School of Population and Public Health, UBC Co-supervisor Anne Gadermann, Assistant Professor, School of Population and Public Health, UBC Co-supervisor Daniel Vigo, Assistant Professor, Psychiatry, UBC Committee Member Brenda Poon, Assistant Professor, School of Population and Public Health, UBC Additional Examiner iii Abstract Post-secondary students are typically assuming greater life responsibilities and independence, which can become emotionally overwhelming, resulting in a high prevalence of psychosocial issues. At a time when many young adults are moving away from their support networks and transitioning from pediatric to adult health care systems, there is a perceived lack of resources and numerous barriers to mental health treatment. A digital psychosocial self-assessment and guidance tool for post-secondary students (HEARTSMAP-U) was adapted to address this growing concern. HEARTSMAP-U has already undergone comprehensive adaptation and evaluation work among both a broad population of clinicians who support young adults in post-secondary education, as well as among post-secondary students themselves. Through a multiphasic study, this research aimed to 1) evaluate the inter-rater reliability of HEARTSMAP-U among young adults pursuing post-secondary education, as applied to a set of fictional cases, and 2) evaluate, on the same set of fictional cases, the scoring agreement between student and clinician assessors. In Phase 1 (n = 15), an iterative process was used to evaluate the fictional vignettes for comprehensiveness and clarity. Feedback received was reviewed and incorporated into the next version of the vignette and this process was conducted four times until all comments were positive and saturated. In Phase 2 (n = 34), HEARTSMAP-U’s inter-rater reliability was evaluated among post-secondary students. Students displayed substantial to near perfect inter-rater scoring agreement in applying HEARTSMAP-U to the finalized fictional clinical vignettes, with weighted kappas on tool domains ranging from 0.72 (Student Health; 95% CI: 0.71, 0.73) to 0.81 (Psychiatry; 95% CI: 0.80, 0.82). In Phase 3, a clinician applied HEARTSMAP-U to the same vignettes and a large proportion of scoring agreement was found between student and clinician responses (median range 0.82-0.85) on concern severity and on iv whether the individual described in the vignette had resources in place (97%). Together, these results indicate that HEARTSMAP-U can be consistently interpreted by young adults pursuing post-secondary education. These study results will add to HEARTSMAP-U’s ongoing evaluation in which HEARTSMAP-U’s predictive validity will be assessed. v Lay Summary The post-secondary years are an important period for an individual’s mental health and general health behavior development. Not only are young adults navigating through normal social, educational, occupational, and financial transitions to adulthood, they are often doing so will reduced family or institutional support. During this time, individuals are at risk of developing a mental health disorder. To address the large burden of mental health concerns in post-secondary students, the scarcity of resources, and the lack of streamlined self-assessment tools, HEARTSMAP-U was adapted. HEARTSMAP-U is a digital psychosocial self-assessment and guidance tool for post-secondary students. This study evaluates the evidence for inter-rater reliability of HEARTSMAP-U by engaging post-secondary students to evaluate fictional clinical vignettes. Findings indicate that HEARTSMAP-U exhibits good evidence of inter-rater reliability when used by young adults pursuing post-secondary education and that students’ responses are similar to a clinician’s assessment. These results will add to HEARTSMAP-U’s ongoing validation. vi Preface All studies were approved by the University of British Columbia’s Children’s and Women’s Research Ethics Board (H19-03347). The study is outlined in Chapter 2 and a version of this work has been submitted for publication. Dr. Quynh Doan, Dr. Anne Gadermann, Dr. Daniel Vigo, Dr. Marna Nelson, Punit Virk, and Yujie (Jenny) Zhang were co-authors and offered critical review during manuscript preparation. Under Dr. Doan’s supervision, I was responsible for: (i) designing and conducting the study; (ii) collecting and analyzing data; and (iii) writing the manuscript. vii Table of Contents Abstract ......................................................................................................................................... iii Lay Summary .................................................................................................................................v Preface ........................................................................................................................................... vi Table of Contents ........................................................................................................................ vii List of Tables ................................................................................................................................ xi List of Figures .............................................................................................................................. xii Acknowledgements .................................................................................................................... xiii Dedication ................................................................................................................................... xiv Chapter 1: Introduction ................................................................................................................1 1.1 Epidemiology Background ............................................................................................. 1 1.1.1 Prevalence of Psychosocial Concerns in Young Adults ......................................... 1 1.1.2 Impacts of Mental Health Challenges Later in Life................................................ 2 1.1.3 Mental Health Stressors Among Post-Secondary Students .................................... 3 1.1.4 Service Utilization Trends....................................................................................... 4 1.2 Barriers to Care ............................................................................................................... 5 1.2.1 Student Perceived Barriers ..................................................................................... 5 1.2.2 Institutional Challenges/Limitations ....................................................................... 7 1.3 Screening and Early Intervention.................................................................................. 10 1.3.1 Importance of Mental Health Screening ............................................................... 10 1.3.2 Existing Screening and Psychosocial Instruments ............................................... 12 1.3.3 Gaps to Existing Instruments ................................................................................ 16 1.4 Precursor Tools to HEARTSMAP-U............................................................................ 18 viii 1.4.1 HEARTSMAP ........................................................................................................ 18 1.4.2 MyHEARTSMAP ................................................................................................... 20 1.4.3 HEARTSMAP-U .................................................................................................... 20 1.5 HEARTSMAP-U Adaptation & User Testing .............................................................. 21 1.5.1 Clinician Evaluation and Feedback...................................................................... 21 1.5.2 End User Evaluation & Feedback ........................................................................ 22 1.6 Validity Evidence Framework ...................................................................................... 23 1.6.1 Validity .................................................................................................................. 23 1.6.2 Reliability .............................................................................................................. 26 1.6.3 Measures of Reliability ......................................................................................... 27 1.6.3.1 Internal Consistency ......................................................................................... 27 1.6.3.2 Test-retest .......................................................................................................... 28 1.6.3.3 Inter-Rater......................................................................................................... 28 1.7 Research Rationale and Thesis Objectives ................................................................... 29 1.7.1 Self-administered psychosocial evaluation ........................................................... 29 1.7.1.1 Phase 1: Development & Piloting Vignettes .................................................... 29 1.7.1.2 Phase 2: Inter-Rater Reliability ........................................................................ 30 1.7.1.3 Phase 3: Clinician Comparison ........................................................................ 30 Chapter 2: Evaluating Evidence of the Reliability of HEARTSMAP-U ................................31 2.1 Summary ....................................................................................................................... 31 2.2 Introduction ................................................................................................................... 32 2.3 Methods......................................................................................................................... 35 2.3.1 Study Design & Objectives ................................................................................... 35 ix 2.3.1.1 Phase 1: Development & Piloting Vignettes .................................................... 35 2.3.1.1.1 Vignette Development ................................................................................. 35 2.3.1.2 Phase 2: Inter-Rater Reliability ........................................................................ 37 2.3.1.3 Phase 3: Clinician Comparison ........................................................................ 37 2.3.2 Study Population and Setting ................................................................................ 37 2.3.3 Instrument ............................................................................................................. 38 2.3.4 Data Collection & Storage ................................................................................... 39 2.3.5 Study Procedures .................................................................................................. 39 2.3.5.1 Participant Training ......................................................................................... 40 2.3.5.2 Phase 1: Development & Piloting Vignettes .................................................... 40 2.3.5.3 Phase 2: Inter-Rater Reliability ........................................................................ 41 2.3.5.4 Phase 3: Clinician Comparison ........................................................................ 41 2.3.6 Power Analysis...................................................................................................... 41 2.3.7 Analytic Approach ................................................................................................ 42 2.3.7.1 Background of Inter-Rater Reliability .............................................................. 42 2.3.7.2 Kappa Analysis ................................................................................................. 43 2.3.7.3 Sensitivity Analysis............................................................................................ 44 2.3.7.4 Clinician Comparison ....................................................................................... 45 2.4 Results ........................................................................................................................... 46 2.4.1 Phase 1: Development & Piloting Vignettes ........................................................ 46 2.4.2 Phase 2: Inter-Rater Reliability ............................................................................ 48 2.4.3 Phase 3: Clinician Comparison ............................................................................ 49 2.5 Discussion ..................................................................................................................... 50 x 2.6 Conclusion .................................................................................................................... 53 Chapter 3: Discussion, Future Directions & Clinical Implications .........................................54 3.1 Overview ....................................................................................................................... 54 3.2 Key Findings ................................................................................................................. 54 3.3 Overall Strengths .......................................................................................................... 55 3.4 Overall Limitations ....................................................................................................... 56 3.5 Future Research ............................................................................................................ 57 3.6 Significance & Implications ......................................................................................... 58 3.7 Conclusory Remarks ..................................................................................................... 59 References .....................................................................................................................................60 Appendices ....................................................................................................................................81 Appendix A - HEARTSMAP-U (paper version) ...................................................................... 81 Appendix B - Example Fictional Vignette................................................................................ 91 xi List of Tables Table 2.1 Basic demographic information of study participants (N1 = 15, N2 = 34).................... 46 Table 2.2 Kappa statistics (95% confidence intervals) for each domain. ..................................... 49 Table 2.3 Median proportion of agreement (IQR) between student participants and study clinician. ........................................................................................................................................ 50 xii List of Figures Figure 2.1 Example of vignette description and corresponding HEARTSMAP-U assessment section. .......................................................................................................................................... 48 xiii Acknowledgements First off, I would like to express my deepest gratitude to my mentor and primary thesis supervisor Dr. Quynh Doan. Thank you for your endless encouragement, patience, guidance, and for helping me grow as a researcher over these past two years. I look up to you and I could not have completed this work without you. Second, I would like to thank Dr. Anne Gadermann for her guidance, insight, questions and perspectives that challenged me to push my limits, as well as Dr. Daniel Vigo for joining my committee and providing valuable support to elevate this project. A big thank you to Karly Stillwell for her administrative support in coordinating this project and moral support in answering my endless emails throughout my time with the Doan lab. In addition, I would like to thank Takuro Ishikawa for his time and expertise in statistical analyses, without whom I would still be struggling with R. Next, I would like to thank Punit Virk. I could not have completed this project without your guidance and support. Thank you for answering my stress calls/emails/text messages at various times of the day and for being a researcher for whom I can learn from. You paved the path for me and made my time here so much more enjoyable. A special thank you to my colleagues and friends in the Doan lab and SPPH for helping me through these past two years. Thank you for keeping me going and for creating a memorable and productive work environment. Finally, a special thank you is owed to my parents, who have supported me throughout my years of education, both morally and financially. Thank you for your endless love, for always supporting my dreams, and for forgiving my infrequent visits home. Thank you. xiv Dedication To my mom and dad. 1 Chapter 1: Introduction 1.1 Epidemiology Background 1.1.1 Prevalence of Psychosocial Concerns in Young Adults In the next two decades, it is estimated that Canada’s leading cause of disability will be mental health related,1 and 75% of lifetime cases of mental health issues are diagnosed by the age of 24.2 The latest Statistics Canada census data on mental health indicators was collected in 2012.3 At that time, in young adults aged 15 to 24 years, 30.7% (N = 1,344,221) self-reported that they had been diagnosed with a mental or substance disorder by a health professional in their lifetime, 18.5% (N = 806,427) of which were within the previous 12 months. Specifically, 12.8% (N = 565,175) had been diagnosed with a mood disorder; 10.7% (N = 474,311) with a major depressive episode; 4.0% (N = 177,597) with bipolar disorder; 6.3% (N = 280,516) with generalized anxiety disorder; 21.3% (N = 937,090) with a substance use disorder; and 14.1% (N = 626,976) with suicidal thoughts in their lifetime.3 Post-secondary students, who on average range from 18 to 24 years old, make up a significant portion of young adults in Canada (over 40%), and enrollment rates are steadily increasing.4–6 Post-secondary students’ mental health is a growing concern. Birth cohort comparison studies show that, over the past several decades, college students have scored increasingly higher on clinical scales of depression, hypomania, hypochondriasis, paranoia, and schizophrenia.7 In recent years, there has been an increase in reported mental health concerns in college student populations,8 with diagnoses increasing from 22% in 2007 to 36% in 2017.9 Unfortunately, the statistics around clinically diagnosed conditions are just the beginning when evaluating the burden of mental health concerns in this vulnerable population. Considering 2 and identifying young people with clinical sub-threshold mental health symptoms is also critical in tackling this problem and is the focus of my thesis. According to the General Health Questionnaire,10 nearly one-third of Canadian undergraduate students reported experiencing elevated psychological distress, and 2009 prevalence data from six Ontario post-secondary institutions estimates that more than 80% of students feel overwhelmed or exhausted.11 These findings are generally consistent with a 2016 study that found that more than half of the students experienced generalized anxiety, over 20% had significant depressive symptoms, and more than 10% considered suicide.12 The Ontario College Health Association found that post-secondary students are more than twice as likely to report mental challenges and elevated distress than non-university youth.13 According to the Spring 2020 American College Health Association-National College Health Assessment, approximately 30% of post-secondary students report receiving psychological or mental health services in the last 12 months, with 27.4% screening positive for anxiety, 22.3% for depression, and 9.4% indicating that they had intentionally self-harmed.14 In this study, approximately 75% of the students surveyed rated their level of stress as either moderate or severe,14 figures that are higher than those reported for the general population.15,16 1.1.2 Impacts of Mental Health Challenges Later in Life Mental health concerns in adolescence and in the young adult years is associated with a number of negative experiences, both over one’s lifetime17 as well as cross-generationally.18 When mental health conditions go untreated, health issues can arise. Mental health conditions have been shown to be associated with obesity,19 substance misuse,20 smoking,21,22 and even suicide.23 Poor mental health in adolescence is further associated with negative outcomes in 3 adulthood, including poor educational achievement, lower levels of employment,24 low earnings, antisocial behaviour, marital problems, and criminal activity.22,25,26 Finally, the economic burden of mental health challenges in Canada is estimated at $51 billion per year. This includes health care costs, lost productivity, and reductions in health-related quality of life.27,28 1.1.3 Mental Health Stressors Among Post-Secondary Students Why are post-secondary students more than twice as likely to report mental health concerns and elevated psychosocial distress than non-university youth? The post-secondary years represent an important period for mental health and general health behaviours. Not only are young adults navigating through normal social, educational, occupational, and financial transitions to adulthood, they are often doing so with reduced family or institutional support.29 The average undergraduate student is between 18-24 years old and has left many familiar sources of stability behind. They are assuming greater life responsibilities and independence that can become emotionally overwhelming, resulting in a high prevalence of psychosocial issues.30 For example, many students may work part-time or full-time to support their studies. Expectations by both the student and parents for academic success, similar to that which may have been previously achieved, adds further pressure to an already stressful situation.13 Factors including lack of sleep and poor nutrition can also significantly interfere with a student’s ability to cope with their new circumstances.13 In one study of students who screened below the threshold for anxiety and depression at entry to university, 9% were above the threshold for depression and 20% were above the threshold for anxiety 18 months into their course.31 International students and recent immigrants may be even more vulnerable to mental health concerns as they are more likely to be living further away from family, loved ones and 4 other supports when compared with their domestic peers. Differences in language and cultural norms may also impede on international students’ ability or willingness to seek out and receive help.13 1.1.4 Service Utilization Trends With the added mental health stressors related to the transition into adult life, post-secondary institutions have seen rising mental health diagnoses in the last decade.9 This, compounded with the perceived decline in stigma associated with mental health, have contributed to a greater demand in mental health services and receptiveness towards prevention of mental health concerns.9 It is well established by epidemiological data that mental health problems are highly prevalent among post-secondary students.32 Equally important, the data on resource utilization shows a steady increase in the number of students requesting mental health care over time.9 Xiao et al., found an increase in all reported aspects of counseling centre utilizations from 2009 to 2014, including the number of scheduled appointments, attended appointments, and students seeking services, with the growth in the utilization numbers 4-6 times the rate of institutional growth.33 Over this time period, 88% of the 340 university and college counselling centres included in the study reported an increase in students seeking treatment.33 These findings are generally consistent with national survey data from the Healthy Minds Study,34 which documented population-level trends in mental health service utilization by college students from 2007 to 2017.9 In this national sample of college students, increases across all measures of service utilization, including past-year treatment, therapy or counselling, psychiatric medication 5 use, and lifetime diagnosis were reported. In addition, the study reported that the most common location for receiving mental health treatment was on the institution’s campus.9 Studies have also found that among participating students, White identifying students were substantially more likely to seek mental health treatment compared with Asian, Black and Hispanic identifying students.35 In addition, treatment use was lower among students who identified as international, religious or heterosexual.35 Eisenberg et al., also found that service use was higher in smaller and private institutions, which may be related to a greater availability of resources.35 These demographic distributions in resource users indicate that the available mental health resources may not be accommodating or accessible to all students and this may be due to a number of student-perceived or institutional barriers. 1.2 Barriers to Care 1.2.1 Student Perceived Barriers In a time when many young adults are moving away from their support networks and transitioning from pediatric to adult health care systems, there is a perceived lack of resources and numerous barriers to mental health treatment. Peer-pressure, perceived stigma, inadequate mental health literacy, lack of understanding of campus counselling services, and a negative previous experiences are all student-perceived barriers to accessing mental health services at a post-secondary institution.36,37 The stigma surrounding mental health concerns can cause students to stay silent about their struggles.38 Stereotypes such as being “weak” or “dangerous” can account for much of the stigma attached to mental health conditions. Numerous studies report high levels of public stigma related to mental health issues.38 Stigma was found to be a “pervasive barrier” to effective 6 mental health treatment by two U.S. federal initiatives: the Surgeon General’s Report on Mental Health;39 and the President’s New Freedom Commission on Mental Health.40 Eisenberg et al. examined 5,555 students from 13 universities and found that the stigma of mental health challenges perceived from peers and others was more prevalent than personal stigma. However, personal stigma was more prevalent than peer stigma among students who identified as male, young, Asian, religious, or from a poor family and was negatively correlated with the likelihood of those students seeking professional support for mental health problems.38 The use of counselling services has also been associated with stigma. Studies have found that individuals using counselling services were perceived negatively compared to individuals who did not receive counselling.41 The perceived stigma associated with mental health concerns may be lessening,42–44 but work still needs to be done to continue this trajectory and ensure safe and comfortable access for all. With limited mental health education, students may not recognize their need for treatment, believing that symptoms of clinical depression and anxiety are typical of college life.45 Eisenberg et al. found that a commonly reported barrier by students was that they “don’t have enough time” to seek care, which suggests that students may see mental health treatment as less important or less urgent as compared to other priorities such as coursework or social activities.35 Another study looking at help-seeking behaviours found that the most commonly reported reason for a student not to seek mental health services was the perception that their concerns were too minor or transient,46 which is consistent with other studies.45,47 The association between the perceived need for help and the use of mental health treatments was supported in a study that found that 50% of students with a mental health concern who perceived a need for help received treatment, whereas only 11% of those with a mental health concern who did not perceive a need 7 for help received treatment.35 The emphasis on self-reliance compounds this issue. When students fail to reach internal or external expectations in their new academic environment the disappointment that may follow can hinder a student’s willingness to seek support.38,48 As young people progress through adolescence, they have a growing need for autonomy and independence and believe they should be able to handle problems themselves.49,50 As a result of both prioritization and the desire to be self-reliant, many students do not get the help they need. Finally, when care is sought, the uncertainty around available and appropriate supports can further hinder student help-seeking.11 On the one hand, while services exist, they may not be promoted efficiently and openly, and accessibility to these services needs to be improved. On the other hand, the timely access to available resources may be the issue. Interviews with post-secondary students show consistent discussion over long wait times as a major barrier to accessing services in a timely and meaningful way.11 In addition, some students have found that even when they pursued services, their demands are not taken seriously without a clinical psychiatric or psychological diagnosis.11,51 Compounding this issue is the perceived lack of cultural sensitivity and staff training to address the unique needs of diverse students. Studies have reported that service use is notably infrequent among students from lower socioeconomic backgrounds, international students and Asian-American students.52 In addition to these student-perceived barriers, there are a number of institutional challenges that need to be addressed. 1.2.2 Institutional Challenges/Limitations The rising statistics surrounding mental health conditions have led to a call for an increase in mental health support initiatives to correct negative perceptions of mental health challenges and, more importantly, to provide services to treat these challenges. Changes at both 8 the national and provincial levels have been made to post-secondary institutions in an attempt to accommodate the mental health needs of students.11 However, post-secondary institutions face many challenges when attempting to prevent, identify and treat mental health concerns on campus. Many institutions have some form of on-campus counselling centre/services, consistent with the increasing demand for these services.53 Nevertheless, resources are insufficient, both in terms of financial investment in mental health projects and in human resource capacity. The capacity issues include insufficient number of therapists11,13, discontinuity of care, and inconsistent communication between providers.54 These all may result in a reactive rather than proactive or preventative approach to mental health programs.13,55 Availability of resources and the gaps within those provided are inextricably linked with funding methods. In Ontario, for example, fund allocation in certain departments is based on institutional enrolment numbers and the number of students using their services.13 Funding for counselling services may come from the institution, from student fees, from third party insurers or from a mixture of these sources. Despite the growing need for services, if post-secondary institutions face ongoing financial constraints, counselling services are subject to budget cuts. In this event, the service becomes more reactive, servicing more urgent cases first, creating longer wait lists, having shorter periods of time for appointments, etc. As a result, mental health concerns can become more severe, before students receive help. Retaining doctors and psychiatrists has also become a significant issue for many post-secondary institutions, in part due to the larger financial incentives paid to them for being affiliated with a hospital.13 In a 2007 survey, members of the American College Counselling Association were asked about the increasing demand for services, and 96% of members responded by saying caseload was an increasing concern and, among those, the most common response was a need for more 9 employees.56 Institutional mental health funding for both physical services (i.e. buildings) as well as personnel (i.e. clinicians) is urgently needed. This strain on mental health services is even more evident at smaller institutions, which tend to have fewer staff and larger financial constraints.57 Funding needs to be restructured in order to adequately address student needs. Like many health care systems, mental health care within post-secondary institutions tends to be reactive, with resources focusing on managing problems as they arise, despite the fact that prevention and early treatment are known to be more effective.13 As mentioned above, this outlook on treatment is typically the result of budget cuts or a general lack of funding in the area. Overall, health promotion is usually minimally funded with health education being the primary focus of this budget, rather than resources and supports.13 While health education is important, it is not enough to address all aspects of mental health. The focus on “reaction” rather than “prevention” also means that campus mental health services focus on short-term therapy, and long-term therapy is generally not provided.55 Off-site referral for longer-term therapy may be associated with an extra financial burden, which may be particularly problematic for students with limited income or insurance.58 Formal follow-up procedures for those requiring long-term (generally off campus) therapy are lacking and needed. Although a number of mental health services are located on university campuses, the degree of coordination and collaboration between services varies due to a number of barriers.13 Physical distance between resources prevents the spontaneous communication between health care providers. Communication and trust are necessary when multiple professionals share the responsibility of helping those with mental health concerns. Students are also constantly transitioning into or out of campus services. Campus providers need access to information from previous mental health providers in order to provide the best care to students. Fragmented 10 services/supports with insufficient continuity of care can lead to students having to repeat their stories multiple times, which can worsen their emotional distress.54 The lack of efficient communication between services is a major challenge to providing the best care. These institutional barriers are extensive and complicated. However, despite the challenges in constructing comprehensive strategies for post-secondary institutions, guidelines have started to emerge.59 Certain actions (beyond funding) can be implemented to improve counselling, such as a focus on strength and recovery, peer counselling, offering individual and group therapy, and a streamlined referral process to other academic services or local hospitals. Student-to-student or peer health educator programs have also been shown to extend the reach of health services.60 1.3 Screening and Early Intervention 1.3.1 Importance of Mental Health Screening As discussed, these institutional challenges, such as a lack of personnel and/or funding, lead to a reactionary rather than preventative approach to mental health care. These systematic barriers then force services to create long wait lists and provide care for shorter periods of time. Reports have stated that wait times can be anywhere from weeks to months, depending on the institution and time of year.13 These institutional limitations are heightened or exacerbated by a lack of recognition of treatment need on the student’s behalf. As mentioned, many students do not recognize their need for treatment, believing that their symptoms are typical of college life or are too minor to address.45,46 As a result, symptoms often worsen to a point of crisis before students attempt to access care.61,62 Instead of getting immediate attention, students often face long wait lists which allows their concerns to progress and worsen. Successful mental health 11 treatment is a matter of timing. Timing in the recognition of a concern and integrating the time it takes to get access to care. To work with these systematic institutional limitations, students need to recognize mental health concerns earlier on in order to access care before they are at a point of crisis. Integrating regular or scheduled psychosocial screening interventions within post-secondary health systems can help improve students’ recognition of mental health concerns earlier on and support the timely connectivity with care, which in turn can lead to improved outcomes.63 As mentioned earlier, psychiatric disorders in childhood and adolescence have been shown to be associated with the development of skills, and substantially increase the likelihood of experiencing health-related, economic-related and social-related disadvantages in adulthood.64–67 However, more than half of all adult patients with mental health challenges retrospectively report that their first symptoms occurred during childhood and adolescence and that there was a chronic progression into adulthood.68,69 Thus, preventative measures, such as mental health screening, have the immense potential in allowing early detection and more effective intervention, thereby lessening the burden of mental health conditions, and preventing or minimizing their progression.63,70 A vital criterion for assessing the utility of screening is the availability of facilities to conduct the screening and provide an intervention.63 Digital technology provides a way to increase screening availability and also increase the access to evidence-based interventions.71 By 2020, there was expected to be approximately 6.1 billion mobile phone users and a net addition of 3 billion smartphone users globally.72 As a result, online health platforms offer a powerful and unique method for monitoring and assessing one’s mental health. There are many advantages to online tools, including: constant availability, greater access, equity of mental health resources, 12 anonymity, tailored content, and increased service capability and efficiency.73 Overall, individuals may feel less embarrassed or self-conscious about reporting sensitive or potentially stigmatizing information during a online assessment than during a face-to-face interview. This can allow for more accurate estimates of mental health behaviors and symptom severity.74 Screening efforts can then be linked to strategies designed to increase the likelihood that a participant will accept a referral and initiate further assessment or treatment. 1.3.2 Existing Screening and Psychosocial Instruments Focusing specifically on mental health screening and assessment tools, below is a summary of a few of the most commonly utilized validated tools. The Patient Health Questionnaire (PHQ) is a free, publicly available, 3-page self-administered questionnaire, used for making criteria-based diagnoses of mental disorders.75 The PHQ assesses 8 diagnoses.76 The PHQ-9 is a 9-item self-report measure designed to assess the presence of DSM-IV symptom criteria for major depressive disorder. For each of the nine-items, participants rate how often they may have experienced the depression symptoms over the last 2 weeks using the following scale: 0 (“not at all”), 1 (“several days”), 2 (“more than half the days”), and 3 (“nearly every day”). The scores are summed and range from 0-27, with higher scores indicating greater severity. A question at the end of the instrument assesses the functional impairment related to the symptoms assessed.77 Thus, in addition to making criteria-based diagnoses of depressive disorders, the PHQ-9 is also shows evidence of reliability and validity for depression severity.76 The evidence for diagnostic validity of the PHQ was originally developed and conducted in the clinical primary care setting. Higher scores on the PHQ-9 were found to be associated with worse psychological functioning, symptom-related difficulties, and 13 health care utilization.76 It also demonstrated good evidence for internal consistency with Cronbach’s alpha values upward of 0.8076 and has been translated and shown to have evidence for validity in diverse cultural groups.78–80 Although initially developed for use in the primary care clinical setting, a 2018 study looked at the utility of the PHQ-9 as a brief depression assessment tool in the college setting.77 Authors examined the psychometric properties (e.g., score reliability, validity, and factor structure) of the PHQ-9 with racially diverse college students using data drawn from the 2007 National Study of Living Learning Programs (NSLLP).81 Overall, their results showed evidence for the validity and reliability of the PHQ-9 as an equivalent measure of depression for both female and male college students. The College Health Questionnaire (CHQ) is an 18-item questionnaire that was developed to supplement standard screening instruments and in particular augment the PHQ. It assesses domains of college-student distress not addressed by the PHQ, including “weight loss, recreational drug use, sexual identity, risky sexual behavior, and common psychosocial college-adjustment issues”.82 It also includes an 11-item subscale, called the College Maladjustment Index which comprises the sections focusing on risky sexual behaviors and common psychosocial college-adjustment issues. Low to moderate inter-item correlations were exhibited within these two subscales of the College Maladjustment Index and evidence for concurrent validity was shown by moderate to high positive correlation between the College Maladjustment Index and the PHQ. Preliminary evaluations of this questionnaire demonstrate that it may be useful in identifying problems in the college student population that are not captured by the PHQ.82 Similarly, the Symptoms and Assets Screening Scale (SASS) was developed as a screening measure for use among the general college population. It is a 34-item measure designed to identify students at-risk for depression, anxiety, substance abuse, eating disorders, 14 and general well-being.83 Psychometric testing showed evidence for the SASS’s internal consistency, test re-test reliability, and initial criterion validity compared against commonly used mental health measures.83 The Generalized Anxiety Disorder scale (GAD-7) is a widely used seven-item scale designed to screen for generalized anxiety disorder (GAD) according to DSM-IV criteria.84 Similar to the PHQ-9, answer choices range from 0 (“not at all”) to 3 (“nearly every day”). The scale score is determined by summing individual item scores with a total ranging from 0 to 21.85 The tool was originally developed and studied in primary care clinics in the United States, however, the psychometric properties of GAD-7 have been reported to be good to excellent in an array of studies, with different populations and from different countries.86 Original evidence for tool validity was established by comparing GAD self-report diagnoses with independent diagnoses made by mental health professionals.84 Despite the prevalence of anxiety in young adults, very few of the original psychometric studies of the GAD questionnaire looked at this age group.87,88 As a result, a 2020 study conducted a secondary analysis of three large cross-sectional studies to assess the psychometric properties of GAD-7 in young adults enrolled in college in the United States.85 Overall, they found strong evidence for reliability and construct validity of the GAD-7 in young adults enrolled in college. The Alcohol Use Disorder Identification Test (AUDIT) was first developed in 1973 as a screening instrument for hazardous and harmful alcohol consumption in primary health care settings.89 It is a 10-item questionnaire that covers concerns related to alcohol consumption, drinking behavior, and alcohol-related problems. Responses are scored from 0 (“never”) to 4 (“daily”) with a sum total score ranging from 0 to 40 and higher scores indicating greater severity. A unique feature of AUDIT is that it was derived from a cross-national data set 15 involving six countries.89 Original psychometric testing results indicated high sensitivity and specificity with relatively little variation from country to country.89 In addition, a more recent review conducted in 2007, concluded that the AUDIT is an effective screening tool with adults across a variety of clinical settings.90 The use of AUDIT in young adult and college populations also suggests that AUDIT is a promising tool.91,92 A college study was conducted in a northeastern university and found that AUDIT performed well in detecting at-risk drinking, with a sensitivity above 80% and specificity just under 80%. These values were consistent with the median levels seen in adult samples.90 Finally, the Suicide Behaviors Questionnaire-Revised (SBQ-R) is a 4-item questionnaire used to identify at-risk individuals and specific risk behaviors for suicide.93 It assesses an individual’s lifetime suicide ideation and/or suicide attempts (scale from 1 to 4), the frequency of suicidal ideation over the past 12 months (scale 1 to 5), the threat of suicide attempts (scale 1 to 3), and evaluates the self-reported likelihood of suicidal behavior in the future (scale 0 to 6); with a total score ranging from 3 to 18.93 Osman et al., examined the evidence for reliability and validity of SBQ-R in a sample that included psychiatric inpatient adolescents, high school students, psychiatric inpatient adults, and undergraduate students.94 With the goal to enhance the generalizability of the findings, they included participants from both clinical and nonclinical settings. The psychometric properties (i.e. sensitivity and specificity) of SBQ-R were found to be at least 0.80 in all participants populations, with a specific sensitivity of 0.93 and a specificity of 0.95 for the undergraduate cohort.94 16 1.3.3 Gaps to Existing Instruments There are numerous e-mental health tools in addition to the instruments just described. Notwithstanding the potential of existing tools, there are gaps to their ultimate implementation for universal psychosocial screening in a post-secondary education institution. First, apart from instruments used in research and in clinical settings (e.g., the ones described above), online tools that are commonly found through a web search are rarely rigorously tested and evaluated before they are accessible to the public. Second, few validated tools were specifically designed in a student-centered manner or validated in that population. Third, many tools target only one mental health area of concern, and/or are somewhat pathologizing in that regard. Finally, few screening tools direct students towards relevant resources after assessment, therefore limiting their utility in the context of universal screening. Despite the large number of available online mental health apps, tools, and platforms, both for mental health assessment and/or intervention, many lack a rigorous evidence base before they are accessible to the public. A 2013 review looking at mobile mental health apps for the most prevalent conditions, as identified by the World Health Organization, found that, among the 1,536 apps identified for depression, only 32 had been studied in published articles.95 A separate 2013 review was conducted to examine the efficacy of mobile mental health apps for all ages.96 They included studies that examined the effects of mental health apps with a pre- to post-test design or that were compared with a control group. The review identified eight papers which described only five apps that met the inclusion criteria. The study’s discussion repeatedly highlighted how research has lagged behind in the field of app development.96 A 2018 review,61 conducted to update this previous 2013 review, found that clinically validated mobile apps for mental health have the potential to be effective in assessing, monitoring or improving symptoms 17 of certain mental disorders. However, this study also stressed the notable difference in the number of commercially available apps and the number of clinically validated apps. This review found that only 14 out of approximately 100 studies reportedly utilized an app which had clinically validated evidence of its effectiveness.61 Second, as summarized above, there are mental health tools that are systematically evaluated. However, few tools are developed with the involvement of mental health experts and/or targeted end-users in the development process;97,98 in this case, college students. Existing psychosocial tools may be subsequently evaluated in a student population, but they are not originally developed for this population. This can reduce the feasibility and utility of these tools in the college student population. Tools may exhibit evidence of validity, but may omit constructs which are specifically relevant to student psychosocial well-being (e.g., relationships, finances, academics).99 Ultimately, screening tools may be more acceptable and utilized in a community if members of such community are involved in the design and testing from the beginning. Third, many tools target only one mental health area of concern, and/or are somewhat pathologizing in that regard.98,100 The need for a “clinical diagnosis” strengthens the stereotypes that lead to stigma. Stigma can significantly impair the quality of life for people living with mental health conditions. The social consequences or shame that results from stigma associated with the label of “mentally ill” can take away opportunities enjoyed by others.100 It can also prevent individuals who might otherwise benefit from clinical services from pursuing treatment in an effort to avoid the label. Research has suggested that many people choose not to pursue mental health services because they do not want to be labeled as a “mental patient”.100 In addition, systematic reviews looking at online psychometric instruments state substantial gaps in 18 the range of mental health conditions that are addressed by these tools.101 These reviews have shown that while there may be an abundance of online instruments for depressive symptoms for example, there is a shortage of instruments for other disorders.101 College screening interventions are often restricted to depression and anxiety, as these are the most common mental health disorders in the student population.102–105 However, screening across a narrow range of psychosocial constructs may not adequately capture and assess a student’s concerns across social factors that may be contributing to, or are the results from, these psychosocial issues (e.g., education, relationships, or housing). Broader screening allows health care providers to better understand a student’s needs and connect them with more customized resources. Finally, few screening tools direct students towards relevant resources after assessment, therefore limiting their utility in the context of universal screening. The intention of students to engage with mental health assessment tools and to seek care is contingent on their perception that these tools are useful. Supplementing screening results with customized recommendations may contribute to the perceived utility of screening and can effectively improve mental health outcomes.106–108 In addition, including concern-specific resources may supplement limited and/or strained campus mental health services, as students can access them in lieu of or while they are waiting to meet with a health care provider. 1.4 Precursor Tools to HEARTSMAP-U 1.4.1 HEARTSMAP Mental health conditions also represent a substantial burden of illness in the pediatric population, affecting approximately 10 to 20% of Canadian children and adolescents.109,110 Pediatric emergency departments (PEDs) are seeing growing numbers of youth requiring mental 19 health care. For example, at BC Children’s Hospital, from 2003-2012, the annual number of mental health-related visits nearly doubled, representing an average increase of 7% per year, far surpassing the 2.5% annual rise in total PED visit volume.3 To address the rising number of pediatric mental health-related visits to the emergency department, the Doan research lab developed a tool called HEARTSMAP.111–113 HEARTSMAP is a rapid psychosocial assessment and management tool for emergency department clinicians, which separates psychiatric, social, and behavioural concerns, allowing for multi-dimensional screening across a range of clinically and theoretically justified psychosocial areas. HEARTSMAP guides clinicians to score the severity and urgency of concerns across 10 psychosocial areas: Home, Education, Alcohol & drugs, Relationships & bullying, Thoughts & anxiety, Safety, Sexual health, Mood, Abuse, and Professional resources, and aids clinician decision-making during their patient interview. For each psychosocial section, concern severity is measured on a 4-point Likert-type scale from 0, indicating “no concern”, to 3, indicating “severe concern”. In addition, for each section, the level of support already accessed is assessed on a separate 2-point dichotomous scale (yes or no). These 10 sections then map to five general domains: Social, Functional, Youth health, Abuse, and Psychiatry. Score inputs feed into a built-in algorithm that triggers service recommendations and time frames in which they should be accessed. HEARTSMAP has been shown to have strong evidence for inter-rater reliability and also evidence for good predictive validity.112,111 It is currently used throughout the province of British Columbia for the assessment of children and youth with an emergency mental health presentation. 20 1.4.2 MyHEARTSMAP As discussed, the early recognition of mental health conditions is associated with the timely access to services and thus improved health outcomes.63 While the American Academy of Pediatrics has recommended universal screening for mental health conditions among youth114, this has yet to be effectively implemented. To help enable mental health screening in the emergency department, the Doan research lab modified and adapted HEARTSMAP to create a self-administered assessment tool for youth and their parents, called MyHEARTSMAP. Youth and/or their parent or guardian are guided through prompting questions to score the severity of the youth’s concerns across the same 10 sections as HEARTSMAP. Severity is recorded on the same scale from 0 to 3, and, as with HEARTSMAP, the user also indicates whether resources are already in place to address each area of concern. This allows the tool to assess the urgency of the youth’s needs. These sections map to the five general domains and a back algorithm, based on the electronically computed individual areas and domain composite scores, generates targeted recommendations for mental health services. Recommendations are specific to the domain, the acuity of the concern, and whether resources are already in place.115 MyHEARTSMAP has been shown to have evidence for good inter-rater relatability and utility.116,117 1.4.3 HEARTSMAP-U In response to the increasing numbers of post-secondary mental health diagnoses and to extend the clinical HEARTSMAP tool, the Doan research lab established HEARTSMAP-U. HEARTSMAP-U is a digital psychosocial self-assessment and guidance tool designed specifically for young adults pursuing post-secondary education. HEARTSMAP-U is an adaptation, building on the HEARTSMAP assessment and management tool designed for use by 21 clinicians in pediatric emergency care settings, and MyHEARTSMAP, its self-administered version. HEARTMSAP-U assess 10 similar psychosocial areas from that of HEARTSMAP and MyHEARTSMAP, which are situated as universal needs under a human development framework and socio-ecological model. Individuals strive towards these universal needs in an ordered manner (Maslow’s Hierarchy of Needs). Individuals start from psychological and safety needs (e.g., Housing & Material Security, Safety) and work towards self-actualization, or self-fulfillment needs (e.g., Education).118,119 Under a socio-ecological model, these areas demonstrate how mental well-being is influenced through the interaction of not only the individual (e.g. Mood, Thoughts & anxiety), but also interpersonal (e.g., Relationships), organizational (e.g., Education & Activities), and community factors (e.g., Professional resources).120–122 HEARTSMAP-U incorporates recommendations and guiding frameworks proposed for e-mental health assessment applications,123 and is currently under rigorous evaluation, of which my thesis is one part. Completed preliminary evaluations of HEARTSMAP-U are explained below. A paper version of HEARTSMAP-U can be found in Appendix A. 1.5 HEARTSMAP-U Adaptation & User Testing 1.5.1 Clinician Evaluation and Feedback HEARTSMAP-U’s comprehensive adaptation and evaluation work was divided into two sub-phases, the first being with clinicians (1a), and the second with end users (1b). The primary aim of phase 1a, which was completed prior to my arrival at the Doan research lab, was to 22 evaluate HEARTSMAP-U’s clinical content in a broader population of clinicians that support young adults in post-secondary education. Mental health providers (n = 28) from large Canadian post-secondary institutions took part in clinically evaluating HEARTSMAP-U. They applied the tool to fictional vignettes which they designed from their own clinical practice with students experiencing a range of psychosocial related concerns. The preliminary analyses showed that a majority of clinicians felt that, across HEARTSMAP-U’s tool sections, the guiding questions and scoring criteria captured important issues and the severity range which they saw in their practice.124 These healthcare professionals also found that HEARTSMAP-U triggered recommendations consistent with or slightly more conservative than students’ level of need. Feedback was grouped into three major categories: “using strength-based language”, “exhaustive coverage of issues/stressors”, and “adding contextualizing details to support screening”. With this feedback, the tool’s language was reframed for ease of interpretation and tool sections were refined to flag broad psychosocial concern and support follow-up with diagnostic evaluation. 1.5.2 End User Evaluation & Feedback Phase 1b of HEARTSMAP-U’s development work was conducted with end users. I was a research assistant on this phase of evaluation. The aims of Phase 1b were three-fold: first, to determine how acceptable psychosocial self-assessment and resource connection is among post-secondary students and what factors contribute to or detract from tool acceptability; second, to modify the presentation and language of the initial HEARTSMAP-U tool in accordance with the lived experiences of a diverse student population to ensure interpretability, relatability and 23 meaningfulness; and third, to work in partnership with students to develop an effective and easy-to-navigate visual interface for the tool. Diverse UBC students (i.e., age/gender/ethnicity/sexual orientation/area of study/housing/lived experiences, n = 54) participated in six focus groups. In these focus groups, we gauged student feedback on the understandability and relevance of HEARTSMAP-U’s content and language. Overall, students provided positive reactions and felt that HEARTSMAP-U would cater to student-specific needs, help destigmatize discussing mental health, and provide a less invasive alternative to in-person assessment. Student feedback generally focused on two major categories: “lay, non-judgemental language” and “scoring gradient fluidity”. We iteratively incorporated student feedback between sessions and received mostly affirming reactions in our fifth and sixth sessions, supporting the prior changes.124 1.6 Validity Evidence Framework 1.6.1 Validity Validity is crucial to the interpretations and decisions we make surrounding test scores. Strong evidence for validity increases our confidence regarding our screening and the subsequent decisions we make. Alternatively, low evidence for validity may lead to misinformed decisions. The definition and concept of validity has evolved over the past century. One of the most common theoretical frameworks was developed by Samuel Messick and describes a unified view of validity, which is contrary to the traditional view which suggested that there were three distinct types of validity: content, criterion, and construct. The theoretical framework developed by Messick emphasizes a unified lens of validity that integrates considerations of content, criteria, and consequences into one framework, 24 encompassing six aspects of validity. These six aspects: content, substantive, structural, generalizability, external, and consequential aspects of construct validity, function as general validity standards for educational and psychological measurements.125 Messick defines validity as “an overall evaluative judgement of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions on the basis of test scores or other modes of assessment”.125 Messick states that validity is not a property of the test or assessment, but rather the meaning and interpretation of the test scores. Therefore, validation is a continuing process dependent on the context of the test. Overall, Messick states that “the process of construct validation evolves from these multiple sources of evidence [the six aspects] a mosaic of convergent and discriminant findings supportive of score meaning.”125 This unified perspective is also endorsed by the Standards for Educational and Psychological Testing (commonly referred to as the Standards).126 The Standards is a set of testing principles developed jointly by the American Educational Research Association (AERA), American Psychological Association (APA) and the National Council on Measurement in Education (NCME).126 The intent of the Standards is to provide a basis for evaluating the quality of testing practices, as it provides criteria for evaluating tests, testing practices, and the effects of test use.127 The original edition was published in 1954 and has since undergone five revisions. In the newest version, the Standards defines validity as “the degree to which evidence and theory support the interpretation of test scores for proposed uses of tests”.126 The authors explain that the process of validation involves the accumulation of evidence to provide a sound basis for the proposed score interpretations. It is the interpretations of these test scores that are evaluated, not the test itself. The authors emphasize that each intended interpretation must be validated. The Standards describe five “types” of validity evidence, which they state can help illuminate 25 different aspects of validity. However, they caution, that the use of these separate “labels” should not be taken to imply distinct types of validity, as validity is a unitary concept.126 These are not different aspects of validity, but describe different research procedures for validation. The first type of validity evidence described is Evidence Based on Test Content. This type of validity evidence is based on “logical analyses and experts’ evaluations of the content of the measure, including items, tasks, formats, working and processes required of examinees”.128 It addresses concerns about the extent to which the content of a measure is “sufficient, clear, and relevant” to its intended purpose as well as incorporating possible aspects of bias in its administration and scoring.128 This type of validity evidence is needed for virtually all measures. The second type of validity evidence is Evidence Based on Response Processes. This aspect examines “the extent to which the tasks or types of responses required of examinees fit the intended, defined construct”.128 An example of this type of evidence would be that users are not simply giving socially desirable answers. The goal is to examine whether the process that respondents engage in, matches the process that is expected theoretically. The third aspect of validity evidence is Evidence Based on Internal Structure and examines the extent “to which the internal components of a test match the defined construct”.128 The Standards describe two main sources of evidence regarding internal structure: dimensionality (which concerns the inter-relationships among the test items, and is often estimated by confirmatory factor analysis) and measurement invariance (which assesses if members from different groups attribute the same meaning to the construct being measured, and is often assessed within the structural equation modeling framework129). The fourth aspect of validity evidence is Evidence Based on Relations to Other Variables. This is an extensive category as it includes criterion-related validity (concurrent and predictive validity) as well as construct validity.126 This is a widely used source 26 of validity evidence which examines the relationships between the scores from the measure of interest with other variables. The most commonly used methods for this type of validity evidence include correlational studies, criterion-group comparison studies, and experimental studies.128 The fifth and final type of validity evidence is Evidence Based on the Consequences of Testing which pertains to the positive and negative, anticipated and unanticipated, consequences of measurement. It assesses the integrity of the proposed interpretation and its intended use. The use of focus groups is one way to investigate this aspect. Overall, the Standards emphasize that validity is not a static property of an instrument or measure, but rather a property of the sample.128 The main take away is that validity is not an inherent characteristic of the measure, and it is not an “all-or-non-phenomenon”, but rather a matter of degree. It concerns the interpretations and use of the scores, given the specific sample and context. Therefore, validity cannot be separated from the sample and context in which the test is administered. It is an ongoing process and one which Messick claims is never complete.130 1.6.2 Reliability Reliability refers to how consistently a method, or system, measures an outcome. If the same result can be consistently achieved by using the same methods under the same circumstances, the measurement is typically considered to have evidence of reliability.131 Put another way, reliability is the ability to reproduce consistent assessment data and scoring over time.132,133 Test reliability is intrinsically tied to the validity of its inferences. Strong evidence of reliability does not necessarily translate to strong evidence of validity, but rather evidence of reliability is typically thought of as a “necessary precursor” to or a “preliminary form” of 27 validity.134 Overall, assessment instruments should have both evidence of reliability and validity in the specific context for study results to be credible. All assessment data, just like other scientific experimental data, need to be reproducible in order for it to be meaningfully interpreted and useful. Considerable inconsistencies in assessment scoring can call into question the utility of the assessment data and the tool altogether, as it is difficult to interpret the information in a meaningful manner.132 Data resulting from assessments with low or no evidence of reliability may have a larger component of random error, and therefore large variations in scores can be expected upon retesting. Thus, it is important to examine reliability in a sample of students from the post-secondary student population to ensure that students can interpret HEARTSMAP-U consistently. 1.6.3 Measures of Reliability There are multiple methods for estimating reliability. Each method depends on a unique set of assumptions about the participants and the testing procedures, and therefore, no one method will provide the single best estimate of reliability evidence under all circumstances. I will expand on a few common measures below. 1.6.3.1 Internal Consistency One of the most widely used methods for estimating the evidence of reliability is to determine the internal consistency of the measure. This method only requires one form of the test (compared to “alternate forms” of reliability), and participants only need to complete the assessment once (compared to “test-retest” reliability). The thought is that if various components or items of a test are measuring the same construct, then the scores on these components or items 28 will tend to correlate.135 Therefore, statistics can be used to summarize the degree of consistency among the responses from the different components. Commonly utilized approaches for internal consistency are the split-half approach, Cronbach’s alpha, and standardized coefficient alpha. 1.6.3.2 Test-retest Test-retest reliability is another common way to assess the evidence of reliability of a measure.136 The researcher has participants take a test once, and then has participants take the same test after a certain time interval. The correlation between the two tests provides an estimate of the test-retest reliability. One assumption of this method is that the participants true scores is stable across the two testing occasions, meaning that the characteristic being measured is stable over time. Therefore, test-retest is typically used with “trait”, rather than “state”, characteristics; for example, intelligence (trait) vs. mood (state).136 The length of time between the testing occasions can influence results and needs to be taken into consideration. If the interval is too short, there is the increased risk of carryover effects. If the interval is too long, there is an increased risk that the characteristic being measured has changed. A period of one- to two-weeks is typically suggested.136 1.6.3.3 Inter-Rater Inter-rater reliability is used to assess the degree to which different raters make consistent estimates of the same experience, sometimes thought of as consistency estimates. Inter-rater agreement generally means that the raters assign the same score when rating the same individual or object. Agreement does not have to be defined as an all-or-none phenomenon, depending on the goal of the researcher. Analyses are typically done through Cohen’s kappa, or variations, 29 such as Fleiss’ or Light’s kappa, which range from -1 to 1, and represent the proportion of agreement corrected for chance. These are generally easy to compute and will indicate rater disparities. Other types of consistency estimates are correlation coefficients, Cronbach’s alphas, and intraclass correlation. For assessment tools such as HEARTSMAP-U, which rely on self-report raters, the inter-rater reliability is an important measure to estimate for evidence of reliability. The use and rationale of this specific type of reliability, and specific kappa, are further explained in the methods section of Chapter 2. 1.7 Research Rationale and Thesis Objectives 1.7.1 Self-administered psychosocial evaluation Previous studies in the adaptation of HEARTSMAP-U have worked to develop and evaluate the tool in terms of its content with both clinicians as well as end users. However, it is necessary to evaluate whether post-secondary students can reliably score the severity and acuity of psychosocial issues using HEARTSMAP-U. With good evidence for inter-rater reliability, indicating consistency of scoring and inferences, post-secondary students can safely participate in their own mental health assessment, in turn empowering them in their responsibility for their own health. Rigorous reliability testing will contribute evidence towards the tool’s ongoing evaluation. 1.7.1.1 Phase 1: Development & Piloting Vignettes Thesis objective one: to ensure that the vignettes used were comprehensive and understandable to the general student population. 30 1.7.1.2 Phase 2: Inter-Rater Reliability Thesis objective two: to evaluate the evidence for inter-rater reliability of HEARTSMAP-U, among a sample of post-secondary students. 1.7.1.3 Phase 3: Clinician Comparison Thesis objective three: to evaluate whether students’ assessments were similar to those of a clinician. 31 Chapter 2: Evaluating Evidence of the Reliability of HEARTSMAP-U 2.1 Summary Purpose: To evaluate the evidence of inter-rater reliability of HEARTSMAP-U, a psychosocial self-assessment and guidance tool, among a sample of post-secondary students. Methods: We conducted a cross-sectional, multiphasic study between June and October 2020. Student participants applied HEARTSMAP-U to evaluate a set of 25 fictional clinical vignettes covering a range of student mental health presentations. In Phase 1 (n = 15), an iterative process was used to evaluate the vignettes for comprehensiveness and clarity. In Phase 2 (n = 34), a separate group of student participants evaluated the finalized fictional clinical vignettes using HEARTSMAP-U and the evidence for inter-rater reliability was determined. In Phase 3, a clinician applied HEARTSMAP-U to the finalized fictional clinical vignettes, and student and clinician assessments were compared. Results: We report substantial to almost perfect agreement, overall, among all students for all psychosocial domains of the tool (κ = 0.72 in the Student Health domain to 0.81 in the Psychiatry domain). In addition, the resulting proportions of scoring agreement indicated substantial agreement between student and clinician responses (0.82 in the Functional domain and 0.85 in all other domains) on concern severity and also with regards to whether or not the individual described in the vignette had resources in place (97%). Conclusions: Our findings show that HEARTSMAP-U is an interpretable psychosocial assessment and guidance tool that can be reliably applied by post-secondary students. Results from this study will contribute to HEARTSMAP-U’s ongoing validation. 32 2.2 Introduction The post-secondary school years represent a crucial period for mental health and general health behavior development. Not only are young adults navigating through social, educational and financial transitions to adulthood, they are often doing so with reduced family or institutional support.29 Post-secondary students are typically assuming greater life responsibilities and independence, which can become emotionally overwhelming, resulting in a high prevalence of psychosocial issues.30 According to the Spring 2020 American National College Health Assessment, approximately 30% of post-secondary students report receiving psychological or mental health services in the last 12 months, with 27.4% screening positive for anxiety, 22.3% for depression, 9.4% indicating that they had intentionally self-harmed, and approximately 75% of the students surveyed rated their level of stress as either moderate or severe.137 These estimates are comparable to the 2019 Canadian reference group survey138 and are part of a worsening trend, with increased mental health diagnoses among post-secondary students from 22% in 2007 to 36% in 2017.9 At a time when many young adults are moving away from their support networks and transitioning from pediatric to adult health care systems, there are a perceived lack of resources and numerous barriers to mental health treatment. Many post-secondary institutions have limited counselling services with long wait times.11 The perceived personal and public stigma associated with mental health concerns can cause students to stay silent about their struggles.11 Moreover, with limited mental health literacy, many students may not recognize a treatment need, believing that symptoms of clinical depression and anxiety are typical of college life.45 Finally, when care is sought, the uncertainty around available and appropriate supports can further hinder student help-seeking behaviour.11 33 Mental health system improvement efforts are increasingly looking into how technology can be integrated into mental health services to engage young adults in a familiar medium. Young people may prefer self-administered assessments over those that require face-to-face disclosure, as it offers privacy, time to effectively describe concerns, and a sense of control over managing their well-being.139 However, despite these positive attributes, these tools are rarely integrated into post-secondary health systems.54 There are few student developed, valid, reliable, and comprehensive psychosocial self-assessment tools for post-secondary students.97,98 The majority of tools are developed for diagnostic purposes rather than universal screening in the student population; and typically for single psychiatric or psychological conditions.76,84,91,92,94 Nevertheless, four instruments are commonly used. First, the Patient Health Questionnaire (PHQ-9) is a widely used 9-item self-report measure designed to assess the presence of major depressive disorder.76 PHQ-9 has been extensively studied in primary and secondary care settings,80 however, to our knowledge, evidence of its application in the university population is limited, with research on its English version restricted to a single secondary analysis.77 Second, the Generalized Anxiety Disorder scale (GAD-7) is a broadly used scale designed to assess for generalized anxiety disorder according to DSM-IV criteria.84 The GAD-7 was originally developed in the primary care setting,84 however, a 2020 study conducted a secondary analysis to assess the psychometric properties of GAD-7 in young adults enrolled in college in the United States.85 Third, the Alcohol Use Disorder Identification Test (AUDIT) was first developed in 1973 as an instrument to detect hazardous and harmful alcohol consumption in primary health care settings.89,92 Finally, the Suicide Behaviors Questionnaire-Revised (SBQ-R) is a questionnaire used to identify at-risk individuals and specific risk behaviors for suicide.93,94 34 These existing tools have shown strong psychometric properties, but the needs of post-secondary students are multi-dimensional and should not be approached in a fragmented manner. There is a need for a short assessment tool that addresses core mental health related concerns (which include anxiety, mood, suicidality, and substance use) but also speaks to the broader-and related-social needs (e.g., relationships and financial security). In addition, these tools should encourage student involvement and feedback throughout their development. In response to this need, the growing concern regarding post-secondary mental health, as well as the support from our institution, our team developed HEARTSMAP-U. HEARTSMAP-U is a digital psychosocial self-assessment and guidance tool for post-secondary students. It is an adaptation of HEARTSMAP, the assessment and management tool designed for use by clinicians in pediatric emergency care settings,111 and MyHEARTSMAP, its self-administered version for children and their parents.116 HEARTSMAP-U assesses similar domains as these two previous tools but was modified and tailored to address the unique needs of post-secondary students.124 This study contributes to the ongoing development of HEARTSMAP-U by evaluating the inter-rater reliability of the tool. Reliability refers to how consistently a method or system measures an outcome. If the same result can be consistently achieved by using the same methods under the same circumstances, the measurement is considered reliable.131 Reliability can be measured in multiple ways, and it is the specific purpose of the assessment tool that dictates what type of reliability is of greatest importance. For assessment tools like HEARTSMAP-U, which rely on students to provide and rate their individual situation, a fundamental threat to reproducibility is poor consistency between different students/raters and therefore inter-rater reliability is a crucial type of reliability to estimate.132 Completing this aspect of its psychometric 35 evaluation will ensure that HEARTSMAP-U can be reliably used by post-secondary students and will ensure consistency in its mental health assessments. 2.3 Methods 2.3.1 Study Design & Objectives 2.3.1.1 Phase 1: Development & Piloting Vignettes We conducted a multiphasic cross-sectional study. In Phase 1, we engaged post-secondary students (n = 15) to use HEARTSMAP-U and perform a psychosocial assessment on a set of 25 fictional clinical vignettes and provide feedback on these vignettes guided by standardized questions. The objective of this phase was to ensure that the vignettes used were comprehensive and understandable to the general student population. As such, all feedback received from each participant (without exception), on each vignette section description, was reviewed and incorporated sequentially into the next version of that vignette. Documentation of each edit was also recorded on a separate document to ensure that no comments were missed in the process. Despite not conducting formal qualitative analyses, this document helped to organize and characterize the feedback. This was done until all comments received were positive and saturated (i.e., no new areas/themes of concern). 2.3.1.1.1 Vignette Development A vignette is a “brief, written case history of a fictitious patient based on a realistic clinical situation”.140 Clinical vignettes have been used for more than 30 years to evaluate variability in clinical practice and decision-making processes,140,141 and in doing so, have been shown to offer many advantages over other methods, such as medical record reviews, analysis of 36 claims data, and standardized patients.140 In addition, clinical vignettes have also been used to measure the inter-rater reliability of clinician assessments of mental health status.142,143 One of the most significant and applicable advantages of vignettes is that they are able to control case mix and isolate the individual rater’s decision making from other factors in the environment. In a prior study, for the purpose of HEARTSMAP-U’s comprehensive development and evaluation, 28 clinicians applied HEARTSMAP-U to up to two fictional vignettes of their own creation based on clinical encounters with students with psychosocial-related clinical presentations to their practice, as well as the clinicians’ empirical knowledge. A total of 47 vignettes were created by these clinicians.124 In the current study, together with student collaborators, which included a second-year medical student and a PhD student, we selected 25 out of the available 47 vignettes and modified them to include a range of concerns and severities typical of post-secondary school students inspired by their own experiences or that of their acquaintances. Each vignette was selected using an a priori set list of criteria, to ensure a breadth of psychosocial related concerns and severities, and to avoid duplication of concerns. As the original vignettes were created by clinicians based on real clinical encounters with students with psychosocial-related concerns, they primarily covered moderate to severe presentations. Our modifications ensured that scenarios representing students with no to mild concerns were also included. In addition, care was taken to ensure representation of international student experiences, varying sexual orientations, and non-binary gendered students. Selection and modification of the vignettes were conducted by myself, as well as student collaborators in parallel to ensure that presentations appeared accurate to student life. After we had each selected a set of 25 vignettes, we compared our selections and discussed any discrepancies. We overlapped on 19 vignettes and worked 37 together to select and modify the last six to include in the final set of 25 vignettes to begin Phase 1. All changes were overseen by a research clinician. A sample vignette is provided in Appendix B. 2.3.1.2 Phase 2: Inter-Rater Reliability In Phase 2, we engaged post-secondary students (n = 34) to perform psychosocial assessments on the 25 finalized fictional clinical vignettes that resulted from Phase 1. The objective of Phase 2 was to evaluate HEARTSMAP-U inter-rater reliability, among young adults pursuing post-secondary education, as applied to a set of fictional cases. 2.3.1.3 Phase 3: Clinician Comparison A cross sectional study was conducted in November and December 2020, in which a clinician used HEARTSMAP-U to perform psychosocial assessments on the same 25 finalized fictional clinical vignettes that resulted from Phase 1. The objective of Phase 3 was to compare, on the same set of fictional vignettes, how similar are student’s assessments with that of a clinician. 2.3.2 Study Population and Setting Convenience and purposive samples were obtained of post-secondary students (17 years and older) studying at a large public research university in Canada. Care was taken to ensure representation on gender, as young men are typically underrepresented in mental health research144, and student type (i.e., undergraduate vs. graduate students), as HEARTSMAP-U was designed for both groups. While we initially enrolled all eligible students who reached out, at the 38 mid-point of our planned sample size, demographic distributions of enrolled participants were reviewed, and sampling aimed to ensure representation that reflects approximately 50:50 male, to female gender ratio and 50:50 undergraduate, to graduate student ratio was applied. Students were recruited through multiple social media platforms (e.g., Facebook, Reddit) and institution affiliated websites (e.g., Student Forums). In addition to our student population, for Phase 3, we recruited a clinician from the Student Health Services at UBC who became a co-investigator on the study. Student Health Services offers a wide range of health assessments and treatments provided by doctors, nurse practitioners, nurses, and specialists to the UBC community. All participants took part in the study remotely. 2.3.3 Instrument The HEARTSMAP-U tool was developed with a student-centered design and has completed extensive preliminary evaluations with both clinicians and student end users.124 HEARTSMAP-U’s clinical content was evaluated by a broad population of clinicians that support young adults in post-secondary education. With the feedback from participating healthcare professionals, the tool’s language was modified to reflect post-secondary students’ unique needs and tool sections were refined to flag broad psychosocial concerns and support follow-up with in-depth diagnostic evaluations.124 Student end user acceptability evaluations were conducted in a diverse sample of post-secondary students through the use of focus groups. Students provided positive reactions and felt that HEARTSMAP-U would cater to student-specific needs, normalize discussing mental health, and provide a less invasive alternative to in-person assessment.124 39 Overall, HEARTSMAP-U facilitates self-reporting across 10 psychosocial sections: Housing and material security, Education and activities, Substances and behavioral dependencies, Relationships, Thoughts and anxiety, Safety, Sexual wellness, Mood, Abuse, and Professional resources. Scoring options on each severity scale have descriptive statements expanding on each score’s conditions, helping students decide on appropriate scores. Psychosocial sections map to service-oriented general domains which are labelled: Social, Functional, Student health, and Psychiatry. For each section, concern severity is measured on a 4-point Likert-type scale from 0 (no concern) to 3 (severe concern), and services already accessed are measured on a separate 2-point scale (yes or no). Inputs from both scales feed into a built-in algorithm, triggering service recommendations with a suggested time frame for access. 2.3.4 Data Collection & Storage All clinical vignettes were stored on Research Electronic Data Capture (REDCap), a standardized online data collection instrument.145 REDCAP also hosted the HEARTSMAP-U tool, which was enabled as an online survey. 2.3.5 Study Procedures All participants provided informed consent prior to participation and were compensated $125 CAD ($5 for each completed vignette). This research was reviewed and approved by the University of British Columbia Children’s and Women’s Hospital ethics review board (H19-03347). 40 2.3.5.1 Participant Training Student participants completed a 45-60 minute training session with a co-investigator (HB) on the GoToMeeting video conferencing platform. Training addressed an overview of the HEARTSMAP-U sections, scoring guidelines, and application to fictional cases (i.e., emphasizing how participants should imagine themselves as the student described in each of the vignettes). The overall aim was to ensure that HEARTSMAP-U scoring is approached in a manner consistent with the tool’s intended use. Upon training completion, each participant received a link via email to access the clinical vignettes and the HEARTSMAP-U tool, allowing for remote completion at a self-directed pace. All responses were captured in REDCap.146 Participants did not receive any additional training. 2.3.5.2 Phase 1: Development & Piloting Vignettes To ensure that the vignettes were understandable to the general student population, multiple rounds of evaluation were carried out to assess vignettes’ comprehensiveness and clarity before the reliability evaluation procedure occurred. Students were recruited in samples of 3-4 to read and score all 25 vignettes. For each section of the HEARTSMAP-U and associated vignettes, participants were asked to comment on the vignette, guided by two main questions with a possible follow up question for each. First, participants were asked: “Was there enough information to confidently score each section?” (yes or no). Only if participants answered “no”, were they asked the follow-up question: “What information do they feel is needed to confidently score that section?” Second, all participants were asked: “Was there ambiguous or confusing information that made the vignette difficult to score?” (yes or no). Only if they answered “yes”, were they asked the follow-up question: “What information needs clarification or removal from 41 the vignette to facilitate confident scoring?” All feedback received from participants were reviewed and incorporated sequentially into the next version of the vignette and tested using a new sample of 3-4 students in the same manner as described above. These changes allowed further refinement of the vignettes (e.g., grammar, word choice, detail in descriptive statements), to improve consistency. This was done until all comments were positive and saturated. 2.3.5.3 Phase 2: Inter-Rater Reliability A separate group of student participants were presented with the finalized 25 fictional clinical vignettes describing a variety of post-secondary mental health related presentations. Participants were asked to apply HEARTSMAP-U to score the severity and acuity of concerns across the tool’s ten sections. Upon completing their assessment of each vignette, scores were recorded automatically to the study’s REDCap database. 2.3.5.4 Phase 3: Clinician Comparison To evaluate whether students’ assessments using HEARTSMAP-U are similar to a clinician’s assessment, a clinician was presented with the same finalized 25 fictional clinical vignettes describing a variety of post-secondary mental health related presentations. This clinician was asked to apply HEARTSMAP-U to score the severity and acuity of concerns across the tool’s ten sections. 2.3.6 Power Analysis Phase 2 sample size was based on an ICC power analysis,147 equivalent to quadratically weighted kappas.148 32 student participants were required to achieve a power of 80% to detect a 42 kappa of 0.60 (substantial agreement) under the alternative hypothesis, assuming a kappa of 0.42 (moderate agreement) under the null hypothesis and a significance level of 5%.149 All study analyses were conducted using RStudio (Version 1.3.1093) and the Microsoft Excel 2021 (Version 16.47) Data Analysis Toolpak (Microsoft, Redmond, Washington). 2.3.7 Analytic Approach 2.3.7.1 Background of Inter-Rater Reliability Reliability can be measured in multiple ways; and it is the specific purpose of the assessment tool that dictates what type of consistency or reliability is of greatest importance. For assessment tools like HEARTSMAP-U, which rely on students to provide and rate their individual situation, a fundamental threat to reproducibility is poor consistency between different students/raters and therefore inter-rater reliability is a crucial type of reliability to estimate.132 Inter-rater reliability is defined as the extent to which an instrument or tool ensures reproducible measurement of its distinct items by two or more raters.150 After taking measurement errors between raters into consideration, this method of analysis determines the degree of true variance in rater scoring.150 Analyzing agreement between raters offers a way of quantifying the degree of agreement and therefore a method of gauging reliability. Inter-rater reliability is separate from validity analysis, which assesses how closely an instrument or tool measures an actual construct rather than how well raters provide similar ratings.150 Instruments may have varying levels of validity regardless of the inter-rater reliability of the tool. 43 2.3.7.2 Kappa Analysis Multiple statistics and variants can be used in an inter-rater reliability analysis. Kappa statistics measure the observed level of agreement between raters for a set of nominal ratings and correct for agreement that would be expected by chance.150 Cohen’s original kappa statistic is the classic summary statistic of inter-rater reliability between a rater pair, which takes into consideration agreement that may have arisen due to chance.151 However, this measure is limited to evaluating the level of agreement between only two-raters.150,151 With three or more raters, kappa-like variants can be used to analyze the inter-rater reliability depending on the specific study design. Fleiss kappa provides kappa-like coefficients that are suitable for studies where “…any constant number of m coders is randomly sampled from a larger population of coders, with each subject rated by a different sample of m coders”.150,152 In other words, each subject is rated by a random sample of the available raters and not by all of them. Fleiss’s coefficient is inappropriate for studies where every subject is rated by every rater.150 For designs in which there are three or more raters and every rater rates every subject, there is Light’s (1971) kappa variant.153 In this variant, a kappa is computed for all rater pairs and then a mean of these estimates is used to provide the overall index of agreement. Light’s kappa variant works for “m observers, who each assign n items, among C categories”.153 This means that there can be any number of raters (e.g., 5), who rate any number of subjects (e.g., 10), using a scale with any number of categories (i.e., it does not have to be bivariate). However, for instruments measuring categorical variables with an ordinal response scoring format, like HEARTSMAP-U, the clinical significance of scoring disagreement varies depending on its magnitude and it is important to retain the hierarchical nature of the categories 44 in the analysis.154 Unfortunately, standard kappas do not account for the degree of rater disagreement. To reflect the magnitude of disagreement, kappas can be weighted, so that greater emphasis is attached to larger differences between ratings than to smaller differences.154 In other words, weighted kappas penalize disagreements in terms of their seriousness, whereas unweighted kappas treat all disagreements equally. Unweighted kappas are therefore inappropriate for scales like HEARTSMAP-U. Quadratic weighted kappas for ordinal scales offer a specific advantage in that they can be substituted interchangeably with intraclass correlation coefficients (ICCs).155 In this study, to extend the measurement of inter-rater reliability to multiple raters (n = 34), in which all raters rated each vignette (n = 25), using a 0-3 scale, Light’s (1971) kappa variant was used. In accordance with Light (1971), kappa values were computed for all evaluator pairs, the overall mean of these pairs were then used as our index of agreement.153,148 We report 95% CIs for all tests and guidelines by Landis and Koch were used to interpret the agreement strength of kappa coefficients.149 Overall, possible values for kappa statistics range from -1 to 1. Values less than zero were interpreted as “no agreement”, values 0-0.20 as “slight agreement”, values 0.21-0.40 as “fair agreement”, values 0.41-0.60 as “moderate agreement”, values 0.61-0.80 as “substantial agreement”, and values 0.81-1.00 as “almost perfect or perfect agreement”. 2.3.7.3 Sensitivity Analysis To monitor the time participants spent on each vignette, and to detect unreasonable speed (e.g., scoring randomly or finishing cases too fast), REDCap’s activity logging feature was used to monitor the duration spent on each case. We conducted sensitivity analyses excluding any 45 outliers with unreasonable vignette completion paces as defined a priori by more than two standard deviations from the group average. 2.3.7.4 Clinician Comparison For each domain (i.e., using domain scores) and for each vignette, I calculated the proportion of student participants (n = 34) who scored the same as the study clinician. I did this for each of the 25 vignettes and calculated the median and the interquartile range (IQR) of these values. For example, if the clinician has a social domain score of 2 for vignette 1, I calculated what proportion of student responders also scored a 2 (e.g., 0.60). I did the same method for each vignette and had 25 proportions (e.g., 0.60, 0.55, 0.9, 0.75, etc.). As these values were not normally distributed, we calculated and reported the median and IQR of the values. In addition, for each applicable psycosocial scoring section (i.e., Housing and material security, Education and activities, etc.), and for each vignette, I calculated the proportion of student participants who scored the same as the study clinician on the resource assessment question. This secondary question appears if raters score the severity of the section from 1-3 (i.e., identifying some concerns). In each vignette, there are 11 sections to score (because the Thoughts & anxiety section is broken into two subsections). Of these 11 sections, 9 have the possibility of triggering the secondary resource assessment question (Abuse and Professional resources do not have this component). Therefore, for 25 vignettes, there are 225 sections where the resource assessment question could be triggered. Out of these 225 sections, we identified the sections in which the clinician identified concerns (i.e., did not score zero). In these sections, we calculated the number of student participants who also identified concerns and from this number calculated the number of student participant responses that agreed with the clinician’s assessment 46 of whether or not the individual described in the vignette had resources in place to address the identified concerns. We calculated and reported the average percent agreement across the identified sections. 2.4 Results 2.4.1 Phase 1: Development & Piloting Vignettes We conducted four rounds of vignette modification, each with 3-4 students, for a total of 15 students, all applying HEARTSMAP-U to 25 fictional clinical vignettes. Additional demographic details of participants are summarized in Table 2.1. Table 2.1 Basic demographic information of study participants (N1 = 15, N2 = 34). Characteristic Phase 1 n (%) Phase 2 n (%) Age (years) 17-21 22-25 26+ 2 (13%) 8 (53%) 5 (33%) 13 (38%) 15 (44%) 6 (18%) Student Type Undergraduate Graduate 4 (27%) 11 (73%) 19 (56%) 15 (44%) Sex Female Male 11 (73%) 4 (27%) 17 (50%) 17 (50%) Gender Female Male 11 (73%) 4 (27%) 17 (50%) 17 (50%) 47 Sexual Orientation Heterosexual Bisexual Gay/Lesbian Other 11 (73%) 3 (20%) 1 (7%) 26 (76%) 2 (6%) 4 (12%) 2 (6%) Ethnic Background European Descent Asian Hispanic Other 3 (20%) 1 (7%) 3 (20%) 8 (53%) 13 (38%) 17 (50%) 1 (3%) 3 (9%) Experienced Mental Health Concerns Before Yes No Choose not to respond 10 (67%) 5 (33%) 17 (50%) 15 (44%) 2 (6%) In round one and two (n = 3 and n = 4, respectively), the overall majority of the feedback from participants was regarding the amount of descriptive detail provided in the vignettes; in general, participants suggested to include more details in the vignette descriptions. In addition, two branching logic errors in the participant demographic survey were identified and corrected. In round three (n = 4), we received the highest number of edits, which consisted mainly of stylistic edits without change in meaning or core content. The majority of these edits were made to clarify sentence structure or add small details to the descriptions. In round four (n = 4), which was the final round of Phase 1, no details were added to the descriptors and only small clarifications, or word choice changes were made to finalize the vignettes. A section of the HEARTSMAP-U tool and vignette description are shown below in Figure 2.1. 48 Figure 2.1 Example of vignette description and corresponding HEARTSMAP-U assessment section. 2.4.2 Phase 2: Inter-Rater Reliability A total of 34 students were recruited to evaluate the 25 finalized fictional clinical vignettes using HEARTSMAP-U. Additional demographic details of these participants are summarized in Table 2.1. Inter-rater reliability analyses were performed to assess the degree that student raters consistently assigned severity scores to the psychosocial domains of the vignettes. Light’s kappa was the appropriate index of inter-rater reliability and a kappa was computed for each rater pair then averaged to provide a single index of agreement.150,153 At the domain level, the resulting kappa statistics indicated substantial agreement149 ranging from 0.67 (95% CI: 0.65, 0.69) in the Student Health domain to 0.76 (95% CI: 0.74, 0.78) in the Psychiatry domain. 49 Upon analysis of REDCap’s activity logging feature, we observed one participant with outlying vignette completion times (mean of 0.9 minutes/vignette, SD 0.5). On average, the other participants (n = 33) completed each vignette in 7.4 minutes (SD 2.8). This participant matched our a priori definition of an outlier (>2 SD lower than the group average) with regard to the time required to read, reflect, and complete each vignette assessment. These low completion times may indicate that this participant did not actively read the vignette contents and likely randomly selected their answers. Sensitivity analyses excluding data from this outlier modestly, but statistically significantly, improved the kappa statistics for each domain. Mean weighted kappa coefficients with and without the outlying data for each psychosocial domain are reported in Table 2.2. Table 2.2 Kappa statistics (95% confidence intervals) for each domain. Domain κ (all participants) κ (outliers removed) Social 0.74 (0.72, 0.76) 0.79 (0.78, 0.80). Functional 0.73 (0.70, 0.75) 0.79 (0.78, 0.80) Student Health 0.67 (0.65, 0.69) 0.72 (0.71, 0.73) Psychiatry 0.76 (0.74, 0.78) 0.81 (0.80, 0.82) 2.4.3 Phase 3: Clinician Comparison To evaluate the degree to which students’ assessments using HEARTSMAP-U are the same as a clinician, a clinician was presented with the finalized 25 fictional clinical vignettes, that resulted from Phase 1, and asked to apply HEARTSMAP-U to score the severity and acuity of concerns across the tool’s sections. At the domain level, without outlying data, the resulting proportions indicated substantial agreement between student and clinician responses (median range 0.82 – 0.85). For the Social 50 Domain, the median proportion of agreement found was 0.85 (0.71-0.97), for the Functional Domain, 0.82 (0.30-0.97), for the Student Health Domain, 0.85 (0.32-0.95), and for the Psychiatry Domain, 0.85 (0.73-0.98). Median proportions of scoring agreement with and without the outlying data for each psychosocial domain are reported in Table 2.3. Table 2.3 Median proportion of agreement (IQR) between student participants and study clinician. Domain Median (IQR) with all participants. Median (IQR) with outliers removed. Social 0.82 (0.68-0.94) 0.85 (0.71-0.97) Functional 0.76 (0.35-0.94) 0.82 (0.30-0.97 Student Health 0.85 (0.35-0.91) 0.85 (0.35-0.95) Psychiatry 0.79 (0.71-0.97) 0.85 (0.73-0.98) Of the 225 psychosocial sections in which the secondary resource assessment question could be triggered, the clinician identified concerns (i.e., did not score zero) in 112 sections. Overall, on average in these 112 sections, 30/33 (91%) student participants (outlier removed) also identified concerns. On average, student participants agreed with the clinician’s assessment of whether or not the individual described in the vignette had resources in place 97% of the time. 2.5 Discussion The current study found that HEARTSMAP-U, designed for universal psychosocial screening and resource support, displays good evidence for inter-rater reliability when used by post-secondary students and applied to fictional cases. 51 There are few valid and comprehensive psychosocial self-assessment tools for post-secondary students, with the majority of tools developed for diagnostic rather than universal screening purposes in the student population. In addition, these tools are typically for single psychiatric or psychological conditions.76,84,91,92,94 The PHQ-9, GAD-7, SQB-R and AUDIT tools have strong psychometric properties; however, their approach, context, purpose, and analytical strategies differ substantially from our study. Due to these differing conceptual models and aims, analytical metrics varied. The PHQ-9, GAD-7, AUDIT, and SBQ-R studies primarily focused on internal reliability estimates rather than inter-user estimates. As HEARTSMAP-U relies on students’ self-reported psychosocial status, poor consistency would threaten tool reproducibility, and thus it is essential to establish its inter-rater reliability. These prior psychological assessment tools (PHQ-9, GAD-7, AUDIT, and SBQ-R) were evaluated for clinical assessment purposes, and thus target students with identified needs. Only the GAD-7 tool has been evaluated with universal screening intentions, but again this was done in a clinical practice among students who had already reached out for care.85 HEARTSMAP-U was developed for universal screening in the larger student population, including students who have not reached out to health care providers. Another feature distinguishing HEARTSMAP-U from previously evaluated instruments is that it is multi-dimensional; the PHQ-9, GAD-7, AUDIT, and SQB-R studies assess the clinical and diagnostic aspects of one psychological concern, depression, anxiety, substance abuse, and suicidality respectively. HEARTSMAP-U shows strong inter-rater reliability in the context of post-secondary students, and in addition, captures and assesses a student’s concerns across a broad range of social factors that may be contributing to, or the result from, their psychosocial issues. Our study is strengthened by its methodology for vignette development and 52 tool administration, using participant training and accountability measures for thoughtful scoring,156 a feature that is infrequently reported in inter-rater studies of psychosocial measures.157 Our iterative approach in Phase 1 allowed us to make ongoing modifications to address participant concerns regarding vignette comprehensiveness and clarity. This ensured that the finalized vignettes utilized in the reliability testing (Phase 2) were user-friendly. In addition, it should be noted that our study was not interrupted by the COVID-19 pandemic. Moreover, with decreased access to in-person care because of safety measures put in place, online screening assessment tools are even more valuable during these times. Mental health concerns are expected to increase as a result of the pandemic; early studies show a possible increase in the number of young adults (ages 18-24) reporting symptoms of anxiety, depression, substance abuse, and/or suicidal thoughts.158 Study limitations include a lack of formal qualitative analysis on the feedback received in Phase 1, preventing us from categorizing and analyzing participant feedback, but providing sufficient documentation for iterative HEARTSMAP-U modifications. In addition, the inter-rater agreement estimates may differ when the tool is applied to patient self-report versus fictional vignettes.159 The use of vignettes required participant training to ensure that all participants could comfortably score the psychosocial information of fictional patients. Finally, all participants took part in the study remotely. Therefore, participant scoring was completed unsupervised, and we cannot ensure that participants remained focused and engaged throughout the study procedures. However, the time participants spent on each vignette was monitored to detect unreasonable speed and a sensitivity analysis was conducted excluding any outliers with unreasonable vignette completion pace. 53 2.6 Conclusion Our findings show that HEARTSMAP-U is an interpretable psychosocial assessment and guidance tool that can be reliably applied by post-secondary students. In addition to good reliability, HEARTSMAP-U was developed in a student-centered design, and covers a broad range of constructs that are important to the psychosocial well-being of students. Following prospective evaluation of its predictive validity, we plan to make HEARTSMAP-U accessible to students as a downloadable or web-based application. 54 Chapter 3: Discussion, Future Directions & Clinical Implications 3.1 Overview This thesis presents findings from one study which contributes to the utility of HEARTSMAP-U, a psychosocial self-assessment and guidance tool for post-secondary students. Through a multiphasic study, this research aimed to 1) evaluate the evidence of inter-rater reliability of HEARTSMAP-U among a cross-section of young adults pursuing post-secondary education, as applied to a set of fictional cases, and 2) evaluate, on the same set of fictional cases, the scoring agreement between student and clinician assessments. 3.2 Key Findings We conducted a cross-sectional, multiphasic study to evaluate the evidence of inter-rater reliability of HEARTSMAP-U among post-secondary students, as well as evaluate the scoring agreement between student and clinician assessments. Overall, students displayed substantial agreement across HEARTSMAP-U’s four domains, when applied to a range of diverse fictional clinical vignettes. In addition, compared to a clinician, students displayed high agreement across all four domains, as well as regarding whether or not the individual described in the vignette had support resources in place. Overall, these findings provide evidence that HEARTSMAP-U is an interpretable psychosocial assessment tool that can be reliably applied by post-secondary students. The preliminary evaluation of HEARTSMAP-U with a clinician offers a methodological quality check, indicating that students understood the vignettes and how to apply HEARTSMAP-U. Overall, this provides preliminary evidence of tool accuracy which will be further assessed in 55 subsequent studies. Taken together, these findings contribute to the foundation for HEARTSMAP-U validation and implementation. 3.3 Overall Strengths Chapter 2 describes specific study strengths and limitations which will be expanded on here. With regards to the study sample population, attention was taken to ensure representation on gender, as young men are typically underrepresented in mental health research.144 As there are significant gender differences in the prevalence of mental health disorders as well as in mental health service utilization144, it is crucial to include male identifying individuals in mental health research, particularly in the development of a novel mental health assessment tool. The reasons for the poor engagement of young men with mental health services are complex (e.g., stigma, masculinity, inadequate mental health literacy, poor emotional competency144). However, engaging young men in the beginning stages of developing a new mental health assessment tool can help engage this population in their own mental health assessment and needs. In addition to our student population, we recruited a clinician from UBC’s Student Health Services. UBC’s Student Health Services offers a wide range of health assessments and treatments provided by health care professionals to the UBC community. This ensured that the clinician in our study was familiar with assessing our population of interest. This study also used extensive vignette methodology. The original vignettes, created for the purpose of HEARTSMAP-U’s comprehensive development and evaluation with clinicians and end users, were created by clinicians based on real clinical encounters with students with psychosocial-related presentations as well as the clinicians’ empirical knowledge. For the current study, these vignettes were then modified with student collaborators to ensure that presentations 56 appeared accurate to student life. In addition, the iterative approach to vignette modification allowed us to make ongoing modifications to address participant concerns regarding vignette comprehensiveness and clarity. This ensured that the finalized vignettes used to assess the evidence of inter-rater reliability were accessible and user-friendly. 3.4 Overall Limitations In this study, HEARTSMAP-U was applied to fictional vignettes, which gave participant raters a psychosocial history pertaining to each of HEARTSMAP-U’s sections, and the inter-rater agreement estimates may differ when the tool is applied to an individual’s self-report. Compared to real-life scenarios, vignettes cannot capture the personal trauma (whether past or present) behind the individual’s current situation and their self-report. Vignettes do not always reflect the complexity and nuances of real people or describe complex psychosocial situations. In addition, practice effects can be an issue in these instances. Each participant evaluated 25 vignettes and it is possible that higher agreement may be seen because of the rater’s prolonged exposure to the tool and improved ability to apply the tool to these cases. As mentioned above under overall strengths, care was taken to ensure representation on gender within the study sample. However, using a convenience and purposive sampling strategy does not guarantee diversity and representativeness on a breadth of demographic factors, such as ethnicity, sexual orientation, or immigration status, which may impact how students interact with the tool. Finally, all study participation was completed remotely. In the original protocol, participant study training was to be completed in person and then subsequent participant scoring was to be completed unsupervised at the participant’s own time. Fortunately, this study was not 57 significantly impacted by the COVID-19 pandemic, however, the study training portion was converted to an online format and completed over the GoToMeeting platform. As a result, the study was completed 100% online with no in-person interaction between participants and study researchers. Without the added personal interaction, it is harder to ensure that participants fully understood study procedures and remained focused and engaged throughout the study process. As an added safety net, the time participants spent on each vignette was monitored to detect unreasonable speed and analyses were conducted excluding any outliers with unreasonable completion pace. 3.5 Future Research HEARTSMAP-U’s psychometric evaluation is ongoing. My work is just one of multiple studies that are being conducted to evaluate the tool’s measurement properties. Nevertheless, this study offers crucial evidence to support the tool’s inter-rater reliability. A pilot predictive validity study is currently underway, and funding has been secured for a larger predictive validation study to commence in Fall 2021 (n = 450). Ensuring the accuracy of HEARTSMAP-U’s assessment data and its tool triggered service recommendations is crucial before tool implementation in the post-secondary student health system. These subsequent investigations will ensure that HEARTSMAP-U can safely identify mental health issues among post-secondary students by determining the sensitivity and specificity in identifying psychosocial issues compared against a clinician’s assessment. In addition to the already completed tool development and user evaluation studies, in order for HEARTSMAP-U to be a suitable candidate for universal screening in the post-secondary student population, it is necessary to also complete additional psychometric testing. 58 3.6 Significance & Implications Screening tools serve as the gateway to intervention. As such, their ability to correctly classify individuals as at-risk or not-at-risk is crucial. HEARTSMAP-U has the potential to be embedded into broader campus-wide mental health promotions initiatives, supporting students year-round and at crucial times of stress (e.g., start of the semester or exam periods). In this regard, HEARTSMAP-U may play a key role in early recognition of mental health issues, students’ connectivity with appropriate levels of mental health care, and longitudinal tracking of students perceived psychosocial well-being through repeated measures. This would also allow health providers to offer long-term care, addressing the dynamic and changing mental health needs of students. A benefit of brief self-report instruments, beyond the ease of administration, cost, and scoring, is the ability to obtain sensitive information from individuals who may have difficulty discussing this information in an interview situation. Mental health disorders that develop early in life, including young adulthood, are linked with an elevated risk for onset of other mental health concerns later in life.160 Thus, early diagnosis and treatment of mental health disorders may also offer protection against serial co-morbidity of other mental health concerns.160 The fully evaluated and implementation-ready HEARTSMAP-U will generate service recommendations based on students’ scores in each section (e.g., concern severity and acuity). HEARTSMAP-U will have a built-in safety net that will include recommendations for immediately accessible services and supports, including self-care resources that may serve as preventative measures (e.g., meditation, mindfulness), crisis hotline information, links to e-counselling, and information about nearby emergency departments. These features may mitigate 59 wait times and saturation of in-person services with e-mental health supports that students may prefer and have immediate access to.161 3.7 Conclusory Remarks This thesis contributes to the ongoing validation of a psychosocial self-assessment and guidance tool for post-secondary students. While HEARTSMAP-U shows great promise as a valuable psychosocial screening tool, further research is needed and is currently underway to continue its evaluation and prepare it for use in the post-secondary student population. 60 References 1. Beatie BE, Stewart DW, Walker JR. A Moderator Analysis of the Relationship Between Mental Health Help-Seeking Attitudes and Behaviours among Young Adults. Can J Couns Psychother / Rev Can Couns psychothérapie. 2016;50(3):290-314. https://cjc-rcc-ucalgary-ca.acces.bibl.ulaval.ca/cjc/index.php/rcc/article/view/2871/pdf_1. 2. Kessler RC, Berglund P, Demler O, Jin R, Merikangas KR, Walters EE. Lifetime Prevalence and Age-of-Onset Distributions of. Arch Gen Psychiatry. 2005;62(June):593-602. doi:10.1001/archpsyc.62.6.593 3. Mental health indicators. https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=1310046501&pickMembers%5B0%5D=1.1&pickMembers%5B1%5D=2.2&pickMembers%5B2%5D=3.1&cubeTimeFrame.startYear=2012&cubeTimeFrame.endYear=2012&referencePeriods=20120101%2C20120101. Accessed January 8, 2021. 4. Nces. Educational Attainment of Young Adults.; 2020. 5. The Condition of Education - Postsecondary Education - Postsecondary Students - College Enrollment Rates - Indicator May (2020). https://nces.ed.gov/programs/coe/indicator_cpb.asp. Accessed March 9, 2021. 6. Education in Canada: Key results from the 2016 Census. https://www150.statcan.gc.ca/n1/daily-quotidien/171129/dq171129a-eng.htm. Accessed March 9, 2021. 7. Twenge JM, Gentile B, DeWall CN, Ma D, Lacefield K, Schurtz DR. Birth cohort increases in psychopathology among young Americans, 1938-2007: A cross-temporal meta-analysis of the MMPI. Clin Psychol Rev. 2010;30(2):145-154. 61 doi:10.1016/j.cpr.2009.10.005 8. Lattie EG, Lipson SK, Eisenberg D. Technology and College Student Mental Health: Challenges and Opportunities . Front Psychiatry . 2019;10:246. https://www.frontiersin.org/article/10.3389/fpsyt.2019.00246. 9. Lipson SK, Lattie EG, Eisenberg D. Increased Rates of Mental Health Service Utilization by U.S. College Students: 10-Year Population-Level Trends (2007-2017). Psychiatr Serv. 2019;70(1):60-63. doi:10.1176/appi.ps.201800332 10. Adlaf EM, Gliksman L, Demers A, Newton-Taylor B. The prevalence of elevated psychological distress among Canadian undergraduates: Findings from the 1998 Canadian campus survey. J Am Coll Heal. 2001;50(2):67-72. 11. Giamos D, Young Soo A, Suleiman A, Heather S, Shu-Ping C. Understanding Campus Culture and Student Coping Strategies for Mental Health Issues in Five Canadian Colleges and Universities. Can J High Educ. 2017;47(3):136-151. http://journals.sfu.ca/cjhe/index.php/cjhe/article/view/187957/pdf. 12. American College Health Association, ACHA-NCHA II. American College Health Association-National College Health Assessment II: Canadian Reference Group Executive Summary Spring 2016. Am Coll Heal Assoc. 2016:1-19. http://www.cacuss.ca/health_data.htm. 13. Ontario College Health Association. Towards a Comprehensive Mental Health Strategy: The Crucial Role of Colleges and Universities as Partners.; 2009. http://www.oucha.ca/pdf/mental_health/2009_12_OUCHA_Mental_Health_Report.pdf. Accessed November 1, 2020. 14. American College Health Association–National College Health Assessment (ACHA- 62 NCHA). Reference Group Executive Summary.; 2020. www.acha.org/NCHA. Accessed November 1, 2020. 15. Rotenstein L, Ramos M, Torre M. Prevalence of depression, depressive symptoms, and suicidal ideation among medical students: A systematic review and meta-analysis. JAMA. 2016;316(21):2214-2236. http://dx.doi.org/10.1001/jama.2016.17324. 16. Glauser W. Medical schools addressing student anxiety, burnout and depression. Can Med Assoc J. 2017;189(50):E1569-E1570. doi:10.1503/cmaj.109-5516 17. Goodman A, Joyce R, Smith JP. The long shadow cast by childhood physical and mental problems on adult life. Proc Natl Acad Sci. 2011;108(15):6032-6037. doi:10.1073/pnas.1016970108 18. Johnston DW, Schurer S, Shields M a. Evidence on the Long Shadow of Poor Mental Health across Three Generations.; 2011. https://www.york.ac.uk/media/economics/documents/herc/wp/12_20.pdf. 19. Simon GE, Von Korff M, Saunders K, et al. Association between obesity and psychiatric disorders in the US adult population. Arch Gen Psychiatry. 2006;63(7):824-830. doi:10.1001/archpsyc.63.7.824 20. Mental health of children and young people in Great Britain, 2004 - NHS Digital. https://digital.nhs.uk/data-and-information/publications/statistical/mental-health-of-children-and-young-people-in-england/mental-health-of-children-and-young-people-in-great-britain-2004. Accessed December 2, 2020. 21. Lasser K, Boyd JW, Woolhandler S, Himmelstein DU, McCormick D, Bor DH. Smoking and mental illness: A population-based prevalence study. J Am Med Assoc. 2000;284(20):2606-2610. doi:10.1001/jama.284.20.2606 63 22. Royal College of Psychiatrists. No Health Without Public Mental Health: The Case for Action. London; 2010. https://www.rcpsych.ac.uk/pdf/Position Statement 4 website.pdf. 23. Cooper J, Kapur N, Webb R, et al. Suicide After Deliberate Self-Harm: A 4-Year Cohort Study. Am J Psychiatry. 2005;162(2):297-303. doi:10.1176/appi.ajp.162.2.297 24. Fryers T, Jenkins R, Melzer D, Jenkins R, Melzer D. Social Inequalities and the Distribution of the Common Mental Disorders. Psychology Press; 2004. doi:10.4324/9780203496206 25. Chance of a lifetime preventing early conduct problems and reducing crime. Cent Ment Heal. November 2009. https://www.bl.uk/collection-items/chance-of-a-lifetime-preventing-early-conduct-problems-and-reducing-crime. Accessed December 2, 2020. 26. Fergusson DM, Horwood LJ, Ridder EM. Show me the child at seven: The consequences of conduct problems in childhood for psychosocial functioning in adulthood. J Child Psychol Psychiatry Allied Discip. 2005;46(8):837-849. doi:10.1111/j.1469-7610.2004.00387.x 27. Smetanin P, Adair C, Ahmad CE, And Khan S. Life at Risk Analysis of The Impact of Mental Illness in Canada Suggested Citation.; 2011. 28. Lim KL, Jacobs P, Ohinmaa A, Schopflocher D, Dewa CS. A new population-based measure of the economic burden of mental illness in Canada. Chronic Dis Can. 2008;28(3):92-98. doi:10.24095/hpcdp.28.3.02 29. Terenzini PT, Rendon LI, Lee Upcraft M, et al. The Transition to College: Diverse Students, Diverse Stories. Vol 35.; 1994. 30. Pedrelli P, Nyer M, Yeung A, Zulauf C, Wilens T. College Students: Mental Health Problems and Treatment Considerations. Acad Psychiatry. 2015;39(5):503-511. 64 doi:10.1007/s40596-014-0205-9 31. Andrews B, Wilding JM. The relation of depression and anxiety to life-stress and achievement in students. Br J Psychol. 2004;95(4):509-521. doi:10.1348/0007126042369802 32. American College Health Association–National College Health Assessment (ACHANCHA) II. Ontario Reference Group Executive Summary.; 2009. www.acha-ncha.org. Accessed October 29, 2020. 33. Xiao H, Carney DM, Youn SJ, et al. Are we in crisis ? National mental health and treatment trends in college counseling centers. Psychol Serv. 2017;14(4):407-415. doi:10.1037/ser0000130 34. Healthy Minds Study - Healthy Minds Network. https://healthymindsnetwork.org/hms/. Published 2018. Accessed November 30, 2020. 35. Eisenberg D, Hunt J, Speer N, Zivin K. Mental health service utilization among college students in the United States. J Nerv Ment Dis. 2011;199(5):301-308. doi:10.1097/NMD.0b013e3182175123 36. Bray SR, Born HA. Transition to University and Vigorous Physical Activity: Implications for Health and Psychological Well-Being. J Am Coll Heal. 2004;52(4):181-188. doi:10.3200/JACH.52.4.181-188 37. Mahmoud JSR, Staten RT, Hall LA, Lennie TA. The relationship among young adult college students’ depression, anxiety, stress, demographics, life satisfaction, and coping styles. Issues Ment Health Nurs. 2012;33(3):149-156. doi:10.3109/01612840.2011.632708 38. University Of Michigan M, Ii S, Eisenberg D, Downs MF, Golberstein E, Zivin K. Stigma and Help Seeking for Mental Health Among College Students. Med Care Res Rev. 65 2009;66(5):522-541. doi:10.1177/1077558709335173 39. U.S. Public Health Service. The Surgeon General’s Call to Action to Prevent Suicide. https://go-gale-com.ezproxy.library.ubc.ca/ps/i.do?p=HRCA&u=ubcolumbia&id=GALE%7C0QXW&v=2.1&it=aboutJournal&sid=kbart. Published 1999. Accessed December 1, 2020. 40. New Freedom Commission on Mental Health. Achieving the Promise: Transforming Mental Health Care in America. Executive Summary. Rockville, MD; 2003. http://hdl.handle.net/2027/pur1.32754077097628. Accessed December 1, 2020. 41. Sibicky M, Dovidio JF. Stigma of Psychological Therapy. Stereotypes, Interpersonal Reactions, and the Self-Fulfilling Prophecy. J Couns Psychol. 1986;33(2):148-154. doi:10.1037/0022-0167.33.2.148 42. College-Aged Adults Face Less Mental Health Stigma | Anxiety and Depression Association of America, ADAA. https://adaa.org/college-aged-adults-face-less-mental-health-stigma. Accessed January 4, 2021. 43. Stigma of mental illnesses decreasing, survey shows | HealthPartners. https://www.healthpartners.com/hp/about/press-releases/stigma-of-mental-illnesses-decreasing.html. Accessed January 4, 2021. 44. Among College Students, Mental Health Diagnosis and Treatment are Up, Stigma is Down. https://www.psychiatry.org/newsroom/news-releases/among-college-students-mental-health-diagnosis-and-treatment-are-up-stigma-is-down. Accessed January 4, 2021. 45. Eisenberg D, Golberstein E, Gollust SE. Help-Seeking and Access to Mental Health Care in a University Student Population. Med Care. 2007;45(7). https://journals.lww.com/lww-medicalcare/Fulltext/2007/07000/Help_Seeking_and_Access_to_Mental_Health_Care_in 66 _a.3.aspx. 46. Czyz EK, Horwitz AG, Eisenberg D, Kramer A, King CA. Self-reported barriers to professional help seeking among college students at elevated risk for suicide. J Am Coll Heal. 2013;61(7):398-406. doi:10.1080/07448481.2013.820731 47. Downs MF, Eisenberg D. Help Seeking and Treatment Use Among Suicidal College Students. 2012. doi:10.1080/07448481.2011.619611 48. Corrigan PW, Backs Edwards A, Qreen A, Thwart SL, Perm DL. Prejudice, Social Distance, and Familiarity with Mental Illness. https://academic.oup.com/schizophreniabulletin/article/27/2/219/1871015. Accessed October 29, 2020. 49. Gould MS, Velting D, Kleinman M, Lucas C, Thomas JG, Chung M. Teenagers’ attitudes about coping strategies and help-seeking behavior for suicidality. J Am Acad Child Adolesc Psychiatry. 2004;43(9):1124-1133. doi:10.1097/01.chi.0000132811.06547.31 50. Rickwood DJ, Deane FP, Wilson CJ. When and how do young people seek professional help for mental health problems? Med J Aust. 2007;187(7 Suppl):S35-9. http://www.ncbi.nlm.nih.gov/pubmed/17908023. 51. Rickwood D, Deane FP, Wilson CJ, Ciarrochi J V. Young people’s help-seeking for mental health problems. Aust e-Journal Adv Ment Heal. 2005;4(3):218-251. doi:10.5172/jamh.4.3.218 52. Hunt J, Eisenberg D. Mental Health Problems and Help-Seeking Behavior Among College Students. J Adolesc Heal. 2010;46(1):3-10. doi:10.1016/j.jadohealth.2009.08.008 53. Watkins DC, Hunt JB, Eisenberg D. Increased demand for mental health services on college campuses: Perspectives from administrators. Qual Soc Work Res Pract. 67 2012;11(3):319-337. doi:10.1177/1473325011401468 54. Jaworska N, De Somma E, Fonseka B, Heck E, MacQueen GM. Mental Health Services for Students at Postsecondary Institutions: A National Survey. Can J Psychiatry. 2016;61(12):766-775. doi:10.1177/0706743716640752 55. Stone GL, Vespia KM, Kanz JE. How good is mental health care on college campuses? J Couns Psychol. 2000;47(4):498-510. doi:10.1037/0022-0167.47.4.498 56. Smith TB, Dean B, Floyd S, et al. Pressing Issues in College Counseling: A Survey of American College Counseling Association Members. J Coll Couns. 2007;10(1):64-78. doi:10.1002/j.2161-1882.2007.tb00007.x 57. Mowbray CT, Megivern D, Mandiberg JM, et al. Campus mental health services: Recommendations for change. Am J Orthopsychiatry. 2006;76(2):226-237. doi:10.1037/0002-9432.76.2.226 58. Blanco C, Okuda M, Wright C, et al. Mental health of college students and their non-college-attending peers: results from the National Epidemiologic Study on Alcohol and Related Conditions. Arch Gen Psychiatry. 2008;65(12):1429. doi:10.1001/archpsyc.65.12.1429.Mental 59. Washburn C, Su-Ting T, Rita K, Jonny M. Post-Secondary Student Mental Health: Guide to a Systemic Approach.; 2013. http://www.cacuss.ca/_Library/ documents/PSSMH_Guide_To_Systemic_Approach_-_ CACUSS-CMHA_-_2013.pdf. 60. Kirsch DJ, Pinder-Amaker SL, Morse C, Ellison ML, Doerfler LA, Riba MB. Population-Based Initiatives in College Mental Health: Students Helping Students to Overcome Obstacles. Curr Psychiatry Rep. 2014;16(12). doi:10.1007/s11920-014-0525-1 61. Wang K, Varma DS, Prosperi M. A systematic review of the effectiveness of mobile apps 68 for monitoring and management of mental health symptoms or disorders. J Psychiatr Res. 2018;107:73-78. doi:10.1016/j.jpsychires.2018.10.006 62. Van Orden ML, Deen ML, Spinhoven P, Haffmans J, Hoencamp E. Five-year mental health care use by patients referred to collaborative care or to specialized care. Psychiatr Serv. 2015;66(8):840-844. doi:10.1176/appi.ps.201400238 63. National Research Council and Institute of Medicine. Preventing Mental, Emotional, and Behavioral Disorders Among Young People: Progress and Possibilites. (O’Connell ME, Boat T, Warner KE, eds.). Washington, D.C.: The National Academies Press; 2009. doi:https://www.ncbi.nlm.nih.gov/books/NBK32775/pdf/Bookshelf_NBK32775.pdf 64. Colman I, Murray J, Abbott RA, et al. Outcomes of conduct problems in adolescence: 40 Year follow-up of national cohort. BMJ. 2009;338(7688):208-211. doi:10.1136/bmj.a2981 65. Althoff RR, Verhulst FC, Rettew DC, Hudziak JJ, Van Der Ende J. Adult outcomes of childhood dysregulation: A 14-year follow-up study. J Am Acad Child Adolesc Psychiatry. 2010;49(11). doi:10.1016/j.jaac.2010.08.006 66. Copeland WE, Adair CE, Smetanin P, et al. Diagnostic transitions from childhood to adolescence to early adulthood. J Child Psychol Psychiatry Allied Discip. 2013;54(7):791-799. doi:10.1111/jcpp.12062 67. Copeland WE, Shanahan L, Costello EJ, Angold A. Childhood and adolescent psychiatric disorders as predictors of young adult disorders. Arch Gen Psychiatry. 2009;66(7):764-772. doi:10.1001/archgenpsychiatry.2009.85 68. Kessler RC, Amminger GP, Aguilar-Gaxiola S, et al. Age of onset of mental disorders: a review of recent literature. Curr Opin Psychiatry. 2007;20(4):359-364. doi:10.1097/YCO.0b013e32816ebc8c 69 69. Kim-Cohen J, Caspi A, Moffitt TE, Harrington HL, Milne BJ, Poulton R. Prior juvenile diagnoses in adults with mental disorder: Developmental follow-back of a prospective-longitudinal cohort. Arch Gen Psychiatry. 2003;60(7):709-717. doi:10.1001/archpsyc.60.7.709 70. M Vogt T, F Hollis J, Lichtenstein E, J Stevens V, Glasgow R, Whitlock E. The Medical Care System and Prevention: The Need for a New Paradigm. Vol 12.; 1998. 71. Musiat P, Tarrier N. Collateral outcomes in e-mental health: A systematic review of the evidence for added benefits of computerized cognitive behavior therapy interventions for mental health. Psychol Med. 2014;44(15):3137-3150. doi:10.1017/S0033291714000245 72. Ericsson A.B. Ericsson Mobility Report: on the Pulse of the Networked Society. http://boletines.prisadigital.com/ericsson-mobility-report-august-2014-interim.pdf. Published 2015. Accessed January 5, 2021. 73. Olff M. Mobile mental health: a challenging research agenda. Eur J Psychotraumatol. 2015;6(1):27882. doi:10.3402/ejpt.v6.27882 74. Taylor CB, Luce KH. Computer- and Internet-Based Psychotherapy Interventions. Vol 12.; 2003. 75. Spitzer RL, Kroenke K, Williams JBW. Validation and utility of a self-report version of PRIME-MD: The PHQ Primary Care Study. J Am Med Assoc. 1999;282(18):1737-1744. doi:10.1001/jama.282.18.1737 76. Kroenke K, Spitzer RL, Williams JBW. The PHQ-9: Validity of a brief depression severity measure. J Gen Intern Med. 2001;16(9):606-613. doi:10.1046/j.1525-1497.2001.016009606.x 77. Keum BT, Miller MJ, Inkelas KK. Testing the factor structure and measurement 70 invariance of the PHQ-9 across racially diverse U.S. college Students. Psychol Assess. 2018;30(8):1096-1106. doi:10.1037/pas0000550 78. Baas KD, Cramer AOJ, Koeter MWJ, Van De Lisdonk EH, Van Weert HC, Schene AH. Measurement invariance with respect to ethnicity of the Patient Health Questionnaire-9 (PHQ-9). J Affect Disord. 2011;129(1-3):229-235. doi:10.1016/j.jad.2010.08.026 79. Adewuya AO, Ola BA, Afolabi OO. Validity of the patient health questionnaire (PHQ-9) as a screening tool for depression amongst Nigerian university students. J Affect Disord. 2006;96(1-2):89-93. doi:10.1016/j.jad.2006.05.021 80. Huang FY, Chung H, Kroenke K, Delucchi KL, Spitzer RL. Using the Patient Health Questionnaire-9 to measure depression among racially and ethnically diverse primary care patients. J Gen Intern Med. 2006;21(6):547-552. doi:10.1111/j.1525-1497.2006.00409.x 81. Rowan-Kenyon H, Soldner ME, Kurotsuchi Inkelas K. The Contributions of Living-Learning Programs on Developing Sense of Civic Engagement in Undergraduate Students. J Stud Aff Res Pract. 2011;44(4). doi:10.2202/1949-6605.1867 82. Alschuler MS KN, Hoodin PhD F, Byrd PhD MR. Rapid Assessment for Psychopathology in a College Health Clinic: Utility of College Student Specific Questions. J Am Coll Heal. 2009;58(2):177-179. doi:http://dx.doi.org/10.1080/07448480903221210 83. Downs A, Boucher LA, Campbell DG, Dasse M. Development and initial validation of the Symptoms and Assets Screening Scale. J Am Coll Health. 2013;61(3):164-174. doi:10.1080/07448481.2013.773902 84. Spitzer RL, Kroenke K, Williams JBW, Löwe B. A brief measure for assessing generalized anxiety disorder: The GAD-7. Arch Intern Med. 2006;166(10):1092-1097. 71 doi:10.1001/archinte.166.10.1092 85. Byrd-Bredbenner C, Eck K, Quick V. Psychometric Properties of the Generalized Anxiety Disorder-7 and Generalized Anxiety Disorder-Mini in United States University Students. Front Psychol. 2020;11:550533. doi:10.3389/fpsyg.2020.550533 86. Plummer F, Manea L, Trepel D, McMillan D. Screening for anxiety disorders with the GAD-7 and GAD-2: A systematic review and diagnostic metaanalysis. Gen Hosp Psychiatry. 2016;39:24-31. doi:10.1016/j.genhosppsych.2015.11.005 87. Lee B, Kim YE. The psychometric properties of the Generalized Anxiety Disorder scale (GAD-7) among Korean university students. Psychiatry Clin Psychopharmacol. 2019;29(4):864-871. doi:10.1080/24750573.2019.1691320 88. Bártolo A, Monteiro S, Pereira A. Factor structure and construct validity of the Generalized Anxiety Disorder 7-item (GAD-7) among Portuguese college students. Cad Saude Publica. 2017;33(9):e00212716. doi:10.1590/0102-311x00212716 89. SAUNDERS JB, AASLAND OG, BABOR TF, DE LA FUENTE JR, GRANT M. Development of the Alcohol Use Disorders Identification Test (AUDIT): WHO Collaborative Project on Early Detection of Persons with Harmful Alcohol Consumption‐II. Addiction. 1993;88(6):791-804. doi:10.1111/j.1360-0443.1993.tb02093.x 90. Reinert DF, Allen JP. The Alcohol Use Disorders Identification Test: An Update of Research Findings. Alcohol Clin Exp Res. 2007;31(2):185-199. doi:10.1111/j.1530-0277.2006.00295.x 91. Cook RL, Chung T, Kelly TM, Clark DB. Alcohol screening in young persons attending a sexually transmitted disease clinic: Comparison of AUDIT, CRAFFT, and CAGE instruments. J Gen Intern Med. 2005;20(1):1-6. doi:10.1111/j.1525-1497.2005.40052.x 72 92. DeMartini KS, Carey KB. Optimizing the use of the AUDIT for alcohol screening in college students. Psychol Assess. 2012;24(4):954-963. doi:10.1037/a0028519 93. CQF. The Suicide Behaviors Questionnaire-Revised {SBQ-R)-Overview Psychometric Properties 1 Cutoff Score Sensitivity Specificity Adult General Population. 94. Osman A, Bagge CL, Gutierrez PM, Konick LC, Kopper BA, Barrios FX. The suicidal behaviors questionnaire-revised (SBQ-R): Validation with clinical and nonclinical samples. Assessment. 2001;8(4):443-454. doi:10.1177/107319110100800409 95. Martínez-Pérez B, De La Torre-Díez I, López-Coronado M. Mobile health applications for the most prevalent conditions by the world health organization: Review and analysis. J Med Internet Res. 2013;15(6). doi:10.2196/jmir.2600 96. Donker T, Petrie K, Proudfoot J, Clarke J, Birch MR, Christensen H. Smartphones for smarter delivery of mental health programs: A systematic review. J Med Internet Res. 2013;15(11). doi:10.2196/jmir.2791 97. Bradford S, Rickwood D. Acceptability and utility of an electronic psychosocial assessment (myAssessment) to increase self-disclosure in youth mental healthcare: a quasi-experimental study. BMC Psychiatry. 2015;15(1):305. doi:10.1186/s12888-015-0694-4 98. Bakker D, Kazantzis N, Rickwood D, Rickard N. Mental Health Smartphone Apps: Review and Evidence-Based Recommendations for Future Developments. JMIR Ment Heal. 2016;3(1):e7. doi:10.2196/mental.4984 99. American College Health Association. National College Health Assessment II: Canadian Consortium Executive Summary Spring 2019. Silver Spring, MD; 2019. 100. Corrigan P. How Clinical Diagnosis Might Exacerbate the Stigma of Mental Illness. Vol 73 52.; 2007. doi:10.1093/sw/52.1.31 101. Ballegooijen W Van, Riper H, Cuijpers P, Oppen P Van, Smit JH. Validation of online psychometric instruments for common mental health disorders : a systematic review. BMC Psychiatry. 2016;16(45). doi:10.1186/s12888-016-0735-7 102. Williams A, Larocca R, Chang T, et al. Web-based depression screening and psychiatric consultation for college students: a feasibility and acceptability study. Int J Telemed Appl. 2014;2014:580786. doi:10.1155/2014/580786 103. Chung H, Klein MC, Silverman D, et al. A Pilot for Improving Depression Care on College Campuses: Results of the College Breakthrough Series-Depression (CBS-D) Project. Vol 59.; 2011. 104. Mackenzie S, Wiegel JR, Mundt M, et al. Depression and suicide ideation among students accessing campus health care. Am J Orthopsychiatry. 2011;81(1):101-107. doi:10.1111/j.1939-0025.2010.01077.x 105. Khubchandani J, Brey R, Kotecki J, Kleinfelder J, Anderson J. The Psychometric Properties of PHQ-4 Depression and Anxiety Screening Scale Among College Students. Arch Psychiatr Nurs. 2016;30(4):457-462. doi:https://doi.org/10.1016/j.apnu.2016.01.014 106. King CA, Eisenberg D, Zheng K, et al. Online suicide risk screening and intervention with college students: A pilot randomized controlled trial. J Consult Clin Psychol. 2015;83(3):630-636. doi:10.1037/a0038805 107. Ebert DD, Franke M, Kählke F, et al. Increasing intentions to use mental health services among university students. Results of a pilot randomized controlled trial within the World Health Organization’s World Mental Health International College Student Initiative. Int J Methods Psychiatr Res. 2019;28(2):e1754. doi:10.1002/mpr.1754 74 108. Bewick BM, West R, Gill J, et al. Providing Web-based feedback and social norms information to reduce student alcohol intake: A multisite investigation. J Med Internet Res. 2010;12(5):e1461. doi:10.2196/jmir.1461 109. Mental Health Commission of Canada. Making the Case for Investing in Mental Health in Canada.; 2010. http://www.mentalhealthcommission.ca/English/system/files/private/document/Investing_in_Mental_Health_FINAL_Version_ENG.pdf. 110. REPORT FROM THE CANADIAN CHRONIC DISEASE SURVEILLANCE SYSTEM : 2015. 111. Lee A, Deevska M, Stillwell K, et al. A psychosocial assessment and management tool for children and youth in crisis. Can J Emerg Med. 2018;21(1):87-96. doi:10.1017/cem.2018.1 112. Virk P, Stenstrom R, Doan Q. Reliability testing of the HEARTSMAP psychosocial assessment tool for multidisciplinary use and in diverse emergency settings. Paediatr Child Heal. 2018;23(8):503-508. doi:10.1093/pch/pxy017 113. Doan Q, Black T. Behavioural disorders in Children. In: Tintinalli J, Stapczynski JS, Ma OJ, Yealy DM, Meckler GD, Cline DM, eds. Tintinalli’s Emergency Medicine: A Comprehensive Study Guide, Ninth Edition. 9e ed. New York: McGraw Hill; 2020:982-988. 114. Dolan MA, Fein JA, Shaw KN, et al. Technical report - Pediatric and adolescent mental health emergencies in the emergency medical services system. Pediatrics. 2011;127(5):e1356-e1366. doi:10.1542/peds.2011-0522 115. Doan Q and BT. Heartsmap psychosocial assessment and mental health resource 75 navigation tools. https://openheartsmap.bcchr.ca/ords/f?p=211:LOGIN_DESKTOP:::::: Published 2018. Accessed November 1, 2020. 116. Virk P, Laskin S, Gokiert R, et al. MyHEARTSMAP : development and evaluation of a psychosocial self- assessment tool , for and by youth. 2019:1-9. doi:10.1136/bmjpo-2019-000493 117. Doan Q, Wright B, Atwal A, et al. Utility of MyHEARTSMAP for Universal Psychosocial Screening in the Emergency Department. J Pediatr. 2020;219:54-61.e1. doi:10.1016/j.jpeds.2019.12.046 118. Gawel JE. Herzberg’s theory of motivation and Maslow’s hierarchy of needs. Pract Assessment, Res Eval. 1997;5(11):1996-1997. 119. Freitas FA, Leonard LJ. Maslow’s hierarchy of needs and student academic success. Teach Learn Nurs. 2011;6(1):9-13. doi:https://doi.org/10.1016/j.teln.2010.07.004 120. Canadian Standards Associations & Mental Health Commission of Canada. National Standard of Canada (Z2003:20): Mental Health and Well-Being for Post-Secondary Students. Toronto; 2020. https://www.mentalhealthcommission.ca/English/studentstandard. 121. Versaevel LN. Canadian Post-Secondary Students, Stress, and Academic Performance – A Socio-Ecological Approach. 2014. https://ir.lib.uwo.ca/cgi/viewcontent.cgi?article=4054&context=etd. 122. Woodgate RL, Gonzalez M, Stewart D, Tennent P, Wener P. A Socio-Ecological Perspective on Anxiety Among Canadian University Students. Can J Couns Psychother. 2020;54(3 SE-Articles/ Articles):486-519. https://cjc-rcc.ucalgary.ca/article/view/67904. 123. Mental Health Apps : How to Make an Informed Choice. 76 https://www.mentalhealthcommission.ca/sites/default/files/2018-01/eMH_app_eng.pdf. Published 2018. 124. Virk, P., Arora, R., Burt, H., Nelson, M., Davidson, J., Gadermann, A., Barbic, S., Cornish, P., Doan Q. Adapting HEARTSMAP-U: a psychosocial self-screening and resource navigational support instrument for post-secondary students. Manuscr Prep. 125. Messick S. Validity of psychological assessment: Validation of inferences from persons’ responses and performances as scientific inquiry into score meaning. Am Psychol. 1995;50(9):741-749. doi:10.1037/0003-066X.50.9.741 126. American Educational Research Association, American Psychological Association, National Council on Measurement in Education. The Standards for Educational and Psychological Testing. Washington, DC, DC; 2014. 127. Plake BS, Wise LL. What Is the Role and Importance of the Revised AERA, APA, NCME Standards for Educational and Psychological Testing? Educ Meas Issues Pract. 2014;33(4):4-12. doi:10.1111/emip.12045 128. Goodwin LD, Leech NL. The Meaning of Validity in the New Standards for Educational and Psychological Testing: Implications for Measurement Courses. Meas Eval Couns Dev. 2003;36(3):181-191. doi:10.1080/07481756.2003.11909741 129. Putnick DL, Bornstein MH. Measurement invariance conventions and reporting: The state of the art and future directions for psychological research. Dev Rev. 2016;41:71-90. doi:10.1016/j.dr.2016.06.004 130. Messick S. Meaning and Values in Test Validation: The Science and Ethics of Assessment. Educ Res. 1989;18(2):5-11. doi:10.3102/0013189X018002005 131. Nelson AA Jr. Research design: measurement, reliability, and validity. Am J Hosp Pharm. 77 1980. https://pubmed.ncbi.nlm.nih.gov/7395882/. Accessed January 8, 2021. 132. Downing SM. Reliability: on the reproducibility of assessment data. Med Educ. 2004;38(9):1006-1012. doi:10.1111/j.1365-2929.2004.01932.x 133. Sullivan GM. A primer on the validity of assessment instruments. J Grad Med Educ. 2011;3(2):119-120. doi:10.4300/JGME-D-11-00075.1 134. Keszei AP, Novak M, Streiner DL. Introduction to health measurement scales. J Psychosom Res. 2010;68(4):319-323. doi:10.1016/j.jpsychores.2010.01.006 135. Salkind N. Internal Consistency Reliability. In: Encyclopedia of Research Design. SAGE Publications, Inc.; 2012. doi:10.4135/9781412961288.n191 136. Salkind N. Test–Retest Reliability. In: Encyclopedia of Research Design. SAGE Publications, Inc.; 2012. doi:10.4135/9781412961288.n457 137. American College Health Association. ACHA-National College Health Assessment. . https://www.acha.org/documents/ncha/NCHA-III_Spring_2020_Reference_Group_Executive_Summary.pdf. Published 2020. Accessed November 1, 2020. 138. American College Health Association. Canadian Reference Group - Executive Summary Spring 2019.; 2019. www.acha-ncha.org. Accessed April 30, 2021. 139. Bradford S, Rickwood D. Young People’s Views on Electronic Mental Health Assessment: Prefer to Type than Talk? J Child Fam Stud. 2015;24(5):1213-1221. doi:10.1007/s10826-014-9929-0 140. Veloski J, Tai S, Evans ASA, Nash DDB. Clinical vignette-based surveys: a tool for assessing physician practice variation. Am J Med Qual. 2005;20(3):151-157. doi:10.1177/1062860605274520 78 141. Peabody JJW, Luck J, Glassman P, et al. Measuring the quality of physician practice by using clinical vignettes: a prospective validation study. Ann Intern Med. 2004;141(10):771-780. http://www.ncbi.nlm.nih.gov/pubmed/15545677. 142. Hjortsø S, Butler B, Clemmesen L, et al. The use of case vignettes in studies of interrater reliability of psychiatric target syndromes and diagnoses: A comparison of ICD‐8, ICD‐10 and DSM‐III. Acta Psychiatr Scand. 1989;80(6):632-638. doi:10.1111/j.1600-0447.1989.tb03036.x 143. Hanssen-bauer K, Gowers S, Aalen OO, et al. Cross-National Reliability of Clinician-Rated Outcome Measures in Child and Adolescent Mental Health Services. Adm Policy Ment Heal Ment Heal Serv Res. 2007;34(6):513-518. doi:10.1007/s10488-007-0135-x 144. Ellis LA, McCabe KL, Rahilly KA, et al. Encouraging young men’s participation in mental health research and treatment: perspectives in our technological age. Clin Investig (Lond). 2014;4(10):881-888. doi:10.4155/cli.14.61 145. Harris PA, Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research electronic data capture (REDCap)-A metadata-driven methodology and workflow process for providing translational research informatics support. J Biomed Inform. 2009;42(2):377-381. doi:10.1016/j.jbi.2008.08.010 146. Harris P a., Taylor R, Thielke R, Payne J, Gonzalez N, Conde JG. Research Electronic Data Capture (REDCap) - A metadata driven methodology and workflow process for providing translational research informatict support. J Biomed Inform. 2009;42(2):377-381. doi:10.1016/j.jbi.2008.08.010.Research 147. Walter SD, Eliasziw M, Donner A. Sample size and optimal designs for reliability studies. Stat Med. 1998;17(1):101-110. doi:10.1002/(SICI)1097-0258(19980115)17:1<101::AID- 79 SIM727>3.0.CO;2-E 148. Hallgren KA. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial. Tutor Quant Methods Psychol. 2012;8(1):23-34. doi:10.20982/tqmp.08.1.p023 149. Landis JR, Koch GG. The Measurement of Observer Agreement for Categorical Data. Biometrics. 1977;33(1):159. doi:10.2307/2529310 150. Hallgren KA. Computing Inter-Rater Reliability for Observational Data: An overview and Tutorial. Tutor Quant Methods Psychol. 2012;8(1):23-34. doi:10.1016/j.pestbp.2011.02.012.Investigations 151. Cohen J. A Coefficient of Agreement for Nominal Scales. Educ Psychol Meas. 1960;20(1):37-46. doi:10.1177/001316446002000104 152. Fleiss JL. Measuring nominal scale agreement among many raters. Psychol Bull. 1971;76(5):378-382. doi:10.1037/h0031619 153. Light RJ. Measures of response agreement for qualitative data: Some generalizations and alternatives. Psychol Bull. 1971;76(5):365-377. doi:10.1037/h0031643 154. Sim J, Wright CC. The Kappa Statistic in Reliability Studies: Use, Interpretation, and Sample Size Requirements. Phys Ther. 2005;85(3):257-268. http://dx.doi.org/10.1093/ptj/85.3.257. 155. Hallgren KA. Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial A Primer on IRR. 156. Shweta, Bajpai RC, Chaturvedi HK. Evaluation of Inter-Rater Agreement and Inter-Rater Reliability for Observational Data: An Overview of Concepts and Methods. J Indian Acad Appl Psychol. 2015;41(3):20-27. 80 157. Rosen J, Mulsant BH, Marino P, Groening C, Young RC, Fox D. Web-based training and interrater reliability testing for scoring the Hamilton Depression Rating Scale. Psychiatry Res. 2008;161(1):126-130. doi:10.1016/j.psychres.2008.03.001 158. Panchal N, Kamal R, Cox C, Garfield R. The Implications of COVID-19 for Mental Health and Substance Use | KFF. https://www.kff.org/coronavirus-covid-19/issue-brief/the-implications-of-covid-19-for-mental-health-and-substance-use/. Published February 10, 2021. Accessed April 30, 2021. 159. Brunner E, Probst M, Meichtry A, Luomajoki H, Dankaerts W. Comparison of clinical vignettes and standardized patients as measures of physiotherapists’ activity and work recommendations in patients with non-specific low back pain. Clin Rehabil. 2016;30(1):85-94. doi:10.1177/0269215515570499 160. Ströhle A, Gensichen J, Domschke K. The Diagnosis and Treatment of Anxiety Disorders. doi:10.3238/arztebl.2018.0611 161. Cunningham CE, Zipursky RB, Christensen BK, et al. Modeling the mental health service utilization decisions of university undergraduates : A discrete choice conjoint experiment. J Am Coll Heal. 2017;65(6):389-399. doi:10.1080/07448481.2017.1322090 81 Appendices Appendix A - HEARTSMAP-U (paper version) Housing & Material Security Guiding Questions: • Are my basic needs (housing, food, rent, tuition, insurance, medication) being met? • How well and easily are my needs met? • Do I have a physically safe and secure place to live? 0 1 2 3 My current situation: My basic needs are satisfactorily met. “Basic needs” defined as housing/utilities, food, tuition, rent/bills. My needs are met, but not always easily or satisfactorily. I am struggling to have my needs met. “Struggling” defined as close to losing housing, barely enough money for food/tuition, extremely stretched. At least one of my basic needs is not being met. Resources: I have accessed an advisor with student services, a community-based support program, or a family member to help me with my housing situation: Yes or No 82 Education and Activities Guiding Questions: • Am I attending, maintaining, and satisfied with my academics (courses, research) and other activities? “Other” defined as non-academic activities done alone or with others. 0 1 2 3 My current situation: I engage with and keep up with academics/other activities. “Engage” defined as your level of engagement or your satisfaction/dissatisfaction with how you’re engaging with academics/activities. “Academics/other activities” defined as anything related to your program of study (e.g., coursework, thesis, Co-op). I feel unengaged or struggle with academics/other activities but still manage them well. I am unengaged or struggle to keep up with academics/other activities and it’s affecting my performance. I have been unable to engage with academics/other activities or might fail out of my academic program. Resources: I have an academic advisor helping me with my educational situation: Yes or No 83 Relationships Guiding Questions: • Am I able to initiate, build, and maintain meaningful social connections? • Do I have emotionally supportive social connections (e.g., family, partner(s), friend(s), roommates, colleagues, educators) • How connected do I feel with my different relationships? *Some of your close relationships might be fine, but others may not be. Score based on any struggling social connections that are significant in your life (losing them would affect how emotionally supported you feel). 0 1 2 3 My current situation: I am emotionally supported and satisfied with my social connections. “Emotionally supported” defined as having social connections that provide me any of the following: empathy, love, trust, encouragement, active listening, or reassurance. I feel emotionally supported but feel challenged building/maintaining social connections. “Challenged” defined as feeling overwhelmed (negatively) by the thought of it. I lack emotional support and feel challenged building/maintaining social connections. I don’t feel emotionally supported at all and haven’t been able to build/maintain social connections. Resources: I have accessed mental health support (e.g., trained peer support, counsellor, family physician, spiritual care) and am being helped with my relationship: Yes or No 84 Thoughts and Anxiety Guiding Questions: • Do I feel in control of my thoughts? Are negative or unwanted thoughts disrupting my daily activities? 0 1 2 3 My current situation: I am in control of my thoughts. “in control of my thoughts” defined as I am comfortable with my thoughts, and if/when I’m not, I can stop/prevent them if I choose. I feel like I sometimes lose control of my thoughts, but I can keep up with self-care/daily activities. “lose control of my thoughts” defined as cannot prevent/suppress unwanted, obsessive, intrusive, or other thoughts. I feel like I often lose control of my thoughts, or I hear voices, which disrupts my self-care/daily activities. I feel like I’m losing control of my thoughts most of the time OR I hear voices or have thoughts telling me to do harmful things. Resources: I have accessed mental health support (e.g., family physician, mental health nurse, counsellor, psychiatrist) and am being helped for my thoughts: Yes or No 85 Thoughts and Anxiety Guiding Questions: • How often do I feel worried or distressed (stressed and anxious)? • Do I get so stressed or anxious I can’t participate in my daily activities (academics, work, extracurriculars)? 0 1 2 3 My current situation: I am currently not anxious. “Anxious” defined as feeling of worry, distress, unease, nervousness. I am feeling anxious, but I can keep up with self-care/daily activities. “Self-care” defined as actively protecting one's well-being and happiness (e.g., nutrition, sleep, exercise, hobbies etc.). I am feeling anxious and and/or been having panic attacks, which makes it difficult to keep up with self-care/daily activities. I don’t leave the house or perform self-care/daily activities on most days because of anxiety and/or panic attacks. Resources: I have accessed mental health support (e.g., family physician, mental health nurse, counsellor, psychiatrist) and am being helped for my anxiety: Yes or No 86 Substances & Behavioural Dependencies Guiding Questions: • Do I use substances (alcohol, marijuana, recreational drugs) or engage in addictive behaviours (excessive sex/gambling/gaming/exercise/eating/spending)? • Do I depend on these to get through the day or deal with stress? • Are they affecting my daily activities and self-care? “Behavioural dependencies” defined as Inability to cutdown or quit using substances (e.g., alcohol, marijuana, prescription or illicit drugs) or engaging in certain behaviors (e.g., excessive sex/gambling/gaming/exercise/eating/spending) despite best efforts. 0 1 2 3 My current situation: I don’t use substances (unless prescribed) and I don’t think I have any addictive behaviours. “Addictive” defined as unable to stop consuming a substance or engaging in a behaviour, despite best efforts and the negative impact they’re having. I sometimes use substances or engage in addictive behaviours, but I can stop/live without them. I often use substances or engage in addictive behaviours, and I can’t stop/live without them. I regularly use substances or engage in addictive behaviors. They’ve deterred my health, but I can’t stop. People around me may have expressed I need professional help. Resources: I have accessed resources to help me with my substance use or addictive behaviours: Yes or No 87 Safety Guiding Questions: • Do I feel trapped or hopeless, that there’s no way out of my suffering? Do I have thoughts about dying? • Have I thought about or tried hurting myself or others recently? • Have I seriously considered or tried ending my life recently? 0 1 2 3 My current situation: I don’t have thoughts about self-harm, dying, suicide, or hurting myself or others. I engage in non-suicidal self-harm, OR sometimes have thoughts about dying/suicide/hurting others but wouldn’t act on them. “Non-suicidal self-harm” defined as damaging one's own body tissue on purpose, NOT in line with social norms or with intention of suicide. I often have thoughts about dying/suicide but wouldn’t act on them. OR I have threatened to harm others but don’t intend to be violent. I am considering suicide (have a plan), I’ve recently attempted suicide, or I’ve been violent towards others and want to hurt them. Resources: A healthcare provider knows about my thoughts AND I have a safety plan with strategies to help me stay safe, and it's been working for me: Yes or No 88 Sexual Wellness Guiding Questions: • If I’m sexually active, do I make healthy sexual decisions? Do I have access to protective measures for contraception and STD prevention? • Do I put myself at-risk for STDs or unwanted pregnancy? 0 1 2 3 My current situation: I am not sexually active, or I make healthy sexual decisions. “Healthy sexual decisions” defined as know long-term partner’s sexual history, consistent protection usage (exception: seeking pregnancy), partner and I are both able to consent (not high/intoxicated). I always use protection, but I am unsure about my sexual partners. “Protection” defined as device used to prevent unplanned pregnancy and/or the spread of sexually transmitted disease (e.g., condoms, dental dams, contraception). My sexual partners and/or I sometimes don't use protection. My sexual partners and/or I regularly engage in sexual activities without protection or when at least one of us doesn’t have the capacity to consent (intoxicated/high). Resources: I have a doctor, nurse, or other health care professional who I see about my sexual health: Yes or No 89 Mood Guiding Questions: • Has my mood been changing a lot recently? • Do I feel unexplainably down (low, sad, irritable), numb (emotionless, flat), or elevated (uncontrollably happy, high) and how often? 0 1 2 3 My current situation: I don't feel down/numb or elevated unless something bad or good happens. I sometimes feel down/numb, or elevated without reason, but I can go about my daily life. I often feel down/numb or elevated. It affects my actions, or I struggle to go about daily activities Most days, I’ve felt down/numb (>2 weeks) or elevated (> 4 days) or been swinging between the two extremes, and it may prevent me from going about daily activities. Resources: I have accessed mental health support (e.g., family physician, mental health nurse, counsellor, psychiatrist) and am being helped for my mood: Yes or No 90 Abuse Guiding Questions: • In the past or present, have I experienced abuse (threats/assault/harassment/coercion) in any form (e.g., verbal, physical, sexual, emotional, or financial)? • Was I able to share these experiences with someone? Have I been able to receive help effectively? 0 1 2 3 My current situation: I don’t believe I’ve been abused in a way I deem significant. “Significant” defined as lasting impact on my emotional/physical/mental well-being or ability to go about my life. I believe I was abused in the past, but I’ve disclosed these experiences and am effectively working through them. “Effectively” defined as occurring to my satisfaction and addressing my needs so I can move forward. I believe I was abused in the past but have either not disclosed these experiences or begun working through them effectively. I believe I am currently being abused or forced to engage in activities against my will/consent. Professionals & Resources (for Mental Health) Guiding Questions: • Have I tried accessing help? Have I experienced barriers in getting help? • Am I satisfied with the help I’m currently receiving? 0 1 2 3 My current situation: I am satisfied with my mental health supports or have no needs currently. I am supported in some mental health needs, but I have outstanding needs I haven’t been connected to resources for. I’ve tried accessing help but haven’t received it yet (e.g., waitlisted) or didn’t find it helpful. I need mental health support but don’t know where to go, or I’ve been referred but haven’t tried accessing it. 91 Appendix B - Example Fictional Vignette Demographics and Overview: Age: 19-year-old. Case Overview: The student has moved to UBC from another country. It is his first time away from home. He is feeling very homesick and sad. Housing and material security: The student lives in a student residence on campus with a roommate. He has no financial concerns. He feels lonelier at home than anywhere else because he feels like all of their floormates are incredibly close with each other but not with him. He has brought up these concerns with his RA, who suggested that the student join some clubs or international student societies. Education and Activities: The student signed up for several student clubs at Clubs Day. He is doing well in his classes and feels that the material is easier than he expected it would be. Relationships: The student has met a few friends from his classes but is sad that he hasn't met anyone from his home country. He feels that none of his friendships are as close as he would like, because he cannot relate to the life experiences of his friends. He has good friends from back home, but it is difficult to stay in touch with them as they have lives of their own and the time difference is hard to manage. He has spoken to his counsellor about these worries. Thoughts and Anxiety: The student feels very homesick and misses his friends and family from his home country very much. He is stressed that he will disappoint his parents if he tells them he wants to go home. This stress does not interfere with his studies. He has spoken to a counsellor about these worries. Substance and Behavioral Dependencies: The student has never tried nicotine, cannabis, or any other recreational substances. He doesn't drink alcohol. Safety: The student has never considered suicide and has also never thought about hurting others. Sexual Wellness: The student is not currently sexually active. Mood: The student feels sad all the time and feels unable to stop crying. He has lost all appetite and rarely finds the motivation to leave the house, preferring to watch lecture recordings from home 92 instead of going to class. He sleeps at irregular hours and feels tired all the time. He has sought counselling for these issues a few times. Abuse: The student has faced some unintentionally racist questions and comments since arriving in Canada. Though these situations always make the student a bit uncomfortable when they occur, he tries not to let the comments bother him too much and is able to go about his daily life. The student has not faced abuse or harassment in the past. Professionals and Resources: The student is aware that he needs help to deal with his homesickness and has accessed counselling services, which have been somewhat helpful. He has also been referred to international student services and he is waiting on an appointment with them.
Thesis/Dissertation
eng
Population and Public Health
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Doan, Quynh
Gadermann, Anne
Universal psychosocial screening for post-secondary students with HEARTSMAP-U : evidence for inter-rater reliability
Text