@prefix vivo: . @prefix edm: . @prefix ns0: . @prefix dcterms: . @prefix skos: . vivo:departmentOrSchool "Applied Science, Faculty of"@en, "Nursing, School of"@en ; edm:dataProvider "DSpace"@en ; ns0:degreeCampus "UBCV"@en ; dcterms:creator "Billows, Jennifer Elizabeth"@en ; dcterms:issued "2011-03-08T00:34:18Z"@en, "2007"@en ; vivo:relatedDegree "Master of Science in Nursing - MSN"@en ; ns0:degreeGrantor "University of British Columbia"@en ; dcterms:description """In 2006, the University of British Columbia (UBC) School of Nursing (SON) received UBC Senate approval to limit admission to their baccalaureate nursing program to third-year, with advanced standing entry. The judicious selection of nursing students is increasingly salient in light of the looming retirement of 'baby-boomer' nurses, the goal of minimizing attrition from educational programs, and the resource-intensive admission procedure currently utilized at UBC, in which the applicants' academic and non-academic backgrounds are evaluated. A retrospective, correlational study of all 1,343 applicants to the program, for entry between January 2003 and January 2006, was undertaken. A series of multiple, logistic, and ordinal regression analyses were conducted to determine whether the current selective admission tools used by the SON added any valuable information over that obtained through the consideration of admission grade point average (GPA), in the selection of students who would be both academically and clinically successful in the program. The admission tools included a supplemental application, comprised of a personal statement and structured resume, which outlined the applicant's leadership, volunteer and employment experiences, motivation for choosing a career in nursing, and ability to work with others. Interviews were conducted of selected applicants to further evaluate applicants' suitability for nursing, particularly with respect to their clinical aptitude. The study variables included the applicants' demographics, their supplemental application and interview scores, and their admission GPA, as well as the outcome variables of interest: offers of program admission and in-program course grades. Although admission GPA was consistently predictive of the students' success, neither the supplemental application nor the interview scores were found to have predictive utility. The supplemental application had particularly poor inter-rater reliability. The variables found to besignificantly predictive of the students' cumulative average grade for selected nursing courses were gender, visible minority status, and admission GPA, accounting for approximately 27% of the variance in grades (pseudo-R²= 0.27). The results provide little evidence to justify continuation of the admission process currently in place."""@en ; edm:aggregatedCHO "https://circle.library.ubc.ca/rest/handle/2429/32144?expand=metadata"@en ; skos:note "J • PREDICTIVE V A L I D I T Y OF A N INNOVATIVE SELECTIVE ADMISSIONS PROCESS FOR B A C C A L A U R E A T E NURSING STUDENTS by JENNIFER E L I Z A B E T H BILLOWS B.A., Wilfrid Laurier University, 1995 B.Sc.N., McMaster University, 2000 A THESIS SUBMITTED IN PARTIAL F U L F I L L M E N T OF THE REQUIREMENTS FOR THE DEGREE OF M A S T E R OF SCIENCE IN NURSING in THE F A C U L T Y OF G R A D U A T E STUDIES THE UNIVERSITY OF BRITISH C O L U M B I A August 2007 © Jennifer Elizabeth Billows, 2007 A B S T R A C T In 2006, the University of British Columbia (UBC) School of Nursing (SON) received U B C Senate approval to limit admission to their baccalaureate nursing program to third-year, with advanced standing entry. The judicious selection of nursing students is increasingly salient in light of the looming retirement of 'baby-boomer' nurses, the goal of minimizing attrition from educational programs, and the resource-intensive admission procedure currently utilized at U B C , in which the applicants' academic and non-academic backgrounds are evaluated. A retrospective, correlational study of all 1,343 applicants to the program, for entry between January 2003 and January 2006, was undertaken. A series of multiple, logistic, and ordinal regression analyses were conducted to determine whether the current selective admission tools used by the SON added any valuable information over that obtained through the consideration of admission grade point average (GPA), in the selection of students who would be both academically and clinically successful in the program. The admission tools included a supplemental application, comprised of a personal statement and structured resume, which outlined the applicant's leadership, volunteer and employment experiences, motivation for choosing a career in nursing, and ability to work with others. Interviews were conducted of selected applicants to further evaluate applicants' suitability for nursing, particularly with respect to their clinical aptitude. The study variables included the applicants' demographics, their supplemental application and interview scores, and their admission GPA, as well as the outcome variables of interest: offers of program admission and in-program course grades. Although admission GPA was consistently predictive of the students' success, neither the supplemental application nor the interview scores were found to have predictive utility. The supplemental application had particularly poor inter-rater reliability. The variables found to be significantly predictive of the students' cumulative average grade for selected nursing courses were gender, visible minority status, and admission GPA, accounting for approximately 27% of the variance in grades (pseudo-R2= 0.27). The results provide little evidence to justify continuation of the admission process currently in place. iv TABLE OF CONTENTS ABSTRACT ii TABLE OF CONTENTS iv LIST OF TABLES vii LIST OF FIGURES ix ACKNOWLEDGEMENTS x DEDICATION xi CHAPTER ONE: INTRODUCTION 1 Background 1 Research Problem 2 Purpose Statement 2 Research Questions 3 CHAPTER TWO: LITERATURE REVIEW 5 Predicting the Future Educational Success of Nursing School Applicants 6 Grade Point Averages 8 Demographics 10 Age 10 Ethnicity 11 The Use of Multiple Indicators in Admission Processes 14 Supplemental Application Components '.. 16 Interviewing Applicants 20 Summary ; 27 CHAPTER THREE: METHODS 28 V Research Setting 28 The Admission Process 29 The Supplemental Application 30 The Interview 33 Changes Made to the Admission Process over Time 34 Students' Grades in Selected Nursing Courses 36 Research Design 38 Ethical Considerations 38 Data Collection 39 Measurement 40 Data Analysis 43 Methodological Limitations 47 CHAPTER FOUR: RESULTS 48 Applicant Demographics 48 Gender 48 Age 49 Grade Point Average 49 Educational Attainment 50 Visible Minority Status 50 Essay Grade 50 Supplemental Score 50 Correlates of the Supplemental Score 51 Factors Associated with the Supplemental Score 52 vi Interview Score 52 Factors Associated with the Likelihood of being Invited to an Interview 55 Offers of Admission 57 Factors Associated with the Likelihood of being Offered Admission 59 Academic Achievement 61 Nursing 310 61 Nursing 350 62 Nursing 450 65 Nursing 452 66 Nursing 330 68 Nursing 431/432 70 Cumulative Average for Selected Nursing Courses 73 Summary 75 CHAPTER FIVE: DISCUSSION 77 In Search of Predictive Validity 77 A Question of New Information 82 The Link to Prior Research 83 Limitations 84 Recommendations 87 Summary 89 REFERENCES 91 Appendix 1 98 Appendix 2 100 LIST OF T A B L E S Table 1. Applicants with Complete Data (September 2002 to January 2006 Entry) (N = 1,343).49 Table 2. Average Supplemental Application Scores by Age Group 51 Table 3. Simultaneous Multiple Regression Predictors for Supplemental Application Score 53 Table 4. Interview Scores by Academic Year 54 Table 5. Applicants Interviewed, Expected vs. Observed, by Age Group (N =1,193) 54 Table 6. Comparison of Mean Interview Scores and Essay Marks (N = 217) 55 Table 7. Logistic Regression Analysis of Interview Disposition on Prediction Variables 56 Table 8. Offers of Admission, Expected vs. Observed, by Age Group (N =1,261) 57 Table 9. Offers of Admission, Expected vs. Observed, by Visible Minority Status (N = 1,251) .58 Table 10. Admission GPA, Supplemental and Interview Scores by Offers of Admission 59 Table 11. Logistic Regression Analysis of Admission being Offered on Prediction Variables.... 60 Table 12. Simultaneous Multiple Regression Analysis of Nursing 310 Grade on Prediction Variables 63 Table 13. Simultaneous Multiple Regression Analysis of Nursing 350 Grade on Prediction Variables 65 Table 14. Simultaneous Multiple Regression Analysis of Nursing 450 Grade on Prediction Variables 67 Table 15. Simultaneous Multiple Regression Analysis of Nursing 452 Grade on Prediction Variables 69 Table 16. Ordinal Regression Analysis of Nursing 330 Grade on Prediction Variables 71 Table 17. Simultaneous Multiple Regression Analysis of Nursing 431/432 Grade on Prediction Variables 72 Table 18. Number of Selected Courses Completed 73 Table 19. Mean Admission GPA by Cumulative Average Grade for Selected Courses 74 Table 20. Ordinal Regression Analysis of Cumulative Average Grade of Selected Nursing Courses on Prediction Variables 75 IX LIST OF FIGURES Figure 1. The UBC SON Admission Process from 1997 to 2007 29 X A C K N O W L E D G E M E N T S The process of creating this thesis was truly enjoyable, thanks in part to my wonderful committee - Dr. Pamela Ratner, Marion Clauson, and Dr. Bernie Garrett. To my supervisor, Pam, working with you was such a pleasure. Not only was your statistical expertise invaluable, you are a mentor extraordinaire. Your continued support means a great deal to me and I feel fortunate, both academically and personally, to have had you as a supervisor. Perhaps just as important as what I learned from you (and there was so much), I had fun! I will sincerely miss our data analysis sessions and chats. To my committee members, Marion and Bernie, your ideas, feedback, and confidence in me facilitated my growth and I cannot thank you enough. It was delightful collaborating with and learning from both of you. I would also like to thank Drs. Joy Johnson and Susan Dahinten for their unique contributions to my academic and personal development. Finally, I wish to thank those in the School of Nursing at U B C who, in their individual ways, supported my endeavour, from clerical and program assistance to the support of Dr. Sally Thorne, Director. XI DEDICATION This work is dedicated to my daughter, Amelie. Ybu provided more inspiration and motivation than you can possibly comprehend. I love you, baby girl. 1 CHAPTER ONE: INTRODUCTION Background The School of Nursing (SON) at the University of British Columbia (UBC) began to allow third-year direct entry to the Baccalaureate of Science in Nursing program for advanced standing applicants in 1997. At that time, the admission process incorporated the evaluation of applicants on the basis of admission grade point average (GPA) only. In 2002, the process was amended to manage a larger number of applicants than there was space for in the program. Evaluation of a supplemental application package, comprised of a structured resume, a personal statement, and two reference letters, was implemented, in addition to an admission interview, a tool to further evaluate an applicant's \"clinical aptitude.\" In 2004, an on-the-spot essay requirement was added for the applicants who had been invited and presented for an interview. With \"baby-boomer\" nurses' retirements looming, increasing numbers of applicants to baccalaureate nursing programs, and the objective of minimizing attrition from undergraduate nursing programs, the selection of nursing students is increasingly salient. Nursing program seats are expensive to fill and universities are unable to re-fill seats when students are lost to attrition. Competitive admission processes are time-consuming and costly. It is essential that the process used be a reliable, valid, and fair method of choosing qualified, top-calibre students who are suitable for a program, are motivated, and have both academic and clinical aptitude. Other Canadian universities, such as McMaster University, the University of New Brunswick, and the University of Toronto, use selective admission processes similar to those of UBC' s SON, while others offer advanced standing admission based on admission GPA alone (e.g., Dalhousie University and the University of Calgary). Few studies have been conducted to examine the predictive power of admission criteria in baccalaureate nursing programs, 2 particularly within the realm of advanced standing degree entry. At U B C , there is little evidence within the School of Nursing to justify the use of the resource- and time-intensive admission process employed. Fraser's (2003) research work, previously conducted at U B C , examined students' clinical self-efficacy as an outcome related to the admission process and found that there was little predictive validity to warrant continuing with the current method of selecting students. Accordingly, it is important to evaluate the relative importance of each criterion in the selective admission process to determine whether any additional valuable information about the applicant is offered above that of GPA alone or if admission GPA is the best predictor of undergraduate nursing academic success. Research Problem The goals of the research presented here were to answer the following questions: 1. Is the U B C SON's admission process valid (i.e., does the application process succeed in identifying applicants that will succeed, both academically and clinically)? 2. Is the intensive process worthy of the investment of resources (i.e., does it offer more information, which is advantageous in the admission decision process than relying on admission GPA alone)? 3. Do the findings support or refute, by analysis of different outcomes, an earlier study concluding that the intensive procedure is not worth the resource expenditure? Purpose Statement The purpose of the present study was to examine whether the admission process (i.e., use of a supplemental application and interview) is a reliable, valid, and fair method of predicting 3 nursing students' success, as measured by the factors associated with being offered admission to the program and specific undergraduate academic and clinical course outcomes (i.e., grades). Of particular interest is whether the supplemental application and interview of applicants contribute any additional value, beyond that of admission GPA, in both selecting entrants and predicting their performance in the program. Research Questions Is the U B C School of Nursing's admission process a reliable and valid means of selecting students who will successfully complete the program? Or, more specifically: 1. What is the inter-rater reliability of: (a) the supplemental application and (b) the admission interview scoring procedures used? 2. What are the relationships between applicants' (a) supplemental scores and (b) interview scores and their (i) gender, age, visible minority status, and educational attainment at the time of application, (ii) admission GPA, (iii) likelihood of being offered admission, and (iv) grades in selected courses (for those admitted)? 3. Do applicants' demographic backgrounds contribute to the variability in their supplemental scores? 4. Do applicants' demographic backgrounds and supplemental scores predict the likelihood of being invited to an admission interview? 5. Do the (a) supplemental scores and (b) interview scores add to the prediction of students' success, as operationalized by their (i) admission status and (ii) grades in selected courses, after all other variables are considered? Summary A series of analyses were conducted to examine whether the selective admission tools used by the SON, namely the supplemental application and interview, added any additional information over that of admission GPA in selecting students who would be both academically and clinically successful in the program and to determine the reliability of these admission tools. CHAPTER TWO: LITERATURE REVIEW Little research has been conducted to analyze the predictive validity and reliability of admission criteria used by baccalaureate nursing programs, with few studies conducted in Canada and even fewer undertaken regarding advanced standing entry. A comprehensive search of the literature was performed to examine the state of the research regarding admission criteria for undergraduate nursing. Because of the paucity of research pertaining to nursing admission criteria, relevant materials were searched within medicine and other allied health professions, as well as the fields of education and employment research. The following databases were searched: Comprehensive Index of Nursing and Allied Health Literature (CINAHL), MEDLINE, Academic Search Primer, PsychlNFO, PsychARTICLES, PsychEXTRA, and the Educational Resource Information Center (ERIC). Retrievals were limited to English language works with a primary focus on the last 17 years (1990 to 2007), although earlier seminal works were considered where appropriate. The following alphabetically listed keywords, and combinations thereof, were searched, depending upon appropriate terms for the database in use: academic achievement, academic performance, admission(s) criteria, advanced standing, autobiographical essay, autobiographical letter, baccalaureate education, baccalaureate nursing, entry criteria, essay, ethnicity, grade point average, interview, non-traditional learners, nursing education, nursing students, personal statement, predictive measurement, school admission(s), student admission(s), supplemental application, and visible minority. This literature review begins by discussing the global topic of admission criteria used in nursing and medicine. The main academic predictor of concern for this research, grade point average, is then discussed, followed by non-academic variables examined in this thesis. Neither 6 scores on nursing licensure examinations1 nor job placement statistics were featured outcomes in this research and are therefore only briefly mentioned within the literature review. Studies specifically examining attrition or student retention as outcomes are similarly only briefly discussed. Predicting the Future Educational Success of Nursing School Applicants Wharrad, Chappie, and Price (2003) stated that there was a dearth of published studies examining nursing admission criteria used in the United Kingdom, a sentiment echoed by Sadler (2003) in the United States and Brown, Carpio, and Roberts (1991) in Canada. Indeed, Watson (2006) expressed his surprise that few of his nursing colleagues make use of a valuable and easily available resource, that is, data about their students, who are undoubtedly the long-term future of nursing. In searching the literature for this thesis, very few articles were found regarding nursing admission criteria in Canada, with only slightly more describing the prediction of admission into nursing education programs elsewhere in the world. Most of these articles discussed retention and scores on licensing examinations as outcomes. Campbell and Dickson (1996) conducted a meta-analysis of ten years' worth of studies (N = 47) examining the prediction of nursing students' success, as defined by retention, graduation, and National Council Licensure Examination for Registered Nurses (NCLEX) success. The authors found that, although there is literature addressing achievement prediction, consistently stable predictor variables have not been identified and there remains a need for 1 See Carpio, O'Mara, and Hezekiah (1996) for Canadian license testing outcomes, or Haas, Nugent, and Rule (2004) for research predicting success on the US National Council Licensure Examination for Registered Nurses. 7 more collaborative research among institutions to allow for consistency in design, methodology, and more refined sampling. Results from the studies that have been conducted are limited in their generalizability due to small convenience samples, descriptive designs, failure to sufficiently report demographics, and the varying predictors and outcomes measured. There has been a trend for health professional programs faced with larger numbers of applicants than available student spaces to select students by using both traditional, 'cognitive' and non-traditional, 'non-cognitive' measures (Dore et al., 2006; Kulatunga-Moruzi & Norman, 2002a; Salvatori, 2001). Although these qualities are often referred to in the literature as 'non-cognitive abilities' (e.g., Kulatunga-Moruzi & Norman, 2002a), Carrothers, Gregory, and Gallagher (2000) suggested that the domain is better described as \"emotional intelligence,\" a concept introduced in the early 1990s by Salovey and Mayer (1990). The term \"emotional intelligence\" is not sufficiently comprehensive to include all the criteria considered in the admission process, such as professional motivation; it does, however, more adequately describe abilities that, though not purely 'academic' in nature, require cognitive ability nonetheless. Gardner (1993) later expanded upon this idea when discussing \"multiple intelligences\". For the sake of consistency with the literature, however, these abilities will be referred to, in this thesis, as either non-cognitive or non-academic. The trend to examine non-cognitive measures has influenced practices in schools of nursing. A 1999 review of 160 American nursing schools found that 87% of the schools used college GPA as a B S N admission criterion, with 28% using reference letters, and 15% incorporating interviews (Crow, Handley, Morrison, & Shelton, 2004), although the authors did not clearly state whether these criteria applied to traditional or accelerated programs. In surveying the admission criteria of 28 US master of science programs in nursing, Streubert-Speziale (2002) found that 100% of the programs used letters of reference, 96% had minimum GPA cut-offs, 82% used personal statements, and only 11% required interviews. The following sections focus on the primary predictors studied in this thesis, these being admission GPA, demographics, ethnicity, supplemental application components, and interviews. Grade Point Averages Prior academic achievement, measured by admission GPA, has been consistently identified as one of the best and most stable predictors of admission to and performance in nursing, medical, and other health professional education programs (Didier, Kreiter, Buri, & Solow, 2006; Kulatunga-Moruzi & Norman, 2002a; Salvatori, 2001). The exact type of GPA used for an admission criterion varies across studies and depends upon the program for which admission is sought. Campbell and Dickson (1996) found in their meta-analysis of nursing studies that clinical, nursing, pre-admission, science, and cumulative GPA were all predictive of success (defined as retention, graduation, and N C L E X success), although general college cumulative GPA was the least predictive among the GPAs examined. Science GPAs were among the most consistently valuable predictors of nursing success, potentially because nursing courses rely heavily upon strong science backgrounds (i.e., courses in anatomy, physiology, biology, and biochemistry). Nursing clinical grades also were consistently predictive of retention, graduation, and N C L E X success. In a U K study of predictors of nursing academic success, the percentage of variation in course marks that could be explained by A-grade pre-registration qualifications decreased dramatically as students progressed through the nursing degree program (Wharrad et al., 2003). The students' performance in the fourth year of their nursing degree was predicted much less 9 accurately by pre-entry grades than for students in first year. It has been suggested that, because the beginning of a nursing program is so focused on course work, as opposed to clinical work, grades prior to admission will necessarily correlate better with grades in the first year of a program when the course work is the heaviest. In another U K study examining the influence of pre-entry qualifications on in-course nursing diploma grades, older female applicants and applicants with higher GPAs outperformed their counterparts in theory-based coursework (Kevern, Ricketts, & Webb, 1999). The use of GPA as a predictor is not without concern. The relationship between GPA and clinical performance is less conclusive (Salvatori, 2001). There is great variability in grades assigned by different institutions; admission committees often have opinions about the reputations of some undergraduate institutions; and grade inflation is always an issue (Didier et al., 2006). GPA can include substantial institutional bias and applicants who attend institutions with more stringent grading policies may be disadvantaged, raising the question of the usefulness of unadjusted GPAs. With respect to medical school admission, for which the Medical College Admission Test (MCAT) is usually written, multicollinearity between scores neither implies redundancy of measures nor lessens the bias in unadjusted GPAs, but rather can call attention to ways of adjusting GPAs based on information from groups of applicants from the same undergraduate institutions (Didier et al.). To examine more closely the problem of multicollinearity and to attempt to create adjusted GPAs, Didier et al. (2006) analyzed the admission decisions about over 17,000 medical 2 Another explanation for this finding is that poorly performing students do not continue to the fourth year. Thus, the academic achievement of the remaining students becomes increasingly homogeneous, which deflates the correlation with predictors. 10 school applicants over an 8-year period, and examined the shared variance between science GPAs and the science portion of the M C A T . The authors found high correlations between the two science indices (which was expected and estimated by cross-validation of the sample), an increase in science M C A T explained variance by using the adjusted science GPA, and improved predictive validity in the upper range of science GPA by removing institutional systematic influences. Although many nursing schools do not require standardized entry tests, the issue of adjusting GPAs is worth considering and lends weight to the argument that GPA alone may not be a sufficient predictor of students' success. Demographics Age In a meta-analysis of the prediction of nursing educational program success, Campbell and Dickson (1996) found that, of 47 studies included in their analysis, only 6 specified age and 3 specified gender as predictor variables. A New Zealand School of Nursing examined the relationship between entry criteria and in-course bioscience grades and found that, although there was no significant correlation with the overall \"gross entry\" criteria (i.e., grades for either high school leaving or mature applicants), age was significantly associated with bioscience grades, with older students achieving higher grades (van Rooyen, Dixon, Dixon, & Wells, 2006). Similarly, Salamonson and Andrew (2006) found that older students (>25 years) achieved higher grades in pathophysiology and nursing practice subjects. As previously mentioned, Kevern et al. (1999) found that older and female applicants achieved higher theory-based course grades compared with their younger and male counterparts. 11 Ethnicity This thesis research also examined the relationship between applicants' and students' success and their ethnicity. Statistics Canada (2007) defines a person as being of visible minority status i f s/he is non-Caucasian in race or non-white in colour. Under the Federal Employment Equity Act, Aboriginal persons are not considered to be of visible minority status (Statistics Canada). The United Nations recently suggested that Canada reconsider using the term visible minority because the use of the phrase itself may be discriminatory and lead to the making of distinctions based on race, colour, descent, or ethnic origin (CBC News, 2007). The United Nations did not, however, suggest an alternate or more appropriate phrase, although 'people of colour' has previously been proposed by others (Dhruvarajan, 2000). A concept often referred to instead is ethnicity. This thesis makes reference to visible minority status and ethnicity interchangeably. Price and Cortis (2000) defined ethnicity as a term describing people who share collective characteristics that distinguish them from others and that 'ethnicity' carries the implication of significant shared cultural concepts. Ethnicity classification poses several inherent concerns. Interestingly, although it may have been earlier socially desirable to 'treat everyone the same' on the premise of equality, colour blindness may in fact preclude giving each person's ethnicity and culture due respect (Price & Cortis). In 2005, McGaghie and Kreiter cited a key 2003 United States Supreme Court ruling that race and ethnicity are indeed legitimate variables to consider in admission processes, but the authors also argued that rigorous numerical admission methods are consistently supported in the literature as more predictive of success than are individual, case-by-case admission decisions based on race or ethnicity. As eloquently argued by Gerrish (2000), collecting ethnic data may be beneficial for highlighting discriminatory practices and advancing social policy change, but the racialization of data could also lead to further discriminatory practices. Researchers collecting ethnic data may move beyond the descriptive usefulness of the data to infer causality based on ethnicity. Offering explanations on the basis of ethnicity is problematic because ethnic categories are social constructs based on both belonging and group identity, are individually defined, and a person can ascribe to more than one ethnic identity simultaneously (Gerrish; Price & Cortis, 2000). Moreover, the categorical terms used often underestimate the heterogeneity within each construct. For example, white can refer to Irish or Greek ancestry, while Asian can denote Punjabi or Chinese heritage. It is essential, therefore, that researchers make undeniably clear the logic they use when creating ethnic categories and how these judgments are assigned and it is imperative that conclusions not be drawn beyond the data. Other variables including language, religion, and country of origin are useful in providing a more complete picture of ethnicity. Statistics Canada (2003), in conducting their Ethnic Diversity Survey, asked respondents to self-identify their ethnic or cultural identity and that of their ancestors, their country of birth, other countries lived in before coming to Canada, religion, racial or cultural group with which they identified, language learned in childhood, and language spoken most frequently (respondents could choose more than one option for many questions).3 3 Respondents were categorized into two main categories of interest: CBFA+ (Canadian, British, French, American, Australian, or New Zealander) and non-CBFA+, before being further distinguished within these two umbrella categories. 13 In attempting to review the literature to ascertain why certain ethnic applicants might be advantaged or disadvantaged in admission processes, however, very little research was found. Much of the available work focused on either reporting outcomes to show that a statistical disadvantage exists or asserting that consideration of ethnicity and race matters, without explaining why the disadvantage may occur. One can hypothesize that language barriers and racial discrimination are two key factors. An interesting article by Gittoes and Thompson (2007) suggested that it is not a question of advantage or disadvantage but rather admission criteria modelling misspecification. That is, in an analysis of Muslim students' likelihood of being offered admission to higher education in England, based on academic achievement, no systematic differences were found compared with white applicants. Repeated applications and omitted variables were two sources of model misspecification; the authors highlighted the need to accurately model variables to reduce error, especially when reporting an issue as sensitive as discrimination. An example of possible misspecification is found in a paper by Powis, James, and Ferguson (2007), where applicants to medical school with 'white', 'Chinese', and 'mixed' ethnic origin had higher pre-entry academic achievement than those of 'Asian', 'black', 'unknown', and 'other' ethnic origin; there was, however, no clear description of which origins the more vague categories of 'mixed' and 'other' may have been comprised.. Much of the nursing research regarding academic outcomes and ethnicity centres upon the relative success of graduates on the N C L E X test compared with white counterparts. In a meta-analysis of studies focused on the prediction of nursing school success, Campbell and Dickson (1996) found that, of 47 studies included in their analysis, only 5 studies specified ethnicity as a predictor. These studies indicated that students of non-white or ethnic status performed at lower standards than those of their comparison groups. For example, Haas, Nugent, and Rule (2004) found that African-American and Asian students had higher N C L E X failure rates than did white students, while Crow et al. (2004) found that Hispanic students' performance was poorer than that of others. Specific to in-course grades, a study from Australia showed that those defined as ethnic students (based on language spoken at home) performed poorly in pathophysiology and nursing practice courses compared with their \"non-ethnic\" counterparts (Salamonson & Andrew, 2006). A recent American study examined the potential impact of increasing the required admission GPA cut-off from 2.7 to 3.0 on the ethnic diversity of nursing students and found that, although the ethnic diversity of students graduating would not differ significantly with the higher GPA cut-off, the actual number of minority students admitted and graduating would fall by as much as 50% for specific ethnicities (DeLapp, 2006). Multivariate analysis was not possible because of insufficient sample size, but Chi-square analysis indicated no significant difference in the overall graduation rates between non-minority and minority students (here defined as Native American, Asian-Pacific Islander, Black, Hispanic, or other). The Use of Multiple Indicators in Admission Processes As early as 1971, accelerated or advanced-standing nursing programs for those with previous university experience began to appear in the United States as a response to the desire for easier exit to the work force, seamless transition to graduate school, and recruitment of those earlier displaced from the workforce by economic hardships (Seldomridge & DiBartolo, 2005). Surprisingly little literature exists, however, on the admission criteria applied and the predictive validity of non-traditional entry processes in relation to offers of admission and academic performance. What literature can be found often pertains to comparisons between traditional 15 high-school entry and non-traditional entry nursing school graduates on their N C L E X examination results (Seldomridge & DiBartolo). Recent work has begun to investigate various outcomes for advanced-standing students but sample sizes are often too small to produce anything beyond correlations (Seldomridge & DiBartolo), with admission GPA and nursing course grades often being associated with N C L E X scores. A multifactorial approach to assessing applicants has been espoused by many researchers, citing the need to assess both academic and non-academic criteria to better choose candidates that will not only succeed academically and professionally, but also to minimize attrition by choosing well-suited and motivated candidates (Campbell & Dickson, 1996). An underlying assumption of the advanced standing, selective admission program is that certain clinical aptitudes - personal and interpersonal qualities, such as communication skills, motivation, empathy, and maturity - are as necessary and valuable in becoming a successful professional nurse as is academic performance. As early as 1996, Leners, Beardslee, and Peters questioned whether the traditional usage of GPA alone was a sufficient enough criterion to successfully choose the type of nurses that the future would demand. In the medical field, it has been argued that predictors such as GPA and M C A T scores should be considered only after the assessment of personal qualities, to enable the selection of applicants whose personal characteristics may outweigh any academic record difficulties (Cohen as cited in Albanese, Snow, Skochelak, Huggett, & Farrell, 2003). This suggestion highlights the importance of assessing personal qualities effectively and in a reliable and valid manner. The range of exactly which personal qualities a professional school may want to assess, however, is wide. Albanese et al. determined that schools of medicine may assess anywhere from 4 to 87 different personal qualities. 16 A study following cohorts of medical school applicants in Belgium confirmed the importance of cognitive predictors such as GPA, while showing that situational judgment tests4 provide incremental predictive utility for students studying curricula with heavy emphases on interpersonal skills, whereas the difference disappears for students studying curricula with minimal interpersonal skill content (Lievens, Buyse, & Sackett, 2005). The tests increased predictive validity throughout the academic years, despite the threat of practice effects. The authors emphasized the need to carefully select and specify admission criteria and prediction constructs. Ferguson, James, O'Hehir, Sanders, and McManus (2003) used hierarchical regression analysis to find that, over the five years of a medical degree, GPA was predictive of pre-clinical (i.e., academic) success, whereas the personal statement (focusing on motivation) was more strongly predictive of clinical performance, supporting their recommendation of a mixed method of assessing medical school applicants. Supplemental Application Components Despite the varying titles attributed to an applicant's written work, the use of an essay, personal statement, or autobiographical letter has become a common tool in the health professions admission processes, with the goal of eliciting personal information that describes attributes not entirely academically related, commonly referred to as non-cognitive qualities (Kulatunga-Moruzi & Norman, 2002a). Despite controversy regarding the assessment of 4 A situational judgment test is an admission tool where applicants are provided a hypothetical scenario and asked to identify a response from a list of alternatives, allowing for the evaluation of applicants' critical thinking skills (Lievens, Buyse, & Sackett, 2005). 17 personal qualities, it is recognized that 'book smarts' do not necessarily translate into clinical success, nor are personal characteristics such as integrity easily taught, making their assessment at application an important task (Kulatunga-Moruzi & Norman). Although the personal statement has become a popular option, little evaluation has been done of its predictive validity in admission processes, which is of particular importance because personal statements often determine who is invited to an admission interview (Ferguson, Sanders, O'Hehir, & James, 2000; Salvatori, 2001). For the purposes of this literature review, the discussion of autobiographical essays and personal statements is combined because the focus of assessing non-academic qualities is often the same, despite the tool used and the time at which the tool may be administered during the admission process. Both the content and the amount of information presented within a personal statement or essay are important to assess, and content analysis may be worth conducting to more clearly elucidate the expressed abilities of the candidates that most closely match those desired by an admission committee (Ferguson et al., 2000). Personal statements may include motivation for or prior experience in the chosen field, methods of problem solving or dealing with interpersonal conflict, or applicants' philosophies of life, for example. Data collected from essays may provide additional information about motivation that could allow programs to better predict attrition; it has been found that those who tend to complete nursing programs had internalized the role of the nurse at the time of application, whereas those who drop out of nursing wrote of nursing as something external to themselves (Sadler, 2003). Personal statements and essays are excellent means of allowing applicants to express compelling personal characteristics that may have resulted in poor academic records, which may otherwise hinder their eligibility for admission (Albanese et al., 2003). Setting high GPA thresholds may prevent the identification of prized students who have suffered great personal setbacks and, in addition to the interview, written portions of the application may allow for these capable students to be discovered. Notwithstanding the potential benefits described above, concerns have been expressed about the use of personal essays or statements in health professional applications (Leners et al., 1996). Having students write essays on the spot may guarantee authorship, which may counteract the benefit gained for those who have had help with their personal statements (Dore et a l , 2006; Leners et al.). Young (1997), however, wrote that, in requiring written statements for residency selection, medical students are encouraged to lie and deceive to achieve their selection choice by offering convincing arguments about their motivation and goals so as to not jeopardize their career paths. Further, Ferguson et al. (2000) found that groupings of content statements may covary because of the tendency for certain activities to naturally covary (e.g., different types of volunteerism) or due to a reflection of an applicant's writing style. The fact remains that more articulate students may be advantaged in any setting. Applications to McMaster University Medical School in Hamilton, Ontario were found to be heavily weighted toward non-cognitive measures, despite these measures having low predictive validity for success post-graduation, as measured by licensing examination scores for both those admitted to McMaster and those refused admission and who completed their medical education elsewhere in Canada (Kulatunga-Moruzi & Norman, 2002a, 2002b). Despite those being admitted having higher non-cognitive scores than those refused admission, there were no significant differences found on the autobiographical ratings of those offered admission on the first round and those offered admission on the second round. The authors suggested that these tools may not be measuring the attributes that they are intended to measure, and are weighted too heavily, with GPA having an effect size of only .39, whereas the effect sizes were .79 for simulated tutorials and 2.37 for interviews (unfortunately the authors did not provide an effect size for the autobiographical submission). As mentioned, in hierarchical regression analysis, the autobiographical submission did not account for any of the variance in any part of the licensing examination outcomes (Kulatunga-Moruzi & Norman, 2002a). Personal statements and essays may reflect gender differences. Ferguson et al. (2000) found that female medical school applicants refer more to volunteerism while male applicants referenced their scientific and athletic propensities. These differences, however, were not statistically significant and the personal statement did not predict medical school success, regardless of gender. Regrettably, no research could be found examining cultural or ethnic influences on personal statements. Concerns also arise regarding the non-independence of ratings of autobiographical submissions (Dore et al., 2006; Kulatunga-Moruzi & Norman, 2002a). Autobiographical statements have been shown to have poor inter-rater reliability (r = .45), although internal consistency can be high (Kulatunga-Moruzi & Norman). Positing that high internal consistency can actually reflect a negative outcome of the scoring because it results from a halo effect (i.e., high performance on the first question of a tool influences raters' subsequent ratings within the same tool), Dore et al. attempted to control for threats to validity by varying the ratings of medical school autobiographical submission questions. Raters were asked to rate all applicants on Question 1 before moving on to rating all applicants on Question 2 (the horizontal method), compared to the control (vertical) method of scoring the entire application of Applicant A before moving on to that of Applicant B. It was determined that the horizontal method not only increased the independence of the observations, but also correlated better with the applicants' 20 interview scores, suggesting that halo effects were decreased and that the autobiographical submission could be used as a valid screening tool. In one of the few studies to address the use of an essay in nursing admission criteria, Brown et al. (1991) found high inter-rater reliability on essay scoring but moderate concurrent validity with the admission interview, and no predictive validity for success in completing the nursing program. They concluded, however, that the poor validity may have been reflective of the fact that admission essays and interviews measure distinct applicant qualities and may therefore be useful. The authors further reflected that more specific, short answer questions were needed for their essay tool, in combination with more in-depth orientation for the essay readers. More recently, Sadler (2003) determined that students who completed their nursing programs had strong correlations between their admission GPAs and essay scores, despite neither criterion being significantly related to completion vs. non-completion of the program. Sadler therefore suggested that when essays are used, they need to be sufficiently refined to assess information about motivation and internalization of the nursing role (see the aforementioned discussion of content analysis) that is not redundant with GPA, and that careful consideration should be given to the weighting of these criteria in the admission process. Albanese et al. (2003), in conducting a review of the assessment of personal qualities for medical school admission, could find no articles assessing the extent to which the personal statement conclusively offers information different than that obtained during the interview. Interviewing Applicants Although there is scant literature about the use of personal statements or essays in health professional school admission processes, the literature is replete with studies addressing the use 21 of the interview. Sixteen years ago, a review of the academic literature showed that there were few attempts made to establish the predictive validity of personal interviews as a tool in the admission process (Shahani, Dipboye, & Gehrlein, 1991). Shortly thereafter, Carpio and Brown (1993) found that no previous examples of interview reliability and validity studies in nursing admission processes existed, although the interview was already a widely established practice in medicine. The admission interview is almost consistently ranked as the most important tool of all selection criteria, particularly in medicine (Kulatunga-Moruzi & Norman, 2002b; Salvatori, 2001; Stansfield & Kreiter, 2007). Format, length, panel composition, structure, content, and scoring methods vary widely, however (Kreiter, Yin, Solow, & Brennan, 2004; Parry et al., 2006; Salvatori). The literature presently does not support the interview as being reliable or valid, however (Stansfield & Kreiter). Many authors have not provided enough statistical information to closely scrutinize the reliability of the interview process used (Campion, Palmer, & Campion, 1997; Kreiter et al., 2004). Others have offered different or imprecise definitions of reliability and how to measure it, thereby rendering a poor interview validity assessment (Kreiter et al.). Most of the studies of the validity of the interview technique are concentrated in the field of medicine (Courneya et al., 2005; Elam & Andrykowski, 1991; Kreiter et al., 2004; Kulatunga-Moruzi & Norman, 2002a, 2002b; Parry et al., 2006; Patrick, Altmaier, Kuperman, & Ugolini, 2001; Stansfield & Kreiter, 2007); however, two studies were found that examined the use of the admission interview in nursing (Carpio & Brown, 1993; Ehrenfeld & Tabak, 2000). The remaining studies were conducted in physiotherapy (Gabard, Porzio, Oxford, & Braun, 1997; Lewis & Smith, 2002), dentistry (Garni, Moshinsky, & Kapitulnik, 2003), pharmacy (Latif, 2005), veterinary medicine (Lewis, van Walsum, Spafford, Edwards, & 22 Turnwald, 2004), grouped health professions (Salvatori, 2001; Turnwald, Spafford, & Edwards, 2001), and general employment or academic research (Campion et al., 1997; Conway, Jako, & Goodman, 1995; Shahani et a l , 1991). Of the applied research examining admission interviews, six studies described three-member panel interviews. Carpio and Brown (1993) assessed the three-member panel interview with simulated applicants, whereas Courneya et al. (2005) had participants volunteer for two separate panel interviews after their admission-determining interview was completed. Ehrenfeld and Tabak (2000) compared individual interview vs. group interview vs. no interview, with admission status as an outcome of success, while Gafni et al. (2003) compared the interview with an open-ended questionnaire to determine admission interview efficacy. A three-member interview panel also was examined by Kulatunga-Moruzi and Norman (2002a, 2002b). Two-member panel interviews were occasionally used (e.g., Kreiter et al., 2004, who analyzed interviews on two separate occasions, and Patrick et a l , 2001). One-to-one interviews were examined by Elam and Andrykowski (1991) and Shahani et al. (1991). A l l of the interviews conducted were semi-structured. In a review of the use of admission interviews across medicine, veterinary medicine, dentistry, and optometry, Turnwald et al. (2001) noted that the interview has five very distinct purposes: (a) to gain new applicant information; (b) for admission decision making; (c) to verify application information; (d) to recruit candidates; and (e) for public relations purposes. The constructs assessed during the interview varied, although motivation was frequently cited (Carpio & Brown, 1993; Courneya et a l , 2005; Ehrenfeld & Tabak, 2000; Gafni et a l , 2003; Shahani et al., 1991). Communication and interpersonal skills are common themes (Carpio & Brown; Courneya et al.; Ehrenfeld & Tabak; Shahani et al.). Carpio and Brown noted that their 23 interview process for nursing sought to discover applicants' awareness of the program, problem-solving abilities, self-appraisal skills, and career goals, while Courneya et al. described a process that sought information about integrity and service from their medical school applicants. Only three of the research teams ascribed acceptable efficacy, reliability, or predictive validity (with admission status as the predicted outcome) to the admission interview (Carpio & Brown, 1993; Courneya et al., 2005; Patrick et al., 2001). Eight research teams declared that the use of the personal interview is questionable or limited in value and requires further evaluation (Ehrenfeld & Tabak, 2000; Elam & Andrykowski, 1991; Gabard et al., 1997; Gafni et a l , 2003; Kreiter et al., 2004; Kulatunga-Moruzi & Norman, 2002a, 2002b; Lewis & Smith, 2002; Shahani et al., 1991). The remaining authors emphasized the need for evidence of better reliability and validity to justify the continued use of structured interviews in the admission process. Turnwald et al. (2001) described important criteria for a structured interview: (a) questions are developed according to a job or position analysis; (b) each applicant is asked the same questions; (c) scoring is done on anchored rating scales with examples; (d) the interview panel records and rates the applicant's answers; and (e) each candidate proceeds through the same process. Highly structured interviews increase the reliability and validity of the technique, yet are very uncommon (Turnwald et al.). Campion et al. (1997) noted that an interview's reliability or validity can be enhanced by including more of the following components of structure: limiting prompting, having longer interviews, controlling the interviewers' viewing of the supplemental information, preventing questions from the candidate, using the same interviewers throughout the process, and preventing discussion between the interviewers. 24 Before the interview, faculty members are encouraged to: (a) develop a structured panel interview, based in part on a job analysis to identify the desired characteristics, skills, and qualities; (b) establish the purpose of the interview; (c) train the interviewers; and (d) establish methods to analyze the criteria (Lewis et al., 2004). Although structure increases the likelihood of protecting oneself and the school in the event of a lawsuit (Latif, 2005), the process of standardizing interviews is complex, time consuming, and costly and may not be worth the effort in terms of predictive value (Ehrenfeld & Tabak, 2000). Reliability is a building block of validity yet biases are often ignored by those who recommend the use of an interview in the admission process and by researchers. Threats to reliability include halo, contrast, first impression, order, and 'similar-to-me' effects, as well as leniency, severity, and central tendency biases (Gabard et al., 1997; Turnwald et al., 2001). A bias against women in interviews has been found, however bias against racial or ethnic minorities is more difficult to uncover (Turnwald et al.). Inter-rater reliability in admission interviews is most often examined and is moderate at best (Turnwald et al., 2001). Reliability varied greatly, from 0.22 to 0.97, in a review of interviews in the health professions, which may not be surprising given the variability of the interview structures and rating methods used across programs and schools (Salvatori, 2001). In the studies examined for this research, low to moderate reliability was common (Carpio & Brown, 1993; Kreiter et al., 2004; Kulatunga-Moruzi & Norman, 2002a). Courneya et al. (2005), however, found the reliability of a semi-structured, panel interview to be higher than that of individual interview inter-rater reliability. Stansfield and Kreiter (2007) recently discussed the conditional reliability of an interview, that is, the reliability of different scale ranges. Many applicants are evaluated on 25 Likert-type scales without sufficient precision to allow for the distinction between, for example, a mediocre and poor applicant, while the identification of stellar candidates may be possible. Given the low-to-moderate reliability, with 17% to 27% of the interview rating variance being accounted for by applicants' differences, it has been recommended that applicants have multiple raters and multiple interviews (Kreiter et al., 2001). To examine the issue of inter-rater disagreement further, Stansfield and Kreiter (2007) analyzed both observed interviews and a Monte Carlo simulation to discern whether inter-rater disagreement is homogeneous across a rating scale, and hypothesized that scale regions with less disagreement would show greater predictive validity. Their findings indicated that raters tended to agree more about the lowest and highest quality applicants and suggested that 3-point Likert scales would be just as informative as 5-point scales. Furthermore, ignoring the moderate ratings is preferable to using them, if the admission scores are weighted, because of the unreliable nature of the mid-range ratings. Treating moderate scores as missing data reduced the statistical 'noise' in their analysis, increased reliability, and allowed for the extreme scores, which had higher predictive validity, to become influential (Stansfield & Kreiter). Most of the authors who have addressed the validity of admission interviews do not support their use as valid predictors of students' success (Kulatunga-Moruzi & Norman, 2002a, 2002b; Turnwald et al. 2001). Kreiter et al. (2004) stated that the admission interview does not have the psychometric precision needed to justify being highly influential in admission decision making. Interview scores are likely to reflect random error, and heavily favouring the interview in the admission process may compromise the validity of the decision to admit (Kreiter et al.). In a 1995 meta-analysis, Conway et al. estimated the upper limits of validity to be 0.67 for highly structured interviews and 0.34 for unstructured interviews. 26 Interview validity depends on many factors, including the accuracy (truthfulness) of the applicant, as well as the interview questions' relevance to a job analysis, yet there is so much variation within fields (e.g., nurse researcher, clinician, or educator) that it is difficult to accurately characterize predictive qualities (Gabard et al., 1997). Erroneous construct measurement is a frequent problem. Academic selection tools may predict academic performance well, yet may lose their validity when predicting clinical performance (Turnwald et al., 2001). Many studies have compared the admission interview, which can attempt to assess non-cognitive dimensions, with various academic indices. Often, these interviews do not actually measure the non-cognitive dimensions for which they were created and instead assess cognitive abilities (Turnwald et al.). It has been found to be difficult to assess non-cognitive dimensions, and traditional cognitive criteria, particularly admission GPA, have the greatest utility in predicting academic and clinical performance (Kulatunga-Moruzi & Norman, 2002a, 2002b). In the studies examined for this research, the majority of authors concluded that interviews are of low predictive value. Elam and Andrykowski (1991) found science GPA and M C A T scores accounted for more than 25% of the variance in interviewers' ratings. Shahani et al. (1991) also found that the interview held less predictive power than academic measures (such as GPA) with respect to both admission decisions and academic success, as defined by first year GPA. Interviews were deemed to be of low predictive value with over 25% of attrition being described as occurring because of unpredictable personal reasons (Ehrenfeld & Tabak, 2000). The interview did not explain a significant amount of variance on outcomes related to medical licensing examinations (Kulatunga-Moruzi & Norman, 2002b). In one case, with low correlations between interview and academic scores, moderate correlations between interviews 27 and questionnaires, and interviews being correlated higher with final admission decisions than with questionnaires, the authors hesitated to abolish interviews altogether and suggested using a questionnaire as a pre-screening tool to then interview a smaller number of applicants (Gafni et al., 2003). Although several limitations were frequently cited in the articles reviewed, of particular relevance is the lack of accurate post-admission measurement of non-academic aptitudes, such as interpersonal and communication skills (Carrothers et al., 2000; Kulatunga-Moruzi & Norman, 2002b; Turnwald et al., 2001). Often only matriculating students are evaluated and generally on the basis of academic indices. The most pertinent gap, however, is the paucity of literature pertaining to the admission interview in nursing admission criteria and advanced standing entry. Only two studies examined interview reliability and validity in nursing admissions (Carpio & Brown, 1993; Ehrenfeld & Tabak, 2000). Summary In short, there is little conclusive evidence to support the continued use of supplemental applications and interviews in admission decision making. The research on the predictive validity of personal statements, essays, and interviews is equivocal. Although these tools may provide useful additional information about candidates, admission GPA remains the most stable predictor of academic success. More research is needed to determine the predictive validity of admission criteria, particularly those tools that putatively assess non-academic qualities. Research is lacking not only in nursing, but also in medicine, with very little work being conducted in Canada. 28 CHAPTER THREE: METHODS Research Setting The U B C School of Nursing (SON) obtained U B C Senate approval, in 2006, to limit admissions to those who met specified criteria for third-year advanced standing entry into their baccalaureate nursing program. The change in admission practices came about because of the demonstrated success of the advanced standing program (students had been admitted into first and third years of the program in 1997-2005), the desire to admit the best and the brightest, and because the number of seats in four-year baccalaureate nursing programs in British Columbia had been increased to address, in part, the nurse shortage. Students with a minimum of 48 credits, including mandatory English and human anatomy prerequisites, may apply to enter directly to the third-year of the program, so that they may complete 75 nursing-focused credits within two calendar years. Admission directly from secondary school was suspended in 2006 (see Figure 1 for an overview of the process and its changes over time). The SON has a large pool of highly qualified applicants and limited enrolment, and has therefore developed an admission process by which several applicant characteristics are considered. Applicants are evaluated on both academic and non-academic criteria. The use of non-academic criteria includes scores obtained from a personal statement (which addresses the applicant's reasons for seeking admission and understanding of the profession of nursing), and a resume (a structured form is provided in the application package) describing employment history, volunteerism, and extra-curricular experiences. In addition, applicants must provide the name and contact information of two individuals that can provide reference information (it is advised that at least one referee be a teacher, instructor, employer or supervisor). These application materials are believed to aid in the selection of those with not only academic 29 potential, but those who show motivation for nursing and the qualities and clinical aptitudes needed to be a professional nurse. Figure 1. The UBC SON Admission Process (1997 to 2007) Years Admission Process Components 1997-2002 Admission GPA is the only criterion for first- and third-year entry 2002 A supplemental application, consisting of a structured resume, personal statement, and written references, and an admission interview are implemented for third-year entry applicants 2004-05 The written references are discontinued and a written essay is required post-interview. Names and contact information for references are requested and used to clarify facts, i f necessary 2006 Third-year entry only is approved by the U B C Senate The Admission Process Applicants are required to have both a minimum GPA of 60% and 48 previous university credits to apply to the advanced standing program (a minimum C average or GPA of 2.0 on a 4-point scale). Currently, admission GPA accounts for 50% of the weight in the decision to be invited to an interview, with the supplemental application package comprising the remaining 50%. Of approximately 350 applicants per year, 30% to 40% with the highest ranked scores are invited to an interview. The interview is the final deciding factor, with approximately 90 applicants gaining admission to the program each year (30 in September and 60 in January). A short-answer essay is written following the interview. Although admission is not offered or 30 declined on the basis of the essay grade, a senior faculty member, who evaluates the essays, uses the grade as a reliability check by visually comparing the essay grade against the interviewers' rankings. When there is a large discrepancy between the scores, the faculty member confers with the interviewers to clarify how an applicant's rating was achieved. In 2001, the School of Nursing at U B C , in conjunction with the School of Midwifery, requested information from the Faculty of Medicine and Schools of Pharmaceutical Science and Rehabilitation Science, about the admission process used by these other health professional programs. The SON developed a system similar to that used in Medicine but focused the language on the profession of nursing. A structured resume, personal statement, and written references were part of the original supplemental application. The application has been adjusted each year with respect to scoring and weighting and, in 2005, the reference letters were removed from the required package. Names and contact information for references are instead requested and only used to clarify facts in few situations. The Supplemental Application The supplemental application score is a mark, given out of 50, based on the scoring of the personal statement and structured resume that the applicants submit with their application form. The personal statement currently comprises 10 marks and the structured resume accounts for the remaining 40 of a possible 50 total marks. The application seeks to gain understanding of an applicant's leadership capacity, ability to work with others, diversity of experiences, and aptitude and suitability for nursing, although the latter category is explored in greater detail, with respect to clinical aptitude, during the applicant's interview (if invited). An applicant's 31 attitudes, values, experiences, motivation, and understanding of health-care situations are appraised. Teams reviewing the supplemental applications are drawn from graduate students at the U B C SON. The training is conducted by a senior faculty member. A l l reviewers are given a 3-hour training session, during which the admission process is described, sample applications are scored, and opportunities for discussion are plentiful. Inter-rater reliability is assessed by comparing sample application scores, and the ratings are discussed to ensure consistency and understanding of the criteria. The Admission Committee expressed concern with the scoring of the supplemental applications for one of the admission cohorts. Two teams of paired raters were responsible for scoring the supplemental applications and the applicants were thus divided into two groups. One group was scored systematically higher than the other. The Admission Committee was faced with deciding how to impose a cut-off to identify suitable applicants for the next stage of the process (an interview) because, had the decision been made to invite the highest ranked supplemental applications, only applicants from the higher ranked group would have been invited, and the second group would have been disadvantaged. The Committee decided therefore to invite the top applicants from each of the two groups, resulting in a cut-off score of 30.5 from the higher ranked group with last names beginning with A - K and a cut-off of 26.0 from the lower ranked group with last names beginning with L-Z. Given their concerns about poor inter-rater reliability, the Admission Committee found it necessary to modify the rules about which raters' scores to use when there were two or more raters and substantial scoring discrepancies. The rationale for these various approaches is not known. In 39 cases, some of the raters' scores were ignored because of the scoring 32 discrepancies. For 25 applicants, the higher of the two scores was used. On one occasion, the lowest score was used, again because the discrepancy was deemed too great. Eight applicants who had 4 raters had only two of those scores used. Six times, an applicant's two lower scores were averaged and twice the two higher scores were averaged. A composite supplemental score was also averaged from three of four raters for 5 applicants, whose single lowest scores were dropped. One score was erroneously averaged and the true score was uncovered during this analysis. Additionally, clerical errors were made in juxtaposing scores when applicants shared same or similar last names. The majority of the errors were discovered after the interviews had been conducted; the remainder of the errors was identified by this researcher and her supervisor upon analysis of the data. As a result of these errors, some candidates were invited to an interview that should not have been, based on their corrected ratings, and, similarly, some applicants were not interviewed that should have been based on their corrected ratings. The Admission Committee made additional interview offers, where appropriate, and withdrew other applicants from consideration when their corrected ratings were found to be below the decided-upon cut-offs for admission GPA and the supplemental score. Some of the students therefore remained legitimate or expected outliers in the data analysis; that is, despite having a supplemental application score that, based on prediction, resulted in them having a high probability of being interviewed or admitted, they were not because of the change in cut-off and errors. Finally, several GPAs were not made available in time, resulting in some interviews being conducted on the basis of the supplemental application score alone. When the GPAs were 33 received by the School of Nursing, some of these applicants were found to be inadmissible and were removed from the list of accepted applicants. Of the 426 applicants invited to an interview over the 4-year study period, enough detailed information was provided to allow for inter-rater reliability assessment of both the interview and supplemental application scores. Forty nine percent of the supplemental applications (n = 210) were scored by two raters, thus allowing for inter-rater reliability analysis. Of those 210 applications scored by two raters (49% of all applications), the inter-rater reliability was found to be unacceptable at r - .52, p < 0.001. One hundred and ninety five applications were reviewed by only one rater (26% of all applications), with the remaining 21 being reviewed by 3 or 4 raters. For applications reviewed by 3 or 4 raters, inter-rater reliability of any 2 of the group of raters increased to r = .80, p < 0.001. The supplemental score inter-rater reliability was further considered upon examination of the aforementioned rule changes. For applications where one or more of the raters' scores were excluded, all original ratings assigned to an applicant were averaged and a new variable describing the 'true' supplemental score was created. This 'true' average was available for the 427 applicants who were invited to an interview. The correlation between this 'true' average and the supplemental score used by the Admission Committee following their rule modifications was r = .94, p < 0.001. This correlation provided support that some of the applicants' relative rankings changed as a result of the rule modifications and errors. The Interview The interviewers of the invited applicants are faculty members of the SON. The interviewers undergo a 3-hour training session with a senior faculty member, during which time 34 the selective admission process is fully explained. A mock 20-minute interview is conducted to give the interviewers an opportunity to practice scoring the interview and to allow for post-interview debriefing. Admission interviews are conducted with pairs of interviewers for each applicant. Each interviewer asks one half of the questions and both interviewers record their responses and their scores during the interview. The content of the interviews includes assessments of: (a) Motivation, (b) Nursing Awareness, (c) Coping and Problem Solving, (d) Ability to Relate, and (e) Self-Perception. At the request of the Admission Committee, the interview questions have been kept confidential. The total possible interview score increased from 70 in 2003 to 90 in 2006, when additional questions were added. Separate from the total score, Communication Skills and Overall Opinion are also each rated out of 5 marks. No probing or clarifying questions are asked during the interview. The applicant's admission GPA and supplemental application score are not known by the interviewers. Upon completion of each interview, the interviewers independently tally their scoring of the applicant's responses but may confer with each other to validate their observations. The two scores are averaged to create one final interview score. Inter-rater reliability for the interview was high, r = .89, p < 0.001 (N = 426). Changes Made to the Admission Process over Time Because of an increased number of seats and therefore increased probability of being admitted, in addition to changes in the scoring of the supplemental application and interview, academic year was entered into all prediction models. The information provided below is for descriptive purposes only. 2002-03 Academic Year 35 Admission GPA accounted for 50% of the rating, the supplemental application score accounted for 30%, and the interview (out of a total 70 marks) accounted for 20%. Decisions regarding the offers of admission were based on this weighting. 2003- 04 Academic Year Again, admission GPA accounted for 50% of the rating, with the supplemental application comprising 30% and the interview making up the final 20%. The interview had an added question regarding Self-Perception, providing a total of 75 possible marks for the interview score. 2004- 05 Academic Year A change in the admission process resulted in the admission GPA accounting for 50% of the initial rating, with the supplemental application score comprising the remaining 50%. From this initial ranking, the top approximately 100 applicants were interviewed, with the interview being the final deciding factor for approximately 65 offers of admission. The interview had 2 questions added to the domain, Ability to Relate, generating a final 85 total possible marks. An additional change for the 2004-05 academic year was the inclusion of a 1-2 paragraph essay to be written after the interview, from the applicant's choice of three topics: (a) The Health-Care System; (b) The Role of the Nurse, and (c) Knowledge. This essay, which a senior faculty member alone rates with a letter grade, was added to serve as a check and balance against the interview ranking and admission offers were not altered based on this essay. 2005- 06 Academic Year The process for the 2005-06 academic year repeated that of the 2004-05 academic year with one exception: the interview had one question added to the domain, Nursing Awareness, to allow a total of 90 possible marks. 36 Students' Grades in Selected Nursing Courses An underlying assumption of the advanced standing, selective admission program is that certain clinical aptitudes — personal and interpersonal qualities, such as communication skills, motivation, empathy, and maturity — are as necessary and valuable in becoming a successful professional nurse as is academic performance. The advanced standing admission criteria have been established in an attempt to measure clinical aptitude as well as academic, therefore an analysis of the validity of these criteria should naturally include both academic and clinical outcomes. For the purposes of this research, grades in six specific undergraduate nursing courses were analyzed as an operationalization of success. Academic aptitude was assessed by analyzing grades in theory courses and clinical abilities were measured via clinical course outcomes. Nursing 310 - The Core of Nursing Practice provides for the study of key concepts and frameworks fundamental to the practice of nursing, followed by Nursing 330 - Nursing Care of Individuals within the Context of Community, where clinical nursing practice is focused on acutely i l l individuals within the larger context of community. Both courses are clinically focused in nature, with students beginning to acquire the necessary skills and knowledge to become professional nurses. Students earn a percentage and letter grade for the completed course however the clinical practice component of the course is graded on a pass/fail basis. A pass grade in the clinical practice component of a course is required. Should a student receive a fail grade in the clinical practice component, both the practice and the theory component of the course must be repeated to advance in the program. In Nursing 350 - The Sociocultural Construction of Health and Illness, students seek to gain an understanding of the historical, social, economic, political, and cultural contexts of 37 health and illness, particularly within the Canadian health-care system. Students reflect on their own socio-cultural positions and how their personal contexts affect their care. Concepts of cultural safety, diversity, access, and vulnerability are explored, with links made to clinical practice and the role of the nurse. Course evaluation is by means of written assignments, presentations, and examinations. Students in Nursing 450 - Introduction to Research Utilization develop the skills to identify, locate, read, critique, and apply different forms of evidence in guiding their nursing practice. Evidence-based nursing and research utilization are emphasized, with the evaluation of students' learning assessed by written assignments and examinations. Nursing 452 - The Ethical Basis of Health Care allows students to examine moral problems in health care, and fosters critical reflection, ethical reasoning, decision making, and value exploration. Largely a discussion focused course, evaluation is based on written assignments, presentations, and examinations. While drawing on clinical practica for examples and discussions, Nursing 450 and 452 are theoretical in nature. In-course clinical aptitude is difficult to measure. A course that may be a better indicator of both clinical and academic success is NURS 431/432, Nursing Practice with Acute and Chronically 111 Populations, a final year course. The course objectives aim to help students achieve clinical competencies while facilitating their understanding of nursing theories and evidence-based practice. It is reasonable to believe that this course is an adequate measure of both academic and clinical abilities, thereby providing valuable insight into the predictive power of the various admission criteria for nursing success. Research Design The goal of the research was to determine the reliability and predictive validity of the admission process at the U B C School of Nursing, incorporating the supplemental application and interview as particular selection tools. A non-experimental, retrospective, correlational study was conducted to examine the relationships between the admission criteria and the outcome variables, namely standing obtained through the admission process and course grades for those admitted. Ethical Considerations The research proposal was approved by the student's supervisory committee. The consent of the Director of the School of Nursing was given (see Appendix 1). Ethical approval was sought from the University of British Columbia, Behavioural Research Ethics Board and was granted before data collection commenced (see Appendix 2). The data file was given to the researcher by a records clerk with unique, anonymized personal identifiers and all relevant data intact. The file was restricted to demographics, admission criteria scores, some annotations made by the clerk, and relevant course marks. Because the research was conducted using a secondary set of data already collected during application to the third-year entry program, none of the applicants was identified nor directly contacted, therefore rendering minimal risk to the applicants. Those who applied to the program had done so with the knowledge that the data would be kept and analyzed by the SON. Because the applicants were not contacted or directly involved in the research, there were no potential benefits to those who contributed information to the data set. 39 Only the supervisor, the members of the thesis committee, and the graduate student had access to the data. The supervisor kept a complete, electronic, secured data file with the applicants' names and student numbers included, to which only she had access. The supervisor was entitled to hold this information, as a member of the Leadership Team of the School of Nursing. The student's data file will be kept for a minimum of five years, as required by the Behavioural Research Ethics Board (BREB). No additional research will be undertaken without prior approval of the SON and BREB. Data Collection A l l applicants to the SON's third-year entry program between January 2003 and January 2006 were included. A l l applicants were assigned unique JD numbers. A l l demographic information, scores, and grades were linked with each individual student's unique identifier, as appropriate. The data entry clerk was asked to add information where relevant when a course was repeated or a student withdrew from either a course or the program. Recoding of variables was undertaken as necessary and is explained where appropriate in the section about measurement. The data were continually cleaned and checked. Scores ascribed to students incorrectly when they applied more than once (i.e., in subsequent years) and clerical errors (e.g., typing 03 for year vs. 03) were remedied. Any outliers in the data were investigated, particularly when applicants were found in the regression models to have large residuals and were classified incorrectly to the outcome of interest on the basis of their predicted scores. When necessary, the data were verified against the original application file by a senior faculty member. Missing data 40 were filled in where possible. The supervisor accessed the university's Student Information System to retrieve missing demographic information or course grades. Using the entire population of applicants during this study period yielded a sample of 1,343 applicants. Data from the applicants who applied in more than one year were treated independently because the application information was not carried over year to year. Although it may be unusual to count cases twice and these cases could be correlated, it is defensible to leave them in the analysis because the application process changed year to year as did the applicant pool with which they were compared. Complete data did not exist for all applicants because not every applicant was successful in progressing through the process or gaining entry to the program. In addition, some applicants did not submit all components of the application package. Accordingly, the number of subjects examined decreased to N = 427 after the final stage admission criterion (i.e., the interview score) was considered. Because the essay criterion was only implemented in 2004, the eligible number of applicants to be evaluated on the basis of its performance was N = 217. Measurement Age Date of birth was provided by the applicant. The age of the applicant was determined by subtracting the date of birth from September 1s t of the year for which the application was made. The data were positively skewed and leptokurtotic and, therefore, recoded into categorical data based on quartiles (i.e., \"< 23 years,\" \"23 - <25 years,\" \"25 - <30 years,\" and \"30+ years\"). Academic Year Academic Year refers to the session date of the year for which the application was made (i.e., \"2002-03,\" \"2003-04,\" \"2004-05,\" and \"2005-06\"). Gender The applicants indicated on their application materials whether they were female or male. Ethnicity The graduate student's supervisor, being part of the Leadership Team of the School of Nursing, had access to private applicant information (i.e., first and last names, ID photos once in the U B C on-line Student Information System, and demographics including country of citizenship) to which the graduate student did not have access. The supervisor conducted the original coding of ethnicity without the presence of the graduate student, to ensure applicant anonymity and confidentiality. The applicants were initially classified on the basis of their first and last names into one of four groups: \"European origin,\" \"Asian origin,\" \"South Asian origin,\" and \"Unknown.\" The coding was then \"verified\" by reviewing the photograph and citizenship status (including country of citizenship) for 44% of the applicants. Following this review, the variable was revised such that there were five codes: \"European origin,\" \"Asian origin,\" \"South Asian origin,\" \"Other origin (visible minority status, e.g., Aboriginal and black applicants)\" and \"unknown\" (for applicants without a photograph and a name that was not suggestive of membership in one of the described groups). Visible Minority Status The Ethnicity variable was collapsed into a binary variable, \"Visible Minority Status\" with \"Asian origin,\" \"South Asian origin,\" and \"Other origin (visible minority status)\" applicants being coded \"yes\" and all others coded \"no\" or \"unknown\". 42 Degree Status upon Admission The educational preparation of the applicants was coded: \"Full Degree Completed\" (which at minimum was a baccalaureate degree) or \"Partial Degree Completed\" (with a minimum of 48 university credits completed). Admission GPA The applicants' Grade Point Average, as of September 1st of the year for which the application was made, was recorded. Supplemental Application Score The applicants' averaged supplemental application scores, as used by the Admission Committee, were recorded. Invited for an Interview The first outcome considered was whether the applicant was invited for an interview (assuming that he or she was eligible; that is, the applicant had not withdrawn from the process, the minimum criteria were met, and the application package was complete). The variable was simply a binary outcome: \"yes\" or \"no.\" Interview Score The averaged interview score, of the two interviewers, as used by the Admission Committee, was treated as a continuous variable. Essay Grade A letter grade from A to D was assigned for each applicant that wrote an essay. The variable was treated as an ordinal variable and also grouped as a categorical variable. Admission Decision 43 Whether an applicant was offered admission was the final outcome of the admission process and was coded as \"yes\" or \"no.\" Those that had withdrawn their applications before a determination was made were not included in the analyses (i.e., they were treated as missing cases). Grades in Nursing 310, 330, 350, 431/432, 450, and 452, by academic year, term, and section The grades for each relevant course earned by the admitted students were recorded for each section, term, and academic year. The course grades were aggregated across the various offerings such that the term in which the students completed the courses was not considered. Cumulative Average Grade in Nursing 310, 330, 350, 431/432, 450, and 452 The students' grades for the 6 relevant courses were averaged. This variable was treated as a continuous variable and grouped into quartiles (i.e., \"< 78.9%,\" \"78.9% - <82.9%,\" \"82.9% -<85.2%,\" and \"85.2+%.\" Data Analysis The data were analyzed using SPSS vl5.0. Because much of the dataset had been created in Microsoft Office Excel by data clerks external to the research, and human errors are unavoidable, the potential existed for data entry mistakes, missing data, or errors in converting the data file to SPSS. The data were therefore first cleaned and checked. Frequency distributions were examined to identify outliers, out of range or implausible values, and non-normality. Upon finding questionable data, the entries were examined for the source of error and possible correction. Transformation of data was conducted where appropriate. The original, hardcopy 44 applicant files were reviewed, when required, by a senior faculty member. The data were continually screened throughout the analysis process. The thesis supervisor accessed the Student Information System Centre when biographical information or missing course grades were needed. To handle outstanding missing data, \"user-defined system missing\" codes were developed (e.g., 99 = unknown) and listwise deletion of missing cases was applied in analyses where appropriate. Although listwise exclusion assumes randomness of missing data, there is some support that when less than 10% of the data is missing, there is little difference in the bias of different forms of treatment for missing data (Othuon & Kishor, 1998). The variables with continuous data were examined for normal distribution. The residuals were examined and assumptions regarding homoscedasticity, multicollinearity, and normality of data were met, unless otherwise noted in the analysis. To further test the normality of the data pertaining to the development of the prediction model for the supplemental scores, Durban-Watson and Cook's Distance statistics were examined and found to be within acceptable ranges (Durban-Watson < 2, Cook's D <1, in keeping with the advice of Cohen, Cohen, West, & Aiken, 2003). The regression model for the supplemental scores was also cross validated by dividing the sample into two random samples. The first sample was run as a screening sample 2 2 2 and the second as a calibration sample. The adjusted R values (R adjScreening = -27; R adjCaiibration = .28) were close enough to verify that no significant differences existed between the two samples. The demographic data were examined via frequency distributions and descriptive statistics appropriate to the level of measurement. To answer questions pertaining to inter-rater reliability, Pearson's r was used. Typically, one thinks of using Cohen's Kappa or intra-class 45 correlation when examining inter-rater reliability. Garson (2007), however, noted that Cohen's Kappa is most appropriate when raters are using a scale measurement, and intra-class correlation is preferred only when sample size is small (n < 15) or when more than two tests are being correlated. With a large sample, Pearson's r does not make assumptions about the raters' means or the magnitude of the differences between the scores. This is therefore appropriate for this sample, where the ratings have varied greatly from year to year. By convention, a value of r > 0.70 is considered acceptable inter-rater reliability, although this fluctuates depending upon the researcher's criteria (Garson). Polit and Beck (2004) noted that, when measures are used to make decisions about individuals, such as nursing school admission, reliability coefficients should be r = 0.90 or higher, whereas r = 0.80 may be acceptable for group-level comparisons. To answer the questions related to the relationships between the demographic information, the admission criteria, and the outcome measures, the applicants' data were analyzed using bivariate correlations to assess the extent to which the variables covaried, Chi-square tests to examine expected vs. observed frequency of occurrences (as tests of association), and Mests or A N O V A to assess group mean differences, where appropriate. Non-parametric tests (Kruskal-Wallis, Martn-Whitney U , Kendall's T ) were conducted when the assumptions of normality were violated (e.g., skewed or kurtotic data). A N O V A group comparisons were conducted on continuous variables, which were also tested for homogeneity. When Levene's test of homogeneity of variance was significant, Games-Howell post-hoc analyses were conducted. With homogeneity present, Scheffe tests were used for the post hoc group comparisons. The main purpose of this research was to determine the predictive validity of the admission criteria used by the SON. Accordingly, particular attention was paid to the 46 supplemental scores, interview invitations and scores, admission offer decisions, and in-program course grades. Specifically of interest was the amount of unique variance each of the predictor variables contributed to each the outcomes examined. Data were entered into regression analyses to examine the relationships among the variables in the prospect of building models of predictive utility, that is, to determine whether academic and clinical achievement was associated with the admission criteria. Regression analyses, whether they be linear, logistic, or ordinal (the three kinds used in this research) are concerned with either prediction or explanation of an outcome dependent variable. The ability to control for certain variables and assess the unique variance that each variable contributes to the outcome makes these regression analyses advantageous to only examining bivariate relationships. A linear regression model was first estimated to analyze the factors associated with the supplemental scores. Because of concerns expressed by the Admission Committee pertaining to the value of the supplemental application, a Pratt Index was calculated in an attempt to determine the relative importance of each variable in the prediction model (i.e., whether it provided information that was not available through consideration of the admission GPA alone). Incorporating the Beta, r, and R 2 values associated with each variable in the supplemental score linear regression model, a Pratt Index, which allows researchers to establish a relative ranking of variable importance, provided a value for each variable, adding to a total of 1.0 across all variables. Allowing for more meaningful interpretation, the Pratt Index may be advantageous over semi-partial correlations, in part due to the Pratt's additive property and its allowance of the ordering of variables and expression of suppressor variables and multicollinearity (Ochieng & Zumbo, 2001). 47 A logistic regression model was then estimated to analyze the factors associated with being invited to an interview and a linear regression model was estimated to examine factors associated with the interview scores. A logistic regression model was then estimated to analyze factors associated with being offered admission to the nursing program. Regression models were also built to analyze the admitted students' nursing course grades. Ordinal regression models were analyzed for the NTJRS 330 grades and the admitted students' cumulative average grades on the six relevant courses; ordinal regression was employed because of violated assumptions of data normality. The remaining course grades were subjected to linear regression analyses. For each regression model, initial runs were calculated with all pertinent variables present in the analysis. As variables were found to be non-significant in the model, they were excluded from further model analyses. Variables that were non-significant in the final models were excluded from the final analysis, allowing parsimonious reporting for the purposes of prediction, to ensure maximal statistical power and inclusion of all information (i.e., minimizing the number of cases lost through listwise deletion). As with all analyses in this research, a .05 level of significance was adopted for the statistical tests. Methodological Limitations As described, concerns about rigor arose, given the issues associated with missing data, clerical errors, rule changes, and varying applicant assessors by year. The chosen solution to address these concerns, after data cleaning and checking, was to enter cohort academic year as a variable in every model. Despite our best efforts, we may not have been able to account for all variability in the data. Moreover, despite the use of multiple testing, we did not employ the Bonferroni correction at the time of analysis. 48 CHAPTER FOUR: RESULTS Applicant Demographics A l l applications to UBC's School of Nursing's Selective Admission (SA) third-year entry program, for commencement of studies between the academic years of September 2002 and January 2006, were examined in this research. Depending on the variable considered, the total number of subjects used in the analyses varied. Some applications to the program were not complete. Not all applicants were invited to participate in the latter stages of the application process (i.e., completing an interview and essay). Overall, 1,343 people applied to the SA program over the four academic years under consideration. Two hundred and ninety-five (295) applications were received for the 2002-03 academic year, 384 for the 2003-04 academic year, 331 for the 2004-05 academic year, and 333 applications were received for the 2005-06 academic year. Each application had the applicant's gender designated and 1,326 applicants provided a birth date. GPA upon application to the program was available for 1,019 applicants and 1,326 denoted their educational attainment to date (degree status). Classification of visible minority status was accomplished for 1,313 applicants. The specific demographics were examined for each year for which the applications were made and, because the cohort profiles did not differ significantly across the years, aggregated data were used for the more extensive statistical analyses (see Table 1). These analyses are discussed in more detail in the following sections. Gender Of the 1,343 applicants to the program over the 4-year period, 1,157 women (86.2%) and 186 men (13.8%; range of 11.2% - 16.3% for each year) submitted documentation. 49 Table 1. Applicants with Complete Data (September 2002 to January 2006 Entry) (N = 1,343) Variable Complete Data (N - missing data) Percentage of Total Applicants Gender 1,343 100% Age 1,326 98.7% Grade Point Average 1,019 75.9% Educational Attainment 1,326 98.7% Visible Minority Status 1,313 97.8% Supplemental Application Score 1,269 94.5% Interview Score 427 31.8% Essay Grade 217 16.2% Age The modal age of the applicants, as of September 1 of the academic year for which the application was made, was 20.8 years. The range of applicant ages spanned from 18.6 to 63.4 years of age, with quartiles of 22.4 (lower), 25.0 (second), and 29.3 (third) years of age. The age distribution over the 4-year academic period examined was non-normal, with a kurtosis of 3.7 and skewness of 1.7. A Kruskal-Wallis test of age across academic years yielded a non-significant result of 0.62, allowing us to conclude that the applicant cohorts did not differ significantly in age distribution across the four years of interest. Grade Point Average 50 Among the 1,019 applicants for whom an admission GPA was reported, the mean GPA was 75.0%, with a range of 53.0% to 93.8% over the 4-year period. Educational Attainment The total number of applicants who submitted an application with their prior education noted was 1,326 over the 4-year study period. Of these 1,326, approximately one half of the applicants (50.2%; range of 48.9% - 53.6% for each year) had a completed baccalaureate or higher degree and one half (48.5%) met the minimum requirement (i.e., at least 48 university credits). An additional 17 applicants did not provide information about their prior education. Visible Minority Status Of the applicants for entry between September 2002 and January 2006, 31.2% were deemed to be of visible minority status, with a range of 27.0% - 34.8% over the four years. Otherwise, 66.6% were classified as being of European origin and 2.2% of the applicants were considered to be of unknown ethnic origin. Essay Grade Following implementation of the post-interview essay requirement, in the last two application years, 217 candidates completed an essay. Of these 217, the modal grade was B (n = 85; 39.2 % of those who wrote the essay). The highest grade achieved was an A (n = 71) and the lowest was a D (n = 4). Supplemental Score Complete supplemental applications were submitted by 1,269 applicants. Of these, the mean score was 21.2 (SD = 6.5), with a range of 3 to 38 out of a possible 50 points. The scores were normally distributed. 51 Correlates of the Supplemental Score Age was significantly associated with the supplemental application scores, F(3, 1264) = 10.79,/? < 0.001. Post-hoc Scheffe analysis revealed that the two youngest age groups (< 23 and 23 - < 25 years) were given equivalent scores, on average, while the two oldest age groups (25 -< 30 and 30+ years) were scored alike. The two younger age groups were each scored significantly differently from each of the two older age groups, with the trend being for older applicants to be scored higher (see Table 2). Table 2. Average Supplemental Application Scores by Age Group Variable Coding Average Supplemental Application Score SD Age Group (years) <23 20.1 6.2 23 - <25 20.2 6.4 25 - <30 22.2 6.6 30+ 22.1 6.6 Male and female applicants achieved similar supplemental application scores (^(1267) = .36, p < 0.72), while applicants with completed academic degrees (n = 650) received higher supplemental scores (21.7; SD = 6.6) than those with a partial degree (20.5; SD = 6.4) (t(1266) = -3.23, p = 0.01). Visible minority applicants (n = 403) received scores lower (19.6; SD = 6.2) than those of non-visible minority applicants (22.0; SD = 6.7) (f(1240) = 5.92, p < 0.001). Supplemental application scores became significantly higher from 2002 to 2006 (F(3, 1265) = 93.80,/? < 0.001), with academic years three (2004-05) and four (2005-06) being significantly higher than each preceding year. Grade Point Average at admission was weakly correlated with supplemental application scores (r - .17, p - 0.01). Factors Associated with the Supplemental Score A simultaneous multiple regression analysis was conducted to identify the factors that contributed unique variance in the supplemental application scores. The model was statistically significant (F(8, 982) = 45.30, p < 0.001), with 26% (R2ad} = .26) of the variance in scores being accounted for by the predictor variables. Age, visible minority status, academic year, and admission GPA were significant predictor variables (see Table 3). Gender and degree status were not significant in the final model. Multicollinearity and variance inflation factor (VTF) values were within appropriate limits, allowing confidence in the final model. Assumptions regarding residual distribution were satisfied. Interview Score During the 4-year study period under examination, 1,193 applicants were eligible for an interview (i.e., they did not voluntarily withdraw from the application process or had incomplete files or missing prerequisite courses) and 427 applicants (35.8% of those eligible) were invited to an interview. The mean interview scores per year are shown in Table 4. Correlates of Interview Invitations and Scores The likelihood of being invited to an interview did not differ on the basis of gender (X2 (1, N = 1,194) = .52,p = 0.47) with 374 of 1,032 female applicants (36.2%) being interviewed and 54 of 162 male applicants (33.3%) interviewed. Forty percent of the degree holding applicants were interviewed (n = 241); they were 1.5 times more likely to be invited for 53 Table 3. Simultaneous Multiple Regression Predictors for Supplemental Application Score Variable Coding B SE Beta t P Pratt Constant 6.44 2.23 — 2.89 .00* — Age Group (years) <23 — — — — — -23 - <25 -.45 .52 -.03 -.86 .39 0.01 25 - <30 1.28 .47 .09 2.73 .01* 0.04 30+ 1.74 .51 .11 3.43 .00* 0.04 Visible Minority Status No — — — ~ — — Yes -1.97 .39 -.14 -5.01 .00* 0.09 Academic Year 2002-03 — — — — — — 2003-04 -.79 .50 -.06 -1.56 .12 0.05 2004-05 1.87 .52 .13 3.60 .00* 0.00 2005-06 6.62 .53 .44 12.59 .00* 0.67 Admission GPA .18 .03 .17 6.18 .00* 0.11 R ' a d j = 0.26. * denotes statistical significance in linear regression model. — denotes referent group. an interview than were those with partial degrees (31.5%; n = 187) (%2 (1, N = 1,193) = 9.66, p < 0.01) (OR = 1.46; 95% CI: 1.15 - 1.85). Applicants over the age of 25 years (85.2%) were more likely to be interviewed than were applicants under the age of 25 years (58.7%) (x2 0, N = 1193) = 22.36, p < 0.001) (see Table 5). 54 Table 4. Interview Scores by Academic Year Academic Maximum N Mean SD Lowest Highest Year Score Possible Interview Score Score Score 2002- 03 70 109 53/7 S\\6 24~3 68.0 2003- 04 75 99 60.8 9.2 33.5 74.5 2004- 05 85 107 67.8 10.3 36.8 84.5 2005- 06 90 112 71.2 11.0 35.5 89.5 Table 5. Applicants Interviewed, Expected vs. Observed, by Age Group (N =1,193) Variable Coding Expected Count Observed Count Percentage of Age Group Interviewed Age Group <23 132.4 109 29.5 (years) 23 - <25 84.7 69 29.2 25 - <30 117.3 137 41.9 30 + 93.6 113 43.3 The mean admission grade point average was significantly greater for those interviewed (77.2%; SD = 5.5) than not interviewed (73.0%; SD = 6.1) (Levene's F= 0.4,/? < .01) (r(944.97, equal variances not assumed) = -11.26,/? < 0.001). Visible minority applicants (N = 110) were 38% less likely to be interviewed compared with non-visible minority applicants (N = 314) (x2 (1. N = 1,171) = 13.61,/? < 0.001) (OR = .62; 95% CI: .47 - .80). The proportion of 55 applicants offered an interview did not differ significantly on the basis of academic year ( j 2 (3, N = 1,194) = 5.91,/? = 0.17), while those with higher supplemental application scores (M = 26.5, SD = 4.5) were more likely to be interviewed (those not interviewed had mean supplemental application scores of 18.0, SD = 5.4) (Levene's F= 6.9,/? < .01) (/(1009.56, equal variances not assumed) = -29.20,/? < 0.001). The interview scores and post-interview essay grades were analyzed; applicants who excelled in the interview also received higher grades on their essay (F(2, 214) = 17.45,/? < 0.001) (see Table 6). Table 6. Comparison of Mean Interview Scores and Essay Grades (N = 217) Year Variable Coding N Mean Interview Score 2004-05 Essay Grade Dto C+ 12 51.2 Bto B+ 51 67.7 A 42 73.0 2005-06 Essay Grade Dto C+ 34 66.5 Bto B+ 49 72.3 A 29 75.0 Factors Associated with the Likelihood of being Invited to an Interview A logistic regression analysis was conducted to identify the factors that were associated with being invited to an interview. This explanatory model accurately predicted 93% of those interviewed and 89% of those not interviewed, for an overall percentage of 91.3% correct; 56 however, the model had a statistically significant Hosmer and Lemeshow test (%2 (8, N = 976) = 116.10, p < 0.001) indicating that the model was a poor fit with the data and predicted values significantly different than those observed. The Nagelkerke pseudo-R was .78, with academic year, admission GPA, and supplemental application score being significant in the model (see Table 7). Table 7. Logistic Regression Analysis of Interview Disposition on Prediction Variables Variable Coding B SE Wald P Odds Ratio 95% CI Constant -29.43 2.39 151.18 .00* — — Academic Year 2002-03 — — — — — — 2003-04 -.73 .34 4.71 .03* .48 .25 - .93 2004-05 -1.53 .36 18.08 .00* .22 .11 - .44 2005-06 -5.16 .50 108.91 .00* .01 .00 - .02 Admission .21 .02 78.02 .00* 1.23 1.18 — 1.29 GPA Supplemental .69 .05 206.83 .00* 1.98 1.81 - 2.18 Score Pseudo-R\" = 0.78. * denotes significance in the logistic regression model. — denotes referent group. 57 Offers of Admission During the 4-year period under examination, of 1,343 applicants to the program, 983 applicants (73%) were refused admission, 294 applicants (22%) were offered admission, and 66 applicants (5%) withdrew their applications and were not further assessed. Two hundred and forty-nine students (85%) accepted the offer of admission, 3 deferred entrance to the program until the following year (1%), and 42 applicants declined the offer (14%). Offers of admission did not differ on the basis of gender (%2 (1, N = 1,277) = .18, p = 0.67) with 255 of 1,098 female applicants (23.2%) receiving offers and 39 of 179 male applicants (21.8%) receiving offers. One hundred and sixty-six applicants with full degrees were offered admission, proportionally significantly more than those with partial degrees ( ( 1 , N = 1,261) = 6.33, p = 0.01) (OR = 1.40; 95% CI: 1.08 - 1.82). Applicants over the age of 25 years were significantly more likely to be offered admission (55.0%) compared with applicants under the age of 25 years (38.3%) (^ 2 (3, N = 1,261) = 14.31,p < 0.01) (see Table 8). Table 8. Offers of Admission, Expected vs. Observed, by Age Group (N =1,261) Variable Coding Expected Observed Count Percentage Count Offered Admission Age Group <23 9L2 73 18?7 (years) 23-<25 57.1 48 19.6 25-<30 79.7 100 29.2 30+ 66.0 73 25.8 58 The admission grade point average was significantly higher for those offered admission (77.6%; SD = 4.9) than for those refused admission (73.6%; SD = 6.3) (Levene's F= 20.88,/? < .01) (r(694.89, equal variances not assumed) = -10.63,/? < 0.001). Visible minority applicants were 49% less likely to be offered admission than were non-visible minority applicants (%2 (1, N = 1,251) = 19.03,/? < 0.001) (OR = .51; 95% CI: .38 - .69) (see Table 9). Table 9. Offers of Admission, Expected vs. Observed, by Visible Minority Status (N = 1,251) Variable Coding Expected Observed Percentage Offered Count Count Admission Within Ethnicity Status Visible Minority No 198.4 229 27/1 Status Yes 95.6 65 16.0 The percentage of applicants offered admission differed significantly by academic year (^ 2 (3, N = 1,277) = 15.15,/? < 0.01), with a range of 61 students admitted in the 2003-04 academic year to 92 students admitted in the 2005-06 year, a function of increased seats in the program. Applicants with higher supplemental scores (M= 27.5, SD = 4.5) were more likely to be offered admission (Levene's F= 16.40,/? < .01) (r(620.21, equal variances not assumed) = -26.58,/? < 0.001) (see Table 10), as were applicants with higher interview scores (M= 68.2, SD = 9.9) 0/(423) = -15.26,/? < 0.001), and higher essay grades ( j 2 ( 2 , N = 214) = 21.29,/? < 0.01). 59 Table 10. Admission GPA, Supplemental and Interview Scores by Offers of Admission Variable Admission Status Mean SD N Grade Point Average Refused 73.6 6.3 695 Offered 77.6 4.9 291 Supplemental Application Score Refused 19.0 5.7 929 Offered 27.5 4.5 293 Interview Score Refused 52.8 9.0 131 Offered 68.2 9.9 294 Factors Associated with the Likelihood of being Offered Admission A logistic regression analysis was conducted to identify the factors that were associated with an applicant's likelihood of being offered admission. The model accurately predicted 93.8% of the applicants who were offered admission and 78.4% of those refused admission, for an overall correct classification of 89.2%. The Nagelkerke pseudo-R2 was .73, with age group, degree status, academic year, admission GPA, supplemental score, and interview score being significant in the model (see Table 11). 60 Table 11. Logistic Regression Analysis of Admission being Offered on Predictor Variables Variable Coding B SE Wald P Odds Ratio 95% CI Constant -49.70 6.26 63.03 .00* — — Age Group <23 — — — — — (years) 23 - <25 -.11 .62 .03 .86 .90 .26- 3.04 25 - <30 -.93 .56 2.72 .10 .40 . 1 3 - 1.19 30+ -1.30 .55 5.50 .02* .27 .09 - .81 Degree Status Partial — — — — — Completed 1.01 .46 4.80 .03* 2.75 1.11 - 6.80 Academic Year 2002-03 — — — — — — 2003-04 -2.02 .52 15.36 .00* .13 .05 - .36 2004-05 -3.22 .64 25.03 .00* .04 .01 - .14 2005-06 -2.18 .66 10.95 .00* .11 .03 -.41 Admission GPA .27 .05 37.46 .00* 1.31 1.20- 1.43 Supplemental Score .42 .06 42.38 .00* 1.52 1.34- 1.73 Interview Score .33 .04 74.93 .00* 1.40 1.30- 1.50 Pseudo-R\" = 0.73. * denotes significance in the logistic regression model. — denotes referent group. 61 Academic Achievement Two of the courses, Nursing 330 - Nursing Care of Individuals within the Context of Community and Nursing 431/432 - Nursing Practice with Acute and Chronically III Populations, had non-normal distributions of grades and were treated categorically. Those courses with grades that were normally distributed are discussed first followed by the two aforementioned courses that required modified analyses. A combined average for the selected courses is also discussed. The courses were chosen to reflect a theory/practice balance (as discussed in the Methods chapter). Nursing 310 Data were analyzed for the 249 students that had completed Nursing 310 - The Core of Nursing Practice. The course had been offered 8 times over the four years of the study period. None of the students in the cohort failed the course; the mean grade was 80.5% (SD = 6.6), and the distribution was very slightly skewed negatively (-0.57). Male students achieved grades that were statistically significantly lower (76.2%; SD = 7.3) than those of the female students (81.2%; SD = 6.3) (t(247) = 4.30, p < 0.001), and visible minority students received grades lower (78.0%; SD = 7.4) than those of non-visible minority students (81.4%; SD = 6.2) « 2 4 7 ) = 3.51,/? < 0.001). Those with a full degree, on admission, had grades significantly higher (81.5%; SD — 6.2) than those with a partial degree (i.e., courses leading to a degree) (79.5%; SD = 7.0) (t(247) = -2.37,p = 0.02). Age group was significantly associated with the grade in Nursing 310 (F(3, 245) = 3.07, p = 0.03). The students aged 30 years and older had significantly lower grades than those of the students aged 25 to <30 years (78.4% vs. 81.7%; Scheffep = 0.03), whereas there were no significant grade differences between all other age groups. Students that completed the course in the 2004-05 academic year had significantly lower grades (M = 77.4%) than all other years under consideration (F(3, 245) = 9.04,/? < 0.001). The Nursing 310 grades were positively correlated with the students' GPA at admission (r = .38,/? < 0.001) and inversely correlated with their interview scores (r = -.13, /? = 0.04); no association was found between the Nursing 310 grades and the supplemental application scores. A simultaneous multiple regression analysis was conducted to identify the factors that contributed unique variance in the Nursing 310 grades. The model was statistically significant (F(10, 236) = 11.33,/? < 0.001), with 30% (i? 2 a d j = .30) of the variance in grades accounted for by the predictor variables. Gender, visible minority status, degree status, age group, academic year, and admission GPA were significant predictor variables (see Table 12). The supplemental scores and interview scores were non-significant in the model. Assumptions regarding residual distribution and multicollinearity were satisfied. Nursing 350 Two hundred and forty-eight students completed Nursing 350 - The Sociocultural Construction of Health and Illness - and one student withdrew during the four-year study period. The mean grade achieved was 80.4% (SD = 5.8) and the data were normally distributed, with a range of grades from 64% to 95%. Male students achieved grades significantly lower (78.5%; SD = 6.4) than those of the female students (80.7%; SD = 5.6) « 2 4 6 ) = 2.09, p = 0.04), and visible minority students received grades lower (78.0%; SD = 5.1) than those of non-visible minority students (81.2%; 63 Table 12. Simultaneous Multiple Regression Analysis of Nursing 310 Grade on Predictor Variables Variable Coding B SE Beta t P Constant 51.59 6.48 — 7.96 .00* Gender Female — — — — Male -2.87 1.10 -.15 -2.60 .01* Visible Minority Status No — — — — — Yes -2.98 .87 -.19 -3.41 .00* Degree Status Partial — — — — Completed 2.08 .85 .16 2.45 .02* Age Group (years) <23 — — — — — 23 - <25 -2.46 1.28 -.14 -1.93 .06 25 - <30 -.91 1.11 -.07 -.82 .42 30+ -3.95 1.16 -.25 -3.42 .00* Academic Year 2002-03 — — — — — 2003-04 -.20 1.09 -.01 -.19 .85 2004-05 -3.72 1.06 -.25 -3.50 .00* 2005-06 -.72 1.05 -.05 -.69 .49 Admission GPA .42 .08 .31 5.38 .00* R\"adj = 0.30. * denotes statistical significance in linear regression model. — denotes referent group. 64 SD = 5.8) (7(246)= 3.74,/? < 0.001). There was no significant difference in the Nursing 350 grades of students with full degrees and those with partial degrees (?(246)= -1.80,/? = 0.07). Age group differences were not significantly associated with Nursing 350 grades (F(3, 244) = 1.04,/? = 0.37), while the academic year in which the course was taken did contribute to variability in the grades (F(3, 244) = 25.48,/? < 0.001). Post-hoc Scheffe analysis revealed that the grades in the 2002-03 academic year were significantly higher (M= 85.8%) than those in each of the other three years under consideration (means ranging from 78.0% in 2004-05 to 79.6% in 2005-06). The grade in Nursing 350 was weakly positively correlated with GPA at admission (r = .28,/? < 0.001) and weakly inversely correlated with interview score (r — -.23, /? < 0.001). No significant association was found between the Nursing 350 grade and the supplemental application score. A simultaneous multiple regression analysis was conducted to identify the factors contributing unique variance to the Nursing 350 grades. The model was statistically significant (F(5, 240) - 22.85, p < 0.001), with 31% (i? 2 a d j = .31) of the variance in grades being accounted for by the predictor variables. Visible minority status, academic year, and admission GPA were significant predictor variables (see Table 13). Gender, degree status, age group, supplemental scores, and interview scores were non-significant in the model. Assumptions regarding residual distribution and multicollinearity were satisfied. 65 Table 13. Simultaneous Multiple Regression Analysis of Nursing 350 Grade on Predictor Variables Variable Coding B SE Beta t P Constant 62.70 5.22 — 12.01 .00* Visible Minority Status No — — — — — Yes -2.07 .74 -.15 -2.81 .00* Academic Year 2002-03 — — — — — 2003-04 -6.50 .94 -.47 -6.91 .00* 2004-05 -6.77 .92 -.52 -7.43 .00* 2005-06 -5.38 .89 -.42 -6.02 .00* Admission GPA .30 .07 .25 4.50 .00* R2adj = 0.31. * denotes statistical significance in linear regression model. — denotes referent group. Nursing 450 Over the 4-year study period, 164 students completed Nursing 450 - Introduction to Research Utilization. None of the students failed the course; six students withdrew, two transferred out, and five had exemptions granted for previous research coursework completed. The mean grade was 83.4% (SD = 5.7), the distribution was slightly skewed negatively (-0.75), and the range of grades was from 66% to 95%. No significant grade differences were found on the basis of gender (t(\\62) = 1.61, p = 0.10), visible minority status (r( 162) = 0.26, p = 0.79), or degree status (t(162)= -0.61, p = 0.54). 66 The academic year in which the course was taken also did not contribute significantly to differences in the grades (F(2, 161) = l.00,p - 0.37). Age group was significantly associated with the grade in Nursing 450 (F(3, 160) = 3.39, p = 0.02). The students aged 30 years and older had significantly lower grades than those of the students aged less than 23 years (81.1% vs. 84.4%; Scheffe p = 0.03); there were no significant grade differences between all other age groups. The Nursing 450 grades were positively correlated with the students' GPA at admission (r = .34, p < 0.001); no association was found between the Nursing 450 grades and the supplemental application or interview scores. A simultaneous multiple regression analysis was conducted to identify the factors that contributed unique variance to the grades in Nursing 450. The model was statistically significant (F(2, 158) = 7.82,/? < 0.001), with 14% (i? 2 a d j = .14) of the variance in the grades accounted for by the predictor variables. Only the age group and admission GPA of the students were significant predictor variables in the final model (see Table 14), with gender, visible minority status, degree status, academic year, supplemental score, and interview score being left out of the final model. Assumptions regarding residual distribution and multicollinearity were satisfied. Nursing 452 Two hundred and forty-eight students completed Nursing 452 - The Ethical Basis of Health Care - during the 4-year study period. Two students transferred out, five withdrew, and six had exemptions granted because of previous coursework completed. There were no failures 67 in the course. The mean grade was 83.3% (SD = 6.6) and the data were slightly negatively skewed (-.71), with a range of grades from 61% to 96%. Table 14. Simultaneous Multiple Regression Analysis of Nursing 450 Grade on Predictor Variables Variable Coding B SE Beta t P Constant 52.18 7.25 — 7.19 .00* Age Group (years) <23 — — — — — 23 - <25 -.99 1.36 -.06 -.73 .47 25 - <30 -1.00 1.08 -.08 -.93 .36 30+ -3.45 1.12 -.26 -3.09 .00* Admission GPA .42 .09 .34 4.47 .00* R\"adj=0.14 * denotes significance in linear regression model. - denotes referent group. Male students achieved grades significantly lower (79.7%; SD = 7.3) than those of the female students (83.9%; SD = 6.4) (t(230)=3A7,p < 0.001). The visible minority students' grades in Nursing 452 did not differ significantly from those of the non-visible minority students (/(230)=1.91,/? = 0.06), nor was there a significant difference between students with full degrees and those with partial degrees (£(230) — -l.ll,p- 0.09). Age group was not significantly associated with Nursing 452 grades (F(3, 228) = 2.30, p = 0.08), while the academic year in which the course was taken did contribute to variability in 68 the grades (F(3, 228) = 16.16,/? < 0.001). Post-hoc Games-Howell analysis revealed that grades in the 2005-06 academic year were significantly higher (M- 87.5%) than those earned in each of the other three years under consideration (means ranging from 80.5% in 2004-05 to 82.5% in 2003-04). The grade in Nursing 452 was moderately positively correlated with GPA at admission (r = .33,/? < 0.001), weakly positively correlated with the supplemental score (r = .21, /? < 0.001) and weakly positively correlated with the interview score (r = .16,/? < 0.001). A simultaneous multiple regression analysis was run to identify the factors that contributed unique variance to the Nursing 452 grades. The model was statistically significant (F(7, 222) = 11.08, p < 0.001), with 24% (i? 2 a d j = .24) of the variance in grades being accounted for by the predictor variables. Gender, age, admission GPA, supplemental scores, and interview scores were significant predictor variables (see Table 15). Visible minority status, academic year, and degree status were non-significant and not included in the final model. Assumptions regarding residual distribution and multicollinearity were satisfied. Nursing 330 Two hundred and forty nine students completed Nursing 330 - Nursing Care of Individuals within the Context of Community. Four of the students failed the course with grades less than 60% (i.e., 29% to 48%) and the mean grade was 80.4% (SD = 7.3). The distribution was negatively skewed (-2.65) and leptokurtotic (14.03), with a range from 29% to 93%. The grades were separated into quartiles of < 77%, 77% to < 81%, 81% to <85%, and 85+%. Nursing 330 grades significantly differed on the basis of gender (%2 (3, N = 249) = 9.14,/? = 0.03), with males overrepresented (23.8%) in the lowest grade category and underrepresented in 69 the highest grade category (5.3%). Visible minority students were significantly overrepresented (73.4%) in the lowest two grade quartiles compared with non-visible minority students (46.6%) (%2 (3, N = 249) = 13.39,p = 0.00). Nursing 330 grades did not significantly differ on the basis of age (x2 (9, N = 249) = 14.78, p = 0.10), degree status (x2 (3, N = 249) = 1.79, p = 0.62), or academic year ( (9, N = 249) = 9.45, p = 0.40). Table 15. Simultaneous Multiple Regression Analysis of Nursing 452 Grade on Predictor Variables Variable Coding B SE Beta t P Constant 27.90 8.30 — 3.36 .00* Gender Female — — — — — Male -2.42 1.17 -.13 -2.07 .04* Age Group (years) <23 — — — — — 23 - <25 .14 1.23 .01 .12 .91 25 - <30 .21 1.02 .02 .21 .84 30+ -2.40 1.14 -.15 -2.10 .04* Admission GPA .52 .09 .37 6.10 .00* Supplemental Score .39 .09 .26 4.33 .00* Interview Score .12 .04 .17 2.85 .00* R\"adj = 0.24. * denotes significance in linear regression model. — denotes referent group. 70 The students' GPAs at admission were significantly associated with their Nursing 330 grades (F(3, 243) = 14.93,p < 0.001), with post-hoc Scheffe analysis revealing that those in the lowest grade quartile for Nursing 330 had a significantly lower mean admission GPA (75.0%) than those in the other three quartiles (76.8% - 80.2%) and those in the second grade quartile also significantly differed from those in the highest grade quartile. There were no significant group differences on the basis of the supplemental score (F(3, 244) = .69, p = 0.56) or the interview score (F(3, 245) = 1.01,/? = 0.39). Ordinal regression analysis was conducted on the independent variables to determine which variables were associated with the Nursing 330 grade. The model was statistically significant (%2 (3) = 58.91, p < 0.001), fit well with the data (%2 (522) = 508.60, p = 0.66), and provided a Nagelkerke pseudo R-square of .23. The assumption of parallel slopes was satisfied. Gender, visible minority status, and admission GPA were significant variables in the final model (see Table 16). Those of the female gender and non-visible minority status were more likely to be in the upper grade quartiles relative to their counterparts and, as admission GPA increased, the likelihood of a student receiving a grade in the upper quartiles also increased. Degree status, age group, academic year, supplemental scores, and interview scores were not significant in the final model. Nursing 431/432 One hundred and sixty-five students completed Nursing 431/432 - Nursing Practice with Acute and Chronically III Populations during the 4-year study period. Two students transferred out and six withdrew from the course. Three students failed the course with grades less than 60% (i.e., 15% to 52%) and the mean grade was 84.2% (SD = 8.9). The distribution was 71 negatively skewed (-5.2) and leptokurtotic (36.0), with a range from 15% to 99%. Examining the data after removing the three outlying failing grades, the distribution was normal. The outliers were left in the analysis and non-parametric tests were conducted. Table 16. Ordinal Regression Analysis of Nursing 330 Grade on Predictor Variables Variable Coding Estimate SE Wald P Gender Male — — — — Female .78 .36 4.60 .03* Visible Minority Status Yes — — — — No .94 .29 10.87 .00* Admission GPA .16 .03 35.34 .00* Pseudo-EC = 0.23. * denotes significance in the ordinal regression model. — denotes referent group. Nursing 431/432 grades differed significantly on the basis of gender, with males scoring a mean rank of 56.7 and females scoring a mean rank of 87.5 (Mann-Whitney U= 1061.5,/? = 0.00). No significant group differences were found on the basis of visible minority status (Mann-Whitney U= 2002.0,/? = 0.36), degree status (Mann-Whitney U= 3248.5,/? = 0.66), or age group (Kruskal Wallis %2 (3, N = 165) = 2.62, /? = 0.45). The academic year in which the course was taken contributed significantly to 431/432 grade differences (Kruskal Wallis (2, N = 165) = 7.25,/? = 0.03), with those taking the course in the 2003-04 academic year achieving a mean rank score of 97.2 (vs. 73.4 in 2002-03 and 78.7 in 2004-05). 72 The students' Nursing 431/432 grades were weakly positively correlated with their GPAs at admission (Kendall's r b = 2\\,p < 0.001), and weakly negatively correlated with both their supplemental scores (Kendall's r b - -.12,p < 0.001) and interview scores (Kendall's r b = -.11,/? = 0.01). A simultaneous multiple regression analysis was conducted to identify the factors that contributed unique variance to the Nursing 431/432 grades. The model was statistically significant (F(2, 161) = 13.25,/? < 0.001), with 13% (^ 2 a d j = .13) of the variance in grades being accounted for by the predictor variables. Gender and admission GPA were significant predictor variables (see Table 17). Academic year was not significant, despite bivariate significance. Age, visible minority status, degree status, supplemental score, and interview score were non-significant and not included in the final model. Assumptions regarding residual distribution and multicollinearity were satisfied. Table 17. Simultaneous Multiple Regression Analysis of Nursing 431/432 Grade on Predictor Variables Variable Coding B SE Beta t P Constant 47.63 11.67 4.08 .00* Gender Female — — Male -5.40 1.89 -.21 -2.87 .00* Admission GPA .55 .14 .28 3.83 .00* R 2adj=0.13. * denotes significance in linear regression model. — denotes referent group. Cumulative Average for Selected Nursing Courses The 249 students' cumulative averages of their grades for the selected six nursing courses were non-normally distributed, with a mean cumulative average grade of 81.7% (SD = 5.2) and the distribution being negatively skewed (-1.5) and leptokurtotic (4.3). The range of cumulative average grades was from 53.5% to 92.0%. The average grades were separated into quartiles of < 78.9%, 78.9% to < 82.9%, 82.9% to < 85.2%, and 85.2+%. Sixty one percent of the students had completed all 6 courses at the time of the analysis (see Table 18). Table 18. Number of Selected Courses Completed Number of Selected Courses Completed Number of Students (N=249) Percent 2 1 0.4 3 8 3.2 4 70 28.1 5 19 7.6 6 151 60.6 The cumulative average of the selected nursing course grades significantly differed on the basis of gender ( j 2 (3, N = 249) = \\1.21,p< 0.01), with males overrepresented (40.7%) in the lowest grade category and underrepresented in the highest grade category (13.3%). Visible minority students were significantly overrepresented (73.3%) in the lowest two grade quartiles compared with non-visible minority students (45%) ( ( 3 , N = 249) = 14.71, p < 0.01). The 74 cumulative average grades did not significantly differ on the basis of degree status (%2 (3, N = 249) = 5.38,p = 0.15) or academic year (%2 (9, N = 249) = 8.86,/? = 0.45). Although the cumulative average grade for the selected nursing courses was not significantly associated with age (F(3, 245) - 2.49,/? = 0.06), supplemental score (F(3, 244) = .26,/? = 0.86), or interview score (F(3, 245) = .53, p = 0.67), it was associated with GPA upon admission (F(3, 243) = 18.21,/? < 0.01). Post-hoc Scheffe analysis revealed that those in the highest quartile of the cumulative course average grades had significantly higher GPAs at admission than those of the students in each of the other quartiles, with the mean admission GPA in the highest quartile being 80.7% (SD = 3.7) (see Table 19). Table 19. Mean Admission GPA by Cumulative Average Grade for Selected Courses Cumulative Average Grade Mean GPA at Admission SD N < 78.9% 75X) 5.1 63 78.9% to < 82.9% 76.5 4.4 65 82.9% to <85.2% 77.7 4.2 60 85.2+% 80.7 3.7 59 Of a possible 6 nursing courses. Ordinal regression analysis was conducted on the independent variables to determine which variables were significantly associated with the cumulative average grade for the selected nursing courses. The model was statistically significant (%2 (3) = 71.47,/? < 0.001), fit well with the data (%2(522) = 527.94,p = 0.42), and provided a Nagelkerke pseudo R-square of .27. The assumption of parallel slopes was satisfied. Gender, visible minority status, and admission GPA were significant variables in the final model (see Table 20). Degree status, age group, academic year, supplemental scores, and interview scores were not significant in the final model. Table 20. Ordinal Regression Analysis of Cumulative Average Grade of Selected Nursing Courses on Predictor Variables Variable Coding Estimate SE Wald P Gender Male — — — — Female 1.32 .38 12.10 .00* Visible Minority Status Yes — — — — No .91 .29 10.11 .00* Admission GPA .17 .03 39.65 .00* Pseudo-R2=0.27. * denotes significance in the ordinal regression model. — denotes referent group. Summary To summarize, the results are presented within the context of the research questions posed. Because the primary focus of this research was predictive validity, the answers to Research Question 2, regarding bivariate relationships among the variables, are embedded within the Results chapter and are not made explicit in this summary. 1. What is the inter-rater reliability of: (a) the supplemental application and (b) the admission interview scoring procedures used? The inter-rater reliability of the supplemental score was very low. The interview inter-rater 76 reliability was very high. 2. Do applicants' demographic backgrounds contribute to the variability in their supplemental scores? The supplemental score was predicted by age, visible minority status, and admission GPA. 3. Do applicants' demographic backgrounds and supplemental scores predict the likelihood of being invited to an admission interview? Invitation to an interview was predicted by admission GPA and supplemental application score. 4. Do the (a) supplemental scores and (b) interview scores add to the prediction of students' success, as operationalized by their (i) admission status and (ii) grades in selected courses, after all other variables are considered? Offers of admission were predicted by all variables except gender and visible minority status. Neither the supplemental application score nor the interview score were predictive of success within the nursing program, with the exception of the prediction model for Nursing 452. Admission GPA was consistently predictive of program success, with gender and visible minority status also frequently found to be predictive. 77 CHAPTER FIVE: DISCUSSION The discussion of the results is presented in the context of the goals of the research conducted. Following the findings is a discussion of the limitations of this research. Finally, suggestions for the U B C SON and other schools of nursing admission committees are offered and recommendations for future research are made. In Search of Predictive Validity The first goal of the research was to determine whether the U B C SON's admission process is valid. The desire was to ascertain whether the application process successfully identifies students that will be both academically and clinically successful in the U B C advanced standing baccalaureate nursing program. The results overwhelmingly indicate that the \"non-cognitive\" tools used in the selective admission process do not appear to facilitate the identification of students who will be successful in the program, and offer no additional predictive value above that provided by the consideration of previous academic achievement (admission GPA). Overall, there were no correlations between the supplemental application and interview scores and the grades earned in the program. There were weak negative correlations between both tools and the grade in Nursing 431/432, a final year theory and clinical course, and there were weak positive correlations between both tools and the grade achieved in Nursing 452. The positive correlation with Nursing 452, an ethics course, could occur because the supplemental application and interview may provide opportunities for applicants to express their morals and beliefs better than grade point averages can obviously allow (Ferguson et al., 2000). Students who were scored highly in the assessment of these types of personal qualities may also be more 78 proficient at expressing the same beliefs within the context of assignments and presentations on similar topics. Because this thesis highlighted the use of Nursing 431/432 grades as an outcome, which purportedly allows for the assessment of both academic and clinical aptitudes by the very nature of the way the students in the course are evaluated, the weak negative correlations found between the grade in this course and both the supplemental score and the interview score speak to a lack of predictive ability of these tools for either academic or clinical success. The negative correlations are difficult to interpret because the course clearly has both academic and clinical outcomes, the break down of which is not provided in the final grade, therefore precluding exact attribution of the correlation with the clinical component, the written assignments, or both. For the remainder of the courses, as well as for the cumulative average grade across the selected nursing courses, no other significant correlations were found with the non-cognitive criteria. Moreover, neither the supplemental score nor the interview score were predictive of the cumulative average grade or the six course grades, with one exception. The significant predictor variables for Nursing 452 included admission GPA, gender, supplemental score, and interview score. Admission GPA was a consistent and strong predictor of success in the program. Admission GPA was positively correlated with all of the six course grades, as well as with the cumulative average grade, and was a significant predictor in each of the regression models. These findings were not entirely surprising; the research literature is replete with studies that show a positive association between admission GPA and academic success (Didier et al., 2006; Kulatunga-Moruzi & Norman, 2002a; Salvatori, 2001). What had previously been uncertain was the relationship between admission GPA and clinical aptitude. Although Kulatunga-Moruzi and 79 Norman (2002a) found admission GPA to be predictive of clinical success in medicine and Carpio et al. (1996) found similar results using high school grades and success in university nursing, Salvatori's (2001) review provided equivocal results. In earlier U B C SON thesis research, Fraser (2003) found no significant correlation between GPA and a clinical self-efficacy tool, although the measured outcome was perceived ability vs. actual grades received, making comparison with this work difficult. Because admission GPA was found to be predictive of all course grades and because some of the courses evaluated for this research are clinical in nature, or are academic with a clinical component, we conclude that admission GPA is a valid predictor of clinical success. Because they contribute no unique variance in the overall success of students, the continued use of the supplemental application and interview is questionable. Kulatunga-Moruzi and Norman (2002a) also failed to find unique variance explained in medical school outcomes by the use of an autobiographical submission, although their research focussed on medical licensing examinations and not course grades. In nursing, Brown et al. (1991) also failed to establish the autobiographical essay as a predictor of program success. Despite the admission interview being frequently cited as the most important tool in admissions, researchers such as Kreiter et al. (2004) and Salvatori (2001) have discussed the equivocal nature of the predictive validity of the admission interview, while others (Ehrenfeld & Tabak, 2000; Elam & Andrykowski, 1991; Shahani et al., 1991) have found the interview to be of low predictive value when compared to other variables, such as admission GPA. Recently, however, Reiter, Eva, Rosenfeld, and Norman (2007) found that the Multiple Mini-Interview (MMI) in use in McMaster University Medical School admission processes was predictive of objective structured clinical examination performance, and clinical and ethical 80 decision-making scores. The authors concluded that the M M I , which focuses on non-cognitive skills, such as communication, moral reasoning, and collaboration, offers predictive validity for both clinical and licensing performance and is more valuable than other non-cognitive assessments, such as panel interviews and autobiographical submissions. Furthermore, given that admission GPA predicted testing score sections different than those predicted by the M M I on professional licensing exams, the authors suggested that the M M I is a tool best used as complementary to admission GPA in medical student selection. One reason why the supplemental application may not contribute any unique variance to the prediction of success could be the problems inherent with the tool, particularly during this study period. As Dore et al. (2006) and Kulatunga-Moruzi and Norman (2002a) found, inter-rater reliability of supplemental application tools can be poor. Certainly, this was the case in these findings. With the tool having inter-rater reliability of only r = .52, one cannot expect predictive validity. What cannot be explained by error or rule changes, however, is why the interview, with a high inter-rater reliability of r = .89, failed to be predictive of program success. These concerns will be further addressed below in a discussion of whether these tools add any new information beyond that offered through the consideration of admission GPA. Surprisingly, gender and visible minority status appeared more consistently predictive of success, the only exceptions being in the regression models for Nursing 450 (where only admission G P A was predictive), Nursing 452 (as mentioned above), and Nursing 431/432 (as mentioned above). Gerrish (2000) cautioned against drawing conclusions beyond the data, wise advice which is followed in this thesis; however, it is evident that some sort of disadvantage is occurring, particularly because these findings occurred with any differences in admission GPA being accounted for. Male students and visible minority students were almost consistently 81 assigned lower grades than those of their counterparts; both gender and visible minority status were significant predictors in almost all of the regression models. There is little literature to explain why these variables may be influential. The literature reviewed for this thesis acknowledged that these disadvantages occur elsewhere (e.g., gender in Kevern et al., 1999), and often in the context of the N C L E X examinations in the United States (e.g., gender and ethnicity in Haas et al., 2004). Although some research shows older applicants faring better in nursing programs (Kevern et al., 1999; Salamonson & Andrew, 2006; van Rooyen et al., 2006), age was only correlated with Nursing 310 and 450, with older students achieving lower grades; age was not a significant variable in any of the regression models. Because Nursing 310 is the first course students take when entering the curriculum, one could hypothesize that older students, who may have been away from an academic environment longer than younger students, may find the transition back to school difficult. Additionally, since Nursing 450 is a research methodology course, older students, who may not have fresh writing and critical thinking skills, could be disadvantaged. There are no data within this research, however, to explain why these two negative correlations occurred. Defining applicant success involves examining the offers of admission made. Despite neither the supplemental score nor interview score having predictive utility for success in the program, both contributed variance to the prediction of admission status, in conjunction with admission GPA and educational attainment. Odds ratios for being offered admission were: admission GPA (OR = 1.31); supplemental score (OR = 1.52); interview score (OR = 1.40), and degree status (OR = 2.75). The concern, however, is that a tool with questionable reliability, that is, the supplemental score, is given serious weighting in the decision to admit an applicant. In 82 the final model for offers of admission, the pseudo-R2 was .73, with age group, degree status, academic year, admission GPA, supplemental score, and interview score being significant predictors. Despite the model's 89% correct classification of offers and refusals, it is apparent that there may be random or systematic errors at play or there may be variables on which applicants are not being explicitly assessed, yet which influence the decision making. The extremely high inter-rater reliability for the interview scores also is of concern. Although the interviewers are instructed not to discuss or compare their marks to calibrate their scores, this high reliability is not seen elsewhere in the literature (Conway et al., 1995; Kreiter et al., 2004). The prediction of which applicants were offered interviews was based mostly upon the supplemental application score, which has been previously found in the literature as predictive of invitations to an interview, and has been discussed as problematic when the reliability of the rating of the personal statement is questionable (Ferguson et al., 2000; Salvatori, 2001). Despite a high percentage of correct classification, our model poorly fit the data, indicating that some applicants were interviewed despite low probabilities of this outcome, based on their relevant variable values. A Question of New Information The second goal of the research was to determine whether the intensive process was worthy of the resources invested (i.e., whether it offers more information that is advantageous in admission decisions than relying upon admission GPA alone). Given the findings, we conclude that the process is not worth the resources invested; we found we could not statistically predict success in the program with the tools in use and no new additional information is provided by the use of these tools. Not only is admission GPA the best and most consistent predictor of 83 success in the program, the concerns raised with both the supplemental application and the interview call into question their predictive utility in admission decisions. Bivariately, it was found that older, non-visible minority students achieved higher supplemental application and interview scores compared with their counterparts. Supplemental application scores were predicted by age, visible minority status, admission GPA, and academic year, with only 26% of the variance in scores being accounted for by these variables. It appears that other variables may be influential, presumably related to the qualities said to be assessed; conversely the unexplained variance could be the result of random or systematic errors, such as the scoring discrepancies encountered. The poor inter-rater reliability of the supplemental application precludes further discussion of its utility. Admission GPA was positively correlated with both tools. Furthermore, the cumulative average grade was predicted by admission GPA (as well as gender and visible minority status) but not by the supplemental score or the interview score. Fraser (2003) had found a negative relationship between the supplemental score and GPA, although the GPA used was for that of first-term marks only. Therefore, although some new information about the candidates may be provided via the use of the supplemental application and interview, the potential for certain students to be disadvantaged, combined with some redundancy with the admission GPA, provides little evidence to justify continuation of these practices. The Link to Prior Research The final research problem posed was to determine whether the findings supported or refuted an earlier study concluding that, through the analysis of clinical self-efficacy, the intensive procedure is not worth the resource expenditure (Fraser, 2003). The failure to find predictive validity of the process for academic and clinical success within the program, and given the results of studies of supplemental application scores and interviews found in the literature, we unequivocally support the conclusions of Fraser. Given the present difficulties with the process, the potential for select groups of students being disadvantaged, and the clear predictive utility of admission GPA, there is little question that the resource- and time-intensive process currently in place at the U B C SON is not worthy of such expenditures. Limitations As with any research, several factors may limit the generalizability of these findings. Regarding visible minority status, despite our best efforts to make accurate and confirmed classifications, the possibility of misspecification exists (Gittoes & Thompson, 2007), particularly for those students for whom photographs were not available or who had taken the European names of their spouses. As is the nature of nursing education, group sizes were vastly different for male and female applicants and students who entered the program. Of the 249 students evaluated in this research who completed the selected courses, 215 were female and 34 were male. Despite the unequal group size, however, we remain confident that our analyses were sound, by employing statistical methods such as applying Mests with equal variances not assumed, in analyses where the Levene's test was significant. The smaller sample size would render our results more susceptible to Type II errors rather than Type I errors. Therefore, in the cases of statistically significant differences in the grades earned by the male and female students, for example, we conclude that the findings are of practical significance. 85 Other important variables in the models may have been misspecified. It is possible that the selection of variables for assessment in the application and, therefore, in these regression models do not present the most relevant variables pertaining to student selection. Important variables such as leadership and employment experiences may be proxies for measuring clinical aptitude but are not specified variables in the application process and therefore may not be adequately captured in the current evaluation system. Content analysis of the various supplemental applications and interview responses may aid in uncovering these variables. The likelihood of being offered admission naturally increased as the number of seats in the program increased, thereby making any associations with academic year a statistical manoeuvre rather than predictive. We adjusted for differences in year because not only did the number of seats change, but application raters and applicant pools also changed from year to year. Admission GPA has been found to be the best and most stable predictor of success, with science GPA being of particular value (Didier et al., 2006; Kulatunga-Moruzi & Norman, 2002a). Prerequisite university anatomy and physiology course grades were not available for this research and therefore the analysis of the predictive validity of using science GPA, alone, was not possible. Furthermore, the institution from which the applicants were applying was also unavailable, making GPA adjustment analysis impossible. Yet, grading practices vary from institution to institution and over time. Poor inter-rater reliability (r = 0.52, p < 0.001) likely accounted for some of the variance in the supplemental scores. The Admission Committee recognized that there were inherent issues with inter-rater reliability and attempted to overcome these discrepancies by making scoring modifications. Inter-rater reliability problems may have significantly distorted the 86 predictability of our model by introducing added variance and error. By necessity, one cannot definitively say that, had the inter-rater reliability not been an issue, the tool would not have had more predictive validity for success within the program. This potential will remain unknown unless further studies are completed. The results of this study must be regarded in the context of having examined a highly unreliable tool. Despite the unreliability of the supplemental application, however, the Admission Committee relied on the information they had acquired to make their selection decisions. The data analyzed here are valid, therefore, as these were the data actually used in the admission process. In some cases, outliers in the data remained, based on the applied scoring cut-offs (e.g., those not invited to an interview despite a high probability of being invited, given their scores on the relevant criteria). These outliers had significant influences on our models, as evidenced by the poor fit of the interview prediction model with the data. Finally, some of the analyses yielded low correlations. Low correlations in regression analyses regarding admission criteria can be interpreted in one of two ways (Kulatunga-Moruzi & Norman, 2002b). Either the admission measures can be deemed to have poor predictive validity or the predictors could be viewed as working as intended, selecting homogeneous cohorts who will perform similarly at the top of their ability range. The low correlations would therefore be indicative of this lack of variability within the sample. Such consideration is worth noting here, because the cohorts admitted to the School of Nursing became very homogenous once pooled post-admission. In an attempt to address this issue, Kulatunga-Moruzi and Norman examined those who were and were not accepted to the McMaster University Medical School and found no significant relationship between admission measures intended to assess non-academic qualities and performance post-graduation, as measured by licensing examination scores, regardless of which academic institution the applicants had been ultimately accepted. Such a study would be worth undertaking should the U B C School of Nursing decide to continue its use of the supplemental application and interview and if Canadian Registered Nursing Examination results could be obtained. Recommendations The following recommendations are specific to the School of Nursing at the University of British Columbia. Although these suggestions apply to a program currently using a selective admission process, which makes use of a supplemental application and interview to select students for an accelerated program, the following points should be considered by any school of nursing currently using or considering implementing an admission process based on more than admission GPA alone. Reflection upon the individual needs of each school of nursing will facilitate the determination of which recommendations may be most applicable. These recommendations are merely springboards for discussion amongst U B C SON Admission Committee members and are not to be taken as prescriptive or comprehensive. 1. Discontinue use of both the supplemental application and the interview; admit students on the basis of admission GPA only. 2. If the process is to be continued, change the supplemental application scoring process. Impose a scoring distribution to prevent extreme scoring or employ a horizontal marking scheme, such as that described by Dore et al. (2006), to prevent scoring bias. 3. Conduct content analyses of the supplemental applications to better elucidate the desired applicant characteristics and further attempt to validate the measure. 4. Change the threshold for offering interviews to utilize statistically calculated cut-offs based on the current year's applicant pool. 5. Instead of making decisions using the current system of admission GPA/supplemental ranking to progress to an interview, and interview score ranking for offers of admission, reweigh each successive step based on the prediction models derived in this research. 6. Adjust the admission GPAs by the applicant's institution (and perhaps by year), as recommended by Didier et al. (2006). 7. Review marking practices within the SON to ensure that no students are being systematically disadvantaged for irrelevant reasons such as gender and visible minority status. The following suggestions are for future research endeavours that could be undertaken by members of the U B C SON and may also prove fruitful for other institutions to consider. Analyze the two separate components of the supplemental application, the structured resume and the personal statement, to discern where the reliability issues are occurring and whether clear disadvantages are found. It could be hypothesized that, for example, some students with certain familial or cultural values may work exceedingly hard at academics and have little time for extracurricular volunteering. These differences could be uncovered in separate analyses of the supplemental application. 89 Examine ethnicity further, including other variables in the analysis to paint a more comprehensive picture of ethnicity, to ensure that the issue is not one of prediction model misspecification, and to determine whether a disadvantage truly exists. Follow the graduates of the program to determine attrition from the profession, which could be used as an alternative measure of success. A l l available in-course data were used in this research and no correlations between the advanced standing admission criteria currently being used and academic or clinical course success were found. If the current processes are being used to assess clinical aptitude, and the use of these tools are not otherwise supported in the literature, there needs to be some other measure of success to justify the continued use of this process. Collaborate with other schools of nursing, or other programs within the University of British Columbia, to conduct admission criteria research, as suggested by Campbell and Dickson (1996). Develop consistent and stable predictors, larger sampling pools, and opportunities for interdisciplinary consultation to facilitate the development of more reliable and valid tools. Summary Current concerns arising from the data analysis require that members of the U B C School of Nursing address inter-rater reliability issues in order to reliably use a supplementary application tool for student admission selection purposes. There is some indication that select groups face disadvantage through the supplemental application currently in use. It is therefore imperative that those involved at the U B C School of Nursing make detailed examination of the current state of their admission process. The Admission Committee may choose to revise the supplemental application to better reflect clinical aptitude while ensuring that visible minority 90 and younger applicants are not disadvantaged, or they may choose to discontinue the use of both the supplemental application and the interview on the grounds of poor predictive validity and unjustifiable resource expenditure. There remains, from this thesis and the published literature to date, little conclusive evidence to support the use of either tool. Future research endeavours may involve alternative definitions of nursing student success, while interdisciplinary collaboration could facilitate the development of reliable and valid tools. 91 REFERENCES Albanese, M . A. , Snow, M . FL, Skochelak, S. E., Huggett, K. N . , & Farrell, P. M . (2003). Assessing personal qualities in medical school admissions. Academic Medicine, 78, 313-321. Brown, B., Carpio, B., & Roberts, J. (1991). The use of an autobiographical letter in the nursing admissions process: Initial reliability and validity. Canadian Journal of Nursing Research, 23, 9-19. Campbell, A . R., & Dickson, C. J. (1996). Predicting student success: A 10-year review using integrative review and meta-analysis. Journal of Professional Nursing, 12, 47-59. Campion, M . A. , Palmer, D. K. , & Campion, J. E. (1997). A review of structure in the selection interview. Personnel Psychology, 50, 655-702. Carpio, B. , & Brown, B. (1993). The admissions process of a bachelor of science in nursing program: Initial reliability and validity of the personal interview. Canadian Journal of Nursing Research, 25, 41-52. Carpio, B., O'Mara, L., & Hezekiah, J. (1996). Predictors of success on the Canadian Nurses Association Testing Service (CNATS) Examination. Canadian Journal of Nursing Research, 28, 115-123. Carrothers, R. M . , Gregory, S. W., Jr., & Gallagher, T. J. (2000). Measuring emotional intelligence of medical school applicants. Academic Medicine, 75, 456-463. C B C News. (2007). Term 'visible minorities' may be discriminatory, U N body warns Canada. March 8, 2007. Retrieved March 10, 2007, from http://www.cbc.ca/canada/story/2007/03/08/canada-minorities.html Cohen, J., Cohen, P., West, S. G., & Aiken, L. S. (2003). Applied multiple 92 regression/correlation analysis for the behavioral sciences (3rd ed.). New Jersey: Lawrence Erlbaum. Conway, J. M . , Jako, R. A. , & Goodman, D. F. (1995). A meta-analysis of interrater and internal consistency reliability of selection interviews. Journal of Applied Psychology, 80, 565-579. Courneya, C. A. , Wright, K. , Frinton, V. , Mak, E., Schulzer, M . , & Pachev, G. (2005). Medical student selection: Choice of a semi-structured panel interview or an unstructured one-on-one interview. Medical Teacher, 27, 499-503. Crow, C. S., Handley, M . , Morrison, R. S., & Shelton, M . (2004). Requirements and interventions used by B S N programs to promote and predict N C L E X - R N success: A national study. Journal of Professional Nursing, 20, 174-186. DeLapp, T. D. (2006). Raising admission standards: Impact on ethnic diversity of B S N graduates. Communicating Nursing Research, 39, 224-224. Dhruvarajan, V . (2000). People of colour and national identity in Canada. Journal of Canadian Studies, 35, 166-176. Didier, T., Kreiter, C. D., Buri, R., & Solow, C. (2006). Investigating the utility of a GPA institutional adjustment index. Advances in Health Sciences Education: Theory and Practice, 11, 145-153. Dore, K . L. , Hanson, M . , Reiter, H . I., Blanchard, M . , Deeth, K. , & Eva, K . W. (2006). Medical school admissions: Enhancing the reliability and validity of an autobiographical screening tool. Academic Medicine, 81(10 Suppl.), S70-73. Ehrenfeld, M . , & Tabak, N . (2000). Value of admission interviews in selecting of undergraduate nursing students. Journal of Nursing Management, 8, 101-106. 93 Elam, C. L., & Andrykowski, M . A . (1991). Admission interview ratings: Relationship to applicant academic and demographic variables and interviewer characteristics. Academic Medicine, 66(9 Suppl), S13-15. Ferguson, E., James, D., O'Hehir, F., Sanders, A. , & McManus, I. C. (2003). Pilot study of the roles of personality, references, and personal statements in relation to performance over the five years of a medical degree. B M J , 326, 429-432. Ferguson, E., Sanders, A. , O'Hehir, F., & James, D. (2000). Predictive validity of personal statements and the role of the five-factor model of personality in relation to medical training. Journal of Occupational and Organizational Psychology, 73, 321-344. Fraser, S. (2003). The relationship between baccalaureate admissions criteria and student outcomes. Unpublished master's thesis, University of British Columbia, Vancouver, British Columbia, Canada. Gabard, D. L. , Porzio, R., Oxford, T., & Braun, R. (1997). Admission interviews: Questions of utility and cost in masters of physical therapy programs in the United States. Physiotherapy Research International, 2, 135-149. Gafni, N . , Moshinsky, A. , & Kapitulnik, J. (2003). A standardized open-ended questionnaire as a substitute for a personal interview in dental admissions. Journal of Dental Education, 67, 348-353. Gardner, H . (1993). Multiple intelligences: The theory in practice. New York: Basic Books. Garson, D. G. (2007). P A 765: Reliability Analysis. Retrieved June 11, 2007, from http://www2.chass.ncsu.edu/garson/pa765/reliab.htm Gerrish, K . (2000). Researching ethnic diversity in the British NHS: Methodological and practical concerns. Journal of Advanced Nursing, 31, 918-925. 94 Gittoes, M . , & Thompson, J. (2007). Admissions to higher education: Are there biases against or in favour of ethnic minorities? Teaching in Higher Education, 12, 419-424. Haas, R. E., Nugent, K. E., & Rule, R. A . (2004). The use of discriminant function analysis to predict student success on the N C L E X - R N . Journal of Nursing Education, 43, 440-446. Kevern, J., Ricketts, C., & Webb, C. (1999). Pre-registration diploma students: A quantitative study of entry characteristics and course outcomes. Journal of Advanced Nursing, 30, 785-795. Kreiter, C. D., Yin, P., Solow, C., & Brennan, R. L. (2004). Investigating the reliability of the medical school admissions interview. Advances in Health Sciences Education: Theory and Practice, 9, 147-159. Kulatunga-Moruzi, C., & Norman, G. R. (2002a). Validity of admissions measures in predicting performance outcomes: The contribution of cognitive and non-cognitive dimensions. Teaching and Learning in Medicine, 14, 34-42. Kulatunga-Moruzi, C., & Norman, G. R. (2002b). Validity of admissions measures in predicting performance outcomes: A comparison of those who were and were not accepted at McMaster. Teaching and Learning in Medicine, 14, 43-48. Latif, D. A . (2005). Including the assessment of nontraditional factors in pharmacy school admissions. Annals of Pharmacotherapy, 39, 721-726. Leners, D., Beardslee, N . Q., & Peters, D. (1996). 21st century nursing and implications for nursing school admissions. Nursing Outlook, 44, 137-140. Lewis, M . , & Smith, S. (2002). Selection of pre-registration physiotherapy students: Changing to a more objective process. Physiotherapy, 88, 688-698. Lewis, R. E., van Walsum, K. L., Spafford, M . M . , Edwards, J. C , & Turnwald, G. H. (2004). 95 Veterinary school admission interviews, part 3: Strategies for increasing interview validity. Journal of Veterinary Medical Education, 31, 128-137. Lievens, F., Buyse, T., & Sackett, P. R. (2005). The operational validity of a video-based situational judgment test for medical college admissions: Illustrating the importance of matching predictor and criterion construct domains. Journal of Applied Psychology, 90, 442-452. McGaghie, W. C , & Kreiter, C. D. (2005). Holistic versus actuarial student selection. Teaching and Learning in Medicine, 17, 89-91. Ochieng, C. O., & Zumbo, B. D. (2001). Examination of a variable ordering index in linear regression models: An assessment of the relative Pratt index in Likert data. Paper presented at the First Annual Bob Conroy Conference on Measurement Evaluation and Research and Methodology, Vancouver, BC. Othuon, L., & Kishor, N . (1998). Bias in regression estimates after different treatments of missing data. Paper presented at the American Educational Research Association, San Diego, CA. Parry, J., Mathers, J., Stevens, A. , Parsons, A. , Lilford, R., Spurgeon, P., et al. (2006). Admissions processes for five year medical courses at English schools: Review. B M J , 332, 1005-1009. Patrick, L. E., Altmaier, E. M . , Kuperman, S., & Ugolini, K. (2001). A structured interview for medical school admission, phase 1: Initial procedures and results. Academic Medicine, 76, 66-71. Polit, D. F., & Beck, C. T. (2004). Nursing research: Principles and methods. Philadelphia, PA: Lippincott Williams & Wilkins. Powis, D., James, D., & Ferguson, E. (2007). Demographic and socio-economic associations with academic attainment (UCAS tariff scores) in applicants to medical school. Medical Education, 41,242-249. Price, K . M . , & Cortis, J. D. (2000). The way forward for transcultural nursing. Nurse Education Today, 20, 233-243. Reiter, H . I., Eva, K. W., Rosenfeld, J., & Norman, G. R. (2007). Multiple mini-interviews predict clerkship and licensing examination performance. Medical Education, 41, 378-384. Sadler, J. (2003). Effectiveness of student admission essays in identifying attrition. Nurse Education Today, 23, 620-627. Salamonson, Y . , & Andrew, S. (2006). Academic performance in nursing students: Influence of part-time employment, age and ethnicity. Journal of Advanced Nursing, 55, 342-349. Salovey, P., & Mayer, J. D. (1990). Emotional intelligence. Imagination, Cognition and Personality, 9, 185-211. Salvatori, P. (2001). Reliability and validity of admissions tools used to select students for the health professions. Advances in Health Sciences Education: Theory and Practice, 6, 159-175. Seldomridge, L. A. , & DiBartolo, M . C. (2005). A profile of accelerated second bachelor's degree nursing students. Nurse Educator, 30, 65-68. Shahani, C , Dipboye, R. L., & Gehrlein, T. M . (1991). The incremental contribution of an interview to college admissions. Educational & Psychological Measurement, 51, 1049. Stansfield, R. B., & Kreiter, C. D. (2007). Conditional reliability of admissions interview ratings: Extreme ratings are the most informative. Medical Education, 41, 32-38. 97 Statistics Canada. (2003). Ethnic Diversity Survey: Portrait of a multicultural society (No. 89-593-XIE). Ottawa, ON: Minister of Industry. Retrieved June 8, 2007, from http://www.statcan.ca/english/freepub/89-593-XIE/89-593-XIE2003001.pdf Statistics Canada. (2007). Definition of concepts and variables - visible minority. Retrieved June 8, 2007, from http://www.statcan.ca/english/concepts/definitions/vis-minorit.htm Streubert-Speziale, H . J. (2002). R N - M S N admission practices and curricula in the Mid-Atlantic region. Nursing Education Perspectives, 23, 294-299. Turnwald, G. H. , Spafford, M . M . , & Edwards, J. C. (2001). Veterinary school admission interviews, part 1: Literature overview. Journal of Veterinary Medical Education, 28, 111-121. van Rooyen, P., Dixon, A. , Dixon, G., & Wells, C. (2006). Entry criteria as predictor of performance in an undergraduate nursing degree programme. Nurse Education Today, 26, 593-600. Watson, R. (2006). Commentary on Salamonson Y. and Andrew S. (2006). Academic performance in nursing students: Influence of part-time employment, age and ethnicity. Journal of Advanced Nursing 55, 342-351. Journal of Advanced Nursing, 55, 350-351. Wharrad, H. J., Chappie, M . , & Price, N . (2003). Predictors of academic success in a bachelor of nursing course. Nurse Education Today, 23, 246-254. Young, T. A . (1997). Teaching medical students to lie. The disturbing contradiction: Medical ideals and the resident-selection process. C M A J , 156, 219-222. Appendix 1 Consent of the Director of the School of Nursing Appendix 2 Behavioural Research Ethics Board Approval "@en ; edm:hasType "Thesis/Dissertation"@en ; edm:isShownAt "10.14288/1.0101029"@en ; dcterms:language "eng"@en ; ns0:degreeDiscipline "Nursing"@en ; edm:provider "Vancouver : University of British Columbia Library"@en ; dcterms:publisher "University of British Columbia"@en ; dcterms:rights "For non-commercial purposes only, such as research, private study and education. Additional conditions apply, see Terms of Use https://open.library.ubc.ca/terms_of_use."@en ; ns0:scholarLevel "Graduate"@en ; dcterms:title "Predictive validity of an innovative selective admissions process for baccalaureate nursing students"@en ; dcterms:type "Text"@en ; ns0:identifierURI "http://hdl.handle.net/2429/32144"@en .