Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Cross-cultural validity and equivalency of oral health-related quality of life (OHQoL) measurement for.. 2012

You don't seem to have a PDF reader installed, try download the pdf

Item Metadata


ubc_2012_fall_seo_jaesung.pdf [ 3.59MB ]
JSON: 1.0072976.json
JSON-LD: 1.0072976+ld.json
RDF/XML (Pretty): 1.0072976.xml
RDF/JSON: 1.0072976+rdf.json
Turtle: 1.0072976+rdf-turtle.txt
N-Triples: 1.0072976+rdf-ntriples.txt

Full Text

CROSS-CULTURAL VALIDITY AND EQUIVALENCY OF ORAL HEALTH- RELATED QUALITY OF LIFE (OHQOL) MEASUREMENT FOR KOREAN OLDER ADULTS  by Jaesung Seo B.Sc. (Hons), The University of Western Ontario, 2009   A THESIS SUBMITTED IN PARTIAL FULFILMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE in THE FACULTY OF GRADUATE STUDIES (Dental Science)  THE UNIVERSITY OF BRITISH COLUMBIA (VANCOUVER)  August 2012 © Jaesung Seo, 2012  ii Abstract Introduction: Oral Health Impact Profile (OHIP) is a widely used psychometric instrument or scale developed in English to measure Oral Health-related Quality of Life (OHQoL), and there has been many translations of the instrument into other languages, including Korean. Purpose: My thesis examines the validity and cultural equivalence of the English and Korean versions of the scale by answering the questions: ―What methods are available to validate the cultural equivalence of psychometric instruments?‖ and ―How culturally appropriate and valid is the Korean version of the short-form of the OHIP (OHIP-14K)? Method: Ten Korean dental experts fluent in English and Korean independently assessed the clarity, relevance, and cultural equivalence of the OHIP-14K and offered suggestions for improving the cultural sensitivity and validity of the instrument content. The item-level Content Validity Index (I-CVI) was used to measure the validity of each item from the experts’ ratings followed by the calculation of Scale-level Content Validity Index (S-CVI) as the proportion of content valid items. Additional analyses including the average deviation index (ADM) and Kappa statistics (Kfree) were performed with the clarity index (CI), relevance index (RI) and cultural equivalence index (CEI) to measure the level of agreement between the experts. Results: The experts rated the OHIP-14K as mostly clear (S-CVI= 0.93), but they were concerned about the relevance of many items to the expected domains of the instrument (S-CVI = 0.42) and about its cultural equivalence (S-CVI = 0.50) to the English version. However, there was much disagreement between the experts as measured by the RI (Kfree = 0.19 to 1.00) and CEI (ADM = 0.36 to 0.96). Conclusion: The relevance and cultural equivalence of the OHIP-14K to the original English version of the OHIP-14 are not strong. Suggestions are offered for improving the OHIP-14K, which needs further testing within the Korean populations.  iii Preface  Ethics approval for the present research was obtained from the Research Ethics Board of the University of British Columbia (BREB#: H10-01023).  I, Jaesung Seo, claim sole authorship of the content of the present thesis.   iv Table of Contents  Abstract.........................................................................................................................................ii Preface..........................................................................................................................................iii Table of Contents.........................................................................................................................iv List of Tables..............................................................................................................................viii List of Figures..............................................................................................................................ix Glossary.........................................................................................................................................x Acknowledgements......................................................................................................................xi Dedication...................................................................................................................................xiii Chapter  1: Introduction..............................................................................................................1 1.1 Objectives and Research Questions .................................................................................. 4 Chapter  2: First Component – Literature Review...................................................................6 2.1 Theoretical Frameworks ................................................................................................... 6 2.1.1 Biomedical Model ...................................................................................................... 7 2.1.2 Sociomedical Model .................................................................................................. 8 Development of the Original Forms of the Oral Health Impact Profile ............. 9 Criticisms of Locker’s Model and OHIP .......................................................... 13 2.1.3 Biopsychosocial Model ............................................................................................ 14 2.1.4 OHQoL Measurement in Korean ............................................................................. 16 2.2 Validity Theory ............................................................................................................... 17 2.2.1 Trinitarian Concepts of Validity and OHIP-14K ..................................................... 17  v 2.2.2 Unitarian Concept of Validity .................................................................................. 20 2.3 Cross-Cultural Equivalence Theory ................................................................................ 21 2.3.1 Method Equivalence ................................................................................................ 23 Scale Bias .......................................................................................................... 23 Administration Bias .......................................................................................... 25 2.3.2 Item Equivalence ..................................................................................................... 27 2.3.3 Construct Equivalence ............................................................................................. 28 Chapter  3: Systematic Synthesis of the Literature.................................................................31 3.1 Search Strategy ............................................................................................................... 31 3.1.1 Forward Translation ................................................................................................. 34 3.1.2 Back-Translation ...................................................................................................... 35 3.1.3 Committee Translation............................................................................................. 36 3.1.4 Pilot Testing ............................................................................................................. 37 3.1.5 Statistical Validation ................................................................................................ 37 3.2 Content Validity .............................................................................................................. 40 3.3 Cross-cultural Equivalence ............................................................................................. 41 Chapter  4: Second Component – Empirical Exercise Methods............................................45 4.1 Selection of Bilingual Subject Matter Experts ................................................................ 46 4.2 Development of Questionnaire on Content Validity ...................................................... 49 4.2.1 Clarity Index ............................................................................................................ 50 4.2.2 Cultural Equivalence Index ...................................................................................... 50 4.2.3 Relevance Index ....................................................................................................... 51 4.3 Administration of Content Validation Questionnaire and Data Collection .................... 52 4.4 Data Analysis .................................................................................................................. 52  vi 4.4.1 Calculating the Item-level Content Validation Index (I-CVI) ................................. 53 Calculating the Average Deviation Mean Index ............................................... 53 Multirater Kappa ............................................................................................... 55 4.4.2 Scale-Content Validity Index ................................................................................... 55 Chapter  5: Results.....................................................................................................................56 5.1 Clarity of OHIP-14K Elements ....................................................................................... 56 5.2 Cultural Equivalence between OHIP-14 and OHIP-14K ............................................... 58 5.3 Relevance of OHIP-14K Domains.................................................................................. 59 5.4 Recommended Revisions or Deletions ........................................................................... 62 5.4.1 Methods.................................................................................................................... 62 5.4.2 Items ......................................................................................................................... 63 5.4.3 Construct .................................................................................................................. 64 Chapter  6: Discussion...............................................................................................................65 6.1 Content Validation Methods ........................................................................................... 67 6.1.1 Response Format on the Relevance Subscale .......................................................... 68 6.1.2 Disagreement Indices ............................................................................................... 69 6.2 Discussion of Content Validation Findings .................................................................... 71 6.3 Implications of the Content Validity Findings................................................................ 79 6.4 Study Limitations ............................................................................................................ 82 Chapter  7: Conclusion..............................................................................................................90 7.1 Study Implications .......................................................................................................... 91 7.2 Future Directions ............................................................................................................ 92 References....................................................................................................................................96 Appendices.................................................................................................................................134  vii Appendix A ............................................................................................................................ 134 A.1 Coding Search for Inclusion in Systematic Synthesis ............................................. 134 Appendix B ............................................................................................................................ 135 B.1 English Version of Oral Health Impact Profile-14................................................... 135 B.2 Oral Health Impact Profile-14K (Bae et al., 2007) .................................................. 137 B.3 Oral Health Impact Profile-14K (Kim & Min, 2009) .............................................. 139 B.4 Oral Health Impact Profile-14K (Kim & Min, 2008) .............................................. 141 Appendix C ............................................................................................................................ 143 C.1 Computation of Item-level ADM for Clarity and Cultural Equivalence Index......... 143 C.2 Computation of ADM for Clarity and Equivalence Index ........................................ 143 C.3 Computation of Multirater Kfree for Relevance index items ..................................... 144 C.4 Interpretation of Multirater Kfree Statistics ............................................................... 144 Appendix D ............................................................................................................................ 145 D.1 Informed Consent ..................................................................................................... 145 D.2 Cover Letter and Content Validation Questionnaire ................................................ 149   viii List of Tables  Table 1. Purposes for which the Oral Health Impact Profile has been applied. ......................... 2 Table 2. The original short form of the Oral Health Impact Profile (OHIP-14) questions and theoretical domains (Slade, 1997)............................................................................................. 12 Table 3. The various methods used to translate and culturally adapt the OHIP ....................... 39 Table 4. Background information of the subject matter experts ............................................... 48 Table 5. Item-level content Validity Index (I-CVI) and Average Deviation from the Mean Index (ADM) of OHIP-14K instruction, response format, event frequency, and items in the Clarity and Equivalence Indices ............................................................................................... 57 Table 6. The SMEs’ classification of items into the seven OHIP domains, Item-level Content Validity Index (I-CVI), Free-Marginal Kappa (Kfree), and levels of agreement on the Relevance Index ........................................................................................................................ 60 Table 7. Suggested revisions of OHIP-14K .............................................................................. 74 Table 8. Suggested revised version of OHIP-14K .................................................................... 81   ix List of Figures  Figure 1. International Classification of Impairment, Disability and Handicap (WHO, 1980)..... 9 Figure 2. Locker's model of oral health (Locker, 1988) [Reproduced with the permission of the editor of Community Dental Health] ........................................................................................... 10 Figure 3. Various aspects of validity and equivalency for cross-cultural measurement and comparison of OHQoL ................................................................................................................ 30 Figure 4. Stages of OHIP-14K development and proposed subscale indices of the content validity questionnaire ................................................................................................................... 49 Figure 5. Possible outcomes of content validation study of OHIP-14K ...................................... 54 Figure 6. An existential model of oral health (MacEntee, 2006) [Reproduced with the permission of the editor of Community Dental Health]. ............................................................. 88 Figure 7. The atomic model of oral health (Brondani et al., 2007) [Reproduced with the permission of the editor of Gerodontology]................................................................................. 89     x Glossary Bias: ―deviation from the truth‖ (Grimes & Schulz, 2002, p. 2). Construct: a theoretical representation of a concept, trait, attribute, or any variable that cannot be directly measured (Messick, 1989). A construct (e.g., quality of life) is often multi-faceted and operationally defined by a theory. Construct validity: how well a scale measures the construct of interest theorized by the underlying model (Hubley & Zumbo, 1996). Content validity: the relevance and representativeness of the scale content (e.g., items or questions) to the construct of interest and its appropriateness for the target population (Beck & Gable, 2001). Convergent validity: ―the notion that the application of two or more different measures of the same trait should lead to highly correlated, or convergent, results‖ (Larcker & Lessig, 1980). Criterion validity: empirical correlation between a test score and concurrent or predictive criteria (Messick, 1989). Cultural adaptation: a process of translating and modifying an existing scale to reflect cultural values and norms of target populations for use in a new country or culture (Guillemin et al., 1993). Discriminant validity: ―the notion that the correlation between different measures of the same trait should be greater than the correlation between different measures of different traits and the same measure of different traits‖ (Larcker & Lessig, 1980).  xi Equivalence: unbiased measurement of actual differences among different cultural groups (Cella et al., 1996). Cross-cultural equivalence is broadly divided into measurement equivalence (the extent to which the item content is interpreted the same way across cultures) and structural equivalence (the degree of congruency in the underlying theoretical structure and relations of items measured by subscales) (Byrne & Watkins, 2003). Face validity: respondents’ perception of the scale’s appropriateness for the purpose of measuring the construct of interest (Nevo, 1985). Fidelity: the extent to which a culturally adapted version of a scale is faithful to the original scale (Corless et al., 2001). Measurement: a process of linking the observable empirical indicants to the underlying unobservable construct (Zeller & Carmines, 1980). Reliability: the extent to which repeated measurement of the same property produces the same result. Scale: ―a standardized procedure for sampling behaviour and describing it with categories or scores‖ (Gregory, 2007). Validity: confidence that the scale measures what it purports to measure in the target culture or population (Sandelowski, 1986). Acknowledgements I offer my utmost thanks to my research supervisor Dr. Mario Brondani for his support in my personal, academic, and professional endeavours. I would also like to thank my co-supervisor  xii Dr. MacEntee and my MSc committee, Drs. Bryant, Graf and Wong, for their teachings, guidance and encouragement. Finally, I would like to thank my wife Ewa Seo and family members for their selfless devotion and unconditional support for turning my academic pursuit into reality.   xiii Dedication  This thesis is dedicated to my wife, Ewa Seo, for always being there for me and enduring struggles and hardships together with me.  1 Chapter  1: Introduction Emerging from the landmark study by Cohen and Jago on sociodental indicators (1976), oral health-related quality of life (OHQoL) has come to represent a psychological construct defined as self-reports from people pertaining to the functional, psychological and social impacts of oral problems on their quality of life (Gift & Redford, 1992). To date, more than a dozen OHQoL measurement tools have been developed, as summarized by Slade (1997), Allen (2003), Johns and colleagues (2004), Hebling and Pereira (2007), and again by Brondani and MacEntee (2007). Unlike clinical indices, their primary objective is to capture biopsychosocial consequences of oral problems that are perceived to be important by lay respondents (Allen, 2003). Among these is an internationally recognized and frequently used scale called the Oral Health Impact Profile (OHIP) (Slade & Spencer, 1994). Originally developed in Australia, OHIP is a self-report measure of dysfunction, discomfort and disability due to oral conditions. It consists of 49 questions representing seven domains (impairment, functional limitation, discomfort/pain, physical, psychological and social disabilities and handicap) and 5-point Likert response categories (4 = ―very often,‖ 3 = ―fairly often,‖ 2 = ―occasionally,‖ 1 = ―hardly ever‖, and 0 = ―never,‖ with an optional ―don’t know‖) (Slade, 1997). A shortened version of the OHIP called OHIP-14 was later developed based on the 14 of original 49 questions (subsets of 2 questions for each of the seven domains, Table 2). The short form is preferred to the longer OHIP-49 by many researchers and respondents because it takes less time to complete, costs less for administration and data management, and imposes less burden on the respondent, especially if they are frail, thus helping to improve the item-response rate (Saub et al., 2005). Both the long and short forms of the OHIP have been widely used as an evaluative and discriminative tool for research and clinical outcomes (Table 1).   2 Table 1. Purposes for which the Oral Health Impact Profile has been applied. Purpose of Application Sources Health care utilization, resource allocation and the quality of care and interventions Atchison, 1997; Pearson et al., 2007; Walker & Kiyak, 2007 Oral health promotion and disease prevention programs Locker, 1995 Patient-centred care-planning Slade, 1997; Jones et al., 2004 Patient outcome-evaluation Gift, 1997; Hebling & Pereira, 2007 Prioritization of research and treatment options Cohen, 1997; Calabresse et al., 1999 Interdisciplinary health-care  Hayran et al., 2009 Measurement of psychological burden of oral problems  Cohen 1997; Calabrese & Friedman, 1999; Sampogna et al., 2009 Patients’ assessment of care  Allen, 2003 Impact of oral disorders on patients with systemic diseases Mumcu et al., 2006; Jeganathan et al., 2011  Although commonly applied across all age groups, OHQoL is particularly important for older adults who have a greater accumulation of chronic oral problems, and where the primary treatment goals are to improve basic functions (Kressin et al., 1996). The growing interest in OHQoL measurement has also spread widely, and the original OHIP in English has been translated into at least 19 languages (Appendix B.1), which has subsequently stimulated cross- cultural comparisons of OHQoL (Allison et al, 1999; Kawamura et al., 2000; Kawamura et al., 2001; Steele et al., 2004; Slade et al., 2005; Bae et al., 2007; Rener-Sitar et al., 2008). This momentum in cross-cultural measurement of OHQoL has occurred also in South Korea, which has one of the fastest-aging populations (Cha, 2009). The National Statistics Office (2007) of South Korea projected that this country will become the most aged country globally by 2050, when 38.2% of the population will be aged 65 and older, nearly double the projected world average of 16.2%. Despite the drastic demographic shift, oral health is perceived as a low  3 priority by South Korean older adults and their caregivers. However, the assessment of OHQoL in this population could bring benefits in terms of detecting oral and nutritional problems (Jung & Shin, 2008) and treatment needs (Bae et al., 2007) by improving public oral health promotions (Ha et al., 2012) and studying oral health outcomes (Cha et al., 2011). The benefits of OHQoL cultural adaptation could be conferred not only to elders in Korea but also to Korean immigrants elsewhere who speak Korean. OHQoL measurement in Korean is important in the Canadian context as Canada is home to the world’s third-largest Korean immigrant population, whose settlement is concentrated in Vancouver and Toronto (Lee, 2010). They represent the seventh-largest incoming group of immigrants in Canada, and the third-largest visible minority group in the Greater Vancouver with approximately 2.3% of the total population (City of Vancouver, 2000). Moreover, 12.0% of them are aged over 55 years (Statistics Canada, 2006). The first generation of elderly Korean immigrants, who are not fluent in English, are most likely to benefit from OHQoL assessment in Korean. Originally developed for use in Southern Australia, the OHIP has been culturally adapted to the Korean population by Bae and colleagues (2007). In lieu of developing an entirely new OHQoL scale (i.e., de novo method), adapting a supposedly validated scale to target cultures has the advantage of providing cheaper, quicker and easier scale preparation; cross-language support; inclusion of immigrants; and fair cross-cultural comparisons (Hambleton & Patsula, 1998; Guillemin et al., 1993). Cultural adaptation of the scale is usually followed by the extensive validation efforts. Moreover, given the vast amount of literature on cross-cultural comparisons of OHIP scores, the researchers worldwide might hold the view that the current methods of cultural adaptation and validation preserve the validity and equivalence of the original version of OHIP for obtaining valid and reliable information. However, it remains unclear how these properties are conceptualized in cross-cultural OHQoL researches and established by the current  4 standards of cultural adaptation and validation methods. More importantly, to what extent a translated version of the OHIP exhibits these properties has yet to be fully understood. 1.1 Objectives and Research Questions Given these introductory remarks, the overriding research aim of my thesis was to describe the concepts of validity and equivalence in the context of cross-cultural OHQoL measurement and to recommend an effective process for investigating these properties. In regard to the aims of my study, I have organized my dissertation into two major components. The first component is more theoretical and is comprised of a synthesis of the existing literature that will allow me to critically examine the theoretical frameworks underpinning the existing scales of OHQoL with a main emphasis on the OHIP. I will then analyze the literature on different strategies for the adapting and validating English-based psychometric measures for other cultures. For this I will pose three research questions, which, in turn, will help me to discuss theories of validity, equivalence and measurement. The first two questions are: 1. How are validity and cultural equivalence conceptualized in cross-cultural settings? 2. What are the strengths and weaknesses of existing methods for culturally adapting and validating the OHIP-14?  The second component of my dissertation is more empirical by using exercises with SMEs to assess the content validity and cultural equivalence of the Korean version of OHIP-14 (OHIP- 14K) with the original OHIP-14. In this section I will address my third research question: 3. To what extent does OHIP-14K exhibit content validity and cultural equivalency in Korean?   5 I will then discuss the methods I used and the results I gathered from this exercise. Finally, I will proceed with the concluding discussion and offer the implications and future directions of my findings.  6  Chapter  2: First Component – Literature Review 2.1 Theoretical Frameworks The theoretical approach to validating the OHIP requires adherence to psychometric theory, which refers to ―all those aspects of psychology which are concerned with psychological testing, both the methods of testing and the substantive findings‖ (Kline, 2000, p. 1). There are two main branches of measurement theory in the field of psychometrics: classical test theory (CTT) and item response theory (IRT). CTT assumes that a respondent has an observed score and a true score, and that these can be interpreted against a reference or norm group. IRT makes similar assumptions about the observed and true scores, but it delves deeper into modelling the relationship between responses to each item and each construct (Reeve & Masse, 2004), allowing for more advanced statistical analyses. Both CTT and IRT are compatible with the concepts of validity and cultural equivalence, but my choice of CTT as the theoretical measurement framework is motivated by its alignment with the OHIP design. Many features of the OHIP are compatible with the assumptions of CTT in terms of equal contribution of individual items to the total score, constancy of the true score from test to retest, and dependence on a sample of participants to determine the properties of the scale, such as item discrimination (Magno, 2009; Wong et al., 2010). Moreover the OHIP, like most OHQoL measures, is validated with methods grounded in CTT, including internal consistency, criterion-related validity, and test-retest reliability (Locker, 2008). IRT, on the other hand, requires unidimensionality of the construct as in the case of the construct of alcohol dependence. (Helzer et al., 2008). Therefore, CTT is a better alternative for  7 studying multidimensional constructs such as OHQoL and used as a guiding principle for my literature review and for the design and analysis of my validation study. Now I will review the literature on the concept of OHQoL. Over the years, efforts to conceptualize and quantify OHQoL have been shaped by the evolving frameworks or models of health and disability. 2.1.1 Biomedical Model Traditionally, the most predominant theoretical framework in oral health research had been the biomedical approach, which focuses primarily on diagnoses and pathologies when defining health (Levasseur et al., 2004). The biomedical view of health and illness was the result of a mechanical view of biology called the Cartesian mechanical paradigm, which constituted the foundation of modern medicine (Capra, 1982). From this biomedical point of view, an absence of oral health is synonymous with the clinical presence of disease and tissue destruction (Slade & Sanders, 2003), as exemplified by WHO’s definition of oral health as ―a state of being free from chronic mouth and facial pain, oral and throat cancer, oral sores, birth defects such as cleft lip and palate, periodontal disease, tooth decay and tooth loss, and other diseases and disorders that affect the oral cavity‖ (2008, para. 1). Pursuant to this definition, oral health was frequently measured with clinical indices such as decayed, missing, filled teeth (DMFT) (Dean, 1942), periodontal index (Russell, 1956), community periodontal index of treatment needs (CPITN) (Cutress et al., 1982), gingival bleeding index and plaque index (Silness & Loe, 1964), and so on. However, the biomedical model can only define the presence or absence of oral problems; it cannot account for behavioural and subjective inner experiences prompted by the problems (Warner, 1999). Researchers have long been perplexed by the lack of correspondence between the clinical status and subjective impacts of oral health (Leao & Sheilham, 1995). The  8 traditional clinical measures are too narrow in scope to capture various dimensions of human experience (Slade & Locker, 1994) and they provide limited scope of the consequences of oral disorders (Reisine, 1985). In response to these criticisms and the need for global scales and broader conceptual frameworks, two main schools of thought have emerged: sociomedical and biopsychosocial models. 2.1.2  Sociomedical Model From the sociomedical perspective, individuals’ oral health status is directly linked to their oral health’s effect on their task performance and usefulness to society, as exemplified by Dolan’s definition: ―a comfortable and functional dentition which allows individuals to continue their desired social role‖ (1993, p. 37). The need for tools with which to measure these social factors and impacts of oral health, such as personal lifestyle or cultural and ecological factors, was first recognized by Cohen and Jago (1976) and has led to development of more than a dozen OHQoL measures (see Slade, 1997; John et al., 2004; MacEntee, 2006; Brondani & MacEntee, 2007; Hebling & Pereira, 2007). They attempt to describe lay people’s perception of their own oral health and, in tandem with clinical measures, to determine health care goals and assess the degree to which they are met (Cushing et al., 1985) (Table 1). Three interdependent sociomedical frameworks have been used as the theoretical basis for formulating OHQoL measures: Parsonian Sick Role Theory (Parsons, 1951); International Classification of Impairments, Disabilities and Handicaps (ICIDH) (WHO, 1980); and Locker’s model of oral health (Locker, 1988).  Resine (1981) first proposed the use of a sociomedical framework called Parson’s sick role theory, which posits that a sickness warrants exemption from fulfilling one’s social role; that it  9 is the role of health professionals to restore a sick person’s health; and that an illness is an undesirable and socially deviant behaviour, so sick people must seek treatment from a health professional in order to substantiate their sick roles (Parsons, 1951). The sick role theory first found its place in the WHO’s framework of disability, known as the ICIDH (WHO, 1980). ICIDH explains consequences of disease in three dimensions (Figure 1):  Impairment: organ-system-level loss of structure or function.  Disability: person-level loss of the ability to function in ways considered normal for a human being.  Handicap: societal-level disadvantage for the person created by the intersection of the impairment or disability with the person’s environment and roles. Development of the Original Forms of the Oral Health Impact Profile Presently, the key concepts of ICIDH and the sick role theory provide the theoretical basis for the majority of OHQoL scales, including the OHIP (MacEntee, 2006). For instance, the theoretical framework of OHIP is the oral health interpretation of ICIDH via Locker’s model of oral health. According to this model, oral disease can have various consequences in a sequential manner, whereby impairments caused by the oral disease lead to functional limitations or discomfort/pain, which lead in turn to disability and ultimately to handicap (Figure 2) (Locker, 1988). In some cases, impairments and discomfort/pain can directly lead to handicap, and intervening variables such as personal and environmental factors can play an important role in oral health (Baker, 2007). The main merit of such a conceptual model is that it no longer defines Disease Impairment Disability Handicap Figure 1. International Classification of Impairment, Disability and Handicap (WHO, 1980)  10 oral health as merely an absence of disease but also as a state that includes functional, social and psychological well-being, thereby enabling researchers to focus on a range of physical, psychological and social roles (Locker, 1988).   In accordance with its theoretical framework, the aim of OHIP is to measure self-reports pertaining to impairment, disability and handicap due to oral conditions. In OHIP, 46 out of 49 items were factual statements identified from the personal interviews with 64 South Australian dental patients to reflect 7 domains hypothesized by Locker’s model of oral health. Three additional questions were added ad hoc by the researchers as none of the 64 participants were concerned with handicap, which Locker had identified as a domain of oral health (Slade, 1997). The long-form OHIP-49 with 49 items was reduced to develop various shorter forms, two of which present 14 items each derived from two different methods: regression method and item- impact method. In the regression method, items are selected based on their prevalence and ability to predict overall scores (Slade, 1997) whereas the item-impact method used respondents’ ratings on the importance of each question (Locker & Allen, 2002). It is suggested that the regression method may be more suitable for studies whose aim is to discriminate Disease Impairment Functional Limitation Disability Physical Psychological Social Discomfort & Pain Handicap Figure 2. Locker's model of oral health (Locker, 1988) [Reproduced with the permission of the editor of Community Dental Health]  11 between individual patients while the item-impact method might be better for describing the OHQoL of populations for epidemiological studies (Locker & Allen, 2002). The OHIP-14 is scored and analyzed for individual theoretical domains or for the entire profile (Table 2) in one of two ways: (1) by simply counting items above the threshold response (e.g., ―fairly often‖), or (2) by adding up the Z scores of individual items (the individual score minus the mean of the population divided by the standard deviation of that population) (Slade, 1997). The former scoring method tends to underestimate scores because not many individuals report impact above the threshold, thereby violating assumptions of parametric distribution. Nevertheless, the summation scoring method is preferred to the latter Z-score method, which can be too laborious and complicated for practical purposes (Slade, 1997). Locker and Allen (2007) recommended that OHIP-14 be scored by the numbers of above-threshold responses out of the14 items. Alternatively, the numeric values of Likert scale responses (e.g., 0 = never) for the 14 items can be combined to give total scores ranging from 0 to 56, higher values indicating more frequent oral health impact. However, the significance of overall profile scores has been questioned by MacEntee (2006) and again by Brondani and MacEntee (2007), as the same scores can be achieved by a different set of answers. For the same reason, the profile scores are frequently reported with domain scores rather than as a single total score.   12 Table 2. The original short form of the Oral Health Impact Profile (OHIP-14) questions and theoretical domains (Slade, 1997) Theoretical Domains OHIP-14 Item Functional Limitation  Q1. Have you had trouble pronouncing any words because of problems with your teeth, mouth or dentures? Q2. Have you felt that your sense of taste has worsened because of problems with your teeth, mouth or dentures? Pain & Discomfort  Q3. Have you had painful aching in your mouth? Q4. Have you found it uncomfortable to eat any foods because of problems with your teeth, mouth or dentures? Psychological Discomfort Q5. Have you been self-conscious because of your teeth, mouth or dentures? Q6. Have you felt tense because of problems with your teeth, mouth or dentures? Physical Disability  7. Has your diet been unsatisfactory because of problems with your teeth, mouth or dentures? Q8. Have you had to interrupt meals because of problems with your teeth, mouth or dentures? Psychological Disability Q9. Have you found it difficult to relax because of problems with your teeth, mouth or dentures? Q10. Have you been a bit embarrassed because of problems with your teeth, mouth or dentures? Social Disability  Q11. Have you been a bit irritable with other people because of problems with your teeth, mouth or dentures? Q12. Have you had difficulty doing your usual jobs because of problems with your teeth, mouth or dentures? Handicap Q13. Have you felt that life in general was less satisfying because of problems with your teeth, mouth or dentures? Q14. Have you been totally unable to function because of problems with your teeth, mouth or dentures?   13 Criticisms of Locker’s Model and OHIP Although Locker’s model is one of the most widely accepted theoretical frameworks of oral health, it is not without limitations. Among the criticisms are its negative portrayal of health; the linear presentation of disease states; and the unequal power distribution between doctor and patient. Owing to its roots in Parson’s sick role theory, Locker’s model of oral health has negative classification labels, such as impairment, disability and handicap, and ignores the positive end of the health spectrum including the social benefits of positive oral aesthetics, for example. This seems to be a common problem among OHQoL measures that are based on the early ICIDH classifications. In review of 17 of the existing OHQoL scales, almost all were found to have items with negative connotations (Brondani & MacEntee, 2007). Consequently, few, if any, ICIDH-based scales capture the full range of oral health states and experiences, such as positive oral health behaviours and beliefs as well as coping and adapting to the oral problems (MacEntee, 2006; Brondani & MacEntee, 2007). Moreover, ICIDH-based scales have retained a vestige of the sick role theory, which dictates that it is the exclusive role of health professionals to reverse the progression of the disease. As a result, ICIDH does not allow for diseases that have unknown causes, nor does it consider convalescence, coping and adapting strategies or reciprocal influences among social, psychological, and biological processes. The negative connotations associated with disability and handicap might further encumber people with disabilities, causing loss of self-esteem and the development of defensive coping and adapting (Barnes & Mercer, 1996). Additionally, the unequal patient-doctor relationship of ICIDH ignores lay perspectives on oral health. As noted by Berkonovic (1972) and Conrad (1990), the sick role theory offers a rather limited scope of health and illness from the standpoint of outsiders, such as health care  14 providers or medical sociologists, while overlooking the subjective perceptions and expectations of those who experience and manage their illness. This type of patient-doctor relationship is also reflected in the sociomedical definition of oral health: ―the objective which the dental profession has in mind in assisting the individual to achieve a satisfactory state of function, comfort, and appearance‖ (Gerrie, 1947, p. 1318). In this reflection, a normal mouth is based on the opinion of a dentist, irrespective of the patient’s concerns (McCollum, 1943; MacEntee, 1997). Given these limitations, the current paradigm of oral health is gradually becoming a biopsychosocial conceptualization. 2.1.3 Biopsychosocial Model In keeping with WHO’s definition of health as a state of complete physical, mental, and social well-being – and not merely the absence of disease (WHO, 1977), Engel (1977) offered an alternative route to expanding the traditional biomedical model. His overarching framework, better known as the biopsychosocial model, is a comprehensive multi-systems model in which a combination of biological (e.g., immune system), psychological (e.g., depression) and social factors (e.g., socioeconomic status) contributes to health in various contexts. The main advantage of the biopsychosocial model is its ability to analyze individual responses to the same disease within a multifaceted perspective on health. Further improvements in the biopsychosocial model included consideration of a patient’s subjective experience and active roles in the traditional patient-doctor relationship (Borrell-Carrio et al., 2004). The emergence of the biopsychosocial view of health demanded expansion of the original ICIDH framework (WHO, 1980) in order to incorporate quality-of-life (Jette, 1993; Tennant & McKenna, 1995) and the lay perspective of patients (Peters, 1996). The WHO (2001) later revised the ICIDH in the context of the International Classification of Functioning, Disability and Health (ICF) –  15 provisionally titled ICIDH-2 – to include new dimensions, such as environmental factors and participation in social activities. However, the new dimensions of health have yet to be reflected in the currently available Socio Dental Indicators (SDIs) (Brondani & MacEntee, 2007).  In accordance with the transition to the biopsychosocial framework in medicine, the concept of oral health acquired a more comprehensive definition: "a standard of the oral and related tissues which enables an individual to eat, speak and socialise without active disease, discomfort or embarrassment and which contributes to general well-being" (UK Department of Health, 1994, p. 55). Moving away from the sociomedical definition that focuses on social roles, the new biopsychosocial definition of oral health offers an integrated and holistic view that is intricately linked to health and quality of life (QoL) by referring to ―an individual's perceptions in the context of their culture and value systems, and their personal goals, standards and concerns‖ (WHO, 2011). As presented earlier on, the need for quality-of-life measures sensitive enough to detect changes in oral health status and the psychosocial impacts of oral problems also led to the development of OHQoL scales, which have been identified also as Socio Dental Indicators (SDIs) (Cohen & Jago, 1976). In summary, both the sociomedical and biopsychosocial models influenced the contemporary conceptualization of OHQoL. However, at the present time, most OHQoL scales – including the OHIP – are based on sociomedical frameworks, such as the older ICIDH, rather than the newer ICF. In my study, because of the theoretical roots of the OHIP, I used the former framework for the purpose of content-validating exercise in accordance with Locker’s ICIDH-based model.    16 2.1.4 OHQoL Measurement in Korean Recently, many OHQoL measures have been made available in Korean (Bae et al., 2007; Jung et al., 2008; Lee et al., 2008; Shin & Jung, 2008; Ha et al., 2009; Kim et al., 2009), including the OHIP-14 (Slade & Spencer, 1994). There are at least three different variations of OHIP-14K (Bae et al., 2007; Kim & Min, 2008; Kim & Min, 2009). Even though they seem analogous (Appendix B.2, B.3, & B.4), it is confusing to have multiple variations of the same scale (van Windenfelt et al., 2005), and possibly in violation of the original OHIP-14 copyright (Hunt et al., 1991). I will use the translation of the OHIP-14 by Bae and colleagues (2007) as the official version of the Korean OHIP because it was the only version whose cultural adaptation and validation have been documented in the literature (Bae et al., 2007). In order to appraise cross-cultural measurement of OHQoL in Korean, I will first describe the theories of validity and cultural equivalence by attempting to answer my first research question: how are validity and cultural equivalencies conceptualized in cross-cultural settings?   17 2.2 Validity Theory Validity is defined as confidence that the scale measures what it purports to measure in the target culture or population (Sandelowski, 1986). Despite this simple definition, there have been many controversies surrounding the concept of validity; however, the Trinitarian and Unitarian theories dominate. 2.2.1 Trinitarian Concepts of Validity and OHIP-14K Owing to its roots in educational and psychological testing, traditional validation efforts are centred on demonstrating the utility of scales by correlating its scores with external variables or criteria, such as reading skills (Thurstone, 1932). Thorndike (1931) and Jenkins (1946) criticized this validation approach on the grounds that the statistically driven approach does not capture the relevance of criteria to the measurement purpose or, more importantly, to the validity of the criteria themselves (Sireci, 1998). To overcome these limitations, Kelley proposed that criterion-related evidence be combined with professional judgment (1927), which led subsequently to the emergence of the Trinitarian concept of construct, criterion, and content validity whereby:  Construct validity is the ultimate goal of psychometric measurement by indicating how accurately a scale reflects the construct of interest theorized by an underlying model selected by the scale developer (Hubley & Zumbo, 1996; Devellis, 2003). In my thesis, the construct validity of OHIP-14K indicates how well the construct of OHQoL hypothesized by Locker’s model of oral health is measured in a Korean context. Construct validity is typically asserted in one of three ways. The first way is through correlation with a gold standard, although construct validity of OHIP measurement in  18 English or in other languages has not been established because there is no accepted gold standard for OHQoL (McGrath & Bedi, 2001). The second way is by the statistics of factor analysis ―whose common objective is to represent a set of variables in terms of a smaller number of hypothetical variables.‖ (Kim & Mueller, 1978, p.57) However, the accuracy of factor analysis is only as good as the validity of the data being modelled. Before performing a factor analysis, any potential bias in the content of the scale must be addressed to identify or explain the causality of score variations (Rahman et al., 2010). The third and final way is by establishing convergent validity by assessing the extent to which two or more scales of the same trait correlate (Larcker & Lessig, 1980; Hubley & Zumbo, 1996).   Criterion validity is an empirical correlation between a test score and concurrent or predictive criteria (Messick, 1989), such as behaviours, personalities, abilities or outcomes, depending on the measurement objective of the scale. The current validation strategies for OHIP rely heavily on criterion-related variables, which vary greatly between the different language versions. In general, one tests for score differences between categories of respondents that are expected to differ (e.g., dentate vs. edentulous populations). However, criterion validity does not provide a theoretical explanation of the relationships between or causality of observed scores. With criterion-related evidence, it is not certain that what is being measured is indeed the construct of interest (e.g., OHQoL) or that the observed score is meaningful to the respondents (Sim & Waterfield, 1997).   19  Content validity refers to expert judgment about how the scale measures the situations or subject matter from which conclusions are drawn (Sireci, 1998). Unlike criterion validity, it is concerned with the clarity, relevance, and precision of the questions (Rulon, 1946; Mosier, 1947; Hubley & Zumbo, 1996; Yagmhmale, 2003). It serves as ―an important indicator of the instrument’s quality‖ (Polit & Beck, 2006, p. 489) and as ―a link between abstract theoretical construct and measurable indicators‖ (Wynd et al., 2003, p. 508). Therefore, content validity is crucial evidence of construct validity (Anastasi, 1988; Messick, 1989; Ding & Hershberger, 2002).  To date, the construct validity of OHIP14-K has been supported by Bae and colleagues (2007), who compared the observed OHIP scores to the criterion-related variables, such as perceived dental needs and number of natural teeth. Other methods for testing the content validity of OHIP-K have been suggested, including consultation with lay participants, or relying upon the judgment of subject-matter experts. A participant’s perception of the scale’s appropriateness is known as the ―face validity‖ (Nevo, 1985). For instance, Montero-Martin and colleagues (2010) examined the face validity of the Short Form Health Survey (SF-36) to validate its content and found that its content domains were indeed relevant to target populations. However, face validity is extremely difficult to quantify and is insufficient as a psychometric property because judgments of validity are based mostly on the appearance of items rather than on their deeper theoretical foundation (Mosier, 1947; American Psychological Association, 1974). For this reason, face validity has been integrated with content validity (Haynes et al., 1995).  Another method of content-oriented validation involves the collection of expert judgments by administrating evaluation surveys to those with adequate experience and knowledge of disease  20 on quality of life (Lynn, 1986; Sireci & Geisinger, 1995; Beitz & van Rijswijk, 1999; Wynd et al., 2003; Hubley & Palepu, 2007). However, it has not been used for validating an OHQoL scale like the OHIP-14. The main advantages of this method are simplicity, ease of use, cost- effectiveness, speed of application, and quantitative assessment of content validity index (CVI), all of which help researchers to make decisions about retaining, revising or eliminating items from a self-report scale (Schilling et al., 2007).  Up to this point, three broad attributes of validity have been reviewed. Although this conventional view of validity is useful for describing different aspects of this phenomenon, it might also benefit to discuss the unifying validity framework. 2.2.2 Unitarian Concept of Validity Within the Unitarian framework, criterion-related, content-related and construct-related validity help to demonstrate construct validity (AERA, APA & NCME, 1999). In other words, construct validity of a measurement is supported by theoretical considerations and empirical evidence (Lawshe, 1975). There are two views of the type and amount of evidence required to validate a psychosocial measurement. Kane (2009) argued that validity should depend on an assessment of the purpose of the measurement rather than on the scale itself. The predictive attributes of the measurements, if they exist, are what really matter. Zumbo (2009) contended that criterion-driven validation overlooks the importance of theoretical explanations for the scores and for item response variations (2009). Vickery (1998) took this argument a step further by claiming that construct validity and reliability are meaningless without content-related evidences because content validity appeals to reason and theoretical justification. This view is shared by Messick (1989, p. 13), who defined validity as ―an  21 integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the interpretations of assessment scores‖. In this context, content validity, the principle for theoretical explanation of the measurement, is in fact an attribute of construct validity. As noted by many validity theorists including Sireci (1998) and Schouwstra (2000), this Unitarian framework is gaining ground as a suitable theoretical basis for studying validity because construct validity cannot be established if the content of the scale is not valid. Another main tenet of this framework is that construct validity can be strengthened by minimizing threats, including construct underrepresentation (i.e., a scale fails to capture important dimensions of the construct) and construct irrelevant variance (i.e., variance due to non-related constructs or method variance) (Messick, 1989). In my thesis, I adopted Messick’s Unitarian theory of validity as the conceptual framework in which content- and criterion-related validities are integrated as a crucial piece of evidence for the measurement’s construct validity (Messick, 1989; Yaghmale, 2003). An investigation of the content of the OHIP-14K, however, requires further discussion of another closely related but distinct concept: cross-cultural equivalence theory. 2.3 Cross-Cultural Equivalence Theory A cross-cultural comparison of OHQoL provides cultural perspectives on the conceptualization and measurement of oral health. Central to the comparison is the notion of cultural equivalency. Because the accuracy of self-reported data is incumbent on the quality of cultural adaptation, linguistically and culturally equivalent scales are needed in multi-regional research. However, establishing cultural equivalence between two different-linguistic versions of the same scale is often challenged by the conflict between fidelity and cultural appropriateness (Corless et al.,  22 2001). There are two competing views on the extent of accommodating cultural appropriateness in the translation in a way that challenges its fidelity to the original version: the emic and etic perspectives. The emic perspective interprets phenomena within a culturally specific context that is significant to its members, whereas the etic perspective studies cultural phenomena from an outsider’s point of view (Pike, 1967; Berry, 1990). The emic perspective has the advantage of providing an insider’s view of culture by dealing with culturally relevant subjects, and preventing an imposition of the researcher’s values and biases. Hence, cultural adaptation of OHIP should consider the surface structure, such as ethnic/cultural characteristics, experiences, norms, values, behavioural patterns and beliefs of a target population, as well as the ―relevant historical, environmental, and social forces‖ influencing the culture (Resnikow et al., 2000, p. 272). However, it is not yet known whether it is possible to penetrate native experiences to provide a compatible framework for cross-cultural comparisons (Cohen, 1974; Alegria et al., 2004).  The etic or culturally comparative approach observes through generalizations of cultural characteristics and values from an outsider’s perspective (Gzagnon & Tuck, 2004). It has been criticized for assuming an idealistic or universal application of the observer’s values and biases to all cultures (Cohen, 1974) and for encouraging cultural homogeneity by imposing ethnocentric constructs on other cultures (Alegria et al., 2004). Consequently it could lead to a culturally insensitive scale that lacks content and construct validity for that target population (Rogler, 1999). Detailed examples of each case will be presented in section 2.4.3, Construct Equivalence, on page 28. Given the merits of etic and emic perspectives in cross-cultural studies, my investigation of OHIP-14K includes both. Such a dual perspective on cultural equivalence has been suggested  23 by many researchers (Cohen, 1974; Bravo et al., 1991; Canino & Bravo, 1994; Herdman et al., 1998; Jones et al., 2001), and it defines cultural equivalence as multi-dimensional properties of cross-cultural comparison conceptualized broadly in terms of method, item, and construct equivalence (Hunt et al., 1991; van de Vijver & Tanzer, 1997). 2.3.1 Method Equivalence Method equivalence, also known as technical (Canino & Bravo, 1994) or operational (Herdman et al., 1998) equivalence, refers to the use of unbiased methods for designing and administering multilingual versions of the same measure (van de Vijver & Tanzer, 1997). Method equivalence is one of the most overlooked aspects of cultural adaptation (Sireci et al., 2006), and to date, there is no benchmark for standardizing the format of the scale or the instructions in different languages. Potential threats to a scale’s method equivalence include biases related to the scale or administrative methods. Scale Bias Scale bias refers to cross-cultural differences in scores due to differences in the format of a questionnaire especially relating to the way in which possible responses are presented (van de Vijver & Tanzer, 1997). Consequently, the structures and format of the original and the translated versions must be preserved while accommodating the cultural needs of the target participants. For example, participants who are unfamiliar with the format of a Likert scale might find it confusing to use (Bracken & Barona, 1991). A scale bias could also occur during the cultural adaptation of a questionnaire if Likert categories are narrow in scope or ambiguous. For example, the Likert scale of the original English version of OHIP allows respondents to answer each question with five choices plus an optional ―don’t know‖ (DK) response (Slade &  24 Spencer, 1994), yet the OHIP-K does not offer the DK option (Bae et al., 2007). The DK options seem to eliminate guessing which should improve the accuracy of the responses. In particular, answering the OHIP questions is a memory-dependent task as it requires respondents to recall related events from the past 3, 6 or 12 months, and the DK options is useful for respondents who cannot recall the events or simply don’t know the answer to the question (Ritter & Sue, 2007). In support of this idea, Courtenay and Weidemann (1985) found that DK option eliminates guessing and produces more accurate answers on Palmore’s Facts on Aging Quiz. On the other hand, there are concerns that DK responses is difficult to interpret for researchers (Holbrook, 2008) and that it might distract respondents from making a choice from the more definite possibilities (Krosnick, 1991; Gilljam & Granberg, 1993). However, these findings mostly apply to opinion-type questions about personal dispositions, attitudes, beliefs or agreements rather than to factual questions such as those posed by the OHIP. Indeed, the low non-response rates from the sampled populations in many oral health surveys suggest that the DK options in the OHIP are less likely to be abused by respondents (Locker, 1993). Overall, it seems that keeping the response formats uniform across different-language versions is necessary to minimize the scale bias for valid cross-cultural comparisons. Another example of a scale bias can be found in the Likert scales used for another existing version of the OHIP-14K proposed by Kim and Min (2008). In this Korean translation of ―very often, fairly often, occasionally, hardly ever and never” in the original English OHIP, we see that the choices ―hardly ever” and ―never ― became ―it’s not like that‖ and ―it is never like that‖ (Appendix B.4). Not only do these response categories have unequal intervals, but they also offer little or no information about the frequency of events.    25 Administration Bias An administration bias is caused by different instructions or other communications for using the instrument (Feinstein, 1987; van de Vijver & Tanzer, 1997). Yet the instructions given in the OHIP-14K differ significantly from their English counterparts in recall periods, frequency of oral problems, and range of oral health problems (Bae et al., 2007). The one-year recall period of oral health-related events in the original OHIP was reduced to three months in the OHIP-14K. Shorter recall periods tend to yield more accurate responses but underestimate the prevalence of problems, and vice versa for longer recall periods (WHO, 2003). Hence, longer recall periods are ideal for describing chronic oral problems, but may miss acute changes in oral health and impose a cognitive burden on respondents, especially elderly respondents with some cognitive impairment. This may support the shortened recall period for the OHIP-14K; however, the difference in recall periods might bias cross-cultural comparisons. For instance, the lower test-retest reliability (intra-class correlation coefficients) for the OHIP- 14K (0.40–0.64) compared with German (0.63–0.92) and Chinese versions (0.63–0.92) lead Bae and colleagues to conclude that OHIP-14K might be measuring acute changes in respondents’ oral health status rather than their oral health status overall.  Unlike the original version of OHIP-14, the OHIP-14K gives a defined set of frequencies per week for each Likert response (e.g., very often = more than 3 times a week). The additional instructions could impose a cognitive burden on respondents and could also reduce bias as it eliminates the variation caused by individual perceptions and judgments of the frequency of the events. Nonetheless, the objective criteria can have unintended consequences by introducing researchers’ biases into their measurement of OHQoL when the aim is to reflect the  26 participants’ subjective perceptions on the impact oral health rather than on their oral health history. There is a chance that response categories arbitrarily set by researchers may be different from those determined by the respondents. For example, ―more than 3 times a week‖ could imply occasional happenings for some, and high frequency for others. Thus, the inclusion of specific frequencies in one language and not in others could lead to misleading or at least incomparable responses across cultures.  Another source of method bias is the limited scope of oral problems defined by OHIP-14K. It only asks about problems with teeth and gums whilst the OHIP includes problems associated with the tongue, dentures or mouth in general. This instruction bias could underestimate the oral problems of Korean respondents. Also, the way in which the directions are presented to respondents may affect their responses. In the original OHIP, many of the oral health-specific questions overlap with systemic health or psychological conditions (e.g., trouble pronouncing words due to physical impairment of vocal cords or muscle tone, not due to oral problems). Therefore, every OHIP question in English has the phrase ―because of problems with your teeth, mouth or dentures‖ to define the topic of interest, while this phrase was omitted from the OHIP- K items. Repeating the instructions in every question may be necessary to improve the respondents’ comprehension and responses (Smith, 2004; Somekh & Lewin, 2005; Dykema et al., 2010).  In light of these findings, method biases, therefore, can have pervasive effects on various elements of OHQoL measurement and adversely affect the outcomes of cross-cultural comparisons. But because method biases do not affect correlations, method bias is undetected by most statistical analyses and is almost never addressed in cultural adaptation or validation  27 studies (van de Vijiver & Poortinga, 2006). While it remains unknown whether the translated versions of the OHIP have standardized measurement methods, the authors have suggested that collective expert judgment could be used to detect potential biases in the instructions, response format, and other features of the scale. 2.3.2 Item Equivalence The quality of self-reports depends on how clearly and accurately the questions are posed, so language and culture must be considered carefully when translating a scale for different cultural groups (Ellis, 1989). Even slight differences in wording can change meaning and elicit different responses (McTavish, 1997). The various forms of item equivalence between source and target versions of the same scale have been consolidated into five categories: semantic equivalence, experiential equivalence, functional equivalence, scalar equivalence and conceptual equivalence (Guillemin et al., 1993; Hui, 2008). 1. Semantic equivalence means the similarity in linguistic meaning across cultures, and it can be attained by ensuring faithfulness to the original texts and cultural appropriateness for a target population through multiple forward-back translations and committee reviews (Canino & Bravo, 1994; Herdman et al., 1998). 2. Experiential equivalence refers to the extent to which experienced situations described by target items are equivalent to those of the source items (Canino & Bravo, 1994). 3. Functional equivalence suggests that target texts convey the meaning of the source texts in a way that preserves the purpose and functions of the original texts (Bullinger et al., 1993). 4. Conceptual equivalence implies that identical concepts have the same meaning across cultures (Usunier, 1998) and is achieved when questionnaires have the same relationship  28 with the underlying concept or theoretical dimensions of both cultures (Herdman et al., 1998). 5. Scalar equivalence means that the scale has same measurement unit across cultures (Canino & Bravo, 1994; Lee et al., 2009). As noted by Bartram (2011), scalar equivalence is difficult to achieve because score differences between cultural groups could be due to the true score variation as well as systematic scale-related biases. Therefore, unlike other types of item equivalence, scalar biases can be tested only by administering the scale and interviewing respondents in order to verify the same degree of construct indeed produce equivalent scores. Although scalar equivalence was beyond the scope of my content-validation, its implications in relation to my findings are examined in the Discussion.  In the literature, various solutions have been proposed to minimize the multiple types of item biases presented above, including the adoption of items without modifications, minor modifications to items, replacement of items, and elimination of items deemed inappropriate. Some problematic items would require re-translation or further verification from bicultural experts (Herdman et al., 1998). 2.3.3 Construct Equivalence The ultimate goal of cross-cultural adaptation is to establish construct equivalence, the degree to which the same construct is measured across populations of interest (van de Vijver & Tanzer, 1997; Herdman et al, 1998; Kristjansson et al., 2003). Potential threats to construct equivalence include biased sampling of behaviours relevant to the construct, incomplete representation of the construct, and a lack of construct overlap (Matsumoto & van de Vijver, 2011). For instance, the  29 biopsychosocial consequences of oral health can vary greatly among cultures depending on the significance given to oral health. It is also suggested that cultural climate can control the expression of social disability and the perception and tolerance of pain (Nayak et al., 2000). Allison and colleagues (1999) found substantial conceptual differences between Australian and Canadian judges in their perceptions of item severity. In more extreme cases, the construct of interest might not be comparable or directly transferable across cultures (Colress et al., 2001). For instance, despite their shared history and language, North and South Koreans have different cultures due to the partition of Korea, and the theoretical foundations underlying OHIP-14K may not respond the same way in the two countries. Therefore, apart from the linguistic factors, construct equivalence should be studied in a broader cultural context, taking into account cultural or sub-cultural factors such as social background, education, ethnicity, age, and gender. Duarte and Rossier (2008) suggested that the cultural approach to construct equivalence involves ensuring that the construct is both relevant to and observable in the target group before adjusting scales. In sum, Figure 3 illustrates various aspects of validity and equivalency for cross-cultural measurement and comparison of OHQoL. As can be seen from the diagram, I found that construct equivalence of the measurement is supported by evidence of both validity and equivalence. To find out what evidence there is to support construct equivalence of OHQoL measurement, I compared and contrasted a variety of methods used to adapt and validate the OHIP-14 in the next chapter.  30           Legend Equivalence Direction of Influence  Q1 Q2 Q3 Q4  English Version Instructions Response Formats Method Equivalence © Jaesung Seo 2010 Domain 1  Construct Validity Item Equivalence OHQoL Measured by Scale Score  OHQoL Measured by Scale Score Construct Equivalence Structural Equivalence Content Validity Cultural Adaptation Korean Version Instructions Response Formats Content Validity   OHQoL  OHQoL  Construct Validity Q1 Q2 Q3 Q4  Domain 2  Domain 1  Domain 2  Figure 3. Various aspects of validity and equivalency for cross-cultural measurement and comparison of OHQoL  31 Chapter  3: Systematic Synthesis of the Literature  3.1 Search Strategy 1) The systematic synthesis I used is considered to be primary research with a clearly defined research question: To what extent does the Korean version of the short-form Oral Health Impact Profile (OHIP-14K) exhibit content validity and cultural equivalency in Korean? Papers were selected based on their relevance to cultural adaptation and validation of OHIP. As the original English version of OHIP was developed in 1994 (Slade & Spencer, 1994), a literature search was performed to identify articles published from 1994 onward through Medline, EMBASE, and CINAHL, using the keywords ―OHIP or Oral Health Impact Profile,‖ ―cultural adaptation‖, and ―validation‖ in subject headings, titles, and abstracts. The examples of search coding schemes are presented in Appendix A.1. I included publications showing: (1) translation and cultural adaptation from the source English version of OHIP to various languages; (2) all types of validity evidence for OHIP obtained in different cultures; and (3) cross-cultural comparisons of responses using different language versions of OHIP. I excluded articles published in languages other than English and Korean along with others that did not describe the process of how the English version was adapted to the target culture. I read the titles and abstracts of the papers identified in this first step of the search for relevance to the second research question, but only the twenty two studies that were considered  relevant to cross-cultural measurement of the OHIP were read fully and analyzed critically (Figure 4).  32     Phase 1: Abstract Screening   Phase 2: Full Article Inspection     The search identified 19 papers describing various cultural adaptation and validation procedures for the OHIP. The most widely used methods were found to be forward and back-translations, pilot testing and post-hoc statistical analysis, usually following the pattern presented in Figure 5. Figure 4. Flow chart of the search process of accepting and rejecting papers MEDLINE (114 Citations) EMBASE (66 Citations) Total Number of Abstracts (180 Citations) Read full text (n= 22) Rejected (n= 158)  Rejected (n=3) - Duplicate study (n=1) - Unrelated to cultural adaptation (n=2) Studies meeting the inclusion criteria (n= 19)  33            Figure 5. Flow chart of cultural adaptation stages for OHIP Source English OHIP  Forward Translation  Back-Translation Committee Review  Pilot Testing  Target Language OHIP   34  In the following sections, I will explain each cultural adaptation method in detail with its relative strengths and weaknesses. 3.1.1  Forward Translation With a notable exception of the English version of OHIP-14 developed specifically for Scottish populations (Fernandes, et al., 2005), 18 other versions were forward-translated from the source. In conjunction with the forward translations of the original English version, the German (John et al., 2002) and Japanese (Yamazaki et al., 2007) versions were developed de novo from personal statements obtained from the respective populations to establish content validity. Forward translation, a process of translating from one language to another, is usually performed by one or more bilingual health professionals familiar with the content and goals of the study to ensure that the clinical meaning is appropriate (King et al., 2011; WHO, 2011). There are two main translational approaches in health research: literal and conceptual (WHO, 2011). Literal translations follow the original texts as closely as possible in tone, grammatical structure and idioms, but are not always semantically equivalent or in context. On the other hand, conceptual translations can be too liberal or lenient, which can weaken their precision and fidelity to the original texts. Therefore, forward translations alone do not necessarily establish equivalence between the source and target languages (Maneesriwongul & Dixon, 2004). Nor is forward translation a bias-free process because there are many words and phrases in one language that have no equivalent meaning in other languages, and their interpretation is at the discretion of the translator. As a result, a translation bias can arise from a number of factors including: bilingual proficiency and experience (Wild et al., 2005; Hambleton, 2006); a lack of training in the construction of psychometric measures (Hambleton & Patsula, 1998); and an  35 inadequate knowledge of OHQoL constructs and their underlying theories. For these reasons, many studies adopted a precautionary step to avoid translational errors by using multiple and independent translators. Examples of this approach include the multiple forward-translations used to create the Italian, Spanish, Dutch, and Brazilian versions of the OHIP by two or more translators working independently and using a mediator or an expert to resolve disagreements (Segu et al., 2005; Lopez & Baelum, 2006; Pires et al., 2006; Meulen et al, 2008). In contrast to single forward translations, multiple forward translations reduce the risk of biased translation caused by one person’s writing style, speech habit or misinterpretations (Wild et al., 2005). 3.1.2  Back-Translation The most frequently used judgmental procedure for detecting bias in forward translations is back-translation, whereby another bilingual translator independently translates the forward translation back into the source language (Guillemin et al., 1993). The back-translations are then compared to the original version for semantic and conceptual equivalence (Brislin, 1970; Hambleton, 2006). However, there is debate, over who should perform the back-translations. Van Winderfelt and colleagues (2005) recommend translators with knowledge about the subject of interest. In contrast, Guillemin (1995) argues that back-translations should be performed by translators who have no familiarity with the subject. Hence, it might be necessary to employ both types of back-translators. Even close similarities in language between the back-translation and the original does not necessarily produce cross-cultural equivalence because translators may share the same set of rules for translating non-equivalent words or compensate for poor forward translation by using a literal translation with no consideration for conceptual differences between the texts (Brislin, 1970; Stansfield, 2003). Therefore, to make a valid inference about the equivalence between the  36 source and target versions it is also necessary to know which type of back-translation was performed. Like forward translation, back-translations can be performed literally or conceptually. The former attempts to produce semantic equivalence while the latter aims to reflect the same construct in different cultures (Herdman et al., 1997). Wild and colleagues (2005) suggested that literal back-translations can be more suitable for medical surveys that are relatively objective as opposed to the more subjective QoL-related items, which require conceptual back-translations. However, unlike most QoL scales, most of the OHIP questions ask about oral health-related experiences (e.g., difficulty chewing), so a conceptual back-translation may not be sensitive enough to detect a potential misrepresentation of the original concepts in forward translations. For this reason, Sen and Mari (1986) recommend both literal and conceptual back-translations to ensure linguistic and conceptual equivalence between the source and target versions. 3.1.3  Committee Translation In contrast to a single forward translation, multiple translations can be performed by a committee of independent translators to detect an individual translator’s errors (Brislin, 1970; Beaton et al., 2002). Parallel and serial translations of the OHIP have been assessed in this way. In the parallel method, independent translators produce forward and back-translations in a parallel fashion, which are later reconciled into one final version by another translator (Brislin, 1970). Twelve out of 19 versions of OHIP were developed using this parallel method with at least two forward translations and two back-translations. The serial or multiple pass method uses a group of bilingual translators who take turns in increasing order of expertise to review each other’s translations and make necessary amendments (Gouadec, 2007). The authors of OHIP-14K opted for this method with a panel of  37 four Korean dentists who reviewed the forward translations in Korean. However, committees can be costly and time-consuming to form with three or more bilingual translators, and they do not necessarily operate without a shared bias or eliminate the potentially distorting influence of a hierarchy of power among members (Drennan et al., 1991; Maneesriwongul & Dixon, 2004). 3.1.4 Pilot Testing Pilot testing a provisional version of OHIP with a small subset of the monolingual participants give the participants an opportunity to judge and if necessary modify the scale. For instance, OHIP-14K was pilot tested with a convenience sample of 10 Korean adults, and, according to the authors, no comprehension difficulties were reported on the OHIP-14K questions. However, this will not necessarily establish the content validity or cross-cultural equivalency of a scale because of the limited advice expected from monolinguals. 3.1.5  Statistical Validation In general, post-hoc statistical analyses involve calculation of internal consistency and test-retest reliability. Internal consistency with Cronbach’s alpha (α) statistics evaluates the interrelatedness of items to the construct of interest, while test-retest reliability refers to the stability of scores obtained at two different times for the same population (Hubley & Zumbo, 1996). However, it is not always indicative of the quality of the scale because an extremely high alpha value (α > .90) can also indicate redundancy of items (Streiner, 2003) and low validity as a result of the scale being too narrow and specific (Kline, 1979). Streiner (2003) recommends that the maximum Cronbach’s alpha value not exceed 0.9. By the same measure, a very high alpha value coupled with high inter-item correlation of the Dutch version of OHIP has led to the conclusion that the five out of seven domains were redundant (Meulen et al, 2008).  38 Test-retest reliability refers to the consistency of psychometric measurements; however, it does not address the problems inherent to the scale because re-administering the same scale only reintroduces the same bias. Also, if the test-retest intervals are too far apart, it might measure changes in subjects’ physical state, severity perception or point of reference, rather than a real change in QoL. Changes in internal standards, values or conceptualizations, coping strategies, social comparison, social support, prioritization, expectations and spiritual practice can occur over time to confuse the results of the test-retest (Sprangers & Schwartz, 1999). These steps of cultural adaptation and validation used for the 19 language versions can be summarized in Table 3.   39  Table 3. The various methods used to translate and culturally adapt the OHIP  1 De novo method refers to the new development of a questionnaire according to the same method used for the original English version of OHIP (John et al., 2002). 2 The long-form OHIP, which was its original length, has 49 items (Slade & Spencer, 1994). 3 The short-form OHIP refers to OHIP-14, which has 14 items (Slade, 1997). Authors Language Version Forward & Back- Translation Pilot Study Committee/ Multiple Translation De novo Development1 Long Form2 Short Form3 Content Validity Wong et al., 2002  Chinese  O  O O X O   O X Meulen et al., 2008  Dutch  O   X O X O O X Allison et al., 1999  Canadian French O O O X O X  X John et al., 2002  German  O O O O O O O Papagiannopoulou et al., 2012 Greek  O O O X X O X Kushnir et al., 2004  Hebrew  O X X X X  O O Szentpetery et al., 2006  Hungarian  O O O X O X X Segu et al., 2005  Italian  O X O X X  O O Yamazaki et al., 2007  Japanese  O O O O O O O Bae et al., 2007  Korean  O O X X O O X Kenig & Nikolovska, 2012  Macedonian  O O O X O O X Saub et al., 2005  Malay  O O N/A  O O O O Pires et al., 2006  Portuguese  O O X X O O O Fernandes et al., 2005  Scottish English X X X X X  O X Stancic et al., 2009  Serbian  O O N/A  X X   O X Ekanayake & Perera, 2004  Sinhalese  O O X X X  O X Lopez & Baelum, 2006  Spanish  O X  O X O  X  X Montero-Martin et al., 2009 Spanish  O O O X X  O   X Larsson et al., 2004  Swedish  O X O X O O   X  40 It is evident from the systematic synthesis findings that the current cultural adaptation and validation strategies do not fully address the content validity and cultural equivalency of the OHIP-14. 3.2 Content Validity First, cross-cultural measurement of OHQoL must demonstrate content validity for the target population. As described earlier, content validity means the relevance and representativeness of scale content (e.g., items or questions) to the construct of interest and its appropriateness for the target population (Beck & Gable, 2001). Whilst a variety of sources of validity evidence is needed for scale validation (American Educational Research Association, American Psychological Association, and National Council on Measurement in Education, 1999), many OHQoL studies have neglected efforts to examine the theoretical aspect of cross-cultural measures on the assumption that the known psychometric properties of an originally validated scale, such as content and construct validity, will remain constant across cultures through cultural adaptation.  Nevertheless, content validity is not a fixed property of a scale and requires ongoing evaluation within the context of multiple sources of evidence (Sireci, 1998, 2007). In some cases, the judgments on content validity made by the originators are not generalizable to cultures outside the mainstream Western culture in which they were made (Rogler, 1999). For instance, linguistic equivalence may not guarantee content validity for another culture. To illustrate this point, consider the following case: when Russia’s Supervisory Natural Resources team toured Korea to see mountain animals, a Korean official told the Russian representative that ―Korea is very interested in Siberian tigers‖ (The Chosunilbo, 2009). The translation of this statement was misunderstood by the Russian delegates as a request for a tiger, so the zoo in Seoul received  41 three Siberian tigers. In this example, the literal translation from Korean to Russian did not make it clear that the Korean were simply admiring the animals and not asking that they receive tigers as a gift. In this light, content validity of a scale is only relevant to the intended culture (Hayness et al., 1995); therefore, linguistic and cultural modifications might be necessary to preserve the scale’s content validity (Debb, 2007). The content validity of the OHIP-14 is especially vulnerable due to the small number of items measured (Locker & Allen, 2002), which can adversely affect the construct validity of its measurements (Haynes et al., 1995; Awad et al., 2008). Thus, further examination is necessary to fully evaluate the content validity of OHIP-14K. 3.3 Cross-cultural Equivalence In order for OHIP to be a viable cross-cultural scale, it must demonstrate not only validity for the Korean population but also cultural equivalence with the original English version. The latter means semantic, colloquial, experiential, functional, and conceptual equivalence between the target and source versions (Guillemin, 1995) for unbiased measurement of the construct between different cultural groups (Cella et al., 1996). The current practices of cultural adaptation and validation are criticized for relying heavily on back-translation, reviews by lay panels, and on statistical techniques, whilst ignoring the conceptual equivalence of the scales (Herdman et al., 1998; King et al., 2011). Hence, the existing methods do not eliminate or control the cross-cultural biases that can adversely affect OHQoL measurements. The consequences can have far-reaching impacts on construct validity and equivalence. Addressing the comparability or cultural equivalence of a scale has been a great challenge for comparative health researchers (Liang, 2001; Ramirez et al., 2005) because no single approach is infallible to  42 the cross-cultural biases that pose threats to measurement validity in self-reported data (Hambleton & Patsula, 1998). The method proposed by Allison and colleagues compares the relative unpleasantness of each impact judged by different cultural groups (1999). The authors found the consistency in the severity (or oral impact) rankings of the OHIP items by Australian and Anglophone- and Francophone-Canadian judges and concluded that OHIP is a cross-culturally valid tool. For instance, difficulty chewing was judged to be more unpleasant than the loss of sense of taste across the three cultural groups. However, the comparison of item impact alone may not yield adequate information on the linguistic or conceptual equivalence of items across cultures for cross-cultural comparison of OHQoL. In fact, linguistic and cultural biases have previously been reported in various versions of the OHIP. When the question, ―have you been self-conscious because of problems with your teeth, mouth or dentures?‖ was translated into Macedonian, the term self-consciousness had a very different meaning than in English, and half of the respondents could not answer the question (Kenig & Nikolovska, 2012). A similar problem with the meaning of self-consciousness was encountered in the Italian version of the OHIP (Segu et al., 2005). Another method used for achieving conceptual and linguistic equivalence between two different-language versions of a scale is to administer both versions to a representative sample of bilingual respondents. For instance, strong positive correlations of scores obtained from Arabic, Spanish (Hunt, 1986), French (Hunt et al, 1991), and English versions of the Nottingham Health Profile were seen as evidence of equivalence. In a different study, positive Pearson’s correlations in the total scores of the Greek and English versions of the General Health Questionnaire have been used to support the conclusion that the Greek translation was  43 accurate and the two versions were equivalent (Garyfallos et al., 1991). However, when discrepancies in the scores arose – as in the Chinese version of the Menstrual Distress Questionnaire – the traditional comparative bilingual approach did not offer effective ways to improve the poorly adapted items or mediate disagreement between experts (Chang et al., 1999), and the causes of score differences is unknown. In another study by Son and colleagues (2000), significantly higher means were noted for two items on the Korean version of the Caregiving Satisfaction Scale when compared to the original English version: ―caring for my elder helps keep her/him from getting sicker than she/he otherwise would‖ and ―caring for my elder has taught me to distinguish the important things in life from the not so important‖. The authors faced similar challenges when it came to explaining the score differences and modifying the items to improve cross-cultural equivalence. There is a need for a structured method to first evaluate the psychometric properties of a translated scale and then to detect and derive solutions to potential biases for meaningful cross- cultural measurement and comparison. The health and educational research literature suggests that content-oriented validation has been used to establish content equivalence of alternate forms of various tests (Ding & Hershberger, 2002; Martin et al., 2010) and between multi- lingual versions of various scales, including the Quebec User Evaluation of Satisfaction with Assistive Technology Questionnaire (Brandt, 2005), Quality of Life Questionnaire (Cestari et al., 2006), Safety Attitudes Questionnaire (Devriendt et al., 2012), Diabetes Empowerment Scale (Shiu et al., 2003), Tobacco Acquisition Questionnaire and Decisional Balance Scale (Chen et al., 2003), Help Questionnaire (Kochinda, 2011), and Adolescent Teasing Scale (Liu, 2011). Although it has never been attempted with an OHQoL scale, many cross-cultural researchers have suggested the possibility of submitting both source and target versions for evaluation by  44 bilingual subject matter experts (SMEs) to revise or remove items that are found to be non- equivalent (Geisinger, 1994; Chang et al., 1999; Beaton et al., 2002; Sireci et al., 2006). For instance, Kleiner and colleagues (2009) employed Chinese, Spanish, and French bilingual survey researchers to evaluate on a 7-point Likert scale the translation quality of the respective language versions of the National Survey of Children with Special Care Needs. The authors noted that a thorough assessment of the translation quality was necessary to enhance the naturalness of expression and ensure that the message is recreated with respect to modes of behaviour that are meaningful within the context of the target culture. In summary, I can conclude from this literature review that a different validation approach is needed for assessing content validity and cross-cultural equivalency of OHIP-14K. I now present the empirical design I used to answer my third research question: To what extent does OHIP-14K exhibit content validity and cultural equivalency?  45 Chapter  4: Second Component – Empirical Exercise Methods To assess the content validity and cultural equivalency of the OHIP-14K, I performed a content validation study based on quantitative judgments of bilingual SMEs who are familiar with both Korean and North American cultures. The significance of establishing the content validity and cultural equivalency of the OHIP-14K centres on certifying that (1) the instruction and items can be accurately understood by Korean respondents in their native language; (2) the OHIP-14K items are relevant to the hypothesized domains of OHQoL as explained by Locker’s model of oral health; and (3) the OHIP-14K is linguistically and culturally equivalent to the original English version. In order to measure these properties of OHIP-14K, I adopted CVIs techniques suggested by Lynn (1985). Content validity assessment is generally divided into scale development and quantification phases. In my study, the development stage was omitted because OHIP-14K was adapted directly from the English OHIP-14, obviating the need for domain identification, item generation, and scale formation. I obtained directly from Bae and colleagues the version of OHIP-14K used for their validation study (2007). I carried out the quantification phase closely following the content validation procedures described by Crocker and Algina (1986): 1. Literature review of the psychological domains of OHQoL and its structured framework (as I already have presented throughout the first component of my dissertation, above); 2. Content validation exercise via selection of qualified SMEs; 3. Administration of CV questionnaires; 4. The data collection and evaluation of CVI for the OHIP-14K.  46 The last three steps are discussed as part of the empirical component of my dissertation. The following series of methods I employed are about the development, administration and analysis of Content Validity Questionnaires. First, however, I will present the selection process for the SMEs. 4.1 Selection of Bilingual Subject Matter Experts The selection of experts should be based on careful consideration of their level of expertise and knowledge in the focal area of the research (Dubois & Dubois, 2000). Accordingly, I selected SMEs who: (a) Had a dental licence for practice in Canada and practised general dentistry at least three times a week for a minimum of 10 years; (b) Knew about of oral problems and their impact on patients’ lives through relevant clinical experience; (c) Understood both the Korean and the main-stream English population of Greater Vancouver Area; (d) Used both English and Korean in their dental practice, but whose native language was Korean and who had resided mostly in Canada for the past 5 years; (e) Were willing to participate in this study. The authors of the OHIP-14K and anyone who had previous exposure to the OHIP were excluded from this study to maintain neutrality and to minimize potential confirmatory bias. The selection of recruited SMEs usually ranges from 3 to 10 experts for studies of content validation (Lynn, 1986; Sireci, 1998; Yaghmale, 2003; Hubley & Palepu, 2007; Domingues et al., 2011), but it also considers personal factors including expertise, scope of practice, and other practical considerations, such as the availability of qualified experts, the range of  47 expertise represented, and completion or drop-out rates (Hayness et al., 1995). Although there is no optimal number of SMEs for a content validation study, Gable (1986) suggested that a minimum of 10 judges was required to yield acceptably consistent responses and to avoid chance agreement. Lynn noted that the maximum number is unlikely to exceed 10 experts (1986). Considering the brevity of OHIP-14K and the limited number of eligible SMEs to whom I had access, I aimed to recruit 10 SMEs. A convenience sample of 20 eligible SMEs was identified through the 2009 College of Dental Surgeons of British Columbia directory by seeking dentists with Korean names. Initial contact was made by phone or email followed by a personal visit to their clinical practices. Of the 20 SMEs identified in Greater Vancouver Area, 17 were contacted, as the other 3 could not be reached at their listed addresses. All potential SMEs received personalized cover letters detailing the research aim and rationale as well as the theoretical basis of Locker’s model (Appendix D.2). From these 17 potential recruits, 11 volunteered to participate while the others refused to participate because of their busy schedule. One of the selected SMEs eventually withdrew from the study because she felt that her Korean was not proficient enough for the content validation exercises. Therefore, I had 10 dentists on the SME panel, with one of them working in academia as well (Table 4).  48  Table 4. Background information of the subject matter experts SME Gender Place(s) of Graduation Clinical Experience (yrs) Education Background Location 1 M University of British Columbia; University of Temple; University of California, San Francisco 15 DDS, PhD, AEGD a  Burnaby, BC 2 M Seoul National University; University of British Columbia 26 DDS/MD Burnaby, BC 3 M Undisclosed 30 DMD Coquitlam, BC 4 M Yonsei University; University of Manitoba; University of British Columbia 19 DMD, MSc, PhD Burnaby, BC 5 F Southwestern University 14 DMD North Vancouver, BC 6 M University of London; Seoul National University; Korea University 41 BDS, DMD, MSc, PhD Vancouver, BC/ Seoul, Korea 7 F University of British Columbia 10 BDS, DDS Burnaby, BC 8 M University of Manitoba 15 DDS North Vancouver, BC/ Lancaster, CA 9 F University of Pennsylvania; Columbia University 12 DDS, MA Coquitlam, BC 10 M Korea University; University of British Columbia 26 DDS (SNU), DMD (UBC), MSD, PhD, PPD Burnaby, BC   a  Advanced Education in General Dentistry  49  4.2 Development of Questionnaire on Content Validity Content validation involving SMEs begins with the construction and administration of a questionnaire for content validation (CV) to gather quantitative and qualitative information on the clarity, cultural equivalency and relevance of the scale (Lynn, 1986; Haynes et al., 1995; Grant & Davis, 1997). I designed a questionnaire with three comprehensive subscales for SMEs to judge: 1) the content validity of OHIP-14K in terms of the technical quality of the items, instructions, and response formats; 2) the relevance of the content to the theoretical domains of Locker’s model; and 3) the cultural equivalency of the translation to the intent of the original OHIP-14 (Appendix D.2). Figure 4 illustrates the scope of each Content Validity subscale with respect to the main stages of the development of OHIP-14K.  On the recommendation of Lynn (1986), the CV questionnaire acquired a 4-point ordinal Likert scale as a response format without neutral ground.         3. RELEVANCE SUBSCALE EVALUATES RELEVANCE OF OHIP-14K CONTENT AGAINST THE THEORETICAL DIMENSIONS OF LOCKER’S MODEL.        2. EQUIVALENCE SUBSCALE: CONFIRMING CROSS-CULTURAL EQUIVALENCE BETWEEN THE ENGLISH AND KOREAN VERSIONS OF OHIP    1. CLARITY SUBSCALE CONFIRMS READABILITY OF OHIP-14K OHIP-14K  OHIP-14E  Locker’s Model  Figure 4. Stages of OHIP-14K development and proposed subscale indices of the content validity questionnaire  50 4.2.1  Clarity Index In this part of the CV questionnaire, the SMEs were asked to rate the clarity of the instructions, items, and response format on the following Likert scale: 0 = not at all clear, 1 = somewhat clear, 2 = mostly clear, and 3 = very clear (Kalton & Schuman, 1982; Ferketich, 1991). In the textboxes provided they were instructed to identify comprehension problems they experienced due to vague wording, ambiguous language, complex sentences, double-barrelled questions, dental jargons, and cognitively challenging or culturally inappropriate questions; and to add a comment to explain their response or suggest changes if necessary. 4.2.2 Cultural Equivalence Index Cultural equivalence implies that the measurement of OHQoL across the two cultures is unbiased. Because all versions of the OHIP, including OHIP-K, are derived from the original in English (Slade & Spencer, 1997), the validity of comparisons between any two cultures primarily depends on the faithfulness and appropriateness of the translation from the source English version (Gagnon & Tuck, 2004). In other words, the original English version is the basis for evaluating the cross-cultural equivalency of the OHIP-14K. In my study, the original OHIP-14 and the OHIP-14K were collated using expert judgments to examine the cross-cultural equivalence of the two versions. The SMEs were asked to evaluate the semantic, colloquial, experiential and conceptual equivalence, rather than linguistic or literal equivalence of the translation using the scale: 0 = not at all equivalent, 1 = somewhat equivalent, 2 = mostly equivalent, and 3 = equivalent. They were encouraged also to comment generally on the translation with suggestions for deletions, additions and modifications (Appendix D.2).  51 4.2.3 Relevance Index Relevance was determined using expert ratings on the relevance index, which gauged the degree to which the OHIP-14K items appropriately sample the theoretical domains of OHIP. As suggested by John and colleagues (2005), the bilingual SMEs were asked to identify for each item the most appropriate theoretical domain of Locker’s model: functional limitation, physical pain, psychological discomfort, physical disability, psychological disability, social disability or handicap. The chosen response method on the relevance index was a multiple- choice question derived from the Q-Sort and inter-rater agreement methods. In Q-sort selection method, experts are asked to place index cards containing items into the most appropriate categories (Block, 1961; Waltz & Bausell, 1991). This method has been widely used to refine items and verify that developed items would correctly load onto expected domains at a pretesting stage (Vaughn & Waters, 1990; Moore & Benbasat, 1991; Bhattacherjee, 2002; Nahm et al., 2002; van Ijzendoorn et al., 2004), which in turn forms the basis of construct validity and improves the reliability of the constructs (Rajesh et al., 2010). However, the Q- sort method I used was modified because the original method would have required the SMEs’ presence during the course of the validation procedure (Tojib & Sugianto, 2006). To overcome this limitation, I combined the Q-Sort with inter-observer agreement to create a multiple- choice response format for measuring the proportion of SMEs who correctly classified an item to its expected domain (Thorn & Deitz, 1989). If a SME felt that an item did not describe any of the seven domains provided, they were asked to suggest a more appropriate domain in the provided textbox. Finally, the SMEs were asked to write their comments and overall opinions of OHIP-14K in open-ended textboxes.   52 4.3 Administration of Content Validation Questionnaire and Data Collection Prior to its administration, the CV questionnaire was pilot tested for clarity by one of the 10 SMEs who recommended that the WHO’s definitions of impairment, disability, and handicap be appended to the questionnaire as background information for the other participants. All information in the cover letters, including the various types of cultural equivalence (e.g., semantic equivalence), were verbally conveyed to the SMEs. After this introductory briefing, the SMEs signed the informed consent form (Appendix D.1) and received a CV questionnaire with an instruction manual and the OHIP-14 in both Korean and English (Appendix D.2). They answered the CVI questionnaires individually, independently, and at their own convenience. I returned within 30 days to collect the questionnaire. During this 30-day period, I telephoned each SME every two weeks to prompt their response, and when I finally collected the completed questionnaire, I had a brief discussion with each of them about their opinions on the process and offered them an honorarium of $50 for their participation.  4.4 Data Analysis The Likert responses were transferred to and analyzed using Microsoft Office Excel®. Each scale was inspected for any missing responses, and the mean score for a missing response was imputed for the mean score for that scale element if the missing data did not exceed 60% and items with missing data did not exceed 15% of the total number of items (Fox-Wasylyshyn & El-Masri, 2005; Meulen et al., 2008; Sanders et al., 2009). The clarity of the OHIP-14K was rated for relevance and equivalence both globally for the entire scale (S-CVI) and individually for the instructions, response format, event frequency, and items (I-CVI).  53 4.4.1 Calculating the Item-level Content Validation Index (I-CVI) The I-CVIs were calculated as the proportion of SMEs who endorsed the validity of each scale (i.e., gave ratings of 3 or 4 on the 4-point Likert scale) (Beck & Gable, 2001). Calculating the Average Deviation Mean Index ADM is an index of disagreement with the ability to measure multirater disagreement in the scale’s units (Burke & Dunlap, 2002) and to uncover hidden disagreement in dichotomous data (i.e., content valid vs. content invalid). ADM for the clarity and cultural equivalence indices was calculated as the sum of the differences between individual ratings and the mean in absolute values divided by the total number of ratings. A lower ADM value indicated stronger agreement between SME-ratings because the ADM is dispersed around the mean. The detailed statistical formulas for calculating ADM for an item and a scale are provided in Appendices C.1 and C.2. Two cut-off values of ADM were used to interpret agreement and disagreement between SMEs. The minimum acceptable level of disagreement (ADM) was calculated as c/6, where c is the number of categories (i.e., 4), which was 0.67 for the 4-point Likert scales (Lynn, 1986). An ADM value of 0.67 or higher indicated a significant level of disagreement among SMEs (Figure 5). Any scale element with an ADM greater than this cut-off value required further inquiry into the opinions and explanations of the SMEs to identify alternative questions or format. On the other hand, a critical ADM value of 0.56 or lower was required for the 5% level of statistically significant agreement (achieving the 5% level of statistical significance) for a study involving 10 experts and 4 response categories (Burket & Dunlap, 2002). ADM values  54 below this cut-off value indicate that agreement was unlikely to have been achieved by chance, and it was necessary to accept the SMEs’ feedbacks and suggestions for making revisions. ADM values between 0.56 and 0.67 indicated that neither significant disagreement nor agreement had been reached, and that no further action was required.  Instrument Element   ADM > 0.67 (High Disagreement)   CVI < 0.80 (Content Invalid)   CVI > 0.80 (Content Valid)   ADM ≤ 0.56 (Significant Agreement)   ADM < 0.67 (High Disagreement)   ADM ≤ 0.56 (Significant Agreement)   No action required Further analysis required Document possible alternative terms & phrases Accommodate the SMEs’ feedback and suggestions Figure 5. Possible outcomes of content validation study of OHIP-14K  55 Multirater Kappa The analysis of nominal or categorical data generated by the relevance index in which the most representative theoretical domain was selected for each item requires non-parametric Kappa statistics (Slocumb & Cole, 1991). The minimally acceptable Kappa cut-off value was 0.4 (Okochi et al., 2005; Gorelick et al., 2008). Kappa values below this cut-off were considered to be poor agreement (Fleiss, 1971; Cicchetti, 1984) and prompted further examination of the SMEs’ comments. Kappa values at and above 0.40 indicated moderate agreement. Detailed calculations and interpretation of free-margin Kfree are provided in Appendices C.2 and C.3. 4.4.2 Scale-Content Validity Index The Scale-level CVI, expressed as the percentage of items whose I-CVI values were equal to or greater than the minimally acceptable CVI of 0.78, provided information on the proportion of elements requiring revisions until the S-CVIs was equal to or greater than 0.80 to confirm the validity of the scale (Grant & Davis, 1997).  56  Chapter  5: Results  5.1 Clarity of OHIP-14K Elements With the exception of the response format, all 16 elements (including items and instructions) had I-CVI values greater than the critical value of 0.80, indicating adequate levels of clarity (Table 5). The deviation from the mean index (ADM) across all elements was below the critical value of 0.56, suggesting that homogeneity in SME ratings was unlikely to have been achieved by chance. For the most part, the OHIP-14K was unequivocally judged to be clear (S-CVI = 0.93, ADM ≤ 0.56). The only scale element whose CVI value fell below the acceptable level was the response format (I-CVI = 0.7). The comments from three SMEs suggested that the response format was vague. They recommended changing the Korean words ―maewoo‖ (very) to ―maewoo jaju‖ (very often), and ―guhee‖ (hardly) to ―guhee junhyu‖ (hardly ever). In addition, three other SMEs commented that the Korean translation of the OHIP-14 asks about symptoms that overlap significantly with systemic illnesses such as depression, so it could potentially have diverse interpretations. They recommended that the specific oral health context be expressed in every question by including the phrase ―because of problems of your mouth, teeth or dentures.‖  57  Table 5. Item-level content Validity Index (I-CVI) and Average Deviation from the Mean Index (ADM) of OHIP- 14K instruction, response format, event frequency, and items in the Clarity and Equivalence Indices  a  Two missing values from two questionnaires (SME 7 and 8) were imputed with the item mean value for that scale element.  Clarity Index 1  Equivalence Index 2  Original Scale Element (back-translation of OHIP14-K) I-CVI  ADM I-CVI  ADM Instruction 0.9 0.38 0.3 0.55 Response Format 0.7 0.34 0.8 0.63 Event Frequencya  1.0 0.40 0.6 0.54 1. Trouble pronouncing any words (Discomfort from not being able to pronounce well) 1.0 0.40 0.6 0.96 2. Sense of taste has worsened (Sense of taste was worse than before) 1.0 0.38 0.7 0.64 3. Painful aching in your mouth (Pain in the tongue, sublingual, cheeks, palate, etc.) 1.0 0.40 0.4 0.70 4. Uncomfortable to eat any foods (Uncomfortable to have meal due to painful or uneasy problems of the mouth) 1.0 0.39 0.7 0.60 5. Self-conscious (Reluctant to meet others because of shame) 0.9 0.38 0.4 0.56 6. Felt tense (Paid attention to) 0.9 0.38 0.6 0.54 7.  Unsatisfactory diet  (Dissatisfied meals) 1.0 0.40 0.9 0.48 8. Meals interrupted  (Interrupted during meals) 1.0 0.40 0.8 0.36 9. Difficult to relax  (Difficulties resting comfortably) 1.0 0.39 0.8 0.36 10. A bit embarrassed  (Embarrassed or perplexed) 0.9 0.39 0.9 0.36 11. A bit irritable with other people  (Get angry easily at others) 0.9 0.37 0.5 0.70 12. Difficulty doing your usual jobs  (Difficult to do normal jobs) 0.9 0.37 0.9 0.56  58  Note: (1) Clarity Index S-CVI = 0.93           (2) Equivalence Index S-CVI = 0.50   5.2 Cultural Equivalence between OHIP-14 and OHIP-14K The SMEs were instructed to evaluate the cross-cultural equivalence between the OHIP-14E and OHIP-14K. Seven elements – which included the response format as well as items 7, 8, 9, 10, 12, and 13 – were deemed content valid (CVI > .80), all with acceptable agreement (ADM ≤ 0.56). On the contrary, the instructions, the event frequency, and items 1, 2, 3, 4, 5, 6, 11, and 14 fell below the minimally acceptable CVI value. Of these scale elements, the instructions, event frequencies, and items 5 and 6 showed statistically significant agreement (I- CVI < 0.80, ADM ≤ 0.56). While neither significant agreement nor disagreement was observed for items 2 and 4, a high level of disagreement was noted for items 1, 3, 11, and 14 with regard to cultural equivalency. - The SMEs were divided over the cultural equivalency of the Korean translation of item 1 trouble pronouncing words (ADM = 0.96). Four SMEs suggested that having trouble pronouncing words has a different meaning than the Korean phrase discomfort from not being able to pronounce well. The suggested revisions for item 1 included ―baleumeul mothaeshinjuk‖ (could not pronounce [any words]) and ―baleumeul mothaesuh himdeushinjuk‖ (having difficulties to pronounce [any words]). 13.  Life in general was less satisfying (Life less satisfying than before) 0.9 0.37 0.8 0.54 14.  Totally unable to function (Psychologically, physically and socially cannot at all do one’s share) 0.8 0.36 0.6 0.72  59 - In addition, 5 SMEs pointed out that the expression ―venting anger at others‖ (item 11) in Korean is not equivalent to the English ―a bit of irritability with other people.‖ Two SMEs offered an alternative expression, ―yakganeu jjajeungeul jal naenjeok,‖ meaning ―showing a little bit of irritability.‖ - Lastly, the experts disagreed over the equivalency of item 14 (ADM = 0.72), ―totally unable to function,‖ which was translated into ―jungshinjeok, shinchejeok, sahoejeokeuro junhyuh jemokeul halsu upsutdun jeok‖ (totally unable to do one’s share psychologically, physically, and socially).  5.3 Relevance of OHIP-14K Domains Table 6 shows the distribution of responses to the question: ―Which one of the domains best represents each item?‖ and a table of descriptive statistics including each item’s CVI value, Kfree, and levels of agreement.     60   Table 6. The SMEs’ classification of items into the seven OHIP domains, Item-level Content Validity Index (I-CVI), Free-Marginal Kappa (Kfree), and levels of agreement on the Relevance Index   Numbers of SMEs who endorsed each domain as the most representative for that item  Descriptive Statistics Item Functional Limitation Physical Pain Psychological Discomfort Physical Disability Psychological Disability Social Disability Handicap  I-CVI Kfree Agreement Q1 8 - - - 1 1 -  0.80 0.56 Moderate Q2 9 - - - 1 - -  0.90 0.77 Substantial Q3 - 10 - - - - -  1.00 1.00 Perfect Q4 6 - - 3 1 - -  0.00 0.30 Fair Q5 - - 5 - - 5 -  0.50 0.35 Fair Q6 - - 10 - - - -  1.00 1.00 Perfect Q7 - - 4 4 2 - -  0.40 0.17 Poor Q8 2 - 2 6 - - -  0.60 0.27 Fair Q9 - - 2 8 - - -  0.00 0.59 Moderate Q10 - - 5 - 3 2 -  0.30 0.19 Poor Q11 - - 1 - 1 8 -  0.80 0.56 Moderate Q12 2 - - 6 - - 2  0.00 0.20 Poor Q13 - - 4 - 6 - -  0.00 0.38 Fair Q14 - - - - 1 - 9  0.90 0.77 Substantial Note: Shaded cells indicate the theoretical domains predetermined by Slade and Spencer (1994).  Examination of each item’s relevance to the expected theoretical domain revealed that the endorsement rates for items 1, 2, 3, 6, 11, and 14 were equal to or above the acceptable CVI  61 value of 0.80 (S-CVIrelevance = 0.42, Kfree > 0.4), confirming that the questions correctly corresponded with the domains hypothesized by the Locker’s model. On the other hand, eight out of 14 items (numbers 4, 5, 7, 8, 9, 10, 12, and 13) fell below the acceptable CVI value, with 5 showing poor or fair agreement (Kfree < 0.4). Closer examination of content-invalid items revealed that item 5, ―self-conscious‖ (psychological discomfort); item 9, ―difficult to relax‖ (psychological disability); and item 13, ―life in general was less satisfying‖ (handicap) were also representing social disability (Kfree = 0.4), physical disability (Kfree = 0.6), and psychological disability (Kfree = 0.4), respectively. Poor agreement was noted for items 7, 10, and 12 (Kfree < 0.2) while items 4 and 8 showed fair agreement (0.2 < Kfree< 0.4). Overall, 7 SMEs indicated that some questions could be interpreted outside of the oral health context. The SMEs recommended specific examples of painful areas (e.g., mouth) and clarifying the oral health context by including a phrase as in the English version: ―because of problems with your mouth, teeth or dentures.‖ For example, item 12, ―difficulties in doing one’s usual jobs,‖ may unintentionally elicit non-dental-related experiences.  62 5.4 Recommended Revisions or Deletions A number of potential sources of biases emerged from the content validation results, along with comments and suggested revisions. The potential biases that achieved acceptable agreement are described below within the framework of equivalence theory: methods, items and construct. 5.4.1 Methods Method biases associated with discrepancies in the instructions and response formats between the OHIP-14E and OHIP-14K were as follows.  OHIP-14E Instruction How often have you had the problem during the last year? OHIP-14K Instruction 지난 3개월 동안 치아나 잇몸 때문에 아래의 경험을 얼마나 자주 겪으셨습니까? (In past 3 months, have you had the experiences below because of teeth or gums?).  OHIP-14E Response Format Very often, fairly often, occasionally, hardly ever, never. OHIP-14K Response Format 매우: 1주일에 2–3회 이상, 자주: 1주일에 1회 정도, 가끔: 한달에 2–3회 정도, 거의: 한달에 1회 이하, 전혀: 경험한 적이 없음  (Very: 2–3 times a week; often: once a week; sometimes: 2–3 times a month; almost: once a month; never: never experienced). According to one SME, the instructions may pose a bias due to the difference in their recall periods (1 year versus 3 months). Another SME noted that a ―don’t know‖ response option is not offered in OHIP-14K.   63 5.4.2 Items  Semantic bias – in item 6, the Korean translation of the word tense has a different meaning than the English word.  OHIP-14E Item 6 Have you felt tense because of problems with your teeth, mouth or dentures? OHIP-14K Item 6 ―Shingyungee mani sseuin jeokee itseupnikka?‖  (Have you paid a lot of attention [to the problem]?). Suggested Revision ―Chia, gugangina, teulni taimunae kinjangee doeshinjeokee itseupnikka?‖ (Have you been tense because of problems with your teeth, mouth or dentures?).   Functional Bias – although item 9 was perceived to be cross-culturally equivalent (CVIequivalence = 0.8, ADM = 0.36), 8 of the SMEs recognized resting comfortably as a physical disability instead of placing it in the intended psychological disability domain (CVIrelevance = 0.0, Kfree = 0.59). One SME suggested a revision, replacing the item with ―kinjangeul pulji mothashinjeoki‖ to describe the inability to ease tension.  OHIP-14E Item 9 Have you found it difficult to relax because of problems with your teeth, mouth or dentures? OHIP-14K Item 9 ―Pyeonanhage swiji mothashinjeoki itseupnikka?‖ (Have you had difficulties resting comfortably?). Suggested revision ―Chia, gugangina, teulni taimunae kinjangeul pulji mothashinjeoki itseupnikka?‖ (Have you been unable to ease tension because of your teeth, mouth or dentures?).  64  Conceptual bias – OHIP-14K item 5 (I-CVIequivalence = 0.4), which asks about psychological discomfort, is different than its English counterpart both in meaning as well as the domain covered. This finding is further supported by half of the SMEs classifying item 5 as a social disability (I-CVIrelevance = 0.5) instead of placing it in the intended psychological discomfort domain.  OHIP-14E Item 5 Have you been self-conscious because of your teeth, mouth or dentures? OHIP-14K Item 5 ―Changpihaeseo dareun sarameul mannagiga kkeoryeojishin jeoki itseupnikka?‖ (Have you been reluctant to meet others because of embarrassment?). Suggested revision ―Chia, gugangina, teulni taimunae shingyeongee mani sseuishinjeoki itseupnikka?‖ (Have you paid attention [to the problem] because of your teeth, mouth or dentures?).  5.4.3 Construct  Construct bias – the seemingly transferrable construct of OHQoL can be understood differently in English and Korean cultures due to differences in priorities, health perceptions and potential impact of a disorder. For example, one SME indicated that OHIP-14K does not adequately capture the aesthetic concerns that patients might have about their mouths. In this regard, item 22 of the long form OHIP-49 (appearance unsatisfied) might be more relevant to Koreans (Bae et al., 2007). In their overall evaluation of OHIP-14K, 6 SMEs recognized the need for better accuracy of the OHIP- 14K items to ensure cross-cultural equivalence.   65 Chapter  6: Discussion The availability of cross-culturally valid and reliable OHQoL measures is beneficial to Korean populations for needs assessment, oral health care planning treatment and outcome and service evaluation. Despite the widespread use of OHIP, a review of the literature revealed that content validity and equivalency of its translated versions – including the OHIP-14K – are rarely addressed. This was compounded by the fact that current cultural adaptation and validation strategies are not resistant to biases that can have serious ramifications to inferences made on cultural differences in OHQoL. Typical validation efforts for the OHIP use criterion- related approaches that are vulnerable to the cross-cultural biases and misunderstandings. However, they have paid little attention to the content of the scale or theoretical foundations. Consequently, the validity of the OHIP translated to other languages for use in cultural groups other than English-speakers in Western countries is suspect. This literature finding prompted an investigation of content validity and cultural equivalency of OHIP-14K in order to support the construct validity and equivalence of OHQoL measurements in Korean. I was also interested in applying the CVI techniques for detecting and addressing potential cross-cultural biases. My thesis began with a literature review to provide an overview of the contemporary conceptualization of OHQoL, validity, and cultural equivalence, followed by a literature synthesis of existing approaches to adapting and validating the OHIP. The second part of my study involved a content validation of OHIP-14K with bilingual Korean SMEs. A number of studies have looked at the content validity of translated versions of psychometric scales. The authors of the Dutch version of the Safety Attitudes Questionnaire used CVIs and Kappa to assess the equivalency with the English version (Devriendt et al., 2012), while Shiu  66 and colleagues (2003) used a similar method on the Chinese version of the Diabetes Empowerment Scale. Both studies concluded that the translations were equivalent to the original, although they did not use bilingual subjects to help with the assessments despite evidence suggesting that bilingual experts yield more effective translation (Guillemin, 1995; Harkness et al., 2003). In contrast to the traditional committee method of establishing equivalency, my study investigated the theoretical foundations of the OHIP-14K and recommended the structured methods for resolving potential biases. In the following 4 sections, I will discuss the method of content validation used, content validation findings, possible implications of my work and study limitations, while tying each of these to my literature review.  67  6.1 Content Validation Methods Content validation studies in nursing, education, and exercise physiology researches used different numbers of SMEs (Chen et al., 2003; Chien & Norman, 2004; Li & Lopez, 2004; Hubley & Palepu, 2007; Devriendt, 2012), but the CVI techniques and indices they used were similar to my method. Chen and colleagues (2003) used a 4-point Likert scales to measure the clarity of a Chinese translation of the Tobacco Acquisition Questionnaire and Decisional Balance Scale with results (S-CVI = 0.96) that were close to my S-CVI of 0.93. The cultural equivalence index, which I adapted from Lynn’s CVI (1986) for appraising the item-level equivalency of OHIP-14K, already existed as the Translation Validation Index, which was also derived from the CVI (Tang & Dixon, 2002). It has been applied also to four different questionnaires translated to Japanese (Kochinda, 2011) and a Chinese version of the Adolescent Teasing Scale (Liu, 2011). Like my study, they all used the same cut-off value of 0.80 for the cultural equivalence index. Similar to my results (CVIequivalence = 0.50), the Japanese versions of the Emotional Response Questionnaire (TVI = 0.46), the Style of Helping Scale (TVI = 0.73), and the Threat-Support Scale (TVI = 0.73) raised serious questions about the equivalency of the translated scales. On the other hand, the Japanese version of the Intensity of Help Scale (TVI = 0.80) and the Chinese version of the Adolescent Teasing Scale (TVI = 0.88) fared much better. Domingues and colleagues (2011) also used a 4-point scale to demonstrate the semantic- idiomatic, cultural and metabolic equivalence of the Brazilian translation of the Veterans Specific Activity Questionnaire (VSAQ). Unlike many other self-reported health measures, VSAQ is a clinical tool used to measure exercise levels and is not grounded in a theoretical model (Myers et al., 1994). The OHIP-14K, in contrast, is based on Locker’s model of oral  68 health to sample seven specific domains. Therefore, the relevance of the OHIP-14K to Locker’s model must be considered in any assessment of equivalency between different translations. In the literature, relevance indices were used by Chien and Norman (2004), who employed 15 bilingual health professionals to determine the appropriateness of the Chinese version of the Family Burden Interview Schedule in addressing the original dimensions of the English version (S-CVI = 0.96) (2004). In a similar study by Li and Lopez, 10 nursing experts evaluated the relevance of the Chinese version of the State Anxiety Scale for Children and also reported a very high score (S-CVI = 0.96) (2004). Both studies showed a substantially higher CVIs than that obtained in this study (S-CVIrelevance = 0.42), suggesting that the low values of relevance for OHIP-14K may not be related to the difficulties of the rating tasks. By comparing these different CVI methods, it is apparent that the indices of clarity, translation equivalence and relevance used in my study provides a thorough appraisal of the OHIP-14K. However, I employed these indices in a slightly different manner in terms of response format on the relevance index and the use of disagreement indices to control chance agreement, as I explain below. 6.1.1 Response Format on the Relevance Subscale The traditional relevance indices used by Chien and Norman (2004) and Li and Lopez (2004) measured how relevant an item is to its theoretical domain. Such an arrangement, however, can bias the SMEs’ judgments by exposing them to a predetermined item-domain association, and it could still remain unclear which dimensions the questions are addressing. For these reasons, the relevance index I used had a multiple-choice format derived from the Q-sort method and hybridized with inter-observer agreement. Slocumb and Cole (1991) demonstrated  69 the utility of this categorization method by which SMEs were given definitions of five theoretical domains (Universal, Competence, Availability, Resource, and Embracing) and asked to assign each item to a domain or identify it as independent of all of them. Rubio and colleagues (2003) referred to this method as factorial validity index (FVI) and used it to determine the extent to which experts correctly identify items with particular domain. The authors recommended a minimally acceptable FVI value of 0.8, which was consistent with my study design. However, the FVI is plagued also by chance agreement. 6.1.2 Disagreement Indices The traditional content validation studies were often criticized for running the risk of possible chance agreement (Cohen, 1960; Polit & Beck, 2006; Tojib & Sugianto, 2006). Apparently, a proportion of agreement indices can both over- and under- estimate agreement without a correction for coincidental agreement and disagreement (Burke et al., 1999). Lynn (1986) developed a minimally acceptable range of agreement by calculating a standard error of proportions with a 95% confidence interval around each I-CVI value. Consequently, I selected a minimum of 8 out of 10 SMEs required to establish content validity beyond the 5% significance level (I-CVI = 0.78).  Apart from the risks of chance agreement, the CVI alone cannot account for the loss of data as a result of collapsing 4-point Likert responses into a bivariate content-valid or -invalid choice. To overcome this limitation, I supplemented the CVI assessment with an additional analysis of ADM and multirater Kfree for better accuracy of and confidence in CVI measurement. ADM, a measure of the divergence of responses, has been used to supplement the CVIs of the Injection Drug User Quality of Life measure with a cut-off of 0.69 for acceptable disagreement (Hubley & Palepu 2007). The lower cut-off value (ADM = 0.56) used in my study meant that a lower  70 degree of disagreement in expert ratings was required to establish consensus on the CVI results.  On the other hand, I applied free-margin multirater Kfree instead of ADM to the data obtained from the relevance index and interpreted the Kfree values with parameters provided by Landis and Koch (1977). In contrast to the CVIs, Kappa is compared to the maximum possible value without chance agreement (Musch et al., 1984). Stemming from the classical test theory, the Kappa statistic was originally developed by Cohen (1960) to correct for chance agreement and disagreement between two judges. Fleiss (1971) extended the Kappa to allow for classification of the non-numerical variables by more than two judges. However, Fleiss’s Kappa was inappropriate for my study because it has fixed-marginal ratings with a predetermined number of items belonging to each domain. Instead, I used multirater Kfree to analyze data gathered from SMEs who were not restricted to assigning a certain number of items to each domain (Randolph, 2005). Wynd and colleagues (2003) used this combination of CVI and multirater Kfree to assess experts’ agreement on the content validity of the Atkins Osteoporosis Risk Assessment Tool (ORAT), which had an overall CVI of 0.65 and poor agreement (0.29 ≤ Kfree ≤ 0.48). Chung and colleagues (2007) also used the same method with similar cut-off values (I- CVI > 0.7, Kfree > 0.4) to examine the content validity of the Chinese version of the Integrative Medicine Attitude Questionnaire and found limited evidence of  content validity (S-CVI = 0.71) and expert agreement (Kfree = 0.09). Similarly, I found a low level of relevance for the OHIP-14K (S-CVIrelevance = 0.42) with varying degrees of agreement between SMEs (0.17 ≤ Kfree ≤ 1.00). Another notable difference between my study and other content validation studies is that the cultural adaptation of OHIP-14K was not guided by a translation theory or model. For example,  71 many cross-cultural studies in nursing adhere to a standardized set of procedures called the state-of-art method of translation and validation (Brack & Barona, 1991; Chien & Norman, 2004; Li & Lopez, 2004). There are also other translation models (Guillemin, 1995; Beaton et al., 2002) characterized by a synthesis of multiple independent translations so as to ensure the quality, faithfulness, and cultural appropriateness of an adaptation. However, my systematic synthesis provided insights into its guiding principles and allowed also for an appraisal of the methods. The results of my synthesis showed that cultural adaptation of OHIP is in general guided by a methodological norm in an ad-hoc fashion rather than by a specific model. Additionally, I found three different S-CVI calculation methods in the literature: universal agreement (S-CVI/UA) (Rubio et al., 2003); averaging method (S-CVI/Avg) (Polit & Beck, 2006); and proportional S-CVI (Lynn, 1986). S-CVI/UA represents the proportion of elements that achieved ratings of 3 or 4 by all the SMEs. Because unanimous agreement by multiple experts is difficult to achieve, Rubio and colleagues (2003) proposed averaging all I-CVIs on the index in order to calculate S-CVI/Avg. However, averaged I-CVI values do not give us any information about what percentage of the items are content valid. Therefore, I used the proportional S-CVI method put forward by Lynn (1986). In summary, my content validation methods were compared and contrasted with those found in the literature. Now I will turn to the discussion of my findings and their implications for cross-cultural measurement and comparison of OHQoL. 6.2 Discussion of Content Validation Findings Within CTT, the obtained CVIs can be interpreted as a combination of true scores and random errors in the experts’ ratings. Measurement errors in content validity are thought to have a mean of zero and do not correlated with either the true scores or with each other (Lord &  72 Novick, 1968). The true scores in the CVIs could be due to the following reasons (Murphy & De Shon, 2000): 1. Shared perceptions of content validity and cultural equivalency. 2. Shared goals and biases (i.e., rater influenced in the direction of leniency). 3. Shared frames of reference (similar standards or expectations). 4. Shared relationships (acculturation of bilingual SMEs).  The scale-level CVIs obtained in my study indicated that wording of the OHIP-14K is clear (S-CVIclarity = 0.93), but it is not cross-culturally equivalent to its English counterpart (S- CVIequivalence = 0.50). The results on the clarity index are expected, considering that OHIP-14K has already undergone rigorous pilot testing with monolingual Korean adults and five Korean dentists (Bae et al., 2007) despite its cultural equivalency never having been fully established.  Moreover, my results seemed to indicate a limited degree of relevance for the entire scale (S- CVIrelevance = 0.42). Cross-culturally valid scales must demonstrate evidence of item relevance and mutual exclusiveness of their theoretical dimensions to achieve proper representation of the construct (Smit et al., 2006). For this reason, many researchers recommend that each domain be represented by at least two content-valid items as a safety net to avoid taking chances with translation quality (Bice & Kalimo, 1971; Hinkin et al., 1992). In my study, only 7 out of 14 OHIP-14K items accurately loaded onto the expected dimensions. The low level of item relevance can imply a departure from the original theoretical model and an inaccurate representation of the construct since the subscales are not measuring what they are supposed to measure. This finding also raises questions about the use of subscale scores as a valid and reliable indicator of OHQoL domains.   73 Contrary to my expectations, a significant disagreement between SMEs was noted on the relevance and cultural equivalence indices. According to CTT, highly correlated ratings reflect the true score while observed disagreement is attributed to random measurement errors (Lord & Novick, 1968). In my study, however, the high levels of disagreement could also indicate a failure to achieve a shared understanding of the theoretical construct or dimensions as well as individual differences in interpretation of oral problems and oral health behaviours. In fact, Murphy and De Shon (2000) offered additional explanations for observed disagreement among SMEs, which might also have been the case in my study.  Potential causes of multirater disagreement 1. Systematic differences in their ratings of CVI and cultural equivalency. 2. Systematic differences in areas or levels of expertise or linguistic proficiency. 3. Systematic differences in access to information other than the scale provided (e.g., previous exposure to OHIP). 4. Individual differences in the interpretation of words or expressions with multiple meanings. 5. Individual differences in background, rating proficiency, clinical knowledge and experience, levels of expertise and education. Scale elements with high disagreement required further inquiry into the SMEs’ comments for possible explanations. Following Winderfelt’s suggestions (2005), the qualitative inputs from the SMEs for the content-invalid items with ADM ratings of 0.67 or higher were recorded to make alternative terms available for further validation studies. On the other hand, the scale elements meeting the minimally acceptable disagreement level (ADM ≤ 0.56 or Kfree > 0.40) but falling below the acceptable value of CVI (0.8) needed to be revised or eliminated according to the experts’ suggestions. Instead of eliminating items from  74 a scale like OHIP-14, which is already short, my study findings suggest revisions of the OHIP- 14K instructions, response format and frequency as well as items 5, 6, and 9. Suggested modifications of the OHIP-14K are presented in Table 7.  Table 7. Suggested revisions of OHIP-14K   As previously discussed, standardizing uniform scale features, such as recall periods, is necessary to minimize construct-irrelevant variance for making valid cross-cultural comparisons. In addition, the inclusion of the phrase ―because of problems with your teeth, mouth or dentures‖ for all items would specify the affected areas and help readers to focus on oral health-related events when answering the questions. Finally, the content validity findings were presented in English for non-Korean readers of my thesis using online translator tools for Scale Element OHIP-14K Suggested Revisions Instruction  3 months 1-year recall period Response Format ―maewoo‖ (very) and ―guhee‖ (hardly)  ―maewoo jaju‖ (very often) and ―guhee junhyu‖ (hardly ever) Q. 5 ―Changpihaeseo dareun sarameul mannagiga kkeoryeojishin jeoki itseupnikka?‖ (Have you been reluctant to meet others because of embarrassment?) ―Chia, gugangina, teulni taimunae shingyeonge mani sseuishinjeoki itseupnikka?‖ (Have you been self-conscious because of problems with your teeth, mouth or dentures?) Q. 6 ―Shingyungee mani sseuin jeoki itseupnikka?‖  (Have you paid a lot of attention [to the problem]?) ―Chia, gugangina, teulni taimunae kinjangee doeshinjeok itseupnikka?‖ (Have you been tense because of problems with your teeth, mouth or dentures?) Q. 9  ―Pyeonanhage swiji mothashinjeoki itseupnikka?‖ (Have you had difficulties resting comfortably?) ―Chia, gugangina, teulni taimunae kinjangeul pulji mothashinjeoki itseupnikka?‖ (Have you been unable to ease tension because of problems with your teeth, mouth or dentures?)  75 informational purposes only (Brondani & He, 2012). Back-translations of translated items and SME comments are commonly used for presenting content validation study results, as was done by Noh and colleagues when presenting their content validity evidence for the Korean version of the Center for Epidemiological Studies Depression Scale (1992). In the literature, various types of biases have been reported in other language versions of the OHIP. When item 3, ―have you had any painful spots or areas in your mouth?‖ was translated into Brazilian Portuguese, it used the Portuguese word ―pontos‖ for spot. However, the word also has a second meaning – suture stitches – which caused confusion. In Chinese, question 4 about ―meal interruption‖ was translated as ―stop eating during meals‖ (Wong et al., 2002). Although these findings were not replicated in my study, a similar translation problem to the one reported by Kenig and Nikolovska (2012) in the Macedonian version of the OHIP was detected in item 5, ―self-consciousness,‖ of OHIP-14K. In Macedonian, the literal translation of the term had a different meaning than intended whereas in the Korean version, the low degrees of translation equivalence (I-CVI = 0.4) and relevance (I-CVI = 0.5) suggest that the Korean translation may have been too liberal. A similar semantic problem was reported with item 5 in the Macedonian version. In this case, the authors decided to eliminate the item because most respondents did not understand its meaning (Segu et al., 2005). There are a number of possible explanations for the reported translation problems. As previously discussed, one possible cause of the observed asymmetry between source and target texts could be the disrupted balance between cultural appropriateness vs. faithfulness in the cultural adaptation. However, striking a balance between the two is a difficult process because a translator’s interpretation of the original concept will not be bias-free, especially without using the original theoretical model as a guiding principle in the cultural adaptation process. In a survey of translators, Harkness and colleagues (2003) found that in absence of adequate task  76 specifications, translators are forced to provide their own interpretations, which can bias the translation outcomes (1996). This subjective nature of cultural adaptation was also noted by van Ommeren and colleagues (1999), who found the translator’s judgment to be the determining factor in the success of cultural adaptations. Thus, the aim of the cultural adaptation must be clearly communicated to translators in order to carefully define the confines and fluidity of meaning as well as the range of interpretation to be considered (Wilss, 1996). To address issues with the personal biases of translators, Kleiner and colleagues (2009) provided them with instructional guidelines that would help them to produce more faithful and culturally appropriate translations of the National Survey of Children with Special Health Care Needs in Chinese, French and Spanish, and they found a resultant improvement in the quality of translation for some of these languages (2009). In a similar approach, I provided SMEs with the theoretical blueprint of OHIP (i.e., Locker’s model and domain definitions) to help them evaluate the OHIP-14K content against the theoretical model. Another source of asymmetry could have been the ambiguities in the source English version itself. As discussed previously, the scale’s ―handicap‖ domain included questions that were not developed from the interviews with the lay Australian respondents (Slade & Spencer, 1997). In the case of my study, none of the SMEs judged item 13, ―life less satisfying,‖ to represent the handicap domain. Moreover, high disagreement was noted for item 14, ―totally unable to function,‖ which was translated into ―jungshinjeok, shinchejeok, sahoejeokeuro junhyuh jemokeul halsu upsutdun jeok.‖ One SME pointed out that the Korean translation means a state of being incapacitated psychologically, physically and socially. Not only can such a triple- barrelled question be confusing for respondents, but it also gives a very vague impression of ―handicap,‖ for which no equivalent word exists in Korean. Another SME suggested replacing the wording with ―jaeneungreokeul mothan jeok‖ (unable to use one’s abilities).  77 Finally, there could have been conceptual differences across cultures in interpreting the semantically equivalent texts. My study found that seemingly equivalent items did not always guarantee their relevance to the expected theoretical domains. For example, although item 9, ―have you had difficulties relaxing?‖ was judged to be equivalent to its English counterpart (I- CVIequivalence = 0.8), it did not load onto the expected psychological disability dimension (I- CVIrelevance = 0.0). A similar translation issue was reported by Room and colleagues (1996), who noted that Korean translations of psychological affective states were easily mistaken for physical states because Korean words regarding feelings do not effectively differentiate between physical sensations and emotions. Other researchers also noted that semantically equivalent items are not always conceptually equivalent (Hunt, 1986; Guillemin et al., 1993). For instance, semantically equivalent translations of ―I have pain in my head‖ might vary in their significance and severity depending on the culture (Hunt, 1996). In relation to such conceptual and semantic discrepancies, Bice and Kalimo (1971) summarized 4 possible translation outcomes: 1) Identical items are semantically and conceptually equivalent across cultures. 2) Items that are conceptually but not semantically equivalent may still be used in comparative studies without changes (Bice & Kalimo, 1971) unless an alternative mode of expression is available (Hunt, 1986). 3) For culture-linked items that are semantically equivalent but measure different constructs in different settings, reconsideration of the meaningfulness of the responses or revisions may be necessary to establish construct equivalence (Bice & Kalimo, 1971; Hunt, 1986). 4) Non-related items that are neither conceptually nor semantically equivalent must be either re-translated or removed.  78 Using this typology, the functional (item 9) and construct (item 5) biases that I presented earlier can be classified as culture-linked and non-related items, respectively. Hence, revisions of these items will be critical to the validity of cross-cultural comparisons of OHQoL.   79 6.3 Implications of the Content Validity Findings My study found limited evidence of content validity for OHIP-14K in terms of item relevance and cultural equivalency with the source English version. This has further implications for the construct validity of OHQoL measurements in Korean populations using OHIP-14K because in the Unitarian framework, content validity of the scale is considered as the prerequisite for establishing the construct validity of the measurement. A scale with poor content validity can bias the empirical results and can even prompt the acceptance or rejection of a hypothesis or a theory (Rossiter, 2008). In line with previous research on the integrative framework of emic and etic perspectives (Tripp-Reimer, 1984; Phillips & Luna, 1996; Morris et al., 1999; Alegria et al., 2004; Cenoz & Todeva, 2009), the SMEs’ suggestions for improving the scale’s content validity underscored the importance of its cultural appropriateness and faithfulness to the source version and theoretical mode. This study finding suggests that OHIP-14K should have the same relationship with the construct of interest both within and across cultural groups. Therefore, a theoretical blueprint for the original scale might be useful as a practical baseline for ensuring semantic and conceptual equivalence in cultural adaptation and validation processes. In addition, the further provision of explanations of the scale’s concepts for translators or researchers might be necessary to avoid the conceptual bias caused by misinterpretation of items (Wild et al., 2005). Another potential implication of my study includes the impact of the detected biases on cross- cultural comparisons of OHQoL. The same degree of construct might elicit different responses on a Likert scale and consequently bias interpretation of domain and total scores (Anderson et al., 1996). This form of bias is called a scalar bias. For example, a physical disability subscale  80 could be measuring social disability domain instead. The fragile relationship between items and their theoretical domains carries significant implications as OHIP scores are calculated for each domain as well as for the entire scale (Locker et al., 2005; Brondani & MacEntee, 2007). However, Brondani and MacEntee (2007) raised a more fundamental problem with the use of a summative score, due to the questionable discreteness and stability of the theoretical domains of Locker’s model. Bakers (2007) also questioned whether or not the OHIP domains were distinguishable and if so, how they would relate to one another as per Locker’s conceptual framework. Brennan and Spencer (2004) provided further empirical evidence in support of the existence of conceptual domains using factor analysis, and the 7 domains of Locker’s model were confirmed by John using expert judgment (2007). Therefore, in regard to the aim of the content validity study, the 7 theoretical domains are integral components of Locker’s model that provide a foundation for specifying and operationalizing the construct of OHQoL. Based on the feedback from the 10 SMEs, a refined version of OHIP-14K was yielded to enhance both semantic and conceptual equivalence (Table 8). However, more work is needed to test the new measure and evaluate its psychometric properties in the target Korean populations (see 7.2. Future Directions).   81  Table 8. Suggested revised version of OHIP-14K 지난 1 년동안 아래의 경험을 얼마나 자주 겪으셨습니까?  매우 자주 자주 가끔 거의 전혀 전혀 1. 치아, 구강이나 틀늬 때문에 발음이 잘 안되어 불편했던 적이 있으십니까? 2. 치아, 구강이나 틀늬 때문에 맛을 느끼는 감각이 예전보다 나빠졌다고 느끼신 적이 있습니까? 3. 혀밑, 빰 입천정 등이 아픈 적이 있습니까?  4. 치아, 구강이나 틀늬 때문에 아프거나 거북스러운 입안의 문제 때문에 음식 먹기가 불편한 적이 있습니까? 5. 치아, 구강이나 틀늬 때문에 신경이 많이 쓰인적이 있습니까?  6. 치아, 구강이나 틀늬 때문에긴장이 되신적이 있습니까?  7. 치아, 구강이나 틀늬 때문에식생활이 불만스러운 적이 있습니까?  8. 치아, 구강이나 틀늬 때문에 식사를 도중에 중단하신 적이 있습니까? 9. 치아, 구강이나 틀늬 때문에 긴장을 풀지 못하신 적이 있습니까?  10. 치아, 구강이나 틀늬 때문에 난처하거나 당황스러웠던 적이 있습니까? 11. 치아, 구강이나 틀늬 때문에 다른 사람들에게 화를 잘 내게 되신 적이 있습니까? 12. 치아, 구강이나 틀늬 때문에 평소 하시던 일을 하기가 어려웠던 적이 있습니까? 13. 치아, 구강이나 틀늬 때문에 살아가는 것이 예전에 비해서 덜 만족스럽다고 느끼신 적이 있습니까? 14. 치아, 구강이나 틀늬 때문에 정신적 신체적 사회적으로 전혀 제 몫을 할 수 없었던 적이 있습니까?    82 6.4 Study Limitations Despite content validity and cultural equivalency being crucial evidence of construct validity for the OHIP-14K, it is important to keep in mind that the findings of my study are based on expert judgment rather than empirical evidence obtained from administering the scale to subject pools. My study was meant to be an adjunct to the mounting evidence gained in the investigation of scale validity from various sources, including the empirical validation of OHIP-14K conducted by Bae and Colleagues (2007). Nonetheless, the findings of the study have three main caveats. The first caveat is related to the literature review and study design. As there was limited literature on content validity assessment for scales measuring OHQoL, the scope of the literature search was expanded to include literature in multiple fields including education, social sciences, business and medicine. Given the breadth of literature and variety of topics covered, the manual literature search might have not been efficient in identifying previous studies that had adopted similar designs under different names while I was formulating my research questions and developing a content validation method. In retrospect, the duplication of my efforts could have been avoided if I had taken a more systematic approach to the literature review. Furthermore, as with any other study design that uses expert judgment, my content validity study was subjective in nature. Although an individual SME’s error or bias could be addressed by group decisions, collective judgment is not infallible either and may mask disagreement between individual experts. Also, the convenience sample used in my study consisted only of dentists. Multi-disciplinary SMEs could have provided a more diverse range of knowledge and experience (Domingues, 2011; King et al., 2011). Moreover, expert judgment did not allow for further examination of the latent relationships in multiple variables (John, 2007), such as theoretical domains and how they relate to each other.  83 Another issue pertinent to relying on the expert judgement of bilingual dentists is their bilingual proficiency and the effects of acculturation on the generalizability of their findings. Despite the stringent expert selection criteria, it could not be confirmed whether every SME met the levels of proficiency in both languages and familiarity with both cultures required for the content validation tasks. An individual expert’s linguistic competence and cultural knowledge could potentially have a huge impact on the study results concerning the equivalence and relevance of OHIP-14K. In anticipation of similar problems, I attempted to attenuate the impact of individual biases on the study results by employing percent agreement and disagreement indices. Apart from the individual biases, the bilingual experts may have had differences in their backgrounds and linguistic processing from the average monolingual Korean layperson. Hence, there is a need to examine how target respondents interpret the scale content since poor understanding of the instructions and questions can affect the accuracy of recall and self- reporting of past experiences (Linn et al., 1991). The second caveat concerns the particular method of content validation used in my study. While privacy and anonymity in data are the strengths of CVIs for encouraging honest feedback and minimizing the influence of bias and peer pressure from other experts, they are also considered weaknesses in that they limit cooperation between SMEs and minimize accountability for their individual judgments (Goodman, 1987). Researchers are divided over whether anonymous or non-anonymous conditions would provide more accurate ratings. Leek et al. (2011) found that collaboration between referees produced a more accurate peer-review process, but there are also studies that support anonymous evaluation because accountable ratings may suffer a significant level of inflation (Afonso et al., 2005; Daberkow et al., 2005).  84 More empirical evidence is needed to determine which settings are more appropriate for a content validity assessment. The other potential drawback of the chosen CVI analysis is related to the use of Kappa statistics for analyzing the data collected on the relevance index. Kappa statistics did not permit examination of the interrelationship between conceptual dimensions; it is designed to analyze nominal data of no natural ordering even though in this case the dimensions were sequentially related, as hypothesized by Locker’s model and ICIDH. For example, one can infer from Locker’s model (Figure 2) that impairment would be more closely related to functional limitation than to physical disability. The third caveat of my study design pertains to the limitations inherent in the original OHIP and Locker’s model themselves. Critics of OHIP are sceptical of its ability to adequately represent OHQoL in the intended populations, let alone non-English-speaking cultures. For instance, it cannot factor in the positive or the many personal and environmental factors of oral health, including diet, economic priorities, personal expectations, health values and beliefs (Bakers, 2007; Brondani et al., 2007). Moreover, it remains unclear whether or not the ICIDH- based Locker’s model can accommodate cross-cultural differences in perceptions of disability and handicap (MacEntee et al., 1997). Therefore, it is safe to assume that culturally adapted versions of OHIP are fraught with similar limitations to the original English version. Another limitation of Locker’s model is its sole focus on oral diseases. Consequently, the scope of Locker’s model is limited to the interactions between diseases and negative consequences, leaving no room for coping and adapting abilities or individual reactions to the same disease.  85 Despite these limitations and shortcomings, validating an existing scale required an appraisal in accordance with the original theoretical framework for which it was developed. Consequently, my data did not include the representativeness of OHIP-14K or its degree of comprehensiveness in the item pool. Polit and Beck (2006) noted a similar limitation in their study, where they concluded that CVI ratings did not certify the inclusion of a comprehensive set of items to represent the construct of interest. Likewise, one SME commented that the OHIP-14K items did not adequately capture aesthetic aspect of oral health, which, according to the same SME’s opinion, was very significant to Korean older adults. Meanwhile, there are alternative frameworks of oral health that could provide a more comprehensive and culturally universal foundation for OHQoL research in cross-cultural settings. ICF is an international classification system that holds the biopsychosocial view of health in which individual selves and social environments are shaped by the constant dynamics between them (Bury, 1982). Unlike its predecessor, ICF accounts for positive experiences and beliefs about health as well as the influence of subjective and environmental factors. For instance, ICF emphasizes the positive components of health, such as activity at the individual level and participation in society, and further states that personal factors (including gender, age, coping styles, social background, education, profession and lifestyle) interact with environmental factors (such as climate, cultural values, social attitude and structure) to affect bodily functions and structures/impairments, activity/limitations and participation/restriction (WHO, 2002). The interactions between various health dimensions are denoted by bidirectional arrows arranged in a non-linear fashion (Figure 6) and can be contrasted to the unidirectional progression presented by the ICIDH in Figure 2. This self-directed yet complex view of health in ICF is crucial for understanding QoL because it can account for QoL’s positive spectrum and the use of coping and adapting strategies for various oral conditions  86 (Brondani & MacEntee, 2007), which help fight the stigma associated with disability (MacEntee, 2006). Moreover, ICF domains are said to be culturally appropriate and universally applicable because the main principles of ICF include respecting different languages and cultures as well as conceptualizing functioning, health and disability on the basis of consensus from various perspectives, including health care, education, social policy, economics, insurance, and labour (Peterson, 2012). Figure 6.  The International Classification of Functioning, Disability and Health (ICF) (WHO, 2010b). On the other hand, the major criticism of ICF stems from the fact that it is too complex and impractical to apply in daily practice (Üstün et al., 2010). It can be very difficult to operationalize for scale development (Ostir et al., 2006) and insensitive to cultural differences (Ingstad & Reynolds, 1995) and subjective dimensions (Ueda & Okawa, 2003). Health Condition (Disorder or disease) Activity (Limitations) Participation (Restrictions) Body Functions & Structures (impairments) Environmental factors Personal factors Contextual factors  87 Nonetheless, there are new, preeminent models of oral health that have emerged from the ICF classification. For example, the existential model of oral health, put forward by MacEntee (2006) (Figure 6) and later refined by Brondani, Bryant, and MacEntee (2007) (Figure 7), is designed to reflect the biopsychosocial aspects of oral health in various contexts associated with personal and socio-cultural factors. The existential model expands on the language of the ICF and provides a more positive conceptualization of the various newly identified environmental and individual factors pertaining to oral health, including expectation, diet, economic priorities, health values and beliefs and influences of personal and social environments (Brondani et al., 2007). As can be seen in Figure 7, the key components of the refined model of oral health were empirically based and included a broader scope of experiences and oral health beliefs from older adults, with the limitation that its empirical basis was mostly Caucasians living independently in Vancouver, BC.   88 Figure 6. An existential model of oral health (MacEntee, 2006) [Reproduced with the permission of the editor of Gerodontology].  89  Figure 7. The atomic model of oral health (Brondani et al., 2007) [Reproduced with the permission of the editor of Gerodontology]        90 Chapter  7: Conclusion The content validation method I used allows me to conclude that - The OHIP-14K demonstrated limited evidence of content validity and cultural equivalency, and potential cross-cultural biases have been identified in its method, items, and construct representation. - Further refinement of the instructions, response format, and items 5, 6, and 9 are necessary to improve the validity and equivalence of OHQoL measurement in Korean. - The CVI technique is an effective tool for evaluating content validity and equivalency of a translated version of the OHIP for culturally specific and general considerations; detecting content biases in the early stages of cultural adaptation within budgetary realities to save cost, effort and time; and documenting the content validation process and quantification of CVIs and disagreement indices for future studies.   91  7.1 Study Implications My study results suggest that current cultural adaptation and validation strategies for OHIP may not warrant adequate content validity or cultural equivalency for making valid and reliable cross-cultural comparisons of OHQoL. The main implication for cross-cultural measurement is the biased representation of the construct across cultures; therefore, caution is needed when comparing the OHIP scores cross-culturally. The differences in the OHIP score between cultural groups could be attributed to practical cultural differences in OHQoL, but also to construct-irrelevant variations, such as discrepancies in the method, items or construct. Rigor is also needed for validating a translated version of a scale, much as it is used for developing a new scale, to ensure the compatibility and comparability of OHQoL measurement outcomes. Standardization of scale content and methods is critical to the validity of cross-cultural comparisons of psychometric constructs (King et al., 2011). The CVIs in conjunction with disagreement indices were useful tools for detecting and deriving solutions to various types of biases and for mediating disagreements between the experts. The expert suggestions and the CVI ratings could be used also to improve the content validity and equivalency of the OHIP-14K, as well as to refine inferences made from cross-cultural measurements of OHQoL.  92  7.2 Future Directions It remains critical to consult lay Koreans to examine the relevance and utility of the OHIP-14K revisions suggested in this study and the implications of Locker’s theory of oral health for the target setting and purpose. Such social implications of the scores have yet to be considered, and further assessment of the refined version presented here should be undertaken through personal interviews or focus groups in a Korean population. In fact, WHO (2010a) specifically recommends conducting cognitive interviews with members of a target culture for making final decisions on words or phrases that conform better to their language. Using the cognitive interviewing method, the items that reflected high levels of disagreement could be appraised and likely resolved through consensus between lay respondents. Alternatively, focus groups can be sought to promote a moderated group discussion about the content validity of the study findings. The main advantage of focus groups is interactions among participants that allow more in-depth discussion of events (Casey & Kreuger, 2000). In the literature, focus groups have been used to content-validate Malay and Chinese versions of OHIP (Wong et al., 2002; Shiu et al., 2003; Saub et al., 2005) and other OHQoL studies (Brondani et al., 2007; Brondani & MacEntee, 2007). Focus groups with Korean older adults can provide lay perspectives on the content validity of OHIP-14K as there is a need to confirm whether lay respondents understand the items the way they were intended and to verify respondents’ subjective perceptions and understandings of their oral health conditions (Brondani & MacEntee, 2007). Moreover, future studies should establish content validity and cultural equivalency of other language versions of OHIP-14 (or other OHQoL scales) and further explore the utility of CVIs and disagreement indices for enhancing cross-cultural OHQoL research paradigms, as there is a need for a continuous evaluation of the scale for the intended target populations in their  93 natural environment (Brondani & MacEntee, 2007). As can be deduced from the low level of translation equivalence between the English and Korean versions of OHIP-14, a comparison of two non-English versions might reveal similar problems, or perhaps even more so because current cultural adaptation and validation methods vary across the different-language versions of the OHIP. Hence, further inquiries into the equivalency of other language versions of OHIP would be necessary to ensure the validity of cross-cultural comparisons. Future content validation studies of OHQoL measures could also benefit from consulting dental hygienists or psychologists who can provide relevant information from different professional perspectives. Now that some of the potential cross-cultural biases in the OHIP-14K have been identified by SMEs, empirical investigations can be directed at substantiating or refuting the impact of the detected biases on self-reporting carried out by non-expert respondents. Both the original and refined OHIP-14K can be administered to a sample of bilingual Koreans to confirm my content validity findings by comparing scores or factors. Factor analysis is commonly used to describe correlations with unobserved factors in data by examining equivalence in the factor loading of individual items and the equality of factorial relations (Byrne & Campbell, 1999). For instance, factor analysis has been applied to test the cross-cultural invariance of psychometric properties in a multi-dimensional measurement of social support across African American, non-Latino white, Chinese and Latino groups, with the authors of the study attaining supporting evidence for the presence of common factors, unidimensionality of items within each domain and invariance of each factors and its item set (Wong et al., 2010). Factor analysis is yet to be widely adopted in the validation of OHQoL scales in cross-cultural settings since it can be too costly and laborious. Moreover, without the knowledge of some of the biases I have presented, the factor analytic approach alone could not account for what may have caused the cross-cultural differences.  94 Final Comments As an aspiring dentist and researcher, a parallel can be drawn between this learning and the very purpose of OHQoL measurement, which can be characterized as an effort to reconcile disagreement between patients’ perceptions and clinicians’ judgments of oral health. The self- report measures of OHQoL provide dental professionals with opportunities to listen to patients about health issues that extend far beyond what can be observed in clinics, and encourages them to work in partnership with their patients to protect their overall health. Although I have to yet enter a dental professional licensing body, I plan to put this holistic view of health into practice and use it to build a rapport with future patients. I also came to realize that it is not easy to reach consensus even between experts – for example, dentists. Although information gathered from the panel of experts was more insightful and comprehensive than individual efforts, collective agreement was difficult to reach in the group decision-making process. I will always keep in mind that disagreement is an inevitable part of human interaction and that I need to skilfully mediate the conflict in order to successfully serve the best interests of patients. Additionally, my graduate experience allowed me to develop creativity as well as interpersonal and problem-solving skills in the process of formulating and conducting my research. Following through the structured content-validation methods, I also learned to be aware of my own bias and stay open-minded in a scientific inquiry. During the literature review, I was overwhelmed by the vast amount of literature to be covered across different fields but came to appreciate the communality of multi-disciplinary studies and how seemingly different fields intersect, allowing for transfer of a method. Finally, I learned to accept the limitations of my own study and consider that the collection of all validity evidence as validation is not a  95 procedure but an ongoing process (Hubley & Zumbo, 1996). In this light, I shall bear in mind that validity claims of any test or measure should not be taken at face value and must undergo credible and careful scientific scrutiny. Outside the classroom, I have experienced tremendous personal growth in terms of settling down in a new city and managing my academic life with my family. Vancouver offered ample opportunities to observe geriatric oral care in multicultural settings and to interact with colleagues and faculty members devoted to this field. Furthermore, caring for my own aging parents with chronic illnesses helped me choose a career path in geriatrics in line with my research as I enter an undergraduate dental educational degree. The overall graduate experience has broadened my perspective on dentistry as part of the health care fabric in Canada and has encouraged me to become a geriatric dentist.  96 References Afonso, N. M., Cardozo, L. J., Mascarenhas, O. A. J., Aranhaand, A. N. F., & Shah, C. (2005). Are anonymous evaluations a better assessment of faculty teaching performance? A comparative analysis of open and anonymous evaluation processes. Family Medicine Journal, 37(1), 43-47. Alegria, M., Vila, D., Woo, M., Canino, G., Takeuchi, M. V., Febo, V., Guamaccia, P., Aguilar- Gaxiola, S. & Shrout, P. (2004). Cultural relevance and equivalence in the NLAAS instrument: Integrating etic and emic in the development of cross-cultural measures for a psychiatric epidemiology and services study of Latinos. International Journal of Methods in Psychiatric Research, 13(4), 270-288. Allen, P. F. (2003). Assessment of oral health related quality of life. Health Qual Life Outcomes, 1, 40-56. Allen, P. F. & Locker, D. (1997). Do weights really matter? An assessment using the oral health impact profile. Community Dental Health, 14, 133–138. Allison, P., Locker, D., Jokovic, A., & Slade, G. (1999). A cross-cultural study of oral health values. Journal of Dental Research, 78(2), 643-649. American Educational Research Association, American Psychological Association, and National Council on Measurement in Education. (1999). Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association.  97 American Psychological Association. (1954). Technical recommendations for psychological tests and diagnostic techniques. Washington, DC: American Psychological Association. American Psychological Association. (1974). Standards for Educational and Psychological Tests. Washington, DC: American Psychological Association. Anastasi, A. (1988). Psychological testing (6th ed.). (New York: Macmillan). Anderson, R. T., McFarlane, M., Naughton, M. J., & Shumaker, S. A. (1996). Conceptual issues and considerations in cross-cultural validation of generic health-related quality of life instruments. In B. Spilker (Ed.) Quality of life and Pharmacoeconomics in Clinical Trials, Second Edition (pp. 605–612). Philadelphia, PA: Lippincott-Raven. Atchison, K. A. (1997). The General Oral Health Assessment Index. In G.D. Slade (Ed.), Measuring Oral Health and Quality of Life (pp. 48-55). Chapel Hill, NC: Dental Ecology, University of North Carolina. Awad, M., Al-Shamrany, M., Locker, D., Allen, F., & Feine, J. (2008). Effect of reducing the number of items of the Oral Health Impact Profile on responsiveness, validity and reliability in edentulous populations. Community Dentistry & Oral Epidemiology, 36, 12-20. Bae, K. H., Kim, H. D., Jung, S. H., Park, D. Y., Kim, J. B., Paik, D. I., & Chung, S. C. (2007). Validation of the Korean version of the oral health impact profile among the Korean elderly. Community Dentistry and Oral Epidemiology, 35(1), 73-79. Bakers, S. R. (2007). Testing a Conceptual Model of Oral Health: a Structural Equation Modeling Approach. Journal of Dental Research, 86, 708-712.  98 Barnes, C. & Mercer, G. (1996). Introduction: Exploring the Divide. In B. Colin & Mercer (Eds), Exploring the Divide: Illness and Disability (pp. 11-16). Leeds: the Disability press. Bartram, D. (2011). Scalar Equivalence of OPQ32: Big Five Profiles of 31 Countries. Journal of Cross-Cultural Psychology. Retrieved from Beaton, D. E., Bombardier, C., Guillemin, F., & Ferraz, M. B. (2000). Guidelines for Process of Cross-Cultural Adaptation of Self-Report Measures. Spine, 25(24), 3186-3191. Beck, C. T., & Gable, R. K. (2001). Ensuring content validity: An illustration of the process. Journal of Nursing Measurement, 9(2), 201-215. Beckstead, J. W. (2009). Content validity is naught. International Journal of Nursing Studies, 46(9), 1274-1283. Beitz, J. M. & van Rijswijk, L. (1999). Using wound care algorithms: a content validation study. J Wound Ostomy Continence Nurs, 26(5), 238-249. Berkonovic, E. (1972). Lay conceptions of the sick role. Sot. Forces, 51, 53-63. Berry, J. W. (1990). Imposed etics, emits and derived emits: Their conceptual and operational status in cross-cultural psychology. In T. N. Headland & M. Harris (Eds.), Emits and etics: The insider/out. Gder debate (pp. 84-89). Newbury Park, CA: Sage. Bhattacherjee, A. (2002). Individual Trust in Online Firms: Scale Development and Initial Test. Journal of Management Information Systems, 19(1), 211-241.  99 Block, J. (1961). The Q-sort method in personality assessment and psychiatric research. Springfield, IL: Charles C. Thomas. Borrell-Carrió, F., Suchman, A. L., & Epstein, R. M. (2004). The biopsychosocial model 25 years later: principles, practice, and scientific inquiry. The Annals of Family Medicine, 2, 576-582. Bracken, B. A, & Barona, A. (1991). State of the art procedures for translating, validating and using psycho-educational tests in cross-cultural assessment. School Psychology International, 12, 119-132. Brandt, Å. (2005). Translation, cross-cultural adaptation, and content validation of the QUEST. Technology and Disability, 17: 205–216. Bravo, M., Canino, G. J., Rubio-Stipec, M., & Woobury-Farifia, M. (1991). A cross-cultural adaptation of a psychiatric epidemiology instrument: The Diagnostic Interview Schedule Adaptation in Puerto Rico. Culture, Medicine and Psychiatry, 15, 1-18. Brennan, D. S. & Spencer, A. J. (2004). Dimensions of oral health related quality of life measured by EQ-5D+ and OHIP-14. Health Quality of Life Outcomes, 2, 35-44. Brislin, R. W. (1970). Back-translation for cross-cultural research. Journal of Cross-cultural psychology, 1(3), 195-216. Brondani, M. A., Bryant, S. R., & MacEntee, M. I. (2007). Elders assessment of an evolving model of oral health. Gerodontology, 24, 189-195.  100 Brondani, M. A. & He, S. (2012). Translating Oral Health-Related Quality of Life Measures: Are There Alternative Methodologies? Translating quality of life measures. Social Indicators Research, 106(2), 1-15. Brondani, M. A. & MacEntee, M. I. (2007). The concept of validity in sociodental indicators and oral health-related quality-of-life measures. Community Dentistry Oral Epidemiology, 34, 472-478. Bullinger, M., Anderson, R., Cella, D., & Aaronson, N. (1993). Developing and evaluating cross-cultural instruments: From minimum requirements to optimal models. Qual Life Research, 2, 451-459. Burke, M. J. & Dunlap, W. P. (2002). Estimating Interrater Agreement with the Average Deviation Index: A User's Guide. Organizational Research Methods, 5, 150-172. Burke, M. J., Finkelstein, L. M., & Dusig, M. S. (1999). On average deviation indices for estimating interrater agreement. Organizational Research Methods, 2, 49-68. Bury, M. (1982). Chronic Illness as Biographical Disruption. Sociology of Health and Illness, 4, 167-182. Byrne, B. M., & Campbell, T. L. (1999). Cross-cultural comparisons and the presumption of equivalent measurement and theoretical structure: A look beneath the surface. Journal of Cross-Cultural Psychology, 30, 555-574. Byrne, B. M. & Watkins, D. (2003). The Issue of Measurement Invariance Revisited. Journal of Cross-Cultural Psychology, 34, 155-175.  101 Calabrese, J. M. & Friedman, P. K. (1999). Using the GOHAI to assess oral health status of frail homebound elders: reliability, sensitivity and specificity. Special Care Dentistry, 19, 214-219. Canino, G., & Bravo, M. (1994). The adaptation and testing of diagnostic and outcome measures for cross-cultural research. International Review of Psychiatry, 6, 281-286. Capra, F. (1982). The Turning Point. Science, Society and the Rising Culture. New York: Simon & Schuster. Casey, M. A., & Kreuger, R. A. (2000). Focus groups: A practical guide for applied research (3 rd  ed., pp. 112-145). Thousand Oaks, CA: Sage. Cella, D. F., Lloyd, S. R., & Wright, B. D. (1996). Cross-cultural instrument equating: Current research and future directions. In B. Spilker (Ed.), Quality of life and pharmacoeconomics in clinical trials (2nd ed., pp. 707-715). Philadelphia: Lippincott- Raven. Cenoz, J. & Todeva, E. (2009). The well and the bucket: the emic an etic perspectives combined In E. Todeva & J. Cenoz (eds.), The Multiple Realities of Multilingualism (pp. 265-292). Berlin: Mouton de Gruyter. Cestari, T. F., Balkrishann R., Weber, M. B., Prati, C., Menegon, D. B., Mazzotti, N. G., & Troian, C. (2006). Translation and cultural adaptation to Portuguese of a quality of life questionnaire for patients with melasma. Medicina Cutánea Ibero-Latino-Americana, 34(6), 270-274.  102 Cha, H. B. (2009). The 20 th IAGG World Congress in Seoul, 2013. The Journal of Nutrition, Health, and Aging, 13(1), 8. Chang, A. M., Chau, J. P., & Holroyd, E. (1999). Translation of questionnaires and issues of equivalence. Journal of Advanced Nursing, 29, 316-322. Chapman, D. W., & Carter, J. F. (1979). Translation procedures for the cross cultural use of measurement instruments. Educational Evaluation and Policy Analysis, 1(3), 71-76. Chen, H. S., Homer, S. D., & Percy, M. S. (2003). Cross cultural validation for the Stages of the Tobacco Acquisition Questionnaire and the Decisional Balance Scale. Research in Nursing & Health, 26, 233-243. Chien, W., & Norman, I. (2004). The validity and reliability of a Chinese version of the family burden interview schedule. Nursing research, 53(5), 314-322. Chosunilbo (2009, November 10). Translation Error Results in Russian Gift of Tigers. The Chosunilbo, Retrieved August 13, 2011, from Chung, V., Wong, E. & Griffth, S. (2007). Content validity of the integrative medicine attitude questionnaire: perspectives of a Hong Kong Chinese expert panel. Journal of Alternative Complement Medicine, 13(5), 563-70. Cicchetti, D. V. (1984). On a model for assessing the security of infantile attachment: Issues of observer reliability and validity. Behavioral and Brain Sciences, 7, 149-150.  103 City of Vancouver (2000). Building community: A framework for services for the Korean community in the lower mainland region of British Columbia. Vancouver, BC: Martin Spigelman Research Associates. Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20, 37–46. Cohen, L. K. & Jago, J. D. (1976). Toward the formulation of sociodental indicators. International Journal of Health Services: planning, administration, and evaluation, 6(4), 681-698. Cohen, Y. A. (ed.) (1974). Man in Adaptation: The Cultural Present (2nd  Ed). Chicago: Aldine. College of Dental Surgeons of British Columbia. (2009). Directory of Dentists. Vancouver: CDSBC. Conrad, P. (1990) Qualitative research on chronic illness: a commentary on method and conceptual development. Social Science & Medicine, 30, 1257-1263. Cook, L.L., & Schmitt-Cascallar, A. A. (2006). Establishing score comparability for tests given in different languages. In R.K. Hambleton et al. (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp.139-171). Hillsdale, NJ: Lawrence Erlbaum Associates. Corless, I. B., Nicholas, P. K., & Nokes, K. M. (2001). Issues in cross-cultural quality-of-life research. Journal of Nursing Scholarship, 33(1), 15-20.  104 Courtenay, B. C. & Weidemann, C. (1985). The effect of a "don't know" response on Palmore's Facts on Aging Quizzes. The Gerontological Society of America, 25 (2), 177-181. Crocker, L., & Algina, J. (1986). Introduction to classical and modern test theory. New York: Harcourt Brace Jovanovich College Publishers. Cushing, A. M., Sheiham, A., & Maizels, M. (1985). Developing socio-dental indicators- the social impact of dental disease. Community Dental Health, 3, 3-17. Cutress, T., Ainamo, J., Barmes, D., Beagrie, G., W, & Sardo-Infirri, J. (1982). Development of the World Health Organization (WHO) Community Periodontal Index of Treatment Needs (CPITN). International Dental Journal, 32, 281-291. Daberkow, D. W., Hilton, C., Sanders, C. V., & Chauvi, S. W. (2005). Faculty evaluations by medicine residents using known versus anonymous systems. Medical Education Online, 10(12), 1-9. Dean, H. T. (1942). The Investigation of Physiological Effects by the Epidemiological Method. In F.R. Moulton (Ed.), Fluorine in Dental Health (pp. 23-31). Washington, DC: American Association for the Advancement of Science. Demetriou, A., Shayer, M., & Efklides, A. (1992). Neo-Piagetian theories of cognitive development: Implications and applications for education. London: Routledge & Kegan Paul. DeVellis, R. F. (2003). Scale development: Theory and applications (2nd ed.). Thousand Oaks, CA: Sage.  105 Devriendt, E., Van den Heede, K., Coussement, J., Dejaeger, E., Surmont, K., Heylen, D., Schwendimann, R., Sexton, B., Wellens, N., Boonen, & S., Milisen, K. (2012). Content validity and internal consistency of the Dutch translation of the Safety Attitudes Questionnaire: An observational study. International Journal of Nursing Studies, 49(3), 327-337. Dolan T. A. (1993). Identification of appropriate outcomes for an aging population. Special Care Dentistry,13(1), 35-9. Domingues, G. B., Gallani, M. C., Gobatto, C. A., Miura, C. T., Rodrigues, R. C., & Myers, J. (2011). Cultural adaptation of an instrument to assess physical fitness in cardiac patients, Rev Saude Publica, 45(2), 276-285. Ding, C. S., & Hershberger, S. L. (2002). Assessing content validity and content validation using structural equation model. Structural Equation Modeling, 9(2), 283-293. Drennan, G., Levett, A., & Swartz, L. (1991). Hidden dimensions of power and resistance in the translation process: a South African study. Culture and Medical Psychiatry, 15(3), 361- 381. Duarte, M. E., & Rossier, J. (2008). Testing and assessment in an international context: Cross- and multi- cultural issues. In J. Athanasou & R. Van Esbroeck (Eds.), International handbook of career guidance (pp. 489-510). New York/Heidelberg: Springer Science. Dubois, D. A., & Dubois, C. L. Z. (2000). An alternate method for content-oriented test construction: An empirical evaluation. Journal of Business and Psychology, 15(2), 197- 213.  106 Dykema, J., Thayer-Hart, N., Elver, K., Schaeffer, N. C. & Stevenson, J. (2010). Survey Fundamentals: A Guide to Designing and Implementing Surveys. Office of Quality Improvement, University of Wisconsin Survey Center: Madison, WI, USA. Ekanayake, L. & Perera, I. (2004). Validation of a Sinhalese translation of the Oral Health Impact Profile-14 for use with older adults.  Gerondontology, 20(2), 95-99. Ellis, B. B. (1989). Differential item functioning: Implications for test translations. Journal of Applied Psychology, 74(6), 912-921. Engel, G. (1977). The need for a new medical model: a challenge for biomedicine. Science, 196 (4286), 129-136. Feinstein, A. R. (1987). The theory and evaluation of sensibility. In: Feinstein A.R. (Ed). Clinimetrics (pp. 141-166). New Haven: Yale University Press. Ferketich, S. (1991). Aspects of item analysis. Research in Nursing & Health, 14, 165-168. Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76, 378-382. Fox-Wasylyshyn, S. M., & El-Masri, M. M. (2005). Focus on Research Methods Handling Missing Data in Self-Report Measures. Research in Nursing & Health, 28, 488-495. Gagnon, A. & Tuck, J. (2004). A systematic review of questionnaires measuring the health of resettling refugee women. Health Care for Women International, 25(2) 111-149.  107 Garyfallos, G., Karastergiou, A., Adamopoulou, A., Moutzoukis, C., Alagiozidou, E., Mala, D., Garyfallos, A. (1991). Greek version of the General Health Questionnaire accuracy of translation and validity. Acta Psychiatr Scand, 84(4), 371-8. Geisinger K. F. (1994). Cross cultural normative assessment: translation and adaptation issues influencing normative interpretation of assessment instruments. Psychology Assessment, 6, 304-312. Gerrie, N. F. (1947). Standards of Dental Care in Public Health Programs for Children. American Journal of Public Health, 37, 1317-1321. Gift, H. C. (1997). Oral health outcomes research—challenges and opportunities. In G. D. Slade. (Ed.), Measuring oral health and quality of life. Chapel Hill: University of North Carolina Department of Dental Ecology. Gift, H. C., & Redford, M. (1992). Oral health and quality of life. Clinics in Geriatric Medicine, 8, 673-683. Gilljam, M., & Granberg, D. (1993). Should we take don't know for an answer? Public Opinion Quarterly, 57, 348-357. Goodman, C. M. (1987). The Delphi Technique: A Critic. Journal of Advanced Nursing, 12, 729-734. Gorelick, M. H., Atabaki, S. M., Hoyle, J. D., Dayan, P. S., Holmes, J. F., Holubkov, R., Monroe, D., Callahan, J. M., Kuppermann, & Percan, N. (2008). Interobserver agreement in assessment of clinical variables in children with blunt head trauma. Academic Emergency Medicine, 15, 812-818.  108 Gouadec, D. (2007). Translation as a Profession (pp. 105). Amsterdam/Philadelphia: John Benjamins Publishing Company. Grant, J. S., & Davis, L. L. (1997). Selection and use of content experts for instrument development. Research in Nursing and Health, 20(3), 269-274. Gregory, R. J. (2007). Psychological Testing: History, Principles, and Application, 5th ED, Boston: MA: Pearson Education. Grimes, D. A. & Schulz, K. F. (2002). Bias and causal associations in observational research. Lancet, 19, 359(9302), 248-250. Guillemin, F. (1995). Cross-cultural Adaptation and Validation of Health Status Measures. Scandinavian Journal of Rheumatology, 24, 61-63. Guillemin, F., Bombardier, C., & Beaton, D. (1993). Cross-cultural adaptation of health related quality of life measures: literature review and proposed guidelines. Journal of Clinical Epidemiology, 46, 1417-1432. Ha, J. E., Han, K. S., Kim, N. H., Jin, B. H., Kim, H. Y., Baek, D. I., & Bae, K. H. (2009). The improvement of oral health related quality of life by the national senile prosthetic restoration program. The Korean academy of dental health, 33(2), 227-235. Ha, J. E., Heo, Y. J., Jin, B. H., Paik, D. I. & Bae, K. H. (2012). The impact of the National Denture Service on oral health-related quality of life among poor elders. Journal of Oral Rehabilitation. doi: 10.1111/j.1365-2842.2012.02296.x  109 Hambleton, R. K. (2006). Issues, Designs, and Technical Guidelines for Adapting Tests Into Multiple Languages and Cultures. In R.K. Hambleton et al. (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp.39-64). Hillsdale, NJ: Lawrence Erlbaum Associates. Hambleton, R. K., & Patsula, L. (1998). Adapting tests for use in multiple languages and cultures. Social Indicators Research, 45(1), 151-171. Han, K. S. & Marvin, C. (2002). Multiple Creativities? Investigating Domain-Specificity of Creativity in Young Children. Gifted Child Quarterly, 46, 98-109. Harkness, J., van de Vijver, F. J. R. & Johnson, T. (2003). Questionnaire Design in Comparative Research. In J. Harkness, F. J. R. van de Vijver & P. Ph. Mohler (eds.), Cross-Cultural Survey Methods, New York: Wiley. Haynes, S. N., Richard D. C. S., & Kubany, E. S. (1995). Content Validity in Psychological Assessment: A Functional Approach to Concepts and Methods. Psychological Assessment, 7(3), 238-247. Hayran, O., Mumcu, G., Inanc, N., Ergun, T., & Direskeneli H. (2009). Assessment of minimal clinically important improvement by using Oral Health Impact Profile-14 in Behçet's disease. Clinical and Experimental Rheumatology, 27(53), 79-84. Hebling, E., & Pereira, A. C. (2007). Oral health-related quality of life: A critical appraisal of assessment tools used in elderly people. Gerodontology, 24(3), 151-161.  110 Helzer, J. E., Kraemer, H. C., & Krueger, R. F. (2008). Dimensional approaches in diagnostic classification – Refining the research agenda for DSM-V. Virginia: American Psychiatric Association. Herdman, M., Fox-Rushby, J. & Badia, X. (1997). 'Equivalence' and the translation and adaptation of health-related quality of life questionnaires. Quality of Life Research, 6(3), 237-247. Herdman, M., Fox-Rushby, J. & Badia, X. (1998). A Model of Equivalence in the Cultural Adaptation of HRQoL Instruments: the Universalist Approach. Quality of Life Research, 7(4), 323-335. Holbrook, A. (2008). Don't Knows (DKs). In P.J. Lavrakas. (Ed.), Encyclopedia of Survey Research Methods (Vol. 1, p. 208). Thousand Oaks: Sage Publications. Hubley, A. M. & Palepu, A. (2007). Injection Drug User Quality of Life (IDUQOL) Scale: Findings from a content validation study. Health and Quality of Life Outcomes, 5, 46-81. Hubley, A. M., & Zumbo, B. D. (1996). A dialectic on validity: Where we have been and where we are going. The Journal of General Psychology, 123, 207-215. Hui, C. H., & Triandis, H. C. (1985). Measurement in cross-cultural psychology. Journal of Cross-Cultural Psychology, 16, 131-152. Hunt, S. M. (1986). Cross-cultural issues in the use of  socio- medical  indicators. Health Policy, 6, 149-158.  111 Hunt, S. M., Alonso, J., & Bacquet, D. (1991). Cross-cultural adaptation of health measures. Health Policy, 19, 33-44. Hwang, K. K. & Yang, C. F. (2000). Introduction to special issue on indigenous, cultural, and cross-cultural psychologies. Asian Journal of Social Psychology, 3, 183. Ingstad, B. & Reynolds, W. (1995). Disability and Culture. Berkeley. University of California Press. Jeganathan, S., Batterham, M, Begley, K., Purnomo, J., & Houtzager, L. (2011). Predictors of oral health quality of life in HIV-1 infected patients attending routine care in Australia. Journal of Public Health Dentistry, 71(3), 248-251. Jenkins, J.  G. (1946). Validity for what?. Journal of Consulting Psychology, 10, 93-98. Jette, A. M. (1993). Using health-related quality of life measures in physical therapy outcomes research. Physical Therapy, 73, 528-537. John, M. T (2007). Exploring dimensions of oral health-related quality of life using experts' opinions. Quality of Life Research, 16, 697-704. John, M. T., Patrick, D. L., & Slade, G. D. (2002). The German version of the Oral Health Impact Profile- translation and psychometric properties. European Journal of Oral Science, 110(6), 425-433. Jones J. A., Kressin, N. R., Miller, D. R., Orner, M. B., Garcia, R. I., & Spiro, A. (2004). Comparison of patient-based oral health outcome measures. Quality of Life Research, 13, 975-985.  112 Jones, P. S., Lee, J. W., Phillips, L. R., Zhang, Z. E., & Jaceldo, K. B. (2001). An adaptation of Brislin’s translation model for crosscultural research. Nursing Research, 50, 300-304. Jung, S. H., Ryu, J. I., Tskos, G., & Sheiham, A. (2008). A Korean version of the Oral Impacts on Daily Performances (OIDP) scale in early populations: Validity, reliability, and prevalence. Health Quality Life Outcomes, 6, 17-25. Jung, Y. M., & Shin, D. S. (2008). Oral health, nutrition, and oral health-related quality of life among Korean older adults. Journal of Gerontological Nursing, 34(10), 28-35. Kalton, G. & Schuman, H. (1982). The effect of the question on survey responses: A review. Journal of the Royal Statistical Society, 145, 42-73. Kane, M. (2006). Content-related validity evidence in test development. In T. Haladyna & S. Downing (Eds.). Handbook of test development. Mahwah, NJ: Lawrence Erlbaum. Kawamura, M., Honkala, E., Widstrom, E., & Komabayashi, T. (2000). Cross-cultural differences of self-reported oral health behaviour in Japanese and Finnish dental students. International Dental Journal, 50(1), 46-50. Kawamura, M., Yip, H. K., Hu, D. Y., & Komabayashi, T. (2001). A cross-cultural comparison of dental health attitudes and behaviour among freshman dental students in Japan, Hong Kong and West China. International Dental Journal, 51(3), 159-163. Kelley, T. L. (1927). The Interpretation of Educational Measurement. Yonkers, NY: World Book company.  113 Kenig, N. & Nikolovska, J. (2012). Assessing the psychometric characteristics of the Macedonian version of the Oral Health Impact Profile questionnaire (OHIP-MAC49). Oral Health Dental Management, 11(1), 29-38. Kim, G. U. & Min, K. J. (2009). The impact of oral health impact profile (OHIP) level on degree of patients' knowledge about dental hygiene. The Korean Academic Industrial Society, 10(4), 717-721. Kim, H. Y., Jang, M. S., Chung, C. P., Paik, D. I., Park, Y. D., Patton, L. L., & Ku, Y. (2009). Chewing function impacts oral health-related quality of life among institutionalized and community-dwelling Korean elders. Community dentistry and oral epidemiology, 37(5), 468-476. Kim, J. H. & Min, K. J. (2008) Research about relationship between the Quality of life, oral health, and total health of adults. Korean Journal of Health Education and Promotion, 25(2), 31-46. Kim, J. O., & Mueller, C. W. (1978). Introduction to factor analysis: What it is and how to do it. Beverly Hills, CA: Sage. King, K. M., Khan, N., Leblanc, P. & Quan, H. (2011) Examining and establishing translational and conceptual equivalence of survey questionnaires for a multi-ethnic, multi-language study. Journal of Advanced Nursing, 67(10), 2267-2274. Kleiner, B., Pan, Y., & Bouic, J. (2009). The Impact of Instructions on Survey Translation: An Experimental Study. Survey Research Methods, 3(3). Retrieved January 10, 2012, from  114 Kline, P. (1979). Psychometrics and Psychology. London: Academic Press. Kline, P. (2000) A Psychometrics Primer. London, UK: Free Association Books. Kochinda, C. (2007). Translation Equivalence of Japanese version Help Questionnaire. Paper presented at Midwest Nursing Research Society conference. Retrieved from Kressin, N., Spiro, A., Bossé, R., Garcia, R. & Kazis, L. (1996). Assessing oral health related quality of life: findings from the normative aging study. Medical Care, 34, 416–27. Kristjansson, E. A., Desrochers, A., & Zumbo, B. (2003). Translating and adapting measurement instruments for cross-linguistic and cross-cultural research: A guide for practitioners. Canadian Journal of Nursing Research, 35(2). Krosnick, J. A. (1991). Response strategies for coping with cognitive demands of attitude measures in surveys. Applied Cognitive Psychology, 5, 213-236. Kushnir, D., Zusman, S. P., & Robinson P. G. (2004). Validation of a Hebrew version of the Oral Health Impact Profile 14. Journal of Public Health Dentistry, 63, 71-75. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33, 159-174. Larcker, D. F. & Lessig. V. P. (1980). Perceived Usefulness of Information: A Psychometric Examination.  Decision Sciences, 11(1), 121-134.  115 Larsson, P., List, T., Lundstrom, I., Marcusson, A., & Ohrbach, R. (2004). Reliability and Validity of Swedish version of the Oral Health Impact Profile. Acta Odontol Scand, 62, 147-152. Lawshe, C. H. (1975). A quantitative approach to content validity. Personnel Psychology, 28, 563-575. Leao, A. & Sheilham, A. (1997). The Dental Impact on Daily Living. In G.D. Slade (Ed.), Measuring Oral Health and Quality of Life (pp. 121-135). Chapel Hill, NC: Dental Ecology, University of North Carolina. Lee, C. C., Li, D., Arai, S. & Puntillo, K. (2009). Ensuring cross-cultural equivalence in translation of research consents and clinical documents: a systematic process for translating English to Chinese. Journal of Transcultural Nursing, 20(1), 77-82. Lee, Y. J. (2010). Current Status of Overseas Compatriots. Retrieved March, 2, 2010, from Ministry of Foreign Affairs and Trade: u=TabMenu6. Leek, J. T., Taub, M. A., & Pineda, F. J. (2011). Cooperation between Referees and Authors Increases Peer Review Accuracy. PLoS ONE, 6(11), e26895. Levasseur, M., Desrosiers, J., & Tribble, D. S. C. (2007). Comparing the Disability Creation Process and International Classification of Functioning, Disability and Health Models. Canadian Journal of Occupational Therapy, 74, 233-242.  116 Li, H. C. W. & Lopez, V. (2004). Psychometric evaluation of the Chinese version of the State Anxiety Scale for Children. Research in Nursing & Health, 27, 198-207. Linn, R., Baker, E. & Dunbar, S. (1991). Complex, Performance-Based Assessment: Expectations and Validation Criteria, Educational Researcher, 28(8), 15-21. Liu, Y. H. (2011). Translation and Psychometric Validation of the Chinese Version of the Child-Adolescent Teasing Scale (PhD Dissertation, Boston University, 2011). Retrieved from =false. Locker, D. (1988). Measuring oral health: a conceptual framework. Community Dental Health, 5, 3-18. Locker, D. (1995). Social and psychological consequences of oral disorders. In E.J. Kay (ED). Turning strategy into action. Manchester: Eden Bianchipress. Locker, D. (2008). Research on oral health and the quality of life – a critical overview. Community Dental Health, 25(3), 130-131. Locker, D., & Allen, F. (2002). Developing Short-form Measures of Oral Health-related Quality of Life. Journal of Public Health Dentistry, 62(1), 13-20. Locker, D., & Allen, F. (2007). What do measures of 'oral health-related quality of life' measure?. Community Dentistry and Oral Epidemiology, 35, 401-411.  117 Locker, D., Mscn, E. W., & Jokovic A. (2005). What do older adults’ global self-ratings of oral health measure?. Journal of Public Health Dentistry, 65, 146-152. Locker, D. & Slade, G. D. (1993). Oral health and quality of life among older adults: the Oral Health Impact Profile. Journal of Canadian Dental Association, 59, 830-844. Lopez, R. & Baelum, V. (2006). Spanish version of the Oral Health Impact Profile (OHIP-Sp). BMC Oral Health, 6(11), 1-8. Lord, F. M. & Novick, M. R. (1968). Starktical theories of mental test scores. Reading: MA: Addison-Wesley. Lynn, M. R. (1986). Determination and quantification of content validity. Nursing Research, 35, 382-385. MacEntee, M. I. (2006). An existential model of oral health from evolving views on health, function and disability. Community Dental Health, 23, 5-14. MacEntee, M. I., Hole, R., & Stolar, E. (1997). The significance of the mouth in old age. Social Science Medicine, 45, 1449-1458. Magno, C., (2009). Demonstrating the difference between Classical Test Theory and Item Response Theory using derived test data. The International Journal of Education and Psychological Assessment, 1, 1-11. Maneesriwongul, W. & Dixon, J. K. (2004). Instrument translation process: a methods review. Journal of advanced nursing, 48(2), 175-186.  118 Martin, J. M., Pereze, M. B., Martinez, A. A., Martin, L. A. H., & Gallardo, E. M. R. (2009). Validation of Oral Health Impact Profile (OHIP-14SP) for adults in Spain. Med Oral Patol Or Oral Cir Bucal, 14(1), 44-50. Martin, M. L., Patrick, D. L., Gandra, S. R., Bennett, A. V., Leidy, N. K., Nissenson, A. R., Finkelstein, F. O., Lewis, E. F., Wu, A. W., & Ware, J. E. (2010). Content validation of two SF-36 subscales for use in type 2 diabetes and non-dialysis chronic kidney disease- related anemia. Quality Life Research, 20(6), 889-901. Matsumoto, D., & van de Vijver, F. J. R. (2011). Cross-cultural research methods in psychology. New York: Cambridge University Press. McCollum, B. B. (1943). Oral Diagnosis. Journal of American Dental Association, 30, 1218- 1233. McGrath, C. & Bedi, R. (2001). An evaluation of a new measure of oral health related quality of life--OHQoL-UK(W). Community 
Dent Health, 18, 138-143. McTavish, D. G. (1997). Scale validity: A computer content analysis approach. Social Science Computer Review, 15(4), 379-393. Messick, S. (1989). Validity. In R. L. Linn (Ed.), Educational measurement (3rd ed.) (pp. 13– 103). Washington, DC: American Council on Education. New York: Macmillan. Meulen, M. J., John, M. T., Naeije, M., & Lobbezoo, F. (2008). The Dutch version of the Oral Health Impact Profile (OHIP-NL): Translation, reliability, and construct validity. BMC Oral Health, 8(11), 1-7.  119 Montero-Martin, J., Bravo-Perez, M., Albaladejo-Martinez, A., Hernandez-Martin, L., & Rosel- Gallardo, E. M. (2009). Validation of Oral Health Impact Profile (OHIP-14sp) for adults in Spain. Med Oral Patol Oral Cir, 14(1), 44-50. Montero-Martin, J., Bravo-Perez, M., Vincente, M. P, Galindo, M. P., Lopez, J. F., & Albaladejo, A. (2010). Dimensional structure of the oral health-related quality of life in healthy Spanish workers. Health and Quality of Life Outcomes, 8, 24-33. Moore, G. C. & Benbasat, I. (1991). Development of an Instrument to Measure the Perceptions of Adopting an Information Technology Innovation. Information Systems Research, 2(3), 192-222. Morris, M. W., Leung, K., Ames, D., & Lickel, B. (1999). Views from inside and outside: Integrating Emic and Etic Insights about Culture and Justice Judgment. Academy of Management Review, 24(4), 781-796. Mosier, C. I. (1947). A critical examination of the concepts of face validity. Education and Psychological Measurement, 7, 191-205. Mumcu, G., Inanc, N., Ergun, T., Ikiz, K., Gunes, M., & Islek, U. (2006). Oral health related quality of life is affected by disease activity in Behçet's disease. Oral Diseases, 12, 145- 151. Murphy, K. R. & Deshon, R. (2000). Interrator correlations do not estimate the reliability of job performance ratings. Personnel Psychology, 53, 873-900.  120 Musch, D. C., Landis, J. R., Higgins, I. T. T., Gilson, J. C., & Jones, R. N. (1984). An application of kappa-type analysis to interobserver variation in classifying chest radiographs for pneumoconiosis. Statistics in Medicine, 3, 73-83. Myers, J., Do, D., Herbert, W., Ribisl, P., & Froelicher, V. F. (1994). A nomogram to predict exercise capacity from a specific activity questionnaire and clinical data. American Journal of Cardiology, 73, 591-596. Nahm, A., Solis-Galvan, L. E., Rao, S., & Ragu-Nathan, T. S. (2002). The Q-sort method: assessing reliability and construct validity of questionnaire items at a pre-testing stage. Journal of Modern Applied Statistical Methods, 1(1), 114-125. National Statistics Office of South Korea. (2007). Population Trends of the World and Korea. Retrieved March, 2, 2010, from National Statistics Office website: &num=164&parent_num=0&page=13&sdate=&edate=&search_mode=&keyword=&po sition=&catgrp=&catid1=&catid2=&catid3=&catid4= Nayak, S., Shifleft, S., Eshun, S., & Levine, F. (2000). Culture and Gender Effects in Pain Beliefs and the Prediction of Pain Tolerance. Cross-Cultural Research, 34(2), 135-151. Nevo, B. (1985). Face validity revisited. Journal of Educational Measurement, 22, 287-293. Noh, S., Avison, W. R., & Kaspar, V. (1992). Depressive symptoms among Korean immigrants: assessment of a translation of the Center for Epidemiologic Studies Depression Scale. Psycholological Assessment, 4, 84-91.  121 Okochi, J., Utsonomiya, S., & Takahashi, T. (2005). Health measurement using the ICF: test- retest reliability study of lCF codes and qualifiers in geriatric care. Health and Quality of Life Outcomes, 3-46. Ostir, G. V., Granger, C. V., Black, T., Roberts, P., Burgos, L., & Martinkewiz, P. (2006). Preliminary results for the PARPRO: a measure of home and community participation. Archives of Physical Medicine and Rehabilitation, 87(8), 1043e1051. Papagiannopoulou, V., Oulis, C. J., Papaioannou, W & Yfantopoulos, J. (2012). Validation of a Greek version of the Oral Health Impact Profile (OHIP-14) for use among adults. Health and Quality of Life Outcomes, 10(7), 1-10. Parsons, T. (1951). The social system. London: Routledge & Kegan Paul. Pearson, N. K., Gibson, B. J., Davis, D. M., Gelbier, S., & Robinson, P.G. (2007). The effect of a domiciliary denture service on oral health related quality of life: a randomised controlled trial. British Dental Journal, 203(2), E3; discussion 100-1. Peters, D. J. (1996). Disablement observed, addressed, and experienced: integrating subjective experience into disablement models. Disability Rehabilitation, 18, 593-603. Peterson, D. B. (2012). An International Conceptualization of Disability: The International Classification of Functioning, Disability and Health. In I. Marini & M. Stebnicki (Eds). The Psychological and Social Impact of Illness and Disability. New York: Springer Publishing Company. Phillips, L. R., & Luna, I. (1996). Toward a cross-cultural perspective of family caregiving. Western Journal of Nursing Research, 18(3), 236-252.  122 Pike, K. L. (1967). Language in Relation to a Unified Theory of the Structures of Human Behavior (2nd ed.). The Hague: Mouton. Pires, C. P. A. B, Ferraz, M. B., & Abreu, M. H. N. G. (2006). Translation into Brazillian Portuguese, cultural adaptation and validation of the oral health impact profile (OHIP- 49). Brazilian Oral Research, 20(3), 263-268. Polit, D. F. & Beck, C. T. (2006). The content validity index: Are you sure you know what's being reported? critique and recommendations. Research in Nursing & Health, 29, 489- 497. Rahman, M. S., Usman, S., Warren, O, & Athanasiou, T. (2010). Questionnaires, Surveys, Scales in Surgical Research: Concepts and Methodology. In T. Athanasiou & A. Darzi (Eds.), Key Topics in Surgical Research and Methodology (pp. 477-494). Berlin, Heidelberg: Springer Verlag. Rajesh, R., Pugazhendhi, S. & K. Ganesh (2010). Taxonomy Architecture of Knowledge Management for Third party Logistics Service Provider Benchmarking. An International Journal, 18(1). Randolph, J. J. (2005). Free-marginal multirater kappa: An alternative to Fleiss' fixed-marginal multirater kappa. Paper presented at the Joensuu University Learning and Instruction Symposium 2005, Joensuu, Finland, October 14-15th, 2005. Reeve, B. B. & Masse, L. C. (2004). Item response theory modeling for questionnaire evaluation. In Methods for testing and evaluating survey questionnaires (pp 247-275). New York: John Wiley & Sons.  123 Reisine, S. (1985). Dental health and public policy: The social impact of dental disease. American Journal of Public Health, 74, 27-30. Rener-Sitar, K., Petričević, N., Čelebić, A., & Marion, L. (2008). Psychometric Properties of Croatian and Slovenian Short Form of Oral Health Impact Profile Questionnaires. Croatian Medical Journal, 49(4), 536-544. Resnikow, K., Baranowski, T., Ahluwalia, J. S., & Braithwaite, R. L. (2000). Cultural sensitivity in public health: defined and demystified. Ethnicity & Disease, 9, 10-21. Ritter, L. A., & Sue, V. M. (2007). New directions for evaluation. American Evaluation Association, 115, 1-65. Rogler, L. H. (1999). Methodological sources of cultural insensitivity in mental health research. American Psychologist, 54, 424-433. Room, R., Janca, A., Bennett, L. A., Schmidt, L., & Sartorious, N. (1996). WHO cross cultural applicability research on diagnosis and assessment of substance use disorders: An overview of methods and selected results. Social Addiction, 91, 199-220. Rossiter, J. R. (2008). Content Validity of Measures of Abstract Constructs in Management and Organizational Research. British Journal of Management, 19, 380–388. Rubio, D. M., Berg-Weger, M., Tebb, S. S., Lee, E. S., & Rauch, S. (2003), Objectifying content validity: Conducting a content validity study in social work. Social Work Research, 27, 94–104.  124 Rulon, P. J. (1946). On the validity of educational tests. Harvard Educational Review, 16, 290- 296. Russell, A. L. (1956). A system of classification and scoring for prevalence surveys of periodontal disease. Journal of Dental Research, 35, 350-359. Sampogna, F., Johansson, V., Axtelius, B., Abeni, D., & Söderfeldt, B. (2009). Quality of life in patients with dental conditions: comparing patients' and providers' evaluation. Community Dental Health, 26(4), 234-238. Sandelowski, M. (1986). The problem of rigor in qualitative research. Advances in Nursing Science, 8(3), 27-37. Sanders, A. E., Slade, G. D., John, M. T., Steele, J. G., Suominen-Taipale, A. L., Lahti, S., Nuttall, N. M., & Allen, P. F. (2009). A cross-national comparison of income gradients in oral health quality of life in four welfare states: application of the Korpi and Palme typology. Journal of Epidemiology and Community Health, 63, 569-574. Saub, R., Locker, D., & Alison, P. (2005). Derivation and validation of the short version of the Malaysian Oral Health Impact Profile. Community Dentistry and Oral Epidemiology, 33, 378-383. Schilling, L. S, Dixon, J. K., Knafl, K. A., Grey, M., Ives, B., & Lynn, M. R. (2007). Determining content validity of a self-report instrument for adolescents using a heterogenous expert panel. Nursing Research, 56(5), 361-366. Schouwstra, S. J. (2000). On testing plausible threats to construct validity. Unpublished doctoral dissertation, University of Amsterdam, The Netherlands.  125 Segu, M., Collesano, V., Lobbia, S., & Rezzani, C. (2005). Cross-cultural validation of a short form of the Oral Health Impact Profile for temporomandibular disorders. Community Dentistry and Oral Epidemiology, 33(2), 125-130. Sen, B. & Mari, J. (1986). Psychiatric research instruments in the transcultural setting: experiences in India and Brazil. Social Science & Medicine, 23, 277–281 Shin, D. S., & Jung, Y. M. (2008). Oral health-related quality of life (OHQoL) and Related Factors among Elderly Women. Journal of Korean Academic Fundamental Nursing, 15(3), 332-341. Shiu, A. T. Y., Wong, R. Y. M., & Thompson, D. R. (2003). Development of a reliable and valid Chinese version of the diabetes empowerment scale. Diabetes Care, 26(10), 2817- 2821. Silness, J. & Loe, H. (1964). Periodontal disease in pregnancy. Correlation between oral hygiene and periodontal condition. Acta Odontologica Scandinavica, 22, 122-135. Sim, J. & Waterfield, J. (1997). Validity, reliability and responsiveness in the assessment of pain. Physiotherapy Theory and Practice, 13(1), 23-37. Sireci, S. G. (1998). The Construct of Content Validity. Social Indicators Research, 45(1), 83- 117. Sireci, S. G. (2007). On validity theory and test validation. Education Researcher, 36 (8), 477- 481.  126 Sireci, S. G., Bastari, B., & Allalouf, A. (1998). Evaluating construct equivalence across adapted tests. Invited paper presented at the meeting of the American Psychological Association, San Francisco. Sireci, S. G. & Geisinger, K. F. (1995). Using subject matter experts to assess content representation: A MDS analysis. Applied. Psychological Measurement, 19, 241-255. Sireci, S. G., Patsula, L., & Hambleton, R. K. (2006). Statistical Methods for Identifying Flaws in the Test Adaptation Process. In R.K. Hambleton et al. (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp.39-64). Hillsdale, NJ: Lawrence Erlbaum Associates. Slade, G.D. (1997). Oral Health Impact Profile. In G.D. Slade (Ed.), Measuring Oral Health and Quality of Life (pp. 94-104). Chapel Hill, NC: Dental Ecology, University of North Carolina. Slade, G. D., Nuttall, N., Sanders, A. E., Steele, J. G., Allen, P. F, & Lahti, S. (2005). Impacts of oral disorders in the United Kingdom and Australia. British Dental Journal, 198, 489- 493. Slade G. D. & Sanders, A. (2003). The ICF and Oral Health. In ICF Australian User Guide V1.0 Application (pp. 114-123). Canberra: Elect Printing. Slade, G. D. & Spencer, A. J. (1994). Development and evaluation of the oral health impact profile. Community Dental Health, 11(1), 3-11. Slocumb, E. M. & Cole, F. L. (1991). A practical approach to content validation. Applied Nursing Research, 4 (4), 192–195.  127 Smit, J., Van den Berg C. E., Bekker, L. G., Stein, D. J., & Seedat, S. (2006). Translation and cross-cultural adaptation of a mental health battery in an African setting. African Health Sciences, 6, 215-222. Smith, T. W. (2004). Context Effects in the General Social Survey. In P. P. Biemer, R. M. Groves, L. E. Lyberg, N. A. Mathiowetz & S. Sudman (eds.), Measurement Errors in Surveys. Hoboken, NJ, USA: John Wiley & Sons, Inc. Somekh, B., & Lewin, C. (2005). Research methods in the social sciences. London: Sage Publications. Son, G. R., Zauszniewski, J. A., Wykle, M. L., & Picot, S. J. (2000). Translation and validation of caregiving satisfaction scale in Korean. Western Journal of Nursing Research, 22(5), 609-622. Sprangers, M. & Schwartz, C. (1999). Integrating response shift into health-related quality of life research: a theoretical model. Social Science and Medicine, 48, 1507-1515. Stancic, I., Sojic, L. T, & Jelenkovic, A. (2009). Adaptation of Oral Health Impact Profile (OHIP) index for measuring impact of oral health on quality of life in elderly to Serbian language. Vonjnosanitetski pregled, 66(7), 511-515. Stansfield, C. W. (2003). Test translation and adaptation in public education in the USA. Language Testing 20(2), 189–207. Statistics Canada. (2006). Immigration in Canada: A Portrait of the Foreign-born Population. Retrieved October 13, 2010, from sa/97-557/index-eng.cfm.  128 Steele, J. G., Sanders, A. E., Slade, G. D., Allen, P. F., Lahti, S., Nuttall, N., & Spencer, A. J. (2004). How do age and tooth loss affect oral health impacts and quality of life? A study comparing two national samples. Community Dental Oral Epidemiology, 32, 107-114. Streiner, D. L. (2003). Starting at the beginning: an introduction to coefficient alpha and internal consistency. Journal of Personality Assessment, 80(1), 99-103. Szentpetery, A., Szabo, G., Marada, G., Szanto, I., & John, M.T. (2006). The Hungarian version of the Oral Health Impact Profile. European Journal of Oral Science, 114, 197-203. Tang, S. T. & Dixon, J. (2002). Instrument translation and evaluation of equivalence and psychometric properties: the Chinese Sense of Coherence Scale. Journal of Nursing Measurement,10(1), 59–76. Tennant, A. & McKenna, S. P. (1995). Conceptualising and defining outcome. British Journal of Rheumatology, 34, 899-900. Thorn, D. W. & Deitz, J. C. (1989). Examining content validity through the use of content experts. The Occupational Therapy Journal of Research, 9(6), 334-346. Thorndike, E. L. (1931). Measurement of intelligence. New York, NY: Bureau of Publishers. Thurstone, L. L. (1932). The Reliability and Validity of Tests. Ann Arbor, Michigan: Edwards Brothers. Tojib, D. R. & Sugianto, L. F. (2006). Content Validity of Instruments in IS Research. Journal of Information Technology Theory and Application, 8(3), 31-56.  129 Tripp-Reimer, R. (1984). Reconceptualizing the construct of health: Integrating emic and etic perspectives. Research in Nursing and Health, 7, 101-109. Ueda, S., & Okawa, Y. (2003). The subjective dimension of functioning and disability: What is it and what is it for? Disability and Rehabilitation, 25, 596-601. Üstün, B., Chatterji, S., Rehm, J., Kennedy, C., Prieto, L., & Epping-Jordan, J. (2010).  World Health Organization Disability Assessment Schedule II (WHO DAS II): development, psychometric testing and applications. World Health Organization, Geneva.  Usunier, J. C. (1998). International and cross-cultural management research. Thousand Oaks, CA: Sage. van de Vijver, F. J. R., & Poortinga, Y. H. (2006). Conceptual and methodological issues in adapting tests. In R.K. Hambleton et al. (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp.39-64). Hillsdale, NJ: Lawrence Erlbaum Associates. van de Vijver, F. J. R., & Tanzer, N. K. (1997). Bias and equivalence in cross-cultural assessment: An overview. European Review of Applied Psychology, 47, 263-279. van Ijzendoorn, M. H., Vereijken, C. M. J. L., Bakermans-Kranenburg, M. J., & Riksen- Walraven, J. M. (2004). Assessing Attachment Security With the Attachment Q Sort: Meta-Analytic Evidence for the Validity of the Observer AQS. Child Development, 75(4), 1188-1213. van Ommeren, M., Sharma, B., Thapa, S., Makaju, R., Prasain, D., Bhattarai, R. & de Jong, J. (1999). Preparing Instruments for Transcultural Research: Use of the Translation  130 Monitoring Form with Nepali-Speaking Bhutanese Refugees. Transcultural Psychiatry, 36(3), 285-302. van Windenfelt, B. M., Treffers, P.D.A., de Beurs, E., Siebelink, B. M., & Koudijs, E. (2005). Translation and Cross-Cultural Adaptation of Assessment Instruments used in psychological research with children and families. Clinical Child and Family Psychology Review, 9(2), 135-147. Vaughn, B. E., & Waters, E. (1990). Attachment behavior at home and in the laboratory: Qsort observations and strange situation classifications of one-year-olds. Child Development, 61, 1965-1973. Vickery,  S.  (1998).  Let’s not overlook content validity.  Decision Line, 29, (4), 10-13. Walker, R. J., & Kiyak, H. A. (2007). The impact of providing dental services to frail older adults: perceptions of elders in adult day health centers. Special Care Dentistry, 27, 139- 143. Waltz, C., & Bausell, R. B. (1981). Nursing research: Design, statistics, and computer analysis. Philadelphia: F. A. Davis. Warner, R. (1999). The emics and etics of quality of life assessment. Social Psychiatry and Psychiatric Epidemiology, 34, 117-121. Wild, D., Grove, A., Martin, M., Eremenco, S., McElroy, S., Verjee-Lorenz, A., & Erikson, P. (2005). Principles of Good Practice for the Translation and Cultural Adaptation Process for Patient-Reported outcomes (PRO) Measures: Report of the ISPOR Task Force for Translation and Cultural Adaptation. Value in Health, 8, 94-104.  131 Wilss, W. (1996).  Knowledge and Skills in Translator Behavior. Amsterdam: John Benjamins. Wong, M. C., Lo, E. C., & McMillan, A. S. (2002). Validation of a Chinese version of the Oral Health Impact Profile (OHIP). Community Dental Oral Epidemiology, 30(6), 423-430. Wong, S. T., Nordstokke, D., Gregorich, S., & Pérez-Stable, E. J. (2010). Measurement of social support across women from four ethnic groups: evidence of factorial invariance. Journal of Cross Cultural Gerontology, 25(1), 45-58. World Health Organization (1977). The third ten years of the World Health Organization. Geneva: World Health Organization. World Health Organization. (1980). International Classification of Impairments, Disabilities, and Handicaps (ICIDH). Geneva: World Health Organization. World Health Organization. (2001) International Classification Functioning, Disability and Health (ICF). Geneva: World Health Organization. World Health Organization. (2002). Towards a Common Language for Functioning, Disability and Health ICF. Retrieved May 16, 2011 from WHO website: World Health Organization. (2002). Cultural practices and Environment and Participation assessment. Retrieved May 20, 2011 from CDC website:  132 World Health Organization (2003). EUROHIS: Developing common instruments for health surveys. In A. Nosikov & C. Gudex (Eds.) World Health Organization, Regional Office for Europe. World Health Organization (2009). Oral Health. Retrieved September, 2, 2009 from WHO website: World Health Organization. (2010a). Process of Translation and adaptation of instruments. Retrieved January 28, 2010 from WHO website: World Health Organization (2010b). ICF introduction. Retrieved May 30, 2010 from WHO website: htpp:// World Health Organization. (2011). WHO Quality of Life-BREF. Retrieved April 12, 2011 from WHO website: Wynd, C. A., Schmidt, B., & Schaefer, M. A. (2003). Two quantitative approaches for estimating content validity. Western Journal of Nursing Research, 25, 508-518. Yaghmale, F. (2003). Content validity and its estimation. Journal of Medical Education, 3(1), 25-27. Yamazaki, M., Inuki, M., Baba, K., & John, M. T. (2007). Japanese version of the Oral Health Impact Profile (OHIP-J). Journal of Oral Rehabilitation, 34, 159-168.  133 Zeller, R. A. & Carmines, E. G. (1980). Measurement in the Social Sciences. In the Link Between Theory and Data. New York, NY: Cambridge University Press. Zumbo, B. D. (2009). Validity as Contextualized and Pragmatic Explanation, and Its Implications for Validation Practice. In Robert W. Lissitz (Ed.) The Concept of Validity: Revisions, New Directions and Applications, (pp. 65-82). IAP - Information Age Publishing, Inc.: Charlotte, NC.       134 Appendices Appendix A A.1 Coding Search for Inclusion in Systematic Synthesis Data Base Search Results MEDLINE  Validation (90 citations) (("oral health"[MeSH Terms] OR ("oral"[All Fields] AND "health"[All Fields]) OR "oral health"[All Fields]) AND Impact[All Fields] AND Profile[All Fields]) AND Validation[All Fields] + Translation (16 citations) (("oral health"[MeSH Terms] OR ("oral"[All Fields] AND "health"[All Fields]) OR "oral health"[All Fields]) AND Impact[All Fields] AND Profile[All Fields]) AND Translation [All Fields] + Cultural adaptation (8 citations) (("oral health"[MeSH Terms] OR ("oral"[All Fields] AND "health"[All Fields]) OR "oral health"[All Fields]) AND Impact[All Fields] AND Profile[All Fields]) AND Cultural adaptation[All Fields]  EMBASE Validation (48 citations) (Oral Health Impact Profile and Validation).mp. [mp=title, abstract, subject headings, heading word, drug trade name, original title, device manufacturer, drug manufacturer] + Translation (13 citations) (Oral Health Impact Profile and Translation).mp. [mp=title, abstract, subject headings, heading word, drug trade name, original title, device manufacturer, drug manufacturer] + Cultural Adaptation (5 citations) (Oral Health Impact Profile and Cultural Adaptation).mp. [mp=title, abstract, subject headings, heading word, drug trade name, original title, device manufacturer, drug manufacturer]    135  Appendix B  B.1 English Version of Oral Health Impact Profile-14 Slade, G.D., & Spencer, A J. (1994). Development and evaluation of the oral health impact profile. Community Dental Health, 11(1), 3-11.  HOW OFTEN have you had the problem during the last year? (circle your answer)   1. Have you had trouble pronouncing any words because of problems with your teeth, mouth or dentures? VERY OFTEN FAIRLY OFTEN OCCAS- IONALLY HARDLY EVER NEVER DON’T KNOW 2. Have you felt that your sense of taste has worsened because of problems with your teeth, mouth or dentures? VERY OFTEN FAIRLY OFTEN OCCAS- IONALLY HARDLY EVER NEVER  DON’T KNOW 3. Have you had painful aching in your mouth? VERY OFTEN FAIRLY OFTEN OCCAS- IONALLY HARDLY EVER NEVER  DON’T KNOW 4. Have you found it uncomfortable to eat any foods because of problems with your teeth, mouth or dentures? VERY OFTEN FAIRLY OFTEN OCCAS- IONALLY HARDLY EVER NEVER  DON’T KNOW 5. Have you been self conscious because of your teeth, mouth or dentures? VERY OFTEN FAIRLY OFTEN OCCAS- IONALLY HARDLY EVER NEVER  DON’T KNOW 6. Have you felt tense because of problems with your teeth, mouth or dentures? VERY OFTEN FAIRLY OFTEN OCCAS- IONALLY HARDLY EVER NEVER  DON’T KNOW 7. Has your diet been unsatisfactory because of problems with your teeth, mouth or dentures? VERY OFTEN FAIRLY OFTEN OCCAS- IONALLY HARDLY EVER NEVER  DON’T KNOW 8. Have you had to interrupt meals because of problems with your teeth, mouth or dentures? VERY OFTEN FAIRLY OFTEN OCCAS- IONALLY HARDLY EVER NEVER  DON’T KNOW 9. Have you found it difficult to relax because of problems with your teeth, mouth or dentures? VERY OFTEN FAIRLY OFTEN OCCAS- IONALLY HARDLY EVER NEVER  DON’T KNOW 10. Have you been a bit embarrassed because of problems with your teeth, mouth or dentures? VERY OFTEN FAIRLY OFTEN OCCAS- IONALLY HARDLY EVER NEVER  DON’T KNOW 11. Have you been a bit irritable with other people because of problems with your teeth, mouth or VERY OFTEN FAIRLY OFTEN OCCAS- IONALLY HARDLY EVER NEVER  DON’T KNOW  136 dentures? 12. Have you had difficulty doing your usual jobs because of problems with your teeth, mouth or dentures? VERY OFTEN FAIRLY OFTEN OCCAS- IONALLY HARDLY EVER NEVER  DON’T KNOW 13. Have you felt that life in general was less satisfying because of problems with your teeth, mouth or dentures? VERY OFTEN FAIRLY OFTEN OCCAS- IONALLY HARDLY EVER NEVER  DON’T KNOW 14. Have you been totally unable to function because of problems with your teeth, mouth or dentures? VERY OFTEN FAIRLY OFTEN OCCAS- IONALLY HARDLY EVER NEVER  DON’T KNOW  137  B.2 Oral Health Impact Profile-14K (Bae et al., 2007)  Bae, K.H., Kim, H.D., Jung, S.H., Park, D.Y., Kim, J.B., Paik, D.I., & Chung, S.C. (2007). Validation of the Korean version of the oral health impact profile among the Korean elderly. Community Dentistry and Oral Epidemiology , 35(1), 73-79.  지난 3 개월 동안 치아나 잇몸 때문에 아래의 경험을 얼마나 자주 겪으셨습니까? 매우 자주 가끔 거의 전혀 ※ 매우: 1 주일에 2-3 회 이상 자주: 1 주일에 1 회 정도 가끔: 한 달에 2-3 회 정도 거의: 한 달에 1 회 이하 전혀: 경험한 적이 없음 1. 발음이 잘 안되어 불편했던 적이 있으십니까?  2. 맛을 느끼는 감각이 예전보다 나빠졌다고 느끼신 적이 있습니까?  3. 혀나 혀밑, 빰 입천정 등이 아픈 적이 있습니까?  4. 아프거나 거북스러운 입안의 문제 때문에 음식 먹기가 불편한 적이 있습니까? 5. 창피해서 다른 사람을 만나기가 꺼려지신 적이 있습니까?  6. 신경이 많이 쓰인 적이 있습니까?  7. 식생활이 불만스러운 적이 있습니까?  8. 식사를 도중에 중단하신 적이 있습니까?  9. 편안하게 쉬지 못하신 적이 있습니까?  10. 난처하거나 당황스러웠던 적이 있습니까?   138 11. 다른 사람들에게 화를 잘 내게 되신 적이 있습니까?  12. 평소 하시던 일을 하기가 어려웠던 적이 있습니까?  13. 살아가는 것이 예전에 비해서 덜 만족스럽다고 느끼신 적이 있습니까? 14. 정신적 신체적 사회적으로 전혀 제 몫을 할 수 없었던 적이 있습니까?    139 B.3 Oral Health Impact Profile-14K (Kim & Min, 2009)  Kim, G.U. & Min, K.J. (2009). The impact of oral health impact profile (OHIP) level on degree of patients' knowledge about dental hygiene. The Korean Academic Industrial Society, 10(4), 717-721.  Ⅲ.구강건강에 관한 질문입니다 . (해당란에 “V”해주세요) OHIPー14 지난 1 년 동안 치아나 입안의 문제로 아래와 같은 느낌이나 문제가 발생한 적이 있었습니까? 매우 자주 있었다 자주있 었다. 가끔 아주 가끔 있었다 전혀 없었다 1.입안의 문제로 발음 곤란을 느끼신 적이 있었습니까? ① ②  ③ ④ ⑤ 2.입안의 문제로 맛을 느끼는데 어려운 적이 있었습니까? ① ②  ③ ④ ⑤ 3.입안의 문제로 혀나 혀 밑, 뺨 입천장 등이 아픈 적이 있었습니까? ① ②  ③ ④ ⑤ 4.입안의 문제로 음식물 먹기가 불편한 적이 있었습니까? ① ②  ③ ④ ⑤ 5.입안의 문제로 타인을 만나기 꺼려 지신 적이 있었습니까? ① ②  ③ ④ ⑤ 6.입안의 문제로 신경이 많이 쓰인 적이 있었습니까? ① ②  ③ ④ ⑤ 7.입안의 문제 때문에 식생활이 불만스러운 적이 있었습니까? ① ②  ③ ④ ⑤ 8.입안의 문제로 식사를 도중에 중단하신 적이 있었습니까? ① ②  ③ ④ ⑤ 9.입안의 문제로 마음 편히 쉬지 못한 적이 있었습니까? ① ②  ③ ④ ⑤ 10.입안의 문제 때문에 창피한 적이 있었습니까? ① ②  ③ ④ ⑤ 11.입안의 문제 때문에 타인에게 화를 내게 되신 적이 있었습니까? ① ②  ③ ④ ⑤ 12.입안의 문제 때문에 평소에 하시던 일을 하기가 어려운 적이 있었습니까? ① ②  ③ ④ ⑤ 13.입안의 문제 때문에 일상생활이 만족스럽지 못 하다고 느끼신 적이 있었습니까? ① ②  ③ ④ ⑤  140 14.입안의 문제 때문에 일상생활을 전혀 할 수 없었던 적이 있었습니까? ① ②  ③ ④ ⑤   141  B.4 Oral Health Impact Profile-14K (Kim & Min, 2008) Kim, J.H. & Min, K.J. (2008) Research about relationship between the Quality of life, oral health, and total health of adults. Korean Journal of Health Education and Promotion, 25(2), 31-46. 현재의 구강상태에 대한 질문입니다. 지난 1년 동안 치아나 잇몸, 해넣은이, 틀니, 혀, 입안의 문제 때문에 불편함이나 통증에 대해 느끼시는 정도를 다섯가지 항목 중 하나를 골라 응답해주세요  매우 자주 자주 가끔 그렇지 않다 전혀 그렇지 않다 1 발음 곤란을 느끼신 적이 있습니까?  2 맛을 느끼는 감각이 예전보다 나빠졌다고 느끼신적이 있습니까?  3 입안이 쑤시고 아픈적이 있습니까?(입천정,혀,뺨안쪽)  4 입안의 문제 때문에 음식물 먹기가 불편한 적이 있습니까?  5 치아,입안의 문제로 다른 사람을 만나기 꺼려지신적이 있습니까?  6 입안의 문제 때문에 신경이 많이 쓰인 적이 있습니까?  7 식사를 만족스럽게 하지 못한 적이 있습니까?  8 입안의 문제로 식사를 도중에 중단하신 적이 있습니까?  9 입안의 문제 때문에 마음 편히 쉬지 못한 적이 있습니까?  10 입안의 문제 때문에 창피한 적이 있습니까?  11 입안의 문제 때문에 다른 사람들에게 화를 잘 내게 되신 적이 있습니까?  12 입안의 문제 때문에평소 하시던 일을 하기가 어려웠던 적이 있습니까?   142  13 입안의 문제 때문에 일상생활이 만족스럽지 못하다고 느끼신 적이 있습니까?  14 입안의 문제 때문에 일상생활을 전혀 할 수 없었던 적이 있습니까?   143  Appendix C C.1 Computation of Item-level ADM for Clarity and Cultural Equivalence Index  ADM for an item j is estimated as follows: where N is the number of judges or observations for an item j, xjk is the kth judge’s rating on item j, and x j is the mean of the judges’ scores on item j. (Burke & Dunlap, 2002)  C.2 Computation of ADM for Clarity and Equivalence Index  or simply averaged ADM(j) (Burke & Dunlap, 2002).  144  C.3 Computation of Multirater Kfree for Relevance index items   N is the number of cases, n is the number of raters, and k is the number of rating categories. (Randolph, 2005)    C.4 Interpretation of Multirater Kfree Statistics   Kappa Level of agreement (Landis & Koch, 1977) > 0 Equal to chance and poor agreement 0.00 – 0.20 Slight agreement 0.21 – 0.40 Fair agreement 0.41 – 0.60 Moderate agreement 0.61 – 0.80 Substantial agreement 0.81 – 1.00 Almost perfect agreement     145 Appendix D D.1 Informed Consent   August 3, 2010 Consent Form (Content Experts) RESEARCH TITLE: Qualitative Validation of Korean version of OH-QoL Measures: Content Validation with Korean Content Experts Principal Investigator: Jaesung Seo, HBSc MSc Candidate at the Faculty of Dentistry, UBC Phone: 604-263- XXXX  Research Supervisors: Dr. Mario Brondani, DDS, MSc, PhD Dr. Michael MacEntee, LDS, Dlp. Prosth, FRCD, Ph.D.  The following information has been provided to you for making an informed decision about your participation in this study. PURPOSE The purpose of this study is to conduct a content validation study of the Korean version of oral health impact profile (OHIP), designed to measure your oral health-related quality of life (OH- QoL). It is being conducted as part of dissertation for Masters in Science degree.  146 POTENTIAL BENEFITS The findings of this research can help delineate common problems in cultural adaptation of OHQoL measures and propose the use of structured content validation methods to improve validity of cross-cultural measurement and comparison of OHQoL. STUDY PROCEDURE You are invited to self-administer the content validation questionnaire electronically on the attached PDF form. The content validation questionnaire is designed to collect information about the content validity of two Korean versions of OHIP. You will be asked to rate each element of the indicators for clarity, relevance, and cultural equivalence. Upon completion of the questionnaire, you can submit the PDF form through FTP server by clicking on the submit button or E-mail the saved PDF form to  CONFIDENTIALITY Please note that you are under no obligation to participate in this study. If you choose to participate, you can withdraw or ignore questions at any time without consequences. Any information obtained in this study and that can be identified with you will remain confidential and will be disclosed only with your permission. Communication in scientific reports will identify the program and individual participants only by a code known to the investigators. Information you give will be stored confidentially, and under no circumstances will it be revealed to other faculty members without your expressed wish and instructions. CDs of audio- recordings and transcripts of interviews will be stored in a locked filing cabinet at UBC and will be destroyed after five years.  KNOWN RISKS  147 There are no known risks associated with participating in this research. Data collection will take place only after you are fully aware of the study, and after your informed consent is obtained.  REMUNERATION/COMPENSATION You will be remunerated $50 for your participation.  CONTACT FOR INFORMATION ABOUT THE STUDY If you have any questions or would like further information with respect this study, you may contact: Jaesung Seo Phone: 604-XXX-XXXX E-mail: CONTACT FOR CONCERNS ABOUT THE RIGHTS OF RESEARCH PARTICIPANTS If you have any concerns about your treatment or rights as a research participant, you may contact the Research Subject Information Line in the UBC Office of Research Services at 604- XXX-XXXX or  CONSENT Your participation in this study is entirely voluntary, and you may refuse to participate or withdraw from the study at any time without consequence. You do not have to answer all questions on the questionnaires. Your signature below indicates that you have received a copy of this consent form for your own records.  148 Your signature indicates that you consent to participate in this study. You are willing to have your interview audio-recorded and give permission for the principal investigator to use the information you are providing as part of a larger study focused on the same issue. X Participant’s signature     Date X Printed name of the participant signing above  Principal Investigator’s signature     Date   149  D.2 Cover Letter and Content Validation Questionnaire              150      151  152       153 WHO’s framework of disability known as International Classification of Impairments, Disabilities, and Handicaps (ICIDH) (1980). ICIDH explains consequences of disease in three dimensions:  Impairment: organ system level loss of structure or function  Disability: person level loss of the ability to function in ways considered normal for a human being  Handicap: societal level disadvantage for the person created by the intersection of the impairment or disability with the environment and that person’s roles.  154   155  156  157  158  159      


Citation Scheme:


Usage Statistics

Country Views Downloads
United States 7 0
Japan 7 0
China 7 0
City Views Downloads
Tokyo 7 0
Beijing 7 0
Washington 3 0
Mountain View 2 0
Unknown 2 0

{[{ mDataHeader[type] }]} {[{ month[type] }]} {[{ tData[type] }]}


Share to:


Related Items