Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

Evaluating the effectiveness of the Healthiest Babies Possible dental program Lin, Diana Louise 2008

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Notice for Google Chrome users:
If you are having trouble viewing or searching the PDF with Google Chrome, please download it here instead.

Item Metadata


24-ubc_2008_spring_lin_diana_louise.pdf [ 8.35MB ]
JSON: 24-1.0066711.json
JSON-LD: 24-1.0066711-ld.json
RDF/XML (Pretty): 24-1.0066711-rdf.xml
RDF/JSON: 24-1.0066711-rdf.json
Turtle: 24-1.0066711-turtle.txt
N-Triples: 24-1.0066711-rdf-ntriples.txt
Original Record: 24-1.0066711-source.json
Full Text

Full Text

EVALUATING THE EFFECTIVENESS OF THE HEALTHIEST BABIES POSSIBLE DENTAL PROGRAM by Diana Louise Lin B.H.E., University of British Columbia, 1985 Dip.DH, Vancouver Community College, 1990 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF MASTER OF SCIENCE in THE FACULTY OF GRADUATE STUDIES (Dental Science) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) April 2008 © Diana Louise Lin, 2008 ABSTRACT Objectives: A community dental public health program in Vancouver has been providing clinical hygiene services and oral health counseling to a limited number of high-risk, low income pregnant women for over 20 years. To enable future program decision-making, a program evaluation was undertaken with the following objectives: 1) to describe program, clients, and their oral health needs; 2) to determine whether program activities have been implemented as intended; and 3) to assess program's effectiveness in improving oral health status of clients and in improving their knowledge and behavior in relation to their own health and those of their children. Methods: The evaluation, undertaken by the resident dental hygienist, had two phases. Phase 1: Evaluability assessment, descriptive and process evaluation (retrospective and concurrent chart reviews, semi-structured interviews, appointment monitoring, and field observations). Phase 2: A short- and medium-term outcome evaluation with a convenience sample who attended over a 1-year period. Data, collected by questionnaires, interviews, clinical indices, chart review, field observations, and appointment statistics, was analyzed with a combination of univariate and bivariate analyses. Results: Phase 1: Stakeholders' goals were identified; a logic model and organizational flowchart were developed. Chart review (N=123) revealed mean client age of 27 years; 28% Canadian-born; 48% had other children; 78% were concerned about "bleeding gums"; and 63% had visible tooth decay. 28% of women referred to the program never attended. Unfamiliarity of clients with the dental "experience", language barriers, and clinic operation and time restraints affected implementation of program's activities. Phase 2: Outcomes in clients (N=61) demonstrated significant improvements (P<0.05) in clinical indices, oral health knowledge and tooth brushing skills, and dental care for clients' children. A positive program experience was ii reported from all the women. However, 79% (48/61) of clients never obtained the "outside" dental services that they needed. Conclusions: Evaluation revealed positive changes, extending into the postnatal period in knowledge, behaviors and clinical outcomes despite language barriers, insufficient resources, and ill-defined program goals. Identified program limitations need to be addressed to further improve outcomes of this promising intervention. Research was partially supported by a grant from the British Columbia Dental Hygienists Association. iii Table of Contents Abstract^ ii Table of Contents  ^iv List of Tables^ .xi List of Figures ^xii List of Abbreviations   .xiii Acknowledgements^ xv Dedication^ xvi I.^Introduction 1 1.1. Introduction^ 1 1.2. Purpose^ .2 1.3. Objectives^ .3 1.4. Review of literature 4 1.4.1.^Oral health .4 1.4.2.^Oral health and pregnancy^ 4 Early childhood caries^ .5 Periodontal disease .5 Pre-term, low birth weight babies^ 6 Preventive dental prenatal programs^ .7 1.4.3.^Access to dental services^ 8 Dental services in Canada^ 8 Publicly-funded dental services in Vancouver^ .9 Dental service and women^ .9 1.4.4.^Dental prenatal programs   10 iv 2. Healthiest Babies Possible Program^ 10 1.4.5. Evaluation 12 Description ^ 12 Insider or internal evaluators  12 Evaluation guidelines^ 13 Evaluability assessment (EA)    16 1.4.4. Evaluations of dental prenatal community programs   18 Published literature^ 18 Limitations in the published literature^ 23 Models for program planning and evaluation^ 24 Unpublished literature^ 24 Limitations in the unpublished literature   25 Methods^ 27 2.1. Overview^ ..27 2.2. Phase One^ .29 2.2.1. Evaluability assessment (EA)  ..29 Overview and objectives^ 29 Main stakeholders 30 Program theory^ 30 Existing program circumstances^ 30 Organizational climate^ 31 Plausibility^ 31 2.2.2. Descriptive evaluation ^32 Overview and objectives^ .32 Program^ ^ .32 3. 2.2.3.^Clientele^ Process evaluation ..33 .33 Overview and objectives^ 33 Interviews and questionnaires 34 Chart reviews and appointment monitoring^ .35 Field observations^ .36 2.3. Phase Two^ 36 2.3.1. Outcome evaluation^ ^36 Objectives .36 Design   .36 Sampling^ .37 Pilot study  37 Recruitment and evaluation activities   .38 Outcome indicators^ 40 Chart review ..41 Questionnaires^ 41 Interviews (self-reports)^ 41 Observation of client brushing .42 Appointment monitoring^ 42 Clinical indices and examination^ 43 Controls (reliability)^ 44 Data collection and analysis^ .45 Results .46 3.1. Phase One^ 46 3.1.1.^Evaluability assessment of HBP Dental Program^ .46 vi Overview^ ^ .46 Identification of main stakeholders   .49 Evaluation scope, purpose, and tasks^ ^49 Objectives of HBP Dental Program^ .50 Roles and responsibilities of stakeholders^ 50 Program theory^ .. ..50 Existing program circumstances^ 51 Receptivity of organization to program and evaluation....54 EA recommendations^ 55 3.1.2. Descriptive evaluation (program description)  .55 Program history^ .55 Objectives ^56 Physical environment^ 56 Organizational context ..57 Personnel^ 59 Activities 60 Resources^ ..63 Funding .. 63 Record-keeping mechanisms^ 63 Clientele^ 64 3.1.3. Process evaluation 73 Referral process^ 73 HBP referral form 74 Appointments^ 74 Medical history forms  76 vii ^utraaid tplm uopoejspus s5uIpug reo!uu ^a5painnouN  ^uogenrena um5oid^•I•t, uoIssnosm .C17.£ ' 9 ' 1 ' Z. E •g ' 1 ' Z. £ ^SSalialWAU pun Jo!setiag17. 1' Z. £ ^wall two pun soNdei2oluap luau' £ ' 1' Z• £ ^5upsal Alfficteffali' Z. 1' Z. £ ^uogedIopird•I ' I 7' £ tIOTIMMA3 311IONTIO • I'Z'E ^OM" °SUM 'Z'E ^samnpou ssaoaid 5uwar20.1 suogenJascio Nay Jatgo^•OI.E.I.E ^Ts!! Jamnald muaa^.6" CI' E uogeonpa twat' Imo^'WE'VE ^saopuas atia!5Ati momij^"L. E. I ' £ ^S3DIAJOS M.I.I3(j^'9' E. I' £ ^MUT U092ULIOJUI luau° Hy^.g.E. I *£ •17 176 176 £6 16 68 S8 ZS I8 6L 6L 6L 6L 8L 8L LL LL 9C IIIA 66 uogenjUA3 JapIsui uop-enfuna atuoolno ^uogertrena SS000Jd ^uogenpna 3ARdI.IDS3(1 luatussossu Almquareng S6 176 66 86 L6 96 S6 ^4.1.4.^Challenges^^Insider evaluation^^Measurements^Other challenges^ ^ 4.1.5.^Recommendations for program evaluation in similar settings ^ 100 ...100 101 102 103 4.2. Healthiest Babies Possible (HBP) Dental Program^ 104 4.2.1.^Strengths 104 Relationship with HBP prenatal program^ 104 Program activities^ 104 Outcomes^ .105 4.2.2.^Challenges 107 Direction  107 Resources^ 108 Referral process and appointments^ 109 Outcomes^ 109 Vancouver Community Dental Program infrastructure...109 4.2.3. Recommendations^ .110 Direction  110 Communication^ 110 Referral process 111 Program activities^ 111 Record-keeping 111 4.3. Future directions^ 112 5.^Conclusion^ 114 Bibliography 116 ix Appendices ^ 129 Appendix A^Summary of Eight Articles in Literature Review^ 129 Appendix B^Certificates of approval^ 133 Appendix C^Process evaluation indicators, methods, and relevance ^ 135 Appendix D^List of interview questions ^ 138 Appendix E^Questionnaire #1 ^ 140 Appendix F^Poster questionnaire for HBP staff (2007)^  .141 Appendix G^Consent form^ 142 Appendix H^NCHO Dental clinic Medical history form^ 145 Appendix I^NCHO Dental Chart^ 146 Appendix J^HBP Dental Referral form .148 Appendix K^Interview recording form (prenatal visits) ^ 150 Appendix L^Interview recording form (postnatal visit)  ^ 151 Appendix M^Questionnaire #2^ 152 Appendix N^Questionnaire #3 ^ 153 Appendix 0^Study appointment monitoring of outcome participants  ^ .154 Appendix P^Description of clinical indices ^ 155 Appendix Q^Clinical indices recording form ^ 156 Appendix R^List of stakeholders' roles and responsibilities^ 157 Appendix S^Client dental information form (2004)^ 161 Appendix T^Appointment monitoring record log sheet 162 Appendix U^List of dental providers offering reduced fees (May 2005)^ 163 List of Tables Table 1. Demographics and characteristics of clients 2004-2005 (N=123)^ ..68 Table 2. Dental demographics of clients 2004-2005 (N=123)^ ..70 Table 3. Visual oral examination at Visit #1 on clients 2004-2005 (N=123)^ 72 Table 4. Dental program attendance of HBP clients referred 2004-2006 74 Table 5. Time between Screening date, Visit #1 and Visit #2 in 2004-2005^ 75 Table 6. Dental services provided in 2004-2005^  .77 Table 7. Client participation, measurement indicator in Time (N=61)^ .80 Table 8. General characteristics of participants Oct 2005-2006 (N=61) 83 xi List of Figures Figure 1. Timeline of evaluation activities 2005-2007^ .28 Figure 2. Flowchart of Outcome evaluation activities 39 Figure 3. Main stakeholders' objectives and goals^ 47 Figure 4. Logic model: HBP Dental Program Theory 48 Figure 5. Organizational context of HBP Dental Program^ .58 Figure 6. Vancouver Community Dental Program personnel 59 Figure 7. HBP Dental Program Activity Flowchart^ 61 Figure 8. Oral health concerns and needs of clients 2004-2005 (N=123)^66 Figure 9. Reported oral concerns of "3-visit" clients (N=40)^ 86 Figure 10. Oral health behavior and awareness of "3-visit" clients (N=40)^88 Figure 11. Correct oral health knowledge (%) at Visits (N=39)^ ..90 Figure 12. Indices of "3-visit" clients (N=36)*^ 92 xii List of Abbreviations BREB^=^Behavioral and Research Ethics Board CDC^=^Center for Disease Control and Prevention CHC^=^Community Health Center CPITN^=^Community Periodontal Index of Treatment Needs DL^=^name of graduate student/evaluator/hygienist EA^=^Evaluability assessment ECC^=^Early childhood caries ECHC^=^Evergreen Community Health Center EXCEL^=^Name of computer software database HBP^=^Healthiest Babies Possible ICY^=^Infant Child and Youth programs Mid-Main^=^Name of dental clinic in Vancouver NCHO^=^North Community Health Office PDI^=^Periodontal Disease Index Portland^=^Name of a dental clinic in Vancouver PTLBW^=^Pre-term low birth weight SPSS^=^Statistical Package for Social Sciences Sunrise^=^Name of a dental clinic in Vancouver UBC^=^University of British Columbia US^=^United States USD^=^United States Dollar VCDP^=^Vancouver Community Dental Program VCC^=^Vancouver Community College VCH^=^Vancouver Coastal Health VCHRI^=^Vancouver Coastal Health Research Institute VCHSDA^=^Vancouver Coastal Health Service Delivery Area VGH^=^Vancouver General Hospital xiv Acknowledgements I would like to express my gratitude to the following people for their contribution... To Dr. Rosamund Harrison, thank you for your mentoring, insight, energy, patience, and care. It was a privilege and blessing to have you as my graduate supervisor. To my committee members, Dr. Jolanta Aleksejuniene, for your enthusiasm, expertise and support; Dr. Chris Clark, Dr. Shafik Dharamsi, Dr. Teri-lee Norfolk for your insights and encouragement; and Dr. Jim Frankish, for opening my eyes to community based research. To Dr. Pam Glassby, Tana Wyman, and Jeannie Dickie, my research and academic experience would not been as successful and memorable without your input, cooperation, and encouragement. To Gina Wong, for encouraging and supporting me to pursue great things, Nicoline and Steve Bassaw, for helping me with editing and formatting, Pauline Imai, for your endless enthusiasm and support for my endeavors, Dr. Susanne Sunell, for showing me how to think outside the box, and Phil Feeley, for your expertise with SPSS. To the staff and clients at the North Community Health Office and Evergreen Community Health Center - Healthiest Babies Possible Program, for your participation in the research; and the British Columbia Dental Hygienist Association, for your research grant. To my family and friends, for your encouragement and prayers. To my husband Chris and children, Daniel and Kimberly, thank you for your countless sacrifices and unwavering support and belief in me. You were my loudest cheering squad. And thanks to my Lord and Savior Jesus Christ for all He has done for me. XV This thesis is dedicated to my loving husband, Chris my two wonderful kids, Daniel and Kimberly and to the memory of my mom, Eva Wong. xvi 1. Introduction 1.1 Introduction Evaluation is widely recognized as being important but it is often a neglected area of practice (Petersen & Kwan, 2004). Evaluation is a process which determines areas of concern, collects and analyzes information, and aids decision-makers in selecting among alternatives (Burry, 1973). The evaluation process allows program decision-makers to assess outputs and outcomes; examine areas for improvement; determine client satisfaction; provide evidence to support changes; demonstrate benefits versus costs; and validate a program for accountability and sustainability. It can even serve as a warning for potential problems with the program (Thompson & McClintock, 2000). Unfortunately, evaluations are often performed under tight time constraints and with limited resources (Smith, 1989; Petersen & Kwan, 2004). As a result, policy makers and program decision-makers may find evaluations expensive and of little value, and evaluators find program objectives and goals to be ill-defined and unrealistic (Smith, 1989). Thus, decision- makers are often left to develop their own evaluations. If they have a limited background in evaluation, they tend to be frustrated with the process and the usefulness of their results. Evaluation research is a type of research which produces findings that are practical, applicable and immediately useful (Kidder, 1981). It differs from "basic" scientific research in its purposes, uses and relationships with political and social organizations and its practical application (Kidder, 1981). Basic scientific research involves testing hypotheses, building theories, and possible future application (Kidder, 1981) and aims to contribute to knowledge (Ovretveit, 2002). However, overlap does exist between evaluation and basic research. Evaluation, like basic research, employs careful and rigorous data collection and statistical 1 analysis methods while basic research, similar to evaluation research, uses comparisons and/or control groups (Ovretveit, 2002). Interest in evaluation is growing due to increasing budget constraints and demand for quality assurance, accountability, better program management, and resource allocation (Rutman, 1980). Currently, relatively few public dental health programs in Canada have been the subject of evaluation or evaluation research, especially public dental prenatal programs. Public health interventions can be improved and better accounted for by effective program evaluation that involves feasible, useful, ethical and accurate procedures (Milstein, 1999). 1.2 Purpose This thesis will evaluate an existing provincial government dental program for pregnant women classified as "high risk" for pre-term and/or low birth weight babies. The goal is to conduct an evaluation of this program, provide a feasible evaluation framework for other public dental prenatal programs, and gather evidence for key decision-makers for program improvement and/or further dental interventions for pregnant women. This thesis will: a) list evaluative objectives; b) conduct a literature review of similar public dental prenatal program evaluations; c) outline methodologies used; d) perform an evaluability assessment, descriptive, process and outcome evaluation for the program of interest; e) analyze results; and f) discuss challenges, limitations, and future implications from this research. Challenges of and reflections about insider evaluation are discussed. 2 1.3 Objectives The objectives of this evaluation of the HBP Dental Program are as follows: 1) To describe the program, its clients and their oral health needs; 2) To determine whether the program's activities have been implemented as intended; and 3)^To assess whether the program is effective in improving the oral health status of clients and in improving their knowledge and behavior in relation to their own health and those of their children. First of all, an evaluability assessment was conducted to determine a feasible evaluation framework for this program. The evaluation was then divided into two phases. Phase One included descriptive and process evaluations which were directed by the evaluation questions, "What is the history, physical environment, and organizational structure of the dental prenatal program? Who are the clientele and what are their oral health needs?" "To what extent are the dental prenatal program activities being implemented effectively?" These results identified the program's day-to-day "reality" and its strengths and challenges, and may lead to strategies to improve the program. Phase Two was the outcome evaluation, directed by the question, "Does the dental program for pregnant women improve their gingival health and their knowledge and behavior in relation to their oral health and those of their children?" The emphasis was to evaluate the success of the program, not the clients of the program. 3 1 .4 Review of literature 1.4.1. Oral health. Oral health is an integral part of overall general health. It encompasses the teeth and the surrounding soft and hard tissues and affects a person's ability to eat, speak, smile, taste, chew, touch, kiss, and swallow (Wener & Lavigne, 2004). Oral health can have a significant impact on quality of life and daily functioning and differs between women and men in areas such as eating disorders, oral health practices and behaviors, esthetic concerns, temporomandibular disorders, and hormonal influences on periodontal health (Covington, 1996). Oral diseases like periodontal disease, dental caries (tooth decay), and oral cancer can be painful, costly, debilitating, and in some extreme cases, fatal (Wener & Lavigne, 2004). In 2000 the United States Surgeon General labeled these diseases as the "silent epidemic" (National Institute of Dental and Craniofacial Research., 2000). More research is emerging on the relationship of inflammation and infection in the oral cavity to systemic conditions like diabetes, leukemia, lupus, heart disease, stroke, respiratory disease and adverse pregnancy outcomes (Covington, 1996; Lux & Lavigne, 2004; Ross, 2006; Fine & Yao, 2007; Lux, 2007;). 1.4.2. Oral health and pregnancy. Pregnant women have increased oral health concerns due to hormonal changes and altered eating habits. Dietary changes include smaller but more frequent meals, different food choices, and cravings. Hormonal changes may lead to nausea, vomiting, and altered oral hygiene habits. These changes, or a combination of these changes, may result in increased plaque retention which promotes dental caries (cavities) and gingival inflammation (gingivitis). When these dental diseases are not addressed, there is increased risk of transmission of "cavity causing" bacteria to the child, progression of periodontal disease, and possible adverse pregnancy outcomes (Jeffcoat et al., 2001; Zanata et al., 2003). 4 Early childhood caries. A relationship has been demonstrated between high levels of cavity-causing bacteria in women and dental caries in their children's primary teeth (Kohler, 1984; Caufield, Cutter, & Dasayanake, 1993). The likelihood of caries is stronger when the transfer occurs early in the life of the child and the child's diet is caries-supportive (Berkowitz, 2003; Hale et al., 2003; Boggess, 2006). Early childhood caries (ECC) or severe tooth decay in children under three years of age is a serious childhood infection. Dental problems early in childhood have serious effects on the child's development of speech and language, ability to eat and thrive, nutrition, comfort, concentration and school participation (Gussy et al., 2006). In British Columbia it has been estimated that more than 5000 children under the age of four are treated annually for ECC under general anesthetic, at an estimated cost of around 10 million dollars (Association of Dental Surgeons of British Columbia, November 23, 2001). Periodontal disease. Dental caries and periodontal disease are highly prevalent in women of childbearing age from low income and minority groups (Boggess, 2006). Periodontal disease is an inflammation of the periodontium which includes the gums (gingiva) and supporting structures around the tooth. Periodontal disease is considered the most prevalent chronic disease affecting children, teenagers, adults and the elderly (Ismail, Lewis, & Dingle, 2006). The two main types of periodontal disease are gingivitis and periodontitis. Gingivitis is inflammation of the gingival tissue and is reversible. Periodontitis is inflammation of the supporting tissues of the tissue and is characterized by bleeding gums, loss of supporting bone, bad breath, and loss of tooth attachment causing tooth mobility (Wener & Lavigne, 2004). Risk factors for periodontitis include low socioeconomic status, immuno-compromised, diabetes, poor oral hygiene, smoking, lack of professional dental maintenance and genetic pre-disposition (Wener & Lavigne, 2004). Untreated gingival disease during pregnancy may advance to a severe form of periodontal disease (Steinberg, 1999). Up to 15% of the women of childbearing age have destructive periodontal disease. Pregnant women are at a greater risk to have some form of periodontal disease (Offenbacher et al., 1996; Davenport et al., 1998; Moore et al., 2001). Periodontal disease may affect 20 — 50% of pregnant women, especially those who are economically disadvantaged (Offenbacher, 2001; Xiong et al., 2006). Pre-term, low birth weight babies. Pre-term low birth weight (PTLBW) babies, that is, babies born less than 37 weeks gestation and weighing less than 2500 grams, are a universal public health concern (McGaw, 2002). Approximately 10% of all live births in North America are PTLBW babies and the cost of medical care for these children is estimated to exceed $5 billion US a year (Offenbacher et al., 1998). According to Statistics Canada, the pre-term birth rate in British Columbia was 9.5% in 2003 (British Columbia Reproductive Care Program, 2005). Pre-term low birth weight infants have a higher risk of developing chronic health problems (Horowitz, 1998). PTLBW is the major cause of neonatal death. (Wener & Lavigne, 2004). Premature and low birth weight infants are associated with heavy financial, emotional, and societal costs. Each pre-term infant requires an average of $500,000 USD for lifetime care (Lewitt et al., 1995). Although there are known risk factors for PTLBW such as multiple births, maternal age, diabetes, smoking/alcohol and substance abuse, genitor-urinary tract infections, low socioeconomic status, African American ancestry, and previous history of PTLBW, 50% of the incidence of PTLBW is unexplained (Dasanayake, 1998; Wener & Lavigne, 2004). The first report of a potential association between periodontal disease and pre- term low birth weight infants was in 1996 (Offenbacher et al., 1996). Extrapolation from this study's data suggested that periodontal disease may have accounted for 18% of the PTLBW babies (Boggess, 2006). Since this report, various non-experimental studies (observational, cross-sectional, 6 longitudinal, cohort, case-control) and experimental studies (intervention, trials) have been conducted. Now, a growing body of research supports the association of periodontal disease and PTLBW rate (Lopez, Smith, & Gutierrez, 2002; Jeffcoat et al., 2003; Lopez et al., 2005) and the concept that treatment of periodontal disease may reduce the rate of pre-term and low birth weight babies (Mitchell-Lewis et al., 2001; Lopez et al., 2002; Jeffcoat et al., 2003). However it is still unclear about whether the mechanism of the periodontal disease-associated adverse pregnancy outcome is causal or a surrogate for other mediators that lead to pre-term birth (Offenbacher et al., 1998; Lux & Lavigne, 2004). Results of randomized controlled trials support a positive correlation between periodontal disease and pre-term low birth weight babies (Mitchell-Lewis et al., 2001; Lopez et al., 2002; Jeffcoat et al., 2003; Lopez, 2005). This association is not without controversy as other research has demonstrated no association between periodontal disease and pre-term births (Davenport et al., 2002; Moore et al., 2004; Lunardelli & Peres, 2005). Lack of effect of treatment of periodontitis on PTLBW rate has also been reported (Hujoel et al., 2006; Michalowicz et al., 2006). Conflicting results among studies have been attributed to differences in study design (case control, cohort, randomized controlled studies, intervention), sample size, population diversity (socioeconomic status, race), outcome measures (pre-term or low birth weight or rates of pre-eclampsia or fetal loss), and criteria (definition of periodontal disease, clinical indices) (Scannapieco, 2003; Bobetsis, Banos, & Offenbacher, 2006; Lux, 2007; Xiong et al., 2007). Thus, caution is recommended on a "causality claim" regarding periodontal disease and pre-term low birth weight babies. Further meta-analysis of high quality randomized controlled trials is recommended (Xiong et al., 2007). Preventive dental prenatal programs. Periodontal disease and ECC are preventable or, at least, controllable oral diseases. Behavioral, educational, and chemotherapeutic interventions that decrease a woman's caries risk could improve her own oral health and reduce her risk of transmitting cavity-causing bacteria to 7 her offspring (Kohler, Andreen, & Jonsson, 1984; Gussy et al., 2006). Preventive dental prenatal programs have been highly effective in reducing these concerns probably because pregnant women are receptive to health education interventions, such as learning new behaviors and developing better habits to improve their own health and that of their expected child (Gunay et al., 1998; Gaffield et al., 2001; Gomez & Weber, 2001; Gomez, Weber, & Emilson, 2001). Research has also supported the key role mothers play in establishment of a healthy "dental home" environment and how provision of basic preventive dental care and education may help delay the onset or severity of ECC (Gunay et al., 1998; Gomez & Weber, 2001; Gomez, Weber, & Emilson, 2001; Watson et al., 2001). Although the associations between periodontal disease and PTLBW are inconclusive as well as the effect of periodontal treatment on PTLBW babies, periodontal treatment is safe for pregnant women and is especially positive for low socioeconomic minority groups (Mitchell-Lewis et al., 2001). Furthermore, in light of the relationship between oral health and general overall health, oral care should be available to these disadvantaged groups. Pregnant women are currently recommended to have periodontal examinations and appropriate preventive services when indicated (American Academy of Peridontology, 2004). Proper dental care and oral health education for pregnant women is important for the overall health of the woman, her unborn child, and eventually her newborn. Unfortunately, as Carolyn Bennett, former Minister of State for Public Health was quoted to say, Canada has a "health-care system where the mouth is not considered a part of the body" (Picard, 2004). 1.4.3. Access to dental services. Dental service in Canada. The majority of dental services in Canada is privately funded through employer- sponsored dental benefits and is provided by independent private practices. According to the 8 Canadian Dental Association, about two-thirds of Canadians have dental insurance coverage (Picard, 2004). Dental services are expensive. The decision to visit a dentist is considerably influenced by one's level of income and dental insurance coverage (Dharamsi, 2003; Boggess, 2006). Adults with dental insurance tend to visit a dentist more than those who do not have any insurance coverage (Picard, 2004). People with lower income, lower education levels, and no dental insurance tend to have more oral diseases partially as a result of their limited access to dental care (Wener & Lavigne, 2004). Publicly-funded dental services in Vancouver. Publicly-funded dental programs for adults are limited in availability and accessibility in Canada. Most publicly-funded dental programs provide services for children while others concentrate on residential care and assisted living. For adults seeking low-cost dental services in Vancouver, British Columbia, their options are limited to a few non-profit clinics, dental or dental hygiene schools, and a government-funded dental program. Unfortunately screening criteria and cost make these options unavailable or inaccessible to many. Existing public dental health programs that have a prenatal component are usually only educational in nature. Dental service and women. Women are more likely to seek dental care if they are of a higher socioeconomic status, have dental insurance or are in the care of health professionals who support oral health care (Gaffield et al., 2001). However, more women than men live in poverty. Childrearing responsibilities and caring for elderly parents often deter many women from seeking health care for themselves, especially dental services. Pregnant women have additional barriers to dental care. About 23 — 43% of pregnant women receive dental care during their pregnancies according to the Center of Disease Control's Pregnancy Risk Assessment Monitoring System (National Institute of Dental and Craniofacial Research., 2000). The timing and type of dental procedures are often influenced by the woman's 9 well-being and comfort level in the dental chair. Non-urgent dental procedures are usually delayed until after the birth of the child. Women who did not intend to become pregnant or who initiated prenatal care after the first trimester were found to be less likely than other pregnant women to seek dental care (National Institute of Dental and Craniofacial Research., 2000). Low-income pregnant women and recently immigrated pregnant women with language and economic challenges have an even increased risk of poor oral health and face great difficulty in accessing dental services. Women employed in low-paying jobs may not have dental benefits while immigrant women and refugees must overcome language barriers and the challenges of acculturation to access health services. Other deterrents to seeking dental care may include dependency on other people for transportation, child care, and translation; cost of services; and preconceived knowledge, opinions, and beliefs about dental treatment and care during pregnancy. Many of these women may be unaware of the available low-cost dental services, unable to communicate their dental concerns or unfamiliar with the importance of dental care. 1.4.4. Dental prenatal programs. Publicly-funded dental programs that have a prenatal component primarily provide oral health counseling. Prenatal programs by non-dental health providers usually disseminate oral health literature; this practice is dependent on the provider's dental knowledge and available dental resources. Unfortunately, the lack of provision of clinical dental care such as periodontal therapy and restorative dentistry increases the woman's risk to periodontal disease and increases her infant's chances of exposure to transmission of caries bacteria. The Healthiest Babies Possible Dental Program. A dental prenatal program has operated for over 20 years in one of the Vancouver Coastal Health (VCH) community health centers, the North Community Health Office (NCHO) in East Vancouver. The NCHO is the site of the only fully provincial government-funded dental 10 program, the Vancouver Community Dental Program (VCDP). The full-time priority of the VCDP is to provide prevention and treatment services to low-income Vancouver children through screening programs and in its NCHO dental clinic. The VCDP also has a prenatal program which provides both clinical hygiene care and oral health counseling. This program is limited to pregnant women referred from the Healthiest Babies Possible (HBP) program. The Healthiest Babies Possible (HBP) Program is a free, prenatal, government-funded outreach program that helps women living in Vancouver and Richmond to have healthier pregnancies and lifestyles. Women accepted into the HBP program have been assessed to be at high risk for delivering pre-term and/or low birth weight babies. Risk factors include economic and language barriers, alcohol/smoking or other types of substance abuse, isolation, maternal age, multiple births, and refugee, recent immigrant or Aboriginal status (Healthiest Babies Possible Program, 2004). The HBP program approached the VCDP at NCHO in 1986 to assist their clients who had difficulty eating nutritious foods. The VCDP currently allocates 12 clinic appointments a month to women referred by HBP staff according to VCDP screening criteria. The criteria are Vancouver residency and either self-identification of a dental concern or last dental visit over two previous years. Each woman is allowed two one-hour appointments, at no cost, for a visual dental exam, limited clinical hygiene services and oral health counseling. No radiographs are taken. If other dental services are required, a referral list of outside dental providers offering reduced costs is given along with educational handouts (infant oral care, brushing and flossing instructions). The educational handouts are available in English, Chinese, Spanish and Vietnamese. Since 1986 over 1600 HBP women have been served by this dental program commonly referred to as the HBP Dental Program (Vancouver Community Dental Program, 2004). The HBP Dental Program is the only provincially funded dental prenatal program in British Columbia that provides both clinical and educational services. Despite changes in 11 staffing, client demographics, resources, and knowledge from research, no formal assessment or evaluation of this program has ever been done other than appointment monitoring. Since the VCDP had limited resources (staff, time, and funding) to conduct an evaluation, the resident dental hygienist of the HBP Dental Program (DL) proposed an insider evaluation to enhance future program decision-making. 1.4.5. Evaluation. Description. Evaluation judges an object of interest by systematically gathering information and comparing it to a standard of acceptability (such as a historical, normative, theoretical, absolute and negotiated/compromised standard) (Ovretveit, 2002; Lovato, 2005;). Program evaluation is an investigation of a program to determine its operations and/or effects, relative to its desired objectives (Smith, 1989). The different stages in program development, from planning to implementation, involve evaluation. Evaluation activities are part of needs assessment (should a program should be planned? what type of program is required?), feasibility analysis (is this program feasible?), process evaluation (is the program being implemented as planned? can it be improved?), and outcome evaluation (what are the program results or effects?) (Herman et al., 1997). Data collected in a needs assessment often acts as the "pre-test" for the outcome evaluation; outcome evaluation is based on the goals and objectives; process evaluation is derived from program activities and implementation (Hodges & Videto, 2005). Insider or internal evaluators. An internal or insider evaluator is someone internal to the organization, an employee of the program, who evaluates the services or policies or treatments of the organization or its divisions (Ovretveit, 2002; Mathison, 2005). An estimated 50 to 75% of evaluations performed in North America are by internal evaluators (Mathison, 2005). There is growing support and 12 acknowledged benefits for insider evaluation (Klein & Johnston, 1979; Bonner & Tohurst, 2002; Shaw & Faulkner, 2006). The main advantages of having insider evaluators are their familiarity with and accessibility to the organization and program; ability to observe the implementation of evaluation recommendations; existing rapport with the staff to help reduce evaluation fear and anxiety; ability to communicate evaluation findings more frequently; and ability to empower program staff to have a better understanding of the program activities and theory (Mathison, 2005). The disadvantages of insider evaluators include research bias (systematic bias), lack of objectivity, and ambiguity in "role duality". These weaknesses however can be negated by adherence to proper evaluation methodology, systematic measurements, and having the internal evaluation process validated by an external evaluator (King, Morris, & Fitz-Gibbon, 1987; Mathison, 2005). Evaluators are also encouraged to adhere to guiding principles of systematic inquiry, competence, integrity/honesty, respect for people, and responsibilities for general and public welfare (American Evaluation Association, 2004). Evaluation guidelines. Evaluation guidelines will ensure proper evaluation design, successful indicators, appropriate criteria and methodology and negate the previously cited disadvantages of insider evaluation. "User-friendly" evaluation reference books are available to provide guidelines for evaluation analysis and to clarify evaluation terminology (Rutman, 1980; King, Morris, & Fitz- Gibbon, 1987; Smith, 1989; Wholey, Hatry, & Newcomer, 1994; Ovretveit, 2002; Mathison, 2005). The Center for Disease Control and Prevention (CDC) recommends that any evaluation should include the following six basic steps: 1) Engage stakeholders; 2) Describe the program; 3) Focus the evaluation design; 4) Gather credible evidence; 5) Justify conclusions; and 6) Ensure use and share lessons learned (Milstein, 1999). 13 Participation of stakeholders (persons or organization with a vested interest in the evaluation and its results) is an essential component to ensure their perspectives, objectives, and concerns are addressed. A program description should have enough detail to allow for proper interpretation of results. A detailed description can have an impact on the success of a program by identifying what worked and the areas needing improvement. It can also serve as a historical record as well as a picture of a poor or a good program (King, Morris, & Fitz-Gibbon, 1987). Program description includes objectives, activities, need, resources, context, stage of development, logic model and expected effects (Milstein, 1999). Supporting data from different sources can be used to confirm the accuracy of the program description. For instance, field observations can confirm descriptions of daily process activities. Determining the evaluation design depends on the purpose, uses, users l , questions/concerns, methods, and agreements (Milstein, 1999). Evaluation designs can be quantitative, qualitative or a mixture of both approaches. A quantitative approach is a goal-based experimental design, and has standards of acceptability as well as internal and external validity; a qualitative evaluation develops and outlines program elements, answers the "how" and "why", generates theory, broadens the observational field for effects and can serve as an adjunct to quantitative designs (Lovato, 2005). Employing both qualitative and quantitative information helps to balance the evidence (Green & Caracelli, 1997). The type of evaluation is dependent on the objective or purpose of the stakeholders. If the purpose is program improvement, the evaluation can be a formative (ongoing) process as well as include an outcome evaluation to provide evidence to support recommendations and demonstrate benefits. Common types of evaluations include formative, summative, process, impact, and outcome (Ovretveit, 2002; Hodges & Videto, 2005). No design or method is The individuals whom the evaluation is conducted for; in position of decision-making or authority. 14 considered better than another since its selection is based on the concerns of the "users" of the evaluation (Milstein, 1999). However mixed methodologies are generally more effective since each method has its own limitation and bias (Frechtling & Sharp, 1997). Information collected for evaluation needs to be seen as credible to strengthen conclusions and recommendations. Credible evidence is more likely to be accepted and to stimulate action (Patton, 1997). The use of multiple procedures for data collection and analysis as well as the participation of stakeholders can enhance credibility, since all data types have limitations (Milstein, 1999). The issue of credibility usually affects sources (observations, people, documents), indicators (measures of program activities, effects), quantity (amount of evidence), quality (design, data collection, coding, controls, personnel), and logistics (timing, methods, and infrastructure in gathering evidence). Conclusions of the evaluation are justified when they are linked to the evidence gathered, the standards of the stakeholders, the analysis/synthesis/interpretation of the findings, judgments, and recommendations (Milstein, 1999). Efforts then need to be taken to ensure the use of the evaluation. These efforts would involve a clear evaluation design, preparation for the eventual use of the results, feedback, active follow-up, and appropriate dissemination of report to the audience (Milstein, 1999). The recommended standards for effective evaluation such as utility, feasibility, propriety, and accuracy apply to all the six CDC evaluation steps previously-mentioned (Milstein, 1999). Any program can be evaluated but the contribution of the evaluation to the program is another issue (Wholey et al., 1994). Evaluations should be relevant, practical and feasible as well as be able to assess results and identify ways to improve the program (Wholey et al., 1994). The four problems which usually arise in evaluations are: 1) disagreement between evaluators and users about the goals, objectives and performance indicators resulting in irrelevant policy and management decisions; 2) unrealistic goals and objectives with given existing resources and activities resulting in the need to change the program; 3) unavailable data leading to inconclusive 15 evaluation results; and 4) lack of feasibility to change the program based on the evaluation leading to evaluations which are useless (Wholey et al., 1994). These problems can be reduced by first of all conducting an evaluability assessment. Evaluability assessment (EA). An evaluability assessment (EA) is a systematic process which describes the program structure (i.e. objectives, logic, activities, and performance indicators) and analyzes plausibility and feasibility of the structure for achieving objectives, its suitability for intensive evaluation, and its acceptability to program managers, policy makers, and program operators (Smith, 1989). Plausibility refers to whether the program has the necessary and sufficient conditions to succeed. Feasibility refers to achievability. An EA can clarify program intent, explore program reality, identify areas of improvement, determine if and where evaluation is warranted and identify feasible evaluation designs. The two primary outcomes of an EA are identification of stakeholder awareness, interest, and objectives for the program and a description of a program theory (Mathison, 2005). Program theory is often graphically communicated through a "logic model". A logic model is a type of visual schematic that illustrates the relationships between contextual factors and program inputs and outcomes (Lovato, 2006). It is similar to hypothesis-testing in basic science research and describes "the nuts and bolts of the program". The main program is defined and factors influencing the program are understood. The basic components of a logic model are resources, program activities, outputs, outcomes and impact (W.K. Kellogg Foundation, 2004). Resources or inputs are what go into the program to make it work: financial, human, community and available organizational resources. Activities are the interventions, tools, actions that would elicit the intended results of the program. Outputs are the products of the activities, such as types, targets or levels of services delivered. Outcomes are the changes in knowledge, skills, behavior in the participants. They can be short-term (within one to three years) or long term 16 (four to six years). Impact is the primary intended or unintended change within seven to ten years. The logic model is based on research, key background information, personal and professional experience, future projections and sources. It is constantly revised during the evaluation period. Several frameworks are available to guide an EA. A general guideline consists of five phases: 1) Connect stakeholders; 2) Describe program; 3) Assess program circumstances; 4) Assess organizational climate; and 5) Determine evaluability (Lovato, 2005). Other frameworks outline the EA steps in greater detail: 1) Determine purpose, secure commitment and identify work group members; 2) Define program boundaries to be studied; 3) Identify and analyze program documents; 4) Develop and clarify program theory; 5) Identify and interview stakeholders; 6) Describe stakeholder perceptions of program by analyzing and summarizing interviews with them; 7) Identify both common understandings and major differences between stakeholders in their perception of the program and its implementation; 8) Determine plausibility of program model; and 9) Draw conclusions and make recommendations (Smith, 1989). One single EA framework may not have sufficient information about its methods or concepts to guide an inexperienced evaluator; therefore a combination of various frameworks is often helpful. Most evaluation literature recommend a thorough program description and identification of program objectives; however, conducting an EA as a separate task helps evaluators avoid making two types of error: a) measuring something that is not there or b) measuring something that is of little value to the decision makers (Scanlon et al., 1979). The EA is, in essence, a "pre- evaluation" which provides stakeholders with preliminary information about the program in a shorter time and determines what areas, if any, are feasible for future in-depth evaluation. For the personnel involved, an EA offers a "safe space" for critical examination and reduces evaluation anxiety because of its iterative nature and program development focus (Thompson & McClintock, 2000; Lovato, 2006). An EA is particularly warranted when a program has 17 unapparent results, unclear criteria and/or objectives, and dispersed management responsibilities. It focuses on whether goals are appropriate, not on whether the program is meeting its goals. An EA is not useful when the purpose of the evaluation is accountability rather than program improvement and/or when evaluation criteria are already identified, that is, objectives, goals and performance indicators are known. 1.4.6. Evaluations of dental prenatal community programs. An electronic search for the existence of evaluations of similar public dental prenatal programs was performed on published and unpublished sources. The purpose of this search, limited to English language and human subjects, was to determine a feasible evaluation design, areas of evaluation (oral health status, knowledge, behavior, program implementation, outcomes, and cost-benefit), and methodological criteria to evaluate the existing HBP Dental Program. In the review of the literature, the following definitions were used. A program is defined as a set of activities and resources, typically under the direction of a single manager or management team, directed toward one or more common goals (Wholey et al., 1994). A public health or community program is a program that is government-funded and not a private practice. A research study is an intervention study which tests a hypothesis and involves evaluative outcomes. A research study of a program is a study where the intervention is the program. It may involve developing and implementing a program for the purpose of testing a hypothesis and usually ceases when the research is completed. A program evaluation is an investigation into a program to determine its operations and/or effects, relative to its desired objectives, to enhance program decision-making (Smith, 1989). Published literature. The review found a limited number of published papers on public dental prenatal program evaluations. The terms "study" and "program" were often used indiscriminately in 18 databases, therefore resulting in many articles on research studies rather than existing programs. Usually the studies only assessed clinical outcomes (Gunay et al., 1998; Gomez & Weber, 2001; Gomez, Weber, & Emilson, 2001). When the search was expanded to include all evaluations of public dental programs, many articles either discuss the topic of "evaluation" or were in reality, research studies. Research studies of a program's development and implementation were found more often than evaluations of existing public dental programs. Eight articles which professed to be evaluations or include program evaluation activities were selected for this review (Petersen, 1989; Petersen & Nortov, 1994; Watson et al., 2001; Harrison & Wong, 2003; Larson, 2003; Brown, Canham, & Cureton, 2005; Cruz et al., 2005; Hyde, Weintraub, & Satariano, Spring 2005). Appendix A provides a synopsis of these articles according to Ovretveit's summary framework of an evaluation (Ovretveit, 2002). Each article was then reviewed according to the CDC recommendations for evaluation strengths and weaknesses. Process and outcome evaluations were conducted on a dental preventive program (clinical and educational) developed for Danish chocolate factory workers (Petersen, 1989). The main strengths of the program were the valuable inclusion of the management and stakeholders in the program activities, the reinforcement strategy, and frequency of preventive visits. The main evaluation strength was the clear description of the program rationale and outcome evaluation method, stakeholders' participation, and presentation of results. However the evaluation lacked clarity and detail on the description of the participants, the educational program, process evaluation activities and clinical indicators. No examples of the interview guide or the questionnaire used were provided nor was there any discussion of dissemination and use of findings. More supporting data concerning implementation activities may have helped to corroborate the perspectives from various levels of factory personnel. This missing information 19 may have been due to poor incentive for program improvement since this was presumably a one- time research study program. An oral health education program for Latino immigrant parents was developed to measure oral health knowledge and behavior as well as to provide an oral health program framework for nurses to use (Brown, Canham, & Cureton, 2005). This program was included because of its culturally sensitive methodology, and its inclusion of instruments for different ethnic groups in its communication, activities, and setting. The strengths of the evaluation were the questionnaire and participation incentives. The weaknesses in the evaluation were a lack of information regarding stakeholders' involvement, insufficient program description, rationale and methods, and a questionable measurement tool, target population, and conclusions. Although the questionnaire required simple answers (yes/no, true/false), its readability level, detail, and length were not suitable for this group of low literacy mothers. Furthermore the authors did not offer any conclusion about the success of the program aside from its potential for use as an oral health framework. Given these weaknesses and the lack of process evaluation, the potential of this program evaluation as a viable framework for evaluation was questionable. An Early Childhood Caries Prevention and Plan Program that took place in Palau2 began with five clearly stated objectives (Larson, 2003). Strengths of the program included the integration of dental services within all prenatal and postnatal care programs and the capacity building of other health professionals. The strengths of the evaluation include linkages of plans and activities with objectives; proposals for evaluation activity goals, e.g. an oral health surveillance system to accurately monitor progress for future routine evaluations; and provision of examples for potential evaluation measures/indicators and data collection. These examples were a modified screening survey to allow measurement of key percentage indicators; a future survey for mothers to assess oral health knowledge, practices and beliefs; and an Oral Health 2 Located near the Philippines and Vietnam 20 Tracking Card (take home) and program process statistics to facilitate data gathering. However, because the program was only in the early stages of planning, little detail on methodologies, sampling and evaluation were provided. A community participatory oral health promotion program was implemented in an inner- city Latino Community using the Precede-Proceed Model to guide the planning and process evaluation (Watson et al., 2001). The program's strengths were its use of existing resources, massive community initiatives, positive community impact, and the continuation of oral health interest and activities. Strengths of the evaluation were its description of the needs assessment, process evaluation activities; formation of steering committee 3 ; application of the Precede- Proceed Model; dissemination and use of project findings with active follow-up. The continuations of various aspects of the program after the initial study period emphasize the positive results. Limitations include unclear users of the evaluation; vague success indicators; insufficient detail on certain program activities, process indicators, and areas of program improvement; and minimal program activities focused on objectives or the target audience. It was not a true community participatory program because it was researcher-controlled and there was lack of representation in the steering committee from the target audience. A demonstration project for an oral health promotion program for inner-city Vietnamese preschool children was implemented and evaluated (Harrison & Wong, 2003). Program strengths include its efforts to tailor the program to the Vietnamese mothers, use of community initiatives, interdisciplinary collaboration, and provision of enabling tools (feeding cup, soother, and toothbrush). Evaluation strengths were its clear description of rationale, project phases and inclusion of some general process evaluation activities/improvements over its seven year time period. Limitations were insufficient participant description; unclear users of the evaluation and outcome evaluation activities; and lack of information on the dissemination and use of evaluation 3 A group of individuals responsible for the planning and implementation of the program of interest. 21 findings. The usefulness of this article for evaluation criteria was limited since the methodology and information about data collection tools were discussed in an earlier unretrieved article (Harrison et al., 1997). A community-based culturally appropriate oral health promotion program was developed and implemented to meet the needs of low income immigrant pregnant women in New York City (Cruz et al., 2005). The program's strengths were its appropriately translated materials and bilingual interviewer, reinforcement gifts, and creation of new educational materials. However it may not have been culturally appropriate since it did not involve the community, nor did it consider differing learning styles and behavior. The strengths of the evaluation include a description of the results of the needs assessment, validity and reliability controls and examples of survey questions. Weaknesses were general lack of evaluation activities especially regarding program implementation; vague success indicators; unclear users; and insufficient detail regarding Phase 2 activities/participants, pre/post test, data collection methods and analysis. A summative evaluation of an existing welfare dental program was performed using a Donabedian program evaluation framework for structure, process and outcome measures (Hyde, Weintraub, & Satariano, 2005). The strengths of the evaluation were the use of a framework to guide evaluation activities; detailed description of program facilities, personnel, treatment and orientation; and informative table presentations. Weaknesses were insufficient information on the Donabedian framework, the data collection method for program description, no identification of areas of improvement, and no apparent dissemination of findings. An outcome evaluation of knowledge, attitudes, behavior and self assessments of dental health was performed on an existing dental program for old-age pensioners in Denmark using a follow-up design with interviews, questionnaires, and clinical data (Petersen & Nortov, 1994). Strengths of the evaluation were the informative tables, description of results, and clinical data supporting observations. Weaknesses included insufficient or lack of information about 22 evaluation users, recruitment process, participant description, value criteria, success indicators, and dissemination and use of findings. A process evaluation would have been helpful to support proper interpretation of results. Limitations in the review of published literature. This review is likely not exhaustive. Success and limitations of this search were influenced by the researcher's knowledge about evaluation and her familiarity with search engines; accessibility of some of the articles; and the ambiguity of some of the abstracts. Most of the articles retrieved used different methodologies, approaches, and indicators for evaluation and had insufficient detail in their designs, program/participant description, objectives; difficulties with reliable measurement indicators; unclear users; and lack of information about the dissemination and use of findings. Although most of the articles were research projects and not expected to fully meet the CDC recommendations for an evaluation, their weaknesses and strengths provided new insight into the challenges of program evaluation. Insights include the benefit of a program to be culturally sensitive in program planning and evaluation; the need to incorporate indicators for future routine evaluations; and the importance of clear examples or descriptions of activities, results (tables and charts), and measurements (questionnaires, interviews). Three of the articles also included an evaluation of the program from a participant and staff perspective (Petersen, 1989; Harrison & Wong, 2003; Hyde, Weintraub, & Satariano, 2005). This is important especially if the program is community-based. The application of the Precede-Proceed or Donabedian Model demonstrated the usefulness of a framework to guide the evaluation. Overall, despite a wide assortment and creative combinations of keywords, no evaluation of a public dental prenatal program similar to the HBP Dental Program was discovered. The lack of success to locate published evaluation literature about public dental prenatal programs may be a result of the following: 1) most dental prenatal programs are mainly educational, not 23 clinical and educational; 2) most "evaluation" articles of dental prenatal programs are focused on clinical effectiveness, not on program evaluation; 3) research studies of programs are concerned with testing new programs, not evaluation of existing programs. Models for program planning and evaluation. The Donabedian Model is a framework which evaluates health care quality by examining structure, process, and outcomes. Its assumption is that structures affect processes which then affect outcomes (Mitchell, Ferketch, & Jennings, 1998). Green's Precede-Proceed Model however provides a broader perspective of the program than the Donabedian's Model and helps identify the factors which affect the program. The Precede-Proceed4 model is a health planning framework which incorporates multiple factors affecting health (social, political, and economical) (Green & Kreuter, 1999). It consists of nine basic phases to help plan, implement, and evaluate programs. The model directs the types of information to collect when planning and implementing a program; identifies specific factors (predisposing, enabling, reinforcing) which affect the program outcome; provides a process evaluation framework to identify areas of improvement and targets for an intervention to promote oral health and dental utilization (Green & Kreuter, 1999). Examination of the enabling and reinforcing factors before and after the intervention provides insight about the success or failure of the program. Awareness of predisposing behaviors, attitudes and knowledge of the participants and evaluation of program processes identifies factors which enable, reinforce or hinder the success and failure of these programs. Unpublished literature. A search for unpublished literature of public dental prenatal programs and evaluations involved informal interviews, telephone conversations, and email correspondence with various 4 PRECEDE is an acronym for predisposing, reinforcing, and enabling constructs in educational/ecological diagnosis and evaluation while PROCEED refers to policy, regulatory and organizational constructs in educational and environmental development (Green & Kreuter, 1999). 24 professionals. The findings from unpublished sources reveal that 1) the evaluation component of public dental health programs was limited; and that 2) publicly-funded dental programs in British Columbia were focused on children. Whenever a prenatal component was present in a public dental program, it was purely educational in nature and directed to either the pregnant women or the dental and nursing community and schools involved with prenatal care. Two other Canadian dental prenatal programs similar to the HBP dental were identified. These two programs, located in Calgary, Alberta and Toronto, Ontario also provide restorative services but unfortunately no evaluation had been done except for basic appointment monitoring. Limitations in the review of unpublished literature. The lack of evaluation of existing public dental prenatal programs may occur for a variety of reasons. First, the status of oral health in the Canadian health care system is reflected in a limited number of government-funded dental programs. The small number of Canadian programs will affect the range of target populations served. If there is a prenatal component in a public dental program, it may not be a high priority; thus, an evaluation would not be considered. Secondly, dental prenatal programs with both clinical and educational components are few therefore a specific evaluation framework has not been created. Thirdly, evaluations of existing public dental programs may not be published due to the privacy concerns of the program. Other reasons for lack of evaluation may include scarce resources in staff, time, and finances; heavy work commitments; organizational structure; leadership focus; poor incentives for evaluation; and minimal evaluation and research knowledge and experience. A community dental program needs to benefit the community of interest. Hence, an evaluation of a community program needs to be evaluated by the community it serves as well as the people involved in the program implementation. To obtain a complete picture of the program, it is necessary to examine the program from various perspectives such as the clients, 25 the referring staff, and the operating staff as well as at different time periods in the history of the program (before, during and after the program). Reliable and valid evidence from an evaluation can improve program effectiveness, provide accountability, and promote the value of public dental prenatal interventions. Only with evidence-based data are decision-makers able to make informed decisions and recommendations regarding public dental prenatal interventions. 26 2. Methods 2.1. Overview This research was approved by the University of British Columbia Behavioral and Research Ethics Board (BREB) and the Vancouver Coastal Health Research Institute (VCHRI) (Appendix B). The program evaluation which utilized a quantitative and qualitative approach with mixed methodologies was divided into two major phases. Phase One included three parts: an evaluation assessment, a descriptive evaluation, and a process evaluation using interviews, field observations, a retrospective and current chart review and appointment monitoring. Phase Two was a short- and medium-term s outcome evaluation with a convenience sample of women seen over a one year period. Data was collected by questionnaires, interviews, appointment monitoring, tooth-brushing observations, chart review, and recording of clinical indices. A timeline of these evaluation activities is detailed in Figure 1. 5 "Short-term" will refer to time interval from the 1st prenatal visit (Visit #1) to the 2nd prenatal visit (Visit #2); "Medium-term" will refer to time interval from the 1st prenatal visit (Visit #1) to the visit after the baby is born (Visit #3). 27 2005 F e b M a r A r M a J^J u u A g S e 0 t N 0 D e c Activity StartDate End Date J a n Pilot study of 3rd visit feasibility Feb-05 Jun-05 Pretest of measurements Feb-05 Sep-05 Study recruitment of HBP clients Oct-05 Oct-06 1st EA meeting with main stakeholders May-05 UBC BREB and VCHRI approvals Jul-05 UBC BREB amendment Sep-05 Meeting #1 with VCDP staff Oct-05 Meeting #1 with HBP staff Nov-05 Figure 1. Timeline of evaluation activities 2005 -2007 Activity StartDate End Date 2006 J a n F e b a r MA pa r M J n uuue J 1 A : S I O c t N oe V D C Study recruitment of HBP clients Oct 05 Oct 06 Congratulatory cards sent Jun-06 Apr-07 3rd visits (clinic/phone) Aug-06 Oct-07 Activity StartDate End Date 2007 J a nbr F e M a A pa r M y J u n J u 1 AS u gpt e O c N o v D e c Congratulatory cards sent Jun-06 Apr-07 3rd visits (clinic/phone) Aug-06 Oct-07 Meeting #2 with VCDP staff Jun-07 Meeting #2 with HBP staff Jun-07 28 All data was entered into an EXCEL (Version 2002) database and analyzed with the Statistical Package for Social Sciences (SPSS) statistical analysis software (Version 15). Reliability testing and triangulation of questionnaires, interviews, and indices measures were done. The insider evaluator was DL, a dental hygienist and graduate student conducting the program evaluation as part of her graduate thesis. Her field observations from her experience as the resident HBP clinical dental hygienist for the past four years were also considered. To ensure that this evaluation will ultimately benefit the HBP community and the HBP Dental Program, attempts were made to include evaluation perspectives from a variety of stakeholders (clients, dental staff, and HBP staff) over a period of three years. 2.2. Phase One 2.2.1 Evaluability assessment (EA). Overview and objectives. Unfortunately, the Healthiest Babies Possible (HBP) Dental Program had limited "historical" data available other than appointment monitoring. Therefore, an EA was needed to clarify program goals, activities and organizational structure as well as determine a feasible evaluation plan. The objectives of the EA were to: 1) determine the evaluation working group, the tasks and scope of evaluation; 2) identify stakeholders 6 and program objectives; 3) describe program theory; 4) describe existing program circumstances; and 5) describe organizational climate. The two EA frameworks cited previously were applied to this program evaluation (Smith, 1989; Lovato, 2005). The primary methods used were semi-structured interviews (individual or 6 Someone who has a vested interest in program (e.g. involved in operations, affected or served by program; users). 29 small groups) either in person, by email, or by telephone; retrospective chart reviews; and field observations (DL). Main stakeholders. The main stakeholders were identified as those personnel with decision-making authority and/or with strong involvement with the HBP Dental Program clientele. A meeting was organized in summer 2005 at the North Community Health Office (NCHO) site with the main stakeholders to discuss the objectives and scope of the evaluation; the rationale for the EA; their role in the evaluation; the program's boundaries and documents; receptivity of organizational climate to evaluation and to program; and to identify other stakeholders associated with the HBP Dental Program. Permission was sought from various administrators in the community health centers and regional Health Authority for data access and evaluation. Program theory. A description of program theory was based on interviews with the main stakeholders to determine the program inputs, activities, outputs, outcomes, and impact. History, funding, activities, organizational structure, roles and responsibilities, and relationships between the dental and prenatal programs as well as expectations and objectives for the HBP Dental Program were also discussed. Other dental stakeholders involved with the program were interviewed as to their roles and responsibilities in relation to the program, associated challenges, and their perception of a HBP Dental Program evaluation. Roles and responsibilities were summarized and confirmed either verbally, in written form or by email by the various stakeholders. Higher organizational level stakeholders' roles were confirmed by main stakeholders. Field observations of program activities and objectives were included. Existing program circumstances The program circumstances, that is, the "day-to-day reality" of the program, were described by questioning the dental personnel directly involved with the program. Existing 30 methods of recording program and client information were examined. Program documents (medical histories, dental charts, client information forms, referral forms, and appointment statistics) were manually retrieved and reviewed for determination of type and amount of data collection mechanisms, clarity, and potential data indicators. A description of the HBP clients seen at the NCHO dental clinic was determined through interviews with the main stakeholders, and dental receptionists, field observations, and a chart review of client records. Observations about the infrastructure of the program, that is, type of data collection systems (manual or computerized systems, filing system) were also noted. Organizational climate An assessment of the organizational climate around the HBP Dental Program and the evaluation was elicited by interviews with the stakeholders and by personal field observations of the evaluator. This assessment meant examining the program resources, capacity, barriers, and support for evaluation. All perceptions of the program and implementation were summarized and compared. Any major differences were explored with further follow-up interviews. Plausibility The plausibility of this program was then examined in regards to clarity of goals and activities; identifiable success indicators and sources; sufficiency of activities; and adequacy of resources to implement these activities. Evaluability was determined after the above phases were done. If the program was ready to be evaluated, the next step would be to explore what type of evaluation was needed; what program areas to evaluate; and the evaluation products desired. If the program was not ready to be evaluated, then recommendations would be made to either delay the evaluation until changes are done or not do an evaluation. Refers to whether or not the program has the necessary and sufficient conditions to succeed (Smith, 1989) 31 Summaries with recommendations were presented to the main stakeholders by DL in an oral presentation with printed handouts. Types of evaluations were then discussed among the main stakeholders. 2.2.2. Descriptive evaluation. Overview and objectives. Since the evaluability of the HBP Dental Program activities, theory, and existing infrastructure will be examined as part of the EA, the objectives of the descriptive evaluation are simply to describe the HBP Dental Program's history, its physical location, organizational structure, personnel and funding as well as the clientele and their oral health needs. Both qualitative and quantitative approaches were used with mixed methods such as semi-structured interviews, field observations, chart reviews. Handwritten interview notes were transferred to Microsoft Word 2002 while chart data was manually retrieved and entered into Microsoft Excel 2002. Simple descriptive frequencies were performed to better understand the data collected. Program. Semi-structured interviews with the main stakeholders (individually or in small groups) were conducted about the history, organizational structure, personnel, and funding. Efforts were made to contact the previous HBP Dental Program hygienists through professional contacts. Supporting data was elicited from HBP Dental Program records dating from 1986, a search of the regional Health Authority website, and interviews with former personnel or HBP staff. Observations of the physical environment, resources, program activities, and data collection mechanisms were also included to facilitate understanding of the program. 32 Clientele. A retrospective review of charts from January 2004 - December 2005 was done to determine the clients' demographics and oral health status and needs. All the medical histories, dental charts, HBP dental referral forms and client information forms were manually retrieved, reviewed and then the extracted data was entered into the Excel database. Nominal, ordinal, and interval scales were used to categorize the demographical data. Attempts were made to retrieve missing information 1) by email, personal, and phone requests to either the HBP coordinator or dental staff and 2) by examining other related dental clinic records. Data from these clients was analyzed for descriptive frequencies. Preliminary frequencies were reported to the main stakeholders to provide feedback and to help sustain their involvement and enthusiasm in program evaluation. This "extra EA step" helped clarify missing data and determine potential data indicators 8 and prepare the program for future descriptive and outcome evaluations. 2.2.3. Process evaluation. Overview and objectives. The examination of the logic model by the main stakeholders identified discrepancies between program activities and outputs. Intended activities often did not match actual activities. The objective of the process evaluation was to determine the extent to which the HBP Dental Program activities were being implemented as intended. The process evaluation used a combination of quantitative and qualitative methods which involved retrospective and current chart reviews, appointment monitoring, field observations, questionnaires, and semi-structured interviews (individual/small groups). Appendix C summarizes the measurement methods used for the process evaluation indicators and the rationale for the choice of indicator. Questions used during the specific interviews are in 8 Measurements of interest to the evaluation 33 Appendix D. Attempts were made to evaluate the program from the perspectives of as many stakeholders as possible, including, the HBP staff who refer women to the dental program, the women seen by the dental program, and the dental staff. The handwritten notes from the interviews and observations were transferred into Microsoft Word 2002; data from questionnaires, charts, and appointment monitoring records was entered to an Excel 2002 database. Descriptive frequencies and cross-tabulations were performed with SPSS (Version 15). Interviews and questionnaires. Semi-structured interviews (personal, telephone or email) were conducted with HBP staff, dental receptionists, main dental stakeholders (VCDP coordinator and clinic supervisor), and outside dental providers listed on the VCDP referral list. Questionnaires were given to HBP staff. Two meetings with the HBP staff and DL were held to 1) understand the screening process of the HBP Dental Program; 2) identify barriers to referrals to HBP Dental Program; 3) enhance mutual understanding of the program; and 4) increase communication between the HBP prenatal program and HBP Dental Program. The purpose of the first meeting (2005) was to "re- familiarize" the HBP staff with the HBP Dental Program and build confidence in how their referred clients were managed. The HBP staff was asked to identify concerns with the referral process and to complete a questionnaire of eight questions to assess their oral health knowledge (Appendix E). Follow-up email contact with DL was encouraged. In a subsequent meeting (2007), an interactive questionnaire displayed on a large poster (Appendix F) was used to obtain feedback about the referral process, client satisfaction with the program, and communication issues. Dental receptionists were asked for feedback regarding activities related to the referral process, appointment scheduling, and communication between the two programs. Main dental 34 stakeholders were interviewed specifically about which oral health topics should be discussed at a HBP dental appointment. Outside dental providers were contacted in December 2005 and July 2006 to verify contact information and description of their low-cost services available to HBP women. Hours of operation, languages spoken, length of waiting list, and fees were clarified. Field observations were added about new outside dental providers. Feedback from the entire group of dental staff about concerns or suggestions regarding the HBP Dental Program and this evaluation was encouraged during the VCDP staff meeting. A designated sheet in the HBP dental folder at the reception area was available for suggestions or feedback. Chart reviews and appointment monitoring Chart reviews were done both retrospectively and concurrently on accessible appointment monitoring records, attendance day sheets, and client forms. Client forms (dental charts, medical histories, and HBP referral forms with client information) prior to 2003 were unavailable since they had been archived. Structural renovations to the dental reception area, upgrades to the computerized information database, and a new filing and charting system affected the retrieval of charts. Information from HBP referral forms (2004-2006) and dental charts (2004-2005) was manually reviewed and entered into the Excel database and analyzed for time intervals and patterns. Attempts were made to retrieve missing referral forms or information either by email, telephone or direct contact with HBP or VCDP staff; manual cross-referencing with other program documents; or a repeated search through filing cabinets and drawers. Similar information found on the referral forms was cross-referenced with dental clinic day sheets and appointment monitoring records to confirm accuracy. Accessible medical history forms (2004- 2006) and HBP referral forms were also cross-referenced manually to assess completeness of information. 35 Appointment monitoring records from 1986 to 2006 were reviewed to determine the annual number of clients seen in the HBP dental program and number of available or broken appointments in those years. Records from 2000 to 2006 were reorganized into an Excel database and compared for patterns. These records (2000-2006) were cross-checked with other program documents (appointment logs and day sheets) to confirm accuracy of information as well as to retrieve any missing data. Supporting documents for the earlier years were not available. Field observations. Field observations by DL were included concerning the referral process, program activities, appointment record-keeping, inter-program communication, relevancy of chart information, and feedback on educational activities, pamphlets and referral list. Presence of any factors to reinforce and enable the HBP Dental Program activities and client's accessibility to dental care were considered. 2.3. Phase Two 2.3.1. Outcome evaluation. Objectives. The objective of the outcome evaluation was to determine if the HBP Dental Program improved the participating women's oral health status and their knowledge, awareness and behaviors in relation to their oral health and that of their children. Design. A repeated measures prospective research study was undertaken over a one year period to evaluate outcomes. A before/after design was chosen because a comparison with a control group was not feasible given the uniqueness of the program and its clientele. All customary clinical procedures were followed since this was an evaluation of an existing program. Activities 36 specific to the outcome evaluation research included a research consent form, qualitative measurements (questionnaires, tooth-brushing observations, interviews), quantitative measurements (questionnaires, chart reviews, appointment monitoring, and clinical indices), and a 3 rd postnatal visit. Measurable outcomes included knowledge, behavior, awareness, clinical indices, and program satisfaction. Short-term outcome results were derived by comparing Visit #1 and Visit #2 (prenatal visits). Medium-term results were derived from a comparison of data on Visit #1 (prenatal) and Visit #3 (postnatal). Sampling The sampling frame was a non-probability convenience sample of women who were Vancouver residents referred from the HBP Program to the North Community Health Office (NCHO) Dental Clinic HBP Dental Program and attended their first visit from October 2005 to October 2006. Inclusion "criteria" were referral to the HBP Dental Program, consent to evaluation encounters (interviews, questionnaires, tooth-brushing observations and clinical indices), and less than 35 weeks gestation. A sample size for this exploratory study was set at 40, based on DL's experience with the HBP Dental Program clientele, the number of HBP dental clients seen in the past five years (an average of 50 clients/year from 1999-2003), known barriers encountered in the program (language and lack of appointments available), and feasibility of evaluation. Pilot study. A pilot study was conducted to determine the feasibility of postnatal recruitment, to finalize the inclusion criteria, and to pre-test the measurement indicators. Women seen from February to June 2005 were asked if they were interested in a postnatal visit. Almost 78 percent (15/19) expressed interest and 33% (5/15) returned for a postnatal visit six to 15 months after their first visit. 37 Recruitment and evaluation activities. The flow of activity of the outcome evaluation is outlined in Figure 2. 38 Figure 2. Flowchart of Outcome evaluation activities HBP Vancouver pregnant women referred to HBP Dental Program @ NCHO dental clinic 1 Visit #1 @ dental clinic Receptionist — consent form, medical history, questionnaire #1 Dentist — oral exam, treatment recommendations Dental hygienist — initial periodontal treatment; oral health education & instruction; handouts; indices; interview; observations Visit #2 @ dental clinic* Receptionist — questionnaire #2 Dental hygienist - ongoing periodontal treatment; review of oral health education; indices; interview; observations 1 Birth of baby Congratulations card sent; 3rd clinic appointment booked* Visit #3 @ dental clinic Receptionist — questionnaire #3 Dentist — exam; treatment recommendations Dental hygienist — periodontal treatment; review of oral health education; indices; interview; observations * If woman unable to attend dental clinic visit, questionnaire and interview is administered by telephone. 39 Over a one-year period, each referred woman who presented for an appointment was approached for consent (Appendix G) by the dental receptionist. Participants were asked to return for a postnatal visit (dental examination, clinical hygiene services and oral health counseling) in addition to the customary two prenatal visits, which included a dental examination, oral health education, clinical periodontal therapy (with hand instruments and/or P5 scaler), and oral hygiene instructions. Prophylaxis, fluoride (either with a fluoride varnish or gel in tray) or anti-bacterial post-rinse were applied as needed. Appropriate educational handouts and a list of dental providers with reduced dental fees were distributed accordingly. All three visits involved questionnaires (oral health knowledge), clinical indices (oral health status), observations of tooth-brushing, and interviews (oral health concerns and behavior habits). If a client could not attend for a follow-up, interviews and questionnaires were administered by telephone. The results of the earlier pilot study revealed that contact with clients was easily lost. Measures to enhance participants' retention for the postnatal visit included mail out of birth congratulatory cards and telephone follow-up reminders to book 3 rd appointments. Included with each card was a copy of the infant oral care educational handout either in English, Punjabi, Vietnamese, Chinese or Spanish previously given at Visit #1 (prenatal). A reminder to book the postnatal appointment was written in the card. A "thank you" gift package was given to each returning participant at their postnatal visit. The package consisted of one child's toothbrush, three bibs, three baby face cloths, a thank you note, and a reminder to bring child for a one year old dental check up. Outcome indicators. The behaviors and oral hygiene skills measured and compared at each visit were brushing frequency, tooth surfaces brushed, dental appointments for other children, and attendance at HBP dental visits. At the 3 rd visit, performance of infant oral care was also recorded. Knowledge 40 outcome indicators were correct answers on the questionnaires at Visit #2 and #3. Awareness was measured by clients' self-reports of bleeding upon brushing, Clinical outcome indicators were changes in gingival, calculus, plaque, bleeding upon probing and probing depths. Outcomes related to satisfaction with the program included satisfaction with referral to the program and the desire to return for future visits. Chart review. General and dental demographics of the participants as well as their oral health concerns and needs were gathered from the medical history form (Appendix H), dental chart (Appendix I), HBP referral form (Appendix J), and interviews (Appendix K and L). Questionnaires. Upon arrival at the dental clinic, each participant was given a questionnaire to complete (Appendix E, M, and N). If the participant was unable to attend the subsequent clinic visit, the questionnaire was administered over the telephone. The questionnaires, previously reviewed by HBP and VCDP staff for clarity and comprehension, had six to nine questions at a fourth grade readability level determined by Flesch-Kincaid test. All questionnaires were pilot-tested on previous HBP dental clients and revised for clarity and client readability. Topics included knowledge concerning the effect of pregnancy on gums, baby's health, future cavity risk; infant oral care; cavity-causing food; recommended tooth-brushing frequency; sleeping with the bottle; and effect of baby teeth on adult teeth. A selection of these questions was repeated either at the —nd2 or 3 rd visit. Other questions explored client's experience with the clinical treatment and her rating on the helpfulness of the verbal/written education content, language preference, importance of oral health, program satisfaction, suggestions for the program, and complaints. Interviews (self-reports). At each appointment women were specifically asked to report their oral health concerns (bleeding, pain, sensitivity, and other concerns), oral hygiene habits (brushing frequency), 41 current professional dental care for other children (if applicable), infant oral care (at postnatal Visit #3), and follow-up of recommended dental treatment. The difference between sensitivity and pain were categorized by DL according to degree: if the problem occurred consistently, moderate, and lasted for over 1 minute, it was classified as "pain". If problem occurred only on contact with a trigger, sporadic, and mild, it was categorized as "sensitivity". Other customary interview questions were also included on the HBP client information form. This data (for example, language, country of origin, last dental visit) was then included as part of the description of client demographics. Some data was based on DL's observations. For example, English capability was categorized as "fluent" if woman was able to converse easily in English and able to discuss a variety of topics in detail. "Conversational" was assigned if the woman was only able to ask and respond to questions pertaining to topics asked. "Translator required" was assigned to women who were unable to answer verbally to questions asked and relied on an interpreter. Observations of client tooth-brushing. Prior to clinical treatment at each visit, each participant was asked to demonstrate where and how they normally brushed with a compact adult toothbrush. The observation of tooth surfaces brushed (buccals, linguals, occlusals) and which teeth were brushed (all teeth or just front teeth) was then recorded by DL. Appointment monitoring. Clinic visits, contact telephone calls and cards, and appointment bookings were monitored and recorded (Appendix 0). Missed second appointments were rebooked by a dental receptionist. Third appointments were usually booked by DL when she contacted the women three to four months after delivery. If returning to the clinic was a problem, DL would administer the 3 rd questionnaire and interview 42 questions over the phone. Reminders to follow-up with regular dental care and the child's first visit were also given. Clinical indices and examination. The periodontal examination at each clinic visit was conducted by DL before any clinical hygiene therapy was performed. Various indices were pre-tested on clients from April to September 2005; the clinical indices selected were based on ease of use, time efficiency, and patient-friendliness. A modified version of the Ramfjords' Periodontal Disease Index (PDI) was selected for calculus, plaque and gingival indices on six selected teeth (16, 11, 25, 36, 41, and 45). These six teeth were to provide an accurate assessment of the total periodontal status of the individual (Ramfjord, 1967). Appendix P describes some of the indices used. The teeth were not stained for recording of plaque and measurement of clinical attachment loss was not feasible due to presence of calculus. Each specific index was derived by totaling all the scores for that index and dividing it by the number of teeth examined. Other indices involved bleeding upon probing (BOP) and measurement of pocket depth (mm). BOP was recorded as bleeding being "present" or "absent" within 10 seconds of probing. Pocket depth (POC) was assigned by the deepest recorded reading using a standardized pressure- sensitive probe9 walked along the six areas of each designated tooth to the nearest mm, from the gingival margin to the base of the pocket. Anything close to 0.5 mm was rounded to the higher number. Absent teeth were substituted with a similar tooth in that quadrant. The readings from the indices were recorded on forms designed by DL for the study (Appendix Q). The regular periodontal description of the entire dentition noted in the dental charting (plaque, calculus, gingivitis, periodontitis, mobility) was included; however this information was used primarily for descriptive purposes rather than for outcome variables. Ramfjord's Periodontal Disease Index classification had been the general guideline for these descriptions but 9 KerrHawe Click-Probe 43 the severity of periodontitis was categorized by DL according to probing depths. Pockets of 5mm were rated as "early" periodontitis; pockets 6-7mm was considered "moderate" while pockets 8mm and over were seen as "severe" periodontitis. The "greater depth of pockets" was used because of the inability to measure clinical attachment loss as well as the frequent presence of pseudo-pockets. Mobility was recorded as being "present" or not when lateral pressure was applied to the tooth with end of probe and mirror. The Community Periodontal Index of Treatment Needs (CPITN) (Darby & Walsh, 2003) was also calculated from the clinical data to determine its feasibility and sensitivity as a data collection tool. A sessional dentist examined each subject for the presence of visible decay; determined recommended treatment (e.g. extractions, night guard, endodontics, antibiotic prescription), and the recommended timing for dental treatment (before or after birth of child). Dental caries was either identified as being "present" or "absent" and treatment recommendations were dependent on the dentist's clinical experience. No radiographs were exposed. All information was entered on the NCHO dental clinic chart (Appendix I). Controls (reliability). Controls consisted of the use of calibrated pressure-sensitive probes and determination of intra-examiner reliability. Inter-examiner calibration among the six dentists regarding diagnosis of caries and recommendation of treatment needs was not feasible for logistical reasons. Reliability testing was performed for questionnaires, self-reports, and indices. At the booking of the 3 rd appointment, the women were asked questions from the questionnaires and selected interview questions over the phone and then asked to complete the questionnaire in person at the 3 rd clinic visit. Kappa Coefficient Reliability testing was done for five questionnaires and six interviews. Intra-class Correlation Reliability testing on indices was randomly performed on five women pre and post dental examination. 44 Data collection and analysis. Statistical analysis was consolidated into a Microsoft EXCEL database and converted to the SPSS (Version 15). Missing data was recorded as blanks or no data or unknown. Attempts were made to retrieve missing data from a variety of sources. This included direct follow-up with women at subsequent visits or by telephone. Univariate descriptive frequencies and two tailed bivariate analysis with normality kurtosis testing (< 1.96) were done. Cross-tabulations of descriptive frequencies compared changes over time in individual responses. Chi square tests (questionnaires and self-reports) compared frequencies. For indices and self-reports, parametric dependent variables were analyzed with t-tests and non-parametric data were analyzed with Wilcoxon Signed Rank Test. Significance was set at p < 0.05. Attrition was also considered in the outcome evaluation results. General frequencies were described for all the participants at baseline but comparisons over time were done only for women from whom data was collected at all three encounters 1° . 10 The women had participated in 3 questionnaires, interviews, clinical assessments, and/or tooth-brushing observations. 45 3. Results 3.1. Phase One 3.1.1 Evaluability Assessment of HBP Dental Program. Overview. The EA generated a list of the main stakeholders' objectives and goals (Figure 3), a list of stakeholders and their roles and the responsibilities (Appendix R), and a logic model describing program theory (Figure 4). 46 VCDP Dental • HBP clinical hygienist • For women to recognize benefit of clinical dental care and oral health education 2 • Improve oral health of woman and child I Figure 3. Main Stakeholders' Objectives and Goals Main stakeholders^Short-term objective^Long-term goal VCDP Dental • Clinic supervisor • Community coordinator • Address inunediate dental concerns of women seen • Reduce cavities in the children of women seen I HBP Prenatal • Coordinator  2 • Improve pregnancy outcomes • Address immediate dental concerns of women seen Dental needs & risks Intermediate 1+ Impact utcomeso  t Dental accessibility it Financial burden de Pre-term low birth weight babies t Healthier pregnancies t Dental caries in children il Oral health in women T Quality of life 1 t OH habits Figure 4. Logic Model: HBP Dental Program Theory Inputs Program activities 11+^ Outputs [ 2 clinic visits [Client history Immediate .4 Dental screening criteria Dental needs • HBP identified pregnant women Appointments [ Dental exam • Staff Periodontal Dental • Clinic Dental & medical interview treatment concerns Oral health education, instruction, aids & handouts • Funding t Oral health Dental services Oral health education and instruction t OH skills and knowledge Dental provider list 1 Legend: OH = oral health^t = improved/increased^it^reduced/decreased The HBP Dental Program was considered important by the VCDP dental clinic supervisor, the VCDP coordinator and the HBP coordinator, but it was a lower priority in comparison to their main program responsibilities which were either the child- focused dental programs or the overall HBP prenatal outreach program. The HBP Dental Program operated only two days a month which is considerably less than the rest of the VCDP's activities in terms of time, staff, clientele, and resources. Only 1% of VCDP's overall budget is allocated to the HBP Dental Program. The interviews and field observations revealed confusion in 1) program objectives; 2) program reality and intent; 3) reporting; 4) organizational structure; 5) funding; and 6) description of HBP dental clientele. The existing record-keeping methods were less than adequate for appointment monitoring and insufficient for evaluation purposes. There was no central database or location for client records and information about HBP dental clients. All data was manually entered and filed. The provider herself (DL) had limited orientation to the HBP prenatal program but there was a general understanding by most of the stakeholders about the HBP Dental Program's activities and it's "target" clientele: low-income pregnant women. Communication challenges were also identified. Identification of main stakeholders. The program's main stakeholders were the VCDP clinic supervisor (NCHO site), VCDP coordinator (NCHO site), HBP coordinator (Evergreen site), and the VCDP-HBP clinical hygienist (DL, NCHO site). These individuals had been associated with the HBP Dental Program either directly or indirectly for 4 to 19 years. In fact, the VCDP coordinator was a previous HBP clinical hygienist. Evaluation scope, purpose, and tasks. It was agreed by the main stakeholders that the evaluation would be limited to those activities specifically related to the HBP Dental Program: the dental screening process by the 49 HBP staff, the HBP Dental Program's clinical activities, and referrals to outside dental offices. The stakeholders acknowledged their lack of awareness about the HBP Dental Program's present reality and the "clientele" who actually attended. The main purpose of the program evaluation was to provide information about the program's clientele, activities, and outcomes and to identify areas for improvement. An analysis of cost-effectiveness was not indicated. DL would perform all aspects of the evaluation with the other stakeholders acting as "consultants" to the process. Periodic updates of findings would be reported back to the stakeholders via small group meetings or through email correspondence. Objectives of the HBP Dental Program No formal written goals or objectives about the HBP Dental Program were available at the time of this EA. Figure 3 summarizes the results of the interviews with the main stakeholders about goals and objectives for the HBP Dental Program. There were discrepancies in the long-term goals although everyone hoped to address the immediate needs of the HBP dental clientele. Roles and responsibilities of stakeholders. Appendix R outlines all the stakeholders identified with the HBP Dental Program and their associated roles and responsibilities. Program theory. The program theory or rationale that was developed from the interviews is depicted by a logic model of the HBP Dental Program (Figure 4). The model was constantly reviewed and refined as more data was collected. Inputs to the program, that is, the resources, were easily identified but details of some inputs (target population, HBP staff, funding) were unclear or unknown to some of the main dental stakeholders. There was limited demographic information about the "HBP women" except that they were classified by the HBP program to be at high risk for giving birth to pre- 50 term, low birth weight babies based on economic hardship, maternal age, language barriers, prior pregnancies or substance abuse. A cursory review of medical histories and dental charts (2000- 2004) revealed missing client data and inconsistency of information collected. The ethnicity of the HBP dental clients seen in 2004, as observed by the receptionists and DL, differed from the other main dental stakeholders' experience from previous years. DL knew little about the HBP staff or their level of oral health knowledge. The budget for the HBP Dental Program was reported by the dental clinic supervisor to comprise 1% of the total VCDP budget. However the process for the budget allocation for the HBP Dental Program and even for the overall dental program was unclear. The HBP Dental Program's activities were easily identified. However, it was unclear as to what the indicators of success were and which activities supported what outcomes and impacts. For example, the achievement of VCDP's goal of decreasing caries in children of HBP dental clients (by reducing maternal transmission of the cavity-causing bacteria) was unlikely since essential clinical services like provision of restorative services were missing from the HBP Dental Program. Furthermore measurement of caries in children of HBP dental clients had never been done. Existing program circumstances. Reports from interviews and field observations identified discrepancies between program activities, desired outputs, and outcomes. The discrepancies identified were as follows: • Referral process: Not every woman referred to the HBP Dental Program attended or received two appointments; the referral forms did not usually arrive at the clinic by the scheduled appointment and sometimes never arrived; completion of the medical history forms often required assistance from the receptionist due to language or comprehension difficulties; the number of clients referred to the HBP Dental Program was not documented, only the number of clients who actually attended. 51 • Appointment issues: Appointments were often difficult to confirm (language barriers, no answering machine); the daily schedule was always fully booked but women often missed confirmed appointments; cancelled appointments were often not rescheduled; the limited number of appointments available resulted in wait lists; confirmation and scheduling of HBP dental clients appointments were usually conducted only a day before the appointment due to reception's self-reported heavy workload from the main dental program and clinic services; • Clinical activities: The presence of client's other children in the operatory area was a distraction to the hygienist providing clinical treatment and education; clients unfamiliarity with dental treatment, especially scaling, necessitated extra time allocation for explanations which resulted in less than desired clinical periodontal treatment; hygienist was frustrated that initial therapy was often not completed; some women were not comfortable with Piezo scaler due to sensitivity to water temperature; sessional dentists were not always immediately available for dental examination and were not calibrated to any diagnostic or referral criteria; • Educational activities: All client educational topics were not always covered during a dental visit because of language barriers; educational handouts were often not available in the client's preferred language; • Time management: Completion of clinical hygiene therapy, education, sterilization procedures, and charting within a one hour appointment was challenging; appointments often ran behind schedule; extra time was required because of language barriers and client's unfamiliarity with dental procedures; difficulty in determining the focus of an appointment: patient education or clinical treatment; • Record-keeping: Limited space in existing client charts and forms to note pertinent information; appointment monitoring statistics, personal and contact information, medical 52 histories were available but had inconsistent or incomplete information; HBP client data was on four different forms and stored in four separate locations; some forms had incomplete information; appointment record-keeping system was unclear; all record- keeping was manually done at the time of the EA; the existing dental chart provided limited information regarding provision of treatment, provider, and oral conditions; • Infrastructure: Ongoing changes in the overall clinic operations (procedures), the physical space, and location of client records were challenging. A review of the HBP dental referrals forms 1986 to 2003 was not feasible since the forms were archived. An examination of the active and inactive charts in year 2004 revealed three missing and 23 incomplete referral forms out of 70. An earlier patient information section located on the reverse side of the referral form (Appendix J) had been updated by DL in April 2004 (Appendix S) to provide more client background and oral health information; however even this revised form lacked the detail needed for evaluation purposes. The relationship between the two programs (HBP prenatal and HBP dental) was perceived by the main stakeholders to be collaborative with the HBP prenatal program acting only as a referral agency to the dental program. It was not a partnership since the HBP program had no role in funding or in decision-making in relation to the HBP Dental Program activities. Communication between DL and the main dental stakeholders was hampered because of differing work schedules. DL acted in an independent role and usually only consulted the other dental stakeholders if there were problems. Communication was often done through handwritten post-it notes or emails. Furthermore DL had no direct communication with the HBP coordinator or staff. Only the VCDP coordinator was in contact with the HBP coordinator. Timely access to inquiries or updates was not feasible since DL did not have an assigned desk area. The majority of the stakeholders had knowledge pertaining to their individual responsibilities to the HBP Dental Program but not of its overall program. Classification of which women were at high risk 53 for pre-term low birth weight infants was unclear as well as which of the HBP staff referred their clients to the program. Receptivity of organization to program and to evaluation. Awareness and knowledge of the HBP Dental Program's existence, its activities and purpose varied from none to limited among VCDP staff, NCHO non-dental health professionals, and at other organizational levels within the regional Health Authority, Vancouver Coastal Health (VCH). The dental staff knew the basics of the HBP Dental Program but was vague about its specific activities and purpose. Within the NCHO site, only a few of the non-dental health professionals were even aware that a dental prenatal program component existed on-site. Most assumed that the dental clinic was only for children. DL's informal contact with staff from other health organizations within VCH revealed only vague awareness of the existence of a public dental health clinic program of any kind at NCHO. An online search of the VCH website, after several links, located the NCHO dental clinic but there was no mention of the HBP Dental Program. Furthermore, the main stakeholders were unclear about the location of the HBP Dental Program in the organizational structure of the regional health authority. DL was also confused as to whom she was to report to and with what information. The extent of the children's dental program activities was also not clearly-defined to DL. VCDP staff reacted positively to the evaluation which was perceived to be gathering evidence to improve the HBP Dental Program. Since the staff felt they played a minor role, they did not appear to feel threatened by the evaluation. The regional Health Authority (VCH) and the site managers and staff of both sites (NCHO, ECHC) were also supportive of the evaluation. The only evaluation concerns raised were by DL herself concerning the extra work of data collection in the already limited appointment times and a possibility of extra work for the receptionists. The main dental stakeholders did not feel the evaluation would place the HBP 54 Dental Program in jeopardy of budget or program cuts since there was a strong desire to help the HBP women. EA recommendations. Based on the findings, the EA recommendations were to conduct a descriptive evaluation to describe program structure, activities, clientele (and their concerns); and a formative (i.e. ongoing) process evaluation to examine the implementation of program activities. The need for a centralized record-keeping system and improved information feedback loop was identified. The purpose of these two types of evaluation was to clarify the program and its activities and to improve implementation (e.g. reporting structure, appointment monitoring system). Based on the EA, an evaluation of the program outcomes did not appear to be feasible given the minimal clinical services provided, language barriers, and unclear program objectives. However, on the main stakeholders' request, an exploratory outcomes evaluation was undertaken to gather preliminary data (knowledge, behavior, oral health status) for the stakeholders with the intention of more in-depth clinical and educational outcome evaluation when process activities were improved and program objectives redefined. 3.1.2. Descriptive evaluation (program description). Program history. The history of the program was primarily based on anecdotal reports from the main stakeholders and previous HBP Dental Program hygienists. Efforts to contact the program's original hygienist were unsuccessful. Interviews were conducted with two previous HBP clinical hygienists, both of who either had also served as VCDP coordinators. In 1986 the HBP coordinator from Evergreen Community Health Center (ECHC) approached the VCDP coordinator for help for their pregnant clients who were unable to eat nutritious foods due to dental concerns. Over 1600 HBP women have since attended this 55 program (Healthiest Babies Possible Program, 2004). Today the goals for the HBP Dental Program have evolved and expanded. Since 1986, seven hygienists have worked as the clinical HBP Dental Program hygienist (including DL) for periods of employment ranging from three months to nine years. Objectives. The objectives of the main stakeholders for the HBP Dental Program were previously described in the EA and presented in Figure 3. No written program information was found at the time of the EA; however a later search through files and folders in the VCDP coordinator's office unearthed an undated copy (two pages) of goals, objectives, referral guidelines, and appointment booking guidelines for the HBP Dental Program. This document concurred with the data gathered from the general observations and interviews regarding the program framework, history, activities, and goals. However, no details were included about the linkages between the objectives, goals, and program activities. Physical environment. NCHO is located in a busy, vibrant multi-ethnic community of small businesses and restaurants on a main street surrounded by houses, multi-family dwellings, with nearby elementary schools, a library, and a community center. The building which houses the NCHO dental clinic is located on the second floor of a two storey corner building with underground paid public parking. City transit stops are in front of the building. Access to the second floor where NCHO is located is by elevator or stairs. A private security guard is on duty twenty-four hours a day. The NCHO staff consists of health professionals from various disciplines and programs ranging from Infant Child Youth (ICY) Programs, Adult Older Adult (AOA) Programs, and Addictions (Safe Injection Site). The VCDP and dental clinic are classified under the ICY Programs. The dental clinic, physically separated on the NCHO site from the other health 56 professionals' offices/cubicles, consists of dental operatories, a sterilization lab, and offices for dental staff. However common meeting rooms, other health program offices, lunchroom and other facilities are accessible to the dental staff. Both the dental clinic and general health unit share a common waiting room with individual reception desks and entrances. On the south side of the clinic is a larger enclosed room for eight desks for the dental assistants, and for storage of the educational handouts/supplies. On the north side of the clinic is a glass-enclosed office for the VCDP dental clinic supervisor. Between these two dental office rooms are five operatories; a laboratory for sterilization and radiograph processing; cabinets for instrument and sundries; a linen closet; a small radiography room; and an open receptionist desk area. The VCDP coordinator and the sessional dentists have office space in the "non-dental" side of the NCHO site. The hygienist (DL) uses any desk space that happens to be available. Two of the five dental operatories are reserved for treatment services by the dentists while the other three are shared between the dental assistants and clinical dental hygienist. Of the three operatories, only one operatory has the necessary water specifications suitable for the HBP Dental Program equipment. This operatory is available only one day per week. Organizational context. Figure 5 outlines the organizational context surrounding the HBP Dental Program. An online search of the VCH website (Vancouver Coastal Health, 2007) provided a general structure of the VCH authority but little detail regarding the VCDP at NCHO. The HBP Dental Program was not mentioned as being part of the VCDP. 57 Figure 5. Organizational context of HBP Dental Program Ministry of Health (British Columbia) Vancouver Coastal Health (VCH) Provincial Health Services Authority andOther Health Authorities: Fraser, Interior, Vancouver Island, Northern Health Vancouver Community Health Service Delivery Area (VCHSDA) Other Health Service Delivery Areas: Richmond, Providence, Vancouver Acute, Coastal #2 Community Health Center (CHC) Other sites: Sheway, Pender, DCHC #3 CHC Evergreen (ECHC) Other CHCs: #1 Three Bridges, #4 Pacific Spirit, #5 Raven Song, #6 South North Community Health Office (NCHO) site Other programs andservices _.... Other programs and services: Adult/Older Adult; Addiction — Infant, Child, Youth Programs Other programs Healthiest Babies Possible (HBP) Program Selectively refer  Provide screening criteriaVancouver Community Dental Program (VCDP) Dental Clinic Community Program Treatment services (Vancouver children) HBP Dental Program (ECHC Vancouver HBP women) Prevention services (Vancouver children) 58 HBP Dental Program • " Sessional Dentists (3) — -0"1 Receptionists (2)^— Chair side Assistants (2)  —04 Certified Dental Assistants (7) Prevention servicesTreatment services Community Program (Coordinator - Dental Hygienist) Dental Clinic (Supervisor - Dentist) Dental Hygienist Personnel. Figure 6 is a flowchart of the VCDP staff and their primary responsibilities. Details of their roles and responsibilities were previously outlined in Appendix R in the EA. Most of the dental staff were full-time employees. Figure 6. Vancouver Community Dental Program personnel Vancouver Community Dental Program "+ Main dental health providers^—0' Support dental health providers The sessional dentists worked either one or two days a week while the HBP clinical hygienist worked two days/month. All dental staff except the HBP dental hygienist (DL) was focused on prevention and treatment services for low income children. The sessional dentists, 59 receptionists, and chair side assistants played supportive roles to the HBP Dental Program by providing dental examinations, scheduling appointments or ordering of supplies respectively. Activities. The program theory is illustrated in the logic model created in the EA (Figure 4). A complete program plan was never found other than the two-pages of basic goals, objectives, referral and appointment booking guidelines previously cited. The intended procedures and activities from the time of screening to the two dental clinic appointments are illustrated in the flowchart (Figure 7). 60 Figure 7. HBP Dental Program Activity Flowchart HBP Vancouver pregnant women screened according to dental criteria by HBP staff HBP staff books appointment for client with VCDP receptionist and notifies client VCDP receptionist confirms appointment one day prior; gives NCHO clinic directions; and requests early arrival Visit #1 @ NCHO dental clinic Receptionist — contact information, medical history Dentist — oral exam, treatment recommendations Dental Hygienist — interview; initial periodontal treatment; peri-natal oral health education & instruction; handouts VCDP receptionist schedules second dental appointment VCDP receptionist confirms appointment one day prior 4 Visit #2 @ NCHO dental clinic Dental Hygienist - follow-up interview on previous treatment effects and recommendations; ongoing periodontal treatment; review of oral health education and instructions 61 HBP women are referred to the dental program if they reside in Vancouver and either self-identifies a dental concern or report not having seen a dentist for over two years. Other verbal "guidelines" given to the HBP staff were to refer women who were known to be responsible at keeping appointments and were in their second trimester (between 16-28 weeks gestation). A dental appointment is booked by the HBP staff and a referral form is sent by inter- office mail from ECHC to the NCHO dental clinic. Upon first arrival at the dental clinic, customary contact and information forms are completed by the client. This information is reviewed by the hygienist with emphasis on oral health concerns and oral care routine for self and other children (when applicable). The peri-natal oral health education topics at the appointment with the hygienist include: • increased risk of dental concerns during pregnancy; • maternal "cavity causing" bacterial transmission to the infant; • adverse effects of child sleeping with the bottle; • periodontal disease progression to tooth loss; • infant oral care; and • importance of healthy baby teeth. Each woman is asked to first pre-rinse with an anti-bacterial mouth rinse and then demonstrate brushing with a compact adult toothbrush. At some time during this appointment, a dental examination is provided by the dentist on duty. Only teeth requiring restorative work or extractions are charted. No radiographs are taken. Clinical scaling by hand or by Piezo scaler is begun by the hygienist. Oral hygiene instruction is demonstrated. Other aids are given when appropriate. Clients are asked to schedule a second appointment. Educational handouts and a referral list of outside dentists are given. The receptionist schedules and confirms a second appointment. 62 Resources. Dental supplies, oral hygiene aids, and handouts are shared throughout the dental clinic. However most of these resources are targeted for children. In fact, the radiography room is not physically appropriate for adults. The small open operatory, available for HBP dental clients, has limited space to accommodate strollers. All additional supplies (instruments, oral health aids) pertaining to the HBP Dental Program are ordered by the clinic, after approval by the dental clinic supervisor. Funding. The HBP Dental Program funding accounts for approximately 1% of the total dental clinic budget, as previously reported. These funds are primarily for salary of the clinical hygienist. A detailed budget was not available to this evaluation. Distribution of funding for the various programs and resources within the VCDP is determined by the dental clinic supervisor. The overall VCDP dental budget is allocated by the Vancouver Coastal Health Service Delivery Area (VCHSDA) Board of Directors with input from the NCHO site manager in consideration of other NCHO program needs. Any request for an increase in the VCDP overall dental budget is made to the Vancouver Coastal Health (VCH) Community Directors. The funding for the HBP prenatal program is from Vancouver Coastal Health (VCH) and the Public Health Agency of Canada — Canada Prenatal Nutrition Program (grant). There is no financial support from the HBP prenatal program directly to the HBP Dental Program. Record-keeping mechanisms. Client data was recorded on a medical history form, dental chart, and referral form. Space was available on the reverse of the referral form for tracking appointments, treatment, oral self-care recommendations, and oral health status (Appendix J). Appointment statistics was manually entered on a log sheet (Appendix T). 63 At the beginning of the evaluation in 2005, all client data was manually collected onto paper forms and stored in filing cabinets and shelves located in the NCHO site. In late 2006 there were two major changes in office structure: computerization of all client records to a mandatory health information database network called Primary Access Regional Information System (PARIS); and introduction of a new charting and filing system. The HBP dental component was not incorporated into PARIS or into the new charts except for contact information and appointments. Statistics for HBP dental clientele are currently not accessible with the PARIS system. Furthermore, there was a turnover in receptionists from 2005 to 2007. The "front-desk" changes provided extra challenges for the record-keeping and appointment monitoring of the HBP Dental Program. Clientele. A woman is accepted into the HBP program after screening and determination of her risk for a pre-term and/or low birth weight birth. Admission criteria include experiencing inadequate nutrition or food security concerns, economic hardship, isolation (language, social or ethnic), history of low birth weight baby, youth pregnancy, single parenthood, abuse, multiple birth, substance use or mental health concerns (Dickie, 2005). The women are screened over the phone by the HBP program assistant to determine residency, risk factors and purpose of inquiry. Non-residents of Vancouver and Richmond, and women seeking only prenatal classes or who have no risk factors are excluded. Those women accepted are further assessed at an initial visit at ECHC by HBP staff. If they do not have risk factors for poor pregnancy outcomes, the women are then referred to their doctor, prenatal classes, or other community resources. In 2004, only eight out of 575 initial visits were considered inappropriate referrals (Dickie, 2005). Once the women are accepted into the HBP program, they are assigned a culturally and needs appropriate HBP staff for nutritional and prenatal counseling for a series of seven-to-nine individual visits based on need. According to the HBP fiscal year report April 1, 2003 — March 64 31, 2004, 68 out of 545 HBP participants were referred to the dental program as per referral criteria previously cited but the HBP coordinator acknowledged underreporting by staff of dental referrals (Dickie, 2005). Such a discrepancy was confirmed by the HBP Dental Program record of 139 appointments in that time period. Oral health concerns and needs of the 123 HBP dental clients seen from January 2004 to December 2005 are illustrated in Figure 8. 65 Figure 8. Oral health concerns and needs of Clients 2004-2005 (N=123) 100% 80% 60% 40% 20% 0% 78% 80% 73% 63% Report bleeding gums Had moderate - severe^Had visible cavities^Need other dental services gingivitis 0 Visit # 1  Other general client demographics included: mean (SD) age of 27.3 (5.4) years, 1.8 (1.0) number of pregnancies, 27.6% (34/123) were Canadian-born, 45.5% (56/123) spoke fluent English, and 47.6% (57/120) had other children. The mean gestational week at first visit was 22.3 (5.4) weeks and 26.0 weeks (5.0) at second visit. The clients represented 26 different countries and spoke a total of 20 different languages. Table 1 provides more detail on the 123 clients. In this entire group, only two out of 117 women eventually had pre-term, low birth weight babies. 67 Table 1. Demographics & characteristics of clients 2004-2005 (N=123) (number) Country of origin* Southern Asia 47.2 (58/123) Northern America 28.4 (35/123) South-Eastern Asia 5.6 (7/123) Central America 2.4 (3/123) Eastern Asia 2.4 (6/123) Western Asia 5.6 (7/123) South America 1.6 (2/123) Other 1.6 (2/123) Dominant language reported No data 0.8 (1/123) English 32.0 (39/122) Punjabi 22.1 (27/122) Other 45.0 (55/122) English capability Translator required 19.5 (24/123) Conversational 35.0 (43/123) Fluent 45.5 (56/123) Years in Canada No data 37.3 (46/123) Canadian-born 44.1 (34/77) Less than 2 years 31.1 (24/77) 2-5 years 12.9 (10/77) Over 5 years 11.7 (9/77) Other dependents No data 2.4 (3/123) Women with other children 47.6 (57/120) Women with no other children 52.4 (63/120) Health concerns No data 0.8 (1/123) No 76.2 (93/122) Yes (thyroid, diabetes, heart etc) 23.8 (29/122) Smoker Current 12.2 (15/123) No 87.8 (108/123) Previous 8.3 (9/108) Pregnancy concerns Nausea & Vomiting 59.3 (64/108) None 33.3 (36/108) Other (back etc) 7.4 (8/108) * Geographical region classification according to United Nations Statistics Division - Standard Country and Area Codes Classification. Accessed August 16, 2006 from littp::/unstats.un.orgitinsd/methods 'n149 lm49rezin .htm 68 Table 2 outlines the dental health status and behavior of these women: 42.2% (51/121) had not seen a dentist for over two years and 52.2% (48/92) had not had a "cleaning" for over two years. Of the women with other children, only 36.8% (21/57) had accessed dental care for these children. 69 Table 2. Dental demographics of clients 2004-2005 (N=123) (number) Dental coverage No data 0.8 (1/123) No dental coverage 92.6 (113/122) Ministry 3.2 (4/122) Indian Affairs 4.0 (5/122) Last dental exam (Canada/International) No data 1.6 (2/123) Less than 2 years 57.8 (70/121) 2 - 5 years 20.7 (25/121) Over 5 years 14.9 (18/121) Never seen a dentist before 6.6 (8/121) Recent emergency dental visit (less than 6 months) No data 4.1 (5/123) Yes 18.6 (22/118) No 81.4 (96/118) Last dental cleaning (Canada or International) No data 25.3 (31/123) Less than 2 years 47.8 (44/92) 2 - 5 years 22.8 (21/92) Over 5 years 14.1 (13/92) Never had scaling before 15.2 (14/92) Brushing frequency Less than 2X/day 30.1 (37/123) 2X/day 52.8 (65/123) More than 2X/day 17.1 (21/123) Flossing None 57.0 (69/121) When needed 10.7 (13/121) Occasional 18.2 (22/121) Daily 14.0 (17/121) Other aids used None 50.8 (60/118) Toothpicks 21.2 (25/118) L isterine/mouth wash 16.9 (20/118) Other* 11.0 (13/118) Other children receiving dental care N/A 51.2 (63/123) No data (on women with children) 5.0 (3/60) Yes 36.8 (21/57) No 63.2 (36/57) * includes combination of toothpicks/Listerine/mouthwash 70 Bleeding was the most commonly reported dental concern for 78% (96/123) of the women at the first visit (Figure 8). By the second visit or encounter, 89.9% (80/89) of those who reported bleeding had less to no bleeding. Sensitivity was reported by 47.1% (58/123) at first visit and pain, by 37.4% (46/123). At the second visit, only 28.9% (26/90) reported sensitivity, and 13.2% (12/91) reported pain. Eight of the 123 clients (6.5%) seen in this time period reported no oral health concerns at first visit. In the second visit, 12.9% (12/93) had no oral health concerns. Table 3 provides the findings of the dental and periodontal exam. 71 Table 3. Visual oral examination at Visit #1 on clients 2004-2005 (N=123) Plaque % (number) No data 88.6 (109/123) No plaque 7.1 (1/14) Light 57.1 (8/14) Moderate 35.7 (5/14) Calculus No data 1.6 (2/123) Light 17.4 (21/121) Moderate 71.9 (87/121) Heavy 10.7 (13/121) Gingivitis No data 1.6 (2/123) No gingivitis 0.0 (0/121) Mild (dark pink, no BOP) 27.3 (33/121) Moderate (red, BOP) 65.3 (79/121) Severe (red, edema, BOP) 7.4 (9/121) Periodontitis * No data 4.1 (5/123) None (pockets <5mm) 14.4 (17/118) Early (5mm pockets) 41.5 (49/118) Moderate (6-7mm) 33.9 (40/118) Severe (8+mm) 10.2 (12/118) Mobility No data 65.0 (80/123) No mobility 74.4 (32/43) Yes 25.6 (11/43) Caries (visible cavities) None 36.6 (45/123) Yes 63.4 (78/123) * pocket measurement only; generous classification to account for edema A majority had moderate gingivitis, visible cavities, and pocket depths 5mm and over. Almost 79.7% (98/123) were advised to seek other dental services, but only 14.5% (11/76) were successful by the second visit. Of the 45.9% (45/98) who required urgent treatment before the birth of their child, only 20.0% (7/35) obtained some or all of the recommended urgent services. 72 Changes in oral health behavior between the two dental visits were improvements in tooth surfaces brushed and an increase in accessing professional dental care for the clients' other children. Statistical analysis was not performed on this data from the chart reviews of clients seen in 2004-2005 because of the incomplete data that was collected. 3.1.3. Process evaluation. Referral process. HBP staff stated that the main goals for the clients referred to the HBP Dental Program were exposure to oral health education and access to low cost dental services. The staff was appreciative of the dental program's support for their clients and hoped for even more services in the future. The challenges expressed regarding access to the dental program were the limited number of available appointments (12 per month) and client difficulty in fulfilling the inclusion criteria (dental concern, no dentist, 2nd trimester referrals, reliable client, Vancouver resident, new HBP dental client, no dental coverage). The results of the questionnaire administered to the HBP staff at the first meeting with DL (as part of the evaluation) indicated their generally good oral health knowledge. All of the eight staff members present correctly answered the questions related to periodontal disease, cavity risk factors, relationship of mother's oral health to that of her child, oral care for newborn, and the relationship of baby teeth and adult teeth. Everyone felt that the dental program would be good for the pregnant women. When questioned a year later in the poster questionnaire session, most of the staff reported that the referral process and communication between the programs had improved, but appointments were still difficult to book due to limited availability of appointments. 73 A chart review revealed that two of the eleven HBP staff accounted for 57.1% (97/170) of the HBP dental referrals; however not all these clients eventually attended the HBP Dental Program. HBP referral form. Overall, 69.5% (117/169) of the forms were complete. The items that were usually incomplete were the need for an interpreter or dental referral; HBP client identification number; and dates of appointment and dental screening. DL considered 11 out of 15 items in the referral form (Appendix J) to be of relevance to the information needs of the evaluation. The referral form's dental section was not useful since DL found the information was more accurate when a client was interviewed directly by her. As mentioned previously, referral forms were sometimes received after the client's appointment or were not sent to the dental clinic at all. Appointments. The dental attendance of HBP clients in 2004-2006 is detailed in Table 4. Table 4. Dental program attendance of HBP clients referred 2004-2006 2004^2005^2006 %^(#)^%^(#)^%^(#) HBP women referred Did not attend 20.9 (18/86) 34.5 (29/84) 18.6 (13/70) Attend clinic 79.1 (68/86) 65.5 (55/84) 81.4 (57/70) One visit* 38.2 (26/68) 9.1 ( 5/55) 19.3 (11/57) Two visits 61.8 (42/68) 90.9 (50/55) 80.7 (46/57) reasons include no shows, too close to due date or no available appointments 74 Over three years, 25% (60/240) of women referred never attended the HBP Dental Program; however 76.7% (138/180) of the women who did attend returned for 2 nd visits. The number of missed and/or rescheduled appointments on the day of appointment was 21% (32/150) in 2004; 28% (39/140) in 2005; and 25% (39/159) in 2006. Rescheduling missed first appointments was difficult if no referral form had been received. A review of HBP Dental Program statistics from 1986 to 2006 revealed that there was a decrease in the number of "broken"" appointments in the period of 2004-2006, ranging from 14.7 — 20.7% in comparison to the range in earlier years of 21.7 — 44.2%. The time intervals between screening and first visit in 2004-2005 are described in Table 5. The average time interval was 4.9 (2.3) weeks but the range could vary from a few days to almost 15 weeks. Table 5. Time between Screening date, Visit #1 and Visit #2 in 2004-2005 Time (weeks) Mean (SD) Median Range Between Screening & Visit #1 (N=111) 5.2 (2.6) 5.0** 0.57 — 14.81 Between Visit #1 & Visit #2 (N=93) 4.5 (2.1)* 4.0 0.57 — 9.97 Time (gestational) Between Visit #1 & Visit #2 (N=93) 4.6 (2.2)* 4.0 0.0 — 10.0 * mean chosen since distribution normal as indicated in kurtosis being <1.96 ** median chosen due to abnormal distribution reflected in kurtosis being >1.96 11 This was the classification of the past appointment records. May refer to missed, rescheduled or cancelled appointments. 75 Both HBP staff and dental receptionists agree that they were frustrated with trying to accommodate the clients in the limited number of available appointments. However, determined efforts were made to either move those clients who were further along in their pregnancy to an earlier appointment or put them on a cancellation list. Medical history forms. The dental receptionists estimated that 30-40% of the HBP women needed assistance completing the medical history forms at first visit. The existing translations (Chinese, Vietnamese, Spanish, and Punjabi) were not appropriate for quite a few clients. Of the medical history forms retrievable from 2004-2006, the majority had incomplete information; for example, only 31.8% (21/66) of the forms in 2004 had complete information on all 39 items. Missing information commonly pertained to physician name and phone number, and emergency contact name and number. The medical questions usually incomplete were about presence of heart murmur or congenital heart lesion or abnormal bleeding. However, the items most often left blank overall were in the dental section regarding dentist name, main dental concern and reason for HBP Dental Program visit. Due to time constraints, DL in her role as clinician had limited time to verify and complete the medical history, but concerns were followed-up verbally. She was not concerned about the incomplete dental portion since she often found the clients gave different answers at the interviews than what was actually written on the history form. All client information forms. As mentioned earlier, the client information form (Appendix S) created by DL prior to the evaluation did not provide enough information for the purposes of the evaluation in regards to client's oral health status and concerns, and oral health topics discussed. However, this client form documented more client information than the existing dental charts. 76 Dental services. Table 6 outlines the dental services performed in 2004-2005. Table 6. Dental services provided in years 2004 -2005 Visit #1 (N=123) %^(number) Visit #2 (N=93) %^(number) Exam (dentist) 100.0 (123/123) 16.1 (15/93) Scaling > 30 minutes (hygienist) 87.8 (108/123) 74.2 (69/93) Polish (hygienist) 0.0 (0/123) 48.4 (45/93) Fluoride (hygienist) 4.1 (5/123) 74.2 (69/93) Instructions/Education (hygienist) 100.0 (123/123) 100.0 (93/93) Other* 4.2 (4/95) * some women did get restorative, radiographs An exam by a dentist was always provided at first visit but also at 2 nd visit if needed. 79.7% (98/123) of the women required other services aside from regular maintenance; 45.9% (45/98) of those women with recommended other services were encouraged to get those concerns addressed before the birth of their child. At Visit #2, only 20.0% (7/35) of the "urgent" clients reported obtaining the recommended services. DL observed that clients often did not seem to remember the treatment recommendations and which teeth required treatment. Clinical hygiene services. Oral health education and instructions were consistently performed at each appointment (Table 6). An average of 2.1(1.0) units of scaling were provided at the first visit and 1.9 (+ 0.7) units at the second visit. Most appointments usually went over time; an estimated 15% kept within allocated time (DL personal observation). Clients were often late arriving, but still had to spend time registering and completing medical histories. This "pre-appointment" delay encroached on 77 valuable appointment time. However, cancelled or missed appointments usually provided a buffer to catch up on charting and performing client services. The biggest challenge for DL was to incorporate all six education topics as well as clinical services, procedures, and charting within an hour. The added time for translation and explanation of dental procedures increased appointment time and affected the amount of education and/or clinical treatment performed. From the clinician's point of view, the situation was extremely frustrating. Oral health education. DL reported delivering counseling on all recommended oral health topics; however the extent and depth of the counseling was dependent on client's comprehension of English and her pre-existing oral health knowledge. The coverage of these topics took about 15 minutes with more time allocated to answering related questions. Some of the educational pamphlets were distributed more often due to their superior illustrations and readability. Of the four translated educational handouts, DL noted that the English and Punjabi versions were the most frequently used. When there was no available translation, the client usually asked for the English version. Dental provider list. The dental provider list (Appendix U) was usually distributed with the educational handouts. The information listed on the handout of dental providers offering reduced costs was generally accurate (confirmed by phone in December 2005 and July 2006). Conversations with the various providers revealed fees to be 10-35% less than customary fees. However certain restrictions applied in terms of residency, age, minimum number of treatments required, and hours of operation. In addition, only two of the seven providers had multilingual staff, with the most common languages being Cantonese, Mandarin, Vietnamese, Spanish and Italian. Some of the providers reported wait lists and would only be able to put the pregnant women on a 78 cancellation list. DL was aware of other dental providers offering reduced cost services; however those offices also had screening restrictions. Other field observations regarding process activities. The staff turnover in dental receptionists and reliance on temporary help often affected appointment scheduling, follow-up of missed appointments, and consistency of HBP screening guidelines. DL often felt reluctant to direct the receptionists and tried to do most re-booking of appointments herself 3.2. Phase Two 3.2.1. Outcome evaluation. Participation. Ninety-three women were referred to the HBP Dental Program from October 2005 to October 2006, 72.0% (67/93) of them attended. Ninety-one percent (61/67) of the attendees consented to participate in the outcome evaluation (Table 7). 79 Table 7. Client participation, measurement in Time (N=61) Client Groups % # "2-visit" Group* Short-term evaluation (1st & 2nd pre-natal ) Participated in clinical indices for both visits 82.0 50 Demonstrated brushing ability at both visits 82.0 50 Answered both questionnaires° 85.2 52 Answered both interview questions° 86.9 53 Unable to contact for 2nd prenatal visit 13.1 8 Medium-term evaluation (1st pre-natal & 3rd post-natal) Participated in clinical indices at Visit #1 and #3 63.9 39 Demonstrated brushing ability at Visit #1 and #3 63.9 39 Completed questionnaire #1 and #3° 70.5 43 Completed interview questions #1 and #3° 70.5 43 Unable to contact for postnatal visit 29.5 18 "3-visit" Group** Participated in 3 clinical indices 59.0 36 Demonstrated brushing ability in 3 clinic visits 59.0 36 Answered 3 questionnaires° 63.9 39 Answered 3 interview questions° 65.6 40 * Includes all study participants ** Participants who completed 3 questionnaires, indices, interviews, and/or brushing demonstrations ° Conducted in person or by telephone 80 Usual reasons for non-participation were uncertainty of future residency or current residence in the protection of a "safe house". Thirty-two (52.5%) of these 61 participants were referred by two of the nine HBP staff members. Of the 61 women, 36 attended all three clinical visits; 39 answered all three questionnaires, and 40 participants completed all three interviews (Table 7). For 3 of the women, questionnaires and interviews were completed by telephone. These participants from whom data was collected at all three encounters are called the "3-visit group". Participants who provided data at two encounters (either Visit #1 and #2, n=50) or (Visit #1 and #3, n=39) were called the "2-visit group". Comparisons of clinical indices and questionnaire responses overtime in both 2-visit and 3-visit groups demonstrated similar results, therefore, only the results from the analysis of the 3- visit group are reported. Baseline characteristics are reported for all 61 participants; short-term and medium-term outcome results are compared over the three encounters for the cohort of "3- visit" women. Means (SD) are stated. When normality testing by kurtosis was > 1.96, medians I2 are given. Reliability testing. Reliability testing (Kappa Coefficient) for the repeated questionnaires and self-reports were 0.62 to 1.00. For the indices, results were similar from 0.66 to 1.00. These readings mean the repeated self-reports, questionnaires and indices were in "substantial to almost perfect" agreement (Landis & Koch, 1977). Spearman's correlation for nonparametric data (ordinal) between the Gingival Index and Bleeding on Probing Index revealed high to very high correlation (0.538 - 0.710) (Hopkins, 2006). 12 The score that divides a distribution exactly in half (Gravetter & Wallnau, 2004) 81 A triangulation methodology to correlate clinical indices (e.g. bleeding on probing, gingival) with results from interviews (e.g. bleeding reports, brushing) was not feasible because different scales for measurement (categorical, ordinal, and interval) were used. Client demographics and oral health. General baseline characteristics of the outcome group are in Table 8. 82 Table 8. General Characteristics of Participants Oct 2005-2006 (N=61) Country of origin* (number) Southern Asia 42.6 (26/61) India 19.7 (12/61) Other 23.0 (14/61) Northern America 31.1 (19/61) Canada 29.5 (18/61) United States 1.6 (1/61) South-Eastern Asia 8.2 (5/61) Central America 11.5 (7/61) Other 6.6 (4/61) Dominant language reported English 36.1 (22/61) Punjabi 16.4 (10/61) Other 47.5 (29/61) English capability Translator required 14.8 (9/61) Conversational 27.9 (17/61) Fluent 57.4 (35/61) Years in Canada No data 4.9 (3/61) Canadian born 27.6 (16/58) Less than 2 years 44.8 (26/58) 2-5 years 13.8 (8/58) 5-10 years 8.6 (5/58) Over 10 years 5.2 (3/58) Other dependents Women having other children 49.2 (30/61) Health concerns None 72.1 (44/61) Other** 21.3 (13/61) Smoker Current 13.1 (8/61) No 77.0 (47/61) Previous 9.8 (6/61) Pregnancy concerns Nausea & Vomiting 57.4 (35/61) None 36.1 (22/61) Other (back etc) 6.5 (4/61) Maternal vitamins Yes 96.7 (59/61) * Geographical region classification according to United Nations Statistics Division - Standard Country and Area Codes Classification. Accessed August 16, 2006 from http:/;unstats.un.orgiunsdlinethods/m49/m49regin.litm **includes thyroid, diabetes, heart disease, hepatitis etc. 83 The age, mean (SD), was 27.7 (5.0) years; 29.5% (18/61) were Canadian-born; 57.4% (35/61) spoke fluent English; and 49.2% (30/61) had other children. The women came from 18 countries and spoke 15 different languages. The median number of pregnancies was two. The mean gestational week at first visit was 22.8 (5.2). There was a median four week interval between screening and first visit, and the same interval between first and second visit (range 0 to 12 weeks). Almost 87% (52/60) had no dental insurance and 65.6% (40/61) had not had a dental examination for over two years. A dental emergency within the past six months was reported by 4.9% (3/61). Of the 28 women who had other children living with them, 64.3% (18/28) reported that their children were not receiving professional dental care. About 31% (19/61) brushed less than two times a day; 85.2% (52/61) did not floss daily; and 52.4% (32/61) relied on brushing alone for home oral care. "Bleeding gums", the most common oral health concern was reported by 90.2% (55/61) of the participants. The other common concerns were sensitivity 42.6% (26/61) and pain 14.8% (9/61). Oral assessments (by DL) found 70.5% (43/61) had moderate gingivitis; 82% (50/61) moderate calculus; 95.1% (58/61) light plaque; and 75.4% (46/61) had probing depths ? 5mm. Examination by the sessional dentists revealed 68.9% (42/61) to have visible dental caries. Almost 79% (48/61) were recommended to seek additional dental treatment. Of those 48 women, 17 or 35.4% had dental needs that required urgent attention before the birth of the baby. Postnatal data available from HBP coordinator revealed that the gestational week at delivery was 39.3 (1.2) with birth weight mean of 3328 (458.1) grams. Of the known birth weight results, only one of the 58 participants had a pre-term and low birth weight baby and two other women had babies born weighing less than 2500 grams, but not pre-term. 84 Behavior and awareness. Eighty-two percent (50/61) of the participants returned for a 2" d prenatal clinic visit. Three more participants had a "2" d visit phone interview" to collect data. Sixty-four percent (39/61) of the women returned for the postnatal 3 rd clinic visit. Phone interviews were conducted with four more participants. The time interval between 2 1.1d and 3 rd visit was a median of 41 weeks (range 31 to 70 weeks). Forty-six weeks was the median time span between 1st and 3 rd visit. The interview questions for the 2" d visit were answered by 86.9% (53/61) of the participants either in person or over the phone. For the 3 rd visit, 70.5% (43/61) responded. Figure 9 compares the oral concerns reported by the 3-visit participants over time. 85 100% - 93% •••••44.4******+.4.441■1■4.,80% .0.4**••••••4.44,•••••••444***4****4***1■•• •■•*444* 63%^63%4,4** 60% *4********4,••••♦*********4**,4,*****4****.4,44*.4*** 40% ******4*************4,•••• 33% 30%4,-01****************41.444.4. 28% 25% 20%■*****, 20% 4.04.44. 18%.0.4***4*** ***** ^*****4*4444+4***Iv.*** *****■ 8% + 0°/0 *****••••••4,444.t• • *****.****4, . 0% 3%►*****•^•►ttttt Figure 9. Reported oral concerns of "3-visit" clients (N 40) Bleeding^Sensitivity^Pain^No concerns at all /9 Visit #1^^ Encounter #2^^ Encounter #3 postnatal p < 0.05, when compared to Visit #1, chi square test Reports of bleeding upon brushing were significantly reduced at 2nd and 3 rd visit compared to baseline (1)=0.000). While the number of women reporting pain at 2nd significantly decreased, reports of pain increased again (p=0.004) by 3 rd visit. Reports of sensitivity did not change significantly over the visits. The number of clients with no dental concerns at 2nd ^3 rd visits significantly improved compared to Visit #1 (p=0.001). Oral health skill changes observed and reported in the 3-visit group are illustrated in Figure 10. 87 Figure 10. Oral Health Behavior and Awareness of "3-visits" clients (N 40) 100% - 83% 81% 98%- 100% 80% 78% 70% 68% 04,11/VVVVV4 44414044444.♦4 4.444•444.4 60% ^••••••+4.4414444.4►444 ••••••44^•••••••*4 .444044...404 4 444.44.44.44♦4.444.4444 4 40% .44.44 .1 4 .44444 . ^••••••+ 4 30% ..****44 4 4404.144..40+4.44 *4401.4 4401444 444.44. 20% .44••••• 4 444144144.4444.4404.44 4..44444.44.4+ 4 8% 444...•4144-44. 1.44**** 4104.••••444414.4 .4******444.44441044444 *444444 0°/0 4..4440 4 4.4444. Brush 2X/day or more^Demonstrate proper brushing (N=36)^Other children receiving dental care MI Visit #1 0 Encounter #2 0 Encounter #3 postnatal + p < 0.05 when compared to Visit #1, chi square test At 1 st visit, only 8% (3/36) demonstrated brushing on all teeth and tooth surfaces. The significant improvement in brushing ability by 2" visit (p=0.000) was maintained also at the postnatal 3 rd visit (p=0.000). All the participants liked the size of the toothbrush provided and found it easier to use on molars. Professional dental care for the client's other children significantly increased overtime (p=0.000). At the postnatal interview, 93% (37/40) of the participants reported performing infant oral care at post-natal interview. It was recommended to 75% (30/40) of the participants that they were in need of other dental services. Only 6.7% (2/30) of the participants had obtained the recommended treatment by Visit #2. By the 3 rd visit (medium-outcome), 23.3% (7/30) obtained the recommended services. Of the women (9/30) who were advised to obtain dental services before delivery, only 22.2% (2/9) were successful by 2" visit and 44.4% (4/9) by the time of the 3 rd visit. Of the 26 women who required dental services after delivery, 15.4% (4/26) of those who returned reported that they obtained the recommended delayed services (after birth). For the 10 women who only required regular maintenance, 100% (10/10) returned for 2 nd and 3 rd visits. Knowledge. Figure 11 compares correct answers on the questionnaires between visits in the 3-visit (N=39) group. 89 82% '95%^ 85% 56°/0_ 80% W4. 444 444 444 444 444 44. 44. 444 59% 444 sss 444 ++4 44. 444 44 4.4 444. 44 444 ++4 444 44 4 44 444 444 ++4 ++4 44 +4 444 444 4+ 444 .44 4+ 444 444 44 .44 444 44 444 +.4 44 444 *4 44. + 4 4+ 4 44 444 444 ^ 44 444 +44 4+ 444 +44 4+ 444 44. 44 444 44 444 44 44. 444 t 44 t l? 444 tt 100% 87% 80% 60% 40% 20% 0% 33% VIM 444 44+ +44 +44 444 444 444 *44 4.4 44* 444 444 82% 67% 69% yrs- . 44 ►44• ►44 ►4. .44 .44 44 . 44 •► ▪ 4+ ►44. ►4* 44 , ►44 ►++ ►44 4++ 444 . 4* ►44 72% 67%— 80% 4 ▪ 4 44 +4 44 44 44 44 44 +4 44 44 4+ 44 Figure 11. Correct oral health knowledge (%) in "3-visits" clients (N=39) 1. Bleeding 2. Sweet foods 3. Brush at^4. Mother's 5. Clean baby's 6. Sleep with 7. Baby teeth normal when cause cavities? least 1X/day?^oral health^mouth?^bottle?^affect adult pregnant?^ affect baby? teeth? 0 Visit #1 ^ Encounter #2 ^ Encounter #3 + p < 0.05, when compared to Visit #1, chi square test Questions #2-4 were only asked at Visit #1 and Encounter #2 Questions #5-7 were only asked at Visit #1 and Encounter #3 A statistical significant improvement in correct answers was found for questions related to bleeding gums (p=0.000), effect of maternal oral health on baby (p=0.005); and sleeping with bottle/on breast (p=0.007). Women also answered questions related to the effect of baby teeth on adult teeth, infant oral care, and the relationship between sweet foods and cavities but percentage of correct answers did not significantly improve. Clinical findings. Clinical assessments were completed for 63.9% (39/61) at Visit #3. For the 36 participants in the 3-visit group, Figure 12 gives the means and p-values of each clinical index. 91 Figure 12. Indices in "3-visits" clients (N=36)* Visit #1 Visit #2 Visit #3 Plaque Index n^mean36^0.81 sd 0.56 n 1 mean  0.24 sd 0.47 n 36 mean^sd 0.18^0.34 i lk p = .000 • • p = .533 f I p = .000 Calculus Index n^mean 36^1.94 sd 0.53 n 36 mean 1.01 sd 0.57 n 36 mean^sd 1.07^0.26 i • • p = .000 p= .672 I p = .000 Gingival Index n^mean 36^1.37 sd 0.58 n 36 mean 0.58 sd 0.61 n 36 mean^sd 0.73^0.49 I • • p = .000 • p = .107 I p = .000 Bleeding on Probing Index n^mean 36^0.50 sd 0.36 n 36 mean 0.23 sd 0.29 n 36 mean^sd 0.31^0.29 I • • p = .000 • p = .041 I p = .004 Pocket Depth n^mean36^3.89 sd 0.49 n 1 mean 3.54 sd 0.41 n 36 mean^sd 3.46^0.28 I •t p = .000 A p = .323 f i p = .000 * Wilcoxon Signed-RankTest 92 All the clinical indices were found to be significantly improved between Visit #1 and #2 (short-term) as well as between Visit #1 and #3 (medium-term). However, between Visit #2 and #3, the bleeding upon probing index (p=0.04) significantly increased, that is, more bleeding was noted. Satisfaction with program. Feedback about program satisfaction was derived from questionnaires. At 1 st visit, 95.1% (58/61) believed the dental program was good for pregnant women; 100% (52/52) confirmed that they were glad they came to the dental program at 2 nd visit; and 100% (43/43) at 3 td visit indicated that they would like to return to this clinic for future cleaning visits. Other results at the 2nd encounter (N=61) were as follows: • 92.3% (48/52) reported their mouth felt better after the first visit; • 88.2% (45/51) indicated that the verbal information helped them a lot; • 78.9% (41/52) said the written information helped them a lot; • 65.4% (34/52) preferred English as the written dental information with other language preferences being Punjabi, Vietnamese, Hindi, Urdu, Farsi, and Spanish; and • 80.4% (41/51) said taking care of their teeth was more important to them now since coming to the HBP Dental Program. Only 2.3% (10/43) of the 3-visit group answered the program comment and suggestion question. Written comments on questionnaire #3 were that the program was "good" (3), "nice, helpful and important", and that "hygienist should be commended". Suggestions for improvements were more flexible program hours, more follow-up visits, follow-up appointments every three to four months, and dental check up for baby when tooth erupts. 93 4. Discussion 4.1. Program Evaluation The process of the program evaluation and the findings of the evaluation of the HBP Dental Program will be explored separately under ■ strengths, ■ challenges and limitations, and ■ recommendations 4.1.1. Overview. Accepted principles and concepts of program evaluation were applied to this evaluation: a logic model of program theory, an organizational chart and a description of the HBP Dental Program were developed. Areas were identified for improvement. This evaluation highlighted the importance of having well-defined and feasible program objectives and goals in order to direct the program's activities to achieve successful outcomes. Identifying the roles of the stakeholders and developing an organizational flowchart and logic model helped clarify reporting, funding structure and process activities. Examining the program from the perspectives of the various stakeholders helped corroborate findings and explore strengths and challenges of the program. Incorporating the essential steps of evaluation helped provide a thorough representation of the program and allow for proper interpretation of its outcomes. The evaluation even questioned some of the program's underlying assumptions e.g. client's ability to access outside dental providers on the referral list. Despite limited resources, language barriers, unclear goals and organizational structure and other challenges, significant positive short- and medium- term outcomes related to oral health behaviors, oral self-care skills, and oral health status of the clients were documented. However, a majority of the women were unsuccessful in obtaining the other dental services recommended to them. 94 Areas identified for improvement were data collection mechanisms, communications, promotion of the HBP Dental Program within and outside the VCDP, collaboration with the dental community, and appointment management (time over-runs, cancellations). Evaluating the program from the perspective of an "insider" facilitated access to data and to personnel, ensured "do-ability" of evaluation activities, reasonableness of recommendations to program and staff, and provided the credible insight to interpretation of "hard data" that an outside evaluator may not have known. However challenges of time commitment, workload, bias, conflicts of interest, and differentiation of evaluator from clinician role were encountered. The resulting information was useful to decision-making stakeholders and the outcome results (behavior, knowledge, and oral health status) were similar to those of previously cited oral health promotion programs (Gunay et al., 1998; Gaffield et al., 2001; Gomez & Weber, 2001; Gomez, Weber, & Emilson, 2001). 4.1.2. Strengths. Evaluability assessment (EA). Performing an EA was invaluable because it definitively identified the stakeholders, their related roles and responsibilities, and the main objectives of the program and of the evaluation. The EA resulted in a logic model of program theory and overall, a feasible evaluation. These benefits from the EA were similar to those reported from a recent evaluation of a Calgary mental health service program where the EA served as a "rapid feedback evaluation" (Thurston, Graham, & Hatfield, 2003). The EA identified areas that needed improvement prior to the evaluation (e.g. mechanisms of data collection) and revealed the necessity to perform both a descriptive and a process evaluation before considering an outcome evaluation. Describing the program theory was critical to identify the inputs, activities, and outputs of the program and to examine the feasibility of desired outcomes and impact. The development of a logic model 95 helped to visualize the program theory to allow for its critical examination and the ongoing problematic assumptions. Furthermore, the EA helped focus on whether goals were appropriate, rather than whether or not a program was meeting its goals. This critical aim of an EA helps avoid measuring outcomes that do not represent real program "possibilities". For example, one of the stated HBP Dental Program goals was to reduce cavities in the children of women who participated. Achieving this objective was not realistic or feasible because the program did not provide a full range of dental services to the women. In addition, there were no factors enabling women to obtain these services (e.g. close collaborations with outside service providers or financial support). In addition, fulfilling the goal of reduced dental decay in children of HBP dental clients would be difficult to measure because 1) long-term follow-up is required and 2) no monitoring system is in place to track the dental health of children of these mothers. For effective evaluation, program objectives not only need to be well-defined but their achievement needs to be measurable and most importantly, plausible (Horst et al., 1974). The results from the EA helped in the selection of success indicators and measurements that were useful, user-friendly and did not interfere with daily program activities. For example, if improvement in periodontal health was selected as a success indicator, the measurement chosen should have been clinical attachment loss (CAL). However, if this measurement was chosen, outcomes would have been unfavorable because, as revealed by the EA, sufficient periodontal debridement was usually not completed for clients to permit an accurate measurement of CAL. Descriptive evaluation. The descriptive evaluation or program description documented facts about the program and its history, the physical environment that housed the program, and the client demographics and their oral health concerns and needs and will become a permanent record of the program. 96 Information about HBP dental clients was collected from the chart review (N=123) over two years (2004, 2005) and from the cohort of women (N=61) who were followed over 3-visits into the postnatal period from 2006 to 2007. Because the data collection for the latter group was more intense and rigorous, the two data sets were not pooled together, but it is of interest to examine what the data "tells us" about HBP dental clients over a four year period. The clients were generally similar regarding program attendance, age, gestational weeks at both prenatal visits, proportion who had PTLBW babies, and most common area of origin. Changing trends were noted in English speaking capability and reported oral health concern. Although the most common area of origin remained Southern Asia n in both groups, the 3-visit group had more women from Central America 14 and fewer from Western Asia 15 . This change may be attributed to the shifting immigrant and refugee populations served by the HBP prenatal program. The 3- visit group also had more women with fluent English-speaking capability (57.4%, 35/61) compared to the chart review (45.5%, 56/123). This change may be a result of a change in Canadian immigration policies to attract immigrants with greater job skills. Although "bleeding gums" remained the most frequently reported oral health concern by both groups, 90.2% reported it from the 3-visit group compared to 78.0% from the chart review group. The difference may be due to the referring HBP staff being more aware that "bleeding gums" were a dental concern, after a 2004 HBP staff meeting. Process evaluation. Process activities are crucial because they influence the future outcomes of the program and of program evaluation. Judging a program completely on outcomes may be inappropriate or impractical, especially if there is a problem with program implementation (King, Morris, & Fitz- 13 Afghanistan, Bangladesh, India, Iran, Nepal, Pakistan, and Sri Lanka 14 El Salvador, Mexico, and Nicaragua 15 Iraq, Lebanon, and Saudi Arabia 97 Gibbon, 1987). Ineffective results may be a result of poor implementation of the program's activities, rather than a result of an ineffective program. Examining the relationship of inputs to outputs in the logic model suggested many areas in need of improvement: 1) centralized client database, 2) updated client and appointment tracking forms, 3) revision of dental and periodontal charting procedure, 4) summarized dental findings and recommendations for client, 5) improved communications between VCDP and HBP staff Some of these recommendations were already being implemented by the program's hygienist (DL) with the support of the main stakeholders. This ongoing formative process evaluation by the insider evaluator (DL) was valuable to the stakeholders because it addressed certain areas they already knew to be "challenging", the improvements were made in a timely manner not at the completion of the "evaluation report", and the changes could be revised and revisited on an ongoing basis. Outcome evaluation. One of the main outcome "successes" was an observed significant improvement in women's tooth brushing skills. Nevertheless, some of the usual cautions related to observational data bear consideration (Kerlinger, 1986). Suggested ways to minimize observer bias and maximize consistency include trained observers, systematic data collection methods and unobtrusive observers (Hodges & Videto, 2005). Obviously, for DL in her clinician role, she could not be unobtrusive. In the observations of brushing, some of the demonstrations could have been influenced by unclear understanding of instructions at the first visit or by the woman's comfort level in demonstrating brushing in front of a stranger (DL). At the second visit, the client's self-consciousness may have diminished. However the significantly improved clinical findings at subsequent visits paralleled the observed improved brushing ability and confirmed DL's ability to evaluate tooth brushing skills. Furthermore, her 18 years of practice as a clinician confirms her as a "trained observer". 98 Another strength of the outcome evaluation included the use of several clinical indices to provide a general overall description of each woman's oral health status. Insider evaluation. The advantages of an insider evaluation were similar to those previously cited: economics, evaluator access to data, program, and to staff, existing relationship, ability to monitor program activities and address areas of concerns as they occur (Klein & Johnston, 1979; Bonner & Tohurst, 2002; Minkler, 2004). The existing relationship allowed positive cooperation from both HBP and VCDP staff in all evaluation activities and facilitated timely implementation of some process recommendations. Furthermore, as an insider, DL was present to implement the improvements as well as to ensure that the findings benefit the program and its clients, not just the researcher. Another benefit was the ability of the insider evaluator to add her personal and "frontline" insight to "hard data" to avoid misinterpretation of results. For example, analysis of referral patterns demonstrated that two of the 11 HBP staff was responsible for referring the majority of the HBP dental clients, despite the fact that all the HBP staff had good oral health knowledge. However, over time DL realized that client groups managed by other HBP staff often simply did not meet some of the screening criteria restrictions: "reliable appointment- keeping" clients and/or no dental insurance. Furthermore, as an insider, she was also aware that the lack of clinical improvement in pocket depth that might be construed by an "outsider" as ineffective treatment was likely a result of insufficient clinical time for clients. 4.1.3. Limitations. The evaluation was limited to the scope outlined at the outset by the main stakeholders (Methods, p.49 - 50). Only 10% of the HBP prenatal population was referred to the HBP Dental Program, thus caution is indicated before generalizing results of the outcome assessment to the 99 entire group of HBP clients. In addition, the cohort of women was a volunteer, convenience sample and the number who attended "3-visits" was modest. However, data from 40 out of 61, or 66%, of women who attended over one-year is credible and at least can be generalized to the HBP women referred to the dental clinic. Any selection bias is within acceptable limits. 4.1.4. Challenges. Insider evaluation. While the advantages of insider evaluation are many, the disadvantages of being an insider evaluator warrant consideration (Klein & Johnston, 1979; Bonner & Tohurst, 2002; Minkler, 2004). The "evaluator-clinician" dichotomy is also of interest. • Conflicts about evaluator tasks versus clinician responsibilities to clients. For example, clinician's concern that probing with the pressure calibrated instrument that was used could be uncomfortable to inflamed tissue versus the need to use a standardized measurement instrument for accurate data collection. • Extra time needed for accurate and comprehensive data collection and implementation of process recommendations left little time for casual conversations with clients and other staff. • Pre-existing relationships with stakeholders may have influenced evaluation activities positively and negatively. For example, DL may have been more comfortable interviewing certain stakeholders compared to others and may have been hesitant to address sensitive and long-standing problem areas. In addition her "evaluator role" may have not been taken seriously by others and her recommendations may have been seen by other staff as promoting "her" program over the main dental program. • Since DL was the hygienist involved in delivering the clinical services, there may have been a perceived bias in data collection and reporting of results (e.g. the finding of 100 improved gingival health). However, program records from 1986 already anecdotally reported improvements in gingival status. Basic clinical hygiene services are well-known to lead to decreased gingival bleeding and improved gingival health status in the short and medium term (Tan & Saxton, 1978). The finding of improved gingival health was not unexpected but was certainly commendable considering the limited time for clinical treatment and the challenges presented by the pool of patients. Another bias of the evaluation may have been the objectivity of a dental hygienist in reporting results. DL may have unconsciously focused more on the dental hygiene issues at the expense of other health and social issues. Bias is always a concern of insider evaluations. However "there is no such thing as a neutral evaluation" (Lovato, 2006). Adherence to appropriate methodologies, assuring validity of measurement instruments, and attention to evaluation frameworks are essential to negate concerns with insider evaluation. The challenge is to recognize and balance the strengths and weaknesses in both evaluator and clinician roles and objectively conduct and evaluate the program. Measurements Clinical measurements were selected to accommodate time restrictions and for ease of use, patient comfort and compliance. Many periodontal indices were considered but most did not meet our "selection criteria" or were simply not feasible due to presence of extensive calculus. It must not be forgotten that seven out of 10 participants either had had no scaling for over two years or ever in their lifetime. The Eastman Interdental Bleeding Index (Darby & Walsh, 2003) was tested but the women found it too invasive. The Periodontal Screening and Recording (PSR) Index (Darby & Walsh, 2003) was considered but the specially designed probe was not available nor was screening the entire dentition feasible in the time available. Absence or decrease of BOP is considered a reliable indicator that gingival health has improved (Darby & Walsh, 2003). 101 Selecting six "index" teeth for measurement of clinical status is considered to accurately represent the dentition (Ramfjord, 1967). However DL felt that limiting the recording to six teeth under-reported the "severity" of the disease because many of the "unselected" molars had extensive periodontal pocketing and mobility. However, the rationale for the selection of only six teeth was related to time available and the method provided reliable data about periodontal concerns. Further testing of practical clinical indices that are also time efficient and easy to use in public health settings is needed. The usual cautions are also applied to self-reported data from the interviews e.g. reported attention to dental care for other children (Reisine & Douglass, 1998). However "appointment tracking" of the children of the HBP dental clients was not feasible or within the scope of this evaluation. Other challenges. • Operational changes in record keeping and administration of front reception desk negatively influenced data retrieval and appointment monitoring. However, exclusion of the HBP Dental Program from the updates highlighted the low priority of the program since the new record-keeping and computer systems were designed specifically for information on child patients, not the HBP clientele. • Successfully contacting the clients for the 3 rd visit due to invalid contact numbers or inability to leave messages. The HBP population tended to be transient, especially those who were young, refugees or new immigrants. Further analysis revealed that the majority of the non-responders for the 3 rd visit were primarily from the young, Canadian- born, English speaking group. Only one out of four from this participant group attended the postnatal visit. • Maintaining the interest and involvement of the main stakeholders in the evaluation over 3 years. 102 • Inability to analyze the correlation between tooth brushing observations and clinical indices, brushing observations and reported bleeding, and reported bleeding and indices due to differing measurement scales. Scales used for measurement should be similar (ordinal, nominal, and interval) to allow triangulation of data. • Obtaining a more complete description of oral health status. However this was not feasible in this public health clinic due to time restrictions. • The evaluator's background (English speaking, graduate student of a middle class background, and extensive experience in private clinical practice) likely influenced the evaluation (e.g. wording on questionnaires, choice of measurement tools and expectations for clinic operations). 4.1.5. Recommendations for program evaluation in similar settings. • Program evaluations are needed in order to provide useful information to enable stakeholders to make informed program decisions. • An Evaluability Assessment should be completed as a first step, especially for programs with limited information available. • Evaluations with both external and internal evaluators may help provide balanced evaluations and will encourage a "positive evaluation" experience for all parties involved, and will ensure that findings are mutually beneficial. • Process evaluations need to emphasize strengths and potential for program to improve; be sensitive to staff during the evaluation to avoid creating dissension; and to incorporate data from multiple stakeholders. • Incorporation of data from several sources e.g. charts, interviews, observations are crucial to allow for accurate data and representation of events. 103 • Frequent feedback or evaluation updates to those involved is essential to maintain interest and participation of main stakeholders as well as to establish trust in evaluation procedures. • Results from evaluation need to be presented to policy-makers and decision-makers to encourage use of results for program improvement. 4.2. Healthiest Babies Possible Dental Program. 4.2.1. Strengths. Relationship with the HBP prenatal program. There has been a positive working relationship between the main coordinators and staff of the VCDP and the HBP prenatal programs for over 20 years despite changes in staff and changes within the organizations themselves. Both programs recognized the oral health needs of HBP women, were committed to helping these women, and were receptive and supportive to improving the HBP Dental Program. The HBP staff demonstrated good peri-natal oral health knowledge and has made appropriate referrals based on dental need to the HBP Dental Program over the years. All the clients who attended required both educational and clinical oral health services. However, staff from both programs recognized that the HBP Dental Program, by providing a dental assessment, oral health education, preliminary hygiene therapy, and referral to low cost dental service providers, only served as a starting point for the client to improve her oral health. Program Activities. The HBP Dental Program had the right "type" of activities, that is, a combination of clinical and educational services. "Chair side" education alone would have been insufficient for these clients. The limited clinical hygiene services combined with education helped reduce the 104 women's oral concerns, especially the frequently reported concern of "bleeding gums". These results supported findings from other intervention programs with both clinical and educational components (Tan, 1979; Petersen, 1989). Outcomes. Despite the many challenges that the HBP Dental Program faced, significant short- and medium- term improvements in outcomes (knowledge, behaviors, skills, and gingival health) were demonstrated. Mothers acquired new and important knowledge on topics like the relationship between a mother's oral health and that of her baby, and the fact that bleeding gums are not a "normal" part of pregnancy. For areas where mother's already had good knowledge, for example, sweet foods cause cavities, significant improvements were not noted. The lack of improvement on these more "common sense" topics was not surprising, given the simplicity of the questions. A variety of factors may have encouraged the clients to seek professional dental care for their other children. Factors included 1) the woman's new awareness of the free dental clinic for children at NCHO; 2) the women, having now established a relationship with the clinic, felt more comfortable bringing their other children for care; 3) encouragement from DL; and 4) women's newly acquired knowledge about the importance of baby teeth. Improved oral self care (tooth brushing) and the modest clinical intervention (superficial periodontal debridement) together led to positive changes in gingival health. The observed, improved brushing technique may have been a result of a combination of the smaller, easier-to- use toothbrush recommended by DL; the provision of one-to-one oral hygiene instruction; and decreased gingival sensitivity to brushing. The improved behavior and knowledge outcomes of this program evaluation generally support other reports regarding the receptivity of pregnant women to new health behavior and knowledge (Gunay et al., 1998; Gaffield et al., 2001; Gomez & Weber, 2001; Gomez, Weber, & 105 Emilson, 2001). Infant oral care was reported by 93% of women despite evidence of a decline in personal oral hygiene. The mean age of the infants at the 3 rd visit was 6.6 (1.6) months. Given the challenges of life with a newborn, we are encouraged by the mother's adoption of and attention to infant self care practices. The demonstrated positive outcomes of this program may have been heightened by a self- selection bias of more motivated clients who attended all 3 visits (Harrison & Wong, 2003). The positive changes observed in 66% (40/61) of the women who attended over time certainly warrants further follow-up with a clinical trial with a more robust sample size. Together the individual patient counseling and translated handouts were reasonably effective in relaying information that translated into positive behaviors across diverse cultural backgrounds despite language barriers, limited availability of translations, and a unilingual hygienist. However, it is noteworthy that the majority of the outcome participants indicated that the verbal information from DL was more helpful than the written information from pamphlets. This finding is well documented in the health education literature (Weinstein, Harrison, & Benton, 2006). DL tried her best to be culturally sensitive based on skills learned in previous workshops she attended and her own immigrant family background. She used simple terms and graphics to enhance each oral health counseling session. In addition, the extra time that she devoted to clients at early visits help establish a relationship with each woman Such a relationship is key to enabling behavior change (Weinstein, Harrison, & Benton, 2004; Weinstein, Harrison, & Benton, 2006). The benefits of using a culturally sensitive approach in oral health education have been previously reported (Harrison & Wong, 2003; Brown, Canham, & Cureton, 2005). A low proportion of PTLBW babies were born to the women in the HBP Dental Program. From 2004 to 2007, less than 2% of HBP dental clients had PTLBW babies and less than 5% had babies who were either low birth weight or premature. In 2004, 7% of women from the HBP 106 program had premature babies; less than 6% had low birth weight babies (Healthiest Babies Possible Program, 2004). Despite the potential high risk of our clientele to have PTLBW infants, the rate was slightly lower than the norm for women from similar programs in B.C. However caution is needed in interpreting these results. We cannot quantify the role, if any, that this dental program played in the low number of PTLBW babies born to participating women. The HBP prenatal program and staff (dietitians, nurses, and counselors) spends considerable time and effort in counseling, and supporting the women to have healthier pregnancies and lifestyles. This low number of PTLBW infants in this sample of women may be a result of the success of the overall HBP prenatal program, over- estimation of the women's risk to PTLBW babies or the simple fact that women who attended the HBP Dental Program even for one visit were generally motivated to improve pregnancy outcomes. We reported the gestational week at birth and birth weight for descriptive purposes only and did not have as an original goal an exploration of the association of periodontal disease and its treatment on birth outcomes for this high risk group of women. 4.2.2. Challenges. Direction. The coordinators from both VCDP and HBP programs have competing priorities and responsibilities which lead to differing goals for the HBP Dental Program. The lack of a coordinator dedicated solely to the HBP Dental program resulted in more general goals, a dispersed information reporting structure, limited program knowledge on a day-to-day basis, minimal integration of the HBP Dental Program with main VCDP operations, decentralized record-keeping of client information, uncertain direction for program's future direction, and inattention to challenges of the program. The goal to help the HBP dental clientele with their 107 oral health concerns has limited potential to be achieved because the objectives, although clearly stated, had no pre-determined success indicators. Resources. Currently the program has insufficient resources (funding, appointments available for clinical hygiene treatment) and is deficient in key dental services (restorative) for HBP women. Available funding determines the availability of appointments. The few appointments per month affect the program in many ways: • client accessibility to the dental program; • few people within and outside NCHO even know the program exists because of its limited nature; • increased missed appointments since first visits are often booked long after the initial screening date; • limited time for appointments which affects depth of treatment provided and intensity of counseling, a situation frustrating to provider and client. The resource allocation of the HBP Dental Program is influenced by the VCDP's primary activities and available funding. For example, the opportunity to fund for "extra" clinic days for the HBP dental clients is dependent on the VCDP's overall dental budget. Furthermore, incorporating restorative procedures for adults into a child-centered dental clinic necessitates additional dental materials and equipment at the risk of affecting choices for low-income children. Both programs were aware of limitations in services, and as the process and outcome evaluation confirmed, the majority of women required more services than what the program was able to offer. Overall, very few women obtained the other dental services advised. Outside dental providers were not accessible in terms of location, finances, wait period, and language for these women Better collaboration with community dental providers is needed to increase 108 accessibility and availability of treatment services for these women. Existing resources limit the "power" of this dental program and its potential to help the women it serves. Referral process and appointments. Although the HBP staff are making appropriate referrals based on dental need, a high percentage of clients miss appointments. For example, one HBP staff worker referred 18 women to the program in 2004-2005, but only one woman attended. Reasons for missed appointments may include client's personal concerns about dental treatment and the challenges of confirming appointments. Further strategies to limit this critical lost appointment time should be explored. Outcomes. While the oral health outcomes generally improved by the postnatal visit (Visit #3), there were setbacks in two areas between Visit #2 and #3: reported pain (Figure 9) and gingival bleeding (Figure 12). Both of these clinical outcomes improved by Visit #3 compared to baseline. The reports of pain may have decreased at Visit #2 due to reduced discomfort from gingival tissues but may have increased again by Visit #3 because so few women had their other dental problems (dental caries, abscessed teeth) treated. Gingival bleeding may have increased between Visit #2 and Visit #3 because the mother's responsibilities for her new baby afforded her less time to address personal self care needs including oral self care (Redford, 1993). The increased gingival bleeding between Visit #2 and #3 also confirms the need for regular professional dental maintenance at "reasonable intervals". The time period between Visits #2 and #3 ranged from 31 to 70 weeks, much longer than the customary recall interval of 24 to 36 weeks in private dental practice settings. Vancouver Community Dental Program infrastructure Since 2004, there have been 5 different teams of full-time receptionists and often temporary clerical staff during transition periods. These transitions in reception personnel affected appointment monitoring of active clients, familiarity with the HBP dental process, 109 record-keeping, and flow of information. In addition, the record-keeping system, PARIS, and the computerized appointment scheduler did not incorporate all necessary HBP program data aside from contact information. With the effectiveness of the HBP Dental Program that has been documented in this evaluation, it is hoped that the HBP Dental Program will achieve well- deserved and increased priority within the VCDP, including record-keeping and data management. 4.2.3. Recommendations. Direction. • Review objectives and goals, management, and future directions of the HBP Dental Program. • Negotiate increased commitment toward HBP Dental Program from coordinators of both VCDP and HBP programs in regards to resources (funding, type of activities). • Develop strategies to increase interest and commitment from community dental providers to providing accessible services to HBP women. Communication. • Determine a proper information feedback loop outside and inside of VCDP to enhance the profile of the program, eliminate misinformation, identify concerns, and improve client attendance. For example, have regular meetings with main stakeholders and with dental staff. The HBP hygienist should strive to be updated on VCDP's activities and to increase communication and promotion of the benefits of dental prenatal programs. • Generate reports on client attendance for HBP staff to allow problem-solving of appointment-keeping issues. • Schedule an annual meeting of HBP dental staff with all HBP staff 110 Referral process. • Problem-solve with main program coordinators those unclear areas for client referrals e.g. woman has dental insurance but is more comfortable for a variety of reasons attending the HBP Dental Program for hygiene than an outside provider. Should she still get an HBP appointment? • Strategize with front desk reception about methods to facilitate an efficient appointment process and follow-up (e.g. confirmation of appointments at least two days prior to allow for efficient use of available appointment times). Program activities. • Discuss standardizing diagnosis of treatment needs and scheduling of treatment by sessional dentists. • Develop guidelines to improve management of hygienist's daily schedule to avoid appointment over-runs. • Explore benefits and risks of separating the clinical services from patient education to allow more strategic use of "operatory" time. • Future mini-evaluations (interviews) regarding process and outcome either annually or every two years Record-keeping. • Centralize the client database and situate client records in one physical location. • Reduce the number of forms being used for collection of client data. 111 4.3. Future directions. • Results of this evaluation need to be presented to various stakeholders, especially to those with decision-making and financial authority. • The results of this evaluation should be disseminated to encourage development of similar dental prenatal interventions or more evaluations of existing dental programs to determine patterns. Other dental providers like university or community college dental clinics and non-profit clinics could be considered as sites for similar dental prenatal services. • Consideration should be given for increased funding from within government (for example, VCHSDA) and outside government (e.g. dental companies or charitable organizations). • Since our postpartum moms were so motivated, the program should be developed into the postnatal period. This longer term program might include postnatal telephone follow-up (after 3 months) to reinforce infant oral care message and the early dental visit or postpartum educational sessions with HBP groups. Oral health programs focusing on the early postnatal period may help to control early childhood caries (ECC) since behaviors that promote health are established in infancy and early childhood (Ismail, 1998). • An economic analysis of the program is warranted considering the positive outcomes demonstrated from this evaluation. If future research demonstrates a reduction in ECC in the children of the HBP dental clientele, an evaluation highlighting the financial savings may encourage additional funding to this HBP Dental Program or similar dental prenatal programs. • Research is warranted to see if increased accessibility and availability of dental services will motivate high-risk pregnant women to obtain the recommended treatment. Improved 112 collaboration with the dental community and increased promotion of the benefits of prenatal oral health is needed to better understand the myths surrounding dental care for pregnant women. Further research should also address the predisposing, enabling, and reinforcing factors that affect access to dental care. 113 5. Conclusion and summary This evaluation has demonstrated the benefits of conducting a program evaluation and the challenges of the evaluation to the program under evaluation, the HBP Dental Program. The insider evaluator perspective added unique insight to the analysis and helped to maintain the interest of the stakeholders. The evaluation developed a record of the program, provided formative and summative evaluation, and prepared the program for future evaluations and research. Interpreting a program by looking only at outcomes is dangerous. A more systematic analysis involving an EA is needed to better understand the relationship between program implementation and outcomes. This evaluation has provided evidence to demonstrate the program's success despite the many challenges and barriers. Strategies have been suggested to make the program more effective. Areas in need of program improvement in the HBP Dental Program were identified. The limited funding, unclear objectives, and lack of collaboration with the dental community were recognized as major program challenges. However, positive improvements in client knowledge, behavior, gingival health as well as client satisfaction with the program were demonstrated. These results support the continuing existence of the HBP Dental Program and provide encouragement for additional resources or similar dental prenatal interventions. Evaluations of publicly-funded dental programs are needed to provide programs with the information needed for improvement. Only with evidence-based data can decision-makers make more informed decisions and recommendations regarding the future of public dental interventions. This evaluation has incorporated various frameworks and the methodology can be adapted to guide future evaluations of publicly-funded dental programs. 114 This evaluation research has transferred research knowledge into practice, that is, knowledge of scientific and evaluation principles and methods have been applied to a work setting. It has provided information useful to stakeholders and practice-based evidence for future dental prenatal intervention and evaluations. 115 Bibliography American Academy of Peridontology. (2004). American Academy of Periodontology Statement regarding periodontal management of the pregnant patient. Journal of Periodontology, 75(3), 495. American Evaluation Association. (July 2004). Guiding principles for evaluators. Retrieved October 24, 2005 from Association of Dental Surgeons of British Columbia. (November 23, 2001). Children's dentistry task force report. Vancouver: ADSBC. Berkowitz, R. J. (2003). Acquisition and transmission of mutans streptococci: Journal of the California Dental Association, (February), 1-6. Bobetsis, Y. A., Barros, S. P., & Offenbacher, S. (2006). Exploring the relationship between periodontal disease and pregnancy complications. Journal of American Dental Association, 137(October), 75-135. Boggess, K. A. (2006). Oral health in women during preconception and pregnancy: Implications for birth outcomes and infant oral health. Maternal and Child Health Journal, (10), 169-174. Bonner, A., & Tohurst, G. (2002). Insider-outsider perspectives of participant observation. Nurse Researcher, 9(4), 7-19. British Columbia Reproductive Care Program. (2005). Obstetric guideline 2A preterm labor: BCRCP perinatal database registry. Retrieved November 6, 2007 from 116 Brown, R. M., Canham, D., & Cureton, V. Y. (2005). An oral health education program for Latino immigrant parents. The Journal of School Nursing, 21(5), 266-271. Burry, J. (1973). R and D in education: Foundations for evaluation. UCLA Educator, 15(Winter), 14-16. Caufield, P. W., Cutter, G. R., & Dasayanake, A. P. (1993). Initial acquisition of mutans streptococci by infants: Evidence of a discrete window of infectivity. Journal of Dental Research, 72(1), 37-45. Covington, P. (1996). Women's oral health issues: An exploration of the literature. The Probe, 30(5), 173-177. Cruz, G. D., Roldos, I., Puerta, D. I., & Salazar, C. R. (2005). Community-based, culturally appropriate oral health promotion program for immigrant pregnant women in New York City. New York State Dental Journal, 71(7), 34-38. Darby, M. L., & Walsh, M. M. (2003). Dental hygiene theory and practice (2nd ed.). St. Louis: Elsivier Science (Saunders). Dasanayake, A. P. (1998). Poor periodontal health of the pregnant woman as a risk factor for low birth weight. Annals of Periodontology / the American Academy of Periodontology, 3(1), 206-212. Davenport, E. S., Williams, C. E., Sterne, J. A., Murad, S., Sivapathasundram, V., & Curtis, M. A. (2002). Maternal periodontal disease and preterm low birthweight: Case-control study. Journal of Dental Research, 81, 313-318. 117 Davenport, E. S., Williams, C. E., Sterne, J. A., Sivapathasundram, V., Fearne, J. M., & Curtis, M. A. (1998). The east London study of maternal chronic periodontal disease and preterm low birth weight infants: Study design and prevalence data. Annals of Periodontology, 3, 213-221. Dharamsi, S. (2003). Discursive constructions of social responsibility. (PhD Thesis, University of British Columbia). 1-288. Dickie, J. (2005). Healthiest Babies Possible Program. Personal Communication, November 9, 2005. Fine, J. B., & Yao, S. (2007). The influence of periodontal inflammation on systemic diseases and medical conditions. Access, May-June (14), 19. Frechtling, J., & Sharp, L. (1997). User-friendly handbook for mixed method evaluations. Arlington, VA: National Science Foundation. Gaffield, M. L., Gilbert, B. J., Malvitz, D. M., & Romaguera, R. (2001). Oral health during pregnancy: An analysis of information collected by the pregnancy risk assessment monitoring system. The Journal of the American Dental Association, 132(7), 1009-1016. Gomez, S. S., & Weber, A. A. (2001). Effectiveness of a caries preventive program in pregnant women and new mothers on their offspring. International Journal of Paediatric Dentistry, 11,117 -122. Gomez, S. S., Weber, A. A., & Emilson, C. G. (2001). A prospective study of a caries prevention program in pregnant women and their children five and six years of age. ASDC Journal of Dentistry for Children, 68(3), 191-195. 118 Gravetter, F. J., & Wallnau, L. B. (2004). In Knight V. and Moneypenny D. (Eds.), Statistics for the behavioral sciences (6th ed.). Australia: Wadsworth, Thomson Learning, Inc. Green, J. C., & Caracelli, V. (Eds.). (1997). Advances in mixed-method evaluation: The challenges and benefits of integrating diverse paradigms. San Francisco, CA: Jossey-Bass. Green, L. W., & Kreuter, M. W. (Eds.). (1999). Health promotion planning: An educational and ecological approach. (3rd ed.). California: Mayfield Publishing Co. Gunay, H., Dmoch-Bockhorn, K., Gunay, Y., & Geurtsen, W. (1998). Effect on caries experience of a long-term preventive program for mothers and children starting during pregnancy. Clinical Oral Investigations, 2(3), 137-142. Gussy, M., Waters, E. G., Walsh, 0., & Kilpatrick, N. M. (2006). Early childhood caries: Current evidence for aetiology and prevention. Journal of Paediatrics and Child Health, 42(1/2), 37-43. Hale, K. J., Weiss, P. A., Czerepak, C. S., Keels, M. A., Thomas, H. F., Webb, M. D., et al. (2003). Oral health risk assessment timing and establishment of the dental home. Pediatrics, 111(5), 1113-1116. Harrison, R. L., Wong, T., Ewan, C., Contreras, B., & Phung, Y. (1997). Feeding practices and dental health in Vietnamese preschool children. ASDC Journal of Dentistry for Children, 64(2), 112-117. Harrison, R. L., & Wong, T. (2003). An oral health promotion program for an urban minority population of preschool children. Community Dentistry and Oral Epidemiology, 31(5), 392- 399. 119 Healthiest Babies Possible Program. (2004). Annual report 2004. Vancouver Coastal Health. Herman, J., Lyons Morris, L., Fitz-Gibbon, C., Horne, T., Linney, J. & Wandersman, A. (1997). Program evaluation tool kit: Program evaluation - the basics. Retrieved January 9, 2008, from Hodges, B. C., & Videto, D. M. (2005). Assessment and planning in health programs. Sudbury, Massachusetts: Jones and Bartlett Publishers, Inc. Hopkins, W. G. (2006). A New view of statistics. Retrieved January 17, 2007, from http://sportsci.or resource/stats/index.html Horowitz, H. S. (1998). Research issues in early childhood caries. Community Dentistry and Oral Epidemiology, 26(Supplement 1), 67-81. Horst, P., Nay, J. N., Scanlon, J. W., & Wholey, J. S. (1974). Program management and the federal evaluator. Public Administration Review, 34(4), 300-308. Hujoel, P. P., Lydon-Rochell, M., Roberston, P. B., & Del Aguila, M. A. (2006). Cessation of periodontal care during pregnancy: Effect on infant birthweight. European Journal of Oral Science, 114, 2-7. Hyde, S., Weintraub, J. A., & Satariano, W. A. (Spring 2005). An evaluation of the San Francisco Department of Human Services Welfare dental program. Journal of Public Health Dentistry, 65(2), 104-109. Ismail, A. I. (1998). Prevention of early childhood caries. Community Dentistry and Oral Epidemiology, 26, 49-61. 120 Ismail, A. I., Lewis, D. W., & Dingle, J. L. (2006). Prevention of periodontal disease. In: Canadian task force on the periodic health examination. Canadian guide to clinical preventive health care (online). Health Canada. Retrieved April 30, 2006 from http://www.ctfphc.ort/Abstracts_printable/Ch37abs.htm Jeffcoat, M. K., Hauth, J. C., Geurs, N. C., Reddy, M. S., Cliver, S. P., Hodgkins, P. M., et al. (2003). Periodontal disease and preterm birth: Results of a pilot intervention study. Journal of Periodontology, 74, 1214-1218. Jeffcoat, M. K., Geurs, N. C., Reddy, M. S., Cliver, S. P., Goldenberg, R. L., & Hauth, J. C. (2001). Periodontal infection and preterm birth: Results of a prospective study. The Journal of the American Dental Association, 132(7), 875-880. Kerlinger, F. N. (1986). Foundations of behavioral research. (3rd ed.). Fort Worth, TX: Rinehart and Winston. Kidder, L. H. (1981). In Loch D., Johnson J. N., and Katz A. (Eds.), Selitiz Wrightsman & Cook's Research methods in social relations (4th ed.). New York: Holt, Rinehart and Winston. King, J. A., Morris, L. L., & Fitz-Gibbon, C. T. (1987). How to assess program implementation (2nd ed.). Newbury Park: Sage Publications. Klein, N., & Johnston, M. (1979). Insider-out: The health worker as researcher. Nursing Research, 28(5), 312-314. Kohler, B., Andreen, I., & Jonsson, B. (1984). The effect of caries-preventive measures in mothers on dental caries and the oral presence of the bacteria streptococcus mutans and lactobacilli in their children. Archives of Oral Biology, 29(11), 879-883. 121 Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159-174. Larson, A. K. (2003). The early childhood caries prevention program in Palau. Pacific Health Dialogue, 10(1), 106-110. Lewitt, E. M., Baker, L. S., Corman, H., & Shiono, P. (1995). The direct cost of low birth weight - the future of children. David and Lucile Packard Foundation. Lopez, N. J., Smith, P. C., & Gutierrez, J. (2002). Periodontal therapy may reduce the risk of preterm low birth weight in women with periodontal disease: A randomized controlled trial. Journal of Periodontology, 73, 911-924. Lopez, N. J., Da Silva, I., Ipinza, J , & Gutierrez, J. (2005). Periodontal therapy reduces the rate of preterm low birth weight in women with pregnancy-associated gingivitis. Journal of Periodontology, 76(11 Suppl), 2144-2153. Lovato, C. (2005). Health care and epidemiology 506 Introduction to program evaluation lectures. Unpublished manuscript. Lovato, C. (2006). Health care and epidemiology 540 Evaluation research lectures. Unpublished manuscript. Lunardelli, N. N., & Peres, M. A. (2005). Is there an association between periodontal disease, prematurity and low birth weight? A population-based study. Journal of Clinical Periodontology, 32, 938-946. Lux, J. A., & Lavigne, S. E. (2004). Your mouth - portal to your body. Probe, 38(4), 154-171. 122 Lux, J. (2007). Review of the oral disease-systemic disease link. Part II: Preterm low birth weight babies, respiratory disease. Canadian Journal of Dental Hygiene, 41(1), 8-22. Mathison, S. (Ed.). (2005). Encyclopedia of evaluation. Thousand Oaks: Sage Publications, Inc. McGaw, T. (2002). Periodontal disease and preterm delivery of low-birth-weight infants. Journal of the Canadian Dental Association, 68(3), 165-169. Michalowicz, B. S., Hodges, J. S., DiAngelis, A. J., Lupo, V. R., Novak, M. J., Ferguson, J. E., et al. (2006). Treatment of periodontal disease and the risk of preterm birth. The New England Journal of Medicine., 355(18), 1885-1894. Milstein, R. (1999). Framework for program evaluation in public health. Morbidity and Mortality Weekly Report, 48(11), 1-40. Minkler, M. (2004). Ethical challenges for the "outside" researcher in community-based participatory research. Health Education & Behavior, 31(6), 684-687. Mitchell, P. H., Ferketch, S., & Jennings, B. M. (1998). Quality health outcomes model. American academy of nursing expert panel on quality health care. Image--the Journal of Nursing Scholarship, 30(1), 43-46. Mitchell-Lewis, D., Engebretson, S. P., Chen, J., Lamster, I. B., & Papapanou, P. N. (2001). Periodontal infections and pre-term birth: Early findings from a cohort of young minority women in New York. European Journal of Oral Sciences, 109, 34-39. Moore, S., Ide, M., Coward, P.Y. Randhawa, M., Borkowska, E., Baylis, R., & et al. (2004). A prospective study to investigate the relationship between periodontal disease and adverse pregnancy outcome. British Dental Journal, 19 7, 251-258; discussion 247. 123 Moore, S., Ide, M., Wilson, R. F., Coward, P. Y., Borkowska, E., Baylis, R., et al. (2001). Periodontal health of London women during early pregnancy. British Dental Journal, 191, 570-573. National Institute of Dental and Craniofacial Research. (2000). Oral health in America: A report of the Surgeon General. Rockville, MD: Department of Health and Human Services. Offenbacher S., Jared, H. L., O'Reilly, P. G., Wells, S. R., Salvi, G. E., Lawrence, H. P., Socransky, S. S., & Beck, J. D.. (1998). Potential pathogenic mechanisms of periodontitis associated pregnancy complications. Annals of Periodontology, 3(1), 233-250. Offenbacher, S., Beck, J. D., Lieff, S., & Slade, G. (1998). Role of periodontitis in systemic health: Spontaneous preterm birth. Journal of Dental Education, 62(10), 852-858. Offenbacher, S., Katz, V., Fertik, G., Collins, J., Boyd, D., Maynor, G., McKaig, R., & Beck, J. (1996). Periodontal infection as a possible risk factor for preterm low birth weight. Journal of Periodontology, 67(10 Suppl), 1103-1113. Offenbacher, S. (2001). Maternal periodontitis and prematurity. Part I. Obstetric outcome of prematurity and growth restriction. Annals of Periodontology, 6(1), 164-174. Ovretveit, J. (2002). Action evaluation of health programmes and changes: A handbook for a user focused approach. United Kingdom: Radcliffe Medical Press. Patton, M. Q. (1997). Utilization-focused evaluation: The new century text. (3rd ed.). Thousand Oaks, CA: Sage Publications. Petersen, P. E. (1989). Evaluation of a dental preventive program for Danish chocolate workers. Community Dentistry and Oral Epidemiology, 17, 53-59. 124 Petersen, P. E., & Kwan, S. (2004). Evaluation of community-based oral health promotion and oral disease prevention--WHO recommendations for improved evidence in public health practice. Community Dental Health, 21(4 Suppl), 319-329. Petersen, P. E., & Nortov, B. (1994). Evaluation of a dental public health program for old-age pensioners in Denmark. Public Health Dentistry, 54(2), 73-79. Picard, A. (2004, August 5). Painful dental bills hurt health care. Globe and Mail, pp. A13. Ramfjord, S. P. (1967). The periodontal disease index (PDI). Journal of Periodontology, 38(6), 30-38. Redford, M. (1993). Beyond pregnancy gingivitis: Bringing a new focus to women's oral health. Journal of Dental Education, 57(10), 742-748. Reisine, S., & Douglass, J. M. (1998). Psychosocial and behavioral issues in early childhood caries. Community Dentistry and Oral Epidemiology, 26, 32-44. Ross, P. E. (2006). Invaders and the body's defenses. In: Oral and whole body health. Scientific American, (supplement), 6-11. Rutman, L. (1980). Planning useful evaluations: Evaluability assessments. Beverly Hills: Sage Publications, Inc. Scanlon, J. W., Horst, P., Nay, J. N., Schmidt, R. E., & Waller, J. D. (1979). Evaluability assessment: Avoiding types III and IV errors. In G. R. Gilbert, & P. J. Conklin (Eds.), Evaluation management: A selection of readings. Washington, D.C.: Office of Personnel Management, Federal Executive Institute. 125 Scannapieco, F. (2003). Periodontal disease as a risk factor for adverse pregnancy outcomes. A systematic review. Annals of Periodontology, 8(1), 70-78. Shaw, I., & Faulkner, A. (2006). Practitioner evaluation at work. American Journal of Evaluation, 27(1), 44-63. Smith, M. F. (1989). Evaluability assessment: A practical approach. Massachussetts: Kiuwer Academic Publishers. Steinberg, B. J. (1999). Women's oral health issues. Journal of Dental Education, 63(3), 271- 275. Tan, H. H. (1979). Effect of dental health care instruction and prophylaxis on knowledge, attitude and behavior in Dutch military personnel. Community Dentistry and Oral Epidemiology, 7, 252-258. Tan, H., & Saxton, C. A. (1978). Effect of a single dental health care instruction and prophylaxis on gingivitis. Community Dentistry and Oral Epidemiology, 6(4), 172-175. Thompson, N. J., & McClintock, H. 0. (2000). Demonstrating your program's worth: A primer on evaluation for programs to prevent unintentional injury. (2nd ed.). Atlanta, GA: Centers for Disease Control and Prevention; National Center for Injury Prevention and Control. Thurston, W. E., Graham, J., & Hatfield, J. (2003). Evaluability assessment: A catalyst for program change and improvement. Evaluation and the Health Professions, 26(2), 206-221. Vancouver Coastal Health. (2007). Vancouver Coastal Health. Retrieved February 14, 2007, from 126 Vancouver Community Dental Program. (2004). Healthiest Babies Possible Dental Program appointment statistics. Unpublished manuscript. W.K. Kellogg Foundation. (2004). Logic model development guide. Retrieved October 15, 2005, from Watson, M. R., Horowitz, A. M., Garcia, I., & Canto, M. T. (2001). A community participatory oral health promotion program in an inner-city Latino community. Journal of Public Health Dentistry, 61(1), 34-41. Weinstein, P., Harrison, R. L., & Benton, T. (2006). Motivating parents to prevent caries. Confirmation of one-year findings. Journal of the American Dental Association, 137(6), 789-793. Weinstein, P., Harrison, R., & Benton, T. (2004). Motivating parents to prevent caries in their young children: One-year findings. Journal of the American Dental Association, 135(6), 731-738. Wener, M. E., & Lavigne, S. E. (2004). Can periodontal disease lead to premature delivery? How the mouth affects the body. AWHONN Lifelines /Association of Women's Health, Obstetric and Neonatal Nurses, 8(5), 422-431. Wholey, J. S., Hatry, H. P., & Newcomer, K. E. (1994). In Wholey J. S., Hatry H. P. and Newcomer K. E. (Eds.), Handbook of practical program evaluation. (1st ed.). San Francisco: Jossey-Bass Inc. Xiong, X., Bueken, P., Fraser, W. D., Beck, J., & Offenbacher, S. (2006). Periodontal disease and adverse pregnancy outcomes: A systematic review. International Journal of Obstetrics and Gynaecology, 113(2), 135-143. 127 Xiong, X., Buekens, P., Vastardis, S., & Yu, S. M. (2007). Periodontal disease and pregnancy outcomes: State-of-the-science. Obstetrical and Gynecological Survey, 62(9), 605-615. Zanata, R. L., Navarro, M. F., Pereira, J. C., Franco, E. B., Lauris, J. R., & Barbosa, S. H. (2003). Effect of caries preventive measures directed to expectant mothers on caries experience in their children. Brazilian Dental Journal, 14(2), 75-81. 128 Appendix A. Summary of Eight Articles in the Literature Review* 1. Petersen, Poul Erik 2..Brown, R., Canham, D., & Cureton, V. 3. Larson, A. 4. Watson, M.R., Horowitz, A.M., Garcia, L, Canto, M.T. o`' t••" Evaluation of a dental preventive program for Danish chocolate workers. Community Dentistry and Oral Epidemiology, 1989:^ An oral health education program for Latino immigrant Parents.^The early childhood caries prevention program in Palau. Pacific 17(2)53-59.^ The Journal of School Nursing, 2005: 21(5):266-271.^Health Dialog, March 2003: 10(1):106-110. A Community Participatory Oral Health Promotion Program in an Inner-city Latino Community. Journal of Public Health Dentistry. 2001:61(1):34-41. (4'1'^Two Danish chocolate factories in Denmark Even Start clasroom located in a Northern California elementary' Koror, Palau (near the Philippines)school district campus An inner city' Latino neighborhood (Mount Pleasant) of Washington, D.C. xi4 ' ^1984 ^ No specific date provided. Estimated in early 2000s^2002+; evaluation pending^ 1995 -1996 Occupational Health Service (OHS) in Denmark and managers of (Assumed) San Mateo County Dental Health Coalition and ^ (Assumed) The Spanish Catholic Center, Mount Pleasant the two chocolate factories; (Assumed) Danish Dental^Redwood City School District, California; other nurses or health (Assumed) Division of Oral Health, Ministry of Health, Palau^Community, Rosemount Center, Mary's Center, the Latin Association^ professionals^ American Youth Center Ag,^Visible plaque index (VPI), gingival bleeding (GB), calculus ces.^index (Cl), DINES; Dental knowledge, attitudes, dental behavior, social network activities, and perceptions of A440^process Oral health knowledge and reported dental behavior % children with decay history, untreated decay, urgent care,^Fulfillment of the principles of community-based research ECC; oral health knowledge, practices and beliefs^outlined by Israel et al. Effect evaluation: Two year follow-up preventive program included health education (active group involvement, instructions, education) and clinical prophylaxis (4 hygiene visit in 1st year; 2 in the 2nd year with the hygienist). Repeated clinical indices, questionnaires at baseline, after 1 year, and after 2 year. Educational material used and dental home care kits given yearly. Process evaluation included qualitative taped interviews (group and individual) after 12 and 24 months. Preventive program for Latino immigrant parents provided two 1.5 hour lessons of oral health education (interactive, slides, oral hygiene instruction/demonstration, meal planning) by a Spanish- speaking nurse with a bilingual dental instructor. Pre and post questionnaire (English and Spanish) administered to measure changes in knowledge and reported behavior related to oral health. A nutritious breakfast was offered prior to each lesson. Educational pamphlets and child toothbrushes distributed. Various pilot programs: 1) Repeated fluoride varnish for kids start (ii; 1 year, 1.5, 2, 3 years of age. Special needs children were treated at specialized clinics. Oral health tracking card, oral hygiene aids and educational materials were distributed. 2) Pregnant women given exams and pre and post oral health counselling. Any required treatment at no cost. A collaborative community "Family Health Day" to promote all areas of health. 3) Oral health surveillance system being developed to monitor process & health outcomes via program process information. 4) Survey for new mothers 5) Preventive oral care training for primary health care workers, physicians and nurses (PROSPECTIVE) Precede-Proceed planning framework. Identified community networks and established a steering committee. Needs assessment of risk, OH education performed. Developed 3 intervention weeks targeting 11 community sites with activities and mass media campaign. Process evaluation done by steering committee and Impact evaluation performed. Twenty women enrolled in Even Start, a California statewide Workers at two Danish chocolate factories (n=112) were offered literacy program targeting Latin American immigrants, were preventive care. 89 (80%) participated. 31 men and 58 women^invited to partipate. All were parents of children in the local 19-61 years of age. (CONVENIENCE SAMPLE)^school district and designated as having a low socioeconomic status. (CONVENIENCE SAMPLE - PURPOSIVE) Each pilot program has own target. Ranges from all children, their Pregnant women and preschool parents in inner-city fluoridated mothers, pregnant women to all health care providers in Palau. No neighborhood in Washington, D.C. Majority recent immigrants details given. from Salvador. (CONVENIENCE SAMPLE) . Framework source: Ovretveit, John. Action Evaluation of Health Programmes and Changes: A handbook for a user-focused approach. United Kingdom. Radcliffe Medical Press Ltd.. 2002. San Mateo County Dental Health Coalition and Redwood City No information at this time Hygienist was calibrated with the author on clinical recordings. Double examinations of 15 individuals were performed in majority of indices and high consistency was found. Interviews School District, California audiotaped. Standardize oral health message by development of educational materials with content areas to be used by all volunteers. Appendix A. Summary of Eight Articles in the Literature Review* Leo ()" 1. Petersen, Pouf Erik Improved dental health; positive dental health behavior; diffuse changes in dental knowledge and attitudes; majority of workers satisfied with program. Concluded that systematic preventive care can improve dental health. 2..Brown, R., Canham, D., & Cureton, V. 5 participants showed improvement in behavior. 10 showed improved knowledge. Concluded that oral health education for parents is essential in any dental program. 3. Larson, A. Preliminary evidence showed improved health in young children. No other outcomes available. Oral health surveillance system still being developed to monitor both process and health outcomes. Preparation of key indicators data Basic process data being collected. 4. Watson, M.R., Horowitz, A.M., Garcia, I., & Canto, M.T. Implementation of program feasible and useful. Substantial interest in oral health issues and participation in oral health prevention activities. Positive impact on community' and organizations. Fulfilled eight of the nine principles. cr Oboe Goal: To develop and evaluate a preventive dental health program at two Danish chocolate factories in order to control oral Goal .  To measure the changes in oral health knowledge andoccupational diseases. purposes: I) To evaluate the effect of reported oral health behaviors following two sessions of oraldelivering preventive services to chocolate workers with respect^ To evaluate and measure for progress through an and healthhealth education.^To provide a framework for schoolto dental health status, dental knowledge, attitudes, and behavior; ^ surveillance system tracking both process and health outcomes.nurses to address the oral health needs of the pediatric population,and 2) to describe the implementation of program and perceptions of preventive care activities among workers and^ especially those of the Latino immigrant community. employers at the chocolate factories. Goal: To explore the feasibility of planning and implementing a community participatory oral health promotion program using the Precede-Proceed Model with community organizational theory methods. Purpose: early prevention of dental caries through education of pregnant women and parents of preschoolers in that community. 6,P^20% recruitment loss; examiner bias; hidden pressure may be C°JJ^placed on workers to report positvely on questionnaires orinterviews Small sample size; self reports can result in inaccurate data: 6 women did not take post-test thus making sample size even Unknown at this time since no actual evaluation done.smaller; questionnaire may be too difficult and detailed for literacy level and patience. Steering committee was comprised mainly of researchers, community' professionals and lay people. Representatives of the target group were not included. atV O Clinical recordings and pretested Liken questionnaires (9 questions). Statistical analysis included McNemar test, paired t- test. Quantitative data on workers' perceptions collected. Each interview was based on an interview guide and lasted about 60 minutes. A tape recorder was used. Pre and post test, developed in collaboration with a dentist, had 13 Basic screening survey tool with modifications. Data collection into Needs assessment used focus groups and health records. Impact knowledge questions and I 1 oral health behavior questions^logbook; entered into SPSS database weekly. Process components evaluation used anonymous self administered open-ended comprised of yes/no, true/false, multiple choice answers. Simple either tracked via database or through health encounter forms. Oral questionnaires and simple multiple choice surveys. Process evaluation used record keeping, inventory list of activities andanalysis on correct answers.^ health tracking card and process forms. resources, discussions, and antedoctal accounts. • Framework source: Ovretveit, John. Action Evaluation of Health Programmes and Changes: A handbook for a user-focused approach. United Kingdom,' Radcliffe Medical Press Ltd., 2002. Appendix A. Summary of Eight Articles in the Literature Review* 5. Harrison, R.L. & Wong, T.^6. Cruz, G.D., Roldos, I., Diva, I.P., & Salazar, C.R.^7. Hyde, S., Weintraub, JA., & Satariano, W.A.^8. Petersen, P.E. & Nortov, B. An oral health promotion program for an urban minority^ An Evaluation of the San Francisco Department of Human Services Evaluation of a Dental Public Health Program for Old-age Community-based culturally appropriate oral health promotion population of preschool children. Community Dentistry and Oral Communitv^" '^ Welfare Dental Program. Journal of Public Health Dentistry, Spring Pensioners in Denmark. Public Health Dentistry, 1994:54(2):73- program. New York State Dental Journal. Dec 2005:71(7)34-38. Epidemiology. 2003:31:392-399.^ 2005:65(2):104-109.^ 79 Personal Assisted Employment Services (PAES) Dental Program in San Francisco, California.^ Ballerup, a suburb of Copenhagen, Denmark .01.6^1994-2001 ^ 2001-2002 ^ 1999-2001 ^ 1987-1990 ot" (Assumed) Vancouver Coastal Health Authority and associated Child Health Clinics, Vancouver Health Department San Francisco Department of Human Services (SEDHS): National (Assumed) Maternal Infancy Care (MIC) center organization, New York State Department of Health , Bureau of Dental Health^ Institute of Dental and Craniofacial Research, National Institues of Municipal Dental Service in Denmark, Danish National Services.^ Health, Department of Health and Human Services; City of San^Association of Public Health Dentists Francisco kre 6N^Frequency of use with bottles, defs Effectiveness of instructional program Dental visit habits and status, attitudes, knowledge and self- Summative evaluation of program description, implementation, care, oral health outcomes, dental services, self assessment oflevels of participation and satisfaction oral health and life-quality. Vancouver, British Columbia New York City (Brooklyn, Manhattan, the Bronx and Queens communities) Basic needs assessment done as rationale. Four phases: 1) information gathering - baseline data collected: interviews with mothers by lay worker re: infant feeding, comforting dental beliefs and practices; clinical assessments by hygienist 2) project planning - via working committee on community project input and on infant dental video, 3) implementation - one to one counselling by lay worker to mothers at immunization visits, follow-up telephone calls; community wide initiatives include presentations, magazine, local festivals, fair, displays, brochure; 4) evaluation - process (formative feedback, record-keeping); impact/outcome - comparison with similar- aged children with mothers at 4 followup clinics Two phase approach: I) Needs assessment of target population and existing resources via survey to all pregnant women visiting Maternal services at specified time. Assessment of printed public educational materials re: oral health & pregnancy via Internet, mail inquiries, phone 2) Implementation of workshops (patients/professionals) via lay health bilingual worker; and distribution of materials. Pre/post tests re effectiveness of instruction program. Donabedian's evaluation framework of structure, process, and outcome measures. Study participants followed thru baseline examination, rehabilitative dental treatment. follow-up exam and completion of patient satisfaction survey. 18 month time period. Participation incentive given. Efforts made to reduce appointment failure rate (postcards, telephone reminder), maximize access to program (maps, directions, location, transportation). Outcome evaluation: All 67 year old pensioners offered free dental care in public system or choice of partly reimbursed private dentists. Pensioners participating interviewed about their self- assessments of dental health, dental knowledge, attitudes, and behavior. Interviews occurred at baseline and 3 year follow- up (before/after). Interviews home based, 35 minutes. Clinical data collected only for pensioners in public program. Private care participants represented reference group. (COMPARATIVE INTERVENTION) Preschoolers and infants of Vietnamese mothers attending the^All pregnant women visiting the Maternal Infancy Care Child Health Clinics for immunizations. 41 children under 5 years centers/Women's Health Services of New York (8 centers) during a of age served as baseline.^ specified time period. 377 of 379 referred PAES recipients participated. Primarily male, African American. high school graduates, 21-63 years old. All worked with welfare social worker for min 3 months and either self- identified or been identified as needing extraordinary dental services. 84% (265 of 377) study participants completed some or all of dental treatment. 65% (173/265) returned patient satisfaction surveys. (CONVENIENCE SAMPLE) 304 67-year old citizens offered free preventive/curative dental care choice 75% participated in public program; I I% private: 14% no dental care 216 participants interviewed at baseline; 235 at follow-up. Clinical data (a; public program, 194 at baseline; 187 at follow-up. Most of community dominated by unskilled and skilled workers; suburban. (CONVENIENCE SAMPLE) * Framework source: Ovretveit. John. Action Evaluation of Health Programmes and Changes'. A handbook for a user-focused approach. United Kingdom, Radcliffe Medical Press Ltd., 2002. defs, true/false questions, observations, daily log-book, meeting minutes, 6 month progress reports, funding agency site visits, followup telephone calls, and questionnaires. Closed ended English/Spanish survey (33 questions), intemet searches, mail inquiries, telephone solicitations, and pilot-tested questionnaires. Appendix A. Summary of Eight Articles in the Literature Review* 5. Harrison, R.L. & Wong, T.^6. Cruz, G.D., Roldos, I., Diva, LP., & Salazar, C.R.^7. Hyde, S., Weintraub, 3.A., & Satariano, W.A.^8. Petersen, P.E. & Nortov, B. os' Baseline survey showed low awarenss of importance of maternal Participating children had fewer decayed surfaces than at baseline. oral health and relationship to infant general and oral health. Significant improvement in infant feeding and comforting^Materials inadequate thus created new education material in 4 practices reported than at baseline. Dental counselling well^languages...over 500 workshops were done, educational packages received in well-baby clinics.^ delivered to 10,000 women, 20,000 brochures disseminated in 4 languages to health care and maternal centers in New York State. 86% of all respondents had regular dental visits at least I X/year Overall high levels of satisfaction with Dental Program experience; versus 46% at baseline. Improvements in self-care, knowledge generally no significant difference in baseline demographic and^and attitudes in oral health. Clinical data showed reduction in clinical profile between those who completed treatment or not; 82% unmet dental needs; more preventive care services given to of budget spent on rehabilitative dental treatment.^attenders of public program. 92% satisfied with public program care versus 72% with private service. To design, implement and evaluate an oral health promotion 0'6\6'^ program for inner-city Vietnamese preschool children. To assess the need for and develop an oral health promotion To describe the PAES's planning and implementation, levels fit}, for low-income immigrant pregnant women in New York of participation and patient satisfaction.y. Goal: To evaluate the oral health outcome of the program in terms of changes in oral health status, self-care in oral health, and life-quality among these 67 year olds during 3 years. Specific program goals: I) provide elderly with sociofunctional dentition: 2) reduce unmet dental needs & prevent oral disease; and 3) improve knowledge, attitudes, & self-care in oral health. Different person performing clinical assessments (hygienist at baseline; dentist at follow-ups); small sample group; possible^Unknown at this time since little evaluation activity Hawthorne effect in providing desirable correct response. Not possible to train or calibrate social workers in administering questionaires; no data regarding whether questionaires were self- administered or required help; possible reluctance in voicing program dissatisfaction since it was participants' only access to dental services. Private group pensioners not a control group; examiner bias; other intervening external factors; suburban community may not be representative of all Danish municipalities, possible recall bias. defs measure at the follow-up clinics (but not at baseline) Piloted questionnaire for face and content validity and modified accordingly. Translated back and forth in English and Spanish. Administered by trained interviewer. Old educational materials Dental exams performed by one trained and calibrated examinerevaluated for quality, adequacy, readability and breadth using a pre- using a mouth minor, explorer and CPI periodontal probe.tested instrument by 3 oral health professionals. New materials pilot- tested by 65 women evaluated & modified. Trained lay health worker. Calibrated dental health interviewers. Pretested questionnaire used. Calibrated dentists. Epidemiological recordings followed WHO guidelines. # services/patient recorded by Municipal Dental Service according to National Health Insurance principles. Patient satisfaction survey (Liken scale) for process & outcomes measures. Process measures: methods used to provide program's Longitudinal data collected to measure program outcome viaservices. Outcome measures: effects & changes resulting from structured Liken questionnaire (72 questions), interviews. clinicalprogram. Structure measures equipment, finances, personnel & data, 6 of services/patient. Statistical Analysis System. Gunmanprogram logistics. All data entry & analysis used the JMP Version 4 statistical analysis software from SAS Institute. Chi-square test, scale indices, paired t-test: McNemar test logistic regression, & paired t-test # Framework source: Ovretveit, John. Action Evaluation of Health Programmes and Changes: A handbook for a user-focused approach. United Kingdom, Radcliffe Medical Press Ltd., 2002. Appendix B. Certificates of Approval 'RTIF CATE OF APPROVAL- 5kigi5KAT:TrqV#Tsti?;47i5k7"--^.RTV N At, RISK 133 SW 3113 r:.).4.:413A,a (..)f -Dan ••• • u341/•1 MrilaW1 Wfq!s•" "d^I4,WW1,,:i:ii qi^ssawaspwwoa „^. •• stawnowuwA liztv^. ^ „,Csislwiaa^„.Ciprwt‘i '33ZIWpS ttlft'ati •UW..nap.- W14001 eur4a Ed - Palltzmaa '.:,:o3reava 1OZ saww a'ant^P4Pc04,1-41 TIVA.OadaV"^'11,VDI^"...i3:1) IVAtaNall „Sprws SS:punustro.:‘., ,^wi.n..osu q'w.,kw,sw colEW MS/MD .,awno-www,.‘ tEI a4n44.4sur tp,m4saN jawnopus.A. :^• •-, •••:, „taA.,r,,,,“1^• Appendix C. Process Evaluation Indicators, Methods, and Relevance Program Activity Value Indicator Measurement Method Relevance Referral process Level of clarity in screeningcriteria Semi-structured interviews to HBP staff in 2005; a simple 6 point interval poster questionnaire in Determines barriers to screening process. 2007 °A) correct oral health knowledge Simple 8 point Likert scale questionnaire to HBP Identifies level of dental knowledge of referree and if answers staff in 2005 barriers to screening process. # of received HBP referral Manual count of received referral forms 2004- Tracks continuity of referral process and determines forms; # of incomplete forms 2006; semi-structured interviews with dentalreceptionists barriers to completing forms. # of referrals from each HBP staff; # of referrals who attend/not attend Manual chart review of received referral forms 2004-2005 Determines possible pattern and barriers to referral process. Level of useful information in referral forms Personal field observations Determines relevant information to hygienist. Appointments # of 1, 2-visits; # of referralsnever attended Manual review of referral forms received and dental charts 2004-2006 Identifies type of visit attended and level of dental program participation. Encourages examination of factors promoting and hindering visits. # of missed, cancelled or Manual review of charts and appointment Determines number of missed appointments, possible rescheduled appointments on day monitoring records in 1986 - 2006; semi-structured reasons, patterns, and investigate strategies to reduce of appointment interviews with dental receptionists incidences. Average time period between screening and 1st visit; 1st and Manual review of client charts 2004-2005 Determine accessibility of dental clinic appointmentsto clients.2nd visit Average gestational week at 1st and 2nd visit Manual review of client charts 2004 - 2005 Identifies pregnancy stage the women is seen at. Identifies if time of referral appropriate or not. Program Activity Value Indicator Measurement Method Relevance Ease of booking appointments Level of clarity of appointment records # of clients seen annually Semi-structured interviews with dental receptionists and HBP staff Personal field observations Manual chart review of appointment monitoring records 1986 - 2006 Determine accessibility of dental clinic appointments to clients and barriers. Determines if recording format is "user-friendly". Determine number of clients served in the HBP dental program and if any patterns present. Medical history Level of clarity % of completed forms Semi-structured interviews with dental receptionists Manual review of accessible forms in years 2004 - Identifies the readability level of history forms, potential language barriers, and workload on receptionists. Determines completeness of information, barriers and relevancy of information. All client information forms Level of useful information to hygienist Personal field observ2a0ti0o6ns of medical history, dental chart, client dental information, and HBP referral forms Determines relevant information to hygienist. Dentist services Type of dental servicesperformed and recommended # of exams at 1st and 2nd visit % of women following urgently recommended treatment referrals by 2nd visit Manual dental chart review 2004 - 2005 Manual dental chart review 2004 - 2005 Manual review of dental charts 2004-2005; field observations Determine type of services dentist provide and potential services required. Determine if standard need for dental exams at both visits. Identifies if treatment recommendations are followed or not and if need to address possible barriers. Program Activity Value Indicator Measurement Method Relevance Amount of information charted on dental charts Field observations and cursory manual chart review 2004-2005 Determines if charted information is sufficient in amount and detail. Clinical hygiene services # scaling units, prophy, fluoride treatment being performed Estimated % of appointments kept within time schedule Manual dental chart review 2004-2005 Semi-structure interview with dental receptionists; field observations Determines type and amount of clinical hygiene services performed. Determines appropriateness of appointment length. Identifies possible barriers. Oral hygiene instruction # of appointments with oral hygiene instruction conducted Manual review of dental charts 2004-2005 Determines if oral hygiene instruction is present. Oral health education # and type of oral health topics discussed during appointment Estimated % of time allocated to educational component Type of dental pamphlets distributed Semi-structure interviews with main dental stakeholders; field observations Semi-structure interviews with main dental stakeholders; field observations Field observations Determines relevancy and amount of topics to cover as well as what is "required" to be covered. Determines time frame required for counselling. Determines if pamphlets are being used or not in program and identify potential barriers. Dental provider list Level of accuracy in contact information Accessibility of services to HBP clientele Phone interviews with each provider listed on handout in 2005; comparative review with list 2003 Phone interviews with each provider listed on handout in 2005 and 2006; field observations To ensure correct access information to these providers for this clientele. Determine enabling and reinforcing factors or barriers to dental services. Appendix D. List of Interview Questions Evaluability Assessment To the main stakeholders: 1. When and how did the HBP Dental Program start? 2. Who are the stakeholders associated with the program? 3. Who is responsible for the HBP Dental Program? 4. Explain how the funding is organized. 5. What is your objective and goal for this program? 6. What do you want to know about the HBP Dental Program? 7. What do you want the evaluation to focus on? 8. Can you rank the following six criteria according to importance for you in this evaluation? (Description, correspondence to objectives, outcomes, efficiency, acceptability, sustainability) 9. What do you expect the HBP dental program to be doing for the women referred? 10. What decisions need to be made about the HBP Dental Program? 11. Who needs and/or will use the information? 12. What kind of relationship does HBP program have with the VCDP dental program? To most stakeholders involved with HBP Dental Program: 1.^What is your role and responsibilities in the HBP Dental Program? Descriptive Evaluation To the HBP coordinator: 1. Can you describe the women who are accepted into the HBP prenatal program? How are they accepted? 2. How do you determine low income or high risk? 3. What % of women seeking HBP prenatal services is accepted based on the HBP screening criteria? If they are not accepted, where are they referred to and what are the screening criteria? 4. How is the HBP program funded? 5. Who are the HBP counselors? 6. What is the % of HBP women referred to the dental clinic? Are there certain HBP activities that tend to promote more referrals? To previous HBP Dental Program hygienists: 1.^Can you tell me how this program started, the rationale, the clientele, and previous clinicians with the program and their work duration? 138 Process evaluation To the HBP staff (2005, 2007 meetings) 1. Are there any challenges in dealing with the HBP dental program? 2. What is important to you about this dental program when you refer the women? 3. What do you expect this dental program to give to your HBP women? 4. Do you think this dental program is useful to HBP women? 5. What do you think is important to your clients about this program? 6. What do you think are some of the barriers preventing some women to come to this clinic? 7. Do any of you have any suggestions or concerns about the program or research? 8. Are you satisfied with the communication between the 2 programs? 9. What do you think of the questionnaire? Would it be difficult for the women to complete? Any suggestions? To VCDP dental receptionists at NCHO dental clinic: 1. Are there any challenges you encounter about the booking appointment process with the HBP staff during the referrals? 2. Are there any challenges in scheduling and confirming appointments and/or completion of forms with the clientele? 3. What do you think about the length and timing (AM, PM) of appointments given for the HBP clients? 4. Any concerns about the communication between the 2 programs I should address with HBP staff? 5. Do any of you have any suggestions or concerns about the program or research? If so, you can write it down on this form anonymously in the HBP dental program folder. To other dental staff at NCHO dental clinic: 1. Do any of you have any suggestions or concerns about the program or research? If so, you can write it down on this form anonymously in the HBP dental program folder. To the main stakeholders: 1. What educational topics or points should be covered in a HBP dental appointment? 2. How much appointment time should be spent covering these topics? 3.^What are some possible reasons to account for discrepancies in reporting of client dental program attendance numbers between the HBP annual reports and VCDP annual reports? To the dental providers on list for reduced fees: 1. Can we confirm and update our contact and description information regarding your clinic? 139 Appendix E. Questionnaire #1 Health question 1: We would like to know what you think. Please circle your response. 1. It is okay for a pregnant woman to have bleeding gums. Yes^No^Not sure 2. Sweet foods and germs may cause "holes" or cavities in my teeth. Yes^No^Not sure 3. Brushing at least once a day is good for my gums. Yes^No^Not sure 4. Problems with my gums may affect my baby before he/she is born. Yes^No^Not sure 5. I should clean my baby's mouth at least once a day. Yes^No^Not sure 6. It is okay to let my baby sleep with a bottle of milk or juice or on my breast. Yes^No^Not sure 7. If my baby's teeth are not healthy, his/her adult teeth may not be healthy either. Yes^No^Not sure 8. This dental program will be good for pregnant women. Yes^No^Not sure ID Number: Date: Version: December 20, 2005 140 Appendix F. Poster Questionnaire for HBP Staff (2007) To help us improve the dental program, please mark your answer on the line with the stickers. 1. Dental Screening Criteria 1^ 1 Difficult for me to screen^ Easy for me to screen 2. Dental Screening Form 1^ Too much information to fill out^ Easy to complete 3. Booking appointments for clients 1^ 1 Difficult to get appointments^ Easy to book 4. Communication between HBP staff and dental 1^ 1 Needs a lot of improvement^ Satisfied with way it is 5. Feedback from your clients who attended dental program 1^ No feedback^Negative feedback^Very happy with program 6. Encouraging your clients to go to dental program 1^ 1 Clients generally resistant^ Clients very receptive to go 141 Appendix G. Consent Form ,::1^5i' 3^C „^3 Cons.. r4 FOIM Effec...L^ • rincipal investigator/fa.•.h,:' 55Ithise • Co -investigator Dian L^(Sroduc.2. S: .-•L^ ••^i • •^; • h N^ :5:- • ••••^..^•^•^. •^ ,., TILL.^•:•  )1" o„Ir sftvg) tO lear • artic ipolte^,••• • Sivly Prc:•;:edures. ;^r • ."^•)) .:or )1,•^• :^•^3.3 •^. At the fit • .^v' tOCtte CI 1^no ^I^. COlt(IIICt^"^by •I•e1c„.pho:•: • •• •,•••.•^• • Li:: ^„•,••• ^• e-gt-^.^• .•, ,••::•:10i v----1;:-• • ShOW tO^ S••• Ver:bn: June 18. 2005 142 31.110:::: '9 4 !•1L1nr 'S(1•1-.11441.4 HMO „400.A 104. o^•^no.A^,-.1111.1411L4 ! S 10 0,k (•^•4!! ^•,41.4^44104 SS!!^:!! „I! .1.•••.0„!..:.<1 G.1.0:4[1L.1b1S 0 S 00 112) GLA 0 0 0 kJ (.4A 41(4) 41 Li 1-1 041 0 H Alp( •^011) L„.1 410 N^03 c:! !AS ...191444.44-4,1 OLIIA100:G4 W1140 LOO4A,^0041: 04. >01'0^04.14LAA GLU4 ALIO .40 00.4S JO G1.0(k....:14W21 (14. GSO!^110),^1„104.1100A „.44!....(1•• ^:1 S1 410114.S (1104 141 L.1040!SOLS.4.-•1 uasuoo : AjaS •LI,:.)--ics.G•el^G.:).:44,0^aq4 u!^u00udok^Li-J0-GsG1 not: '.Apnls^IL„loop41.E:^o so ..s.tq.6!„iInoqo ^AOC WAS44,1 104i 41 AVSOZ-!!!'„n•--;^, 3,2r1^p.:nL...u.osoa^pr,IL.oc) •sql^0(11 0 LI CO1. 0 LI ^LL G40 444  :4 • •!• • • • „.. D S .14 skint) AU o aAoL.1 040 LA G P4Sa.q.; ploclo uottoutiolu!^.13o4u-oo •Ss„ .•^POLOPi S'• • !( 12' •!• S'••(:i AO 54, '.!••• i\LID J04 4.S00 OU^GIGW„ 114105LiadlisiO3.1LIOtnieunw4 ^puo Gq-o.t-sJG:.„.1.1.oLu lo4 suiolboici 4.i.parq 100 GA0:40W!^1:4•A ;4:•)4.0d4-344.414440 .4410)4 'AA^')H Q Jovybei :::4t..4. o „.tod ou^Go!.AJes 5,,„fluoGio p(„10 Lion.Ou!LLIOXG 40.1,0(4,0 0 G4A1G0(41..! 111AA u..11.:'.1"1-0 •:141 s4;•••.!S21-4..;sOG4. 'A^411101i010 •SO •Gq0.140444400 SO 1110A 0(41L4414 Ok L,C• •^il•-/wL100.1111AA 44410WW...!. _4 ^ :A;:^A' LAAOL4 OU SOL1 Ouppep^_Jo piomssoo .Aq Gq4 LAO^A-L....O."' !!^t5uN^o u! .:•J!^o^g.),944.u.ap!^s.i.u.awn3op ..Aue u! aulou Lettoolluepguoo '104V, ALAL) LL: 441404A^Asou. eL41^sow:43a4 Hvyk nokk !41-4csait :DUO^G50000 4044 111AA^•.C.:04S S404 40 440(1 aq 4.ou^Gopap noA !GU 0 0 CI^0 4.14.:41 (-^!1-1>1,G0 (141 111,N1 Sk.1'.41.SE11-1411 G(44 S.4!S14,A, O41.144.! (11.44 404 0114110^4.c.)uuryq floA et,4 ,;"“Dc::10 >:;HLu^10;..JA.,, :Duo^inoclo suottsi9r,b ,„,„,,,E.).1^J.,),,H„v,..s.uo £17 I G.1114..OU O!S SSG U fs. Delcins eq4. o GUI 0 N OG1 !„1 d •• Appendix H. NCHO Dental Clinic Medical History Form V.ANA71:0(1\rf:RIRIIIM()ND^'17.! BOARD PRO( CONEr()ENTI M H ThAi. IHISTORY • Nu, 14. 145 TAL RECORD 12 55 54 53 64 65 7,1 48 47 46 45 44 43 42 35 3(> 38 VV,./ Crt 4ded or rn X lb be exlract 1.,»w,. itige (Page omitted by student)  147 Appendix J. HBP Dental Referral Form DATF :OR :DM 148 6171 •••••'•'(^•^( 1, t .^,.• •^•^.• C^7:10 iVLU ,^St1.1.."•••• COII "^•^Ln":".) js^TIVI.NOCIOTaaa GIIG1l.„12V I.Kativ.wao,TKI C 2 3^4 Yes Boys ages^Girls ages^ No _Yes Private/NCHO/Both^_Yes (bleeding/caries/pain/other)^Undiagnosed Yes often^by adult/self/both Appendix K. Interview Recording Form (Prenatal Visits) Name:^  Due Date:^ HBP Worker:^ HBP dental program freq: ^ First visit date:  Gestational week:^ Pregnancy up to present^No concerns^Nausea Vomiting Take maternal vitamins^No^Yes^Other Smoker^No Previous HBP#:^ Age: ^ Other Current, how often ^ ORAL HEALTH Problems reported during this pregnancy which affects eating None^_Pain ^Other _Sensitivity to cold/hot/sweet/pressure....general/local^ Bleeding upon brushing/flossing Last dental exam ^years (Canada/International^) _Emergency _Fillings _Extractions _Scaling _Polish/Fluoride^Other Last scaling ŷears (Canada/International^) Present oral hygiene Brushes ^ /day (man/power) with toothpaste Flosses^No^_When needed^_Yes ... X/day/week Toothpicks^No Mouthwashes/Listerine —Yes No Yes^Other  OTHER Number of pregnancies^1 Other children^No Some/all see dentist Dental problems No Brushing^No English comprehension None^Translator present (1 St)^(2nd) ^Able to communicate concerns and ask questions Speaks English fluently Preferred spoken language(s) ^written language(s)^ Country of origin^ Dominant language ^Years in Canada^ INITIAL EXAMINATION: Tooth surfaces brushed buccal/occl 3 sides, no molars all surfaces Plaque^None Light Moderate Heavy Calculus None Light Moderate Heavy Gingivitis None Mild Moderate Severe Periodontitis None Early Moderate Severe Caries No Yes (3 rd) Mobility in any teeth^_No _Yes Recommended treatment _Restorative (urgent/later/B) Extraction (caries, broken teeth) (urgent/later/B) ^Extraction (3 rd molar impaction) (urgent/later/B) Undiagnosed Other _Reg. exam/hygiene/x-rays ^Periodontist (urgent/later/B) SECOND VISIT date: ^Gestational week:^ Urgent treatment done: _Yes _No _Some^N/A Other children now seeing dentist ŷes ^no N/A Brushing freq ^Surfaces brushed _(Buc/occl) _3 sides, no molars _All surfaces Reported results^bleeding (same/more/less/none) ^pain (same/more/less/none) ^sensitivity (same/more/less/none)^OHI method (likes/dislikes) other File status: 150 Appendix L. Interview Recording Form (Postnatal Visit) Date of Today's 3"I Visit/Phone Call:^ First visit date:^ Second visit date: Name: Language:^ English comprehension: None^Conyers.^Fluent Interpreter present? No^Yes Country of origin: ^Years in Canada:^ HBP #:^Previous HBP referring staff member: ORAL HEALTH Dental problems reported: None^Pain (Less/more than before/new) ^ Sensitivity to cold/hot/pressure (less/more/same/new) Other Bleeding reported: None^Sometimes^All the time Brushing frequency: <2X/day^2X/day >2X/day Brushing method:^Buccals only Buccal/occlusals^3 sides, not molars All surfaces Other aids:^None flossing^toothpicks^Listerine/mouthwash Sensodyne^Other Prey. urgent recom. treatment (exo/restorative) followed?^None Some All^N/A Prey. other recom. treatment (hygiene/restorative) followed?^None Some All ^ HBPdent3rd only^N/A EXAMINATION Plaque Calculus Gingivitis Periodontitis Caries None^Light None Light None^Mild None Early No^Yes Moderate Moderate Moderate Moderate Heavy Heavy Severe Severe Mobility^No Yes Today's recommended treatment Reg. exam/hygiene/radiographs TREATMENT RENDERED Exam ^scaling units (hand/P5) prophy ^fluoride^OHI^ Handouts:^recommendations^referral list^child oral care ECC brush/floss Language DEPENDENT(S) Infant birth date:^ Infant gender:^Male Female Birth gestational week: ^Infant birth weight: ^ Infant age today: Current oral hygiene:^wiping gums^<2X/day ^2X/day^>2X/day Brushing teeth <2X/day^2X/day^>2X/day None Infant dental concerns b Other y parent?  If other children, are they seeing a dentist? No^Yes (Private/NCHO/Both) N/A Undiagnosed Restorative^Extraction^ Other 151 Appendix M. Questionnaire #2 Health question 2: We would like to know what you think. Please circle your response. 1. It is okay for a pregnant woman to have bleeding gums. Yes^No^Not sure 2. Sweet foods and germs may cause "holes" or cavities in my teeth. Yes^No^Not sure 3. Brushing at least once a day is good for my gums. Yes^No^Not sure 4. Problems with my gums may affect my baby before he/she is born. Yes^No^Not sure 5. One week after my dental visit, my mouth seemed... better^worse^no different 6. The dental information that the dental hygienist told me at the clinic... helped me a lot^helped me a little^did not help me 7. a. The written dental information (pamphlets) that I got... helped me a lot^helped me a little^did not help me b. I would prefer the dental information (pamphlets) in the following language: 8. Since coming to this dental clinic, taking care of my teeth is... more important to me^less important to me^just as important 9. I am glad that Healthiest Babies Possible told me to come to this dental program. Yes^No^Not sure ID Number: Date: Version: December 20, 2005 152 Appendix N. Questionnaire #3 Health question 3: We would like to know what you think. Please circle your response. 1. It is okay for me to have bleeding gums. Yes^No^Not sure 2. It is okay to let my baby sleep with a bottle of milk or juice or on my breast. Yes^No^Not sure 3. I should brush my baby's teeth or wipe his/her gums at least once a day. Yes^No^Not sure 4. If my baby's teeth are not healthy, his/her adult teeth may not be healthy either. Yes^No^Not sure 5. Now that my baby is born, I would like to come back to clinic for future cleaning visits. Yes^No^Not sure If no, why? 6. Is there anything you would suggest to help us improve this dental program? ID Number: Date of visit/telephone Version: December 20, 2005 153 Appendix 0. Study Appointment Monitoring of Outcome Participants Study # 1st visit ID # Client Name HBP # Study 2nd visit Due date Birth date Gest. Week Sex Birth weight (gms) 1st post natal contact date Card sent Handout language sent 2nd Preferred language post natal contact date 3rd visit Seen Appendix P. Description of Clinical Indices Gingival status (GI)* 0 = absence of signs of inflammation 1 = mild to moderate inflammatory gingival changes, not extending around the tooth 2 = mild to moderately severe gingivitis extending around the tooth 3 = severe gingivitis characterized by marked redness, swelling, tendency to bleed and ulceration Calculus index (CI)* 0 = absence of calculus 1 = supragingival calculus extending only slightly below the free gingival margin (not more than 1 mm) 2 = moderate amount of supra and subgingival calculus or subgingival calculus alone 3 = an abundance of supra and subgingival calculus Plaque index (PI)* 0 = no plaque present 1 = plaque present on some but not on all interproximal buccal and lingual surfaces of the tooth 2 = plaque present on all interproximal, buccal and lingual surfaces, but covering more than one half of these surfaces 3 = plaque extending over all interproximal, buccal and lingual surfaces, and covering more than one half of these surfaces Bleeding on probing (BOP) index "present" = bleeding noted within 10 seconds of probing "absent" = no bleeding noted within 10 seconds of probing Pocket depth (POC) index Deepest recorded probe measurement (mms) of the six surfaces of the tooth *Source: Ramfjord, S.P. The Periodontal Disease Index. Journal of Periodontology, 1967. 38 (6, Suppl), 30-38. 155 Appendix 0. Clinical Indices Recording Form ID Number:^ Date: Visit Number:^1 Tooth 16 21 25 Subtotal Total Plaque 0-3 Calculus 0-3 Gingival 0-3 BOP Pockets Tooth 45 41 36 Plaque 0-3 Calculus 0-3 Gingival 0-3 BOP Pockets ID Number:^ Date: Visit Number:^2 Tooth 16 21 25 Subtotal Total Plaque 0-3 Calculus 0-3 Gingival 0-3 BOP Pockets Tooth 45 41 36 Plaque 0-3 Calculus 0-3 Gingival 0-3 BOP Pockets ID Number:^ Date: Visit Number:^3^clinic/ hone Tooth 16 21 25 Subtotal Total Plaque 0-3 Calculus 0-3 Gingival 0-3 BOP Pockets Tooth 45 41 36 Plaque 0-3 Calculus 0-3 Gingival 0-3 BOP Pockets 156 Appendix R. List of stakeholders' roles and responsibilities Dental clinic supervisor of Vancouver Community Dental Program (VCDP) at NCHO: Dentist in charge of all the VCDP dental clinic-related programs, staff, equipment, resource allocations, and budget. Focus is on providing preventive and restorative services for low-income Vancouver children and/or from Vancouver community screenings. Works cooperatively with VCDP community coordinator and reports to NCHO site manager. HBP role is supervisor to HBP clinical hygienist and has decision-making authority over distribution of program budget, resources, and clinic availability. Has been involved with the HBP Dental Program for 6 years. Community coordinator of VCDP at NCHO: Community dental hygienist who oversees all the community dental programs and staff in Vancouver related to early childhood caries intervention (e.g. school screenings); works cooperatively with Dental Clinic supervisor and reports to NCHO site manager. Also acts as the liaison to community dental providers on list. HBP role is to act as liaison for Evergreen-HBP program coordinator; provide NCHO dental screening criteria, referral forms, and annual appointment monitoring report of HBP dental clientele to HBP prenatal coordinator. HBP Dental Program hygienist of VCDP at NCHO: Clinician (part-time 2 days/month) in charge of implementing HBP Dental Program services (except for dental exam). Duties include: review medical and dental histories with client; collect information about oral health behavior and concerns; provide peri-natal oral health counseling, oral hygiene/home care instructions, and appropriate information handouts; assess periodontal status and treats accordingly; records data in client forms and appointment record log; follow appropriate clinical guidelines; manage inventory of program supplies; order supplies as needed with chair side assistant; and report to VCDP dental clinic supervisor for clinic-related issues and VCDP coordinator for HBP-related issues. 157 VCDP sessional dentists (3): Dentists' (part-time, alternate days) main responsibility is to provide preventive and restorative treatment for children in clinic; HBP role is perform dental exam on HBP referred women on first visit (and as needed), make treatment recommendations, and assess urgency of treatment before or after delivery. VCDP dental receptionists (2): In charge of all VCDP dental clinic front-desk duties; role in HBP Dental Program is to book client appointments with the HBP prenatal program staff for their clients when the staff phones the clinic. After HBP program staff sends dental referral form to NCHO dental clinic via interoffice mail, receptionist attaches it to newly made chart. Receptionist confirms appointment; registers contact information, requests women to fill out NCHO medical form upon first visit and books subsequent 2 nd appointment. Any cancelled or missed appointments are rebooked. VCDP chair side assistants (2): Responsible for all of clinic-related operations and assist dentists on duty with treatment services. HBP role is to receive supply orders from clinical hygienist, obtain approval of order from dental clinic supervisor, and then order accordingly. North Community Health Office (NCHO) site manager: Person in charge of overseeing all programs, resources, and staff at #2 Community Health Center (CHC) - NCHO site. Provides input about the VCDP overall dental budget to the VCHSDA board of directors. Not directly involved in allocation of HBP Dental Program resources within the dental budget. Works collaboratively with ECHC site manager. Evergreen Community Health Center (ECHC) - HBP coordinator: Oversees the entire HBP program and staff in Vancouver and Richmond from the #3 CHC — Evergreen Community Health site; HBP role is to instruct her HBP staff regarding referrals to the NCHO dental clinic via screening criteria and referral forms; provides birth information from HBP database; liaison is with community coordinator of 158 VCDP in regards to any program-related issues. Not involved in managing HBP Dental Program activities. ECHC - HBP referred women: HBP women, living in Vancouver, who either self-identify a dental concern or have not seen a dentist for over 2 years are referred by their HBP program support worker to the HBP Dental Program. They are the intended beneficiaries of the HBP Dental Program. They are informed of their dental appointment by their HBP staff worker and also by the dental receptionists shortly before the appointment. ECHC - HBP prenatal program staff (8-11): Each staff member is responsible for a group of women assigned from HBP program coordinator. Their HBP Dental Program role is to selectively refer HBP women to the dental program based on designated NCHO-dental screening criteria, personally book the client appointment with the dental receptionist, informs the client of the appointment, and then send dental referral forms to the NCHO dental clinic via interoffice mail. ECHC Infant Child and Youth (ICY) manager: Person in charge of overseeing all the infant, child and youth programs, resources, and staff at ECHC; has no direct involvement with HBP Dental Program but needs to be informed on any evaluation associated with HBP program. Works collaboratively with NCHO site manager. Vancouver Community Health Service Delivery Area (VCHSDA) Board of Directors: Group who determines financial budget for each of the six Community Health Centers (CHC) in consultation with the respective site managers; has the decision making authority over the NCHO programs and resources, which includes the VCDP; reports to Vancouver Coastal Health (VCH). Vancouver Coastal Health (VCH): Oversees funding and direction in all the five Health Service Delivery Areas including VCHSDA; reports to the Ministry of Health. 159 Ministry of Health (British Columbia): Responsible for program and funding decisions in the Provincial Health Services Authority and five BC Health Authorities including VCH. Mid-Main, REACH, UBCNCC dental schools, VGH clinic, Portland, Sunrise: Vancouver dental non-profit clinics and schools offering services at a reduced cost to the general public. Has no responsibility or commitment to HBP Dental Program but benefits in patient referrals and advertising exposure. 160 Appendix S. Client Dental Information Form (2004) Date: Name:^  Due Date:^ Pregnancy up to present^No concerns^NauseaVomiting Others Taking maternal vitamins^No^Yes Smoker^No^If Yes, how often? Other children No Yes Boys ages^ Girls ages ORAL HEALTH Problems prior to pregnancy None Bleeding upon brushing^Cold/hot/sweet sensitivity...gen/localized Pain ^Duration?^Precursor? Problems during this pregnancy^None Bleeding upon brushing^Cold/hot/sweet sensitivity....gen/localized Pain^Duration?^Precursor? Other Present oral hygiene Brushes^/day with toothpaste^ Manual/Electric^Fluoridated/non-fluoridated TP Flosses^No^If yes, how often? Toothpicks No^If yes, when^ Mouthwashes No Yes If other children ^ Seeing regular dentist No^Yes Problems with their teeth^No^Yes Their current oral hygiene regime at home Brushing^No^ often^by^ Flossing^No^Yes Past dental experience LDV^ Scaling Fillings^ Polish/Fluoride  Extractions Language comprehension Nil — Minimal^Translator present ^ Seems to be able to converse and understand Good grasp Spoken language^Able to read^ Country of origin April 2004 161 Appendix T. Appointment Monitoring Record Log Sheet Healthiest Babies Possible Dental Statistics Month of: Date # of appointments available # of 1 St appointments broken # of 2nd appointments broken # of 1 st appointments cancelled # of 2nd appointments cancelled # completed on 1 St appointment # completed on 2nd appointment Total # clients to date (year) 162 ;ry lipeo^„.t. ; • • .• Prevporroo (Sept.^.lonel th Via ric , :,,, , ..',,,,:,-,: , ..lotrimunity Dental Services* for Adults •--- ^. 7^ACY -ipe- '^''.I^r‘f AkVE t„?., CON) . ^'-‘,/ i^PHONE^P.U.:STP1C i : -)NS^SERVICES !^„.,......^.^.; 1 , 7„:•4437373;1302^l 7,.!: .,ointrient required^ l Complete Dentiotry '^ r rii.c,"1 Clinc:i 6(A-254-1331^i^—^— l1 hp frommerlt reourer: COmplete Oeoristry : .I 1145 C •• , •• forP' • '^ l l^ : p,^C.'entro 1^.• :^• .^,^, : ff5f„)^' , .^„,,,:," r^..,:•, , •:.'...3. of th. , •' . .,..... ,.........1^I r',.)roplete Om:Pstry^.i^.-.". 'J^:^•^m •7, ,;: .. . : f^ : = I el, ' k :^,.^ i 3( ..)". 2, • ':, ',..•. •‘•^i tkieSt be . ' '.....k.•:'`.., ... k. '—^1 c.lonipiete 1)etthstry bac or: r,,l ,„.:1;•..., I 21 . 7; r^:^:. .k kkki^ 1 :^ 1^; : 1i Akf; .:''' vents ;^,^„ ..^,. • '^1^.-Th,...•.,,:;:2.p r:.p,^..,....,.,^,,;;• 1.;^,---,:perlo f,,,,,,,,,,• ;^ .^offn rcA c6 rekbe *11 CD CD.^.^.


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items