UBC Community, Partners, and Alumni Publications

What a tangled web we weave : improving performance reporting and accountability in BC : proceedings… University of British Columbia. Centre for Health Services and Policy Research Oct 31, 2004

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata


52387-Tangled_web.pdf [ 1.15MB ]
JSON: 52387-1.0048297.json
JSON-LD: 52387-1.0048297-ld.json
RDF/XML (Pretty): 52387-1.0048297-rdf.xml
RDF/JSON: 52387-1.0048297-rdf.json
Turtle: 52387-1.0048297-turtle.txt
N-Triples: 52387-1.0048297-rdf-ntriples.txt
Original Record: 52387-1.0048297-source.json
Full Text

Full Text

What a tangled web we weave:Improving performance reporting and accountability in BCHEALTH SERVICES AND POLICY RESEARCHCentre forCollege of  Heal th  Discipl inesTHE UNIVERSITY OF BRITISH COLUMBIAOctober 2004Proceedings of the 16th annual health policy conference of the Centre for Health Services and Policy ResearchNovember 7, 2003 - Vancouver, BC“What a tangled web we weave: improving performance reporting and accountability in BC. Proceedings of the 16th annual health policy conference of the Centre for Health Services and Policy Research, November 7, 2003, was produced by:Centre for Health Services and Policy ResearchThe University of British Columbia429-2194 Health Sciences MallVancouver, BC Canada V6T 1Z3Tel: (604) 822-1949Fax: (604) 822-5690Email: enquire@chspr.ubc.caThis conference was supported by a financial contribution from theBritish Columbia Ministry of Health Services.Prepared from conference transcripts byMerrie-Ellen Wilcox, Inkstone Writing and EditingYou can download this publication from our website at www.chspr.ubc.ca.This document is protected by copyright. It may be distributed for educational and non-commercial use, provided the Centre for Health Services and Policy Research is credited.HEALTH SERVICES AND POLICY RESEARCHCentre forCollege of  Heal th  Discipl inesTHE UNIVERSITY OF BRITISH COLUMBIAThe Centre for Health Services and Policy Research (CHSPR) is an independent research centre based at the University of British Columbia. CHSPR’s mission is to stimulate scientific enquiry into issues of health in population groups, and ways in which health services can best be organized, funded and delivered. Our researchers carry out a diverse program of applied health services and population health research under this agenda.CHSPR aims to contribute to the improvement of population health by ensuring our research is relevant to contemporary health policy concerns and by working closely with decision makers to actively translate research findings into policy options. Our researchers are active participants in many policy-making forums and provide advice and assistance to both governmental and non-governmental organizations in BC, Canada and abroad. CHSPR receives core funding from the BC Ministry of Health Services to support research with a direct role in informing policy decision-making and evaluating health reform, and to enable the ongoing development of the BC Linked Health Database. Our researchers are also funded by competitive external grants from provincial, national, and international funding agencies. Much of CHSPR’s research is made possible through the BC Linked Health Database, a valuable resource of data relating to the encounters of BC residents with various health care and other systems in the province. These data are used in an anonymized form for applied health services and population health research deemed to be in the public interest.CHSPR has developed strict policies and procedures to protect the confidentiality and security of these data holdings and fully complies with all legislative acts governing the protection and use of sensitive information. CHSPR has over 30 years of experience in handling data from the BC Ministry of Health and other professional bodies, and acts as the access point for researchers wishing to use these data for research in the public interest.For more information about CHSPR, please visit www.chspr.ubc.ca.IntroductionAccountability has become increasingly prominent in Canadian discussions about health care system reform. Performance reporting is seen as a promising approach to informing, guiding, and evaluating system change in British Columbia. Other jurisdictions, notably the United States and United Kingdom, can offer important insights into the opportunities and challenges in embarking on this agenda. CHSPR’s 16th annual health policy conference—What a tangled web we weave: Improving performance reporting and accountability in BC—aimed to identify the reasons behind our increased focus on accountability, as well as options for moving forward and highlighting “lessons learned” by others who have moved further down this path. Speakers at the conference included researchers, independent consultants, clinicians, and high-level public servants and administrators from BC, Alberta, Saskatchewan, and the UK. (Speaker biographies are provided on page 17.)Numerous definitions of accountability, as well as a range of approaches to accountability and performance reporting were presented. Accountability in health care was compared with accountability in the private sector and industry. We heard examples of accountability approaches and results from other jurisdictions. There was debate about public performance reporting—about what, how, and how much to report. And there were varying conclusions about how we're doing in our accountability efforts.CHSPR’s 16th annual health policy conferenceKEY MESSAGESThere are many different definitions and aspects of accountability.Although the key interaction in the health care system is between patient and clinician, there are many factors that affect that interaction. The health care system creates a complex web of accountability relationships, in which each agent is accountable to several prinicipals, who may themselves be accountable to others.Given the complexity of the accountability web, it is essential to be very clear about exactly who makes what decisions about what issues.Performance reporting has several possible purposes and four key audiences or users: patients, clinicians, managers, and government.Public performance reporting is an essential component of accountability, but it can be challenging. Performance information is complex and easily misinterpreted.There is still a great deal of variation in approaches to performance measurement and reporting. There is also debate about what to measure, how much to measure, and how to measure it—and what to do with the results, to ensure they are meaningful for the various users.Democratic accountability—with the requisite public engagement and debate—is essential to the health care system, but has suffered in the last decade and we need to work hard to renew it.1Defining accountabilityNumerous ways of defining accountability were presented throughout the conference. George Orwell once remarked that there are certain words that have a vague definition but a recognized general meaning that makes them great thought-savers. Accountability is one of those “soggy” words: while it is useful for bringing groups to the table, it makes reaching agreement difficult. “Various parties with different goals will choose accountability as their weapon of choice,” Alan Thomson said, “without pinpointing a clear definition.” Keynote speaker Peter Smith cited John Stewart’s definition of accountability as having two necessary components: a report on performance, and a reaction to the report that reinforces good performance or punishes bad performance. In accountability between a principal and an agent, the agent first renders an account, in the form of a performance report or an inspection. The principal is then able to act on the report in some way, or to hold the agent to account, by, for example, taking business elsewhere, voting the agent out of office, or offering incentives. Professor Smith also defined two types of accountability: executive and democratic. Executive accountability requires holding organizations in a devolved health care system accountable to a higher level, such as the federal or other levels of government, while democratic accountability requires holding health care providers accountable to the people they serve.BC’s Auditor General, Wayne Strelioff, described the essence of accountability as “the obligation to report on and answer questions about responsibilities.” Performance information helps assess whether responsibilities are fulfilled well, and helps legislators make decisions about resource allocation and responsibilities. Wendy Armstrong defined public accountability as including access to reliable, relevant, and understandable information; context, so the information makes sense and its reliability can be verified; and opportunities to question and evaluate mechanisms that enable citizens and communities to influence final decisions, pursue remedies and redress, and make changes.  Peter Smith also reinforced the distinction between raw information, or data, which does not in itself provide accountability, and information paired with a mechanism that alerts people to difficulties or opportunities and prompts them to take action, which is a kind of accountability.Improving performance reporting and accountability2CHSPR’s 16th annual health policy conferenceMany accountability and performance measurement concepts used in the management of health care systems have been borrowed from private industry. But Steven Lewis described some important differences between health care and other industries. The products of other industries are usually easier to define than the end product of health care. People managing a car factory know that their product is a car. The output of the factory is measured in the number of cars, or the rate of production. But in the complex factory of health care, people are still trying to understand what the product is, and how to measure it. The issue is clearer in some health care environments than in others. For example, someone who works in cardiac care has a comprehensible domain. But when health care is meant to produce population health, citizenship, and democratic accountability, figuring out the end product is far more difficult. Is it health, quality of life, or distribution of resources? This complexity makes it difficult to define “performance” and set meaningful targets.In addition, private industries have for a long time been able to base spending decisions on measurements of the value gained per unit of money spent. While the literature is filled with measures of health care system performance, few targetsbased on optimal performance—for example the number of diagnostic scans required per thousand people to keep the population healthy—have ever been identified.Finally, accountability in health care differs from other industries in that there are many autonomous actors in health care. Many health care systems are structured so that no single actor is accountable for an organization’s performance. In BC’s system, where authority is devolved, Mr Lewis said, regional health authorities “will only declare that they are accountable to government under duress.” Additionally, the professionalism of clinicians, and their tradition of autonomy, will inherently conflict with the increasing expectation of organizational coherence and accountability.BORROWING FROM PRIVATE INDUSTRY3Improving performance reporting and accountabilitythese relationships play out.” Three critical components in an accountability relationship are: holding people to account so that everyone involved in the relationship understands it better and feels more responsible; encouraging participation on all sides, to help build trust; and context.Many health care systems are structured so that no single actor is accountable for an organization’s performance. However, for people who work in BC’s health authorities, Brian Schmidt said, “it often feels as though we are the only actors in the health system being held to account ... Managers and governors of health systems are feeling totally obliged to ensure their organization is accountable and performance agreements are maintained. But many other groups, such as politicians, clinicians, and patients, are not.” While no one can be truly accountable for the whole system, it is important to view it as a system, with everyone being accountable—including the clinicians who play a role on the front lines and the patients who make decisions about their health care.The need for accountability in our health systems stems from very fundamental problems. There is limited money to treat a growing burden of disease, and our current approaches and outcomes aren’t satisfactory. We need to recognize these issues, but also accept that they are the result of the action (or inaction) of actors at many different levels—not just the local clinician or manager.      				      		    	      —Brian SchmidtPerformance agreements can enhance accountability, in part by clarifying accountability relationships. But it is important that agreements clearly state who is making the decisions. For example, in the case of BC’s regional health authorities, Auditor General Wayne Strelioff hasThe accountability relationship: who’s responsible?The key interaction in the health care system is between the patient and the clinician. All discussions about accountability really concern the different mechanisms that can help ensure that interaction is as good as it can be.  However, the patient and the clinician operate within a system that has multiple effects on their relationship. Peter Smith likened the complex network of accountability relationships created by health care systems to a spider’s web. Each agent is accountable to several principals, who may themselves be accountable to others. If we look at just one node on this web, such as the provider organization, we can identify several principals to which it will be accountable—the patient, the government, and the insurer (if they are different)—as well as agents that are accountable to it (e.g. the clinician). An accountability regime must therefore service numerous potential relationships. But in the accountability debate, there has been an unfortunate tendency to talk about accountability relationships in general, without ever defining a specific relationship. To disentangle the web of accountability we need to define the relationship more carefully. “A shift from generalities to specifics would take away some of the fears of the people who are held accountable,” said Lillian Bayne, “as well as some of the concerns and frustrations of those on the outside who are watching4CHSPR’s 16th annual health policy conferencerecommended that only the Minister of Health and the chair of the authority sign the performance agreement, with the deputy minister reporting to the minister and the CEO reporting to the chair. While some people believe that multiple accountability—i.e. having everyone sign the agreement—works quite well, because everyone comes to the table and understands everyone else’s roles and views, Mr Strelioff believes that at the end of day, it’s too hard to figure out who carries the responsibility.In BC, there is also the difficult question of what works better: top-down control from the ministry, or autonomy at the level of health care delivery. The health authorities need autonomy if they are to arrive at creative solutions and be empowered to make changes. But at the same time, Alan Thomson pointed out, when politicians and the government are held accountable to the public, they must have some control over the health authorities.So the question remains: who’s in control, and who’s responsible?How useful is performance reporting?The purposes of performance reportingPerformance reporting has numerous purposes, but they are often poorly defined. What do we want performance reporting to achieve? Are we looking for better health outcomes, more efficient use of resources? Do we want hospitals to have an accountability to patients and taxpayers like companies do to their shareholders? Are we trying to increase public confidence in the health care system? Or do we just want our public institutions to behave differently? Charles Normand said that, to some extent, performance reporting is a means of encouraging hospitals to consider their patients as “customers rather than inconveniences.” It can also be a form of arms-length management, as national or provincial governments set standards for local health organization performance. In the US, performance reporting is often designed to inform patient choice. In the UK, it is intended to help citizens get the most from their health care system.This is the essence of accountability—the obligation to report on and answer questions about responsibilities. Performance information is relevant because it helps assess whether responsibilities are fulfilled well, and it helps legislators make decisions about resource allocations and responsibilities. 			              	                  —Wayne Strelioff5Who uses performance reports?One of the challenges of performance reporting is catering to multiple audiences with different information needs. The four key users of health system performance information are patients, clinicians, managers, and government. The patientEven among patients, there are at least two separate groups with different objectives as information users. The established patient knows there is something wrong with him or her, and that a medical intervention is required. Because they know they will use the system and may have the chance to exercise choice, established patients have a natural incentive to scrutinize performance reports. They will want information about which provider will give the best care. The potential patient does not have a specific, immediate health care need. Because they are not choosing a treatment or provider, potential patients have little incentive to scrutinize performance reports. They simply want reassurance that if something were to happen, the health care system would be governed well and respond to their needs. In the US, numerous web-based performance reports are available. Reports designed for potential patients tend to give information on the quality of the health plan rather than the health care provider. Other, more specific, reports are designed to serve the needs of established patients. They may include information on particular procedures, such as coronary bypass surgery (a common focus of performance measurement), abdominal surgery, aortic or aneurysm repair.The clinicianClinicians have different information needs than their patients. In general, clinicians do not find most public reporting helpful. Although the information itself may be useful, the publicity often carries negative side effects. When clinicians do use public reports, it may be to achieve one of two quality improvement goals. The first is safety: identifying poor performers who could be operating in an unsafe way. For this, clinicians need outcome data to track performance over time to detect problems and address them at a specified threshold. The other goal is to improve the overall quality of health care (continuous improvement). To achieve this, reports should identify top-performing clinicians to be emulated. This could be the strategy with the greatest potential for gain in health care system performance, because the majority of practitioners are neither unsafe nor top performers; they are in the middle and could improve. The managerManagers generally use performance reporting as an instrument in the governance of health care systems. Managers need information—not necessarily reported publicly—on the services they purchase, the services they produce, and comparative information on the performance of other systems for benchmarking. For purchasing purposes, performance reporting provides the basis for negotiating and monitoring formal contracts; it needs to be verifiable, and is often associated with incentives.Improving performance reporting and accountability6CHSPR’s 16th annual health policy conferenceIDENTIFYING SAFETY CONCERNSTwo recent scandals in the UK highlight the need for comprehensive systems capable of identifying safety concerns.The first, at the Bristol Royal Infirmary, concerned complex open-heart surgery on infants. Despite systematically poor outcomes (a mortality rate eventually determined to be double that of similar surgical units in England), the team of surgeons continued to operate for nearly five years after concerns were first raised. The subsequent inquiry found that while the performance information was available, no one acted on it, or reacted appropriately when issues were identified. The inquiry sparked major reforms to the National Health Service, focusing on making it safer and more accountable.The second scandal involved a general practitioner, Harold Shipman, who is thought to have murdered at least 215 of his patients over a period of 23 years. Because he particularly targeted elderly women, mortality rates in his practice were nearly ten times higher than those of neighbouring medical practices. Concerns were raised, but in this case, preliminary investigations failed to piece together the available information. Shipman killed three more patients before finally being arrested a year later. Following the criminal investigation, an independent inquiry was established to determine what changes to current systems should be made to safeguard patients in the future.For benchmarking purposes, information is used to compare organizations, with the emphasis on managerial processes associated with good performance. In the UK, for example, some managers use benchmarking schemes to find out whether their organization delivers as well as other organizations. The government The government is a player in both executive and democratic accountability. As mentioned previously, executive accountability means holding organizations in a devolved health care system accountable to a higher level, such as the provincial government. A system designed for executive accountability should measure whether organizations are adhering to legitimate policy constraints in measurable areas. The natural users of this kind of information are the system managers. Democratic accountability, on the other hand, means holding health care providers accountable to the patients they serve. It must ask, “Is this the sort of health care system patients want?” The natural users of performance reports for democratic accountability are voters, the media, and politicians.Public reporting versus publicityA great deal of discussion at the conference focused on the public’s need for and use of performance information. Citizens need enough information to be able to participate in active debate, so they can put pressure on their health care system to improve in the ways they want. But what is enough? And how much does the public actually want to know? The irony of public performance reporting is that reporting on every drawback of the system may in turn sacrifice people’s trust in that system. While quality7		           	        		           Improving performance reporting and accountabilitySuccess or failure?Wendy Armstrong believes that accountability and performance measurement and reporting are failing the public. She gave three reasons. First, most of the time, no one is tracking, evaluating, or even acknowledging the dramatic and largely invisible changes, both internal and external, that have occurred in the Canadian health care system over the last decade. These include:– a dramatic shift from in-hospital care to out-patient 	   and out-of-hospital care;– the delegation of responsibility and control to newly    formed health authorities in most provinces, and the    replacement of civil servants by commercially-        oriented consulting firms;– the replacement of small professional concerns with    large international business concerns;– auto insurance, workers’ compensation plans and       employer disability plans picking up health care costs    no longer covered by public health care plans; and– an unprecedented explosion of commercial marketing    and private sales of new and often experimental drugs    and technologies.Second, “spin”—rather than plain English—is being used to introduce changes to public health plans, many under the guise of health care reforms. For example, while Canadians are told that hospitals and physicians cannot charge user fees or extra billings, patients are then forced to pay user fees for a whole array of services that are no longer considered hospital and physician services. We are told that home nursing is covered; but we aren’t told that the hands-on care and support thatmay improve, confidence could still decline. People want to trust the health system. When they are sick, they don’t want to have to track the evidence to find out how their clinicians are performing; they simply want them to perform well. The human appetite for information ranges from moderate to voracious. Our nature is such that when all our human needs are being met by the systems we rely upon, we are amazingly complacent people, and we ask little about those systems. However, when the systems fail and our needs are not met, or our safety or security is threatened, we suddenly demand a lot of information, including who is accountable for this unsatisfactory situation.					      —Dennis KendelReporting on performance is a useful step towards public accountability. But one of the challenges of releasing information publicly is that it can be easily misinterpreted. “The public is intelligent, and more sophisticated than we often realize with respect to data,” Brian Schmidt said. But while the public might treat the information with the right amount of attention, the media, in searching for a “story” may provide a distorted interpretation.Charles Normand believes that when performance measures are reported to the public, the best approach is to release information routinely but without fanfare. Those who need or want the information can get it, but the reports are not pushed into the news headlines. Heavily promoted reports tend to evoke reactions—most of them negative—from government, the media, and other groups. So the quiet and routine release of information can accomplish the benefits of public reporting, without the drawbacks of publicity.8CHSPR’s 16th annual health policy conferencewe called nursing care until 15 years ago has been renamed personal care and removed from coverage.Third, there is confusion over the purpose of Medicare. The primary purpose of a universal prepaid public health plan is to provide families with security, peace of mind, and freedom from the fear of financial hardship and unnecessary suffering. But in Alberta, and many other parts of Canada, so much of the burden of cost and care has been shifted that we are beginning to see substantial costs resulting from lack of access to care. And few—if any—performance indicators tell us whether our current system or reforms are working. Instead, performance indicators look at health status, which is the responsibility of government, not of Medicare.One result of the failure of accountability and performance measurement and reporting is that citizens are becoming disengaged. A December 2002 Health Insider poll found that a significant number of Canadians feel disenfranchised from debate and policy making. In his Citizen’s Guide to Public Accountability, Henry McCandless notes that citizens are increasingly bewildered by their loss of control over people who are in authority and who are responsible for preventing harm in society.And while there is a lot of information available about health care, it isn’t meaningful or relevant to many people. Until they need the system, few Canadians have any idea what public or private health insurance pays for, what the price of common services are, what the safety or value of goods or services are, who makes decisions, or how to obtain remedies.Mixed resultsOn the other hand, Peter Smith described the impact of public performance reporting as mixed: some initiatives have shown evidence of quality improvement, but have also attracted negative responses from health care providers. To date, the main users of public reports have been provider organizations—particularly hospitals—to identify what has gone wrong in their organizations, and to find areas of improvement. Clinicians, in general, don’t find public report cards useful, because they lack the detail needed to make specific improvements. Patients have made only limited use of available data, as have purchasers. Patients and purchasers may not be using the results because they aren’t communicated effectively; little work has been done to determine how this information should best be presented. Various public reporting initiatives have been trialled, but we still don’t know the best way of getting users the information they want, in a way that will be useful to them. Overall, performance reporting appears to serve a useful role in promoting executive accountability, but it has only had a limited influence on democratic accountability and patient involvement. Professor Smith believes that in the future, if we want more informed and assertive citizens, we need to identify the best ways of serving their information needs. And we should think carefully about which mechanisms give citizens, clinicians, and others the incentives to search out and scrutinize the system when it isn’t performing well.9Improving performance reporting and accountabilityIn any case, despite the complexity and challenges of accountability measures in health care systems, public reporting is here to stay. “The question is not whether to report publicly,” Professor Smith said. “Citizens expect it, the information is there, and it’s a central element of the accountability of any health system. The question is how can we do it most effectively?”Making it workThere has been an explosion of interest in recent years in the reporting—both private and public—of health system information, partly as a result of advances in information technology. But there is still great variation in approaches to performance measurement and reporting. If we want to increase the usefulness of performance reporting, and thereby to increase accountability, we have to measure the right things, measure them the right way, and then do the right things with the data. Currently, while there is a lot of information available about health care, much of it is neither relevant nor meaningful to the different people who need it. We are collecting large amounts of information without a clear purpose in mind. What to measureThree key issues were raised on the question of what to measure: decision-relevant information versus information for monitoring; simplicity versus complexity; and cost. Wayne Strelioff advocates gathering information that will be useful in making decisions—not just gathering information for the sake of it. If no one uses the information, it becomes less credible because everyone knows no one uses it.  	On the other hand, Alan Thomson argued that data should be gathered for monitoring purposes, with decisions made only when monitoring shows that performance is slipping. Until that point, it’s difficult to know what sort of information is needed. And while it’s expensiveCASE STUDY: THE CABG SURVEYThe most celebrated US reporting scheme is the New York State Coronary Artery Bypass Graft (CABG) survey. All New York hospitals (approximately 30) were ranked according to an estimate of their CABG mortality. Some hospitals had such high mortality statistics that an informed patient might avoid them. Individual surgeons were also ranked, according to the risk-adjusted ratio of observed to expected patient mortality. The results of this work have clearly driven up certain aspects of quality in this area of clinical practice in New York. But evidence now shows that doctors are turning down high-risk patients—or operating too readily on low-risk patients—in order to secure a favourable ranking.10ACCOUNTABILITY THROUGH THE STAR RATINGS SYSTEMCHSPR’s 16th annual health policy conferenceThe method currently being used in the UK to enforce executive accountability in health care is the star-ratings system. Health organizations are given zero, one, two, or three stars on the basis of their performance, and are rewarded or punished accordingly. The primary criteria for evaluating an organization’s performance are targets included in the public service agreements established by the Ministry of Health. These targets include reducing waiting times, guaranteeing GP access, improving patient experience, and reducing inequities in health. For each key target, the organization is given a mark of achieved (zero penalty points), underachieved (two penalty points), or significantly underachieved (five penalty points). To reach the final score, the organization is also scored on 28 additional indicators, with clinical, patient, capacity, and capability focuses. Zero-star organizations receive serious sanctions: if there is no improvement within three months, managers or chief executive officers lose their jobs. Three-star organizations receive some autonomy over how their funding is spent, and win the chance to apply for “foundation” status, which allows greater freedom from government oversight. There is some confusion about the goal of the star-ratings system. While some people say the star ratings are used by patients and clinicians, few patients know about their hospital’s star rating, and many continue to seek treatment from zero-star hospitals. Others say the system is a marketing device aimed at reassuring the public that action is being taken on health care quality, or at encouraging competition among clinicians. But the main purpose of the system is executive accountability—getting hospitals to do what the government wants them to do. There is also concern that aggregated measures, such as a star rating, which assess numerous individual aspects of the health system and combine it into a single score, loses too much information to be useful. And aggregation implicitly incorporates value judgments. “For the purposes of democratic debate,” Peter Smith said, “the priority should not be the production of ranked league tables, but the release of information into the public domain without prejudging what is important or how it shouldbe used.”11Setting targetsSetting measurable targets for performance—and targets that are meaningful to all players—is equally challenging. Performance targets must be directly under the control of the clinicians, managers, or governors being held accountable. A study produced by Peter Smith and his colleagues showed how the UK equivalent of local health areas in BC were directly responsible for their performance in certain areas but not in others. For example, only eight per cent of the variation in standard mortality rate (ages 0–74) could be explained by what the district itself was doing, while 80 per cent of the variation in waiting times for inpatient surgery was directly attributable to the district’s activities. To help ensure that targets really are under the control of those who are being held accountable, and to secure buy-in from everyone involved, targets should be developedImproving performance reporting and accountabilityto collect data with no apparent immediate use, if you don’t collect it regularly, even if you don’t know how it will be used, it will be very difficult to determine why performance is slipping.The biggest issue is that we’ve got access to all kinds of information, but it’s not the information we need.				                 —Wendy ArmstrongSimple assessments won’t capture the complexity of health care systems. However, complicated assessments can be too difficult to understand and interpret. For example, wait lists, a performance measure with enormous public support, is not clinically useful, because it tells us only one important thing: how many people are on a wait list. The process by which patients get on and off wait lists is complicated. The length of the list doesn’t tell us how long a serious case will have to wait, or whether there will be adverse effects from waiting. And then there is the issue of cost. Data collection—especially complex data collection—takes resources. And those resources must necessarily come out of the money available to fund health care. “If the flow of money from taxpayer to health care provider is visualized as a stream of water,” Charles Normand said, “there are already numerous sponges waiting in between payer and provider to soak up resources. Conducting extensive data collection adds another sponge.” But performance assessment requires good information.DATA COLLECTION THROUGH ELECTRONIC HEALTH RECORDSIt has been suggested that electronic health records be used to make data collection and performance management easier. But which data from health records would be useful? Managers would need time to look at the data and see patterns—and it would be difficult to see patterns in data involving issues that are difficult to compare, such as home care, Pharmacare, and medical care services. These different services can’t be “added” together unless they’re turned into dollar values, which will not help a manager determine the quality of care.12CHSPR’s 16th annual health policy conferencewithin a partnership of those asking for accountability and those being held to account.Targets also must also be set with attention to the question of whether we have the capacity to make changes. Brian Schmidt argues that, in our current structure in BC, we do not. The focus on redesigning the process of health care delivery yields incremental quality improvements, but does not push a health care system to the next target level. “The system needs heath care delivery research capacity,” he said, “and the receptivity within health authorities to generate, understand, and act on information.” In BC, the health care system spends between one and two percent of its budget on information management. Top performing organizations spend between five and ten per cent. This means that, at present, we do not have the information capacity to be very accountable for our services.Steven Lewis pointed out that while we are lacking targets for both specific and overall performance, we also don’t have targets based on optimal performance. If we had this information, we could more easily describe the value gained per unit of money spent to help with target-setting and spending decisions.In addition, targets may actually conflict with one another. For example, by trying to increase accessibility and quality at the same time with limited resources, we may be working at cross-purposes. Local autonomy may conflict with a high-quality outcome, and efficiency can conflict with professional self-regulation. “Rather than resolving these conflicts, we tend to add another dimension to the performancemeasurement and tally up the score,” Mr. Lewis said. “Or we decide not to tally up the score, because information would be lost. Either way, we are encouraging health care systems to pursue contrary goals.”Finally, targets change. Once information is validated and used to refine a particular domain of performance measurement, research may turn up entirely different findings that must again be addressed, making the establishment of stable and reliable performance indicators a challenge.What to do with the dataOnce data have been collected, there is the question of what to do with them. Some “adjustment” is often necessary—for example, accounting for variation in patient risk profiles in order to avoid blaming clinicians for poor outcomes that are outside their control—but adjusting data too much can make them meaningless.  It’s important to recognize that performance indicators contain value judgments: the more data are refined and aggregated, the more value judgments are incorporated. Peter Smith and others favour an approach that keeps the information as raw as possible, where data are aggregated and manipulated (e.g. using risk adjustment) only enough to make them understandable and usable. IncentivesAnother challenge in performance reporting is incentives. If there are winners, there are losers. And if the winners are rewarded, then those who are in a bad shape get worse and those in good shape get better. If the winners13ACCOUNTABILITY THROUGH ACCREDITATION Canada currently uses a system of accreditation to address some issues of management accountability. The Canadian Council on Health Facilities Accreditation program is voluntary, although accreditation is mandatory for teaching facilities and many others consider it necessary. Accreditation is awarded through self-assessment and a periodic evaluation by a team of surveyors, who generally approach this evaluation in a consultative rather than judgmental capacity. The three-year accreditation may require varying degrees of supervised improvements depending on the surveyors’ recommendations. Although the goal of the accreditation system is accountability and quality improvement, there are no mandatory indicators, and there is no national system of comparisons across facilities. Most hospital managers report that the self-assessment is the most valuable step: during this process, the health care teams can identify and resolve quality issues from a bottom-up perspective, rather than through directives from the top.Improving performance reporting and accountabilityare not rewarded, then there is no incentive to be a winner. Charles Normand pointed out that these issues need to be resolved in order to avoid a downward spiral among poor performing hospitals. Clear incentives against cheating are also needed; when designing accountability systems, it’s important to consider how cheaters can be caught, and what the consequences will be.14CHSPR’s 16th annual health policy conferenceActing on accountabilityFinding a balanceEven if every physician, nurse and physiotherapist performed perfectly each day, we would be misled if we believed that nothing will go wrong with the system. This is because many of the factors that cause risk go beyond the performance of individual practitioners; much of the risk of harm comes from broader systemic factors. However, completely abandoning the “name-and-blame game” and looking only at systemic factors is a risk in itself: this approach is only valid if every individual in the system is competent. “If you walk away from mechanisms that certify the competence of individual practitioners,” Dennis Kendel said, “you will have kicked an important leg out from under the safety stool.”  Anticipating changeEven the most perfect system of performance reporting will become less useful over time. The declining utility of antibiotics reminds us that organisms respond to their environment by changing. “People being assessed by performance measurement tools do the same thing: they learn how to perform to those tools,” Charles Normand said. “Organizations will align their priorities according to measured targets, and facets of clinical care that are not measured may be ignored, and suffer.” Any sustainable system of performance reporting must therefore adapt and evolve to continue challenging health care providers.Eternal vigilanceFrom the point of view of public engagement in the health care system, Wendy Armstrong believes we’ve dropped the ball over the last decade. But there’s a good reason for this: we’ve been inundated with change, not just in health care but also in the media, insurance, agriculture—in every area of society.  “We must stop and forgive ourselves for our indiscretions, but then pick up the ball and run with it again,” Ms. Armstrong said. “Most importantly, we need to remember that rights and quality of life don’t just happen. If we want a well-functioning health care system, fair markets, good government, and a just society, the price is eternal vigilance.”  Today we’ve learned that there’s a huge appetite for the truth, and the public can handle the truth. we just have to find ways of telling the truth a little more quickly and robustly.We must also remember that, although as diagnosticians we have given a semi-dismal picture of where we are, we are immensely further ahead that we were five years ago, and infinitely further ahead than we were ten years ago, and we will be a hell of a lot further ahead five years from now as well, as long as we are vigilant. 					       —Steven Lewis15Improving performance reporting and accountabilityOUR CHALLENGESSteven Lewis wrapped up the conference by presenting the challenges highlighted by the day’s discussions: Communication When we talk about accountability and performance measurement, we are dealing with very complex information. We need to present that information responsibly. The antibiotic metaphorPeople being assessed by performance measurement tools will learn how to perform to those tools; even the most perfect system of performance reporting will become less useful over time. However, in the best sense, measuring performance can also make you better, because you are being observed.The utilitarian challenge How do we find the balance between the individual patient and the collective public world? Where do reasonableness and affordability come in? We must decide: how utilitarian should we become? Explaining what we doWe need to move away from accountability as due diligence reporting to the more difficult task of explaining what we do, without linguistic confusion and propaganda. DatabaseWe need to create a real-time, accurate database of good, reliable performance information. Progress How do we make real progress on accountability and performance measurement when power is so highly diffused that it’s almost impossible to locate? How do we proceed without focusing exclusively on technical performance and ignoring political and democratic performance? How do we avoid skewing the system towards cardiac care, cancer care, and hip replacements (because you can measure them) while ignoring mental health issues and social determinants?The deliberative challengeWe want more accountability and performance measurement, but we don’t know how to handle it. The best response to information is to deliberate—hard, collectively, and across disciplines. Unfortunately, our culture doesn’t promote deliberation: the response to information is to put it in a headline, make a ranking, and rush to pass judgment. We need to respond to performance information with deliberation. Fixing itWe can know a lot about performance without being able to do anything about it. We haven’t done enough conceptual work on a logic model to understand the mechanism of action in any performance domains. Improving performance requires a deep understanding of the kind of data you need and how to use it. System performance We need to develop processes that measure missing pieces at the system level, focusing on philosophical and ethical performance dimensions?16CHSPR’s 16th annual health policy conferenceSPEAKER BIOGRAPHIESWENDY ARMSTRONG Independent Public Policy Analyst and Board Member of Consumers' Association of Canada (Alberta) and PharmawatchWendy Armstrong is an independent consultant, public interest researcher and policy analyst focusing on community and societal impacts. She is a frequent conference presenter on public policy issues (particularly relating to health policy) from the public and community perspective, and is the author of three reports on the impact of health reforms in Alberta on citizens. Wendy is a board member of the Consumers’ Association of Canada (Alberta) and PharmaWatch.Wendy has broad knowledge of insurance, standards, marketplace practices, alternative therapies, food safety, prescription drugs and drug safety, medical device safety issues, health care financing and delivery, and access to information and privacy issues. She has served on such committees as the National Privacy Working Group on Health Information, the Public Advisory Committee to the Law Reform Commission of Canada, the Alberta Interim Committee on Drug Utilization, and the Alberta Steering Committee for Health Technology Assessment.  LILLIAN BAYNEBC Regional Officer, Canadian Health Services Research FoundationLillian Bayne is an independent consultant working in the field of health policy, planning and research. Between April of 2001 and November 2002, she served as associate executive director and special advisor to the Commission on the Future of Health Care (the Romanow Commission). Lillian was regional director-general for Health Canada’s BC/Yukon Region from 1999 until she joined the Commission. Prior to that, Lillian was an assistant deputy minister with the BC Ministry of Health, where she held portfolios including community health, mental health and forensic psychiatric services, legislation and professional regulation, and intergovernmental health relations. Lillian has extensive committee experience, including as chair of the Federal/Provincial/Territorial Advisory Committee on Health Services, as secretary treasurer of the BC Health Research Foundation, and as a member of the BC Health Professions Council. She is currently BC Regional Officer for the Canadian Health Services Research Foundation.CHARLYN BLACKDirector, Centre for Health Services and Policy ResearchCharlyn Black MD, ScD, joined the Centre for Health Services and Policy Research in 2002. She was previously a founding member and co-director of the Manitoba Centre for Health Policy, where she played a key role working at the interface between research and policy. Between September 2000 and August 2001, Dr Black worked as a senior advisor to the president of the Canadian Institutes of Health Research and as a visiting scientist and senior advisor to the president and CEO of the Canadian Institute for Health Information. She has served on a number of influential committees, including the Federal/Provincial/Territorial Advisory Committee on Health Services, the Canadian Population Health Initiative Council and the Steering Committee of the Western Canadian Waiting List Project. Her research interests focus on applications of population-based information systems, uses of administrative data to assess and monitor quality, effectiveness and outcomes of medical care, and the development of data-driven information tools to inform and improve health care delivery.DENNIS KENDELRegistrar, College of Physicians and Surgeons of SaskatchewanDennis Kendel has served as the CEO of the College of Physicians and Surgeons of Saskatchewan since 1986, and played a leadership role in a wide range of provincial and national organizations including the Heath Services and Research Commission, the Saskatchewan Health Information Network, the Health Quality Council, Saskatchewan Blue Cross, the Federation of Medical Licensing Authorities of Canada, and the Medical Council of Canada. During the first fifteen years of his career practising family medicine, Dr Kendel also held a number of leadership roles with the Saskatchewan Medical Association and the Canadian Medical Association.17Improving performance reporting and accountabilityDr Kendel has a keen interest in patient safety and health care quality improvement. He is committed to greater public accountability in health care and greater public engagement in health policy formulation, and has demonstrated this commitment by successfully lobbying for greater public transparency and involvement in the governance of the College of Physicians and Surgeons.STEVEN LEWISPresident, Access ConsultingSteven Lewis is a health policy and research consultant based in Saskatoon, and Adjunct Professor of Health Policy at the University of Calgary. Prior to resuming a full-time consulting practice he headed a health-research granting agency and spent seven years as CEO of the Health Services Utilization and Research Commission in Saskatchewan. He has served on various national boards and committees, including the National Forum on Health, and the Governing Council of the Canadian Institutes of Health Research. He worked closely with the Fyke Commission in Saskatchewan, and co-edits the annual Canadian Institute for Health Information Health Care in Canada reports. Steven was recently named to Canada’s first Health Quality Council in Saskatchewan. CHARLES NORMANDProfessor of Health Economics, London School of Hygiene & Tropical MedicineCharles Normand’s research has focused on the finance, delivery and evaluation of health services. He has written two guidebooks on health care finance for the World Health Organization and the International Labour Office, and is a member of a team that runs a large UK Department for International Development program to work on development of health systems, and ways of improving access for the poor. He has recently carried out studies on the effects of organizational mergers in the health sector in the UK, and on alternative ways of allocating government resources.Charles has played a leading role in health sector management training through the World Bank Flagship Programme, is President of the Association of Schools of Public Health in the European Region and chairs the steering committee of the WHO European Observatory on Health Systems. He took up a new post as Edward Kennedy Professor of Health Policy and Management at Trinity College, University of Dublin in January 2004. ALECK OSTRY Assistant Professor, Department of Health Care and Epidemiology, University of BCAleck Ostry is an assistant professor in the Department of Health Care and Epidemiology at the University of British Columbia, and a faculty associate with the Centre for Health Services and Policy Research. He is the recipient of a five-year Canadian Institutes of Health Research New Investigator salary award, beginning July 2000, and a Michael Smith Foundation for Health Research Scholar Award, beginning July 2002. Aleck’s research covers four main areas: 1) the social determinants of health with a focus on labour market and workplace determinants; 2) the measurement of workplace stress in industrial and healthcare workforces; 3) the evolution of nutrition policy in Canada; and 4) history of medicine and public health. For the past several years, he has also been involved both with the population health program at the Canadian Institute of Advanced Research. Aleck has an MSc in health services planning, an MA in history, and a PhD in epidemiology. BRIAN SCHMIDTSenior Vice President, Strategic Health Development and Performance Management, Provincial Health Services AuthorityThe Provincial Health Services Authority ensures the planning, coordination or operations, and quality of service of province-wide health care programs and services in British Columbia. Prior to joining the PHSA in July 2002, Brian Schmidt served as the chief operating officer for the British Columbia Cancer Agency for fourteen years, and previously served in a number of administrative positions within hospitals in British Columbia.Brian is a surveyor for the Canadian Council on Health Services Accreditation, and sat on their National Advisory Committee to advise on the development of the new AIM18CHSPR’s 16th annual health policy conferenceStandards. He has been active for a number of years on the Board of Directors of long-term care and home-care organizations, and has just completed a three-year term as chair of the board of directors of the Health Employer’s Association of British Columbia. He also serves as Clinical Associate Professor in the Department of Health Care and Epidemiology at the University of British Columbia.PETER SMITHProfessor of Economics, Centre for Health Economics, University of YorkPeter Smith has published widely on his primary research interests of the financing, efficiency and performance management of health care. He has been a member of many UK policy committees in health and the broader public sector, including the Advisory Committee on Resource Allocation and the Treasury's Performance Information Panel. He has also advised numerous other national and international agencies on performance, including the OECD, the World Health Organization and the World Bank, and he leads a US/UK research initiative on incentives to improve clinical quality. He is currently funded by a three-year fellowship from the UK Economic and Social Research Council to enable him to spend more time on his research.WAYNE STRELIOFFAuditor General, British ColumbiaWayne Strelioff was appointed Auditor General of British Columbia in May 2000. The Auditor General’s report—A Review of Performance Agreements Between the Ministry of Health Services and the Health Authorities—was released in May 2003. Wayne discussed his perspective on how these performance agreements can strengthen governance, accountability and performance in British Columbia’s regional health system.Born and raised in Saskatoon, Wayne served as the Provincial Auditor of Saskatchewan from November 1990 to April 2000. He graduated from the University of Saskatchewan with a Bachelor of Arts and a Bachelor of Commerce, and in 1978, obtained his Chartered Accountant designation with Peat, Marwick & Mitchell (now KPMG) in Saskatoon. After receiving his designation, Wayne returned to the University of Saskatchewan and obtained a Masters of Science degree in accounting.ALAN THOMSONExecutive Director, Standards and Performance Development, Performance Management and Improvement Division, BC Ministry of Health ServicesDr Thomson has 27 years experience working in the health ministries in two provinces. He is a physician with further training in epidemiology and health service administration, with interests in the relationship between population health (care) needs and resource allocation. He is particularly interested in the use of data, especially population-based rates, to look at variations in utilization and its relation to variations in needs.Dr Thomson’s recent activities include the direction of the development and implementation of a population needs-based funding model for resource allocation to BC health authorities; the development of the performance agreements between the health authorities and the BC Ministry of Health Services; and the development and introduction of acute care access standards and guidelines for BC health authorities to guide their restructuring plans. On the national level, he is currently chair of the board of the Canadian Coordinating Office for Health TechnologyAssessment.19HEALTH SERVICES AND POLICY RESEARCHCentre for429 - 2194 Health Sciences MallVancouver, BC Canada V6T 1Z3Tel:Fax:www.chspr.ubc.ca(604) 822-1949(604) 822-5690For more information: enquire@chspr.ubc.caThis document is protected by copyright. It may be distributed for educational and non-commercial use, provided the Centre for Health Services and Policy Research is credited.


Citation Scheme:


Citations by CSL (citeproc-js)

Usage Statistics



Customize your widget with the following options, then copy and paste the code below into the HTML of your page to embed this item in your website.
                            <div id="ubcOpenCollectionsWidgetDisplay">
                            <script id="ubcOpenCollectionsWidget"
                            async >
IIIF logo Our image viewer uses the IIIF 2.0 standard. To load this item in other compatible viewers, use this url:


Related Items