Open Collections

UBC Theses and Dissertations

UBC Theses Logo

UBC Theses and Dissertations

The splash and the ripples : assessing sustainability transition experiments Williams, Stephen 2019

Your browser doesn't seem to have a PDF viewer, please download the PDF to view this item.

Item Metadata

Download

Media
24-ubc_2019_november_williams_stephen.pdf [ 6.83MB ]
Metadata
JSON: 24-1.0383246.json
JSON-LD: 24-1.0383246-ld.json
RDF/XML (Pretty): 24-1.0383246-rdf.xml
RDF/JSON: 24-1.0383246-rdf.json
Turtle: 24-1.0383246-turtle.txt
N-Triples: 24-1.0383246-rdf-ntriples.txt
Original Record: 24-1.0383246-source.json
Full Text
24-1.0383246-fulltext.txt
Citation
24-1.0383246.ris

Full Text

THE SPLASH AND THE RIPPLES:  ASSESSING SUSTAINABILITY TRANSITION EXPERIMENTS   by  STEPHEN WILLIAMS  B.A., The University of Western Ontario, 1997 M.B.A., Simon Fraser University, 2002 Graduate Diploma, University of Waterloo, 2013     A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF  DOCTOR OF PHILOSOPHY  in  THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES  (Resource Management and Environmental Studies)   THE UNIVERSITY OF BRITISH COLUMBIA  (Vancouver)         September 2019  © Stephen Williams, 2019  ii  The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the dissertation entitled: The Splash and the Ripples: Assessing Sustainability Transition Experiments  submitted by Stephen Williams in partial fulfillment of the requirements for the degree of Doctor of Philosophy in Resource Management and Environmental Studies  Examining Committee: Dr. John Robinson Supervisor  Dr. Terre Satterfield Supervisor Dr. David Kahane Supervisory Committee Member Dr. Mark Warren University Examiner Dr. Hisham Zerriffi University Examiner    iii Abstract Fostering sustainability transition is a pressing global societal challenge. Rising GHG emissions, severe social and economic inequality, unsustainable production and consumption activities, and global environmental degradation and its human consequences are all indicators of an unsustainable world. Sustainability Transition Experiments (STEs) have been proposed as a method to accelerate sustainability transitions in response to these challenges. We need methods by which to evaluate STEs, to understand what is happening within transition processes, and to provide insight to STE designers and facilitators as to the efficacy of their processes. This need led to the development of my research questions: How can we conceptualize, and evaluate, the contributions to sustainability transitions of STEs? Was the Energy Futures Lab effective in supporting sustainability transitions in Alberta? I develop a three-part evaluation framework of process, societal effects, and sustainability transition impacts leveraging a development pathway framing. I apply this framework to the case of the Energy Futures Lab in Alberta, Canada which has the goal of fostering transition to a sustainable energy system. Through participant interviews, document reviews, then coding all data in Nvivo, I assess the EFL and the evaluation framework itself. I conclude the framework to be an effective method with which to evaluate STEs. The most significant contribution of the framework is in conceptualizing and assessing sustainability transition impacts. By leveraging the framing of development pathways, this framework provides a detailed set of evaluation categories and indicators that can be used to assess development pathway change and the contribution of an STE to such change.   iv The EFL itself strongly demonstrated commitment to process elements such as fairness, inclusivity, and transparency in its process design and implementation and included many elements that support transition. The EFL has demonstrable societal effects across all categories including individual capacity development, enhanced networks, and institutional change in policy and organizations. There are also examples where the EFL might have influenced characteristics of development pathway change. The evidence introduced in this dissertation leads me to conclude that, despite the limited evidence of direct impact, markers of transition are present illustrating the EFL’s contribution to sustainability transition impacts.   v Lay Summary Fostering sustainability transition is a pressing global societal challenge. Rising GHG emissions, severe social and economic inequality, unsustainable production and consumption activities, and global environmental degradation and its human consequences are all indicators of an unsustainable world. Sustainability Transition Experiments (STEs), which are collaborations between governments, NGOs, universities, and business, have been proposed as a method to accelerate sustainability transitions in response to these challenges. The question I attempt to answer is how to evaluate whether these STEs are actually making a contribution to sustainability transition and, if so, how? I develop a method to answer these questions and apply my evaluation framework to the Energy Futures Lab which is attempting to shift Alberta’s energy system in a more sustainable direction. The framework, and my analysis, will help STE designers and facilitators better achieve their goals of contributing to a much needed sustainability transition.  vi Preface I identified the EFL as a potential case study and designed the research program for this dissertation. I developed the interview protocols, conducted the majority of participant interviews, and developed the assessment framework used in my analysis. Some interviews were conducted by Energy Futures Lab team members as described further in Section 4.3.2. I was responsible for interview and document coding along with all analysis of research data.  Kimberly Slater, Gregoire Benzakin, Pani Pajouhesh, and John Robinson at the University of Toronto contributed to the development of the ‘three-level’ concept in my evaluation framework as published in Robinson, J., Slater, K., Williams, S., Pajouhesh, P., and Benzakin, G. (2018). TAF Evaluation Framework Outline. Report prepared for TAF. Slater led the writing of the report and developed the ‘light touch’ version of the evaluation framework, Pajouhesh contributed to the report and introduced a social justice lens to our analysis, and Robinson introduced the 3-level concept. The report was based on my evaluation framework and I contributed to the adaptation of that report to the new context.  Portions of Chapter 2 (Section 2.3.3) and Chapter 3 (Section 3.2) will be published in Williams, S. and Robinson, J. (In Review). Sustainability Transition Impacts: Evaluating Transdisciplinary Sustainability Transition Experiments. I conducted the literature review, wrote most of the manuscript, and developed the integration of SDGs into sustainability transition impacts evaluation. Robinson contributed to the integration of development pathways as a conceptual framing for the sustainability transitions impacts level of the framework along with the concept of ‘markers of transition’.  vii Portions of Chapter 2 (Section 2.3.2.1) have been published in Williams, S. and Doyon, A. (2019). Justice in Energy Transitions. Environmental Innovation and Societal Transitions. Vol 31, June 2019, pp. 144-153. I was lead author on the paper and developed the conceptual framing along with the assessment framework presented. Doyon wrote most of the literature review, especially the sections on environmental justice. Portions of Chapter 1 (Sections 1.2 & 1.3) and 2 (Sections 2.2 & 2.3) will be published in Williams, S. (In Press). The Alberta Energy Futures Lab: A Case Study in Socio-Cultural Transition through Public Engagement in Creating Spaces of Engagement. Eds. Wiebe, S. & Ney, T. University of Toronto Press. Content in Section 2.1 on sustainability transition experiments and 2.2.3 on societal effects evaluation was published in Williams, S. (2017, June). Evaluating Transition Experiments in Times of Rapid Change. Paper presented at International Sustainability Transitions 2017. Chalmers University, Gothenburg, Sweden. Content in Section 2.3.3.3 was published in Larsson, J., Williams, S. & Holmberg, J. (2018, June). Guiding Systemic Transition: A Cross-case Analysis of “transition labs” in Canada and Sweden Paper presented at International Sustainability Transitions 2018. University of Manchester, Manchester, UK. I wrote the EFL case study section and the analytical component applying our framework to the EFL. Larsson led the development of our analytical framework. Larsson wrote the case study on Challenge Labs in Sweden and the introductory material. Holmberg contributed to the integration of Sustainable Development Goal principles to frame our analysis. Small portions of Chapter 2 (Section 2.3.3.1) have been published in Rodina, L., Shah, S., Burt, J., Chapman, M., Gregr, E., McDowell, G., Williams, S., Wilson, N. (2018). Unpacking  viii Social-Ecological Transformations: Conceptual, Ethical, and Methodological Insights. The Anthropocene Review. Vol 5, Issue 3, 250-265. I contributed case examples of energy system transformation and a summary of MLP theory to this paper. Rodina and Shah led the collaborative writing process and developed the overall structure of the paper. The balance of authors contributed to the paper and wrote short case studies from the perspective of their respective disciplines. Content in Section 2.2.2 on co-production was published in Williams, S. 2017. Evaluating Societal Effects of Transdisciplinary Co-production Processes: Final Report. Report prepared for Mistra Urban Futures, Gothenburg, Sweden. An early version of my assessment framework developed in Chapter 3 has been published in Rosenbloom, D., Meadowcroft, J., Sheppard, S., Burch, S. & Williams, S. (2018). Transition experiments: Opening up low-carbon transition pathways for Canada through innovation and learning. Canadian Public Policy. Vol 44, Issue 4, pp. 368-383. I wrote the sections on evaluating transition experiments. Rosenbloom was lead author and wrote the bulk of the content.  Discussion of the LEDlab in Section 3.1.2 was published in Bird, K., Rueda, I. & Williams, S. (2017). Local Economic Development Lab Field Book. I wrote the evaluation section of this report. Bird wrote the majority of the content of the Field Book with contributions and editing from Rueda. Content in Section 3.1.6 on the Suncor Energy Foundation was published in Williams, S. (2017). Suncor Energy Foundation Social Innovation Investment Evaluation Report. Calgary, AB. Content in Section 3.1.8 on Sustainable Canada Dialogues was published in Potvin, C., Sharma, D., Williams, S., Meadowcroft, J., Rosenbloom, D., Hall, M., Hoffmann, M., Robinson, J.,  ix Sheppard, S., McVey, I., Ashley, B., Wielinski, G. and Morency, C. (2018). A framework to evaluate low-carbon energy transition learning projects. Report prepared for Natural Resources Canada. I wrote most of the literature review, led the development of the evaluation framework, created the figure illustrating the framework, and wrote the evaluation section. Sharma led and wrote the case study sections while Potvin was lead author and wrote the structure and framing of the report. Remaining authors reviewed and made substantive contributions to the report. Content in Section 3.1.9 on the Institute for Advanced Sustainability Studies was published in Schirmer, J. & Williams, S. 2018. Criteria and indicators for the societal effects of IASS activities. IASS. Potsdam, Germany. Schirmer led report writing and literature review. I contributed to the literature review and wrote sections on societal effects measures, co-production, and evaluation methods. This research project was approved by the UBC Behavioural Research Ethics Board under Certificate Number H15-00633.  x Table of Contents Abstract .......................................................................................................................................... iii Lay Summary .................................................................................................................................. v Preface ............................................................................................................................................ vi Table of Contents ............................................................................................................................ x List of Tables ............................................................................................................................... xix List of Figures .............................................................................................................................. xxi Acknowledgements ..................................................................................................................... xxii Chapter 1: Introduction ............................................................................................................... 1 1.1 Goals and research approach ...................................................................................... 1 1.2 Case study: The Energy Futures Lab (EFL) ............................................................... 4 1.3 EFL 2.0 and beyond .................................................................................................... 7 1.4 Action research and role of researcher in evaluating transition .................................. 8 1.5 Thesis structure ......................................................................................................... 10 Chapter 2: Theorizing Sustainability Transition Experiment Evaluation ................................. 13 2.1 Sustainability transition experiments ........................................................................ 13 2.1.1 Challenges evaluating sustainability transition experiments ............................ 15 2.1.2 Research approach and questions ..................................................................... 16 2.2 Review of evaluation literatures ............................................................................... 17 2.2.1 Theoretical framings of evaluating sustainability transition experiments ........ 17 2.2.2 Process evaluation ............................................................................................. 19 2.2.3 Societal effects evaluation ................................................................................ 22 2.2.3.1 Implications for evaluation ........................................................................... 24 2.3 Sustainability transition impacts ............................................................................... 26 2.3.1 Transition theory and the Multi-Level Perspective .......................................... 26  xi 2.3.1.1 Governing and managing transitions ............................................................ 28 2.3.1.2 Energy system transitions ............................................................................. 31 2.3.1.3 Learning and evaluation in transitions .......................................................... 32 2.3.1.4 Implications for evaluation framework ......................................................... 33 2.3.1.5 Critiques of transition theory ........................................................................ 35 2.3.2 Beyond transition theory ................................................................................... 36 2.3.2.1 Justice, power and politics in transitions ...................................................... 36 2.3.2.1.1 Just transitions ......................................................................................... 37 2.3.2.1.2 Does justice in transitions matter? .......................................................... 40 2.3.2.1.3 Implications for evaluative framework ................................................... 41 2.3.2.2 Governance ................................................................................................... 42 2.3.2.2.1 Implications for evaluation framework ................................................... 45 2.3.2.3 Actors & agency in transitions ...................................................................... 46 2.3.2.3.1 Social practice ......................................................................................... 48 2.3.2.3.2 Implications for evaluation framework ................................................... 51 2.3.3 Sustainability in transitions literature ............................................................... 53 2.3.3.1 Transformations ............................................................................................ 55 2.3.3.2 Procedural sustainability ............................................................................... 56 2.3.3.3 UN Sustainable Development Goals ............................................................ 58 2.3.3.4 Implications for evaluation ........................................................................... 61 2.3.4 Evaluating sustainability transition impacts ..................................................... 62 2.3.4.1 Leveraging the Sustainable Development Goals .......................................... 68 2.4 Conclusion ................................................................................................................ 74 Chapter 3: From Theory to Practice: Operationalizing the Framework ................................... 82 3.1 Evolving the framework through practice ................................................................ 82  xii 3.1.1 Energy Futures Lab (2014-2018) ...................................................................... 82 3.1.2 Local Economic Development Lab (LEDlab) (2015-2018) ............................. 85 3.1.3 Public Scholars workshops (Feb & April 2017) ............................................... 88 3.1.4 National Status of Women Canada (NSW) (2017-2018) ................................. 89 3.1.5 Mistra Urban Futures (MUF) (2017) ................................................................ 92 3.1.6 Suncor Energy Foundation (2017-2018) .......................................................... 94 3.1.7 The Atmospheric Fund low-Carbon neighbourhood pilots (2018-2019) ......... 97 3.1.8 Sustainable Canada Dialogues (2018) ............................................................ 100 3.1.9 Institute for Advanced Sustainability Studies (IASS)  (2018) ........................ 106 3.1.10 Summary of case study influences on evaluative framework ......................... 109 3.2 Operationalizing framework ................................................................................... 112 3.2.1 Process ............................................................................................................ 113 3.2.2 Societal effects ................................................................................................ 119 3.2.3 Sustainability transition impacts ..................................................................... 125 3.2.4 Additional contextual factors .......................................................................... 134 3.2.5 Evaluation framework summary ..................................................................... 135 Chapter 4: Methodology ......................................................................................................... 139 4.1 Introduction ............................................................................................................. 139 4.2 The role of the “self-reflexive scientist” ................................................................. 141 4.3 Data sources & collecting data ............................................................................... 145 4.3.1 Meeting observation & feedback surveys ....................................................... 145 4.3.2 Semi-structured participant interviews ........................................................... 147 4.3.3 Semi-structured design team and partner interviews ...................................... 150 4.3.4 External document review .............................................................................. 151 4.3.5 Design document review ................................................................................. 152  xiii 4.3.6 Secondary data reuse ....................................................................................... 152 4.4 Data analysis methods ............................................................................................. 153 4.4.1 Coding ............................................................................................................. 153 4.4.2 Outcome harvesting ........................................................................................ 154 4.5 Contribution and causality ...................................................................................... 155 4.5.1 Projecting impact ............................................................................................ 157 Chapter 5: Assessing the Energy Futures Lab Process ........................................................... 160 5.1 Introduction ............................................................................................................. 160 5.2 Inputs ....................................................................................................................... 161 5.2.1 Design and facilitation capacity ...................................................................... 162 5.2.2 Funding and partners ....................................................................................... 163 5.2.3 Motivations of project participants ................................................................. 165 5.3 External context ...................................................................................................... 166 5.4 Enabling conditions ................................................................................................ 168 5.4.1 Inclusivity ....................................................................................................... 168 5.4.2 Fairness ........................................................................................................... 173 5.4.3 Transparency ................................................................................................... 174 5.4.4 Trust ................................................................................................................ 176 5.4.5 Enabling conditions summary ......................................................................... 177 5.5 STE methods ........................................................................................................... 178 5.5.1 Dialogue .......................................................................................................... 178 5.5.2 Negotiation ...................................................................................................... 180 5.5.3 Collective problem solving ............................................................................. 182 5.5.4 Reflexivity ....................................................................................................... 183 5.5.5 Iterative adaptation/learning ........................................................................... 186  xiv 5.5.5.1 EFL design changes .................................................................................... 187 5.5.5.2 EFL strategy changes .................................................................................. 189 5.5.6 Methods summary ........................................................................................... 191 5.6 Supporting transition ............................................................................................... 192 5.6.1 Conceptualizing sustainability ........................................................................ 193 5.6.2 Experimentation and learning ......................................................................... 197 5.6.3 Aligning innovations ....................................................................................... 200 5.6.4 Niche-regime interaction ................................................................................ 201 5.6.5 Supporting transition summary ....................................................................... 202 5.7 Scope ....................................................................................................................... 203 5.7.1 Scope summary ............................................................................................... 206 5.8 Governance ............................................................................................................. 206 5.8.1 Power relations ................................................................................................ 207 5.8.2 Stakeholder capacities ..................................................................................... 209 5.8.2.1 STE support for niche innovations ............................................................. 210 5.8.3 Engaging future and non-human actors .......................................................... 211 5.8.4 Recognition ..................................................................................................... 212 5.8.5 Governance summary ..................................................................................... 216 5.9 Patterns .................................................................................................................... 216 5.10 Conclusion .............................................................................................................. 217 5.10.1 Theory ............................................................................................................. 217 5.10.1.1 Diversity ...................................................................................................... 218 5.10.1.2 Agreement, consensus and contestation ..................................................... 220 5.10.2 EFL process assessment .................................................................................. 222 Chapter 6: Assessing Energy Futures Lab Societal Effects .................................................... 225  xv 6.1 Introduction ............................................................................................................. 225 6.2 Individual capacity .................................................................................................. 226 6.2.1 Understanding and learning ............................................................................ 227 6.2.2 Agency and empowerment ............................................................................. 230 6.2.2.1 Reflexivity on role in transition .................................................................. 231 6.2.3 Changed or broadened perspectives, attitudes and expectations .................... 234 6.2.3.1 Awareness of oppositional arguments ........................................................ 236 6.2.4 Individual capacity summary .......................................................................... 237 6.3 Usable products ....................................................................................................... 237 6.3.1 Innovative technologies and social innovation adoption/scale ....................... 238 6.3.2 Action plans .................................................................................................... 239 6.3.3 Media .............................................................................................................. 239 6.3.4 Publications (academic & non-academic) ...................................................... 241 6.3.5 Written reports, research and practice overviews ........................................... 241 6.1.1 Usable products summary ............................................................................... 242 6.4 Networks and relationships ..................................................................................... 243 6.4.1 Expanded and strengthened networks ............................................................. 244 6.4.2 Sharing with networks .................................................................................... 248 6.4.3 New ways of working, collaborations ............................................................ 249 6.4.4 Development of social capital ......................................................................... 253 6.4.5 Networks and relationships summary ............................................................. 255 6.5 Institutional changes - policy .................................................................................. 255 6.5.1 New evidence introduced into policy/strategy ................................................ 256 6.5.2 Direct impact on policy and practice .............................................................. 257 6.5.2.1 Alberta Indigenous Climate Leadership program (AICL) .......................... 257  xvi 6.5.2.2 Emissions Reductions Alberta (ERA) ........................................................ 259 6.5.3 Indirect impact on the knowledge of policy-makers and practitioners ........... 260 6.5.4 Institutional changes - policy summary .......................................................... 262 6.6 Institutional change - organizational ....................................................................... 263 6.6.1 Shifts in organizational responsibilities/roles, changes in job description ..... 263 6.6.2 Shifts in investment strategy ........................................................................... 264 6.6.2.1 Regime renewables investment ................................................................... 265 6.6.2.2 BioJet/ERA/Alberta Innovates .................................................................... 266 6.6.2.3 Investment strategy summary ..................................................................... 267 6.6.3 Changes in organizational decision-making process or operating models ..... 268 6.6.4 Governance ..................................................................................................... 268 6.6.5 Institutional change – organizational summary .............................................. 270 6.7 Climate/energy effects ............................................................................................ 271 6.8 Patterns and cross-level connections ...................................................................... 273 6.8.1 Patterns ............................................................................................................ 273 6.8.2 Connecting process to effects ......................................................................... 275 6.9 Conclusion .............................................................................................................. 275 6.9.1 Theory ............................................................................................................. 275 6.9.2 Assessment ...................................................................................................... 276 Chapter 7: Assessing Energy Futures Lab Sustainability Transition Impacts ........................ 281 7.1 Introduction ............................................................................................................. 281 7.2 Reported development pathway changes ................................................................ 285 7.3 Socio-technical systems and governance ................................................................ 290 7.3.1 Governance roles and relationships ................................................................ 291 7.3.1.1 New actors and issues in public spaces and discourse ................................ 291  xvii 7.3.1.2 Changes in practices of participation in governance processes .................. 293 7.3.2 Reduced barriers to transition ......................................................................... 293 7.3.3 Socio-technical systems summary .................................................................. 297 7.4 Interlinking regime rules and behaviours ............................................................... 298 7.4.1 Justice .............................................................................................................. 298 7.4.2 Changes in regime routines ............................................................................. 302 7.4.3 Interlinking regime rules summary ................................................................. 305 7.4 Reinforcement at multiple levels ............................................................................ 305 7.4.1 Niche-landscape alignment ............................................................................. 305 7.4.2 Multi-level change .......................................................................................... 308 7.4.3 Reinforcement at multiple levels summary .................................................... 310 7.5 Actors and practices ................................................................................................ 311 7.5.1 Actor roles and relationships .......................................................................... 311 7.5.2 Changes in collective practices ....................................................................... 313 7.5.3 Actors and practices summary ........................................................................ 316 7.6 Social and ecological systems ................................................................................. 316 7.6.1 Alignment to SDGs ......................................................................................... 318 7.6.2 Socio-ecological summary .............................................................................. 321 7.7 Patterns and cross-level connections ...................................................................... 321 7.7.1 Patterns ............................................................................................................ 321 7.7.2 Connecting process to effects to impacts ........................................................ 322 7.8 Conclusion .............................................................................................................. 324 7.8.1 Theory ............................................................................................................. 324 7.8.1.1 Transformational actions and transformational impacts. ............................ 324 7.8.1.2 Markers of transition ................................................................................... 326  xviii 7.8.1.3 EFL as niche or regime ............................................................................... 327 7.8.1.4 Conceptualizing development pathway change .......................................... 328 7.8.2 Assessment ...................................................................................................... 329 Chapter 8: Conclusion ............................................................................................................. 332 8.1 Contributions ........................................................................................................... 335 8.2 Strengths and limitations ......................................................................................... 336 8.3 Potential applications of research findings ............................................................. 338 8.4 Areas of further research ......................................................................................... 339 8.5 Personal reflections ................................................................................................. 343 References ................................................................................................................................... 345 Appendices .................................................................................................................................. 364 Appendix A List of EFL Participants ................................................................. 364 Appendix B EFL Design Team and Governance Structure ................................ 367 Appendix C EFL Initiatives ................................................................................ 370 Appendix D EFL Workshops, Organizational, and Public Engagement ............ 374 Appendix E Interview protocols ......................................................................... 379 Appendix F Code list .......................................................................................... 388 Appendix G Code Sentiment Analysis ............................................................... 393 Appendix H Documents reviewed ...................................................................... 400 Appendix I EFL Products ................................................................................... 403  xix List of Tables Table 1: Process evaluation elements ........................................................................................... 21 Table 2: Evaluation elements from transitions literature .............................................................. 34 Table 3: Five characteristics of development pathways ............................................................... 65 Table 4: Mapping SDGs to sustainability transitions impact evaluation framework. .................. 71 Table 5: Key evaluation insights from literature review .............................................................. 79 Table 6: Evaluation framework used with Suncor Energy Foundation.. ...................................... 96 Table 7: Mapping evaluative questions to effects categories ..................................................... 109 Table 8: Operationalization of process evaluative measures  ..................................................... 115 Table 9: Operationalization of societal effects evaluative measures .......................................... 121 Table 10: Operationalization of sustainability transition impact evaluative measures ............... 126 Table 11: Mapping research methods to evaluative framework levels ....................................... 140 Table 12: Participant interview labelling scheme ....................................................................... 154 Table 13: Process evaluation categories and elements ............................................................... 161 Table 14: Inputs evaluation elements. ........................................................................................ 162 Table 15: Enabling conditions evaluation elements ................................................................... 168 Table 16: EFL participants by MAP category ............................................................................ 169 Table 17: Methods evaluation elements ..................................................................................... 178 Table 18: Supporting transition evaluation elements .................................................................. 192 Table 19: EFL vision, success principles, and SDGs ................................................................. 196 Table 20: Governance evaluation elements ................................................................................ 207 Table 21: Societal effects evaluation categories and elements. .................................................. 226 Table 22: Selected EFL media mentions .................................................................................... 240 Table 23: Selected EFL blog posts ............................................................................................. 242 Table 24: EFL network connectivity changes ............................................................................ 244  xx Table 25: Investment/granting influenced by EFL ..................................................................... 265 Table 26: Sustainability transition impact evaluation categories and elements ......................... 283 Table 27: Assessing EFL activities by SDG ............................................................................... 320 Table 28: EFL design team members ......................................................................................... 367 Table 29: EFL advisory council members .................................................................................. 368 Table 30: EFL partners ............................................................................................................... 369 Table 31: List of EFL initiatives ................................................................................................. 370 Table 32: List of EFL workshops ............................................................................................... 374 Table 33: Selected EFL organizational engagements ................................................................. 378 Table 34: Selected EFL media mentions .................................................................................... 403 Table 35: Academic publications related to Energy Futures Lab ............................................... 404 Table 36: Selected EFL blog posts ............................................................................................. 406   xxi List of Figures Figure 1: EFL phases ...................................................................................................................... 5 Figure 2: Sustainable Development Goals .................................................................................... 60 Figure 3: Using MLP to frame indicators ..................................................................................... 83 Figure 4: LEDlab evaluating systems change approach ............................................................... 87 Figure 5:NSW evaluation Framework.. ........................................................................................ 91 Figure 6: Proposed evaluation framework. ................................................................................. 102 Figure 7: Capturing societal effects of IASS .............................................................................. 108 Figure 8: Relationships between evaluation components. .......................................................... 137 Figure 9: Limitations of EFL Project Evaluation ....................................................................... 187 Figure 10: EFL results framework .............................................................................................. 188 Figure 11: Shifts in EFL strategy ................................................................................................ 190 Figure 12: Participant knowledge change ................................................................................... 227 Figure 13: Participant confidence change ................................................................................... 230 Figure 14: Connections before EFL ............................................................................................ 245 Figure 15: Connections after EFL (as of June 2018) .................................................................. 245 Figure 16: EFL connectivity initiatives ...................................................................................... 251 Figure 17: Alberta hydrocarbon production ............................................................................... 317 Figure 18: GHG emissions by sector .......................................................................................... 317 Figure 19: Greenhouse gas emissions by province and territory, Canada .................................. 318 Figure 20: EFL public engagement activities as of November 2018 .......................................... 378    xxii Acknowledgements This dissertation would not have been possible without the support of many, many people. First, I would like to acknowledge the financial support for this project provided by Mitacs Canada, the Public Scholars Initiative (PSI) at the University of British Columbia, the Social Sciences and Humanities Research Council, the Suncor Energy Foundation, and The Natural Step Canada.  The ideas in this dissertation benefited from collaborations – formal and informal – with colleagues around the world. In particular I would like to thank Kim Slater for introducing me to Social Learning, providing a key paper on social practice and MLP, and being a ‘critical friend’ over the years; Kim, Pani Pajouhesh, and Grégoire Benzakin for collaboratively developing the concept of a three part evaluation framework and testing with TAF; Andréanne Doyon for collaborations on environmental justice and energy transition; Johan Larsson and John Holmberg for collaborations at Chalmers University and elaborating the connection between sustainability transition and the Sustainable Development Goals; Pani and David Boyd for providing thoughtful comments on environmental justice and transitions; Darcy Riddell for insight into foundations and limitations of participatory process theory; Christopher Luederitz for helping me think through MLP and evaluation; Pani for her thesis insight and helping me understand how to use theory; Johnnie Manson for his elaboration of mid-range theory and thoughts on ontology and epistemology; Dimitry Anastakis for helping me think through my dissertation structure and outline; Stine Bigum Pedersen for help with refining my Nvivo coding process; Lori Hewson for always finding ways to put theory into practice; and Mark Cabaj for generously sharing his experience with evaluation and thinking through ways to capture systems change. Key elements of this dissertation were shaped by feedback from participants in Evaluating Complex Change workshops in Edmonton and Vancouver along with comments from conference presentations in Gothenburg, Montreal,  xxiii Edinburgh, Toronto, Vancouver, Sydney, and Potsdam. The Energy Futures Lab Fellows and the EFL design team were incredibly generous with their time, challenging with their comments, and were crucial in my ability to complete this research project.  Members of my Supervisory Committee provided many thoughtful and challenging comments, in particular a discussion on evidence, bias, and assessment. I would specifically like to thank my supervisor John Robinson who continually pushed me to answer the question ‘So what?’ His comments, suggestions, introductions to colleagues, and ideas have made me a better scholar and made this dissertation much stronger. I would like to thank my dear friends Kristi Joba and Dale Littlejohn who provided me with space to write by the ocean on the Sunshine Coast, Kim at Prison Café in Stege for the flat whites while writing, Mette Lea Mortensen for continual encouragement, support and friendship from near and far, and JD Derbyshire for friendship and always being on the other end of the phone when I’m freaking out. My work and research have taken place on the unceded traditional territory of the Musqueam, Squamish and Tsleil-Waututh First Nations. I have also been honoured to be a guest in the territories of the Lubicon Cree, Athabasca Chipewyan, Stoney Nakoda, Blood Tribe, Kanai, and Métis Nations. I am grateful for the opportunity to be a guest on this land and am mindful of the privileges that come along with being a settler on unceded land. Finally, I would like to express my incredible appreciation for the love and support of my family. Chris, Tania, Roman & Lily have hosted me many times in Toronto and I am so grateful for all they have done for me in the process of becoming ‘Dr. Steve’. And last, but not least, Mum and Dad. I cannot count the ways they have supported and encouraged me in my work, my research, and in all areas of my life. I could not have done this without them. 1 Chapter 1: Introduction  “Generally speaking, whether something is logical or isn’t, what’s meaningful about it are the effects. Effects are there for anyone to see, and can have a real influence.  But pinpointing the cause that produced the effect isn’t easy.  It’s even harder to show people something concrete that caused it, in a ‘Look, see?’ kind of way.  Of course there is a cause somewhere. Can’t be an effect without a cause. You can’t make an omelet without breaking some eggs. Like falling dominoes, one domino (cause) knocks over the adjacent domino (cause), which then knocks over the domino (cause) next to it.  As this sequence continues on and on, you no longer know what was the original cause.  Maybe it doesn’t matter. Or people don’t care to know.  And the story comes down to ‘What happened was, a lot of dominoes fell over’.  The story I’ll be telling here may very well follow a similar route.” Haruki Murakami, Killing Commendatore, p. 58.  1.1 Goals and research approach Fostering sustainability transition is a pressing global societal challenge. Rising GHG emissions, severe social and economic inequality, unsustainable production and consumption activities, and global environmental degradation and its human consequences are all indicators of an unsustainable world. In the words of the recently released Global Environmental Outlook 6 report from the United Nations Environment Program: Unsustainable production and consumption patterns and trends as well as inequality, combined with population growth-driven increase in resource use, put at risk the healthy planet needed to attain sustainable development. These trends are deteriorating planetary health at unprecedented rates with increasingly serious consequences especially for poorer people and regions (UNEP, 2019, p. 2).  These societal challenges are beyond the capacity of any single government, business, or non-governmental organization to address. They cross geographical and jurisdictional boundaries and  2 impact at multiple levels of scale ranging from individuals to the planet itself. Adding to the complexity of these challenges is wildly varying degrees of agreement on what the challenges are (or even if challenges like climate change exist) and on potential solutions. Humanity has experienced many societal transitions throughout history such as the transition from hunter-gatherer to agrarian societies or the transition from a biomass to a coal-based energy system. A key difference today is the desire to collectively guide transition in a more sustainable direction. This will require forms of innovation, collaboration, and governance that are very different to those in place today. Sustainability Transition Experiments (STEs) leveraging a transdisciplinary research approach have recently been proposed as a method to accelerate sustainability transitions (Wiek et al, 2014; Schapke et al. 2018) in response to these global challenges. However, there is a presumption in the literature and by practitioners that STEs will lead, as a result of societal effects, to sustainability transition. We need methods by which to evaluate this claim, to understand what is happening within transition processes, and to provide insight to STE designers and facilitators as to the efficacy of their processes. The need for frameworks and methods to evaluate STEs led to the development of my research questions: o How can we conceptualize, and evaluate, the contributions to sustainability transitions of STEs? o Was the Energy Futures Lab (my primary case study) effective in supporting sustainability transitions in Alberta? With a goal of evaluating transdisciplinary sustainability transition experiments, this thesis develops an evaluative framework through engagement with multiple literatures and a series of  3 transdisciplinary research case studies. Evaluating transdisciplinary research processes is a necessary component of the evaluation framework as a key element of STEs is engaging stakeholders and how that is done matters. In addition, several literatures point to the link between process and outcomes, i.e. the design and execution of a process influences the kinds and qualities of outcomes expected. Evaluating societal effects captures the short-term outputs and medium-term outcomes of an STE that, presumably, will lead to sustainability transition impacts. However, these societal effects do not capture such sustainability transition impacts. Assessing governance and power relationships, actor roles and responsibilities, and changes in social practice through the lens of development pathways allow us to do just that. The integration of these elements leads to my proposed three-part evaluation framework of process, societal effects, and sustainability transition impacts. I propose a development pathway approach to organize elements of sustainability transition impact into a coherent framework that highlights the inter-relationships between levels of scales in systems transition and foregrounds the role of changes in governance roles and relationships and the role of politics in transitions. This thesis extracts the key insights from each literature generating a set of indicators to be used in assessing sustainability transition experiments. I then provide more detail on the process by which I operationalize the framework with a set of detailed indicators used in assessment of my primary case study - the Energy Futures Lab in Alberta, Canada. This thesis provides insights into the challenge of evaluating the sustainability transition impacts of sustainability transition experiments and provides a crucial bridge between the evaluation of processes, societal effects and their link to sustainability transition impacts.  4 1.2 Case study: The Energy Futures Lab (EFL) The EFL is a multi-stakeholder process in Alberta, Canada comprised of participants from across the energy system who participated in a collaborative 5-year leadership development and rapid prototyping program designed to answer the question “how can Alberta’s leadership position in today’s energy system serve as a platform for the energy system the future requires of us?” (EFL, 2018). The EFL consisted of over 60 energy system leaders (known as Fellows) coming from oil and gas companies, renewable energy firms, municipal, provincial and federal government agencies, academics, First Nations and NGOs. Convening partners of the EFL were the Natural Step Canada (TNSC), the Suncor Energy Foundation (SEF) (funded by one of Canada’s largest oil companies), the Pembina Institute (an environmental NGO), the Banff Centre (leadership and development organization), and the Government of Alberta. The majority of the funding for EFL came from the Suncor Energy Foundation (see Table 30 for a complete list of EFL funders). The EFL was designed with three phases (in addition to a pre-lab planning phase) (see Figure 1): 1. Commit to collaborating and begin to co-create a new, shared narrative about Alberta’s energy system. 2. Map the desired transition in the system, prioritize collaborative projects, and engage stakeholders beyond the lab. 3. Co-ordinate action across a broad range of organizations, which in turn begins to shift the public narrative about energy issues. (Energy Futures Lab, 2018b)  5   Figure 1: EFL phases. Source: https://energyfutureslab.com/about-the-efl/ This design framework emerged from TNSC’s experience with backcasting (Holmberg & Robert, 2000) and their desire to apply their sustainability principles along with design and facilitation techniques to systems transitions challenges. In practice, the EFL diverged slightly from Holmberg and Robert’s conception of backcasting for a number of reasons. First, the widely diverse membership required a lot of time to build mutual trust. Oil companies, environmentalists and government officials were initially quite skeptical of the collaboration. Second, these different stakeholders had very different view of the current state of the energy system in Alberta and the challenges it faced. Partly this was due to ideological differences but also due to the complexity of the system itself. For example, electricity generation and distribution uses different industry and regulatory structures than oil & gas extraction, processing and transportation which is different again from the renewable energy industry. These two design challenges led to additional time being spent on system sensing.  6 Early Design Team discussions questioned whether the role of the EFL was to shift the system (a transitions approach) or to disrupt the system with innovations. The Design Team decided that the “goal is to disrupt the system in a way that accelerates the transition to a sustainable and resilient energy system [original emphasis]” (Energy Futures Lab, 2015a). The EFL set out to achieve this goal through three streams of activity. The first was a cohort of Fellows that met for 2-3 day long workshops 3-4 times per year. These workshops were organized around different themes such as prototyping, backcasting, and initiative development. Through the workshops (see Appendix D for details on each workshop), EFL participants were engaged in learning activities about sustainability and systems change through different lenses including the multi-level perspective of transition theory, the Natural Step’s Framework for Strategic Sustainable Development (FSSD) (Broman & Robèrt, 2017), innovation processes, systems thinking, strategic foresight, the role of narratives and a number of facilitation and process methods such as open space (used to generate and explore new ideas) and fishbowl conversations (used to facilitate emotionally charged conversations in a safe space). This learning has been facilitated by lecture style presentations, hands on workshop activities and interactive games. The second stream was an organizational engagement process that conducted workshops inside participant organizations such as Suncor and the National Energy Board delivering customized versions of EFL content. The final stream was public engagement that connects EFL to the public through media, events and public workshops.  The EFL was designed and facilitated by a core design team (see 5.2.2.1 for further details). The design team was supported by an Advisory Council and Steering Committee. The Steering Committee was a group of senior influencers in the Alberta energy system and served a largely hands-off role. The Advisory Council (see Appendix B) was smaller group of energy system leaders and played a more hands-on role. The Advisory Council provided feedback to the Design  7 Team on EFL design and received regular updates from the team on progress. Partners also played a role in shaping the EFL design (see 5.2.1.3 for further detail on EFL partners). 1.3 EFL 2.0 and beyond The first phase of EFL (“EFL 1.0”) funding from SEF expired June 30, 2018. The EFL was able secure bridge funding from the SEF from June – December 2018 while the team, in collaboration with the Fellows and Partners, designed the next phase of EFL – “EFL 2.0”. EFL 2.0, which officially kicked off with a workshop in Edmonton in March 2019, retains the Vision and general approach of EFL 1.0 with a refined set of objectives. The objectives of EFL 2.0 are: 1. Working with innovators to continue to identify, develop and pressure-test innovative solutions for accelerating the transition to the energy system the future requires of us; 2. Strengthening the capacity of business and government to adopt and support such innovative solutions; 3. Aligning a growing community of leaders and their organizations behind a bold and transformative vision for the future of energy in Canada and the possible transition pathways to achieve it; 4. Inspiring and sharing narratives among communities that help depolarize the public conversation about energy in Canada and accelerate progress toward a shared vision. 5. Staying open and responsive to emergent opportunities to leverage EFL tools, brand and lessons to extend its impact to jurisdictions beyond Alberta. (Energy Futures Lab, 2018c, p. 4) The study period for this thesis is from Summer 2015 to December 2018 and data collection for this thesis concluded in November 2018. Therefore, I do not include detailed analysis of EFL 2.0 or a comparison of EFL 1.0 and 2.0. However, many design changes emerged as a result of feedback from EFL participants and partners that illustrate the reflexivity and adaptive nature of the EFL. For example, many EFL 1.0 Fellows reported that attending a 3-day workshop was difficult in the middle of a work-week. In response, EFL 2.0 has shorter workshops (1.5 days).  8 This will make it easier for Fellows to attend but has the consequence of reduced time for formal and informal networking which was reported as one of the biggest values by Fellows. Where relevant to the discussion, I will briefly mention how EFL 2.0 has responded to challenges or shifted direction from EFL 1.0. Future research will track the evolution of EFL 2.0 and its effects and impacts. 1.4 Action research and role of researcher in evaluating transition In contrast to a traditional evaluative process, I am taking an active role in the design and delivery of the Energy Futures Lab in the tradition of Action Research. An Action Research approach has recently become more common in transitions research (Asquith et al., 2018; Fazey et al., 2017; Steward, 2018; Wittmayer & Schapke, 2014) and presents both opportunities and challenges for researchers. The EFL is the focus of my PhD thesis research and I have been involved with the project since its inception. As a consultant, I contributed to the original project proposal to the Suncor Energy Foundation in 2014 and participated in planning and design meetings. As a researcher, I facilitated portions of EFL workshops, and reported interim findings from my research to both the EFL Design Team and EFL Fellows. I introduced the MLP framework to the EFL Design Team in early planning meetings in 2014 and 2015 and have continued to share my observations, findings, and recommendations with the Design Team and with Fellows. This engagement puts me in an interesting position as a researcher. My theoretical background in systems transition theory and evaluation of participatory processes has led me to contribute knowledge from the literature to Design Team discussions and advocate for the adoption of the recommendations from MLP described above as I believe that this approach to systems analysis is most fruitful for organizations aiming for systems transition. I have also provided early insights from my own interview synthesis to the EFL design team which has been used to inform strategy.  9 In other words, I am not a disinterested observer of the process and am in some ways assessing my own activities. I believe this to be an unavoidable situation, especially when a researcher is embedded in a project team throughout the life of the project. As Fazey et al. note, “researchers are inevitably embedded within, and not separate from, the systems they seek to observe” (2017, p. 56). As I move through the data collection and analysis phases of my research, awareness and recognition of my own place in the work must remain both clear and transparent. For transparency, I have included in my discussion points where I made direct interjections into the design process. There are two important points of clarification to make here about conflict, influence, and ethics. First is my relationship with research “subjects”, i.e. the EFL Design Team. In Fall 2016, I presented early findings from my research at a seminar at the University of Toronto. Members of the EFL Design Team who subsequently saw the presentation were not pleased as they felt I had misrepresented their intent and activities. Upon further review, this was in fact the case. I was not privy to conversations and meetings that were crucial to my interpretation of the EFL’s activities. In response, we came to an agreement where I would share my work with the EFL Design Team in advance of presentation or publication for review, comment, and clarification. I have continued to do so throughout my research including providing draft copies of thesis chapters for review.  However, no members of the EFL Design Team have provided comments or critiques on my research findings nor have they made suggestions or requests to change the content of this dissertation. Beyond this, I feel that in an action research approach, based on principles of co-production, research subjects should have the opportunity to contribute to the research itself from within. Quoting again from Fazey et al.,   10 conceptualising research as being from within enables the goal to focus on social improvement as opposed to primarily knowledge production which dominates research that is viewed as being conducted from the outside (2017, p. 62). The second point is that my consulting firm – Constructive Public Engagement, Inc. – has been engaged to conduct developmental and summative evaluation for EFL 2.0. This contract was executed in April 2019, after my research data had been collected and the study period ended. An advantage of this engagement with EFL 2.0 is access to continuing data. EFL 2.0 is projected to run from January 2019 – December 2021. By that time, I will have 7 ½ years of cumulative data which will prove invaluable in assessing sustainability transition impacts. The disadvantage, of course, is the potential conflict inherent in having a financial relationship with my research subjects. Explicit in conversations with the EFL was an agreement that research findings – positive or negative – would not influence the contractual relationship.  In Section 4.1.3, I provide an extended discussion of these issues in which I investigate my own normative stance, values, beliefs, assumptions, and role within the EFL. I provide a careful exposition of how I allowed for potential bias, and then describe how I am explicit in the text when I reach a judgement on a given indicator, that my judgement is informed explicitly by my theoretical framework and the evidence I cite (e.g. participant comments, external evidence). I also clarify that while I inevitably bring my own conceptions of sustainability transition to my analysis, I assess the EFL using its own framing of sustainability.  1.5 Thesis structure Chapter 2 of this thesis introduces the theoretical background and rationale for an innovative evaluation framework in three levels: process, societal effects, and sustainability transition impacts. I also introduce the concept of development pathways to characterize sustainability  11 transition impacts which are one of the most challenging sets of impacts to measure. Chapter 3 traces the evolution of the evaluation framework through a series of case studies where the framework was tested in practice and operationalizes the framework with a detailed set of indicators. These case studies from projects around the world illustrate how the framework can be applied in practice, limitations in application, and how learning from each case has influenced the development of the framework. Chapter 4 describes research methods and includes a detailed discussion on evidence, bias, and positionality in my research process. This is an important discussion as the position of ‘reflexive researcher’ who is embedded in an STE is by no means a neutral one. Elaborating on how I have responded to this issue is crucial in order for readers to interpret my findings. Chapters 5, 6, and 7 assess the Energy Futures Lab through the three levels of my evaluative framework. Chapter 5 assesses the EFL process, design, and implementation. While ensuring that STEs are inclusive, representative and fair is important in and of itself, this chapter also assesses the degree to which the EFL process lays the foundation for achieving societal effects and transition impacts in a more sustainable direction. Chapter 6 assesses the societal effects of the EFL and provides a summary of how the EFL has fostered individual learning and development, creation of tangible outputs, and enhanced networks and relationships. The chapter also assesses the effects of the EFL on institutional change in policy and organizational decision making. Chapter 7 traces the sustainability impact of the EFL through the lens of development pathways. The Alberta energy system has been in a state of turbulence for the lifetime of the Lab and this chapter attempts to tease out the relationship between EFL actions and changes in the Alberta energy system. Within each chapter I also briefly address where elements are co-occurring and how, for example, process elements support societal effects and how effects support impacts. In sum, this dissertation represents an innovative theoretical exploration resulting in an  12 evaluation framework that has been applied in case studies around the world. The framework, as applied to the Energy Futures Lab, is a demonstration of what is possible. STEs designers, facilitators, funders, and researchers will all benefit from the concepts, theoretical background, and methods embedded in this novel evaluation framework as they continue their work in fostering sustainability transition.   13 Chapter 2: Theorizing Sustainability Transition Experiment Evaluation 2.1 Sustainability transition experiments1 There have long been collaborative projects between citizens and local governments designed to foster sustainability transitions but there are now emerging new types of transition experiments that are “characterized by cross-organizational collaboration between actors from academia and society (government, industry and citizenry) with the aim of collaboratively fostering transformational change and progress towards greater sustainability” (Luederitz et al., 2016, p. 1). Transition experiments may be focused on low-carbon (Rosenbloom et al. 2018), energy (Grübler, 2012), climate governance (Turnheim et al., 2018), or the urban context (Schapke et al., 2018a). However, as Cherp et al. point out, these domains are not mutually exclusive and there is substantial overlap between them: “Low-carbon transitions may occur outside of the energy sector (e.g. in urban planning, industry, agriculture and forestry). ‘Sustainability’ transitions may also include changes in food systems, distribution of wealth, human rights, governance and conflicts” (2018, p. 176).  'Sustainability transition experiments’ (STEs) can be used as a term that encompasses transdisciplinary, collaborative, multi-stakeholder interventions that focus on transforming a specific issue such as food production, child poverty or energy, and may include living labs,  1 The following section is adapted from Williams, S. (2017, June). Evaluating Transition Experiments in Times of Rapid Change. Paper presented at International Sustainability Transitions 2017. Chalmers University, Gothenburg, Sweden.  14 transition labs and social innovation labs (Schapke et al., 2018a) and I will use that term to describe the case of the Energy Futures Lab throughout my thesis. Common among the literature describing STEs is the premise that a new way of working together is required. As Loorbach et al. note, “transitions are complex and unstructured processes of change, a process of learning-by-doing and doing-by-learning is the only way to adapt, change, and transform existing dominant cultures, structures, and practices” (2017, p. 614). STEs need to develop a common vision of a desirable future, understand the present system, identify gaps between the present and the desirable future and opportunities for intervention, and find ways of scaling interventions to achieve systems change. This implies a need for experimentation and prototyping (Pajouhesh, 2016), multi-stakeholder convening (Loorbach & Rotmans, 2010), visioning processes (Holmberg & Larsson, 2018), and a focus on co-production processes that facilitate social learning (Schapke et al., 2017). In this chapter I explore theoretical perspectives on evaluating sustainability transition experiments and their contribution to sustainability transitions. I begin by reviewing literature on evaluation then turn to literatures of transitions theory, environmental justice, social practice, transdisciplinary co-production, transformations and social learning. As described more fully in the balance of this chapter, all of these literatures contribute elements to better understand the impacts of sustainability transition experiments. The chapter also extracts the key insights from each literature generating a set of indicators to be used in assessing sustainability transition experiments. Chapter 3 complements this theoretical discussion with a set of case studies where a preliminary evaluation framework is applied in practice, then describes in more detail the process by which I operationalize the framework with a set of detailed indicators used in my assessment of the Energy Futures Lab  15 2.1.1 Challenges evaluating sustainability transition experiments Evaluating sustainability transition experiments poses several challenges beyond those of evaluating sustainability projects such as public awareness campaigns, home retrofit programs, or deployment of renewable energy technologies. As numerous authors have pointed out, transitions are inherently boundary-spanning and affect multiple domains (e.g. social, political, cultural and technical) (Hölscher et al., 2017).  Developing tools and methods to capture change across such wide-ranging domains is difficult both conceptually and practically. In addition, transitions occur within complex systems which implies interdependence between system elements, emergent phenomena that cannot be predicted a priori, and discontinuous or non-linear effects of systems interventions (Andersson, 2014; Rotmans & Loorbach, 2009; Westley et al., 2011; Yukawa, 2015). Arnold et al. note, “Complexity introduces a big problem for theory-based intervention design and evaluation: since the system involved changes its characteristics over time, the intervention logic and its components do not necessarily remain stable” (2018, p. 17). Not only is the system constantly changing, which poses challenges of attribution, system transitions (especially energy system transitions) tend to take place over long time periods (Grübler, 1998; Schot & Kanger, 2018; Smil, 2010). Capturing the long-term contribution of an intervention that runs for a period of months or even years to system transition is very difficult; even more so given that, due to finite resources, evaluations may only focus on a sub-set of the project lifecycle. Finally, there is a tension between the need for evaluation that generates learning for those designing and managing sustainability transition experiments and accountability for ‘results’ driven by funders and governments. A finding of “’we had a good process and learnt things’ does little to address the need for accountability in government interventions.” (Arnold et al., 2018 p. 39). This tension is especially acute for low-carbon and sustainability transition experiments that have a specific goal  16 of, for example, reducing GHG emissions in a given sector, geographic area, or target population (Rosenbloom et al., 2018; Potvin et al., 2018). Despite these challenges, it is crucial to evaluate STEs for a number of reasons. Fostering sustainability transition is a pressing global societal challenge. STEs have been more and more commonly proposed as a method to accelerate such sustainability transitions. However, there is a presumption in the literature and by practitioners that STEs will lead, as a result of societal effects, to sustainability transition. We need methods by which to evaluate this presumption, to understand what is happening within transition processes, and to provide insight to STE designers and facilitators as to the efficacy of their processes. Although transitions take time, are boundary-spanning, and take place within complex systems, transitions work is happening now – and needs to happen now – and evaluation of STEs can support and guide this work. Note that while STEs are typically focused on transition within a single domain such as food, energy or transportation, these domains are connected and interdependent reflecting the complex and unpredictable nature of transitions. Robinson (Personal communication, February 23, 2019) argues that there are in fact many possible transition pathways, many possible transitions, and many possible sustainabilities. However, the EFL’s convening question “How can Alberta’s leadership position in today’s energy system serve as a platform for transitioning to the energy system the future needs?” (EFL, 2019a) implies a pathway to a transition. As the process unfolded, the EFL introduced multiple “Innovation Pathways” and attempted to reconcile different conceptions of sustainabilities within the EFL Vision. 2.1.2 Research approach and questions These challenges lead to my research questions:  17 1. How can we conceptualize, and evaluate, the contributions to sustainability transitions of STEs? 2. Was the EFL effective in supporting sustainability transitions in Alberta? In order to answer these questions, I propose to build a framework for evaluating sustainability transition experiments and apply it to the EFL process. In this chapter, I start by reviewing the extant literature on evaluation. This literature has much to say on the evaluation of processes themselves as well as outputs and outcomes from these processes. However, the evaluation literature does not capture sustainability transition impacts – i.e. whether and how these experiments are contributing to a sustainability transition. I then turn to the transitions literature to explore how that literature conceptualizes transitions. This provides further guidance but, as Schapke et al. note, “there is a lack of understanding as to how the core societal effects of transition management are related to sustainability as well as the lack of a framework from the field of transition studies to assess this” (2017, p. 3). To address this gap, I turn to literatures of environmental justice, social practice, transdisciplinary co-production, transformations and social learning. In the following chapter I translate these theoretical foundations into an operationalized evaluation framework. Translating theory to practical application is a difficult task and I will further elaborate on the steps I took to make this leap. 2.2 Review of evaluation literatures  2.2.1 Theoretical framings of evaluating sustainability transition experiments Experimentation implies an effort to assess the impact of an initiative by means of a structured comparison. Traditional experimental thinking understands that comparison is to a so-called baseline or some other control group that functions as a counterfactual. In the language of  18 experimentation, the counterfactual is used to tell us what would have happened otherwise. However, given the complexity of multiple social-ecological-political-economic systems that may be at play, such quantitative evaluation is not always possible (Andersson, 2014; Rotmans & Loorbach, 2009). Evaluating STEs leads to a tension between the desire for the ‘gold standard’ of experimentation, as in randomised control trials, and the complex real-world systems of transition experiments where such controlled settings are never possible (Park, 2017). Instead, evaluating transition experiments will necessitate working from best available data while recognizing that this data may be incomplete or inconclusive. An example of this approach is contribution analysis (Mayne, 2001; 2012). Respecting that attributing direct effects on complex systems from a given intervention is challenging, if not impossible, contribution analysis “builds detailed theories of change that are then tested empirically to show whether interventions contributed to changes” (Arnold et al., 2018, p. 14). Sengers et al. illustrate the difference between approaches saying  Natural science experiments can broadly be interpreted as a practice that take place in the confines of a laboratory or an otherwise strictly controlled environment as a way to find hard objective truths about material reality. Sociotechnical experimentation, on the other hand, implies a more engaged and social constructivist position: society is itself a laboratory and a variety of real-world actors commit to the messy experimental processes tied up with the introduction of alternative technologies and practices in order to purposively re-shape social and material realities (2016, pp. 1-2). Even without a baseline and randomised replications, a key element of evaluation is to have clear answers to the questions: What worked for whom? In what circumstances? Why? What did not work for whom? In what circumstances? Why?  The “messy experimental processes” (Sengers et al. 2016, p. 2) of STEs imply a need to include evaluation of the process itself, i.e. how it was designed and implemented. STEs are designed to produce results therefore we must assess the short and medium term effects of STE processes. Finally, inherent in STEs (which differ from other stakeholder engagement processes) is an explicit  19 aim to foster sustainability transition. We must also then assess the sustainability transition impacts of a given STE. In the following sections, I review a series of literatures that speak to each of these three evaluation levels.  2.2.2 Process evaluation Transition experiments are participatory by nature. Labs, transition arenas, low-carbon, and sustainability transition experiments all have elements where multiple stakeholders are engaged in transition processes (Loorbach & Rotmans, 2010; Rosenbloom & Meadowcroft, 2017; Schapke, et al., 2018b; Westley et al., 2011). We can look then to the literature on evaluation of participatory processes for insights into evaluating sustainability transition experiments. These evaluations tend to focus on the design and procedural elements of the process itself or on the outcomes of the process. Procedural evaluation looks at whether processes are inclusive, fair, and present unbiased information (Abelson et al., 2003; Black, Burkhalter, Gastil, & Stromer-Galley, 2008; Carpini, Cook, & Jacobs, 2004). Rowe and Frewer (2005), following Abelson et al. (2003), propose a framework of fairness and competence/efficiency to measure the effectiveness of public engagement processes. Fairness is a measure of perception - do participants (and the wider public) believe the exercise was designed and conducted in a fair and representative manner. However, Rowe and Frewer extend the definition of competence to include how efficient the information flow is and how well that information is processed (i.e. into policy or other outcomes and objectives).  There has been much work already done on the evaluation of transdisciplinary co-production processes (collaborations between academic researchers, government, business, and civil society), for example understandings of co-production and the roles of participants, along with  20 recommendations for successful projects (Hansson & Polk, 2017; Cvitanovic et al. 2016, Fazey et al. 2014). This literature points to the need for co-production processes to be iterative, interactive and reflexive, provide transparent discourse and collaboration, and embed broad and diverse participation and engagement.2 There is an important link  between diversity and reflexivity. While diversity is commonly measured through demographic traits and/or representativeness of a given population, also important is a diversity of discourses or ‘discursive representation’ (Dryzek & Niemeyer, 2008). This concept asks to what extent a given process is working within a single discourse or includes multiple discourses. Discursive reflexivity is a measure of how an STE questions fundamental dynamics of the broader system. Note that in addition to process evaluation, co-production approaches also have implications for societal effects evaluation. I address these in more detail in Section 2.3.3 This review of the evaluation literature implies the need to assess the number, type and participation rates of STEs along with the fairness, inclusivity and breadth of participation, transparency of processes.3 Co-production literatures suggest also assessing whether processes are iterative, interactive, and reflexive. Table 1 below summarizes key elements of process evaluation.   2 This section adapted from Williams, S. (2017). Evaluating Societal Effects of Transdisciplinary Co-production Processes: Final Report. Mistra Urban Futures. Gothenburg, Sweden. 3 Content in this section adapted from Adapted from Robinson, J., Slater, K., Williams, S., Pajouhesh, P., and Benzakin, G. (2018). TAF Evaluation Framework Outline.  21 Table 1: Process evaluation elements Literature Process Participatory Process Number, type and participation rates of engagement processes Fairness Inclusivity Breadth of participation Scope Transparency of processes Co-Production Iterative Interactive Reflexive Transparent  While the goal of STEs is sustainability transition, process evaluation is important in and of itself. Fiorino (1990) proposes three rationales for engaging publics in processes such as STEs. The first is normative – that publics have a right to be consulted on, and participate in, decisions that affect them. The second is substantive – that publics have knowledge and understanding of issues that are not available to experts and policy makers. Therefore, if publics are engaged in the decision making process, the result of the decision are likely to be better. The third rationale is instrumental – that publics are more likely to be supportive of the results of a decision if they have been part of the process. Dusyck argues that there is value in the generation of “new, potentially more robust, forms of knowledge” (2013, p. 41) – what she calls an “epistemic” benefit - through participatory processes in addition to merely trying to educate the public. Finally, Kearnes & Chilvers suggest that “practices of public participation actively produce publics, public issues, material commitments and forms of democratic engagement” (Kearnes & Chilvers, 2016, p. 312), similar to what Dusyck calls a “constitutive benefit”. Collectively, these authors highlight the importance of engagement processes. It is therefore critical that we have a robust framework through which to evaluate these processes.  22 2.2.3 Societal effects evaluation4 Wiek et al. (2014), Cornish (2013), Robinson (2006) and others have extended the literature on the impact of participatory sustainability projects  to encompass the societal effects of such projects (Blackstock et al., 2007; Robinson et al., 2011; Talwar et al., 2011; Walter et al., 2007). This framework categorizes societal effects into first-order effects which are direct outcomes and outputs of a project (the short term “splash” from a specific event or process) such as enhanced capacity, networks and usable products (e.g. action plans, reports, web sites, new technologies) and second-order effects (“the ripples” which are consequences of first order effects) such as structural changes (e.g. new policies, organizational changes), decisions and actions. Note that these effects categories are interdependent. For example, an intervention such as the EFL might produce a new low carbon oil extraction technology (usable products) while teaching a group of engineers within the oil sands industry (network) how to use the technology (enhanced capacity). Through experimentation and pilot testing of the technology, the EFL might also have direct effects on water and energy consumption (solution implemented). The use of the technology might then be required by regulation (socio-economic effect) and spur the creation of new departments inside oil sands companies to further refine and implement the technology (organizational effects). Wiek et al.’s approach broadened the categories of effect from previous work in the field by including structural changes as evaluative criteria. The framework acknowledges the challenges of attributing effects due to time delays between processes of deliberation or equivalent events that  4 This section is adapted from Williams, S. (2017, June). Evaluating Transition Experiments in Times of Rapid Change. Paper presented at International Sustainability Transitions 2017. Chalmers University, Gothenburg, Sweden.  23 have occurred and their effect. The framework introduces the concepts of 1st and 2nd order effects. 1st order effects are those that occur as a direct result of a given project. 2nd order effects occur as a result of 1st order effects and are therefore further removed temporally from the original project. In addition, making causal attributions between a given project and 2nd order effects becomes more challenging.  While the 1st and 2nd order effects framing is very useful, the definitions and indicators of 2nd order structural effects are not precise, for example conflating societal shifts such as norms and behaviour change with policy and institutional effects. These are very different types of effects therefore analysis would have more clarity by separating these effects. I have done so by including policy and organizational institutional effects (e.g. policy changes, changes in organizations) as societal effects. Structural changes (i.e. broad economic shifts from manufacturing to services) are indicators of development path change and are addressed as sustainability transition impacts. While measures of participatory process effectiveness are quite well established (see Section 2.3.2), outcome measures are much less developed in the participatory process literature. Outcome evaluation looks at both impact of the process on participants (e.g. in terms of increased knowledge or level of civic participation), on the products of the process (e.g. reports) and the use of those products by decision makers (Barrett et al., 2012; Caddy, 2005; Gaventa & Barrett, 2010; Mutz, 2008; Pincock, 2012; Ryfe, 2005). A commonality of this literature is that it comes from a governance perspective asking how citizen engagement will link to policy and decision making (Darcy Riddell, Personal Communication, April 12, 2017). Arnstein (1969) developed the concept of a “ladder of engagement” ranging from manipulation by decision makers through consultation to citizen power. This typology implies a link between process design and possible effects (e.g. affecting policy decisions). Sustainability transition experiments are often started outside of  24 government and may seek to disrupt, rather than engage with, existing governance methods aim to take place high on this ladder of engagement (because they want to have effects).  Schapke et al. (2017) discuss the importance of social learning effects in transition processes and the literature on co-production of knowledge provides insights on how to capture these learning effects (I address social learning literature as an element of sustainability transition impact in section 2.4.2). As Walter et al. (2007) point out, there is a strong correlation between outcomes and “involvement as measured by the number of engagement activities that took place during the project” however more research needs to be done in order to draw such clear links between the process elements of a given experiment and outcomes.  2.2.3.1 Implications for evaluation In summary, societal effects evaluation literatures suggest that assessing effects of STEs involves direct effects (1st order) such as tangible products of the experiment, enhanced individual capacities, deepened or strengthened networks. These direct effects may lead to 2nd order effects such as influence on changes in policy, and organizational decision making and actions. Note that policy and organizational change may be 1st order (i.e. direct) effects depending on the activities and scope of an STE. Alternatively, they may occur as a result of 1st order effects. Societal effect evaluation should also assess the contribution to improvements in the given focus area of an STE (e.g. carbon reduction or strengthened socio-ecological integrity).5 In addition, effects of participatory processes can be assessed as to how results of the engagement are integrated into  5 Content in this section adapted from Adapted from Robinson, J., Slater, K., Williams, S., Pajouhesh, P., and Benzakin, G. (2018). TAF Evaluation Framework Outline.  25 policy or other outcomes, and social learning effects such as relationship/trust building, and the development of problem solving, complex systems thinking, dialogue and reflexivity capacities in participants. Co-production processes may generate new forms of knowledge, values, and social relations and possibly changes in roles and relationships (e.g. challenging the distinction between researcher and practitioner). While this section has been focused on describing elements of societal effects evaluation, through exploration of the literature, process evaluation elements also emerge. For example, social learning emphasizes the importance of shared inquiry, dialogue, and negotiation that deepen understanding of different points of view, and enable collective problem solving, double-loop learning (Argyris & Schon 1978), action and reflection over time, inclusivity and engagement with diverse perspectives, which afford opportunities for assumptions to be challenged. These process elements are echoed by the co-production literature which also highlights the value of inclusivity, transparency, reflexivity, and the iterative nature of transdisciplinary research processes. Societal effects evaluation gives us important information on the short term outputs and medium term impacts of sustainability transition experiments. However, these approaches (e.g. Wiek et al. 2014 and Leuderitz et al. 2016) do not explicitly address the issue of societal transition because they assume these impacts will happen as a result of societal effects. For example, Hansson & Polk note that “transdisciplinary co-production approaches are built on the assumption that the intermediate or direct effects of participatory research contribute indirectly to transformational societal change” (2017, p. 134). In the same paper, the authors conclude that  there are no clear mechanisms that link participatory features to impact; there is instead a complex web of relationships, institutional cultures, and political agendas that require that we open up the categories to see how they are conceived of by different actors internal as well as external to the project (p.141).  26 The literature on societal effects is also vague on how ‘transformational societal change’ is to be conceptualized, not to mention how to assess the contribution of an STE to such change. In the following section, I address these gaps by exploring the sustainability transitions theory literature which purports to theorize the process by which sustainability transitions emerge and how such transitions can be fostered or accelerated through processes such as STEs. 2.3 Sustainability transition impacts  In order to evaluate sustainability transition impacts, I turn first to the transitions literature which is explicitly formed around the question of how societal transitions take place. However, as we will see, additional literatures are required to broaden the perspective of transitions theory and address gaps in, and critiques of, the theory. I then propose the concept of development pathways as elaborated in Burch et al. (2014) as an organizing framework through which to categorize and illustrate relationships between elements of sustainability transition impacts I identify in my literature review. In addition, although I intended to use these literatures to identify sustainability transition impacts, I found implications for evaluating both process and societal effects which I highlight in the sections below. 2.3.1 Transition theory and the Multi-Level Perspective Socio-technical approaches characterize transitions as major, long-term technological changes in the way societal functions are fulfilled. Technological transitions do not only involve changes in technology, but also changes in user practices, regulation, industrial networks, infrastructure, and symbolic meaning or culture (Geels & Schot, 2002, p. 1257). A commonality of socio-technical approaches is a focus on the inter-connected nature of socio-technical change. As Wilson & Grubler note, “technologies and their institutional settings co- 27 evolve. Change in these different areas is mutually dependent, mutually enhancing and mutually dampening” (2011, p. 165). The dominant conceptual framing in the transitions literature is the multi-level perspective (MLP) which describes the inter-relationships between the levels of niche, regime, and landscape (Berkhout et al., 2004; Geels, 2002; 2011; Smith et al., 2010). Niches are protected spaces where radical innovations are generated “by small networks of dedicated actors, often outsiders or fringe actors” (Geels & Schot, 2007, p. 477). Regimes are “a shared set of rules and routines embedded in socio-technical systems to ensure that they can provide the relevant social function” (Cherp et al., 2018, p. 180). Regimes include the set of dominant actors, processes and routines embodied in “industrial networks, techno-scientific knowledge, culture, sectoral policy, infrastructure, markets, user practices, and technology” (Geels, 2002). Regimes are not monolithic and when examining transition processes, it is important to consider inter-regime interactions and interdependencies. For example, “battery-electric vehicles link transport and electricity systems” (Geels, 2011, p. 40). Change (or resistance to change) in one of these linked regimes may have amplifying or dampening effects on the other. Landscapes consist of “heterogeneous factors such as oil prices, economic growth, wars, emigration, broad political coalitions, cultural and normative values, [and] environmental problems” (Geels, 2002, p. 1260) and provide a macro-context for changes to the system.  Exogenous landscape level changes have a dramatic effect on transitions as landscape factors can create conditions for change, have a stabilizing impact on regimes, or induce a dampening effect on the impact of emerging disruptive niche innovations (Burch et al., 2014; Davison, 2001).  The MLP posits that changes in socio-technical systems arise mainly from the intersection of two processes. One in which collections of niche innovations align to put pressure on the policy regime, and the second wherein landscape level changes (i.e. economic pressures) create windows of  28 opportunity within the regime for niche innovations to take hold. While “socio-ecological and socio-economic perspectives emphasise the potential for grassroots innovation and local initiatives to catalyse macro-scale change” (Asquith et al. 2018, p. 10), the socio-technical perspective emphasises the interdependent relationships of external landscape and niche innovations as the key mechanism for bringing about systems transition. Governing transitions is therefore a matter of [enhancing] variation by stimulating the emergence and diffusion of niche innovations [and changing] the selection environment by enhancing pressure on regimes through economic instruments and regulation (e.g. taxes, carbon emission trading, environmental legislation) (Geels, 2018b, p. 63). Interactions between niche and regime are contested and multi-dimensional including economic competition between old and new technologies; business struggles between new entrants and incumbents; political struggles over adjustments in regulations, standards, subsidies and taxes; discursive struggles over problem framings and social acceptance; and struggles between new user practices and mainstream ones (Geels, 2018a, p. 227). 2.3.1.1 Governing and managing transitions Several models have been proposed for governing and managing transitions. Strategic niche management (SNM) focuses on the intentional creation and nurturing of protected spaces, or ‘niches’, that may serve as a source of change to the incumbent regime, possibly triggering a shift to a new constellation of actor/network/technology relations (Burch et al., 2014, p. 470). The approach presumes that the support of technological niches will foster a shift to a more sustainable socio-technical regime (Sengers et al., 2016, p. 3). Technological innovation systems (TIS) “focuses on the emergence of new technologies and their institutional or organizational contexts, as well as drivers and barriers to this emergence” (Burch et al., 2014, p. 470). Loorbach & Rotmans define transition management (TM)  “as a deliberative process to influence governance  29 activities in such a way that they lead to accelerated change directed towards sustainability ambitions” (Loorbach & Rotmans, 2010, p. 238). The transitions management literature is rich with examples of transition processes and tends to focus more on the practicalities of organizing governance processes rather than assessing whether transition management activities have contributed to societal transition (Loorbach & Rotmans, 2010).  Of the various socio-technical approaches described, the MLP “presents a logic that explains how the interplay of innovation and disruption at different scales brings about systemic change” (Asquith et al. 2018, p. 25). In many ways, the different approaches described in the previous paragraph can be seen as different proposals on how to foster transition within the MLP context – i.e. they are not competing theories. SNM and TIS have much to say about the development and nurturing of niche innovations while TM describes an overall approach to managing transitions. The MLP includes a wide variety of actors in transition analysis, e.g.  “not only the agents of financial and production capital but also by scientists, engineers, policy-makers, users, media, social movements, and so on” (Schot, 2017, p. 9). The MLP’s focus on the process of how systems actually change, its ability to represent complex and interdependent systems, and it’s provision of space for the role of a diverse set of actors within systems change, leads me to make the MLP my primary theory of societal transition for this thesis. Common throughout the different strands of transitions literature is the concept that regime transition occurs from the intersection of bottom-up pressures from niche innovations and top-down pressure from landscape shifts. However, transitions literature approaches the question of how to evaluate transitions from a number of different perspectives. These perspectives range from how to assess when transitions are happening to where specific processes that are designed to  30 support/foster/facilitate transition have made a contribution to transition. However, as Rohracher & Späth note while discussing urban sustainability transitions, what happens in practice is often not transitions ‘according to plan’ or other forms of coordinated transformation processes; such paths and plans were just one element of intervention in a constant process of renegotiating what sustainability could mean in concreto, and to what extent these normative orientations could accommodate various actors’ interests, power structures and political strategies in cities (2017, p. 297). This suggests that markers or early indicators of transition may be used in evaluation. For example, do systemic changes point to transitions? Do they incorporate features that might be expected to support transition? Given its roots in innovation studies, transitions scholars have proposed tracing the scaling and adoption of innovations as a key measure of systems transition (Bos & Hofman, 2016). Some theorists take a political lens and have put a focus on “evaluating shifts in power relations, the extent to which power relations have shifted, development priorities have changed, or new identities have developed” (Burch et al., 2018, p. 308) while others focus more on changes in governance roles and processes as indicators of transition (Meadowcroft 2009). Geels (2014) and others (e.g. Walker, 2012) discuss the barriers to change that stand in the way of transition. We can assess the state of those barriers and if they are changing, or what contribution an intervention is are making towards those changes in, for example, economic, political and social lock in mechanisms (Geels, 2018b). To assist in this effort to capture policy impact, knowledge mobilization researchers have developed frameworks that evaluate how and where research has been adopted by policymakers with Meagher et al. (2008) suggesting a spectrum of impact ranging from more conceptual uses such as awareness, knowledge and understanding, and attitudes, perceptions and ideas, to more instrumental uses such as practice and policy change. Lindquist (2001) echoes this sentiment arguing that setting the bar as high as “policy change” can be very  31 challenging for programs and instead suggests intermediate impacts such as expanding policy capacities and broadening policy horizons (i.e. through debate and dialogue) as measurable objectives. 2.3.1.2 Energy system transitions The approaches described above refer to transitions in general but energy transition scholars have proposed methods for evaluating energy system transitions specifically. Gawel et al. suggest we might evaluate a “successful” socio-technical transformation  by whether and how far it succeeds in organizing the comprehensive sustainability of the energy supply in a highly industrialized country within a reasonable period of time and in maintaining profitability, security of supply, environmental compatibility, and social acceptability in equal measure (2014, p.6). Wilson & Grubler define a sustainable energy transition as “mitigating climate change, providing universal access to modern forms of energy, ensuring secure markets and supply chains, reducing air pollution and human health impacts” (2011, p. 165) while Geels et al. propose the following definition of a sustainable energy system transition as one in which major changes in buildings, energy, and transport systems that substantially enhance energy efficiency, reduce demand, or entail a shift from fossil fuels to renewable inputs. These system transitions entail not only technical changes, but also changes in consumer behaviour, markets, institutions, infrastructure, business models and cultural discourses (2016, p. 577). Common among these definitions is a focus on technology, markets, and policy with relatively little emphasis on the role of citizens, governance, and changes in social relationships.  32 2.3.1.3 Learning and evaluation in transitions Transition management (and strategic niche management) highlight the importance of social learning:  Social learning, as a reflexive learning process that involves and goes beyond individual participants, is considered a precondition of change within the transition management literature. It is based on bringing together different actors’ perspectives and a variety of options in participatory settings. Joint learning of participants can contribute to the development of alternative and visionary solutions to complex challenges. This results in new types of discourse as well as changing perspectives (Schapke et al., 2017, p. 5). Reed et al. (2010) offer the following criteria for social learning which provide a useful set of elements for my evaluation framework: “social learning must result in changes in understanding and/or behavior (e.g. new norms, rules, practices and narratives) resulting from social interactions, which become situated in collectives such as communities of practice, networks or societies.” (p. 12).6 Transition management also argues that there is a link between social learning and empowerment. This empowerment is also a by-product of transition as Loorbach has said: “the ultimate goal of transition management should be to influence and empower civil society in such a way that people themselves shape sustainability in their own environments, and in doing so contribute to the desired transitions to sustainability” (2007, p. 284).  The sustainability transition literature emphasises the need for managing STEs for deepening (i.e., learning), broadening (i.e., replicating) and scaling up (i.e., creating a regime shift) (Kivimaa et al., 2017; Raven et al. 2010). Across the literature, a desirable evaluation is seen to be reflexive,  6 Content in this section adapted from Robinson, J., Slater, K., Williams, S., Pajouhesh, P., and Benzakin, G. (2018). TAF Evaluation Framework Outline.  33 flexible, continuous or long-term (occurring before, during and after the experiment), generic (i.e., comparable and usable across different kinds of experiments) and comprehensive. In addition, multiple types of evaluation may be used: “ex-ante evaluation prior to the implementation of pilots to inform their design; formative evaluation to adjust and improve ongoing ones; and, ex-post evaluation to appraise the contribution of pilots to sustainability after completion” (Luederitz et al., 2016, p. 2).  Wiek et al. (2017) elaborate on this work to propose a framework more explicitly focused on urban sustainability transition experiments (called labs in their paper). A key development in the evaluative framework is assessment of potential outcomes by extrapolating “what outcomes could result if the outputs would be widely generated or applied” (2017, p. 254). However, simply extrapolating from outputs to outcomes runs the risk of masking the complexity and emergence in transitions. Wiek et al. assume here that it is the outputs that lead to change. To assess such a link, Arnold et al. suggest that “an evaluation approach…should endeavour to link systemic or macro evaluation with the evaluation of individual sub-programmes at the micro level” (2018, p. ii) but do not come to a conclusion on how to do so. Looking for ‘markers’ of transition that point to potential transition may be a more fruitful approach.  2.3.1.4 Implications for evaluation framework Although intended to produce an approach to evaluating sustainability transition impacts, this review of transition theory literature also suggests implications for evaluating STE processes and their societal effects along with sustainability transition impacts. Table 2 below summarizes these elements.  34 Table 2: Evaluation elements from transitions literature  Transition theory scholars note a series of process characteristics that should be employed to increase the likelihood of achieving listed societal effects and transition impacts. We can assess how STE niche innovations and innovators interact with the regime, create cross-level networks between niche and regime, align with or reinforce other niche innovations, embody reflexive and adaptive learning processes, and create spaces for experimentation and learning. An evaluation framework should assess the influence of projects at the niche and regime levels where 1st and 2nd order societal effects may occur.  Transition theory raises several potential methods of assessing the contribution of a process to sustainability transition. These include shifts in power and governance relationships that change how decisions are made and influence resource flows, reduced barriers to transition (such as institutional inertia and path dependence of built infrastructure), and changes in roles and relationships between actors that change who has access to decision making power. While related to effects at the level of niche and regime, these transition impacts instead reflect lasting structural changes that become embedded – or normalized – within the regime. This also reflects a shift in relationship between niche and regime, one in which niche innovations have succeeded in Process Societal Effects Sustainability Transition Impacts Niche innovations and innovators interacting with the regime Creating cross-level networks between niche and regime Aligning with or reinforcing other niche innovations Reflexive and adaptive learning processes Spaces for experimentation and learning 1st  and 2nd order niche and regime effects (usable products, individual capacity, networks, institutional and organizational change) Climate/energy outcomes (e.g. carbon reduction)  2nd order structural change (policy & organizational) Actor relationships Governance Reduced barriers to transition  35 disrupting the regime and its associated practices. However, while transitions theory raises these issues as considerations for transitions impacts, it is not clear from the literature how to assess such impacts. The mechanisms through which these changes in governance and actor roles and relationships change, or how changes in practice become embedded, are also unclear. I turn to additional literatures in the following sections to explore these issues more deeply and develop a more comprehensive set of sustainability transition impacts.  2.3.1.5 Critiques of transition theory Transition theory, and the MLP in particular, is not without its critics. Critiques of MLP include its conception of sustainability, addressing agency and lack of attention to justice, power and politics. From a sustainability lens, the MLP has been critiqued for a focus on transition processes rather than sustainability impacts (Asquith et al. 2018, p. 13), a relatively narrow conception of sustainability (Geels, 2011; Loorbach et al., 2017; Markard et al., 2012; Schapke et al., 2017), and a focus on experiments and single innovation adoption rather than whole systems change (Geels, 2018a; Markard, 2018). Where agency is located in the MLP has been a recurring critique over the past years (Geels, 2011; Meadowcroft, 2005; Smith and Stirling, 2007; Hodson and Marvin, 2010). Critics have characterized the MLP as overly rational and structuralist in its approach (Genus & Coles, 2008; Smith et al., 2005) to the exclusion of individual agency towards transition. Many authors have pointed to a lack of attention to justice and power within transitions studies in general and specifically within the MLP (c.f. Shove & Walker, 2007; Avelino & Rotmans, 2009; Meadowcroft 2009; Lawhon & Murphy, 2012; Geels, 2014). Where just transitions have been addressed, a narrow definition of justice is often used (van Steenbergen & Schipper, 2017; Swilling & Annecke, 2012).  36 These critiques, and my review of the transitions literature, suggest the existence of four significant gaps. First is a superficial inclusion of justice in transitions. This is important both for normative and instrumental reasons. Second is an inadequate focus on governance and actor roles and relationships. This is highly relevant due to the inherently political nature of transitions. Third, transitions theory has not deeply engaged with theories of social practice. This matters because practices transcend actors and constrain choices and actions. Finally, transition theory has a narrow definition of sustainability which is important because sustainability goes beyond socio-technical terms like changes in energy mix. In the following sections, I engage with additional literatures to address these gaps and integrate key insights from different traditions into my evaluative framework.  2.3.2 Beyond transition theory In this section I engage with literatures that attempt to respond to previously identified gaps in the transitions literature: a superficial inclusion of justice in transitions, an inadequate focus on governance and actor roles and relationships, a lack of engagement with theories of social practice, and a narrow definition of sustainability.  2.3.2.1 Justice, power and politics in transitions7 Early criticisms of transitions research noted the lack of power and politics in the analysis of transitions (c.f. Shove & Walker, 2007; Avelino & Rotmans, 2009; Meadowcroft, 2009; Lawhon  7 Content in this section is adapted from Williams, S. and Doyon, A. (2019). Justice in Energy Transitions. Environmental Innovation and Societal Transitions. Vol 31, June 2019. pp. 144-153. See this paper for an extended discussion of justice in energy system transitions.  37 & Murphy, 2012; Geels, 2014). Some, like Avelino and colleagues, developed new frameworks to offer “a ‘power-laden transition storyline’” (2009, p. 563), or to better understand politics by conceptualising (shifting) power relations between actors in transitions (Avelino & Wittmayer 2015). Miller & Levenda called for more attention to be paid to the “social and political dimensions of sustainability” (2017, p. 347). Others have suggested that the field of transitions could better incorporate issues of power and politics by gaining insights from the fields of political economy (Geels, 2014), political ecology (Lawhon & Murphy, 2012), as well as political geography (Murphy, 2015). However, “despite growing attention to power and political dimensions of transitions (e.g. Avelino et al., 2016) several authors claim that transition scholars have actually very little to say about equity and justice, and the political economy of transitions” (van Steenbergen & Schipper 2017, p. 3). 2.3.2.1.1 Just transitions Within the transitions literature, earlier engagements with justice have come from the concept of the ‘just transition’. Swilling and Annecke (2012) provide a comprehensive overview of global environmental and sustainability challenges from the perspective of the Global South, where the concept of a ‘just transition’ reconciles sustainable consumption with a commitment to sufficiency – essentially arguing for a balanced global approach to resource use and management that addresses the distribution of costs and benefits of transition. Newell and Mulvaney (2013) approach just transitions from a political economy perspective, whereby they argue for the need of equity and justice to be included in efforts to support the transition to a low-carbon future. In particular, they focus on issues related to labour and energy justice, as well as notions of climate justice and vertical forms of environmental justice. They argue that there is a need to understand “who defines what is just, and for whom,” (Newell & Mclvaney 2013, p. 138), and how these  38 questions are related to existing power structures in different contexts. More recent work by Jasanoff (2018) reiterates the need to consider justice in energy transitions from a global, planetary-boundary perspective.  Another approach to incorporating justice can be referred to as ‘justice in transitions’, i.e. how justice has been addressed in the transitions literature. van Steenbergen and Schipper (2017, p. 2) state that “when dealing with transitions one is automatically entangled in moral and ethical questions”. They argue that justice should be understood as a process, and not an end point, meaning justice should be “an essential and integral part of systemic change” (p. 8). Within a more economic focused approach to transitions, Silveira and Pitchard (2018) draw on Sen’s (2009) ideas of justice to incorporate ‘justice in transitions’ in relation to the shift to a sustainable, low-carbon economy. Another approach is offered by Heffron and McCauley (2018) who recommend bringing together different framings of justice from climate justice, environmental justice, and energy justice with transitions theories and legal geography. These different approaches to incorporating justice all highlight the importance of the issue but only address parts of a comprehensive approach to justice. Environmental justice provides such an integrated approach. Environmental justice has been defined as “the fair treatment and meaningful involvement of all people regardless of race, color, national origin, or income, with respect to the development, implementation, and enforcement of environmental laws, regulations, and policies” (US EPA, 2018). It is rooted in the principle that disadvantaged communities should not be subject to disproportionate environmental impacts (Schlosberg 2013). As an approach, it supports political or activist activities against injustice as no group should be unequally burdened by negative environmental impacts (Agyeman & Evans 2004; Agyeman et al. 2010; Schlosberg & Carruthers 2010). Environmental justice takes  39 place across different scales: justice for people, justice for communities, and justice for non-human species and ecosystems (Schlosberg 2013).  There are three key concepts of justice in environmental justice: distributive justice, procedural justice, and justice as recognition. Distributive justice, perhaps the most popular concept of justice, focuses on the distribution of environmental goods, costs and benefits. Bell (2004) provides three questions needed to construct a distributive justice claim: 1) Who are the recipients of environmental justice? 2) What is to be distributed? 3) What is the principle of distribution? It is important to recognize that with regards to distribution, it is not only about the direct environmental burden or benefit, but other intersecting dimensions such as vulnerability, need, and responsibility (Walker, 2012). In addition to environmental burdens/benefits, distributional justice addresses questions of access to resources and opportunities that are deemed to be critical to redress social injustices (c.f. Schlosberg, 2007). Procedural justice, on the other hand, is about inclusion and exclusion in decision-making processes around environmental and social issues: “Many definitions of environmental justice convey the importance of fairness in procedure or process as a distinct concept of justice” (Walker 2012, p. 47-8). Procedural injustices occur when environmental information is unavailable, as well as when there is exclusion and inequity in relation to public participation in policy, decision-making, and access to the formal justice system.  Finally, justice as recognition focuses on the recognition, misrecognition, or non-recognition of various groups, and is related to prejudice and discrimination of all forms. At the root of these injustices are cultural and institutional processes and legacies that have that have explicitly or implicitly given individuals, communities, or social groups unequal recognition (Walker, 2012). Conceptions of environmental justice in the literature have evolved from a relatively narrow conception of distributive justice to include the additional dimensions of procedural and  40 recognition. Recognition is deemed to better engage with pluralist needs, issues and solutions providing a comprehensive conception of justice useful for transition scholarship (Nozick, 2017). 2.3.2.1.2 Does justice in transitions matter? Readers may ask: how important is justice? Does justice in transitions really matter? I propose normative, substantive, and instrumental responses (following Fiorino, 1990) to these questions. From a normative standpoint, I argue that we cannot achieve a sustainability transition without justice, indeed that an unjust transition is not sustainable. A substantive response is that by actively seeking to include a diverse representation of society, diverse worldviews and conceptions of sustainability, a transition process gains access to knowledge not otherwise available. This knowledge contributes to substantively better decision making that that conducted simply by traditional elite policy makers. An instrumental response is that not considering justice erodes political support for transitions efforts (Fiorino, 1990). Rohracher and Späth, in arguing for more socio-political analysis of transitions, note that  changes in the energy or transport system are often strongly shaped by [socio-political] discourses and dynamics – e.g. of cities positioning themselves in global economy competition, dealing with social problems or trying to attract a young and educated population – and not by urban energy plans, transitions targets or the aim to learn about new technologies (2017, p. 291). Recent populist swings in the United States and the United Kingdom illustrate this argument. Many Trump and Brexit supporters are from fading coal mining communities and manufacturing centres that have lost jobs due to globalization – both areas “lost” during the global economic transition still underway. Voters that feel they were not consulted, that their way of life is threatened, that they do not have decision making power, may respond by supporting populist politicians who promise to restore lost greatness and take control of policy back from outsiders (e.g. the WTO or  41 EU) (Batel & Devine-Wright, In Press). These themes have been taken up in recent work by Roberts et al. who argue for the importance of considering  the role of coalitions in supporting and hindering acceleration; the role of feedbacks, through which policies may shape actor preferences which, in turn, create stronger policies; and the role of broader contexts (political economies, institutions, cultural norms, and technical systems) in creating more (or less) favourable conditions for deliberate acceleration (2018, p. 304). 2.3.2.1.3 Implications for evaluative framework The procedural dimension of environmental justice suggests that we should evaluate how an STE is addressing not just inclusivity and diversity of participants but the capacity of stakeholders to adequately participate, power asymmetries between participants (and process designers, facilitators, and funders), and how future generations and non-human actors are represented in the process. The recognition dimension suggests an assessment of how conflicting knowledges are represented and integrated, how marginalized peoples and alternative worldviews are recognized, and how institutional and cultural inequalities are addressed. The distributive justice lens implies paying close attention to where the costs and benefits of STE outputs and outcomes are accruing, what spatial, temporal and jurisdictional scales are being considered, and reflecting on the scope of analysis for the evaluation (e.g. a specific project within an STE, the STE itself, or a whole system). Sustainability transition impacts arising from environmental justice literatures are whether the distribution of benefits and costs of transition is equitable, stakeholders are meaningfully engaged in transition governance, and historical (and current) marginalization of peoples has been reduced. This assessment of impacts reflects the character of changes in governance roles and relationships,  42 not just whether they are occurring. In other words, an environmental justice lens provides a way for us to assess whether transitions are moving in a more sustainable direction. 2.3.2.2 Governance Although governance is recognised as one of the key elements in managing sustainability transitions (Grin, Rotmans, & Schot, 2010), as a particular strand of research it has continued to evolve within the transitions field. This is highlighted by the recognition of multi-scales and multi-sectors involved in transitions research, and the use of the multi-level governance framework. Important to note here is that the governance literature is a huge field. In this section, I discuss governance with reference to specific conversations within the transitions field. Within that context, governance research is important because it contributes to understanding the historical contextualization of transitions. It emphasises the embedded patterns, actions, and structures, and how changes within these domains are influenced by exogenous trends (Grin et al. 2010). Governance for sustainable development is about steering society towards a more sustainable future, therefore, it is important to acknowledge who is steering and to what ends, as well as how and where it takes place (Meadowcroft 2007). Grin et al. (2010) argue that governance research highlights the power and politics inherent to processes of profound change, because “politics and political processes lie at the heart of governance for sustainable development” (Meadowcroft 2009, p. 335). A broader critique of how transition theory addresses governance comes from the view that contestation is, rather than a barrier to transition that needs to be managed, an integral part of transition. Cher et al. argue that when politics is addressed in transitions literature, regulations and policymakers [are conceptualized] as elements of a socio-technical system rather than as  43 techno-economic and political entities and processes [that] are neither independent of nor subsumed into socio-technical systems. Rather they make up semi-autonomous systems with their own dynamics which co-evolve along with socio-technical ones (2018, p. 181).  The argument here is that by addressing politics as an independent entity – much like a set of technology or infrastructure – transition theory does not adequately consider the interdependent relationship of politics to all other elements of the socio-technical system and downplays the critical role that politics plays in decision making and system transition (c.f. Meadowcroft, 2009). Burch et al. foreground politics as involving “interactions through which the identity of actors is shaped, their legitimacy established, and their values articulated in the public realm” and further that the “challenge for urban transformations will be  finding ways to negotiate and resolve (or accept) differences in order to reach collaborative outcomes” (2018, pp. 304-305). Kenis et al. note that “acknowledging conflict, contradictory interests and radical forms of pluralism is a condition for avoiding that large parts of citizens become alienated from transition discourses elaborated by enlightened elites” (2016, p. 15). These different arguments all point to the value of contestation in transition processes rather than seeking consensus. The implication is that how contestation is encouraged, fostered, and managed is an important part of evaluating STEs and will have implications for how processes, effects and sustainability transition impacts are conceptualized. This issue becomes especially relevant when considering methods such as Transition Management that are explicitly designed to incorporate a wide array of interests, stakeholders and viewpoints in transition experiments.  Kenis et al. argue that  transition management can be understood as a specific variant of an ongoing tendency towards ‘governance’, which, in a guise of bottom-up processes and participation, risks to redistribute power from what should be ‘all citizens’ towards non-elected groups of ‘important’ business and civil society actors. The implication of this is that ‘the people’ do not get a place as possible actors of change in this transition process. Sometimes, they are taken into consideration in their role as consumers, but even that is not always the case (2016, p. 3).  44 They further argue that by  relying on a deliberative model of democracy, transition management fails to fully acknowledge power relations, radical pluralism and from the possibly constitutive role of conflict in society. In so far as conflict gets a place, it is framed as a market oriented instrument stimulating innovation (Kenis et al., 2016, p. 3). These critiques point out an inherent tension in sustainability transition experiments between the desire for a truly representative sample of ‘the people’ and the practicalities of managing an experiment with limited time and resources. Kahane et al. (2013) highlight this tension by distinguishing between stakeholder and citizen engagement processes. Kahane et al. argue that stakeholders (i.e. representatives of an organized group) can be helpful in participatory processes by providing, for example, an efficient route to distributed groups, have influence and power with decision makers to actually implement recommendations and any shifts may travel back to impact that organization, sector or industry. However, inclusion of stakeholders can raise challenges as well such as the appearance (or reality) of bias. As Kahane et al. note,  in Alberta, for example, consultation processes about resource extraction are often assumed by environmentalists to incline toward the interests of extractive industries in particular, and away from the perspectives of citizens (2013, pp. 9-14). Inclusion of representatives of the extractive industries can exacerbate this assumption even though it could be argued including the dominant industry is a requirement in making any meaningful policy change. As with stakeholders, engaging citizens has its own benefits and challenges. However,  governments may not have the capacity to effectively organize citizen engagement processes, citizens may not have the capacity to interpret large volumes of (potentially biased) information or the capacity maintain a level of participation over time, and “deliberation between citizens tends to reiterate structural inequalities between social groups” (2013, p. 17) such as those between men and women, the role of marginalized groups, and class and educational  45 differences. Integrating a procedural and recognition based justice lens (see Section 2.4.2.1) into STE evaluation allows us to assess how these issues have been addressed. 2.3.2.2.1 Implications for evaluation framework Assessing how STEs address contestation and negotiation is a key process element to evaluate along with how power relations and dynamics within an STE (e.g. between highly resourced incumbent entities and civil society) are addressed. In addition, we should evaluate how an STE addresses representativeness and mitigates the risk of a (real or perceived) elite making transitions decisions on behalf of the broader population. Medium term outcomes we may assess are whether governance roles and relationships are changing. For example, where is the locus of decision making within a given STE among a range of stakeholders? We may also assess how dominant incumbent actors are incorporating the interests of a different range of stakeholders (as represented by an STE) in policy and organizational decision making. To capture sustainability transition impacts, what is important from a governance lens is how (and whether) governance relations and social inequities are changing. For example, are citizens more broadly engaged in decision making processes about transition? Are new models of governance (e.g. polycentric governance (Ostrom, 2009; Stirling, 2014b)) emerging? This echoes the procedural justice lens but adds the additional element of how these changes in governance becomes embedded in a system. For example, an STE may foster deep engagement with a range of stakeholders and disrupt traditional governance and decision making models. However, if that disruption only lasts for the duration of the STE and does not affect dominant governance models, it is unlikely that the STE has fostered a sustainability transition.   46 2.3.2.3 Actors & agency in transitions The role of agency – what ability individuals or organizations have in creating sustainability transitions – is a key challenge to the idea of managing transitions. Geels suggests that there is a role for actors in improving and expanding new elements in a regime (2002) while emphasizing that “actors at niche and regime levels cannot influence [the landscape] in the short run (2011, p. 28). Geels & Schot suggest that niche actors can critique the existing regime, develop new innovations that compete with the regimes while regime actors can adopt these innovations or adjust. However, they do not suggest that this type of agency can guide socio-technical transitions writ large (2007, p. 415). Despite Geels’ claim that the MLP is “shot through with agency” (2011), many authors (cf. Meadowcroft, 2005; Smith et al., 2005; Smith and Stirling, 2007; Genus and Coles, 2008; Hodson and Marvin, 2010) have argued that the MLP does not pay enough attention to the role that individuals, groups and organizations can have in contributing to transitions. In response to these critiques, there has been much recent work on the role of actors (Markard et al. 2012, Farla et al. 2012, Grin et al. 2010), attention to shifts in power (Rotmans and Loorbach 2010, Avelino 2011) and agency (Geels, 2014; Elzen et al., 2011; Penna and Geels, 2012; Baker et al., 2014; Kanger and Schot, 2016). Westley et al. argue that social and institutional entrepreneurs have agency in affecting transition by “work[ing] simultaneously at building innovation niches into innovation regimes and at destabilizing the dominant landscape and regime to secure the required resources” (2011, p. 771). Jørgensen goes further suggesting that  actors cannot analytically be attached only to one level, as e.g. niche-actors, regime-actors, or even actors with special roles and emphasis on the landscape level. In empirical terms, actors are engaged in transforming and intervening at all levels, without necessarily being very explicit about distinguishing between them (2012, p. 1000).   47 This recent work has  made it clear that the process of transition is far from a moderate and rational consensus-oriented debate about best solutions to clearly defined problems: instead it is rife with struggles between regime-actors and niche-actors with conflicting interests, differing timescales, problem definitions and perceived best courses of action (Schot & Kanger, 2018, p. 1), an assessment shared by Fischer and Newig (2016). Loorbach et al. (2017, p. 164) note that transitions research therefore seeks to understand how different types and forms of agency influence the speed and direction of transitions and how they can be engaged, can be empowered, and can more effectively contribute to desired transitions. However, this leaves open the question of how to practically address the roles of agency and actors in transitions research. This range of authors points to the importance of actors in transition but does not provide a coherent analytical framework by which to do so.  In attempt to provide such an analytical framework of actors in transition, Avelino & Wittmayer (2017) propose a Multi-Actor Perspective (MAP) to analyze transition actors with community, market, state and associations (non-profits) as the primary categories of actors. Within each category, they distinguish between individual roles (e.g. politician, consumer, activist, resident) and organizational roles (e.g. municipalities, SMEs, NGOs, networks) recognizing, per Jørgenson (2012), that actors play multiple roles and act at multiple levels of the MLP. There is a benefit of this more nuanced approach rather than clustering, for example, NGOs, trade unions, universities into a generic ‘civil society’ category. The  tendency in empirical analyses and in management applications [is] to equate the regime with ‘government and big business’, while associating niches with ‘small entrepreneurs and/or civil society’ (Avelino & Wittmayer, 2017, p. 281). By applying this lens of analysis to case studies, MAP can reveal  48 (1) how agency manifested in (individual) actor roles can be both promoting as well as hindering and overlooking regarding socio-technical innovation, and (2) the deeply and inextricably political nature of (the history of) urban [and other] infrastructure (Avelino & Wittmayer, 2017, p. 276). Each case study can be read as a “narrative about urban (un)sustainability as a result of power relations, negotiations and dependencies between different actors, among different urban actors, but also among urban, national and international actors” (Avelino & Wittmayer, 2017, p. 280) highlighting the socio-political nature of transitions in addition to the socio-technical. The MAP approach is a valuable analytical tool for an evaluative framework in that it provides a rich structure for describing the range of actors engaged in systems transition and, more importantly, their interactions at multiple levels of scale. This allows for a deeper understanding of actor roles and relationship and how they are changing as a result of STE processes.  2.3.2.3.1 Social practice An alternative conception of the role of actors comes from social practice theory. Practices, such as driving or cooking, are made up of elements:  materials – including things, technologies, tangible physical entities, and the stuff of which objects are made; competences – which encompasses skill, know-how and technique; and meanings – in which we include symbolic meanings, ideas and aspirations”  and that “practices emerge, persist, shift and disappear when connections between elements of these three types are made, sustained or broken (Shove et al., 2012, pp. 14-15). Shove argues strongly that actors are not “autonomous agents of choice and change” but are in fact carriers of practice. In other words, social practice foregrounds collective processes of activity – practices – rather than individual behaviours.  Social practice theory implies that if we attempt to manage transitions through attempting to change individual behaviours without paying attention to practices that transcend individual actors,  49 we will not be successful in fostering transitions (see discussion on climate change and behaviour change in Shove et al., 2012, pp. 139-162).  Going even further, Shove and Walker say that by looking at practices (i.e. asking ‘what is energy [used] for?’) “is to take a different view of the social. It is to see society not as an outcome of intersecting systems, like geological forces pressing this way and that, but as emergent from, and defined by, social practice” (2014, p. 46).  Rohracher and Späth note that  anchoring interrelations of emerging niches (such as new forms of renewable energy generation), regimes (the fossil-fuel based centralised energy system) and landscapes (deeply entrenched values and institutions, such as neo-liberal governance ideals) in localised contexts of cities directs our view to the embedding of, for example, energy generation and use in wider social practices beyond the energy system, and to potential clashes of energy system change with other logics and  fields of action (2017, p. 288 emphasis added). Social practice then emphasizes “the horizontal circulation of elements and argues for a flatter model [than that proposed by MLP] characterized by multiple relations (rather than hierarchical [or vertical] levels) of reproduction across different scales” (Shove and Walker 2010, p. 474). The key difference here is that the MLP emphasizes the ‘vertical’ relationships between niche, regime, and landscape. Social practice highlights the importance of ‘horizontal’ relationships among niche actors, between regime actors, and within the broader social-cultural landscape. In fact, a practice view of transition implies a radical flattening of the hierarchical levels of the MLP arguing that actors are not simply independent units of analysis within an exogenous system, but are interdependently connected in a network of practice (Geels, 2011).  MAP and Social Practice take quite different approaches to the role of actors and their agency. MAP places actors at the centre of analysis ascribing agency in making and changing systems to actors. Social practice, on the other hand, posits that actors are carriers of practice and enacting  50 routines that are not the result of conscious choice. Despite their differences, both are useful in assessing sustainability transition impacts. MAP provides an analytical framework to assess how governance roles and relationships are changing, how actors (playing different roles and acting at different levels of the MLP) are behaving and influencing system transition, and highlights the impact of structural barriers to action within a system. Social practice provides a framework through with to assess how (or if) sustainability is becoming “normalized” within a system. This might be through changes in routines within a regime (e.g. routines of decision making within energy companies or government agencies) or within the landscape (e.g. practices of transportation, food consumption, or housing that have massive energy implications). For example, Shove suggests sustainability transitions are those in  which contemporary rules of the game are eroded; in which the status quo is called into question; and in which more sustainable regimes of technologies, routines, forms of know-how, conventions, markets, and expectations take hold across all domains of daily life (Shove, 2010). My evaluative framework does not propose to create a meta-theory that integrates these two quite different approaches. However, I argue that these different theoretical lenses give useful perspectives into sustainability transition impacts. Hargreaves et al. propose an analytical approach that integrates the ‘vertical’ levels of MLP with the ‘horizontal’ flows of social practice:  MLP allows one to examine the emergence of novelty through the interactions between the vertically ordered levels of niche, regime, and landscape, while [social practice theory] focuses attention instead on the horizontal dynamics of practices that cut across multiple regimes as they follow their circuits of reproduction. (Hargreaves et al. 2013, p. 407). They introduce an example where a ‘vertical’ analysis of a niche innovation (organic food delivery service) attempted to disrupt the regime by replacing supermarkets with farmer’s markets and  51 home delivery services of seasonal produce. A ‘horizontal’ analysis illustrated that the intended disruption was not successful partially due to a lack of attention to social practices of food purchasers and consumers. For example, the practice of cooking dinner includes planning a menu, creating a shopping list, then stopping at the store on the way home from work to pick up food. Farmers markets that are only open on weekends and delivery of an unknown selection of vegetables did not fit this practice. Shove (2012) notes the same challenges when attempting to foster pro-environmental behaviour change through policy and public education that does not consider barriers to transition raised by existing practices or alternatively, the supports to transition from latent practices. In evaluation, we may ask what are these embedded rules and do STEs address job descriptions, evaluation criteria, professional standards, and changes in those institutional rule systems (largely explicit at the regime level) needed to make transformational change along with changes in cultural norms at the landscape level (largely invisible and implicit). Rather than unifying these theories, an analysis “should retain the distinction between regimes and practices and actively explore the nature of the relationships between these two units of analysis as they intersect and cut across one another in the course of innovation processes” (Hargreaves et al., 2013, p. 418). As Geels et al. note, “full integration of different approaches is not possible because of fundamental differences in ontological assumptions and methods” (2016a). However, a potentially promising strategy is 'bridging', based on dialogue and interaction of independent approaches (Geels et al., 2018). This integrative approach is supported by Moore et al. (2018) who argue for the integration of social practice theory with the multi-level perspective and socio-ecological systems analysis.  2.3.2.3.2 Implications for evaluation framework  52 Given the importance of actors in transition, at the process level, we can assess which actors are part of an STE process and which are not. The MAP framework provides a useful analytical tool through which to do so. At the level of societal effects, we may trace the actions of a range of actors related to an STE and assess how agency is being exercised. In addition, we may assess how actors play multiple roles and act at multiple levels of a system. Effects through a social practice view will be changes in institutional rules and policies that are more sustainable. Avelino & Wittmayer (2017) point out that actor roles may promote or hinder innovation. Sustainability transition impacts would include assessing where and how actor roles and relationships are changing to reduce barriers to and promote sustainability transition. From a social practice lens, we may assess how regime routines or socio-cultural landscape norms (as embodied by materials, competencies and meanings) that “normalize” sustainability are embedded within institutions and cultures. Note that this social practice lens echoes sustainability transition impacts suggested by social learning literature. Social learning processes may lead to impacts such as the embedding of low carbon / sustainable behaviours in collectives (networks, organizations, societies) evidenced by new narratives, practices, or norms. Moore et al. argue that “wholesale changes to the constituent elements of a practice, or the relations between bundled practices, are required for development path change” (2018, p. 12) which implies the need for evaluation methods that capture such changes. Social learning points to a mechanism through which new social practices may develop in, and become embedded within, collectives. By evaluating an STE through the contrasting lenses of MAP and social practice, we may also assess how existing or changing social practices help or hinder transition. For example, niche innovations in food consumption (which has large energy and climate implications) may be  53 hindered in adoption not by institutional rules and regulations, but by embedded practices of work, shopping, and cooking (Hargreaves et al., 2013). 2.3.3 Sustainability in transitions literature In a recent literature review, Geels notes that “the MLP focuses more on processes than on sustainability impacts of transitions. Therefore, it does not really indicate to what degree environmental problems will be alleviated if certain 'green' innovations lead to system change” (Geels, 2018b, p. 67). In addition, the MLP provides “only limited analysis of the potential environmental impacts and feedbacks of large-scale adoption of new technologies and innovations” (Asquith et al., 2018, p. 23). While these critiques focus on the environmental dimensions of sustainability, the same limitations apply to discussions of the social dimensions of sustainability. Broader cultural impacts along with changes in governance roles and relationships are also not well addressed through MLP approaches. Transitions Management approaches (and sustainability transition experiments) attempt to foster a purposive transition by convening ‘transition arenas’ made up of stakeholders that co-create a vision of a sustainable future and develop a portfolio of interventions to achieve that vision (Loorbach & Rotmans, 2010; Schapke et al., 2017).  While transitions theory discusses the processes and characteristics of whole systems change,  the empirical focus has, so far, mostly [with the notable exception of urban sustainability transitions work] been on sectors and systems with clearly identifiable technological components (e.g. electricity, transport, heat and buildings) (Geels, 2018b, p. 66). This is a limitation when conceptualizing sustainability transition. If systems transition on large scales along multiple dimensions is required for sustainability, and the empirical literature focuses  54 on single-sector/technology cases, how can we assess systemic transformation in a more sustainable direction? Geels suggests the need for further conceptual development in “broadening from singular niche-innovations to ‘whole system’ change” (2018a, p. 230). Rohracher and Späth echo this assessment noting that  by far the most analyses of low-carbon transitions have either focused on particular governance levels (e.g., national energy policy), particular sectors (e.g., the electricity system) or particular actor perspectives (e.g., supply or demand-side perspectives) (2017, p. 287). An exception to this single-innovation/single-sector/single-system focus has been recent work in urban sustainability transitions studies and the spatial dimensions of sustainability transitions (cf. Burch et al., 2018; Frantzeskaki et al., 2017; Nevens et al.,, 2013; Rohracher & Späth, 2017). While this work on urban sustainability transitions is addressing the calls for deeper investigation into inter-niche and inter-system interactions, especially as “cities cut across niche/regime boundaries” and governance levels (Frantzeskaki, 2017, p. 291) there are limitations to the broader applicability and scaling of findings from transition in the urban environment to sectoral, regional, national, or global scales. For example, cities in urban transition case studies tend not to have large fossil fuel producing industries in their jurisdictions that come along with substantial economic and employment dependencies as does the Province of Alberta. Bosman et al. do address the issue of the role of fossil incumbents in transition in their study of the Port of Rotterdam (2018). Luederitz et al. (2016) attempt to measure impacts of sustainability transition experiments by asking whether the experiment strengthened socio-ecological integrity, enhanced livelihood sufficiency and opportunity and other measures of sustainability. However, this framing is evaluating specific interventions not the aggregate portfolio of experiments that are part of STEs such as the Energy Futures Lab. This leads the framework to under-represent the importance of  55 mutual reinforcement dynamics between experiments (Grin, 2011; Riddell, 2015) and where the lab fits into (i.e. both impacting and being impacted by) a broader set – or ecology – of processes (Chilvers & Kearnes, 2016). In addition, this framework does not address how a given process is contributing to a sustainability transition which is different from evaluating specific indicators of sustainability. These conceptions of sustainability transition within the MLP literature have a lack of specificity on what the sustainability impacts of transitions actually are except in socio-technical terms (e.g. change in energy mix). As Robinson points out (Personal communication, 2018) these two points are connected: if the end is sustainability, it goes well beyond socio-technical characteristics. The transformations literature attempts to do just this.  2.3.3.1 Transformations  Transformations research uses many concepts from transitions theory to fill the normative sustainability gap within the transitions literature by integrating a socio-ecological perspective to bring in conceptions of sustainability to systems change (Hölscher et al., 2017; O'Brien et al., 2018). While transitions research tends to be focused on interactions within socio-technical regimes, “transformation is more commonly applied to refer to large-scale changes in whole societies, which can be global, national or local, and involve interacting human and biophysical system components” (Hölscher et al., 2017). Key to transformations is that they “always represent a fundamental rethinking of how a system (such as a city, sector, or level of government) should or could function” (Burch et al., 2018, p. 304). The distinction between the two is not always clear as transitions (as practiced within transition management) and transformations both start with a normative stance on the desirability of transition from a current unsustainable state to a more sustainable one (Hölscher et al., 2017; Loorbach et al., 2017; Shah et al., 2018). For example, in a recent paper, Geels notes that he will “use the terms ‘transformation’ and ‘transition’  56 interchangeably to refer to substantial change (depth) in energy, mobility, agro-food systems across multiple dimensions (scope)” (Geels, 2018a, p. 225). However, what is common in the transformations literature is a focus on the direction or character of systemic change,  whether change is seen as intentional or unexpected, managed or emergent, the central focus in much of the transformations work is prescriptive, describing how the system should change, either proactively or reactively (Shah et al., 2018, p. 252, emphasis in original). The attempt to broaden the conception of sustainability in the transitions discourse is welcome. However, a challenge with the transformations approach to sustainability is its focus on a limits discourse using planetary boundaries (e.g. Rockstrom et al. 2009, Steffan et al. 2015; Folke et al. 2010) and thresholds framing of sustainability “which support an assessment of potentially detrimental implications of undesirable transformations and orient desirable transformations towards ‘safe and just operating spaces’” (Hölscher et al., 2017). This framing has been challenged by changes in the “concept of the role and nature of science that suggests that our understandings of the world are necessarily socially mediated, and constructed in terms of deeply held values and theoretical presuppositions” (Robinson & Cole 2015, p. 137). Recent work has highlighted not only the importance but the desirability of contestation in sustainability transitions (Burch et al., 2018; Cherp et al., 2018; Kenis et al., 2016). Indeed Kenis et al. note that “acknowledging conflict, contradictory interests and radical forms of pluralism is a condition for avoiding that large parts of citizens become alienated from transition discourses elaborated by enlightened elites” (2016, p. 15). 2.3.3.2 Procedural sustainability An implication of the role of contestation in sustainability transitions is that espousing a fixed definition of sustainability in transitions is neither feasible nor desirable. This is especially relevant  57 in guiding or governing transitions. Smith et al. argue that “intended regime transformation … would require some level of agreement on the appropriate prescriptive measures and their objectives” (2005, p. 1498). In other words, sustainability is not a purely socio-technical concern, but an essentially contested concept (Cronon 2006; Dryzek 2013; Guha & Martinez-Alier 1997; Robinson, 2004) with strong normative, ethical, and political dimensions. A fixed conception of sustainability downplays the importance of contestation in sustainability transitions; the normative, ethical, and political dimensions of transition; and the role of power and politics in determining  question of what to agree on, what to do, what the goals of a sustainability transition are, and who gets to decide on all of this. These questions can be addressed using the concepts of procedural sustainability. In this conception, sustainability is an emergent property of dialogue which  involves creating processes of discussion and negotiation in order to address the inherently normative and ethical question of how we should live, and what choices we want to make, given the best available scientific knowledge (Robinson and Cole 2014, p. 137, c.f. Robinson 2004; Robinson and Tansey 2006; Miller et al. 2014; Maggs and Robinson 2016). This approach bridges the gap between top-down conceptions of sustainability and bottom-up emergent definitions accepting and embracing the role of contestation. Given the complex and varied nature of sustainability transitions, a procedural sustainability approach will prove more fruitful and flexible in application to multiple domains (e.g. energy, mobility, food, urban systems) than a pre-determined definition. For example, a given community may say that, in addition to traditional environmental framings of sustainability, income inequality and social justice are key components of a sustainable future and therefore should be considered when guiding or governing a sustainable transition. A common critique of procedural sustainability approaches is that by not recognizing the findings of science on issues such as environmental limits and planetary  58 boundaries, the approach appears to suggest that there are no meaningful limits or constraints to human activity. However, procedural sustainability does not reject scientific findings but places science as providing the best available evidence for decision making, and at the same time, recognizes the incompleteness and provisional nature of scientific knowledge. Limits and boundaries may be an input to a procedural sustainability approach but would not be seen as the sole determining factor in what constitutes sustainability. This opens the door to the use of scientific findings in informing scenario development, participatory modelling, deliberative dialogues, and other methods for engaging publics in conversations about sustainability. In addition, per Fiorino (1990), this participatory approach surfaces non-expert knowledge that contributes to, and improves, available scientific data (Jasanoff, 2010; Wynne, 1992). 2.3.3.3 UN Sustainable Development Goals 8 The UN Sustainable Development Goals provide a mechanism through which to validate the comprehensiveness of the above conceptions of sustainability in transition impacts. Much policy and practice are expected to be guided by Agenda 2030 and its 17 Sustainable Development Goals (see Figure 1) during the coming decade. The Agenda recognizes the demand for transformations on various levels in society as necessary processes to realize sustainability (United Nations, 2015). Sustainability is here represented by 17 Sustainable Development Goals, that are integrated and interrelated emphasising three dimensions of sustainable development: the social, the economic and the environmental. The process thereto is outlined in the very title of the Agenda:  8 Content in this section adapted from Larsson, J., Williams, S. & Holmberg, J. (2018, June). Guiding Systemic Transition: A Cross-case Analysis of “transition labs” in Canada and Sweden Paper presented at International Sustainability Transitions 2018. University of Manchester, Manchester, UK.  59 “Transforming our world” and it is emphasised that “no one should be left behind”. While the use of SDGs has been critiqued for taking a top-down approach to governance (Bowen et al., 2017; Hajer et al., 2015), difficulty in translating goals to measurable actions (Biermann, Kanie, & Kim, 2017; Hák, Janoušková, & Moldan, 2015; Kanter et al., 2016), and the lack of capacity of developing nations and non-state actors to address the SDGs (Easterly, 2009; El-Zein et al., 2016; horton, 2015), they are generally recognized across institutions and national governments, and provide a set of goals across the dimensions of social, economic, and environmental sustainability. I use the SDGs to assess the comprehensiveness of my proposed evaluation framework in section 2.3.4.1 below. I recognize here that the SDGs were generated through a procedural sustainability approach. In procedural sustainability approaches, we can compare different approaches to sustainability. The value here of the SDGs is in their comprehensiveness.  60  Figure 2: Sustainable Development Goals. Source: United Nations, 2015, p. 18.  61 2.3.3.4 Implications for evaluation The implication of opening up conceptions of sustainability per a procedural approach is that we may assess how STE participants are engaged in conceptualizing sustainability and how differing views are reconciled, and how conflict and the value of contestation are recognized. Societal effects of STE related to climate action / energy use reduction can be captured. For example, Geels (2015) highlights increased energy efficiency, reduced demand, and shifting supply away from fossil fuels as outcomes of energy system transitions. The transformations literature also speaks to a specific set of individual capacity development such as reflexivity, systems thinking, and changes in worldviews. These effects may be 1st order as direct effects of an STE or may be 2nd order indirect effects. Note also that climate impact or GHG reduction effects may occur in a longer time frame than a given STE is running. This challenge may be mitigated by integrating evaluation of changes in governance, practice, etc. to provide markers of directionality of change. In other words, has an STE had effects that are likely to lead to GHG reductions or changes in energy use in the future? However, whether a system has transitioned to one that is truly ‘sustainable’ is not simply a matter of adding up GHG reductions. Transition impact is captured in the changes in governance roles and responsibilities, social learning, justice in transitions, changes and actor roles and relationships, the development of alternative visions and imaginaries, and changes in social practice as described in above sections. Note that, as highlighted by Burch et al. (2014) and Robinson (Personal Communication, March 29, 2019), systems change is not reducible to these elements. The transformations literature foregrounds the importance of changes in values and worldviews as described above. As with changes in governance and practice, what is relevant for  62 transition impact is the extent to which these changes are embedded in systems or whether they remain as niche outliers while the unsustainable system remains largely unchanged.  2.3.4 Evaluating sustainability transition impacts The preceding literature review highlights that the MLP is typically focused on socio-technical systems and largely ignores or downplays changes in power, politics, social organizations, environmental conditions, etc., except as outcomes (i.e. doesn’t see them as key components of the systems under examination themselves). Through an integration of environmental justice, theories of governance, actor relationships, social practice, procedural sustainability, and transformations, I am able to develop a richer set of descriptions of sustainability transition impacts than those provided by the transitions literature. An environmental justice approach provides a lens through which to capture changes in power dynamics and the equitable nature of transitions. Crucially, this lens starts to address the directionality of transition and provide a set of criteria to assess the sustainability of transitions. Governance literatures provide a complementary set of indicators of sustainability transition. Key elements that emerge here are how governance models, relationships and decision making are changing and how stakeholders are integrated into these decision-making processes. In this case these new governance models may surface (and in fact embody) social sustainability issues that go far beyond environmental impacts. A focus on the role of actors, how those roles change and interact to reduce barriers to, and promote, sustainability transition is also a key element in sustainability transition impacts. While the MLP structuralist view is useful to understand systemic change, actors in fact have agency at multiple levels of the system structure. Understanding how those actors are influencing and exercising agency to embed sustainable change is critical in capturing sustainability transition  63 impacts. However, agency is not solely an individual endeavour. Social practice and social learning foreground the role of the collective in both developing and embedding sustainable practices and narratives in systems. This normalization of sustainability may in fact be the most salient of all sustainability transition impacts. Changes in institutions, infrastructure, governance models, etc. are all most valuable if they support and embed more sustainable social practices in collectives. This is echoed in the transformations approach that argues for the need for changes in collective values and worldviews to achieve sustainability transition. These changes may be captured in the collective practices and narratives described in social practice and social learning literatures. I propose a development pathway approach as a framework with which to characterize transformational societal change and a means to capture contributions of an STE to such change.9 A development path was defined in an IPCC report as the complex array of technological, ec