Medicine, Faculty of
DSpace
UBCV
Hathaway, Brett Alexander
Doctor of Philosophy - PhD
University of British Columbia
Deficits in cost/benefit decision making are a critical risk factor for substance use disorder and behavioural addictions such as gambling disorder. Reward-concurrent cues may mediate this relationship, as such stimuli enhance risk preference and can potentiate cocaine intake in rats. To investigate the influence of cues on brain functioning and cost/benefit decision making, we utilize the rat Gambling Task. This task was designed such that maximal reward is attained by avoiding the high-risk, high-reward options and instead favouring the options associated with lower per-trial gains, as they feature less frequent and shorter time-out penalties. Adding reward-paired audiovisual cues to the task leads to greater risky choice on average. As inflexible pursuit of rewards is a critical component of drug and behavioural addictions, I tested whether these cues can also induce inflexibility in choice patterns. Results showed that win-paired cues but not loss-paired or randomly occurring cues drive risky choice as well as inflexibility in responding following shifts in reinforcer value. I also investigated potential neurobiological mechanisms of the impact of cues on decision making. I showed that altering serotonergic signaling in the lateral OFC can both ameliorate cue-induced risky choice and restore sensitivity to reinforcer devaluation. Furthermore, I showed that altering lateral OFC activity differentially affects choice patterns during early task learning on the uncued versus cued rGT, further demonstrating that OFC output is substantially impacted by the inclusion of cues on the task. I also showed that dopamine activity in the dorsal striatum is involved in the development and performance of choice patterns on the cued rGT, at least in females; inhibiting dopamine pathways projecting to the dorsal striatum resulted in an initial increase in risky choice followed by a subsequent long-term improvement in decision making. Thus, the role of dopamine in guiding choice towards versus away from risky options on the cued rGT shifts over time. Overall, these results shed light on the interaction between reward-paired cues, risky choice, and inflexible responding, thus providing insight into cue-induced compulsive risky behaviours that underlie both substance use disorder and gambling disorder.
COGNITIVE AND NEURAL UNDERPINNINGS OF THE INTERACTION BETWEEN REWARD-PAIRED CUES, RISKY CHOICE, AND COGNITIVE FLEXIBILITY by Brett Alexander Hathaway B.A. (Hon), The University of British Columbia, 2017 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Neuroscience) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) April 2023 © Brett Alexander Hathaway, 2023 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the dissertation entitled: Cognitive and neural underpinnings of the interaction between reward-paired cues, risky choice, and cognitive flexibility submitted by Brett Alexander Hathaway in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Neuroscience Examining Committee: Dr. Catharine Winstanley, Professor, Psychology, UBC Supervisor Dr. Rebecca Todd, Associate Professor, Psychology, UBC Supervisory Committee Member Dr. Manu Madhav, Assistant Professor, Biomedical Engineering, UBC Supervisory Committee Member Dr. Lynn Raymond, Professor, Psychiatry, UBC University Examiner Dr. Dale Griffin, Professor, Marketing and Behavioural Science, UBC University Examiner Additional Supervisory Committee Members: Dr. Christian Schütz, Associate Professor, Psychiatry, UBC Supervisory Committee Member iii Abstract Deficits in cost/benefit decision making are a critical risk factor for substance use disorder and behavioural addictions such as gambling disorder. Reward-concurrent cues may mediate this relationship, as such stimuli enhance risk preference and can potentiate cocaine intake in rats. To investigate the influence of cues on brain functioning and cost/benefit decision making, we utilize the rat Gambling Task. This task was designed such that maximal reward is attained by avoiding the high-risk, high-reward options and instead favouring the options associated with lower per-trial gains, as they feature less frequent and shorter time-out penalties. Adding reward-paired audiovisual cues to the task leads to greater risky choice on average. As inflexible pursuit of rewards is a critical component of drug and behavioural addictions, I tested whether these cues can also induce inflexibility in choice patterns. Results showed that win-paired cues but not loss-paired or randomly occurring cues drive risky choice as well as inflexibility in responding following shifts in reinforcer value. I also investigated potential neurobiological mechanisms of the impact of cues on decision making. I showed that altering serotonergic signaling in the lateral OFC can both ameliorate cue-induced risky choice and restore sensitivity to reinforcer devaluation. Furthermore, I showed that altering lateral OFC activity differentially affects choice patterns during early task learning on the uncued versus cued rGT, further demonstrating that OFC output is substantially impacted by the inclusion of cues on the task. I also showed that dopamine activity in the dorsal striatum is involved in the development and performance of choice patterns on the cued rGT, at least in females; inhibiting dopamine pathways projecting to the dorsal striatum resulted in an initial increase in risky choice followed by a subsequent long-term improvement in decision making. Thus, the role of dopamine in guiding choice towards versus away from risky options on the cued rGT shifts over time. Overall, these results shed light iv on the interaction between reward-paired cues, risky choice, and inflexible responding, thus providing insight into cue-induced compulsive risky behaviours that underlie both substance use disorder and gambling disorder. v Lay Summary The lights and sounds of a casino are exciting and increase enjoyment. However, these stimuli, termed cues, can cause intense cravings in those with pathological gambling. I investigated the impact of cues on decision making using the rat Gambling Task. In this task, most rats learn to favour the options associated with low reward (1-2 sugar pellets) and low risk (shorter and less frequent time-out penalties), which maximize reward earned over time. Pairing cues with reward delivery results in more rats favouring the disadvantageous high-risk options. I discovered that adding these cues specifically to winning trials not only increases risky choice, but also decreases flexibility in choice patterns. I also showed that experimental manipulation of the orbitofrontal cortex or the nigrostriatal dopamine pathway could reverse the negative effect of cues on risky decision making, albeit at different timepoints, suggesting there are nuanced mechanisms of cue-induced risky choice across task learning. vi Preface The candidate and Dr. Winstanley designed Experiment 1 (Chapter 3). The candidate, Salwa B. A. Malhas, Kelly M. Hrelja, and Cailean B. W. Harris collected the behavioural data and performed the reinforcer devaluation test. The candidate conducted the baseline data analyses. The candidate, Dexter R. Kim, and Dr. Angela Langdon analyzed task learning dynamics using reinforcement learning models designed by Dr. Langdon. The candidate interpreted the results and wrote the chapter. Dr. Langdon and Dr. Winstanley reviewed and revised the chapter. A version of Experiment 2 (Chapter 4) has been previously published in manuscript form: Hathaway, B. A., Schumacher, J. D., Hrelja, K. M., and Winstanley, C. A. (2021). Serotonin 2C antagonism in the lateral orbitofrontal cortex ameliorates cue-enhanced risky choice and restores sensitivity to reinforcer devaluation in male rats. eNeuro, 8, ENEURO.0341-21.2021. The candidate, Jackson Schumacher, and Dr. Winstanley designed the experiment. The candidate collected all behavioural data, prepared the drugs for administration, sectioned brains, and performed the histological analysis. The candidate and Jackson Schumacher performed surgeries and conducted the microinfusion procedures, with technical support from Kelly M. Hrelja. The candidate analyzed the data, interpreted the results, and wrote the chapter. Dr. Winstanley reviewed and revised the chapter. The candidate and Dr. Winstanley designed Experiment 3 (Chapter 5). The candidate, Yi Qing Zhao, and Elizabeth Chang prepared the drug, filled syringes, and collected behavioural data. The candidate performed the injections, viral infusion surgeries, and histological analysis. Kelly M. Hrelja assisted with injections and surgeries. The candidate and Yeo Soo Seo sectioned vii brains. The candidate analyzed the data, interpreted the results, and wrote the chapter. Dr. Winstanley reviewed and revised the chapter. The candidate and Dr. Winstanley designed Experiment 4 (Chapter 6). Sukhbir Kaur bred and genotyped the rats, with assistance from the candidate. The candidate and Andrew Li collected behavioural data. The candidate, Mason M. Silveira, and Melanie Tremblay performed the viral infusion surgeries. The candidate, Andrew Li, Melanie Tremblay, and Mason M. Silveira prepared the drug and performed injections. Hannah G. Brodie provided technical support during surgeries and injections. The candidate, Yeo Soo Seo, and Richard Zhong sectioned brains. The candidate performed the histological analysis, analyzed the data, interpreted the results, and wrote the chapter. Dr. Cole Vonder Haar and Dr. Craig Hutton provided input on regression analyses. Dr. Winstanley reviewed and revised the chapter. All animal testing was conducted in accordance with the Canadian Council of Animal Care and all experimental protocols were approved by the Animal Care Committee of the University of British Columbia, certificate numbers A17-0022, A21-0021 (experimental protocols), and A19-0014 (breeding protocol). Adeno-associated virus delivery was performed in accordance with approved standard operating procedure ACC WINSTANLEY CAW116. All procedures are standard for working with rAAVs classified as Risk Group 1, and have been approved by the UBC Biosafety Committee (protocol B15-0027). viii Table of Contents Abstract ......................................................................................................................................... iii Lay Summary .................................................................................................................................v Preface ........................................................................................................................................... vi Table of Contents ....................................................................................................................... viii List of Tables .............................................................................................................................. xiv List of Figures ...............................................................................................................................xv List of Abbreviations ................................................................................................................ xvii Acknowledgements .................................................................................................................... xix Chapter 1: General introduction ..................................................................................................1 1.1 Cues and addiction .......................................................................................................... 2 1.2 Cognitive flexibility ........................................................................................................ 6 1.2.1 Behavioural measures of cognitive flexibility ............................................................ 6 1.2.2 Neurobiological mechanisms of cognitive flexibility ................................................. 8 1.3 Goal-directed versus habitual control of behaviour ...................................................... 11 1.3.1 Behavioural measures ............................................................................................... 11 1.3.1.1 Model-based versus model-free reinforcement learning .................................. 13 1.3.2 Factors influencing goal-directed versus habitual control ........................................ 16 1.3.3 Neurobiological mechanisms of goal-directed versus habitual control .................... 19 1.3.3.1 Dorsomedial striatum ........................................................................................ 19 1.3.3.2 Orbitofrontal cortex .......................................................................................... 20 1.3.3.3 Dorsolateral striatum ......................................................................................... 22 ix 1.3.3.4 Dopamine .......................................................................................................... 23 1.3.3.5 Cholinergic interneurons ................................................................................... 26 1.3.3.6 Serotonin ........................................................................................................... 26 1.3.4 Goal-directed control and habit formation: Sequential or parallel processes? ......... 27 1.3.5 Compulsivity versus impulsivity .............................................................................. 29 1.3.6 The role of habits in compulsive drug use/gambling ................................................ 31 1.4 Animal models that assess the impact of cues on behaviour ........................................ 33 1.4.1 The rat Gambling Task ............................................................................................. 38 1.4.2 Neurobiological mechanisms of cue-mediated behaviours ...................................... 39 1.5 Sex differences in relevant cognitive processes ............................................................ 41 1.6 Insights from clinical populations ................................................................................. 44 1.6.1 Disordered eating ...................................................................................................... 45 1.6.2 Obsessive compulsive disorder ................................................................................. 46 1.6.3 Parkinson’s disease ................................................................................................... 46 1.7 Summary and objectives ............................................................................................... 48 Chapter 2: General methods .......................................................................................................51 2.1 Subjects ......................................................................................................................... 51 2.2 Behavioural apparatus ................................................................................................... 51 2.3 Behavioural testing ....................................................................................................... 52 2.3.1 Pre-training ............................................................................................................... 52 2.3.2 The rat Gambling Task ............................................................................................. 53 2.4 Behavioural measures ................................................................................................... 57 2.5 Data analysis ................................................................................................................. 57 x Chapter 3: Audiovisual cues must be predictable and win-paired to drive risky choice ......60 3.1 Introduction ................................................................................................................... 60 3.2 Additional methods ....................................................................................................... 61 3.2.1 Task variants ............................................................................................................. 61 3.2.2 Reinforcer devaluation .............................................................................................. 64 3.2.3 Hierarchical modeling of learning from wins and losses .......................................... 64 3.3 Results ........................................................................................................................... 68 3.3.1 Choice analysis ......................................................................................................... 68 3.3.2 Premature responding ............................................................................................... 73 3.3.3 Other variables .......................................................................................................... 73 3.3.4 Reinforcer devaluation .............................................................................................. 75 3.3.4.1 Reinforcer devaluation: other variables ............................................................ 77 3.3.5 Analysis of task learning using RL models .............................................................. 77 3.3.5.1 Model comparison ............................................................................................ 77 3.3.5.2 Parameter comparison for the two best-fitting models ..................................... 81 3.4 Discussion ..................................................................................................................... 87 Chapter 4: Serotonin 2C antagonism in the lateral orbitofrontal cortex ameliorates cue-enhanced risk preference and restores sensitivity to reinforcer devaluation .........................93 4.1 Introduction ................................................................................................................... 93 4.2 Additional methods ....................................................................................................... 95 4.2.1 Surgery ...................................................................................................................... 95 4.2.2 Drug preparation ....................................................................................................... 95 4.2.3 Microinfusion procedure ........................................................................................... 95 xi 4.2.4 Reinforcer devaluation .............................................................................................. 96 4.2.5 Histology ................................................................................................................... 96 4.3 Results ........................................................................................................................... 97 4.3.1 Cannulae placements ................................................................................................ 97 4.3.2 Baseline behaviour .................................................................................................... 99 4.3.3 Microinfusions into the lateral OFC ....................................................................... 101 4.3.3.1 Choice ............................................................................................................. 101 4.3.3.2 Premature responding and other variables ...................................................... 101 4.3.3.3 Reinforcer devaluation .................................................................................... 103 4.3.4 Microinfusions into prelimbic region ..................................................................... 107 4.3.4.1 Choice ............................................................................................................. 107 4.3.4.2 Premature responding and other variables ...................................................... 107 4.4 Discussion ................................................................................................................... 109 Chapter 5: Modulating activity in the lateral orbitofrontal cortex differentially impacts risky decision making in the presence versus absence of reward-paired cues .....................117 5.1 Introduction ................................................................................................................. 117 5.2 Additional methods ..................................................................................................... 118 5.2.1 Viral vector injections ............................................................................................. 118 5.2.2 CNO preparation and administration ...................................................................... 119 5.2.3 Histology ................................................................................................................. 119 5.3 Results ......................................................................................................................... 120 5.3.1 DREADDs expression ............................................................................................ 120 5.3.2 Task learning ........................................................................................................... 122 xii 5.3.2.1 Choice ............................................................................................................. 122 5.3.2.2 Other variables ................................................................................................ 125 5.3.3 Task performance.................................................................................................... 127 5.4 Discussion ................................................................................................................... 129 Chapter 6: Dopamine activity in the nigrostriatal pathway alters cue-induced risky choice patterns in female rats ...............................................................................................................136 6.1 Introduction ................................................................................................................. 136 6.2 Additional methods ..................................................................................................... 137 6.2.1 Viral vector injections ............................................................................................. 137 6.2.2 CNO preparation and administration ...................................................................... 138 6.2.3 Histology ................................................................................................................. 138 6.3 Results ......................................................................................................................... 139 6.3.1 DREADDs expression ............................................................................................ 139 6.3.2 Inhibition/activation of nigrostriatal dopamine neurons during task acquisition ... 141 6.3.2.1 Choice ............................................................................................................. 141 6.3.2.2 Other variables ................................................................................................ 145 6.3.3 Inhibition/activation of nigrostriatal dopamine neurons during task performance . 149 6.4 Discussion ................................................................................................................... 151 Chapter 7: General discussion ..................................................................................................158 7.1 Summary of experimental findings ............................................................................. 158 7.2 Goal-directed versus habitual control of decision making on the rGT ....................... 161 7.3 Theoretical implications and future directions ........................................................... 167 7.3.1 Neurobiological mechanisms of the impact of cues on decision making ............... 167 xiii 7.3.2 Cognitive mechanisms of the impact of cues on decision making ......................... 169 7.4 Limitations and critical considerations ....................................................................... 171 7.5 Concluding remarks .................................................................................................... 173 References ...................................................................................................................................175 xiv List of Tables Table 2.1: Auditory and visual cues used on the cued rGT .......................................................... 56 Table 3.1: Summary of RL models ............................................................................................... 66 Table 3.2: P1-P4 choice comparisons for rGT variants ................................................................ 72 Table 3.3: Risk score comparisons for rGT variants .................................................................... 72 Table 3.4: Premature responding comparisons for rGT variants .................................................. 74 Table 3.5: Collect latency comparisons for rGT variants ............................................................. 74 Table 3.6: Devaluation in risk-preferring rats: P1-P4 choice ....................................................... 76 Table 3.7: Devaluation in risk-preferring rats: Risk score ............................................................ 76 Table 4.1: Intra-lOFC RS 102221: Other task variables ............................................................. 102 Table 4.2: Reinforcer devaluation: P1-P4 choice ....................................................................... 106 Table 4.3: Reinforcer devaluation: Other task variables ............................................................ 106 Table 5.1: Performance phase: Other task variables ................................................................... 128 Table 6.1: Nigrostriatal manipulation during task performance: Other variables ...................... 150 xv List of Figures Figure 2.1: The rat Gambling Task. .............................................................................................. 55 Figure 3.1: Variants of the rat Gambling Task ............................................................................. 63 Figure 3.2: Differences in baseline performance between task variants ...................................... 70 Figure 3.3: Effects of reinforcer devaluation on choice preference .............................................. 76 Figure 3.4: Differences in WAIC values compared to independent model .................................. 78 Figure 3.5: Simulated risk score using model parameter estimates .............................................. 80 Figure 3.6: Group-level posterior estimates of independent cost model parameters .................... 83 Figure 3.7: Group-level posterior estimates of scaled + offset cost model parameters ................ 84 Figure 3.8: Subject-level parameter estimates .............................................................................. 86 Figure 4.1: Histological analysis of cannulae implantation .......................................................... 98 Figure 4.2: Choice preferences at baseline ................................................................................. 100 Figure 4.3: Intra-lOFC RS 102221: P1-P4 choice ...................................................................... 102 Figure 4.4: Intra-lOFC RS 102221: Reinforcer devaluation ...................................................... 105 Figure 4.5: Intra-PrL RS 102221: P1-P4 choice ......................................................................... 108 Figure 5.1: Histological analysis of DREADDs expression in the lateral OFC ......................... 121 Figure 5.2: Learning phase: P1-P4 choice .................................................................................. 124 Figure 5.3: Learning phase: Other task variables ....................................................................... 126 Figure 5.4: Performance phase: P1-P4 choice and premature responding ................................. 128 Figure 6.1: Histological analysis of DREADDs expression in the SNc ..................................... 140 Figure 6.2: Nigrostriatal inhibition during task learning: Choice ............................................... 143 Figure 6.3: Nigrostriatal activation during task learning: Choice ............................................... 144 xvi Figure 6.4: Nigrostriatal manipulation during task learning: Premature responding and latencies..................................................................................................................................................... 147 Figure 6.5: Nigrostriatal manipulation during task learning: Trials and omissions ................... 148 Figure 6.6: Nigrostriatal manipulation during task performance: Choice .................................. 150 xvii List of Abbreviations 5CSRT – 5-choice serial reaction time 5-HT - serotonin BLA – basolateral amygdala CIN – cholinergic interneuron CS – conditioned stimulus DLS – dorsolateral striatum DMS – dorsomedial striatum DREADD – designer receptor exclusively activated by designer drugs DT5 – discrete trials fixed ratio-5 EGM – electronic gaming machine HDI – highest density interval HSD – honest significant difference IDED – intradimensional/extradimensional IGT – Iowa gambling task LDW – losses disguised as wins LiCL – lithium chloride MSN – medium spiny neuron OCD – obsessive compulsive disorder OFC – orbitofrontal cortex PFC – prefrontal cortex PIT – Pavlovian to instrumental transfer PrL – prelimbic region xviii rGT – rat Gambling Task RI – random interval RL – reinforcement learning RR – random ratio SNc – substantia nigra pars compacta VTA – ventral tegmental area WAIC – Watanabe-Akaike information criterion WCST – Wisconsin card sorting task xix Acknowledgements Firstly, I’d like to thank Dr. Winstanley for her incredible mentorship and support throughout the years. I’m very grateful to have done my PhD in your lab, and truly appreciate your leadership style, support and care for your students, and enthusiasm for science. I can’t believe it’s gone by so fast! Secondly, I’d like to thank my parents for their unending encouragement, guidance, love, and the values they instilled in me growing up. I could not have accomplished so much without you. Thank you to my brother Brandon for your curiosity in my work, good laughs, and great conversations. And to our cat Monty – you’re the best thesis writing buddy a person could ask for. I am extremely grateful for my close friends in Nanaimo and Vancouver. I truly treasure all the great memories and the joy and stress relief you provided me. Thank you for listening while I vented about rats and for reminding me what’s important in life. Next, I’d like to thank past and current members of the Winstanley lab and BNS area. To Mason, Jacquie, Melanie, and Michael – you set the standards high and taught me so much. I am indebted to Kelly, Jackson, and Hannah whose efforts were essential to the completion of these projects. Thank you also to the undergraduate research assistants I worked with – you all made unique and important contributions to this work. Lastly, thank you to the rats, the animal care technicians who cared for them, my supervisory committee, and to CIHR for awarding me funding. I’d also like to acknowledge that this work took place on the traditional, ancestral, and unceded land of the xʷməθkʷəy̓əm (Musqueam), sə̓lílwətaʔɬ/Selilwitulh (Tsleil-Waututh) and Sḵwx̱wú7mesh (Squamish) Peoples. I am grateful for their stewardship of this land for thousands of years. 1 Chapter 1: General introduction The brain is geared towards associative learning. This is particularly true for reward seeking – remembering salient stimuli that signal reward availability is very helpful for obtaining those rewards again in the future. This clearly serves an adaptive purpose. However, these stimulus-reward associations can exert a high degree of control over behaviour, even in the face of negative consequences and/or little to no earned reward. Ivan Pavlov (1927) was the first to discover that salient stimuli, also called cues, that predict reward can trigger the same response as the reward, which he termed a conditioned stimulus (CS) triggering a conditioned response. A considerable amount of research has been conducted on the role of such cues in addiction. One prominent theory is incentive sensitization (Berridge & Robinson, 2016). According to this theory, drug-paired cues acquire incentive salience, meaning that exposure to these cues triggers intense cravings that can motivate subsequent drug-seeking behaviour. Craving, or “wanting” of the drug, is separated from the drug’s pleasurable effects, or “liking”, such that drug tolerance or negative consequences of drug taking do not reduce cue-triggered drug seeking. Drug seeking may initially be a motivated action with the goal of obtaining the drug in mind, but eventually becomes an automatic behaviour, or habit. Indeed, Everitt and Robbins (2016) have theorized that the basis of drug addiction is a transition from goal-directed drug seeking, to habitual use, to compulsive seeking that is highly resistant to associated consequences. Goal-directed behaviour is based on having a specific outcome in mind, as well as knowledge about the actions needed to obtain it (Balleine & Dickinson, 1998). Conversely, a habit is an ingrained response pattern triggered by a cue rather than desire for the outcome. Habits can certainly be adaptive, as they free up higher-level cognitive processes from 2 overseeing routine patterns of behaviour. However, habits are maladaptive when they become insensitive to negative outcomes, such as those incurred by repeated drug use. As will be discussed below, these processes are also involved in gambling disorder. This disorder is characterized by persistent problematic gambling behaviour despite personal, financial, or health-related consequences (Ladouceur et al., 1994). About 1-2% of the population suffers from this disorder (Wolfling et al., 2011). Cue-driven behaviours play a prominent role in problem gambling, due to the salient lights and sounds associated with casinos and other gambling environments. There is considerable disagreement in the literature as to whether these cue-driven behaviours are caused by impairments in goal-directed control, enhanced habit mechanisms, or both. Increasing our understanding of how these two processes contribute to the ability of cues to spur maladaptive and persistent pursuit of rewards despite high costs may lead to critical developments in addiction treatment. Thus, the goal of my research is to investigate how reward-paired cues impact cost/benefit decision making, with a particular focus on goal-directed versus habitual control of behaviour. 1.1 Cues and addiction The ability of drug-paired cues to induce cravings and subsequent drug seeking has been demonstrated in both humans and rodents (Pitchers et al., 2018). The control of cues over behaviour presents a critical difficulty in addiction recovery. Indeed, even after prolonged abstinence, exposure to drug-related cues can trigger relapse (Pitchers et al., 2018). This process is thought to be dependent on the sensitization of dopaminergic reward circuitry (Jasinska et al., 2014). Given that drugs of abuse drive dopamine activity, one may conclude that such drugs 3 “hijack” the reward system to motivate an individual to seek drugs, triggered by exposure to a drug-associated cue. Evidence suggests that a similar phenomenon is present in individuals with gambling disorder. Increased reactivity to gambling-related cues has been observed in those with gambling disorder compared to healthy controls in regions related to motivation and visual processing (Goudriaan et al., 2010). This effect was replicated by another group, who additionally showed that increased cue reactivity was specific to gambling cues and not food cues (Limbrick-Oldfield et al., 2017). Additionally, cue reactivity in the insula and ventral striatum, as measured by fMRI signals, was positively correlated with subjective reports of craving (Limbrick-Oldfield et al., 2017). A negative correlation was observed between corticostriatal connectivity and reported cravings, indicating impaired top-down control (Limbrick-Oldfied et al., 2017). It has also been shown that cue-induced gambling behaviour is an important distinguishing factor between those with gambling disorder and healthy controls (Genauck et al., 2020). Thus, cue reactivity is not merely driven by increased dopamine activity due to the consumption of drugs of abuse, and is a common feature of both substance use disorder and gambling disorder. While reward-paired cues can trigger craving and subsequent reward seeking, it is not clear how such cues may influence decisions to seek rewards despite large financial and personal consequences. Decision-making deficits are present in both those with substance use disorder and gambling disorder. The Iowa Gambling Task (IGT) has been used to characterize such deficits (Bechara et al., 1994). In this task, participants choose from four decks, each with different probabilities and magnitude of wins and losses. Participants must learn to avoid the two decks offering big per-trial gains, as the losses are disproportionately large, ultimately resulting in a net loss of points. Instead, the optimal strategy is to choose from the two decks that feature 4 small per-trial gains but result in better long-term payout, due to the smaller losses. Most healthy participants are able to determine the optimal strategy and tend to favour the advantageous decks after about 100 trials (Bechara et al., 1994; Bull et al., 2015). Conversely, participants with either gambling or substance use disorder tend to maintain choice of disadvantageous decks, exhibiting a failure to learn from the frequent losses (Leeman & Potenza, 2011). Deficits on this task are even more severe in gamblers who prefer electronic gaming machines (EGMs; Goudriaan et al., 2005). EGMs may be a particularly addictive form of gambling. Salient reward-related cues feature prominently in EGMs, which are specifically designed to encourage excessive gambling (Griffiths, 1993). In particular, the lights and sounds associated with wins lead to an increased state of immersion and heightened physiological arousal (Dixon et al., 2014; Murch et al., 2017). EGMs with sound and faster speed of play incur higher ratings of enjoyment and excitement from those with gambling disorder (Loba et al., 2001). Healthy participants prefer machines with casino-related cues paired with wins, and such cues enhance win memory (Spetch et al., 2020), as well as causing an overestimation of the number of wins (Dixon et al., 2014). Thus, at least in the context of EGMs, win-related cues can impact cognitive processes such as memory of outcomes and feedback processing. It is important to note that sensitivity to cues in the context of addiction differ between men and women. In animal models of drug seeking, female rodents show a greater propensity to incentive sensitization following intermittent access to cocaine (Kawa & Robinson, 2019). A phenomenon referred to as “telescoping” has also frequently been reported, in which women’s first experiences with drugs occurs at a later age but they then escalate to dependence at a quicker rate compared to men (Becker & Koob, 2016). Differences in the development of 5 gambling disorder have also been noted; women are more likely to develop gambling disorder to relieve negative affect and stress, whereas men are more likely to gamble due to high levels of risk taking and sensation seeking (Bos et al., 2013). In addition, a later age of onset but more rapid escalation has been observed in women with gambling disorder compared to men (Bos et al., 2013). Thus, while men are at slightly higher risk of developing substance use disorder or gambling disorder (Becker & Koob, 2016; Bos et al., 2013), there are clear differences in the acquisition of these disorders between men and women. Thus, understanding sex differences in light of the impact of cues on decision making is an important area of study. In addition to deficits in cost/benefit decision making, cognitive inflexibility is also associated with substance use and gambling disorders. Those with gambling disorder exhibit deficits in their ability to flexibly adapt to shifts in action-outcome contingencies and in attentional set-shifting (Leeman & Potenza, 2011; Goudriaan, 2020). Across these tasks, those with gambling disorder show a tendency towards compulsive responding (van Timmeren et al., 2018). Intuitively, parallels could be drawn between these behaviours measured by laboratory tasks and compulsive gambling despite large financial losses. Indeed, animal models of compulsive reward seeking despite negative consequences (such as foot shock) are a major component of addiction literature (Everitt & Robbins, 2016). In this framework, compulsive seeking is viewed as a particularly entrenched and inflexible habit, in which actions are decoupled from the value of the goal. Interestingly, among clinical populations, perseverative responding on tasks measuring flexibility is more reliably found in those with gambling disorder than substance use disorder (Leeman & Potenza, 2011; Jara-Rizzo et al., 2020). Furthermore, among those with gambling disorder, a tendency towards perseverative responding is predictive of treatment dropout and subsequent relapse (Goudriaan, 2020). Given the impact of reward-6 paired cues on motivated behaviours, whether such cues can also mediate inflexibility in decision making is of particular interest. The rat Gambling Task (rGT) is an animal model based on the IGT that can assess the impact of reward-paired cues on both cost/benefit decision making and inflexibility in choice patterns (Zeeb et al., 2009). As in the IGT, optimal performance on this task is attained by avoiding the two high-risk, high-reward options and instead favouring the low-risk options associated with lower per-trial gains. The addition of reward-paired audiovisual cues leads to a higher proportion of rats establishing a disadvantageous risky decision-making profile (Barrus & Winstanley, 2016). A similar effect of reward-paired cues on risky decision making has been observed in humans (Cherkasova et al., 2018). In addition to enhanced risk preference, reward-concurrent cues may also lead to inflexibility in choice patterns. However, the potential interactions between cues, risky choice, and flexible behaviour have not been sufficiently characterized. The experiments included in this dissertation were specifically designed to probe this relationship. 1.2 Cognitive flexibility 1.2.1 Behavioural measures of cognitive flexibility Cognitive flexibility involves the monitoring of predicted outcomes associated with actions that are modified accordingly when changes in the environment are detected. Several different cognitive processes fall under the umbrella of cognitive flexibility and can be separated experimentally. Sensitivity to changes in action-outcome contingencies is one such aspect. This relies on both knowledge of the causal relationship between actions and their associated outcomes, as well as the ability to recognize when either changes to the probability of the outcome or omission of the outcome occur. Reversal learning tasks are used to assess this 7 (Izquierdo et al., 2017). In these tasks, participants must learn to initially select an option that offers reward either 100% of the time or with a higher probability (usually 80%) over a second non-reward or low-probability (usually 20%) option. The contingencies then switch, and participants must learn based on feedback alone to switch their choice to the other option. Extinction tests, in which reward delivery is omitted following a previously rewarded instrumental action, are also a measure of this type of cognitive flexibility. Another aspect of cognitive flexibility is set shifting, in which an individual must shift their attention from one type of stimulus (called a set) to another. This differs from reversal learning in that it requires more abstract, rule-based learning. It involves multiple cognitive processes, such as arbitrating between competing responses, error detection, and tracking reinforcement rates. Tasks measuring set shifting are frequently used to assess executive dysfunction among clinical populations (Bissonnette et al., 2013). It is commonly measured with the Wisconsin Card Sorting Task (WCST; Grant & Berg, 1948). In this task, participants are asked to match stimulus cards based on a dimension that is not explicitly told to them (such as colour, shape, or number), and must be determined based on feedback about correct or incorrect matching. Once the initial rule is learned, the sorting rule changes without notice, and participants must recognize the rule change and learn to sort according to a different dimension. Thus, this task measures the ability of the participant to disengage their attention from a certain stimulus set (e.g., shape) and instead attend to a different set (e.g., colour). Deficits on the WCST in individuals with gambling disorder have been previously reported, specifically in the number of trials to learn the rule, number of categories completed, and perseverative responding to a previous rule (Goudriaan et al., 2006; Rugle & Melamed, 1993; Marazziti et al., 2008). 8 The intradimensional/extradimensional (IDED) set shifting task further divides set shifting into two categories. Stimuli initially only differ on one dimension, and participants must learn the correct rule (i.e., selecting green lines instead of orange lines). Initial rule changes are intradimensional – the same dimension is reinforced but the exemplar changes (i.e., shift from green over orange to blue over yellow). Later, extra-dimensional shifts are introduced and subjects must switch the dimension they are paying attention to (e.g., green lines to orange triangles). Individuals with problem gambling show poorer performance on the IDED set-shifting task compared to healthy controls (Grant et al., 2010). Thus, together with reported deficits on the WCST task, impairments in updating attentional biases between stimulus sets is a frequently reported characteristic of gambling disorder. 1.2.2 Neurobiological mechanisms of cognitive flexibility The neurobiological mechanisms of extra-dimensional shifts and sensitivity to changing action-outcome contingencies can be dissociated. The medial prefrontal cortex (PFC) in rodents, corresponding to the dorsolateral PFC in primates, has been implicated in IDED set shifting, but not reversal learning (Dias et al., 1996a; Bissonnette et al., 2008). Specifically, the primate dorsolateral PFC and rodent medial PFC are involved in abstract rule learning, while the striatum controls stimulus-response patterns once the rule is established (Dias et al., 1996b; Antzoulatos & Miller, 2011, Birrell & Brown, 2000; Ghods-Sharifi et al., 2008). The anterior cingulate cortex monitors error rates and solves conflicting responses, particularly when cues are difficult to distinguish from each other (Rushworth et al., 2003; Ng et al., 2007). Thus, the medial PFC/dorsolateral PFC, striatum, and anterior cingulate cortex are critical to accurate performance on set shifting tasks. 9 Reversal learning, and tracking shifts in action-outcome contingencies more broadly, is instead dependent on the lateral portion of the orbitofrontal cortex (OFC; Izquierdo, 2017; Panayi & Killcross, 2018). Dalton et al. (2016) investigated the contributions of different PFC regions to the reversal learning task. Inactivating the medial OFC resulted in impairments both in the initial discrimination between the two options and subsequent reversals, indicating a role in value comparison based on past reward history. Inactivation of the lateral OFC specifically impaired reversal learning, whereas inactivation of the medial PFC did not, confirming the double dissociation of these regions’ contributions to contingency shifts and set shifts, respectively. These effects were only observed in a probabilistic variant of the task, not when outcomes were assured, suggesting differential recruitment of neural circuitry when outcomes are uncertain. This distinction has also been found for other regions such as the ventral striatum (Dalton et al., 2014). Neural recordings in the lateral OFC during the probabilistic reversal learning task shed light on the role this region plays in mediating flexible responding as action-outcome contingencies change (Amodeo et al., 2017). It was found that over half of neurons in the lateral OFC increased their activity following reward delivery. However, this responding was suppressed following wins on an incorrect trial (i.e., selection of the 20% probability lever). Thus, lateral OFC activity differentiates between rewarded optimal choice and rewarded suboptimal choice. Following a reversal, responses to wins on the now incorrect lever were initially high but gradually reduced as rats’ choice preferences switched to the other lever. This indicates that lateral OFC activity can mediate flexible responding by guiding behaviour towards the most rewarding option and away from suboptimal options based on shifting rates of feedback, as well as diminishing the impact of rewarded suboptimal choice. 10 Serotonergic activity in the OFC is particularly important for facilitating flexible responding (Clarke et al., 2004; Clarke et al., 2007; Boulougouris & Robbins, 2010; Clarke et al., 2011; Hatakama et al., 2022). Matias et al. (2017) conducted an in-depth analysis of responses in serotonergic neurons within the dorsal raphe nucleus to unconditioned and conditioned stimuli, and both positive and negative reversals (i.e., unexpected reward or unexpected omission). They discovered that serotonin neurons respond positively to unexpected outcomes regardless of valence. These neurons also responded to the CS in a manner that correlated with the value of the predicted reward (larger response to the CS that predicted large versus small reward). Thus, serotonin neurons showed evidence of both a traditional reward prediction error usually attributed to dopamine (Schultz et al., 1997) but also showed a broader range of responses to unexpected outcomes of positive, negative, or neutral valence. Serotonin and dopamine responses also exhibited different temporal dynamics following reversals, with serotonin showing a slower but more persistent shift in responding. From these results, the authors concluded that serotonin signals surprise, and could facilitate trial-by-trial unlearning of dopamine-driven learning when outcome predictions are no longer accurate. Thus, serotonin plays an integral role in informing the lateral OFC, and ostensibly other regions to which the dorsal raphe nucleus projects, when outcome predictions require updating. The dorsomedial subregion of the striatum (DMS) is also important for flexibly adapting to shifting reward probabilities (Castañé et al., 2010; Schumacher et al., 2021; Young et al., 2022). Either excessive or reduced dopamine in the caudate of marmosets (corresponding to the DMS in rodents) can impact task performance on a reversal learning task; reduced dopamine leads to a general increase in errors following reversals and excess dopamine specifically leads to increased perseveration (Clarke et al., 2011). Similar effects of dopamine depletion have been 11 found in rodents (O’Neill & Browne, 2007). Additionally, individual differences in dopamine synthesis capacity influence reinforcement learning; high dopamine synthesis capacity leads to better reversal learning indicated by unexpected rewards versus punishments, whereas individuals with low dopamine synthesis capacity show better reversal learning indicated by unexpected punishments (Cools et al., 2009). This may line up with the finding of increased perseveration with excess caudate dopamine, as reversal learning tasks often involve reward omission, which could be considered an unexpected punishment (Clarke et al., 2011). Furthermore, accuracy on the reversal learning task has been shown to depend on the relative balance between striatal dopamine and serotonergic activity within the OFC (Groman et al., 2013). Thus, individual differences in basal levels of monoamines within the dorsal striatum and PFC play a critical role in cognitive flexibility. 1.3 Goal-directed versus habitual control of behaviour 1.3.1 Behavioural measures Goal-directed behaviour can be defined as deliberative action taken to obtain a reward. Responding in this context is controlled by representations of outcome value and knowledge of the contingencies between actions and desired outcomes (Balleine & Dickinson, 1998). Goal-directed behaviour relies on cognitive flexibility to ensure action-outcome contingencies are accurate. Conversely, habitual behaviour is defined as persistent responding that is insensitive to outcome devaluation or contingency degradation. Once triggered by a cue, the habit is completed without conscious oversight, often referred to as a stimulus-response pattern. In this sense, habitual actions are unrelated to the individual’s current goal, and the stimulus-response pattern is stored separately from information about the outcome. 12 Outcome devaluation is a common test to assess the level of goal-directed versus habitual control of behaviour. A considerable amount of research has been conducted on sensitivity to devaluation of outcomes, both in humans and animal models. Two manipulations are commonly used. The first is sensory-specific satiety (Hetherington & Rolls, 1996). In this paradigm, animals are sated on the outcome used to reinforce an instrumental action (often, ad libitum access to sucrose pellets used as the reinforcer). This decreases the value of the reinforcer and thus, if goal-directed control is maintained, animals should make fewer instrumental actions to obtain it. Results are usually compared to satiation on food not used as the reinforcer (e.g., grain pellets), to rule out a general satiety-induced decrease in motivation. Another commonly used manipulation is pairing reinforcer consumption with an injection of lithium chloride (LiCL), which causes gastrointestinal illness and should therefore devalue the reinforcer (Holland & Straub, 1979). Different behavioural mechanisms are thought to underlie these two manipulations, with sensory-specific satiety involving reduced motivation, and LiCL-induced illness based on associative learning (Schreiner et al., 2019). Behavioural insensitivity to devaluation following LiCL pairing may be more relevant to compulsivity, as both involve persistence in responding despite negative consequences paired with the outcome. In either case, insensitivity to devaluation is considered a hallmark of habitual control of behaviour, although impaired goal-directed control is an equally viable hypothesis. Another test of goal-directed versus habitual control is sensitivity to changes in action-outcome contingencies, which, as discussed in Section 1.2.1, is a major component of cognitive flexibility. This is often accomplished with omission, extinction, or degradation tests (Schreiner et al., 2019). Contingency degradation occurs when non-contingent rewards are delivered in equal amounts to contingent rewards – if the behaviour is goal-directed, animals should reduce 13 instrumental actions, as obtaining reward is not dependent on their actions. In extinction tests, no reward is delivered, and behaviours under goal-directed control should gradually stop. Lastly, in an omission test, reward is only delivered if no action is taken – thus, animals should learn to inhibit actions in order to earn reward. It has been noted that testing for habitual behaviour using these paradigms relies on a negative finding (i.e. lack of goal-directed control; Vandaele & Janak, 2017). As mentioned above, this makes it difficult to differentiate between enhanced habitual control and impairments to goal-directed processes. Thus, other behavioural measures of goal-directed versus habitual control have been developed. The slips-of-action task more readily measures goal-directed versus habitual control of behaviour, and is based on outcome devaluation (de Wit et al., 2012). Here, participants first learn a stimulus-response-outcome association (i.e., a key response to a yellow square earns points). Some actions are then devalued (key response to yellow square does not earn points), while others are not. Finally, a Go/No-go task is administered, in which participants should make a key response to valued but not devalued stimuli. To perform well on the Go/No-go task, participants must flexibly update outcome values in the previous stage and learn to suppress cue-triggered learned responses. The number of key responses to valued versus devalued stimuli gives a devaluation sensitivity index, which directly measures the relative balance of goal-directed versus habitual control. 1.3.1.1 Model-based versus model-free reinforcement learning Goal-directed behaviour may be characterized as model-based control (Daw et al., 2005; Dolan & Dayan, 2013). The concept of model-based control is based on reinforcement learning (RL) principles. RL algorithms assume that task structures can be broken down into states. Each state represents a specific segment of a task, in which certain actions can be taken. Performing an 14 action causes a transition to a different state. In RL modeling, a rule, or policy, is used to guide action selection, which is usually based on maximizing the probability of reward (Sutton & Barto, 1998). The value of each state, and estimated probability of reward associated with each action, is learned via feedback as an individual progresses through the states. Model-based RL is based on the creation of an internal model of the task environment that includes all possible future states and state transitions. To select an action, the individual performs a tree search of this model to identify the state transitions that maximize the probability of getting the desired outcome. This process is computationally expensive, but data efficient, as it requires minimal experience with different outcomes to build an accurate model. The individual can therefore react more quickly to changed circumstances such as devalued outcomes. Daw et al. (2005) used computational modeling of simulated data to demonstrate that model-based control of behaviour results in less uncertainty about action-outcome contingencies earlier in training, suggesting that this type of control is particularly beneficial during the early phases of task learning. The alternative to model-based RL is model-free RL. Here, action values in a given state represent the value of all possible future outcomes, without creating an internal model of those future states and the actions needed to transition through them (Daw et al., 2005; Dolan & Dayan, 2013). Reward predictions are updated iteratively through repeated experience, and prediction errors incrementally update the values of all preceding states. This process is not data efficient. The incremental nature of updates to preceding states following prediction errors means that it takes longer for individuals to adjust to changed circumstances such as outcome devaluation, as the devalued outcome must be experienced multiple times. Model-free RL is therefore computationally simpler than model-based RL, but slow to update with changes to 15 outcome value or action-outcome contingencies. As such, model-free behaviour is thought to represent habitual control (Daw et al., 2005). The two-step task was designed to measure the degree of model-based versus model-free control in human subjects (Daw et al., 2011). As the name suggests, this task features two stages. In the first stage, the participant chooses between two stimuli (e.g., apple and orange). Selection of either stimulus leads to the second stage, in which participants choose between two additional stimuli associated with their first-stage choice (e.g., apple leads to choice between cat and dog; orange leads to choice between horse and cow). This occurs in 70% of trials and is termed a common transition. Selection of a second-stage stimulus may be rewarded, with varying probabilities across stimuli and trials (e.g., selection of the cat is rewarded 50% of the time). In 30% of trials, selection of a stimulus in the first stage will lead to the two stimuli that are usually associated with the other first-stage stimulus (e.g., apple leads to choice between horse and cow). The behaviour of the participant after this rare transition is informative of the degree of model-based versus model-free control of action selection. If the participant has created an internal model of the task, then after earning a reward from a second-stage stimulus following a rare transition, they should switch their choice to the other first-stage stimulus that commonly transitions to the rewarding second-stage stimulus (i.e., apple  horse is rewarded, so the individual chooses orange in the first stage of the next trial). However, if the participant is employing a model-free strategy, then they would repeat their selection of the same first-stage stimulus after earning a reward, regardless of whether it was a common or rare transition (apple  horse is rewarded, and the individual picks apple again). This is because the actions leading to reward would all be increased in value, without a model of the probability of state transitions to 16 guide choice. Using the application of computational modeling techniques, the degree of model-based and model-free control can be estimated independently of each other. On this task, there is a high level of individual variability in the degree to which participants employ model-based and model-free strategies (Daw et al., 2011). The devaluation sensitivity index as measured by the slips-of-action task correlates with usage of a model-based strategy (Sjoerds et al., 2016). This indicates that model-based RL overlaps with other measures of goal-directed control. However, Sjoerds et al. (2016) note that the shared variance between the measures is low. Thus, sensitivity to devaluation and model-based RL may also pick up on distinct aspects of goal-directed control. Interestingly, the degree of model-free control in the two-step task does not correlate with the devaluation sensitivity index. Thus, these two tasks may be picking up on different aspects of habitual control. One hypothesis is that because the two-step task requires continual learning due to shifting reward probabilities of the second-stage stimuli across trials, it may be resistant to the formation of inflexible stimulus-response patterns (Sjoerds et al., 2016). Thus, the use of model-free RL on this task may not fit in with the traditional definition of habitual control of behaviour. 1.3.2 Factors influencing goal-directed versus habitual control Elements of the two-step task structure can influence the degree to which model-based and model-free strategies are employed. For example, model-based control can be boosted by fine tuning task parameters so that using a model-based strategy more clearly results in higher reward (Kool et al., 2016). Since creating an internal model of the task is computationally expensive, it is only worth it to employ this strategy when more reward can be earned by doing so. Indeed, putting cognitive demands on individuals during the two-step task results in a shift 17 towards model-free control (Otto et al., 2013), suggesting that expenditure of cognitive resources is a critical factor in model-based versus model-free RL. As alluded to above in the discussion of model-based strategies, goal-directed behaviour is advantageous early in task training, when deliberative effort and high cognitive control over actions is needed to efficiently navigate a new environment. Classic experiments on lever pressing have shown that instrumental actions are sensitive to devaluation early but not late in training (Adams, 1982). This has also been demonstrated on a T-maze that uses audio and tactile cues to instruct animals to choose the left or right arm of the maze to earn reward (Thorn et al., 2010). Additional evidence comes from the seeking-taking chained schedule task, in which animals must first make a response on the seeking lever to gain access to a drug administration lever (Olmstead et al., 2001). Extinction training on the drug-taking lever results in reduced responding on the seeking lever only during the early training period (Olmstead et al., 2001; Zapata et al., 2010). Conversely, extended training results in the development of habitual control. Lever pressing for a sucrose pellet reward, performance on the cued T-maze, and drug seeking in the seeking-taking procedure all become insensitive to devaluation after extended training (at least 10 additional sessions after the task is learned; Adams, 1982; Smith et al., 2012; Everitt & Robbins, 2016). This appears to be specific to certain types of instrumental actions, as lever pressing becomes insensitive to devaluation with overtraining but a chain pull does not (Faure et al., 2020). Interestingly, goal-directed control is also retained in more complex environments. Choice patterns on multi-arm tasks that feature at least two actions leading to two different outcomes remain goal-directed even with overtraining (Colwill & Rescorla, 1985). Additionally, the strength of the association between action and outcome can influence the degree to which 18 behaviour remains goal-directed despite extensive training. Lever pressing on a random ratio (RR) schedule of reinforcement is sensitive to devaluation despite lengthy training, whereas random interval (RI) schedules are not (Dickinson et al., 1983). This is hypothesized to be because the association between actions and outcomes is more readily apparent in RR schedules. RI schedules generate higher uncertainty about the relationship between actions and reward delivery and thus behaviours are more readily relegated to habitual control (Vandaele & Janak, 2018). Salient stimuli may also facilitate the development of inflexible habitual patterns. Fixed ratio schedules of reinforcement are rendered insensitive to devaluation if trials are bracketed by lever insertion and removal, on a task referred to as discrete trials fixed ratio-5 task (DT5; Vandaele et al., 2017). This trial bracketing may break down behaviours into clearly defined action sequences, which in turn enables the development of rigid stimulus-response patterns. By this process, salient cues may expedite the development of habitual control of behaviour. Motor skill learning may represent another form of habitual learning. Similar regions in the striatum mediate shifts from unskilled to skilled motor sequences (Durieux et al., 2012; Graybiel et al., 2015). There are also some key differences; for example, when learning a new skill, positive reinforcement does not always need to be present (Lerner, 2020). Additionally, automatized motor sequences can form to execute a series of actions, with the overall behavioural pattern still being sensitive to devaluation (Garr & Delamater, 2019). Thus, while the learning of action sequences may be an aspect of habit formation, not all learned action sequences are inflexible according to the classical definition of a habit. 19 1.3.3 Neurobiological mechanisms of goal-directed versus habitual control 1.3.3.1 Dorsomedial striatum The striatum is largely composed of medium spiny neurons (MSNs), which receive both dopaminergic input from the midbrain and glutamatergic input from the PFC(Graybiel et al., 2015). Reciprocal connections between the PFC and striatum via the thalamus have been traced in both humans and rodents (Alexander et al., 1986; Lehéricy et al., 2004). Multiple corticostriatal loops are present in the brain, with distinct differences in regional connectivity between the PFC, striatum, as well as other subcortical structures such as the amygdala and dopaminergic midbrain (Groenewegen et al., 1990; Yin et al., 2008). In particular, connectivity differs from the ventromedial to dorsolateral striatum (Burton et al., 2015; Sharpe et al., 2018). The ventral striatum is thought to mediate motivated behaviours and cue-reward associations, supported by dopaminergic input from the ventral tegmental area (VTA) as well as glutamatergic inputs from the anterior insula and OFC (Brog et al., 1993; Gorelova & Yang, 1997; Ikemoto, 2007). The DMS is innervated by dopamine neurons from the medial region of the substantia nigra pars compacta (SNc) as well as cortical input from the lateral OFC and medial PFC (Lerner et al., 2015; Burton et al., 2015). The dorsolateral striatum (DLS) receives dopaminergic input from the lateral SNc and cortical input from the lateral OFC, as well as sensorimotor regions such as the supplementary motor area (Gremel & Costa, 2013; Lerner et al., 2015; Burton et al., 2015). The DMS is thought to support goal-directed behaviour by maintaining new choice patterns both early in training and when contingencies shift, such as in reversal learning (described in section 1.2.2). Thus, the DMS mediates flexible responding across multiple measures of cognitive flexibility and goal-directed behaviour (Schreiner et al., 2019). Lesioning 20 the DMS during early training of instrumental actions results in the loss of goal-directed control, as measured by outcome devaluation and contingency degradation (Yin et al., 2005). On the seeking-taking task, similar results have been found following DMS lesions, in which removal of the taking lever does not reduce responding for the seeking lever, even early in training (Everitt & Robbins, 2013). The DMS also supports the learning of new skills. On the accelerating rotarod task, the DMS is highly engaged early in task training and reduces in activity over time, which is correlated with the emergence of skilled performance (Kupferschmidt et al., 2017). The DMS also plays a role in cue-guided decision making. The DMS is required for optimal choice between a small, certain reward or a large, probabilistic reward for which good or bad odds are signalled by instructive cues (Schumacher et al., 2021). On the cued T-maze task, the DMS exhibits the highest level of activity after the mice learned to respond to instructional audio cues and are subsequently being trained on a tactile version of the task, in which instructions to go down the left or right arm are indicated by different tactile sensations on the floor of the maze (Thorn et al., 2010). Activity in the DMS correlated with differences in performance between the audio and tactile cues. The authors hypothesized that this was indicative of each mouse’s ability to resolve conflicting demands in plasticity - consolidation of the audio version and new learning of the tactile version. Thus, the DMS plays a role both in the acquiring and consolidation of instrumental responses when cues are present to guide learning. 1.3.3.2 Orbitofrontal cortex The lateral OFC has been shown to support goal-directed behaviour, in conjunction with its previously described role in cognitive flexibility (Section 1.2.2). This region is particularly involved in model-based control of behaviour. Schoenbaum et al. (2016) describe in detail how the lateral OFC can estimate novel predictions based on inferred information from a complex 21 environment. Their conclusions were largely drawn from studies using the Pavlovian over-expectation task (Rescorla, 1970). In this task, two cues are independently associated with a small reward (e.g., one sucrose pellet). When the two cues are subsequently presented together as a compound stimulus, the animal summates the expected value from the two individual cues and the animal expects a large reward (e.g., two sucrose pellets). This is evidenced behaviourally by increased responding at the food cup compared to prior presentation of the single cues. This heightened responding is considered evidence of a novel prediction, as animals had not encountered the compound stimulus before this point. Additionally, a subsequent probe test in which each cue is presented on its own results in decreased levels of responding at the food cup compared to the first stage of training. As reward was not delivered following the compound stimulus, a decrease in expected value is transferred to each individual cue and the reward seeking response at the food cup is reduced. Both increased responding to the compound cue and subsequent reduced responding during the probe test have been shown to be dependent on lateral OFC activity (Takahashi et al., 2009; Takahashi et al., 2013). From these results, Schoenbaum et al. (2016) conclude that the OFC is required for the mental simulation of potential future outcomes. The lateral OFC receives extensive sensory information from subcortical regions (Wallis, 2007), and as such is well situated to incorporate the detection of salient stimuli into subsequent decision making. Indeed, this region is particularly involved when reward-predictive cues signal potential outcomes (Zeeb et al., 2011; Izquierdo, 2017). For example, the lateral OFC mediates choice between a small, immediate reward or a larger delayed reward, but only when cues are present to signal delays (Zeeb et al., 2010). Furthermore, lesion or chemogenetic inhibition of the lateral OFC impairs cue-induced reinstatement of cocaine, ethanol, and sucrose seeking (Fuchs et 22 al., 2004; Hernandez et al., 2020; Arinze & Moorman, 2020), indicating that this region facilitates the control of reward-paired cues over behaviour. The lateral OFC has also been shown to play a role in Pavlovian but not instrumental devaluation (Panayi & Killcross, 2018). Thus, this region plays an integral role in model-based behaviour and guiding subsequent actions based on available information, including salient cues in the environment. Exposure to drugs of abuse may interfere with this function, as Stalnaker et al. (2006) found that cocaine exposure impaired OFC signaling to cues predicting aversive outcomes, and after reversal of the cue-outcome associations, cue-related activity was slow to update in cocaine-exposed animals. As such, OFC dysfunction may be a critical component of substance use disorder. Evidence suggests that the DMS and OFC work together to support goal-directed behaviour. The lateral OFC sends dense projections directly to the DMS (Schilman et al., 2008). By measuring activity within the OFC and DMS during training on the RR versus RI reinforcement schedules within the same animals, Gremel & Costa (2013) showed that differences in activity between these two schedules were correlated with the level of goal-directed control of behaviour, as measured by sensitivity to sensory-specific satiety. Furthermore, experimental manipulation of the lateral OFC activity using chemogenetics and optogenetics could shift the relative balance of goal-directed versus habitual control (Gremel & Costa, 2013). Thus, circuitry between the lateral OFC and DMS causally contributes to goal-directed control. 1.3.3.3 Dorsolateral striatum The DLS is thought to mediate habitual control of behaviour. Lesioning the DLS results in control of behaviour shifting to goal-directed systems. This has been found following outcome devaluation via sensory-specific satiety on RI schedules (Gremel & Costa, 2013) and responding 23 for a devalued sucrose reward that has been paired with LiCL after extended training (Yin et al., 2004). DLS lesion groups show reduced responding following these manipulations, whereas sham and DMS lesion groups do not. Similarly, on the seeking-taking paradigm, well-trained cocaine seeking depends on the DLS; lesioning of this region results in reinstatement of goal-directed control, as behaviour becomes sensitive to omission of the drug-taking lever (Zapata et al., 2010). Similar mechanisms of habit formation have been observed in humans (Tricomi et al., 2009). Based on these results, it is thought that the DMS controls behaviour in early training to support goal-directed learning of task contingencies and state space, and is subsequently shifted to the DLS to enact stimulus-response patterns that don’t require cortical input but are consequently inflexible in the face of changing contingencies or outcome value. In line with this, model-free RL is also suggested to be dependent on striatal circuitry, independent of the PFC (Daw et al., 2005). 1.3.3.4 Dopamine The nigrostriatal dopamine pathway has long been implicated in action initiation and the acquisition of motor skills (Yin et al., 2009; Jin & Costa, 2010; Thorn et al., 2010). In rats, optogenetic stimulation of nigral dopamine neurons results in vigorous but undirected movement, but no cue-directed approach, which is observed following stimulation of VTA dopamine neurons (Saunders et al., 2018). This pathway also contributes to higher-order cognitive functions such as decision making. There is now an established role of the dorsal striatum in reinforcement learning, particularly in learning the value of actions and their associated outcomes (Hollon et al., 2021, Ramayya et al., 2014). Dynamic changes in nigrostriatal dopamine activity causally contribute to action selection within the striatum (Howard et al., 2017). Furthermore, instrumental learning in the absence of a primary reinforcer 24 can be supported by self-stimulation of dopamine neurons in the SNc, indicating the role of this dopamine pathway in the learning of action sequences (Rossi et al., 2013). Reward prediction errors are classically thought to be carried by the VTA, in which unexpected reward induces burst firing, and omission of expected reward induces a pause (Schultz et al., 1997). Dopaminergic neurons in the medial SNc also show characteristics of prediction errors, and have distinct intrinsic properties and inputs from SNc neurons located more laterally (Bromberg-Martin et al., 2010; Lerner et al., 2015; Zhang et al., 2017). In both humans and rodents, neurons within the medial SNc increase their firing in response to appetitive events, whereas firing is inhibited in response to aversive events (Lerner et al., 2015; Zhang et al., 2017). Conversely, the lateral portion of the SNc tracks salience, as it is equally excited by appetitive and aversive events (Zhang et al., 2017). This medial to lateral gradient in function is preserved in the striatum, with the medial SNc innervating the DMS and the lateral portion innervating the DLS (Lerner et al., 2015). In correspondence with their projections to the DMS and DLS, dopamine activity within the medial and lateral SNc support goal-directed and habitual behaviour, respectively. In monkeys, it has been shown that both medial and lateral SNc (presumed dopaminergic) neurons respond to visual cues that predict reward delivery (Kim et al., 2015). However, medial SNc neurons primarily responded in a learning context (initial cue-outcome pairing, and value reversal), whereas lateral SNc neurons responded stably across contexts, including during passive cue presentation when no reward was expected. Thus, medial SNc neurons support goal-directed learning about cues, whereas lateral SNc neurons may encode the salience of predictive cues regardless of the current goal. Furthermore, activation of DMS-projecting dopamine neurons results in a decrease in grooming frequency and duration, considered a measure of 25 compulsive behaviour, as well as an increase in exploration in an open-field test (Casado-Sainz et al., 2021). Overtraining of lever pressing remains sensitive to outcome devaluation via sensory-specific satiety when the SNc to DLS pathway is lesioned (Faure et al., 2005). When the spatial locations of the optimal and risky options are switched on a gambling task similar to the rGT, inflexible rats (who perseverate on the same spatial location despite switched contingencies) have increased dopamine content in the DLS compared to rats who flexibly shift their choice patterns (Dellu-Hagedorn et al., 2018). Thus, dopaminergic activity in the DLS supports inflexible choice patterns and habitual control of behaviour. Transfer of control from goal-directed to habitual systems is traditionally thought to rely on dopaminergic transmission proceeding through a series of overlapping loops between the midbrain and striatum, starting in the VTA and ending in the DLS. This “ascending spiral” hypothesis originated from research on anatomical connections that followed this pattern (Haber et al., 2000). Based on this work, it was suggested that the nucleus accumbens could disinhibit DMS dopamine activity via projections to the SNr, and striatal projections from the DMS could subsequently disinhibit DLS dopaminergic neurons. Some subsequent research has supported this hypothesis. For example, activity in the nucleus accumbens core can modulate dopamine activity in the DLS during late-stage cocaine self-administration (Murray et al., 2012). However, in a recent review, Lerner (2020) posited several caveats to this hypothesis. Both inhibitory and disinhibitory circuitry is present between the striatum and dopaminergic midbrain. For example, inhibitory connections from the dorsal striatum comprise 50% of inputs to the SNc, and there is also a significant portion from the ventral striatum (Lerner et al., 2015; Beier et al., 2015). Additionally, lateral to medial projections in the dorsal striatum are particularly strong, indicating that information can flow in the opposite direction of the ascending spiral hypothesis 26 (Lerner et al., 2015). In support of this “backwards” flow of information, lesioning of SNc dopamine neurons increases extracellular dopamine levels in the ventral striatum (van Oosten et al., 2015). This could mediate a shift from habitual back to goal-directed control. Indeed, some evidence suggests that the SNc can signal when greater cognitive control is needed to solve response conflicts (Boehler et al., 2011), which may trigger a shift to goal-directed systems. 1.3.3.5 Cholinergic interneurons Local circuitry within the striatum may contribute to shifts in control between striatal subregions. Striatal interneurons can directly modulate the activity of MSNs, as well as glutamatergic cortical inputs and terminal dopamine release (Collins & Saunders, 2020). Cholinergic interneurons (CINs), comprising 2% of striatal neurons, have been a hot area of study recently. In the DMS, CINs support flexible responding during shifts in action-outcome contingencies (Ragozzino et al., 2009). Removing cholinergic release from CINs in the dorsal striatum leads to reduced dopamine release specifically in the DMS (Favier et al., 2020). Behaviourally, this results in insensitivity to outcome devaluation on the RR schedule of lever pressing as well as deficits in reversal learning. Furthermore, CINs in the DMS receive input about the current state from the OFC, thus representing information relevant to the current context (Stalnaker et al., 2016). Thus, CINs may coordinate the release of dopamine within the DMS to support goal-directed behaviours and cognitive flexibility. 1.3.3.6 Serotonin As discussed in Section 1.2.2, serotonergic activity, particularly in the lateral OFC, is critical to flexible responding when reward predictions and/or action-outcome contingencies shift by signaling unexpected outcomes (Clarke et al., 2011; Matias et al., 2017). Additionally, 27 serotonin is involved in the degree of habitual versus goal-directed control. Tryptophan depletion, which causes low serotonin in humans, results in enhanced habitual control on the slips-of-action task (Worbe et al., 2015). However, questions have been raised about the specificity of tryptophan depletion on depleting serotonin, as it can have other effects in the brain (Young, 2013). Thus, more research into whether low serotonin levels lead to enhanced habit formation is needed. 1.3.4 Goal-directed control and habit formation: Sequential or parallel processes? As discussed above, the DMS, in conjunction with the OFC, is thought to mediate goal-directed behaviour, while the DLS is responsible for habit formation. As instrumental actions are learned, a shift in control occurs from the DMS to DLS, in conjunction with a behavioural shift from flexible to inflexible responding in the face of value or contingency changes. However, some results provide evidence against the transition of control from the DMS to the DLS. On the DT5 task, in which lever insertion and removal bracket fixed interval schedule trials, DLS inactivation late in training only modestly affected behaviour, and the DMS remained consistently engaged across training (Vandaele et al., 2019). Similarly, in comparisons of the RR versus RI schedule tasks, both the DMS and DLS were responsive across training (Gremel & Costa, 2013). In a study examining the generalization of instrumental actions, it was shown that the DLS can support early instrumental learning when the DMS is lesioned before task training (Hilario et al., 2012). During motor skill learning, as assessed by the accelerating rotarod paradigm, both the DMS and DLS disengage across training, albeit with different time courses (Kupferschmidt et al., 2017). Instead, changes in the patterning of neuronal activity were associated with skill learning, rather than the proportion of task-activated neurons in the DMS versus DLS. 28 These results suggest that action comparison and selection within the DMS and DLS operate in parallel rather than in sequence, and can either compete or cooperate in control of behaviour, depending on the circumstances of the task, cognitive demands, motivational state, and other factors. Several groups have suggested hypotheses for how this could work. These systems could be organized hierarchically, in which high-level, goal-directed choice then triggers habitual motor sequences to attain that goal (Balleine & Dezfouli, 2013; Bradfield & Balleine, 2013). Or, goal selection could be the result of habitual patterns, followed by goal-directed planning and deliberation to attain those goals (Cushman & Morris, 2015). In the context of RL algorithms, Gershman et al. (2014) posited that the model-free system controls behavioural output, while the model-based system guides behaviour by training the lower level model-free system through replaying previous experience and simulating possible future outcomes. It is also possible that different aspects of task performance are simultaneously under different levels of goal-directed versus habitual control, in a non-hierarchical manner. Indeed, Schreiner et al. (2019) have noted that because different neural substrates underlie value versus contingency changes, each mechanism could inform action value comparison and selection, resulting in goal-directed and habitual systems simultaneously contributing to decision making. Furthermore, they note that multiple cognitive processes are involved in both devaluation and contingency degradation. For example, flexibly reacting to shifts in action-outcome contingencies relies on memory of past actions, accurate representation of the action-outcome contingency, incorporation of contextual information, as well as cognitive flexibility. Thus, individual components of decision making and action selection could differentially be controlled by these two processes. 29 1.3.5 Compulsivity versus impulsivity Compulsivity is characterized by repetitive responding despite negative consequences that is unrelated to an individual’s overall goal (Luigjes et al., 2019). Impairments to goal-directed control may be responsible for compulsive behaviours across a wide range of psychiatric conditions (Gillan et al., 2016). If compulsivity is conceived as a failure of top-down control to inhibit ineffectual behaviours, then it may have some properties in common with impulsivity, of which one aspect can be defined as a failure to inhibit planned or already initiated actions (Bari & Robbins, 2013). Indeed, overlapping corticostriatal circuits as well as both dopaminergic and serotonergic signalling have been implicated in compulsivity and impulsivity (Fineberg et al., 2010), suggesting similar underlying neural mechanisms. Impulsivity can be broken down into two processes – impulsive choice and impulsive action. Impulsive choice is defined as preference for small but immediate reward over larger delayed reward. It is often measured with the delay discounting task (Ainslie, 1975), which assesses the length of time subjects are willing to wait for a large reward before it is discounted in favour of an immediate, small reward. Conversely, impulsive action is a failure to withhold a motor response, often measured by the Go/No Go task in humans (Donders, 1969). In animals, it can be measured as premature nose-poke responses made at the stimulus array during inter-trial intervals on tasks such as the 5-choice serial reaction time task (5CSRT; Carli et al., 1983). While impulsive choice and impulsive action don’t correlate in healthy populations (Winstanley et al., 2006) and their neural mechanisms can be dissociated (Wang et al., 2016), both play a role in gambling disorder and substance use disorder. A meta-analysis synthesizing studies on delay discounting and addiction found that addiction severity is strongly correlated with high impulsive choice (Amlung et al., 2016). In animal models of compulsive drug use, 30 high levels of premature responding as measured on the 5CSRT task predict the development of compulsive drug taking (Belin et al., 2008), indicating that a switch from impulsivity to compulsivity may be one pathway to drug addiction. High levels of premature responding on the 5CSRT task are also associated with high rates of cocaine self-administration (Dalley et al., 2007). Similarly, high impulsive choice and action are associated with gambling disorder (Verdejo-Garcia et al., 2008). Deficits in impulsive action may be particularly relevant to the development of severe gambling problems (Brevers et al., 2012). Interestingly, despite associations between addictive behaviours and both compulsivity and impulsivity, measures of impulsivity and compulsivity are generally not correlated with each other in either animal models nor individuals with substance use disorder, or in other clinical populations such as those with obsessive-compulsive disorder (Meunier et al., 2012; Dellu-Hagedorn et al., 2018). Furthermore, while corticostriatal loops have been implicated as a neural mechanism for both behaviours, differences in the involved subregions have been identified (i.e., OFC and dorsal striatum for compulsivity, anterior cingulate cortex and ventral striatum for impulsivity; Fineberg et al., 2010). Nevertheless, compelling evidence points to an impulsive phenotype as a risk factor for the development of a compulsive pattern of drug taking (Dalley et al., 2011). A measure of premature responding, similar to that of the 5CSRT task, is included in the rGT; thus, the experiments in this dissertation will be able to assess potential interactions between inflexible responding and impulsive action. Given the complexity of the interaction between goal-directed and habitual systems, questions have been raised about the role of compulsive habits in addiction. Indeed, Hogarth (2020) has argued that some evidence for increased habit formation in individuals with substance use disorder can alternatively be explained by cognitive deficits that result in impaired storage 31 and/or retrieval of action-outcome contingencies. In the context of substance use disorder, habitual versus goal-directed control could also differ between drug seeking and drug taking. The next section will cover evidence for and against the role of habit in compulsive use of drugs and gambling. 1.3.6 The role of habits in compulsive drug use/gambling Compulsive drug use is defined as persistence in drug seeking despite negative consequences. On the seeking-taking task, about 20% of rodents will persistently respond on the seeking lever despite unpredictable foot shock (Pelloux et al., 2007). Considerable evidence suggests that this compulsive, habitual behaviour is the basis of addiction (Everitt & Robbins, 2016). Seminal work in this field demonstrated that alcohol seeking in rats is insensitive to devaluation and contingency degradation (Dickinson et al., 2002). Alcohol-dependent humans also show increased reliance on stimulus-response patterns when goal-directed control would be advantageous for earning reward (Sjoerds et al., 2013). Multiple studies have shown that noncontingent drug exposure renders food seeking insensitive to devaluation (Nelson & Killcross, 2006; Corbit et al., 2012; Hogarth et al., 2013), suggesting that drugs of abuse disrupt the balance between goal-directed and habitual control of behaviour, even for natural rewards. Furthermore, neural correlates of compulsivity have been implicated in substance use disorder. Individuals with substance use disorder show reduced OFC functional connectivity, and low gray matter in this region is correlated with compulsivity measures (Meunier et al., 2012). OFC inputs to the DMS, which are critical to goal-directed control (Section 1.3.3.2), are disrupted by chronic intermittent alcohol exposure, leading to increased reliance on the habitual system and an inability to switch to goal-directed control (Renteria et al., 2018). Serotonergic utilization within the PFC, critical to cognitive flexibility (Section 1.2.2), is reduced in rats that 32 exhibit compulsive drug seeking (Pelloux et al., 2012). The role of serotonergic activity within the PFC in drug seeking was further confirmed by showing that forebrain serotonin depletion increased punished seeking in non-compulsive rats on a short-access model of cocaine administration (Pelloux et al., 2012). There is also some evidence for enhanced neural mechanisms of habit formation. Long-term nicotine self-administration is associated with amplified dopamine activity in the SNc and DLS (Clemens et al., 2014). In the case of gambling disorder, individuals with problem gambling show increased amphetamine-induced dopamine release in the dorsal striatum (Boileau et al., 2014). However, enhanced habit mechanisms do not necessarily translate into compulsive patterns of behaviour, as transfer of control of cocaine seeking from the ventral to the dorsal striatum can occur in a short-access model of cocaine self-administration that typically does not lead to compulsive seeking (Willuhn et al., 2012). An alternative possibility is that habit formation in addiction disorders is at comparable levels to that in healthy controls, but impaired top-down control of goal-directed behaviour results in compulsive use (Vandaele & Janak, 2018). Indeed, individuals with alcohol dependence show impairments in model-based control, and no increase in model-free control (Sebold et al., 2014). In an animal model of drug seeking, it was shown that compulsive seeking could develop under the control of the goal-directed system (Singer et al., 2018). In this study, rodents were required to solve unique daily puzzles to gain access to drug seeking, which therefore precluded the formation of habitual motor sequences, and was independent of dopamine activity in the DLS. Indeed, Hogarth (2020) has posited that compulsive drug seeking results from excessive goal-directed control, driven by greater expected value of the drug. As 33 alluded to in the previous section, there could also be a fractionation of control over separate decision-making processes. For example, drug seeking may be insensitive to changes in action-outcome contingencies, but drug consumption remains sensitive to devaluation (Halbout et al., 2016). Drug-associated cues may play an important role in biasing decision making towards drug taking, in which cue exposure can trigger goal-directed choice leading to habitual automatic sequences (Vandaele & Janak, 2018). The hijacking of dopamine processes could play a critical role in the control of cues over behaviour (Keramati & Gutkin, 2013). Thus, the next section will cover paradigms that examine the impact of cues on motivated behaviours and decision making, with a focus on goal-directed versus habitual processes. 1.4 Animal models that assess the impact of cues on behaviour Several animal models have been investigated to test the impact of cues on reward processing. Ivan Pavlov (1927) was the first to discover that a salient reward-associated stimulus in the environment could elicit a conditioned response as if it were the reward itself. These cues can subsequently motivate reward-seeking behaviour. For example, stimuli paired with food delivery can trigger eating even in sated rats (Weingarten, 1983), indicating that cues can motivate behaviour that is dissociated from the intrinsic drives (in this case, hunger) of the animal. In autoshaping experiments, some animals learn to approach and interact with stimuli that predict reward, such as pecking at a key that illuminates prior to food delivery (Brown & Jenkins, 1968). Autoshaping is often measured by interaction with a lever that is inserted during food pellet delivery (termed the CS+), versus interaction with a lever that is inserted without reward delivery (CS-). Animals who exhibit increased interaction with the CS+ versus the CS- 34 are termed sign-trackers, as opposed to goal trackers who primarily interact with the food cup where reward is delivered (Hearst & Jenkins, 1974). Sign-tracking is often viewed as evidence for increased incentive salience of the cue, in which the reinforcing properties of the reward are transferred to the CS+. Propensity for sign-tracking may also correlate with inflexibility in stimulus-response patterns. In the original experiments, autoshaping occurred even though reward delivery was not contingent on instrumental actions, indicating that sign-tracking is not dependent on outcome value – a hallmark of habitual behaviour. To further investigate this hypothesis, Nasser et al. (2015) tested whether sign-tracking is correlated with inflexibility in stimulus-response associations. In this experiment, rats were first trained to associate a light cue with food pellet delivery. The food pellets were then devalued by pairing them with LiCL injections. A probe test was then conducted to observe the number of food cup entries during presentation of the light cue. Non-sign trackers flexibly reduced the number of food cup entries during cue presentation following outcome devaluation, indicating that the shift in outcome value decreased the incentive properties of the cue. Conversely, sign-trackers persistently responded during cue presentation even after outcome devaluation. Both groups showed reduced consumption of the food pellets following devaluation. Thus, although the value of the reward was decreased in sign-trackers, this did not reduce the incentive salience of the cue. Interestingly, sign-trackers do show sensitivity to outcome devaluation on the autoshaping procedure (Amaya et al., 2020). Thus, among sign-trackers, there may be differences in inflexibility of behaviours invigorated by cue presentation (persistent food cup entries in Nasser et al., 2015) and cue-approach behaviour (reduced interaction with the lever in Amaya et al., 2020). In other words, in a subset of individuals, cues can persistently motivate 35 reward-seeking behaviour following reward devaluation despite intact sensitivity to reward value on other measures. This is consistent with theories on compulsive drug seeking in rats. On the seeking-taking task, extended drug exposure results in seeking responses becoming insensitive to devaluation of the drug-taking response through extinction training (Zapata et al., 2010). Thus, in this model, despite sensitivity to reinforcer value (extinguished responses on drug-taking lever), reward seeking is rendered insensitive (persistent responding on the seeking lever). The Pavlovian to Instrumental Transfer (PIT) paradigm demonstrates the motivational power of reward-predictive cues (Cartoni et al., 2016). In this paradigm, a Pavlovian association between a predictive cue and reward is trained separately from an instrumental action (usually lever pressing) to obtain that reward. Lever pressing for reward is then tested in the presence versus absence of the conditioned stimulus to determine whether cue presentation can invigorate reward seeking. This can be further divided into specific transfer, in which cue presentation results in increased responding for the specific reward associated with the cue, or general transfer, in which responding for other outcomes of the same valence (in this case, appetitive) is increased as well. These two processes have been shown to have distinct neural mechanisms mediated by the nucleus accumbens core and shell, respectively (Corbit & Balleine, 2016). It has been hypothesized that general PIT may be related to sustaining inflexible drug seeking during periods of unpredictable drug reinforcement (Everitt & Robbins, 2016). In this scenario, the conditioned stimulus retrieves the drug’s affective value to reinforce seeking, regardless of whether the drug is actually obtained. In support of this, repeated cocaine exposure results in rats exhibiting a stronger PIT effect as well as insensitivity to reinforcer devaluation (LeBlanc et al., 2013). This suggests that frequent drug use leads to both increased control of cues over behaviour and increased reliance on habitual choice patterns. 36 In this sense, drug-associated cues may be acting as conditioned reinforcers. Tests of conditioned reinforcement involve pairing a conditioned stimulus with reward delivery, then testing the extent to which animals will lever press for the presentation of that cue (Robbins, 1978). This therefore measures the reinforcing properties of the cue itself, independent of the value of the reward. Drug-associated cues can sustain drug seeking in rats, as lever pressing for cocaine takes longer to extinguish when a conditioned stimulus is presented contingently with lever pressing (Arroyo et al., 1998). Furthermore, cues paired with intravenous cocaine infusions acts as stronger conditioned reinforcers compared to food-paired cues (Turnstall & Kearns, 2016). Interestingly, this runs counter to rats’ preference for the primary reinforcer (grain pellets were preferred over cocaine infusions), indicating a disconnect between primary and conditioned reinforcement for drugs of abuse. In conjunction with results discussed above, drug-associated cues may thus be able to sustain drug seeking that is insensitive to devaluation, or in the case of building tolerance to the drug, lack of positive reinforcement. As discussed in Barrus et al. (2016), sign-tracking, PIT, and conditioned reinforcement could be considered hierarchically according to their level of control of behaviour: sign-tracking measures motivated interest in cues, PIT measures the ability of the cue to invigorate reward seeking, and in conditioned reinforcement, the cue acquires rewarding properties itself and animals will work to obtain it. The nucleus accumbens core has been implicated in all three behavioural measures and thus have shared neural mechanisms (Everitt & Robbins, 2013). Experiments testing cue-induced reinstatement of drug seeking exhibit how these three measures are inter-related. In this paradigm, presentation of a cue previously paired with drug administration can lead to reinstatement of drug seeking after extinction training. Importantly, cue-induced reinstatement occurs without actual administration of the drug, and thus seeking 37 behaviours are separated from the physiological impacts of drug consumption. Studies have shown that sign-tracking is associated with cue-induced reinstatement for nicotine, cocaine, and methamphetamine (Versaggi et al., 2016; Saunders & Robinson, 2010; Everett et al., 2020). Furthermore, cue-induced drug seeking persists in sign-tracking rats even when seeking is punished with footshock (Saunders et al., 2013). Thus, rats who are prone to attributing incentive salience to cues show inflexible patterns of drug seeking that are resistant to aversive consequences. While considerably less research has been conducted on the context of cues in gambling disorder, it has been posited that the reward uncertainty and salient lights and sounds of the casino may promote gambling disorder through similar mechanisms (Anselme & Robinson, 2020). Research has also been conducted using tasks that feature cues that signal reward probability at the time of choice (i.e., preceding the instrumental response). In this context, the degree of sign-tracking has been shown to be sensitive to the probability of reward. Lever pressing (i.e., sign-tracking) was higher for the low probability cue, whereas more food port entries were observed with the high probability cue (Marshall et al., 2017). Thus, reward uncertainty can modulate the control of cues over behaviour. Cues that predict probability of reward have also been studied using the “Blackjack” task, in which rats choose between a small/certain option offering a one pellet reward or a large/risky option offering a reward of four pellets, where the odds of the latter are indicated by different auditory cues (12.5 or 50%; Floresco et al., 2018). Animals learn to flexibly adapt their responding depending on the odds indicated by the presented cue – increased selection of the safe lever when the “bad” odds cue plays, and increased selection of the risky option when the “good” odds cue plays. Thus, in this 38 context, cues carry pertinent information about uncertainty in the environment, and can support optimal choice and flexibility in behaviour. Each of the cue paradigms discussed thus far may tap into a different aspect of gambling behaviour observed in humans. Biased attention towards gambling-related cues may line up with sign-tracking (Anselme & Robinson, 2020). Sign-tracking responses are more pronounced when uncertainty in reward delivery following CS presentation is introduced (Anselme et al., 2013), and thus may be particularly relevant to gambling, given that the context of a casino predicts uncertain reward. The ambient lights and sounds of a casino can invigorate responding on the IGT (Brevers et al., 2015), which may be similar to PIT. Losses disguised as wins (LDWs) are a common feature of EGMs, in which win-paired cues play when the player wins a smaller amount than was initially bid (i.e., technically a loss). LDWs tend to be miscategorized as wins (Jensen et al., 2013). This likely involves conditioned reinforcement mechanisms, in which the reward-associated cues serve as a reinforcer of betting behaviour, despite losing money. Lastly, tasks in which cues predict probability can be used to model gambling behaviour on games where the odds are known, such as roulette. While salient stimuli within the environment can clearly impact reward-seeking behaviour, a distinction must be drawn between reward-predictive cues and reward-paired cues. For example, in cue-induced reinstatement, contextual cues that predict reward availability have different mechanisms for invigorating drug seeking than Pavlovian cues paired with reward (Namba et al., 2018). Thus, the next section will discuss similarities and differences between the cued rGT and other cue paradigms. 1.4.1 The rat Gambling Task The rGT may tap into a different facet of gambling compared to other paradigms. On this task, cues signal the success of the instrumental action, which mimics the role of the salient 39 lights and sounds of an EGM. Such cues can interfere with estimations of the probability of success in humans (Dixon et al., 2010). While this may seem similar to conditioned reinforcement, responding for a conditioned reinforcer does not correlate with risky choice, nor does this measure differ between rats trained on the uncued versus cued rGT (Ferland et al., 2019; Ansary et al., 2021). In paradigms assessing responding for a conditioned reinforcer, the action-cue association is 100%, whereas on the rGT, the action-cue association is 40-90%, depending on the option chosen. Introducing uncertainty into outcome delivery can influence multiple cognitive processes related to decision making (see Monosov, 2020 for review), so it stands to reason that underlying mechanisms would differ between conditioned reinforcement and the rGT. Inconsistent results in the association between sign-tracking and risk preference on the cued rGT have been found, although these may be due to methodological differences (Ferland et al., 2019; Swintosky et al., 2021). It is important to note that in sign-tracking paradigms, cue presentation precedes approach behaviours, and cue presentation is not contingent on instrumental responses, and thus may involve different mechanisms compared to measures in which cue presentation is dependent on instrumental actions. 1.4.2 Neurobiological mechanisms of cue-mediated behaviours Dopamine, particularly its activity in the nucleus accumbens, is involved in all the cue paradigms discussed thus far (Everitt & Robbins, 2013; Floresco et al., 2018; Hynes et al., 2020). On the cued rGT, D3 receptor agonists administered after the task is learned or D2/D3 receptor agonists administered chronically during task training both drive risk preference, but not on the uncued task (Barrus & Winstanley, 2016; Mortazavi et al., 2022). Thus, dopamine signaling is critical to mediating the impact of cues on behaviour. 40 Differences in neural mechanisms between the cued rGT and other cue paradigms have also been found. Previous literature has strongly implicated the serotonin 2C receptor in cue-mediated behaviours, such as cue-induced cocaine seeking and responding for a conditioned reinforcer, through its regulation of the mesolimbic dopamine system and the medial PFC (Pentkowski et al., 2010; Anastasio et al., 2014; Swinford-Jackson et al., 2016; Browne et al., 2017). However, while 2C antagonists can increase cue-induced drug seeking and responding for a conditioned reinforcer (Browne et al., 2017; Filip & Cunningham, 2003), systemic delivery of a 2C antagonist actually ameliorates cue-induced deficits on the cued rGT (Adams et al., 2017). This may be due to increased flexibility, as 2C antagonism delivered specifically to the lateral OFC can improve flexibility on a reversal learning task (Boulougouris & Robbins, 2010). Thus, there are different cognitive and neural mechanisms underlying cue-induced risky choice on the rGT and the impact of cues on behaviour in other paradigms. It is important to note that distinct sex differences have been observed in the impact of experimental manipulations on the cued rGT (Hynes et al., 2020; Hynes et al., 2021). The majority of studies in this section have exclusively featured males. Given the differences in the acquisition of addiction between genders described in the Section 1.1, understanding the differential role of critical neurobiological targets in addiction-related traits such as risk preference and inflexibility is important. In this thesis, one study investigating the role of nigrostriatal dopamine in cue-enhanced risky choice was conducted solely in females to begin to build insight into these sex differences. Due to practical constraints and to facilitate comparison to previous literature, the three other chapters were conducted in males. In light of this, the next section will cover sex differences relevant to decision making. 41 1.5 Sex differences in relevant cognitive processes Males and females show different endogenous release patterns of sex hormones (Yoest et al., 2018). Male hormonal release is tonic, under homeostatic control, and follows circadian rhythms. In females, there is a cyclic release of the sex hormones estradiol and progesterone. In rats, this follows a 4-5 day cycle – low estrogen and progesterone during metestrous and diestrous, and high levels during proestrous and estrous stages. During estrous, when estrogen and progesterone levels are high, dopamine release in the dorsal striatum is also high, and there is a general enhancement in the excitability of dopamine terminals (Yoest et al., 2018). This is thought to occur by disinhibition of dopamine terminals via inhibition of GABA release from striatal neurons. Compensatory changes in D2 receptor affinity state and increased dopamine transporter activity may mitigate some of these effects (Yoest et al., 2018). Genomic estrogen receptors -alpha and -beta have been shown to mediate the impact of estradiol on striatal activity (Schultz et al., 2009; Yoest et al., 2019). Many of the effects of estradiol are also mediated by GPER-1 and membrane-bound ER-alpha and ER-beta, which are membrane receptors that can impact neuronal activity on the timescale of a few milliseconds to minutes (Quigley et al., 2021). The GPER-1 receptor plays differential roles in drug-related behaviours in males versus females – activating the GPER-1 receptor in the DLS results in increased motivation to self-administer cocaine and greater drug-induced reinstatement in female but not male rats (Quigley, Logsdon, Graham et al., 2021). There is also some evidence that the rapid effects of estradiol can potentiate conditioned place preference in female rats, perhaps jointly with genomic receptors (Peart et al., 2022). Thus, estradiol and its various receptor subtypes play a key role in dopamine activity and at least some behavioural responses related to drug taking. 42 Increased habit formation of instrumental actions has been reported in female rodents, which appears to be dependent on the XX chromosome rather than sex hormones (Quinn et al., 2017; Haines & Czachowski, 2022). Interestingly, estradiol can also increase sensitivity to stress (Quigley et al., 2021). Stress can impact pathways involved in habit formation (Porcelli & Delgado, 2017). Acute stress reduces the prefrontal response to outcome devaluation, and chronic stress can lead to insensitivity to devaluation coupled with atrophy in the medial PFC and DMS as well as increased DLS activity. Thus, there may be an interaction in the impact of the XX chromosome and estradiol-induced stress sensitivity on habit formation. This may be particularly relevant to compulsive drug seeking, in which unpredictable foot shock ostensibly would cause a stress response within the animal. Indeed, corticosterone administration during adolescence augments cue-induced reinstatement of alcohol seeking only in female rats, suggesting a potential interaction between stress, sex hormones, and the impact of cues on drug seeking (Bertholomey et al., 2016). Androgens such as testosterone and 5-alpha-DHT can also influence relevant behaviours. Local synthesis of androgens occurs in mesocorticolimbic circuitry, even in gonadectomized animals, suggesting they could play a role in decision making mediated by this pathway (Tobiansky et al., 2018). Furthermore, high levels of androgens have been implicated in impaired reversal learning and IDED set shifting in male rats, as well as impaired performance on the IGT in humans (Stanton et al., 2011; Wallin & Wood, 2015). Thus, high androgens may result in cognitive rigidity and impairments to cost/benefit decision making (Fattore & Melis, 2016). Men who use anabolic-androgenic steroids tend to show increased risk-taking, further indicating the role of androgens in decision-making processes (Middleman et al., 1995; Midgley et al., 2000; Bolding et al., 2002). In general, male rodents tend to favour “exploit” strategies over 43 exploration, perhaps supported by increased cognitive rigidity (Orsini & Setlow, 2018). Men also tend to be more impulsive, although interestingly, the opposite relationship has been reported among those with substance use disorders, with women showing higher levels of impulsivity than men (Fattore & Melis, 2016; Tobiansky et al., 2018). Overall sex differences in cost/benefit decision-making tasks have been reported, which are likely to be at least partially explained by the activity of sex hormones described above. On the IGT, women are more sensitive to losses on advantageous decks, as well as more exploratory in early training, resulting in a slower development of the optimal strategy (Bos et al., 2013). Similar differences in rodent models of the IGT have been reported (Shanksy et al., 2018; Orisini & Setlow, 2018). These may be the result of differences in lateral OFC function, as the lateral OFC is more active in men on the IGT (Bolla et al., 2004). Furthermore, the dynamics of lateral OFC activity when making risky versus safe choices differs between men and women on other risky decision-making tasks (Lee et al., 2009; Crowley et al., 2015), further suggesting that activity in this region mediates differences in cost/benefit decision making. Sex differences in the SNc have also been reported. Gillies & McArthur (2010) found that men showed a higher number and increased density of dopamine neurons, while the SNc in females is larger. However, no differences in basal levels of dopamine in the striatum have been reported. Thus, differential mechanisms in neuronal dynamics must exist between males and females, resulting in similar levels of striatal dopamine. Interestingly, estradiol is protective against damage from Parkinson’s disease, suggesting that estradiol can engage in compensatory activity when dopamine neurons are lost (Currie et al., 2004; Hyun Lee et al., 2019). Finally, sex differences on the cued rGT have been found. In males, inhibiting 44 dopaminergic neurons in the VTA reduces risky choice on the cued rGT (Hynes et al., 2020, 2021). Surprisingly, inhibition of the mesolimbic pathway during task learning has the opposite effect in females – they exhibit an increase in risky choice (Hynes et al., 2021). Female rats also show increased risky choice with the systemic administration of the D2/D3 agonist ropinirole during task learning (Russell et al., 2022). As increased mesolimbic dopamine activity cannot explain the risk-inducing effect of ropinirole, other dopaminergic pathways may mediate this effect. Dopamine activity in the dorsal striatum, innervated by the SNc, has not yet been investigated on this task, and may contribute to the development of risky choice patterns. Given the surprising effect in females following manipulation of the VTA on the cued rGT, and the impact of estradiol on dopaminergic signaling in the dorsal striatum, examining the role the nigrostriatal pathway in females is of particular interest. Chapter 6 will therefore investigate the role of the SNc in the acquisition and performance of the cued rGT in female rats. 1.6 Insights from clinical populations As discussed throughout this introduction, reward-associated cues can impact addiction processes such as cravings, motivated approach, and reinstatement of drug seeking, in addition to their effect on risky decision making. Cues may also impact the development of inflexible habits and/or impaired top-down control in gambling disorder and substance use disorder. This hypothesis will be tested in the context of cost/benefit decision making in the following chapters. In addition to substance use and gambling disorders, relevant neurocircuitry and cognitive processes have been implicated in other clinical populations. These will be briefly covered below to highlight shared neural mechanisms across clinical disorders. 45 1.6.1 Disordered eating Increased reliance on model-free systems has been observed in individuals with binge eating disorder (Voon et al., 2015). These effects were not observed in obese individuals, suggesting that enhanced habit formation is specific to disordered eating rather than obesity. Similarly, individuals with restrictive anorexia show increased habit formation on the slips-of-action task, which is correlated with measures of cognitive inflexibility (Favier et al., 2020). This was linked to an animal model of disordered eating, in which both binge eating patterns and anorexia-like behaviour were caused by an imbalance in striatal dopamine release and were associated with a propensity to habit formation (Favier et al., 2020). Animal models of diet-induced obesity have shed light on potential mechanisms for shifts in habit formation observed in clinical populations. Following exposure to a high-fat diet, obese rats show impaired outcome devaluation for a food reward, coupled with distinct changes in OFC function (Seabrook & Borgland, 2020). Decreased GABAergic transmission in the OFC, resulting in noisy pyramidal neurons, as well as an increase of pyramidal neuron dendrite branching, have been reported following prolonged exposure to a high-fat diet (Thompson et al., 2017). Noisy lateral OFC activity has been causally linked to insensitivity to outcome devaluation (Seabrook et al., 2022). Thus, obesogenic diets may disrupt the ability of the lateral OFC to appropriately update action-outcome contingencies to guide the seeking of food rewards. High-fat diets with or without high-sugar have also been shown to impact cue-guided learning, particularly when cue-outcome associations must be discriminated or updated (Winocur & Greenwood, 1993; Kanoski et al., 2007; Kosheleff et al., 2018). While the role of the OFC in these effects has not been directly assessed, it is likely that diet-induced disruptions to this region described above are involved in these behavioural impairments. 46 1.6.2 Obsessive compulsive disorder Obsessive-compulsive disorder (OCD) is characterized by intrusive thoughts driving compulsive stereotyped behaviours to reduce the likelihood of negative outcomes (Fineberg et al., 2010). In individuals with OCD, reduced explicit knowledge of action-outcome associations have been reported, resulting in increased slips of action (Gillan et al., 2011). Individuals with OCD exhibit similar deficits in OFC functional connectivity and reduced gray matter to those with substance use disorder, and reduced connectivity was correlated with measures of compulsivity in both groups (Meunier et al., 2012). Thus, compulsive behaviour in both disorders may be mediated by disruptions to OFC circuitry. Excessive avoidance habits in OCD have been associated with an over-active caudate (analogous to the DMS), not putamen (analogous to the DLS), suggesting that disordered habit formation in OCD is associated with impairments to goal-directed activity, not enhanced habit systems (Gillan et al., 2015). 1.6.3 Parkinson’s disease Progressive loss of the nigrostriatal dopamine pathway in Parkinson’s disease results in profound difficulties in initiating voluntary movements (Armstrong & Okun, 2020). Due to this pathway’s role in habitual control of behaviour, studies have investigated whether patients with Parkinson’s disease show differences in habit formation from healthy controls. Conflicting results have been reported for habitual control as measured by the slips-of-action task, with some reporting increased and others reporting decreased responding to devalued outcomes (Bannard et al., 2019; Mi et al., 2021). Responding to devalued outcomes was shown to correlate with symptom severity, which may explain heterogenous results between different groups (de Wit et al., 2011). Additionally, medications such as levodopa, a dopamine precursor, can influence the formation of habits as well as action-outcome contingency updates following unexpected 47 punishments (Cools et al., 2006; Marzinzik et al., 2011). As patients with more severe symptoms of Parkinson’s disease are more likely to be on medication, there may be a consistent effect across studies in which the impact of unexpected punishments is reduced with increased mesolimbic versus nigrostriatal dopamine activity. Thus, the relative balance of dopamine activity driven by the VTA and along the medial-lateral gradient of the SNc appears to be a more important factor than overall increases or decreases of dopamine activity. This is in line with conclusions drawn from Section 1.3.3.4, in which medial to lateral or lateral to medial communication between striatal regions can toggle between goal-directed and habitual control of behaviour. Differences in attentional biases have also been noted in Parkinson’s patients. Notably, extradimensional set shifting is impaired (Downes et al., 1989). In unmedicated patients, this is a result of both perseverating on the irrelevant dimension and deficits in shifting attention to the new dimension that was previously irrelevant (Owen et al., 1993). Dopamine depletion in monkeys also results in these deficits, suggesting an overall role of dopaminergic transmission in shifting attention away from irrelevant stimuli and towards newly relevant stimuli (Crofts et al., 2001). Interestingly, patients with Parkinson’s disease exhibit deficits in the internal control of attention across multiple tasks (Dirnberger & Jahanshahi, 2013). Individuals with Parkinson’s disease perform normally when salient external cues are present to guide attention, suggesting an enhanced reliance on cues to inform behaviour. Thus, investigating the role of the nigrostriatal dopamine pathway on the cued rGT is of particular interest. 48 1.7 Summary and objectives Evidence suggests that cue-induced impairments in cost/benefit decision making may be a critical risk factor for the development and maintenance of behavioural addictions such as gambling disorder. Salient stimuli that predict reward availability or delivery can not only bias subsequent decision making, but can also fundamentally change the neural mechanisms that are recruited into the decision-making process. Previous literature has alluded to the impact of such cues on goal-directed versus habitual control of behaviour. For example, research in rodent models of addiction indicate that drug-paired cues trigger compulsive drug-seeking habits that are highly resistant to punishment and extinction. On the other hand, some have argued that persistent reward seeking results from maladaptive goal-directed control, rather than compulsive habits. Corticostriatal circuitry and dopaminergic inputs from the midbrain to the striatum have been heavily implicated in goal-directed versus habitual control, as well as the impact of cues on behaviour. Thus, assessing the role of these regions in rGT learning and performance may shed light on the contributions of goal-directed versus habitual systems to cue-induced inflexibility. Using the rGT, it is possible to simultaneously assess the impact of reward-paired cues on cost/benefit decision making as well as flexibility in response patterns. It has previously been shown that the addition of reward-paired cues to the rGT can increase risky choice; this also renders decision making more sensitive to dopaminergic manipulations. However, it is unknown whether this is accompanied by reduced flexibility in choice patterns, nor have the potential contributions of impaired or otherwise maladaptive top-down control been investigated. Furthermore, sex differences in the role of dopamine in cue-induced risky choice have been uncovered but not fully elucidated. Thus, the experiments included in this dissertation were designed to directly investigate these gaps in knowledge. 49 Experiment 1 (Chapter 3) tests which aspects of the cue-reward association are necessary to drive risky choice and induce inflexibility in choice patterns. Based on prior research, the cognitive mechanisms of cue-induced risky choice are unclear. Higher levels of arousal due to enhanced sensory stimulation may explain impairments in decision making. The importance of the specificity of the cue-reward association has also not been previously tested. Furthermore, on the cued rGT, the magnitude and complexity of cues scale with reward size, and as such, the impacts of two factors (salience of the cue and reward size) are confounded. Lastly, the impact of cuing losses instead of wins has not been investigated, on either the rGT or other animal models of cost/benefit decision making. Thus, we designed several different variants of the rGT to investigate these possibilities and identify precisely how reward-paired cues drive risky choice. A reinforcer devaluation procedure was also used to assess which variants would lead to inflexibility in choice patterns. Computational analyses were applied to investigate differences in learning dynamics across the cue variants. Experiment 2 (Chapter 4) assesses whether increasing cognitive flexibility via antagonism of the serotonin 2C receptor in the lateral OFC can ameliorate cue-induced risky choice. Systemic administration of a serotonin 2C receptor antagonist can attenuate cue-induced risk preference on the rGT. However, the neurocognitive mechanisms mediating this effect are currently unknown. We therefore investigated whether the serotonin 2C receptor antagonist RS 102221 is able to attenuate cue-enhanced risk preference via its actions in the lateral OFC. To assess the neural specificity of any observed effects, we also infused RS 102221 into the prelimbic region of the medial PFC. A reinforcer devaluation test was used in this experiment to determine whether cue-induced inflexibility could also be ameliorated by this manipulation. 50 Experiment 3 (Chapter 5) assesses whether altering activity within the lateral OFC during task acquisition differentially impacts choice patterns in the uncued versus cued rGT. Significant research has implicated the lateral OFC in cognitive flexibility and cue-guided learning. Furthermore, the lateral OFC is an important neural locus for model-based learning strategies. Thus, if aberrant goal-directed behaviour is responsible for cue-induced risky choice, then altering lateral OFC activity during task acquisition should differentially impact choice patterns on the uncued versus cued rGT. To investigate this hypothesis, we used chemogenetics to chronically inhibit or activate the lateral OFC across training of the uncued and cued rGT. Experiment 4 (Chapter 6) investigates the role of the nigrostriatal dopamine pathway in cue-induced risky choice. Striking sex differences in the impact of VTA dopamine on cue-induced risky choice on the rGT have been previously observed. Chronic administration of the D2/D3 agonist ropinirole can exacerbate cue-induced risky choice in females, but VTA inhibition also increases risky choice; thus, other dopaminergic mechanisms must explain the effect of ropinirole. The nigrostriatal pathway is a potential candidate for mediating this effect. Furthermore, given this pathway’s role in habit formation, investigating the contribution of nigrostriatal dopamine to decision making on the cued rGT may shed light on the role of habit mechanisms in guiding choice on this task. Thus, we used chemogenetics in transgenic female rats to inhibit or activate dopamine neurons in the SNc during both the learning and performance phases of the cued rGT. Overall hypothesis: reward-paired cues both increase risky choice and induce inflexibility in decision making, and these effects are at least partially mediated by neural regions implicated in goal-directed versus habitual control of behaviour. 51 Chapter 2: General methods 2.1 Subjects Subjects used for all experiments were either male Long Evans rats (Chapters 3-5; Charles River Laboratories, St Constant, QC, Canada) or female transgenic rats bred in-house against a Long-Evans background (Chapter 6). Transgenic rats expressed Cre recombinase (Cre) in neurons containing tyrosine hydroxylase (TH:Cre rats from: Long–Evans-Tg(TH-Cre)3.1Deis, RRRC #00659; Rat Resource and Research Centre; WT rats obtained from Charles River Laboratories). Female rats bred in-house were weaned at postnatal day 21, housed in groups of 3 or 4 animals per cage, and had access to ad libitum standard rat chow until an average weight of ∼250 g was reached. Rats were then transferred to the main vivarium. Rats obtained from the outbred facility weighed 225-300 g upon arrival and had access to ad libitum standard rat chow for one week or until an average weight of ~300 g was reached. Male rats were housed in groups of 2 or 3 animals per cage. Animals were then food-restricted and maintained at least 85% body weight of an age- and sex-matched control. Water was available ad libitum. Rats were housed under a reverse 12 h light/dark cycle in a temperature-controlled colony room maintained at 21°C. Huts and paper towel were provided as environmental enrichment. Behavioural testing took place 5 days per week. Experimental protocols were approved by the Animal Care Committee of the University of British Columbia, and were conducted in accordance with the Canadian Council of Animal Care. 2.2 Behavioural apparatus Testing took place in 32 standard five-hole operant chambers, each of which was enclosed in a ventilated, sound-attenuating chamber (Med Associates Inc, Vermont). Chambers 52 were fitted with an array composed of five equidistantly spaced response holes. A stimulus light was located at the back of each hole, and nose-poke responses into these apertures were detected by vertical infrared beams. On the opposite wall, sucrose pellets (45 mg; Bioserv, New Jersey) were delivered to the magazine via an external pellet dispenser. The food magazine was also fitted with a tray light and infrared sensors to detect sucrose pellet collection. A house light could illuminate the chamber. The operant chambers were operated by software written in Med-PC by CAW, running on an IBM-compatible computer. 2.3 Behavioural testing 2.3.1 Pre-training Rats were first habituated to the operant chambers in two daily 30-minute sessions, during which sucrose pellets were present in the nose-poke apertures and food magazine. Rats were then trained on a variant of the 5CSRT task (Carli et al., 1983), in which rats were required to make a nose-poke response in one of the four apertures used for the rGT, indicated by a stimulus light. A correct response was rewarded by the delivery of one sugar pellet to the food magazine. The location of the stimulus light varied between holes 1, 2, 4, and 5 across the session. Sessions lasted 30 minutes and consisted of approximately 100 trials. Stimulus duration started at 30 seconds. Rats were trained until they reached a criteria of ≥ 30 correct responses, after which stimulus duration was reduced to 20 seconds, and finally 10 seconds. Pre-training was considered complete when rats completed ≥50 correct responses with ≥ 80% accuracy and ≤ 20% omissions. Rats were then trained on a forced-choice variant of the rGT for 3-5 sessions, in which rats were presented with one of the four options in each trial in a pseudo-random fashion. This ensured that rats had equal exposure to each reinforcement contingency prior to training on the free-choice version of the task. 53 2.3.2 The rat Gambling Task A task schematic of the rGT is provided in Figure 2.1. During the 30-minute session, trials were initiated by making a nose-poke response within the illuminated food magazine. This response extinguished the light, and was followed by a five-second inter-trial interval (ITI) in which rats were required to inhibit their responses to proceed with the trial. Any response in the five-hole array during the ITI was recorded as a premature response and punished by a five-second time-out period, during which the house light was illuminated and no response could be made. Following the ITI, apertures 1, 2, 4, and 5 in the five-hole array were illuminated for 10 seconds. A lack of response after 10 seconds was recorded as an omission, at which point the food magazine was re-illuminated and rats could initiate a new trial. A nose-poke response within one of the illuminated apertures was either rewarded or punished according to that aperture’s reinforcement schedule. Probability of reward varied among options (0.9-0.4, P1-P4), as did reward size (1-4 sucrose pellets). Punishments were signalled by a light flashing at 0.5 Hz within the chosen aperture for the duration of a time-out penalty which lasted for 5-40 seconds depending on the aperture selected. The cued rGT was identical in structure, except for the addition of 2-second compound tone/light cues that occurred concurrently with reward delivery. Cue complexity and variability scaled with reward size, such that the P1 cue consisted of a single tone and illuminated aperture, and the P4 cue featured multiple tones and flashing aperture lights presented in 4 different patterns across rewarded trials (see Table 2.1). The task was designed such that the optimal strategy to earn the highest number of sucrose pellets during the 30-minute session would be to exclusively select the P2 option, due to the relatively high probability of reward (0.8) and short, infrequent time-out penalties (10 s, 0.2 probability). While options P3 and P4 provide higher per-trial gains of 3 or 4 sucrose pellets, the longer and more frequent time-54 out penalties associated with these options greatly reduces the occurrence of rewarded trials. Consistently selecting these options results in fewer sucrose pellets earned across the session and are therefore considered disadvantageous. The position of each option was counterbalanced across rats to mitigate potential side bias. Half the animals in each project were trained on version A (left to right arrangement: P1, P4, P2, P3) and the other half on version B (left to right arrangement: P4, P1, P3, P2). 55 Schematic of the cued rGT. A nose poke response in the food tray extinguished the traylight and initiated a new trial. After an inter-trial interval (ITI) of 5 s, four stimulus lights were turned on in holes 1, 2, 4, and 5, each of which was associated with a different number of sugar pellets. The order of the options from left to right was counter-balanced within each cohort to avoid development of a simple side bias (version A (shown): P1, P4, P3, P2; version B: P4, P1, P3, P2). The animal was required to respond at a hole within 10 s. This response was then rewarded or punished depending on the reinforcement schedule for that option. If the animal lost, the stimulus light in the chosen hole flashed at a frequency of 0.5 Hz for the duration of the time-out penalty, and all other lights were extinguished. The maximum number of pellets available per 30 min session shows that P1 and P2 are more optimal than P3 and P4. The percent choice of the different options is one of the primary dependent variables. A score variable is also calculated, as for the IGT, to determine the overall level of risky choice as follows: [(P1 + P2) – (P3 + P4)]. Figure is modified from Winstanley & Floresco (2016). Figure 2.1: The rat Gambling Task. 56 Option Cue duration Auditory cues Visual cues Variable? P1 2s 1 tone Flash H1, 2.5 Hz No P2 2s 2 tones, 1 s each Flash H4, 2.5 Hz No P3 2s 3 tones, 0.2 s each Flash H5, 5 Hz Flash H2-4, 5 Hz Yes, 2 patterns P4 2s 6 tones, 0.2 s each Flash H2, 5 Hz Flash H1-5, 5 Hz Yes, 4 patterns Table 2.1: Auditory and visual cues used on the cued rGT 57 2.4 Behavioural measures All statistical analyses were completed using SPSS Statistics 27.0 software (IBM) and RStudio. As per previous reports, the following rGT variables were analyzed: percentage choice of each option (number of times option chosen/total number of choices × 100), risk score (calculated as choice of [(P1 + P2) − (P3 + P4)]), percentage of premature responses (number of premature responses/total number of trials initiated × 100), sum of omitted responses, sum of trials completed, and average latencies to choose an option and collect reward. Variables that were expressed as a percentage were subjected to an arcsine transformation to limit the effect of an artificially imposed ceiling (i.e., 100%) (McDonald, 2009). For experiments targeting the performance phase after task acquisition, a statistically stable baseline was determined by a repeated-measures ANOVA across data from three consecutive sessions, in which both the session factor and session × choice interaction were not significant, prior to experimental manipulation. 2.5 Data analysis Animals with a mean positive baseline risk score were designated as “optimal decision-makers”, whereas rats with negative risk scores were classified as “risk-preferring”. For the analysis of the performance phase, choice data were analyzed with a two-way repeated measures ANOVA with choice (four levels: P1, P2, P3, and P4) and dose (Chapter 4,6) or session (Chapter 3,5,6) as within-subject factors. For all other variables, dose or session was the only within-subjects factor. Experimental condition and risk status (optimal, risk-preferring) were included as between-subjects factors for all statistical analyses. If sphericity was violated as determined by Mauchley's test, a Huynh–Feldt correction was applied, and corrected p values' degrees of freedom were rounded to the nearest integer. Any main effects or interactions of significance 58 were further analyzed via post hoc one-way ANOVA or paired samples t-tests with a Bonferroni correction applied for the number of comparisons made, except for Chapter 3, which used Tukey’s tests. For the analysis of data from the acquisition experiments (Chapters 5 and 6), we used a multilevel model with two levels: sessions nested within subjects. P1-P4 choice was analyzed using a binomial mixed effects logistic regression model, predicting choice of a given option out of all outcomes. Other dependent variables (DV; risk score, premature responses, omissions, trials completed, and latencies to choose an option and collect reward) were analyzed using linear mixed effects models. For each DV, the full model included session as the level-1 predictor, and experimental group and final risk score after the dosing period (final score in equations below) as level-2 predictors. We accounted for subject-level differences in the intercepts by including a random intercept component that varied across subjects, and modeled individual differences in the evolution of the dependent variable across sessions by including a random slope component for session. This resulted in the following model: Level-1 model: 𝐷𝐷𝐷𝐷𝑖𝑖𝑖𝑖 = 𝛽𝛽0𝑖𝑖 + 𝛽𝛽1𝑖𝑖𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑖𝑖𝑖𝑖 + 𝑠𝑠𝑖𝑖𝑖𝑖 Level-2 model: 𝛽𝛽0𝑖𝑖 = 𝛾𝛾00 + 𝛾𝛾01𝑓𝑓𝑠𝑠𝑠𝑠𝑓𝑓𝑓𝑓 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑖𝑖 × 𝑠𝑠𝑒𝑒𝑒𝑒𝑠𝑠𝑠𝑠𝑠𝑠𝑒𝑒𝑠𝑠𝑠𝑠𝑒𝑒𝑓𝑓𝑓𝑓 𝑔𝑔𝑠𝑠𝑠𝑠𝑔𝑔𝑒𝑒𝑖𝑖 + 𝛾𝛾02𝑓𝑓𝑠𝑠𝑠𝑠𝑓𝑓𝑓𝑓 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑖𝑖+ 𝛾𝛾03𝑠𝑠𝑒𝑒𝑒𝑒𝑠𝑠𝑠𝑠𝑠𝑠𝑒𝑒𝑠𝑠𝑠𝑠𝑒𝑒𝑓𝑓𝑓𝑓 𝑔𝑔𝑠𝑠𝑠𝑠𝑔𝑔𝑒𝑒𝑖𝑖 + 𝑔𝑔0𝑖𝑖 𝛽𝛽1𝑖𝑖 = 𝛾𝛾10 + 𝛾𝛾11𝑓𝑓𝑠𝑠𝑠𝑠𝑓𝑓𝑓𝑓 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑖𝑖 × 𝑠𝑠𝑒𝑒𝑒𝑒𝑠𝑠𝑠𝑠𝑠𝑠𝑒𝑒𝑠𝑠𝑠𝑠𝑒𝑒𝑓𝑓𝑓𝑓 𝑔𝑔𝑠𝑠𝑠𝑠𝑔𝑔𝑒𝑒𝑖𝑖 + 𝛾𝛾12𝑓𝑓𝑠𝑠𝑠𝑠𝑓𝑓𝑓𝑓 𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑠𝑖𝑖+ 𝛾𝛾13𝑠𝑠𝑒𝑒𝑒𝑒𝑠𝑠𝑠𝑠𝑠𝑠𝑒𝑒𝑠𝑠𝑠𝑠𝑒𝑒𝑓𝑓𝑓𝑓 𝑔𝑔𝑠𝑠𝑠𝑠𝑔𝑔𝑒𝑒𝑖𝑖 + 𝑔𝑔1𝑖𝑖 59 To determine the best-fitting model for each DV, we compared the Aikake Information Criterion (AIC) between the full model and reduced models in which each fixed effect, interaction term, and random slope was removed respectively. Significant main effects and interactions involving experimental group for the best-fitting model were further analyzed by examining simple slopes. For interaction terms with final score, post hoc analyses at the mean and +/- 1 SD were performed. For all models, the level-1 residual variances/covariances followed a first-order autoregressive structure; level-2 residual variances/covariances were unstructured: 𝑠𝑠𝑖𝑖𝑖𝑖~𝑁𝑁(0,𝜎𝜎2) Σ = 𝜎𝜎2⎣⎢⎢⎢⎡1𝜌𝜌 1𝜌𝜌2 𝜌𝜌 1⋮ ⋮ ⋱ ⋱𝜌𝜌19 𝜌𝜌18 … 𝜌𝜌 1 ⎦⎥⎥⎥⎤ �𝑔𝑔0𝑖𝑖𝑔𝑔1𝑖𝑖𝑔𝑔2𝑖𝑖� ~ 𝑀𝑀𝐷𝐷𝑁𝑁��000� , �𝜏𝜏00𝜏𝜏01 𝜏𝜏11𝜏𝜏02 𝜏𝜏12 𝜏𝜏22�� For all analyses except trials completed, data were excluded from any animal that performed <20 trials (Winstanley et al., 2011). Results were deemed to be significant if p values were less than or equal to an α of 0.05. Analyses yielding a p value between 0.05 and 0.09 were reported as a trend. 60 Chapter 3: Audiovisual cues must be predictable and win-paired to drive risky choice 3.1 Introduction As discussed in the general introduction, evidence suggests that cue-induced impairments in cost/benefit decision making may be a critical risk factor for the development and maintenance of behavioural addictions such as gambling disorder. However, the impact of salient audiovisual cues on decision making has not been well characterized. Investigating the learning dynamics of the uncued versus cued rGT using a series of RL models revealed that potentiated learning from the cued rewards does not drive risk preference on the cued rGT, as might be expected (Langdon et al., 2019). Instead, rats on the cued task were relatively insensitive to the time-out penalties, particularly for the risky options featuring lengthy and more frequent penalties. Several theories could explain these results. One possibility is that higher levels of arousal resulting from exposure to the cues persists through the time-out penalties on subsequent trials and thereby alters the processing of the punishment signal. Alternatively, reward-paired cues may change the representation of task structure, such that punishments are not correctly integrated into the stored action-outcome contingencies as the rats learn to choose between the options. Salient cues may cause rats to represent winning outcomes as different “states” to losing outcomes, where learning about one state does not generalize to another (Niv, 2019). In that case, time-out penalties would not appropriately devalue the risky options and rats would tend to choose the options offering the highest per-trial reward. 61 Here, to test these theories and further investigate impairments in risky decision making induced by reward-paired cues, we designed several variants of the rGT to specifically manipulate the complexity and contingency of the outcome-paired cues in the task. In the standard-cued task, the audiovisual cues scale in magnitude and complexity with reward size. To determine whether this scaling is a necessary feature to drive risky choice, we implemented an inverse relationship between cue complexity/magnitude and reward size. We next tested whether cuing all outcomes, ostensibly making trial outcomes more similar and perhaps permitting correct integration into each option’s learned value, would similarly impact risky choice as solely cuing the wins. To test whether increased sensory stimulation is sufficient to increase risky choice, cues were played randomly on 50% of trials, regardless of outcome. Lastly, we paired cues with losses instead of wins to investigate whether win-paired cues are necessary to drive risky choice. A reinforcer devaluation procedure was also utilized at the end of training to determine which cue-outcome associations would lead to inflexibility in choice. 3.2 Additional methods 3.2.1 Task variants Six variants of the task were used in this experiment (n = 28-32 rats per task variant; Figure 3.1). On the uncued task, winning trials were signalled by the illumination of the food magazine alone. On the standard-cued task, reward delivery occurred concurrently with 2-second compound tone/light cues. Cue complexity and variability scaled with reward size, such that the P1 cue consisted of a single tone and illuminated aperture, and the P4 cue consisted of multiple tones and flashing aperture lights presented in four different patterns across rewarded trials. The reverse-cued task featured an inversion of the cue-reward size relationship, such that the longest and most complex cue occurred on P1 winning trials, and P4 winning trials were accompanied 62 by a single tone and illuminated aperture. On the outcome-cued task, all trial outcomes were accompanied by an audiovisual cue (i.e., during reward delivery and at the onset of the time-out penalty). The random-cued task consisted of cues that occurred on 50% of trials, regardless of outcome. Lastly, on the loss-cued task, cues occurred only on losing trials, at the onset of the time-out penalty. Cue complexity and magnitude scaled with reward size/time-out penalty length for the outcome-, random-, and loss-cued variants of the task (i.e., same pattern as the standard-cued task). 63 On the uncued variant, no audiovisual cues were present. The standard task featured audiovisual cues that scaled in complexity and magnitude with reward size. The reverse-cued variant inverted this relationship, such that the simplest cue was paired with the largest reward, and vice versa. Audiovisual cues were paired with both wins and losses for the outcome-cued variant. For the random-cued variant, cues were played on 50% of trials, regardless of outcome. Lastly, for the loss-cued variant, cues were only paired with losing outcomes, at the onset of the time-out penalty. Figure 3.1: Variants of the rat Gambling Task 64 3.2.2 Reinforcer devaluation 128 rats (n = 12-28 per task version) underwent a reinforcer devaluation procedure. This procedure took place across two days. On the first day, half of the rats were given ad libitum access to the sucrose pellets used as a reward on the rGT for 1 hour prior to task initiation. The remaining rats completed the rGT without prior access to sucrose pellets. Following a baseline session day for which no sucrose pellets were administered prior to the task to any rats, the groups were then reversed and the other half were given 1-hour access to sucrose pellets. 3.2.3 Hierarchical modeling of learning from wins and losses The modeling approach taken here follows the methods described in Langdon et al. (2019). Valid choice trials from the first 5 sessions were concatenated into one long session and trial-by-trial preferences were modeled using variations on the Q-learning algorithm from reinforcement learning (Sutton & Barto, 1998). Each model was fit separately to each task variant group, thus allowing for the possibility that different RL models might perform better at predicting choice for each of the groups. Data from 11 rats were excluded due to missing sessions, experimenter error, or technical issues. This left a total of 24 rats in the uncued task group, 32 rats in the standard-cued task group, 25 rats in the outcome-cued task group, and 28 rats in the reverse-, random-, and loss-cued task groups. Each of these models assumes that choice on every trial probabilistically follows latent Q-values for each option, and these are updated iteratively according to the experienced outcomes. For our models, the probability of choosing option Px on each trial follows the learned Q-values for x = [1,2,3,4] according to the softmax decision rule: 𝑒𝑒(𝑃𝑃𝑥𝑥) = 𝑠𝑠𝛽𝛽𝑄𝑄𝑥𝑥∑ 𝑠𝑠𝛽𝛽𝑄𝑄𝑦𝑦4𝑦𝑦=1 , 65 where p(Px) is the probability of choosing option Px, Qx is the learned latent value of option x, and β is the inverse temperature parameter that controls how strongly choice follows the latent Q-values rather than a random (uniform) distribution over the four options. In each learning model, we assume learning of latent Q-values from positive outcomes follows a simple delta-rule update: 𝑄𝑄𝑥𝑥𝑛𝑛𝑛𝑛𝑛𝑛 = 𝑄𝑄𝑥𝑥𝑜𝑜𝑜𝑜𝑜𝑜 + 𝜂𝜂+(𝑅𝑅𝑡𝑡𝑡𝑡 − 𝑄𝑄𝑥𝑥𝑜𝑜𝑜𝑜𝑜𝑜), where η+ is a learning rate parameter that governs the step-size of the update, Rtr > 0 is the number of pellets delivered on a given winning trial, and Qx is the latent value for the chosen option x on a given trial. Q-values for learning from punishments were updated differently depending on the model. In each case, we sought to model the negative impact of time-out penalties on choice by transforming the duration of the penalty into an equivalent “cost” in sucrose pellets. Each model tests a different hypothesis on the transform of the punishments, with a separate negative learning rate η–. These are summarized in Table 3.1. 66 Model name Parameters Punishment update Scaled 4 𝑄𝑄𝑥𝑥𝑛𝑛𝑛𝑛𝑛𝑛 = 𝑄𝑄𝑥𝑥𝑜𝑜𝑜𝑜𝑜𝑜 + 𝜂𝜂−(𝑒𝑒𝑚𝑚𝑡𝑡𝑡𝑡 − 𝑄𝑄𝑥𝑥𝑜𝑜𝑜𝑜𝑜𝑜) Scaled + offset 5 𝑄𝑄𝑥𝑥𝑛𝑛𝑛𝑛𝑛𝑛 = 𝑄𝑄𝑥𝑥𝑜𝑜𝑜𝑜𝑜𝑜 + 𝜂𝜂−(𝑏𝑏 + 𝑒𝑒𝑚𝑚𝑡𝑡𝑡𝑡 − 𝑄𝑄𝑥𝑥𝑜𝑜𝑜𝑜𝑜𝑜) Independent 7 𝑄𝑄𝑥𝑥𝑛𝑛𝑛𝑛𝑛𝑛 = 𝑄𝑄𝑥𝑥𝑜𝑜𝑜𝑜𝑜𝑜 + 𝜂𝜂−(𝜔𝜔𝑥𝑥𝑚𝑚𝑡𝑡𝑡𝑡 − 𝑄𝑄𝑥𝑥𝑜𝑜𝑜𝑜𝑜𝑜) Table 3.1: Summary of RL models The three models used to investigate the learning dynamics of the different rGT variants differed in how the time-out penalties were transformed into an equivalent cost in pellets. In the scaled cost model, a linear transform (m parameter) was applied. In the scaled + offset cost model, a linear transform (m) plus a global offset (b parameter) were used. Lastly, for the independent cost model, each time-out penalty length was transformed independently according to separate weight parameters (𝜔𝜔1,𝜔𝜔2,𝜔𝜔3,𝜔𝜔4 for P1-P4, respectively). 67 In the scaled punishment model, we assume that the equivalent punishment for a time-out penalty on each losing trial scales linearly with the duration of the punishment. Ttr>0 is the time-out penalty duration in seconds on a given losing trial and m is a scaling parameter that maps time-out duration into an equivalent cost in pellets (i.e., has units pellets/s). The scaled + offset model is the same as the scaled punishment model but features an additional offset parameter b, which removes the constraint that the linear transform between time-out penalty duration and equivalent cost is zero for zero duration. Lastly, in the independent model, we model individual punishment weights for each outcome, allowing a nonlinear mapping between experienced duration and the equivalent cost in pellets on each trial. Equivalent costs for each option are controlled independently by ωx for each option Px. For every model, Q-values were initialized at zero for the first session, and we assumed Q-values at the start of a subsequent session (on the next day for example) were the same as at the end of the previous session (i.e., we modeled no intersession effects on learning). Each model was fit to the entire set of choices for each group of rats using Hamiltonian Monte Carlo sampling with Stan to perform full Bayesian inference and return the posterior distribution of the model parameters conditional on the data and specification of the model (Carpenter et al., 2017). In each case, we partially pooled choice data across individual rats in a hierarchical model to simultaneously determine the distribution of individual- and group-level model parameters. We implemented a noncentered parameterization for group-level β, η+, and η− in each model, as this has been shown to improve performance and reduce autocorrelation between these group-level parameters in hierarchical RL models (Ahn et al, 2017). Each model was fit using four chains with 1000 or 2000 steps each (after an initial 1000 burn-in), yielding a total of 4000 or 8000 posterior samples. To assess the convergence of the 68 chains, we computed the 𝑅𝑅� statistic (Gelman et al. 2013), which measures the degree of variation between chains relative to the variation within chains. Across all three models, no group-level or subject-level parameter had 𝑅𝑅� > 1.1, and the mode was 1.00, indicating that for each model all chains had converged successfully. To measure the difference between group-level parameters, we used highest density intervals (HDI; Kruschke, 2014). The HDI is the interval which contains the required mass such that all points within the interval have a higher probability density than points outside the interval. Differences were considered credible when the 95% HDI for the sample difference between two mean estimates did not include zero. To compare the overall performance of each model, we computed the Watanabe–Akaike information criterion (WAIC; Watanabe, 2010), which, like AIC or BIC, provides a metric to compare different models fit to the same dataset. The WAIC is computed from the pointwise log-likelihood of the full posterior distribution (thereby assessing model fit) with a second term penalizing for model complexity. 3.3 Results 3.3.1 Choice analysis Differences in decision making induced by the different cue variants were assessed by comparing the percent choice of the four options (optimal: P1, P2; risky: P3, P4) averaged from 4 sessions at the end of training, once a statistically stable baseline was reached (sessions 35-39, +/- 2). On average, across the rGT task variants, rats showed a preference for the optimal (i.e., reward-maximizing) option P2. However, there was marked variation between the cohorts in preference for the high-risk options P3 and P4. When comparing P1-P4 choice in an omnibus ANOVA, a significant choice x task interaction was observed (F(14,415) = 2.16, p = .009; see 69 Figure 3.2A). Group comparisons showed differences in P1-P4 choice between task versions featuring win-paired cues (standard-cued, reverse-cued, outcome-cued) and those without (uncued, random-cued, loss-cued) (Table 3.2). Average risk score at the end of training differed significantly between tasks (F(5, 170) = 6.62, p < .0001, see Figure 3.2B and Table 3.3). In general, average risk score on tasks featuring win-paired cues was lower than on the uncued task, corresponding to a greater proportion of individual rats with a low or negative risk score, while the random-cued task did not differ significantly from the uncued task. Interestingly, rats that learned the loss-cued task exhibited the greatest preference for the optimal options (highest risk score) among all tasks. Task differences in choice preference may have been driven by increased prevalence of risk-preferring rats on the standard-cued and outcome-cued variants of the rGT, as a significant choice x task x risk status interaction was also observed (F(11,415) = 2.76, p = .002), and only risk-preferring rats exhibited task differences (risk-preferring: F(12,96) = 1.83, p = .05; optimal: F(10,248) = 1.12, p = .35). However, only one post-hoc comparison reached marginal significance among risky rats, likely due to the relatively low number of risk-preferring rats for some task variants. 70 Figure 3.2: Differences in baseline performance between task variants Comparative baseline performance on variants of the rGT. (A) Percent choice of each option in the six rGT task variants. (B) Average risk score shows risk preference is significantly modulated by the presence and contingency of outcome-paired cues, with preference for the high-risk options (P3 and P4) strongly enhanced in task variants in which the audiovisual cues scale with outcome magnitude and occur on winning trials. (C) Premature responding across the rGT variants, and (D) for risky versus optimal decision-makers (all tasks). (E) Latency for reward collection on winning trials across the variants of the rGT. Data are expressed as mean + SEM. 71 Task comparison Mean difference Significance P1 Uncued Standard .17 0.007 Reverse 0.14 0.06 Outcome .18 0.006 Random 0.09 0.45 Loss 0.08 0.62 Standard Reverse -0.03 1.00 Outcome 0.01 1.000 Random -0.08 0.62 Loss -0.09 0.45 Reverse Outcome 0.04 0.98 Random -0.05 0.93 Loss -0.06 0.83 Outcome Random -0.09 0.54 Loss -0.10 0.38 Random Loss -0.01 1.000 P2 Uncued Standard 0.20 0.52 Reverse 0.06 1.00 Outcome 0.21 0.48 Random -0.07 0.99 Loss -0.20 0.52 Standard Reverse -0.14 0.85 Outcome 0.01 1.000 Random -0.27 0.20 Loss -.40 0.01 Reverse Outcome 0.15 0.81 Random -0.13 0.89 Loss -0.26 0.28 Outcome Random -0.28 0.19 Loss -.41 0.01 Random Loss -0.13 0.90 P3 Uncued Standard -0.16 0.47 Reverse -0.20 0.26 Outcome -.35 0.004 Random -0.04 0.999 Loss 0.08 0.95 Standard Reverse -0.04 0.998 Outcome -0.19 0.35 Random 0.13 0.76 Loss 0.25 0.10 Reverse Outcome -0.15 0.65 Random 0.17 0.53 Loss .29 0.04 Outcome Random .32 0.02 Loss .44 0.0002 Random Loss 0.12 0.82 72 Task comparison Mean difference Significance P4 Uncued Standard -0.14 0.37 Reverse 0.06 0.98 Outcome 0.02 1.000 Random 0.11 0.72 Loss 0.14 0.45 Standard Reverse 0.20 0.09 Outcome 0.16 0.28 Random .25 0.01 Loss .28 0.004 Reverse Outcome -0.04 0.996 Random 0.05 0.99 Loss 0.08 0.90 Outcome Random 0.09 0.86 Loss 0.12 0.63 Random Loss 0.03 0.999 Table 3.2: P1-P4 choice comparisons for rGT variants Comparisons of P1-P4 between task variants using Tukey’s honest significant differences (HSD) test. Bolded values indicate a significant difference. Task comparison Mean difference Significance Uncued Standard 45.51 <.0001 Reverse 25.32 .006 Outcome 51.75 <.0001 Random -3.29 .99 Loss -22.23 .03 Standard Reverse -20.19 0.06 Outcome 6.24 0.95 Random -48.81 <.0001 Loss -67.75 <.0001 Reverse Outcome 26.43 0.006 Random -28.62 0.002 Loss -47.56 <.0001 Outcome Random -55.05 <.0001 Loss -73.99 <.0001 Random Loss -18.94 0.11 Table 3.3: Risk score comparisons for rGT variants Comparisons of risk score between task variants using Tukey’s HSD test. Bolded values indicate a significant difference. 73 3.3.2 Premature responding We next tested whether rats trained on each task variant differed in their level of motor impulsivity, measured by the proportion of premature responses made during the 5-second intertrial interval relative to the total number of trials. A significant difference was observed between tasks that was not dependent on risk status (F(5,164) = 5.48, p = .0001; see Figure 3.2C). Post-hoc multiple comparisons showed that rats trained on the loss-cued task had the lowest rate of premature responding compared to all other tasks (Table 3.4). Rats trained on the reverse-cued and random-cued task variants also exhibited a lower level of premature responding compared to the uncued, standard-cued, and outcome-cued rats. Across all task groups, risk-preferring rats had a significantly higher proportion of premature responses that optimal rats (F(1,164) = 23.41, p < .0001; see Figure 3.2D). 3.3.3 Other variables Rats differed in their latency to collect reward across the task variants (F(5,151) = 2.47, p = .04; see Figure 3.2D). Results from the post-hoc multiple comparisons are displayed in Table 3.5, showing that rats trained on the random-cued task were significantly slower to collect reward than all other rats. No differences between task variants were observed in latency to choose an option, trials completed, or omissions. Across all tasks, risk-preferring rats completed significantly fewer trials that optimal rats (F(1,151) = 99.28, p < .0001), as expected given that they experienced a higher number of lengthy time-out penalties. 74 Task comparison Mean difference Significance Uncued Standard -0.02 1.00 Reverse 0.08 0.16 Outcome -0.03 0.96 Random .10 0.02 Loss .24 <0.0001 Standard Reverse .095 0.05 Outcome -0.01 1.00 Random .12 0.005 Loss .25 <0.0001 Reverse Outcome -.11 0.02 Random 0.02 0.98 Loss .16 <0.0001 Outcome Random .13 0.002 Loss .27 <0.0001 Random Loss .13 0.001 Table 3.4: Premature responding comparisons for rGT variants Comparisons of premature responding between task variants using Tukey’s HSD test. Bolded values indicate a significant difference. Task comparison Mean difference Significance Uncued Standard 0.13 0.93 Reverse 0.05 1.00 Outcome -0.09 0.99 Random -.72 <0.0001 Loss -0.02 1.00 Standard Reverse -0.08 0.99 Outcome -0.21 0.61 Random -.84 <0.0001 Loss -0.14 0.90 Reverse Outcome -0.13 0.93 Random -.76 <0.0001 Loss -0.06 1.00 Outcome Random -.63 0.0002 Loss 0.07 1.00 Random Loss .70 <0.0001 Table 3.5: Collect latency comparisons for rGT variants Comparisons of collect latency between task variants using Tukey’s HSD test. Bolded values indicate a significant difference. 75 3.3.4 Reinforcer devaluation Data from the reinforcer devaluation test were compared to a separate baseline session during which no experimental manipulation occurred. A significant devaluation x choice x task effect was observed that was dependent on risk status (devaluation x choice x task x risk status: F(15, 314) = .44, p = .002). This effect was marginally significant in risk-preferring rats (F(15,66) = 1.79, p = .06). Effects broken down by task for risk-preferring animals can be found in Table 5; risk-preferring rats on the uncued, random-cued, and loss-cued tasks were grouped together due to low n (1-3 per task). Among the risky rats, only those trained on tasks without win-paired cues exhibited changes in choice patterns following reinforcer devaluation. In Figure 3.3A, choice of the P1-P4 options in risk-preferring rats are depicted as a difference in % choice between baseline and devaluation sessions (baseline subtracted from devaluation) for each task variant. This was done to highlight shifts in choice separate from overall group differences in the selection of the different options. While the effects of devaluation did not differ by task in optimal rats (F(13,226) = 0.95, p = .50), a marginally significant choice x devaluation effect was observed in these rats (F(3,255) = 2.62, p = .06; see Figure 3.3B), indicating that some degree of shifting occurred in optimal rats that was not influenced by the presence or absence of cues. These observed shifts in P1-P4 choice resulted in a significant task-dependent shift in risk score in risk-preferring rats (devaluation x task: F(5,22) = 3.90, p = .01) but not optimal rats (F(5,85) = 1.53, p = .19). Results are summarized by task in Table 3.7. Similar to the choice results, only rats trained on tasks without win-paired cues exhibited shifts in risk preference following reinforcer devaluation. 76 (A) P1-P4 choice preference after reinforcer devaluation compared to baseline preference for risk-preferring rats. Devaluation did not shift choice patterns selectively in task variants featuring consistent win-paired cues (standard, outcome-cued, reverse-cued). (B) P1-P4 choice preference after reinforcer devaluation compared to baseline preference in optimal rats. Reinforcer devaluation induced a slight shift in choice preference, with no differences found between tasks. Data are expressed as the mean change in % choice from baseline + SEM to highlight effects independent of differences in preference for each option between cohorts. Task F value Degrees of freedom P value Uncued/random/loss (n=7) 4.17 3,9 .04 Standard (n = 7) 0.14 3,15 .93 Reverse (n = 3) 2.98 3,6 .12 Outcome (n = 14) 1.47 3,36 .24 Table 3.6: Devaluation in risk-preferring rats: P1-P4 choice Choice x devaluation interactions for each task in risk-preferring rats. Bolded values indicate a significant difference. Task F value Degrees of freedom P value Uncued/random/loss (n=7) 55.49 1,3 .005 Standard (n = 7) 0.02 1,5 .89 Reverse (n = 3) 4.88 1,2 .16 Outcome (n = 14) 3.45 1,12 .09 Table 3.7: Devaluation in risk-preferring rats: Risk score Risk score x devaluation interactions for each task in risk-preferring rats. Bolded values indicate a significant difference. Figure 3.3: Effects of reinforcer devaluation on choice preference 77 3.3.4.1 Reinforcer devaluation: other variables Reinforcer devaluation impacted multiple measures of motivated behaviour on the task. While latency to collect reward did not shift in response to devaluation (F(1,107) = .55, p = .46), latency to choose an option significantly increased across all tasks (F(1,107) = 71.38, p < .0001), as did omissions (F(1,107) = 9.75, p = .002). Number of completed trials decreased in all rats, particularly optimal decision-makers (devaluation x risk status: F(1,107) = 6.72, p = .01; optimal: F(1,85) = 80.66, p < .0001; risky: F(1,22) = 9.41, p = .006). Premature responding significantly decreased across all groups (F(1,107) = 63.32, p < .0001). 3.3.5 Analysis of task learning using RL models 3.3.5.1 Model comparison These results indicate that salient audiovisual cues reliably paired with wins increase the number of individual rats that display a preference for risky options, and that this pattern of responding is insensitive to devaluation selectively in this subset of rats. Next, we asked how choice preferences on each rGT task variant related to learning dynamics during the initial sessions of each task, and how this differed between risk-preferring and optimal rats. For each task variant, the best-fitting model was assessed using WAIC, with lower WAIC indicating a better explanation of the data. Among the models tested, the independent cost RL model best captured the pattern of choice during the learning phase of each task variant (Fig 3.4; ΔWAIC >0 for all models compared to the independent punishment model), with the scaled + offset cost model performing similarly in the standard-, outcome- and loss-cued task variants. Together, this suggests the subjective cost associated with a loss for each option was decoupled from the experienced duration of the time-out penalty, at least during the initial sessions of learning. 78 Figure 3.4: Differences in WAIC values compared to independent model Difference in WAIC between each model and the independent model for each of the rGT task variants. Lower WAIC indicates a better explanation of the data. Error bars are SEM. 79 To confirm that the best-fitting independent cost model captured the dominant features of the behavioural data, we simulated the probability of each option on each trial for 20 sessions, using the subject-level model parameter estimates (Figure 3.5A). We then calculated the risk score for sessions 18-20 of the simulated data. While fewer significant differences were observed in the simulated data compared to the actual data, the overall pattern of results was largely preserved, with simulated choices on task variants featuring win-paired cues exhibiting lower risk scores than those from the uncued and loss-cued rGT, and simulated choices on the loss-cued task displaying the highest score and thus the strongest preference for the safe options. Notably, and in contrast with the actual data, simulated choices on the random task variant with this model overestimated choice of the risky options and exhibited the lowest risk score, potentially related to issues with model non-identifiability, resulting in highly uncertain parameter estimates for the outcome weights in this setting (see next section). Simulated choices for the scaled + offset cost model also reproduced the qualitative pattern of risk score for the standard-, outcome- and reverse-cued task variants compared to the uncued or loss-cued rGT, though it did not capture the reduction in risk score to the same degree for the win-paired cue task variants, nor is the elevated risk score for the loss-cued task present. However, the random-cued task has a similar risk score to the uncued task, as observed in the actual data. In addition, the uncued task is significantly different from the standard-cued and outcome-cued task, which is not present in the independent model data simulation. 80 Figure 3.5: Simulated risk score using model parameter estimates Average risk score (sessions 18-20) for the independent and scaled + offset models simulated with the subject-level parameter estimates for each task variant. 81 3.3.5.2 Parameter comparison for the two best-fitting models Next, we asked how the model parameters that control learning differed between the rGT task cohorts in these early sessions of training. Since both the independent cost and the scaled + offset cost models performed similarly for task variants in which outcomes were meaningfully cued (i.e., standard-, outcome- and loss-cued), we compared the group-level mean posterior estimates for each parameter for both of these models across the task variants. Differences between parameter estimates were considered credible when the 95% highest-density interval (HDI) for the sample difference between two mean estimates did not include zero. Differences between task variants were mainly found in the learning rate estimates for both the independent cost (figure 3.6) and scaled + offset cost (figure 3.7) models. Notably, pairing cues with wins reduces learning from punishments in these early sessions. All tasks featuring reward-paired cues (standard-, outcome-, reverse-cued tasks) exhibit a much lower negative learning rate than the loss-cued task variant for the independent cost model. Comparisons to the uncued and random-cued tasks for this model are inconclusive, as the unusually low estimate for the negative learning rate and diffuse estimates for the P1-P4 weights indicate this model did not perform well for these datasets. This is also indicated by the data simulation for these tasks, which did not recapitulate the observed behavioural data. Nevertheless, the low negative learning rate for the standard-cued task is clearly present in the scaled + offset model. Thus, win-paired cues can reduce the impact of negative outcomes. Across both models, the loss-cued task exhibited the highest learning rate for loss trials, suggesting that cuing losses increases their impact on subsequent choice. However, if wins are also cued, as in the outcome-cued task, then this increased learning from losses does not occur; for this task, the estimate was similar to cuing wins alone. Learning from wins was also affected 82 by outcome-associated cues. Cuing all outcomes resulted in a high positive learning rate, whereas the reverse-cued task exhibited the lowest estimate. The high level of sensory stimulation occurring in the outcome-cued task may have boosted the effect of wins on decision making. By contrast, the inverted relationship between cue complexity and reward size for the reverse-cued task diminished learning from wins. 83 Asterisks within the inset tables mark parameters for which the 95% HDI of the sample what difference did not contain zero, indicating a credible difference. For each distribution, the box demarcates the interquartile interval and the whiskers demarcate the 95% HDI. Figure 3.6: Group-level posterior estimates of independent cost model parameters 84 Asterisks within the inset tables mark parameters for which the 95% HDI of the sample difference did not contain zero, indicating a credible difference. For each distribution, the box demarcates the interquartile interval and the whiskers demarcate the 95% HDI. Figure 3.7: Group-level posterior estimates of scaled + offset cost model parameters 85 We next tested whether any of the subject-level parameter estimates in the independent or scaled + offset model could reliably predict risk preference scores at the end of training. For the independent model, we found that the parameters controlling the weight of the time-out penalties specifically for the risky options (P3 weights: R2 = .056, F(1,163) = 9.60, p = .002, Figure 3.8A; P4 weights: R2 = .026, F(1,163) = 4.35, p = .04, Figure 3.8B) were predictive of rats’ risk preferences during stable responding in the later sessions. The positive learning rate reached marginal significance (R2 = .021, F(1,163) = 3.53, p = .06). These predictive relationships indicate that risk-preferring rats experienced significantly lower time-out penalty costs for the risky options compared to optimal rats. For the scaled + offset model, the negative learning rate parameter estimates predicted risk score at the end of training (R2 = .029, F(1,163) = 4.80, p = .03, Figure 3.8C). This indicates that a lower negative learning rate was associated with greater risk preference. 86 Subject-level parameter estimates for the risky option weights from the independent model (A,B) and the negative learning rate from the scaled + offset model (C) predicted risk preference at the end of training. Figure 3.8: Subject-level parameter estimates 87 3.4 Discussion Here, we showed that audiovisual cues drive risky choice on the rat gambling task (rGT) only if they are reliably, but not exclusively, win-paired. This was demonstrated by higher levels of risky choice in rats trained on the standard-cued, reverse-cued, and outcome-cued variants of the rGT. Computational analysis of the acquisition phase using reinforcement learning models revealed that differences in decision making were largely captured by parameters that control learning from punishments. These parameters predicted risk score at the end of training, indicating that risk-preferring rats discounted losses to a greater degree than optimal rats. There was also some evidence of increased learning from rewarded trials, particularly when all outcomes were cued. These results largely confirm and build upon the previous report investigating learning dynamics of the cued versus uncued rGT (Langdon et al., 2019). We previously observed that the addition of reward-paired cues to the task resulted in insensitivity to punishments, particularly on the risky options, and that time-out penalty weights could predict risk score at the end of training. The current studies extend these results to suggest that this relationship can be bidirectionally modulated, as loss-paired cues reduce risky choice and increase learning from losing outcomes. Furthermore, pairing cues with wins seems to dominate the decision-making process. Cuing the losses when the wins are also cued has no risk-reducing effect, and in fact may even further potentiate risky choice. While the corroborating results across models lends strength to these conclusions, the independent model did not precisely capture the learning dynamics for some groups, despite being the best fitting for all tasks. In particular, the uncued and random-cued tasks had diffuse group-level estimates for some parameters. The unusually peaked distribution near zero for the 88 negative learning rate and the diffuse distributions for the P1-P4 weights suggest that there was difficulty in isolating the influence of these parameters. This may be because, given the low number of risk-preferring rats on these tasks, there were relatively few of certain trial types (e.g., losses on P3/P4) and the more complex model could not identify precise estimates for the negative learning rate versus the independent weights representing the cost of each time-out penalty. Alternatively, rats on the uncued- and random-cued task may have had a more variable choice pattern early in training because of the nature of the task. Adding predictable cues to the task may encourage consistency in the development of choice patterns, albeit in opposite directions depending on the cued outcome; win-paired cues elicit risky choice, and loss-paired cues elicit risk aversion. Conversely, rats trained on tasks that lack salient cues or with cues that do not carry useful information about outcomes may not be as stereotyped in their behaviour early in training. For example, they could switch strategies as sessions progress, or there may be more subjects who maintain a high degree of variability across sessions. Therefore, cues (provided they have an interpretable relationship with outcomes) may drive consistency and constrain the way in which the task is interpreted. Developing models of learning on the rGT in which parameter estimates evolve over time may allow for potential differences in decision-making variability to come to light. Results from the reinforcer devaluation test provide additional support for differences in decision-making processes when win-paired cues are present, in that risk-preferring rats trained on these tasks were not sensitive to changes in reinforcer value. This indicates that such cues can render choice patterns inflexible. However, no differences were found between tasks for optimal rats, and they were overall less sensitive to changes in reinforcer value. Optimal rats may 89 therefore be indifferent to fluctuations in reward value such as occasional reward omission on the low-risk options, or in this case, reinforcer devaluation. Additionally, if we assume that frequency of winning probability becomes more desirable than reward size, optimal rats can only move to P1 to maximize win frequency, whereas risk-preferring rats have more options to which they may shift. Nevertheless, the fact that risk-preferring rats trained on these tasks did not shift suggests that win-paired cues, particularly when they track reward size, inhibit flexible responding in the face of devaluation of those rewards. In addition to inducing inflexibility, win-paired cues on the outcome-cued and standard-cued tasks impacted choice patterns in a relatively comparable manner. These results run counter to the “state” hypothesis described in the introduction, in which we suggested that increasing the similarity of winning and losing trials may permit better integration of the time-out penalties into the learned values of each option. We instead observed that outcome-cued rats were equally risky as those trained on the standard-cued paradigm. Cuing losses in this paradigm did not increase learning about those trial types; instead, it may have disguised them as wins. Losses disguised as wins (LDWs) are a feature of modern multi-line slot machines in which win-related cues are played when a small payout that is less than the bid is won, leading the player to believe they’ve won money when in fact they have experienced a net loss. These LDWs can therefore be miscategorized as wins (Dixon et al., 2010; Jensen et al., 2013). Marshall and Kirpatrick (2017) applied a reinforcement learning model to behavioural data from their task investigating the LDW effect in rodents and showed that playing win-related sensory feedback during losses elevated stay biases on the high-risk option by increasing its value. A similar mechanism may be at play in the outcome-cued task. Conversely, cuing only losses may oppose the LDW effect, as others have shown that playing loss-associated cues during LDWs can permit subjects to 90 correctly categorize them as losses (Dixon et al., 2015). It is interesting to note that rats on the outcome-cued task had the lowest beta parameter estimate. This may suggest the model did not capture the development of their choice patterns to the same degree as the other tasks. It could be that adding a stay-bias parameter similar to the model by Marshall and Kirkpatrick (2017) would better encapsulate the learning dynamics for rats on this task. An alternate hypothesis for the impact of win-paired cues on decision making comes from research investigating the role of dopamine in the perception of time. The timing of dopamine signals can influence whether subjective time speeds up or slows down (Soares et al., 2016; Jakob et al., 2021). Hence, it could be that cued rewards alter the subjective experience of the time-out penalty duration via a dopaminergic mechanism. Indeed, the standard-cued task is more sensitive to dopamine manipulations (Barrus & Winstanley, 2016). Dopamine signals provoked by win-paired cues may reduce the experienced duration of the time-out penalties such that their impact on the latent value of each option is diminished. Measurements of dopamine signals on-task using fiber photometry could be incorporated into a model to test this hypothesis (see Jakob et al., 2021). While pairing cues with wins is sufficient to drive risky choice, cue complexity and magnitude appear to play a minor role, as rats on the reverse-cued task were significantly less risky than the outcome-cued rats, and marginally different from the standard-cued rats. Additionally, parameter estimates for rats trained on the reverse-cued task did not always align with the other tasks featuring win-paired cues (e.g., lower learning rate for rewarded trials). These rats also exhibited a lower rate of premature responding compared to all other tasks except for the loss-cued task. This may indicate that matching cue size and complexity to reward size can potentiate motor impulsivity. Indeed, when the salience of reward-predictive cues matches 91 the size of the reward, activity in the nucleus accumbens is amplified (Knutson et al., 2001), which may be diminished when cue size inversely scales with reward. As activity within the nucleus accumbens is critically involved in motor impulsivity on similar behavioural tasks (Economidou et al., 2012; Pattij et al., 2007), reduction of this signal could explain the low rate of premature responding in these rats. Consistently pairing cues with wins proved to be a necessary component to induce risky choice, as playing cues randomly on 50% of trials regardless of outcome did not significantly shift risk preference compared to the uncued variant. We originally thought that the increased sensory stimulation from the random cues could increase arousal and therefore risk preference. However, rats instead learned to disregard these cues and were perhaps less engaged in the task, as indicated by the longer latencies to collect reward and reduced levels of premature responding. That being said, this finding does not disprove the hypothesis that increased arousal leads to riskier choice patterns; it may be that the cue-reward relationship increases arousal in a way that random cues cannot. Recent results implicating norepinephrine in cue-induced risky choice would suggest that arousal may contribute to the impact of cues on decision making (Chernoff et al., 2021). It would be interesting to pretrain rats on the association between the cues and reward prior to rGT training on the random-cued task; in that case, increased risk preference may be observed. This may represent an intriguing model of the effect of ambient lights and sounds of a casino on gambling behaviour. Pairing cues with losses would also ostensibly increase arousal, however their behavioural impact was quite distinct from that of win-paired cues. Indeed, rats trained on the loss-cued task were the least risk-preferring out of all the groups, including the uncued task. This would suggest that, while the uncued task is usually regarded as a control for the cued task(s), it 92 may also be a deviation from how optimal rats can be. Indeed, Langdon et al. (2019) found that all rats across both the uncued and standard-cued task were globally less sensitive to the time-out penalties, and that differences in risk preference arose from the degree of this reduced sensitivity. Conversely, rats on the loss-cued task appear to be more sensitive to losses. Thus, they could be more proactively risk-avoiding, and rats on the uncued task may be more willing to sample from the risky options despite having an overall optimal decision-making profile. Overall, the results from these studies indicate that outcome-associated cues play a significant role in decision-making processes, and their effect is highly dependent on the outcome type with which they are associated. Differences in choice patterns are largely a result of changes to the relative impact of losses on decision making, as revealed by the effect of different cue paradigms on group-level parameter estimates capturing learning from losses in the tested reinforcement learning models. These analyses demonstrate the power of combining modeling approaches with careful behavioural manipulations to inform our understanding of action selection in complex decision-making scenarios. Furthermore, the findings provide critical insight into the influence of the rich sensory environment in casinos and other forms of gambling, particularly the addictive allure of electronic gaming machines. 93 Chapter 4: Serotonin 2C antagonism in the lateral orbitofrontal cortex ameliorates cue-enhanced risk preference and restores sensitivity to reinforcer devaluation 4.1 Introduction In Chapter 3, it was found that distinct behavioural differences are apparent when win-paired cues are added to the rGT. Decision making on the cued rGT is also subject to unique pharmacological regulation. Systemic administration of a serotonin (5-HT) 2C receptor antagonist SB242,084 decreased risky choice selectively on the cued rGT while increasing premature responding on both the cued and uncued versions of the task (Adams et al., 2017). This finding was somewhat unexpected, given the previously described role of the 5-HT2C receptor in cue-mediated behaviours such as cue-induced reinstatement of cocaine seeking and responding for a conditioned reinforcer (Pentkowski et al., 2010; Browne et al., 2017). Modulation of both mesolimbic dopamine release and activity within the medial PFC were implicated in these results, and the findings were attributed to alterations in the incentive salience of the cues. Increased premature responding induced by 5-HT2C antagonism on the rGT (Adams et al., 2017) and 5CSRT (Winstanley et al., 2004) may also depend on the nucleus accumbens, as systemic administration enhances accumbal dopaminergic release (Browne et al., 2017), which can increase this form of impulsivity (Pattij et al., 2007; Economidou et al., 2012). Indeed, infusions of a 5-HT2C antagonist into the accumbens of rats increased premature responding on the 5CSRT, whereas microinjections into the medial PFC had little effect (Robinson et al., 2008). 94 These findings suggest that 5-HT2C receptor antagonism in the nucleus accumbens would enhance the control of cues over behaviour, and 5-HT2C agonism would attenuate it. However, we instead found that systemic administration of the antagonist ameliorated the risk-enhancing effect of cues on the rGT, while the agonist was without effect. As such, 5-HT2C antagonism may mitigate cue-driven risky choice through a different mechanism than described above, and distinct neural loci. As described in Section 1.2.2 of the general introduction, 5-HT transmission in the lateral OFC is critically involved in mediating flexible behaviour. Administration of a 5-HT2C antagonist into rats’ lateral OFC but not the medial PFC reduced perseveration during reversal learning (Boulougouris and Robbins, 2010). On the uncued rGT, the lateral OFC is involved in determining the optimal decision-making strategy (Zeeb and Winstanley, 2011). As shown in Chapter 3, win-paired cues render choice patterns insensitive to reinforcer devaluation on the rGT. As such, the inclusion of cues may impair decision making by altering the establishment of accurate action-outcome contingencies in the lateral OFC. Theoretically, manipulating serotonergic activity in this region could reintroduce flexibility in the stored action-outcome contingencies and thereby ameliorate this effect. We therefore administered the 5-HT2C antagonist RS 102221 directly into the lateral OFC and assessed performance on the cued rGT. To confirm the regional specificity of any observed effects, we also targeted the prelimbic region (PrL) of the medial PFC. We hypothesized that intra-lateral OFC, but not intra-PrL, RS 102221 would attenuate risky decision making. We also tested whether intra-lateral OFC RS 102221 could restore sensitivity to reinforcer devaluation, as expected if 5-HT2C antagonism improves decision making by reinstating flexibility. 95 4.2 Additional methods 4.2.1 Surgery When baseline performance was deemed statistically stable (following approximately 40 training sessions), 32 animals were anesthetized with 2% isoflurane in O2 and 23-gauge stainless steel guide cannulae were implanted above the lateral OFC (n = 20; AP = +3.5 mm, ML = ±2.6 mm from bregma, DV = −2.9 mm from dura) or the PrL (n = 12; AP = +3.0 mm, ML = ±0.7 mm from bregma, DV = −2.8 mm from dura), using standard stereotaxic techniques. Guide cannulae were fixed to the skull via 4 stainless steel screws and dental acrylic, and obdurators flush with the end of the cannulae were inserted. Animals were given at least 1 week of recovery in their home cages before subsequent testing. 4.2.2 Drug preparation Three concentrations of the compound RS 102221 hydrochloride (Tocris Bioscience, MN) were prepared each dosing day. Firstly, 1 mg of the drug was suspended in 300 μL of 0.1M HCl via sonication. The pH level was then adjusted to 6-7 with 1.0 M and 0.1 M NaOH and saline to a final concentration of 2 mg/mL. Two aliquots of the solution were further diluted to 0.2 and 0.6 mg/mL. The highest concentration dose was vortexed prior to each infusion to prevent precipitation of the drug during the procedure. The vehicle solution consisted of saline that was pH-adjusted to 6-7 with NaOH. 4.2.3 Microinfusion procedure Following recovery, animals performed 10 free-choice sessions, after which all individuals displayed stable behaviour. Animals were then habituated to the microinfusion process with 2 mock infusions, during which 30-gauge dummy injectors were inserted for 2 min but no infusion was performed, followed by a behavioural testing session initiated 10 min later. 96 Infusions adhered to a 3-day cycle starting with a baseline session, followed by a drug or vehicle injection session, and then by a non-testing day. 0.5 μL per hemisphere injections of saline or RS 102221 (0.1, 0.3, or 1.0 μg of drug per hemisphere) were administered bilaterally at a rate of 0.3 μL/min with injectors that extended 0.8 mm beyond the guide cannulae. Injectors were left in place for an additional minute to allow for diffusion. Animals received each dose of RS 102221 plus vehicle, counterbalanced in a Latin Square design (For doses A thru D: ABCD, CADB, BDAC, DCBA). Once the microinfusions were completed, injectors were removed, obdurators replaced, and animals were placed in the operant chambers for 10 min prior to initiation of the cued rGT. 4.2.4 Reinforcer devaluation Twelve lateral OFC-cannulated rats and 16 surgically-naïve rats underwent a reinforcer devaluation procedure. For the naïve rats, methods followed those described in Chapter 4, Section 3.2.2. In cannulated rats, to prevent the accumulation of damage of multiple infusions from impacting the results of this procedure, only one session of reinforcer devaluation was completed. Ten min prior to task initiation, half of the rats received 1.0 μg of RS 102221 per hemisphere, according to the microinfusion procedure specified above. The other half received a vehicle dose. All rats in this group were given ad libitum access to sucrose pellets for 1 hour prior to task initiation. 4.2.5 Histology Following completion of all behavioural testing, animals were anesthetized with isoflurane and euthanized by carbon dioxide exposure. Brains were extracted and fixed in 4% formaldehyde for at least 24 hours, transferred to a 30% sucrose solution, and then frozen and cut via cryostat into 40-μm coronal sections. These sections were stained with cresyl violet for 97 visualization, and the projected locations of the injector tips protruding from the guide cannulae were mapped onto standard sections from Paxinos and Watson (1998). 4.3 Results 4.3.1 Cannulae placements The locations of all acceptable placements are depicted in Figure 4.1A for the lateral OFC cohort and 4.1B for the PrL cohort. One animal in the lateral OFC condition did not survive surgery. One rat was excluded from the lateral OFC analyses due to inaccurate placement of the cannulae. All PrL cannulae placements were acceptable. This left a total of 18 (n = 9 risk-preferring; n = 9 optimal) and 12 (n = 6 risk-preferring; n = 6 optimal) rats for the lateral OFC and PrL analyses, respectively. 98 Location of all acceptable lOFC (A) and PrL region (B) infusions and an example microscopy image for each region in C and D are shown. Coordinates are relative to bregma. Plates modified from Paxinos and Watson (1998). Figure 4.1: Histological analysis of cannulae implantation 99 4.3.2 Baseline behaviour One rat was excluded from the PrL condition due to poor task performance (mean of 30 trials completed per session, >30 s reward collection latency). As expected, risk-preferring rats selected the risky options at a significantly higher proportion than those who developed an optimal decision-making strategy for all baseline and saline sessions (choice x risk preference: F(2,148) = 41.77, p<.0001; optimal vs risk-preferring: P1: t(75) = 1.60, p = .11; P2: t(75) = 12.89, p<.0001 ; P3: t(68) = -5.95, p<.0001; P4: t(66) = -3.97, p = .0002). Across both cohorts, risk-preferring rats completed significantly fewer trials (risk preference: F(1,25) = 38.43, p<.0001), had a significantly higher proportion of premature responses (risk preference: F(1,25) = 6.76, p = .02), and exhibited shorter latencies to collect reward (risk preference: F(1,25) = 14.67, p = .001). The risk-preferring and optimal rats in the lateral OFC and PrL cohorts exhibited slightly different choice patterns (choice x brain region x risk preference: F(2,51) = 5.11, p = .009). This was particularly evident in optimal rats; risk-preferring rats exhibited only a trending difference in choice preference (risk-preferring rats: choice x brain region: F(2,17) = 3.45, p = .07, see Figure 4.2A; optimal rats: choice x brain region: F(3,38) = 3.99, p = .02, see Figure 4.2B). Optimal rats in the PrL group chose P2 at a significantly higher rate than rats in the lateral OFC group (t(12) = 2.92, p = .01). To assess whether damage associated with cannula implantation and the microinfusion procedure impacted decision making on the rGT, 4 pre-surgery sessions were binned and compared to baseline data collected between infusion days. No effect was observed on P1-P4 choice, indicating that procedure-associated damage did not significantly impact their decision making (data bin x choice: F(3,66) = 0.40, p = .75). 100 Figure 4.2: Choice preferences at baseline Baseline preferences for each option in the lOFC and PrL cohorts in risk-preferring rats (A) and optimal decision-makers (B). Optimal rats in the PrL cohort selected P2 at a significantly higher rate than optimal rats in the lOFC cohort. Data are expressed as mean + SEM. *p < .05 according to an independent-groups t test. 101 4.3.3 Microinfusions into the lateral OFC 4.3.3.1 Choice We observed a significant shift in choice when comparing all doses in an omnibus ANOVA, that was dependent on the rats’ choice patterns at baseline (dose x choice x risk status: F(7,117) = 7.34, p = .04). This effect was only present in risk-preferring rats (dose x choice, risk-preferring: F(9,72) = 3.09, p = .003, see Figure 4.3A; optimal: F(9,72) = 0.01, p = .30; see Figure 4.3B). Post hoc analyses revealed a significant reduction in P3 choice (t(8) = 2.49, p = .04) and a significant increase in P1 choice (t(8) = -2.74, p = .03) in risk-preferring rats, when comparing vehicle to the highest dose. This resulted in a significant increase in risk score in these rats (dose: F(3,24) = 4.39, p = .01; vehicle vs 1.0 ug: t(8) = -2.53, p = .04). No effect was observed on P2 or P4 choice (P2: t(8) = -1.52, p = .17; P4: t(8) = -1.59, p = .15). 4.3.3.2 Premature responding and other variables No effect on premature responding was observed (dose: F(3,48) = 1.10, p = .36). A significant shift in omissions was evident in optimal rats only (dose x risk status: F(3,38) = 3.66, p = .03; dose, optimal: F(3,21) = 3.56, p = .03; risk-preferring: F(1,11) = 1.20, p = .32). Post hoc analyses revealed a significant reduction in omitted trials in these rats, when comparing vehicle to the highest dose (t(8) = 2.40, p = .04). However, these rats showed a significantly higher level of omissions at vehicle compared to their baseline data (t(8) = 2.41, p = .04). No effect was observed on latency variables or trials completed (all F < 1.33; p > .27; see Table 4.1). 102 Figure 4.3: Intra-lOFC RS 102221: P1-P4 choice Effects of intra-lOFC infusions of the 5-HT2C receptor agonist RS 102221 in (A) risk-preferring animals (n = 9) and (B) optimal decision-makers (n = 9) on P1-P4 choice. Antagonism of the 5-HT2C receptor in the lOFC decreased P3 choice and increased P1 choice on the cued rGT in risk-preferring rats. Data are expressed as mean + SEM. *p < .05 compared to vehicle Region Dose Premature responses Choice latency Collect latency Omissions Trials completed lateral OFC 0 28.62 ± 4.05 1.28 ± 0.19 1.32 ± 0.07 1.50 ± 0.53 75.38 ± 5.28 0.1 29.66 ± 3.65 1.20 ± 0.12 1.19 ± 0.04 0.94 ± 0.22 77.56 ± 5.11 1.0 31.90 ± 4.71 1.22 ± 0.12 1.24 ± 0.03 0.39 ± 0.18 75.63 ± 5.05 3.0 25.92 ± 3.72 1.30 ± 0.13 2.01 ± 0.42 0.94 ± 0.51 83.63 ± 6.41 PrL 0 34.05 ± 5.35 1.13 ± 0.12 1.42 ± 0.16 1.36 ± 0.72 69.98 ± 7.94 0.1 31.94 ± 6.59 1.14 ± 0.12 1.29 ± 0.05 0.91 ± 0.44 73.37 ± 9.67 1.0 28.64 ± 6.30 1.31 ± 0.17 1.21 ± 0.07 1.18 ± 0.46 75.41 ± 10.12 3.0 24.19 ± 5.00 1.34 ± 0.15 1.24 ± 0.12 0.91 ± 0.44 78.66 ± 10.78 Table 4.1: Intra-lOFC RS 102221: Other task variables Mean values of all other task variables (% premature response, choice latency, collect latency, omissions, trials completed) at each dose of RS 102221 delivered into either the lOFC or PrL. Data are mean ± SEM. 103 4.3.3.3 Reinforcer devaluation Behavioural data from devaluation sessions for each group of rats (naïve, vehicle, or drug) was compared to their baseline data (naïve group: no manipulation; vehicle group: vehicle dosing data; drug group: highest RS 102221 dose data). In Figure 4.4A, choice of the P1-P4 options are depicted as a difference in % choice between baseline and devaluation sessions (baseline subtracted from devaluation) for each group. This was done to highlight shifts in choice separate from overall cohort differences in the selection of the different options. Mean values and SEMs for each session and group can be found in Table 4.2. Risky and optimal rats are grouped together as statistical analyses did not reveal any effects that were dependent on risk status. We observed a significant shift in P1-P4 choice in response to reinforcer devaluation that was dependent on group but not risk status (devaluation x choice x group: F(9,57) = 2.56, p = .02; see Figure 4.4A). Naïve rats did not demonstrate a shift in their choice profile (devaluation x choice: F(3,36) = 1.58, p = .21). Similarly, we did not observe any change in the choice profile of rats that received a vehicle dose in addition to devaluation, versus vehicle alone (devaluation x choice: F(3,9) = 1.39, p = .31). Conversely, rats in the drug + devaluation condition exhibited a shift in their decision making that was significantly different from the effect of the drug alone (devaluation x choice: F(3,12) = 5.64, p = .01). When comparing P1-P4 choice between the drug versus drug + devaluation condition with post hoc analyses, we observed a trending reduction in P2 choice (t(5) = 2.35, p = .07) and a significant increase in P4 choice (t(5) = -3.76, p = .01). A comparison of the drug + devaluation versus vehicle condition in these rats reached marginal significance (devaluation x choice: F(2,6) = 3.80, p = .08), resulting from a significant increase in P1 and P4 choice (P1: t(5) = -4.92, p = .004; P4: t(5) = -5.95, p = .002). Thus, rats who received an infusion of the highest dose of RS 102221 into the lateral OFC were uniquely 104 sensitive to the effects of reinforcer devaluation on choice, exhibiting a choice profile that differed from their decision-making patterns after receiving either vehicle or drug alone. We observed a significant shift in trials completed that was dependent on group (group x devaluation: F(3,19) = 6.25, p = .004; Figure 4.4B). Only rats that received intra-lateral OFC RS 102221 completed significantly fewer trials in response to devaluation (drug: F(1,4) = 24.12, p = .008; vehicle: F(1,3) = 4.64, p = .12; naïve: F(1,14) = 2.50, p = .14). Rats in all groups exhibited decreased premature responding (F(1,19) = 39.37, p < .0001; Figure 4.4C) and increased latencies to choose an option (devaluation: F(1,19) = 17.52, p = .001; Figure 4.4D) in response to reinforcer devaluation. No effect was observed on omissions or latencies to collect reward (Figure 5E; all F < 1.16, all p > .30). See Table 4.3 for the mean values and SEMs for the reported variables in each group and session. 105 Only rats receiving intra-lOFC RS 102221 shifted their choice profile (A) and completed significantly fewer trials (B) in response to devaluation. Animals in all groups exhibited a significant reduction in premature responding (C) and increased choice latencies (D). No shift in latencies to collect reward were observed in any group (E). In panel A, data are expressed as the mean change in % choice from baseline + SEM to illustrate effects independent of differences in preference for each option between cohorts. Data are expressed as mean ± SEM in all other panels. *p < .05, **p < .01, **** p < .0001, # p = .07 compared to each group’s own baseline Figure 4.4: Intra-lOFC RS 102221: Reinforcer devaluation 106 Group Condition P1 P2 P3 P4 Surgically-naïve Baseline 4.82 ± 1.33 51.72 ± 9.17 24.93 ± 8.68 18.53 ± 7.18 Devaluation 3.62 ± 1.33 53.50 ± 9.25 22.09 ± 8.12 20.79 ± 6.66 Drug + devaluation Vehicle 1.59 ± 0.79 56.57 ± 7.93 25.91 ± 11.55 15.93 ± 8.51 Drug 7.13 ± 1.67 58.16 ± 5.44 18.67 ± 8.12 16.04 ± 6.56 Devaluation 9.29 ± 1.73 37.67 ± 10.06 18.94 ± 8.34 34.10 ± 12.09 Vehicle + devaluation Vehicle 4.83 ± 2.46 41.01 ± 13.12 40.03 ± 19.23 14.13 ± 5.56 Devaluation 4.92 ± 1.72 39.91 ± 10.92 31.19 ± 17.17 23.98 ± 9.85 Table 4.2: Reinforcer devaluation: P1-P4 choice Percent choice of the different options in response to acute sucrose pellet devaluation in surgically-naïve rats, rats who received intra-lOFC RS 102221 in addition to devaluation, and rats who received intra-lOFC vehicle and devaluation. Data are mean ± SEM. Mean values of all other task variables (% premature response, choice latency, collect latency, omissions, trials completed) in response to acute sucrose pellet devaluation in surgically-naïve rats, rats who received intra-lOFC RS 102221 in addition to devaluation, and rats who received intra-lOFC vehicle and devaluation. Data are mean ± SEM. Group Condition Premature responses Choice latency Collect latency Omissions Trials completed Surgically-naive Baseline 27.96 ± 3.00 0.93 ± 0.08 1.12 ± 0.11 0.25 ± 0.11 79.89 ± 7.63 Devaluation 8.82 ± 2.09 1.71 ± 0.21 1.38 ± 0.15 1.31 ± 0.55 63.14 ± 9.87 Drug + devaluation Vehicle 26.72 ± 5.72 1.35 ± 1.20 1.57 ± 0.35 1.50 ± 1.11 80.87 ± 11.71 Drug 19.96 ± 6.94 1.26 ± 0.25 1.23 ± 0.08 0.50 ± 0.34 92.18 ± 9.46 Devaluation 6.61 ± 2.81 2.40 ± 0.52 1.51 ± 0.06 1.33 ± 0.49 32.00 ± 7.93 Vehicle + devaluation Vehicle 37.74 ± 5.72 1.10 ± 0.21 1.33 ± 0.19 1.20 ± 0.97 71.06 ± 7.63 Devaluation 14.00 ± 6.68 2.04 ± 0.43 1.31 ± 0.14 0.80 ± 0.37 51.62 ± 5.40 Table 4.3: Reinforcer devaluation: Other task variables 107 4.3.4 Microinfusions into prelimbic region 4.3.4.1 Choice When examining choice in an omnibus ANOVA, we observed a significant effect of dose that did not interact with risk status or the different options (dose: F(3,27) = 3.39, p = .03), potentially indicative of increased variability or noise in rats’ response patterns that doesn’t reliably load on one option or another. Figure 4.5A and 4.5B depicts the % choice of each option at each dose in risk-preferring and optimal rats, respectively. Comparing the choice of P1-P4 between vehicle and each dose did not reveal any significant effects (all t < 1.99, all p > .09). Correspondingly, there was no significant effect on risk score (dose: F(3,27) = 0.64, p = .60). These results indicate that while rats’ decision-making patterns became more variable across doses, there was no clear pattern of an increase or decrease of choice for any particular option. 4.3.4.2 Premature responding and other variables No significant effect was observed on premature responding, latency variables, omissions, or trials completed (all F < 1.68; p > .20; see Table 4.1 for mean values and SEMs of task variables at each dose). 108 Figure 4.5: Intra-PrL RS 102221: P1-P4 choice Effects of intra-PrL infusions of the 5-HT2C receptor agonist RS 102221 in (A) risk-preferring animals (n = 5) and (B) optimal decision-makers (n = 6) on P1-P4 choice. Antagonism of the 5-HT2C receptor in the PrL did not significantly increase or decrease choice of any particular option in either risk-preferring (A) or optimal (B) rats. Data are expressed as mean + SEM. 109 4.4 Discussion Results from this study demonstrated that infusing a 5-HT2C antagonist directly into the lateral OFC, but not the PrL region, improved decision making in risk-preferring rats without negatively impacting impulsivity measures. 5-HT2C antagonism in the lateral OFC also restored behavioural sensitivity to reinforcer devaluation, indicating that flexibility in reward valuation was increased by this manipulation. In contrast, intra-PrL infusions of RS 102221 did not increase optimal choice. The ability of 5-HT2C antagonism to ameliorate cue-enhanced risky choice therefore exhibits at least some regional specificity within the frontal cortices. While the damage associated with surgeries and infusions is typical of studies using this technique, we cannot rule out that this may have altered the functioning of prefrontal circuitry and influenced the observed results. Nevertheless, within-subjects comparisons to vehicle dosing, along with counterbalanced orders of doses among the rats, supports the conclusion that the effect of 5-HT2C antagonism in the lateral OFC is not simply due to procedure-associated damage. Furthermore, choice patterns during baseline sessions did not shift throughout the infusion procedure and were not significantly different from data collected prior to surgery. Although the ratio of 5-HT2A to 5-HT2C receptors in the medial PFC is positively correlated with levels of premature responding on a simplified version of the 5CSRT (Anastasio et al., 2015), we did not see any increase in premature responding when RS 102221 was delivered into the PrL, similar to a previous report (Robinson et al., 2008). There were fewer rats in the PrL cohort (n = 11) than the lateral OFC cohort (n = 18), but the sample size for both experiments was certainly comparable to other studies using this technique. Considerable data suggest an association between 5-HT2C receptor activity in the medial PFC, motor impulsivity, and relapse-like cocaine-seeking in rats (Filip and Cunningham, 2003; Anastasio et al., 2014; 110 Swinford-Jackson et al., 2016). Furthermore, increased risky decision making following cocaine self-administration correlates with at least one measure of relapse vulnerability (Ferland and Winstanley, 2017). Given that both greater risky choice and higher levels of premature responding are associated with behavioural markers of cocaine addiction in rats (Section 1.3.5) and are significantly correlated at the population level (Barrus et al., 2015), we might expect a greater overlap in the neurobiological control of these cognitive processes. However, numerous studies now show that these phenomena are subject to differential neural and pharmacological regulation (e.g., Zeeb et al., 2009; Barrus and Winstanley, 2016; Adams et al., 2017; Betts et al., 2021; Chernoff et al., 2021). While impulsive action and risky decision making may interact synergistically in the manifestation of impulse control and addiction disorders, they ultimately may represent somewhat dissociable pathways to addiction. In contrast to the effects of systemic administration, which impacted both optimal and risk-preferring rats, intra-lateral OFC administration of a 5-HT2C antagonist only improved decision making in those that exhibited a preference for the risky options at baseline. The neural locus whereby 5-HT2C antagonism further optimizes choice in animals already making advantageous decisions remains to be determined, but is clearly neither the lateral OFC nor the PrL. The selective effects of drug infusions in risk-preferring rats also suggests that the neural architecture underlying the decision-making process in optimal vs. risky rats differs – either in the regions forming the network, the weight given to the output of those regions in guiding choice, or the computational analyses performed by key nodes. Indeed, data across species support the view that individual differences in choice preference can be attributed to differential activity across brain networks. Cues present in the environment can also influence the adoption of a behavioural strategy. The considerable literature on sign-trackers vs goal-trackers, as 111 discussed in Section 1.4 of the general introduction, may best exemplify the significant individual differences in how such cues can be used to guide behaviour (Flagel et al., 2009; Saunders and Robinson, 2010; Flagel et al., 2011; Saunders and Robinson, 2013). Illuminating a cue light during lengthy delays between response and reward delivery can decrease delay discounting (i.e., reduction in a reward’s subjective value due to waiting period) in rats (Cardinal et al., 2000). However, the presence of this cue does not eliminate the large individual differences in animals’ preferences for smaller-sooner vs larger-later rewards. Inactivation of the lateral OFC only reduced choice of the larger-later reward in rats that showed a high baseline preference for this option; an effect that was also observed following local infusions of dopamine antagonists (Zeeb et al., 2010). Thus, the hypothesis that recruitment of the lateral OFC into the decision-making process depends on the presence of cues, and that decision-making patterns are only lateral OFC-dependent in a subpopulation of individuals that use those cues to guide behaviour, has some precedent. It is interesting to note that the effect in risk-preferring animals was specific to a decrease in P3 (their preferred risky option) and an increase in P1, the optimal option offering the most frequent wins but the smallest reward size (one sucrose pellet). This could be due to either an increased sensitivity to the length and/or frequency of time-out penalties, or increased impact of frequent winning trials. This may help explain the specific effect observed in risk-preferring animals; for optimal rats, the difference between P1 and P2 in time-out penalty length/frequency and reward frequency may not be large enough to shift decision-making patterns away from their preferred option. Investigating neural activity following rewards and time-out penalties on each option within the lateral OFC could shed light on these hypotheses. 112 Indeed, identifying how audiovisual cues modulate lateral OFC neuronal firing as well as the impact of 5-HT2C antagonism are important next steps. The 5-HT2C receptor is an excitatory GPCR found throughout the rat central nervous system (Clemett et al., 2000). Previous studies have demonstrated that 5-HT2C receptors are primarily located in the deep layers of the prefrontal cortex in rats (Pompeiano et al., 1994; Liu et al., 2007). In the medial PFC, at least 50% of receptors are localized to GABAergic interneurons, and are hypothesized to regulate the output of pyramidal cells (Liu et al., 2007). Considerably less is known about the localization and function of 5-HT2C receptors in the OFC. Interneurons in this region play a key role in reversal learning, so it is possible that modulation of GABAergic interneuron activity by the 5-HT2C antagonist may drive the improvement in choice seen here (Bissonette et al., 2015). Previous literature has strongly implicated the 5-HT2C receptor in cue-mediated behaviours such as cue-induced reinstatement of cocaine seeking and responding for a conditioned reinforcer through its regulation of the mesolimbic dopamine system (Section 1.4.2; Pentkowski et al., 2010; Browne et al., 2017). Based on this work, 5-HT2C receptor antagonists should increase motor impulsivity and potentiate the risk-promoting effect of cues in the rGT. While the former is true, the latter is clearly not. 5-HT2C receptor antagonism must therefore alter decision-making through an alternate, yet concurrent mechanism. The current data implicate the lateral OFC as one critical locus, and an increase in behavioural flexibility as a potential cognitive mechanism, through which decision making may improve. As noted in the general introduction, serotonergic activity within the OFC, and the 2C receptor, have been implicated in cognitive flexibility by multiple previous studies (Boulougouris and Robbins, 2010; Alsiö et al., 2015; Barlow et al., 2015; Alsiö et al., 2021). Previous results have indicated that circuitry between the OFC and basolateral amygdala (BLA) supports shifts in choice following reinforcer 113 devaluation on the uncued rGT (Zeeb and Winstanley, 2013). It may be that this circuitry is also involved in cue-induced inflexibility on the task. This pathway certainly plays a role in cue-based decision making, as BLA-OFC projections are essential for guiding decision making based on cue-triggered reward representations (Lichtenberg et al., 2017). As discussed in Section 1.3.1, insensitivity to outcome devaluation suggests that the behaviour in question is under habitual rather than goal-directed control. However, it could be that the behaviour is instead reinforced by another aspect of the environment. Given that the cues are concurrent with reward delivery, it could be argued that “reward + cue” has formed a compound reinforcer. Selective devaluation of only one component of this reinforcer may therefore fail to alter response patterns. As outlined in Section 1.4, devaluation of drug taking does not necessarily translate to the devaluation of drug-associated cues that have acquired incentive salience. However, this is believed to be a highly aberrant state, driven by supraphysiological drug-induced dopamine release amplifying associative learning between cues and drugs. Why sound and light cues should exert the same effect in the current task is unclear, given that rats typically show a dramatic reduction in responding for reward-paired cues following devaluation of that reward (Hatfield et al., 1996; Pickens et al., 2005; McDannald et al., 2014). This is not the first time that similarities have been observed between responding on the cued rGT and responding for addictive drugs, as there is some evidence that risky choice on-task and cocaine self-administration may cross-sensitize and/or substitute for one another (Ferland et al., 2019; Hynes et al., 2021). However, systemic 5-HT2C antagonism increases responding for drug (Fletcher et al., 2002), yet decreases risky choice here, indicating that the pharmacological regulation of these processes is not uniform. 114 Instead of the cues acquiring incentive motivational properties that are now independent of reward value, another alternative is that these highly salient audiovisual cues overshadow sucrose pellet delivery to some extent (Pavlov, 1927), such that a change in the sucrose pellet value doesn’t dominate behaviour in the presence of the cues. To our knowledge, neither the lateral OFC nor 5-HT2C receptor signalling have been evaluated in overshadowing experiments, and as such further discussion of this hypothesis is premature. If overshadowing is taking place, then the attenuation of learning about the devalued state of the reward is highly specific to the decision-making process, as devaluation still impacted latencies and motor impulsivity, indicating a reduction in the ability of the sucrose pellet rewards to motivate and invigorate behaviour. Given that 5-HT2C receptor antagonism can clearly increase incentive motivation for rewards and reward-paired cues, likely through actions in the mesolimbic dopamine pathway, the following question remains: why doesn’t this mechanism dominate the decision-making process? One potential answer comes from human literature examining the impact of serotonin depletion on model-based behaviour: goal-directed choice is impaired when learning from rewards, but enhanced when learning from punishments (Worbe et al., 2016). Computational modeling analyses have revealed that the addition of cues to the rGT specifically impairs learning from the time-out penalties (Chapter 3; Langdon et al., 2019). As such, manipulating serotonergic activity within the lateral OFC may promote the correct integration of punishments into the stored action-outcome contingencies for risk-preferring rats, rather than influencing the incentive motivation of the cues. This is in line with the finding that responding for a conditioned reinforcer does not correlate with risky choice on the cued rGT (Ferland et al., 2019), indicating that incentive motivation is not primarily responsible for cue-induced risky choice on this task. The activity of 115 striatal neurons, although influenced by neuromodulators like dopamine, is still primarily driven by cortical inputs. It is therefore possible that 5-HT2C-receptor mediated modulation of lateral OFC output is sufficient to dominate the behavioural response. Overall, these results indicate that 5-HT2C antagonism in the lateral OFC can ameliorate cue-induced disadvantageous risky choice in rats with pre-existing preferences for these risky options. The lack of effect in optimal rats, together with computational modeling analyses presented in Chapter 3, suggests there are underlying differences in the processing or storage of action-outcome contingencies between optimal and risk-preferring animals. It is currently unknown whether this is due to distinct activity patterns within the lateral OFC or differential involvement of downstream targets. Future studies could examine activity in the lateral OFC during cued rGT learning and performance in risk-preferring vs optimal rats to address this question. Furthermore, these results suggest that targeting flexibility may be a viable approach to improving decision making in individuals with impaired cost/benefit decision making, specifically in the presence of cues. This would have implications for the treatment of behavioural addictions and substance use disorders, in which individuals show marked impairments in disadvantageous risky decision making and processing of reward-associated cues (Goudriaan et al., 2005; Limbrick-Oldfield et al., 2017; Zilberman et al., 2019). Recent clinical trials for the 5-HT2C agonist in the treatment of substance use disorder have been unsuccessful. Results from these studies indicate the necessity to attend to individual differences, as only rats with pre-existing deficits in decision making show improvements in response to the antagonist. Greater specificity of targeting to regions may also improve treatment, as the antagonist can increase impulsivity via other pathways. Allosteric modulators of the 5-HT2C receptor may be 116 worth pursuing in this regard. Given that numerous effective psychoactive medications act on the serotonin system, there is every reason to feel cautiously optimistic that a viable serotonergic medication could be developed for disorders hallmarked or exacerbated by risky decision making. 117 Chapter 5: Modulating activity in the lateral orbitofrontal cortex differentially impacts risky decision making in the presence versus absence of reward-paired cues 5.1 Introduction On the rGT, salient audiovisual cues paired with reward render rats insensitive to reinforcer devaluation (Chapter 3, Chapter 4). Perseverative responding is also present in individuals with gambling disorder (de Ruiter et al., 2009). As such, the ability of reward-paired cues to increase risky choice and reduce flexibility in decision making could be a critical factor in the progression from casual to problem gambling. The lateral OFC has been shown to mediate both cognitive flexibility and cue-guided learning (sections 1.2.2 and 1.3.3.2). Thus, this region may be involved in the interaction between reward-paired cues, risky choice, and inflexible responding. The lateral OFC has been implicated in the evaluation of available options in the environment (Rudebeck & Murray, 2014). It appears to be particularly involved in the creation of a model of the latent structure of the environment, when reward probability must be inferred (Zhou et al., 2021; Takahashi et al., 2009). Thus, the lateral OFC is involved in learning from past experience to guide choice when outcomes are uncertain, rather than choosing between options with known probabilities. Perseverative responding resulting from lateral OFC inactivation is likely based on the inability to update expected reward probability, rather than outcome-independent repetition of choice, as others have shown that the lateral OFC activity can bias choice towards a risky option following a win, but is not involved in the repeating a choice based on its spatial location (Constantinople et al., 2019). This again demonstrates this region’s 118 role in tracking action-outcome contingencies to estimate the probability of reward under conditions of uncertainty. The lateral OFC may be critical to differences in decision making on the uncued versus cued rGT. Lesions of the lateral OFC prior to task training on the uncued rGT results in difficulties in discriminating between the two low-risk, low-reward options to determine the best overall choice early in training; lesions after the task is learned do not impact decision making (Zeeb & Winstanley, 2011). This indicates that at least on the uncued rGT, the lateral OFC is involved in determining the optimal strategy when reward probabilities are ambiguous. Conversely, on the cued rGT, modulating serotonergic transmission in the lateral OFC can reinstate flexible responding and reduced risky choice in risk-preferring rats (Chapter 4). This suggests that the lateral OFC plays a role in cue-induced impairments on decision making in this task. However, its differential contributions to the development of choice preference in the presence versus absence of reward-paired cues are currently unknown. Thus, this chapter’s studies employed chemogenetics to investigate the impact of inhibition and activation of the lateral OFC on both the uncued and cued rGT. 5.2 Additional methods 5.2.1 Viral vector injections To assess the impact of modulating lateral OFC activity on the uncued versus cued rGT, Designer Receptors exclusively activated by designer drugs (DREADDs) techniques were used (Armbruster et al., 2007; Smith et al., 2016). Viral vectors were used to insert either an inhibitory receptor, an excitatory receptor, or a control fluorescent tag into neurons within the lateral OFC (n = 16 male rats per group). These receptors are exclusively activated by systemic administration of an exogenous ligand, thereby granting experimental control of neuronal 119 activity within this region. Rats were anesthetized with isoflurane (5% induction, 2-3% maintenance) prior to viral infusion surgeries. Ketoprofren (5 mg/kg) was given as an analgesic and bupivicaine was administered at the surgical site for local anesthesia. For each experimental group, 0.6 µl of the viral vector (excitatory: 2.0×1012 genomic copies/µl of AAV8-hSyn -hM3D(Gq)-mCherry; inhibitory: 7.0×1012 genomic copies/µl of AAV8-hSyn -hM4D(Gi)-mCherry; control: 1.0×1013 genomic copies/µl of AAV8-hSyn-mCherry, diluted 50% with sterile 0.9% saline solution (UNC Vector Core, USA)) was injected bilaterally in the lateral OFC (coordinates relative to bregma: AP: +3.5, ML: ±2.6, DV: -3.7 from dura), at a rate of 0.2 µl/minute. Clozapine-N-Oxide (CNO) was used as the ligand for DREADDs receptors and was administered to all animals, including controls. CNO administration started minimum 3 weeks after surgeries, when virus expression is known to be stabilized. 5.2.2 CNO preparation and administration CNO (Toronto Research Chemicals) was dissolved in 100% DMSO and diluted with saline (0.9% NaCl) to a final DMSO concentration of 3%. CNO was injected at 1 ml/kg via the intraperitoneal route 30 min prior to the start of the behavioural task. Animals received 1 mg/kg CNO prior to each rGT session until a stable performance baseline was reached (27 sessions). 5.2.3 Histology Rats were deeply anesthetized and perfused transcardially with ice-cold phosphate-buffered saline followed by 4% paraformaldehyde in PBS. Brains were removed, stored in 4% paraformaldehyde solution, cryoprotected in a 30% sucrose solution, and sectioned (40 µm) with a cryostat. Free-floating sections were then immunostained with a chicken mCherry primary (1:1000; Abcam ab205402) and goat anti-chicken secondary stain (Alexa 633, 1:500; Invitrogen 120 A21103) and mounted on slides. DREADD virus transfection and spread was verified using a confocal microscope (Leica TCS SP8). 5.3 Results 5.3.1 DREADDs expression Viral vectors were highly expressed by cell bodies within the lateral OFC (Figure 5.1). Rats with unilateral expression were excluded from analyses (n = 3 for uncued experiment, n = 4 for cued experiment). Expression was well-contained within the lateral OFC. Some light expression in the motor cortex dorsal to the lateral OFC and insular cortex lateral to the OFC was noted in a small subset of rats. 121 Location of all acceptable lateral OFC expression patterns (A) and an example microscopy image (B) are shown. Coordinates are relative to bregma. Plates modified from Paxinos and Watson (1998). Figure 5.1: Histological analysis of DREADDs expression in the lateral OFC 122 5.3.2 Task learning 5.3.2.1 Choice To assess the evolution of choice preferences during the learning phase, we analyzed choice of the four options using a multilevel logistic regression model across the first 20 sessions, weighted by the number of trials (see Section 2.5). On the uncued rGT, activation of the lateral OFC resulted in an increased slope for the most optimal option P2 across sessions for rats at the mean final score (M = 63.02), such that experimental rats selected this option more than controls as training progressed (session x group: β = 0.071, p = .03; activation: β = .094; control: β = .080; activation vs control: t(877) = 2.14, p = .06; Figure 5.2B). This effect did not significantly differ across different values of final score (session x group x final score: β = -.0008, p = .4). Additionally, rats in the inhibition group exhibited a flatter slope across sessions than controls for P3 (session x group: β = .071, p = .03; control: β = -.081; inhibition: β = -.010; inhibition vs. control: t(877) = 2.17, p = .06; Figure 5.2C, middle panel). However, this did not result in different levels of P3 choice at the start or end of task learning (session 1: t(877) = -1.19, p = .4; Session 20: t(877) = 1.76, p = .1). No significant differences in P1 or P4 choice were found (all p > .11; Figure 5.2A and 5.2D), nor were any differences in overall risk score observed (all p >.25; Figure 5.2I). Thus, this set of results indicate that lateral OFC activity supports choice of the most optimal option P2 on the uncued rGT as rats develop their choice preferences. However, this is not due to increased or decreased risk preference (i.e., boosted choice of P2 comes at the expense of both risky and optimal options). On the cued rGT, activation of the lateral OFC resulted in a significant shift in the development of P1 choice compared to that of control rats, dependent on final score (session x 123 group x final score: β = 0.001, p = .03). Rats at the mean final score (M = -8.21) had a significantly lower intercept (t(875) = -2.55, p = .02) and flatter slope (activation: β = -0.06; control: β = -0.15; t(875) = 3.82, p = .0003), indicating that activation of the lateral OFC resulted in reduced choice of P1 early in training, but control rats decreased more rapidly across training such that no difference in P1 selection was present in later sessions (Figure 5.2E, middle panel). A similar pattern was observed in optimal rats (M + 1 SD final score = 51.88); these rats had a lower intercept than controls (t(875) = -2.48, p = .03), and did not significantly shift across sessions (β = .013, p = .67; Figure 5.2E, right panel), indicating a consistently lower level of P1 choice across training. Additionally, the activation group differed from controls in their selection of P3 at Session 1, dependent on final score (group x final score: β = .012, p = .05); however, post-hoc comparisons to the control group were not significant (all p >.2). No other differences in P2 or P4 choice were observed (all p > .18; Figure 5.2F and H). No differences in the selection of P1-P4 were observed with chronic inhibition of the lateral OFC during cued rGT training (all p >.12). However, we did observe differences in score, dependent on rats’ ultimate level of risk preference (session x group x final score: β = 0.03, p = .006). Inhibiting the lateral OFC in risk-preferring rats delayed the development of a risky choice profile, as they exhibited a marginally higher (i.e., less risky) score at the first session compared to controls (t(38) = 2.19, p = .06) but a significantly steeper decline across sessions (inhibition: β = -5.27; control: β = -3.02; t(830) = -2.61, p = .02), such that groups did not differ by session 20 (t(38) = -0.31, p = .92; Figure 5.2J, left panel). Inhibition of the lateral OFC did not affect the development of risk preference in rats with higher final scores (all p >.29; Figure 5.2J, middle and right panels). Additionally, no differences in risk score were observed between activation and control groups (all p >.29). 124 Simple slopes analyses of the experimental group x final score x session interaction for P1-P4 choice and score on the uncued (A-D, I) and cued (E-H, J) rGT. Panels show the predicted probability of P1-P4 choice across the first 20 dosing sessions, split by the mean ± 1 SD final score. Effects of inhibiting lateral OFC neurons: on the uncued rGT, flatter slope of P3 choice (C, middle panel); on the cued rGT, delayed development of risk preference in risk-preferring rats (J, left panel). Effect of activating lateral OFC neurons: on the uncued rGT, boosted choice of P2 towards the end of training in rats at the mean final score (B, middle panel); on the cued rGT, reduced selection of P1 in rats at the mean final score (E, middle panel) and optimal rats (E, right panel). Red boxes indicate a significantly different slope between experimental groups (p<.05). Figure 5.2: Learning phase: P1-P4 choice 125 5.3.2.2 Other variables On the uncued rGT, inhibition of the lateral OFC resulted in elevated premature responding early in training, which declined to similar levels as controls over sessions (group x session: β = -.033, p = .04; inhibition: β = -.040; control: β = -.006; inhibition vs. control: t(875) = -2.05, p = .08; Figure 5.3A). Both inhibition and activation of the lateral OFC impacted latency to choose an option in a manner that was dependent on final score (inhibition: group x final score: β = -.027, p = .005; activation: group x final score: β = -.017, p = .07). Among the controls, rats with a higher risk preference were quicker to choose an option than rats that chose more optimally (β = .015, p = .03); this relationship was not present in the inhibition group (β = -.0043, p = .43) and was trending in the opposite direction in the activation group (β = -.0088, p = .08; see Figure 5.2C). Similarly, the number of omitted trials was significantly impacted by both inhibition and activation of the lateral OFC, dependent on final score (inhibition: group x final score: β = -.062, p = .05; activation: group x final score: β = -.092, p = .003). Control rats with a higher risk preference omitted fewer trials (β = .040, p = .07), which was not present in inhibition rats (β = -.018, p = .31), and the inverse relationship was observed with lateral OFC activation - rats with a higher risk preference omitted more trials (β = -.039, p = .02). Together, these results would suggest that when lateral OFC activity was shifted, task engagement was reduced in rats with a relatively higher risk preference; particularly for lateral OFC activation. No other significant or trending effects were observed (all p > .24). On the cued rGT, latency to choose an option was elevated above controls in the activation group early in training (group x session: β = -.02, p = .04l; Session 1: t(41) = 2.16, p = .04, Figure 5.3D). No differences in premature responding (Figure 5.3B), latency to collect reward, trials completed, or omitted trials were found (all p >.12). 126 Simple slopes analyses of the experimental group x session interaction for premature responding on the uncued (A) and cued (B) rGT. Figures show predicted probability of % premature responding. Rats in the inhibition group had a significantly elevated level of premature responding at the start of training. C,D: Panels show predicted choice latencies in seconds across the first 20 dosing sessions, split by the mean ± 1 SD final score. Red box indicates a significantly different slope between experimental groups (p<.05). Figure 5.3: Learning phase: Other task variables 127 5.3.3 Task performance Once a stable performance baseline was reached, differences between groups were analyzed using a repeated-measures ANOVA to determine whether shifts in lateral OFC activity during task learning resulted in persistent changes to task performance. On the uncued rGT, experimental groups differed in P1-P4 choice (choice x group: F(5,96) = 2.60, p = .03; Figure 5.4A). This effect was only present in the activation group compared to controls (choice x group: F(3,64) = 2.96, p = .05); inhibition of lateral OFC did not significantly shift P1-P4 choice in the performance phase (choice x group: F(2,62) = 0.39, p = .7). Post-hoc analyses revealed that activation of the lateral OFC reduced choice of P1 (group: F(1,25) = 7.91, p = .009; Figure 5.4A). This did not result in any significant shifts in risk score (p = .6). Additionally, omissions significantly differed between groups (group: F(2,38) = 3.52, p = .04); again, this was only present in activation versus controls (group: F(1,25) = 3.83, p = .06; inhibition: F(1,26) = 0.04, p = .8), with activation resulting in an increased number of omitted trials (Table 5.1). No other effects were observed (all F<1.64, all p>.21, Table 5.1). On the cued rGT, premature responding differed between groups (F(2,38) = 3.54, p = .04; Figure 5.4D). Neither experimental group was different from controls; however, the inhibition group was significantly elevated above the activation group (F(1,24) = 9.99, p = .004; See Figure 5.4D). No differences in P1-4 choice (Figure 5.4B) or other variables were found (all F < 2.27, all p > .12, Table 5.1). 128 Figure 5.4: Performance phase: P1-P4 choice and premature responding Differences in the performance phase of the rGT between experimental groups for P1-P4 choice and premature responding. On the uncued task (A), rats in the activation group showed a significantly lower choice of P1 compared to controls. No differences in P1-P4 were found on the cued task (B). Premature responding: on the cued task (D), the inhibition group was significantly higher compared to the activation group. While in the same direction, no significant differences in premature responding were observed on the uncued task (C). Data are mean + SEM. * = p < .05 Mean values of all other task variables (choice latency, collect latency, omissions, trials completed) during the performance phase for rats in the control, activation, and inhibition groups. Data are mean ± SEM. Group Task Choice latency Collect latency Omissions Trials completed Control Uncued 1.63 ± 0.22 1.67 ± 0.20 2.30 ± 0.72 104.51 ± 7.91 Cued 1.67 ± 0.14 1.37 ± 0.14 1.08 ± 0.37 75.02 ± 8.58 Activation Uncued 2.04 ± 0.29 1.98 ± 0.33 3.80 ± 1.57 94.63 ± 6.76 Cued 1.98 ± 0.16 1.34 ± 0.16 1.91 ± 0.93 65.51 ± 6.29 Inhibition Uncued 1.43 ± 0.20 1.46 ± 0.14 1.25 ± 0.58 96.99 ± 6.72 Cued 1.59 ± 0.19 1.42 ± 0.19 1.66 ± 0.58 63.27 ± 3.82 Table 5.1: Performance phase: Other task variables * * 129 5.4 Discussion Here, we showed that the inclusion of reward-paired cues on a risky decision-making task substantially alters the contributions of the lateral OFC to learning of the optimal strategy. On the uncued rGT, activity within the lateral OFC supported optimal choice, as enhancing activity led to a stronger preference for the most optimal option, P2, earlier in training. Conversely, on the cued rGT, the lateral OFC did not support optimal choice and may even contribute to the development of cue-induced risk preference, as inhibiting the lateral OFC during cued rGT training delayed the development of a risky choice pattern in rats which ultimately preferred those options. Across both tasks, inhibiting the lateral OFC increased motor impulsivity, and activation reduced the selection of the less optimal option P1, indicating some consistency in the role of the lateral OFC on the rGT. However, differences in the selection of the other options as well as risk preference indicate that the presence of reward-paired cues substantially impacts how the lateral OFC contributes to risky choice. Thus, we have identified the lateral OFC as one neural locus for mediating the impact of reward-paired cues on decision making. While the large number of rats used for this study lends strength to these conclusions, it also precluded the inclusion of females to study potential sex differences, due to practical constraints. Sex differences in the OFC have been previously established. Female mice have higher levels of a marker for myelination in the OFC (Darling & Daniel, 2019), which may account for lower levels of motor impulsivity in female rodents (Weafer & de Wit, 2014). In humans, sex differences in medial versus lateral OFC activation have been observed during performance of the IGT (Bolla et al., 2004). Furthermore, sex differences in lateral OFC activation, measured indirectly via c-Fos expression, have been observed on a rodent version of the IGT (van Hasselt et al., 2012). Given that distinct differences between males and females in 130 the role of other regions such as the VTA have been observed on the rGT (Hynes et al., 2021), it would be worthwhile to investigate whether the observed results differ from effects in females. Additionally, while care was taken to restrict expression to the lateral OFC, some light expression extending into the insular cortex located laterally to the OFC was noted in some rats. As discussed in Izquierdo (2017), the region located at the boundary between the lateral OFC and insula appears to play a functionally similar role to the lateral OFC itself in a variety of tasks such as reversal learning, reinforcer devaluation, and cue-outcome associations. Thus, manipulation of neurons at the OFC-insula boundary may have resulted in similar behavioural effects as expression patterns located more medially. Targeting the insula specifically could shed light on potential divergences in contributions to task learning and performance between these regions. Importantly, no expression in the medial OFC was observed, which as will be discussed below, plays a functionally distinct role (Izquierdo, 2017; Noonan et al., 2010). Results on the uncued task indicate that activity within the lateral OFC supports the selection of the option that maximizes reward gain over time, as rats in the activation group developed a preference for P2 more quickly than controls or rats in the inhibition group. This may have been at the expense of the less optimal option P1, as we also observed a small but significant reduction in P1 selection at the end of task training. This lines up with the effect of lesioning the OFC prior to training on the uncued rGT, in which rats took longer to develop a preference for P2 and exhibited increased selection of P1 (Zeeb & Winstanley, 2010). Increased lOFC activity may have amplified the difference between P2 and the other options during learning. On the probabilistic reversal learning task, the lateral OFC differentially responds to reward following optimal versus suboptimal choices (Amodeo et al., 2017; section 1.2.2). Thus, activation of the lateral OFC may have preferentially increased neuronal responses to wins on the 131 optimal P2 versus the other options, resulting in an accelerated development of P2 preference across training. Differences in choice did not last into the performance phase, indicating that shifts in lateral OFC activity influence the speed of learning, rather than the overall level of choice preference after contingencies are known. Indeed, on the IGT in humans, activity in the lateral OFC declines across training (Zha et al., 2022), which underscores the importance of this region in learning rather than choosing between options of known value. The lack of effect on the score variable suggests that on the uncued rGT, the lateral OFC is involved in determining the best option out of all four available choices that maximizes reward over time, rather than boosting choice of the safe options over risky choices. This would be consistent with the lateral OFC’s proposed role in model-based learning (Schoenbaum et al., 2016; section 1.3.3.2). Conversely, the medial OFC may be more involved in preference for the risky versus optimal options – this has been shown on the IGT in humans (Zha et al., 2022) and on the probabilistic discounting task in rats (Stopper et al., 2014). Additionally, direct comparison of the function of these OFC subregions in macaques has shown that the lateral OFC guides learning-related processes, whereas the medial OFC is important for choosing between valued options (Noonan et al., 2010). Differences in medial versus lateral OFC functioning could explain why the effects of chemogenetic inhibition of the lateral OFC did not line up with those observed by Zeeb & Winstanley (2010), as both the medial and lateral OFC were lesioned prior to training. In the present study, we did not observe any increase in the selection of P1 nor slowed development of preference for P2 with chemogenetic inhibition on the uncued rGT. Thus, combined loss of the medial and lateral OFC may be necessary for these effects to materialize. Differential choice of the risky option P3 across task learning was evident in the current study, but the effect was small 132 and did not result in a significant shift in score. Compensatory activity in the medial OFC or perhaps the prelimbic and infralimbic regions, which also contribute to optimal choice (Zeeb et al., 2015), may explain the lack of results observed here. Assessing activity levels in other brain regions on the uncued rGT resulting from chemogenetic inhibition of the lateral OFC could shed light on whether other cortical regions are compensating for reduced OFC function. Modulating lateral OFC activity on the cued rGT resulted in markedly different effects compared to those described above. One exception is the reduced choice of P1 with activation of the lateral OFC, which occurred on both the uncued and cued tasks, albeit at different timepoints. Thus, the lateral OFC is important for disregarding the option offering highly frequent but small rewards that ultimately does not maximize gains. However, on the cued rGT, this effect was only apparent in rats that generally made good decisions, suggesting that the lateral OFC plays different roles in risky versus optimal rats when reward-paired cues are present. Indeed, as shown in Chapter 4, experimental manipulations within the lateral OFC differentially impact risky versus optimal rats on the cued rGT. Furthermore, individual differences in the impact of lateral OFC inactivation are also observed in delay discounting, but only when the delays are cued (Zeeb et al., 2010). Thus, when reward-related cues are present, differential function of the lateral OFC may be at least partly responsible for individual differences in decision making. Despite the decreased choice of P1, there was no change in the selection of P2 with lateral OFC activation on the cued rGT. Thus, rats could disregard P1 but could not selectively shift their choice to the option that maximizes reward. This may indicate that reward-paired cues interfere with the accuracy of action-outcome contingencies in the lateral OFC. As discussed in Chapter 3 and Langdon et al. (2019), win-paired cues reduce sensitivity to negative feedback on 133 the rGT. Thus, the reduced impact of the time-out penalties may have biased reward predictions in the lateral OFC. One potential mechanism for this effect of reward-paired cues on outcome predictions is circuitry between the lateral OFC and basolateral amygdala. Projections from the lateral OFC to the BLA are responsible for encoding the value of rewards, which are then retrieved by the lateral OFC to guide future choice (Malvaez et al., 2019; Sias et al., 2021). In support of this, on the uncued rGT, connections between these two regions is critical for flexible responding following outcome devaluation (Zeeb & Winstanley, 2013). Furthermore, BLA projections to the OFC are necessary for Pavlovian cues to invigorate reward seeking in the PIT test (Lichtenberg et al., 2017), indicating that this pathway facilitates the impact of cue-reward associations on motivated behaviour. As such, it is possible that on the cued rGT, cue-reward associations mediated by BLA-OFC circuitry dominate the decision-making process, rather than the learned value of the actions that may be stored elsewhere (e.g., the DMS; section 1.3.3.1). This could also explain the improved decision making observed on the loss-cued task in Chapter 3, in which the loss-cue association may control the decision-making process to bias choice away from the disadvantageous options. Future studies investigating the contributions of the OFC-BLA pathway to decision making on the cued rGT could shed light on this hypothesis. Another possibility is that on the cued rGT, dopamine activity in the nucleus accumbens, which drives risky choice on the cued rGT in males (Hynes et al., 2020, 2021), overrides inputs from the lateral OFC to this region, thus biasing action selection towards the risky options. This could explain why increasing lateral OFC activity does not result in increased choice of P2. However, this hypothesis cannot explain why lateral OFC inhibition delayed the development of a risky choice pattern in rats that ultimately preferred the risky options at the end of training. 134 This result instead suggests that the lateral OFC actively contributes to disadvantageous risky choice during the learning phase on the cued rGT, rather than reduced top-down control resulting from aberrant dopamine activity. Indeed, others have also found that lateral OFC activity can bias choice towards cued risky options during learning (Constantinople et al., 2019). The ventral striatum, phasic dopamine release from the VTA, and the lateral OFC have all been implicated in model-based but not model-free reinforcement learning (sections 1.3.1.1, 1.3.3; mcDannald et al., 2011; Langdon, Sharpe, et al., 2019; Sharpe et al., 2020). Thus, these regions may cooperate to increase disadvantageous risky choice due to cue-driven biases in model-based reinforcement learning, rather than competing for control of behaviour. That inhibition of both VTA dopamine (Hynes et al., 2021) and the lateral OFC can reduce risky choice, at least in males, suggests that reducing model-based control of behaviour actually improves decision making on the cued rGT. As such, the addition of reward-concurrent cues to the rGT may bias or corrupt the generation of an accurate model of the task’s reinforcement structure. It is possible that the ability of lateral OFC inhibition to delay development of a risky choice pattern in risk-preferring rats is not specific to the cued rGT, but that we are underpowered to detect this on the uncued rGT due to a low number of risky rats (1-2 per experimental group). However, the lack of increased choice of P2 on the cued rGT, even in rats with the same level of risk preference as those that exhibited an increase in P2 selection on the uncued rGT, would suggest that the contribution of the lateral OFC to decision making depends on the presence versus absence of reward-paired cues, at least for some effects. Nevertheless, we cannot rule out that biased processing of action-outcome contingencies in the lateral OFC occurs in risk-preferring rats on the uncued rGT as well. Examination of neural activity following different trial 135 outcomes using techniques such as fiber photometry could shed light on whether computations within the lateral OFC truly differ between the uncued and cued rGT. Increased motor impulsivity resulting from lateral OFC inhibition is in line with the previously established role of this region in inhibiting prepotent motor responses (Chudasama et al., 2003; Eagle et al., 2008), although it has also been posited that the lateral OFC mediates response conflict, rather than directly impacting motor impulsivity (Bryden and Roesch, 2015). Given the role of the lateral OFC in waiting for reward (Dalley et al., 2011), it may be that increased premature responding with OFC inhibition resulted from decreased willingness to wait for stimulus onset, rather than disinhibition of motor responses. While this effect was present in both tasks, it only reached significance in the cued rGT after the task was learned. Thus, the presence of win-paired cues may delay the time point at which the lateral OFC can influence motor impulsivity. Given the non-overlapping time courses of the impact of lateral OFC inhibition on choice patterns versus premature responding across both tasks, these effects may be mediated by different downstream targets (e.g., ventral versus dorsal striatum). Indeed, as discussed in Chapter 4, while risky choice and impulsivity on the rGT are correlated at the population level (Barrus et al., 2015), neural mechanisms can be dissociated experimentally, lending support to this hypothesis. Overall, these results shed light on the role of the lateral OFC in facilitating cue-induced risky choice. Furthermore, the proposed framework of aberrant model-based control of choice patterns may provide a basis for further research into neural circuitry underlying cue-induced risky choice, such as OFC-BLA circuitry and pathways from the lateral OFC to the dorsal and ventral subregions of the striatum. 136 Chapter 6: Dopamine activity in the nigrostriatal pathway alters cue-induced risky choice patterns in female rats 6.1 Introduction Extensive research has addressed the critical role of dopaminergic activity originating from the VTA in modulating risky choice (Bissonette & Roesch, 2016; Sugam et al., 2014), which greatly differs between sexes (Hynes et al., 2020, 2021). Substantially less work has examined the role of nigrostriatal dopamine activity in risky decision making, and whether sex differences impact its function. In humans, amphetamine-induced dopamine release in the dorsal striatum is correlated with gambling severity (Boileau et al., 2014). Thus, understanding the contribution of the nigrostriatal pathway to risky choice and the role of biological factors such as sex could advance our understanding of gambling disorder. Female rats show increased risky choice with the systemic administration of the D2/D3 agonist ropinirole during task learning, which cannot be explained by increased activity along the mesolimbic dopamine pathway (section 1.5; Hynes et al., 2021; Russell et al., 2022). Other dopaminergic pathways may therefore mediate this effect. Dopamine activity in the dorsal striatum, innervated by the SNc, has not yet been investigated on this task, and may contribute to the development of choice patterns. Given that estradiol impacts the excitability of dopamine terminals and D2 receptor binding in the dorsal striatum (section 1.5), dopamine activity in this region may play a particularly important role in decision making in females versus males. Dopamine release from the medial SNc in the DMS contributes to flexible, goal-directed behaviour, whereas dopamine neurons in the lateral SNc project to the DLS to support habit formation (section 1.3.3.4). At least in males, reward-paired cues reduce flexibility in choice 137 patterns on the rGT following reinforcer devaluation, indicative of lower goal-directed control of decision making (Chapter 3, 4). As such, nigrostriatal dopamine activity may be involved in the establishment of risky choice patterns on the cued rGT. While previous studies examining the function of the SNc and dorsal striatum have included males and females, few have reported whether sex differences were present. Given the surprising effect in females following manipulation of the VTA on the cued rGT, and the impact of estradiol on dopaminergic signaling in the dorsal striatum, examining the role the nigrostriatal pathway in females is of particular interest. Thus, the current set of studies employed the DREADDs technique to manipulate activity in SNc dopamine neurons during both the acquisition and performance phases of the cued rGT in female rats, with the goal of determining whether the nigrostriatal pathway contributes to the development of risky choice patterns when reward-paired cues are present. 6.2 Additional methods 6.2.1 Viral vector injections A similar approach employing chemogenetic methods as described in Chapter 5 was used for the present experiments. Four cohorts of 32-48 female transgenic rats (total N = 160) expressing Cre recombinase in neurons containing tyrosine hydroxylase were bred in-house against a Long-Evans background. Rats were anesthetized with isoflurane (5% induction, 2-3% maintenance) prior to surgery. Ketoprofren (5 mg/kg) was given as an analgesic and bupivicaine was administered at the surgical site for local anesthesia. For the inhibitory DREADD experiment, 1 µl of 1.0×10^9 genomic copies/µl of AAV5-hSyn-DIO-hM4D(Gi)-mCherry (UNC Vector Core, USA) was injected bilaterally in the SNc (coordinates relative to bregma: AP: -5, ML: ±2.1, DV: -7.5), at a rate of 0.1 µl/minute. This low rate resulted in frequent 138 blockages within the Hamilton syringes used for infusions and therefore multiple rats exhibited unilateral or no viral expression. Accordingly, the rate was increased to 0.2 µl/minute for the remaining experiments. The volume of viral vector (excitatory: 1.0×10^9 genomic copies/µl of AAV5-hSyn-DIO-hM3D(Gq)-mCherry; UNC Vector Core, USA) was reduced to 0.8 µl to reduce the spread of viral particles to the VTA. CNO administration started at least 6 weeks after surgeries, allowing sufficient time for viral expression to stabilize. 6.2.2 CNO preparation and administration CNO (Toronto Research Chemicals) was dissolved in 100% DMSO and diluted with saline (0.9% NaCl) to a final DMSO concentration of 3%. CNO was injected at 1 ml/kg via the intraperitoneal route 30 min prior to the start of the behavioural task. For acquisition experiments, animals received 1 mg/kg CNO prior to each rGT session for 30 sessions, at which point performance was statistically stable across all measures (see data analysis section). For baseline experiments, animals received CNO (0, 1, 3 mg/ml) in a modified digram-balanced Latin square drug design (for doses A-C: ABC, BAC, CAB, CBA). Drug injections were given on a 3-day cycle, starting initially with a baseline session. The following day, rats received a drug or vehicle injection prior to testing. On the third day, animals were not tested and remained in their homecages to ensure no carry-over effects were observed. 6.2.3 Histology Rats were deeply anesthetized and perfused transcardially with ice-cold phosphate-buffered saline followed by 4% paraformaldehyde in PBS. Brains were removed, stored in 4% paraformaldehyde solution, cryoprotected in a 30% sucrose solution, and sectioned (40 µm) with a cryostat. Free-floating sections were then immunostained with a chicken mCherry primary (1:1000; Abcam ab205402) and goat anti-chicken secondary stain (Alexa 633, 1:500; Invitrogen 139 A21103) and mounted on slides. DREADD virus transfection and spread was verified using a confocal microscope (Leica TCS SP8). 6.3 Results 6.3.1 DREADDs expression Viral vectors were highly expressed by cell bodies within the SNc (Figure 6.1). Rats with no or unilateral expression were excluded from analyses (n = 15 of 80 Tg+ rats). Expression was well-contained within the dopaminergic midbrain, largely centered on the medial SNc. Some light expression in lateral VTA was noted in a small subset of rats. . Nine rats across all four cohorts died as a result of complications from surgery or from ill health not obviously related to the experimental manipulations. Data were also excluded from rats that failed to complete at least 20 trials per session (inhibition during acquisition: n = 5; inhibition during performance: n = 4) resulting in the following sample sizes for each experimental cohort: SNc DA inhibition/activation during acquisition: 14 Tg+, 15 Tg-/ 12 Tg+, 15 Tg-; SNc DA inhibition/activation during performance: 12 Tg+, 20 Tg-/ 18 Tg+, 23 Tg-. 140 Location of all acceptable SNc expression patterns (A) and example microscopy images at 63x magnification (B) and 10x magnification (C) are shown. Coordinates are relative to bregma. Plates modified from Paxinos and Watson (1998). Figure 6.1: Histological analysis of DREADDs expression in the SNc 141 6.3.2 Inhibition/activation of nigrostriatal dopamine neurons during task acquisition 6.3.2.1 Choice To assess the impact of modulating SNc activity on risky choice across training, we analyzed the evolution of score across the 30 dosing sessions using a multilevel logistic regression model (see Section 2.5). Following chemogenetic inhibition of the nigrostriatal dopamine neurons, we observed a significant transgene x session x final score interaction (β = -0.014, p = .03) for the score variable, suggesting an alteration in the rate at which rats developed a preference for the optimal or risky options. Post hoc analyses revealed that Tg+ risk-preferring rats (mean – 1 SD final score) had a lower intercept compared to Tg- risk-preferring rats (session 1: β = -36.10, t(25) = -2.42, p = .02) and a flatter slope (Tg+: β = -0.42, Tg-: β = -2.28, t = 3.03, p = .003; Figure 6.2E). As such, Tg+ rats that ultimately preferred the risky options at the end of training developed this preference significantly earlier than their Tg- counterparts, even though levels of risk-preference were comparable by the final session. No significant differences were found between transgene groups at the mean or mean + 1SD level of risk preference (all t < 1.12, all p > .27; Figure 6.2B and C). Somewhat in keeping with the effect of inactivating SNc DA neurons on score reported above, we also observed a trend-level difference between transgene groups in the intercept for P3, indicating that Tg+ rats tended to choose P3 more at the start of training (transgene: β = 0.93, p = .08; Figure 6.2C). This was not dependent on final score (transgene x final score: β = .00005, p = .8). More surprisingly, we also observed a significant group x session effect for P2 (β = 0.04, p = .01; Fig 6.2B). Simple slope analyses revealed a significant difference in slopes between groups across sessions (Tg+: β = 0.03, Tg-: β = 0.07, p = .03), indicating that Tg+ rats selected P2 more as training progressed. Again, this change was 142 not dependent on final score, indicating this effect was also present in those that ultimately preferred the risky options (transgene x session x final score: β = -6 x 10-6, p = .98). There was also a very weak trend for selection of P1 to be lower in Tg+ rats as training progressed (session x group: β = -.04, p = .09; Figure 6.2A), whereas no transgene-dependent effects for P4 were observed (all p > .15). This set of results indicates that Tg+ rats initially tended to show a greater preference for P3, but developed a stronger preference for P2 by the end of the 30 sessions. Activation of the dopaminergic projections from the SNc did not result in any significant change in score (session x group: β = 0.54, p = .3; Figure 6.3E). When analysing choice of each option, we also did not observe any significant or trending terms involving transgene group, with the exception of a very weak trend for increased selection of P1 over time (session x group: β = 0.029, p = .09, Figure 6.3A; all other p > .2, Figure 6.3B-D). 143 Simple slopes analysis of the transgene group x final score x session interaction, comparing SNc inhibition to controls. Tg+ risk-preferring rats were significantly riskier early in training compared to Tg- risky rats (E, left panel); no differences were found between transgene groups for rats at the mean (E, middle panel) or optimal risk level (E, right panel). A-D: Predicted probability of P1-P4 choice across the 30 dosing sessions, split by the mean and +/- 1 SD final score. Tg+ rats show a significantly greater choice of P2 across training (B), and a tendency to show an initial stronger preference for P3 (C) combined with a weak trend for lower choice of P1 (A). Red boxes indicate a significant difference in slopes and/or intercepts between groups, p <.05. Light red boxes indicate a trend (p between .05 and .09). Figure 6.2: Nigrostriatal inhibition during task learning: Choice 144 Simple slopes analysis of the transgene group x final score x session interaction, activation versus controls. No significant differences were found on P1-P4 choice (A-D) or score (E), with the exception of a weakly trending increase in P1 selection across sessions (A). Figure 6.3: Nigrostriatal activation during task learning: Choice 145 6.3.2.2 Other variables Daily chemogenetic inhibition of SNc DA neurons resulted in significantly fewer premature responses across training (session x group: β = 0.005, p = .005; group- session 1: t(27) = 6.54, p <.0001; session 30: t(27) = 5.50, p <.0001; Fig 6.4A). The rate of premature responding declined progressively in control rats with training (β = -0.006, p<.0001) yet stayed consistently low in the experimental group (β = -.0005, p = .69). In contrast, there was a very weak trend for premature responding to be higher in Tg+ rats at the first session following activation of SNc DA neurons; no difference in the slopes across sessions was found (group: β = .08, p = .09; group x session: β = -.0002, p = .95; Figure 6.4B). Following inhibition of SNc DA neurons, Tg+ rats were slower to make a choice, and became slower over time, while Tg- rats sped up as training progressed (group x session: β = .023, p = .001; Tg-: β = -.009, Tg+: β = .014; Tg- vs Tg+: t(827) = -3.28, p = .001; Figure 6.4C). By contrast, following SNc activation, there was a trending difference between transgene groups at the first session that was dependent on final score (transgene x final score: : β = .005, p = .07). Simple slope analyses revealed that risk-preferring rats were significantly quicker to choose than risky control rats at Session 1 (final score mean – 1 SD = -36.37; t(23) =2.78, p = .01; Figure 6.4D, left panel). Reward collection latency also tended to be slower in Tg+ rats following inhibition, but significantly decreased across sessions, whereas control rats remained constant (group x session: β = -.02, p = .08; Tg+: β = -.03, p = .002; Tg-β = -.008, p = .16; Fig 6.4E). However, high initial values in the Tg+ were likely driven by an outlier that was very slow to collect reward in early sessions (e.g., latency of 15 sec on session 5). Residuals were not normally distributed for this model, further suggesting that the effect was skewed by outliers. In keeping with this pattern of generally lower and slower activity following inhibition of SNc DA 146 neurons, while control rats that ultimately developed the most optimal choice profile completed progressively more trials as training progressed, this tended to be less apparent in their Tg+ counterparts (session x group x final score: β = -.008, t = .06; mean + 1 SD final score; Tg+: β = 0.68, Tg-: β = 1.39; Tg+ vs Tg-: t(837) = -1.75, p = .08; Fig 6.5A, right panel). Activation of SNc DA caused the opposite effect, in that Tg+ rats completed progressively more trials per session as training progressed, whereas control rats did not (group x session: β = 0.64, p = .08; Tg+: β = 0.71, p = .01; Tg-: β = .07, p = .76; Figure 6.5B). In the inhibition condition, Tg+ rats also made progressively more omissions, whereas control rats reliably omitted low numbers of trials (Session x group: β = 0.24, p = .0001; Tg+: β = 0.26, p <.0001; Tg-: β = 0.03, p = .47; Fig 6.5C). Activation of SNc DA neurons did not alter omission rates (Fig 6.5D). 147 Simple slope analyses of premature responding, choice latency, and collect latency variables. Inhibition of the nigrostriatal dopamine pathway resulted in decreased premature responding (A), whereas there was a trending increase with nigrostriatal activation (B). Inhibition also increased choice latencies (C) and activation resulted in a decrease this measure (D). Inhibition may have also increased latency to collect reward, but this was likely driven by an outlier (E); no effects were found in the activation condition (F). Red boxes indicate a significant difference in slopes and/or intercepts between groups, p <.05. Light red boxes indicate a trend (p between .05 and .09). Figure 6.4: Nigrostriatal manipulation during task learning: Premature responding and latencies 148 Simple slope analyses of the trials and omissions variables. A bidirectional effect on trials completed was observed in optimal rats; in the inhibition condition, optimal rats completed fewer trials than controls as training progressed (A, right panel); conversely, in the activation condition, optimal rats completed more trials than controls as training progressed (B, right panel). Inhibition of SNc dopamine neurons also resulted in significantly more omitted trials (C); no differences were found with activation (D). Red boxes indicate a significant difference in slopes and/or intercepts between groups, p <.05. Figure 6.5: Nigrostriatal manipulation during task learning: Trials and omissions 149 6.3.3 Inhibition/activation of nigrostriatal dopamine neurons during task performance Choice: In the inhibition condition, Tg+ tended to be riskier than Tg- rats at the highest dose, indicative of greater risky choice, but were not significantly different from their vehicle risk preference (dose x transgene: F(2,48) = 2.82, p = .07; Tg+ vs Tg- at 3 mg/kg: t(28) = 1.89, p =.07; Tg+, vehicle vs 3 mg/kg: t(10) = -1.09, p = .3, Figure 6.6A). No significant differences in choice of any individual option were observed (dose x choice x Tg: F(5,138) = 1.33, p = .2, data not shown). In the activation condition, score increased somewhat dose-dependently with CNO administration, in Tg+ rats only (dose x Tg: F(3,111) = 3.25, p = .03; dose- Tg+: F(3,48) = 3.12, p = .04; Tg-: F(3,63) = 1.68, p = 0.18; Tg+, vehicle vs 3 mg/kg: t(17) = -1.81, p = .09; Figure 6.6B). When choice of each option was analysed, the dose x choice x transgene interaction reached trend-level significance (F(9,333) = 1.49, p = .06). While the choice patterns of Tg+ animals varied significantly following administration of CNO (Tg+, dose x choice: F(9,144) = 1.96, p = .05), comparison between the vehicle and highest dose did not reveal significant effects for any particular option (all t < 1.71, all p > .11, data not shown). Tg- rats were unaffected by CNO administration (Tg-: dose x choice: F(9,189) = 1.25, p = .27). Other variables: In the inhibition condition, transgene positive rats completed significantly fewer trials at the highest dose (Dose x Tg: F(2,54) = 44.71, p < .0001; dose- Tg+: F(2,20) = 32.52, p < .0001; Tg-: F(2,32) = 0.86, p = .43; Tg+, vehicle vs 3 mg/kg: t(15) = 2.64, p = .02; Table 6.1). No other dose-dependent or transgene-dependent effects were found (all F < 1.94, all p > .15). In the activation condition, no transgene-dependent effects of CNO were observed for any variable analysed (all F < 2.02, all p > .12; Table 6.1). However, the highest dose tended to increase choice latency in all rats by an average of 0.19 seconds (dose: F(3,111) = 2.47, p = .07; vehicle vs. 3 mg/kg: t(40) = -2.13, p = .04). 150 Impact of nigrostriatal dopamine modulation on score during rGT task performance. Inhibition of SNc dopamine neurons (A) tended to decrease score (i.e., more risky choice) compared to controls; conversely, activation of SNc dopamine neurons (B) tended to increase score (decreased risky choice). Mean values of all other task variables (% premature response, choice latency, collect latency, omissions, trials completed) following acute CNO dosing for transgene-positive and -negative rats in the inhibition and activation experiments. Data are mean ± SEM. Dose Dose (mg/kg) Premature responses Choice latency Collect latency Omissions Trials completed Inhibition Tg+ 0 11.77±2.49 1.51±0.17 1.18±0.05 0.67±0.67 63.44±5.37 1.0 10.51±3.24 2.18±0.31 1.40±0.21 3.00±1.00 44.25±5.20 3.0 11.05±2.73 2.38±0.35 1.28±0.14 2.67±0.88 40.31±6.06 Inhibition Tg- 0 11.69±1.95 1.77±0.16 1.22±0.05 0.43±0.20 71.96±5.10 1.0 7.56±1.32 1.75±0.15 1.23±0.06 0.86±0.34 75.15±4.76 3.0 11.70±1.89 1.78±0.16 1.25±0.05 1.57±0.53 73.43±5.94 Activation Tg+ 0 23.91±3.44 1.65±0.21 1.08±0.07 2.77±0.62 75.46±9.77 0.1 27.86±4.42 1.78±0.23 1.07±0.06 3.44±0.70 72.30±9.25 1.0 18.03±2.82 1.86±0.24 1.20±0.43 3.83±0.77 77.11±9.67 3.0 19.57±3.61 1.98±0.25 1.25±0.06 4.11±0.81 79.76±10.12 Activation Tg- 0 21.49±3.42 1.84±0.22 1.33±0.05 2.39±0.51 70.33±8.23 0.1 22.16±3.32 1.84±0.22 1.30±0.05 2.61±0.45 72.37±8.56 1.0 22.33±3.90 1.96±0.23 1.75±0.08 3.00±0.48 69.93±8.41 3.0 24.30±4.17 1.93±0.24 1.34±0.06 2.70±0.52 69.07±8.27 Table 6.1: Nigrostriatal manipulation during task performance: Other variables Figure 6.6: Nigrostriatal manipulation during task performance: Choice 151 6.4 Discussion Here we showed that in female rats, the nigrostriatal pathway contributes to choice preferences on the cued rGT. Inhibiting this pathway throughout acquisition accelerated the development of risk preference in risky rats during the early learning phase, but also increased selection of the most optimal choice later in training. However, activating this pathway had little effect on the development of choice preferences. Once the task was learned, increasing or decreasing activity in the nigrostriatal pathway had bidirectional impacts on score, with inhibition increasing while activation decreasing risky choice. In keeping with a prominent role for nigrostriatal dopamine signaling in motor function, inhibition also generally slowed motor output during task acquisition, as indicated by lower motor impulsivity, and higher omissions and choice latencies. Inhibition during task performance also decreased the number of trials completed. Such observations increase confidence that the chemogenetic inhibition was effective. However, generally speaking, effect sizes were small, and this was particularly true for the activation experiments. This may be partly explained by the use of the DREADDs technique, which modulates neuronal activity to a lesser degree than techniques such as optogenetics or inactivation via GABA agonists. Inhibitory DREADDs receptors have been reported to dampen neuronal firing rates to about 60% of basal levels (Chang et al., 2015; Smith et al., 2016). However, that is also a strength of this technique, as it may emulate naturally-occurring fluctuations in neural activity, as opposed to completely silencing or hyper-stimulating a particular neuronal subgroup. Alternatively, the smaller effect sizes for the activation experiments may suggest that increased dopamine activity along this pathway has only minimal effects on behaviour, or is more tightly regulated than inhibition, such that any impacts on behaviour are minimized. Based on our immunohistochemistry results, 152 excitatory hM3 and inhibitory hM4 DREADDs expression was comparable across experiments, indicating that the marginal behavioural effects of activating SNc dopamine neurons cannot be attributed to differences in viral transduction. However, it remains possible that the downstream effects of additional excitatory g-protein coupled receptor insertion triggers compensatory mechanisms in this specific neuronal population. While we can find no evidence of this in the literature at the time of writing, there have been remarkably few studies published that document the effects of excitatory DREADDs targeted to SNc dopamine neurons. Certainly, the chemogenetic activation of VTA DA neurons is highly efficacious in our hands (Chernoff et al., 2022). Future studies may shed light on the reason for these discrepancies, but the current data suggest that potentiating SNc dopamine activity using this technique has only minimal effects on decision making, at least in females. As outlined in Chapters 3 and 4, choice patterns on the cued rGT, at least in males, are less goal-directed than choice on the uncued rGT. More specifically, choice is insensitive to reinforcer devaluation when rats are trained on cue-outcome schedules that promote risky choice. It is thought that the control of behaviour can shift over time from the ventral striatum to the DMS to the DLS through a series of spiraling loops passing through the dopaminergic midbrain (section 1.3.3.4; Haber et al., 2000). For example, as drug seeking becomes more resistant to devaluation in rats, the locus of dopaminergic control shifts from the ventral to the dorsal striatum (Zapata et al., 2010; Murray et al., 2012). If this shift is evident in risk-preferring female rats performing the cued rGT, we would expect that dampening nigrostriatal signaling would promote optimal decision making in this group by reinstating goal-directed control. If the development of a bias towards the risky options was underpinned by a premature transition from goal-directed to habitual performance, somewhat similar to a shift from exploration to 153 exploitation, we would also expect the effect of inhibiting SNc dopamine to manifest earlier rather than later in training. While we did observe an increase in preference for the best option, P2, when SNc dopamine neurons were inhibited during acquisition, this effect was most pronounced towards the end of training. The timing at which this effect emerges therefore does not support the hypothesis that cue-induced risky choice is driven by a more rapid switch to habitual control. The increased choice of P2 was also evident in all rats, including those which showed a strong preference for the best options. Again, these data argue against our hypothesis that risk-preferring rats would benefit the most from inhibition of SNc dopamine neurons. Finally, unlike chemogenetic inhibition of VTA dopamine neurons (Hynes et al., 2021), the effect on P2 choice was too small to result in an overall change in score. Although SNc dopamine signalling may therefore promote suboptimal choice as training progresses, these data strongly suggest it is not the major determinant of overall choice bias. While it is possible that our failure to detect the effects we expected may be due to an inadequate sample size, the number of animals used was sufficient to detect robust effects on non-decisional variables, and these were largely as predicted from previous reports (i.e. inhibition of SNc dopamine reduces motor output while activation facilitates action; Benazzouz et al., 1993; Saunders et al., 2018). Comparable numbers of rats were also used in our previous work targeting the VTA, and these effect sizes were substantial. Furthermore, in the current study, we were able to detect significant differences in score specific to risk-preferring rats. However, the direction of this effect was opposite to our prediction: rather than reducing preference for the risky options early in training, rats developed this preference more rapidly and sustained this bias throughout training when SNc dopamine neurons were inhibited. This could be attributed to early onset of a preference for P3, which is associated with maximal uncertainty 154 regarding the nature of the outcome (p = 0.5). The final score was nevertheless comparable across Tg+ and Tg- rats, indicating that it was the rate at which the preference was developed, rather than the size of the preference itself, that was influenced by inhibiting SNc dopamine projections. Chemogenetic inhibition of VTA dopamine neurons, in contrast, dramatically increased the level of risky choice in females throughout training, resulting in a significantly more negative final score (Hynes et al., 2021). Whereas VTA dopamine activity can therefore directly determine choice preference, the output of SNc DA neurons may alter the rate at which such preferences come to dominate choice, such that dampening this signal results in more rapid biases towards an option once it becomes preferred, as seen here with P2 and P3. It is worth considering whether the differing effects observed in early versus late training could be explained by a shift in dopamine activity from the DMS to the DLS as the task is learned (Burton et al., 2015; Everitt & Robbins, 2013). Thus, the initial increase in risky choice may be caused by inhibition of dopaminergic activity within the DMS, which is critical for goal-directed behaviour and linking actions to outcomes (Section 1.3.3.1; Yin et al., 2005; Balleine & Dickinson, 1998). Increased selection of the most optimal P2 option later in training could then be due to inhibition of dopamine release in the DLS, preventing choice patterns from stabilizing and becoming rigid or inflexible (Yin et al., 2004). However, if the control of behaviour really does shift from DMS to DLS as training proceeds, and more optimal choice results from dampening dopaminergic signaling to the DLS, then inhibition of the nigrostriatal pathway in the performance phase, after months of task training, should similarly improve choice. We instead observed an increase in selection of the risky options, P3 and P4. It may be important to note that rats in this cohort were largely risk-preferring, with only a small number of optimal decision-makers. As such, the effect of inhibiting SNc dopamine projections after the task had been 155 acquired may mimic what was observed early in training, with risk-preferring rats exhibiting a more pronounced bias towards their preferred options. However, this result doesn’t fit easily with the hypothesis that inhibition of SNc dopamine leads to an increase in preference for P2 later in training due to attenuation of dopaminergic signals to the DLS. Another key variable that alters as training progresses is the learning rate. This will be highest during the initial sessions, when subjects are exploring the reinforcement contingencies associated with each outcome. This information should be used by the subjects to build an internal representation or model of the task’s reinforcement structure (Section 1.3.1.1). Exclusive selection of any option, including the most optimal, will inevitably result in delivery of time-out penalties on a set proportion of trials. Developing and then maintaining an optimal strategy therefore requires subjects to discern when a penalty should instigate a negative re-evaluation of that option’s value, and therefore prompt a shift in choice patterns, and when to ignore it as a relatively rare event. If subjects are building an accurate model of the task, reward prediction errors generated following penalties after selection of P1 or P2 should increasingly fall into the latter category, and have been described as “stability-demanding prediction errors” (Trempler et al.,2020). In contrast, RPEs following penalties incurred by selection of P3 or P4 should be considered “flexibility-demanding prediction errors” that signal the need for adaptation. Whether an RPE can be considered flexibility or stability-demanding depends critically on an accurate probability estimates. Patients with PD can show deficits in both selecting or inhibiting actions, sometimes reacting to rare events as if they signal a need for immediate behavioural change, while also failing to switch strategy following repeated negative outcomes on set-shifting tasks (Wylie et al., 2009; Wylie et al., 2010; Cools et al., 2001; Section 1.6.3). 156 One interpretation of these seemingly contradictory behavioural patterns- both too much and too little flexibility- is that subjects are struggling to differentiate between these two types of RPE, perhaps as a result of insufficient embedding of the probability context. This may explain why dampening SNc dopamine signaling here produced both an increase in preference for P3 in some animals early in learning, followed by an increase in P2 later in training: once animals started to form a preference for a particular option, this choice became more likely to be repeated, like a snowball rolling down-hill, perhaps due to a premature categorization of RPEs as stability-demanding. As to why this manipulation didn’t amplify choice of P1 or P4, it is perhaps notable that choice of P1 and P4 were generally lower in Tg+ animals from the outset of free choice. These options are associated with either the smallest reward (P1) or the longest penalty (P4), both of which are delivered far more frequently than not. There is arguably less of a nuanced cost/benefit trade-off associated with these options, and they can be discounted without an accurate calculation of the probability of events. In contrast to both chemogenetic inhibition of dopaminergic output from either the VTA or the SNc, chronic administration of the D2/3 agonist ropinirole during acquisition significantly increases risky choice in both females (Russell et al., 2022) and males (Mortazavi et al., 2022). Thus, the mechanism of action of this effect in females is still unclear. One possibility is that ropinirole increases risky choice by impacting dopamine release from terminals in the striatum rather than influencing neuronal activity in the midbrain. Sex differences exist in the regulation of dopamine release and reuptake by D2 autoreceptors in the striatum (Ford, 2014; Zachry et al., 2021). Furthermore, gonadal hormones such as estradiol and progesterone have profound effects on the excitability of striatal dopamine terminals and D2 receptor binding (Yoest et al., 2018). 157 Thus, the neural locus of the risk-inducing effect of ropinirole may differ between the sexes – increased midbrain dopamine activity in males, and changes to terminal release in females. Overall, we have shown that the nigrostriatal pathway contributes to the development of risk preference in females, as well as the maintenance of choice patterns once the decision-making task is well learned, in addition to impacting measures of motor output. Gambling disorder is on the rise in women, and differences have been observed in the type of gambling that women partake in, the speed of their progression to problem gambling, and their motivations to gamble (González-Ortega et al., 2019). Thus, understanding the contribution of mesostriatal dopamine in females to factors associated with gambling disorder, such as deficits in cost/benefit decision making, could shed light on gender differences observed in clinical populations. 158 Chapter 7: General discussion 7.1 Summary of experimental findings Here I showed that reward-paired cues can both boost disadvantageous risky choice and induce inflexibility in decision making. In Experiment 1, I demonstrated that the effect of such cues is dependent on the cue-reward association, rather than a general enhancement in sensory stimulation. I also showed that decision making can be improved by cuing the losses instead of the wins. These cue associations specifically affect the processing of the time-out penalties: reduced learning from negative outcomes occurs with win-paired cues, and enhanced learning occurs with loss-paired cues. Thus, the impact of punishments on subsequent choice can be altered according to the cue-outcome association. Similarly, in humans, adding sound concurrent with wins on an EGM results in an overestimation in the number of wins (Dixon et al., 2010), which can be ameliorated by cuing the losses instead (Dixon et al., 2015). Experiment 1 also demonstrated that when wins and losses are cued, rats are both riskier and inflexible, indicating that the impact of win-paired cues on decision making is greater than loss-paired cues. This suggests that animals are preferentially influenced by salient stimuli paired with reward. I also found that there are overlapping neural mechanisms for the impact of reward-paired cues on decision making and flexibility. In Experiment 2, I showed that both disadvantageous risky choice and insensitivity to outcome devaluation can be ameliorated by antagonism of the serotonin 2C receptor in the lateral OFC. Both serotonergic transmission and the lateral OFC have been heavily implicated in cognitive flexibility (Section 1.2.2). 2C receptor antagonists acting in the lateral OFC can reduce perseverative responding on a reversal learning task (Boulougouris & Robbins, 2010). Interestingly, bidirectional modulation of the 2C receptor also impacts motor impulsivity, cue-mediated behaviours, and drug seeking, likely through its 159 effects on dopamine transmission in the striatum (Section 1.4.2; Winstanley et al., 2004; Robinson et al., 2008; Pentkowski et al., 2010; Cunningham, 2015; Adams et al., 2017; Howell & Browne et al.,2017). However, these effects are at odds with those observed in reversal learning tasks: 2C agonists decrease impulsivity, drug seeking, and the control of cues over behaviour, whereas 2C antagonists increase it. From this, one may predict that a 2C antagonist would increase the risk-enhancing effect of cues on the rGT, but we instead observed that delivery of the antagonist to the lateral OFC improved cue-induced risky choice. This suggests that impaired flexibility is central to the impact of cues on risky choice. This was further explored in Experiment 3, in which I assessed the role of the lateral OFC during task learning of the uncued versus cued rGT. Boosting lateral OFC activity on the uncued rGT resulted in increased choice of the most optimal option, P2, suggesting that this region biases decision making towards reward maximization over time. However, the lateral OFC is not necessary for optimal choice, as inhibition did not substantially affect decision making on the uncued rGT, suggesting that other neural circuitry can support optimal choice when lateral OFC output is dampened. Conversely, lateral OFC activity on the cued rGT did not contribute to optimal choice and even exacerbated the development of cue-induced risky choice in risk-preferring rats. Thus, the presence of reward-paired cues significantly alters the role that the lateral OFC plays in guiding learning about options with different action-outcome contingencies. This implicates maladaptive top-down control in mediating the deleterious impact of cues on risky choice. This is consistent with other research showing that noisy signaling in the lateral OFC can cause deficits in value updating (Seabrook et al., 2022). In addition to these experiments on the lateral OFC, I also investigated whether neurocircuitry involved in habit formation could influence choice on the cued rGT, in 160 Experiment 4. While the above experiments utilized male rats, I used females in Experiment 4 to examine the role of the nigrostriatal dopamine pathway. This was because distinct sex differences in the impact of the mesolimbic pathway have been observed, and critically, alterations to VTA dopamine signaling cannot explain why the D2/D3 agonist increases risky choice on the cued rGT in females (Hynes et al., 2020; Hynes et al., 2021; Russell et al., 2022). In Experiment 4, I found that the role of the nigrostriatal dopamine pathway in guiding choice on the cued rGT is complex; inhibiting SNc dopamine neurons accelerated the development of risky choice in risk-preferring rats but also led to an increase in choice of the optimal option P2 later on. SNc dopamine neurons can carry both value and salience signals (Lerner et al., 2015; Zhang et al., 2017). However, few studies have examined how these signals are influenced by the presence of reward-paired cues. If nigrostriatal dopamine induces habitual control of behaviour and is responsible for increasing risky choice, then we’d expect to see more optimal choice early in training following inhibition of this pathway – however, improvements in choice occurred at the end of training, and inhibited habitual control during task learning is not easily reconciled with an accelerated development of risk preference. Thus, in females, either nigrostriatal dopamine does not induce habitual control of behaviour, or a more rapid switch to habitual control does not cause risky choice. Studies on Parkinson’s patients, in whom SNc dopamine neurons are progressively lost, may help explain these results. Parkinson’s patients exhibit increased reliance on external cues to guide attention, as well as decreased detection of prediction violations (Dirnberger & Jahanshahi, 2013; Trempler et al., 2020). Thus, initial increases in risky choice may be the result of biased attention towards the larger cues on the risky options, as well as reduced detection of the need to flexibly update choice patterns following time-out penalties. However, as evidence accumulated 161 across sessions that P2 was the best option, experimental rats may have shifted to exploiting that option instead and became less sensitive to losses on P2 than controls. Increased exploitation of P2 choice appears to be dependent on long-term inhibition of the nigrostriatal pathway across task experience, as acute inhibition of this pathway after the task was learned resulted in increased risky choice, mirroring the effect observed in early training. Regardless, the complexity of these results suggests that the interaction of habit formation and choice patterns on the rGT are not straightforward. Overall, it appears this pathway impacts the speed of development of stable choice patterns, rather than general levels of risk preference, which may be more dependent on activity within the mesolimbic dopamine pathway. 7.2 Goal-directed versus habitual control of decision making on the rGT It is notable that in Experiment 1, decision making in risk-preferring rats on the task variants featuring win-paired cues was not altered by reinforcer devaluation, or in vehicle-treated rats in Experiment 2, in contrast to the effects of this manipulation in the absence of cues (Zeeb and Winstanley, 2013). Decision making in other tasks that require considerably more sessions to train remain sensitive to changes in outcome value, indicating that simple repetition of actions in complex cognitive tasks is not sufficient to produce habitual behaviour through procedural motor learning (e.g., Cocker et al., 2012). Indeed, goal-directed control can be maintained following prolonged training even if automatization of certain action sequences occurs (Garr and Delamater, 2019). Acute satiety with regular chow does not shift choice patterns on the uncued rGT (Zeeb & Winstanley, 2013) and thus the effect of reinforcer devaluation on choice can be attributed to shifts in goal-directed action rather than reduced motivation. While acute satiety with chow has not been tested on the cued rGT, it stands to reason that reduced motivation alone would similarly leave choice patterns unaffected. As such, decision making on the cued rGT may 162 fail one of the critical tests of true goal-directed behaviour, in that it is insensitive to changes in the goal’s value (Balleine and Dickinson, 1998). While differences in reinforcer devaluation tests in risky versus optimal rats where not found in Experiment 2 or by Zeeb & Winstanley (2013) on the uncued rGT, the cohort sizes of Experiment 1 far exceed other reports, which may have been underpowered to detect such differences. The effect in risk-preferring rats could be due to either enhanced habit formation or hypoactive, or otherwise maladaptive, goal-directed control. In Experiment 2, I showed that altering serotonergic signaling in the lateral OFC can restore sensitivity to reinforcer devaluation in rats trained on the cued rGT, indicating that prefrontal cortices, and presumably impaired goal-directed control, play a role in inflexibility induced by reward-paired cues. Indeed, given the role of the lateral OFC in the uncued versus cued rGT explored in Chapter 3, as well its previously established role in updating stored action-outcome contingencies and cue-guided learning (Izquierdo, 2017; Amodeo et al., 2017, Zeeb et al., 2010), this region could be mediating both the differential processing of rewards and punishments across different variants of the rGT and cue-induced inflexibility. Alternatively, the task bracketing hypothesis from Vandaele et al. (2017) posits that the inclusion of salient cues in decision-making tasks (lever insertion, in their case) encourages the formation of rigid stimulus-response patterns, rather than generally reducing cognitive flexibility. Thus, inflexible decision making on the cued rGT could be a result of cue-driven rigidity in stimulus-response patterns, rather than overall shifts in flexible decision making. Regardless of experimental condition, motivation to engage in the task declined with reinforcer devaluation in both Experiments 1 and 2; premature responses decreased when animals were sated, while the latency to choose an option increased. Interestingly, while latency 163 to collect reward is sensitive to satiety on the uncued rGT (Zeeb and Winstanley, 2013), this measure did not significantly increase with reinforcer devaluation in any group. Furthermore, the inclusion of cues on the rGT results in decreased collection latencies, particularly in risky rats (Hathaway et al., 2021). It therefore may be that the presence of the cues invigorates responding to reward delivery, and the effect is not dependent on the value of the reward or computation within the lateral OFC, as this measure was unaffected by devaluation with or without drug administration in Experiment 2. In addition, the number of trials completed decreased in all rats in response to devaluation, but the effect only reached significance in rats who also received the 2C antagonist. On the uncued rGT, trial completion was reduced in response to devaluation, but this was prevented by disconnecting the BLA and lateral OFC (Zeeb and Winstanley, 2013). Thus, the lateral OFC may play a key role in inhibiting perseverative responding on the rGT. That intra-lateral OFC infusion of the 2C antagonist selectively rendered decision-making and trial completion more sensitive to reinforcer devaluation, without altering the effects of satiety on other variables, further supports the hypothesis that local 5-HT2C antagonism facilitated some form of cognitive flexibility. In Experiment 1, risk-preferring rats were only sensitive to reinforcer devaluation on tasks that did not feature consistent win-paired cues, and optimal rats were sensitive to this manipulation regardless of the task, thus suggesting that inflexible responding and decision-making deficits have overlapping mechanisms. However, from visual inspection of Figure 3.3, it is clear that shifts in P1-P4 choice among risk-preferring rats that did show sensitivity to devaluation are quite pronounced, whereas shifts in optimal rats are, while statistically significant, rather modest in size. Thus, insensitivity to devaluation and cue-induced risky choice do not completely overlap. Indeed, in order to maintain choice of the optimal options, optimal 164 rats must remain insensitive to infrequent punishments on these options. A relatively higher preference for the risky options on tasks that don’t feature win-paired cues might be driven by increased sensitivity to immediate feedback (i.e., higher tendency to switch after a loss, even on optimal options), at the expense of long-term payouts. On the probabilistic reversal learning task, increased sensitivity to immediate feedback resulting from inactivation of the prelimbic region actually increases the number of reversals completed in a session (Dalton et al., 2014). Thus, there is some support that increased impact of the immediately preceding trial is associated with increased flexibility. Given that inactivation of the prelimbic region impairs decision making on the uncued rGT (Zeeb et al., 2015), higher sensitivity to the preceding outcome may hinder the development of the optimal strategy in the case of the rGT. Indeed, some have posited that differences in feedback sensitivity can explain gender differences in the development of an optimal choice strategy on the IGT in humans (Bos et al., 2013). Thus, on the uncued rGT, and other variants that do not feature consistent reward-paired cues, relatively higher preference for the risky options as well as bigger shifts in choice following reinforcer devaluation may both be explained by increased sensitivity to immediate feedback (i.e., higher levels of win-stay/lose-switch) leading to a higher degree of variability in decision making. Differences in striatal D2 receptor availability is an intriguing potential neural mechanism underlying this effect; D2 receptor availability in the striatum predicts sensitivity to positive feedback (i.e., increased win-stay), leading to better reversal learning performance (Groman et al., 2011, 2014). Furthermore, administration of a D2 antagonist improved performance on the uncued rGT (Zeeb et al., 2009). Thus, increased D2 receptor availability in the striatum may lead to more variability in choice on the uncued rGT, resulting in relatively 165 more selection of the risky options but also a higher degree of flexibility when responding to changes in outcome value. While this may be the case for riskier rats on the uncued task and other tasks lacking consistent win-paired cues, it clearly is not the case for tasks that do feature such cues. Risk-preferring rats on the cued rGT do not shift their choice in response to reinforcer devaluation. Based on previous literature, one may conclude from this that goal-directed control of behaviour is reduced in risk-preferring rats on the cued rGT (Balleine & Dickinson, 1998). However, the experimental manipulations presented in this dissertation instead suggest that goal-directed control is maladaptive rather than hypoactive in these rats. As discussed in Section 1.3.1.1 of the introduction, model-based RL is a strategy for task learning that, if not the same as goal-directed behaviour, at least largely overlaps with it. In Experiment 3, I showed that inhibiting the lateral OFC actually delays the development of risk preference. As inhibition of VTA dopamine transmission also improves choice in males (Hynes et al., 2020, 2021), and has been implicated in model-based control (Langdon, Sharpe, et al., 2019; Sharpe et al., 2020), the internal model of the task structure that is created by lateral OFC, ventral striatum, and perhaps DMS circuitry may be incorrect on the cued rGT, such that inhibiting these regions improves choice. In this framework, the results of Experiment 2 may indicate that 2C antagonism in the lateral OFC improves the accuracy of the created model by reintroducing flexibility into action-outcome contingencies and thereby permitting the correct integration of time-out penalties, thus rectifying erroneous lateral OFC output and improving choice and reducing inflexibility. This process seems to be specific to risk-preferring rats, as manipulations on the cued rGT in both Experiments 2 and 3 only improved decision making in rats that preferred the risky options. Thus, the lateral OFC may play differential roles in risky versus optimal rats depending on the 166 presence versus absence of reward-paired cues. At the very least, if lateral OFC output is also erroneous in optimal rats on the cued rGT, they are able to disregard this, and other cortical regions may instead control choice patterns. The bidirectional effect of lateral OFC versus nigrostriatal dopamine inhibition in risk-preferring rats may suggest that these two regions contribute to opposing learning strategies. If nigrostriatal activity supports model-free reinforcement learning, then the inhibition of this learning strategy may lead to increased control of model-based systems that erroneously guide choice towards the risky options, thus causing the transient decrease in score that was observed in Experiment 4. However, differences in effects may be instead owed to sex differences. As discussed in Section 1.5, distinct sex differences are present in both dorsal striatal dopamine release in rats and OFC activity on the IGT in humans. Thus, further research into the role of the lateral OFC in females and the nigrostriatal dopamine pathway in males is needed to confirm whether inhibition of these regions leads to opposing effects in risk-preferring rats during early training on the cued rGT within the same sex. Regardless, the effects of both Experiments 3 and 4 were largely constrained to the learning phase of the rGT. Certainly, there is evidence that goal-directed/model-based control of behaviour dominates in early training, when knowledge of the task structure and contingencies is highly uncertain (reviewed in Section 1.31, 1.3.2). Thus, it makes sense that the impact of maladaptive goal-directed control would be most obvious in early sessions. The degree of goal-directed versus habitual control may therefore be more relevant to the speed of determining the best strategy during learning, rather than the overall degree of risk preference. Other brain regions and/or neurotransmitters must be responsible for the continued decline in risky choice beyond early sessions. Nevertheless, it is clear that the regions and transmitters implicated by the 167 current studies, as well as the regions and transmitters that regulate overall risk preference, are all impacted by the presence of predictable, win-paired cues: in Experiment 1, the presence of such cues affected both early learning (as assessed by RL models) and final levels of risk preference (as assessed by baseline analyses). Thus, reward-paired cues impact processing in multiple brain regions mediating diverse functions across different task stages to modulate feedback processing in early learning, increase endpoint risky choice, and induce inflexibility. 7.3 Theoretical implications and future directions 7.3.1 Neurobiological mechanisms of the impact of cues on decision making One implication from these results is that the lateral OFC may be functioning differently in risky versus optimal rats on the uncued versus cued rGT. While Experiments 2 and 3 strongly suggest this region mediates differences in decision making across the two tasks, it is unclear how neuronal dynamics differ on a trial-by-trial basis between tasks and preference groups. It’s possible that optimal choice is supported by differential responding to wins on optimal versus risky options within the lateral OFC, similar to what has been observed on the probabilistic reversal learning task (Amodeo et al., 2017). However, Experiment 1 implicates differential processing of losses as a key element of risky choice, and as such it could be possible that neuronal responses to losses differ across tasks and/or decision-making profiles. Another possibility is that the signal-to-noise ratio is reduced when win-paired cues are added to the task. Indeed, noisy lateral OFC signaling due to reduced firing of GABAergic interneurons can develop after long-term exposure to a high-fat, high-sugar diet, leading to impairments in sensitivity to outcome devaluation (Seabrook et al., 2022). Directly measuring neural activity in the lateral OFC using techniques such as calcium imaging while rats learn the uncued versus cued rGT could provide critical evidence for determining which hypothesis is correct. 168 Additionally, several questions remain about the inputs and outputs that may contribute to differential processing within the lateral OFC, and how computations within the lateral OFC are translated into biased action selection. One possibility, as discussed earlier, is lateral OFC inputs to the ventral striatum. Another possibility is inputs to the DMS. In support of this, in an animal model of intermittent alcohol exposure, difficulties in switching between goal-directed versus habitual control were causally attributed to the OFC to DMS pathway (Renteria et al., 2018). Additionally, inputs from the lateral OFC to cholinergic interneurons in this striatal region are particularly important for model-based control of behaviour (Section 1.3.3.5). A third pathway of interest is circuitry between the BLA and OFC. Experiment 1 results suggest that the cue-reward association is particularly important for driving risky choice on the rGT. Reciprocal BLA-OFC circuitry has been implicated in mediating the impact of reward-associated cues on behaviour (Lichtenberg et al., 2017; Malvaez et al., 2019; Sias et al., 2021). This pathway also plays a role in learning of the uncued rGT, as well as reinforcer devaluation on this task variant (Zeeb & Winstanley, 2013); however, its role in regulating choice patterns on the cued rGT is currently unknown. Investigating these three pathways (OFC  ventral striatum, OFC  DMS, OFCBLA) would provide valuable insight into how these regions work together to either support optimal choice or bias selection towards the risky options, depending on the task. Pathway-specific DREADDs would be helpful in assessing the role of these pathways. The use of pathway-specific DREADDs could also be employed to resolve whether the DMS and DLS, as well as dopaminergic innervation of each subregion, make differential contributions to either cue-induced risky choice or inflexibility on the rGT. Considerable research has demonstrated the different roles of these regions in goal-directed versus habitual control, although results do not always cleanly divide these functions between the medial and 169 lateral dorsal striatum (Sections 1.3.3.1, 1.3.3.2, 1.3.4). Furthermore, the nigrostriatal circuit been shown to carry a wide variety of information, such as value, salience, and probability of outcomes, as well as guiding action initiation and selection (Surmeier et al., 2009; Lerner et al., 2015; Zhang et al., 2017; Saunders et al., 2018; Tricomi & Lempert, 2015). Given the nuanced effects observed in Experiment 3, there is clearly more to uncover in how the nigrostriatal pathway contributes to decision making on the cued rGT. It’s also possible, given that the dorsal striatum has been implicated in a wide variety of decision-making processes, that this region plays a role in decision making on the uncued rGT, which has not yet been assessed in males or females. Retrograde viral vectors infused into the DMS or DLS in TH:Cre rats could permit selective control over the medial versus lateral SNc, which could provide valuable insight into how dopaminergic innervation of different striatal subregions contributes to rGT learning and/or performance. 7.3.2 Cognitive mechanisms of the impact of cues on decision making While the reinforcer devaluation results of Experiments 1 and 2 indicate that reward-paired cues induce inflexibility in decision making, future research could further test this hypothesis. Contextual cues may be a critical factor in whether decision making is sensitive to devaluation or not. Amaya et al. (2020) found that sign-trackers were only sensitive to outcome devaluation when the outcome was devalued in the same context where autoshaping occurred (i.e., the operant box). They hypothesized that when devaluation took place in a different context (i.e., home cages), sign-trackers had difficulties generalizing this learning to the operant box, rather than being truly insensitive to devaluation. Indeed, sign-trackers versus goal-trackers are differentially affected by discrete versus contextual cues (Saunders et al.,, 2014; Pitchers et al., 2017), and these two cue types have distinct neural mechanisms (Namba et al., 2018), so there is 170 some support for the idea that individual differences in cue-mediated behaviours might depend on context. Thus, measures of inflexibility specifically within the context of the rGT might shed light on how win-paired cues impact flexible responding. One possibility would be to omit sucrose pellet delivery (i.e., extinction training), and observe how long it takes for trial completion to diminish in rats trained on the cued versus uncued rGT. A second possibility would be to switch rats between version A and version B of the task (General methods 2.3.2), between which the spatial locations of the P1-P4 options differ. Indeed, this type of reversal learning has been used by others as a measure of flexibility (Dellu-Hagedorn et al., 2018). Inconsistent results have been found regarding the degree to which sign-tracking versus goal-tracking correlates with risky choice on the cued rGT (Ferland et al., 2019; Swintosky et al., 2021). However, given the importance of context, as described above, it’s possible that observation of sign-tracking versus goal-tracking during performance of the rGT itself would be a more valid measure. Video analysis of behaviour during task performance combined with a slight re-designing of the task could assess this. At the moment, delivery of the light cues occurs both in the stimulus array and the pellet tray. If the location of the light cue was separated from the choice locations and reward delivery location (for example, the lights above the levers in the operant box), then interaction with the lights versus the pellet tray could serve as a measure of sign-tracking versus goal-tracking during performance of the cued rGT. Lastly, while conditioned reinforcement does not correlate with risky choice, it’s possible that rats trained on the cued rGT would be more willing to exert physical effort to initiate a trial compared to rats on the uncued rGT. For example, a progressive ratio schedule of lever pressing to initiate a trial could be employed to assess breakpoints in rats trained on the uncued versus cued rGT. This would perhaps be more relevant to compulsive gambling behaviour, as there 171 would be a physical “cost” to gamble, in as far as an rGT trial can be considered gambling. Furthermore, it may shed light on whether excessive goal-directed control underlies cue-induced risky choice, as higher breakpoints indicate higher effectiveness of reinforcement (Roane, 2008). 7.4 Limitations and critical considerations The first limitation is that no direct comparisons between sexes were made. While practical constraints due to large cohort sizes did not permit the inclusion of both sexes, this is certainly a weakness in the current studies. As mentioned throughout the dissertation, sex differences in the functioning of the brain regions and circuitry examined in the present experiments have been previously reported (Section 1.5). Thus, running the same studies but in females for chapters 1-3, and males for chapter 4, would be useful for understanding how the impact of reward-paired cues may contribute to gender differences that have been observed in both gambling disorder and substance use disorder in humans (Becker & Koob, 2016; Bos et al., 2013). Secondly, while the use of the DREADDs technique is powerful and well validated (Smith et al., 2016), it comes with drawbacks as well. It’s impossible to know if there are compensatory actions that are diminishing the effects of experimental manipulations over time. Furthermore, it is not possible to know how neuronal activity and their output pathways are being affected without simultaneously employing neuronal recording techniques, which is challenging to do, and would certainly be difficult to employ while rats are performing the rGT. For example, DREADDs receptors have been noted to only transfect about 60% of neurons (Smith et al., 2016; Gremel & Costa, 2013); the remaining 40% of cells can be modulated in the opposite direction due to circuit changes (Chang et al., 2015). Differences between observed results in Experiment 3 and a previous study examining the impact of lesioning the OFC prior to 172 task training (Zeeb & Winstanley, 2011) may be explained by unintended effects on local circuitry in either or both studies. Additionally, the viral approach used for Chapter 5 targeted both excitatory and inhibitory neurons, which may differentially impact behaviour compared to selective activation or inhibition of pyramidal neurons in the OFC. Markers of neuronal activity such as c-Fos could be analyzed to shed light on how exactly regional activity is being manipulated; ideally this would occur at the timepoint when the effects on decision making were observed. Thus, a small study in which DREADDs are inserted into the lateral OFC and rats are then euthanized ~5 sessions into rGT training for the purposes of immunohistochemistry analysis may be warranted. For Chapter 6, follow-up studies could be conducted to specifically confirm the role of the nigrostriatal pathway versus other regions that SNc dopamine neurons may project to; for example, CNO could be administered via guide cannulae directly to the dorsal striatum. Another limitation is that differences in effects between risky and optimal rats are easier to study on the cued rGT, as it is common to get a 50/50 split in a single cohort. Thus, smaller cohort sizes may be used to interrogate the impacts of manipulations across rats with diverse decision-making profiles. Conversely, risky rats are uncommon on the uncued rGT. There are likely shared neurobiological mechanisms in risky rats across tasks, but we are underpowered to detect them. It’s possible to get around this by doing a median split on risk score, and compare relatively more versus less risky rats on the uncued rGT, even though few are truly risk-preferring rats. This approach was taken in Experiment 3. Nonetheless, conclusions about differences in risky rats on the uncued versus cued task could not be drawn. Data collection methods that result in a rich dataset from single subjects, such as calcium imaging, may allow for a more in-depth investigation of risky rats on the uncued task. Given that the IGT is “uncued” so to speak, and individuals with gambling disorder and substance use disorder show deficits on this 173 task that are predictive of relapse and treatment dropout (Stevens et al., 2013; Bolla et al., 2003; Goudriaan et al., 2005), studying risk-preferring rats on the uncued task may be of great clinical relevance. Lastly, great efforts were taken to carefully employ computational modeling techniques in Experiment 1 to shed light on how learning dynamics differed between task variants. As this was the first time these task variants were analyzed, and only the second time that RL models have been applied to rGT data (the first being Langdon et al., 2021), the focus was to examine model fits and parameter estimates at the group-level rather than subject-level. As such, there is ample room for further investigation into individual differences. For example, different rats could be better fit by different models, despite the overall groups being best fit by the independent cost model. Examining subject-level parameter estimates to determine that parameters associated with learning from negative outcomes were predictive of a lower risk score at the end of training demonstrates that valuable insight can be gained from examining individual rat behaviour. Most likely, there is heterogeneity in the reasons why rats are risky; it may be that the majority of risk-preferring rats discount the losses and thus favour the risky options, but there may be some that hyper-learn from the large wins, or can’t understand or remember the contingencies across the four options. Indeed, in some cases, cognitive deficits can explain the use of habitual over goal-directed control (Sjoerds et al., 2016; Hogarth, 2020; Sebold et al., 2014). Thus, attending to individual differences in conjunction with modelling approaches would be a fruitful way forward in understanding risky choice on the rGT. 7.5 Concluding remarks The experiments included in this dissertation have shed light on how reward-paired cues impact both risky choice and flexibility in decision-making patterns in an animal model of 174 cost/benefit decision making. I have implicated the lateral OFC, and by extension, aberrant goal-directed control, as an important neural locus and cognitive mechanism for contributing to both of these effects. I have also provided evidence for the nuanced role that the nigrostriatal pathway plays in influencing cost/benefit decision making in female rats, which are critically understudied in investigations of habitual control of behaviour. These results are relevant to understanding the contribution of reward-paired cues to maladaptive decision-making patterns in individuals with gambling disorder and substance use disorder. Gambling disorder causes significant personal and financial consequences in those affected. Similarly, substance use disorders incur considerable costs at both the individual and societal level. Developing effective treatments for these behavioural disorders is of utmost importance. The research in this dissertation made significant headway in precisely identifying the key cognitive and neurobiological mechanisms involved in risky, inflexible choice induced by reward-paired cues. Results from the included experiments delineate the contribution of maladaptive goal-directed control to the development of disadvantageous risky choice patterns. By doing so, this research could spur development of medical and behavioural treatments that reduce the pervasive nature of compulsive reward-seeking associated with addiction and help to prevent relapse, by targeting the control of cues over behaviour. 175 References Adams, C. D. (1982). Variations in the sensitivity of instrumental responding to reinforcer devaluation. The Quarterly Journal of Experimental Psychology Section B, 34(2), 77–98. https://doi.org/10.1080/14640748208400878 Adams, W. K., Barkus, C., Ferland, J. N., Sharp, T., & Winstanley, C. A. (2017). Pharmacological evidence that 5-HT2C receptor blockade selectively improves decision making when rewards are paired with audiovisual cues in a rat gambling task. Psychopharmacology, 234(20), 3091–3104. https://doi.org/10.1007/s00213-017-4696-4 Ahn, W., Haines, N., & Zhang, L. (2017). Revealing neurocomputational mechanisms of reinforcement learning and decision-making with the hBayesDM package. Computational Psychiatry, 1, 24-57. doi:10.1162/CPSY_a_00002 Ainslie, G. (1975). Specious reward: A behavioral theory of impulsiveness and impulse control. Psychological Bulletin, 82(4), 463–496. https://doi.org/10.1037/h0076860 Alsiö, J., Lehmann, O., McKenzie, C., Theobald, D. E., Searle, L., Xia, J., Dalley, J., & Robbins, T. (2021). Serotonergic innervations of the orbitofrontal and medial-prefrontal cortices are differentially involved in visual discrimination and reversal learning in rats. Cerebral Cortex, 31, 1090-1105. https://doi.org/10.1093/cercor/bhaa277 Alter, A. (2017). Irresistible: The rise of addictive technology and the business of keeping us hooked. New York: Penguin Press. Amaya, K. A., Stott, J. J., & Smith, K. S. (2020). Sign-tracking behavior is sensitive to outcome devaluation in a devaluation context-dependent manner: Implications for analyzing habitual behavior. Learning & Memory, 27(4), 136–149. https://doi.org/10.1101/lm.051144.119 Amlung, M., Vedelago, L., Acker, J., Balodis, I., & MacKillop, J. (2017). Steep delay discounting and addictive behavior: A meta-analysis of continuous associations. Addiction, 112(1), 51–62. https://doi.org/10.1111/add.13535 Amodeo, L. R., McMurray, M. S., & Roitman, J. D. (2016). Orbitofrontal cortex reflects changes in response–outcome contingencies during probabilistic reversal learning. Neuroscience, 345, 27–37. https://doi.org/10.1016/j.neuroscience.2016.03.034 Anastasio, N. C., Lanfranco, M. F., Bubar, M. J., Seitz, P. K., Stutz, S. J., McGinnis, A. G., Watson, C. S., & Cunningham, K. A. (2010). Serotonin 5-HT2C receptor protein expression is enriched in synaptosomal and post-synaptic compartments of rat cortex. Journal of Neurochemistry, 113(6), 1504–1515. https://doi.org/10.1111/j.1471-4159.2010.06694.x Anastasio, N. C., Stutz, S. J., Fink, L. H. L., Swinford-Jackson, S. E., Sears, R. M., DiLeone, R. J., Rice, K. C., Moeller, F. G., & Cunningham, K. A. (2015). Serotonin (5-HT) 5-HT2A Receptor (5-HT2AR):5-HT2CR imbalance in medial prefrontal cortex associates with motor 176 impulsivity. ACS Chemical Neuroscience, 6(7), 1248–1258. https://doi.org/10.1021/acschemneuro.5b00094 Anastasio, N. C., Stutz, S. J., Fox, R. G., Sears, R. M., Emeson, R. B., DiLeone, R. J., …, Cunningham, K. A. (2014). Functional status of the serotonin 5-HT2C receptor (5-HT2CR) drives interlocked phenotypes that precipitate relapse-like behaviors in cocaine dependence. Neuropsychopharmacology, 39(2), 370–382. https://doi.org/10.1038/npp.2013.199 Ansary, S., Russell, B., Winstanley, C. A. (2021). The link between conditioned reinforcers and rodent decision-making on a cued gambling task. Poster presented at Society for Neuroscience, 2021. Online. Anselme, P., & Robinson, M. J. F. (2020). From sign-tracking to attentional bias: Implications for gambling and substance use disorders. Progress in Neuro-Psychopharmacology & Biological Psychiatry, 99, 109861. https://doi.org/10.1016/j.pnpbp.2020.109861 Anselme, P., Robinson, M. J. F., & Berridge, K. C. (2013). Reward uncertainty enhances incentive salience attribution as sign-tracking. Behavioural Brain Research, 238, 53–61. https://doi.org/10.1016/j.bbr.2012.10.006 Antzoulatos, E. G., & Miller, E. K. (2011). Differences between neural activity in prefrontal cortex and striatum during learning of novel abstract categories. Neuron, 71(2), 243–249. https://doi.org/10.1016/j.neuron.2011.05.040 Arinze, I., & Moorman, D. E. (2020). Selective impact of lateral orbitofrontal cortex inactivation on reinstatement of alcohol seeking in male Long-Evans rats. Neuropharmacology, 168, 108007. https://doi.org/10.1016/j.neuropharm.2020.108007 Armstrong, M. J., & Okun, M. S. (2020). Diagnosis and treatment of Parkinson disease: A review. JAMA, 323(6), 548–560. https://doi.org/10.1001/jama.2019.22360 Arroyo, M., Markou, A., Robbins, T. W., & Everitt, B. J. (1998). Acquisition, maintenance and reinstatement of intravenous cocaine self-administration under a second-order schedule of reinforcement in rats: Effects of conditioned cues and continuous access to cocaine. Psychopharmacology, 140(3), 331–344. https://doi.org/10.1007/s002130050774 Balleine, B. W., & Dezfouli, A. (2019). Hierarchical Action Control: Adaptive Collaboration Between Actions and Habits. Frontiers in Psychology, 10, 2735. https://doi.org/10.3389/fpsyg.2019.02735 Balleine, B. W., & Dickinson, A. (1998). Goal-directed instrumental action: Contingency and incentive learning and their cortical substrates. Neuropharmacology, 37(4–5), 407–419. https://doi.org/10.1016/s0028-3908(98)00033-1 Bari, A., & Robbins, T. W. (2013). Inhibition and impulsivity: Behavioral and neural basis of response control. Progress in Neurobiology, 108, 44–79. https://doi.org/10.1016/j.pneurobio.2013.06.005 177 Barlow, R. L., Alsiö, J., Jupp, B., Rabinovich, R., Shrestha, S., Roberts, A. C., Robbins, T. W., & Dalley, J. W. (2015). Markers of serotonergic function in the orbitofrontal cortex and dorsal raphé nucleus predict individual variation in spatial-discrimination serial reversal learning. Neuropsychopharmacology (New York, N.Y.), 40(7), 1619–1630. https://doi.org/10.1038/npp.2014.335 Barreiros, I. V., Panayi, M. C., & Walton, M. E. (2021). Organization of afferents along the anterior-posterior and medial-lateral axes of the rat orbitofrontal cortex. Neuroscience, 460, 53-68. https://doi.org/10.1016/j.neuroscience.2021.02.017 Barrus, M. M., & Winstanley, C. A. (2016). Dopamine D3 receptors modulate the ability of win-paired cues to increase risky choice in a rat gambling task. Journal of Neuroscience, 36(3), 785–794. https://doi.org/10.1523/JNEUROSCI.2225-15.2016 Barrus, M. M., Hosking, J. G., Zeeb, F. D., Tremblay, M., & Winstanley, C. A. (2015). Disadvantageous decision-making on a rodent gambling task is associated with increased motor impulsivity in a population of male rats. Journal of Psychiatry and Neuroscience, 40(2), 108–117. https://doi.org/10.1503/jpn.140045 Bayless, D. W., Darling, J. S., Stout, W. J., & Daniel, J. M. (2012). Sex differences in attentional processes in adult rats as measured by performance on the 5-choice serial reaction time task. Behavioural Brain Research, 235(1), 48–54. https://doi.org/10.1016/j.bbr.2012.07.028 Bechara, A., Damasio, A. R., Damasio, H., & Anderson, S. W. (1994). Insensitivity to future consequences following damage to human prefrontal cortex. Cognition, 50(1–3), 7–15. https://doi.org/10.1016/0010-0277(94)90018-3 Becker, J. B. (2016). Sex differences in addiction. Dialogues in Clinical Neuroscience, 18(4), 395–402. https://doi.org/10.31887/DCNS.2016.18.4/jbecker Beier, K. T., Steinberg, E. E., DeLoach, K. E., Xie, S., Miyamichi, K., Schwarz, L., Gao, X. J., Kremer, E. J., Malenka, R. C., & Luo, L. (2015). Circuit architecture of VTA dopamine neurons revealed by systematic input–output mapping. Cell, 162(3), 622–634. https://doi.org/10.1016/j.cell.2015.07.015 Benazzouz, A., Gross, C., Féger, J., Boraud, T., & Bioulac, B. (1993). Reversal of rigidity and improvement in motor performance by subthalamic high-frequency stimulation in MPTP-treated monkeys. The European Journal of Neuroscience, 5(4), 382–389. https://doi.org/10.1111/j.1460-9568.1993.tb00505.x Berg, E. A. (1948). A simple objective technique for measuring flexibility in thinking. The Journal of General Psychology, 39(1), 15–22. https://doi.org/10.1080/00221309.1948.9918159 Bertholomey, M. L., Nagarajan, V., & Torregrossa, M. M. (2016). Sex differences in reinstatement of alcohol seeking in response to cues and yohimbine in rats with and without 178 a history of adolescent corticosterone exposure. Psychopharmacology, 233(12), 2277–2287. https://doi.org/10.1007/s00213-016-4278-x Birrell, J. M., & Brown, V. J. (2000). Medial frontal cortex mediates perceptual attentional set shifting in the rat. The Journal of Neuroscience: The Official Journal of the Society for Neuroscience, 20(11), 4320–4324. https://doi.org/10.1523/JNEUROSCI.20-11-04320.2000 Bissonette, G. B., & Roesch, M. R. (2016). Neurophysiology of reward-guided behavior: Correlates related to predictions, value, motivation, errors, attention, and action. Current Topics in Behavioral Neurosciences, 27, 199–230. https://doi.org/10.1007/7854_2015_382 Bissonette, G. B., Martins, G. J., Franz, T. M., Harper, E. S., Schoenbaum, G., & Powell, E. M. (2008). Double dissociation of the effects of medial and orbital prefrontal cortical lesions on attentional and affective shifts in mice. The Journal of Neuroscience, 28(44), 11124–11130. https://doi.org/10.1523/JNEUROSCI.2820-08.2008 Bissonette, G. B., Powell, E. M., & Roesch, M. R. (2013). Neural structures underlying set-shifting: Roles of medial prefrontal cortex and anterior cingulate cortex. Behavioural Brain Research, 250, 91–101. https://doi.org/10.1016/j.bbr.2013.04.037 Bissonette, G. B., Schoenbaum, G., Roesch, M. R., & Powell, E. M. (2015). Interneurons are necessary for coordinated activity during reversal learning in orbitofrontal cortex. Biological Psychiatry (1969), 77(5), 454–464. https://doi.org/10.1016/j.biopsych.2014.07.023 Boileau, I., Payer, D., Chugani, B., Lobo, D. S. S., Houle, S., Wilson, A. A., Warsh, J., Kish, S. J., & Zack, M. (2014). In vivo evidence for greater amphetamine-induced dopamine release in pathological gambling: A positron emission tomography study with [11C]-(+)-PHNO. Molecular Psychiatry, 19(12), Article 12. https://doi.org/10.1038/mp.2013.163 Bolding, G., Sherr, L., & Elford, J. (2002). Use of anabolic steroids and associated health risks among gay men attending London gyms. Addiction, 97(2), 195–203. https://doi.org/10.1046/j.1360-0443.2002.00031.x Bolla, K. I., Eldreth, D. A., London, E. D., Kiehl, K. A., Mouratidis, M., Contoreggi, C., …, Ernst, M. (2003). Orbitofrontal cortex dysfunction in abstinent cocaine abusers performing a decision-making task. NeuroImage, 19(3), 1085–1094. https://doi.org/10.1016/s1053-8119(03)00113-7 Bolla, K. I., Eldreth, D. A., Matochik, J. A., & Cadet, J. L. (2004). Sex-related differences in a gambling task and its neurological correlates. Cerebral Cortex, 14(11), 1226–1232. https://doi.org/10.1093/cercor/bhh083 Boothman, L., Raley, J., Denk, F., Hirani, E., & Sharp, T. (2006). In vivo evidence that 5-HT2C receptors inhibit 5-HT neuronal activity via a GABAergic mechanism. British Journal of Pharmacology, 149(7), 861–869. https://doi.org/10.1038/sj.bjp.0706935 179 Boulougouris, V., & Robbins, T. W. (2010). Enhancement of spatial reversal learning by 5-HT2C receptor antagonism is neuroanatomically specific. The Journal of Neuroscience, 30(3), 930–938. https://doi.org/10.1523/jneurosci.4312-09.2010 Bradfleld, L., Morris, R., & Balleine, B. W. (2017). OCD as a failure to integrate goal-directed and habitual action control. In Obsessive-compulsive disorder: Phenomenology, pathophysiology, and treatment (pp. 343–352). Oxford University Press. Brevers, D., Cleeremans, A., Verbruggen, F., Bechara, A., Kornreich, C., Verbanck, P., & Noël, X. (2012). Impulsive action but not impulsive choice determines problem gambling severity. PLOS ONE, 7(11), e50647. https://doi.org/10.1371/journal.pone.0050647 Brevers, D., Noël, X., Bechara, A., Vanavermaete, N., Verbanck, P., & Kornreich, C. (2015). Effect of casino-related sound, red light and pairs on decision-making during the Iowa gambling task. Journal of Gambling Studies, 31(2), 409–421. https://doi.org/10.1007/s10899-013-9441-2 Brog, J. S., Salyapongse, A., Deutch, A. Y., & Zahm, D. S. (1993). The patterns of afferent innervation of the core and shell in the “accumbens” part of the rat ventral striatum: Immunohistochemical detection of retrogradely transported fluoro-gold. The Journal of Comparative Neurology, 338(2), 255–278. https://doi.org/10.1002/cne.903380209 Brown, P. L., & Jenkins, H. M. (1968). Auto-shaping of the pigeon’s key-peck. Journal of the Experimental Analysis of Behavior, 11(1), 1–8. https://doi.org/10.1901/jeab.1968.11-1 Browne, C. J., Ji, X., Higgins, G. A., Fletcher, P. J., & Harvey-Lewis, C. (2017). Pharmacological modulation of 5-ht2c receptor activity produces bidirectional changes in locomotor activity, responding for a conditioned reinforcer, and mesolimbic DA release in C57BL/6 Mice. Neuropsychopharmacology (New York, N.Y.), 42(11), 2178–2187. https://doi.org/10.1038/npp.2017.124 Bryden, D. W., & Roesch, M. R. (2015). Executive control signals in orbitofrontal cortex during response inhibition. The Journal of Neuroscience, 35, 3903–3914. https://doi.org/10.1523/JNEUROSCI.3587-14.2015 Bubar, M. J., Seitz, P. K., Thomas, M. L., & Cunningham, K. A. (2005). Validation of a selective serotonin 5-HT2C receptor antibody for utilization in fluorescence immunohistochemistry studies. Brain Research, 1063(2), 105–113. https://doi.org/10.1016/j.brainres.2005.09.050 Bull, P. N., Tippett, L. J., & Addis, D. R. (2015). Decision making in healthy participants on the Iowa Gambling Task: New insights from an operant approach. Frontiers in Psychology, 6. https://www.frontiersin.org/articles/10.3389/fpsyg.2015.00391 180 Burton, A. C., Nakamura, K., & Roesch, M. R. (2015). From ventral-medial to dorsal-lateral striatum: Neural correlates of reward-guided decision-making. Neurobiology of Learning and Memory, 0, 51–59. https://doi.org/10.1016/j.nlm.2014.05.003 Carli, M., Robbins, T. W., Evenden, J. L., & Everitt, B. J. (1983). Effects of lesions to ascending noradrenergic neurones on performance of a 5-choice serial reaction task in rats; Implications for theories of dorsal noradrenergic bundle function based on selective attention and arousal. Behavioural Brain Research, 9(3), 361–380. https://doi.org/10.1016/0166-4328(83)90138-9 Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., . . . Riddell, A. (2017). Stan: A probabilistic programming language. Journal of Statistical Software, 76, 1-32. doi:10.18637/jss.v076.i01 Cartoni, E., Balleine, B., & Baldassarre, G. (2016). Appetitive Pavlovian-instrumental Transfer: A review. Neuroscience & Biobehavioral Reviews, 71, 829–848. https://doi.org/10.1016/j.neubiorev.2016.09.020 Casado-Sainz, A., Gudmundsen, F., Baerentzen, S. L., Lange, D., Ringsted, A., Martinez-Tejada, I., …, Palner, M. (2022). Dorsal striatal dopamine induces fronto-cortical hypoactivity and attenuates anxiety and compulsive behaviors in rats. Neuropsychopharmacology, 47(2), Article 2. https://doi.org/10.1038/s41386-021-01207-y Castañé, A., Theobald, D. E. H., & Robbins, T. W. (2010). Selective lesions of the dorsomedial striatum impair serial spatial reversal learning in rats. Behavioural Brain Research, 210(1), 74–83. https://doi.org/10.1016/j.bbr.2010.02.017 Chang, S. E., Todd, T. P., Bucci, D. J., & Smith, K. S. (2015). Chemogenetic manipulation of ventral pallidal neurons impairs acquisition of sign-tracking in rats. The European Journal of Neuroscience, 42(12), 3105–3116. https://doi.org/10.1111/ejn.13103 Cherkasova, M. V., Clark, L., Barton, J. J. S., Schulzer, M., Shafiee, M., Kingstone, A., Stoessl, A. J., & Winstanley, C. A. (2018). Win-concurrent sensory cues can promote riskier choice. The Journal of Neuroscience, 38(48), 10362–10370. https://doi.org/10.1523/JNEUROSCI.1171-18.2018 Chernoff, C. C., Hynes, T., Hrelja, K. M., Tse, M., Betts, G. D., Lysenko-Martin, M. M., …, Winstanley, C. A. (2022) Chemogenetic sensitization of midbrain dopamine neurons exacerbates cue-provoked risk taking and amplifies cocaine self-administration. Poster presented at Dopamine 2022, Montreal, QC. Chernoff, C. S., Hynes, T. J., & Winstanley, C. A. (2021). Noradrenergic contributions to cue-driven risk-taking and impulsivity. Psychopharmacology, 238, 1765-1779. doi:10.1007/s00213-021-05806-x 181 Chow, J. J., Smith, A. P., Wilson, A. G., Zentall, T. R., & Beckmann, J. S. (2017). Suboptimal choice in rats: Incentive salience attribution promotes maladaptive decision-making. Behavioural Brain Research, 320, 244–254. https://doi.org/10.1016/j.bbr.2016.12.013 Chudasama, Y., Passetti, F., Rhodes, S. E. V., Lopian, D., Desai, A., & Robbins, T. W. (2003). Dissociable aspects of performance on the 5-choice serial reaction time task following lesions of the dorsal anterior cingulate, infralimbic and orbitofrontal cortex in the rat: Differential effects on selectivity, impulsivity and compulsivity. Behavioural Brain Research, 146(1), 105–119. https://doi.org/10.1016/j.bbr.2003.09.020 Clarke, H. F., Dalley, J. W., Crofts, H. S., Robbins, T. W., & Roberts, A. C. (2004). Cognitive inflexibility after prefrontal serotonin depletion. Science (New York, N.Y.), 304(5672), 878–880. https://doi.org/10.1126/science.1094987 Clemett, D., Punhani, T., Duxon, M. S., Blackburn, T. P., & Fone, K. C. F. (2000). Immunohistochemical localisation of the 5-HT2C receptor protein in the rat CNS. Neuropharmacology, 39(1), 123–132. https://doi.org/10.1016/S0028-3908(99)00086-6 Cocker, P. J., Hosking, J. G., Benoit, J., & Winstanley, C. A. (2012). Sensitivity to cognitive effort mediates psychostimulant effects on a novel rodent cost/benefit decision-making task. Neuropsychopharmacology, 37(8), 1825–1837. https://doi.org/10.1038/npp.2012.30 Collins, A. L., Aitken, T. J., Huang, I.-W., Shieh, C., Greenfield, V. Y., Monbouquette, H. G., Ostlund, S. B., & Wassum, Kate. M. (2019). Nucleus accumbens cholinergic interneurons oppose cue-motivated behavior. Biological Psychiatry, 86(5), 388–396. https://doi.org/10.1016/j.biopsych.2019.02.014 Collins, P., Wilkinson, L. S., Everitt, B. J., Robbins, T. W., & Roberts, A. C. (2000). The effect of dopamine depletion from the caudate nucleus of the common marmoset (Callithrix jacchus) on tests of prefrontal cognitive function. Behavioral Neuroscience, 114(1), 3–17. https://doi.org/10.1037//0735-7044.114.1.3 Colwill, R. M., & Rescorla, R. A. (1990). Effect of reinforcer devaluation on discriminative control of instrumental behavior. Journal of Experimental Psychology. Animal Behavior Processes, 16(1), 40–47. Constantinople, C. M., Piet, A. T., Bibawi, P., Akrami, A., Kopec, C., & Brody, C. D. (2019). Lateral orbitofrontal cortex promotes trial-by-trial learning of risky, but not spatial, biases. ELife, 8, e49744. https://doi.org/10.7554/eLife.49744 Cools, R., Altamirano, L., & D’Esposito, M. (2006). Reversal learning in Parkinson’s disease depends on medication status and outcome valence. Neuropsychologia, 44(10), 1663–1673. https://doi.org/10.1016/j.neuropsychologia.2006.03.030 Cools, R., Frank, M. J., Gibbs, S. E., Miyakawa, A., Jagust, W., & D’Esposito, M. (2009). Striatal dopamine predicts outcome-specific reversal learning and its sensitivity to 182 dopaminergic drug administration. The Journal of Neuroscience, 29(5), 1538–1543. https://doi.org/10.1523/JNEUROSCI.4467-08.2009 Corbit, L. H., & Balleine, B. W. (2011). The general and outcome-specific forms of Pavlovian-instrumental transfer are differentially mediated by the nucleus accumbens core and shell. Journal of Neuroscience, 31(33), 11786–11794. https://doi.org/10.1523/JNEUROSCI.2711-11.2011 Corbit, L. H., Nie, H., & Janak, P. H. (2012). Habitual alcohol seeking: Time course and the contribution of subregions of the dorsal striatum. Biological Psychiatry, 72(5), 389–395. https://doi.org/10.1016/j.biopsych.2012.02.024 Crofts, H. S., Dalley, J. W., Collins, P., Van Denderen, J. C., Everitt, B. J., Robbins, T. W., & Roberts, A. C. (2001). Differential effects of 6-OHDA lesions of the frontal cortex and caudate nucleus on the ability to acquire an attentional set. Cerebral Cortex, 11(11), 1015–1026. https://doi.org/10.1093/cercor/11.11.1015 Crowley, T. J., Dalwani, M. S., Mikulich-Gilbertson, S. K., Young, S. E., Sakai, J. T., …, Banich, M. T. (2015). Adolescents’ Neural Processing of Risky Decisions: Effects of Sex and Behavioral Disinhibition. PloS One, 10(7), e0132322. https://doi.org/10.1371/journal.pone.0132322 Currie, L. J., Harrison, M. B., Trugman, J. M., Bennett, J. P., & Wooten, G. F. (2004). Postmenopausal estrogen use affects risk for Parkinson disease. Archives of Neurology, 61(6), 886–888. https://doi.org/10.1001/archneur.61.6.886 Cushman, F., & Morris, A. (2015). Habitual control of goal selection in humans. Proceedings of the National Academy of Sciences of the United States of America, 112(45), 13817–13822. https://doi.org/10.1073/pnas.1506367112 Dalley, J. W., Everitt, B. J., & Robbins, T. W. (2011). Impulsivity, compulsivity, and top-down cognitive control. Neuron, 69(4), 680–694. https://doi.org/10.1016/j.neuron.2011.01.020 Dalton, G. L., Phillips, A. G., & Floresco, S. B. (2014). Preferential involvement by nucleus accumbens shell in mediating probabilistic learning and reversal shifts. Journal of Neuroscience, 34(13), 4618–4626. https://doi.org/10.1523/JNEUROSCI.5058-13.2014 Dalton, G. L., Wang, N. Y., Phillips, A. G., & Floresco, S. B. (2016). Multifaceted contributions by different regions of the orbitofrontal and medial prefrontal cortex to probabilistic reversal learning. Journal of Neuroscience, 36(6), 1996–2006. https://doi.org/10.1523/JNEUROSCI.3366-15.2016 Darling, J. S., & Daniel, J. M. (2019). Pubertal hormones mediate sex differences in levels of myelin basic protein in the orbitofrontal cortex of adult rats. Neuroscience, 406, 487–495. https://doi.org/10.1016/j.neuroscience.2019.03.041 183 Daw, N. D., Niv, Y., & Dayan, P. (2005). Uncertainty-based competition between prefrontal and dorsolateral striatal systems for behavioral control. Nature Neuroscience, 8(12), Article 12. https://doi.org/10.1038/nn1560 de Ruiter, M. B., Veltman, D. J., Goudriaan, A. E., Oosterlaan, J., Sjoerds, Z., & van den Brink, W. (2009). Response perseveration and ventral prefrontal sensitivity to reward and punishment in male problem gamblers and smokers. Neuropsychopharmacology, 34(4), 1027–1038. https://doi.org/10.1038/npp.2008.175 de Wit, S., Barker, R. A., Dickinson, A. D., & Cools, R. (2011). Habitual versus goal-directed action control in Parkinson disease. Journal of Cognitive Neuroscience, 23(5), 1218–1229. https://doi.org/10.1162/jocn.2010.21514 de Wit, S., Watson, P., Harsay, H. A., Cohen, M. X., van de Vijver, I., & Ridderinkhof, K. R. (2012). Corticostriatal connectivity underlies individual differences in the balance between habitual and goal-directed action control. The Journal of Neuroscience, 32(35), 12066–12075. https://doi.org/10.1523/JNEUROSCI.1088-12.2012 Dellu-Hagedorn, F., Rivalan, M., Fitoussi, A., & De Deurwaerdère, P. (2018). Inter-individual differences in the impulsive/compulsive dimension: Deciphering related dopaminergic and serotonergic metabolisms at rest. Philosophical Transactions of the Royal Society B: Biological Sciences, 373(1744), 20170154. https://doi.org/10.1098/rstb.2017.0154 Dias, R., Robbins, T. W., & Roberts, A. C. (1996a). Dissociation in prefrontal cortex of affective and attentional shifts. Nature, 380(6569), 69–72. https://doi.org/10.1038/380069a0 Dias, R., Robbins, T. W., & Roberts, A. C. (1996b). Primate analogue of the Wisconsin Card Sorting Test: Effects of excitotoxic lesions of the prefrontal cortex in the marmoset. Behavioral Neuroscience, 110(5), 872–886. https://doi.org/10.1037//0735-7044.110.5.872 Dickinson, A., Nicholas, D. J., & Adams, C. D. (1983). The effect of the instrumental training contingency on susceptibility to reinforcer devaluation. The Quarterly Journal of Experimental Psychology Section B, 35(1), 35–51. https://doi.org/10.1080/14640748308400912 Dickinson, A., Wood, N., & Smith, J. W. (2002). Alcohol seeking by rats: Action or habit? The Quarterly Journal of Experimental Psychology. B, Comparative and Physiological Psychology, 55(4), 331–348. https://doi.org/10.1080/0272499024400016 Dirnberger, G., & Jahanshahi, M. (2013). Executive dysfunction in Parkinson’s disease: A review. Journal of Neuropsychology, 7(2), 193–224. https://doi.org/10.1111/jnp.12028 Dixon, M. J., Collins, K., Harrigan, K. A., Graydon, C., & Fugelsang, J. A. (2013). Using sound to unmask losses disguised as wins in multiline slot machines. Journal of Gambling Studies, 31, 183-196. doi:10.1007/s10899-013-9411-8 184 Dixon, M. J., Harrigan, K. A., Sandhu, R., Collins, K., & Fugelsang, J. A. (2010). Losses disguised as wins in modern multi-line video slot machines. Addiction, 105, 1819-1824. doi:10.1111/j.1360-0443.2010.03050.x Dixon, M. J., Harrigan, K. A., Santesso, D. L., Graydon, C., Fugelsang, J. A., & Collins, K. (2013). The impact of sound in modern multiline video slot machine play. Journal of Gambling Studies, 30, 913-929. doi:10.1007/s10899-013-9391-8 Dolan, R. J., & Dayan, P. (2013). Goals and habits in the brain. Neuron, 80(2), 312–325. https://doi.org/10.1016/j.neuron.2013.09.007 Doll, B. B., Simon, D. A., & Daw, N. D. (2012). The ubiquity of model-based reinforcement learning. Current Opinion in Neurobiology, 22(6), 1075–1081. https://doi.org/10.1016/j.conb.2012.08.003 Donders, F. C. (1969). On the speed of mental processes. Acta Psychologica, 30, 412–431. https://doi.org/10.1016/0001-6918(69)90065-1 Downes, J. J., Roberts, A. C., Sahakian, B. J., Evenden, J. L., Morris, R. G., & Robbins, T. W. (1989). Impaired extra-dimensional shift performance in medicated and unmedicated Parkinson’s disease: Evidence for a specific attentional dysfunction. Neuropsychologia, 27(11–12), 1329–1343. https://doi.org/10.1016/0028-3932(89)90128-0 Eagle, D. M., Baunez, C., Hutcheson, D. M., Lehmann, O., Shah, A. P., & Robbins, T. W. (2008). Stop-signal reaction-time task performance: Role of prefrontal cortex and subthalamic nucleus. Cerebral Cortex, 18(1), 178–188. https://doi.org/10.1093/cercor/bhm044 Economidou, D., Theobald, D. E. H., Robbins, R. W., Everitt, B. J., & Dalley, J. W. (2012). Norepinephrine and dopamine modulate impulsivity on the five-choice serial reaction time task through opponent actions in the shell and core sub-regions of the nucleus accumbens. Neuropsychopharmacology, 37, 2057-2066. doi:10.1038/npp.2012.53 Everett, N. A., Carey, H. A., Cornish, J. L., & Baracz, S. J. (2020). Sign tracking predicts cue-induced but not drug-primed reinstatement to methamphetamine seeking in rats: Effects of oxytocin treatment. Journal of Psychopharmacology, 34(11), 1271–1279. https://doi.org/10.1177/0269881120954052 Everitt, B. J., & Robbins, T. W. (2013). From the ventral to the dorsal striatum: Devolving views of their roles in drug addiction. Neuroscience and Biobehavioral Reviews, 37(9 Pt A), 1946–1954. https://doi.org/10.1016/j.neubiorev.2013.02.010 Everitt, B. J., & Robbins, T. W. (2016). Drug addiction: Updating actions to habits to compulsions ten years on. Annual Review of Psychology, 67, 23–50. https://doi.org/10.1146/annurev-psych-122414-033457 185 Faure, A., Haberland, U., Condé, F., & Massioui, N. E. (2005). Lesion to the nigrostriatal dopamine system disrupts stimulus-response habit formation. Journal of Neuroscience, 25(11), 2771–2780. https://doi.org/10.1523/JNEUROSCI.3894-04.2005 Favier, M., Janickova, H., Justo, D., Kljakic, O., Runtz, L., Natsheh, J. Y., …, El Mestikawy, S. (2020). Cholinergic dysfunction in the dorsal striatum promotes habit formation and maladaptive eating. Journal of Clinical Investigation, 130(12), 6616–6630. https://doi.org/10.1172/JCI138532 Ferland, J. N., Hynes, T. J., Hounjet, C. D., Lindenbach, D., Vonder Haar, C., Adams, W. K., Phillips, A. G., & Winstanley, C. A. (2019). Prior exposure to salient win-paired cues in a rat gambling task increases sensitivity to cocaine self-administration and suppresses dopamine efflux in nucleus accumbens: Support for the reward deficiency hypothesis of addiction. The Journal of Neuroscience, 39(10), 1842–1854. https://doi.org/10.1523/JNEUROSCI.3477-17.2018 Filip, M., & Cunningham, K. A. (2003). Hyperlocomotive and discriminative stimulus effects of cocaine are under the control of serotonin 2C (5-HT2C) receptors in rat prefrontal cortex. Journal of Pharmacology and Experimental Therapeutics, 306(2), 734–743. Fineberg, N. A., Potenza, M. N., Chamberlain, S. R., Berlin, H. A., Menzies, L., Bechara, A., …, Hollander, E. (2010). Probing compulsive and impulsive behaviors, from animal models to endophenotypes: A narrative review. Neuropsychopharmacology, 35(3), 591–604. https://doi.org/10.1038/npp.2009.185 Fletcher, P. J., Grottick, A. J., & Higgins, G. A. (2002). Differential effects of the 5-HT(2A) receptor antagonist M100907 and the 5-HT(2C) receptor antagonist SB242084 on cocaine-induced locomotor activity, cocaine self-administration and cocaine-induced reinstatement of responding. Neuropsychopharmacology, 27(4), 576–586. https://doi.org/10.1016/S0893-133X(02)00342-1 Floresco, S. B., Montes, D. R., Tse, M. M. T., & Holstein, M. van. (2018). Differential contributions of nucleus accumbens subregions to cue-guided risk/reward decision making and implementation of conditional rules. Journal of Neuroscience, 38(8), 1901–1914. https://doi.org/10.1523/JNEUROSCI.3191-17.2018 Ford, C. P. (2014). The role of D2-autoreceptors in regulating dopamine neuron activity and transmission. Neuroscience, 282, 13–22. https://doi.org/10.1016/j.neuroscience.2014.01.025 Fuchs, R. A., Evans, K. A., Parker, M. P., & See, R. E. (2004). Differential involvement of orbitofrontal cortex subregions in conditioned cue-induced and cocaine-primed reinstatement of cocaine seeking in rats. The Journal of Neuroscience, 24(29), 6600–6610. https://doi.org/10.1523/JNEUROSCI.1924-04.2004 Gardner, M. P. H., Sanchez, D., Conroy, J. C., Wikenheiser, A. M., Zhou, J., & Schoenbaum, G. (2020). Processing in lateral orbitofrontal cortex is required to estimate subjective preference 186 during initial, but not established, economic choice. Neuron, 108(3), 526-537.e4. https://doi.org/10.1016/j.neuron.2020.08.010 Garner, J. P., Thogerson, C. M., Würbel, H., Murray, J. D., & Mench, J. A. (2006). Animal neuropsychology: Validation of the Intra-Dimensional Extra-Dimensional set shifting task for mice. Behavioural Brain Research, 173(1), 53–61. https://doi.org/10.1016/j.bbr.2006.06.002 Garr, E., & Delamater, A. R. (2019). Exploring the relationship between actions, habits, and automaticity in an action sequence task. Learning & Memory, 26(4), 128–132. https://doi.org/10.1101/lm.048645.118 Gelman, A., Carlin, J., Stern, H., Dunson, D., Vehtari, A., & Rubin, D. (2013). Bayesian data analysis. Boca Raton: Chapman and Hall/CRC. Genauck, A., Andrejevic, M., Brehm, K., Matthis, C., Heinz, A., Weinreich, A., Kathmann, N., & Romanczuk-Seiferth, N. (2020). Cue-induced effects on decision-making distinguish subjects with gambling disorder from healthy controls. Addiction Biology, 25(6), e12841. https://doi.org/10.1111/adb.12841 Genauck, A., Matthis, C., Andrejevic, M., Ballon, L., Chiarello, F., Duecker, K., Heinz, A., Kathmann, N., & Romanczuk-Seiferth, N. (2021). Neural correlates of cue-induced changes in decision-making distinguish subjects with gambling disorder from healthy controls. Addiction Biology, 26(3), e12951. https://doi.org/10.1111/adb.12951 Gershman, S. J., Markman, A. B., & Otto, A. R. (2014). Retrospective revaluation in sequential decision making: A tale of two systems. Journal of Experimental Psychology. General, 143(1), 182–194. https://doi.org/10.1037/a0030844 Ghods-Sharifi, S., Haluk, D. M., & Floresco, S. B. (2008). Differential effects of inactivation of the orbitofrontal cortex on strategy set-shifting and reversal learning. Neurobiology of Learning and Memory, 89(4), 567–573. https://doi.org/10.1016/j.nlm.2007.10.007 Gillan, C. M., Apergis-Schoute, A. M., Morein-Zamir, S., Urcelay, G. P., Sule, A., Fineberg, N. A., Sahakian, B. J., & Robbins, T. W. (2015). Functional neuroimaging of avoidance habits in obsessive-compulsive disorder. The American Journal of Psychiatry, 172(3), 284–293. https://doi.org/10.1176/appi.ajp.2014.14040525 Gillan, C. M., Kosinski, M., Whelan, R., Phelps, E. A., & Daw, N. D. (2016). Characterizing a psychiatric symptom dimension related to deficits in goal-directed control. ELife, 5, e11305. https://doi.org/10.7554/eLife.11305 Gillan, C. M., Papmeyer, M., Morein-Zamir, S., Sahakian, B. J., Fineberg, N. A., Robbins, T. W., & de Wit, S. (2011). Disruption in the balance between goal-directed behavior and habit learning in obsessive-compulsive disorder. The American Journal of Psychiatry, 168(7), 718–726. https://doi.org/10.1176/appi.ajp.2011.10071062 187 Gillan, C. M., Robbins, T. W., Sahakian, B. J., van den Heuvel, O. A., & van Wingen, G. (2016). The role of habit in compulsivity. European Neuropsychopharmacology, 26(5), 828–840. https://doi.org/10.1016/j.euroneuro.2015.12.033 González-Ortega, I., Echeburúa, E., de Corral, P., & Polo-López, R. (2019). Pathological gambling: Clinical gender differences. In M. Sáenz-Herrero (Ed.), Psychopathology in women: Incorporating gender perspective into descriptive psychopathology (pp. 825–838). Springer International Publishing. https://doi.org/10.1007/978-3-030-15179-9_35 Goudriaan, A. E., de Ruiter, M. B., van den Brink, W., Oosterlaan, J., & Veltman, D. J. (2010). Brain activation patterns associated with cue reactivity and craving in abstinent problem gamblers, heavy smokers and healthy controls: An fMRI study. Addiction Biology, 15(4), 491–503. https://doi.org/10.1111/j.1369-1600.2010.00242.x Goudriaan, A. E., Oosterlaan, J., de Beurs, E., & van den Brink, W. (2005). Decision making in pathological gambling: A comparison between pathological gamblers, alcohol dependents, persons with Tourette syndrome, and normal controls. Brain Research. Cognitive Brain Research, 23(1), 137–151. https://doi.org/10.1016/j.cogbrainres.2005.01.017 Goudriaan, A. E., Oosterlaan, J., de Beurs, E., & van den Brink, W. (2006). Neurocognitive functions in pathological gambling: A comparison with alcohol dependence, Tourette syndrome and normal controls. Addiction, 101(4), 534–547. https://doi.org/10.1111/j.1360-0443.2006.01380.x Grant, D. A., & Berg, E. (1948). A behavioral analysis of degree of reinforcement and ease of shifting to new responses in a Weigl-type card-sorting problem. Journal of Experimental Psychology, 38, 404–411. https://doi.org/10.1037/h0059831 Grant, J. E., Chamberlain, S. R., Odlaug, B. L., Potenza, M. N., & Kim, S. W. (2010). Memantine shows promise in reducing gambling severity and cognitive inflexibility in pathological gambling: A pilot study. Psychopharmacology, 212(4), 603–612. https://doi.org/10.1007/s00213-010-1994-5 Gremel, C. M., & Costa, R. M. (2013). Orbitofrontal and striatal circuits dynamically encode the shift between goal-directed and habitual actions. Nature Communications, 4(1), Article 1. https://doi.org/10.1038/ncomms3264 Griffiths, M. D. (1993). Fruit machine gambling: The importance of structural characteristics. Journal of Gambling Studies, 9, 101-120. doi:10.1007/BF01014863 Groenewegen, H. J., Berendse, H. W., Wolters, J. G., & Lohman, A. H. (1990). The anatomical relationship of the prefrontal cortex with the striatopallidal system, the thalamus and the amygdala: Evidence for a parallel organization. Progress in Brain Research, 85, 95–116; discussion 116-118. https://doi.org/10.1016/s0079-6123(08)62677-1 188 Haar, C. V. (2020). Challenges and opportunities in animal models of gambling-like behavior. Current Opinion in Behavioral Sciences, 31, 42-47. doi:10.1016/j.cobeha.2019.10.013 Haber, S. N., Fudge, J. L., & McFarland, N. R. (2000). Striatonigrostriatal pathways in primates form an ascending spiral from the shell to the dorsolateral striatum. The Journal of Neuroscience, 20(6), 2369–2382. https://doi.org/10.1523/JNEUROSCI.20-06-02369.2000 Hadj-Bouziane, F., Benatru, I., Brovelli, A., Klinger, H., Thobois, S., Broussolle, E., Boussaoud, D., & Meunier, M. (2012). Advanced Parkinson’s disease effect on goal-directed and habitual processes involved in visuomotor associative learning. Frontiers in Human Neuroscience, 6, 351. https://doi.org/10.3389/fnhum.2012.00351 Haines, K. M., & Czachowski, C. L. (2022). Evaluating habit formation across pairs of female and male selectively bred alcohol-preferring and non-preferring rats. Alcohol, 102, 11–22. https://doi.org/10.1016/j.alcohol.2022.04.003 Hankosky, E. R., Westbrook, S. R., Haake, R. M., Willing, J., Raetzman, L. T., Juraska, J. M., & Gulley, J. M. (2018). Age- and sex-dependent effects of methamphetamine on cognitive flexibility and 5-HT2C receptor localization in the orbitofrontal cortex of Sprague-Dawley rats. Behavioural Brain Research, 349, 16–24. https://doi.org/10.1016/j.bbr.2018.04.047 Hatakama, H., Asaoka, N., Nagayasu, K., Shirakawa, H., & Kaneko, S. (2022). A selective serotonin reuptake inhibitor ameliorates obsessive–compulsive disorder-like perseverative behavior by attenuating 5-HT2C receptor signaling in the orbitofrontal cortex. Neuropharmacology, 206, 108926. https://doi.org/10.1016/j.neuropharm.2021.108926 Hearst, E., & Jenkins, H. M. (1974). Sign-tracking: The stimulus-reinforcer relation and directed action. Psychonomic Society. Hernandez, J. S., Binette, A. N., Rahman, T., Tarantino, J. D., & Moorman, D. E. (2020). Chemogenetic inactivation of orbitofrontal cortex decreases cue-induced reinstatement of ethanol and sucrose seeking in male and female Wistar rats. Alcoholism: Clinical and Experimental Research, 44(9), 1769–1782. https://doi.org/10.1111/acer.14407 Hervig, M., Piilgaard Petersen, L., Božič, T., Alsiö, J., & Robbins, T. (2020). Glutamatergic and serotonergic modulation of rat medial and lateral orbitofrontal cortex in visual serial reversal learning. Psychology and Neuroscience, 13, 438-458. https://doi.org/10.1037/pne0000221 Hetherington, M. M., & Rolls, B. J. (1996). Sensory-specific satiety: Theoretical frameworks and central characteristics. In Why we eat what we eat: The psychology of eating (pp. 267–290). American Psychological Association. https://doi.org/10.1037/10291-010 Higgins, G. A., Ouagazzal, A. M., & Grottick, A. J. (2001). Influence of the 5‐HT2C receptor antagonist SB242,084 on behaviour produced by the 5‐HT2 agonist Ro60‐0175 and the 189 indirect 5‐HT agonist dexfenfluramine. British Journal of Pharmacology, 133(4), 459–466. https://doi.org/10.1038/sj.bjp.0704082 Hilario, M., Holloway, T., Jin, X., & Costa, R. M. (2012). Different dorsal striatum circuits mediate action discrimination and action generalization. The European Journal of Neuroscience, 35(7), 1105–1114. https://doi.org/10.1111/j.1460-9568.2012.08073.x Hogarth, L. (2020). Addiction is driven by excessive goal-directed drug choice under negative affect: Translational critique of habit and compulsion theory. Neuropsychopharmacology, 45(5), Article 5. https://doi.org/10.1038/s41386-020-0600-8 Hogarth, L., Balleine, B. W., Corbit, L. H., & Killcross, S. (2013). Associative learning mechanisms underpinning the transition from recreational drug use to addiction. Annals of the New York Academy of Sciences, 1282, 12–24. https://doi.org/10.1111/j.1749-6632.2012.06768.x Holland, P. C., & Straub, J. J. (1979). Differential effects of two ways of devaluing the unconditioned stimulus after Pavlovian appetitive conditioning. Journal of Experimental Psychology. Animal Behavior Processes, 5(1), 65–78. https://doi.org/10.1037//0097-7403.5.1.65 Hollon, N. G., Williams, E. W., Howard, C. D., Li, H., Traut, T. I., & Jin, X. (2021). Nigrostriatal dopamine signals sequence-specific action-outcome prediction errors. Current Biology, 31(23), 5350-5363.e5. https://doi.org/10.1016/j.cub.2021.09.040 Howell, L. L., & Cunningham, K. A. (2015). Serotonin 5-HT2 receptor interactions with dopamine function: Implications for therapeutics in cocaine use disorder. Pharmacological Reviews, 67(1), 176–197. https://doi.org/10.1124/pr.114.009514 Hynes, T. J., Ferland, J.-M. M., Feng, T. L., Adams, W. K., Silveira, M. M., Tremblay, M., …, Winstanley, C. A. (2020). Chemogenetic inhibition of dopaminergic projections to the nucleus accumbens has sexually dimorphic effects in the rat gambling task. Behavioral Neuroscience, 134(4), 309–322. https://doi.org/10.1037/bne0000372 Hynes, T. J., Hrelja, K. M., Hathaway, B. A., Hounjet, C. D., Chernoff, C. S., Ebsary, S. A., …, & Winstanley, C. A. (2021). Dopamine neurons gate the intersection of cocaine use, decision making, and impulsivity. Addiction Biology, 26(6), e13022. https://doi.org/10.1111/adb.13022 Ilango, A., Kesner, A. J., Keller, K. L., Stuber, G. D., Bonci, A., & Ikemoto, S. (2014). Similar roles of substantia nigra and ventral tegmental dopamine neurons in reward and aversion. The Journal of Neuroscience, 34(3), 817–822. https://doi.org/10.1523/JNEUROSCI.1703-13.2014 190 Izquierdo, A. (2017). Functional heterogeneity within rat orbitofrontal cortex in reward learning and decision making. The Journal of Neuroscience, 37, 10529-10540. doi:10.1523/JNEUROSCI.1678-17.2017 Izquierdo, A., Brigman, J. L., Radke, A. K., Rudebeck, P. H., & Holmes, A. (2017). The neural basis of reversal learning: An updated perspective. Neuroscience, 345, 12–26. https://doi.org/10.1016/j.neuroscience.2016.03.021 Izquierdo, A., Suda, R. K., & Murray, E. A. (2004). Bilateral orbital prefrontal cortex lesions in rhesus monkeys disrupt choices guided by both reward value and reward contingency. The Journal of Neuroscience, 24(34), 7540–7548. https://doi.org/10.1523/JNEUROSCI.1921-04.2004 Jakob, A. M. V., Mikhael, J. G., Hamilos, A. E., Assad, J. A., & Gershman, S. J. (2021). Dopamine mediates the bidirectional update of interval timing. bioRxiv, 2021.11.02.466803. doi:10.1101/2021.11.02.466803 Jara-Rizzo, M. F., Navas, J. F., Rodas, J. A., & Perales, J. C. (2020). Decision-making inflexibility in a reversal learning task is associated with severity of problem gambling symptoms but not with a diagnosis of substance use disorder. BMC Psychology, 8(1), 120. https://doi.org/10.1186/s40359-020-00482-6 Jasinska, A. J., Stein, E. A., Kaiser, J., Naumer, M. J., & Yalachkov, Y. (2014). Factors modulating neural reactivity to drug cues in addiction: A survey of human neuroimaging studies. Neuroscience and Biobehavioral Reviews, 38, 1–16. https://doi.org/10.1016/j.neubiorev.2013.10.013 Jensen, C., Dixon, M. J., Harrigan, K. A., Sheepy, E., Fugelsang, J. A., & Jarick, M. (2013). Misinterpreting 'winning' in multiline slot machine games. International Gambling Studies, 13, 112-126. doi:10.1080/14459795.2012.717635 Jin, X., & Costa, R. M. (2010). Start/stop signals emerge in nigrostriatal circuits during sequence learning. Nature, 466(7305), 457–462. https://doi.org/10.1038/nature09263 Kanoski, S. E., Meisel, R. L., Mullins, A. J., & Davidson, T. L. (2007). The effects of energy-rich diets on discrimination reversal learning and on BDNF in the hippocampus and prefrontal cortex of the rat. Behavioural Brain Research, 182(1), 57–66. https://doi.org/10.1016/j.bbr.2007.05.004 Kendig, M. D., Cheung, A. M. K., Raymond, J. S., & Corbit, L. H. (2016). Contexts paired with junk food impair goal-directed behavior in rats: Implications for decision making in obesogenic environments. Frontiers in Behavioral Neuroscience, 10, 216. https://doi.org/10.3389/fnbeh.2016.00216 191 Keramati, M., & Gutkin, B. (2013). Imbalanced decision hierarchy in addicts emerging from drug-hijacked dopamine spiraling circuit. PLOS ONE, 8(4), e61489. https://doi.org/10.1371/journal.pone.0061489 Kim, H. F., Ghazizadeh, A., & Hikosaka, O. (2015). Dopamine neurons encoding long-term memory of object value for habitual behavior. Cell, 163(5), 1165–1175. https://doi.org/10.1016/j.cell.2015.10.063 Klanker, M., Sandberg, T., Joosten, R., Willuhn, I., Feenstra, M., & Denys, D. (2015). Phasic dopamine release induced by positive feedback predicts individual differences in reversal learning. Neurobiology of Learning and Memory, 125, 135–145. https://doi.org/10.1016/j.nlm.2015.08.011 Knutson, B., Adams, C. M., Fong, G. W., & Hommer, D. (2001). Anticipation of increasing monetary reward selectively recruits nucleus accumbens. The Journal of Neuroscience, 21, 159-RC159. doi:10.1523/JNEUROSCI.21-16-j0002.2001 Kobayakawa, M., Koyama, S., Mimura, M., & Kawamura, M. (2008). Decision making in Parkinson’s disease: Analysis of behavioral and physiological patterns in the Iowa gambling task. Movement Disorders, 23(4), 547–552. https://doi.org/10.1002/mds.21865 Kosheleff, A. R., Araki, J., Tsan, L., Chen, G., Murphy, N. P., Maidment, N. T., & Ostlund, S. B. (2018). Junk food exposure disrupts selection of food-seeking actions in rats. Frontiers in Psychiatry, 9. https://www.frontiersin.org/articles/10.3389/fpsyt.2018.00350 Kruschke, J. (2014). Doing Bayesian data analysis. Saint Louis: Elsevier Science & Technology. Küppers, E., Krust, A., Chambon, P., & Beyer, C. (2008). Functional alterations of the nigrostriatal dopamine system in estrogen receptor-alpha knockout (ERKO) mice. Psychoneuroendocrinology, 33(6), 832–838. https://doi.org/10.1016/j.psyneuen.2008.03.007 Kuwabara, M., Kang, N., Holy, T. E., & Padoa-Schioppa, C. (2020). Neural mechanisms of economic choices in mice. ELife, 9, e49669. https://doi.org/10.7554/eLife.49669 Ladouceur, R., Dubé, D., & Bujold, A. (1994). Prevalence of pathological gambling and related problems among college students in the Quebec metropolitan area. The Canadian Journal of Psychiatry, 39(5), 289–293. https://doi.org/10.1177/070674379403900509 Langdon, A. J., Hathaway, B. A., Zorowitz, S., Harris, C. B. W., & Winstanley, C. A. (2019). Relative insensitivity to time-out punishments induced by win-paired cues in a rat gambling task. Psychopharmacology, 236, 2543-2556. doi:10.1007/s00213-019-05308-x Lawrence, N. S., Jollant, F., O’Daly, O., Zelaya, F., & Phillips, M. L. (2009). Distinct roles of prefrontal cortical subregions in the Iowa gambling task. Cerebral Cortex, 19(5), 1134–1143. https://doi.org/10.1093/cercor/bhn154 192 LeBlanc, K. H., Maidment, N. T., & Ostlund, S. B. (2013). Repeated cocaine exposure facilitates the expression of incentive motivation and induces habitual control in rats. PLOS ONE, 8(4), e61355. https://doi.org/10.1371/journal.pone.0061355 Lee, T. M. C., Chan, C. C. H., Leung, A. W. S., Fox, P. T., & Gao, J.-H. (2009). Sex-related differences in neural activity during risk taking: An fMRI study. Cerebral Cortex, 19(6), 1303–1312. https://doi.org/10.1093/cercor/bhn172 Lee, Y. H., Cha, J., Chung, S. J., Yoo, H. S., Sohn, Y. H., Ye, B. S., & Lee, P. H. (2019). Beneficial effect of estrogen on nigrostriatal dopaminergic neurons in drug-naïve postmenopausal Parkinson’s disease. Scientific Reports, 9(1), Article 1. https://doi.org/10.1038/s41598-019-47026-6 Lehéricy, S., Ducros, M., Van de Moortele, P.-F., Francois, C., Thivard, L., Poupon, C., …, Kim, D.-S. (2004). Diffusion tensor fiber tracking shows distinct corticostriatal circuits in humans. Annals of Neurology, 55(4), 522–529. https://doi.org/10.1002/ana.20030 Lerner, T. N., Shilyansky, C., Davidson, T. J., Evans, K. E., Beier, K. T., Zalocusky, K. A., …, Deisseroth, K. (2015). Intact-brain analyses reveal distinct information carried by SNc dopamine subcircuits. Cell, 162(3), 635–647. https://doi.org/10.1016/j.cell.2015.07.014 Lichtenberg, N. T., Pennington, Z. T., Holley, S. M., Greenfield, V. Y., Cepeda, C., Levine, M. S., & Wassum, K. M. (2017). Basolateral amygdala to orbitofrontal cortex projections enable cue-triggered reward expectations. Journal of Neuroscience, 37(35), 8374–8384. Limbrick-Oldfield, E. H., Mick, I., Cocks, R. E., McGonigle, J., Sharman, S. P., Goldstone, A. P., . . . Clark, L. (2017). Neural substrates of cue reactivity and craving in gambling disorder. Translational Psychiatry, 7, e992. doi:10.1038/tp.2016.256 Liu, S., Bubar, M. J., Lanfranco, M. F., Hillman, G. R., & Cunningham, K. A. (2007). Serotonin2C receptor localization in GABA neurons of the rat medial prefrontal cortex: Implications for understanding the neurobiology of addiction. Neuroscience, 146(4), 1677–1688. https://doi.org/10.1016/j.neuroscience.2007.02.064 Loba, P., Stewart, S. H., Klein, R. M., & Blackburn, J. R. (2001). Manipulations of the features of standard video lottery terminal (VLT) games: Effects in pathological and non-pathological gamblers. Journal of Gambling Studies, 17(4), 297–320. https://doi.org/10.1023/a:1013639729908 López‐Giménez, J. F., Mengod, G., Palacios, J. M., & Vilaró, M. T. (2001). Regional distribution and cellular localization of 5-HT2C receptor mRNA in monkey brain: Comparison with [3H]mesulergine binding sites and choline acetyltransferase mRNA. Synapse, 42(1), 12–26. https://doi.org/10.1002/syn.1095 193 Luigjes, J., Lorenzetti, V., de Haan, S., Youssef, G. J., Murawski, C., Sjoerds, Z., …, Yücel, M. (2019). Defining compulsive behavior. Neuropsychology Review, 29(1), 4–13. https://doi.org/10.1007/s11065-019-09404-9 Malvaez, M., Shieh, C., Murphy, M. D., Greenfield, V. Y., & Wassum, K. M. (2019). Distinct cortical-amygdala projections drive reward value encoding and retrieval. Nature Neuroscience, 22(5), 762–769. https://doi.org/10.1038/s41593-019-0374-7 Mar, A. C., Walker, A. L. J., Theobald, D. E., Eagle, D. M., & Robbins, T. W. (2011). Dissociable effects of lesions to orbitofrontal cortex subregions on impulsive choice in the rat. The Journal of Neuroscience, 31(17), 6398–6404. https://doi.org/10.1523/JNEUROSCI.6620-10.2011 Marazziti, D., Catena Dell’Osso, M., Conversano, C., Consoli, G., Vivarelli, L., Mungai, F., Di Nasso, E., & Golia, F. (2008). Executive function abnormalities in pathological gamblers. Clinical Practice and Epidemiology in Mental Health, 4, 7. https://doi.org/10.1186/1745-0179-4-7 Marquardt, K., Josey, M., Kenton, J. A., Cavanagh, J. F., Holmes, A., & Brigman, J. L. (2019). Impaired cognitive flexibility following NMDAR-GluN2B deletion is associated with altered orbitofrontal-striatal function. Neuroscience, 404, 338–352. https://doi.org/10.1016/j.neuroscience.2019.01.066 Marshall, A. T., & Kirkpatrick, K. (2017). Reinforcement learning models of risky choice and the promotion of risk-taking by losses disguised as wins in rats. Journal of Experimental Psychology. Animal Behavior Processes, 43, 262-279. doi:10.1037/xan0000141 Marshall, A. T., Munson, C. N., Maidment, N. T., & Ostlund, S. B. (2020). Reward-predictive cues elicit excessive reward seeking in adolescent rats. Developmental Cognitive Neuroscience, 45, 100838. https://doi.org/10.1016/j.dcn.2020.100838 Marzinzik, F., Wotka, J., Wahl, M., Krugel, L. K., Kordsachia, C., & Klostermann, F. (2011). Modulation of habit formation by levodopa in Parkinson’s disease. PloS One, 6(11), e27695. https://doi.org/10.1371/journal.pone.0027695 Matias, S., Lottem, E., Dugué, G. P., & Mainen, Z. F. (2017). Activity patterns of serotonin neurons underlying cognitive flexibility. ELife, 6, e20552. https://doi.org/10.7554/eLife.20552 McDannald, M. A., Lucantonio, F., Burke, K. A., Niv, Y., & Schoenbaum, G. (2011). Ventral striatum and orbitofrontal cortex are both required for model-based, but not model-free, reinforcement learning. The Journal of Neuroscience, 31(7), 2700–2705. https://doi.org/10.1523/JNEUROSCI.5499-10.2011 Meunier, D., Ersche, K. D., Craig, K. J., Fornito, A., Merlo-Pich, E., Fineberg, N. A., …, Bullmore, E. T. (2012). Brain functional connectivity in stimulant drug dependence and 194 obsessive-compulsive disorder. NeuroImage, 59(2), 1461–1468. https://doi.org/10.1016/j.neuroimage.2011.08.003 Mi, T.-M., Zhang, W., McKeown, M. J., & Chan, P. (2021). Impaired formation and expression of goal-directed and habitual control in Parkinson’s disease. Frontiers in Aging Neuroscience, 13. https://doi.org/10.3389/fnagi.2021.734807 Middleman, A. B., Faulkner, A. H., Woods, E. R., Emans, S. J., & DuRant, R. H. (1995). High-risk behaviors among high school students in Massachusetts who use anabolic steroids. Pediatrics, 96(2), 268–272. Midgley, S. J., Heather, N., Best, D., Henderson, D., McCarthy, S., & Davies, J. B. (2000). Risk behaviours for HIV and hepatitis infection among anabolic-androgenic steroid users. AIDS Care, 12(2), 163–170. https://doi.org/10.1080/09540120050001832 Mimura, M., Oeda, R., & Kawamura, M. (2006). Impaired decision-making in Parkinson’s disease. Parkinsonism & Related Disorders, 12(3), 169–175. https://doi.org/10.1016/j.parkreldis.2005.12.003 Mitchell, S. H. (2019). Linking delay discounting and substance use disorders: Genotypes and phenotypes. Perspectives on Behavior Science, 42(3), 419–432. https://doi.org/10.1007/s40614-019-00218-x Monosov, I. E. (2020). How outcome uncertainty mediates attention, learning, and decision-making. Trends in Neurosciences, 43(10), 795–809. https://doi.org/10.1016/j.tins.2020.06.009 Moreno, M., Economidou, D., Mar, A., López-Granero, C., Caprioli, D., Theobald, D., …, Dalley, J. (2013). Divergent effects of D2/3 receptor activation in the nucleus accumbens core and shell on impulsivity and locomotor activity in high and low impulsive rats. Psychopharmacology, 228(1), 19–30. https://doi.org/10.1007/s00213-013-3010-3 Murch, W. S., Chu, S. W. M., & Clark, L. (2017). Measuring the slot machine zone with attentional dual tasks and respiratory sinus arrhythmia. Psychology of Addictive Behaviors, 31, 375-384. doi:10.1037/adb0000251 Murphy, E. R., Dalley, J. W., & Robbins, T. W. (2005). Local glutamate receptor antagonism in the rat prefrontal cortex disrupts response inhibition in a visuospatial attentional task. Psychopharmacology, 179(1), 99–107. https://doi.org/10.1007/s00213-004-2068-3 Namba, M. D., Tomek, S. E., Olive, M. F., Beckmann, J. S., & Gipson, C. D. (2018). The winding road to relapse: Forging a new understanding of cue-induced reinstatement models and their associated neural mechanisms. Frontiers in Behavioral Neuroscience, 12, 17. https://doi.org/10.3389/fnbeh.2018.00017 195 Nasser, H. M., Chen, Y.-W., Fiscella, K., & Calu, D. J. (2015). Individual variability in behavioral flexibility predicts sign-tracking tendency. Frontiers in Behavioral Neuroscience, 9, 289. https://doi.org/10.3389/fnbeh.2015.00289 Nelson, A., & Killcross, S. (2006). Amphetamine exposure enhances habit formation. Journal of Neuroscience, 26(14), 3805–3812. https://doi.org/10.1523/JNEUROSCI.4305-05.2006 Ng, C.-W., Noblejas, M. I., Rodefer, J. S., Smith, C. B., & Poremba, A. (2007). Double dissociation of attentional resources: Prefrontal versus cingulate cortices. The Journal of Neuroscience, 27(45), 12123–12131. https://doi.org/10.1523/JNEUROSCI.2745-07.2007 Nimitvilai, S., Lopez, M. F., & Woodward, J. J. (2018). Effects of monoamines on the intrinsic excitability of lateral orbitofrontal cortex neurons in alcohol-dependent and non-dependent female mice. Neuropharmacology, 137, 1–12. https://doi.org/10.1016/j.neuropharm.2018.04.019 Niv, Y. (2019). Learning task-state representations. Nature Neuroscience, 22, 1544-1553. https://doi.org/10.1038/s41593-019-0470-8 Nonkes, L. J. P., Tomson, K., Mærtin, A., Dederen, J., Roald Maes, J. H., & Homberg, J. (2010). Orbitofrontal cortex and amygdalar over-activity is associated with an inability to use the value of expected outcomes to guide behaviour in serotonin transporter knockout rats. Neurobiology of Learning and Memory, 94(1), 65–72. https://doi.org/10.1016/j.nlm.2010.04.002 Noonan, M. P., Walton, M. E., Behrens, T. E. J., Sallet, J., Buckley, M. J., & Rushworth, M. F. S. (2010). Separate value comparison and learning mechanisms in macaque medial and lateral orbitofrontal cortex. Proceedings of the National Academy of Sciences, 107(47), 20547–20552. https://doi.org/10.1073/pnas.1012246107 O’Neill, M., & Brown, V. J. (2007). The effect of striatal dopamine depletion and the adenosine A2A antagonist KW-6002 on reversal learning in rats. Neurobiology of Learning and Memory, 88(1), 75–81. https://doi.org/10.1016/j.nlm.2007.03.003 Olmstead, M. C., Lafond, M. V., Everitt, B. J., & Dickinson, A. (2001). Cocaine seeking by rats is a goal-directed action. Behavioral Neuroscience, 115(2), 394–402. Otto, A. R., Gershman, S. J., Markman, A. B., & Daw, N. D. (2013). The curse of planning: Dissecting multiple reinforcement-learning systems by taxing the central executive. Psychological Science, 24(5), 751–761. https://doi.org/10.1177/0956797612463080 Owen, A. M., Roberts, A. C., Hodges, J. R., Summers, B. A., Polkey, C. E., & Robbins, T. W. (1993). Contrasting mechanisms of impaired attentional set-shifting in patients with frontal lobe damage or Parkinson’s disease. Brain: A Journal of Neurology, 116, 1159–1175. https://doi.org/10.1093/brain/116.5.1159 196 Panayi, M. C., & Killcross, S. (2018). Functional heterogeneity within the rodent lateral orbitofrontal cortex dissociates outcome devaluation and reversal learning deficits. ELife, 7, e37357. https://doi.org/10.7554/eLife.37357 Pattij, T., Janssen, M. C. W., Vanderschuren, L. J. M. J., Schoffelmeer, A. N. M., & van Gaalen, M. M. (2007). Involvement of dopamine D1 and D2 receptors in the nucleus accumbens core and shell in inhibitory response control. Psychopharmacology, 191(3), 587–598. https://doi.org/10.1007/s00213-006-0533-x Pavlov, I. (1927). Conditioned reflexes: An investigation of the physiological activity of the cerebral cortex. Annals of Neurosciences, 17(3), 136–141. https://doi.org/10.5214/ans.0972-7531.1017309 Peart, D. R., Andrade, A. K., Logan, C. N., Knackstedt, L. A., & Murray, J. E. (2022). Regulation of cocaine-related behaviours by estrogen and progesterone. Neuroscience & Biobehavioral Reviews, 135, 104584. https://doi.org/10.1016/j.neubiorev.2022.104584 Pelloux, Y., Everitt, B. J., & Dickinson, A. (2007). Compulsive drug seeking by rats under punishment: Effects of drug taking history. Psychopharmacology, 194(1), 127–137. https://doi.org/10.1007/s00213-007-0805-0 Pentkowski, N. S., Duke, F. D., Weber, S. M., Pockros, L. A., Teer, A. P., Hamilton, E. C., Thiel, K. J., & Neisewander, J. L. (2010). Stimulation of medial prefrontal cortex serotonin 2C (5-HT) receptors attenuates cocaine-seeking behavior. Neuropsychopharmacology, 35(10), 2037–2048. https://doi.org/10.1038/npp.2010.72 Peterson, A. C., Zhang, S., Hu, S., Chao, H. H., & Li, C.-S. R. (2017). The effects of age, from young to middle adulthood, and gender on resting state functional connectivity of the dopaminergic midbrain. Frontiers in Human Neuroscience, Journal Article. https://doi.org/10.3389/fnhum.2017.00052 Pitchers, K. K., Phillips, K. B., Jones, J. L., Robinson, T. E., & Sarter, M. (2017). Diverse roads to relapse: A discriminative cue signaling cocaine availability is more effective in renewing cocaine seeking in goal trackers than sign trackers and depends on basal forebrain cholinergic activity. The Journal of Neuroscience, 37(30), 7198–7208. https://doi.org/10.1523/JNEUROSCI.0990-17.2017 Pitchers, K. K., Sarter, M., & Robinson, T. E. (2018). The hot “n” cold of cue-induced drug relapse. Learning & Memory, 25(9), 474–480. https://doi.org/10.1101/lm.046995.117 Pompeiano, M., Palacios, J. M., & Mengod, G. (1994). Distribution of the serotonin 5-HT2 receptor family mRNAs: Comparison between 5-HT2A and 5-HT2C receptors. Brain Research. Molecular Brain Research, 23(1–2), 163–178. https://doi.org/10.1016/0169-328x(94)90223-2 197 Price, A. E., Sholler, D. J., Stutz, S. J., Anastasio, N. C., & Cunningham, K. A. (2019). Endogenous serotonin 5-HT2A and 5-HT2C receptors associate in the medial prefrontal cortex. ACS Chemical Neuroscience, 10(7), 3241–3248. https://doi.org/10.1021/acschemneuro.8b00669 Quinn, J. J., Hitchcott, P. K., Umeda, E. A., Arnold, A. P., & Taylor, J. R. (2007). Sex chromosome complement regulates habit formation. Nature Neuroscience, 10(11), Article 11. https://doi.org/10.1038/nn1994 Ramayya, A. G., Misra, A., Baltuch, G. H., & Kahana, M. J. (2014). Microstimulation of the human substantia nigra alters reinforcement learning. The Journal of Neuroscience, 34(20), 6887–6895. https://doi.org/10.1523/JNEUROSCI.5445-13.2014 Renteria, R., Baltz, E. T., & Gremel, C. M. (2018). Chronic alcohol exposure disrupts top-down control over basal ganglia action selection to produce habits. Nature Communications, 9(1), Article 1. https://doi.org/10.1038/s41467-017-02615-9 Rescorla, R. A. (1970). Reduction in the effectiveness of reinforcement after prior excitatory conditioning. Learning and Motivation, 1(4), 372–381. https://doi.org/10.1016/0023-9690(70)90101-3 Roane, H. S. (2008). On the applied use of progressive-ratio schedules of reinforcement. Journal of Applied Behavior Analysis, 41(2), 155–161. https://doi.org/10.1901/jaba.2008.41-155 Robbins, T. W. (1978). The acquisition of responding with conditioned reinforcement: Effects of pipradrol, methylphenidate, d-amphetamine, and nomifensine. Psychopharmacology, 58(1), 79–87. https://doi.org/10.1007/BF00426794 Robinson, E. S., Dalley, J. W., Theobald, D. E., Glennon, J. C., Pezze, M. A., Murphy, E. R., & Robbins, T. W. (2008). Opposing roles for 5-HT2A and 5-HT2C receptors in the nucleus accumbens on inhibitory response control in the 5-choice serial reaction time task. Neuropsychopharmacology, 33(10), 2398–2406. Rossi, M. A., Sukharnikova, T., Hayrapetyan, V. Y., Yang, L., & Yin, H. H. (2013). Operant self-stimulation of dopamine neurons in the substantia nigra. PloS One, 8(6), e65799. https://doi.org/10.1371/journal.pone.0065799 Rudebeck, P. H., & Murray, E. A. (2014). The orbitofrontal oracle: Cortical mechanisms for the prediction and evaluation of specific behavioral outcomes. Neuron, 84(6), 1143–1156. https://doi.org/10.1016/j.neuron.2014.10.049 Rugle, L., & Melamed, L. (1993). Neuropsychological assessment of attention problems in pathological gamblers. The Journal of Nervous and Mental Disease, 181(2), 107–112. https://doi.org/10.1097/00005053-199302000-00006 198 Rushworth, M. F. S., Hadland, K. A., Gaffan, D., & Passingham, R. E. (2003). The effect of cingulate cortex lesions on task switching and working memory. Journal of Cognitive Neuroscience, 15(3), 338–353. https://doi.org/10.1162/089892903321593072 Russell, B., Kwok, C., Ansary, S., Chang, T., Hrelja, K. M., Betts, G. D., Winstanley, C. A., Bains, A. (2022). The effects of dopamine D2 and D3 receptors agonists on rodent decision-making. Poster presented at Annual Society for Neuroscience meeting, San Diego, CA. Saunders, B. T., & Robinson, T. E. (2010). A cocaine cue acts as an incentive stimulus in some, but not others: Implications for addiction. Biological Psychiatry, 67(8), 730–736. https://doi.org/10.1016/j.biopsych.2009.11.015 Saunders, B. T., O’Donnell, E. G., Aurbach, E. L., & Robinson, T. E. (2014). A cocaine context renews drug seeking preferentially in a subset of individuals. Neuropsychopharmacology, 39(12), 2816–2823. https://doi.org/10.1038/npp.2014.131 Saunders, B. T., Richard, J. M., Margolis, E. B., & Janak, P. H. (2018). Dopamine neurons create Pavlovian conditioned stimuli with circuit-defined motivational properties. Nature Neuroscience, 21(8), 1072–1083. https://doi.org/10.1038/s41593-018-0191-4 Schilman, E. A., Uylings, H. B. M., Graaf, Y. G., Joel, D., & Groenewegen, H. J. (2008). The orbital cortex in rats topographically projects to central parts of the caudate–putamen complex. Neuroscience Letters, 432(1), 40–45. https://doi.org/10.1016/j.neulet.2007.12.024 Schoenbaum, G., Chang, C.-Y., Lucantonio, F., & Takahashi, Y. K. (2016). Thinking outside the box: Orbitofrontal cortex, imagination, and how we can treat addiction. Neuropsychopharmacology, 41(13), 2966–2976. https://doi.org/10.1038/npp.2016.147 Schreiner, D. C., Renteria, R., & Gremel, C. M. (2020). Fractionating the all-or-nothing definition of goal-directed and habitual decision-making. Journal of Neuroscience Research, 98(6), 998–1006. https://doi.org/10.1002/jnr.24545 Schultz, K. N., von Esenwein, S. A., Hu, M., Bennett, A. L., Kennedy, R. T., Musatov, S., …, Becker, J. B. (2009). Viral vector-mediated overexpression of estrogen receptor-alpha in striatum enhances the estradiol-induced motor activity in female rats and estradiol-modulated GABA release. The Journal of Neuroscience, 29(6), 1897–1903. https://doi.org/10.1523/JNEUROSCI.4647-08.2009 Schultz, W., Dayan, P., & Montague, P. R. (1997). A neural substrate of prediction and reward. Science, 275(5306), 1593–1599. https://doi.org/10.1126/science.275.5306.1593 Schumacher, J. D., van Holstein, M., Bagrodia, V., Le Bouder, H. B., & Floresco, S. B. (2021). Dorsomedial striatal contributions to different forms of risk/reward decision making. Neurobiology of Learning and Memory, 178, 107369. https://doi.org/10.1016/j.nlm.2020.107369 199 Seabrook, L. T., Naef, L., Baimel, C., Judge, A. K., Kenney, T., Ellis, M., …, Borgland, S. L. (2022). Disinhibition of the orbitofrontal cortex biases decision-making in obesity. Nature Neuroscience, 1–15. https://doi.org/10.1038/s41593-022-01210-6 Sebold, M., Schad, D. J., Nebe, S., Garbusow, M., Jünger, E., Kroemer, N. B., …, Huys, Q. J. M. (2016). Don’t think, just feel the music: Individuals with strong Pavlovian-to-instrumental transfer effects rely less on model-based reinforcement learning. Journal of Cognitive Neuroscience, 28(7), 985–995. https://doi.org/10.1162/jocn_a_00945 Seymour, B., Daw, N. D., Roiser, J. P., Dayan, P., & Dolan, R. (2012). Serotonin selectively modulates reward value in human decision-making. The Journal of Neuroscience, 32(17), 5833–5842. https://doi.org/10.1523/JNEUROSCI.0053-12.2012 Sharpe, M. J., Batchelor, H. M., Mueller, L. E., Yun Chang, C., Maes, E. J. P., Niv, Y., & Schoenbaum, G. (2020). Dopamine transients do not act as model-free prediction errors during associative learning. Nature Communications, 11(1), 106. https://doi.org/10.1038/s41467-019-13953-1 Sharpe, M. J., Chang, C. Y., Liu, M. A., Batchelor, H. M., Mueller, L. E., Jones, J. L., Niv, Y., & Schoenbaum, G. (2017). Dopamine transients are sufficient and necessary for acquisition of model-based associations. Nature Neuroscience, 20(5), 735–742. https://doi.org/10.1038/nn.4538 Sias, A. C., Morse, A. K., Wang, S., Greenfield, V. Y., Goodpaster, C. M., Wrenn, T. M., …, Wassum, K. M. (2021). A bidirectional corticoamygdala circuit for the encoding and retrieval of detailed reward memories. ELife, 10, e68617. https://doi.org/10.7554/eLife.68617 Singer, B. F., Fadanelli, M., Kawa, A. B., & Robinson, T. E. (2018). Are cocaine-seeking “habits” necessary for the development of addiction-like behavior in rats? Journal of Neuroscience, 38(1), 60–73. https://doi.org/10.1523/JNEUROSCI.2458-17.2017 Sjoerds, Z., Dietrich, A., Deserno, L., de Wit, S., Villringer, A., Heinze, H.-J., Schlagenhauf, F., & Horstmann, A. (2016). Slips of action and sequential decisions: A cross-validation study of tasks assessing habitual and goal-directed action control. Frontiers in Behavioral Neuroscience, 10, 234. https://doi.org/10.3389/fnbeh.2016.00234 Skandali, N., Majuri, J., Joutsa, J., Baek, K., Arponen, E., Forsback, S., Kassinen, V., & Voon, V. (2021). The neural substrates of risky rewards and losses in healthy volunteers and patient groups: A PET imaging study. Psychological medicine, 52, 3280-3288. https://doi.org/10.17863/CAM.64002 Smith, K. S., Bucci, D. J., Luikart, B. W., & Mahler, S. V. (2016). DREADDs: Use and application in behavioral neuroscience. Behavioral Neuroscience, 130(2), 137–155. https://doi.org/10.1037/bne0000135 200 Smith, K. S., Virkud, A., Deisseroth, K., & Graybiel, A. M. (2012). Reversible online control of habitual behavior by optogenetic perturbation of medial prefrontal cortex. Proceedings of the National Academy of Sciences, 109(46), 18932–18937. https://doi.org/10.1073/pnas.1216264109 Soares, S., Atallah, B. V., & Paton, J. J. (2016). Midbrain dopamine neurons control judgment of time. Science, 354, 1273–1277. doi:10.1126/science.aah5234 Spetch, M. L., Madan, C. R., Liu, Y. S., & Ludvig, E. A. (2020). Effects of winning cues and relative payout on choice between simulated slot machines. Addiction, 115(9), 1719–1727. https://doi.org/10.1111/add.15010 Stalnaker, T. A., Berg, B., Aujla, N., & Schoenbaum, G. (2016). Cholinergic interneurons use orbitofrontal input to track beliefs about current state. Journal of Neuroscience, 36(23), 6242–6257. https://doi.org/10.1523/JNEUROSCI.0157-16.2016 Stalnaker, T. A., Roesch, M. R., Franz, T. M., Burke, K. A., & Schoenbaum, G. (2006). Abnormal associative encoding in orbitofrontal neurons in cocaine-experienced rats during decision-making. European Journal of Neuroscience, 24, 2643–2653. https://doi.org/10.1111/j.1460-9568.2006.05128.x Stanton, S. J., Liening, S. H., & Schultheiss, O. C. (2011). Testosterone is positively associated with risk taking in the Iowa Gambling Task. Hormones and Behavior, 59(2), 252–256. https://doi.org/10.1016/j.yhbeh.2010.12.003 Stevens, L., Betanzos-Espinosa, P., Crunelle, C. L., Vergara-Moragues, E., Roeyers, H., Lozano, O., …, Pérez-García, M. (2013). Disadvantageous decision-making as a predictor of drop-out among cocaine-dependent individuals in long-term residential treatment. Frontiers in Psychiatry, 4, 149. https://doi.org/10.3389/fpsyt.2013.00149 Stopper, C. M., Green, E. B., & Floresco, S. B. (2014). Selective involvement by the medial orbitofrontal cortex in biasing risky, but not impulsive, choice. Cerebral Cortex, 24(1), 154–162. https://doi.org/10.1093/cercor/bhs297 Surmeier, D. J., Plotkin, J., & Shen, W. (2009). Dopamine and synaptic plasticity in dorsal striatal circuits controlling action selection. Current Opinion in Neurobiology, 19(6), 621–628. https://doi.org/10.1016/j.conb.2009.10.003 Swinford-Jackson, S. E., Anastasio, N. C., Fox, R. G., Stutz, S. J., & Cunningham, K. A. (2016). Incubation of cocaine cue reactivity associates with neuroadaptations in the cortical serotonin (5-HT) 5-HT2C receptor (5-HT2C R) system. Neuroscience, 324(Journal Article), 50–61. https://doi.org/10.1016/j.neuroscience.2016.02.052 Swintosky, M., Brennan, J. T., Koziel, C., Paulus, J. P., & Morrison, S. E. (2021). Sign tracking predicts suboptimal behavior in a rodent gambling task. Psychopharmacology, 238(9), 2645–2660. https://doi.org/10.1007/s00213-021-05887-8 201 Takahashi, Y. K., Chang, C. Y., Lucantonio, F., Haney, R. Z., Berg, B. A., Yau, H.-J., Bonci, A., & Schoenbaum, G. (2013). Neural estimates of imagined outcomes in the orbitofrontal cortex drive behavior and learning. Neuron, 80(2), 10.1016/j.neuron.2013.08.008. https://doi.org/10.1016/j.neuron.2013.08.008 Takahashi, Y. K., Roesch, M. R., Stalnaker, T. A., Haney, R. Z., Calu, D. J., Taylor, A. R., Burke, K. A., & Schoenbaum, G. (2009). The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes. Neuron, 62(2), 269–280. https://doi.org/10.1016/j.neuron.2009.03.005 Thompson, J. L., Drysdale, M., Baimel, C., Kaur, M., MacGowan, T., Pitman, K. A., & Borgland, S. L. (2017). Obesity-induced structural and neuronal plasticity in the lateral orbitofrontal cortex. Neuropsychopharmacology, 42(7), 1480–1490. https://doi.org/10.1038/npp.2016.284 Thorn, C. A., Atallah, H., Howe, M., & Graybiel, A. M. (2010). Differential dynamics of activity changes in dorsolateral and dorsomedial striatal loops during learning. Neuron, 66(5), 781–795. https://doi.org/10.1016/j.neuron.2010.04.036 Thrailkill, E. A., Michaud, N. L., & Bouton, M. E. (2021). Reinforcer predictability and stimulus salience promote discriminated habit learning. Journal of Experimental Psychology. Animal Learning and Cognition, 47(2), 183–199. https://doi.org/10.1037/xan0000285 Trempler, I., Bürkner, P.-C., El-Sourani, N., Binder, E., Reker, P., Fink, G. R., & Schubotz, R. I. (2020). Impaired context-sensitive adjustment of behaviour in Parkinson’s disease patients tested on and off medication: An fMRI study. NeuroImage, 212, 116674. https://doi.org/10.1016/j.neuroimage.2020.116674 Tricomi, E., & Lempert, K. M. (2015). Value and probability coding in a feedback-based learning task utilizing food rewards. Journal of Neurophysiology, 113(1), 4–13. https://doi.org/10.1152/jn.00086.2014 Tricomi, E., Balleine, B. W., & O’Doherty, J. P. (2009). A specific role for posterior dorsolateral striatum in human habit learning. European Journal of Neuroscience, 29(11), 2225–2232. https://doi.org/10.1111/j.1460-9568.2009.06796.x van den Bos, R., Homberg, J., & de Visser, L. (2013). A critical review of sex differences in decision-making tasks: Focus on the Iowa Gambling Task. Behavioural Brain Research, 238, 95–108. https://doi.org/10.1016/j.bbr.2012.10.002 van Eimeren, T., Ballanger, B., Pellecchia, G., Miyasaki, J. M., Lang, A. E., & Strafella, A. P. (2009). Dopamine agonists diminish value sensitivity of the orbitofrontal cortex: A trigger for pathological gambling in Parkinson’s disease? Neuropsychopharmacology, 34(13), 2758–2766. https://doi.org/10.1038/npp.2009.124 202 van Eimeren, T., Pellecchia, G., Cilia, R., Ballanger, B., Steeves, T. D. L., Houle, S., …, Strafella, A. P. (2010). Drug-induced deactivation of inhibitory networks predicts pathological gambling in PD. Neurology, 75(19), 1711–1716. https://doi.org/10.1212/WNL.0b013e3181fc27fa van Hasselt, F. N., de Visser, L., Tieskens, J. M., Cornelisse, S., Baars, A. M., Lavrijsen, M., …, & Joëls, M. (2012). Individual variations in maternal care early in life correlate with later life decision-making and c-fos expression in prefrontal subregions of rats. PloS One, 7(5), e37820. https://doi.org/10.1371/journal.pone.0037820 van Timmeren, T., Daams, J. G., van Holst, R. J., & Goudriaan, A. E. (2018). Compulsivity-related neurocognitive performance deficits in gambling disorder: A systematic review and meta-analysis. Neuroscience & Biobehavioral Reviews, 84, 204–217. https://doi.org/10.1016/j.neubiorev.2017.11.022 Vandaele, Y., & Ahmed, S. H. (2021). Habit, choice, and addiction. Neuropsychopharmacology, 46(4), Article 4. https://doi.org/10.1038/s41386-020-00899-y Vandaele, Y., & Janak, P. H. (2018). Defining the place of habit in substance use disorders. Progress in Neuro-Psychopharmacology & Biological Psychiatry, 87, 22–32. https://doi.org/10.1016/j.pnpbp.2017.06.029 Vandaele, Y., Mahajan, N. R., Ottenheimer, D. J., Richard, J. M., Mysore, S. P., & Janak, P. H. (2019). Distinct recruitment of dorsomedial and dorsolateral striatum erodes with extended training. ELife, 8, e49536. https://doi.org/10.7554/eLife.49536 Vandaele, Y., Pribut, H. J., & Janak, P. H. (2017). Lever insertion as a salient stimulus promoting insensitivity to outcome devaluation. Frontiers in Integrative Neuroscience, 11, 23. https://doi.org/10.3389/fnint.2017.00023 Verdejo-García, A., Lawrence, A. J., & Clark, L. (2008). Impulsivity as a vulnerability marker for substance-use disorders: Review of findings from high-risk research, problem gamblers and genetic association studies. Neuroscience & Biobehavioral Reviews, 32(4), 777–810. https://doi.org/10.1016/j.neubiorev.2007.11.003 Versaggi, C. L., King, C. P., & Meyer, P. J. (2016). The tendency to sign-track predicts cue-induced reinstatement during nicotine self-administration, and is enhanced by nicotine but not ethanol. Psychopharmacology, 233(15–16), 2985–2997. https://doi.org/10.1007/s00213-016-4341-7 Volkow, N. D., Wang, G.-J., Telang, F., Fowler, J. S., Logan, J., Childress, A.-R., …, Wong, C. (2006). Cocaine cues and dopamine in dorsal striatum: Mechanism of craving in cocaine addiction. The Journal of Neuroscience, 26(24), 6583–6588. https://doi.org/10.1523/JNEUROSCI.1544-06.2006 203 Voon, V., Derbyshire, K., Rück, C., Irvine, M. A., Worbe, Y., Enander, J., …, Bullmore, E. T. (2015). Disorders of compulsivity: A common bias towards learning habits. Molecular Psychiatry, 20(3), 345–352. https://doi.org/10.1038/mp.2014.44 Wallin, K. G., & Wood, R. I. (2015). Anabolic-androgenic steroids impair set-shifting and reversal learning in male rats. European Neuropsychopharmacology, 25(4), 583–590. https://doi.org/10.1016/j.euroneuro.2015.01.002 Wallis, J. D. (2007). Orbitofrontal cortex and its contribution to decision-making. Annual Review of Neuroscience, 30, 31–56. https://doi.org/10.1146/annurev.neuro.30.051606.094334 Wang, Q., Chen, C., Cai, Y., Li, S., Zhao, X., Zheng, L., …, Xue, G. (2016). Dissociated neural substrates underlying impulsive choice and impulsive action. NeuroImage, 134, 540–549. https://doi.org/10.1016/j.neuroimage.2016.04.010 Watabe-Uchida, M., Zhu, L., Ogawa, S. K., Vamanrao, A., & Uchida, N. (2012). Whole-brain mapping of direct inputs to midbrain dopamine neurons. Neuron, 74(5), 858–873. https://doi.org/10.1016/j.neuron.2012.03.017 Watanabe, S. (2010). Asymptotic equivalence of bayes cross validation and widely applicable information criterion in singular learning theory. Journal of Machine Learning Research, 11, 3571-3594. Weafer, J., & de Wit, H. (2014). Sex differences in impulsive action and impulsive choice. Addictive Behaviors, 39(11), 1573–1579. https://doi.org/10.1016/j.addbeh.2013.10.033 Weingarten, H. P. (1983). Conditioned cues elicit feeding in sated rats: A role for learning in meal initiation. Science, 220(4595), 431–433. https://doi.org/10.1126/science.6836286 Welborn, B. L., Papademetris, X., Reis, D. L., Rajeevan, N., Bloise, S. M., & Gray, J. R. (2009). Variation in orbitofrontal cortex volume: Relation to sex, emotion regulation and affect. Social Cognitive and Affective Neuroscience, 4(4), 328–339. https://doi.org/10.1093/scan/nsp028 Wickens, J. R., Horvitz, J. C., Costa, R. M., & Killcross, S. (2007). Dopaminergic mechanisms in actions and habits. The Journal of Neuroscience, 27(31), 8181–8183. https://doi.org/10.1523/JNEUROSCI.1671-07.2007 Willuhn, I., Burgeno, L. M., Everitt, B. J., & Phillips, P. E. M. (2012). Hierarchical recruitment of phasic dopamine signaling in the striatum during the progression of cocaine use. Proceedings of the National Academy of Sciences, 109(50), 20703–20708. https://doi.org/10.1073/pnas.1213460109 Winocur, G., & Greenwood, C. E. (1993). High-fat diets impair conditional discrimination learning in rats. Psychobiology, 21(4), 286–292. https://doi.org/10.3758/BF03327147 204 Winstanley, C. A., Theobald, D. E., Dalley, J. W., Glennon, J. C., & Robbins, T. W. (2004). 5-HT2A and 5-HT2C receptor antagonists have opposing effects on a measure of impulsivity: Interactions with global 5-HT depletion. Psychopharmacology, 176(3–4), 376–385. https://doi.org/10.1007/s00213-004-1884-9 Woon, E. P., Sequeira, M. K., Barbee, B. R., & Gourley, S. L. (2020). Involvement of the rodent prelimbic and medial orbitofrontal cortices in goal-directed action: A brief review. Journal of Neuroscience Research, 98(6), 1020–1030. https://doi.org/10.1002/jnr.24567 Worbe, Y., Palminteri, S., Savulich, G., Daw, N. D., Fernandez-Egea, E., Robbins, T. W., & Voon, V. (2016). Valence-dependent influence of serotonin depletion on model-based choice strategy. Molecular Psychiatry, 21(5), 624–629. https://doi.org/10.1038/mp.2015.46 Xin, J., & Costa, R. M. (2010). Start/stop signals emerge in nigrostriatal circuits during sequence learning. Nature, 466(7305), 457–462. https://doi.org/10.1038/nature09263 Yager, L. M., Garcia, A. F., Wunsch, A. M., & Ferguson, S. M. (2015). The ins and outs of the striatum: Role in drug addiction. Neuroscience, 301, 529–541. https://doi.org/10.1016/j.neuroscience.2015.06.033 Yin, H. H., & Knowlton, B. J. (2006). The role of the basal ganglia in habit formation. Nature Reviews. Neuroscience, 7(6), 464–476. https://doi.org/10.1038/nrn1919 Yin, H. H., Knowlton, B. J., & Balleine, B. W. (2004). Lesions of dorsolateral striatum preserve outcome expectancy but disrupt habit formation in instrumental learning. The European Journal of Neuroscience, 19(1), 181–189. https://doi.org/10.1111/j.1460-9568.2004.03095.x Yin, H. H., Mulcare, S. P., Hilário, M. R. F., Clouse, E., Holloway, T., Davis, M. I., …, Costa, R. M. (2009). Dynamic reorganization of striatal circuits during the acquisition and consolidation of a skill. Nature Neuroscience, 12(3), Article 3. https://doi.org/10.1038/nn.2261 Yin, H. H., Ostlund, S. B., & Balleine, B. W. (2008). Reward-guided learning beyond dopamine in the nucleus accumbens: The integrative functions of cortico-basal ganglia networks. European Journal of Neuroscience, 28(8), 1437–1448. https://doi.org/10.1111/j.1460-9568.2008.06422.x Yoest, K. E., Quigley, J. A., & Becker, J. B. (2018). Rapid effects of ovarian hormones in dorsal striatum and nucleus accumbens. Hormones and Behavior, 104, 119–129. https://doi.org/10.1016/j.yhbeh.2018.04.002 Young, M. K., Conn, K.-A., Das, J., Zou, S., Alexander, S., Burne, T. H. J., & Kesby, J. P. (2022). Activity in the dorsomedial striatum underlies serial reversal learning performance under probabilistic uncertainty. Biological Psychiatry Global Open Science. https://doi.org/10.1016/j.bpsgos.2022.08.005 205 Young, S. N. (2013). Acute tryptophan depletion in humans: A review of theoretical, practical and ethical aspects. Journal of Psychiatry & Neuroscience, 38(5), 294–305. https://doi.org/10.1503/jpn.120209 Yu, A. J., & Dayan, P. (2005). Uncertainty, neuromodulation, and attention. Neuron, 46(4), 681–692. https://doi.org/10.1016/j.neuron.2005.04.026 Zachry, J. E., Nolan, S. O., Brady, L. J., Kelly, S. J., Siciliano, C. A., & Calipari, E. S. (2021). Sex differences in dopamine release regulation in the striatum. Neuropsychopharmacology, 46(3), 491–499. https://doi.org/10.1038/s41386-020-00915-1 Zeeb, F. D., & Winstanley, C. A. (2011). Lesions of the basolateral amygdala and orbitofrontal cortex differentially affect acquisition and performance of a rodent gambling task. The Journal of Neuroscience, 31, 2197-2204. doi:10.1523/JNEUROSCI.5597-10.2011 Zeeb, F. D., & Winstanley, C. A. (2013). Functional disconnection of the orbitofrontal cortex and basolateral amygdala impairs acquisition of a rat gambling task and disrupts animals’ ability to alter decision-making behavior after reinforcer devaluation. The Journal of Neuroscience, 33(15), 6434–6443. https://doi.org/10.1523/JNEUROSCI.3971-12.2013 Zeeb, F. D., Baarendse, P. J. J., Vanderschuren, L. J. M. J., & Winstanley, C. A. (2015). Inactivation of the prelimbic or infralimbic cortex impairs decision-making in the rat gambling task. Psychopharmacology, 232(24), 4481–4491. https://doi.org/10.1007/s00213-015-4075-y Zeeb, F. D., Floresco, S. B., & Winstanley, C. A. (2010). Contributions of the orbitofrontal cortex to impulsive choice: Interactions with basal levels of impulsivity, dopamine signalling, and reward-related cues. Psychopharmacology, 211(1), 87–98. https://doi.org/10.1007/s00213-010-1871-2 Zeeb, F. D., Robbins, T. W., & Winstanley, C. A. (2009). Serotonergic and dopaminergic modulation of gambling behavior as assessed using a novel rat gambling task. Neuropsychopharmacology, 34(10), 2329–2343. https://doi.org/10.1038/npp.2009.62 Zha, R., Li, P., Liu, Y., Alarefi, A., Zhang, X., & Li, J. (2022). The orbitofrontal cortex represents advantageous choice in the Iowa gambling task. Human Brain Mapping, 43(12), 3840–3856. https://doi.org/10.1002/hbm.25887 Zhang, Q. J., Wang, S., Liu, J., Ali, U., Gui, Z. H., Wu, Z. H., …, Chen, L. (2010). Unilateral lesion of the nigrostriatal pathway decreases the response of interneurons in medial prefrontal cortex to 5-HT 2A/2C receptor stimulation in the rat. Brain Research, 1312, 127–137. https://doi.org/10.1016/j.brainres.2009.11.052 Zhang, Y., Larcher, K. M.-H., Misic, B., & Dagher, A. (2017). Anatomical and functional organization of the human substantia nigra and its connections. ELife, 6, e26653. https://doi.org/10.7554/eLife.26653 206 Zhou, J., Gardner, M. P., & Schoenbaum, G. (2021). Is the core function of orbitofrontal cortex to signal values or make predictions? Current Opinion in Behavioral Sciences, 41, 1–9. https://doi.org/10.1016/j.cobeha.2021.02.011 Zhou, J., Jia, C., Feng, Q., Bao, J., & Luo, M. (2015). Prospective coding of dorsal raphe reward signals by the orbitofrontal cortex. Journal of Neuroscience, 35(6), 2717–2730. Zilberman, N., Lavidor, M., Yadid, G., & Rassovsky, Y. (2019). Qualitative review and quantitative effect size meta-analyses in brain regions identified by cue-reactivity addiction studies. Neuropsychology, 33(3), 319–334. https://doi.org/10.1037/neu0000526
Thesis/Dissertation
eng
Neuroscience
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Winstanley, Catharine
Cognitive and neural underpinnings of the interaction between reward-paired cues, risky choice, and cognitive flexibility
Text