Applied Science, Faculty of
Engineering, School of (Okanagan)
DSpace
UBCO
Gupta, Kashish
Doctor of Philosophy - PhD
University of British Columbia
Reinforcement learning has long been advertised as the one with the capability to intelligently mimic and understand human learning and behavior. While the upshot of the field's advances is not underrated, its applicability and extension to large, complex and highly dynamic environments remain inefficient, inaccurate or unsolved. As the complexity of objectives increase, trend in reinforcement learning research is to tackle it with more computational power and training samples. The inspiration for the proposed methodology is derived from human learning, where increasingly complex objectives are learned with limited computational power by re-purposing previously learned skills. An intuitive and elegant concept is presented which exploits abstract symmetries present in training environments to learn a naïve agent and bypass some of the core challenges of reinforcement learning training. The naïve agent, trained in a local environment, can then be used in numerous ways to improve training in high-dimensional state-space environments. The proposed solution is incorporated with heuristic-based planning, learning from demonstration and state-space abstraction methods to present the efficacy and ease of adaptability of the proposed concept on a range of domains. The proposed method provides a structured approach to the training process and improves the trained agent's generalization capabilities and training sample efficiency. While the presented concept benefits from additional information about the state-space structure, it is notably different from traditional reinforcement learning approaches which augments the training process with expensive domain-specific knowledge to improve sample efficiency. Experiments and analyses are presented for challenging navigation and control environments, solved with augmented off-policy family of reinforcement learning methods.
Reinforcement Learning in ComplexEnvironments With Locally TrainedNa¨ıve AgentsbyKashish GuptaB.Tech., Vellore Institute of Technology, Chennai, India, 2016A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFDOCTOR OF PHILOSOPHYinTHE COLLEGE OF GRADUATE STUDIES(Electrical Engineering)THE UNIVERSITY OF BRITISH COLUMBIA(Okanagan)November 2021© Kashish Gupta, 2021The following individuals certify that they have read, and recommendto the College of Graduate Studies for acceptance, a thesis/dissertation en-titled:Reinforcement Learning in Complex Environments With Lo-cally Trained Na¨ıve Agentssubmitted by Kashish Gupta in partial fulfilment of the requirements ofthe degree of Doctor of PhilosophyDr. Homayoun Najjaran, School of EngineeringSupervisorDr. Abbas Milani, School of EngineeringSupervisory Committee MemberDr. Chen Feng, School of EngineeringSupervisory Committee MemberDr. Solomon Tesfamariam, School of EngineeringUniversity ExaminerDr. William Melek, University of WaterlooExternal ExamineriiAbstractReinforcement learning has long been advertised as the one with thecapability to intelligently mimic and understand human learning and be-havior. While the upshot of the field’s advances is not underrated, its appli-cability and extension to large, complex and highly dynamic environmentsremain inefficient, inaccurate or unsolved. As the complexity of objectivesincrease, trend in reinforcement learning research is to tackle it with morecomputational power and training samples. The inspiration for the proposedmethodology is derived from human learning, where increasingly complexobjectives are learned with limited computational power by re-purposingpreviously learned skills. An intuitive and elegant concept is presentedwhich exploits abstract symmetries present in training environments to learna na¨ıve agent and bypass some of the core challenges of reinforcement learn-ing training. The na¨ıve agent, trained in a local environment, can then beused in numerous ways to improve training in high-dimensional state-spaceenvironments. The proposed solution is incorporated with heuristic-basedplanning, learning from demonstration and state-space abstraction methodsto present the efficacy and ease of adaptability of the proposed concept on arange of domains. The proposed method provides a structured approach tothe training process and improves the trained agent’s generalization capa-iiibilities and training sample efficiency. While the presented concept benefitsfrom additional information about the state-space structure, it is notably dif-ferent from traditional reinforcement learning approaches which augmentsthe training process with expensive domain-specific knowledge to improvesample efficiency. Experiments and analyses are presented for challengingnavigation and control environments, solved with augmented off-policy fam-ily of reinforcement learning methods.ivLay SummaryReinforcement Learning in Artificial Intelligence is a formal frameworkfor trial-and-error-based learning. While learning a task, the artificial learnertakes a series of actions, in expectation of a positive experience. The feed-back from the task, then guides the learning process and helps the learneridentify an optimal series of actions. The process is analogous to an infanttrying to learn from its environment. As the complexity of task increases,large amounts of positive experiences are required for learning in this fash-ion. An able human learner on the other hand, utilizes their past knowl-edge and skills to learn faster; by identifying similarities or making abstractconnections with their previous knowledge. The presented research takesinspiration from this intuition and attempts to improve the reinforcementlearning process by initiating training on simpler, abstractly symmetricalversions of the complex task and using that information to effectively learnor solve the complex task itself.vPrefaceThe work presented in this thesis was conducted at the Advanced Con-trol and Intelligent Systems (ACIS) Laboratory at the University of BritishColumbia, Okanagan under the supervision of Dr. Homayoun Najjaran.During my thesis, I’ve had the opportunity to work on multiple percep-tion related problems - from autonomous identification of a landing pad forquadcopter landing to wrinkle detection on deformed fabrics draped on dou-ble curved surfaces. Making sense of raw sensor data constitutes the firststep in completely autonomous agents. The perception output then guidesthe decision-making component of intelligent systems. In other words, per-ception gives objectivity to the logic behind the decision-making process.While the primary contribution of the presented work is on autonomousdecision-making and control, numerous exciting works on perception werepublished during my PhD. The following list sheds light on the publicationsand components of the presented work.− Chapter 1 introduces the proposed concept with motivation from thenatural learning process, and highlights the benefit of the proposedconcept from reinforcement learning’s perspective.− Chapter 2 presents a comprehensive literature review for the presentedvimethodology. Parts of the review are published in the internationaljournal of robotic engineering; K. Gupta, H. Najjaran ”Curriculum-Based Deep Reinforcement Learning for Adaptive Robotics: A Mini-Review”, IJRE, Vol6(1), 2021 and K.Gupta, D. Mukherjee, H. Naj-jaran, ”Extending the Capabilities of Reinforcement Learning ThroughCurriculum: A Review of Methods and Applications”, SN CS Journal,vol 3(1).− The concept of EASE proposed in Chapter 3, along with results fromChapter 3 and 4 are submitted for review to the IEEE Internationalconference on robotics and automation (ICRA) 2022 K. Gupta, H.Najjaran ”Exploiting Abstract Symmetries in Reinforcement Learningfor Complex Environments”.− Works have originated from my PhD that focus on perception forunmanned systems and allow them to autonomously execute complexmaneuvers like UAV landing, based on the data from landing platformdetection. The works include i) K. Gupta, B. J. Emran, H. Najjaran,”Vision-based pose estimation of a multi-rotor unmanned aerial ve-hicle”, IJIUS, April, 2019, pp 0030 and ii) E. Nowak, K. Gupta, H.Najjaran, ”Development of a Plug-and-Play Infrared Landing Systemfor Multirotor Unmanned Aerial Vehicles”, CRV, 2017, pp 256-260.− Numerous other works have originated from the work on perceptionduring my PhD that study quality inspection of deformed fabrics inmanufacturing setting. The works include i) K. Gupta, M. Ko¨rber,F. Krebs, H. Najjaran, ”Vision-based deformation and wrinkle de-viitection for semi-finished fiber products on curved surfaces”, CASE,August, 2018, pp 618-623, ii) K. Gupta, M. Ko¨rber, A. Djavadifar, F.Krebs, H. Najjaran, ”Wrinkle and boundary detection of fiber productsin robotic composites manufacturing”, Assembly Automation, 2019,and iii) A. Djavadifar, J. B. Grahan-Knight, K. Gupta, M. Ko¨rber,P. Lasserre, H. Najjaran, ”Robot-assisted composite manufacturingbased on machine learning applied to multi-view computer vision”,Smart Multimedia, 2019.− Last but not the least, works spanning efficient computer vision X.Chen, Y. Chen, K. Gupta, J, Zhou, H. Najjaran, ”SliceNet: A profi-cient model for real-time 3D shape-based recognition”, Neurocomput-ing, 2018, vol 316, surveying human-robot collaboration methods D.Mukherjee, K. Gupta, L. H. Chang, H. Najjaran, ”A Survey of RobotLearning Strategies for Human-Robot Collaboration in Industrial Set-tings”, RCIM, 2022, vol 73, pp 102231, and teaching tool simulationscovering the basics of autonomous vehicle technology M. O’Brien, K.Gupta, B. Emran, H. Najjaran, ”Autonomous Vehicle Control: Teach-ing Tool and Simulation”, FIE, 2019, have been published.viiiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiiList of Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . .xviiiList of Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . xxGlossary of Notation . . . . . . . . . . . . . . . . . . . . . . . . .xxiiiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . .xxviiDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .xxviiiChapter 1: Introduction . . . . . . . . . . . . . . . . . . . . . . . 11.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1ixTABLE OF CONTENTS1.1.1 Nature inspired Artificial Intelligence . . . . . . . . . . 11.1.2 The Elusive Solution . . . . . . . . . . . . . . . . . . . 21.2 Reinforcement Learning . . . . . . . . . . . . . . . . . . . . . 41.3 Learning with EASE . . . . . . . . . . . . . . . . . . . . . . . 51.4 Objectives and Contributions . . . . . . . . . . . . . . . . . . 10Chapter 2: Background . . . . . . . . . . . . . . . . . . . . . . . 152.1 Reinforcement Learning Framework . . . . . . . . . . . . . . 152.1.1 Robotics and Reinforcement Learning . . . . . . . . . 192.2 EASE paradigms . . . . . . . . . . . . . . . . . . . . . . . . . 202.2.1 Hierarchical Reinforcement Learning . . . . . . . . . . 212.2.2 Transfer Learning . . . . . . . . . . . . . . . . . . . . 222.2.3 Curriculum Learning . . . . . . . . . . . . . . . . . . . 222.3 Guiding samples and MDP Homomorphism . . . . . . . . . . 232.4 Curriculum-based Reinforcement Learning . . . . . . . . . . . 242.4.1 Curriculum through Teacher-Student interaction . . . 282.4.2 Curriculum through intermediate goals . . . . . . . . . 302.4.3 Curriculum through Self-Play . . . . . . . . . . . . . . 342.4.4 Curriculum through Attributes and Distillation . . . . 352.5 Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 382.5.1 Deep Q-Network Learning . . . . . . . . . . . . . . . . 402.5.2 Deep Deterministic Policy Gradient . . . . . . . . . . 41Chapter 3: EASE - Concept and Insights . . . . . . . . . . . . 453.1 EASE as a concept . . . . . . . . . . . . . . . . . . . . . . . . 463.1.1 Reacher . . . . . . . . . . . . . . . . . . . . . . . . . . 50xTABLE OF CONTENTS3.1.2 Benefits of local training . . . . . . . . . . . . . . . . . 553.2 Heuristics with EASE . . . . . . . . . . . . . . . . . . . . . . 56Chapter 4: Data generation with EASE . . . . . . . . . . . . . 764.1 Demonstrations with EASE . . . . . . . . . . . . . . . . . . . 794.2 Abstraction with EASE . . . . . . . . . . . . . . . . . . . . . 86Chapter 5: Conclusions . . . . . . . . . . . . . . . . . . . . . . . 975.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 975.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 1005.2.1 EASE with Heuristics . . . . . . . . . . . . . . . . . . 1005.2.2 EASE as a generative model . . . . . . . . . . . . . . 1015.2.3 EASE with feature extraction . . . . . . . . . . . . . . 101Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103xiList of TablesTable 3.1 Comparison of success rates as a function of accuracyrequirements de for the local and global agents afterconvergence. Both the agents are given the same testset and the success rate is reported as an average over50 trials for each de. The agents are trained for de = 0.02 53Table 3.2 Success rates reported as a function of deviation fromthe fixed initial state for which the local and globalagents were trained. Two de values are reported forthe global agent to further emphasize that the agentis rarely able to generalize around the start state evenwith relaxed accuracy requirements . . . . . . . . . . . 54xiiList of FiguresFigure 1.1 A car agent at point A is trained to find the optimalpath between the points A and B. The objective is toefficiently navigate and avoid obstacles on the way.The trajectories τ1 and τ2 shows random explorationwith reinforcement learning, while trajectories τ3 andτ4 represents reinforcement learning guided throughsome predefined milestones M31,M32,M41,M42 andM43. τ∗ represents the optimal path between the twopoints. . . . . . . . . . . . . . . . . . . . . . . . . . . 13xiiiLIST OF FIGURESFigure 1.2 The gray shaded area represents the local environ-ment as a part of the global environment. For thelocal environment, the reset state for ω0 is boundedbetween [0, π/2] or [−3π/2, 0] depending on the direc-tion of measure. There are no limitations on ω1. Thetarget is denoted by the red dot, with its velocity vec-tor denoted by the black broken arrow. The targetis also bounded in region A for local training. Thevertical lines represent the partition over the globalenvironment, generating 4 abstractly similar regionsA,B,C and D . . . . . . . . . . . . . . . . . . . . . . 14Figure 2.1 Reinforcement Learning General Framework . . . . . 16Figure 2.2 The Figure shows a modified tower of hanoi problem,where the robotic arm is expected to identify and lo-cate different sized rings and stack them on the tower,in the decreasing order of their radii. It demonstratestwo possible curriculum and their progress towardsthe final goal, with Figures (a), (b) and (c) represent-ing common steps between the two curriculum, (d1),(d2) as curriculum 1, (e1), (e2) as curriculum 2 witha hierarchical component and (f) as the final finishedtask. . . . . . . . . . . . . . . . . . . . . . . . . . . . 27xivLIST OF FIGURESFigure 3.1 Reacher environment depiction. The gray shaded arearepresents the local environment as a part of the globalenvironment. For the local environment, the resetstate for ω0 is bounded between [0, π/2] or [−3π/2, 0]depending on the direction of measure. There areno limitations on ω1. The target is denoted by thered dot, with its velocity vector denoted by the blackbroken arrow. The target is also bounded in regionA for local training. The vertical lines represent thepartition over the global environment, generating 4abstractly similar regions A,B,C and D. . . . . . . 51Figure 3.2 Training progression of the local and global agentswith fixed start state and randomly sampled targetlocations . . . . . . . . . . . . . . . . . . . . . . . . . 53Figure 3.3 A mapping example of an environment with N = 4,with a recursive (a) and representative (b) map on toa smaller environment with n = 2. Here N = n2, l =2 > 1. The figure demonstrates how larger environ-ments can be abstracted (either by representative orrecursive mapping) onto a smaller environment andsolved as a combination of solutions of the smallerenvironments. . . . . . . . . . . . . . . . . . . . . . . 61xvLIST OF FIGURESFigure 3.4 The figure denotes the possible choices for the selec-tion of a subgoal for the region A and agent positionae. From ae, the agent could choose to go to theneighbourhood vertices (change regions) or select an-other vertex of the current region A. The functionthen greedily picks the vertex end goal that has theleast distance to goal sg in environment EN . . . . . 62Figure 3.5 Flowchart of the presented methodology . . . . . . . 65Figure 3.6 Case 1: Implementation of EASE - Solution to N =16 environment using an RL agent trained on an n =4 environment. The agent solves for a region, thengreedily selects for a subgoal either within the sameregion or a neighbourhood region. The heuristics theniteratively maps different regions onto En and solvesthe environment . . . . . . . . . . . . . . . . . . . . . 67Figure 3.7 Case 1: Implementation of EASE without recursivemapping - Solution to N = 16 maze using an RLagent trained on an n = 4 environment. The agentsolves different user-defined regions (A, B, C and D)while the solution for traversal in between the regionswas pre-defined . . . . . . . . . . . . . . . . . . . . . 68xviLIST OF FIGURESFigure 3.8 Case 2: Implementation of EASE with obstacles -Solution to N = 16 environment using an RL agenttrained on an n = 4 environment, with randomlyplaced obstacles. The agent greedily selects for asubgoal after solving each region. The vertices thatare also obstacles, are removed from consideration forpossible end goals. The heuristics then recursivelymap the different regions onto En and the agent solvesfor the path until the environment EN is solved . . . 69Figure 3.9 Successful training of an RL agent in a n = 4 grid-world environment without obstacles . . . . . . . . . 73Figure 3.10 Successful training of an RL agent in a n = 4 grid-world environment with obstacles . . . . . . . . . . . 74Figure 3.11 Failed training of an RL agent in a N = 16 gridworldenvironment without obstacles . . . . . . . . . . . . . 74Figure 4.1 Training progression of the DDPG agent with andwithout the EASE augmented data. . . . . . . . . . . 85Figure 4.2 Robot workspace with δ units away from the end-effector reset position . . . . . . . . . . . . . . . . . . 90Figure 4.3 Robot workspace for which a global agent is expectedto learn . . . . . . . . . . . . . . . . . . . . . . . . . . 91Figure 4.4 Recurring δ unit areas in the robot workspace withgripper position at the centre . . . . . . . . . . . . . . 91xviiLIST OF FIGURESFigure 4.5 The figure shows the sensitivity of the algorithm tothe tuning factor ζ. The factor controls the amount ofdata utilized from the local agent. As suspected, thesample efficiency increases with increasing ζ whichconfirms that the data generate by the local agentsignificantly helps the global learning process . . . . . 93Figure 4.6 This figure shows the significant improvement in globalagent learning with the proposed algorithm δ-EASEcompared to the standalone DDPG agent . . . . . . . 94Figure 4.7 This figure shows how the training progress destabi-lizes for certain ζ values, when the global agent isinitialized with the weights of the local agent. Thetraining eventually converges . . . . . . . . . . . . . . 95xviiiList of Algorithms1 Deep Q-Learning (DQN) . . . . . . . . . . . . . . . . . . . . . 422 Deep Deterministic Policy Gradient (DDPG) . . . . . . . . . . 433 Update model weights . . . . . . . . . . . . . . . . . . . . . . . 444 EASE with demonstrations (demo-EASE) . . . . . . . . . . . . 875 EASE with δ-abstraction (δ-EASE) . . . . . . . . . . . . . . . 92xixList of AbbreviationsA2C Advantage Actor Critic. 20A3C Asynchronous Advantage Actor Critic. 20AI Artificial Intelligence. 1ALP Absolute Learning Progress. 30BC Behavior Cloning. 80CER Competitive Experience Replay. 20CL Curriculum Learning. 6CNN Convolutional Neural Network. 48CRL Curriculum-based Reinforcement Learning. 23DDPG Deep Deterministic Policy Gradient. 38DDPGfD Deep Deterministic Policy Gradient from Demonstrations. 80demo-EASE EASE from demonstrations. 83DL Deep Learning. 4xxList of AbbreviationsDOF Degree Of Freedom. 45DP Dynamic Programming. 59DQN Deep Q-Network Learning. 38EASE Exploitation of Abstract symmetry in Environment. 5GAN Generative Adversarial Network. 31GMM Gaussian Mixture Models. 30GPS Guided Policy Search. 23HER Hindsight Experience Replay. 20HRL Hierarchical Reinforcement Learning. 6i.i.d. Independent and Identically Distributed data. 39LfD Learning from Demonstration. 6LQR Linear Quadratic Control. 23MDP Markov Decision Process. 16ML Machine Learning. 4NN Neural Network. 28OO Object-Oriented. 37OU Ornstein-Uhlenbeck Noise. 41xxiList of AbbreviationsPER Prioritized Experience Replay. 80PG Procedurally Generated. 29POMDP Partially Observable Markov Decision Process. 28PPO Proximal Policy Optimization. 18RL Reinforcement Learning. 1TL Transfer Learning. 22TRPO Trust Region Policy Optimzation. 18xxiiGlossary of NotationE Environment. 39EN Environment E parameterized with a sizemetric N . 47P Transition Probability of the states in envi-ronment. 17Qϕ Q(s, a) function, parameterized by ϕ. 38Qϕtarg Q Target Network, parmeterized by ϕtarg. 40Φ(.) Mapping function: S × S. 50Φ−1 Inverse Mapping Function Φ−1 : S l → S. 77β Angle of rotation required for global to localmap. 83ω˙ Angular Velocity. 83ϵ-greedy ϵ-greedy exploration strategy. 40ϵthresh ϵ threshold for ϵ-greedy exploration. 42γ Discount Factor. 17λ Tuning Factor for the Behavior Cloning Loss.80A Action space of the environment. 17xxiiiGlossary of NotationAl Action space of local environment. 47B Experience Replay Memory Buffer. 39Bdemo Demonstration Replay Buffer. 80Bbatch A batch sampled from the experience replaybuffer B. 39M Markov Tuple. 17P Partition over the state-space. 47S State space of the environment. 17S l Local State Space for the na¨ıve agent. 20T Set of Tasks. 28Ta Subtask sampled from T . 29ω Robot Arm joint angle. 45π Policy to learn. 17πn Na¨ıve policy trained in a local environment.Interchangeable with πl. 87πθ Policy parameterized by Theta. 17πlθ Local policy trained on a local environment(na¨ıve agent). 8πθtarg Policy Target Network, parmeterized by θtarg.40ρ Controls the lag of target parameter updates.41θ Parameters of Policy. 17ζ Tuning parameter for controlling teh localagent data generation for δ-EASE. 90xxivGlossary of Notationa∗ Optimal action in an environment. 48al∗Optimal action in the local environment. 48at Action at time t. 16cg Goal Cell. 56cs Start Cell. 56d Done signal. 39de Accuracy requirement while solving for a task.50gv Target Velocity. 83ot Observation from environment at time t. 16ot+1 Observation at next time step. 16p|g Gripper position coordinates. 50p|t Target position coordinates. 50rt numerical reward signal. 16s State of the environment. 17s′ Next state of the environment, after s. 17sl Local state. 20st state at time t. 16t time, denoted as subscript. 16F (s) Find subgoal funtion. 60LQ Value calculated for maximization of Q func-tion. 44Lπ Value calculated for optimization minimiza-tion of π. 44LBC Behavior Cloning Loss. 44xxvGlossary of NotationON Set of obstacle cells in a gridworld environ-ment. 56Qmask Mask for comparing Q values between globaland demonstration agent. 44Qmask Mask to identify if demonstration action isbetter than the policy action. 81R(st, st+1) Reward Function from st to st+1, often underacton a. 16α Learning Rate for NN model update. 44E Expectation value. 171 Binary Indicator Function. 31G Set of Goals for the learning process. 31N Gaussian Noise Function. 43∇ Gradient. 44g Goal for a task. 31q(s, a) q value of a state-action pair obtained fromthe Q-function. 49sn Na¨ıve State, interchangeable with sl. 87sg Goal State. 79si Iniitial State. 79xxviAcknowledgementsFirst and foremost, I would like to express my deepest gratitude to myadvisor, Dr. Homayoun Najjaran. Throughout my PhD, he has been a con-stant support and motivation for exciting avenues of research. Discussionswith him left me invigorated, instilled with new ideas and a rekindled inter-est in research.I would like to extend my gratitude to all my colleagues at the AdvancedControl and Intelligent Systems (ACIS) Laboratory for their support, help,insightful discussions and contributions towards my PhD. I would also liketo thank my committee members for their guidance and feedback and thesupport staff of UBC School of Engineering for answering all my questions.Finally, I express my most profound gratitude to my parents, friends andfamily for their unfailing support, encouragement and prayers throughoutthe process. Their unseen contributions helped shape this work, and I amvery grateful for it!xxviiDedicationTo my parents Mrs. and Mr. Sangita and Rajiv Gupta; fortheir unending support, encouragement and ever so positive out-look on life. To my brother Sanchit; for his kindness and unbrokenpursuit of knowledge. To my grandparents and my entire family;for their constant nourishment and affection. To my friend Ann;for her invigorating belief in me.And to the memory of my uncle Dr. Amit Gupta, for his unde-terred resolve, ceaseless strength and selfless care, truly the only“Dr. Gupta” my family would ever know.xxviiiChapter 1Introduction“The greatest challenge to any thinker is stating the problem in a waythat will allow a solution - Bertrand Russell”1.1 Motivation1.1.1 Nature inspired Artificial IntelligenceNature has always been an inspiration to science and technology leadingto great innovations. Right from the study of honey bees to create intelligentoptimization algorithms or the microscopy of gecko’s toe pad structure tobuild grippers for space repairs, the behavioural and structural adaptabilityin nature has led the era of technological leap-forwards. With its immenselyresourceful assembly of techniques, nature inspired learning now fuels thefield of Reinforcement Learning (RL) in Artificial Intelligence (AI). Learningby conditioning (interacting with the environment) is arguably the seed ofevery learning-based thought process and the natural way of acquiring, de-ciphering and connecting information for animals and more so humans. Theprocess is incredibly complicated for adults incorporating years of evolvingexperiential and sensory data with innate qualities and structured knowl-edge from literature, teachers, media or even the word of mouth. However,11.1. MOTIVATIONan infant’s learning process is significantly less involved and hence it may besimulated, artificially. An infant learns strictly through interaction with theenvironment by perceiving through their senses and responding by takingan action, often without clear a priori goals. This distinction presents analluring opportunity for AI agents to learn in an infant’s fashion, though inpractice this seemingly simple framework presents a great number of chal-lenges. Evidently, learning through conditioning alone cannot be expectedto learn compounded behaviors and patterns that are required to under-stand the complex intricacies of the real-world interactions. Augmentationof RL agents with sophisticated information and data acquisition methodsimprove the learning outcome, however still face numerous challenges. Forexample, the trade-off between exploration and exploitation, credit assign-ment, delayed rewards and partial observability of the scene: all leading tohighly inefficient and expensive learning. While the AI and RL research com-munity has made remarkable advances to alleviate some of these challengesin numerous domains, sample and computational efficiency of the processremains a significant obstacle to practical applicability of RL systems.1.1.2 The Elusive SolutionWe live in exciting times. The AI is at a level where a general refer-ence to “intelligence” in a day to day dialogue with a colleague, may causeambiguity. So much so, that it may be imperative to clarify “human” or“artificial”? In the recent years, trend in AI has been to tackle increasinglycomplex problems, often at the cost of exponentially rising training samplesand compute power. In a mere span of 6 years, from training of AlexNet21.1. MOTIVATION[1] in 2012 to AlphaZero [[2]] in 2018, the amount of compute used grewby 300,000 times! [3]. Humans on the other hand, do not have this lux-ury. One might wonder, how do we do it? How do we optimize and solvefor increasingly complex problems with limited compute and training sam-ples? One simple attribute could be the concept of chunking, where humansstore large information or knowledge in chunks of smaller information. Withpractise, smaller chunks of information or task-required actions become au-tomatic and allows us to focus on new or more complex chunks. The studyof these questions is rapidly gaining traction in AI research and dubbedas meta-reasoning; efficient use of computational resources (thinking aboutthinking) and meta-learning; efficient use of data (learning to learn) [4], [5].It could be argued however, that a key component of human intelligence thatallows for effortless execution of complex tasks is our ability to re-purposepast skills and intrinsically break down the complex problem into a seriesof sub-problems that we understand or can solve. In other words, insteadof trying to increase our computational power (albeit impossibly, of course),we make the problem simpler. In contrast to an infant learner describedin Section 1.1.1, an able human learner when trying to achieve a hard andcomplex goal, initiates by breaking down the complex problem into a seriesof simple problems. The concept is a momentous base, from which numer-ous methods, fields and algorithmic innovations have stemmed [6] and yetpotential is such that its studied in inspired and prescient works about AIas an elusive solution [7].31.2. REINFORCEMENT LEARNING1.2 Reinforcement LearningRL presents a poised methodology to deliver some of the beguilingpromises of AI. Ever since the advent of AI and Machine Learning (ML),the research community has pushed towards finding autonomous ways ofdecision-making to supplement the domains of supervised and unsuper-vised learning. Traditional automation and control strategies often requiremanually crafted rules and models of complex systems to find effective so-lutions. In contrast, model-free reinforcement learning framework allowsfor autonomous decision-making through an interactive and goal-seekingparadigm and has made significant strides in a variety of domains and appli-cations. Continuously evolving scope of the reinforcement learning domainwarrants its ever expanding application base with extensions to real-worldchallenges and solutions [8]. Much like machine and Deep Learning (DL),the field has immensely benefited from advancements in computer hardwareand graphical processors. In general, reinforcement learning employs trial-and-error to identify the best action for a given situation, which not onlypresents a strong framework to deal with current real-world requirements,but is also equipped to take on challenges that may arise in an aggressivelyadvancing data-driven world. Intuitively, this kind of approach, howevermuch effective, would require a lot of samples and data to effectively con-struct an explicit bridge between distinct scenarios and actions to be taken.This sample inefficiency is a bottleneck for many potential reinforcementlearning applications. With such a potential and rather limiting challenges,extensions to the field are evident. Researchers have continued to develop41.3. LEARNING WITH EASEand enhance the efficiency and performance of reinforcement learning algo-rithms [9] [10] [11]. In specific, exhaustive exploration of large state-spacesystems presents an analytically intractable challenge to the state-of-the-art Reinforcement Learning (RL) agents. The challenge significantly limitsthe applicability of the learning framework to simulated and comparativelysimpler environments. While a notable amount of research potential hasbeen expended towards advancing the exploration methodologies to allowfor efficient combinatorial optimization [12], [13], the challenge and hence ap-plication in real-world scenarios remain either inefficient or unsolved. Whileadvance exploration strategies continues to solve this challenge, a simplifiedoutlook on the problem itself is presented. A lof of the challenges associ-ated with training and combinatorial optimization would dissipate, if onlythe state-space could be made smaller. With this intuition, the concept ofExploitation of Abstract Symmetry in Environments (EASE) is introduced,to simplify learning and advance the generalization capabilities and sampleefficiency of the RL training paradigm.1.3 Learning with EASEHuman’s are faced with complex tasks in their day to day lives. Even thetasks that may seem trivial, requires huge amounts of processing, thinkingand thousands of primitive actions for completion. Consider a scenariowhere a human plans to drive to work from home. The plan for this scenariois always abstract e.g. walk to the garage, drive to work, park at work. It issimply not possible to plan each and every primitive action that is required51.3. LEARNING WITH EASEfor a human to finish this sequence of tasks. Such primitive actions areplanned and executed within each abstraction of task. If there is a roadclosure on the way to work, only that section of the plan needs to be changedand adapted. Walking to garage and looking for parking stays the same. Onthe other hand, while an autonomous agent may be able to plan primitiveactions for part of the tasks, there is no way for the agent to autonomouslyidentify these abstractions, or high-level decisions. It is unreasonable for anRL agent to be able to solve for the entire sequence of primitive actions,while considering the stochasticity and uncertainties in the environment e.g.road closure. This example perfectly summarizes the current state-of-the-art in RL training research. On one hand, there are end-to-end trainingmethods for complex objectives that attempts to plan for each primitiveaction which renders combinatorial optimization in state space infeasible andhighly inefficient. On the other hand, there exists manually designed stateabstractions and high-level options for multi-level learning e.g. HierarchicalReinforcement Learning (HRL) which learns actions over multiple temporalabstractions of the state space [14], [7], [15]. HRL has proven very effectivefor solving unstructured and complex environments, however the granularityof the manual abstraction poses a significant challenge. In addition, EASEcan be applied to efficiently solve within such hierarchies.Some recent extensions to the field like Curriculum Learning (CL), Learn-ing from Demonstrations (LfD) and predicate logic based training addressessome of these challenges by bringing in simple concepts from human learn-ing, logical reasoning and past research directions. However, the majority ofthe research trend is towards end-to-end training methods, often irrespective61.3. LEARNING WITH EASEof the need for additional computational and training samples. An abstrac-tion that hasn’t been discussed yet, is the ability of the human driver todrive anywhere and not just to work. In case the human driver decides todrive somewhere else, they won’t have to start learning tabula rasa. Instead,the driver would count on their “ability to drive” and look for abstract sim-ilarities that would allow them to drive other places. Inspired by our uniqueability of re-purposing previous knowledge and skills, this research aims toconceptually augment RL algorithms to improve their generalization abilityand sample efficiency for practical adaptability of RL trained agents. In par-ticular, the focus in this work is on robot control and navigation challenges,and the following questions motivate the presented work.− Could an abstract metric be devised in an RL setting, that would allowto break a complex task into simpler yet representative tasks?− Would learning on the simpler tasks be easier? If yes, what benefits itmay provide?− Can the simply trained agent now help to either learn or solve thecomplex task directly?Let’s elaborate on the driving thought process here to further highlightthe concept of EASE. From Figure 1.1, consider an Agent (car) at point A,learning to navigate to a certain point B while avoiding all obstacles (shownas green solid objects). The figure shows four different trajectories (τ1 to τ4)as a result of a trial-and-error-based interaction. τ1 shows the agent execut-ing random movements and finally reaching point B via random exploration.Similar to τ1, τ2 also executes a completely random exploration, but gets71.3. LEARNING WITH EASEstuck around a particular obstacle and never reaches the final destination.τ3 and τ4 shows the exploration via a predefined set of milestones M31,M32for τ3 and M41,M42,M43 for τ4. The milestones subtly guides the agenttowards the final goal that immensely helps in training and navigation. τ∗shows the optimal path between point A and point B that the agent canpotentially navigate.The introduction of milestones for better learning is base case of CL-based RL training methods. Let’s analyze the same problem from a dif-ferent perspective. Say a locally trained agent πlθ is trained to go frompoint A to milestone M41. This agent is expected to converge better andfaster than an agent trying to learn the whole trajectory at one go, becausethe state space that this agent has to traverse is significantly smaller thanthe entire space. Now, can this locally trained agent solve from M41 →M42,M42 → M43,M43 → B? Moreover, could the same agent, also solveA → M31,M31 → M32,M32 → B, knowing that’s a different trajectory al-together?. The multiple sections of this environment share the same spatio-temporal structure and are abstractly similar. In other words, the agenthas to learn “how to drive” and it can figure any path. One immediateconstraint is that the policy trained locally may not recognize the statefrom other regions of the environment. This representation requires map-ping function and under the said mapping function and simple heuristics,the agent can not only solve local trajectories, but also for any possiblemilestone goals, which an end-to-end agent trained in this fashion is unableto do. This abstract symmetry within the environments has not been ex-plored in RL literature. It is referred to as “abstract” because the different81.3. LEARNING WITH EASEsubsections of the trajectories are not exactly similar, but rather have aconceptual spatial configuration that is symmetrical. The conceptual sym-metrical configuration in this context refers to a small state-space section ofthe entire environment which shares its transition properties with the globalenvironment, but is easier to learn due to the small state-space. One exam-ple of such a setting could be the placement and shape of objects that aredifferent within each section, but navigation around those objects follow thesame principle. With the presented work, it is shown how simple mappingfunctions and locally trained agents can solve for large state-space problemsand complex tasks with simple heuristics in Chapter 3 and further use theconcept and locally trained agents as a generative model to train end-to-end global agents for highly dynamic environments in Chapter 4. Figure1.2 represents an example of conceptually similar regions extracted from acomplex global environment. The entire region is divided into conceptuallysimilar abstract local areas to facilitate training. The example is reintro-duced in detail in Chapter 3 with a deeper insight. Let’s revisit the drivingexample. Say, the agent wishes to drive a 1km × 1km, which leads to theexploration in 1km2 = 1, 000, 000m2 region. If the entire state space is di-vided in 4 parts, the exploration area reduces to 250m× 250m which leadsto 62, 500m2 which is only 1/16th of the original area. With each new di-mension, the state space grows exponentially posing a significant challengeto the convergence of traditional RL agents. The challenge is further exac-erbated for high dimensional action spaces as that adds to the number ofstate-action pairs that needs to be explored.91.4. OBJECTIVES AND CONTRIBUTIONS1.4 Objectives and ContributionsThe proposed concept presents a simplified outlook on the RL problem.While the research community tries to improve RL algorithms to cater tocomplex problems, the presented work simplifies the problem itself. Muchlike how a human driver, learning to drive in a test area, can drive anywhereelse as long as the same rules of driving are followed. Learning in the testarea can be compared to learning in “local” or “na¨ıve” environment, whiledriving anywhere else represents the ability of the local agent to either inferor help train an agent in the global environment. When compared to theresearch trend, the presented approach may be different, but the algorithmicinnovation addresses the same issues. The objectives of the presented workare highlighted below:− Theorize and study the effects of exploitation of abstract symmetryin environments (EASE), to present a simplified outlook on complexproblems− Utilize EASE to improve the sample efficiency of traditional reinforce-ment learning algorithms− Utilize EASE to improve generalization capabilities of traditional re-inforcement learning algorithmsWith the novel concept of EASE, numerous test case scenarios are pre-sented to demonstrate its efficacy and flexibility with different methods.Often, simulations used for benchmarking RL control agents are unable tocapture complexities of real-world environments. The simulations are aug-101.4. OBJECTIVES AND CONTRIBUTIONSmented to present implementations on highly dynamic environments, whichpresent a challenge to traditional RL algorithms, to further display the po-tential of EASE. The contributions of this thesis can be summarized below:− A novel concept to augment reinforcement learning agent training toexploit abstract symmetries in an environment. The concept is re-ferred to as EASE (Exploitation of Abstract Symmetry in Environ-ment) (Chapter 3, Section 3.1)− Amalgamation of EASE with simple heuristics-based method to solvelarge and dynamic state spaces (Chapter 3, Section 3.2)− Amalgamation of EASE with learning from demonstration-based rein-forcement learning to aid global training that improves generalizationcapabilities of the agent (4, Section 4.1)− A novel algorithm with EASE and abstraction in state space to aidglobal training that improves sample efficiency 4, Section 4.2)Perception, planning and control, together, forms the base case for anyautonomous and intelligent agent. Perception identifies meaningful featuresand information from raw sensor data which in turn helps the planning andcontrol process and further allows for autonomous decision-making. Inspiredfrom the natural flow of perceptual decision-making, the author’s researchfocus was initially on efficient and real time machine perception, rangingfrom autonomous unmanned quadcopter landing [16], [17] to autonomouswrinkle detection on preformed dry carbon-fibre fabrics on double curvedsurfaces [18], [19], [20]. These exciting works further lead to the develop-111.4. OBJECTIVES AND CONTRIBUTIONSment of efficient computer vision algorithms for fast and efficient objectdetection in 3D space [21]. The amalgamation of the three key componentslead to the development of an autonomous vehicle teaching tool [22] andstudy and survey of applications like human-robot collaboration methods[23]. The features extracted with deep representational networks often learnan abstract representation, which encourages the question: how can thesefeatures be used to take decisions? This is where the research focus turnsto efficient reinforcement learning methods [24] [25]. The proposed conceptand algorithms in this thesis are presented with applications on various usecases.121.4. OBJECTIVES AND CONTRIBUTIONSFigure 1.1: A car agent at point A is trained to find the optimal path betweenthe points A and B. The objective is to efficiently navigate and avoid obsta-cles on the way. The trajectories τ1 and τ2 shows random exploration withreinforcement learning, while trajectories τ3 and τ4 represents reinforcementlearning guided through some predefined milestonesM31,M32,M41,M42 andM43. τ∗ represents the optimal path between the two points.131.4. OBJECTIVES AND CONTRIBUTIONSFigure 1.2: The gray shaded area represents the local environment as a partof the global environment. For the local environment, the reset state forω0 is bounded between [0, π/2] or [−3π/2, 0] depending on the direction ofmeasure. There are no limitations on ω1. The target is denoted by the reddot, with its velocity vector denoted by the black broken arrow. The targetis also bounded in region A for local training. The vertical lines representthe partition over the global environment, generating 4 abstractly similarregions A,B,C and D14Chapter 2Background2.1 Reinforcement Learning FrameworkRL features a powerful framework for goal seeking agents to interactivelylearn from their environments by optimizing a numerical reward signal [26].The unique interactive and goal-seeking nature of the learning makes ita unique paradigm in the machine learning family. The RL’s reliance onhuge and quality datasets is usually addressed with the use of high-fidelitysimulations which render to be an inexpensive data collection method. Inrecent years, RL has made massive advancements demonstrating potentialin a broad range of virtual, simulated and real-world applications whileaccomplishing exceptional performance in disciplines like robotics, finance,marketing, web systems and gaming industries [27]. Some noteworthy ac-complishments of the field boasts applied agents that beat the world Gochampion [2], competed and won against top-level professional Dota andStarCraft players [28] [29], inferred and executed impressively complicatedrobotic tasks in simulation and reality [30] [31] [32], learned to play multipleAtari games from pixels as inputs alone and achieved ”super human” perfor-mance on 49 of these games [33] [34]. The advances further strengthen thefield’s applicability to real-world problems with real-time demands. Figure152.1. REINFORCEMENT LEARNING FRAMEWORKFigure 2.1: Reinforcement Learning General Framework2.1 shows the general framework of RL algorithms as a cyclic communica-tion between the Agent and the Environment. The agent takes an actionat at time t to manipulate the environment, when presented with an Ob-servation ot. This results in the environment responding with the changedobservation, or the observation at the next time step ot+1 and a correspond-ing reward signal rt. The state st of the system is a function of the obtainedobservation ot which identifies and communicates meaningful, low-level andcoherent information to the agent. Apart from the agent and the environ-ment, a general RL algorithm consists of the following primary elements:i) a policy - acting function that maps state inputs to action, ii) a rewardfunction - calculates the reward for a state, action or state-action pair, iii) avalue function - estimates the maximum reward possible from a given stateand optionally, iv) a model - model of the environment.For RL framework to effectively learn, it is imperative to construct a re-lationship between different action-based state transitions. This relation-ship in reinforcement learning, is formulated as a Markov Decision Process(MDP). At any given time, the agent chooses an action at, available atthe current state st, leading to the next state st+1 and an associated re-ward rt+1 =R(st, st+1). The transition from the current state st to next162.1. REINFORCEMENT LEARNING FRAMEWORKstate st+1 is guided by the transition probability function of the systemP (st, st+1). As all the states of this nature follow the Markov Property(MP), the MDP chain can be represented as a 4-tuple M= {S, A, P ,R}, where S represents the state space, A represents the Action space,P (s,s′) = P (s′ = st+1|s = st, a = at) is the transition probability of state sto state s′ and r(s, s′) = R(s′ = st+1|s = st, a = at) is the immediate rewardof the transition from s to s′. Different algorithms add different variables tothis tuple as per their usage and version of the Markov chain, while keepingthe base case same. The representation of a state s at a particular time tdoes not restrict the application to obvious Markov chains only. Even stateswhere the current information may depend on previous observations (likemotion prediction in a video where multiple frames may be required to theestimate current motion vector) can be transformed into a Markov processfor RL training to solve. The state may not necessarily be obtained from asingle observation and can be extracted from a series of observations defin-ing the current process.The objective of the policy π, parameterized by θ, πθ, is to maximize thecumulative reward over the entire run of a trial. To avoid situations withvery high or infinite rewards and ensure statistical convergence, the futurepredicted rewards are discounted by a discount factor γ. Thus the objectiveof the process can be defined to maximize the equationEst+1∼Pat (st,st+1)[∞∑t=0γtRat(st, st+1)]where at = πθ(st) or πθ(at|st) and 0 ≤ γ ≤ 1.At any given time, the agent either exploits the learned behavior by172.1. REINFORCEMENT LEARNING FRAMEWORKpicking an action with expected highest reward or explores by picking sub-optimal action in pursuit of a higher reward. This dilemma is termed asexploration-exploitation tradeoff. The tradeoff is a well studied challenge inRL and multiple works have surfaced contributing efficient exploration tech-niques to improve sample efficiency. [35] extended count-based explorationtechniques for high-dimensional continuous state space systems. Prior tothis work, count-based exploration techniques were limited to low dimen-sional tabular discrete reinforcement learning algorithms. Work presentedin [36] unifies count-based and intrinsic motivation while [37] learns a hier-archy for intrinsic motivation to enable efficient exploration while learning.Another interesting conceptual design was proposed in [38] where the ex-ploration is driven by curiosity. The work formulates curiosity as the errorin the agent’s ability to predict the outcome of produced actions. The workwas applied to high dimensional continuous state space systems and studiedin detail on multiple environments and benchmarks by [39].For inexpensive data collection, most training processes acquire datafrom simulations. A simulation along with an obvious reward signal canalso be exploited to receive multiple pseudo signals corresponding to thetask being learned. The work in [40] proposed to maximize such pseudosignals to learn a faster representation of the actual task. The work showsphenomenal result but significantly increases the computational complex-ity of the training process. Multiple works have been proposed to betterthe optimization process irrespective of the training, specific to large non-linear policies like neural networks. The works include Trust Region PolicyOptimization (TRPO) [41], Proximal Policy Optimization (PPO) [42] and182.1. REINFORCEMENT LEARNING FRAMEWORKscalable trust region methods [43]. [44] proposes to maximize reward whilemaximizing the entropy of the system for stabilized training process.Some works suggest re-labelling strategies that involves modifying theexperience replay buffers in the training process leading to competitive ex-ploration [45], prioritized experience generation [46] and reward re-labellingin hindsight [47]. Finally, some of the recent works proposed in the field fo-cus on practical implementation of RL system with integration of traditionaland newly introduced algorithms [48], surprise based systems [49] [50] alongwith hierarchical information-primitive networks [51]. Researchers have alsodiscussed the fields compatibility with symbolic planning for interpretabilityof tasks [52] and employ reverse reinforcement learning to execute tasks onthe basis of mere language-based instructions [53].2.1.1 Robotics and Reinforcement LearningRobotics is one of the primary and a classical applied field for reinforce-ment learning agents and algorithms. The use of RL algorithms for roboticenvironments eliminates the need of explicit programming of every task andallows for truly adaptive robots potentially optimizing actions for multiplegoals and tasks [54] [8]. Like games, robot simulation also presents multiplechallenges relating to the state, action and input dimensionality of differ-ent tasks and systems. In addition to these, there are challenges associatedwith improper modeling of the robotic system (low-fidelity simulations, in-complete system dynamics, model uncertainties) and the unavailability of adirect reward signal [55]. Moreover, there is an added challenge of transfer-ring the learnt policies from simulation to the real world, often dubbed as192.2. EASE PARADIGMSsim-to-real [56] [57] [58] [59]. Many works have surfaced to alleviate thesechallenges, both in simulation and real-world. For sample efficiency, rewardre-labelling strategies like Hindsight Experience Replay (HER) [47], Com-petitive Experience Replay (CER) [45] and efficient exploratory techniqueslike intrinsic motivation [60] [61], curiosity [62] [63] [39] and surprise [64] [65][66] based exploration have been successfully demonstrated. It also inspiredextensions to algorithmic families like Actor-Critic methods [67] [68], Ad-vantage Actor-Critic (A2C), Asynchronous A2C (A3C) [69], skill, optionsdiscovery and auxiliary task methods [70] [71] [40] [72] for large scale roboticstasks.2.2 EASE paradigmsIn this section, other paradigms of reinforcement learning extensions arereferred, which can be related to the proposed novel concept of EASE. Let’srefer to the whole environment to be solved as the “global” environment withthe state space S and the restricted environment as “local” with the statespace S l, where S l ⊂ S. EASE trains a na¨ıve agent in a local abstractly sim-ilar version of the global environment, to help accelerate training in globalenvironment. The concept and considerations for setting the state spaces isdiscussed in detail in Chapter 3, but is provided as a general case in thissection. The state variables sampled from the two state spaces are denotedas s ∈ S and sl∈ S l, and the agents trained as the global and local or na¨ıveagents for the global and local environments, respectively. EASE exploitsthe fact that under certain abstractly symmetrical restrictions, a na¨ıve agent202.2. EASE PARADIGMStrained in a local environment may be able to generate high quality optimaldata to either help train a global agent or solve a complex task using simpleheuristics-based planning at inference time. This concept of exploiting ab-stract symmetries for aiding RL training is an unexplored area. A notabledesign key here is to ensure that both the local and global environmentshave the same action space A. The fundamental concept of EASE imple-mentation can be compared to the mechanisms discussed in the followingsections.2.2.1 Hierarchical Reinforcement LearningThe heuristics-based planning algorithm discussed in Section 3.2, at in-ference time, defines a sequence of high-level actions for the local agent tosolve. Besides a heuristic-based approach, the determination of high-levelactions can further be automated using a simple rule-based scheduling al-gorithm or another auxiliary agent. Here, the high-level actions can becompared to temporally abstract options defined for the HRL agent train-ing. Options are high-dimensional actions that can be learned along differenttemporal abstractions of the state space, with the final agent trained in se-ries to perform primitive actions in A. While the final agent in series iscomparable to the local agent, the action space for different abstractions ofthe state-space varies in the HRL setting and is often defined by an expert.Moreover, learning options enforces training over a semi-MDP environment,whereas EASE employs MDPs with symmetry-based similarities. In bothcases, a high-level goal is passed on to a local agent to solve. End-to-endtraining of a global agent using the local agent, on the other hand (Chapter212.2. EASE PARADIGMS4) does not follow the hierarchical structure.2.2.2 Transfer LearningThe concept of training in a source environment for alleviating chal-lenges in a target environment is the core principle behind Transfer Learning(Transfer Learning (TL)). In the case of EASE, the local agent can be con-sidered as the source, while the global agent acts like the target. The sourceagent is often re-trained in the target environment with the expectationof improving some metric of performance e.g., sample efficiency, general-ization capabilities, asymptotic performance. From an RL perspective, there-training in source environment benefits from the previously learned infor-mation and allow for meaningful and efficient exploration. In comparison,EASE does not require any re-training of the local agent. In fact, heuristicscould allow for a zero-shot utilization of the template agent in the entiretarget environment. In Chapter 4, EASE is presented as a generative modelthat comes close to TL setting, however instead of re-training the local agent,a separate global agent is trained with data generated by the local agent.2.2.3 Curriculum LearningCL augments training by initiating the process on a form of a simplerenvironment and gradually moving towards the complex environment. InEASE, as the local environment is an abstract version of the global envi-ronment, the training process creates a natural curriculum, however dis-continuous. Different from usual CL practices, EASE implementation doesnot require any kind of schedule over task-specific parameters. Given the222.3. GUIDING SAMPLES AND MDP HOMOMORPHISMclose nature of Curriculum-based RL (CRL) methods with EASE, a detailedliterature review of CRL methods is presented in Section 2.4.2.3 Guiding samples and MDP HomomorphismIn Chapter 4 the local agent is utilised to generate optimal datapointsand trajectories to train the global agent. A close resemblance to this workis presented in [73] where guiding samples present high reward optimal tra-jectories which are then employed for efficient training. Guiding samples inGuided Policy Search (GPS) and initial state slices in [74] are both anal-ogous to the trained local agent. In GPS, guiding samples are generatedthrough adaptive Linear Quadratic Control (LQR) which requires a modelof the system. In contrast, a local agent training can be accomplished withsome restrictions on the global agent state-space and is significantly easierto implement. On the other hand, initial-state slices in [73] requires anagent training in each slice which is then distilled against a central policy.In EASE, a single local agent is equipped to operate in all slices with simpleabstractions and state-space mappings. The work in [75], which is concep-tually similar to [73], is designed for multi-task domains. [76] shows thatdistilling against a central policy significantly improves performance. [31]utilises an approach where an RL agent is trained with a mix of demonstra-tions from human experts and environment interaction data. The proposedapproach does not require demonstrations from humans, but rather gener-ates high quality and high accuracy data from the local agent itself. Theproposed approach can be studied as a direct extension of symmetry-reduced232.4. CURRICULUM-BASED REINFORCEMENT LEARNINGMDP’s. Multiple prior works attempt to minimize models based on symme-tries. They often try to find homomorphic features within MDPs to improvethe sample efficiency of the training process. The proposed methodology,on the other hand, utilizes the concept of abstract symmetry and relies onthe expertise of a locally trained agent that can solve for a small part of thestate space even when it’s not exactly symmetrical. The contribution of theproposed work is not to minimize model based on symmetries, but rather touse abstract spatial symmetries to train local agents that can further helpdemonstrate optimal actions for larger spaces. The proposed method by-passes the need for context restricted MDP’s or MDP model minimization.Simple group transformation functions in cartesian coordinate are used toobtain local state equivalents of global observations, which can be solvedby the local agent. Due to the abstract nature of symmetries, no explicitgroup transformation functions have to be designed or learned. The datais mapped between different partitions of S with trivial euclidean geometrictransformations or simple rotations and translations.2.4 Curriculum-based Reinforcement LearningOne of the salient extension to the field of RL is CL [77], which aug-ments the training data among other attributes of the learning frameworkfor the learning agent to effectively converge. It’s similar to curriculumdesigns for academic purposes, where a complicated topic or subject is bro-ken down into multiple sub-parts for a student to understand and buildupon. Curriculum is essentially an ordered set of non-uniformly sampled242.4. CURRICULUM-BASED REINFORCEMENT LEARNINGtraining data (mini-batches in some cases) that allows for simple or easiertasks to be presented and learnt on first, while gradually building it’s way toharder tasks (increasing complexity of tasks/training data). This idea hasimmensely helped reinforcement learning researchers, methodologies and al-gorithms making it possible to learn extremely complicated tasks, wherestandalone reinforcement learning agents either fail or perform poorly, whileimproving sample efficiency.A more direct comparison to CRL methods can be made with the HRL,presented here as a brief digression. Both learning paradigms help scaleRL to sizeable and complicated state-action spaces by structuring knowl-edge and task requirements. The difference, however, lies in their executionand computation strategy. HRL, internally, comprises of multiple layers ofcontrol, each responsible to act for a different level of temporal abstraction.CRL on the other hand, is more progressive, building on a comparativelyeasier base task with each iteration. The segregation of tasks can range frommanually designed to autonomous [14] [78] [79]. In general, a hierarchy inHRL tasks can be seen as a curriculum, but a curriculum might not alwaysbe hierarchical e.g. consider a robotic arm tasked to stack multiple rings indecreasing order of their radii (like a tower of hanoi) as shown in Figure 2.2.Different steps required for such a task (Figure 2.2(a)) are quite straightfor-ward; identify different rings, get a comparative sense of their radii (eitherabsolute or relative) (Figure 2.2(b)), pick and place the largest ring at thedesignated spot (Figure 2.2(c)), identify and locate the second-largest ring,pick and place the identified ring on top of the previously placed ring (Fig-252.4. CURRICULUM-BASED REINFORCEMENT LEARNINGure 2.2(e1)) and continue to execute the steps in a similar fashion (Figure2.2(e2)) until all rings are placed (Figure 2.2(f)). The steps describe anobvious hierarchy to the process, do a certain task first, then proceed onto second. Also, with the given granularity of broken down tasks, one canassume to construct multiple policies, e.g. a policy to identify and locate thering, a second policy to control the robotic arm, a third policy to grasp thering and finally a fourth policy to place it at the required spot. All policiestogether cater to the final objective; placing the ring and constructing thetower. The approach is also a curriculum as it breaks down the larger taskinto easy or attainable tasks. A different approach would be to have a singlepolicy that first, in an end-to-end fashion, learns to stack one ring (Figure2.2(c)), then stack two rings (Figure 2.2(d1) and Figure 2.2(d2)) and even-tually learn to stack all the rings (Figure 2.2(f)). This approach is strictlycurriculum and lacks the hierarchical component. On the other other hand,if the robot could learn how to stack one ring, and then use that informa-tion to either stack or learn to stack other rings; forms the base case of theproposed EASE concept, where abstract symmetries present in the processof stacking multiple rings with slightly changing radii, are exploited.With CRL, it is imperative to design and uphold a good curriculum, asa badly designed curriculum can hamper the learning instead of helping it[25], [80]. The following section breaks down the field of curriculum-basedreinforcement learning in terms of different methods, proposed works andpresents a discussion on their applications and merits.262.4. CURRICULUM-BASED REINFORCEMENT LEARNINGFigure 2.2: The Figure shows a modified tower of hanoi problem, where therobotic arm is expected to identify and locate different sized rings and stackthem on the tower, in the decreasing order of their radii. It demonstrates twopossible curriculum and their progress towards the final goal, with Figures(a), (b) and (c) representing common steps between the two curriculum,(d1), (d2) as curriculum 1, (e1), (e2) as curriculum 2 with a hierarchicalcomponent and (f) as the final finished task.272.4. CURRICULUM-BASED REINFORCEMENT LEARNING2.4.1 Curriculum through Teacher-Student interactionStudent and tutoring agents in response to strategic student problemhave been previously seen in the robotics community [81][82], but it’s ex-tension and integration with reinforcement learning is rather unexplored.Initial works on teacher-student framework in reinforcement learning [83][84] [85] focused on the teacher advising the student on how to take actionsor what actions to take next, around a conceptual but strict, budget. Thebudget is defined as the number of times the teacher can suggest or ask foran action [85]. In a curriculum learning setting, formally introduced in [86],Teacher-Student Curriculum Learning (TSCL) defines two communicatingcomponents: a Teacher and a Student. The student component is the agentbeing trained on a certain RL task, while the teacher is a pseudo-explicit su-pervisor. Within the same family of implementations, the teacher can rangefrom a simple mathematical guiding function to an independent Neural Net-work (NN) based approximator. Teacher decides on a task that the studentshould learn or practice, based on the performance (or the probability ofsuccess) of the student. The task as well as the communication between thestudent and the teacher, is formulated as a partially observable Markov De-cision Process (Partially Observable Markov Decision Process (POMDP)),with an action at representing the task chosen by the teacher for the studentto learn, observation ot being the performance of the student on the selectedtask and reward Rt as the difference between the previous and current im-mediate episodic rewards rt−rt−1. In [86], the performance measure for thestudent in reinforcement learning setting was taken to be the total rewardfor an episode. The setup requires a set of tasks that T , from which the282.4. CURRICULUM-BASED REINFORCEMENT LEARNINGteacher samples a sub-task Ta for the student to learn. As student’s perfor-mance improves on the sub-task Ta (increasing slope of the learning curve- reward vs iterations), teacher greedily selects the same sub-task. As thelearning curve starts to flatten, teacher moves on to the next sub-task Tb,while sometimes re-iterating on sub-task Ta to avoid forgetting. The workcombined the process with Botzmann distribution for sampling tasksp(a) =eQt(a)/τ∑Ni=1 eQt(i)/τwhere N is the number of sub-tasks in task set T , Q is the Q-values asso-ciated with each task sub-set and τ is the temperature of the Boltzmanndistribution. The work was configured and applied to a gaming Minecraftenvironment (Project Malmo, Microsoft) with varying doors, bridges andnumber of rooms (massive exploration space), and the curriculum tasksquantified based on the number of rooms and their connectivity (door orbridge).The above described process generates an automatic curriculum like in[87], but requires a human defined set of tasks, from which the teacher sam-ples. In addition, the method was proposed and applied for a discrete stateand action space environment. [88] extends the work to a continuous spacewith varying generalized environments. Traditionally, such varying environ-ments are Procedurally Generated (PG) and have a massive stronghold inthe gaming industry (Designing of Maps, checkpoints, other game-specificattributes), among other applications [89]. The teacher in [88], samples adistribution for the environment, instead of sub-tasks, and maintains histor-ical data for each interaction to compute final student returns (competence).292.4. CURRICULUM-BASED REINFORCEMENT LEARNINGThe work proposes to evaluate the student on Absolute Learning Progress(ALP), compared to the single task return in [86], with sampling basedon Gausian Mixture Models (GMM), and thus the algorithms ALP-GMM.The experiments on parameterized bipedal environment [90], shows fasterlearning curve and improved overall performance of reinforcement learningagents, with the use of curriculum.A not so distant field of RL, which can be directly associated to theteacher-student setup is master-slave based learning frameworks. Master-slave [91] framework, although very different in terms of computation andapplication, can be conceptually equated to the teacher-student setup, wherethe master acts as a teacher and designs workload for slave or student agents.Master-slave architecture has shown potential in Multi-Agent Deep Rein-forcement Learning [91] settings.2.4.2 Curriculum through intermediate goalsA goal can be loosely referred to as a desired final state of an environ-ment. In other words, the objective of the learning agent is to be able tomanipulate the environment to reach a goal state. For sparsely rewardedenvironments, setting a complex goal state would almost always lead to ex-tremely inefficient exploration, significantly affecting the learning progress.The challenges intensify while scaling this process for multiple tasks anddynamic goals. A properly developed curriculum for such a setting can notonly improve sample efficiency (by limiting exploration towards the generaldirection of goals), but also help the agent learn a range of tasks. An obvious302.4. CURRICULUM-BASED REINFORCEMENT LEARNINGapplication of this type of system can be seen in adaptive robotics wherea robotic software agent learns to navigate complex environments or learnmultiple tasks.[92] introduced a Goal Generative Adversarial Network (Goal GAN) asa variant to GAN [93]. A GAN network is typically used for generating newdata which is statistically similar to the training data. It comprises of agenerative and a discriminative network, that is responsible for generationand evaluation of data, respectively. Consider a set of goals g ∈ G, whereeach goal g maps directly to a particular state sg of the environment. Tooptimize and ensure learning on multiple tasks, [92] defines a sparse rewardfunction, that is common for all the goals, rg(st, at, st+1 = 1{st+1 ∈ Sg}for a time t and an indicator function 1. The total expected reward over afinite episodic time T and a policy π is given by:.Rg(π) = Eπ(.|st,g)1{∃t ∈ [1...T ] : st ∈ Sg}The agent, then tries to optimize the following objective.π∗(at|st, g) = argmaxπEg∼pg(.)Rg(π)A Goal GAN is trained on some initially labelled goal states and in turn,generates other possible goal states at similar difficulty levels for the agentto train. The start states for Goal GAN are a set of goals called Goals of In-termediate Difficulty, which ensure that the goal states are neither too hardnor too easy for the agent to achieve. This is implemented by selecting goalstates for which the expected reward by following the policy is greater than312.4. CURRICULUM-BASED REINFORCEMENT LEARNINGthe minumum reward possible, but lesser than the largest reward allowed.The method was successfully applied and demonstrated on different roboticnavigation and manipulation tasks [92].Another interesting approach to similar reinforcement learning problemsis the use of reverse curriculum learning [94]. Instead of picking goal statesfor a set of start or initial states, reverse curriculum methodology generatesa curriculum of start states for a fixed goal state. This technique is speciallyuseful for problems with narrow optimal trajectory spaces, like attaching agear to an axle or putting a ring on a peg. With every iteration, the environ-ment resets to a nearby start state that can be assumed to find the goal staterather easily and obtain the positive reward. This encourages the agent tolook for this positive reward, even when starting from further away stages.Once learnt from the picked start states, the agent takes brownian stepsin random directions to identify other start states with increased complex-ity. Similar to goal generation, reverse curriculum method also picks statesfrom where attaining the goal would be neither too hard nor too easy. Inother words, states from which goal can be sometimes reached are labelledas good start states. The methodology was applied and proven on roboticsimulations trying to insert a ring on a peg, guiding a key into a keyholeand point mass robotic navigational mazes.Similar to reverse curriculum generation, [95] introduces BaRC algorithmthat utilized the capacity of the agent to be able to calculate the backwardreachability set of states based on an approximate physical model of the sys-322.4. CURRICULUM-BASED REINFORCEMENT LEARNINGtem. The algorithm requires decomposition methods and software tools toefficiently calculate the backward reachability from intermediate states andmodify the start state distribution. The use of a low-fidelity physical modelwith accelerated software tools for decomposition ensures comparable train-ing times to RL agents with no curriculum. In contrast, a different work [96]uses task descriptors to learn an inverse model of the environment under acontext C to identify a curriculum of goals through intrinsic motivation, inan unsupervised fashion.[97] presents a goal masking methodology to generate an automatic cur-riculum for continuous environments. A goal mask M = {1, 0}n, whenapplied to a goal g ∈ Rn with attributes n, gives a masked goal gmt for anobservation ot in time t.gmt = g ⊙m+ ot ⊙ (−m+ 1)In other words, the masked goal basically is a reduced version of the intendedgoal, where the agent is rewarded even if it reaches the intended goal insome aspects, while staying the same in others e.g. for a goal defined as apoint in a 3D coordinate plane, a masked goal can be a 2D projection ofthe 3D point on any one of the planes with respect to the agent’s currentposition. To effectively scale this approach to goals in larger dimensions, thework assumes a conditional dependence of successful trials among differentsub-goals derived by differently masking the intended goal. The techniquewas demonstrated on robotic manipulation environments of FetchPush andFetchPickandPlace from OpenAI Gym [90]. Similar to goal masking, an332.4. CURRICULUM-BASED REINFORCEMENT LEARNINGintended goal can vary in terms of the resolution of required for the agentto achieve it. This resolution can form a type of curriculum, presented in[98] with a tuning factor based on competence progress. This work waspresented on a continuous Reacher environment.2.4.3 Curriculum through Self-PlaySelf-play, in itself, is a very interesting approach typically catered to-wards gaming systems and agents trying to master these systems. Tra-ditional methods of creating game-playing agents exploit extreme domain-specific knowledge and hand crafted techniques to master a particular game.Through self-play and RL, [99] presented a general game-playing algorithmicagent, trained with random play and no domain-specific knowledge (tabularasa), that eventually went on to defeat the world leading programs in thegames of chess and shogi. Use of self-play to develop curriculum howeverstays unexplored, though with a lot of potential. [100] presents an inter-esting self-play technique combined with intrinsic motivation to generatecurriculum to better explore large spaces. This simple yet elegant techniqueinvolves an agent with two units Alice and Bob. Alice starts from an initialstate s0 and executes a series of random actions to reach st, which formu-lates the task for Bob. Bob is then expected to either trace it’s way back tos0 from st for reversible environments or reach st from s0 for environmentsthat allow resetting. The rewards for these units is what allows for efficientexploration. If RA and RB are rewards associated with Alice and Bob re-spectively, then RB = −γtB and RA = γmax(0, tB − tA), where tA and tBdenotes the time taken by Alice and Bob to complete the task respectively342.4. CURRICULUM-BASED REINFORCEMENT LEARNINGand γ ensures a comparable score between the intrinsically reward and thetarget task reward. Alice is rewarded to find tasks where Bob takes longer,but if Bob fails at a certain task then the time for Bob tB is set to betMAX − tA, which encourages Alice to find hard tasks, bot not hard enoughfor Bob to fail. This naturally creates a curriculum between Alice and Bobwhere Alice soon learns to find increasing harder tasks for Bob to achieve.Bob, then is executed on the final target task. The approach was success-fully implemented on Mazebase (Light Key)[101], RLLab (SwimmerGatherand Mountain Car) [11] and StarCraft environments.2.4.4 Curriculum through Attributes and DistillationThis section combines the works where curriculum is defined as somesort of an ordered set of an intrinsic attribute to the training frameworkor as a distillation of policies. Complexity of different training objectivesor tasks can be an internal parameter of an environment e.g. a curriculumbased on number of user slots in a task completion dialogue managementservice [102] or an environment consisting of multiple obstacles that can beexploited by an operator to define a curriculum based on the obstacle ortask attributes (big/small, sparse/dense) giving rise to a curriculum [103][104].[105] presents a Distributed Proximal Policy Optimization (DistributedPPO) algorithm and argues that the presence of an implicit curriculum dueto varying complexities of tasks in rich environments, significantly enhancesthe training performance of DPPO. DPPO is a distributed extension to PPOwhich averages the gradients and data over all the workers. The algorithms352.4. CURRICULUM-BASED REINFORCEMENT LEARNINGuses K-step windows on value functions to bootstrap advantage estimatesAt in time t.Aˆt =K∑i=1γi−1rt+i + γK−1V (st+K)− V (st)The algorithm was tested, reported and benchmarked on continuousOpenAI gym environments like Planar Walker, Quadruped and Humanoid.Other extension to the variation to PPO along with other traditional RLalgorithms is presented as an experiment in a multi-agent domain by [106].Disintegrating a hard task to generate smaller sub-tasks is the most intu-itive curriculum design for any RL application [107]. [108] presents a similardisintegration to develop a multi-level curriculum to aid decision making forautonomous vehicles. The algorithm uses Value function estimates to pre-dict the complexity level of a given task and prioritizes tasks with higherestimates. A different approach [109] develops an automatic curriculum byrewarding temporally rare pre-defined events to explore different objectivesfor a game of Doom. The frequency of the occurrence of any event (byextension - say a sub-task) defines the reward for that event. Less visitedevents (or accomplished sub-tasks) are rewarded more, to encourage theagent to explore these rare events over other events. This creates a naturalcurriculum in the exploration domain. Previously mentioned HER [47] isanother interesting after-the-fact reward relabelling strategy that allows forexperience replay transition modifications and better sample efficiency byconverting failed scenarios into successful ones. [110] presents a curriculum-guided HER that conducts non-uniform prioritized sampling of transitionsfrom HER buffer. The modified (reward-relabelled) transitions are sampled362.4. CURRICULUM-BASED REINFORCEMENT LEARNINGto encourage proximity to the intended goal as well as well as the diversity ofpseudo-goals (failed scenarios relabelled to successful trials). The transitionsin HER buffer are prioritized based on a similarity function, that provides amathematical measure of the transitions proximity and diversity, while alsobiasing proximity over diversity at the later stages of training (exploration-exploitation). The method was demonstrated on complex robotic simula-tions comprising of experiments with OpenAI [90] environments; FetchReachand HandManipulate with Block, Egg and Pen. A different approach wouldbe to prioritize the replay transitions as a function of the transitions tem-poral error and model age [111]. The work introduces a method to evaluatethe complexity of the transition with the use of coverage penalty. This com-plexity then further helps in developing a priority function for the storedtransitions. The approach is analogous to EASE data generation in Chap-ter 4, where tuple information from local agent is re-labeled to optimize theglobal agent.Based on a similar principal [112] proposed the use of Object-Oriented(OO) MDPs to develop an OO curriculum. The work also proposes a simi-larity function to identify tasks (via task descriptors) that are similar to thetarget tasks, encourages exploration by picking tasks dissimilar to previouslytrained upon tasks and have a smaller state space. The work is presentedas an extension to [113] in terms of the evaluation metric. [113] proposed toconstruct a curriculum graph, instead of a linear ordered list of attributes.The evaluative metric of transfer potential, helps the framework identify theeffect of the learning of a certain task by a certain agent on the overall targetlearning on target task.372.5. ALGORITHMS[114] presents a Mix & Match framework which, in essence, combinesmultiple policies in a single agent and quantifies these policies in terms oftheir complexities. A mix of policies, say π1 to πK , for K different policiesconstructs a single agent where the objective of policies π1 to πK−1 is tobe able to train πK . The policies are sampled based on a probability massfunction parameterized by a categorical random variable. The work intro-duces a distillation-like cost D to enable a collective gain and learning alongwith the knowledge transfer and RL losses.Lmm(θ) =K∑i,j=1D(πi(·|·, θi), πj(·|·, θj), i, j, α)where mm represents the mix& match component of the loss function, poli-cies π are parameterized by weights θ and α corresponds to the tuningfactor for the categorical random variable and selection of policies. Ideally,towards the end of a successful training iteration, αK is expected to convergeto unity, which represents a trained final policy πK . Another policy distilla-tion method presented in [115] combines progressive learning with transferlearning to enable skill acquisition for different tasks and domains.2.5 AlgorithmsThis section describes the traditional RL algorithms that are augmentedin the presented work. Two algorithms are focused - Deep Q-Learning (DQN- algorithm 1) for discrete action spaces and Deep Deterministic Policy Gra-dient (DDPG - algorithm 2) for continuous domains. Both the algorithmswork on training a Qϕ function that maps a state-action pair to expectedreward Q : S × A → R and learning a policy π. The Q function outputs382.5. ALGORITHMSthe expected value of reward estimated for a state s after taking an actiona and following the policy π thereafter. The function, approximated withdeep neural network models, is represented as a Bellman equation, oftenupdated with value iteration methods. The algorithms fall under the familyof off-policy methods where the policy that takes an action can be differ-ent from the policy that collects the data. This makes the training robustagainst the source of data and allows for the opportunity to improve train-ing performance through data generated and acquired through innovativepolicies and concepts e.g. EASE. Data for any NN training is assumed tobe Independent and Identically Distributed (i.i.d.) to make sure that sam-ples obtained from this data are unbiased and fairly represents the entiredataset. The data acquired sequentially through interaction with an envi-ronment (E) however, is highly correlated and violates the i.i.d. assumption.To break the correlation of the data, the algorithms utilize an experiencereplay buffer B where sequential transitions are stored in memory and thenrandomly sampled for batch updates Bbatch. The sampled data batch is thenused to perform stochastic gradient descent on Q loss and stochastic gra-dient ascent on policy output (algorithm 3). In addition, the environmentresponds with a done signal d that indicates if an episode has ended (either amax episode duration has reached or the target has been achieved). DDPGcan be considered a continuous domain extension of DQN. The work hasbeen extended in multiple forms and incorporated with actor-critic meth-ods to further investigate its efficacy and proved useful in multiple domains[116] [117] [118], [119], [120]. Given the policy predicting the actions are alsothe one’s that get updated after the action is taken, the target for learning392.5. ALGORITHMSbecomes highly non-stationary. This non-stationary nature of the learningparadigm raises a number of challenges. A simple solution is to implementtarget networks that calculates the target for learning (πθtarg , Qϕtarg). Thesetarget networks can either be updated every few iterations or can slowly fol-low (lag behind) the actual networks.Algorithm 3 acts as a function for network parameter updates that calcu-lates the Q, Policy and Behavior Cloning losses (Note the superscript n isidentical to superscript l and used for avoiding confusion between l and ′,introduced in subsequent chapters.2.5.1 Deep Q-Network LearningAs DQN operates in discrete action space, it is possible to generate a Qvalue estimate for all possible actions, given a state. Then, an action can besampled based on the exploration strategy. With ϵ-greedy exploration, theaction with the highest Q-value is chosen with the probability (1− ϵ) and arandom action otherwise (with probability ϵ), with exponentially annealingϵ. This strategy effectively denotes a policy, and thus a separate network tolearn the policy in DQN is not required. A Q-function and correspondingoptimal action a∗ for a state can be calculated as:Q∗(s, a) = Es′∼E [r + γmax′aQ∗(s′, a′)|s, a] (2.1)a∗(s) = argmaxaQ∗(s, a) (2.2)The entire algorithm is presented in algorithm 1.402.5. ALGORITHMS2.5.2 Deep Deterministic Policy GradientGiven the continuous nature of the action space in DDPG setting, it’snot possible to calculate the Q-value for all possible actions. In fact, allpossible actions are infinite, due to their continuous nature. So a maxoperator over the Q-values cannot be learned. A separate non-linear policythat learns to output an action to maximize Q on the other hand, canbe learned. Exploration in DDPG can be obtained by adding Ornstein-Uhlenbeck (Ornstein-Uhlenbeck Noise (OU)) noise to the action output,although gaussian noise has also shown to be equally effective and is easierto generate. Some works show that noise in parameter space also presents agood exploration alternative. The target networks in DDPG lags the actingnetworks controlled with ρ, usually close to 1. In continuous space, theQ-function and the optimal action can be calculated as:Q∗(s, a) = Es′∼E [r + γQ∗(s′, π∗(s′))] (2.3)a∗(s) = π∗(s) (2.4)The entire algorithm is presented in algorithm 2.412.5. ALGORITHMSAlgorithm 1: Deep Q-Learning (DQN)Initialize: Q-function Q with parameters ϕ, replay buffer B ;Set: target parameters ϕtarg ← ϕ, ϵthresh close to 1 ;for however many environment interactions doObserve state s and get action a with ϵ-greedy exploration ;a =∼randomA for ϵ < ϵthreshmaxaQ(πθ(s), a) otherwiseAnneal ϵthresh ;Execute a in environment and store tuple {s, a, r, s′, d} in B ;if s′ is terminal or max episode length reached thenReset environment ;endif time to update thenSample transitions batch Bbatch = {s, a, r, s′, d} ∼ B ;Calculate targetsy(r, s′, d) = r + γ(1− d) maxa′Qϕtarg(s′, a′)Update model weights (Q), Algorithm 3 ;if update target networks thenϕtarg ← ϕ;endendend422.5. ALGORITHMSAlgorithm 2: Deep Deterministic Policy Gradient (DDPG)Initialize: policy π with parameters θ, Q-function Q withparameters ϕ, replay buffer B ;Set: target parameters θtarg ← θ, ϕtarg ← ϕ ;for however many environment interactions doObserve state s and get action aa = clip(πθ(s) + anoise, alow, ahigh), anoise ∼ N (1, 0) ;Execute a in environment and store tuple {s, a, r, s′, d} in B ;if s′ is terminal or max episode length reached thenReset environment ;endif time to update thenSample transitions batch Bbatch = {s, a, r, s′, d} ∼ B ;Calculate targetsy(r, s′, d) = r + γ(1− d)Qϕtarg(s′, πθtarg(s′))Update model weights (Q, π), Algorithm 3 ;Update target networks;ϕtarg ← ρϕtarg + (1− ρ)ϕ, θtarg ← ρθtarg + (1− ρ)θ;endend432.5. ALGORITHMSAlgorithm 3: Update model weightsif Q thenCalculate gradient descent step and update Q-function modelQϕ ;LQ =1|Bbatch|∑{s,a,r,s′,d}∈Bbatch(Qϕ(s, a)− y(r, s′, d))2ϕ← ϕ− α∇ϕLQendif π thenCalculate gradient ascent step and update policy model πθ ;Lπ =1|Bbatch|∑s∈BbatchQϕ(s, πθ(s))θ ← θ + α∇θLπendif BC thenCalculate Lπ, gradient ascent step and update policy model πθ ;Qmask = Boolean(Q(sn = ϕ(s), an = πn(sn)) > Q(s, a))LBC = λ1|Qmask ̸= 0|∑a∼π(s),an∼πn(sn)(a− an)2QmaskL = Lπ − LBCθ ← θ + α∇θLend44Chapter 3EASE - Concept and InsightsIn RL, every MDP is considered to be a separate task, expected to besolved by the agent. Tasks with high dimensional state spaces, can presentan intractable challenge for the RL agent to learn. Let’s consider a sim-ple environment. A two-dimensional 2-Degree Of Freedom (DOF) roboticarm, where the states represent the angles ω of the joints s = {ω0, ω1}. Arandomly sampled state, where each ω can be sampled from [0, 2π] with aresolution of 0.1rads gives a total of ( 2π0.1( 2π0.1)) = 1.7e46 possibilities. Thisnumber alone is orders of magnitude larger than the total number of starsavailable in the universe. Combinatorial optimization from a space that bigis simply not possible. While using NN models as non-linear function ap-proximators, generalization in such a space is possible, although explorationremains to be an unsolved challenge.In this section, the concept of EASE is presented in detail with mathe-matical insights into its working. The conceptual EASE is then combinedwith simple heuristics and mapping functions to empirically show its po-tential for different problems. Three primary ingredients of any intelligentmachine and its autonomous decision-making processes are perception, plan-ning and control. Perception refers to information acquisition from raw sen-sor data, while planning and control utilizes this information to take optimal453.1. EASE AS A CONCEPTdecisions. Planning is often executed with discrete set of actions, while con-trol is continuous. Some deep RL methods train the entire process in anend-to-end fashion that allows them to take decision actions or calculatecontrol input directly from the raw sensor data. While perception is out ofscope of the presented work, the use cases are presented in both discrete andcontinuous action spaces to highlight the flexibility and adaptability of theproposed concept. For the low level continuous control problem, the concep-tual EASE is presented on the reacher environment and inspects the benefitsof learning in a local area. Then, a simple heuristics based EASE algorithmis developed for a custom-built gridworld environment that elaborates onthe concept’s ability to solve complex planning and navigation challengesin a discrete action space. Both the environments are modified to presenthighly dynamic and challenging cases, representative of the challenges anagent may face in the real world.3.1 EASE as a conceptThis section analyses the proposed concept of EASE and present insightson its efficacy from an experimental and mathematical perspective.In addition to exploration-based challenges, large amount of initial statevariations during training also leads to high variance in gradient estimatesthat inhibits the capacity of the agent to learn. Moreover, identical trajec-tories from different directions would add noise to the gradient estimates,which further hampers training. Finally, estimates from different tasks canhamper each other as they all don’t have the same optimum. Numerous463.1. EASE AS A CONCEPTworks have surfaced to address some of these challenges. The family of pol-icy gradient algorithms and their variants actively try to reduce variancein such estimates to allow for better and faster learning. The challengessignificantly escalate for sparsely rewarded problems, where the agent seesonly a few trajectories with high reward returns. Intuitively, a lot of thesechallenges would be easier to deal with, if the state-space could just besmaller.Consider an environment, with a size parameter N , represented as EN ,N ∈ N. State space of this environment S is large and too complex. De-fine a partition P over the state space S that breaks S into multiple littlestate spaces, each one referred to as a local environment S l. Say, the sizeparameter defining the local parameter is n, n ∈ N. Thus by design, n < N ,|S l| << |S| and the partition is such that each S l is a local abstraction ofthe global environment that follows the conditions below:1. The action space in the local environment En is homogeneous withthe action space in the global environment EN . Al := AThis condition ensures that any action calculated in the local statespace can be directly applied to the global space without any mapping.This is particularly important as the condition would allow a localagent to solve or plan for an optimal trajectory, which can directlyform a part of the global solution.2. The local environment shares the same spatio-temporal structure asthe global environment. In other words, the local environment has thesame transition probability function P (sl, sl′) as the global environ-473.1. EASE AS A CONCEPTment P (s, s′)This condition ensures that a local state in any partition can bemapped to a different local environment region, with simple grouptransformations. This would allow for the local agent trained in a lo-cal environment to solve for any S l throughout S. In addition, similarspatio-temporal structure would also allow a restricted global state tobe mapped on to a local state.In practice, defining a local environment reduces to restricting the re-set and goal state of the environment in a particular region. That makesthe local environment a region-restricted MDP of the global environment.Implementation of this sort makes the conditions rather general and allowthem to extend to multiple other scenarios and domains, both simulatedand real-world. Multiple local environments in a large state space can becompared to the equivariance property of Convolutional Neural Network(CNN) training in SL. In an image classification problem, a well trainedCNN model is able to recognise or classify an object from an input imageirrespective of the object’s position in the image. This ability is due to thefact that different layers and nodes of the NN model share their weights. Itcan be argued that region-restricted MDPs present equivariance and trans-lational invariance in action space. In other words, an optimal action inlocal environment al∗would be the same as the optimal action a∗ for thecorresponding state in the global environment al∗:= a∗. This leads us to anotable implication - Locally trained models can be augmented with simplegeometric transformations to generate data for larger spaces. In addition,as |S l| << |S|, local agent training is expected to be significantly easier and483.1. EASE AS A CONCEPTfaster. The intuition behind this is as follows.Each batch update for the local agent would sample the same statemultiple times due to the region-restricted MDP sampling. The batch, thuswould have seemingly lesser random data as the agent visits the same statemultiple times, with less initial state variations. The variance in gradientestimates would reduce. If an estimate of a quality of a state given an actionis given by q(s, a) = Q(s, a), then the quality of the estimate itself dependson the number of times the agent witnesses the actual state-action pair.Due to the stochastic nature of the environment, with each state-actiondatapoint, the estimate can be improved.q∗(s, a) =1hi=h∑i=0qi(s, a) (3.1)And q → q∗, when h→∞ (3.2)Training in a local region would sample a particular state more oftenthan the global environment, ensuring hl > h and thus predicting a betterestimate for the quality of the state-action pair and by extension, the Q-function. The fundamental concept of EASE is different from RL approacheswhere the training process is augmented with domain-specific knowledge toimprove training [121]. While domain-specific information and informationengineering can improve the training process, such information is usuallyexpensive to encode and works in a constrained domain. EASE on the otherhand is a flexible and adaptive concept that exploits inherent symmetries,by abstractly defining local regions.493.1. EASE AS A CONCEPT3.1.1 ReacherThe concept is implemented on a Gym Reacher environment. The en-vironment is a two DOF robotic arm that attempts to reach a target in a2D space. The agent is expected to learn the low level torque control of thisarm to manipulate the gripper position p|g and reach the target positionp|t. With a static target, the challenge can be rather easily solved. Theenvironment is modified to change the target from static to dynamic. Theagent is now expected to intercept the target while the target is in motion.The target’s initial velocity is randomly sampled at the time of reset. Figure1.2 depicts the reacher environment with a 2 joint robotic arm. The shadedarea represents the local area, within which the reset state is bounded forlocal agent training. The four regions A,B,C and D are abstractly similarin a sense that a locally trained agent can start in any one of the regionsand intercept the moving target within that region, when subjected to co-ordinate transformations that force regions B,C and D to look like regionA.Figure 3.1 represents how different local region states in the global en-vironment can be mapped onto the region A, where a local policy hasbeen trained. The mapping function Φ(.) represents both ϕ : S l × SA andϕ : S × SA.With this setup, agents were trained with a fixed initial state with littlenoise si, while the target state was randomly sampled from the respectiveenvironments st ∼ EN , slt ∼ En. The environment is sparsely rewarded,where the agent gets a reward only if it reaches within a certain accuracyrequirement de of the target. Comparing the training graphs of the two503.1. EASE AS A CONCEPTFigure 3.1: Reacher environment depiction. The gray shaded area representsthe local environment as a part of the global environment. For the localenvironment, the reset state for ω0 is bounded between [0, π/2] or [−3π/2, 0]depending on the direction of measure. There are no limitations on ω1. Thetarget is denoted by the red dot, with its velocity vector denoted by the blackbroken arrow. The target is also bounded in region A for local training. Thevertical lines represent the partition over the global environment, generating4 abstractly similar regions A,B,C and D.513.1. EASE AS A CONCEPTagents wouldn’t be a fair comparison (Figure 3.2) the local agent is sub-jected to only a part of the entire state-space, however the performanceof the trained agents after convergence can be compared. Convergence forboth agents is defined when asymptotic performance (running average cu-mulative reward) of the training progress reaches max(R)/1.5. As shownin Table 3.1, the local agent holds its success rate for a range of accuracyrequirements. On the other hand, the global agent’s success rate drops asaccuracy requirement increases. Interestingly, both the agents were trainedfor an accuracy requirement of 0.03m. The success rate is reported for anaverage of 50 trials after the training process. For a fair comparison, boththe agents are given the same initial start state and same target locations.It is interesting to note that despite convergence, the local agent is only ableto solve the environment only 66.36% of the times. A closer look revealed,that the global policy converges to a point where it starts spinning the robotarms at the highest possible speed. The sparse nature of the reward does notpenalize the agent for using excessive torque, and prioritizes reaching thegoal. Given the 2D nature of the environment, there is a good chance thatsuch a technique would intercept the target. This was further confirmed bycomparing the average torques applied to the robot by the agents. Both av-erage torque over an episode and average episode duration were significantlylower for the local agent, compared to the global agent. The global agentconverges far from an optimal policy.From Table 3.1, another interesting point of note is that the local agentperforms equally good for accuracy requirements smaller than the originaltraining. This is an interesting behavior and suggests generalization capa-523.1. EASE AS A CONCEPTTable 3.1: Comparison of success rates as a function of accuracy require-ments de for the local and global agents after convergence. Both the agentsare given the same test set and the success rate is reported as an averageover 50 trials for each de. The agents are trained for de = 0.02de /Agent0.05m 0.04m 0.03m 0.02m 0.01mLocal 100.0 100.0 99.27 98.88 74.19Global 89.65 79.87 66.36 42.42 14.98Figure 3.2: Training progression of the local and global agents with fixedstart state and randomly sampled target locations533.1. EASE AS A CONCEPTTable 3.2: Success rates reported as a function of deviation from the fixedinitial state for which the local and global agents were trained. Two devalues are reported for the global agent to further emphasize that the agentis rarely able to generalize around the start state even with relaxed accuracyrequirementsDeviation /Agent±0.1rad ±0.4rad ±π/4radLocal 99.48 99.23 98.1Global de =0.05m89.65 71.54 50.10Global de =0.03m66.36 41.28 12.10bilities. A second set of inference experiments were conducted to see howwell the two agents can generalize within a local domain. Although thetrained agents were subjected to a fixed initial state during training, at thisinference stage, their initial states were randomly sampled from the localenvironment and success rates are reported in Table 3.2. Evidently, the lo-cal agent is able to perform even when the initial state is radically changingcompared to the global agent. A change of ±0.4rad in the local environmentessentially suggests that the agent can start from anywhere in the environ-ment and still reach the target. Global agent on the other hand is rarelyable to perform, despite relaxed accuracy requirements. With these insightsinto the working of EASE with a dynamic environment, benefits of localtraining are summarized in Section 3.1.2543.1. EASE AS A CONCEPT3.1.2 Benefits of local trainingThe experiments with the modified Reacher environment concretely iden-tifies the benefits of local training. The key highlights are presented below:1. Local policy is able to learn in a dynamic and sparsely rewarded en-vironment. In contrast, the global policy gets stuck in a local optimawhich performs poorly during inference time. The local policy is alsoeasier and faster to train2. Average torque and episode duration were found to be significantlylower for the local policy, compared to the global agent, when subjectedto the same start and target states. The lesser variance in training datahelps the local policy to converge to an optima3. Local policy achieves better asymptotic performance4. Local policy is able to solve even stricter tasks (accuracy requirements)than what it was initially trained to do. Global policy’s success ratefalls dramatically with increasing accuracy requirements5. The local policy demonstrates excellent generalization where it is ableto handle varying initial states during inference time. Global policyon the other hand, is unable to generalize or even perform adequatelyfor the original training scenario. While generalization in ML is of-ten studied in different forms, generalization in this context refers tothe ability of the agent to perform in a wide range of varying initialconditions and is restricted to this context553.2. HEURISTICS WITH EASEIn future sections, the effect of this strong local training in global con-vergence is studied.3.2 Heuristics with EASEConsider an optimization problem within a two-dimensional grid worldenvironment of size N × N , N ∈ N, where {EN} represents the set ofall cells within the grid world and EN represents the environment to besolved. The environment has randomly placed obstacles ON ⊂ {EN} thatchanges with every episode. The optimization objective is to navigate froma randomly sampled start cell cs to a randomly sampled goal cell cg where{cs, cg} ∼random{EN} and {cs, cg} ∩ ON = ∅, under the assumption thatat least one path exists from cs → cg. For such a dynamic environment,RL training is too complex and computationally expensive. The challengessignificantly escalate for large N environments, often rendering the prob-lem intractable. Intuitively however, a smaller grid world En of size n× n,n << N,n ∈ N, and cell set {En} represent a similar environment with com-parable dynamics with the exception of notably smaller |En| space. While,most methods use the training on En as an initial step to learn the funda-mental dynamics of the environment, while building an agent that is directlyapplicable in EN . It can be seen that along with the dynamics of EN , Enalso boasts a crucial symmetry in state and action spaces with EN thatstays unexplored. Moreover, with the right choice of En one could mapevery element in {EN} onto {En}; that is an agent trained on En can beemployed to solve EN without any further training, but with mapping from563.2. HEURISTICS WITH EASEthe states of EN to the states of a series of En.With this intuition, an execution approach to tackle the challenges posedby large EN -like environments is introduced that attempts to solve such en-vironments by employing an agent trained on the En environment alone.The following section materializes a heuristic-based algorithmic that sys-tematically maps {EN} to {En}, present insights to the choice of n anddiscuss possible extensions to the work. In contrast to the research trendin reinforcement learning that focuses on improving the training statisticsincluding sample efficiency, generalization and exploration capabilities, theproposed method takes advantage of heuristics to strengthen run-time ex-ecution of a na¨ıvely trained agent. This intuition is analogous to a driverbeing trained in a limited driving environment though expected to use thelearned skills and do well everywhere.Methodology and AlgorithmThe idea behind the proposed Heuristics with EASE is elegantly intu-itive. The following section highlights the different features and implementa-tion steps of the proposed algorithm. The proposed algorithm trains a na¨ıveRL agent on a template version of the environment. The so-called templateenvironment, by design, has two important properties; i) dynamic featuresto encompass different local areas of the state space and ii) fundamentallysimilar features (e.g., spatial arrangement) and possible actions as the largerenvironment. Overall, the template environment is a compact representa-tion of the abstract states of the actual environment, in its entirety. TheRL agent is then trained on the template environment and augmented with573.2. HEURISTICS WITH EASEheuristics for deployment in the actual environment. The algorithm requiresdomain-specific knowledge for template and heuristic design. It is worth toemphasize that the proposed solution is intrinsically different from the ma-jority of the reported RL research in that it focuses on the use of a na¨ıveagent that takes advantage of an abstract symmetry in the environment,rather than augmenting the decision-making power of the agent by boostingthe learning strategies.EnvironmentTwo resettable environment MDPs with N and n size parameters are de-fined: an objective environment MDP: EN that is aimed to be solve, and atemplate environment MDP En under the assumption that En is spatiallysimilar to EN . Let’s denote all possible states of EN as S and En as Sl.For each step in EN and transition: state s → s′, {s, s′} ∈ S, a mappingfunction sl = Φ(s|.) exists that maps the transition to sl → sl′ , {sl, sl′} ∈ Sl.The non-stationary and dynamic nature of En warrants the employment ofa robust agent that can learn to solve for the entire state space of Sl. As|Sl| << |S| by design, a fair assumption would be that learning a policyπθ(sl), ∀s ∈ S⋖ would be significantly easier compared to a policy πθ(s),∀s ∈ S. RL agents present a strong framework to learn dynamic and non-stationary environments such as En and forms the obvious choice for thepresented implementation. Theoretically, an agent could be trained to learnover the entire set of {EN}, but the large state space and lack of conver-gence guarantees for na¨ıve exploration techniques render the problem eitherintractable or computationally expensive [12].Mapping583.2. HEURISTICS WITH EASEA mapping function Φ(.|nature) : S→ Sl that defines the state space Sl as aprototypical representation of S and allows the nature of the representationto be either recursive or representative is defined. The value of n can bechosen such that, N = nl, l > 1 for l ∈ N. This choice has two important im-plications that allow EASE to scale to large and complex environments. Fora d-dimensional environment, the implications can be clearly demonstratedby the cardinality of EN as follows:|EN | = Nd = ndl = |En|l= nd × nd(l−1)= nd × [nd × nd(l−2)]=l∏i=1nd → l, En problems(3.3)1. Recursive mapping - The EN problem is decomposed into a se-quence of l times En problems with their start and end points defined,heuristically. This gives rise to a recursive nature of En being a DPsubproblem of EN . In other words, the solution for EN can be repre-sented as the recursive solution sequence: En → En2 → ...→ Enm →... → Enl = EN , for a monotonically increasing m ∈ N, 1 ≤ m ≤ l.This provides the solution of EN using only a single agent trained onEn. Fig. 3.3(a) represents a two-dimensional example of recursivemapping.2. Representative mapping - As the En environment is a subprob-593.2. HEURISTICS WITH EASElem of the EN environment, a similar spatial arrangement is assumedwithin the elements of the two environments implying a common tran-sitional probability function p(.|.). This assumption allows the envi-ronment Enmto exhaustively cover the elements in {En(m+1)} throughspatially arranged repetitions of {Enm} without any overlaps to en-sure that for any 1 ≤ m ≤ l solutions to the Enm environment canbe linearly combined to solve the En(m+1)environment. The currentposition of the agent in EN identifies the current region that is thenmapped onto En as the observation to the agent (Fig. 3.3(b)).SubgoalA Find Subgoal function is defined sle = F (s) that finds a suitable subgoalfor the current state position sp : slp. Note that for an agent to move fromone region to another, it is enforced to be at one of the vertices of the currentregion (e.g., region A in Fig. 3.4). This allows for specific options for thefunction to greedily select an end goal. The function F (.) identifies thecurrent region of sp and finds possible subgoals as depicted in Fig. 3.4. Thepossible subgoals for a vertex position could either be the other vertices inthe same region or the vertices of the neighbourhood regions. The functionthen greedily picks the subgoal sle that minimizes the distance |sle − sg|.This choice then reflects the value of the nature argument for the mappingfunction.nature =recursive, ∀ sle ∈ {neighbourhoodpositions}representative, otherwiseAlgorithm Steps603.2. HEURISTICS WITH EASEFigure 3.3: A mapping example of an environment with N = 4, with arecursive (a) and representative (b) map on to a smaller environment withn = 2. Here N = n2, l = 2 > 1. The figure demonstrates how larger envi-ronments can be abstracted (either by representative or recursive mapping)onto a smaller environment and solved as a combination of solutions of thesmaller environments.613.2. HEURISTICS WITH EASEFigure 3.4: The figure denotes the possible choices for the selection of asubgoal for the region A and agent position ae. From ae, the agent couldchoose to go to the neighbourhood vertices (change regions) or select anothervertex of the current region A. The function then greedily picks the vertexend goal that has the least distance to goal sg in environment EN623.2. HEURISTICS WITH EASEThe Figure 3.5 depicts the flowchart of the proposed EASE heuristics algo-rithm realized in the following seven steps:1. Consider a large resettable EN environment with a size parameter N ,N ∈ N for which optimization in the state space is computationallyexpensive, if not intractable2. Define a resettable template-like small environment En with a sizeparameter n where N = nl, l ∈ N, l > 1, n ∈ N that represents theprototypical features of the large environment EN3. Train a robust RL policy to solve for the environment En with dynamicfeatures, allowing it to learn all possible mapping configurations. Thepolicy returns an action a for a current state sla = πθ(sl)4. Start with m = 1 at the current state position sp in the environmentEnmand find subgoal and nature [sle, nature] = F (sp)5. Employ mapping sl = Φ(s|nature) for the region of the current posi-tion of the agent in environment Enmto obtain En.If nature is recursiveslp =int(sp/n) + 1, if int(sp/n)! = 0int(sp/n), otherwise, or if nature is representativeslp =sp%n, if sp%n = 0n, otherwise633.2. HEURISTICS WITH EASEwhere slp is the agent position in the environment En6. Employ the trained agent to solve from the current position to subgoal{slp, sp} → sle7. Incrementm and iteratively follow steps 4-6 for monotonically increas-ing m until m = l i.e., the final goal in EN (i.e., sg) is achievedThe algorithm allows a na¨ıvely trained agent to solve for any randomlysampled initial and goal state pairs in the EN environment, {s, sg} ∼randomSsuch that {s, sg} ∩ {ON} = ∅ under the assumption that at least one pathexists from s→ sg.ImplementationThe algorithm was implemented on a two dimensional grid world with N =16, n = 4 and l = 2 to demonstrate heuristics with EASE. A duelling DQNagent was trained for the 4 × 4 maze. Two case studies, a maze with andwithout obstacles, were implemented to verify the efficacy of the algorithm.Case 1: The maze was considered without any obstacles to examinethe algorithm on a stationary environment representation with a relativelystraightforward RL agent training. This case was investigated with tworecursive mapping implementations: automatically selected next region (Fig.3.6), and user-defined next region (Fig. 3.7). More precisely, the formerutilizes the full EASE algorithm with representative and recursive mappingto solve the entire maze. On the other hand, the latter only adopts therepresentative mapping and pre-defined region traversal (the A → B → C→ D sequence) is introduced to the agent that only tries to reach the subgoalin each region. This is an important implementation as it demonstrates the643.2. HEURISTICS WITH EASEFigure 3.5: Flowchart of the presented methodology653.2. HEURISTICS WITH EASEability of the algorithm to extend to environments that violate the N = nlcondition. Pre-defined traversal between regions ensure continuity betweenmultiple n-environment solutions, rendering the need for recursive mappingobsolete. Thus, the agent can solve for any n sized environments whereN = nk for some k ∈ N. The addition of user-defined heuristics still followthe core principles of EASE that exploits the symmetry found in differentregions of a large and complex environment and utilizes a na¨ıve agent for anear-optimal solution.Case 2: The environment was constructed with randomly placed obsta-cles. Figure 3.8 shows the solution of a maze with obstacles occupying upto 30% of the cells. It is worth to mention that for the purpose of thisimplementation a single-step recursive mapping is used as a special case ofrecursive maps where it is ensured that the region jumps are at most a sin-gle step away from the current region. This is further strengthened by theFind Subgoal function F (.) where the neighbourhood vertices considered arealso always only a single step away from the current location. This has twoimplications. First, the action in the En(m+1)recursive environment will onlybe a single step, and it can be directly executed on the agent in Enmen-vironment without any mapping in the action space. Second, the recursiveand representative mapping solutions will be inherently continuous and noadditional steps are required to combine the two solutions. This implemen-tation also allows for a direct conversion of obstacle positions in EN to Enenvironment, On = Φ(ON |representative).The proposed algorithm was successfully implemented on an N = 16 en-vironment with an agent trained for n = 4 environment. Empirical evidence663.2. HEURISTICS WITH EASEFigure 3.6: Case 1: Implementation of EASE - Solution to N = 16 environ-ment using an RL agent trained on an n = 4 environment. The agent solvesfor a region, then greedily selects for a subgoal either within the same regionor a neighbourhood region. The heuristics then iteratively maps differentregions onto En and solves the environment673.2. HEURISTICS WITH EASEFigure 3.7: Case 1: Implementation of EASE without recursive mapping -Solution to N = 16 maze using an RL agent trained on an n = 4 environ-ment. The agent solves different user-defined regions (A, B, C and D) whilethe solution for traversal in between the regions was pre-defined683.2. HEURISTICS WITH EASEFigure 3.8: Case 2: Implementation of EASE with obstacles - Solution toN = 16 environment using an RL agent trained on an n = 4 environment,with randomly placed obstacles. The agent greedily selects for a subgoalafter solving each region. The vertices that are also obstacles, are removedfrom consideration for possible end goals. The heuristics then recursivelymap the different regions onto En and the agent solves for the path untilthe environment EN is solved693.2. HEURISTICS WITH EASEconfirms that the proposed algorithm successfully finds a near-optimal pathin large state-space environments with an agent trained in a small template-like environment and can efficiently tackle dynamic features like randomlysampled obstacles. It was further verified that a DQN agent trained directlyon N = 16 environment does not converge, while n = 4 presents a rathereasier training (figures 3.11, 3.9, 3.10). The implementation was verifiedwith multiple randomly sampled start and goal states both with and with-out obstacles. Owing to the nature of the F (.) function of greedily selectinga subgoal, instances where optimal neighbourhood vertices are blocked withobstacles may push the algorithm to repeatedly select vertices within thesame region and get trapped in a local loop. However, a randomly sam-pled end goal in this case will ensure a solution to the maze at the costof optimality, which is often the case with heuristics. The proposed workhighlights a bottom-up approach that starts with the solution in a smallenvironment and takes one-step action to recursively change regions. Thisallows for seamless integration of the two mapping outputs as the actionin one domain can be directly transferred to the other. With this setup,multiple extensions to the work are theorized:− Direct extensions: While the presented experiments involve a dis-crete state space, the work can be directly extended to continuousstate-space domains.− Top-down approach: The presented bottom-up approach exploitsthe single-step region traversals to form a continuous solution to alarge state space. The same can be re-purposed to allow for a top-703.2. HEURISTICS WITH EASEdown approach where the environment EN mapped onto En(l−1) issolved first, identifying regions required for the solution. These regionscan then be solved in a similar fashion until each region could berepresentatively mapped onto En. The solution to large regions wouldrequire a multi-step recursive approach and an abstraction functionthat maps the dynamic features of the Enmenvironment on to En(m−1).− Mapping approximators: With increasingly large and complex statespaces and environments, a polynomial mapping may not be straight-forward. A function approximator could be employed to learn themapping from EN to En, through unsupervised or pattern-recognitionmethods.The RL agent was trained with the following properties:− A state vector of dimension [2n2, 1] and [3n2, 1] for cases 1 and 2respectively. For Case 2, given the dynamic nature of the obstacles,its important for the state vector to reflect the location of the obstacles.Thus an agent with an [n2, 1] additional information compared to case1, is trained. The state vectorx =if sl(idx(slc))1, at sl(n2 + 1 + idx(sle))at sl(2n2 + 1 + idx(On), case 2 only0, everywhere elserepresents a flattened maze concatenated multiple times with the firstflat array representing the current position [1 : n2 = 1 : 16], second713.2. HEURISTICS WITH EASEthe end position [n2 + 1, 2n2 = 17 : 32] and third with all the obstaclepositions [2n2 + 1, 3n2 = 33 : 48] (if applicable).− Action space for the agent A = {N,E, S,W} - Agent can move North,East, South or West at any step− A sparse reward functionR =+5.0 if slc = xe,with obstacles+10 if slc = sle,without obstacles−0.5, if agent bumps into an edge−2.0, if agent bumps into an obstacle−0.1, otherwise− The agent was trained as a fully connected neural network with thestructure [FC(state dimensions), FC(100), ReLU, FC(200), ReLU,FC(75), ReLU, FC(50), ReLU, FC(25), ReLU, FC(10), ReLu, FC(output= 4)] where FC(j) represents a fully connected perceptron layer withj inputs and ReLU denotes a Rectified Linear Unit layer. The trainingwas implemented with a learn rate α = 0.001 and greedy-ϵ exploration,ϵ− decay = 0.0001Fig. (3.9) and Fig. (3.10) show the training progress of an agent in a4 × 4 environment with and without obstacles, respectively. The trainedagents are then employed with EASE heuristics to solve a variety of N = 16environment without any additional training. Fig. (3.11) shows the trainingprocess of the agent directly in N = 16 environment without obstacles. The723.2. HEURISTICS WITH EASEFigure 3.9: Successful training of an RL agent in a n = 4 gridworld envi-ronment without obstaclesfigure also demonstrates the inability of the agent to learn to navigate in alarge state-space environment, even for an easy, no obstacle scenario.The chapter elaborates on the concept of EASE and presents exper-iments on the efficacy of the proposed method in different scenarios. Itstarts by demonstrating the benefits of a locally trained agent and com-pares the performance of a locally trained agent vs the global one, in theirrespective environments. This comparison and the strong convergence of thelocal agent showcases the effects of the training process in a smaller state-space and forms the motivation for additional work. Following the concept,the local agent is amalgamated with guiding heuristics where the qualityof the local convergence is exploited to solve a much larger and complexenvironment. The fact that a local agent with only a small foresight is ableto efficiently solve a much larger environment with no additional trainingfurther exemplifies the proposed concept. Evidently, the local agent is able733.2. HEURISTICS WITH EASEFigure 3.10: Successful training of an RL agent in a n = 4 gridworld envi-ronment with obstaclesFigure 3.11: Failed training of an RL agent in a N = 16 gridworld environ-ment without obstacles743.2. HEURISTICS WITH EASEto generate solutions directly in the global environment. While in this chap-ter, heuristics combine these local solution for a global inference, definingsuch heuristics for continuous robotic environments is not straightforwardand hence presents a challenge. However, the data generated by the localagent is still optimal in its region and can be used to aid global trainingand convergence. This ability of EASE is further studied and extended tothree-dimensional complex robotic scenarios discussed in the next chapter(Chapter 4.75Chapter 4Data generation with EASEThe na¨ıve agent’s ability to solve local areas in complex environmentsshows a significant potential and a range of possible extensions, as shownin Chapter 3 and further discussed in Chapter 5. Along with the discussedbenefits, one advantage that may not immediately become apparent is it’sability to generate quality data for the global agent training. The questionthat guides the process in this section, is if the na¨ıve agent is able to generatequality, optimal even, trajectories for complex global learning? Local exper-tise of the na¨ıve agent allows for numerous interpretations and extensions ofthe fundamental concepts of EASE. One notable fact that sets EASE apartfrom other methods is the homogeneity of the action space across the localand global environments. This homogeneity can be exploited to generate ex-pert trajectories that may aid in global agent training. The concept closelyresembles Learning from Demonstrations (LfD) in RL setting and derives itsmotivation from the field of Behavior Cloning (BC) and Imitation Learning(IL). While LfD often requires global expert demonstrations which the agentcan then learn to either imitate or explore, this section proposes a novel al-gorithm that utilizes local solutions as expert trajectories for optimizationin complex environments. The concept of EASE as a generative model isexamined from an LfD perspective to improve sample efficiency and explo-76ration in large state-space environments. In EASE, the na¨ıve environmentis a masked representation of the global environment. This fairly leads tothe assumption that for any mapping function Φ : S → S l there exists aninverse mapping function Φ−1 : S l → S such that sl = ϕ(s) and s = Φ−1(sl)where s ∈ S, sl ∈ S l. For goal based RL, the agent’s output is a series ofactions that respectively manipulate the target and template environmentsfrom an initial state si, sli to the goal state sg, slg in a closed-loop formalism .If Φ(.) is a spatially abstract mask over S, then a spatial goal can be definedsuch that slg = ϕ(sg) and |sli−slg| < δ, where δ represents a threshold for thesize metric of the local environment. This condition ensures that the localenvironment can solve the given task, and the goal does not lie outside S l.δ can be considered as a more general form of n from Chapter 3. A generalform of the MDP tuple used for RL training in the global environment is{s, as, s′, rs} where s′ represents the next state after taking action a in states with transition probability p(s|a) and receiving reward r.For the local environment the tuple follows a similar format {sl, a, sl′ , r}.Expert trajectories in LfD and IL are recorded with the intuition thatthe source and target environments closely resemble each other (often thesame) and the expert trajectories either replace the need for a reward func-tion [122] or helps in exploration [123]. With EASE, a local agent tuple{sl, a, sl′ , r} has elements of the target environment and can be rewrittenas {sl = ϕ(s), a, sl′ = ϕ(s′), r}. Given the local and global environmentsand the inverse mapping function, state-action pair (s = ϕ−1(sl), a) in thetarget environment can be implemented to obtain s′ and r. The resultingtuple {s, a, s′, r} now denotes an expert trajectory in the target domain,77derived from a local trajectory by the source agent. This tuple relabellingregime has powerful implications in learning a complex environment underthe assumption that the optimal action suggested by the local agent is atleast near optimal in the global environment. Some previous works e.g.,[124] utilize the symmetry of environments to improve sample efficiency ofthe training process, but fall short of utilizing the same symmetry to aid theagents in complex environments. Intuitively, it might not seem likely that alocally expert agent may be able to generate optimal data for a global agentas the local expert never witnesses most of the state space. However, it canbe seen from the experiments with heuristics and local mappings in Chapter3 on environment ablations that na¨ıve agents can optimally solve for largespaces under some constraints. Training in the global environment, if suc-cessful, can reduce dependence on such constraints however, suffers due tothe inherent challenges of the learning process. The high reward trajectoriesgenerated by the local agent can be helpful for training sparsely rewardedscenarios, where only a few environment interactions from long rollouts havehigh or positive rewards. The local and global trajectories share the rewardr and done d signal as they usually are a metric of progress for every episode,which stays the same e.g. relative distance between the gripper coordinatesand the target coordinates. With that insight, the concept of EASE is ex-tended to generate quality data to allow for better training of a global agent.Two distinct methodologies are presented in this regard. Section 4.1 extendsthe field of Learning from Demonstrations (LfD) with data generated by ana¨ıve agent and presents a sensitivity study on the effect of EASE generateddemonstrations while Section 4.2 presents a novel algorithm to highlight the784.1. DEMONSTRATIONS WITH EASEcapacity of conceptual EASE in improving the training sample efficiencythrough state abstractions.4.1 Demonstrations with EASEThe above-described tuple relabelling scheme can easily construct abuffer of expert demonstrations that can further aid in training of any off-policy RL algorithm. A pre-trained na¨ıve agent locally solves randomly sam-pled initial and goal states such that the initial and goal states are withinthe same region {Φ(si),Φ(sg)} ⊂ S l and relabels the tuple for storage in theexpert buffer Bdemo for training in the target environment. The loss for theoptimization update is closely associated with the loss commonly used inBehavior Cloning applications (modified after the work in [123], Algorithm3).For resettable environments, the inverse mapping function is only re-quired to find the initial state si, while the following states are extractedby implementing the state-action pair. This allows us to loosely define theinverse mapping function which is in fact used only while resetting the en-vironment. If a lookup table of possible initial states in corresponding localand global environments can be generated, the goal states can be randomlyselected based on the ability threshold constraint. In fact, as the local agentshares a part of the state-space with the global environment, training in thisspace can be done with the same reset states and even the need for a lookuptable can be omitted. The inverse mapping function is defined for analyticalpurposes only and may not be needed in practice. The local environment794.1. DEMONSTRATIONS WITH EASEmay be nontrivial to create for many instances, but given a mapping func-tion, the global environment could itself be represented as a template to theagent, with restrictions on the state sampling.Learning from Demonstrations extension to DDPG (DDPGfD) [31] israther straightforward where the experience buffer replay of the trainingprocess is augmented with data generated by experts. A separate buffer withdemonstrated data Bdemo is maintained which provides sampled throughoutthe training process. The effect of these samples on the calculated loss isoften conditioned to a tuning parameter λ. Work presented in [125] priori-tizes sampling of demonstration data over the data generated by the agentin-training by employing Prioritized Experience Replay (PER) where tran-sitions are sampled with probability inversely proportional to their mostrecent temporal difference error and demonstrations given a slightly higherprobability than others. Other works anneal λ as training progresses tobalance the effect of trajectories sampled from the demonstration bufferand experience buffer. The actions sampled from the demonstration bufferad ∼ Bdemo are considered optimal and a Behavior Cloning (BC) loss is in-troduced in the gradient update step of algorithm 2 to force current policyaction output a to be closer to the demonstration action.LBC =λ|Bdemobatch |∣∣∣∣∣∣ad − πθ(sd)∣∣∣∣∣∣2(4.1)Some works [126] allow for learning from imperfect demonstrations by ad-justing the gradient update by either including optimal or near-optimaldemonstrations or allowing for exploration in demonstration data to findbetter policies. Previous works have also shown that the exploration process804.1. DEMONSTRATIONS WITH EASEof the training agent can be completely replaced by utilizing the demonstra-tions [123]. Finally, given that the algorithm trains a separate Q-functionto estimate the quality of an action, a mask can be generated that comparesthe action from demonstration with the action calculated by the policy. Thisensures that the data from demonstration will improve the agent and notinclude actions where the current Q-network estimates that the reward ishigher for the policy action compared to the action from demonstrations(Equations 4.2 and 4.3).Qmask = Boolean(Q(sd, ad) > Q(s, a)) (4.2)LBC =λ|Qmask ̸= 0|∑a∼π(s),ad∼Bdemobatch(a− ad)2Qmask (4.3)Using demonstrations for learning in an RL setting presents a few challenges.The BC loss may overfit the agent in-training to the demonstration data ifthe demonstration samples and exploration is not properly balanced. Thisusually is handled by the implementation expert and can be sensitive to thechoice of tuning parameters. Additionally, BC loss in imitation learning isknown to suffer from compounded errors in long trajectories which can leadto catastrophic failures due to lack of generalization [127]. Finally, collec-tion of demonstration data with an expert is an expensive process. Theuse of EASE to generate demonstration data alleviates a number of thesechallenges. The trajectories are equivalent to being generated by an expertdemonstrator, as they are locally optimal in the region, while being inexpen-sive to generate. While the data itself can be considered optimal, mapping aglobal state and action pair to a local environment may cause stochasticity814.1. DEMONSTRATIONS WITH EASEin the process, specially for environments with abstract symmetry. In sucha setting, the agent in-training can benefit with exploration, even on actionssampled from the demonstration buffer. The experiments with ablationsof the method were conducted to study the effect of the choice of λ on thetraining process. The experiments showed that the implementation is robustagainst the choice of λ and improves the training efficiency and performanceover the standalone DDPG algorithm, while the additional exploration al-low it to generalize over a wide range of initial and target states. Finally,the implementation of Qmask, allows LBC to guide Lπ while ensuring thatonly good actions contribute to the optimization update. As the range ofaction-space A is significantly smaller than the range of the reward functionR, the magnitude of LBC is notably lesser than Lπ. This distinction givesrise to subtle improvement in the training progress and makes sure that theagent does not overfit to the demonstration data.Eqns 4.2 and 4.1 can be rewritten with notation from the na¨ıve agent.Qmask = Boolean(Q(sn = ϕ(s), an = πn(sn)) > Q(s, a)) (4.4)LBC =λ|Qmask ̸= 0|∑a∼π(s),an∼πn(sn)(a− an)2Qmask (4.5)A notable difference from traditional DDPGfD algorithm implementationis that the EASE trained na¨ıve agent does not generate optimal trajecto-ries for the entire environment. Thus its imperative to adjust the sam-pling process to allow for start-goal combinations which the na¨ıve agent cansolve. The sampling process established in Chapter 3 is utilized and furtherelaborated earlier this section, to start the training process by generating824.1. DEMONSTRATIONS WITH EASEdata for the demonstration buffer by sampling start-goal pairs such that,{Φ(si),Φ(sg)} ⊂ S l. This sample is fed to the local agent to generate op-timal actions ad = a∗ = al∗ = πl(Φ(s)). The solved tuple is then stored inboth B and Bdemo. The entire algorithm is shown in algorithm 4.Implementation and ResultsThe proposed DDPGfD with EASE algorithm (demo-EASE) is implementedon the Reacher environment with a dynamic target. The target’s initial ve-locity is sampled from a higher range compared to the experiments presentedin Chapter 3 to further increase the complexity of the training progress. Theenvironment and training is sparsely rewarded, where the agent gets a pos-itive reward only when its able to manipulate the end-effector of the robotwithin a strict accuracy requirement de of the target coordinates. The statevector consists of the joint angles {ω1, ω2}, angular velocities ω˙ = {ω˙1, ω˙2},target coordinates p|t, target velocity gv, end-effector position p|g and thedistance between the end-effector and target coordinates ||p|g − p|t||2. Withdifferent regions of the state-space, the angular positions, gripper and tar-get coordinates, and velocity vectors would change and require mapping tothe local area, while the angular velocities and distance between the tar-get and end-effector coordinates stay the same. This structure allows fordirect transformation (rotation in euclidean space) between the local andglobal state-space, as long as the region of start-state and goal-state are thesame. Say β denotes the angle of rotation required. This rotation indicatesthe map from any region within the global state-space to the local region.834.1. DEMONSTRATIONS WITH EASEThen, following Figure 1.2,β =0 if 0 ≤ ω1 < π/4 Region A: Local Region−π/4 if π/4 ≤ ω1 < π/2 Region B−π/2 if π/2 ≤ ω1 < 3π/2 Region C−3π/2 if 3π/2 ≤ ω1 < 2π Region D(4.6)Then,plxply =cos(β) −sin(β)sin(β) cos(β)pxpy ∀{px, py} ∈ {p|g, p|t, gv} (4.7)If pl=ˆsl ⊂ S l, an action can be calculated for the demonstration buffer.Figure 4.1 shows the cumulative results plot of the training process of theDDPG algorithm, with and without the EASE augmented data. The DDPGwith EASE (equivalent to DDPGfD with EASE generated demonstrations)clearly outperforms the standalone agent, improving the sample efficiency ofthe training process. While it may seem like the global learning is catchingup, the trained global agent displays similar trend as in Chapter 3, whereit gets stuck in a local optima, tries to find a single policy output thatcould achieve all goals and starts to randomly spin on its axis. The agenttrained with demo-EASE on the other hand, is able to optimally solve forrandomly sampled start and goal states while reducing training samples,thereby improving the generalization and sample efficiency of the process.The plots in Figure 4.1 are reported as an average over 4 runs from differentinitial random seeds, with the standard deviation shown in lightly shadedarea around the mean. The global, local and demo-EASE policies sharedthe same structure and hyperparameters. For demo-EASE, the training844.1. DEMONSTRATIONS WITH EASEFigure 4.1: Training progression of the DDPG agent with and without theEASE augmented data.was found to be surprisingly robust to the choice of λ. Multiple valuesbetween the range of λ = [0.2, 1] were selected and tested, all leading tosuccessfully results with slight changes. Interestingly, keeping λ constant vsannealing λ also arrived at very similar results. A possible explanation tothis observation is the fact the magnitude of LBC is too small compared to Lπand the LBC simply guides the direction of the gradient with subtle changes,which improves the training progress over the grand scheme of learning. Thisalso allows for the demonstration data to be re-sampled throughout thetraining progress without forcing the agent into a local minima. λ valuesoutside the reported range, either slows training (low λ reduces the impactof demonstrations) or overfits to a single strategy (high λ forces the agent tofollow high-reward trajectories from the demonstration buffer and reducesthe impact of the samples generated by the global agent). Evidently, thedemonstration observations can only improve the learning process up to a854.2. ABSTRACTION WITH EASEcertain point, which is expected as the demonstrations span a wide butlocal trajectories. The demonstrations are most utilized for reducing thestart state variation in training and helps further in encouraging the conceptbehind the algorithm presented in Section 4.2.4.2 Abstraction with EASEIn this section, a novel methodology that utilizes the concept of EASEto learn end-to-end control of a high dimensional robotic arm in GymFetchReach simulation, is proposed. The motivation behind the presentedalgorithm is derived by the successful empirical demonstrations thus far.Section 4.1 showed how locally optimal trajectories can be used to speed uptraining of a global agent and help it generalize outside their region. Thefact that a local agent knows how to solve for a part of the environment canbe exploited in a number of ways. The abstract symmetries can also be de-fined in various ways for better and faster implementations. The algorithmpresented in this section utilizes error as one of these abstract symmetrymetrics, that abstracts parts of global trajectories into local state-space tra-jectories. Error can be one of the most defining metrics of abstraction forstate-space based control. Control theory applications primarily depend onthe error in the system to calculate the change in control input. One couldargue that the use of the error signal helps the controller reduce dependenceon the nuances of the plant, and streamlines the objective to a single gener-alizable state; reduce the error. A similar idea is used to define error-basedstate space abstractions in the presented training. Consider a robotic sim-864.2. ABSTRACTION WITH EASEAlgorithm 4: EASE with demonstrations (demo-EASE)Initialize: policy π with parameters θ, Q-function Q withparameters ϕ, replay buffer B ;Set: target parameters θtarg ← θ, ϕtarg ← ϕ ;Train: Na¨ıve policy in local environment πn using algorithm 2 withstart state si ∈ Si and goal state sg ∈ Sg, where Si ⊂ S,Sg ⊂ Sand {si, sg} belongs to the same region;Fill demonstration buffer Bdemoπn(sn), sn = ϕ(s), if p|g , p|t in same region Equations (4.6, 4.7) ;for however many environment interactions doObserve state s and get action a, a = πθ(s) ;Noise a = clip(πθ(s) + anoise, alow, ahigh), anoise ∼ N (1, 0) ;Execute a in environment and store tuple {s, a, r, s′, d} in B ;Reset environment: If s′ is terminal or reached episode duration;if time to update thenSample transitions batch Bbatch = {s, a, r, s′, d} ∼ B anddemo batch Bdemobatch ;Calculate targetsy(r, s′, d) = r + γ(1− d)Qϕtarg(s′, πθtarg(s′))Update model weights (π, Q, BC), Algorithm 3 ;Update target networks;ϕtarg ← ρϕtarg + (1− ρ)ϕ, θtarg ← ρθtarg + (1− ρ)θ;endend874.2. ABSTRACTION WITH EASEulation (FetchReach) with 5 DOF arm with a fixed start state. The goalstate in this environment can be sampled from anywhere in the entire statespace and an agent is expected to learn a low-level joint torque controller.The high variational goal state of the environment would inhibit learningfor traditional off-policy algorithms. Usually for such goal-conditioned train-ing, methodologies like HER prove useful. Intuitively, if the goal state spacecould be restricted, the agent would be able to sample more high-rewardtrajectories as there is a higher chance for the agent to arrive at the re-ward, even with random actions, as it exploited by methods like CurriculumLearning, where the training starts with a restricted goal space, assumedto be an easier task for the agent. Let’s consider this restricted area tobe δ units away from the reset gripper position (Figure 4.2). In contrast,Figure 4.3 shows the entire workspace of the robot arm for which the globalagent is expected to learn. Note that the original FetchReach environmenthas been modified to expand its workspace. In this implementation, theglobal workspace is at least 8 times more voluminous than the local δ re-gion. Within the entire workspace however, a recurring δ sphere could beimagined with the robotic arm always at the centre of it (Figure 4.4. δrepresents the maximum error that the local agent is able to solve (or thesize metric for which the local agent was trained), while the reset positionis virtually transformed to showcase the δ region around various reset po-sitions in the global workspace. Interestingly, the action sequences withinthe multiple δ areas of the workspace remain the same while mappings fromarea to area are simple translations. This follows both the conditions nec-essary for implementation of EASE as discussed in Chapter 3. With any884.2. ABSTRACTION WITH EASEtask represented as an MDP, the optimal solution to the MDP are actionsthat incrementally decrease a pre-defined error metric, under optimizationconstraints. For the purpose of the implementation in this section, a simpleerror metric is proposed. The metric gives the distance between the grip-per and target coordinate position δ = ||p|g − p|t||2. The distance metricin global space is referred to as δg, while δ is a constant that representsthe size of the local environment. With each action, the end-effector wouldassume a new coordinate position and updated δg = ||p|g − p|t||2. As theagent learns to reduce deltag, and satisfies δg ≤ δ, the MDP solution can beprovided by the local agent, given the local agent was trained within thisarea. At this stage, if the local agent takes over and finishes the sequence,a part of the global MDP would most certainly be optimal and does notrequire learning. Thus for every successful MDP task in global learning, theMDP sequence from δ → 0 can be produced by the local agent. This formsa natural curriculum where the global agent only has to traverse from thestart state to δ-units close to the goal. The training however, occurs onthe entire trajectory, significantly improving the training time and sampleefficiency. The solution to the larger MDP is effectively broken down intotwo consecutive parts, solution from start state to any state in Local MDP,then following local policy thereafter. The local MDP then becomes the ter-minal state for any global MDP (internally, without the need of any explicitindication). The principle of optimality suggests that if the solution fromstart state to the local MDP state is optimal, then the entire trajectory isoptimal: as local agent is considered to be optimal. The proposed algorithmis dubbed δ-EASE and multiple ablations are presented for studying the894.2. ABSTRACTION WITH EASEFigure 4.2: Robot workspace with δ units away from the end-effector resetpositioneffect of the local agent on trajectory generation after δg < δ. The localagent’s data generation is controlled through a tuning parameter ζ that actsas the probability with which the local agent takes an action.For the implementation of this method, the FetchReach simulation is trainedon only the end-effector coordinate position and target coordinate positionas the state, whereas the action space is the low-level torque output for therobot joints. The mapping function is rather straightforward. If δg < δ,sl = Φ(s) = s− si =px|g − pxi |gpy|g − pyi |gpz|g − pzi |gpx|t − pxi |tpy|t − pyi |tpz|t − pzi |t(4.8)The entire method is summarised in algorithm 5.904.2. ABSTRACTION WITH EASEFigure 4.3: Robot workspace for which a global agent is expected to learnFigure 4.4: Recurring δ unit areas in the robot workspace with gripperposition at the centre914.2. ABSTRACTION WITH EASEAlgorithm 5: EASE with δ-abstraction (δ-EASE)Initialize: policy π with parameters θ, Q-function Q withparameters ϕ, replay buffer B ;Set: target parameters θtarg ← θ, ϕtarg ← ϕ ;Train: Na¨ıve policy πn using algorithm 2 with start state si (fixed)and goal state sg, ∀∣∣∣∣∣∣pg|g − pi|t∣∣∣∣∣∣2 ≤ δ), where sg ∈ Sg ⊂ S,pi =̂ si and pg =̂ sg;for however many environment interactions doObserve state s and get action a ;a =πn(sl), sl = Φ(s), if∣∣∣∣∣∣p|g − p|t∣∣∣∣∣∣2≤ δwith probability ζ Equation 4.8πθ(s) otherwiseNoise a = clip(πθ(s) + anoise, alow, ahigh), anoise ∼ N (1, 0) ;Execute a in environment and store tuple {s, a, r, s′, d} in B ;Reset environment: If s′ is terminal or reached episode duration;if time to update thenSample transitions batch Bbatch = {s, a, r, s′, d} ∼ B ;Calculate targetsy(r, s′, d) = r + γ(1− d)Qϕtarg(s′, πθtarg(s′))Update model weights (Q, π), Algorithm 3 ;Update target networks;ϕtarg ← ρϕtarg + (1− ρ)ϕ, θtarg ← ρθtarg + (1− ρ)θ;endend924.2. ABSTRACTION WITH EASEFigure 4.5: The figure shows the sensitivity of the algorithm to the tuningfactor ζ. The factor controls the amount of data utilized from the localagent. As suspected, the sample efficiency increases with increasing ζ whichconfirms that the data generate by the local agent significantly helps theglobal learning processFigure 4.5 shows the change in training progress with change in ζ. Withincreasing ζ, the local agent action is sampled with a higher probability,which directly improves learning. The figure proves that the more the localagent is utilized, better the training progress is achieved. Best results wereobtained for ζ = 1, which is equivalent to the local agent deciding on anaction, every time δg < δ. Figure 4.6 shows the comparison betweent thetraining progress of two global agents: δ-EASE and DDPG. The proposedmethodology significantly outperforms the standalone agent. ζ is set to 1for this comparison.An obvious extension to the work would be to initialize the global net-work with the learned weights of the local agent to enable a TL type training.934.2. ABSTRACTION WITH EASEFigure 4.6: This figure shows the significant improvement in global agentlearning with the proposed algorithm δ-EASE compared to the standaloneDDPG agentWhile this may be an intuitively beneficial step, it takes away some flexibilityfrom the proposed methodology. With the global agent initialized indepen-dent of the local agent, there are no restrictions on the global agent’s size,architecture, shape or even state input. By training in the proposed fashion,the algorithm quickly compensates for the lack of transfer initialization aslocal agent starts to generate high-reward trajectories. Moreover, the localagent is fitted to a local state-space of the environment. For it to extendto the global space, it may have to forget a bit to allow for bigger and far-sighted representations. This is evident in Figure 4.7 where the presence ofthe local agent, destabilizes training for certain ζ values. The training even-tually recovers and still works converges faster than the standalone agent,an independent global agent with randomized initial weights is found to befaster and easier to train.944.2. ABSTRACTION WITH EASEFigure 4.7: This figure shows how the training progress destabilizes forcertain ζ values, when the global agent is initialized with the weights of thelocal agent. The training eventually convergesThe chapter extends the concepts discussed in Chapter 3 to high dimen-sional and challenging environments and studies the efficacy of data gener-ated by the local agent. The data and trajectories from the local agent areextended to two scenarios i) when they are considered to be expert demon-strations (optimal and sub-optimal) and ii) when a part of the trajectory isoptimal. Both the settings significantly improve the global training processand helps the global agent explore effectively and avoid local optimum. Inthe presented scenarios, the proposed method utilized only 25% and 12%of the samples required to train the standalone global agents with demo-EASE and δ-EASE respectively. The structure of the proposed methodsin this chapter is flexible wherein the local agents could be replaced withother effective control strategies to generate data. The idea emphasized inthis setting is to encourage the use of symmetries in training with EASE to954.2. ABSTRACTION WITH EASEfacilitate global RL training and is not limited to the use of RL for localagents.96Chapter 5Conclusions5.1 SummaryEASE (Exploitation of Abstract Symmetry in Environment) is a novelconcept in AI to exploit abstract symmetries present in complex environ-ments and redefine the outlook on RL training. The presented work demon-strates how locally trained agents can be used for better training of RLagents. In essence, the concept of EASE redefines the usual research ap-proach in RL and shifts the focus from advancing algorithms to findinga more computationally tractable problem. The different outlook on theproblem itself, is inspired by human learning and their ability to re-purposepreviously learned skills and simplifying complex problems for better under-standing. A notable feature of EASE is that instead of utilizing the informa-tion encoded within the local learner, it utilizes the ability of the local expertto solve simpler problems in complex domains. The solution to that simpleproblem, then is utilized to solve the originally complex problem. The veryability of using local expertise to solve complex problems resonate perfectlywith human learning. This ideology itself has been applied in numerousdomains and in fact forms the origin of RL through dynamic programming.The key components of intelligent and autonomous systems are perception,975.1. SUMMARYplanning and control. With this thesis, the focus is on planning and con-trol in complex environments while perception stays as an internal stateof the simulations and simulated sensors. Concept’s amalgamation withsimple heuristics allows it to solve large state-space navigation challenges(Chapter 3) that are intractable with standalone RL. On the other hand, itsextension with demonstrations and abstractions (Chapter 4) outperforms itstraditional RL counterparts and proves better against many direct variationsand ablations. Evidently, the implementations improve the sample efficiencyof the training algorithms and help the agent generalize better without in-creasing the need for adding computational resources or additional trainingsamples, but instead utilizing what it already knows.The conditions necessary for the implementation of EASE also forms thelimitations of the proposed concept. The homogeneity of the action spacebetween the local and global environments restrict the application to set-tings where such an environment could be defined. Different transformationsfor state and action spaces for such environments would be required and areout of scope of the presented work. The second condition for EASE is ratherflexible and can allow for easier local abstraction from the global environ-ment, however requires manual separation. This condition of a commontransition probabilities between the local and global environment is alsowhat allows the seamless data usage between the two agents. Finally, thecurrent methodology can only be augmented with off-policy family of meth-ods due to its dependence on the data generated by the older or local policies.Special cases of the proposed concept of EASE can be compared to the985.1. SUMMARYstate-of-the-art RL approaches that tries to improve the training processlike Transfer Learning, Curriculum Learning and Learning augmented withdomain-specific knowledge. EASE presents a generalized approach and asimplified outlook on the training process itself, something the other meth-ods fail to do. The concept of abstract symmetries covers a wide rangeof possible meanings, as demonstrated among different sections of the the-sis. The theorized symmetry does not require MDP model-minimizationor complex group transformations, but rather straightforward and simplestate coordinate transformations. This allows the concept to be extremelyflexible and be applied to a range of applications. In robotics settings,this symmetry is easy to identify, and in fact inherently available. In RLanother big challenge is considered to be a measure of performance. Inmany instances, different measures of progress may force different conclu-sions about the training progress, shown in Chapter 4, where the modelseems to be converging; however is actually stuck in a local optima. WithEASE, the possibility of this happening is reduced, by limiting the explo-ration in state-space. Note that the simulation is not actually restricted,but rather figuratively restricted where the reset and goal states are sam-pled from the same region, and thus the agent gets no reward to leave theregion whatsoever. This limitation, then improves a multitude of trainingprogress metrics, including asymptotic performance, episode duration, one-shot extension to new reset states and in robotic examples, average torque.These benefits further motivate the potential of the EASE concept.995.2. FUTURE WORK5.2 Future WorkIn this section, possible extensions to the proposed concept are presented.5.2.1 EASE with HeuristicsChapter 3 discusses the extension of EASE with a simple heuristic methodand shows how a na¨ıve agent can successfully navigate large state-space envi-ronments, which are intrinsically intractable. An extension of the proposedalgorithm could be to use the local learner with advanced planning algo-rithms or curriculum generating methods. If an agent is able to output aset of subgoals required to achieve a particular optimal trajectory in a largestate-space system, the local learner can solve from subgoal to subgoal, evenin stochastic environments. This extension utilizes the best of both worlds:ability of the local learner to handle varying dynamics of the system at thelow-level control, and the ability of advance planning and navigation algo-rithms to render subgoal waypoints, without worrying about randomnessand noises. Some complex environments may share the same end goal irre-spective of the local environment region e.g., a robotic arm trying to raisean object as high as possible. In this scenario, the local environments canbe constructed such that they share the same end state; robotic arm fullyupwards erected. In such scenarios, heuristics can be completely omittedand the local agent itself can solve for the entire global environment withgroup transformation on the state space.1005.2. FUTURE WORK5.2.2 EASE as a generative modelBuilding on the arguments in Section 5.1, the internal information en-coded within the local learner could further help to improve the globalagent’s training process. Specifically, information in the local learner couldbe distilled against the global policy, while the local learner continues toslowly expand its area and match the global environment. The learner aug-mented with mappings can create different local policies for different regionsof the global environment, all of which can then be distilled against the cen-tral global policy. This can be considered as a representational sharingadvancement on Guided Policy Search based methods. The proposed canbe extended to on-policy family of algorithms. Let’s posit that a local agentπl is an I-projection of global policy π. The distributions from multiple πlcan then act as guiding distributions pl that minimizes the Kullback-Leiblerdivergence DKL(pl||p). This comes very close to the policies defined onstart state slices presented in [73] and [74] algorithms though this approachstill requires only a single policy. Another interesting extension to EASEcould be to see if the local learner can continue to train a series of locallearners, each one slightly better than before, until they match the globalenvironment. This extension can be considered as a curriculum over theagent ability itself.5.2.3 EASE with feature extractionThe core concept of EASE lies within the ability of an agent to learn asmall section of the global environment, which is spatio-temporally recur-ring. With simple mappings and group coordinate transformations, other1015.2. FUTURE WORKsections of the global environment are forced to look like the original train-ing section, thereby utilizing agent skills beyond their training. It would beinteresting to see if a feature extraction model could be learned that is ableto identify multiple recurring features throughout the global environmentand construct an optimal local environment for the agent to learn and ex-pand.The algorithmic innovations and technical advancements discussed in thepresented work through the concept of EASE encourages the research fieldof reinforcement learning to explore the benefits of training in abstractlysimilar environments. With the current direction of the field, most researchfocus is expended towards end-to-end learning and improving already satu-rated agents. Automated symmetries and learned mappings within differentenvironments would also allow for emerging behaviors and patterns thatcannot be defined a-priori. This creates a very exciting direction of re-search within the field with task agnostic applications and implementations.102Bibliography[1] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “Imagenet classificationwith deep convolutional neural networks,” Advances in neural infor-mation processing systems, vol. 25, pp. 1097–1105, 2012. → pages3[2] D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang,A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton, Y. Chen, T. Lillicrap,F. Hui, L. Sifre, G. van den Driessche, T. Graepel, and D. Hassabis,“Mastering the game of Go without human knowledge,” Nature, vol.550, no. 7676, pp. 354–359, oct 2017. → pages 3, 15[3] D. Amodei and D. Hernandez, “Ai and compute,” May 16 2018 Online.→ pages 3[4] T. L. Griffiths, F. Callaway, M. B. Chang, E. Grant, P. M. Krueger,and F. Lieder, “Doing more with less: meta-reasoning and meta-learning in humans and machines,” Current Opinion in BehavioralSciences, vol. 29, pp. 24–30, 2019. → pages 3[5] M. T. Cox and A. Raja, Metareasoning: Thinking about thinking.MIT Press, 2011. → pages 3103[6] R. Bellman, “Dynamic programming,” Science, vol. 153, no. 3731, pp.34–37, 1966. → pages 3[7] S. Russell, Human compatible: Artificial intelligence and the problemof control. Penguin, 2019. → pages 3, 6[8] P. Kormushev, S. Calinon, and D. Caldwell, “ReinforcementLearning in Robotics: Applications and Real-World Challenges,”Robotics, vol. 2, no. 3, pp. 122–148, jul 2013. [Online]. Available:http://www.mdpi.com/2218-6581/2/3/122 → pages 4, 19[9] X. Xu, L. Zuo, and Z. Huang, “Reinforcement learning algorithmswith function approximation: Recent advances and applications,” In-formation Sciences, vol. 261, pp. 1–31, mar 2014. → pages 5[10] A. Gosavi, “Reinforcement Learning: A Tutorial Survey and RecentAdvances,” Tech. Rep. → pages 5[11] Y. Duan, X. Chen, C. X. B. Edu, J. Schulman, P. Abbeel, and P. B.Edu, “Benchmarking Deep Reinforcement Learning for ContinuousControl,” Tech. Rep., 2016. [Online]. Available: https://github.com/→ pages 5, 35[12] T. Barrett, W. Clements, J. Foerster, and A. Lvovsky, “Exploratorycombinatorial optimization with reinforcement learning,” in Proceed-ings of the AAAI Conference on Artificial Intelligence, vol. 34, no. 04,2020, pp. 3243–3250. → pages 5, 58[13] I. Bello, H. Pham, Q. V. Le, M. Norouzi, and S. Bengio, “Neural com-104binatorial optimization with reinforcement learning,” arXiv preprintarXiv:1611.09940, 2016. → pages 5[14] A. G. Barto and S. Mahadevan, “Recent Advances in HierarchicalReinforcement Learning,” pp. 343–379+382, oct 2003. → pages 6, 25[15] H. A. Simon, “The architecture of complexity,” in Facets of systemsscience. Springer, 1991, pp. 457–476. → pages 6[16] K. Gupta, B. J. Emran, and H. Najjaran, “Vision-based pose estima-tion of a multi-rotor unmanned aerial vehicle,” International Journalof Intelligent Unmanned Systems, pp. IJIUS–10–2018–0030, apr 2019.→ pages 11[17] E. Nowak, K. Gupta, and H. Najjaran, “Development of a Plug-and-Play Infrared Landing System for Multirotor Unmanned Aerial Vehi-cles,” in 2017 14th Conference on Computer and Robot Vision (CRV).IEEE, may 2017, pp. 256–260. → pages 11[18] K. Gupta, M. Ko¨rber, F. Krebs, and H. Najjaran, “Vision-based de-formation and wrinkle detection for semi-finished fiber products oncurved surfaces,” in IEEE International Conference on AutomationScience and Engineering, vol. 2018-August. IEEE, aug 2018, pp.618–623. → pages 11[19] K. Gupta, M. Ko¨rber, A. Djavadifar, F. Krebs, and H. Najjaran,“Wrinkle and boundary detection of fiber products in robotic com-posites manufacturing,” Assembly Automation, 2019. → pages 11105[20] A. Djavadifar, J. B. Graham-Knight, K. Gupta, M. Ko¨rber,P. Lasserre, and H. Najjaran, “Robot-assisted composite manufactur-ing based on machine learning applied to multi-view computer vision,”in International Conference on SMART MULTIMEDIA. San Diego:Springer, 2019, p. Submitted. → pages 11[21] X. Chen, Y. Chen, K. Gupta, J. Zhou, and H. Najjaran, “SliceNet:A proficient model for real-time 3D shape-based recognition,” Neuro-computing, vol. 316, pp. 144–155, nov 2018. → pages 12[22] M. O’brien, K. Gupta, B. Emran, and H. Najjaran, “Autonomous Ve-hicle Control: Teaching Tool and Simulation,” in Proceedings - Fron-tiers in Education Conference, FIE, vol. 2018-Oct. IEEE, oct 2019,pp. 1–9. → pages 12[23] D. Mukherjee, K. Gupta, L. H. Chang, and H. Najjaran, “A survey ofrobot learning strategies for human-robot collaboration in industrialsettings,” Robotics and Computer-Integrated Manufacturing, vol. 73,p. 102231, 2022. → pages 12[24] K. Gupta and H. Najjaran, “Curriculum-based deep reinforcementlearning for adaptive robotics: A mini-review,” International Journalof Robotic Engineering, vol. 6, no. 1, 2021. → pages 12[25] K. Gupta, D. Mukherjee, and H. Najjaran, “Extending the capabilitiesof reinforcement learning through curriculum: A review of methodsand applications,” SN Computer Science, vol. 3, no. 1, pp. 1–18, 2022.→ pages 12, 26106[26] R. S. Sutton and A. G. Barto, Reinforcement Learning, An Introduc-tion, second edition, 2017. → pages 15[27] K. Arulkumaran, M. P. Deisenroth, M. Brundage, and A. A. Bharath,“A Brief Survey of Deep Reinforcement Learning,” aug 2017. → pages15[28] O. Five, “OpenAI Five,” 2018. → pages 15[29] DeepMind, “AlphaStar: Mastering the Real-Time Strategy GameStarCraft II — DeepMind,” 2018. → pages 15[30] A. Mart´ınez-Tenor, J. A. Ferna´ndez-Madrigal, A. Cruz-Mart´ın, andJ. Gonza´lez-Jime´nez, “Towards a common implementation of rein-forcement learning for multiple robotic tasks,” Expert Systems withApplications, vol. 100, pp. 246–259, jun 2018. → pages 15[31] M. Vecerik, T. Hester, J. Scholz, F. Wang, O. Pietquin, B. Piot,N. Heess, T. Rotho¨rl, T. Lampe, and M. Riedmiller, “LeveragingDemonstrations for Deep Reinforcement Learning on RoboticsProblems with Sparse Rewards,” jul 2017. [Online]. Available:http://arxiv.org/abs/1707.08817 → pages 15, 23, 80[32] J. Kober, J. A. Bagnell, and J. Peters, “Reinforcement learning inrobotics: A survey,” The International Journal of Robotics Research,vol. 32, no. 11, pp. 1238–1274, sep 2013. [Online]. Available: http://journals.sagepub.com/doi/10.1177/0278364913495721 → pages 15[33] V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou,107D. Wierstra, and M. Riedmiller, “Playing Atari with Deep Reinforce-ment Learning,” Tech. Rep. → pages 15[34] V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G.Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski,S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Ku-maran, D. Wierstra, S. Legg, and D. Hassabis, “Human-level controlthrough deep reinforcement learning,” Nature, vol. 518, no. 7540, pp.529–533, feb 2015. → pages 15[35] H. Tang, R. Houthooft, D. Foote, A. Stooke, X. Chen, Y. Duan,J. Schulman, F. De Turck, and P. Abbeel, “#Exploration: A Studyof Count-Based Exploration for Deep Reinforcement Learning,” nov2016. → pages 18[36] M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton,G. Deepmind, and R. Munos, “Unifying Count-Based Exploration andIntrinsic Motivation,” Tech. Rep. → pages 18[37] J. Zhang, N. Wetzel, N. Dorka, J. Boedecker, and W. Burgard, “Sched-uled Intrinsic Drive: A Hierarchical Take on Intrinsically MotivatedExploration,” mar 2019. → pages 18[38] D. Pathak, P. Agrawal, A. A. Efros, and T. Darrell, “Curiosity-drivenExploration by Self-supervised Prediction,” may 2017. → pages 18[39] Y. Burda, H. Edwards, D. Pathak, A. Storkey, T. Darrell, andA. A. Efros, “Large-Scale Study of Curiosity-Driven Learning,” 7thInternational Conference on Learning Representations, ICLR 2019,108aug 2018. [Online]. Available: http://arxiv.org/abs/1808.04355 →pages 18, 20[40] M. Jaderberg, V. Mnih, W. M. Czarnecki, T. Schaul, J. Z.Leibo, D. Silver, and K. Kavukcuoglu, “Reinforcement Learningwith Unsupervised Auxiliary Tasks,” 5th International Conference onLearning Representations, ICLR 2017 - Conference Track Proceedings,nov 2016. [Online]. Available: http://arxiv.org/abs/1611.05397 →pages 18, 20[41] J. Schulman, S. Levine, P. Moritz, M. I. Jordan, and P. Abbeel, “TrustRegion Policy Optimization,” feb 2015. → pages 18[42] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov,“Proximal Policy Optimization Algorithms,” jul 2017. → pages 18[43] Y. Wu, E. Mansimov, S. Liao, R. Grosse, and J. Ba, “Scalabletrust-region method for deep reinforcement learning using Kronecker-factored approximation,” aug 2017. → pages 19[44] T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft Actor-Critic:Off-Policy Maximum Entropy Deep Reinforcement Learning with aStochastic Actor,” jan 2018. → pages 19[45] H. Liu, A. Trott, R. Socher, and C. Xiong, “CompetitiveExperience Replay,” 7th International Conference on LearningRepresentations, ICLR 2019, feb 2019. [Online]. Available: http://arxiv.org/abs/1902.00528 → pages 19, 20109[46] T. Schaul, J. Quan, I. Antonoglou, and D. Silver, “Prioritized Expe-rience Replay,” nov 2015. → pages 19[47] M. Andrychowicz, F. Wolski, A. Ray, J. Schneider, R. Fong,P. Welinder, B. McGrew, J. Tobin, P. Abbeel, and W. Zaremba,“Hindsight Experience Replay,” Tech. Rep., 2017. [Online]. Available:https://goo.gl/SMrQnI. → pages 19, 20, 36[48] G. Schoettler, A. Nair, J. Luo, S. Bahl, J. A. Ojea, E. Solowjow,and S. Levine, “Deep Reinforcement Learning for Industrial InsertionTasks with Visual Inputs and Natural Rewards,” jun 2019. → pages19[49] S. H. Huang, M. Zambelli, J. Kay, M. F. Martins, Y. Tassa, P. M.Pilarski, and R. Hadsell, “Learning Gentle Object Manipulation withCuriosity-Driven Deep Reinforcement Learning,” mar 2019. → pages19[50] M. Hessel, H. Soyer, L. Espeholt, W. Czarnecki, S. Schmitt, and H. vanHasselt, “Multi-Task Deep Reinforcement Learning with PopArt,”Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33,pp. 3796–3803, jul 2019. → pages 19[51] A. Goyal, S. Sodhani, J. Binas, X. B. Peng, S. Levine, and Y. Bengio,“Reinforcement Learning with Competitive Ensembles of Information-Constrained Primitives,” jun 2019. → pages 19[52] D. Lyu, F. Yang, B. Liu, and S. Gustafson, “SDRL: Interpretableand Data-Efficient Deep Reinforcement Learning Leveraging Symbolic110Planning,” Proceedings of the AAAI Conference on Artificial Intelli-gence, vol. 33, pp. 2970–2977, jul 2019. → pages 19[53] J. Fu, A. Korattikara, S. Levine, and S. Guadarrama, “From Languageto Goals: Inverse Reinforcement Learning for Vision-Based InstructionFollowing,” feb 2019. → pages 19[54] S. G. Khan, G. Herrmann, F. L. Lewis, T. Pipe, and C. Melhuish,“Reinforcement learning and optimal adaptive control: An overviewand implementation examples,” Annual Reviews in Control, vol. 36,no. 1, pp. 42–59, apr 2012. → pages 19[55] M. P. Deisenroth, P. Englert, J. Peters, and D. Fox, “Multi-task policysearch for robotics,” in Proceedings - IEEE International Conferenceon Robotics and Automation. Institute of Electrical and ElectronicsEngineers Inc., sep 2014, pp. 3876–3881. → pages 19[56] X. B. Peng, M. Andrychowicz, W. Zaremba, and P. Abbeel, “Sim-to-Real Transfer of Robotic Control with Dynamics Randomization,”in Proceedings - IEEE International Conference on Robotics and Au-tomation. Institute of Electrical and Electronics Engineers Inc., sep2018, pp. 3803–3810. → pages 20[57] J. Tan, T. Zhang, E. Coumans, A. Iscen, Y. Bai, D. Hafner,S. Bohez, and V. Vanhoucke, “Sim-to-Real: Learning AgileLocomotion For Quadruped Robots,” apr 2018. [Online]. Available:http://arxiv.org/abs/1804.10332 → pages 20[58] Y. Chebotar, A. Handa, V. Makoviychuk, M. MacKlin, J. Issac,111N. Ratliff, and D. Fox, “Closing the sim-to-real Loop: Adaptingsimulation randomization with real world experience,” in Proceedings- IEEE International Conference on Robotics and Automation, vol.2019-May. Institute of Electrical and Electronics Engineers Inc., may2019, pp. 8973–8979. → pages 20[59] A. A. Rusu, M. Vecerik, T. Rotho¨rl, N. Heess, R. Pascanu, andR. Hadsell, “Sim-to-Real Robot Learning from Pixels with ProgressiveNets,” oct 2016. [Online]. Available: http://arxiv.org/abs/1610.04286→ pages 20[60] A. G. Barto, “Intrinsic motivation and reinforcement learning,” inIntrinsically Motivated Learning in Natural and Artificial Systems.Springer-Verlag Berlin Heidelberg, nov 2013, vol. 9783642323751, pp.17–47. → pages 20[61] M. G. Bellemare, S. Srinivasan, G. Ostrovski, T. Schaul, D. Saxton,G. Deepmind, and R. Munos, “Unifying Count-Based Exploration andIntrinsic Motivation,” Tech. Rep., 2016. → pages 20[62] M. Frank, J. Leitner, M. Stollenga, A. Fo¨rster, and J. Schmidhuber,“Curiosity driven reinforcement learning for motion planning onhumanoids,” Frontiers in Neurorobotics, vol. 7, no. JAN, p. 25,jan 2014. [Online]. Available: http://journal.frontiersin.org/article/10.3389/fnbot.2013.00025/abstract → pages 20[63] S. Still and D. Precup, “An information-theoretic approach to112curiosity-driven reinforcement learning,” Theory in Biosciences, vol.131, no. 3, pp. 139–148, sep 2012. → pages 20[64] J. Achiam and S. Sastry, “Surprise-Based Intrinsic Motivationfor Deep Reinforcement Learning,” mar 2017. [Online]. Available:http://arxiv.org/abs/1703.01732 → pages 20[65] N. Ranasinghe and W. M. Shen, “Surprise-based learning for devel-opmental robotics,” in Proceedings of the 2008 ECSIS Symposium onLearning and Adaptive Behaviors for Robotic Systems, LAB-RS 2008,2008, pp. 65–70. → pages 20[66] A. White, J. Modayil, and R. S. Sutton, “Surprise and Curiosityfor Big Data Robotics,” Tech. Rep., jun 2014. [Online]. Available:www.aaai.org → pages 20[67] J. Peters and S. Schaal, “Natural Actor-Critic,” Neurocomputing,vol. 71, no. 7-9, pp. 1180–1190, mar 2008. → pages 20[68] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan,V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine, “SoftActor-Critic Algorithms and Applications,” dec 2018. [Online].Available: http://arxiv.org/abs/1812.05905 → pages 20[69] V. Mnih, A. Puigdome`nech Badia, M. Mirza, A. Graves, T. Harley,T. P. Lillicrap, D. Silver, and K. Kavukcuoglu, “Asynchronous Meth-ods for Deep Reinforcement Learning,” Tech. Rep., 2016. → pages20113[70] G. Konidaris and A. Barto, “Skill Discovery in Continuous Reinforce-ment Learning Domains using Skill Chaining,” Tech. Rep., 2009. →pages 20[71] A. S. Lakshminarayanan, R. Krishnamurthy, P. Kumar, andB. Ravindran, “Option Discovery in Hierarchical ReinforcementLearning using Spatio-Temporal Clustering,” may 2016. [Online].Available: http://arxiv.org/abs/1605.05359 → pages 20[72] P. Mirowski, R. Pascanu, F. Viola, H. Soyer, A. J. Ballard, A. Banino,M. Denil, R. Goroshin, L. Sifre, K. Kavukcuoglu, D. Kumaran, andR. Hadsell, “Learning to navigate in complex environments,” in 5thInternational Conference on Learning Representations, ICLR 2017 -Conference Track Proceedings. International Conference on LearningRepresentations, ICLR, nov 2019. → pages 20[73] S. Levine and V. Koltun, “Guided policy search,” in Internationalconference on machine learning. PMLR, 2013, pp. 1–9. → pages 23,101[74] D. Ghosh, A. Singh, A. Rajeswaran, V. Kumar, and S. Levine,“Divide-and-conquer reinforcement learning,” arXiv preprintarXiv:1711.09874, 2017. → pages 23, 101[75] Y. W. Teh, V. Bapst, W. M. Czarnecki, J. Quan, J. Kirkpatrick,R. Hadsell, N. Heess, and R. Pascanu, “Distral: Robust multitaskreinforcement learning,” 2017. → pages 23114[76] I. Mordatch and E. Todorov, “Combining the benefits of function ap-proximation and trajectory optimization.” in Robotics: Science andSystems, vol. 4, 2014. → pages 23[77] Y. Bengio, J. Louradour, R. Collobert, and J. Weston, “Curriculumlearning,” in ACM International Conference Proceeding Series,vol. 382. New York, New York, USA: ACM Press, 2009, pp.1–8. [Online]. Available: http://portal.acm.org/citation.cfm?doid=1553374.1553380 → pages 24[78] M. M. Botvinick, “Hierarchical reinforcement learning and decisionmaking,” pp. 956–962, dec 2012. → pages 25[79] A. S. Vezhnevets, S. Osindero, T. Schaul, N. Heess, M. Jaderberg,D. Silver, and K. Kavukcuoglu, “FeUdal Networks for HierarchicalReinforcement Learning,” 34th International Conference on MachineLearning, ICML 2017, vol. 7, pp. 5409–5418, mar 2017. [Online].Available: http://arxiv.org/abs/1703.01161 → pages 25[80] S. Narvekar, B. Peng, M. Leonetti, J. Sinapov, M. E. Taylor, andP. Stone, “Curriculum learning for reinforcement learning domains: Aframework and survey,” arXiv preprint arXiv:2003.04960, 2020. →pages 26[81] M. Lopes and P. Y. Oudeyer, “The strategic student approach forlife-long exploration and learning,” in 2012 IEEE International Con-ference on Development and Learning and Epigenetic Robotics, ICDL2012, 2012. → pages 28115[82] B. Clement, D. Roy, P.-Y. Oudeyer, and M. Lopes, “Multi-ArmedBandits for Intelligent Tutoring Systems,” oct 2013. [Online].Available: http://arxiv.org/abs/1310.3174 → pages 28[83] L. Torrey and M. E. Taylor, “Teaching on a budget: Agents advisingagents in reinforcement learning,” in International Conference on Au-tonomous Agents and Multiagent Systems (AAMAS), 2013. → pages28[84] F. Leno Da Silva, R. Glatt, and A. H. Reali Costa, “SimultaneouslyLearning and Advising in Multiagent Reinforcement Learning,” Tech.Rep. [Online]. Available: www.ifaamas.org → pages 28[85] M. Zimmer, P. Viappiani, and P. Weng, “Teacher-Student Framework:a Reinforcement Learning Approach,” Tech. Rep., 2014. [Online].Available: https://hal.archives-ouvertes.fr/hal-01215273 → pages 28[86] T. Matiisen, A. Oliver, T. Cohen, and J. Schulman, “Teacher-StudentCurriculum Learning,” IEEE Transactions on Neural Networksand Learning Systems, pp. 1–9, jul 2017. [Online]. Available:http://arxiv.org/abs/1707.00183 → pages 28, 30[87] A. Graves, M. G. Bellemare, J. Menick, R. Munos, andK. Kavukcuoglu, “Automated Curriculum Learning for NeuralNetworks,” 34th International Conference on Machine Learning,ICML 2017, vol. 3, pp. 2120–2129, apr 2017. [Online]. Available:http://arxiv.org/abs/1704.03003 → pages 29[88] R. Portelas, C. Colas, K. Hofmann, and P.-Y. Oudeyer, “Teacher116algorithms for curriculum learning of Deep RL in continuouslyparameterized environments,” oct 2019. [Online]. Available: http://arxiv.org/abs/1910.07224 → pages 29[89] R. Van Der Linden, R. Lopes, and R. Bidarra, “DesigningProcedurally Generated Levels,” Tech. Rep., nov 2013. [Online].Available: www.aaai.org → pages 29[90] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman,J. Tang, and W. Zaremba, “Openai gym,” 2016. → pages 30, 33, 37[91] X. Kong, B. Xin, F. Liu, and Y. Wang, “Revisiting the Master-SlaveArchitecture in Multi-Agent Deep Reinforcement Learning,” dec2017. [Online]. Available: http://arxiv.org/abs/1712.07305 → pages30[92] C. Florensa, D. Held, X. Geng, and P. Abbeel, “Automatic GoalGeneration for Reinforcement Learning Agents,” 35th InternationalConference on Machine Learning, ICML 2018, vol. 4, pp. 2458–2471,may 2017. [Online]. Available: http://arxiv.org/abs/1705.06366 →pages 31, 32[93] I. J. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,S. Ozair, A. Courville, and Y. Bengio, “Generative adversarial nets,”in Advances in Neural Information Processing Systems, vol. 3, no.January. Neural information processing systems foundation, jun 2014,pp. 2672–2680. → pages 31[94] C. Florensa, D. Held, M. Wulfmeier, M. Zhang, and P. Abbeel,117“Reverse Curriculum Generation for Reinforcement Learning,” jul2017. [Online]. Available: http://arxiv.org/abs/1707.05300 → pages32[95] B. Ivanovic, J. Harrison, A. Sharma, M. Chen, and M. Pavone, “BaRC:Backward reachability curriculum for robotic reinforcement learning,”in Proceedings - IEEE International Conference on Robotics and Au-tomation, vol. 2019-May. Institute of Electrical and Electronics En-gineers Inc., may 2019, pp. 15–21. → pages 32[96] S. Forestier, Y. Mollard, and P.-Y. Oudeyer, “Intrinsically MotivatedGoal Exploration Processes with Automatic Curriculum Learning,”aug 2017. [Online]. Available: http://arxiv.org/abs/1708.02190 →pages 33[97] M. Eppe, S. Magg, and S. Wermter, “Curriculum goal masking forcontinuous deep reinforcement learning,” in 2019 Joint IEEE 9th In-ternational Conference on Development and Learning and EpigeneticRobotics, ICDL-EpiRob 2019. Institute of Electrical and ElectronicsEngineers Inc., aug 2019, pp. 183–188. → pages 33[98] P. Fournier, O. Sigaud, M. Chetouani, and P.-Y. Oudeyer, “Accuracy-based Curriculum Learning in Deep Reinforcement Learning,” jun2018. [Online]. Available: http://arxiv.org/abs/1806.09614 → pages34[99] D. Silver, T. Hubert, J. Schrittwieser, I. Antonoglou, M. Lai, A. Guez,M. Lanctot, L. Sifre, D. Kumaran, T. Graepel, T. Lillicrap, K. Si-118monyan, and D. Hassabis, “A general reinforcement learning algorithmthat masters chess, shogi, and Go through self-play,” Science, vol. 362,no. 6419, pp. 1140–1144, dec 2018. → pages 34[100] S. Sukhbaatar, Z. Lin, I. Kostrikov, G. Synnaeve, A. Szlam,and R. Fergus, “Intrinsic Motivation and Automatic Curricula viaAsymmetric Self-Play,” 6th International Conference on LearningRepresentations, ICLR 2018 - Conference Track Proceedings, mar2017. [Online]. Available: http://arxiv.org/abs/1703.05407 → pages34[101] S. Sukhbaatar, A. Szlam, G. Synnaeve, S. Chintala, and R. Fergus,“MazeBase: A Sandbox for Learning from Games,” nov 2015.[Online]. Available: http://arxiv.org/abs/1511.07401 → pages 35[102] A. Saito, “Curriculum Learning Based on Reward Sparseness for DeepReinforcement Learning of Task Completion Dialogue Management,”pp. 46–51, 2018. → pages 35[103] K. Shao, Y. Zhu, and D. Zhao, “StarCraft Micromanagement WithReinforcement Learning and Curriculum Transfer Learning,” IEEETransactions on Emerging Topics in Computational Intelligence,vol. 3, no. 1, pp. 73–84, apr 2018. → pages 35[104] Y. Wu and Y. Tian, “Training Agent for First-Person Shooter Gamewith Actor-Critic Curriculum Learning,” Tech. Rep. [Online]. Avail-able: http://vizdoom.cs.put.edu.pl/competition-cig-2016/results →pages 35119[105] N. Heess, D. TB, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa,T. Erez, Z. Wang, S. M. A. Eslami, M. Riedmiller, and D. Silver,“Emergence of Locomotion Behaviours in Rich Environments,” jul2017. [Online]. Available: http://arxiv.org/abs/1707.02286 → pages35[106] J. K. Gupta, M. Egorov, and M. Kochenderfer, “Cooperative Multi-agent Control Using Deep Reinforcement Learning,” in Lecture Notesin Computer Science (including subseries Lecture Notes in ArtificialIntelligence and Lecture Notes in Bioinformatics), vol. 10642 LNAI.Springer Verlag, 2017, pp. 66–83. → pages 36[107] A. Karpathy and M. Van De Panne, “Curriculum learning for mo-tor skills,” in Lecture Notes in Computer Science (including subseriesLecture Notes in Artificial Intelligence and Lecture Notes in Bioin-formatics), vol. 7310 LNAI. Springer, Berlin, Heidelberg, 2012, pp.325–330. → pages 36[108] Z. Qiao, K. Muelling, J. M. Dolan, P. Palanisamy, and P. Mudalige,“Automatically Generated Curriculum based Reinforcement Learningfor Autonomous Vehicles in Urban Environment,” in IEEE IntelligentVehicles Symposium, Proceedings, vol. 2018-June. Institute of Electri-cal and Electronics Engineers Inc., oct 2018, pp. 1233–1238. → pages36[109] N. Justesen and S. Risi, “Automated Curriculum Learning by Reward-ing Temporally Rare Events,” in IEEE Conference on Computatonal120Intelligence and Games, CIG, vol. 2018-August. IEEE ComputerSociety, oct 2018. → pages 36[110] M. Fang, T. Zhou, Y. Du, L. Han, and Z. Zhang, “Curriculum-guidedHindsight Experience Replay,” in Advances in Neural InformationProcessing Systems 32, 2019, pp. 12 623–12 634. [Online]. Available:https://github.com/mengf1/CHER. → pages 36[111] Z. Ren, D. Dong, H. Li, and C. Chen, “Self-Paced Prioritized Curricu-lum Learning with Coverage Penalty in Deep Reinforcement Learn-ing,” IEEE Transactions on Neural Networks and Learning Systems,vol. 29, no. 6, pp. 2216–2226, jun 2018. → pages 37[112] F. Leno Da Silva and A. H. Reali Costa, “Object-Oriented Curricu-lum Generation for Reinforcement Learning,” in Proceedings of the17th International Conference on Autonomous Agents and MultiAgentSystems, Stockholm, Sweden, 2018, pp. 1026–1034. → pages 37[113] M. Svetlik, M. Leonetti, J. Sinapov, R. Shah, N. Walker,and P. Stone, “Automatic Curriculum Graph Generation forReinforcement Learning Agents,” Tech. Rep. [Online]. Available:www.aaai.org → pages 37[114] W. M. Czarnecki, S. M. Jayakumar, M. Jadcrbcrg, L. Hasenclever,Y. W. Tch, S. Osindero, N. Heess, and R. Pascanu, “Mix & match- Agent curricula for reinforcement learning,” in 35th InternationalConference on Machine Learning, ICML 2018, vol. 3. International121Machine Learning Society (IMLS), jun 2018, pp. 1761–1773. → pages38[115] G. Berseth, C. Xie, P. Cernek, and M. Van de Panne, “ProgressiveReinforcement Learning with Distillation for Multi-Skilled MotionControl,” 6th International Conference on Learning Representations,ICLR 2018 - Conference Track Proceedings, feb 2018. [Online].Available: http://arxiv.org/abs/1802.04765 → pages 38[116] T. P. Lillicrap, J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa,D. Silver, and D. Wierstra, “Continuous control with deep reinforce-ment learning,” sep 2015. → pages 39[117] H. van Hasselt, A. Guez, and D. Silver, “Deep Reinforcement Learningwith Double Q-learning,” sep 2015. → pages 39[118] D. Silver, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller, “De-terministic Policy Gradient Algorithms,” Tech. Rep. → pages 39[119] Z. Wang, T. Schaul, M. Hessel, H. Van Hasselt, M. Lanctot, andN. De Frcitas, “Dueling Network Architectures for Deep Reinforce-ment Learning,” in 33rd International Conference on Machine Learn-ing, ICML 2016, vol. 4. International Machine Learning Society(IMLS), nov 2016, pp. 2939–2947. → pages 39[120] Z. Wang, V. Bapst, N. Heess, V. Mnih, R. Munos, K. Kavukcuoglu,and N. de Freitas, “Sample Efficient Actor-Critic with Experience Re-play,” nov 2016. → pages 39122[121] T.-H. Teng, A.-H. Tan, and J. M. Zurada, “Self-organizing neuralnetworks integrating domain knowledge and reinforcement learning,”IEEE transactions on neural networks and learning systems, vol. 26,no. 5, pp. 889–902, 2014. → pages 49[122] P. Abbeel and A. Y. Ng, “Apprenticeship learning via inverse rein-forcement learning,” in Proceedings of the twenty-first internationalconference on Machine learning, 2004, p. 1. → pages 77[123] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba, and P. Abbeel,“Overcoming exploration in reinforcement learning with demonstra-tions,” in 2018 IEEE International Conference on Robotics and Au-tomation (ICRA). IEEE, 2018, pp. 6292–6299. → pages 77, 79,81[124] A. Mahajan and T. Tulabandhula, “Symmetry learning for func-tion approximation in reinforcement learning,” arXiv preprintarXiv:1706.02999, 2017. → pages 78[125] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman,E. Todorov, and S. Levine, “Learning complex dexterous manipulationwith deep reinforcement learning and demonstrations,” arXiv preprintarXiv:1709.10087, 2017. → pages 80[126] Y. Gao, H. Xu, J. Lin, F. Yu, S. Levine, and T. Darrell, “Rein-forcement learning from imperfect demonstrations,” arXiv preprintarXiv:1802.05313, 2018. → pages 80[127] D. Ghosh, A. Gupta, J. Fu, A. Reddy, C. Devin, B. Eysenbach, and123S. Levine, “Learning to reach goals without reinforcement learning,”2019. → pages 81124
Thesis/Dissertation
eng
Electrical Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Najjaran, Homayoun
Reinforcement learning in complex environments with locally trained naïve agents
Text