Open Collections

Gou, Ruiyu

University of British Columbia

Deep reinforcement learning has had significant success at learning motor control tasks. Typically, these policies are fully closed loop or ‘state indexed’, implying a control policy that is queried at every control time step with the current state in order to estimate the best current action corresponding to that state. However, this approach ignores the inherent predictability of many systems, wherein the future states and actions are often quite predictable and can thus be controlled in an open-loop or ‘time indexed’ fashion. Chunking of action sequences is a well-established mechanism in cognitive systems to enhance learning capabilities and learning efficiency. By modeling actions in time-indexed chunks, one reduces the computational and perceptual demands required for control. Learning this type of temporal action abstraction remains under-explored. We present a method that learns a chunk-based state-and-time-indexed policy from any existing state-indexed reinforcement learning policy, with minimal added complexity. We show that with a straightforward multi-layer neural network, the chunk-based policy can decrease the required control frequency significantly. In particular, we show a reduction from 60Hz to 10Hz for the control of a 3D humanoid capable of robust and realistic movement across varying terrain. We further propose an adaptive runtime algorithm that can leverage long action chunks while reverting to single-step actions as needed in order to achieve robust behavior.

Text

2024-08-27

Attribution-NonCommercial-NoDerivatives 4.0 International

http://hdl.handle.net/2429/89057

Computer Science

University of British Columbia

UBCV

http://creativecommons.org/licenses/by-nc-nd/4.0/