Open Collections

Fuhrman, Robert

University of British Columbia

Speech communication is a multimodal process. In this process, the act of producing symbolic linguistic structure affects the patterning of various energy media in the surround, which can be sensed, and subsequently transformed, into neural processes which likewise influence the cognitive state of a perceiver. With respect to vision in particular, there is likewise widespread recognition that the distribution of perceivers’ attention – where they fixate- plays a fundamental role in this process, but there is still considerable uncertainty as to what information in the visible signal is beneficial to perceivers, and why fixation patterns are structured in a way which tends to be distributed across various physiological structures, including, but not limited to, both the eyes and the mouth. In this dissertation, I approach this problem from a computational perspective, specifically via the development of a Deep Reinforcement Learning (DRL) agent that learns to adjust its visual fixations in order to predict the spatiotemporal structure of the acoustic speech signal. The reward signal that is maximized by this agent is based upon an ecologically oriented optimization principle – the Maximum Power Principle (MPP; [1, 2]) – which asserts that self-organizing living systems behave in such a way as to maximize the throughput of available energies in the environment. In experiments with two Audiovisual Speech datasets, I demonstrate that the agent’s fixation behaviours, when structured optimally with respect to this criterion, support successful learning of cross-modal relations between visible and auditory speech. More generally, I also suggest that this explicit link between the algorithmic notion of reward – which has recently been proposed to be sufficient for explaining all manner of intelligent behavior [3] - with the physical notion of energy could serve to explain how generally intelligent behavior manifests across all kinds of different environments: a universal heuristic that may well apply to biological and computational organisms alike.

Text

2025-03-17

Attribution-NonCommercial-NoDerivatives 4.0 International

http://hdl.handle.net/2429/90496

Linguistics

University of British Columbia

UBCV

http://creativecommons.org/licenses/by-nc-nd/4.0/