Explicit and implicit warping for accurate human pose estimation and low-latency neural rendering
department or school: Science, Faculty of
department or school: Computer Science, Department of
data provider: DSpace
degree campus: UBCV
creator: Yu, Frank
issued: 2023-04-20T20:29:38Z
issued: 2023
related degree: Master of Science - MSc
degree grantor: University of British Columbia
description: Deep neural networks have become an integral part of modern advances in the field of computer vision. However, these solutions are not practical when relying on increasingly large neural networks and diverse datasets to scale. Prior works demonstrate that embedding domain/application-specific knowledge in both the architecture design and training procedure is one way to improve scalability. In this thesis, we propose two methods that leverage domain knowledge, defined through explicit and implicit warping, to create more data and runtime efficient networks in two applications. First, we compute an explicit warping to disentangle the learning of camera intrinsic parameters from the human pose estimation pipeline. Our explicit warping takes into account the region of interest and the camera's focal length to define a perspective-correct crop. By including this as a preprocess or end-to-end component in the network, we significantly increase performance, especially in cases where the subject is near the boundary of the image. Second, we leverage the knowledge that sequential frames in talking-head video conferencing have significant visual overlap. Therefore, we design a simple and effective implicit warping strategy between timesteps to greatly decrease the latency and increase the framerate of talking-head neural rendering. Our proposed methods demonstrate significant improvements in accuracy and latency in their respective applications.
aggregated CHO: https://circle.library.ubc.ca/rest/handle/2429/84370
note: Explicit and Implicit Warping for Accurate Human PoseEstimation and Low-Latency Neural RenderingbyFrank YuB.Sc., University of Manitoba, 2020A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFMaster of ScienceinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Computer Science)The University of British Columbia(Vancouver)April 2023© Frank Yu, 2023The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Explicit and Implicit Warping for Accurate Human Pose Estimationand Low-Latency Neural Renderingsubmitted by Frank Yu in partial fulfillment of the requirements for the degree ofMaster of Science in Computer Science.Examining Committee:Helge Rhodin, Assistant Professor, Computer Science, UBCSupervisorLeonid Sigal, Associate Professor, Computer Science, UBCSupervisory Committee MemberiiAbstractDeep neural networks have become an integral part of modern advances in thefield of computer vision. However, these solutions are not practical when relyingon increasingly large neural networks and diverse datasets to scale. Prior worksdemonstrate that embedding domain/application-specific knowledge in both the ar-chitecture design and training procedure is one way to improve scalability. In thisthesis, we propose two methods that leverage domain knowledge, defined throughexplicit and implicit warping, to create more data and runtime efficient networks intwo applications. First, we compute an explicit warping to disentangle the learn-ing of camera intrinsic parameters from the human pose estimation pipeline. Ourexplicit warping takes into account the region of interest and the camera’s focallength to define a perspective-correct crop. By including this as a preprocess orend-to-end component in the network, we significantly increase performance, es-pecially in cases where the subject is near the boundary of the image. Second,we leverage the knowledge that s…
has type: Thesis/Dissertation
date issued: 2023-05
is shown at: 10.14288/1.0431314
language: eng
degree discipline: Computer Science
provider: Vancouver : University of British Columbia Library
publisher: University of British Columbia
rights: Attribution-NonCommercial-NoDerivatives 4.0 International
rights URI: http://creativecommons.org/licenses/by-nc-nd/4.0/
scholar level: Graduate
contributor: Rhodin, Helge
type: Text
identifier URI: http://hdl.handle.net/2429/84370