Open Collections

Ajisafe, Daniel Abidemi

University of British Columbia

Human motion capture either requires multi-camera systems or is unreliable using single-view input due to depth ambiguities. Meanwhile, mirrors are readily available in urban environments and can take the role of additional views. When picturing a person in front of a mirror, the mirror image provides a second view of the person using only a single camera. Prior work has hence exploited this additional constraint to improve 3D human pose reconstruction. Going beyond existing mirror approaches, we utilize mirrors for learning a complete body model, including shape and appearance. Our main contribution is extending articulated neural radiance fields (NERFs) to include a notion of a mirror and making it sample-efficient. We integrate this into an entire system that succeeds without any 3D annotation by automatically calibrating the camera, estimating mirror orientation, and subsequently lifting 2D keypoint detections to 3D skeleton pose that is used to condition the mirror-aware NeRF. We empirically demonstrate the benefit of learning a body model and accounting for mirror-occlusion in challenging mirror scene setups. We show continuous improvements on time-varying articulated 3D joint estimation, reconstruct the body geometry from only mirror images and 2D detections, and synthesize novel views from unobserved viewpoints.

Text

2023-01-20

Attribution 4.0 International

http://hdl.handle.net/2429/83704

Computer Science

University of British Columbia

UBCV

http://creativecommons.org/licenses/by/4.0/