Science, Faculty of
Computer Science, Department of
DSpace
UBCV
Yu, Frank
Master of Science - MSc
University of British Columbia
Deep neural networks have become an integral part of modern advances in the field of computer vision. However, these solutions are not practical when relying on increasingly large neural networks and diverse datasets to scale. Prior works demonstrate that embedding domain/application-specific knowledge in both the architecture design and training procedure is one way to improve scalability. In this thesis, we propose two methods that leverage domain knowledge, defined through explicit and implicit warping, to create more data and runtime efficient networks in two applications. First, we compute an explicit warping to disentangle the learning of camera intrinsic parameters from the human pose estimation pipeline. Our explicit warping takes into account the region of interest and the camera's focal length to define a perspective-correct crop. By including this as a preprocess or end-to-end component in the network, we significantly increase performance, especially in cases where the subject is near the boundary of the image. Second, we leverage the knowledge that sequential frames in talking-head video conferencing have significant visual overlap. Therefore, we design a simple and effective implicit warping strategy between timesteps to greatly decrease the latency and increase the framerate of talking-head neural rendering. Our proposed methods demonstrate significant improvements in accuracy and latency in their respective applications.
Explicit and Implicit Warping for Accurate Human PoseEstimation and Low-Latency Neural RenderingbyFrank YuB.Sc., University of Manitoba, 2020A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFMaster of ScienceinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Computer Science)The University of British Columbia(Vancouver)April 2023© Frank Yu, 2023The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Explicit and Implicit Warping for Accurate Human Pose Estimationand Low-Latency Neural Renderingsubmitted by Frank Yu in partial fulfillment of the requirements for the degree ofMaster of Science in Computer Science.Examining Committee:Helge Rhodin, Assistant Professor, Computer Science, UBCSupervisorLeonid Sigal, Associate Professor, Computer Science, UBCSupervisory Committee MemberiiAbstractDeep neural networks have become an integral part of modern advances in thefield of computer vision. However, these solutions are not practical when relyingon increasingly large neural networks and diverse datasets to scale. Prior worksdemonstrate that embedding domain/application-specific knowledge in both the ar-chitecture design and training procedure is one way to improve scalability. In thisthesis, we propose two methods that leverage domain knowledge, defined throughexplicit and implicit warping, to create more data and runtime efficient networks intwo applications. First, we compute an explicit warping to disentangle the learn-ing of camera intrinsic parameters from the human pose estimation pipeline. Ourexplicit warping takes into account the region of interest and the camera’s focallength to define a perspective-correct crop. By including this as a preprocess orend-to-end component in the network, we significantly increase performance, es-pecially in cases where the subject is near the boundary of the image. Second,we leverage the knowledge that sequential frames in talking-head video conferenc-ing have significant visual overlap. Therefore, we design a simple and effectiveimplicit warping strategy between timesteps to greatly decrease the latency andincrease the framerate of talking-head neural rendering. Our proposed methodsdemonstrate significant improvements in accuracy and latency in their respectiveapplications.iiiLay SummaryArtificial intelligence has produced impressive results in computer vision by learn-ing from large datasets. However, simply feeding data into a generic network is notscalable because further improvements require larger models and datasets, as theycannot utilize prior knowledge about the world (such as the laws of physics). Priorworks have shown how to improve these systems by including application-specificinformation, we follow in this line of work. For 3D human pose estimation, we canleverage information about how the camera projects 3D space onto a 2D sensorto improve model performance. Furthermore, for applications where we want tosynthesize a video of a person talking, similar to a teleconferencing setting, we canexploit the fact that objects move smoothly in space and time. Therefore, we canreuse neural network computations between time steps to reduce the latency of themodel.ivPrefaceAll work presented in this thesis was written and produced by Frank Yu underthe supervision of Professor Helge Rhodin. All software used and presented inthis work is written by Frank Yu unless otherwise stated, for example, based onprior codebases. A version of the work presented in Chapter 2 was originally pub-lished at The Conference on Computer Vision and Pattern Recognition (CVPR)2021 [90]. A version of the work presented in Chapter 3 was originally publishedat IEEE/CVF Winter Conference on Applications of Computer Vision (WACV)2023 [91] and Section 3.7 was presented at SIGGRAPH Asia XR 2022 [84]. FrankYu is the lead author of the CVPR and WACV work, and the second author of theSIGGRAPH Asia XR demonstration. Large portions of these works are includedin this thesis. Data collected for the work in Chapter 3 was done using the con-sent form from the Behavioural Research Ethics Board (BREB), reference numberH19-02997, with equipment provided by Frank Yu and Helge Rhodin. Further-more, access to end-to-end teleconferencing hardware was generously provided byProfessor Sidney Fels and the Human Communication Technologies (HCT) Lab.Computing resources used throughout my studies were generously provided byProfessor Helge Rhodin and Compute Canada.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xList of Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Accurate Human Pose Estimation via Explicit Warping . . . . . . . 32.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.3 Perspective Crop Layer (PCL) . . . . . . . . . . . . . . . . . . . . 82.3.1 Motivation and Rectangular Crops . . . . . . . . . . . . . 82.3.2 Defining a Virtual Camera . . . . . . . . . . . . . . . . . 92.3.3 Differentiable Network Layer . . . . . . . . . . . . . . . 152.3.4 Implementation Details . . . . . . . . . . . . . . . . . . . 18vi2.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182.4.1 PCL for 2D to 3D Keypoint Lifting . . . . . . . . . . . . . 232.4.2 Diversity of Crop Positions in H3.6M and MPI-INF-3DHP 242.4.3 PCL for Temporal Convolutional Neural Networks . . . . 242.4.4 PCL for Convolutional Neural Network Architectures . . . 262.4.5 Ablation Studies . . . . . . . . . . . . . . . . . . . . . . 262.5 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303 Latency-Reduced Neural Rendering for Telepresence Using ImplicitWarping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343.3 Preliminaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . 353.4 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 373.4.1 Operation Modes and Parallel Execution . . . . . . . . . . 393.4.2 Improved Neural Head Rendering . . . . . . . . . . . . . 413.5 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . 423.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 433.6.1 Latency and Runtime Improvement . . . . . . . . . . . . 463.6.2 Offline Reconstruction Quality . . . . . . . . . . . . . . . 473.6.3 Online Reconstruction Quality . . . . . . . . . . . . . . . 483.6.4 Novel View Synthesis Quality . . . . . . . . . . . . . . . 493.6.5 Retargeting . . . . . . . . . . . . . . . . . . . . . . . . . 503.6.6 Ablation Studies . . . . . . . . . . . . . . . . . . . . . . 513.7 Live System Integration . . . . . . . . . . . . . . . . . . . . . . . 543.8 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 564 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . . . 584.1 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 584.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605.1 General Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . 60viiBibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62A Supporting Materials . . . . . . . . . . . . . . . . . . . . . . . . . . 73A.1 Consent Form for Collecting Training Data . . . . . . . . . . . . 73viiiList of TablesTable 2.1 Reported MPJPE and PCK for 2D to 3D keypoint lifting testsperformed on H3.6M and MPI-INF-3DHP. . . . . . . . . . . . 22Table 2.2 Temporal CNN tests . . . . . . . . . . . . . . . . . . . . . . . 25Table 2.3 Results with 2D-3D keypoint lifting on H3.6M while decreas-ing the model complexity of our PCL embedded model (MLP +PCL) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28Table 2.4 Results with 2D-3D keypoint lifting on MPI-INF-3DHP whenapplying half and full pose rotations. . . . . . . . . . . . . . . 28Table 2.5 MPJPE in millimeters for all tests conducted on the Cube Datasetand its variations. . . . . . . . . . . . . . . . . . . . . . . . . 30Table 3.1 Offline evaluation on the beard dataset. . . . . . . . . . . . . . 48Table 3.2 High-fps comparison . . . . . . . . . . . . . . . . . . . . . . . 49Table 3.3 Ablation study results for our warping network . . . . . . . . . 52Table 3.4 Ablating caching layers for our warping network using thesequential variant (1x warp), showing the improvement afteradding each component on the beard dataset. . . . . . . . . . . 54ixList of FiguresFigure 2.1 Perspective effects and correction with PCL crops. . . . . . . . 4Figure 2.2 PCL virtual camera. . . . . . . . . . . . . . . . . . . . . . . . 9Figure 2.3 PCL focal length settings. . . . . . . . . . . . . . . . . . . . . 12Figure 2.4 Effect of virtual camera focal length. . . . . . . . . . . . . . . 14Figure 2.5 PCL integration into existing architectures. . . . . . . . . . . . 16Figure 2.6 3D pose change. . . . . . . . . . . . . . . . . . . . . . . . . 17Figure 2.7 H3.6M and MPI-INF-3DHP qualitative examples. . . . . . . 19Figure 2.8 Improvement of reconstruction error on H3.6M. . . . . . . . . 21Figure 2.9 Improvement of reconstruction error on MPI-INF-3DHP . . . 24Figure 2.10 2D histogram of hip joint locations for H3.6M and MPI-INF-3DHP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Figure 2.11 Evaluating the effectiveness of PCL on approximated camerafocal lengths . . . . . . . . . . . . . . . . . . . . . . . . . . 27Figure 2.12 Toy-Cube examples. . . . . . . . . . . . . . . . . . . . . . . 29Figure 3.1 Neural rendering with warping. . . . . . . . . . . . . . . . . 32Figure 3.2 Our low latency neural face-rendering pipeline . . . . . . . . 38Figure 3.3 Our rendering pipelines for sequential (top) and parallel exe-cution (bottom). . . . . . . . . . . . . . . . . . . . . . . . . . 40Figure 3.4 Reconstruction performance vs. latency (left) and frames persecond (FPS) (right) . . . . . . . . . . . . . . . . . . . . . . . 46Figure 3.5 Reconstructed frames and their error maps . . . . . . . . . . . 47Figure 3.6 Online face reconstruction setting. . . . . . . . . . . . . . . . 49Figure 3.7 Comparison against Wang et al. [82]. . . . . . . . . . . . . . 50xFigure 3.8 Novel views while retargeting. . . . . . . . . . . . . . . . . . 51Figure 3.9 Retargeting comparison . . . . . . . . . . . . . . . . . . . . 52Figure 3.10 Ablation on warp distance . . . . . . . . . . . . . . . . . . . 54Figure 3.11 Our end-to-end system (OpenTeleView). . . . . . . . . . . . . 55Figure 3.12 Block diagram of OpenTeleView. . . . . . . . . . . . . . . . 55Figure 3.13 Limitations of head reconstruction . . . . . . . . . . . . . . . 56xiList of AbbreviationsCNN convolutional neural networkCPU central processing unitCUDA Compute Unified Device ArchitectureDNR deferred neural rendererFPS frames per secondGAN generative adversarial networkGPU graphics processing unitGT ground truthH3.6M human 3.6 million datasetLPF low pass filterxiiMLP multilayer perceptronMPI-INF-3DHP Max Planck Institute for Informatics 3D human poseMPJPE mean per joint position errorNERF neural radiance fieldNN neural networkPCA principal component analysisPCK percentage of correct keypointsPCL Perspective Crop LayerPSNR peak signal-to-noise ratioRC root centeringROI region of interestRTX Ray Tracing Texel eXtremeSH spherical harmonicsxiiiSSIM structural similarity indexSTN Spatial Transformer NetworkT-CNN temporal convolutional neural networkVR virtual realityxivAcknowledgmentsI would first like to thank my advisor, Professor Helge Rhodin, for their con-stant support, guidance, and mentorship, always helping me to become a betterresearcher. I would also like to thank Professor Sidney Fels and the HCT Lab fortheir collaboration on the talking head project which was an incredible opportunityto learn and work on a multidisciplinary team. Furthermore, I want to thank Pro-fessor Leonid Sigal for their valuable feedback during the thesis writing processand for being a member of my thesis committee. I also thank my labmates and theoverall computer vision group at The University of British Columbia (UBC) forbeing a constant beacon of support, both inside and outside the lab. In particular,I would like to thank Shih-Yang Su and Professor Bastian Wandt for their invalu-able support throughout my degree. A big thanks also go out to Abi Kuganesan,Xingzhe He, Daniel Ajisafe, Luis Bolanos for helping me with everything fromproofreading papers to being a part of my projects. Another big thanks go out toKaseya Xia, Rosaline Baek, Emily Jia, and Beibei Xiong, without which we couldnot have completed our work on developing an end-to-end teleconferencing systemnor carried out a successful live demonstration.Lastly, I would like to thank my parents and brother for always supporting meand helping me through the ups and downs of my degree and for encouraging meto pursue my ambitions.xvChapter 1IntroductionIn recent years, there has been a surge in data-driven deep-learning methods; andat the center of these developments lies a neural network. Neural networks havedemonstrated tremendous representational capacities, producing state-of-the-art re-sults in applications including human pose estimation and reconstructing virtualhumans. However, these improvements come at the cost of an increasing need fordata and increased computation time during both training and inference for larger(more expressive) networks. However, for certain applications, it is not feasibleto collect an exorbitant amount of data. Furthermore, implementing larger andlarger networks to account for increased data availability is not a scalable solution.Therefore, it is critical to effectively use available data and design efficient neuralnetwork architectures. While black-box deep learning has shown impressive re-sults, further improvements can be introduced through explicit or implicit priorson network architecture, loss functions, or input data.In this thesis, we build on this idea and explore two methods that leveragedomain knowledge, through explicit and implicit warping to improve the perfor-mance of deep neural networks. We experiment with two applications, monocular3D human pose estimation, and talking-head synthesis. For the former, we aim toimprove the accuracy of the network without introducing more expressive networksor additional data. This is achieved by leveraging camera intrinsics to compute anexplicit image crop and warp to correct the perspective distortion in the image-capture process. Even though classic 3D human pose estimation methods used1accurate camera models the state-of-the-art deep learning models neglect perspec-tive effects. However, doing so forces the neural network backbone to learn how tomodel these effects. Moreover, learning the perspective effects from scratch is im-possible if the network is only given a local crop, without the context of where thecrop was taken. Our experiments show that this results in worse performance andcauses the underlying neural network to overfit to training focal lengths–thereforefailing to generalize to new focal lengths. In comparison, explicit modeling cancreate smaller neural networks with the same performance as their larger counter-parts. Furthermore, it is impractical to collect data for every possible pose in everypossible position in a camera, and for every focal length. By explicitly accountingfor these effects we show a 2-10% improvement compared to the baseline whichexcludes the explicit warping on average and up to a 25% improvement on regionswith strong perspective effects.For the latter application, we propose a system to reduce the end-to-end sys-tem latency of a neural talking-head synthesis network. We propose an implicitwarp for neural network features between timesteps to quickly synthesize futureframes. We define a learned implicit warping for a neural renderer rather than anexplicit warping paired with a traditional rendering pipeline due to the flexibilitythat jointly learning both the warping and the renderer provides. The key intuitionbehind our time-saving implicit warping is that in talking-head situations, thereis limited movement, mostly changes in expression. Therefore, most neural ren-dering approaches will be redundantly processing unchanged pixels in each framebetween timesteps. Therefore, by sharing, and conditioning a smaller network, op-erating on the current timestep, on the features of a larger network from a previoustime step, we can save a significant amount of time. Thereby enabling real-time3D teleconferencing by reducing the latency it takes to synthesize an image forthe viewer. By including the warping we are able to reduce the latency by up to70% when compared to the warping-less, per-frame, alternatives and increase theframerate by up to 300% with only a slight degradation to reconstruction quality.2Chapter 2Accurate Human Pose Estimationvia Explicit WarpingThe following text is taken from: Yu, Frank, et al. ”PCLs: Geometry-awareneural reconstruction of 3D pose with perspective crop layers.” Proceedingsof the IEEE/CVF Conference on Computer Vision and Pattern Recognition.2021.2.1 IntroductionConvolutional neural networks (CNNs) have proven highly effective for image-based prediction tasks because of their translation invariance and the locality of thecomputation they perform. For 3D pose estimation, this allows them to focus onimage locations that carry information about the pose while discarding other ones[13, 39, 43, 50, 58, 59, 67, 71, 73, 78, 86, 94].Convolutions in the image plane, however, ignore the perspective effects causedby projecting a 3D scene in 2D. For example, as shown in Figure 2.1, a person cap-tured by a static camera in a fixed pose and moving in a constant direction is seenfrom different angles as their image location changes. Applying the same convolu-tional filter at the top-left image corner and at the bottom-right one will thereforeyield different features, even though the pose is the same. In practice, this is typi-cally tackled by increasing the width and depth of the network, so that different fil-3PCLCropLabelRectangularCropLabelMatchMismatchFigure 2.1: Perspective effects and correction with PCL crops. The skierlooks as if they turn from front to backward facing, even though theyare recorded with a static camera and are going straight. PerspectiveCrop Layers (PCLs) correct the stretching originating from the projec-tion onto the image plane and matches the 3D pose label to the localview direction of the crop.ters and layers can model the same 3D pose perspective-distorted in different ways.The effectiveness of this procedure, however, strongly depends on the availabilityof large amounts of training data, which is far from being a given for pose esti-mation in the wild. Notably, two-stage approaches that lift 3D pose from 2D poseestimates using multilayer perceptrons (MLPs) [7, 15, 46, 49, 51, 53, 60, 62] rely4also on translational invariance by centering the 2D pose on a root joint, therebylosing important cues on perspective distortion too.In this chapter, we introduce Perspective Crop Layers (PCLs) to explicitly ac-count for perspective distortion within CNNs and other neural networks. Specifi-cally, we use a homography to map the input image to a virtual camera with pre-defined intrinsic parameters that point to the region of interest (ROI). The homog-raphy parameters are functions of the ROI’s location and scale. Hence, this yieldsa synthetic view in which the location-dependent perspective deformations are un-done. The 3D pose inferred from this synthetic view can then be projected into theoriginal image. This requires a priori knowledge about the intrinsic camera pa-rameters, which is rarely a problem in real-world situations because they either arereadily available from the camera specifications or can be inferred from the inputimages alone [25, 83]. We will further show that our PCLs are robust to calibrationinaccuracies. Ultimately, all the operations performed by our PCLs are differen-tiable, and thus amenable to end-to-end learning, while removing the need for theCNN to learn the already-known perspective geometry. Our contributions can besummarized as follows:• We showcase the influence of perspective effects on 3D pose estimates thatincrease for poses away from the image center, which is disregarded by vir-tually all state-of-the-art algorithms;• We derive the equations to compensate for these effects across the image ina location-dependent manner;• We encapsulate our formalism into generic neural network (NN) layers,dubbed PCLs, that naturally integrate into existing deep learning frame-works.We demonstrate the benefits of our PCLs for 3D human pose estimation of bothrigid objects and articulated people. PCLs yield a consistent boost in performance,of 2−10% on average and up to 25% at the image boundary where perspective ef-fects are strongest. Notably, the improvements attributable to our PCLs are consis-tent across the baseline we seek to improve, which validates our claim that even themost-advance deep networks do not learn these perspective effects on the existing5datasets. This includes a PCL variant that undoes the perspective effect on 2D key-points, thus allowing us to showcase the benefits of our approach on state-of-the-art3D pose estimation methods that lift 2D keypoint detections to 3D poses [46, 61].Our code is publicly available at github.com/yu-frank/PerspectiveCropLayers.2.2 Related WorkIn this section, we discuss existing ways of handling image distortions and reviewthe existing attention window mechanisms upon which PCLs are built.Handling perspective effects. Many works sidestep perspective effects by train-ing and testing on synthetic renderings [5, 14, 87, 88] or real images [24, 88] wherethe object of interest is centered manually. However, these methods are not appli-cable to natural images where the object can be at an arbitrary location. If theobject location is known in advance, perspective distortion can be undone in a pre-processing stage. For instance, [47] propose to rotate locally inferred 3D posesback to the camera frame. This strategy has later been adopted by [31], but nei-ther of these works undistorts the input images or input 2D pose. [65, 66] applyan image correction, however, only approximating the homography with an affinetransformation. In other words, the above-mentioned approaches neither modelthe perspective correction geometrically accurately nor formulate it as a differen-tiable layer. However, differentiability is an important prerequisite for end-to-endtraining on natural images, particularly for unsupervised approaches, that deal withunknown object locations.Related are classical methods that undo the Keystone effect, wherein distortionis caused when an image is projected to a non-perpendicular wall [69]. The diffi-culty lies in deriving the deformation parameters, which we derive analytically forour setting while keystoning requires fitting.Radial undistortion. Fisheye cameras and others with a large field of view yieldlarge deformations when mapped to a rectangular pixel grid. They are better rep-resented with spherical images, thereby avoiding location-dependent deformationentirely. This, however, gives rise to challenges when one wants to process the6resulting non-rectangular pixel grids with convolutions. [9] compute convolutionson spherical harmonics, but such frequency-domain networks do not yet reach theaccuracy of regular CNNs. A common workaround is to unfold the spherical im-ages along the azimuth and longitude dimensions, which leads to lesser artifactsthan perspective projection to a planar image. Nevertheless, extreme stretching atthe sphere poles remains. This deformation has been handled by learning filtersthat have the same response as processing local planar patches [68] or by usingDeformable Convolutional Networks [11]. However, any convolution is locationinvariant and misses the geometric position that caused the deformation. To coun-teract this, [42] propose to add the pixel coordinates as an input feature. In ourpreliminary experiments, however, we observed this to lead to overfitting and de-graded results.Convolution can be defined directly on the sphere, by sampling points reflect-ing the sphere curvature [10, 35]. This leads to high accuracy and position invari-ance, but is computationally expensive because non-regular convolution kernelsare needed at each neural network layer, which hampers parallelization and cacheefficiency. By contrast, we target a single undistortion layer that works in harmonywith regular CNNs, MLPs, and attention mechanisms.Attention windows. Processing ROIs instead of the entire input image leads tocomputationally more efficient and more accurate models. Most prominent andrelated to our approach are Spatial Transformer Networks (STNs) [29] that learninvariance to translation, scale, rotation, and more generic warping by spatiallytransforming the feature maps with an affine or free-form deformation. MultipleSTNs have also been stacked [41] to model more complex transformations. STNsproceed in two steps: First, a grid of sample points is defined in the original image,either by direct regression or by predicting the parameters of a restricted familyof transformations, such as a 3× 3 matrix for affine transformations. Second, thepixel value at each grid point is mapped to the target by bilinear interpolation ofthe neighboring image pixels. This yields differentiability and enables end-to-endtraining as an ordinary layer within deep network architectures. It also applies to3D transformations [87]. In this work, we generate a sampling grid that undoesperspective effects in the ROI and use the STN to maintain differentiability with7respect to the ROI position and scale.2.3 Perspective Crop Layer (PCL)We start our derivation by formulating local processing and existing cropping solu-tions mathematically as an affine transformation between a real and virtual cameraof fixed orientation. Subsequently, we derive the perspective transformation under-lying PCL, which corresponds to a rotation of the virtual camera frame, and finallyintroduce the implementation of PCL via two neural network layers that sandwichthe backbone prediction network.2.3.1 Motivation and Rectangular CropsEach point qˆ of a rescaled rectangular or trapezoidal image patch can be expressedin terms of the original image coordinates q. This affine transformation can bewritten in projective coordinates asqˆ = Cq , with C =sx cx axcy sy ay0 0 1 , (2.1)where a = [ax,ay] defines a 2D translation, s = [sx,sy] are scalings in two differentdirections, and c= [cx,cy] are skew parameters. Therefore, as shown in Figure 2.2,a cropped image can be thought of as being taken by a virtual camera with intrinsicparameters Kvirt = CK, where K is the true 3× 3 matrix of intrinsic parameters.As the translation a is usually chosen so that the patch contains an object of inter-est, the optical center of the virtual camera depends on the target location, whichmeans that objects projected far from the image center are deformed differentlyfrom those near it, as shown in Figure 2.1. To remedy this, our goal is to designa crop operation such that the optical center of the virtual camera is always at thecenter of the patch, which makes perspective distortion independent from imagelocation.The centering of 2D human pose commonly done in the state-of-the-art 2D-to-3D lifting approaches is a form of rectangular cropping, too. A pose is root-centered by multiplication with an affine matrix C, in which a is the pelvis/root8-Figure 2.2: PCL virtual camera.Underlying to PCL is the projection from the original image plane onto a virtualcamera pointing at the crop location. This figure visualizes the various quantitiesneeded to infer the mapping.position, s = 1, and c = 0. The subsequent root-centered processing with an MLPhas the same downsides as cropping in STNs and convolution in CNNs in thatinformation about the image location is removed while being affected by position-dependent perspective effects.2.3.2 Defining a Virtual CameraWe introduce a virtual camera with the same optical center as the real one butwhose optical axis points at the center of the target patch p = [px,py] and whosefocal length is chosen to zoom onto the region of interest with factor s = [sx,sy], asshown in Figure 2.2. Below, we derive the virtual extrinsic parameters, in the form9of rotation matrix Rvirt→real, and virtual intrinsic parameter matrix, Kvirt, such thatthese constraints are fulfilled.Cropping as a change of camera perspective. We aim to find camera parameterssuch that mapping a pixel from the original image to the cropped patch can be doneby multiplying an image coordinate in homogeneous coordinates, (u,v,1)⊤, by thematrixΓ(u,v,s,K) = KvirtR−1virt→realK−1 . (2.2)This transformation undoes the original projection using K−1, rotates the resultingpoint to the virtual camera with R−1virt→real and projects it using Kvirt. As for the typ-ical rectangular cropping defined in Eq. 2.1, mapping from the image to the patchremains a warp and does not depend on the generally unknown scene geometry.By contrast to rectangular cropping, this warp is non-linear.Extrinsic parameters. Let Rvirt→real be the 3× 3 rotation matrix that defines thevirtual camera orientation. It can be written as RyRxRz, where Rx, Ry, and Rz arethe Euler rotation matrices that rotate counter-clockwise around the x, y, and z axesof the original camera coordinate system, as depicted by Figure 2.2. Two degreesof freedom of Rvirt→real, Rx and Ry, are determined by pinning the center of thevirtual camera to the back-projected point p = K−1(u,v,1)⊤ in the real camera.Formally, we computeRvirt→real =1√1+p2x−pxpy√(1+p2x+p2y)(1+p2x)px√1+p2x+p2y0√1+p2x√1+p2x+p2ypy√1+p2x+p2y−px√1+p2x−py√(1+p2x+p2y)(1+p2x)1√1+p2x+p2y . (2.3)The yaw angle around the optical axis is unconstrained. We set it to zero (point-ing upwards) in our experiments. Instead, Rz could be controlled, to normalizesubject orientation to pose human subjects upright in the virtual view.Rotation matrix derivation. The rotation that maps from the virtual to the realcamera, Rvirt→real, stems from the definition of rotation matrices. We use the right-handed rule, i.e., counter-clockwise rotation for positive angles, and a right-handed10coordinate system with the y-axis pointing downwards, the x-axis rightwards, andpositive z pointing in the camera direction. The definition of Rvirt→real = RyRxreads therebyRyRx = cos(φ) 0 sin(φ)0 1 0−sin(φ) 0 cos(φ) 1 0 00 cos(θ) −sin(θ)0 sin(θ) cos(θ)= cos(φ) sin(φ)sin(θ) cos(θ)sin(φ)0 cos(θ) −sin(θ)−sin(φ) sin(θ)cos(φ) cos(φ)cos(θ) , (2.4)where φ and θ are, respectively, the vertical and horizontal rotations depicted inFigure 2.2. Therefore Equation 2.3 follows from the following trigonometric rela-tions,sin(φ) =px√1+p2x, cos(φ) =1√1+p2x,sin(θ) =−py√1+p2x +p2y, cos(θ) =√1+p2x√1+p2x +p2y, (2.5)where p is the point on the original image plane to which the virtual camera isrotated.Intrinsic parameters. LetKvirt =fvirtx 0 tvirtx0 fvirty tvirty0 0 1 (2.6)be the 3×3 matrix of intrinsic parameters of the virtual camera. Putting the opticalcenter in the middle of the patch means that tvirtx = tvirty = 0.5. The virtual focallengths are fvirt = [fvirtx , fvirty ] = hvirts , where hvirt is a function of the original focallength in the horizontal and vertical direction stored in K, and s determines thecrop scale in relation to the full image. Together with p, it defines the area ofinterest and is the input to PCL.11Original imageplaneCentral cropof 10 pixelsPerspectivecropsCameraoptical centerOption AOriginal focal lengthOption B Option CVirtual focal length optionsFigure 2.3: PCL focal length settings. We propose three different variantsfor inferring the focal length of the camera to match the pixel scales inthe original and transformed view. The intersection of the green coneswith the image plane is the fraction of pixels that is cropped by PCL.Only Option C maintains a consistent scaling between center and off-center positions.There is no universal way for choosing hvirt, the virtual camera’s focal length,without scaling to the smaller crop size. We propose the following three alterna-tives and evaluate their influence empirically in Section 2.4:A. Setting hvirt to f, the original focal length.B. Setting hvirt to f∥p∥, so that the virtual image plane intersects with the realone at p.C. Setting hvirtx = fx∥p∥√p2x +1 and hvirty = fy∥p∥2√p2x+1to preserve pixel scales.12Although the nature of such a perspective transformation cannot maintain scalein all parts of the image, the last choice of parameters guarantees that scale betweenthe original and virtual image is preserved along the vertical and horizontal axis,while scaling non-linearly in the diagonal direction. Figure 2.1 visualizes this be-havior, and Figure 2.3 illustrates the crop width after each of the three choices.Virtual focal length selection. The focal length of the virtual camera defines thezoom level of the PCL crop. It controls the crop size in the original images. There-fore, it needs to be set individually for every crop target, depending on its desiredsize and position in the image. In the following, we justify our selection of OptionC by defining virtual cameras for a synthetic cube dataset, shown in Figure 2.4.A. By setting hvirt to f, the camera is only rotated, without any change in zoom.A crop is obtained by scaling with factor s, that means, f = hs . Figure 2.4,second row, shows that this simple choice leads to inconsistent crop sizes.The object appears smaller the further away it is.B. By multiplying the virtual length with ∥p∥, the distance of the crop targetposition on the image plane to the camera center, this distance-related ef-fect is compensated. However, as the third row in Figure 2.4 shows, thismatch is not perfect as it does not account for the foreshortening effect whenprojecting from the original image plane onto the virtual one.C. Our final choice models foreshortening withhvirtx = fx∥p∥√p2x +1 andhvirty = fy∥p∥2√p2x+1. It is derived as follows.Derivation of pixel scale preserving virtual focal length. Let p = (x,y,z)⊤ bethe target crop position on the image plane, a 3D position. By construction, pwill be at the image center. Therefore, projecting the infinitesimal motion offsetp+(δx,δy,0)⊤ and comparing the ratio of the offset in the original and projectionyields the desired scale estimate. Formally, we write(u,v,1)T = P(p+(δx,δy,0)⊤), (2.7)13Original ImageGround TruthOption AOption COption BSelectedMatchingFigure 2.4: Effect of virtual camera focal length. The proposed options forsetting the virtual focal length scale differently with respect to the imageposition. When given the pixel cube width as input (as a factor of theentire image resolution), only option C maintains the desired marginof 10 % between cube and crop boundary. The ground truth is a cubeplaced at the center of the screen rotated by the same angle that thevirtual camera is rotated by. The remaining differences in color stemfrom the position-dependent illumination effects.where P projects points in the original coordinate system to the virtual one, asdefined in Chapter 2. For the sake of simpler equations, we do computations incamera coordinates with the origin at the image center and the focal length f = 1.In this case, P = R−1virt→real. Using the definition of Rvirt→real above, the identityR−1virt→real = R⊤virt→real, and computing the partial derivatives with respect to δx andδy at δx = δy = 0 we obtain14∂ (u,v,1)T∂u∣∣∣δu=0=1√(1+ x2)∥p∥ (2.8)and∂ (u,v,1)T∂v∣∣∣δv=0=√(1+ x2)∥p∥2 . (2.9)These equations compute the horizontal and vertical pixel scale ratio between theoriginal and virtual image at p. To maintain the scale, the focal length must be setto the inverse of this scaling factor, which is Option C for f virt that also incorporatesthe original focal length and the desired crop scale.Note that the equations for hvirtx and hvirty are not equivalent with x and y ex-changed because the x and y axes in the virtual camera do not, in general position,project to perpendicular lines in the original view. Figure 2.1 provides examplesof this perspective effect. In our definition, the up-direction is kept fixed and thehorizontal axis is rotated, therefore, behaving differently in the slant compensation.An equivalent formulation could be derived with the horizontal axis fixed and thevertical axis rotated.Maintaining the original aspect ratio. To enable computations with a differentfocal length in the x and y directions, which models the case of non-square pixels,the above notation was performed individually for the horizontal and vertical di-rections. This, however, can lead to stretched crops if different scales are predictedfor s in horizontal and vertical directions. To maintain the original aspect ratio,we set the focal length to the minimum of the axis-specific lengths. This leads toa crop that is the same or larger than the original one with a mismatching aspectratio, thereby strictly containing the object of interest.2.3.3 Differentiable Network LayerPCLs are designed to facilitate perspective correction within existing deep neuralnetwork architectures. We propose the two forms that are depicted in Figure 2.5.Two layers are involved, the projection to the virtual camera and the transformationof the reconstruction to the original camera.15CNN STN CNNCNN+ STNMLPRoot centering (RC)MLP + RCCNN PCL CNNPCL InverseCNN+ PCLPCL InverseMLPPCLMLP + PCLFigure 2.5: PCL integration into existing architectures. PCLs are appliedin pairs, sandwiching the original neural network backbone with thePCL mapping from original to virtual image space and the PCLinv thatmaps 3D predictions back to the original camera coordinates.PCL for lifting 2D keypoints to 3D with MLPs. For networks taking 2D keypointsas input, such as the locations of the human body parts detected in the image plane,Eq. 2.2 can be applied directly on every 2D coordinate and becomes a simple pre-processing that normalizes the 2D pose for perspective effects. The target centerlocation p can be chosen as the mean of all joints, or a root joint. We use the pelvislocation as crop target for human pose estimation.PCL for CNNs. Applying PCL to CNNs requires a two-stage CNN architecture.First, one or more RoIs (p,s)Ni=1 are predicted from the input image I ∈ RW×H×Fusing a detection network. Subsequently, the input is cropped to focus the attentionof the subsequent reconstruction network. PCL replaces the cropping by imple-menting Eq. 2.2 and Eq. 2.3. The pixels are warped using bilinear interpolation, as163D pose(in original camera) 3D pose (in virtual camera)originalcamera frustumvirtual camera frustum(in virtual camera coordinates)camera -axiscamera -axisvirtual camera frustum(in original camera)Figure 2.6: 3D pose change. Because networks equipped with a PCL layeroperate in a virtual camera, the 3D pose prediction living in these coor-dinates is transformed back from virtual to original camera coordinatesin the PCLinv layer by applying Rvirt→real.in the conventional STNs [29] we introduced in the related work section. Becausethe transformation Γ, the definition of Rvirt→real, and the virtual camera matrix Kvirtrely on simple algebraic operations, the entire process is analytically differentiable.To improve efficiency and numerical stability, we parametrize Rvirt→real in terms oflength measures (Eq. 2.3) instead of angles and computationally-expensive trigono-metric functions in the general definition of Euler angles.PCL Inverse (PCLinv): back-transformation to the real camera. The derived per-spective crop has the regular grid structure of a normal image or feature map. Anyneural image processing steps can be performed on it as is. However, the derivedquantities will live in the coordinates of the virtual camera. As for classical atten-tion windows, if the spatial context is important, the processed crop needs to betranslated back to the original camera coordinates. For 3D quantities, such as 3Dhuman pose, this amounts to applying Rvirt→real on the reconstruction, as shown in17Figure 2.6. This PCLinv layer is the same for MLPs and CNNs.2.3.4 Implementation DetailsWe normalize the 2D input poses and 3D output poses with their mean and stan-dard deviation. On the input side, the mean and standard deviation are computedafter the PCL layer and in the case of root centering (RC) after the crop and scal-ing operation. On the output side, the mean and standard deviation for PCL andthe baselines are computed in the pelvis-centered coordinates. We found it moreeffective to multiply the output by the computed standard deviation and add themean instead of doing the inverse operation on the label. This ensures that thenetwork output has mean zero and unit standard deviation, which fares well withnetwork layer initialization, while the loss operates on the scales of the originaloutput space.2.4 ExperimentsWe evaluate the improvements brought about by PCL on the task of 3D humanpose estimation from either images or 2D keypoints, and show that they hold forneural networks of diverse complexity. The benefits of PCL for the 2D to 3D liftingtask on human 3.6 million dataset (H3.6M) [27] and MPI-INF-3DHP dataset [48]are shown qualitatively in both Figure 2.7.Baselines. We integrate PCL into the three neural network architectures for 3Dpose estimation discussed below and compare the resulting networks with the orig-inal ones.MLP+RC: As a first baseline, we use the four-layer MLP from [45] with rootcentering. To ensure a fair comparison, we scale the 2D input of the baseline bythe crop scales s that are used in PCL.temporal convolutional neural network (T-CNN): Our second baseline con-sists of the temporal convolution 2D to 3D lifting approach of Pavllo et al. [61],which operates on pose sequences. At the time of publishing, it was the mostaccurate method in its class.CNN+STN: ResNet [23] is the most widely used backbone for predicting 3D18H3.6M MPI-INF-3DHPImg + CropsImg + CropsImg + CropsImg + CropsMLP + RCMLP + PCLMLP + RCMLP + PCLMPJPE: 41.9MPJPE: 61.6MPJPE: 40.7MPJPE: 45.9MPJPE: 54.7MPJPE: 32.7MPJPE: 91.2MPJPE: 37.9Figure 2.7: H3.6M and MPI-INF-3DHP qualitative examples. For both,H3.6M (left) and MPI-INF-3DHP (right), PCL improves 3D pose es-timation significantly by predicting the orientation of limbs more pre-cisely. The MLP+PCL output is shown in blue and the baseline w/oPCL is in red. Individual mean per joint position error (MPJPE) scores(in mm) are reported for each, which relates the visual and quantitativeimprovements.pose from images. As a baseline, we use an STN that takes a 265×265 image asinput and outputs a 128×128 patch. We do almost all tests with ground-truth croplocations determined by bounding box annotations in the dataset. This guaranteesthat the differences in performance that we measure during evaluation are entirelydue to the type of cropping and not differences in a detector network. We test aversion with an additional L2 loss on the crop location predicted by a ResNet-18detector network, trained end-to-end with the 3D reconstruction objective. We testResNet backbones of depth 18 and 50.Each baseline is extended with PCLs as follows, and as sketched in Figure 2.5.MLP+PCL: Our method replaces the traditional hip-centering in [45] with thePCL and PCLinv layers.19T-CNN+PCL: We apply PCL to [61] by transforming the sequence of framesto the virtual camera pointing to the image coordinates in the middle of the se-quence. Note that this centering is a significant difference to the original [61],which works with absolute 2D positions as input. For simplicity, we assume theoptical center is at the image center.CNN+PCL: We simply replace the rectangular STN crop with our PCL layer,as detailed in Section 2.3.3.Datasets. H3.6M: We evaluate the effectiveness of PCL on the popular H3.6Mdataset [27] that features eleven subjects performing 14 different actions and pro-vides ground truth (GT) 3D poses and camera calibration. We use the establishedtrain/validation/test split, 17-joint skeleton, and the pre-processing of [56]. We setthe rectangular and PCL crop location to the pelvis 2D joint and compute the cropscale as the width and height of a tightly-fitting bounding box. We also experi-ment with using the GT depth for scale estimation. We compare variants using2D detections from [70, 85] and ground truth as input for 2D to 3D lifting. Whenwe compare to [61], we use their preprocessing and train/validation/test split sinceconsecutive frames are required. For simplicity, we assume that the image size is1000×1000, although the size varies from 1000 to 1002.Max Planck Institute for Informatics 3D human pose (MPI-INF-3DHP):We also evaluate our approach on the MPI-INF-3DHP dataset [48], which, com-pared to H3.6M, contains more extreme poses, outdoor environments, and is shotwith wide field-of-view cameras, leading to stronger perspective effects. The cam-eras are calibrated, and all frames are labeled with 3D pose. We use the color aug-mentation from [56] and the official test set and training subjects 1-8 for training,while withholding the first sequence of subject 4 and the last sequence of subject 8for validation. We set the rectangular and PCL crop location to the pelvis joint for2D to 3D lifting and at the mean of the 2D poses for the image-based variants. Thecrop scale is computed as the width and height of a tightly fitting bounding box.ToyCube: We introduce a synthetic dataset containing images of a renderedcube of edge length 0.5 m. We use this toy example to ablate individual factors ofvariation, such as the effect of illumination and pose distribution.20Distance from the image centerAverage MPJPE errorMPJPE gain over 2D image regionsMLP + RCMLP + PCLMLP + RC < MLP + PCL MLP + RC > MLP + PCL Figure 2.8: Improvement of reconstruction error on H3.6M. binned withrespect to the image position. Left: MLP+RC suffers from perspectiveeffects away from the center, while MLP+PCL effectively compensatesthese leading to improvements of up to 25%. Right: The consistentdifference of MLP+RC and MLP+PCL is also reflected over a 2D tiling,showing the average MPJPE error difference of cells with 10 or moreframes on the validation set.Training setup. The 3D pose is trained on an L2 loss using Adam [36] with alearning rate of 0.001 for the 2D to 3D lifting models and 0.0005 for the imageto 3D networks. The temporal convolution networks are trained using Amsgrad[64] with an initial learning of 0.001 and a learning rate decay factor of 0.95 ap-plied after each epoch. We train 2D to 3D lifting methods for 200 epochs andbatch size 64 and the temporal convolution networks for 80 epochs and batchsize 1024 with up to 243 frames in each batch element. Lastly, image-to-3D net-works using a ResNet-50 backbone are trained for 40 and 150 epochs on H3.6Mand MPI-INF-3DHP respectively. The ResNet-18 backbone trained on H3.6M istrained for 60 epochs.Metrics. To quantitatively evaluate 3D pose accuracy, we use the MPJPE, com-puted as the average Euclidean distance of the predicted 3D joints to the ground-truth ones, where both poses are centered at the pelvis. All MPJPE results arereported in millimeters. We also report the percentage of correct keypoints (PCK),encoding the proportion of joints whose distance to the ground truth is less than athreshold, using thresholds of 50 and 100 millimeters.21H3.6M MPI-INF-3DHPInput Model MPJPE ↓ PCK @ 50mm ↑ PCK @ 100mm ↑ MPJPE ↓ PCK @ 50mm ↑ PCK @ 100mm ↑2D GT + 3D Root GT MLP + RC ([45]) 45.3±0.2 66.8±0.3 91.4±0.2 69.4±0.4 46.4±0.5 77.0±0.32D GT + 3D Root GT MLP + PCL (Ours) 40.1±0.5 72.8±0.5 93.3±0.2 45.6±0.5 70.1±0.5 89.7±0.32D GT MLP + RC ([45]) 48.4±0.4 62.9±0.5 90.3±0.2 74.1±0.5 43.0±0.3 74.7±0.72D GT MLP + PCL (Ours) 43.8±0.1 68.3±0.1 92.2±0.0 50.1±0.2 65.5±0.2 87.8±0.12D Detection MLP + RC ([45]) 69.7±0.2 46.3±0.5 80.5±0.1 - - -2D Detection MLP + PCL (Ours) 67.0±0.1 48.7±0.2 82.1±0.0 - - -Image CNN (ResNet50) + STN 96.5 32.7 64.6 117.7 33.6 60.1Image CNN (ResNet50) + PCL (Ours) 94.1 34.1 65.8 109.5 40.3 66.2Image CNN (ResNet18) + STN 95.9 35.4 65.6 - - -Image CNN (ResNet18) + PCL (Ours) 93.9 37.1 66.6 - - -Table 2.1: Reported MPJPE and PCK for 2D to 3D keypoint lifting tests performed on H3.6M and MPI-INF-3DHP. The reported mean and standard deviation is computed over three runs with varying random seed. ForMPJPE, lower values are better and for PCK, higher values are better. We can see from the table that our methodsignificantly outperforms the baselines that do not use PCL. We bold the best-performing models in each category.222.4.1 PCL for 2D to 3D Keypoint LiftingThe results for the 2D to 3D lifting task on H3.6M and MPI-INF-3DHP are pro-vided in Table 2.1. For H3.6M, MLP+PCL achieves an MPJPE of 67.0 mm vs.69.8 mm MPJPE of the MLP+RC baseline [45] when using 2D detections from[70, 85] as input, a 4% improvement. Even larger improvements are achievedwhen using the GT 2D pose as input and when using the 3D root joint position forscale estimation. Notably, our method with a scale computed from the 2D posestill outperforms the STN baseline using 3D ground truth for scale prediction.We obtain even larger improvements with PCL on the MPI-INF-3DHP dataset,with 2.4 cm in MPJPE and 22 PCK points. This dramatic improvement is no sur-prise since the larger field of view (smaller focal length f ) of the MPI-INF-3DHPcameras leads to stronger perspective effects and, therefore, a larger difference be-tween the corrected and uncorrected views. These experiments also show that PCLis not specific to any particular focal length.In Figure 2.8, we analyze the position-dependent effect of our method onH3.6M. As shown by the plot, the baseline MPJPE increases with the distancefrom the subject to the image center, hinting at the negative effect of perspectivedistortion. PCLs undo this effect, leading to a more stable MPJPE and outper-forming the baseline by a growing margin as the distance increases. PCL evendecreases with the distance to the center, which is surprising. We believe this is be-cause the most complex poses in H3.6M, such as sitting and lying on the ground,are performed in the image center while walking dominates for the off-center ones.We also perform the same analysis on the MPI-INF-3DHP dataset. Figure 2.9depicts significant improvements from using PCL. The error by PCL is stable oreven improving with the distance to the center, while the MLP+RC model degradesdue to perspective effects. PCL even gains an improvement at the image center.This is unexpected at first glance but can be explained with the MPI-INF-3DHPdataset using different cameras for training and testing. The MLP+RC model seem-ingly overfits to the perspectives seen during training while the automatic correc-tion of PCL leads to better generalization irrespective of the image location.230.0 0.2 0.4 0.6 0.8Distance From the Center [Relative]020406080MPJPE [mm]STNPCL1.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.001.000.750.500.250.000.250.500.751.00402002040Figure 2.9: Improvement of reconstruction error on MPI-INF-3DHPbinned with respect to the image position. Left: MLP+RC suffers fromperspective effects away from the center, while MLP+PCL effectivelycompensates these leading to improvements of approximately 50% .Right: The consistent difference of MLP+RC and MLP+PCL is alsoreflected over a 2D tiling, showing the average MPJPE error differenceof cells with 10 or more frames on the validation set.2.4.2 Diversity of Crop Positions in H3.6M and MPI-INF-3DHPIn Figure 2.10, we can see the distribution of hip joints in the testing dataset forH3.6M and MPI-INF-3DHP. While H3.6M has most of the images around thecenter of the frame, MPI-INF-3DHP contains poses that are widely spread outacross the image. This along with the fact that MPI-INF-3DHP uses a wider field-of-view camera explains the significant improvement we see from introducing PCLon MPI-INF-3DHP.2.4.3 PCL for Temporal Convolutional Neural NetworksAs shown in Table 2.2 incorporating PCL into the temporal convolutional networkof [61] does not improve results on their original implementation. Our followinganalysis shows that this is due to [61] already learning position-dependent effectsby operating on unnormalized 2D pose. This, however, overfits to the camera usedat training time. By contrast, PCL generalizes perfectly when the camera changesat test time so long as its properties are known approximately.241.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.001.000.750.500.250.000.250.500.751.0001002003004005006007008001.00 0.75 0.50 0.25 0.00 0.25 0.50 0.75 1.001.000.750.500.250.000.250.500.751.000255075100125150175Figure 2.10: 2D histogram of hip joint locations for H3.6M and MPI-INF-3DHP in normalized image coordinates over the test dataset forH3.6M (left) and MPI-INF-3DHP (right). The MPI-INF-3DHP setshows a wider and less regular distribution. Brighter cells indicate ar-eas in the images where subjects appear more often.Model Original f New fT-CNN 47.3±0.0 72.7±0.5T-CNN + RC 51.5±0.1 51.5±0.1T-CNN + PCL (known f ) 48.8±0.3 48.9±0.2Table 2.2: Temporal CNN tests , computed as the MPJPE over two runs withvarying seed on H3.6M. While the baseline performs the best using theoriginal camera, it cannot generalize to new camera settings. The PCL-equipped version strikes the best compromise.To analyze the effect, we artificially change the focal length of the test se-quences by multiplying all 2D testing poses by a factor of 2/3. This simulates acamera with a smaller focal length and larger field of view. T-CNN generalizespoorly, as it seemingly overfits to the global position and scale of the training set,while PCL can adapt perfectly to the new capture setup without loss in perfor-mance.To facilitate a fair comparison, we created a second baseline, T-CNN+RC. Itcenters and scales the input pose sequence by subtracting the root joint of the cen-tral frame and scaling by its horizontal and vertical size; this is the same procedure25that is used for the other RC baselines. This maintains the root motion while re-moving the absolute scale and position in the image. This variant generalizes muchbetter than T-CNN does but has an overall higher error than T-CNN+PCL. In sum-mary, PCL strikes the best compromise between accuracy on the original domainwhile being applicable to new camera geometries and capture setups.2.4.4 PCL for Convolutional Neural Network ArchitecturesAs shown in the last four rows of Table 2.1, when using images as input to a ResNetregressing 3D pose on H3.6M, the baselines achieves an MPJPE of 96.5 mm, whileour model with PCL yields 94.1 mm, a 2.5% reduction. The improvement is lowercompared to 2D to 3D lifting, likely because the overall higher error for image-to-3D pose prediction compared to 2D to 3D lifting is dominated by other errorfactors.On the MPI-INF-3DHP dataset, PCLs’ improvement is more pronounced, im-proving by 8mm and 6 PCK points, which further validates the previous findingsthat perspective effects are stronger on MPI-INF-3DHP, therefore leading to a clearimprovement despite higher total errors.2.4.5 Ablation StudiesRobustness to Errors in Focal LengthAlthough PCL requires knowledge of the camera’s focal length, it is often knownor can be estimated approximately from image features [25, 83]. To evaluate therobustness of PCL to erroneous estimates, we experiment with an artificial distur-bance to the true focal length at test time. The 2D input poses to the MLP+PCLnetwork are deformed as if they would stem from a camera with a different zoom.As shown in Figure 2.11, PCL is relatively robust when the estimated focal lengthranges between 0.7 and 1.5 times the true one. One interesting point is that rectan-gular cropping is equivalent to an infinite virtual camera focal length; however, it’sclear that the performance lines of MLP+PCL and MLP+RC intersect far beforethat. We hypothesize this is due to the MLP+RC method implicitly modeling someperspective effects.26MLP + RCMLP + PCL Multiplicative Factor of GT Focal LengthMPJPE [mm]Figure 2.11: Evaluating the effectiveness of PCL on approximated cam-era focal lengths We can see from the figure that for reasonable ap-proximations of the focal length, PCL still outperforms rectangularcropping.Model Efficiency from PCLTo demonstrate the effectiveness of using PCL (taking perspective distortions intoaccount), we reduce the dimension of the hidden state of our 2D-3D keypoint liftingmodel and report the performance on the validation set for H3.6M. From Table 2.3we can see that even with about half the number of parameters as the baseline MLPwe are able to capture a more precise reconstruction.Axis-based Rotation ExperimentTo study the effects of using PCL as a post-process on existing trained models, wetrain the baseline 2D-3D keypoint lifting MLP and apply the PCL-defined rotationmatrix on the predicted 3D pose. To this end, we experiment on MPI-INF-3DHPsince that dataset contains more perspective distortions and will have more pro-nounced effects from adding the PCL-defined rotation. Along with this we also27Model Linear Size # Parameters MPJPE (mm)MLP + RC 1024 4,296,755 48.4MLP + PCL 1024 4,296,755 43.8MLP + PCL 896 3,300,915 45.4MLP + PCL 768 2,436,147 46.5MLP + PCL 512 1,099,827 50.8Table 2.3: Results with 2D-3D keypoint lifting on H3.6M while decreas-ing the model complexity of our PCL embedded model (MLP + PCL)while maintaining that of the baseline (MLP + RC). We can see here thatdespite having roughly 50% of the parameters of the baseline, the PCL-equipped model is still able to outperform the baseline.MPI INF 3DHP2D GT + 3D Root GT 2D GTSTN 69.4 74.1STN + Half Rotation (x) 67.6 73.1STN + Full Rotation (x, y) 66.3 70.0PCL 45.6 50.1Table 2.4: Results with 2D-3D keypoint lifting on MPI-INF-3DHP whenapplying half and full pose rotations. We can see that even after in-cluding information from PCL into the baseline model, it still falls shortin terms compared to the model that is trained with PCL.perform a ”half-rotation” and ”full-rotation” defined as rotation along only the x-axis and rotation about the x and y-axis respectively. Furthermore, we show re-sults for when the 3D root is given and when the scale is estimated from the 2Dpose. From Table 2.4, we can see that when we compensate the baseline modelwith PCL-defined rotations it improves but still falls short of our model trainedend-to-end with PCL, showing the importance of incorporating PCL during modeltraining.Cube DatasetThe Cube Dataset contains images of a single coloured cube with edge length 0.5mat random locations and orientations within the frame. Figure 2.12 shows an exam-ple of the cube used in this dataset as well as demonstrating the perspective effectsthat occur when objects move away from the image center. To further demonstrate28Original Image PCL Rect.Figure 2.12: Toy-Cube examples. While cubes projected to the sides of animage appear distorted and look stretched when cropped, PCL undoesthese perspective effects.the effect that ignoring perspective effects have on 3D pose estimation we intro-duce a variation of this dataset in which the cube has a random orientation but isalways placed at the center of the frame. We refer to this variation as the CenteredCube Dataset. We now train models on the Centered Cube Dataset with a centralcrop and analyze their generalization capabilities to crops from the general datasetwith the position of the cube given. On this dataset, the baseline attains an MPJPEof 13mm and our PCL variant improves to 8.2mm. The improvement for end-to-end training is equivalent, with a 4.4mm improvement from 11.2 to 6.8mm. Whilethis test is simplistic, the synthetic nature allows us to analyze the generalizationcapabilities of PCL by training on a version of the cube dataset where the cube isalways centered in the training images and tested on the original version with gen-eral position, with the ground truth 2D crop location provided. PCL shows goodgeneralization, outperforming the baseline by 36.1% on the unseen test set. Ta-ble 2.5 compares the performance of PCL and STN models on these two datasets.We also investigated the effect of illumination by switching from point lights toambient illumination, which had a negligible effect on the reconstruction quality.29Matching train-test set Unseen test setModel Detector centered generalCNN + STN GT Loc. + GT Scale 6.9 13.0 13.3CNN + PCL (Ours) GT Loc. + GT Scale 6.9 8.2 8.5CNN + STN Trained End-to-End 5.9 11.2 -CNN + PCL (Ours) Trained End-to-End 5.9 6.8 -Table 2.5: MPJPE in millimeters for all tests conducted on the CubeDataset and its variations. For this metric, lower values are better. Wecan see that our method produces more accurate results while at the sametime generalizing better to unseen instances.2.5 LimitationsThe gained improvements come at the cost of requiring an estimate of the focallength f . Yet, the robustness towards errors in f shows that improvements canstill be obtained with a rough guess. It is worth noting that PCL compensates forlocation-dependent perspective effects. Those effects that originate from varyingdistances of the object to the camera, e.g., selfie vs. third person picture, can notbe resolved with image warps but would require knowledge of the 3D geometry.Data-driven approaches have been proposed to compensate for these [63, 93].30Chapter 3Latency-Reduced NeuralRendering for Telepresence UsingImplicit WarpingThe following text is taken from: Yu, Frank, et al. ”Scaling Neural FaceSynthesis to High FPS and Low Latency by Neural Caching.” Proceedingsof the IEEE/CVF Winter Conference on Applications of Computer Vision.2023.3.1 IntroductionTelepresence via photo-realistic 3D avatars promises to better connect people. Re-cent advances in neural rendering already enable near photo-realistic image quality,but the underlying deep neural networks limit the best possible latency with theirsequential, layer-wise processing. This is a problem for virtual reality applicationsas these require low latency upon head motion of the user to convey an immersiveexperience. For example, [97] creates a high-fidelity system using 120Hz pro-jectors and user viewpoint tracking with a tracker having 60Hz updates and 3mslatency to minimize users’ perception of warping of the scene when they move.However, none of the existing neural renderers reaches the required motion to pho-tons latency, i.e., the time it takes from the user input, such as moving the head in31UV MapCV ReconstructionMeshNeural Texture (NT)LearningInferenceWarpFigure 3.1: Neural rendering with warping. While a single frame methodrequires a deep neural network to synthesize a realistic head image fromrough geometry, our implicit warping yields low latency for a new frameat t+1 with a shallow network.virtual reality (VR), until the display updates. It is henceforth an open problem tofind improved neural models that strike a better trade-off between speed and imagequality.We develop a parallel implementation, inspired by previous approaches forimproving the frame rate for other video processing tasks [4] by spreading thecomputation over several frames over multiple graphics processing units (GPUs).By itself, parallel execution only improves frame rate, not the latency that is socrucial for VR, since the effective network depth of sequentially executed layersremains the same. To nevertheless reach the desired latency reduction and framerate, we combine parallel execution with a dedicated warp layer that is tailored forneural face synthesis and acts as a skip connection between consecutive frames.This neural caching approach re-uses information from the previous time stepto improve latency from the current one with a shallow network for image genera-tion and a deep network for computing the cache while waiting for the next inputframe. Note that this caching strategy by itself already improves latency on a sin-gle GPU and scales in frame rate on increasingly parallel hardware by breakingthe sequential layer dependencies and offloading the image cache generation intomultiple threads.32The difficulty lies in finding the right representation for the cache to succeedwith a low-capacity network. Our approach is inspired by classical low-latencyrendering on VR displays [79], which compute only the position of objects for in-between frames and warp color from the previous. However, explicit warping doesnot apply well to neural renderers where the underlying geometry is approximateand the neural texture is high-dimensional making warping operations much morecostly.To warp the neural representation, we introduce an implicit warp that providesa skip connection that is tailored for neural rendering by taking into account thehead model parameterization. The result is a warp network that models the image-space motion from one frame to the next given the desired viewpoint and headmodel parameters. Figure 3.1 outlines the main components.Our design is geared towards novel-view-synthesis of a talking head given adynamically changing viewpoint, such as the user’s head motion in VR. We buildupon the deferred neural renderer (DNR) [77] that uses a neural texture learned attraining time. Our contributions towards scaling frame rate and latency on parallelhardware are:• Demonstrating that the proposed neural caching can reduce latency by up to70% with minimal degradation in image quality (only 1% PSNR).• Extending a DNR to generate a high-resolution output with low latency viacaching and implicit warping.• Developing a parallel scheduler that supports warping and synchronizes mul-tiple threads using queues.• Refining the representation of facial expression, head pose, and camera angleto improve the implicit warp.• Adding head stabilization and tweaking the backbone and training strategyfor noisy real-world conditions.Our experiments highlight the importance of how to cache as well as what andhow information from the current frame is passed to the shallow implicit warpingnetwork. Figure 3.4 compares the most related methods. Ours strikes the bestlatency and frames per second (FPS) improvement with the least image qualitytrade-off.333.2 Related WorkIn this section we discuss recent high-quality image generation methods and con-trast with those that optimize runtime with a focus on face generation.Photo-realistic synthesis. High-quality photorealistic rendering is booming, us-ing either implicit scene representations such as neural radiance fields (NERFs)[52] or deep neural networks trained on generative adversarial network (GAN) ob-jectives [32, 33], which can also be conditioned on viewpoint and pose changes [12,74]. However, these implicit models all rely on very deep neural networks that donot run at high-enough frame rates or are limited to static scenes via pre-computedacceleration structures [89]. Recently, acceleration methods have been proposed torender NeRFs in real-time [6, 18, 54], albeit for static scenes, due to their efficientray sampling and small neural network architectures. However, extending theseworks to dynamic scenes is an open problem.Talking head models. For dynamic faces, it is most common to start from aparametric face model that parameterizes identity and expression in the form ofblend shapes that are linearly combined with the full mesh [1, 40]. The coef-ficients of these models are low-dimensional and therefore suitable for drivingavatars in telepresence and for reenactment by mixing identity and pose informa-tion of two subjects. While earlier models focus on only modeling the facial re-gion [72, 75–77, 81] conditioning on a complete head model, such as FLAME [40],enables novel view synthesis of side views [21] suitable for view-dependent 3Dsystems[96]. We follow this line of work and extend it by reducing the render-ing latency for interactive applications. Other works [22] aim to learn person-specific head meshes from multiple images given a coarse estimate of geometry.This method is related to ours but largely orthogonal as ours aims solely to reducelatency. However, our method should be applicable to reduce the latency of theirinference pipeline.Another line of work developed subject-agnostic face synthesis [19, 20, 55, 57]that is conditioned on single or multiple images instead of a subject-specific 3Dmodel. However, this comes at the price of reduced details and entanglement of34expressions and head motion [82], particularly if the reference and target posesdiffer largely, and may require explicit warping operations that are relatively costlyand ill-defined for occluded regions [92]. Although [92] is already extremely faston medium image resolution with 4ms runtime, there is still room for improve-ment when rendering high-resolution images and medium resolution on embeddeddevices.Caching approaches. Re-using information from the previous time step has beenused for many computer vision tasks, among them: object detection [44], video ac-tion recognition [16], and segmentation [26]. Carreira et al. [4] gives an excellentoverview of different architectures for video processing, including: i) depth paral-lel architectures that execute a deep neural network over several iterations, leadingto a delay equal to the depth of the layers; ii) depth parallel + skip in which thehead (last couple of network layers) of a depth parallel network is updated with thenew input via a skip connection; and, iii) multi-rate clock architectures in whichthe input features for the head are not updated at every time step and the head andbackbone operate at different clock rates. Our caching approach follows the multi-rate clock pattern. However, none of the existing parallel models has demonstratedimage generation. Our contribution is to tailor this general concept to the problemof novel-view synthesis of faces via neural rendering by a suitable form of warping.3.3 PreliminariesOur goal is an efficient neural renderer that outputs an image I ∈ RH×W of a faceparameterized as a surface mesh with vertices v ∈R3×K , triangle indices i ∈R3×K ,and associated texture coordinates u∈R2×K and neural texture N∈RD×H×W [77].These are the same inputs that a normal renderer would expect, except that thetexture N is D-dimensional instead of storing three color values.Deferred neural rendering. Our starting point is the deferred neural renderer in-troduced by Thies et al. [77], which approximates the complex and computationallyexpensive rendering equation with a convolutional neural network G. Figure 3.1gives an overview including the differences of [77] applied to our full model includ-35ing caching. Initially, a rasterizer renders UV maps U ∈ R2×H×W of the texturedmesh. These are of the same dimension as the output image and store for everypixel the corresponding texture coordinate. Sampling these locations from N givesthe feature map F ∈ RD×H×W . For classical deferred rendering, we would samplefrom a color texture and combine it with light position information to form the finalimage. In the case of the neural renderer, the texture has more than three channelsforming learnable features. The network G turns F into the final image, replacinggeometric illumination computations and material shaders in classical rendering.This forward pass is traced with blue arrows in Figure 3.1 while the backwardsinformation flow during training is marked in green.Training and face reconstruction. The parameters of the involved neural networkG as well as the neural texture itself are trained on a large dataset that has examplesof the input 3D mesh and a high-resolution image of the face—the desired output.We use real videos of the person as input and reconstruct vertices v∈R3×K , expres-sion coefficients e ∈ R50, principal component analysis (PCA) shape parameterss ∈ R100, and head pose θ ∈ R6 of the FLAME parametric model [40] alongsidecamera position p ∈ R3 using the off-the-shelf estimator DECA [17]. Internally, itis using the 2D keypoint detector from [3]. The reconstructed face overlays wellwith the image when re-projected, but details, such as hair and ears are often mis-aligned, which puts a larger burden on the neural renderer to synthesize these. Thisreconstruction step is marked with a red arrow in Figure 3.1.The generator G is a U-Net and the neural texture N is initialized at randomand subsequently optimized by backpropagation to store details of the training ob-ject locally. The loss function is the L1 difference between the rendered and thereference image in the dataset. This backward pass is traced with green arrows inFigure 3.1. Training is on cropped images, which speeds up training.Base architecture. We use the 10-layer U-Net as in [28] and a multi-scale neuraltexture with four levels of detail as in [77]. Furthermore, to better model viewpoint-dependent effects, the view direction is projected to 9 spherical harmonics coeffi-cients which are subsequently multiplied to channels 4 through 12 of the feature36map. This enables explicit encoding of view-dependent effects similar to positionalencoding [80].Real-time viewpoint-dependent rendering. Our primary application field is 3Dteleconferencing, where a person must be rendered at a high frame rate, with lowlatency, from the viewpoint of the user that is roughly frontal. Given a new viewdirection, e.g., from an eye-tracker, our focus is on generating a natural-lookingimage of this novel view as quickly as possible to mitigate motion sickness, reducewarping artifacts [95], and avoid discomfort. The object motion capture couldbe performed offline or through a slower channel, as usually only the viewpoint-dependent rendering demands low latency. In telepresence applications, bandwidthis dictated by the size of the FLAME model; estimated on the source side, trans-ferred, and rendered by our system given a new view from the receiver.3.4 MethodIn this section, we introduce our neural caching approach and propose two vari-ants that operate on single and multi-GPU systems and are respectively tuned forlatency and frame rate. We cache information from the immediately precedingframes. Thereby, the motion that must be bridged from the cached informationto the current frame is small, which allows us to introduce an implicit warp thatattains maximum performance.Neural cache. For our neural caching, we first run the deep and slow image gen-erator G(Nt ,Ut ,pt) that is conditioned on the neural texture N, UV-map Ut andview direction (e.g., user’s head motion in VR) of the current frame t. Figure 3.2provides a detailed overview of our pipeline. We cache features C(3)t , C(4)t , andC(5)t from the last 3 layers of the generator together with camera position pt , andspherical harmonics (SH) encoding of the pose, hobjt . Furthermore, we add the UVmap Ut , expression et , and the pose θt that are specific to rendering faces. Formallywe write the combined cache C asCt := [C(3)t ,C(4)t ,C(5)t ,θt ,pt ,et ,hobjt ,Ut ]. (3.1)37LPF = Low Pass FilterFigure 3.2: Our low latency neural face-rendering pipeline caches the pre-vious frame (features from the generator (left) marked orange and red)and the pose of the person to generate the subsequent frames with onlytwo additional up-convolution layers (right) as soon as a new view di-rection is available. The generator and warping networks are learnedend-to-end and can be applied in parallel at inference time.Implicit warping. Previous work used an explicit warp operation from a referenceframe, which requires preceding neural network layers to predict the warp and isimplemented as a texture sampling step that is relatively slow due to random accesspatterns for every pixel. We propose an implicit warp with a neural network W thatis composed of only two up-convolution layers. These two layers take in cached38information Ct , rendered UV map and the new camera, pose, and expression fromthe new frame t + 1, to reconstruct the image It+1. This network is intentionallykept shallow to decrease the latency of image generation.In addition, we found in a detailed ablation study that giving as input the newcamera position pt+1, object pose θt+1, expression et+1 and their differences to theprevious frame, via a single-layer MLP, M, works best. This yieldsθ¯t+1 = M(pt+1,∆pt+1,θt+1,∆θt+1,et+1∆,et+1,hobjt+1), (3.2)where the ∆ refers to the change in a quantity between two frames, here θt+1−θt .Additionally, we also include the UV map Ut+1. Together with the cached featuresprocessed by the first warping layer W1 givesF1 =W1(C(3)t ,C(4)t ∆hobjt+1, θ¯t+1,∆Ut+1) (3.3)where ∆ht+1 and θ¯t+1 are broadcasted to the resolution of the cache and ∆ht+1modulates the feature map by multiplication, similar to how positional encodingworks, but here for rotational changes. These features are further processed tooutput the image with It+1 =W2(F1,C(5)t ∆hobjt+1). Note that W warps the previous tothe current frame, but without a spatial transformation layer—it does so implicitlyvia a local approximation of how the image changes with respect to changes inpose.3.4.1 Operation Modes and Parallel ExecutionOur approach scales easily between single and parallel GPU execution. Whenonly a single GPU is available, Figure 3.3-top visualizes the principle of reducinglatency with a shallow warp network, compared to running the slower generator.Here, the cache is updated in sequence with the warp, thereby reducing latency butnot frame rate.When multiple GPUs are available, we can combine the proposed implicit warpwith parallel execution, thereby rendering at a two or more times higher framerate by warping a single or multiple images It ,It+1, · · · while the generator runson a separate GPU on a separate thread. Figure 3.3-right showcases this scheme.39V W V WGGSequential (1x warp)Parallel (2x warp)VWGV WThread 1Thread 2VGW V WVV GV WFigure 3.3: Our rendering pipelines for sequential (top) and parallel exe-cution (bottom). A network W warps a cache from the previous frameas soon as a new viewpoint V is ready for rendering, producing the out-put image much quicker than the full generator G could. Leveragingtwo threads further increases the output frame rate.Notably, even though the warping is an approximation, multiple warping does notreduce image quality when operating online on high-frame-rate video streams. Thehigher processing speed of warping twice in parallel reduces the distance betweenframes that can be processed and thereby makes the two-frame warp as difficult asa single-frame warp operating at half the frame rate.We experimented with different job assignments and synchronization schemesbetween the two threads. We found that threads alternating between image gen-eration and warping are the most efficient and easy to implement. In this model,the main thread distributes newly arriving viewpoint information to the two workerthreads via a queue, each associated with one GPU. These in turn wait for new datain the queue. Upon receiving new data, they alternate between caching and warp-40ing as visualized in Figure 3.3-bottom. Thereby, the warping is executed on thesame GPU as the generator, such that the cache can remain on the same GPU. Thealternative of using a dedicated warp and cache thread has a much lower perfor-mance since the cache would have to be moved from one GPU to central processingunit (CPU) and then again from CPU to the target GPU. The required synchroniza-tion of ques has a negligible overhead in our implementation on two RTX 2080swith only 0.25ms/frame. Moreover, our preliminary attempt of using manual locksinstead of queues was more complex without improving performance. We willmake our implementation publicly available to facilitate further research.3.4.2 Improved Neural Head RenderingOur starting point, DNR [77], is a general rendering approach. In the following,we explain our architectural changes toward tuning it for face synthesis.Predicting foreground masks. Because the background geometry is hard to recon-struct, e.g., requires offline structure from motion [77], we instead impose a staticbackground assumption and only parameterize the head geometry. This bringsabout the problem of segmenting the foreground. The FLAME head model couldbe used to render a foreground mask, however, its parametric form does not modelhair and other facial details. Instead, we add an additional output channel to DNR,such that it has three channels for color and an additional one to predict the fore-ground mask of that frame. At training time, the background image is estimatedas the mode of the pixel value over the input image or via inpainting if the headis mostly static during the performance. We finally linearly combine the recon-structed foreground with the background.Head stabilization. To reduce jitter and flicker of the head, we ensure that the vir-tual camera that we use to generate our UV map is always centered on the subject’shead and has consistent scale when generating videos by centering the camera onthe head midpoint and scaling by the projected ear-to-ear distance. Because thedriving motion capture signal is often unstable, we further smooth the global headposition pt with a delayed Gaussian filter of size five and fix the identity s of the41FLAME model to be the mean identity estimated by DECA on the training set. Tomaintain facial expressions and lip motion faithfully, the jaw orientation in θ andexpression parameters e are left untouched.Loss function. For our baselines which only predict the image at time t we definethe loss function asLtrain = λtexLtex+λimgLimg+λpLp, (3.4)which measures the L1 distance between the first three channels of the sampledneural texture and the ground truth image (Ltex), the L1 photometric loss (Limg),and the perceptual loss [30] between the predicted image and ground truth image(Lp). We weigh these loss terms with coefficients 1, 1, and 0.1. For the warping,we add λimg and Lp on the future frame t+1 and down-weight the existing Limgby 0.1 to put much more weight on the prediction for It+1 that is used at inferencetime.Architectural changes. Based on the work of [34] we make the following im-provements to the base DNR [77] network to improve output image quality. First,we replace the transpose convolutions in the latter half of the U-Net with a bilin-ear upsample layer followed by a 2D convolution (up-convolution). This has beenshown to increase the final image quality of output and reduce grid-like noise inreconstructed images. Furthermore, we apply a Gaussian low pass filter (LPF) onthe smallest spacial features of the U-Net architecture. We refer to the baselinesthat utilize all these improvements as DNR+.3.5 Implementation DetailsMeshes are generated using the forward function of the FLAME model imple-mented in Python [40]. UV maps are generated by rendering the FLAME headmodel using the PyTorch3D rasterizer, which implements rasterization on the GPUusing custom Compute Unified Device Architecture (CUDA) kernels but it is not asfast as classical rasterization. The explicit warp is implemented using the PyTorch42grid sample function, which is the same as used by [8] for their explicit warp.Generating novel views. Our method relies on both the processed UV maps, ini-tialized with the 3D mesh estimated by DECA [17]. Therefore, to generate novelviews for our method we require to modify these parameters. In our tests, we foundthat the best generalization to novel views is obtained by rotating the vertices out-putted by the FLAME model, as modifying the joint angles in FLAME model toextreme poses may lead to parameter ranges not seen during training.Multithreading library. Our multi-GPU operation modes are implemented usingthe PyTorch multithreading library and two threads. For synchronization and com-munication between threads, we use the torch.multiprocessing.mp.Queue queues:one input queue for each thread (two for two GPUs), one image output queue, andone queue to signal the completion of caching. Listing 1 provides pseudocode forhow caching and warping threads are orchestrated from the main thread, makingsure that the output images arrive in sequence and that the caching completes be-fore warping on the same worker while also ensuring that the parallel workers areindependent otherwise for maximal performance. For simplicity, the pseudocodeis simplified for 2x GPU and 2x warping for every caching operation. A differentnumber of GPUs and warp operations only require changing the condition in lines4 and 8 that distribute caching and warping operations over worker threads. Wepublish our multithreading code so others can build upon it with their own neuralmodels.3.6 ExperimentsWe evaluate our Neural Warping technique with respect to our goal of maximizingimage quality and minimizing the latency by applying our Neural Warping. Fig-ure 3.4 summarizes our main results on the possible trade-offs between accuracyvs. latency and accuracy vs. FPS, comparing our two variants to the most relatedwork and showing an improvement in FPS of up to 300% and a reduction in la-tency of 70%. We also provide an ablation study to identify specific trade-offswith respect to individual components of our warping network.43Algorithm 1: Parallel Implementation (2x GPUs, 2x warp)/* Main thread */1 Queue viewpoint queue[NGPU=2], cache queue, out queue// Queues used for synchronization2 cache tid ← 0 // Pointer to assign worker responsiblefor caching3 for t=0; t++; doInput: viewpoint, expr4 if t % NUM WARPS == 0 then5 cache queue.get() // Wait for caching to complete6 cache tid ← (cache tid+1) % NGPU // Switch role ofthreads// Post viewpoint to worker and assigncaching role7 (viewpoint, expr, warp=False) → viewpoint queue[cache tid]// Post viewpoint to the other worker andassign warping role8 (viewpoint, expr, warp=True) → viewpoint queue[(cache tid+1) %NGPU]// Display output image once availableOutput: out queue.get()/* Worker thread, one for each GPU */9 do in parallel10 NeuralCache cache // variable local to each thread11 (viewpoint, expression, warp) ← viewpoint queue.get() // Waitfor new input to process12 if warp = True then13 image ← runWarpNet(viewpoint, expression, cache)// Generate the image14 image → out queue // Return output image viaqueue15 else16 mesh ← runFaceModel(viewpoint, expression) // Face meshfrom expr. parameters17 cache ← runDecoder(viewpoint, expression, mesh) // Cacheneural representation18 ”caching done” → cache queue // Notify cachecompletion via queue44Baselines. DNR [77] is the backbone we use as our reference. DNR+ improvesDNR with recent neural network architecture improvements from [34]. We fur-thermore add DNR++ that uses the current and past frame as input. DNR+ andDNR++ both act as a theoretical upper bound for our model’s image quality andtherefore, provide a good measure of the effectiveness and efficiency of our ap-proach. We also compare against the recent method from Wang et al. [82], usingtheir online interface (http://imaginaire.cc/vid2vid-cameo/) and in terms of run-time to Zakharov et al. [92] using the same 512× 512 image resolution. We donot provide peak signal-to-noise ratio (PSNR) numbers as it was designed for amuch smaller resolution of 256× 256. In addition, we create a naive baselinewhere we shift input frames by one to emulate the delay that incures when runninga large image generator without warping. It serves as an expected lower bound onaccuracy.Datasets. We use the talking head sequences from [77] to compare against priorwork as well as a beard and a high-fps sequence with more difficult facial hairthat was recorded with ethics board approval. An unsigned copy of the consentform we used can be found in the appendix. Videos for the beard and high-fpsdataset were recorded at 1920x1080 and split into, respectively, 2604/558/558 and3600/500/1000 frames for training/validation/testing. Trump [77] sequences is a1280x720 of 431 frames. The Obama [77] sequence is a 512x512 of 2412 frames.The male and female [77] sequences are both 768x768 with 2380 frames. Thehigh-fps recording has 60 FPS, all others run at 30 FPS.Training setup. We train all our models for 150 epochs using Adam [37] withbetas equal to {0.9, 0.999}, learning rate of 1e-4 for both the Generator, G andWarp network W , and 1e-3 for the neural texture.Metrics. We evaluate our image reconstruction accuracy using an L1 reconstruc-tion error between the ground truth and reconstructed image as well as PSNR andstructural similarity index (SSIM). Furthermore, to show our increase in speed,we report the latency and frames per second of our models and baselines. For all4510 20 30 40 50 60Latency [ms]25.425.625.826.026.226.426.6PSNRDNR+DNR+ (Naive)DNR++Ours (1x warp)ExWarp (1x warp)Ours (2x warp)ExWarp (2x warp)PSNR vs. Latency [ms]10 20 30 40 50 60 70Frames-Per-Second (FPS)25.425.625.826.026.226.426.626.827.0PSNRDNR+DNR+ (Naive)DNR++ Ours (1x warp)ExWarp (1x warp) Ours (2x warp)ExWarp (2x warp)PSNR vs. FPSFigure 3.4: Reconstruction performance vs. latency (left) and FPS (right)for our models and the baselines on the beard dataset. Optimal perfor-mance is in the top-left and top-right corners, indicated by the yellowstar in each respective figure. The vertical blue line marks the latencyand FPS of [92]. Models suitable for parallel execution are run usingtwo GPUs.methods, we only measure the time it takes to process the input UV maps and thecorresponding skeleton (pose), expression, and camera (extrinsic) information; ex-cluding the time it takes to render the UV map because they are implementationdependent and with negligible overhead when implemented in a rasterizer. Simi-larly, we only account for the time the inference generator takes in [92], ignoringthe processing of the conditional keypoint image. All latency and FPS metrics arecomputed on NVIDIA RTX 2080 GPUs. For our approaches, we report metricsand timings when we cache every frame (1x warp) and every second frame (2xwarp) to show our method is robust to varying deviations between input frames.3.6.1 Latency and Runtime ImprovementThe generator backbone has a runtime of 47.02ms and equivalent latency. Ourwarp net has a runtime of 14.62ms and therefore deduces latency by a factor of 3.2.Running on multiple GPUs improves frame rate from 28.5 to 67.6 while inducingonly a negligible 0.25 ms increase in latency due to the required synchronization.46ImageGT DNR+ DNR+ (Naive) Ours (1x w.) Ours (2x w.)ErrorImageErrorFigure 3.5: Reconstructed frames and their error maps for our model (1xand 2x warp), DNR+, and naive baseline. Based on the L1 error wecan see all models perform similarly, differences are visible in the errormaps for high-frequency details.Note that parallel execution across multiple GPUs can by itself not improve thelatency of a sequential process. When processing even and odd frames on differentGPUs, the time from input view to rendering output remains the same. The impactof different model configurations on latency is evaluated in Table 3.3.3.6.2 Offline Reconstruction QualityTo test image generation quality, we use a held-out test video and drive the trainedmodels using the FLAME head model reconstructed on the reference video. Theresults from Figure 3.5 and Table 3.1 evaluate the difference to the reference videoin offline processing mode. When comparing individual frames between the base-line and our model, Figure 3.5 shows that single (1x warp) and multi-frame warping47(2x warp) work nearly as well in terms of error. Table 3.1 reveals, somewhat sur-prisingly that 1x warp slightly outperforms the DNR+ baseline in terms of PSNRdespite having to warp with a shallow network. This is possible since it has ac-cess to the current and past frame that helps to correct errors in facial expressionestimation. To this end, we introduced the DNR++ baseline as a new upper boundthat also has access to the two previous frames. In summary, at a small reductionin accuracy compared to the baseline models, latency and frame rate are greatlyimproved by a factor of two or more.3.6.3 Online Reconstruction QualityOnline reconstruction requires the algorithm to run at the native frame rate of thevideo. This has a significant influence on the performance of our algorithm as thewarping operation becomes simpler for high-frame-rate videos where the motionbetween two frames is reduced, leading to even higher performance gains thanfor the previous offline evaluation. We test this effect on the high-fps sequenceshown in Figure 3.6. Table 3.2 reveals that running online on 60 FPS videos withOurs (2x warp) improves on Ours (1x warp), as the latter can only process everyother frame requiring larger warps. Hence, warping multiple times is beneficial,jointly improving in latency, runtime, and image quality, when parallel hardware isavailable. The basic DNR baselines do not even run at 30 FPS and are hence notModel #GPU L1 ↓ PSNR ↑ SSIM ↑ Latency [ms] ↓ FPS ↑timing baseline [92] 1 - - - 16.60 60.2DNR+ (Naive) 1 0.0278 25.56 0.8970 46.81 21.4DNR+ 1 0.0240 26.59 0.9165 46.81 21.4DNR++ 1 0.0237 26.67 0.9168 49.37 20.3ExWarp as in [92] (1x warp) 2 0.0257 26.33 0.9108 18.84 26.4ExWarp as in [92] (2x warp) 2 0.0260 26.24 0.9094 18.84 60.8Ours (1x warp) 1 0.0244 26.66 0.9107 14.62 16.3Ours (1x warp) 2 0.0244 26.66 0.9107 14.87 28.5Ours (2x warp) 1 0.0251 26.45 0.9069 14.62 26.3Ours (2x warp) 2 0.0251 26.45 0.9069 14.87 67.6Table 3.1: Offline evaluation on the beard dataset. As expected, our modeldoes not achieve the best metrics on the image reconstruction metrics,but they outperform the baselines in terms of latency and FPS. Timingresults for Ours 2x warp are using parallel execution.481x warp@30 fps2x warp@60 fpsGTFigure 3.6: Online face reconstruction setting. When running at their na-tive frame rate, high-fps models (2x warp) improve as they have tobridge a smaller gap between frames.comparable in the high-fps online setting.Note that absolute PSNR numbers differ across subjects and scenes since facesare smaller/bigger and also contain more or less high-frequency details, includingthe texture on the shoulder region which is of little concern for the facial featurereconstruction fidelity. Hence, it does not make sense to relate absolute but onlyrelative numbers between the beard and high-fps scores.3.6.4 Novel View Synthesis QualityReproducing a pre-recorded sequence does not necessarily need low latency sincethe entire video could be cached. Yet, latency is crucial for rendering a face froma novel viewpoint to account for the user’s head motion in VR and in general forviewpoint-dependent displays. We generate novel views of characters by rotatingOperation Mode L1 ↓ PSNR ↑ SSIM ↑Ours (1x warp @ 30fps) 0.0361 23.19 0.8148Ours (2x warp @ 60fps) 0.0359 23.23 0.8162Table 3.2: High-fps comparison for our warping network using realistic op-eration settings on the high-fps dataset. 2x warp not only improves speedbut even slightly improves the rendering quality.49(1x warp)-20.0 -10.0 0.0 10.0 20.0Ours Wang et al.'21Ours(2x warp)Figure 3.7: Comparison against Wang et al. [82]. While both models showsimilar levels of detail, ours is anchored in a 3D representation whichgives us more fine-grained and independent control.their underlying 3D mesh (which is used to generate our input UV maps) whileholding other parameters fixed. Figure 3.8 shows the retargeting of poses and viewsacross videos and 3.7 shows synthetically generated views and compares it to theresults from [82]. Because we condition on a full 3D face model, our rotation ismore precise and keeps the pose unchanged compared to the learned 3D featuresfrom Wang et al. that leads to the opening of the mouth and up-rotation. The qualityby [82] is expected to be slightly lower as it is not person-specific, which validatesour motivation to train a person-specific model.3.6.5 RetargetingTo show the flexibility of our approach, we retarget facial and head motions fromone person to another on in-the-wild videos that were established as a benchmarkin [77]. In this setting we train the neural texture and renderer on our target sub-ject and use these learned models at inference time, driven by the head motionreconstructed from a source sequence with a different actor. In Figure 3.9 we showthat our approach is capable of retargeting with just transferring expressions (asin [77]) and also when mapping the global head orientation. The original results,provided from [77] do not include head stabilization. Nevertheless, this compari-50Source Target -20.0 -10.0 0.0 10.0 20.0Figure 3.8: Novel views while retargeting. Our model is capable of gener-ating realistic, view-dependent novels views (Section 3.6.4) while mim-icking the source’s facial expressions (Section 3.6.5). These examplesare generated using Ours (1x warp) using the target actors from [77].son shows that our approach (1x warp) is reproducing or even outperforming theirimage quality.Furthermore, as we condition on a full 3D head model, we are able to gener-ate novel views while performing the retargeting as shown in Figure 3.8. Sinceour approach only approximates the background for each scene, we use a staticbackground in our predictions.3.6.6 Ablation StudiesNetwork Backbones.To show our method’s generality to other backbones, we compare our UNet to thewidely successful ResNet-backbone using residual blocks. As expected, warpingis effective in reducing latency from 113 ms to 26 ms, but the ResNet does notoutperform the UNet architectures. On the beard sequence, the ResNet backboneoperating using single warps gives a PSNR of 23.70 (UNet 26.66) and 23.18 PSNRusing two warps (UNet 26.45).51Sourcet=0 t=25 t=50 t=75 t=100 t=125TargetOurs (1x warp)Retarget w/ SourceOurs (1x warp)RetargetDNR'19Figure 3.9: Retargeting comparison without and with the transfer of theglobal head motion, including the comparison to [77].Concat UV Use Θ Use MLP SH Pose SH Skips ExWarp Exp. L1 ↓ PSNR ↑ SSIM ↑ Latency [ms] ↓ Rel. Latency [ms] ↓✓ 0.0276 25.6799 0.8987 12.80 -✓ ✓ 0.0275 25.6994 0.8989 13.24 0.44✓ ✓ ✓ 0.0271 25.8378 0.9012 13.61 0.81✓ ✓ ✓ ✓ 0.0260 26.1627 0.9038 13.79 0.99✓ ✓ ✓ ✓ ✓ 0.0244 26.5705 0.9105 14.40 1.60✓ ✓ ✓ ✓ ✓ ✓ 0.0257 26.3326 0.9108 18.84 6.03✓ ✓ ✓ ✓ ✓ ✓ 0.0244 26.6562 0.9107 14.62 1.82Table 3.3: Ablation study results for our warping network using the se-quential variant (1x warp), showing the relative improvement resultingfrom including each component on the beard dataset.Model Components.Adding the parameters and transformations used in Our full model one-by-one in-creases image quality (L1, PSNR, and SSIM) while only incurring a slight decrease52in latency and FPS (see rel. latency and relative FPS column, the relative latencycompared to our full model). Table 3.3 presents the results for the sequential oper-ation mode on the beard sequence over the following components:• Concat UV: using the difference of the current and cached UV maps in C3.• Use θ : concatenating the cached pose and camera extrinsics to C3.• Use MLP: passing θ , e, p through an MLP M before concatenation with C3.• SH Pose: includes the spherical harmonics Sob j in θ .• SH Skips: ’rotationally’ encode C4,C5 using ∆Sob j.• ExWarp: explicit warping from a reference by sampling the learned neuraltexture with Ut+1 and concatenating it with C3.• Exp: cache and concatenate the expression e with θ .The explicit warping (ExWarp, second-last row) adds a large latency increase whilenot improving the quality metrics consistently. Hence, we favour the implicit warpin our full model. When applied in parallel on multiple GPUs, these reduced la-tencies translate directly to improved frame rate. The synchronization overhead inthe parallel implementation is only 0.25ms/frame, which we measured by runningthe sequential model with the same threading and queue synchronization as for theparallel mode and taking their latency difference.Effect of Warp DistanceThis ablation study measures the performance drop incurred from warping moreframes ahead. Figure 3.10 shows that the reconstruction quality decreases linearlywith the warp distance on the beard dataset, which was recorded at 30fps. Pleasenote that this increase in error is expected as the proposed neural caching and warp-ing is a local approximation designed for high-fps videos where the warp distanceis small.Ablation on Caching LayersIn this ablation, we sequentially remove the caching layers (C3, C4, C5) and mea-sure the reconstruction quality in terms of L1 loss, PSNR, and SSIM. Each config-uration was trained on the beard dataset for 50 epochs, and metrics were computedon the held-out test set. Table 3.4 shows that adding the C4 cache layer is the most531 2 3 4 5Warp Distance [Frames]27.026.526.025.5PSNRPSNR vs. Warp DistanceFigure 3.10: Ablation on warp distance as we increase the warp distance,or the number of warping operations per caching operation, the recon-struction quality decreases linearly.C3 C4 C5 L1 ↓ PSNR ↑ SSIM ↑✓ 0.0261 26.13 0.9048✓ ✓ 0.0247 26.48 0.9096✓ ✓ ✓ 0.0246 26.54 0.9102Table 3.4: Ablating caching layers for our warping network using the se-quential variant (1x warp), showing the improvement after adding eachcomponent on the beard dataset.impactful, but since performance increases after adding all layers, we include allcache layers in our final configuration.3.7 Live System IntegrationThe following text is taken from: Ziyi Xia, Frank Yu, et al. 2022. Tele-ViewDemo: Experience the Future of 3D Teleconferencing. In SIGGRAPHAsia 2022 XR (SA ’22). Association for Computing Machinery, New York,NY, USA, Article 12, 1–2.To demonstrate the effectiveness of our approach we integrate our method into anend-to-end telepresence pipeline (TeleViewDemo). An example teleconferencing54Figure 3.11: Our end-to-end system (OpenTeleView). The Sender sidecamera captured image (left) is encoded using DECA to produceFLAME head parameters. These parameters are sent to the re-ceiver side where our neural renderer generates a photo-realistic view-dependent rendering which is shown on the Receiver’s 3D display(right).Figure 3.12: Block diagram of OpenTeleView. Our method makes up theDecoder, and the Encoder is replaced with DECA. At inference time,head-tracking is fed from the receiver to the Decoder to render view-dependent images.55Figure 3.13: Limitations of head reconstruction We can see that extremeangles and glasses present a problem to the capture algorithm and pro-duces inaccurate UV maps and parameters.frame with both the sender and receiver views is shown in Figure 3.11 and Fig-ure 3.12 shows a block diagram of the overall pipeline where our Neural Render-ing approach replaces the Decoder. In order for our work to act as the Decoder weimplement DECA as the Encoder which sends the per-frame-estimated FLAMEparameters. Furthermore, we utilize our method’s ability to generate novel viewsto render to a viewpoint-dependent display. We demonstrate the overall pipeline’sflexibility by implementing our system on 3 view-dependent displays: fish-tankvirtual reality, flat screen, and virtual reality. In our system implementation, weachieved a consistent 22 FPS and 172ms of latency in the system. A visualiza-tion of our end-to-end system integration is shown in Figure 3.11. The differencebetween the performance results we report in our isolated experiments and the end-to-end pipeline stem from the difference in data loading. In the isolated setting wepreload all the preprocessed data (from a fixed dataset) into memory whereas, in thelive end-to-end setting, we have to dynamically send data from the encoder/DECAand process the received data based on the receiver’s head position.3.8 LimitationsBecause our generator, G, and warping network, W , learn how to generate an imagewithout an explicit rendering equation, we require a diverse set of training views toensure that we can perform novel view synthesis and accurate warping at inferencetime. We can see in Figure 3.8 that the novel views break at extreme angles aroundthe ears of the target subject as these are not seen during our training videos. The56limiting factor here is the face reconstruction algorithm which becomes unreliablewhen large parts of the face are occluded, at extreme poses, shown in Figure 3.13.Eyes and wide-open mouths can also pose a problem since they are represented asholes in the underlying FLAME model and therefore the direction of the eye gazeand tongue cannot be modeled. Furthermore, because the FLAME model onlyestimates 3D models for the head, the network struggles in cases where users haveglasses or expressions that it cannot express. Improving image quality in thesedirections is largely orthogonal to our contributions towards low latency and highframe rate.57Chapter 4Limitations and Future Work4.1 LimitationsWhile we present two methods that improve performance on their respective base-lines, they are not without limitations. One integral flaw with defining any warpingis the need for either specific supporting data or application-specific knowledge,like camera parameters or an assumed spatiotemporal relationship. This meansthat techniques struggle with scale. For PCLs we assume that ground truth cameraintrinsics are available. This is a valid assumption when working on videos/im-ages recorded in a controlled setting but would not apply to in-the-wild imagesand videos. There are some methods that can estimate camera intrinsics, how-ever, without testing, we cannot quantify the expected performance improvements.Our method disentangles the camera intrinsics from the model learning pipeline.Therefore, we do expect the performance of future methods which include PCLsto improve alongside future intrinsic estimation methods.For our work on talking-head synthesis, the critical assumption is that there isnot much visual change between input frames, as such we can define a light im-plicit warping model. Therefore, for high-framerate videos and videos with slowermovements, our method works very well. However, this may pose a problem incurrent live settings where hardware can limit the input framerate (ie. few webcamsare capable of recording video at or above 60 FPS). As technology develops, andhardware improves, not only will this pose less of a limitation, it actually validates58our assumption. As described in the chapter, as frame rates increase, there willexist fewer visual differences between frames. Furthermore, in order to performthe warping we require per-frame codes (for example, expression and pose codes)for changes in appearance that would not scale to scenes outside our talking-headapplication. For example, a scene of a ball rolling down a hill would pose a chal-lenge due to the lack of per-frame signals we can extract and leverage. Lastly, inorder to synthesize truly realistic talking heads, we must be able to model and ad-just the lighting. Our current model ”bakes in” the lighting at training time makingit unsuitable for relighting applications.4.2 Future WorkThe natural extension for PCLs to adapt it to diverse, in-the-wild image sourceswould be to devise a way to properly estimate per-frame camera intrinsics. Someprior work already attempts this [2, 38]. Integrating this work with ours wouldhelp bridge the gap between the performance we would see in a naive baselineand those we present in our experiments. Furthermore, the natural extension forour talking head work would be to extend it to videos with more visual differencesbetween frames or low frame rate inputs. It would then be possible to extend this tofull-body motions. For generic scenes, it would highly depend on the accuracy andefficiency of low-level, scene agnostic, conditioning signals, like optical or sceneflow, which have shown promise on purely synthetic scenes.59Chapter 5ConclusionRecent advances in deep learning and their architectures have been accelerated byincreased GPU sizes and speeds. However, relying on the continued scaling ofGPU advances is infeasible as expectations and demands for higher quality anddeeper neural networks also increase. Therefore, it’s inevitable that we will needto efficiently utilize available resources. We already see examples of warping frommajor companies like Nvidia. In this thesis, we present two different methods ofwarping in order to maximize both available data and neural network inferencelatency. We hope these findings motivate further research to train and design in-creasingly efficient neural networks.5.1 General ImpactRecent developments have made virtual communication commonplace in daily life.Therefore, it is more important than ever to produce technological solutions thatbridge the gap between the virtual and physical worlds. Our solutions facilitatethe development of virtual avatars by improving the underlying skeleton accuracyand the inference speeds to make them interactive. We promote two solutions thatimpact a wide array of everyday imaging solutions. Nevertheless, as we developtechnologies that can generate realistic images, especially of people, we also openthe possibility for generating deep fakes. Deep fakes pose a considerable threat dueto their potential to cause everything from personal defamation to mass misinfor-60mation. Therefore, we must be careful about how we develop and deploy theseapplications and the data we use to train them. While our proposed methods couldbe used to generate deep fakes, we do not explicitly improve the reconstructionquality, only the latency and frame rates at inference time.61Bibliography[1] V. Blanz and T. Vetter. A morphable model for the synthesis of 3D faces.1999. → page 34[2] O. Bogdan, V. Eckstein, F. Rameau, and J.-C. Bazin. Deepcalib: a deeplearning approach for automatic intrinsic calibration of wide field-of-viewcameras. pages 1–10, 12 2018. doi:10.1145/3278471.3278479. → page 59[3] A. Bulat and G. Tzimiropoulos. How far are we from solving the 2d & 3dface alignment problem? (and a dataset of 230,000 3d facial landmarks). InInternational Conference on Computer Vision, 2017. → page 36[4] J. Carreira, V. Patraucean, L. Mazare, A. Zisserman, and S. Osindero.Massively parallel video networks. In Proceedings of the EuropeanConference on Computer Vision (ECCV), pages 649–666, 2018. → pages32, 35[5] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li,S. Savarese, M. Savva, S. Song, H. Su, et al. Shapenet: An information-rich3d model repository. arXiv preprint arXiv:1512.03012, 2015. → page 6[6] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. Tensorf: Tensorial radiancefields. In European Conference on Computer Vision (ECCV), 2022. → page34[7] C.-H. Chen and D. Ramanan. 3d human pose estimation= 2d poseestimation+ matching. In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition, pages 7035–7043, 2017. → page 4[8] X. Chu, B. Zhang, H. Ma, R. Xu, and Q. Li. Fast, accurate and lightweightsuper-resolution with neural architecture search. In 2020 25th InternationalConference on Pattern Recognition (ICPR), pages 59–64. IEEE, 2021. →page 4362[9] T. Cohen, M. Geiger, J. Ko¨hler, and M. Welling. Convolutional networks forspherical signals. arXiv preprint arXiv:1709.04893, 2017. → page 7[10] B. Coors, A. Paul Condurache, and A. Geiger. Spherenet: Learningspherical representations for detection and classification in omnidirectionalimages. In Proceedings of the European Conference on Computer Vision(ECCV), pages 518–533, 2018. → page 7[11] J. Dai, H. Qi, Y. Xiong, Y. Li, G. Zhang, H. Hu, and Y. Wei. Deformableconvolutional networks. In Proceedings of the IEEE internationalconference on computer vision, pages 764–773, 2017. → page 7[12] Y. Deng, J. Yang, D. Chen, F. Wen, and X. Tong. Disentangled andcontrollable face image generation via 3D imitative-contrastive learning. InCVPR, 2020. → page 34[13] Y. Du, Y. Wong, Y. Liu, F. Han, Y. Gui, Z. Wang, M. Kankanhalli, andW. Geng. Marker-less 3D human motion capture with monocular imagesequence and height-maps. In European Conference on Computer Vision(ECCV), pages 20–36. Springer, 2016. → page 3[14] H. Fan, H. Su, and L. Guibas. A Point Set Generation Network for 3DObject Reconstruction from a Single Image. In Conference on ComputerVision and Pattern Recognition, 2017. → page 6[15] H. Fang, Y. Xu, W. Wang, X. Liu, and S. Zhu. Learning pose grammar toencode human body configuration for 3d pose estimation. In S. A. McIlraithand K. Q. Weinberger, editors, Proceedings of the AAAI Conference onArtificial Intelligence, pages 6821–6828. AAAI Press, 2018. → page 4[16] C. Feichtenhofer. X3d: Expanding architectures for efficient videorecognition. In Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition, pages 203–213, 2020. → page 35[17] Y. Feng, H. Feng, M. J. Black, and T. Bolkart. Learning an animatabledetailed 3D face model from in-the-wild images. volume 40, 2021. URLhttps://doi.org/10.1145/3450626.3459936. → pages 36, 43[18] S. Fridovich-Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa.Plenoxels: Radiance fields without neural networks. In CVPR, 2022. →page 3463[19] O. Fried, A. Tewari, M. Zollho¨fer, A. Finkelstein, E. Shechtman, D. B.Goldman, K. Genova, Z. Jin, C. Theobalt, and M. Agrawala. Text-basedediting of talking-head video. ACM TOG, 2019. → page 34[20] J. Geng, T. Shao, Y. Zheng, Y. Weng, and K. Zhou. Warp-guided GANs forsingle-photo facial animation. ACM TOG, 2018. → page 34[21] P. Ghosh, P. S. Gupta, R. Uziel, A. Ranjan, M. J. Black, and T. Bolkart. Gif:Generative interpretable faces. In 2020 International Conference on 3DVision (3DV), pages 868–878. IEEE, 2020. → page 34[22] P.-W. Grassal, M. Prinzler, T. Leistner, C. Rother, M. Nießner, and J. Thies.Neural head avatars from monocular rgb videos. arXiv preprintarXiv:2112.01554, 2021. → page 34[23] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residualnetworks. In European conference on computer vision, pages 630–645.Springer, 2016. → page 18[24] G. Hinton, A. Krizhevsky, and S. Wang. Transforming Auto-Encoders. InInternational Conference on Artificial Neural Networks, pages 44–51, 2011.→ page 6[25] Y. Hold-Geoffroy, K. Sunkavalli, J. Eisenmann, M. Fisher, E. Gambaretto,S. Hadap, and J.-F. Lalonde. A perceptual measure for deep single imagecamera calibration. In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition, pages 2354–2363, 2018. → pages 5, 26[26] P. Hu, F. Caba, O. Wang, Z. Lin, S. Sclaroff, and F. Perazzi. Temporallydistributed networks for fast video semantic segmentation. In Proceedings ofthe IEEE/CVF Conference on Computer Vision and Pattern Recognition,pages 8818–8827, 2020. → page 35[27] C. Ionescu, I. Papava, V. Olaru, and C. Sminchisescu. Human3.6M: LargeScale Datasets and Predictive Methods for 3D Human Sensing in NaturalEnvironments. IEEE Transactions on Pattern Analysis and MachineIntelligence, 2014. → pages 18, 20[28] P. Isola, J.-Y. Zhu, T. Zhou, and A. A. Efros. Image-to-image translationwith conditional adversarial networks. 2017. → page 36[29] M. Jaderberg, K. Simonyan, A. Zisserman, and K. Kavukcuoglu. SpatialTransformer Networks. In Advances in Neural Information ProcessingSystems, pages 2017–2025, 2015. → pages 7, 1764[30] J. Johnson, A. Alahi, and L. Fei-Fei. Perceptual losses for real-time styletransfer and super-resolution. In European conference on computer vision,pages 694–711. Springer, 2016. → page 42[31] A. Kanazawa, S. Tulsiani, A. Efros, and J. Malik. LearningCategory-Specific Mesh Reconstruction from Image Collections. EuropeanConference on Computer Vision, 2018. → page 6[32] T. Karras, S. Laine, and T. Aila. A style-based generator architecture forgenerative adversarial networks. In CVPR, 2019. → page 34[33] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila.Analyzing and improving the image quality of StyleGAN. In CVPR, 2020.→ page 34[34] T. Karras, M. Aittala, S. Laine, E. Ha¨rko¨nen, J. Hellsten, J. Lehtinen, andT. Aila. Alias-free generative adversarial networks. arXiv preprintarXiv:2106.12423, 2021. → pages 42, 45[35] R. Khasanova and P. Frossard. Graph-based classification of omnidirectionalimages. In Proceedings of the IEEE International Conference on ComputerVision, pages 869–878, 2017. → page 7[36] D. Kingma and J. Ba. Adam: A Method for Stochastic Optimisation. InInternational Conference on Learning Representations, 2015. → page 21[37] D. Kingma and J. Ba. Adam: A method for stochastic optimization. InICLR, 2015. → page 45[38] S.-E. Lee, K. Shibata, S. Nonaka, S. Nobuhara, and K. Nishino. Extrinsiccamera calibration from a moving person. IEEE Robotics and AutomationLetters, 7(4):10344–10351, 2022. doi:10.1109/LRA.2022.3192629. → page59[39] S. Li, L. Ke, K. Pratama, Y.-W. Tai, C.-K. Tang, and K.-T. Cheng. Cascadeddeep monocular 3d human pose estimation with evolutionary training data.In Proceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition (CVPR), June 2020. → page 3[40] T. Li, T. Bolkart, M. J. Black, H. Li, and J. Romero. Learning a model offacial shape and expression from 4d scans. ACM Trans. Graph., 36(6):194–1, 2017. → pages 34, 36, 4265[41] C.-H. Lin and S. Lucey. Inverse compositional spatial transformer networks.In Proceedings of the IEEE Conference on Computer Vision and PatternRecognition, pages 2568–2576, 2017. → page 7[42] R. Liu, J. Lehman, P. Molino, F. P. Such, E. Frank, A. Sergeev, andJ. Yosinski. An intriguing failing of convolutional neural networks and thecoordconv solution. In Advances in Neural Information Processing Systems,pages 9605–9616, 2018. → page 7[43] C. Luo, X. Chu, and A. L. Yuille. Orinet: A fully convolutional network for3d human pose estimation. In British Machine Vision Conference (BMVC),page 92, 2018. URL http://bmvc2018.org/contents/papers/0289.pdf. →page 3[44] M. Malinowski, G. Swirszcz, J. Carreira, and V. Patraucean. Sideways:Depth-parallel training of video models. In Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition, pages11834–11843, 2020. → page 35[45] J. Martinez, R. Hossain, J. Romero, and J. Little. A Simple Yet EffectiveBaseline for 3D Human Pose Estimation. In International Conference onComputer Vision, 2017. → pages 18, 19, 22, 23[46] J. Martinez, R. Hossain, J. Romero, and J. J. Little. A simple yet effectivebaseline for 3d human pose estimation. In International Conference onComputer Vision (ICCV), 2017. → pages 4, 6[47] D. Mehta, H. Rhodin, D. Casas, P. Fua, O. Sotnychenko, W. Xu, andC. Theobalt. Monocular 3D Human Pose Estimation in the Wild UsingImproved CNN Supervision. In International Conference on 3D Vision,2017. → page 6[48] D. Mehta, H. Rhodin, D. Casas, P. Fua, O. Sotnychenko, W. Xu, andC. Theobalt. Monocular 3d human pose estimation in the wild usingimproved cnn supervision. In 3D Vision (3DV), 2017 Fifth InternationalConference on. IEEE, 2017. doi:10.1109/3dv.2017.00064. URLhttp://gvv.mpi-inf.mpg.de/3dhp dataset. → pages 18, 20[49] D. Mehta, H. Rhodin, D. Casas, P. Fua, O. Sotnychenko, W. Xu, andC. Theobalt. Monocular 3d human pose estimation in the wild usingimproved cnn supervision. In International Conference on 3D Vision (3DV).IEEE, 2017. doi:10.1109/3dv.2017.00064. URLhttp://gvv.mpi-inf.mpg.de/3dhp dataset. → page 466[50] D. Mehta, S. Sridhar, O. Sotnychenko, H. Rhodin, M. Shafiei, H.-P. Seidel,W. Xu, D. Casas, and C. Theobalt. Vnect: Real-time 3d human poseestimation with a single rgb camera. In ACM Transactions on Graphics,volume 36, 7 2017. doi:10.1145/3072959.3073596. URLhttp://gvv.mpi-inf.mpg.de/projects/VNect/. → page 3[51] D. Mehta, O. Sotnychenko, F. Mueller, W. Xu, M. Elgharib, P. Fua, H.-P.Seidel, H. Rhodin, G. Pons-Moll, and C. Theobalt. XNect: Real-timemulti-person 3D motion capture with a single RGB camera. volume 39,2020. → page 4[52] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi,and R. Ng. Nerf: Representing scenes as neural radiance fields for viewsynthesis. In European conference on computer vision, pages 405–421.Springer, 2020. → page 34[53] F. Moreno-Noguer. 3d human pose estimation from a single image viadistance matrix regression. In Proceedings of the Conference on ComputerVision and Pattern Recognition (CVPR), 2017. → page 4[54] T. Mu¨ller, A. Evans, C. Schied, and A. Keller. Instant neural graphicsprimitives with a multiresolution hash encoding. ACM Trans. Graph., 41(4):102:1–102:15, July 2022. doi:10.1145/3528223.3530127. URLhttps://doi.org/10.1145/3528223.3530127. → page 34[55] K. Nagano, J. Seo, J. Xing, L. Wei, Z. Li, S. Saito, A. Agarwal, J. Fursund,and H. Li. paGAN: real-time avatars using dynamic textures. ACM TOG,2018. → page 34[56] A. Nibali, Z. He, S. Morgan, and L. Prendergast. 3d human pose estimationwith 2d marginal heatmaps. In 2019 IEEE Winter Conference onApplications of Computer Vision (WACV), pages 1477–1485. IEEE, 2019.→ page 20[57] K. Olszewski, Z. Li, C. Yang, Y. Zhou, R. Yu, Z. Huang, S. Xiang, S. Saito,P. Kohli, and H. Li. Realistic dynamic facial textures from a single imageusing GANs. In ICCV, 2017. → page 34[58] S. Park, J. Hwang, and N. Kwak. 3d human pose estimation usingconvolutional neural networks with 2d pose information. In EuropeanConference on Computer Vision (ECCV), pages 156–169, 2016.doi:10.1007/978-3-319-49409-8\ 15. URLhttps://doi.org/10.1007/978-3-319-49409-8 15. → page 367[59] G. Pavlakos, X. Zhou, K. G. Derpanis, and K. Daniilidis. Coarse-to-finevolumetric prediction for single-image 3d human pose. Conference onComputer Vision and Pattern Recognition (CVPR), pages 1263–1272, 2017.→ page 3[60] G. Pavlakos, X. Zhou, and K. Daniilidis. Ordinal depth supervision for 3dhuman pose estimation. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition, pages 7307–7316, 2018. → page4[61] D. Pavllo, C. Feichtenhofer, D. Grangier, and M. Auli. 3d human poseestimation in video with temporal convolutions and semi-supervisedtraining. In Proceedings of the IEEE Conference on Computer Vision andPattern Recognition, pages 7753–7762, 2019. → pages 6, 18, 20, 24[62] M. Rayat Imtiaz Hossain and J. J. Little. Exploiting temporal informationfor 3d human pose estimation. In European Conference on Computer Vision(ECCV), 2018. → page 4[63] A. Recasens, P. Kellnhofer, S. Stent, W. Matusik, and A. Torralba. Learningto zoom: a saliency-based sampling layer for neural networks. InProceedings of the European Conference on Computer Vision (ECCV),pages 51–66, 2018. → page 30[64] S. J. Reddi, S. Kale, and S. Kumar. On the convergence of adam andbeyond. arXiv preprint arXiv:1904.09237, 2019. → page 21[65] H. Rhodin, M. Salzmann, and P. Fua. Unsupervised Geometry-AwareRepresentation for 3D Human Pose Estimation. In European Conference onComputer Vision, 2018. → page 6[66] H. Rhodin, J. Spoerri, I. Katircioglu, V. Constantin, F. Meyer, E. Moeller,M. Salzmann, and P. Fua. Learning Monocular 3D Human Pose Estimationfrom Multi-View Images. In Conference on Computer Vision and PatternRecognition, 2018. → page 6[67] G. Rogez, P. Weinzaepfel, and C. Schmid. LCR-Net:Localization-Classification-Regression for Human Pose. In Conference onComputer Vision and Pattern Recognition (CVPR), pages 1216–1224,Honolulu, United States, July 2017. IEEE. doi:10.1109/CVPR.2017.134.URL https://hal.inria.fr/hal-01505085. → page 368[68] Y.-C. Su and K. Grauman. Learning spherical convolution for fast featuresfrom 360 imagery. In Advances in Neural Information Processing Systems,pages 529–539, 2017. → page 7[69] R. Sukthankar and M. D. Mullin. Automatic keystone correction forcamera-assisted presentation interfaces. In Advances in MultimodalInterfaces (ICMI), 2000. → page 6[70] K. Sun, B. Xiao, D. Liu, and J. Wang. Deep high-resolution representationlearning for human pose estimation. In CVPR, 2019. → pages 20, 23[71] X. Sun, J. Shang, S. Liang, and Y. Wei. Compositional human poseregression. In Proceedings of the IEEE International Conference onComputer Vision, pages 2602–2611, 2017. → page 3[72] S. Suwajanakorn, S. M. Seitz, and I. Kemelmacher-Shlizerman.Synthesizing Obama: learning lip sync from audio. ACM TOG, 2017. →page 34[73] B. Tekin, I. Katircioglu, M. Salzmann, V. Lepetit, and P. Fua. Structuredprediction of 3d human pose with deep neural networks. In British MachineVision Conference (BMVC), 2016. URLhttp://www.bmva.org/bmvc/2016/papers/paper130/index.html. → page 3[74] A. Tewari, M. Elgharib, G. Bharaj, F. Bernard, H.-P. Seidel, P. Pe´rez,M. Zo¨llhofer, and C. Theobalt. StyleRig: Rigging StyleGAN for 3d controlover portrait images. In CVPR, 2020. → page 34[75] J. Thies, M. Zollho¨fer, M. Nießner, L. Valgaerts, M. Stamminger, andC. Theobalt. Real-time expression transfer for facial reenactment. ACMTOG, 2015. → page 34[76] J. Thies, M. Zollhofer, M. Stamminger, C. Theobalt, and M. Nießner.Face2Face: Real-time face capture and reenactment of rgb videos. In CVPR,2016.[77] J. Thies, M. Zollho¨fer, and M. Nießner. Deferred neural rendering: Imagesynthesis using neural textures. ACM Transactions on Graphics (TOG), 38(4):1–12, 2019. → pages 33, 34, 35, 36, 41, 42, 45, 50, 51, 52[78] D. Tome, C. Russell, and L. Agapito. Lifting from the deep: Convolutional3d pose estimation from a single image. In The IEEE Conference onComputer Vision and Pattern Recognition (CVPR), July 2017. → page 369[79] J. Van Waveren. The asynchronous time warp for virtual reality onconsumer hardware. In Proceedings of the 22nd ACM Conference on VirtualReality Software and Technology, pages 37–46, 2016. → page 33[80] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,Ł. Kaiser, and I. Polosukhin. Attention is all you need. In NeurIPS, 2017.→ page 37[81] D. Vlasic, M. Brand, H. Pfister, and J. Popovic. Face transfer withmultilinear models. ACM TOG, 2005. → page 34[82] T.-C. Wang, A. Mallya, and M.-Y. Liu. One-shot free-view neuraltalking-head synthesis for video conferencing. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages10039–10049, 2021. → pages x, 35, 45, 50[83] S. Workman, C. Greenwell, M. Zhai, R. Baltenberger, and N. Jacobs.Deepfocal: A method for direct focal length estimation. In 2015 IEEEInternational Conference on Image Processing (ICIP), pages 1369–1373.IEEE, 2015. → pages 5, 26[84] Z. Xia, F. Yu, B. Xiong, E. Jia, S. Baek, J. Gregson, X. He, H. Rhodin, andS. Fels. Televiewdemo: Experience the future of 3d teleconferencing. InSIGGRAPH Asia 2022 XR, SA ’22, New York, NY, USA, 2022. Associationfor Computing Machinery. ISBN 9781450394734.doi:10.1145/3550472.3558404. URLhttps://doi.org/10.1145/3550472.3558404. → page v[85] B. Xiao, H. Wu, and Y. Wei. Simple baselines for human pose estimationand tracking. In European Conference on Computer Vision (ECCV), 2018.→ pages 20, 23[86] J. Xu, Z. Yu, B. Ni, J. Yang, X. Yang, and W. Zhang. Deep kinematicsanalysis for monocular 3d human pose estimation. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR), June 2020. → page 3[87] X. Yan, J. Yang, E. Yumer, Y. Guo, and H. Lee. Perspective transformernets: Learning single-view 3d object reconstruction without 3d supervision.In Advances in Neural Information Processing Systems, pages 1696–1704,2016. → pages 6, 770[88] J. Yang, S. Reed, M.-H. Yang, and H. Lee. Weakly-SupervisedDisentangling with Recurrent Transformations for 3D View Synthesis. InAdvances in Neural Information Processing Systems, pages 1099–1107,2015. → page 6[89] A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa. Plenoctrees forreal-time rendering of neural radiance fields. arXiv preprintarXiv:2103.14024, 2021. → page 34[90] F. Yu, M. Salzmann, P. Fua, and H. Rhodin. Pcls: Geometry-aware neuralreconstruction of 3d pose with perspective crop layers. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages9064–9073, 2021. → page v[91] F. Yu, S. Fels, and H. Rhodin. Scaling neural face synthesis to high fps andlow latency by neural caching. In Proceedings of the IEEE/CVF WinterConference on Applications of Computer Vision, pages 3474–3483, 2023. →page v[92] E. Zakharov, A. Ivakhnenko, A. Shysheya, and V. Lempitsky. Fast bi-layerneural synthesis of one-shot realistic head avatars. In ECCV, 2020. → pages35, 45, 46, 48[93] Y. Zhao, Z. Huang, T. Li, W. Chen, C. LeGendre, X. Ren, A. Shapiro, andH. Li. Learning perspective undistortion of portraits. In Proceedings of theIEEE International Conference on Computer Vision, pages 7849–7859,2019. → page 30[94] K. Zhou, X. Han, N. Jiang, K. Jia, and J. Lu. Hemlets pose: Learningpart-centric heatmap triplets for accurate 3d human pose estimation. InProceedings of the IEEE International Conference on Computer Vision,pages 2344–2353, 2019. → page 3[95] Q. Zhou, G. Miller, K. Wu, I. Stavness, and S. Fels. Analysis and practicalminimization of registration error in a spherical fish tank virtual realitysystem. volume 10114, pages 519–534, 03 2017. ISBN 978-3-319-54189-1.→ page 37[96] Q. Zhou, K. Wu, G. Miller, I. Stavness, and S. Fels. 3dps: Anauto-calibrated three-dimensional perspective-corrected spherical display. In2017 IEEE Virtual Reality (VR), pages 455–456. IEEE, 2017. → page 3471[97] Q. Zhou, G. Hagemann, D. Fafard, I. Stavness, and S. Fels. An evaluation ofdepth and size perception on a spherical fish tank virtual reality display.IEEE transactions on visualization and computer graphics, 2019. URLhttps://ieeexplore.ieee.org/abstract/document/8642347. → page 3172Appendix ASupporting MaterialsA.1 Consent Form for Collecting Training DataAttached is the consent form we gave volunteers prior to recording their data totrain personalized models for our work in Chapter 3.73 November 22, 2021 H19-02997 1 of 3 Participant Consent Form Markerless Motion Capture dataset recording, a new benchmark for motion capture algorithms You are being invited to participate in a study entitled “Next Generation 3D Content Creation for End Users on Mobile Devices” that is being conducted by Helge Rhodin. Helge Rhodin is the principal investigator and can be reached by mail and phone . The purpose of this study is to record a large dataset of casual motions performed in every-day environments and appearances. The dataset will be used to train and evaluate computer vision and machine learning algorithms for reconstructing and modelling human motion and appearance. The output of the algorithm is a virtual copy of the person with capabilities to change the persons pose, shape and appearance. However, use is restricted to alterations that are nonviolent, non-offensive and respectful. The use of the dataset for the purpose of surveillance applications as well as any non-ethical use is explicitly excluded. The construction of such dataset is important because it allows us to understand whether the developed tools are efficient in reconstructing video recordings of persons. Example applications are telepresence and motion capture for sports, medicine, and entertainment. If you volunteer to participate in this research, your participation may include: • Recording of scripted every-day motions such as walking, running, gesturing, hugging, waving, jumping, and reaching. • Recording of you in your personal clothing. • Recording of your facial expressions and facial appearance, should you consent. The research session is expected to take approximately 30 to 45 minutes and, depending on the research stage, will take place at one of the following locations: • In a closed room in the UBC Computer Science Department. • At a public place such as plaza, open grass field, or wide path. There are only minor anticipated risks to you by participating in this research since only every-day motions are performed and no personal information is retained except video recordings of your scripted performance and optionally your age, height, weight and ethnicity. It is of equal or lower risk as posting a video of you on Youtube. COVID-19 poses additional risk that will be mitigated with the following measures. Hand sanitizer will be available at the recording site and contact objects will be disinfected before and after every use. Recordings will be performed under the strict physical distancing rules (physical separation and 2m minimum distance between persons) and other measures installed at UBC/CS (e.g., one-way staircases). Public transport should be avoided by the participants. Only a single subject will be recorded at a time until physical distancing rules are relaxed. We Department of Computer Science University of British Columbia 201-2366 Main Mall Vancouver, V6T 124 Canada 74 November 22, 2021 H19-02997 1 of 3 rank the risk for participants by COVID as extremely low; yet doing many group activities would increase the accumulated risk for society and must be limited until treatments are available. Taking part in this study is entirely up to you. You have the right to refuse to participate in this study. If you decide to take part, you may choose to pull out of the study at any time without giving a reason and without any negative impact. If you do withdraw, we will ask whether we may use your data for data analysis and destroy your data otherwise. Your confidentiality and the confidentiality of the data will be protected by identifying data only with a participant number rather than your name. All data stored in computer files will be encrypted and password- protected. Only the principal investigator and co-investigators will have access to any personal information you provide. We will give access to the main study data via an open-access research repository, earliest three months after data recording. From that point on, your will not be able to withdraw your data. Note that all personal identifiers are removed beforehand, unless otherwise agreed upon, and this poses minimal risk to participants. We anticipate that parts of this dataset will be shared with others in the following ways: • Published articles and their supplemental material • Conference presentations • Theses • Internet project descriptions • This study is sponsored by NSERC and our commercial project partner, Huawei Canada Inc. You may give consent to allow Huawei to use the recordings for internal research and development, under the same conditions as the public datasets, except for allowing commercial use. Only algorithmic development is allowed. For instance, algorithm performance may be evaluated on recorded videos to select the best model. The captured data and its derivatives, such as machine learning algorithms trained on it, cannot be sold nor re-distributed to third parties. Data from this study will be stored for at least 10 years but at most 25 years, unless consent is given or earlier termination is requested. Electronic data will be erased and paper copies will be shredded. Imagery published in conferences and journals cannot be deleted. If you have any concerns or complaints about your rights as a research participant and/or your experiences while participating in this study, contact the Research Participant Complaint Line in the UBC Office of Research Ethics at or if long distance e-mail or call toll free . Your signature below indicates that you have received a copy of this consent form for your own records and implies your consent to participate for the duration of the study today and that the recorded data may be used for the purposes outlined in this document. You can withdraw your participation at any time, as outlined above. 75 November 22, 2021 H19-02997 1 of 3 You may provide additional consent for open-access, retaining raw recordings, and indefinite storage: ☐ Please tick this box if you allow publishing your recordings in open-access repositories and papers. Note that data cannot be revoked after publication. Open-access is only given three months after recording and you can still withdraw at any time until then. ☐ Please tick this box if you allow Huawei to use your recordings in for internal algorithmic development. Note that selling or deployment of the data in products will still be excluded, it is for R&D only. ☐ Please tick this box if you give additional consent for storing and re-distributing your raw video recordings. Raw video enables people who know you or have imagery with identification information to identify you in the video. Please note that you will not be able to withdraw any footage that is printed and displayed in journal and conference articles and their supplemental documents nor data provided on public-access repositories. ☐ Please tick this box if you give additional consent for storing your data indefinitely, beyond the 25 years. __________________________ ________________________________ ________________________ Name of Participant Signature Date A copy of this consent will be left with you, and a copy will be taken by the researcher. 76
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Rhodin, Helge
Explicit and implicit warping for accurate human pose estimation and low-latency neural rendering
Text