Science, Faculty of
Computer Science, Department of
DSpace
UBCV
Bolanos, Luis
Master of Science - MSc
University of British Columbia
High-fidelity character models form an integral part in multiple domains, including visual effects, animation, gaming, telepresence, fashion, and more. Recently, approaches for learning and creating 3D animatable avatars through neural rendering have obtained impressive rendering results, even when learning from monocular video data. These neural characters, while achieving highly detailed geometry and texture reconstruction, bake lighting and shadow information into the model, leading to artifacts when generating novel views, novel poses, or when relighting the model. The highest quality models are trained on data that is uniformly lit, and will otherwise contain artifacts in the shape reconstruction if using illuminated data. Consequently, the use of these neural characters is limited and has not seen widespread adoption in the aforementioned industries. In this thesis, we address the limitations of neural character lighting, specifically shadow casting, enabling explicit and efficient light calculations while disentangling shadows in the learned texture. Our primary contributions include: 1) A framework for approximating the shape of neural characters using a sum of 3D anisotropic Gaussians; 2) The derivation of a simple analytic formula for integrating the sum of anisotropic Gaussians along a ray; 3) A framework to perform light and shadow calculations using a deferred rendering approach for easy integration on existing models. These contributions enable the training of neural characters from outdoor lit data, and the relighting of the neural characters in a wide array of environments, increasing their utility in industry.
Gaussian Shadow Casting for NeuralCharactersbyLuis BolanosB.Sc., The University of British Columbia, 2022A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFMASTER OF SCIENCEinThe Faculty of Graduate and Postdoctoral Studies(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)April 2024© Luis Bolanos 2024The following individuals certify that they have read, and recommend tothe Faculty of Graduate and Postdoctoral Studies for acceptance, the thesisentitled:Gaussian Shadow Casting for Neural Characterssubmitted by Luis Bolanos in partial fulfillment of the requirements forthe degree of Master of Sciencein Computer ScienceExamining Committee:Helge Rhodin, Assistant Professor, Department of Computer Science, UBCSupervisorLeonid Sigal, Professor, Department of Computer Science, UBCSupervisory Committee MemberDinesh K. Pai, Professor, Department of Computer Science, UBCSupervisory Committee MemberiiAbstractHigh-fidelity character models form an integral part in multiple domains,including visual effects, animation, gaming, telepresence, fashion, and more.Recently, approaches for learning and creating 3D animatable avatars throughneural rendering have obtained impressive rendering results, even when learn-ing from monocular video data. These neural characters, while achiev-ing highly detailed geometry and texture reconstruction, bake lighting andshadow information into the model, leading to artifacts when generatingnovel views, novel poses, or when relighting the model. The highest qualitymodels are trained on data that is uniformly lit, and will otherwise containartifacts in the shape reconstruction if using illuminated data. Consequently,the use of these neural characters is limited and has not seen widespreadadoption in the aforementioned industries.In this thesis, we address the limitations of neural character lighting,specifically shadow casting, enabling explicit and efficient light calculationswhile disentangling shadows in the learned texture. Our primary contribu-tions include: 1) A framework for approximating the shape of neural charac-ters using a sum of 3D anisotropic Gaussians; 2) The derivation of a simpleanalytic formula for integrating the sum of anisotropic Gaussians along aray; 3) A framework to perform light and shadow calculations using a de-ferred rendering approach for easy integration on existing models. Thesecontributions enable the training of neural characters from outdoor lit data,and the relighting of the neural characters in a wide array of environments,increasing their utility in industry.iiiLay SummaryA major component of high-fidelity character models is lighting, which in-volves the simulation of how light rays reflect on surfaces to determine shad-ing, while checking for occlusions to determine shadows. Accurate lightinggrounds the characters in the environment in which they are shown, improv-ing the viewer’s experience. Typically, accurate lighting calculations requireexplicit ray-tracing to simulate how light paths travel in a scene. In thisthesis, we introduce a new approach for explicit light calculations via ray-tracing using a Gaussian density proxy for neural characters. The Gaussianmodel is smooth and differentiable, and can be jointly optimized withoutthe need for any templates, which provide an initial estimate of the bodysurface. Our proposed method can be integrated into existing works andenables accurate and realistic lighting while being efficient.ivPrefaceThe written content of this thesis is the original work of Luis Bolanos underthe supervision of Helge Rhodin. Human motion data was collected using theconsent form from the Behavioural Research Ethics Board (BREB), referencenumber H19-02997, with equipment provided by Luis Bolanos and HelgeRhodin.Chapter 3 is based on a publication accepted at CVPR 2024.Luis Bolanos, Shih-Yang Su, and Helge Rhodin. Gaussian ShadowCasting for Neural Characters. In The Conference on ComputerVision and Pattern Recognition (CVPR), 2024In this work, Luis Bolanos developed the idea of using Gaussians as a meansfor light calculations, implemented and experimented with all algorithmsand components, and wrote the paper. Shih-Yang Su developed the originalneural character model code base which this work extends and helped runexperiments on the cluster. Both Helge Rhodin and Shih-Yang Su helpedwith writing the paper, and offered invaluable feedback and support duringthe project development.vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . xiv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32.1 Neural Radiance Fields . . . . . . . . . . . . . . . . . . . . . 32.1.1 Scene Representation . . . . . . . . . . . . . . . . . . 42.1.2 Volumetric Rendering . . . . . . . . . . . . . . . . . . 42.2 Neural Body Models . . . . . . . . . . . . . . . . . . . . . . . 52.2.1 Skeleton Representation . . . . . . . . . . . . . . . . . 52.2.2 NeRF Input Encoding . . . . . . . . . . . . . . . . . . 52.2.3 Results and Limitations . . . . . . . . . . . . . . . . . 73 Gaussian Shadow Casting for Neural Characters . . . . . . 83.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 103.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.3.1 Deferred Neural Illumination . . . . . . . . . . . . . . 133.3.2 Gaussian Shadow Casting . . . . . . . . . . . . . . . . 143.3.3 1D Gaussian Parameters Derivation . . . . . . . . . . 173.3.4 Shadow Map Rendering . . . . . . . . . . . . . . . . . 18viTable of Contents3.4 Optimization & Training Details . . . . . . . . . . . . . . . . 193.4.1 Loss Functions & Regularizations . . . . . . . . . . . 193.4.2 Scheduled Learning . . . . . . . . . . . . . . . . . . . 213.5 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.5.1 Albedo Estimates on Training Poses . . . . . . . . . . 233.5.2 Novel-Pose Rendering with Shadows . . . . . . . . . . 233.5.3 Novel-View Rendering with Shadows . . . . . . . . . 273.5.4 Render Time Comparison . . . . . . . . . . . . . . . . 273.5.5 Relighting with Environment Maps . . . . . . . . . . 283.5.6 Ablation Study . . . . . . . . . . . . . . . . . . . . . 303.5.7 Non-Human Characters . . . . . . . . . . . . . . . . . 313.5.8 Limitations . . . . . . . . . . . . . . . . . . . . . . . . 313.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334 Gaussian Shadow Casting with Environment Maps . . . . 344.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . 344.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364.2.1 Importance Sampling . . . . . . . . . . . . . . . . . . 374.2.2 Environment Map Fitting - Fixed Neural Body . . . . 404.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434.3.1 Relighting with Importance Sampling . . . . . . . . . 434.3.2 Environment Map Fitting . . . . . . . . . . . . . . . . 434.4 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 455 Future Work and Conclusion . . . . . . . . . . . . . . . . . . 495.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 495.2 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . 505.3 Risk and Ethics . . . . . . . . . . . . . . . . . . . . . . . . . 50Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52AppendicesA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58A.1 Data Collection Consent Form . . . . . . . . . . . . . . . . . 58viiList of Tables3.1 Light intensity modulation: Our method is invariant tomoderate light intensity values. . . . . . . . . . . . . . . . . . 143.2 Novel-pose synthesis (all test frames). Our GaussianShadow Casting model achieves better PSNR scores consis-tently for novel pose renderings as it properly models thehard shadows cast by the limbs in novel positions. Existingmethods only shine in perceptual metrics (SSIM and LPIPS)as these normalize contrast and hence lessen the impact ofproper shadows and shading. . . . . . . . . . . . . . . . . . . 263.3 Novel-pose synthesis (subset of test set with observedself-casting shadows). Our Gaussian Shadow Casting ren-ders novel poses with strong hard shadows well. Our scoresdrop marginally on these hard frames compared to all framesin 3.2, while the baselines drop significantly. . . . . . . . . . . 263.4 Novel-pose in novel-view synthesis (all test frames).As our method is explicit, large changes is view direction stillresult in accurate shadowing unlike the baselines. . . . . . . 273.5 Render time. The overhead of Gaussian Shadow Casting(GSC) is minimal on DANBO with diffuse shading (DANBO+ DS) and enables casting many rays (64 for GSC-HDRi-64).By contrast, NeRF shadow casting (NeRFSC) doubles theruntime with every light source, making training prohibitivelyslow and HDRi relighting impractical. . . . . . . . . . . . . . 293.6 Ablation on Synthetic Sequence. . . . . . . . . . . . . . . 31viiiList of Figures1.1 Overview: We present a method for explicit and efficientshadow casting on implicit neural characters. Utilizing a sumof 3D anisotropic Gaussians, we are able to realistically relightthe avatars in novel environments. Our method is able to op-timize the parameters of the Gaussians jointly with the neuralbody model without any templates, and learn the unknowndirectional light source direction while disentangling shadowsfrom the texture in the neural field. All faces are blurredfor anonymity. . . . . . . . . . . . . . . . . . . . . . . . . . 12.1 NeRF Rendering. NeRF parametrizes a scene as a con-tinuous function Fϕ, mapping a query point in 3D space qisampled along a ray r with direction rd to a color c and den-sity σ. These values are volumetrically rendered to produce agiven pixel value. Figure modified from [43]. . . . . . . . . . 32.2 Skeleton representations. The template-less neural bodymodels utilize a skeleton for driving the animation. These3D poses are estimated with off-the-shelf pose estimators [22]as shown above on the MonoPerfCap dataset, a monoculardataset [52]. . . . . . . . . . . . . . . . . . . . . . . . . . . . 62.3 DANBO Encoding. We provide a copy of DANBO’s [44]Figure 2, Method Overview, for the reader’s convenience whichwe explain in Section 2.2.2. . . . . . . . . . . . . . . . . . . . 62.4 DANBO Novel Poses Results. DANBO [44] produceshigh-quality renders in novel pose synthesis but bakes in light-ing and shadows as part of the texture which can produceartifacts during novel poses as seen on the examples on theright. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7ixList of Figures3.1 Gaussian Shadow Casting (GSC): Our method is able toreconstruct 3D neural characters from a sparse set of videosin settings with strong directional illumination. GSC uses asum of Gaussians density model to cast secondary shadowrays efficiently with an analytic formula. Our method learnsto remove shadows from the neural color field, allowing us torelight in novel illuminations. All faces are blurred foranonymity. . . . . . . . . . . . . . . . . . . . . . . . . . . . 83.2 Method Overview. Our method takes as input images andposes of a person. Using a neural radiance field as a backbone[44]1, density, normals, and albedo values are volumetricallyreconstructed and rendered. We fit a sum of 3D anisotropicGaussian density model to approximate the neural densityfield and compute shadow maps using our novel anisotropicGaussian ray occlusion equations. The shadow map is com-bined with a diffuse shading pass to produce the lit image.The whole model is optimized with a photometric loss againstthe training images. Our method is able to optimize the lightdirection and ambient intensity without any initialization. Italso separates albedo from shading and shadow, allowing usto relight the model. . . . . . . . . . . . . . . . . . . . . . . . 123.3 Gaussian Density Model. The approximation to the neu-ral density field using a sum of 3D anisotropic Gaussians us-ing: a) 2 Gaussians per bone, b) 4 Gaussian per bone, and c)8 Gaussian per bone; d) is the groundtruth mesh. Note: ellip-soids are scaled to 2.5 STD of the Gaussians (99th percentile) 153.4 3D Anisotropic Gaussian Raytracing. a) A cross-sectionof a 3D anisotropic Gaussian with rays passing through theGaussian. b) The computed 1D Gaussians resulting fromour derivation in Section 3.3.2 (colored solid), compared tosampling the 3D Gaussian directly (dashed), with their exactmatch validating the correctness. c) The transmittance alongeach ray which is used as the shadow map value. . . . . . . . 163.5 Regularization on Gaussian Density Model. a) The reg-ularization function for the standard deviations of the Gaus-sians, constraining the size. b) The regularization functionfor the means of the Gaussians, keeping the Gaussians closeto the center of the bones. . . . . . . . . . . . . . . . . . . . 20xList of Figures3.6 Weight Modulation. How the weights αi(t) change through-out the training. First focusing on fitting a grey silhouetteand the Gaussian model, later transitioning to fit the RGBalongside ambient regularization. . . . . . . . . . . . . . . . . 223.7 Albedo Estimation. Our method can better separate shad-ows and lighting from training images to obtain better albedoestimates without lighting artifacts compared to a) Relight-ing4D [8] and b) RANA [18]. . . . . . . . . . . . . . . . . . . 243.8 Novel Pose Rendering. Our method can more accuratelyreproduce the shadow in novel poses compared to the base-lines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.9 Outdoors in sunlight (Real Sequence 1). Novel posesrendered using DANBO [44], NPC [45] and our method. Ourmethod has more consistent lighting and shadows whereas thebaselines suffer from large shadow changes from small posevariations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253.10 Novel-view Synthesis (Synthetic Sequence). Due to theexplicit nature of our lighting module, our models are robustto changes in view. The baselines are highly pose-dependantwith small changes in pose affecting the image drastically. . . 283.11 HDRi Relighting on MonoPerfCap [52]. Our Gaussianrelighting method can work on recordings done under uniformillumination, even on monocular datasets. . . . . . . . . . . . 293.12 HDRi Relighting on MonoPerfCap using NPC [45]as the backbone. Our Gaussian relighting method can beutilized at inference with higher quality models on data thatis uniformly lit. . . . . . . . . . . . . . . . . . . . . . . . . . 303.13 Ablation Comparisons (novel-pose). a) The model trainedwith the groundtruth light direction. b) The model trainedwhile detaching the gradients from the surface normals duringdiffuse shading. c) The model trained without diffuse shading.d) Our full model. . . . . . . . . . . . . . . . . . . . . . . . . 323.14 HDRi Relighting on the Animal Dataset. Our methodcan extend beyond human avatars as we do not necessitateany templates. . . . . . . . . . . . . . . . . . . . . . . . . . . 32xiList of Figures4.1 Environment Map Projections. These are the three mostcommon types of environment maps. The left showcases spheremapping, also known as the mirror ball, where the texturerepresents what a perfectly mirrored sphere would look likein the scene rendered through an orthographic camera. Themiddle showcases cube mapping, where the environment isprojected onto the faces of a cube. Lastly, the right show-cases an equirectangular projection, mapping a sphere onto arectangular texture space. . . . . . . . . . . . . . . . . . . . . 354.2 Equirectangular Projection. Our work utilizes environ-ment maps as a global illumination estimate using the 2Dequirectangular projection which maps a sphere onto a rect-angular texture space using Equation 4.1. . . . . . . . . . . . 364.3 Environment Maps. Examples of equirectangular environ-ment maps gathered through Poly Haven [16]. . . . . . . . . 364.4 Normals on the Equirectangular Projection Map. Theunnormalized (left) and normalized (middle) view direction/ normal values mapped onto the equirectangular projectionmap. This map can be used to compute the cosine anglebetween the environment map and a given normal to producea modulation map to attenuate the sampling process given asurface normal (right). The queried normal for obtaining thediffuse modulation map is highlighted as the red circle. . . . 404.5 Importance Sampling. In order to approximate the inte-gral of the incoming light at a given surface point, we castsecondary rays towards the environment map using an impor-tance sampling technique. The method uses the pixel value in-tensities multiplied by a diffuse modulation to obtain the finalprobability density function from which to sample rays from.The figure’s columns showcase 1) The environment map. 2)The filtered and down-sampled intensity maps with 128 ran-dom samples overlaid in red using our method. 3) The diffusemodulation map given the normal marked in red. And 4)The resulting PDF from which we sample from by uniformlysampling the PDF’s cumulative distribution function. . . . . 41xiiList of Figures4.6 Synthetic Data. Examples of the rendered images from oursynthetic data sequences. On the left is the subject beingilluminated by a uniform white light for albedo-like renders,the middle is from a sky texture where the sun was high inthe sky, and the right showcases a more complex illuminationcondition with 3 colored point sources. . . . . . . . . . . . . 424.7 Relighting Results. Results for relighting using our morerobust and accurate importance sampling as described in Sec-tion 4.2.1. NPC [45] was used as the backbone for the neuralbody model using the MonoPerfCap dataset [52]. All environ-ment maps except "tricolor_points" were obtained throughPoly Haven [16]. . . . . . . . . . . . . . . . . . . . . . . . . . 444.8 Importance Sampling Ablation. a) The importance sam-pling if the diffuse modulation is not used in either p(x) orq(x). The resulting relighting is overtly smooth with no de-tails b) The importance sampling if p(x) is kept uniform butq(x) has the diffuse modulation. The resulting relighting con-tains very sharp features. c) Our full proposed importancesampling method. The results contains details while not be-ing overtly sharp. . . . . . . . . . . . . . . . . . . . . . . . . 454.9 Environment Optimization Results. The viability studyconfirmed that we can optimize environment maps through aphotometric reconstruction loss between the training imagesand a neural body model lit using our importance samplingtechnique. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 464.10 Regularization Ablation. We compare results for an op-timized environment map without (top) and with (middle,bottom) the regularization discussed in Section 4.2.2. . . . . 464.11 Sampling Ablation in Optimization. The ablation studycomparing environment map optimization where the impor-tance sampling directly follows Section 4.2.1 (top) which leadsto areas not not being sampled and optimized if they becomedark, and one where we add a uniform constant to the in-tensity map in q(x) (middle) to mitigate the problem. Themethod where we add a constant has the optimized environ-ment map align more closely with the groundtruth (bottom). 47xiiiAcknowledgementsFirst, I want to thank my supervisor, Professor Helge Rhodin, for theirconstant support and guidance throughout my degree, and for cultivatingsuch an amazing lab environment. His constructive feedback always pushedmy work to be the best it could be, and made me a better researcher. Iam honoured to have been a part of his lab and will forever cherish thesepast two years. I would also like to thank Professor Leonid Sigal and DineshK. Pai for their valuable feedback on this thesis and being a part of mycommittee, as well as Professor Timothy Murphy for letting me be a partof his lab during my undergrad and beginning my research career. Thiswork was supported in part by an NSERC Discovery Grant awarded toHelge Rhodin, an NSERC CGS-M scholarship awarded to Luis Bolanos, andthe computational resources and services provided by Advanced ResearchComputing at The University of British ColumbiaI want to thank all of the professors in the Computer Science departmentthroughout my undergrad and graduate studies for their amazing courses, inparticular, Professor Helge Rhodin, Leonid Sigal, Kwang Moo Yi, Dinesh K.Pai, and Alla Sheffer. Their teaching ability is unparalleled and it is thanksto them that I was able to learn so much throughout my degree.I also want to thank all of my X660 lab mates who made my degreesuch a fun, loving, and engaging experience: Shih-Yang Su, Frank Yu, OliviaPerryman, James Ross, James Tang, Shashwat Suri, Andrew Evans, XingzheHe, Chunjin Song, Daniel Ajisafe, Ahmad Khajehnejad, Floria Gu, CamiloTalero, Yuni Fuchioka and Suzuran Takikawa. I want to express my biggestgratitude towards Shih-Yang Su who was, and still is, my role model forresearch. His work is undeniably top-tier, and his guidance and help wasinvaluable.Finally, I want to dedicate this thesis to my parents, brother, and sister.I would not have made it this far without their constant love and support.xivChapter 1IntroductionFigure 1.1: Overview: We present a method for explicit and efficientshadow casting on implicit neural characters. Utilizing a sum of 3Danisotropic Gaussians, we are able to realistically relight the avatars in novelenvironments. Our method is able to optimize the parameters of the Gaus-sians jointly with the neural body model without any templates, and learnthe unknown directional light source direction while disentangling shadowsfrom the texture in the neural field. All faces are blurred for anonymity.In recent decades, digital copies of people have become an integral as-pect of the entertainment industry. The visual effects (VFX) industry, forinstance, uses digital humans to age and deage actors and actresses or toapply other effects onto the copy [35]. They are also widely used to gener-ate large animated crowds in stadiums, and to even bring someone who haspassed away back onto the screen [24]. The gaming industry likewise seesa large use of digital copies from Esports titles using models of athletes asthe cohort of playable characters, to reconstructions of actors where motioncapture and facial capture technologies can be employed for realistic cut-scene interactions and dialogue. These industries often utilize sophisticatedequipment, custom-made for obtaining a digital human such as a capturedome [10, 26, 34]. These are systems which use hundreds of high-resolutioncameras all synchronized with bright and addressable LED lighting for il-1Chapter 1. Introductionlumination of the subject. The captured data must also undergo a lengthyand complex procedure to obtain a final model including mesh retopology,UV unwrapping, rigging, and grooming. Evidently, the process of humandigitalization has, for the most part, been limited to a select few groups andindividuals world-wide with expert knowledge.Recent advancements in deep learning, however, have facilitated thegeneration of digital humans and other avatars through inverse rendering[8, 19, 27, 34, 44–46]. These methods have achieved impressive renderingresults while easing the constraints of having hundreds of synchronized cam-eras. Instead, they are able to create a digital copy of a human from a setof sparse-view videos to even monocular data, allowing a wider range of in-dividuals to explore content creation with personalized 3D avatars. Whilethe reconstructed avatars are high fidelity with impressive detail quality andanimation ready, their use has been limited due to their inability to be lit re-alistically in a novel environment with the added restriction that the trainingdata should be recorded in a uniformly lit environment.In this thesis, we build upon the recent advancements in template-lessanimatable neural body models and introduce a novel lighting module to re-alistically light the character Figure 1.1. We explore two methods of lightinga model using our module. In Chapter 3, we demonstrate the ability to traina neural body model from scratch using data that was recorded outdoorswith harsh lighting conditions. We assume a single directional light source,the sun, in which our method is able to optimize the unknown directionof and further demonstrate the removal of shadows from the texture of theneural body model, achieving a more albedo-like result. In Chapter 4, weexplore an extension of our work to learn an environment map with an un-known number of lights with various colors and intensities as well as exploremethods for robust importance sampling of the environment in a less biasedway.2Chapter 2BackgroundIn this chapter, we provide a technical overview of the two major componentsthat our work builds upon to aid unfamiliar readers. We begin by coveringNeural Radiance Fields (NeRFs) [32], which form the basis of the neuralbody models that our work extend. We then summarize key features of theanimatable neural body models we use [44, 45], and their limitations whichour work attempts to mitigate.2.1 Neural Radiance FieldsNeural Radiance Fields [32] are a recent development which popularized im-plicit static scene representations. Their method uses a differentiable volumerendering approach which allows scenes to be learned from a set of N imagesof width W and height H, {It ∈ RH×W×3}Nt=1. NeRF demonstrates impres-sive high-quality reconstructions with the ability to render novel-views of ascene, a highly sought out feature for its applications.camerarrdrdFigure 2.1: NeRF Rendering. NeRF parametrizes a scene as a continuousfunction Fϕ, mapping a query point in 3D space qi sampled along a ray rwith direction rd to a color c and density σ. These values are volumetricallyrendered to produce a given pixel value. Figure modified from [43].32.1. Neural Radiance Fields2.1.1 Scene RepresentationUnlike explicit scene representations, such as meshes which are typically usedin the gaming and VFX industries, NeRFs embed the scene as parameters ofa Multi-layer Perceptron (MLP). This results in an implicit scene represen-tation, a continuous function which maps a point in 3D space to a particularcolor and density which can be volumetrically rendered. An overview of howNeRF parametrizes and renders a scene is shown in Figure 2.1. In particular,NeRF takes the form:Fϕ(Γ(q),Γ(rd)) 7→ (c, σ), with c ∈ R3, σ ∈ R,q ∈ R3, and rd ∈ R3,(2.1)with q being a query point in 3D world space, rd the view direction thatthe query is being viewed from, and Γ being the Positional Encoding [48]mapping which lifts the 3D query point and view direction into a higherdimensional space such that the MLP Fϕ can learn high frequency details.The output of Fϕ is the color c and density σ at the query location.2.1.2 Volumetric RenderingTo generate an image I′t, a pinhole camera with projection matrixP = KRt =fx 0 cx 00 fy cy 00 0 1 0[R t0 1](2.2)is used to cast primary rays rp from the camera towards the scene at everygiven pixel (u, v). In the above equation, K is the camera’s intrinsic matrixwith (fx, fy) being the focal lengths and (cx, cy) being the image centers orprincipal points. Rt is the camera’s extrinsic matrix with rotation R ∈ R3×3and translation t ∈ R3×1. The primary rays will have a ray origin ro = tand ray directionrd = R−1 (u− cx)/fx−(v − cy)/fy−1 . (2.3)Query points along each ray, Q, are sampled using a stratified approachand are volumetrically rendered following [31].Cϕ(rp) =Q∑i=1Ti (1− exp(−σiδi)) ci, where Ti = exp− i−1∑j=1σjδj (2.4)42.2. Neural Body Modelswhere δi is the distance between adjacent samples, Ti is the accumulatedtransmittance from q0 to qi, and ci, σi are the color and density outputtedby the MLP.2.2 Neural Body ModelsAs previously mentioned, NeRFs were originally developed for static scenerepresentations. They were not suitable for modelling animatable avatarsdue to the inconsistent mapping between world space coordinates and theexpected density and color. To circumvent this problem, neural body meth-ods apply an encoding which facilitates a mapping between the world co-ordinates and the character’s surface in a consistent manner. This allowsthe neural fields to be trained, as well as provide the means for controllabil-ity and animation. In the next subsections, we provide a quick overview ofhow DANBO [44], the template-less neural body model our work primarilyextends, defines this encoding and facilitates animation.2.2.1 Skeleton RepresentationDANBO [44], as well as other template-less neural body methods [45, 46],utilize a skeleton-based approach for handling animations, which furtherprovides the means by which we turn world-space queries into consistentlocal encodings that the neural field can train with.Skeletons are the industry standard way for animating characters in theVFX and gaming industries. They define deformation and poses of a char-acter as a series of weighted linear transformations. In typical computergraphics with explicit meshes, this process is called Linear Blend Skinning[25], where each vertex of a mesh is moved using a linear combination of thetransformation matrices associated to each bone. The skeletons are definedwith a rest pose Figure 2.2 (left) where all of the bones have the 4×4 identitytransformation matrix. Posing of the skeleton involves applying rotations,and possibly translations if a given bone is not connected to its parent, orlacks a parent, to each bone. The final location of a specific bone is com-puted by chaining all of its ancestors’ transformation matrices, in a processknown as forward kinematics.2.2.2 NeRF Input EncodingDANBO uses these skeletons to facilitate a canonical-like transformation be-tween world-space coordinates and pose-space. An overview of DANBO’s52.2. Neural Body ModelsFigure 2.2: Skeleton representations. The template-less neural bodymodels utilize a skeleton for driving the animation. These 3D poses are esti-mated with off-the-shelf pose estimators [22] as shown above on the MonoP-erfCap dataset, a monocular dataset [52].Figure 2.3: DANBO Encoding. We provide a copy of DANBO’s [44]Figure 2, Method Overview, for the reader’s convenience which we explainin Section 2.2.2.method is shown in Figure 2.3. Briefly, a graph neural network (GNN) isused to encode pose dependent features which form the basis for the factor-ized bone volumes. The factorized bone volumes encapsulate the body partwhich it is associated with as shown in Figure 2.3 (green). During rendering,the query location x along ray d is transformed to the local coordinates ofeach bone volume by applying the inverse of the bone’s global transforma-tion matrix (kinematic chain applied). Using the local query, the factorizedvolume vectors along each axis plane are sampled and concatenated. Theseconcatenated features vectors are present for each bone for a single query,where the aggregation network A learns weights to aggregate the featurevectors Figure 2.3 (red). The aggregated feature vector is then used as theinput to a standard NeRF MLP where the color c and density σ can bepredicted Figure 2.3 (blue).62.2. Neural Body ModelsFigure 2.4: DANBO Novel Poses Results. DANBO [44] produces high-quality renders in novel pose synthesis but bakes in lighting and shadows aspart of the texture which can produce artifacts during novel poses as seenon the examples on the right.2.2.3 Results and LimitationsDANBO achieves high quality results during novel pose and novel view syn-thesis tasks as shown in Figure 2.4. Unfortunately, the implicit nature ofbody models, not only DANBO, results in an inability to control key aspectsof the digital copy. For instance, textures cannot be edited, and clothing andhair-styles are fixed with no current way of changing them besides trainingan entirely new body model from scratch using new data. Furthermore, un-less the data used to train the body model was lit uniformly as you would ina capture dome, the resulting texture will contain lighting and shadows asbaked-in colors. Due to the large long-distance effects that shadows have, forexample the arm casting a shadow onto the torso, novel pose and novel viewsynthesis succumb to artifacts such as incorrect lighting and shadows as seenin the examples on the right in Figure 2.4. Furthermore, there existed veryfew methods of relighting the model in novel environments, which limits theuse of the body models significantly.In the following chapters, we describe possible solutions to alleviate theaforementioned problem of lighting and shadows by introducing an explicitlighting module that uses a sum of 3D anisotropic Gaussians where secondaryray tracing for shadow casting can be done in an efficient and differentiablemanner.7Chapter 3Gaussian Shadow Casting forNeural CharactersFigure 3.1: Gaussian Shadow Casting (GSC): Our method is able toreconstruct 3D neural characters from a sparse set of videos in settings withstrong directional illumination. GSC uses a sum of Gaussians density modelto cast secondary shadow rays efficiently with an analytic formula. Ourmethod learns to remove shadows from the neural color field, allowing us torelight in novel illuminations. All faces are blurred for anonymity.This chapter is based on [6], published at CVPR 2024.3.1 IntroductionIt is now possible to reconstruct animatable 3D neural avatars from videobut methods do not account for accurate lighting and shadows. They have torely on recordings that have soft uniform lighting, which precludes recordingoutdoors in direct sun light and on film sets with spotlights, and most areunable to relight characters in novel environments, limiting their applicabilityin content creation.The most recent body models [19, 23, 34, 44, 45, 49] which are basedon neural radiance fields (NeRFs) [32], approximate the light transport bycasting primary rays between the camera and the scene, sampling the un-derlying neural network dozens of times along each ray to obtain the density83.1. Introductionand color. As they do not include an illumination model, the color that theNeRF learns includes lighting, shadow, and view-dependent effects. Learninga body model in a challenging scene with a strong directional light source,such as the sun, leads to the neural field overfitting to the observed shadows.It does not generalize to novel poses, as the cast shadows are global effectswhere movement of a joint could affect the appearance of other distant ar-eas of the body. Figure 3.1 shows such setting. This is in contrast to localshading effects such as wrinkles in clothing which current body models cansuccessfully reconstruct.To cast shadows within NeRF, secondary ray tracing from the recon-structed body model to the light source is an option. Although the predom-inant NeRF formulation enables casting secondary rays without change, itcomes with a massive computational cost. For each sample along the pri-mary ray, an equal number of secondary rays would have to be computed,each with multiple samples, leading to a quadratic, instead of linear, com-plexity in the number of samples per pixel. As a result, current re-lightingmodels only support diffuse reflection [18], hard shadows that do not gen-eralize to novel poses [8], and soft dynamic shadow maps by approximatesphere tracing [53].Our core contribution is introducing an additional volumetric densityfield that is approximate but significantly speeds up dynamic shadow cast-ing while still maintaining differentiability and smoothness for gradient-basedoptimization. We introduce an anisotropic Gaussian density model and as-sociated rendering functions that approximate the fine-grained density ofthe NeRF. The Gaussians have the beneficial property that we can integratetheir density along a ray in closed form, thereby avoiding any sampling steps.Our derivation and implementation differs significantly from existing workusing Gaussians for rendering. Compared to Gaussian Splatting [20, 37, 40],we neither require an affine approximation nor back-to-front ordering. Com-pared to Gaussian density models we alleviate their sampling [37] with ananalytic integration and extend the existing analytic integration [38] to ap-ply to anisotropic Gaussians. Notably, the Gaussian density is optimizedalongside the NeRF without requiring a reference mesh such as SMPL [30];it is template-free.To further reduce runtime, we use a deferred shading approach [8] inwhich the first rendering pass computes the albedo, depth, and normal foreach pixel. The second pass casts only one secondary ray per pixel from theestimated surface point to the light source. This makes shadow computationsindependent of the number of samples in the NeRF, avoiding the mentionedquadratic complexity.93.2. Related WorkOur experiments with strong directional light and cast shadows demon-strate that our explicit lighting reduces the occurrence of artifacts in novel-view and novel-pose synthesis tasks. Figure 3.1 shows how our method isable to disentangle lighting and shadows from the avatar’s albedo givensparse-view data from only a single illumination. We take advantage ofthe dynamic aspect of the data where we can observe the same body partin multiple illuminations as the subject moves. We further demonstrate theability to optimize the unknown light directions without any user input orcareful initialization. Moreover, relighting of the neural character enables usto composite recorded motions into novel scenes realistically, making themdirectly applicable in computer graphics and entertainment industries, asdemonstrated by the HDRi re-lighting in Figure 3.1-right.3.2 Related WorkWe build on neural body models using NeRF [32], which we introduce briefly.The subsequent discussion focuses on relighting methods for 3D scenes andbody models as well as how Gaussians are used in rendering and reconstruc-tion.Neural avatars model dynamic performances by conditioning the neural ren-dering model on a template mesh driven by skeleton motion [4, 18, 23, 28,29, 49, 59, 60] or template-free by linking neural fields directly to a skeleton[27, 33, 44–46]. Our implementation uses the more flexible template-freeapproach but it is general enough to extend to any NeRF-based model.Static NeRF scene relighting approaches can be categorized by either implicit[7, 11, 39, 41, 58] or explicit [15, 51] implementations. In implicit methods,the NeRF’s MLP is extended to further output illumination data such asshadow, direct and indirect illumination or occlusion maps [11, 39, 41], ordecompose the scene into material properties such as metallicity and rough-ness which can be used in a Bidirectional Reflectance Distribution Function(BRDF) lighting model [7, 58]. These extended MLPs are conditioned attraining and test time on lighting information such as spherical harmon-ics coefficients [39], or light direction [11]. Implicit methods require largeamounts of data in both multi-view and multiple illuminations with lightinginformation known or estimated [11, 39]. Explicit methods simulate how reallight interacts with the environment which improves the generalizability tonovel illuminations but are difficult to extend to dynamic scenes or objects.These methods either utilize a secondary data structure such as proxy ge-ometries where lighting computations can be done using established methods103.2. Related Work[51], or attempt to cast the necessary secondary rays within the neural field’svolume which comes with a significant computational burden [15].Dynamic neural character relighting has been built on top of volume ren-dering methods [5, 8, 26, 36, 53, 54, 61] as well as 2D CNN based models[18]. Implicit methods again require large amounts of data which can only becaptured using light stages with known illumination [5, 26, 54, 61] or, acrossmultiple subjects for faces that are self-similar, each captured in a differentsetting with in-the-wild illumination [36]. Our model provides dynamic andexplicit shading and is most closely related to the following three methods.RANA [18] uses SMPL+D [3] to estimate the coarse geometry of a personand extract an albedo texture map using TextureNet [17]. Given a targetpose, they render person-specific neural features alongside coarse albedo andnormals from the SMPL-D model. These are passed through two CNNs torefine the albedo and normal maps. Finally, they generate a light map usingspherical harmonics and the normal map which is multiplied by the albedomap to obtain the final lit image. While spherical harmonics allow a widearray of lighting conditions to be simulated, cast shadows are not present,e.g., an arm casting a shadow on the body. Our work implements a Gaussiandensity model [37, 38] to facilitate fast and efficient secondary ray tracing tocompute these cast shadows.Likewise, Relighting4D [8] uses SMPL [30] to condition a 4D neural fieldof latent features which are trilinearly interpolated based on the nearby ver-tices to the query location. The latent features are passed through an MLP toobtain geometry, occlusion, and reflectance properties which are fed througha BRDF to get the final lit image. It is able to estimate the light probe, andat inference time, be able to switch the light probe to a new illumination.However, Relighting4D was not designed to work with hard shadows in novelposes, which is the focus point of our work.Finally, Xu et al. [53] utilize a signed distance field (SDF) based ap-proach to learn a neural human avatar which utilizes SMPL-based inverseLinear Blender Skinning (LBS) and a displacement field to obtain canoni-cal features. They utilize Hierarchical Distance Queries (HDQ) to computeminimum distances from world space to surface locations and perform spheretracing to obtain material and surface properties for each camera ray. Theyfurther take advantage of HDQ through the SDF by computing soft visibilitymaps towards a learned light probe. While HDQ allows for fast occlusionchecks, their solution focuses on soft approximate shadows whereas our workenables hard shadow casting.Gaussians have been used in rendering applications as differentiable methodsfor computing visibility and occlusions [37, 38, 42], as components of envi-113.2. Related WorkFigure 3.2: Method Overview. Our method takes as input images andposes of a person. Using a neural radiance field as a backbone [44]1, density,normals, and albedo values are volumetrically reconstructed and rendered.We fit a sum of 3D anisotropic Gaussian density model to approximate theneural density field and compute shadow maps using our novel anisotropicGaussian ray occlusion equations. The shadow map is combined with adiffuse shading pass to produce the lit image. The whole model is optimizedwith a photometric loss against the training images. Our method is able tooptimize the light direction and ambient intensity without any initialization.It also separates albedo from shading and shadow, allowing us to relight themodel.ronment maps [56], or as a means to improve rendering efficiency for neuralscenes [20]. Most methods are limited to spherical Gaussians [37, 38], whileGaussian Splatting uses an affine approximation that is only accurate whenmany small Gaussians are used [20], and Sridhar et al. use an approxima-tion by perspective projection of ellipsoids [40]. Our work extends Rhodin etal. [37, 38] to use anisotropic Gaussians, without introducing any approxima-tion, and tailors the analytic formulas and implementation towards shadowcasting.123.3. Method3.3 MethodOur method reconstructs a neural character from a set of N images of widthW and heightH, {It ∈ RH×W×3}Nt=1, and corresponding character poses θt ∈RJ×4×4. The pose is represented as a skeleton with one 4× 4 local-to-worldtransformation matrix for each of the J joints. Figure 3.2 gives an overview ofour method. A key element of our design is a deferred illumination model [47]that separates the rendering into computing albedo, a ∈ R3, surface normal,nˆ ∈ R3, and depth, d ∈ R, in a first pass and subsequently adding shadingand shadow, s ∈ [0, 1], in a second pass. Our key contribution is the closedform formula for the shadow s.3.3.1 Deferred Neural IlluminationOur volumetric body model is optimized on a reconstruction objective, LRGBthat minimizes the squared difference between the input images It and therendering of the model. We test our method using DANBO [44]. It outputsa color and density for samples x along the primary view rays. These aresubsequently integrated to compute a color, which we interpret as the albedo,a. The illuminated color for a given pixel of the reconstructed image, cˆ, iscomputed by a Lambertian reflectance model,cˆ = a(θt)(Lˆamb + s(θt)Lcol(Lˆdir · nˆ(θt))). (3.1)This diffuse shading model illuminates the entire body with an ambient lightLˆamb and a directional light with color Lcol. The directional light intensityis attenuated by the cast shadows s and the cosine angle between the surfacenormal nˆ and light direction Lˆdir.Shading extensions. The benefit of the deferred rendering approach isthat it lets us compute lighting information only once for each pixel, asopposed to at every sample location of the volumetric ray tracing leadingto significantly faster computation. To be applicable, we extend DANBOto yield surface normals nˆ and depth d for a given view ray. The formerwe attain by switching the density formulation to a signed distance functionwith an Eikonal loss. The normal is then readily estimated by differentiatingthe distance with respect to the original query location x as in [55]. Wecompute d likewise to albedo a by integrating the sample’s x positions alongthe ray, weighted by their density and transmittance. Furthermore, we fix theintensity of the directional light to white with a magnitude of 1.5. Without133.3. Methodfixing the directional light intensity, the equation would be over parametrizedand lead to ambiguities. Our model is invariant to light intensities between1.0 and 2.0, as shown by Table 3.1.Table 3.1: Light intensity modulation: Our method is invariant to mod-erate light intensity values.Novel View & PoseLight Intensity: 1.0 1.25 1.50 2.0 5.0PSNR ↑ 24.46 24.52 24.37 24.00 18.98LPIPS ↓ 0.165 0.164 0.165 0.163 0.1853.3.2 Gaussian Shadow CastingFor modeling shadows more efficiently, we represent the body shape with aset of Gaussians rigidly attached to the skeleton model. The relative posi-tions, orientation, and size of the Gaussians are optimized to approximate thedensity of the neural field and to allow for a fast, efficient, and closed-formsolution for integration along a ray (occlusion checking). Our model extendsprevious work [37, 38] by using anisotropic Gaussians (variable scale and ro-tation along each axis) and avoids the need for sampling during integrationas in NeRF.Anisotropic Gaussian body model. We define the anisotropic Gaus-sian density model as the matrix G ∈ RJ×K×13, with K being the numberof Gaussians per joint, typically ∼ 8, and the columns representing the 3Dmean (µx, µy, µz), the axis aligned standard deviations (σx, σy, σz), the rota-tion defined using the 6 DOF representation (R0,0, R0,1, R0,2, R1,0, R1,1, R1,2)[62], and density (C). Figure 3.3 gives examples with varying numbers ofGaussians.The 3D density function,G(x), defines the density of the Gaussian modelat the query location x in world space. We define the density function of asingle 3D anisotropic Gaussian asGi(x) = C exp[−0.5 ((µ− x)T )Σ−1(µ− x))], (3.2)where the precision matrix Σ−1 = RTDR and R is the rotation matrix com-puted from the 6 DOF representation and D = diagonal(1/σ2x, 1/σ2y , 1/σ2z).The density of the entire Gaussian model is the sum of the density of each.The query location is transformed to the local space of the given Gaussian’s143.3. Methoda. b. c. d.Figure 3.3: Gaussian Density Model. The approximation to the neuraldensity field using a sum of 3D anisotropic Gaussians using: a) 2 Gaussiansper bone, b) 4 Gaussian per bone, and c) 8 Gaussian per bone; d) is thegroundtruth mesh. Note: ellipsoids are scaled to 2.5 STD of the Gaussians(99th percentile)joint j at time-step t using the world-to-local transformation matrix θ−1t,j ,rigidly attaching the Gaussians to the underlying skeleton and facilitatinganimation,G(x) =J×K∑i=0Gi(θ−1t,j x). (3.3)We jointly fit the parameters of the Gaussian density model to the neural fieldby minimizing LgDensity, the L2 error between the density function G(x) andthe target neural density field at query location x. We detach the gradientsof the neural density field to optimize the Gaussians, fitting the Gaussiansto the neural field and not the other way around.Gaussian Ray Tracing. Figure 3.4 shows how casting a ray, r, with rayorigin ro ∈ R3×1 and direction rd ∈ R3×1, through a 3D anisotropic Gaussianresults in a 1D Gaussian density along the ray. Through the Gaussian bodymodel, this equates to a sum of 1D Gaussians for which analytic integralscan be computed. The amount of occlusion these rays experience is equal tothe sum of the integrals of each of the 1D Gaussians across the rays. Thetransmittance value, T , used as the shadow map value, s, is the exponentialof the negative integral from the start of the ray, t = 0, to the length of theray, t = l,s = Tr = exp[−J×K∑i=0∫ l0Gri]. (3.4)153.3. Methoda. b. c.densitytransmittancexyt tt=0Figure 3.4: 3D Anisotropic Gaussian Raytracing. a) A cross-sectionof a 3D anisotropic Gaussian with rays passing through the Gaussian. b)The computed 1D Gaussians resulting from our derivation in Section 3.3.2(colored solid), compared to sampling the 3D Gaussian directly (dashed),with their exact match validating the correctness. c) The transmittancealong each ray which is used as the shadow map value.Gri is the 1D Gaussian created by the ray, r, going through the 3D anisotropicGaussian, Gi with mean µ ∈ R3×1 and precision matrix Σ−1 ∈ R3×3. The1D Gaussian’s density function takes the formGrsi = C¯ · exp(−(µ¯− x)22σ¯2), (3.5)whereC¯ = C exp(−0.5((µ− ro)TΣ−1(µ− ro)− µ¯2σ¯2)),µ¯ =rTdΣ−1(µ− ro)rTdΣ−1rd, andσ¯ =√1rTdΣ−1rd. (3.6)This formula is more complex than in [38], as it now accounts for anisotropicGaussians with an arbitrary covariance instead of isotropic Gaussians. Thefull derivation is shown in the next subsection. The comparison to samplingthe 3D Gaussian in Figure 3.4 validates their correctness. It also lets us com-pute the cumulative density function (CDF) analytically, thereby avoidingthe sampling in classical NeRFs,∫ x0Grsi = C¯ · 0.5 ·(1 + erf(x− µ¯σ¯√2)). (3.7)163.3. MethodTogether with Equation 3.4, this integral computes the shadow s when ap-plied to the secondary ray with origin rso, as the point on the subject’s surfacecomputed from the depth map d, and direction rsd towards the light.3.3.3 1D Gaussian Parameters DerivationHere we present the full derivation of our equations to parametrize the 1DGaussians formed by a ray passing through a 3D anisotropic Gaussian. Ourgoal is to derive a 1D function Gr(t) that represents the density along a rayr that we can integrate to acquire the cast shadow. We start our derivationfrom the 3D anisotropic GaussianG(x) that we use to approximate the bodydensity field,G(x) = C exp [−0.5(µ− x)TΣ−1(µ− x)]= C exp [−0.5(µTΣ−1µ− 2µTΣ−1x+ xTΣ−1x)] . (3.8)We can infer the density along the ray, Gr(t), by parameterizing the 3Dpositions along the ray by the distance t from the origin ro, with x = ro+t·rd,and substituting it into Equation 3.8,Gr(t)= Cˆ exp [−0.5(t2rTdΣ−1rd − 2trTdΣ−1(ro − µ))]= Cˆ exp [− 0.5rTdΣ−1rd(t2 − 2trTdΣ−1(ro − µ)rTdΣ−1rd)], (3.9)where rd is the ray direction, and Cˆ consists of the terms that are constantswith respect to the ray distance t, separated by the equality exp(a + b) =exp(a) exp(b),Cˆ = C exp [−0.5(µ− ro)TΣ−1(µ− ro)] . (3.10)We then reorganize Equation 3.9 by substituting σˆ = 1√rTd Σ−1rand µ¯ =rTd Σ−1(ro−µ)rTd Σ−1rd,Gr(t) = Cˆ exp [− 0.5(t2 − 2tµ¯)σ¯2]= Cˆ exp [− 0.5(t2 − 2tµ¯+ µ¯2)− µ¯2σ¯2]= Cˆ exp [− 0.5(t− µ¯)2 − µ¯2σ¯2]= C¯ exp[−(µ¯− t)22σ¯2], (3.11)173.3. Methodwhere, again, C¯ absorbs the terms that are constant to t,C¯ = C exp[−0.5((µ− ro)TΣ−1(µ− ro)− µ¯2σ¯2)], (3.12)and we arrive at Gr(t) that assumes the form of a 1D Gaussian densityfunction with mean µ¯, std σ¯, and scaling factor C¯. As the integral througha Gaussian can be computed in closed form through the error function, thisenables analytical integration of the density along the ray, which in turnenables our method to cast shadows efficiently.3.3.4 Shadow Map RenderingAs previously mentioned in 3.3.1, we apply lighting to the rendered NeRFoutputs as a secondary pass. First, a depth map is computed within NeRFand surface normals are estimated by differentiating the density with respectto the original query location x as in [55]. Our implementation uses an SDFbased density to improve the normal quality, but it is not a requirement.Following equation 3.1, we render a diffuse shading pass as the cosineangle between the surface normal nˆ and the optimized light direction Lˆdirclamped to [0, 1]. This is combined with the cast shadow map which iscomputed for each pixel containing the subject. For each of those pixels, wedetermine the secondary ray origin rso, as the point on the subject’s surfacecomputed from the previously estimated depth map. The secondary raydirection is always towards the optimized light direction Lˆdir for the case ofa directional light source. The 1D Gaussians along the secondary ray arecomputed using Equation 3.6, and the shadow map value is computed usingequations 3.4 and 3.7.The diffuse shading augments the Gaussian cast shadows as it is ableto more faithfully reproduce the light intensity fall-off as the light directionbecomes more incident with the surface. Furthermore, the Gaussians are acoarse approximation of the body model which would lack high frequencydetails such as wrinkles and small extremities (i.e. nose, ears, and fingers).The Gaussian cast shadows focus more on the far-reaching, low frequencyeffects.183.4. Optimization & Training Details3.4 Optimization & Training Details3.4.1 Loss Functions & RegularizationsOur primary objective is the accurate reconstruction of the neural characterrenders Iˆ and the training images I. We use a standard photometric recon-struction loss between the pixel color values in the training image, c, andreconstruction, cˆ,LRGB = |cˆ− c|. (3.13)We augment this by also using a mask loss that operates between theintegrated accumulation, ρˆ, and the foreground mask, ρ,Lmask = |ρˆ− ρ|. (3.14)Our final photometric loss is one we dub the grey loss. Its objective isto initialize the RGB head of the NeRF to output a light grey value suchthat when the standard RGB loss starts to have an influence on the training,it is not prone to getting stuck in a local minimum with shadows alreadylearnt caused by an initialization resulting in darker color values. It alsoprovides enough time for the light direction to optimize and fit before theNeRF overfits to the shadows as we interpolate between the grey loss andthe RGB lossLgrey = |cˆ− 0.75|. (3.15)To fit the Gaussians, we introduce a Gaussian Density loss which mini-mizes the squared distance between the Gaussian density function at a givenquery location, G(x), and the density head of the NeRF at the same querylocation, D(x),LgDensity = ||G(x)−D(x)||2. (3.16)We regularize our Gaussians by supervising their mean and standard de-viations. The standard deviation regularization limits the size of the Gaus-sians to approximately be within 2.5 and 50 centimeters, while the meanregularization prevents Gaussians from drifting too far from the bone cen-ters, b,LgSigma ={2e−5σ σ ≤ 0.02100(σ − 0.02)4 + 0.001 σ > 0.02 (3.17)193.4. Optimization & Training Detailsa. b.losslossGaussian σ μ distance (m)Figure 3.5: Regularization on Gaussian Density Model. a) The regu-larization function for the standard deviations of the Gaussians, constrainingthe size. b) The regularization function for the means of the Gaussians, keep-ing the Gaussians close to the center of the bones.LgMean = (100(µ− b)4 + 1)14 − 1. (3.18)These loss functions are visualized in Figure 3.5.We also regularize the ambient intensity, Lˆamb, to be somewhat darkvalues to prevent the model from setting a bright ambient value and learningall of the shadows as part of the NeRF color,Lamb = ||Lˆamb − 0.1||2. (3.19)We adopt the SDF-based density field from VolSDF [55] for improvednormals, and regularize the SDF network using an Eikonal loss [14] LEikonalto predict proper level sets,LEikonal = ||nˆs(x)− 1||22 (3.20)and use a curvature loss, LCurvature, to smoothen out the geometry by mini-mizing the difference between neighbouring normals,LCurvature = ||nˆs(x)− nˆs(x+ ϵ)||2, (3.21)with ϵ a small random perturbation.Our total loss is the sum of all of these losses and regularization terms,each with a weighing function, αi(t), for loss term i and training iteration203.4. Optimization & Training Detailst. Training takes around 20 hours on an NVIDIA RTX 3090, similar toRelighting4D [8].3.4.2 Scheduled LearningOur scheduled learning can be split into 3 segments. Denoting a change inthe weights for each of the loss terms throughout the training using linearinterpolation.Segment 1: Density Fitting ∼ 1k Iterations. In this step, the maingoal for the model is to train the neural field’s density to fit the silhouette ofthe character. It begins with high weights for only Lmask and Lgrey alongsidethe regularizers for curvature, LCurvature, and Eikonal constraints LEikonal.We need this first stage as our shading computations rely on accurate depthmaps, normal maps and accurate Gaussian fits, which the latter requires anaccurate density field to fit to. Training the RGB head directly from thestart results in many artifacts that the network cannot recover from due tothe deferred nature of our shading computations.Segment 2: Gaussian Density Model Fitting ∼ 4k Iter. This seg-ment marks the addition of the Gaussian density loss, LgDensity, and itsregularizers, LgMean and LgSigma. At which point the parameters of theGaussians, G, are optimized to fit to the pretrained NeRF’s density.Segment 3: Light Fitting & RGB Fitting This step switches fromusing the grey loss to the RGB loss. Interpolation between the two ensures asmooth transition between purely optimizing for the silhouette and our targetcolor reconstruction. In our experiments, 1k iterations were sufficient to fullyoptimize the light direction, at which point the diffuse and Gaussian shadowcomputation is fairly accurate, allowing the neural color field to learn colorwithout shadows, more closely resembling the albedo, see the SupplementalVideo.Weight Modulation: As previously mentioned, our total loss is the sumof all of our loss terms each with a weighing function αi(t) which modulatesthe weight during training to allow the previously mentioned stages to trainproperly. We plot the value of each of the weighing functions over the trainingiterations in Figure 3.6. They are linearly interpolating between two valuesover a number of iterations with a hold-off period.213.5. ResultsGreyMaskRGBCurvatureAmbientGaussian DensityGaussian SigmaGaussian Mean Figure 3.6: Weight Modulation. How the weights αi(t) change throughoutthe training. First focusing on fitting a grey silhouette and the Gaussianmodel, later transitioning to fit the RGB alongside ambient regularization.3.5 ResultsWe evaluate our method on synthetic sequences, as done in prior work [18].However, this does not test performance in real world conditions. Hence, wecaptured a new dataset in direct sunlight and compare to the most closelyrelated baselines, showing significantly improved relightable body models.Synthetic datasets. We test our model on the RANA dataset [18], andfurther create our own synthetic sequence by obtaining a textured mesh of asubject with a 3D full body scanner (VITUS 3D Body Scanner). A Blender[9] cloth simulation was applied to a shirt over the scan and the character wasautomatically rigged and animated using Mixamo [2]. We use the ‘swing-dance’ animation as the driving motion as it contains a variety of poses fromall body angles. Four (3 train, 1 test) cameras are placed around the subjectat 90 degrees from each other. A directional light source illuminates thescene with a slight ambient contribution such that the shadowed areas werenot fully black. The animations are rendered using the Cycles render engine.In addition, ground-truth pose and segmentation masks are exported. Wesplit the dataset into 600/114 images for the train/test sets, with the testset including 57 novel view images and 57 novel poses, out of which 15 havea strong hard cast shadows that we test separately.223.5. ResultsOutdoor sunlight dataset. We recorded two sequences of real humanmotion in an outdoor scene during a sunny day. This setting has largely beenunexplored in neural body models and few publicly available datasets areavailable. We capture the data using 3 cameras (Canon EOS R8, Canon EOS70D, iPhone12) and obtain SMPL estimates using EasyMocap [1, 12, 13].Segmentation masks are obtained using the Segment Anything Model (SAM)[21]. We divide the frames into 600/200 images for the train/test splits, usingall three cameras for training.Baselines We evaluate our method using the hard illumination datasetagainst Relighting4D [8]. Due to code being unavailable, we were not able tocompare against RANA [18] and Xu et al. [53] using our datasets. We insteadquantitatively compare albedo estimates between our model and RANA ontheir dataset as these results were kindly made available. We further compareour work with other template-less neural body models [44, 45], highlightingthe drawback when not explicitly modeling lighting. DANBO [44] is ourneural field backbone. NPC [45] forms the current state-of-the-art template-less neural character model.3.5.1 Albedo Estimates on Training PosesWe ran the official implementation of Relighting4D (R4D) [8] on our outdoordataset, providing the same segmentation masks and SMPL body model asto our method (our method only uses the skeleton, not the surface). As thefirst stage of R4D is NeuralBody [34] which does not take shadowing into ac-count, it produces dark floaters in the space to approximate the hard shadow,hindering their subsequent relighting module from estimating shadow andshading correctly as seen in Figure 3.7a.To compare against RANA [18], we run our method on subject 1 of theirsynthetic dataset. Even with the dataset being monocular (light comes fromthe same direction relative to the camera), our method was able to accuratelyestimate the light direction (error of 9.9 degrees) and obtain albedo estimateswith fewer lighting artifacts compared to RANA, see the back of the left legin Figure 3.7b.3.5.2 Novel-Pose Rendering with ShadowsIn this setting, the camera and illumination are unchanged and only novel-poses are tested. These poses created new shadow casts that resulted inlarge appearance changes distant to the changed body part. As expected,233.5. ResultsFigure 3.7: Albedo Estimation. Our method can better separate shadowsand lighting from training images to obtain better albedo estimates withoutlighting artifacts compared to a) Relighting4D [8] and b) RANA [18].Figure 3.8: Novel Pose Rendering. Our method can more accuratelyreproduce the shadow in novel poses compared to the baselines.existing methods (NPC, DANBO) overfit the training poses and the shadowscreated in the novel poses are highly inaccurate. Figure 3.8 shows how forframes that had body parts casting shadows on other regions, our methodproduced more accurate shadows. Table 3.2 quantifies the gains across novel243.5. ResultsFigure 3.9: Outdoors in sunlight (Real Sequence 1). Novel posesrendered using DANBO [44], NPC [45] and our method. Our method hasmore consistent lighting and shadows whereas the baselines suffer from largeshadow changes from small pose variations.253.5. Resultsposes and Table 3.3 across the subset of the novel poses that has a shadowcast across the body.Table 3.2: Novel-pose synthesis (all test frames). Our GaussianShadow Casting model achieves better PSNR scores consistently for novelpose renderings as it properly models the hard shadows cast by the limbs innovel positions. Existing methods only shine in perceptual metrics (SSIMand LPIPS) as these normalize contrast and hence lessen the impact ofproper shadows and shading.Synthetic (N = 57) Real S1 (N = 200) Real S2 (N = 200) AveragePSNR↑SSIM↑LPIPS↓ PSNR SSIM LPIPS PSNR SSIM LPIPS PSNR SSIM LPIPSDANBO [44] 17.52 0.756 0.195 16.57 0.599 0.328 17.69 0.588 0.325 17.26 0.648 0.283NPC [45] 17.57 0.758 0.188 16.33 0.590 0.334 17.47 0.575 0.328 17.12 0.641 0.283Ours 22.04 0.829 0.166 17.57 0.592 0.356 18.29 0.577 0.351 19.30 0.666 0.291Table 3.3: Novel-pose synthesis (subset of test set with observedself-casting shadows). Our Gaussian Shadow Casting renders novel poseswith strong hard shadows well. Our scores drop marginally on these hardframes compared to all frames in 3.2, while the baselines drop significantly.Synthetic (N = 15) Real S1 (N = 41) Real S2 (N = 36) AveragePSNR↑SSIM↑LPIPS↓ PSNR SSIM LPIPS PSNR SSIM LPIPS PSNR SSIM LPIPSDANBO [44] 17.78 0.740 0.209 15.11 0.559 0.354 16.55 0.547 0.353 16.48 0.615 0.305NPC [45] 17.81 0.741 0.201 14.88 0.553 0.357 16.51 0.538 0.355 16.40 0.611 0.304Ours 22.13 0.821 0.175 16.88 0.572 0.365 17.40 0.544 0.371 18.81 0.646 0.303For the synthetic sequence, improvements were consistent across all threemetrics. In the real outdoor sequence, all methods attain a lower qualityas the cameras are spaced further apart and the segmentation masks and3D input pose, estimated with off-the-shelf 2D pose detection and liftingmethods, are less reliable. Nevertheless, Figure 3.8 shows how our modelcan accurately optimize the light direction and predict realistic shadows,including on the ground. To map shadows to the ground, we estimate theground plane from the reconstructed foot positions and cast the Gaussianshadow on it by modulating the static background with the ground shadowmap.Table 3.2 and Table 3.3 show that our method consistently improves thePSNR while the perceptual metrics SSIM [50] improves only in one and thebaselines perform better for LPIPS [57]. This lower performance in percep-tual metrics is expected because these metrics normalize for brightness andcontrast differences, thereby lessening the importance of producing proper263.5. Resultsshading and shadowing. In addition, the texture and geometry detail of ourmethod is slightly lower, which we attribute to the separation into shadingand albedo imposing additional constraints, thereby leading to slightly lessdetailed reconstructions. Additional novel-pose rendering examples for theoutdoor sequence is shown in Figure 3.9.3.5.3 Novel-View Rendering with ShadowsWe also test the ability to render novel views using our method. We findthat due to the explicit nature of our shading computations, the lighting andshadows are still accurate. The baselines fail to produce accurate illumina-tions and are much more susceptible to small pose variations leading to largechanges, as seen in Figure 3.10. In contrast to the training cameras, thenovel view is backlit with large parts of the trunk in shadow. Consequently,the quantitative analysis in Table 3.4 demonstrates even larger performancegains than the novel pose evaluation in the main document. The resultfurther highlights the importance of our explicit shadow-casting method.Table 3.4: Novel-pose in novel-view synthesis (all test frames). Asour method is explicit, large changes is view direction still result in accurateshadowing unlike the baselines.Novel View Synthetic (N = 57)PSNR↑ SSIM↑ LPIPS↓DANBO 18.85 0.773 0.179NPC 20.64 0.823 0.157Ours 27.19 0.873 0.1533.5.4 Render Time ComparisonTable 3.5 lists the render time of our baseline compared to our full model.Casting shadows with GSC has minimal overhead (0.3s for one ray, only2% of the entire render time), enabling efficient training alongside NeRFoptimization. Casting a shadow with the NeRF baseline requires processingtwice the number of samples by the NeRF. The deferred shading modelcreates one occlusion ray and each of these secondary rays requires a similarnumber of samples as for the primary ray. Already with a single light source,this increases runtime by 25%, a ten-fold difference to GSC.273.5. ResultsFigure 3.10: Novel-view Synthesis (Synthetic Sequence). Due to theexplicit nature of our lighting module, our models are robust to changes inview. The baselines are highly pose-dependant with small changes in poseaffecting the image drastically.3.5.5 Relighting with Environment MapsThe shadow computation not only benefits training time but also enablescomputing irradiance maps for environment maps. Figure 3.1 shows relight-ing with two different HDRi maps (obtained from Poly Haven [16]) by casting64 secondary light rays towards the environment map for each pixel. One ofthose light rays is importance sampled, going towards the brightest regionin the HDRi. In both cases, the bright sun casts a strong shadow while thecolored light from the environment leads to natural shading that matchesthe character with the environment. This enables placing the reconstructedcharacters into new environments and giving them a natural and consistentlook with respect to the rest of the scene while still containing cast shadows.Further exploration into using environment maps as training targets and forrelighting is described in Chapter 4.Our Gaussian Shadow Casting method can also be used as a standalonerelighting tool for datasets captured under uniform illuminations. These283.5. ResultsTable 3.5: Render time. The overhead of Gaussian Shadow Casting (GSC)is minimal on DANBO with diffuse shading (DANBO + DS) and enablescasting many rays (64 for GSC-HDRi-64). By contrast, NeRF shadow casting(NeRFSC) doubles the runtime with every light source, making trainingprohibitively slow and HDRi relighting impractical.Method render time [s]DANBO + DS 17.13DANBO + DS + GSC 17.47DANBO + DS + NeRFSC 21.4DANBO + DS + GSC-HDRi-8 20.70DANBO + DS + GSC-HDRi-16 23.57DANBO + DS + GSC-HDRi-32 29.49DANBO + DS + GSC-HDRi-64 41.22datasets have very few lighting and shading effects baked-in to the texturewhich allows for the direct interpretation of the neural color field as thealbedo. We can simply fit the Gaussian density model and relight the learnedavatars with HDRis. We test this paradigm on a monocular sequence fromthe MonoPerfCap dataset [52]. See Figure 3.11 for the results.Figure 3.11: HDRi Relighting on MonoPerfCap [52]. Our Gaussianrelighting method can work on recordings done under uniform illumination,even on monocular datasets.Furthermore, our method is plug-and-play with other neural body modelsdue to the deferred rendering approach. It can be used with higher qualityvolumetric neural models without degraded quality when training on uni-formly lit data, and using GSC soley for relighting as shown in Figure 3.12.In those renderings, NPC [45], was used as the neural field backbone andillumination was applied using our deferred shading approach. NPC pro-293.5. Resultsvides significantly improved geometry with high quality textures comparedto DANBO [44], which can be readily seen when comparing Figure 3.11 andFigure 3.12. Unfortunately, using NPC as the NeRF backbone during jointlight, Gaussian, and neural field optimization proved to be a challenge. Thepoint-based approach of NPC requires K-Nearest Neighbours (KNN) sam-pling for encoding the input to the NeRF resulting in very blobby/sphericalartifacts in the normals. These artifacts would not dissipate until muchlater in the training yet impact the lighting module significantly as the dif-fuse shading computation relies on the normals. Furthermore, the densityof the neural body model is much slower to converge to a solid shape whichaffects the fitting of the Gaussians.Figure 3.12: HDRi Relighting on MonoPerfCap using NPC [45] asthe backbone. Our Gaussian relighting method can be utilized at inferencewith higher quality models on data that is uniformly lit.3.5.6 Ablation StudyWe test a variety of implementation details in our model,including usingonly diffuse shading on top of DANBO (DANBO + Diff. Shading), onlyusing the Gaussians to cast shadows (Ours w/o Diff. Shading), providingground truth lighting (Ours-GT Light), and detaching the normals priorto the diffuse shading (Ours-Detached Normals). The results of which canbe seen in Table 3.6, which shows that each of our contributions improvesreconstruction quality at test time.Diffuse Shading. Shadowing alone does not account for accurate shadingbased on how incident the light hits the surface. Moreover, the Gaussianscast long-range shadows, but their smooth and low-resolution approximationto the NeRF’s density prevents them from representing finer details such as303.5. ResultsTable 3.6: Ablation on Synthetic Sequence.Training Novel Pose Novel ViewPSNR↑SSIM↑LPIPS↓ PSNR SSIM LPIPS PSNR SSIM LPIPSDANBO 27.66 0.913 0.133 17.52 0.756 0.195 18.85 0.773 0.179DANBO + Diff. Shading 24.82 0.860 0.187 18.60 0.785 0.208 24.40 0.875 0.167Ours w/o Diff. Shading 27.17 0.879 0.161 21.22 0.802 0.178 25.08 0.867 0.163Ours-Detached Normals 25.90 0.863 0.182 20.43 0.814 0.192 26.18 0.893 0.158Ours-GT Light 25.22 0.861 0.180 21.23 0.830 0.184 26.68 0.895 0.155Ours 26.60 0.876 0.165 22.30 0.827 0.176 27.32 0.882 0.154small extremities (nose, fingers). As a result, finer shading details are missedas seen in Figure 3.13c. On the other hand, using only shading (DANBO +Diff. Shading) already reduces texture detail quality as seen by the trainingscores, but achieves improved performance in novel pose and novel viewconditions. However, the missing cast shadows play the most significant rolein improving test scores.Detached Normals. We compare results between a model where networkgradients could, Figure 3.13d, and could not, Figure 3.13b, backpropagatethrough the surface normals used in the diffuse shading to see whether ornot artifacts in the shading would smoothen out the geometry. We find thatthe surface is indeed affected by the gradients backpropagating through thediffuse computation and observe a smoother geometry reconstruction.GT Light. Our method is able to fit the direction of the light source and theambient intensity with little user input. We observe accurate light recoverywhen the light is initialized randomly, e.g. when coming from the backthe angle error is only 1.36 degrees on our synthetic sequence. We foundproviding the ground truth light direction did not improve results and mayhave hindered the model due to the added constraints, Figure 3.13a.3.5.7 Non-Human CharactersDue to the template-less nature of our implementation, we are able to learnbodies with Gaussian density models for non-human characters. We testthis using the Animal dataset [27] as seen in Figure 3.14 during an HDRirelighting task.3.5.8 LimitationsThe Gaussian cast shadows model long-range effects, such as the arm castinga shadow on the leg but the smooth Gaussians lack high frequency details.313.5. ResultsFigure 3.13: Ablation Comparisons (novel-pose). a) The model trainedwith the groundtruth light direction. b) The model trained while detachingthe gradients from the surface normals during diffuse shading. c) The modeltrained without diffuse shading. d) Our full model.Figure 3.14: HDRi Relighting on the Animal Dataset. Our methodcan extend beyond human avatars as we do not necessitate any templates.This is a minor drawback since the diffuse shading already faithfully repro-duces the light intensity fall-off as the light direction becomes more incidentwith the surface and therefore shades the back side of small extremities (i.e.nose and fingers) well. A future extension could be to integrate mid-scaleeffects with screen-space ambient occlusion and shading.Moreover, we noticed that disentangling color into shading and albedo,compared to the original DANBO backbone, leads to slightly lower imagereconstruction metrics when shading effects are minimal. We attribute thisto the additional constraints that are imposed on the model. However, theoverall performance in novel light conditions is still improved significantlyby our model.323.6. Conclusion3.6 ConclusionWe enabled the 3D reconstruction of human motions in uncontrolled envi-ronments by a Gaussian shadow model that applies to dynamic scenes andis differentiable for iterative refinement. The reconstructed characters sup-port reposing and relighting in novel environments. They are equipped withglobal shadow computation, diffuse shading, geometric reconstruction, and aconsistent albedo, much like hand-crafted computer graphics models wouldprovide.33Chapter 4Gaussian Shadow Casting withEnvironment Maps4.1 IntroductionIn the previous chapter, we experimented with jointly fitting the neural bodymodel alongside the Gaussian density model and lighting. This resultedin improved albedo estimates on the neural color field as the lighting andshadows were explicitly computed in a separate pass. A major limitation ofthat work is that it assumed the training data was illuminated by a singledirectional light source. In practice, this condition is rare beyond outdoordaylight recordings. In this chapter, we explore the beginnings of relaxingthat assumption and directly using environment maps as the target lightsource.Environment maps, also known as high dynamic range images (HDRis),are a popular solution in the computer graphics domain as an approximationfor global illumination, and as a method for reflection mapping. These mapsstore omni-directional illumination values, which is to say that, for a givendirection, a directional light source exists with color and intensity denotedby the pixel value in the map. These maps are rendered at infinity whichemit light onto the 3D scene, meaning that only rotation changes affect theobserved illumination while changes in position do not. Environment mapsare high dynamic range images typically using 32-bit floats for each colorchannel such that the details in both the dark regions and brighter regionscan be preserved. They have advantages over simply using a lot of point ordirectional light sources as they can represent much more complex illumina-tion conditions, while being easy to visualize and be captured from real-worlddata. There exist a variety of mappings to store these as texture files in-cluding sphere mapping or mirror balls, cube mapping, and equirectangularprojection maps. These are shown in Figure 4.1.We choose to use the equirectangular projection maps as these are themost common type of environment map shared online. These are rectangular344.1. Introductionsphere map cube map equirectangular mapFigure 4.1: Environment Map Projections. These are the three mostcommon types of environment maps. The left showcases sphere mapping,also known as the mirror ball, where the texture represents what a per-fectly mirrored sphere would look like in the scene rendered through anorthographic camera. The middle showcases cube mapping, where the envi-ronment is projected onto the faces of a cube. Lastly, the right showcasesan equirectangular projection, mapping a sphere onto a rectangular texturespace.textures which map onto a sphere, similar to sphere maps, with the advantageof having a more uniform texture distribution as seen in Figure 4.2. Examplesof equirectangular environment maps gathered through Poly Haven [16] areshown in Figure 4.3.To query a texture map given a normalized direction vector (x,y,z), theequirectangular projection follows the equationF (x, y, z)→[uv]=[arctan2 (y,x)+π2πarccos (z)π ,](4.1)where the output u and v range between [0, 1]. These floating pointcoordinates are transformed into image coordinates by multiplying by thewidth and height of the environment texture respectively. The inverse ofthe projection, which maps the normalized image coordinates, (u, v) ∈ [0, 1],into a unit direction vector isF−1(u, v)→xyz =sin (θ) cos (ϕ)sin (θ) sin (ϕ)cos (θ) where θ = πv and ϕ = 2πu− π.(4.2)These two equations form the basis of our HDRi sampling method whichwe cover in the next section.354.2. Methodenvironment map sphere equirectangular projection mapFigure 4.2: Equirectangular Projection. Our work utilizes environmentmaps as a global illumination estimate using the 2D equirectangular pro-jection which maps a sphere onto a rectangular texture space using Equa-tion 4.1.veranda_4k.exrautumn_field_4k.exr mossy_forest_4k.exrFigure 4.3: Environment Maps. Examples of equirectangular environ-ment maps gathered through Poly Haven [16].4.2 MethodOur main goal for this extension is to be able to learn an environment mapalongside the neural body and Gaussian density models. In particular, thereare three cases we want to explore.Joint Optimization - Single Environment. The first case is what theoptimal goal would be, where a neural body model is jointly learnt along-side the Gaussian density model and the environment map. This conditionwould employ the simplest input data where a subject is being lit by a singleenvironment map of interest. Following a similar procedure to Section 3.4.2,we want the environment map that is being optimized to approximate thetrue illumination conditions that the data was recorded in. We theorize thatthis is the most challenging condition due to the large ambiguities that arepresent in this formulation. The problem is naturally over-parametrized asit is unclear whether an observed color in the training images is a cause of364.2. Methodthe subject’s texture, or the illumination. The fact that the subject changespose, and therefore experiences multiple illuminations, would provide cuesas to the true color of the character, but it is unclear whether it is sufficient.Joint Training - Multiple Environments. The second case is the mostcomplex in terms of the input data and the training targets. In this condition,we still jointly learn the lighting, Gaussians, and neural body model followingSection 3.4.2, but the training data is comprised of the same subject beingilluminated by multiple environment maps. We optimize for all of theseenvironment maps jointly and theorize that in doing so, the ambiguitiesdescribed in the single environment condition have a higher chance to bereduced.Fixed Neural Body. The final case is the simplest, used as a viabilitystudy. In this condition, a pretrained neural body model that was trainedon uniform illumination is used as a proxy to learn a single environmentmap. The environment map is trained from data of the same subject beingilluminated by some environment of interest. This study is used to con-firm the ability to optimize a texture map that is randomly sampled in anon-differentiable way while using the neural body as an intermediate stepbetween the observed image data and the optimized environment map.4.2.1 Importance SamplingIn Section 3.5.5, we described a method for sampling an environment mapfor HDRi relighting. In that method, a single importance sample was castedtowards the brightest pixel value in the environment map with the rest of thesecondary rays following a diffuse sampling technique based on the surfacenormal. This method worked well and was efficient for environment mapscontaining a single bright light source such as the sun. It, however, wouldfail on environment maps which contained multiple light sources as the singleimportance sample would fail to capture all major lights. In this section,we describe a more robust importance sampling technique for environmentmaps.Illuminating a surface patch requires the integration of the environmentmap along the hemisphere created by the surface normal. This in turn isequal to a probability weighted average of the function f(x), also known asthe expectation,374.2. Method∫p(x)f(x)dx = Ep[f(x)]. (4.3)In practice, it is too computationally expensive to calculate this integral,and instead is approximated using a Monte Carlo method where we draw Nrandom samples which follow the probability distribution p(x) and averageout the values.Ep[f(x)] ≈ 1NN∑i=1f(xi) xi ∼ p(x). (4.4)Here, f(xi) denotes the environment map value at xi where the samplesxi follow the probability density function p(x). In this case, p(x) is a uniformdistribution as the incoming light to a surface patch has equal probabilityof hitting a surface from any angle as long as it comes from the hemispherealong the normal (not being illuminated from the back). This approximationof the integral leads to very noisy renders, especially when the environmentmap contains high frequency details such as bright sources of light. This isdue to points in the environment map, which have the same probability ofbeing drawn as any other point, yet impact the expectation significantly dueto being a lot brighter.To overcome this, Importance Sampling is used, where a second prob-ability distribution, q(x), is used to sample these important regions moreoften and are reweighed to unbias the final average. Importance samplingstates thatEp[f(x)] =∫p(x)f(x)dx=∫q(x)[p(x)q(x)f(x)]dx= Eq[p(x)q(x)f(x)](4.5)as long as q(x) > 0 for values where p(x)f(x) ̸= 0. This can again beapproximated through Monte Carlo estimationEq[p(x)q(x)f(x)]≈ 1NN∑i=1p(x)q(x)f(xi) xi ∼ q(x). (4.6)384.2. MethodAgain, f(xi) denotes the environment map value at xi where in this case,the samples xi follow the probability density function q(x). This method ofsampling reduces the variance in the output as long as q(x) is large when|p(x)f(x)| is also large. In our implementation, we choose to combine adiffuse modulation distribution and pixel intensity distribution to constructq(x). We are able to sample the bright regions much more frequently aswell as regions in which the Bidirectional Reflectance Distribution Function(BRDF) is impacted more.We compute q(x) as the product of a diffuse modulation map and a pixelintensity map. We first create a projection of all surface normals onto theequirectangular map, Figure 4.4 (left, middle), which is used to constructthe diffuse modulation map by taking the cosine angle between each direc-tion and a given surface normal Figure 4.4 (right). The diffuse modulationmap is clipped to values between [0, 1] and is multiplied by the greyscalepixel intensity values of the environment map which is then normalized tosum to 1 (a PDF). This results in random samples that more frequently gotowards brighter regions in the environment map while being biased alongthe surface normal direction. This on its own does not recreate the BRDF’seffect of more incident light contributing less to the irradiance of a surfacepatch as the ratio p(x)q(x) corrects for the biased sampling. To properly recreatethe BRDF effect, we change p(x) to follow the diffuse modulation distribu-tion. Examples of our sampling method can be seen in Figure 4.5 whichshowcases: 1) The environment map, f(x), which in practice is blurred anddownsampled for diffuse objects. 2) The greyscale intensity map. 3) A dif-fuse modulation map for a given surface normal (marked in red). And 4) theresulting probability distribution q(x) which we use to sample 128 randompoints (shown as the red dots on the intensity maps).While the importance sampling results in less noisy light estimates, thereexists a bias in the sampling. The equirectangular projection has some dis-tortion along the latitude direction as seen in Figure 4.2. The poles of thesphere cover a smaller area than at the equator, yet map to the same sizedgrid in the projection. Due to our sampling choices coming from the pro-jected space, we expect the poles to be sampled more frequently in worldspace. While we do not expect this to affect our optimization results as thesampling is the same at both training and inference times, it is somethingthat should be corrected in the future.394.2. Method+z-z+x-x +y-y+z-z+x +y-y+z-z+x -x-x-x +y-ydi use intensity modulationtrue normals on equirectangular map normals on equirectangular mapFigure 4.4: Normals on the Equirectangular Projection Map. Theunnormalized (left) and normalized (middle) view direction / normal valuesmapped onto the equirectangular projection map. This map can be used tocompute the cosine angle between the environment map and a given normalto produce a modulation map to attenuate the sampling process given a sur-face normal (right). The queried normal for obtaining the diffuse modulationmap is highlighted as the red circle.4.2.2 Environment Map Fitting - Fixed Neural BodySynthetic Data. In a similar fashion to Section 3.5, we generate newsequences of synthetic human motion data for the environment map opti-mization experiments. Briefly, a 3D mesh scan of a subject was obtained,rigged and animated using Mixamo [2], and had a cloth simulation appliedusing Blender [9]. In contrast to our previous synthetic sequence, this scenewas illuminated using environment maps instead of a directional light. Werender the same animation sequence in multiple illuminations, the first beinga pure white environment to achieve an albedo-like (with ambient occlusion)render. The next render was illuminated using a daylight environment map,"sky_01", and the last, a more complicated illumination pattern with 3bright colored point sources, "tricolor_points". Examples of the renderedimages are shown in Figure 4.6Training We first train a neural body using the uniformly lit data to createa model with a pseudo-groundtruth albedo color field. The body is jointly fitalongside the Gaussian density model but is optimized without the lightingmodule enabled as there is no light source to fit.Using a separate dataset of the same subject illuminated by some envi-ronment of interest, the previously trained neural body model has its param-eters fixed and is used to render the albedo which will be lit with the learnedenvironment map. We create an empty environment map as the parametersfor optimization, typically of size 64× 32px, with all of the values initializedto 1—a uniformly white map. This is such that there is no bias during the404.2. Methodveranda_4kkiara_8_sunset_4kcinema_lobby_4kautumn_field_4kHDRi f(x) intensity map di use modulation sampling q(x)tricolor_pointsFigure 4.5: Importance Sampling. In order to approximate the integral ofthe incoming light at a given surface point, we cast secondary rays towardsthe environment map using an importance sampling technique. The methoduses the pixel value intensities multiplied by a diffuse modulation to obtainthe final probability density function from which to sample rays from. Thefigure’s columns showcase 1) The environment map. 2) The filtered anddown-sampled intensity maps with 128 random samples overlaid in red usingour method. 3) The diffuse modulation map given the normal marked in red.And 4) The resulting PDF from which we sample from by uniformly samplingthe PDF’s cumulative distribution function.start of training where some pixels are sampled more if they are brighter.Furthermore, the environment map is constrained to have intensity values> 0 such that the requirements of q(x) are satisfied in the importance sam-414.2. MethodFigure 4.6: Synthetic Data. Examples of the rendered images from oursynthetic data sequences. On the left is the subject being illuminated by auniform white light for albedo-like renders, the middle is from a sky texturewhere the sun was high in the sky, and the right showcases a more complexillumination condition with 3 colored point sources.pling equation. The optimized environment map is used to light the albedorenders using the importance sampling method previously described and theloss function values are computed as the photometric error between the litimages and the training images of the subject in the target illuminationcondition.While the sampling is non-differentiable due to the randomness, gradientsare able to back-propagate between the rendered lit images and the queriedenvironment map values. This poses a problem when environment mapvalues are close to 0 and are sampled very rarely or not at all as thoseregions will not be optimized. To mitigate this, we add a uniform constantto the optimized environment’s intensity map during the q(x) computationin the importance sampling equation to ensure that all regions continueto be sampled during the training. This is similar to exploration in theReinforcement Learning domain.Regularization. We can introduce priors into the optimized environmentmap to achieve more accurate results with faster convergence. For instance,424.3. Resultswe can regularize the environment map intensity values with the functionLsqrt = λUVU∑u=1V∑v=1√||f(xu,v)||2, (4.7)where ||f(xu,v)||2 is the L2 norm of the environment map at the coordi-nates u and v, and λ is the weighing factor. This has the effect of making theenvironment map sparse to get bright sources of light to be more point-like,for example, those environment maps which contain the sun.4.3 Results4.3.1 Relighting with Importance SamplingWe first showcase results of our importance sampling technique for relight-ing using NPC [45] as the backbone, shown in Figure 4.7. The imagesare rendered with 128 secondary rays per pixel which achieved minimalspeckle noise. The 4k resolution environment maps were preprocessed usinga Gaussian Filter with a kernel size of (127, 127) and were downsampled to128 × 64px prior to importance sampling. The results showcase a more ac-curate illumination compared to our previous sampling technique with theability to handle environment maps with multiple bright light sources as seenby the "tricolor_points" example.We run an ablation test to showcase how an ill-formed importance sam-pling equation affects the rendering. Figure 4.8 showcases the results of thisablation. If the diffuse modulation is not a part of the equation, the resultingrenders are smooth, due to the low frequency nature of the Gaussian densitymodel, with no details formed by the normal map as it is not a part of theequation, Figure 4.8a. On the other hand, if the diffuse modulation is usedin q(x) to sample along the normal more often, the resulting renders havetoo sharp of features as there is a large discrepancy between normals thathave the sun within their hemisphere and those which do not, Figure 4.8b.Using our proposed equation with p(x) being the diffuse modulation mapand q(x) also containing the diffuse modulation map, we achieve the bestresults as the detail quality is there without being too sharp, Figure 4.8c.4.3.2 Environment Map FittingWe run the viability study on both the sequence illuminated by a brightdaylight environment as well as the more complex "tricolor_points" map.434.3. ResultsFigure 4.7: Relighting Results. Results for relighting using our more ro-bust and accurate importance sampling as described in Section 4.2.1. NPC[45] was used as the backbone for the neural body model using the MonoP-erfCap dataset [52]. All environment maps except "tricolor_points" wereobtained through Poly Haven [16].We observe fairly accurate results in the optimized environment maps asshown in Figure 4.9. The environment maps begin as uniform maps andslowly approximate the true environment map.We compare how the regularization affects the optimized environmentmap using the sequence illuminated by the daylight environment map. Wefind that the regularization added a useful prior to more quickly optimizethe map to contain a single bright light source. It however, made the mapstoo sparse and less realistic as seen in Figure 4.10.We further compare the optimization if we did not add a constant to theintensity maps in the q(x) equation. Without this mitigation, it is possible444.4. DiscussionFigure 4.8: Importance Sampling Ablation. a) The importance sam-pling if the diffuse modulation is not used in either p(x) or q(x). The result-ing relighting is overtly smooth with no details b) The importance samplingif p(x) is kept uniform but q(x) has the diffuse modulation. The result-ing relighting contains very sharp features. c) Our full proposed importancesampling method. The results contains details while not being overtly sharp.for the learned environment map to not fully match with the groundtruthand be unable to recover due to the incorrect areas no longer being sampledand optimized. This can be seen by the incorrectly spaced gaps in Figure 4.11(top) between the point sources of light whereas with the mitigation (middle)the lights align with the groundtruth (bottom).4.4 DiscussionWhile we showcased good results on the viability study, proving the abil-ity to optimize an environment map, we have yet to prove whether or not454.4. Discussioniteration: 1000 iteration: 5,000 iteration: 10,000 iteration: 30,000 groundtruthFigure 4.9: Environment Optimization Results. The viability studyconfirmed that we can optimize environment maps through a photometricreconstruction loss between the training images and a neural body model litusing our importance sampling technique.iteration: 1000 iteration: 5,000 iteration: 10,000 iteration: 30,000w/out regularizationw/ regularization λ = 0.01w/ regularization λ = 0.05Figure 4.10: Regularization Ablation. We compare results for an opti-mized environment map without (top) and with (middle, bottom) the regu-larization discussed in Section 4.2.2.this method can be used while jointly training the neural body model andGaussian density model. It is also possible that our optimization methodcan be improved. In this section, we discuss possible avenues for improvingthe accuracy and convergence of the optimized environment map which canhelp future experiments.It is important to note that we cannot expect the learned environmentmaps to be as detailed as the true illumination maps. This is because weare using a highly diffuse shading model which averages out large portionsof the environment map as they are integrated into a single surface patch.Using more reflective objects, where the BRDF becomes more shallow, couldprovide these high-frequency cues that would allow for more detailed envi-ronment maps to be learned. Nevertheless, it may not be necessary to learn464.4. Discussioniteration: 250 iteration: 1,000 iteration: 5,000 iteration: 10,000 iteration: 20,000groundtruthw/out constantw/ constantFigure 4.11: Sampling Ablation in Optimization. The ablation studycomparing environment map optimization where the importance samplingdirectly follows Section 4.2.1 (top) which leads to areas not not being sampledand optimized if they become dark, and one where we add a uniform constantto the intensity map in q(x) (middle) to mitigate the problem. The methodwhere we add a constant has the optimized environment map align moreclosely with the groundtruth (bottom).these highly detailed maps as long as the lighting and shadows in the neuralcharacter are properly modeled, and can therefore be disentangled from thecolor field to allow for relighting.The first possible avenue for improving the optimization of the envi-ronment map is to bias the loss function to place more importance in theshadows. Areas with cast shadows can provide the most information for highfrequency light sources as being in a shadow would push every environmentpixel not occluded to be dark. It is however unclear how one can detectthese shadowed regions and bias the loss function to place more importanceon them.A second avenue for improving performance is to modify the proposedregularization. While we do want high frequency bright lights most of thetime. We rarely want the environment map to truly be sparse and havemost values reach 0. Oftentimes, values less than 1, are typical for nonemissive objects, and should not be penalized. Instead, only the extremelybright regions should be made more sparse. A possible solution is utilizinga learned prior that is trained on typical environment maps.Finally, a third avenue for possible accuracy improvement is to correct forthe biased sampling. Again equirectangular projection naturally has somedistortion along the latitude direction. To unbias the sampling, q(x) andp(x) would have to be multiplied by a correction map to sample the polesless frequently. Likewise, any regularization, or loss function, that acts on474.4. Discussionthe environment map texture should also have the correction applied.One interesting thing we did observe from the learned environment mapsare slight ghosting opposite of all light sources. One theory is that this is themodel trying to simulate bounce lighting which we do not explicitly modelin our lighting module. The subject had a white shirt in the training datawhich reflected a lot of light onto other body parts. Another theory is thatthese are due to boundary effects between the neural body model and thebackground pixels during rendering. Further tests should be conducted todiscern this effect.48Chapter 5Future Work and Conclusion5.1 Future WorkIn Chapter 4, we demonstrated the ability to optimize an environment tex-ture through the rendering of a subject illuminated by an environment ofinterest. It, however, utilized a fixed neural body with a pretrained albedocolor field. A natural extension of our work is to conduct the other twoexperiments listed in Section 4.2 which aim at jointly training a neural bodymodel with the environment map and Gaussian density model. Likewise,investigating the possible avenues for improving the optimization and con-vergence of the environment maps as listed in Section 4.4, would be of benefitfor future work that utilizes HDRis. In particular, modifying the loss func-tion to place more importance in shadowed regions has the best potential asthese areas provide the most information for high-frequency light sources inthe environment texture. It is, however, the most challenging avenue as it isunclear how one would detect these regions.A larger extension of our work is to integrate the explicit lighting com-putations of Gaussian Shadow Casting within the very popular GaussianSplatting work [20]. In that work, they also utilize a sum of 3D anisotropicGaussians to directly represent an entire high-fidelity static scene. Inte-grating our method could be the beginning of relighting large static worldsrealistically which has many applications in the VR/AR sectors as well astelepresence. Code for obtaining depth estimates and performing deferredrendering would be necessary, and most likely, an acceleration structure toquery the Gaussians along a shadow ray would have to be implemented asthe amount of Gaussians they use to represent a scene is in the hundredsof thousands while our work had at most 200. Another approach would beto represent the scene in a coarse-to-fine way, in a sense, creating levels ofdetail, which for lighting, the coarsest level of detail would be used, whilethe actual rendering would use the highest level of detail.Finally, another possible extension of our work is to utilize the Gaussiansfor more than just occlusion checking. For example, bounce lighting candrastically affect the final observed color of a surface patch. Using multiple495.2. Conclusionbounces for each ray would simulate global illumination in a scene, addingto the realism. This extension would require the Gaussian density model tohave some color associated with each Gaussian and would most likely requiresampling along the rays.5.2 ConclusionIn this thesis, we presented a novel method for lighting of neural characters.Using a sum of 3D anisotropic Gaussians an an approximation to the neu-ral density field, we can analytically compute occlusions along a ray whichforms the basis of our explicit shadow map computations. In Chapter 3,we demonstrate the ability to jointly optimize a neural body model and theGaussian density model alongside the direction of a light source, resultingin a color field that is more albedo-like; disentangling the shadows from thetexture in the body model. We further showcase the ability to relight theneural characters using high dynamic range images which obtain highly re-alistic lighting and shadows. In Chapter 4, we present a robust method forimportance sampling of environment maps and demonstrate the ability tooptimize an unknown environment texture through our lighting module.Our work directly extends recent advancements in computer vision whichaim at creating high fidelity personalized digital copies of subjects withoutrequiring expensive hardware setup and data processing. Our hope is that byadding our lighting computations, neural body models will be more practicalin the visual effects industry and in general. Prior works were restricted tolearn body models in uniform illuminations, and could render novel posesand views but were limited to be rendered in the same environment forrealistic compositing. Our work mitigates this by allowing the training ofthe neural body in non-uniformly lit environments and be relit in a diverseset of illuminations such that the compositing onto new backgrounds resultsin much more realistic renders.5.3 Risk and EthicsTo conclude, we showcased the creation of 3D digital copies of humans whichcan be animated and controlled by a user. These naturally have a riskassociated with them as users with ill intentions could utilize this technologyfor harm. They can generate digital doubles of people without their consentand create negative deepfake content.505.3. Risk and EthicsWe should note that our work only deals with the lighting computationsand is a general solution not specific to humans, but it is important to ac-knowledge that this we showcase sensitive work and that future contributionsshould be aware of the dangers. As researchers, we should be conscious oftechnologies we create that can be used to harm individuals and should takeprecautions to mitigate the risk. Future research on this topic should revolvearound the safety of human digital copies and ways of safeguarding once’sidentity. Outside of research, raising public awareness about these tools willbe integral. Likewise, the addition of regulations to protect individuals willbe crucial.51Bibliography[1] Easymocap - make human motion capture easier. https://github.com/zju3dv/EasyMocap, 2021. 23[2] Adobe. Mixamo. www.mixamo.com, 2023. 22, 40[3] Thiemo Alldieck, Marcus Magnor, Bharat Lal Bhatnagar, ChristianTheobalt, and Gerard Pons-Moll. Learning to reconstruct people inclothing from a single RGB camera. In CVPR, 2019. 11[4] Timur Bagautdinov, Chenglei Wu, Tomas Simon, Fabian Prada,Takaaki Shiratori, Shih-En Wei, Weipeng Xu, Yaser Sheikh, and Ja-son Saragih. Driving-signal aware full-body avatars. ACM TOG (Proc.SIGGRAPH), 2021. 10[5] Sai Bi, Stephen Lombardi, Shunsuke Saito, Tomas Simon, Shih-En Wei,Kevyn Mcphail, Ravi Ramamoorthi, Yaser Sheikh, and Jason Saragih.Deep relightable appearance models for animatable faces. ACM TOG(Proc. SIGGRAPH), 2021. 11[6] Luis Bolanos, Shih-Yang Su, and Helge Rhodin. Gaussian shadow cast-ing for neural characters. In CVPR, 2024. 8[7] Mark Boss, Raphael Braun, Varun Jampani, Jonathan T. Barron,Ce Liu, and Hendrik P.A. Lensch. Nerd: Neural reflectance decom-position from image collections. In ICCV, 2021. 10[8] Zhaoxi Chen and Ziwei Liu. Relighting4d: Neural relightable humanfrom videos. In ECCV, 2022. xi, 2, 9, 11, 21, 23, 24[9] Blender Online Community. Blender - a 3d modelling and renderingpackage. http://www.blender.org, 2023. 22, 40[10] Paul Debevec, Tim Hawkins, Chris Tchou, Haarm-Pieter Duiker, West-ley Sarokin, and Mark Sagar. Acquiring the reflectance field of a hu-man face. In Proceedings of the 27th Annual Conference on Computer52BibliographyGraphics and Interactive Techniques, SIGGRAPH ’00, page 145–156,USA, 2000. ACM Press/Addison-Wesley Publishing Co. 1[11] Dawa Derksen and Dario Izzo. Shadow neural radiance fields for multi-view satellite photogrammetry. In CVPR, 2021. 10[12] Junting Dong, Qi Fang, Wen Jiang, Yurou Yang, Hujun Bao, and Xi-aowei Zhou. Fast and robust multi-person 3d pose estimation and track-ing from multiple views. In TPAMI, 2021. 23[13] Junting Dong, Qing Shuai, Yuanqing Zhang, Xian Liu, Xiaowei Zhou,and Hujun Bao. Motion capture from internet videos. In ECCV, 2020.23[14] Amos Gropp, Lior Yariv, Niv Haim, Matan Atzmon, and Yaron Lipman.Implicit geometric regularization for learning shapes. In ICML, 2020.20[15] Michelle Guo, Alireza Fathi, Jiajun Wu, and Thomas Funkhouser.Object-centric neural scene rendering, 2020. 10, 11[16] Poly Haven. Poly haven. www.polyhaven.com/hdris, 2023. xii, xiii, 28,35, 36, 44[17] Jingwei Huang, Haotian Zhang, Li Yi, Thomas Funkhouser, MatthiasNießner, and Leonidas J Guibas. Texturenet: Consistent localparametrizations for learning from high-resolution signals on meshes.In CVPR, 2019. 11[18] Umar Iqbal, Akin Caliskan, Koki Nagano, Sameh Khamis, PavloMolchanov, and Jan Kautz. Rana: Relightable articulated neuralavatars. arXiv preprint arXiv:2212.03237, 2022. xi, 9, 10, 11, 22, 23, 24[19] Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and AnuragRanjan. Neuman: Neural human radiance field from a single video. InECCV, 2022. 2, 8[20] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and GeorgeDrettakis. 3d gaussian splatting for real-time radiance field rendering.ACM TOG (Proc. SIGGRAPH), 2023. 9, 12, 49[21] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rol-land, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C.Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything.arXiv:2304.02643, 2023. 2353Bibliography[22] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Dani-ilidis. Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In ICCV, 2019. ix, 6[23] Youngjoong Kwon, Dahun Kim, Duygu Ceylan, and Henry Fuchs. Neu-ral human performer: Learning generalizable radiance fields for humanperformance rendering. NeurIPS, 2021. 8, 10[24] Brady Langmann. How j.j. abrams pulled off carrie fisher’s cgi flashbackin star wars: The rise of skywalker. Esquire, 2020. 1[25] J. P. Lewis, Matt Cordner, and Nickson Fong. Pose Space Deformation:A Unified Approach to Shape Interpolation and Skeleton-Driven Defor-mation. Association for Computing Machinery, New York, NY, USA, 1edition, 2000. 5[26] Junxuan Li, Shunsuke Saito, Tomas Simon, Stephen Lombardi, Hong-dong Li, and Jason Saragih. Megane: Morphable eyeglass and avatarnetwork. In CVPR, 2023. 1, 11[27] Ruilong Li, Julian Tanke, Minh Vo, Michael Zollhofer, Jurgen Gall,Angjoo Kanazawa, and Christoph Lassner. Tava: Template-free ani-matable volumetric actors. In ECCV, 2022. 2, 10, 31[28] Zhe Li, Zerong Zheng, Yuxiao Liu, Boyao Zhou, and Yebin Liu. Po-sevocab: Learning joint-structured pose embeddings for human avatarmodeling. In ACM TOG (Proc. SIGGRAPH), 2023. 10[29] Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar,Jiatao Gu, and Christian Theobalt. Neural actor: Neural free-viewsynthesis of human actors with pose control. ACM TOG (Proc. SIG-GRAPH Asia), 2021. 10[30] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll,and Michael J. Black. SMPL: A skinned multi-person linear model.ACM TOG (Proc. SIGGRAPH Asia), 2015. 9, 11[31] N. Max. Optical models for direct volume rendering. IEEE Transactionson Visualization and Computer Graphics, 1(2):99–108, 1995. 4[32] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T.Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes asneural radiance fields for view synthesis. In ECCV, 2020. 3, 8, 1054Bibliography[33] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada. Neuralarticulated radiance field. In ICCV, 2021. 10[34] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai,Hujun Bao, and Xiaowei Zhou. Neural body: Implicit neural represen-tations with structured latent codes for novel view synthesis of dynamichumans. In CVPR, 2021. 1, 2, 8, 23[35] Gabriel Ponniah. How (& why) sean young’s rachael was deaged inblade runner 2049. Screen Rant, 2021. 1[36] Anurag Ranjan, Kwang Moo Yi, Jen-Hao Rick Chang, and Oncel Tuzel.Facelit: Neural 3d relightable faces. In CVPR, 2023. 11[37] H. Rhodin, N. Robertini, C. Richardt, H. Seidel, and C. Theobalt. Aversatile scene model with differentiable visibility applied to generativepose estimation. In ICCV, 2015. 9, 11, 12, 14[38] Helge Rhodin, Nadia Robertini, Dan Casas, Christian Richardt, Hans-Peter Seidel, and Christian Theobalt. General automatic human shapeand motion capture using volumetric contour cues. In Bastian Leibe,Jiri Matas, Nicu Sebe, and Max Welling, editors, ECCV, 2016. 9, 11,12, 14, 16[39] Viktor Rudnev, Mohamed Elgharib, William Smith, Lingjie Liu,Vladislav Golyanik, and Christian Theobalt. Nerf for outdoor scenerelighting. In ECCV, 2022. 10[40] Srinath Sridhar, Helge Rhodin, Hans-Peter Seidel, Antti Oulasvirta, andChristian Theobalt. Real-time hand tracking using a sum of anisotropicgaussians model. In 3DV, 2014. 9, 12[41] Pratul P. Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik,Ben Mildenhall, and Jonathan T. Barron. Nerv: Neural reflectance andvisibility fields for relighting and view synthesis. In CVPR, 2021. 10[42] Carsten Stoll, Nils Hasler, Juergen Gall, Hans-Peter Seidel, and Chris-tian Theobalt. Fast articulated motion tracking using a sums of gaus-sians body model. ICCV, 2011. 11[43] Shih-Yang Su. From videos to animatable 3d neural characters. PhDthesis, 2024. ix, 355Bibliography[44] Shih-Yang Su, Timur Bagautdinov, and Helge Rhodin. Danbo: Disen-tangled articulated neural body representations via graph neural net-works. In ECCV, 2022. ix, x, xi, 2, 3, 5, 6, 7, 8, 10, 12, 13, 23, 25, 26,30[45] Shih-Yang Su, Timur Bagautdinov, and Helge Rhodin. Npc: Neuralpoint characters from video. In ICCV, 2023. xi, xiii, 3, 5, 8, 23, 25, 26,29, 30, 43, 44[46] Shih-Yang Su, Frank Yu, Michael Zollhöfer, and Helge Rhodin. A-nerf:Articulated neural radiance fields for learning human shape, appear-ance, and pose. In NeurIPS, 2021. 2, 5, 10[47] Justus Thies, Michael Zollhöfer, and Matthias Nießner. Deferred neuralrendering: Image synthesis using neural textures. ACM TOG (Proc.SIGGRAPH), 2019. 13[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, LlionJones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attentionis all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach,R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neu-ral Information Processing Systems, volume 30. Curran Associates, Inc.,2017. 4[49] Shaofei Wang, Katja Schwarz, Andreas Geiger, and Siyu Tang. Arah:Animatable volume rendering of articulated human sdfs. In ECCV,2022. 8, 10[50] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.Image quality assessment: from error visibility to structural similarity.IEEE transactions on image processing, 13(4):600–612, 2004. 26[51] Zian Wang, Tianchang Shen, Jun Gao, Shengyu Huang, JacobMunkberg, Jon Hasselgren, Zan Gojcic, Wenzheng Chen, and SanjaFidler. Neural fields meet explicit geometric representations for inverserendering of urban scenes. In CVPR, 2023. 10, 11[52] Weipeng Xu, Avishek Chatterjee, Michael Zollhöfer, Helge Rhodin,Dushyant Mehta, Hans-Peter Seidel, and Christian Theobalt. Monoper-fcap: Human performance capture from monocular video. ACM Trans.Graph., 37(2):27:1–27:15, May 2018. ix, xi, xiii, 6, 29, 4456[53] Zhen Xu, Sida Peng, Chen Geng, Linzhan Mou, Zihan Yan, JiamingSun, Hujun Bao, and Xiaowei Zhou. Relightable and animatable neuralavatar from sparse-view video, 2023. 9, 11, 23[54] Haotian Yang, Mingwu Zheng, Wanquan Feng, Haibin Huang, Yu-KunLai, Pengfei Wan, Zhongyuan Wang, and Chongyang Ma. Towardspractical capture of high-fidelity relightable avatars. In ACM TOG(Proc. SIGGRAPH Asia), 2023. 11[55] Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume ren-dering of neural implicit surfaces. In NeurIPS, 2021. 13, 18, 20[56] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and NoahSnavely. PhySG: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In CVPR, 2021. 12[57] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and OliverWang. The unreasonable effectiveness of deep features as a perceptualmetric. In CVPR, 2018. 26[58] Xiuming Zhang, Pratul P. Srinivasan, Boyang Deng, Paul Debevec,William T. Freeman, and Jonathan T. Barron. Nerfactor: Neural factor-ization of shape and reflectance under an unknown illumination. ACMTOG (Proc. SIGGRAPH), 2021. 10[59] Zerong Zheng, Han Huang, Tao Yu, Hongwen Zhang, Yandong Guo, andYebin Liu. Structured local radiance fields for human avatar modeling.In CVPR, 2022. 10[60] Zerong Zheng, Xiaochen Zhao, Hongwen Zhang, Boning Liu, and YebinLiu. Avatarrex: Real-time expressive full-body avatars. ACM TOG(Proc. SIGGRAPH), 2023. 10[61] Taotao Zhou, Kai He, Di Wu, Teng Xu, Qixuan Zhang, Kuixiang Shao,Wenzheng Chen, Lan Xu, and Jingyi Yu. Relightable neural humanassets from multi-view gradient illuminations. In CVPR, 2023. 11[62] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. Onthe continuity of rotation representations in neural networks. In CVPR,2019. 1457Appendix AA.1 Data Collection Consent FormAttached below is the consent form used during the data collection of humanmotion to train personalized neural characters of.58 February 23, 2023 H19-02997 1 of 3 Participant Consent Form Markerless Motion Capture dataset recording, a new benchmark for motion capture algorithms You are being invited to participate in a study entitled Human Body Modeling that is being conducted by Helge Rhodin. Helge Rhodin is the principal investigator and can be reached by mail and phone . The purpose of this study is to record a large dataset of casual motions performed in every-day environments and appearances. The dataset will be used to train and evaluate computer vision and machine learning algorithms for reconstructing and modelling human motion and appearance. The output of the algorithm is a virtual copy of the person with capabilities to change the persons pose, shape and appearance. However, use is restricted to alterations that are nonviolent, non-offensive and respectful. The use of the dataset for the purpose of surveillance applications as well as any non-ethical use is explicitly excluded. The construction of such dataset is important because it allows us to understand whether the developed tools are efficient in reconstructing video recordings of persons. Example applications are telepresence and motion capture for sports, medicine, and entertainment. If you volunteer to participate in this research, your participation may include: • Recording of scripted every-day motions such as walking, running, gesturing, hugging, waving, jumping, and reaching. • Recording of you in your personal clothing. • Recording of your facial expressions and facial appearance, should you consent. The research session is expected to take approximately 30 to 60 minutes and, depending on the research stage, will take place at one of the following locations: • In a closed room in the UBC Computer Science Department. • At a public place such as plaza, open grass field, or wide path. There are only minor anticipated risks to you by participating in this research since only every-day motions are performed and no personal information is retained except video recordings of your scripted performance and optionally your age, height, weight and ethnicity. It is of equal or lower risk as posting a video of you on Youtube. COVID-19 poses additional risk that will be mitigated with the following measures. Hand sanitizer will be available at the recording site, contact objects will be disinfected before and after every use, and a social distance of 2m be maintained whenever possible. Taking part in this study is entirely up to you. You have the right to refuse to participate in this study. If you decide to take part, you may choose to pull out of the study at any time without giving a reason and without any negative impact. If you do withdraw, we will ask whether we may use your data for data analysis and destroy Department of Computer Science University of British Columbia 201-2366 Main Mall Vancouver, V6T 124 Canada 59 February 23, 2023 H19-02997 1 of 3 your data otherwise. Your confidentiality and the confidentiality of the data will be protected by identifying data only with a participant number rather than your name. All data stored in computer files will be encrypted and password- protected. Only the principal investigator and co-investigators will have access to any personal information you provide. We will give access to the main study data via an open-access research repository, earliest three months after data recording. From that point on, your will not be able to withdraw your data. Note that all personal identifiers are removed beforehand, unless otherwise agreed upon, and this poses minimal risk to participants. We anticipate that parts of this dataset will be shared with others in the following ways: • Published articles and their supplemental material • Conference presentations • Theses • Internet project descriptions • This study is sponsored by NSERC, Mitacs and by our commercial project partners, Huawei Canada Inc., FlexAI, Providence Health Care (PHC), and FlawlessAI. In addition to the UBC internal study, you may give consent to allow the project partners to use the recordings for internal research and development, under the same conditions as the public datasets, except for allowing commercial use. Only algorithmic development is allowed. For instance, algorithm performance may be evaluated on recorded videos to select the best model. The captured data and its derivatives, such as machine learning algorithms trained on it, cannot be sold nor re-distributed to third parties. Data from this study will be stored for at least 10 years but at most 25 years, unless consent is given or earlier termination is requested. Electronic data will be erased and paper copies will be shredded. Imagery published in conferences and journals cannot be deleted. If you have any concerns or complaints about your rights as a research participant and/or your experiences while participating in this study, contact the Research Participant Complaint Line in the UBC Office of Research Ethics at or if long distance e-mail or call toll free . Your signature below indicates that you have received a copy of this consent form for your own records and implies your consent to participate for the duration of the study today and that the recorded data may be used for the purposes outlined in this document. You can withdraw your participation at any time, as outlined above. You may provide additional consent for open-access, retaining raw recordings, and indefinite storage: ☐ Please tick this box if you allow publishing your recordings in open-access repositories and papers. Note that data cannot be revoked after publication. Open-access is only given three months after recording and you can still withdraw at any time until then. 60 February 23, 2023 H19-02997 1 of 3 ☐ Please tick this box if you allow Huawei, FlexAI, PHC, and Flawless AI to use your recordings in for internal algorithmic development. Note that selling or deployment of the data in products will still be excluded, it is for R&D only. ☐ Please tick this box if you give additional consent for storing and re-distributing your raw video recordings. Raw video enables people who know you or have imagery with identification information to identify you in the video. Please note that you will not be able to withdraw any footage that is printed and displayed in journal and conference articles and their supplemental documents nor data provided on public-access repositories. ☐ Please tick this box if you give additional consent for storing your data indefinitely, beyond the 25 years. __________________________ ________________________________ ________________________ Name of Participant Signature Date A copy of this consent will be left with you, and a copy will be taken by the researcher. 61
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-ShareAlike 4.0 International
Graduate
Rhodin, Helge
Gaussian shadow casting for neural characters
Text