Science, Faculty of
Computer Science, Department of
DSpace
UBCV
Wu, Zhijie
Master of Science - MSc
University of British Columbia
We present a novel method to provide efficient and highly detailed reconstructions. Inspired by wavelets, we learn a neural field that decompose the signal both spatially and frequency-wise. We follow the recent grid-based paradigm for spatial decomposition, but unlike existing work, encourage specific frequencies to be stored in each grid via Fourier features encodings. We then apply a multi-layer perceptron with sine activations, taking these Fourier encoded features in at appropriate layers so that higher-frequency components are accumulated on top of lower-frequency components sequentially, which we sum up to form the final output. We demonstrate that our method outperforms the state of the art regarding model compactness and convergence speed on multiple tasks: 2D image fitting, 3D shape reconstruction, and neural radiance fields. Our code is available at https://github.com/ubc-vision/NFFB.
Neural Fourier Filter BankbyZhijie WuB.Sc., Sichuan University, 2016A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFMaster of ScienceinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Computer Science)The University of British Columbia(Vancouver)August 2023© Zhijie Wu, 2023The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Neural Fourier Filter Banksubmitted by Zhijie Wu in partial fulfillment of the requirements for the degree ofMaster of Science in Computer Science.Examining Committee:Kwang Moo Yi, Assistant Professor, Computer Science, UBCSupervisorVered Shwartz, Assistant Professor, Computer Science, UBCSupervisory Committee MemberiiAbstractWe present a novel method to provide efficient and highly detailed reconstruc-tions. Inspired by wavelets, we learn a neural field that decompose the signalboth spatially and frequency-wise. We follow the recent grid-based paradigm forspatial decomposition, but unlike existing work, encourage specific frequenciesto be stored in each grid via Fourier features encodings. We then apply a multi-layer perceptron with sine activations, taking these Fourier encoded features in atappropriate layers so that higher-frequency components are accumulated on topof lower-frequency components sequentially, which we sum up to form the finaloutput. We demonstrate that our method outperforms the state of the art regard-ing model compactness and convergence speed on multiple tasks: 2D image fit-ting, 3D shape reconstruction, and neural radiance fields. Our code is available athttps://github.com/ubc-vision/NFFB.iiiLay SummaryIn this thesis, we introduce a novel artificial neural network tailored to synthe-size high-quality results while minimizing computation budgets and maintainingdesirable method complexities. Our motivation stems from observing that sig-nals exhibit varying frequency changes across spatial regions. Additionally, high-frequency signals rapidly change over short spatial spans, while low-frequency sig-nals remain stable over larger expanses. Our main contribution involves segment-ing signals into smaller parts and explicitly modulating high-frequency signals inlocalized regions, while controlling low-frequency signals in broader areas. Ourmethodology undergoes evaluation across three tasks: the fitting of 2D images, thereconstruction of 3D shapes, and the generation of images utilizing camera data.Empirical validation highlights the potential of our approach to yield favorableoutcomes with reduced computational expenses and increased compactness.ivPrefaceThe work presented in this thesis is developed by the author, Zhijie Wu, and alsois in the collaboration with Yuhe Jin. Specifically, Yuhe provided invaluable as-sistance in the technical discussions near the CVPR deadline and helped make allthe figures and tables in the manuscript. This work is under the supervision ofProf. Kwang Moo Yi who provided invaluable feedback on the idea formulation,experiment consolidations, the writing of the manuscript, and many other aspects.A version of this work has been published at:• Zhijie Wu, Yuhe Jin, and Kwang Moo Yi. “Neural Fourier Filter Bank”. InComputer Vision and Pattern Recognition (CVPR), June 2023 [81]vTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.1 Neural Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.2 Neural Radiance Fields . . . . . . . . . . . . . . . . . . . . . . . 62.3 Wavelets in deep nets . . . . . . . . . . . . . . . . . . . . . . . . 83 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.1 The Fourier grid features . . . . . . . . . . . . . . . . . . . . . . 103.1.1 Multi-scale grid . . . . . . . . . . . . . . . . . . . . . . . 113.1.2 Converting grid features to Fourier features . . . . . . . . 123.2 Composing the field value . . . . . . . . . . . . . . . . . . . . . 12vi3.3 Implementation details . . . . . . . . . . . . . . . . . . . . . . . 143.3.1 More details about Fourier grid features . . . . . . . . . . 143.3.2 Common Implementations . . . . . . . . . . . . . . . . . 154 Experimental Results . . . . . . . . . . . . . . . . . . . . . . . . . . 164.1 2D Image Fitting . . . . . . . . . . . . . . . . . . . . . . . . . . 164.1.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164.1.2 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . 184.1.3 Experimental Settings . . . . . . . . . . . . . . . . . . . 194.1.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 194.2 3D Shape Reconstruction . . . . . . . . . . . . . . . . . . . . . . 204.2.1 Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214.2.2 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . 224.2.3 Experimental Settings . . . . . . . . . . . . . . . . . . . 224.2.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 234.3 Novel View Synthesis . . . . . . . . . . . . . . . . . . . . . . . . 244.3.1 Network Adaptation & Experimental Setting . . . . . . . 264.3.2 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . 264.3.3 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . 264.4 Ablation Study . . . . . . . . . . . . . . . . . . . . . . . . . . . 284.4.1 General Design Justification . . . . . . . . . . . . . . . . 284.4.2 The effect of grid resolutions . . . . . . . . . . . . . . . . 294.4.3 The effect of the Fourier feature variance. . . . . . . . . . 294.4.4 The effect of the scaling factor αi. . . . . . . . . . . . . . 304.4.5 The effect of Fourier encodings . . . . . . . . . . . . . . 304.4.6 The effect of fully-connected layer size. . . . . . . . . . . 314.4.7 The effect of the Fourier grid level. . . . . . . . . . . . . 314.5 Other Discussions . . . . . . . . . . . . . . . . . . . . . . . . . . 314.5.1 Comparison with ModSine . . . . . . . . . . . . . . . . . 314.5.2 Discussions about Runtime . . . . . . . . . . . . . . . . . 325 Conclusions and Future Work . . . . . . . . . . . . . . . . . . . . . 335.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33vii5.2 Broader Impact . . . . . . . . . . . . . . . . . . . . . . . . . . . 345.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35viiiList of TablesTable 4.1 2D Fitting – We report the reconstruction comparisons in termsof Peak Signal-to-Noise Ratio (PSNR), Structural Similarity In-dex Metric (SSIM) [27] and Learned Perceptual Image PatchSimilarity (LPIPS) [91]. Our method provides the best trade-off between model size and reconstruction quality. . . . . . . 18Table 4.2 Performance under varying T and L – Our method showshigher PSNR values for ‘Tokyo’ image with various T and Lsettings. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Table 4.3 3D Fitting – We report the Intersection over Union (IoU), F-Score and Chamfer distance (CD) after performing marchingcubes to extract surfaces. Our method performs best, with theexception of F-score on ‘Asian Dragon’, which is due to BA-CON preferring blobby output, as demonstrated by the higherChamfer distance and worse IoU. . . . . . . . . . . . . . . . . 24Table 4.4 Neural Radiance Fields (NeRF) – We report the novel viewrendering performance in terms of Peak Signal-to-Noise Ra-tio (PSNR) and Structural Similarity Index Metric (SSIM). Ourmethod provides comparable rendering quality as the state ofthe art, while having the smallest size among the grid-basedmethods (middle rows) that provide fast training, providing thebest trade-off between quality and model size. See the appendixfor runtime discussions. . . . . . . . . . . . . . . . . . . . . . 25ixList of FiguresFigure 1.1 Teaser – We propose neural Fourier filter bank to performspatial and frequency-wise decomposition jointly, inspired bywavelets. Our method provides significantly improved recon-struction quality given the same computation and storage bud-get, as represented by the PSNR curve and the error imageoverlay. Relying only on space partitioning without frequencyresolution (InstantNGP) [49] or frequency encodings withoutspace resolution (SIREN) [65] provides suboptimal performanceand convergence. Simply considering both (ModSine) [46] en-hances scalability when applied to larger scenes, but not interms of quality and convergence. . . . . . . . . . . . . . . . 2Figure 1.2 A wavelet-inspired framework – In our framework, givena position x, low- and high-frequency filters are used to de-compose the signal, which is then reconstructed by accumulat-ing them and using the intermediate outputs as shown. Here,we utilize a multi-scale grid to act as if they store these high-frequency filtering outcomes at various spatially decomposedlocations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3xFigure 3.1 Framework overview – Based on the input query, e.g., theposition x, our neural Fourier filter bank uses both a grid anda Multi-Layer Perceptron (MLP) to compose the final esti-mate. Specifically, grid features are extracted via interpolationat multiple scale levels, which are then encoded to appropriatefrequencies for each layer via the Fourier Feature layers. TheMLP uses these encoded features as the higher-frequency com-ponent in Fig. 1.2, while the earlier layer outputs as the lowerfrequency ones, similarly to wavelet filter banks. Intermediateoutputs are then aggregated as the final estimate. . . . . . . . 13Figure 4.1 2D Fitting – Qualitative results for the Tokyo image. Ourmethod provides the best reconstruction quality at various scalelevels, from nearby regions to far away ones, demonstrating theimportance of considering both space and frequency jointly. . 17Figure 4.2 2D fitting result for ‘Einstein’ image. Our entire reconstructedimage is presented on the left while four close-up views arepresented on the right. Note how our reconstructions are near-perfect for both coarse and fine details. . . . . . . . . . . . . 20Figure 4.3 2D fitting result for ‘Tokyo’ image. Our entire reconstructedimage is presented on the top while four close-up views arepresented on the bottom. Our method provides near-perfectreconstruction. . . . . . . . . . . . . . . . . . . . . . . . . . 21Figure 4.4 3D Fitting – Qualitative comparisons for the ‘Bearded Man’shape. Our method is the most compact among the comparedmethods, and is capable of reconstructing both coarse and finedetails without obvious artifacts. . . . . . . . . . . . . . . . 22Figure 4.5 3D fitting result for the ‘Asian Dragon’. The left sub-image isthe ground truth shape while six zoomed insets are shown onthe right for better detail visualizations. . . . . . . . . . . . . 23Figure 4.6 Novel View Synthesis – Although more compact, our methodcan synthesize comparable or better results. . . . . . . . . . . 25xiFigure 4.7 Qualitative results for novel view synthesis with neural radi-ance fields. Our method is able to clearly reconstruct the tex-tures (e.g., , the chair on 2nd row) and the geometric details(e.g., the lego on the last row). . . . . . . . . . . . . . . . . . 27Figure 4.8 Ablation study – We compare against variants of our methodwith the Fourier grid feature and/or the proposed MLP compo-sition architecture disabled. Having both components togetheris critical for performance. . . . . . . . . . . . . . . . . . . . 28Figure 4.9 Ablation studies for the Fourier feature variance (left) and thescaling factor α in fully-connected layers (right). Both pa-rameters highly affect how frequency is dealt with within ourframework, and thus require optimal values to be set. Theseparameters are mostly task dependant. . . . . . . . . . . . . 29Figure 4.10 The ablation study for the base resolution Nmin. With largerNmin, fine details are better preserved. . . . . . . . . . . . . . 30Figure 4.11 The ablation study for the scaling coefficient cg. With larger cg,reconstruction quality improves, with more fine details beingpreserved and with higher spatial resolution. . . . . . . . . . 30Figure 4.12 The ablation study results for the MLP size. As the MLP sizeincreases, the network becomes better at composing signalsfrom various levels, thus various frequencies, leading to a bet-ter final outcome. . . . . . . . . . . . . . . . . . . . . . . . 31Figure 4.13 The ablation study for the number of levels for the Fourier gridfeature. More levels lead to a drastic increase in the quality offine details. . . . . . . . . . . . . . . . . . . . . . . . . . . . 31xiiAcknowledgementsI would like to acknowledge the contributions of various individuals and institu-tions that have made this research possible and have significantly impacted myacademic journey during my master’s study at UBC.First and foremost, I am deeply grateful to Prof. Kwang Moo Yi, my supervi-sor, who has been an exceptional mentor and one of the kindest individuals I haveever had the privilege to meet. When I first arrived in Vancouver and began mystudies at UBC, I faced significant challenges with my oral English skills as a for-eign student. However, Kwang displayed remarkable patience and provided invalu-able guidance during our research discussions. Furthermore, he generously offeredme ample room for exploration, accompanied by insightful suggestions and ac-cess to rich computation resources when I ventured into the realm of neural fields.Above all, his meticulous academic attitude has greatly influenced and shaped myresearch sensibilities. It is through his unwavering support and encouragement thatmy academic journey at UBC has become an extraordinary experience.I would also like to express my sincere appreciation to the Computer ScienceDepartment at UBC for granting me the chance to pursue my studies in such a pres-tigious program. Additionally, I am grateful for the support received from variousorganizations, including the Natural Sciences and Engineering Research Councilof Canada (NSERC) Discovery Grant, Digital Research Alliance of Canada, andAdvanced Research Computing at UBC.Furthermore, I extend my heartfelt thanks to the students in Kwang’s researchgroup, who have been a constant source of companionship and warmth throughoutmy journey. In particular, I would like to acknowledge Yuhe Jin for his insightfulconversations and exceptional paper polishing skills. Their presence has greatlyxiiienriched my research experience.Finally, I want to express my deepest gratitude to my parents and family. Theirconstant support and encouragement have been instrumental in helping me achievethis milestone in my academic career. Without their love and understanding, noneof this would be possible.Once again, I would like to extend my heartfelt appreciation to all those whohave contributed to my research and academic growth.xivChapter 1IntroductionNeural fields [83] have recently been shown to be highly effective for various tasksranging from 2D image compression [16, 92], image translation [6, 66], 3D recon-struction [54, 64], to neural rendering [1, 48, 49]. Since the introduction of earlymethods [48, 50, 64], efforts have been made to make neural fields more efficientand scalable. Among various extensions, we are interested in two particular direc-tions: those that utilize spatial decomposition in the form of grids [10, 49, 72] thatallow fast training and level of detail; and those that encode the inputs to neuralfields with high-dimensional features via frequency transformation such as peri-odic sinusoidal representations [48, 65, 73] that fight the inherent bias of neuralfields that is towards low-frequency data [73]. The former drastically reduced thetraining time allowing various new application areas [17, 74, 82, 86], while thelatter has now become a standard operation when applying neural fields.While these two developments have become popular, a caveat in existing worksis that they do not consider the two together—all grids are treated similarly andinterpreted together by a neural network. We argue that this is an important over-sight that has a critical outcome. For a model to be efficient and accurate, differentgrid resolutions should focus on different frequency components that are properlylocalized. While existing grid methods that naturally localize signals—can learnto perform this frequency decomposition, relying purely on learning may lead tosub-optimal results as shown in Fig. 1.1. This is also true when locality is notconsidered, as shown by the SIREN [65] example. Explicit consideration of both1FrequencySpaceFrequencySpaceFrequencySpaceFrequencySpaceInstantNGP SIREN ModSine OursPSNRIterationsIntensity Error (intensity is between 0 and 255)Figure 1.1: Teaser – We propose neural Fourier filter bank to perform spatialand frequency-wise decomposition jointly, inspired by wavelets. Ourmethod provides significantly improved reconstruction quality giventhe same computation and storage budget, as represented by the PSNRcurve and the error image overlay. Relying only on space partitioningwithout frequency resolution (InstantNGP) [49] or frequency encod-ings without space resolution (SIREN) [65] provides suboptimal per-formance and convergence. Simply considering both (ModSine) [46]enhances scalability when applied to larger scenes, but not in terms ofquality and convergence.together is hence important.This caveat remains true even for methods that utilize both grids and frequencyencodings for the input coordinates [49] as grids and frequency are not linked,and it is up to the deep networks to find out the relationship between the two.Thus, there has also been work that focuses on jointly considering both space andfrequency [26, 46], but these methods are not designed with multiple scales in mindthus single-scale and are designed to be non-scalable. In other words, they can bethought of as being similar to short-time Fourier transform in signal processing.Therefore, in this work, we propose a novel neural field framework that de-composes the target signal in both space and frequency domains simultaneously,analogous to the traditional wavelet decomposition [62]; see Fig. 1.1. Specifi-cally, a signal is decomposed jointly in space and frequency through low- andhigh-frequency filters as shown in Fig. 1.2. Here, our core idea is to realize thesefilters conceptually as a neural network. We implement the low-frequency path inthe form of Multi-Layer Perceptrons (MLP), leveraging their frequency bias [73].For the high-frequency components, we implement them as lookup operations on2+++XXXLow Frequency FilterHigh Frequency FilterFigure 1.2: A wavelet-inspired framework – In our framework, given a po-sition x, low- and high-frequency filters are used to decompose the sig-nal, which is then reconstructed by accumulating them and using theintermediate outputs as shown. Here, we utilize a multi-scale grid toact as if they store these high-frequency filtering outcomes at variousspatially decomposed locations.grids, as the grid features can explicitly enforce locality over a small spatial areaand facilitate learning of these components. This decomposition is much resem-blant of filter banks in signal processing, thus we name our method neural Fourierfilter bank.In more detail, we utilize the multi-scale grid structure as in [28, 49, 72], butwith a twist—we apply frequency encoding in the form of Fourier Features justbefore the grid features are used. By doing so, we convert the linear change ingrid features that arise from bilinear/trilinear interpolation to appropriate frequen-cies that should be learned at each scale level. We then compose these grid fea-tures together through an MLP with sine activation functions, which takes thesefeatures as input at each layer, forming a pipeline that sequentially accumulateshigher-frequency information as composition is performed as shown in Fig. 1.2.To facilitate training, we initialize each layer of the MLP with the target frequencyband in mind. Finally, we sum up all intermediate outputs together to form theestimated field value.We demonstrate the effectiveness of our method under three different tasks: 2Dimage fitting, 3D shape reconstruction, and Neural Radiance Fields (NeRF). We3show that our method achieves a better trade-off between the model compactnessversus reconstruction quality than the state of the arts. We further perform anextensive ablation study to verify where the gains are coming from.To summarize, our contributions are as follows:• we propose a novel framework that decomposes the modeled signal both spatiallyand frequency-wise;• we show that our method achieves better trade-off between quality and mem-ory on 2D image fitting, 3D shape reconstruction, and Neural Radiance Fields(NeRF);• we provide an extensive ablation study shedding insight into the details of ourmethod.4Chapter 2Related WorkOur work is in line with those that apply neural fields to model spatial-temporal sig-nals [7, 8, 36, 47, 48, 50, 58]. In this section, we survey representative approacheson neural field modeling [3, 42, 49, 50, 72, 73] and provide an overview of workon incorporating the wavelet transform into deep network designs [11, 19, 30].2.1 Neural FieldsNeural Field is a type of network which takes a coordinate as input and is trained toparameterize physical properties of scenes or objects across space and time. Thusit is also referred as the coordinate-based network. A compressive survey can befound in [83]. Here we briefly discuss representative work. The CompositionalPattern Producing Networks (CPPNs) make the first attempt to treat coordinates asinput and combine various non-linear activations within a genetic algorithm frame-work [67]. While existing methods have achieved impressive performance on mod-eling various signals that can be represented as fields [12, 23, 24, 47, 49, 50, 53, 72],neural fields can still fall short of representing the fine details [24], or incur highcomputational cost due to model complexity [31]. Prior works attempt to solvethese problems by frequency transformations [48, 65, 73] and grid-based encod-ings [24, 49, 72].For frequency transformations [49], Vaswani et al. [76] encode the input fea-ture vectors into a high-dimension latent space through a sequence of periodic5functions. Tancik et al. [73] carefully and randomly choose the frequency of theperiodic functions and reveal how they affect the fidelity of results. Sitzmann etal. [65] propose to use periodic activation functions instead of encoding featurevectors. [18, 39] further push analysis in terms of the spectral domain with amulti-scale strategy, improving the capability in modeling band limited signals inone single model. To further understand the success of these methods, [4, 90] ana-lyze the implicit representations from the perspective of a structured dictionary andFourier series, respectively.For grid-based encodings [49, 72], the core idea is to encode the input tothe neural field by interpolating a learnable basis consisting of grid-point features(space partitioning). A distinctive benefit of doing so is that one can trade memoryfor faster training—bigger networks can be used to represent complex scenes, aslong as the entire grid used is within memory. To reduce this memory footprint,compact hash tables [49] and volumetric matrix decomposition [10] have beenintroduced. These recent methods, however, do not, at the very least explicitly,consider how grid resolutions and frequency interact.Thus, some works try to combine both directions. For example, SAPE [26]progressively encodes the input coordinates by attending to time-spatial informa-tion jointly. Mehta et al. [46] decompose the inputs into patches, which are usedto modulate the activation functions. They, however, utilize a single space resolu-tion, limiting their modeling capability. Instead, we show that by using multiplescale levels, and a framework that takes into account the frequencies that are tobe associated with these levels, one can achieve faster convergence with higheraccuracy.2.2 Neural Radiance FieldsThe Neural Radiance Field (NeRF) [48] employs a neural field to synthesize newperspectives of intricate 3D environments, utilizing a limited collection of 2D im-ages. This technique involves utilizing input images that portray a scene, and thenutilizing interpolation to generate a comprehensive representation of the scene.Since its introduction in 2020, NeRFs have gained popularity due to their poten-tial for photo-realistic and immersive 3D scene reconstruction from calibrated im-6ages. The concept of neural volumetric rendering, as exemplified by NeRFs, hassparked a surge of subsequent research endeavors. These subsequent works delveinto leveraging this idea to enable a multitude of exciting applications and tacklefundamental challenges in various domains, including computer vision, graphics,robotics, and beyond.Several works have aimed to enhance the standard NeRF model. To improvesynthesis quality, Mip-NeRF [1] introduced Integrated Positional Encoding (IPE)and used cone tracing instead of ray tracing for volume rendering. Mip-NeRF360 [2] extended Mip-NeRF to handle unbounded scenes, while Ref-NeRF [77]focused on better modeling reflective surfaces. PointNeRF [85] proposed usingfeature point clouds as an intermediate step for supervision, relying on expecteddepth rather than just RGB images. This approach led to faster convergence andhigher final quality.In recent times, significant efforts have been dedicated to improving the train-ing and inference speed of NeRF models. One approach is the training, precom-putation, and storage of NeRF MLP evaluation results in accessible data struc-tures, resulting in substantial improvements in inference speed [9, 21, 59, 88]. Onthe other hand, alternative methods like InstantNGP [49] and TensoRF [10] fo-cus on learning separate scene features from the MLP parameters. This approachenhances both training and inference speed, although it comes at the expense ofincreased memory usage.In addition to the advancements mentioned earlier, there have been notable pro-gressions in other NeRF-related areas. These include: 1) Training NeRF modelswith few-shot or sparse training views: Approaches like PixelNeRF [89], MVS-NeRF [5], and SinNeRF [84] explore techniques to effectively train NeRF modelsusing limited training views. These methods enable accurate scene reconstructioneven with sparse data. 2) Generative NeRF models for 3D geometry generation:Works such as NeRF-VAE [34], DreamFusion [56], and Magic3D [38] focus onconditioning NeRF models on texts, images, or latent codes to generate diverseand realistic 3D content. These advancements highlight the progress in trainingNeRF models with limited data and the development of generative NeRF models,expanding the capabilities and applications of NeRF-based techniques.72.3 Wavelets in deep netsWavelet transform is a widely adopted approach for efficiently representing com-plex signals, such as images, through a decomposition process that expresses sig-nals as a linear combination of translated and scaled versions of a short pulse,known as a wavelet. Unlike Fourier transform, wavelet transform offers finite spacesupport, allowing for faster approximation speed. As a result, it finds extensive ap-plications in diverse fields such as solving ordinary differential equations [71], im-age compression [13, 63], motion detection and tracking [69], filter design [14, 68],as well as other image processing [25]and video processing tasks [78].The use of wavelet transforms has been well-studied in the deep learning lit-erature. For example, they have been used for wavelet-based feature pooling op-erations [20, 41, 80], for the improvements on style transfer [19, 87], for denois-ing [40], for medical analysis [32], and for image generation [29, 44, 55, 79, 79].Recently, Liang et al. [37] reproduced wavelets through linearly combining ac-tivation functions. Gauthier et al. [22] introduce wavelet scattering transform tocreate geometric invariants and deformation stability. Phung et al. [55] use Haarwavelets with diffusion models to accelerate convergence. In the 3D vision do-main, De Queiroz et al. [11] propose a transformation that resembles an adaptivevariation of Haar wavelets to facilitate 3D point cloud compression. Isik et al. [30]directly learn trainable coefficients of the hierarchical Haar wavelet transform, re-porting impressive compression results. Concurrently, Rhoet al. [60] propose usingwavelet coefficients to improve model compactness.While our work shares a similar spirit as those that utilize wavelets, to the bestof our knowledge, ours is the first work aimed at a general-purpose neural fieldarchitecture that jointly and explicitly models the spatial and frequency domains.Meanwhile, our work stands out from previous wavelet-based approaches by tak-ing a different approach to coefficient computation in wavelet space. Instead ofexplicitly calculating coefficients, we establish a direct association between thespatial cell size and a specific frequency transformation coefficient. Furthermore,we iteratively merge the representations of low-frequency and high-frequency com-ponents in a multi-scale fashion. This unique strategy allows us to achieve both fastrunning speed and high-quality signal reconstructions simultaneously. By bypass-8ing explicit coefficient computation and leveraging the direct relationship betweenspatial cell size and frequency transformation, we optimize the efficiency of ourapproach without compromising the accuracy and fidelity of the reconstructed sig-nals.9Chapter 3MethodIn this work, we aim for a multi-resolution grid-based framework that also ties inthe frequency space to these grids, as is done with wavelets, and an architectureto effectively reconstruct the original signal. As shown in Fig. 1.2, we constructour pipeline, neural Fourier filter bank, composed of two parts: a Fourier-spaceanalogous version of grid features (Sec. 3.1); and an MLP that composes the finalsignals from these grid values (Sec. 3.2). We discuss these in more detail in thefollowing subsections.3.1 The Fourier grid featuresAs discussed earlier in Chapter 1, we use a grid setup to facilitate the learning ofhigh-frequency components via locality. Specifically, we aim for each grid level inthe multi-grid setup to store different frequency bands of the field that we wish tostore in the neural network. The core idea in how we achieve this is to combinethe typical grid setup used by, e.g., [49], with Fourier features [73], which we theninitialize appropriately to naturally encourage a given grid to focus on certain fre-quencies. This is analogous to how one can control the frequency details of a neuralfield by controlling the Fourier feature [73] encoding of the input coordinates, buthere we are applying it to the grid features.In more detail, the grid feature at the i-th level is defined as a continuous map-10ping from the input coordinate x P Rn to m dimension feature space:κi : RnÑ Rm. (3.1)We set n “ 2, 3 for 2D images and 3D shapes respectively. As shown in Fig. 3.1,κi consists of two parts: a lookup tableΦi which has Ti feature vectors with dimen-sionality F ; and a Fourier feature layer [73] Ωi.3.1.1 Multi-scale gridWe apply a trainable hash table [49] to implement Φi for a better balance betweenperformance and quality. For the i-th level, we store the feature vectors at thevertices of a grid, the resolution of which Ni is chosen manually. To utilize this gridin a continuous coordinate setup, one typically performs linear interpolation [49,72]. Hence, for a continuous coordinate x, to get the grid points, for each dimensionwe first scale x by Ni before rounding down and up, which we write with a slightabuse of notation (ignoring dimensions) as:txiu“ tx ¨Niu , rxis“ rx ¨Nis . (3.2)Here, txiu and rxis, for example occupies a voxel with 2n integer vertices. As in[49], we then map each corner vertex to an entry in the matching lookup table,using a spatial hash function [49, 75] as:hpx¯q “#nľi“1x¯i ¨Πi+mod Ti, (3.3)where x¯ represents the position of a specific corner vertex,Źdenotes the bit-wiseXOR operation and Πi are unique, large prime numbers. As in [49], we chooseΠ1 “ 1, Π2 “ 2654435761 and Π3 “ 805459861.Finally, for x, we perform linear interpolation for its 2n corner feature vectorsbased on their relative position to x within its hypercube as wi “ xi´ txiu. Specif-ically, we use bilinear interpolation for 2D image fitting and trilinear interpolationfor 3D shape modeling. We denote the output features through the linear interpo-lation over the lookup table Φi as ϕpx;Φiq.11It is important to note that this linear interpolation operation makes these fea-tures behave similarly to how the input coordinates affect the neural field out-put [73]—introducing bias toward slowly changing components. Thus, in orderfor each grid level to focus on appropriate frequency bands it is necessary to ex-plicitly take this into account.3.1.2 Converting grid features to Fourier featuresThen, to associate the spatial area with the specific frequency level, we applyFourier feature encoding to vi “ ϕpx;Φiq before we utilize them:γ ipviq “ rsinp2π ¨Bi,1 ¨vJi q, . . . ,sinp2π ¨Bi,m ¨vJi qsJ, (3.4)where tBi,1,Bi,2, ¨ ¨ ¨ ,Bi,mumeans trainable frequency transform coefficients on i-thlevel. We then utilize γ ipviq in our network that converts these into desired fieldvalues.Importantly, we directly associate the frequency band on the i-th level with de-sired grid size by explicitly initializing tBi,1,Bi,2, ¨ ¨ ¨ ,Bi,mu with adaptive Gaussiandistribution variance similarly to Gaussian mapping [73, Sec. 6.1]. We choose toinitialize with different variances, as it is difficult to set a specific frequency rangefor a given grid a priori. Instead of trying to set a proper range that is hard toaccomplish, we initialize finer grids with larger variance and naturally bias finergrids towards higher frequency components since the multiplier for v will then belarger—they will be biased to converge to larger frequencies [26].3.2 Composing the field valueTo compose the field values from our Fourier grid features, we start from twoimportant observations:• The stored Fourier grid features at different layers, after going through a deepnetwork layer for interpretation, are not orthogonal to each other. This calls forthe need for learned layers when aggregating features from different levels sothat this non-orthogonality is mitigated.• The Fourier grid features should be at a similar ‘depth’ so that they are updated12+ ++L1O1L2O2L3O3X+L Fully Connected LayerO Output LayerFourier Feature LayerInterpolationOutput SignalFigure 3.1: Framework overview – Based on the input query, e.g., the posi-tion x, our neural Fourier filter bank uses both a grid and a Multi-LayerPerceptron (MLP) to compose the final estimate. Specifically, grid fea-tures are extracted via interpolation at multiple scale levels, which arethen encoded to appropriate frequencies for each layer via the FourierFeature layers. The MLP uses these encoded features as the higher-frequency component in Fig. 1.2, while the earlier layer outputs as thelower frequency ones, similarly to wavelet filter banks. Intermediateoutputs are then aggregated as the final estimate.simultaneously. This makes residual setups preferable.We thus utilize an MLP, which takes in the Fourier grid features at various layers.As shown in Fig. 3.1, each layer takes in features from the previous layer, as wellas the Fourier grid features, then either passes it to the next layer or to an outputfeature that is then summed up to form a final output.Mathematically, denoting the MLP as a series of fully-connected layers L “tL1,L2, ¨ ¨ ¨ u, we writefi “ sinpαi ¨Wigi´1`biq, gi “ fi` γ ipviq, (3.5)where Wi and bi are trainable weight and bias in the i-th layer Li, and αi is thescaling factor for this layer that control the frequency range that this layer focuseson, which is equivalent to the w0 hyperparamter in SIREN [65]. Note here that ficorresponds to the output of the lower-frequency component, and the Fourier gridfeatures γ ipviq are the higher-frequency ones in Fig. 1.2. For the first layer, as there13is no earlier level, we use the input position x. Thus,f1 “ sinpα1 ¨W1x`b1q, g1 “ f1` γ1pv1q. (3.6)Then, with gi, we construct the per-level outputs oi “Woi gi`boi with output layersO“ tO1,O2, ¨ ¨ ¨ u with another trainable parameters settWo1,Wo2, ¨ ¨ ¨ ,bo1,bo2, ¨ ¨ ¨ u. We then sum up oi to obtain the final estimated fieldvalue asF pxq “ři“1 oi.Importance of the composition architecture. A simpler alternative to composingthe field signal estimate would be to simply use Fourier grid features in an existingpipeline [49, 72] that utilizes grids. However, as we will show in Sec. 4.4, this re-sults in consistently inferior performance compared to our method of composition.3.3 Implementation details3.3.1 More details about Fourier grid featuresIn Sec. 3.1 of the main text, the multi-level Fourier grid features are defined tocompute the continuous mapping between the input coordinate x P Rn and the mdimension feature space. Following instantNGP, we set the base resolution Nminand a scaling coefficient cg between adjacent levels to define the resolution for acertain level l as:Nl “ Nmin ¨ clg, (3.7)where the level index l starts from 0. We adjust the total number of levels to balancethe ability to model fine details and the complexity of the model itself. To computethe variance values that we use to initialize the Fourier features, we apply a similarscaling strategy:σl “ σmin ¨ clf , (3.8)where σmin and c f represent the base variance value and its corresponding scalingcoefficient. Roughly, we set σmin “ 2?Nmin and cg « c f . However, the optimalchoice of these values is circumstantial and we modify Nmin, cg, σmin and c f foreach task.143.3.2 Common ImplementationsDepending on the target applications, some implementation details vary—the lossfunction, the number of training iterations, and the network capacity are task de-pendant and we elaborate on them later in their respective subsections. Other thanthe task-specific components we keep the same training setup for all experiments.We implement our method in PyTorch [52]. We use the Adam optimizer [33] withdefault parameters β1 “ 0.9 and β2 “ 0.99. We use a learning rate of 10´4, anddecay the learning rate to half every 5,000 iterations. We set the dimension of gridfeatures as F “ 2. We train our method on a single NVidia RTX 3090 GPU. Here,for brevity, we note only the critical setup for each experiment. For more details onthe hyperparameter and detailed experimental settings under different evaluationtasks, please see the Sec. 4.1.3, Sec. 4.2.3 and Sec. 4.3.1 in Chapter 4.15Chapter 4Experimental ResultsWe evaluate our method on three different tasks: 2D image fitting (Sec. 4.1), 3Dshape reconstruction using signed distance functions (Sec. 4.2), and novel viewsynthesis using NeRF (Sec. 4.3). Ablation study is shown in Sec. 4.4. More exper-iment discussions can be found in the appendix.4.1 2D Image FittingWe commence by confirming the efficacy of our approach through the 2D ImageFitting task. Our objective is to employ a single neural network to accurately matchextensive 2D images, thereby demonstrating the network’s proficiency in model-ing signals across broad frequency spectra. To achieve it, each model takes a 2Dposition as input and is trained to regress the RGB value at that location with meansquared error. Hence, our loss function for this task isLimg “ }y´ygt}22 , (4.1)where y is the neural field estimate and ygt is the ground-truth pixel color.4.1.1 DataTo keep our experiments compatible with existing work, we follow ACORN [45]and evaluate each method on two very high-resolution images. The first image16Ground Truth SIREN ModSineSAPEInstantNGP OursFigure 4.1: 2D Fitting – Qualitative results for the Tokyo image. Our methodprovides the best reconstruction quality at various scale levels, fromnearby regions to far away ones, demonstrating the importance of con-sidering both space and frequency jointly.is a photo of ‘Einstein’1, already shown in Fig. 1.1. This image has a resolutionof 3250ˆ 4333 pixels, with varying amounts of details in different regions of theimage, making it an interesting image to test how each model is capable of repre-senting various levels of detail—background is blurry and smooth, while the eyeand the clothes exhibit high-frequency details. Another image is a photo of thenightscape of ‘Tokyo’ [45] with a resolution of 6144ˆ 2324, where near and farobjects provide a large amount of detail at various frequencies.1Collected from https://github.com/NVlabs/tiny-cuda-nn.17Tokyo AlbertSize (MB)Ó PSNRÒ SSIMÒ LPIPSÓ Size (MB)Ó PSNRÒ SSIMÒ LPIPSÓInstantNGP [49] 36.0 33.38 0.9452 0.201 3.7 41.61 0.9623 0.152SIREN [65] 5.2 28.52 0.8921 0.474 5.0 42.51 0.9661 0.478SAPE [26] 3.2 21.64 0.5357 0.745 3.2 34.26 0.9219 0.399ModSine [46] 3.5 23.23 0.7587 0.607 4.2 36.74 0.9184 0.438Ours 10.0 33.62 0.9555 0.141 3.7 43.83 0.9763 0.142Table 4.1: 2D Fitting – We report the reconstruction comparisons in termsof Peak Signal-to-Noise Ratio (PSNR), Structural Similarity IndexMetric (SSIM) [27] and Learned Perceptual Image Patch Similarity(LPIPS) [91]. Our method provides the best trade-off between modelsize and reconstruction quality.T “ 217 T “ 219 T “ 221L“ 8 L“ 10 L“ 12 L“ 8 L“ 10 L“ 12 L“ 8 L“ 10 L“ 12InstantNGP 28.18 30.89 31.93 29.38 33.37 36.41 30.41 36.37 41.28Ours 30.31 32.86 33.82 31.28 34.53 37.56 31.43 36.86 41.36Table 4.2: Performance under varying T and L – Our method shows higherPSNR values for ‘Tokyo’ image with various T and L settings.4.1.2 BaselinesWe compare our method against four different baselines designed for this task: In-stantNGP [49], which utilizes grid based space partitions for the input; SIREN [65],which resembles modeling the Fourier space; and two methods (SAPE [26] andModSine [46]) that consider both the frequency and the space decomposition butnot as in our method. For all methods, we use the official implementation bythe authors but change their model capacity (number of parameters, and grid/hashtable size) and task-specific parameters. Specifically, for SIREN, we set the fre-quency parameter ω0 “ 30.0 and initialize the network with 5 hidden layers withsize 512ˆ 512. For SAPE, we preserve their original network size. For Instant-NGP, we adjust its maximum hashtable size as T “ 217 and the grid level to L“ 8for the ‘Einstein’ image and set T “ 219 and L “ 16 for ‘Tokyo’ to better caterto complex details. To allow all models to fully converge, we report results after50,000 iterations of training.18Experimental setting for InstantNGP. Generally, our choices are based [49, Sec. 3.].As shown in [49, Fig. 5.], F“2 and L“16 are good choices for the feature dimen-sion F and feature level L. For the hash table size T , we choose 219 as it is whenthe performance starts being throttled as shown in [49, Fig. 4.]. For the ‘Einstein’image in Tab. 4.1 of the main text, we reduced the model’s capacity as the imageis simpler.In addition, we use various settings for T and L for both InstantNGP and ourmethod and report the results for ‘Tokyo’ image in Tab. 4.2. Regardless of thehyperparameter settings, our method outperforms InstantNGP consistently.4.1.3 Experimental SettingsTo roughly match the model capacity used by other methods, for the ‘Tokyo’ imagewe use fully-connected layers with 128 neurons, and for the ‘Einstein’ image 256.All fully connected layers (tLiu) are using sine activations as previously describedin the main text. However, we apply a two-layered fully-connected layers with Reluactivation as the output layer (tOiu) to avoid high-frequency artifacts. Additionally,for the ‘Tokyo’ image we use Nmin “ 64, cg “ 1.5, σmin “ 5.0 and c f “ 2.0, and forthe ‘Einstein’ image we use Nmin“ 64, cg“ 2.0, σmin“ 10.0 and c f “ 2.0. For bothimages, we train our network with 50,000 iterations to ensure full convergence—our method already converges after 20,000 iterations. To well-reconstruct complexhigh-frequency signals, we set αi in Eq. (3.5) to 100.4.1.4 ResultsWe provide qualitative results for the ‘Tokyo’ image in Fig. 4.1, and report thequantitative metrics in Tab. 4.1. As shown, our method provides the best tradeoffbetween model size and reconstruction quality, both in terms of Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Metric (SSIM) [27], and LearnedPerceptual Image Patch Similarity (LPIPS) [91]. Among these, note that the gapin performance is larger with SSIM and LPIPS, which better represents the localstructure differences. This is also visible in Fig. 4.1, where our method providesresults that are nearly indistinguishable from the ground truth.We note that the importance of considering both frequency and space is well19OursGround TruthFigure 4.2: 2D fitting result for ‘Einstein’ image. Our entire reconstructedimage is presented on the left while four close-up views are presentedon the right. Note how our reconstructions are near-perfect for bothcoarse and fine details.exemplified in Fig. 4.1. As shown, while InstantNGP provides good details fornearby regions (second row), as further away regions are investigated (third andlast row), artifacts are more visible. This demonstrates that even when multiscalegrid is used, without consideration of the frequencies associated with these scales,results degrade. Other baselines, SIREN, ModSine, and SAPE, are all single-scaleand show results as if they are focusing on a single frequency band. Ours on theother hand does not suffer from these artifacts.Moreover, in Fig. 4.2 and Fig. 4.3, we provide more detailed look into the2D reconstruction results. Both results provide highly impressive reconstructions,without any discernable differences to the ground truth.4.2 3D Shape ReconstructionWe further evaluate our method on the task of representing 3D shapes as signeddistance fields (SDF). For this task, we input a set of 3D positions and train thenetworks to regress the signed distance field values with the square of the MeanAbsolute Percentage Error (MAPE) [49] as training objective, to facilitate detailmodeling. We thus train models by minimizing the loss:Lsd f “ }y´ygt}22 {´ε`}ygt}22¯, (4.2)20Figure 4.3: 2D fitting result for ‘Tokyo’ image. Our entire reconstructed im-age is presented on the top while four close-up views are presented onthe bottom. Our method provides near-perfect reconstruction.where ε denotes a small constant to avoid numerical problems, y is the neural fieldestimate, and ygt is the ground-truth SDF value.4.2.1 DataFor this task, we choose two standard textured 3D shapes for evaluation: ‘BeardedMan’ (with 691K vertices and 1.38M faces); and ‘Asian Dragon’ (3.6M verticesand 7.2M faces). Both shapes exhibit coarse and fine geometric details. Whentraining with these shapes, we sample 3D points x P R3 with a 20/30/50 split—20% of the points are sampled uniformly within the volume, 30% of the points are21Figure 4.4: 3D Fitting – Qualitative comparisons for the ‘Bearded Man’shape. Our method is the most compact among the compared methods,and is capable of reconstructing both coarse and fine details without ob-vious artifacts.sampled near the shape surface, and the rest sampled directly on the surface.To obtain the ground-truth SDF values, we use pysdf2. We use the originalmesh files and normalize them into a unit sphere to standardize shapes. Whentraining each model, for each batch, we sample 49152 points for training where20% of the points are sampled uniformly within the volume, 30% of the points aresampled near the shape surface, and the rest are sampled directly on the surface.4.2.2 BaselinesWe compare against the same baselines as in Sec. 4.1, and additionally BACON,which also utilizes frequency decomposition for efficient neural field modeling.For BACON and SIREN, we use networks with 8 hidden layers and 256 hiddenfeatures, and again ω0 “ 30.0 for SIREN. For ModSine, we set the grid resolutionas 64ˆ 64ˆ 64 and apply 8 hidden layers and 256 hidden features for both themodulation network and the synthesis network. For SAPE and InstantNGP, use theauthor-tuned defaults for this task. All models are trained for 100K iterations forfull training.4.2.3 Experimental SettingsFor this task, we train our network for 50,000 iterations on 26 million sampledpoints with a batch size of 49,152, to maximize GPU memory utilization. Note that,2Github link: https://github.com/sxyu/sdf22Figure 4.5: 3D fitting result for the ‘Asian Dragon’. The left sub-image is theground truth shape while six zoomed insets are shown on the right forbetter detail visualizations.training models with the same learning rate decay strategy for 50 epochs whereeach epoch occupies 100 iterations can produce similar results. But longer trainingyields more detailed geometry textures. As the SDF has varying level-of-detail—e.g., , smooth regions can be very smooth, while detailed regions can have high-frequency detail—we set the number of levels to five for the Fourier grid feature.For each level, we use fully-connected layers (tLiu) with 256 neurons. To keepthe consistency between the Sine-activated fully connected layers and the outputlayers tOiu, we initialize tOiu with the same αi as in tLiu. We further set Nmin “ 8,cg “ 1.3, σmin “ 5 and c f “ 1.2. For both shapes in Tab. 4.3, we choose αi “ 45,which we empirically found to provide the best balance between high- and low-frequency details for this task.4.2.4 ResultsWe present our qualitative results in Fig. 4.4-Fig. 4.5 and report quantiative scoresin Tab. 4.3. To extract detailed surfaces from each implicit representation we applymarching cubes with a resolution of 10243. As shown, our method provides thebest performance, while having the smallest model size. Note that in Tab. 4.3 ourresults are worse in terms for F-score for the Asian Dragon, while the other metricsreport performance comparable to InstantNGP with 30ˆ smaller model size. Thelower F-score but higher Chamfer distance is due to our model having lower recallthan BACON, which provides more blobby results, as demonstrate by the IoU andChamfer distance metrics. We also note that for the ‘Bearded Man’, our method23Size (MB)Ó Asian Dragon Bearded ManF-scoreÒ IoUÒ Cham distÓ F-scoreÒ IoUÒ Cham distÓInstantNGP [49] 46.5 0.8714 1.0 0.00191 0.999 0.9970 0.00272SIREN [65] 2.0 0.8593 0.998 0.00234 0.997 0.9951 0.00302BACON [39] 2.0 0.9200 0.995 0.00242 0.716 0.9932 0.00285SAPE [26] 3.2 0.3210 0.959 0.00584 0.284 0.9837 0.00438ModSine [46] 12.0 0.6892 0.995 0.00238 0.873 0.9952 0.00307Ours 1.4 0.8717 1.0 0.00189 0.999 0.9985 0.00272Table 4.3: 3D Fitting – We report the Intersection over Union (IoU), F-Scoreand Chamfer distance (CD) after performing marching cubes to extractsurfaces. Our method performs best, with the exception of F-score on‘Asian Dragon’, which is due to BACON preferring blobby output, asdemonstrated by the higher Chamfer distance and worse IoU.outperforms all other methods.This difference in quantitative metrics is also visible in Fig. 4.4. As shown, ourmethod provides high-quality reconstruction for both zoomed-in regions, whereasother compared methods show lower-quality reconstructions for at least one ofthem. For example, SIREN provides good reconstruction for the beard region(second row), but not for the region around the ears (top row), where sinusoidalartifacts are visible. InstantNGP also delivers high-quality reconstruction for the‘Bearded Man’, but with much higher memory requirement.In Fig. 4.5, we further provide the qualitative results for regressing the ‘AsianDragon’ shape SDF. Our method and InstantNGP both provide results with veryfine details, but ours is more compact.4.3 Novel View SynthesisAs our last task, we apply our method to modeling Neural Radiance Fields (NeRF) [48].Because we are interested in comparing the neural field architectures, not the NeRFmethod itself, we focus on the simple setup using the synthetic Blender dataset.We train all architectures with a pure NeRF setup [48], where volumetric ren-dering is used to obtain pixel colors, which are then compared to ground-truth24Ground Truth NeRF DVGO InstantNGP OursFigure 4.6: Novel View Synthesis – Although more compact, our methodcan synthesize comparable or better results.Steps Size (MB) Ó Time Ó PSNR Ò SSIM ÒNeRF[48] 300k 5.0 ą 30h 31.01 0.947Plenoxels [61] 128k 778.1 11.4m 31.71 0.958DVGO [70] 30k 612.1 15m 31.95 0.957InstantNGP [49] 30k 46.6 3.4m 32.08 0.955Ours 30k 14.7 13.1m 32.04 0.955Table 4.4: Neural Radiance Fields (NeRF) – We report the novel view ren-dering performance in terms of Peak Signal-to-Noise Ratio (PSNR) andStructural Similarity Index Metric (SSIM). Our method provides compa-rable rendering quality as the state of the art, while having the smallestsize among the grid-based methods (middle rows) that provide fast train-ing, providing the best trade-off between quality and model size. See theappendix for runtime discussions.values for training. Specifically, a pixel color is predicted asCˆ prq “nÿi“1Ti p1´ expp´σiδiqqci,Ti “ expp´i´1ÿj“1σ jδ jq,(4.3)where ci and σi denote the color and density estimated at the i-th queried locationalong the ray r and δi is the distance between adjacent samples along a given ray.25Then, the mean-squared loss for training is:Lrec “ÿrPR››Cˆprq´Cgtprq››22 , (4.4)where R is the whole ray set and Cgt is the ground truth.4.3.1 Network Adaptation & Experimental SettingFor this task, we found that the complexity of the task, estimating both the colorand the density, requires appending our pipeline with an additional MLP that de-codes deep features into either the color or the density. Thus, instead of directlyoutputting these values from our framework, we output a deep feature, which isthen converted into color and density. Specifically, as in NeRF [48], we apply two64ˆ 64 linear layers to predict density value and a low-dimension deep feature,which is further fed into three 64ˆ64 linear layers for RGB estimation.Additionally, we closely follow the experimental setup of InstantNGP [49],including the four levels for the grid. For our method, we use 128 neurons in eachfully-connected layer (tLiu) to match a similar model capacity as the baseline. Wefurther set αi “ 20.0, Nmin “ 64, cg “ 2.0, σmin “ 8.0,c f “ 1.5 to balance modelcomplexity and synthesis quality.4.3.2 BaselinesWe compare against five baselines: NeRF [48] which utilizes the frequency domainvia positional encoding; Plenoxels [61], DVGO [70], and InstantNGP [49], whichare grid-based methods.4.3.3 ResultsIt is shown in Fig. 4.6 that both our method and InstantNGP can successfully re-generate the circular patterns on 1st row and the high-frequency geometric detailson 2nd row. We also report the quantitative metrics in Tab. 4.4. Our method pro-vides similar performance as other state-of-the-art methods qualitatively and quan-titatively, but with a much smaller model size, showing our empirical effectiveness.We provide more qualitative results for novel view synthesis in Fig. 4.7. As26Figure 4.7: Qualitative results for novel view synthesis with neural radiancefields. Our method is able to clearly reconstruct the textures (e.g., , thechair on 2nd row) and the geometric details (e.g., the lego on the lastrow).27Figure 4.8: Ablation study – We compare against variants of our methodwith the Fourier grid feature and/or the proposed MLP composition ar-chitecture disabled. Having both components together is critical forperformance.shown, our method is able to provide synthesis results with both low-frequencydetails as shown by the ‘Lego’ scene, and high-frequency details as shown by the‘Ficus’ scene with thin structures.4.4 Ablation StudyAs discussed in Chapter 3 of the main text, our key idea is the Fourier grid features,and the wavelet-inspired composition. Here, we further justify our design choicesbased on experiments. Note that, for the ‘Tokyo’ image, the PSNR scores arecomputed based on the intermediate ‘image’ data instead of the raw images as inTab. 4.1.4.4.1 General Design JustificationTo justify the design choices of our method we explore three variants of our method:our method where only Grid features are used as ‘Only Grid’; our method withGrid and the Fourier features encoding as ‘Grid+FF’; and finally when only us-ing the MLP architecture for composition without the grid as ‘Only MLP’. For afair evaluation of the effects of the MLP part, we adjust the ‘Only MLP’ model topossess similar number of trainable parameters as the full model. We report ourresults for the ‘Tokyo’ image in Fig. 4.8. As shown, all variants perform signifi-cantly worse. Interestingly, simply applying Fourier Features to the grid does nothelp, demonstrating the proposed MLP architecture is also necessary to achieve itspotential.28PSNRIterationsPSNRIterationsFigure 4.9: Ablation studies for the Fourier feature variance (left) and thescaling factor α in fully-connected layers (right). Both parametershighly affect how frequency is dealt with within our framework, andthus require optimal values to be set. These parameters are mostly taskdependant.4.4.2 The effect of grid resolutionsIn Chapter 1, we discuss how grid resolution relates to what frequency range amodel can reconstruct. In Fig. 4.10, we illustrate that this is indeed the case byvarying Nmin. Also in Fig. 4.11, we show how the scaling factor cg affects finalresults. As expected, whether fine details are preserved or not depends highly onthe two parameters.4.4.3 The effect of the Fourier feature variance.In Sec. 3.1, we discuss how our initialization strategy leads to the natural biasing offrequency components. Thus, this variance has a strong impact on the performanceof the method—too high variance would lead to the method focusing only on high-frequencies, while too low variance would cause the opposite. Thus, this varianceshould be selected with care. In Fig. 4.9, we show how the variance σmin affectsthe final reconstruction performance—σmin should roughly be in a proper range, as29Figure 4.10: The ablation study for the base resolution Nmin. With largerNmin, fine details are better preserved.Figure 4.11: The ablation study for the scaling coefficient cg. With larger cg,reconstruction quality improves, with more fine details being preservedand with higher spatial resolution.demonstrated by the σmin “ 1 and σmin “ 8 results.4.4.4 The effect of the scaling factor αi.Similarly, αi is another parameter that highly impacts how each layer combinesgrid features and the features from the previous layer. We set a single global valuefor all layers for simplicity, and experiment with multiple values to demonstrateits effect in Fig. 4.9. As expected, a properly tuned value is necessary for optimalperformance. We found this parameter to be highly task dependant.4.4.5 The effect of Fourier encodingsWe also demonstrate the influences of applying Fourier encodings to the low di-mensional grid features by only preserving the Grid+MLP components. We trainthis ablated model for the ‘Tokyo’ image which gives the PSNR of 30.48, whereasthe full model yields 31.57. To further evaluate the effects of activations for gridfeatures, we implement by replacing the sine activation functions with Relu andproduce 30.39, highlighting the necessity of current design choices.30SSIM PSNRIterations Iterations Size 32 Size 64 Size 128 Size 256 Ground TruthFigure 4.12: The ablation study results for the MLP size. As the MLP sizeincreases, the network becomes better at composing signals from var-ious levels, thus various frequencies, leading to a better final outcome.Figure 4.13: The ablation study for the number of levels for the Fourier gridfeature. More levels lead to a drastic increase in the quality of finedetails.4.4.6 The effect of fully-connected layer size.The size of the MLP also plays an important role, as it allows for more complexcomposition of signals coming from different frequencies. In Fig. 4.12, we illus-trate the importance of the MLP size— the larger the better, but with an increase incomputation and model complexity.4.4.7 The effect of the Fourier grid level.Finally, we demonstrate how the number of Fourier grid levels affects our results.As expected, we observe in Fig. 4.13 that the models with higher grid levels con-sistently provide better results.4.5 Other Discussions4.5.1 Comparison with ModSineWe use the local representation with a tile size of 64 for 2D && 3D signal fitting,under the auto-decoding setup. For ModSine [46], we have taken the network from31the official implementation3 and included it in our training and evaluation code,to keep all training aspects identical to ours. We note, however, that our resultsmight not have optimal hyperparameter settings, as some of the experimental se-tups (layer number, layer size, batch size, and learning rate) were chosen by us asthey were unavailable in [46].4.5.2 Discussions about RuntimeAs shown in Tab. 4.4 of the main text, our current implementation is not utilizingCUDA libraries (e.g. tiny-cuda-nn4) in places other than the hash grid, thus slowerthan InstantNGP as of now. Our current implementation requires around 13 min-utes to train a Blender scene for the NeRF task, whereas InstantNGP takes around3–4 minutes. However, we suspect that with a more efficient implementation, forexample with a full CUDA-integrated implementation such as InstantNGP, wouldgreatly accelerate our method, as our method only introduces a few small linearlayers and Fourier Feature embedding layers, which should not increase the com-putation load significantly. Finally, recall that as shown in Fig. 1 of the main paper,our method converges faster in terms of number of optimization steps than othermethods, including InstantNGP.3Code from https://ishit.github.io/modsine/4Code from: https://github.com/NVlabs/tiny-cuda-nn32Chapter 5Conclusions and Future Work5.1 ConclusionsIn this work, we introduce the neural Fourier filter bank, inspired by wavelets,which enables high-quality reconstruction using more compact models. Our keyidea is to jointly decompose the space and frequency domains in an adaptive man-ner, similar to wavelet transformation. To achieve this, we explicitly decomposethe frequency domain of input signals into different bins. Within each bin, wefurther decompose the entire space domain into smaller regions. The finer spa-tial regions correspond to high-frequency features, while the coarser regions matchlow-frequency features.To realize our approach, we propose a novel neural field framework that incor-porates a group of learnable Fourier filters. For frequency decomposition, thesefilters are grouped into different levels, where each level corresponds to a specificfrequency bin. Fourier-based grid features are applied within each level for spatialdecomposition. Additionally, we introduce a Sine-activated fully connected layerthat transforms lower frequency features into higher frequency ones. To regress thetarget signals, we concatenate the features from all levels.Overall, the neural Fourier filter bank offers a fresh perspective for develop-ing a powerful neural field model. Through our investigation of various networkdesigns, we highlight the significance of Fourier grid features for spatial decompo-sition and Sine-activated fully connected layers as critical components that strike33the optimal balance between result quality and model compactness in tasks such as2D image reconstruction, 3D shape representation, and novel-view synthesis usingNeRF (Neural Radiance Fields) [48].5.2 Broader ImpactOur work is of fundamental nature and is not immediately linked to any particularapplication. However, our method would facilitate efficient neural field represen-tations, which can widen the potential application area of neural fields. In addi-tion, our method, being more efficient, would reduce the amount of computing andpower consumption required for the application of these methods.5.3 Future WorkOur work has some limitations that should be acknowledged. Firstly, we assume astationary neural field, which means it is not conditioned. However, our wavelet-inspired framework has the potential for broader applications beyond our currentevaluation tasks. Exploring the application of the Fourier filter bank frameworkwith specific network designs in other areas, such as dynamic NeRF [17, 51, 57],editing NeRF representations [35, 43], and data compression [15, 16], could be apromising direction for future research.Although our running speed is comparable to state-of-the-art methods like In-stantNGP [49], the inclusion of fully connected layers in our network introducesdense computation costs, which limit achieving optimal running efficiency. Sim-ilarly, using dense grids for grid features significantly increases the network size.Therefore, enhancing network training speed through more compact model designswould further expand the range of applications.Furthermore, we acknowledge that, like existing grid-based methods [10, 49,72], our approach encounters difficulties when modeling very large scenes or rep-resenting thin, volume-less geometric structures. Addressing challenges related toapplying the Fourier filter bank architecture to model city-level 3D scenes or tree-like geometric textures presents an additional and potentially interesting researchdirection.34Bibliography[1] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, andP. P. Srinivasan. Mip-NeRF: A Multiscale Representation for Anti-AliasingNeural Radiance Fields. 2021. Ñ pages 1, 7[2] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman.Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. 2022. Ñpage 7[3] I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. V. Le. Attention AugmentedConvolutional Networks. 2019. Ñ page 5[4] N. Benbarka, T. Ho¨fer, A. Zell, et al. Seeing Implicit Neural Representationsas Fourier Series. 2022. Ñ page 6[5] A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su. Mvsnerf:Fast generalizable radiance field reconstruction from multi-view stereo.2021. Ñ page 7[6] Y. Chen, S. Liu, and X. Wang. Learning Continuous Image Representationwith Local Implicit Image Function. 2021. Ñ page 1[7] Z. Chen and H. Zhang. Learning Implicit Fields for Generative ShapeModeling. 2019. Ñ page 5[8] Z. Chen, T. Funkhouser, P. Hedman, and A. Tagliasacchi. MobileNeRF:Exploiting the Polygon Rasterization Pipeline for Efficient Neural FieldRendering on Mobile Architectures. 2022. Ñ page 5[9] Z. Chen, T. Funkhouser, P. Hedman, and A. Tagliasacchi. MobileNeRF:Exploiting the Polygon Rasterization Pipeline for Efficient Neural FieldRendering on Mobile Architectures. 2023. Ñ page 7[10] Chen, Anpei and Xu, Zexiang and Geiger, Andreas and Yu, Jingyi and Su,Hao. TensoRF: Tensorial Radiance Fields. 2022. Ñ pages 1, 6, 7, 3435[11] R. L. De Queiroz and P. A. Chou. Compression of 3D point clouds using aregion-adaptive hierarchical transform. 2016. Ñ pages 5, 8[12] B. Deng, K. Genova, S. Yazdani, S. Bouaziz, G. Hinton, and A. Tagliasacchi.CvxNet: Learnable Convex Decomposition. 2020. Ñ page 5[13] R. A. DeVore, B. Jawerth, and B. J. Lucier. Image Compression ThroughWavelet Transform Coding . IEEE Transactions on information theory,1992. Ñ page 8[14] D. L. Donoho and I. M. Johnstone. Threshold selection for waveletshrinkage of noisy data. In Proceedings of 16th annual internationalconference of the IEEE engineering in medicine and biology society, 1994.Ñ page 8[15] E. Dupont, A. Golin´ski, M. Alizadeh, Y. W. Teh, and A. Doucet. COIN:COmpression with Implicit Neural representations. Int. Conf. Learn.Represent. Workshop, 2021. Ñ page 34[16] E. Dupont, H. Loya, M. Alizadeh, A. Golinski, Y. W. Teh, and A. Doucet.COIN++: Neural Compression Across Modalities. 2022. Ñ pages 1, 34[17] J. Fang, T. Yi, X. Wang, L. Xie, X. Zhang, W. Liu, M. Nießner, and Q. Tian.Fast Dynamic Radiance Fields with Time-Aware Neural Voxels. 2022. Ñpages 1, 34[18] R. Fathony, A. K. Sahu, D. Willmott, and J. Z. Kolter. Multiplicative FilterNetworks. 2021. Ñ page 6[19] R. Gal, D. C. Hochberg, A. Bermano, and D. Cohen-Or. SWAGAN: AStyle-based Wavelet-driven Generative Model. ACM Trans. Gr., 2021. Ñpages 5, 8[20] X. Gao and H. Xiong. A Hybrid Wavelet Convolution Network withSparse-Coding for Image Super-Resolution. 2016. Ñ page 8[21] S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin.FastNeRF: High-Fidelity Neural Rendering at 200FPS. 2021. Ñ page 7[22] S. Gauthier, B. The´rien, L. Alsene-Racicot, M. Chaudhary, I. Rish,E. Belilovsky, M. Eickenberg, and G. Wolf. Parametric Scattering Networks.2022. Ñ page 836[23] K. Genova, F. Cole, D. Vlasic, A. Sarna, W. T. Freeman, and T. Funkhouser.Learning Shape Templates with Structured Implicit Functions. 2019. Ñpage 5[24] K. Genova, F. Cole, A. Sud, A. Sarna, and T. Funkhouser. Local DeepImplicit Functions for 3D Shape. 2020. Ñ page 5[25] L. He and L. Carin. Exploiting Structure in Wavelet-Based BayesianCompressive Sensing. IEEE Transactions on Signal Processing, 2009. Ñpage 8[26] A. Hertz, O. Perel, R. Giryes, O. Sorkine-Hornung, and D. Cohen-Or.SAPE: Spatially-Adaptive Progressive Encoding for Neural Optimization.2021. Ñ pages 2, 6, 12, 18, 24[27] A. Hore and D. Ziou. Image Quality Metrics: PSNR vs. SSIM. 2010. Ñpages ix, 18, 19[28] T. Hu, S. Liu, Y. Chen, T. Shen, and J. Jia. EfficientNeRF: Efficient NeuralRadiance Fields. 2022. Ñ page 3[29] H. Huang, R. He, Z. Sun, and T. Tan. Wavelet Domain GenerativeAdversarial Network for Multi-scale Face Hallucination. 2019. Ñ page 8[30] B. Isik, P. Chou, S. J. Hwang, N. Johnston, and G. Toderici. LVAC: Learnedvolumetric attribute compression for point clouds using coordinate basednetworks. Frontiers in Signal Processing, 2021. Ñ pages 5, 8[31] C. Jiang, A. Sud, A. Makadia, J. Huang, M. Nießner, T. Funkhouser, et al.Local Implicit Grid Representations for 3D Scenes. 2020. Ñ page 5[32] E. Kang, J. Min, and J. C. Ye. A deep convolutional neural network usingdirectional wavelets for low-dose X-ray CT reconstruction. Medical physics,2017. Ñ page 8[33] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. 2014.Ñ page 15[34] A. R. Kosiorek, H. Strathmann, D. Zoran, P. Moreno, R. Schneider,S. Mokra´, and D. J. Rezende. NeRF-VAE: A Geometry Aware 3D SceneGenerative Model. 2021. Ñ page 7[35] Z. Kuang, F. Luan, S. Bi, Z. Shu, G. Wetzstein, and K. Sunkavalli.PaletteNeRF: Palette-based Appearance Editing of Neural Radiance Fields.2023. Ñ page 3437[36] M. Levoy and P. Hanrahan. Light Field Rendering. In SIGGRAPH, 1996. Ñpage 5[37] S. Liang, L. Lyu, C. Wang, and H. Yang. Reproducing activation functionfor deep learning. 2021. Ñ page 8[38] C.-H. Lin, J. Gao, L. Tang, T. Takikawa, X. Zeng, X. Huang, K. Kreis,S. Fidler, M.-Y. Liu, and T.-Y. Lin. Magic3D: High-Resolution Text-to-3DContent Creation. 2023. Ñ page 7[39] D. B. Lindell, D. Van Veen, J. J. Park, and G. Wetzstein. BACON:Band-limited Coordinate Networks for Multiscale Scene Representation.2022. Ñ pages 6, 24[40] L. Liu, J. Liu, S. Yuan, G. Slabaugh, A. Leonardis, W. Zhou, and Q. Tian.Wavelet-Based Dual-Branch Network for Image Demoireing. 2020. Ñ page8[41] P. Liu, H. Zhang, W. Lian, and W. Zuo. Multi-level Wavelet ConvolutionalNeural Networks. IEEE Access, 2019. Ñ page 8[42] R. Liu, J. Lehman, P. Molino, F. Petroski Such, E. Frank, A. Sergeev, andJ. Yosinski. An Intriguing Failing of Convolutional Neural Networks and theCoordConv Solution. 2018. Ñ page 5[43] S. Liu, X. Zhang, Z. Zhang, R. Zhang, J.-Y. Zhu, and B. Russell. EditingConditional Radiance Fields. 2021. Ñ page 34[44] Y. Liu, Q. Li, and Z. Sun. Attribute-aware Face Aging with Wavelet-basedGenerative Adversarial Networks. 2019. Ñ page 8[45] J. N. Martel, D. B. Lindell, C. Z. Lin, E. R. Chan, M. Monteiro, andG. Wetzstein. ACORN: Adaptive Coordinate Networks for Neural SceneRepresentation. SIGGRAPH, 2021. Ñ pages 16, 17[46] I. Mehta, M. Gharbi, C. Barnes, E. Shechtman, R. Ramamoorthi, andM. Chandraker. Modulated Periodic Activations for Generalizable LocalFunctional Representations. 2021. Ñ pages x, 2, 6, 18, 24, 31, 32[47] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger.Occupancy Networks: Learning 3D Reconstruction in Function Space.2019. Ñ page 538[48] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi,and R. Ng. NeRF: Representing Scenes as Neural Radiance Fields for ViewSynthesis. 2020. Ñ pages 1, 5, 6, 24, 25, 26, 34[49] T. Mu¨ller, A. Evans, C. Schied, and A. Keller. Instant Neural GraphicsPrimitives with a Multiresolution Hash Encoding. SIGGRAPH, 2022. Ñpages x, 1, 2, 3, 5, 6, 7, 10, 11, 14, 18, 19, 20, 24, 25, 26, 34[50] J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove.DeepSDF: Learning Continuous Signed Distance Functions for ShapeRepresentation. 2019. Ñ pages 1, 5[51] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, andR. Martin-Brualla. Nerfies: Deformable Neural Radiance Fields. 2021. Ñpage 34[52] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen,Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang,Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang,J. Bai, and S. Chintala. PyTorch: An Imperative Style, High-PerformanceDeep Learning Library. 2019. Ñ page 15[53] T. Peleg, P. Szekely, D. Sabo, and O. Sendik. Im-net for high resolutionvideo frame interpolation. 2019. Ñ page 5[54] S. Peng, M. Niemeyer, L. Mescheder, M. Pollefeys, and A. Geiger.Convolutional Occupancy Networks. 2020. Ñ page 1[55] H. Phung, Q. Dao, and A. Tran. Wavelet Diffusion Models are fast andscalable Image Generators. arXiv preprint, 2022. Ñ page 8[56] B. Poole, A. Jain, J. T. Barron, and B. Mildenhall. DreamFusion: Text-to-3Dusing 2D Diffusion . 2023. Ñ page 7[57] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer. D-NeRF:Neural Radiance Fields for Dynamic Scenes. 2021. Ñ page 34[58] C. R. Qi, H. Su, M. Nießner, A. Dai, M. Yan, and L. J. Guibas. Volumetricand Multi-View CNNs for Object Classification on 3D Data. 2016. Ñ page5[59] C. Reiser, S. Peng, Y. Liao, and A. Geiger. KiloNeRF: Speeding up NeuralRadiance Fields with Thousands of Tiny MLPs. 2021. Ñ page 739[60] D. Rho, B. Lee, S. Nam, J. C. Lee, J. H. Ko, and E. Park. Masked WaveletRepresentation for Compact Neural Radiance Fields. 2023. Ñ page 8[61] Sara Fridovich-Keil and Alex Yu, M. Tancik, Q. Chen, B. Recht, andA. Kanazawa. Plenoxels: Radiance Fields without Neural Networks. 2022.Ñ pages 25, 26[62] C. E. Shannon. Communication in the Presence of Noise. Proceedings ofthe IRE, 1949. Ñ page 2[63] J. M. Shapiro. Embedded Image Coding Using Zerotrees of WaveletCoefficients. IEEE Transactions on signal processing, 1993. Ñ page 8[64] V. Sitzmann, M. Zollho¨fer, and G. Wetzstein. Scene RepresentationNetworks: Continuous 3D-Structure-Aware Neural Scene Representations.2019. Ñ page 1[65] V. Sitzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein. ImplicitNeural Representations with Periodic Activation Functions. 2020. Ñ pagesx, 1, 2, 5, 6, 13, 18, 24[66] I. Skorokhodov, S. Ignatyev, and M. Elhoseiny. Adversarial Generation ofContinuous Images. 2021. Ñ page 1[67] K. O. Stanley. Compositional pattern producing networks: A novelabstraction of development. Genetic programming and evolvable machines,2007. Ñ page 5[68] G. Strang and T. Nguyen. Wavelets and Filter Banks. SIAM, 1996. Ñ page8[69] Subramaniam, Daly, and Rind. Wavelet transforms for use in motiondetection and tracking application. In Image Processing And ItsApplications, 1999. Seventh International Conference on (Conf. Publ. No.465), 1999. Ñ page 8[70] C. Sun, M. Sun, and H. Chen. Direct Voxel Grid Optimization: Super-fastConvergence for Radiance Fields Reconstruction. 2022. Ñ pages 25, 26[71] W. Sweldens. The Construction and Application of Wavelets in NumericalAnalysis. PhD thesis, PhD thesis, Department of Computer Science,Katholieke Universiteit Leuven . . . , 1994. Ñ page 840[72] T. Takikawa, J. Litalien, K. Yin, K. Kreis, C. Loop, D. Nowrouzezahrai,A. Jacobson, M. McGuire, and S. Fidler. Neural Geometric Level of Detail:Real-time Rendering with Implicit 3D Shapes. 2021. Ñ pages1, 3, 5, 6, 11, 14, 34[73] M. Tancik, P. Srinivasan, B. Mildenhall, S. Fridovich-Keil, N. Raghavan,U. Singhal, R. Ramamoorthi, J. Barron, and R. Ng. Fourier Features LetNetworks Learn High Frequency Functions in Low Dimensional Domains.2020. Ñ pages 1, 2, 5, 6, 10, 11, 12[74] M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan,J. T. Barron, and H. Kretzschmar. Block-NeRF: Scalable Large SceneNeural View Synthesis. 2022. Ñ page 1[75] M. Teschner, B. Heidelberger, M. Mu¨ller, D. Pomerantes, and M. H. Gross.Optimized Spatial Hashing for Collision Detection of Deformable Objects.In VMV, 2003. Ñ page 11[76] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,Ł. Kaiser, and I. Polosukhin. Attention Is All You Need. 2017. Ñ page 5[77] D. Verbin, P. Hedman, B. Mildenhall, T. Zickler, J. T. Barron, and P. P.Srinivasan. Ref-NeRF: Structured View-Dependent Appearance for NeuralRadiance Fields. 2022. Ñ page 7[78] M. Wakin, J. Laska, M. Duarte, D. Baron, S. Sarvotham, D. Takhar, K. F.Kelly, and R. G. Baraniuk. Compressive Imaging for Video Representationand Coding. In Picture Coding Symposium, 2006. Ñ page 8[79] J. Wang, X. Deng, M. Xu, C. Chen, and Y. Song. Multi-level Wavelet-basedGenerative Adversarial Network for Perceptual Quality Enhancement ofCompressed Video. 2020. Ñ page 8[80] T. Williams and R. Li. Wavelet Pooling for Convolutional Neural Networks.2018. Ñ page 8[81] Z. Wu, Y. Jin, and K. M. Yi. Neural Fourier Filter Bank. 2023. Ñ page v[82] Y. Xiangli, L. Xu, X. Pan, N. Zhao, A. Rao, C. Theobalt, B. Dai, and D. Lin.CityNeRF: Building NeRF at City Scale. 2022. Ñ page 1[83] Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari,J. Tompkin, V. Sitzmann, and S. Sridhar. Neural Fields in Visual Computingand Beyond. Computer Graphics Forum, 2022. Ñ pages 1, 541[84] D. Xu, Y. Jiang, P. Wang, Z. Fan, H. Shi, and Z. Wang. SinNeRF: TrainingNeural Radiance Fields on Complex Scenes from a Single Image. 2022. Ñpage 7[85] Q. Xu, Z. Xu, J. Philip, S. Bi, Z. Shu, K. Sunkavalli, and U. Neumann.Point-NeRF: Point-based Neural Radiance Fields. 2022. Ñ page 7[86] K. Yin, J. Gao, M. Shugrina, S. Khamis, and S. Fidler. 3DStyleNet: Creating3D Shapes with Geometric and Texture Style Variations. 2021. Ñ page 1[87] J. Yoo, Y. Uh, S. Chun, B. Kang, and J.-W. Ha. Photorealistic Style Transfervia Wavelet Transforms. 2019. Ñ page 8[88] A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa. PlenOctrees forReal-time Rendering of Neural Radiance Fields. 2021. Ñ page 7[89] A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixelNeRF: Neural RadianceFields from One or Few Images. 2021. Ñ page 7[90] G. Yu¨ce, G. Ortiz-Jime´nez, B. Besbinar, and P. Frossard. A StructuredDictionary Perspective on Implicit Neural Representations. 2022. Ñ page 6[91] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. TheUnreasonable Effectiveness of Deep Features as a Perceptual Metric. 2018.Ñ pages ix, 18, 19[92] Y. Zhang, T. van Rozendaal, J. Brehmer, M. Nagel, and T. Cohen. ImplicitNeural Video Compression. In Int. Conf. Learn. Represent. Workshop, 2022.Ñ page 142
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Yi, Kwang Moo
Neural fourier filter bank
Text