Applied Science, Faculty of
Electrical and Computer Engineering, Department of
DSpace
UBCV
Wafa, Abrar
Doctor of Philosophy - PhD
University of British Columbia
Light field technology offers a truly immersive experience, having the potential to revolutionize entertainment, education, virtual and augmented reality, autonomous driving, and digital health. There are two common techniques for capturing LF: multiple-camera arrays and microlens array (i.e., plenoptic camera). However, LF capturing techniques face a trade-off between spatial and angular resolution. That is, camera arrays capture high-spatial-resolution LF images with sparse angular sampling (i.e., fewer views), while plenoptic cameras capture dense angular sampling (i.e., more views) with low-spatial-resolution LF images due to the size of the plenoptic camera sensors. Since constructing an LF camera array is expensive, time consuming, and often impractical, many efforts have been made to improve the spatial resolution of LF images captured by plenoptic cameras. In this thesis, we propose two novel methods for LF spatial super-resolution (SR). First, we propose a learning-based model for spatial SR which takes advantage of the epipolar image plane (EPI) information to ensure smooth disparity between the generated views and in turn construct high-spatial-resolution LF images. In our second contribution, we exploit the full four-dimensional (4D) LF images by proposing a deep-learning spatial SR approach that considers the spatial and angular information (i.e., information within each view and information among other views) and progressively reconstructs high-resolution LF images at different upscaling levels. Another challenge when dealing with LF images is the enormous amount of data generated, as they require significant increase in bandwidth. A possible solution may be to drop specific views at the transmitting end and effectively synthesize them at the receiver end, thus minimizing the amount of data needed to be transferred or stored. Accordingly, our third contribution focuses on LF angular SR by synthesizing virtual LF views from a sparse set of input views using two novel approaches. First, a deep recursive residual network is applied using the EPI information to generate one in-between view. Second, a generative adversarial network approach is proposed, which generates up to five in-between views, using LF spatial and angular information for an efficient angular SR with minimal impact on the visual quality on the generated LF content.
LIGHT FIELD SPATIAL AND ANGULAR SUPER-RESOLUTION by Abrar Wafa B.S., Effat University, 2012 M.A.Sc, University of British Columbia, 2016 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF Doctor of Philosophy in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Electrical and Computer Engineering) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) December 2022 © Abrar Wafa, 2022 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the dissertation entitled: LIGHT FIELD SPATIAL AND ANGULAR SUPER-RESOLUTION submitted by Abrar Wafa in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Electrical and Computer Engineering Examining Committee: Panos Nasiopoulos, Professor, Electrical and Computer Engineering Department, UBC Supervisor Victor Leung, Professor, Electrical and Computer Engineering Department, UBC Supervisory Committee Member Matthew Yedlin, Professor, Electrical and Computer Engineering Department, UBC Supervisory Committee Member Shahriar Mirabbasi, Professor, Electrical and Computer Engineering Department, UBC University Examiner Jim Little, Professor, Computer Science Department, UBC University Examiner iii Abstract Light field technology offers a truly immersive experience, having the potential to revolutionize entertainment, education, virtual and augmented reality, autonomous driving, and digital health. There are two common techniques for capturing LF: multiple-camera arrays and microlens array (i.e., plenoptic camera). However, LF capturing techniques face a trade-off between spatial and angular resolution. That is, camera arrays capture high-spatial-resolution LF images with sparse angular sampling (i.e., fewer views), while plenoptic cameras capture dense angular sampling (i.e., more views) with low-spatial-resolution LF images due to the size of the plenoptic camera sensors. Since constructing an LF camera array is expensive, time consuming, and often impractical, many efforts have been made to improve the spatial resolution of LF images captured by plenoptic cameras. In this thesis, we propose two novel methods for LF spatial super-resolution (SR). First, we propose a learning-based model for spatial SR which takes advantage of the epipolar image plane (EPI) information to ensure smooth disparity between the generated views and in turn construct high-spatial-resolution LF images. In our second contribution, we exploit the full four-dimensional (4D) LF images by proposing a deep-learning spatial SR approach that considers the spatial and angular information (i.e., information within each view and information among other views) and progressively reconstructs high-resolution LF images at different upscaling levels. Another challenge when dealing with LF images is the enormous amount of data generated, as they require significant increase in bandwidth. A possible solution may be to drop specific views at the transmitting end and effectively synthesize them at the receiver end, thus minimizing the iv amount of data needed to be transferred or stored. Accordingly, our third contribution focuses on LF angular SR by synthesizing virtual LF views from a sparse set of input views using two novel approaches. First, a deep recursive residual network is applied using the EPI information to generate one in-between view. Second, a generative adversarial network approach is proposed, which generates up to five in-between views, using LF spatial and angular information for an efficient angular SR with minimal impact on the visual quality on the generated LF content. v Lay Summary Light Field (LF) is a rapidly evolving technology, supporting capturing and displaying of visual information that approximates the human eye, with applications ranging from entertainment to digital health. Light field’s unsurpassed immersive realism in every direction (unlike the limited horizontal 3D) and its ability to refocus on multiple distances even after acquisition, define new heights for remote non-intrusive surgery, avionics, augmented and virtual reality and autonomous driving. However, the low resolution of plenoptic LF content and the huge amount of data compromise the quality of experience and consequently the performance of the application. In this thesis, first we propose two novel contributions to increase the resolution of LF images. To address the challenges of storage and bandwidth requirements, we present two view synthesis approaches that allow us to drop LF views at the transmitting end and accurately recover them at the receiver end. vi Preface This thesis presents research conducted by Abrar Wafa under the guidance of Dr. Panos Nasiopoulos. A list of publications resulting from the work presented in this thesis is provided below. The content of Chapter 2 appears in the journal publication [P1] and the content of Chapter 3 is taken from the submitted journal paper presented in [P2]. The main body of Chapter 4, section 4.1, is taken from the publication in [P3] and section 4.2 of the same Chapter is taken from the submitted paper in [P4]. The work presented in [P1] – [P4] was primarily performed by Abrar Wafa, including literature review, designing, and implementing the proposed methods, performing all evaluations, analyzing the results, and writing the manuscripts. Dr. Panos Nasiopoulos and Mahsa T. Pourazad provided guidance and editorial input into the manuscript writing. Finally, it should be noted that subjective studies in this work were covered under the UBC Ethics Board (H12-00308). List of Publications Based on Work Presented in this Thesis [P1] A. Wafa, M. T. Pourazad, and P. Nasiopoulos “A Deep Learning Based Spatial Super-Resolution Approach for Light Field Content”, IEEE Access, vol. 9, pp.2080-2092, December 2020. [P2] A. Wafa, and P. Nasiopoulos “GAN-based Light Field Spatial Super-Resolution Using Spatial and Angular Information”, Manuscript Submitted, June 2022. vii [P3] A. Wafa, M. T. Pourazad, and P. Nasiopoulos, "Learning-Based Light Field View Synthesis for Efficient Transmission and Storage", 2021 IEEE International Conference on Image Processing (ICIP), pp.354-358, September 2021. [P4] A. Wafa, and P. Nasiopoulos, "Light Field GAN-based View Synthesis using full 4D information", European Conference on Visual Media Production (CVMP), London, United Kingdom, December 2022. viii Table of Contents Abstract ................................................................................................................................. iii Lay Summary ......................................................................................................................... v Preface .................................................................................................................................. vi Table of Contents ……………………………………………………………………………viii List of Tables ......................................................................................................................... xi List of Figures ...................................................................................................................... xii List of Abbreviations ........................................................................................................... xvi Acknowledgements ............................................................................................................ xviii Chapter 1: Introduction ............................................................................................................ 1 1.1 Plenoptic Function of Light ......................................................................................... 2 1.2 Light Field Capturing .................................................................................................. 3 1.2.1 Plenoptic Camera ..................................................................................................... 4 1.2.2 Camera Array .......................................................................................................... 8 1.3 Representation of Light Field Data ........................................................................... 10 1.3.1 Macro-pixel Representation .................................................................................. 10 1.3.1.1 Macro-pixel Image Reconstruction Process .................................................. 11 1.3.2 Sub-aperture Representation ................................................................................. 15 1.3.3 Epipolar Image Plane Representation ................................................................... 16 1.4 Literature Review on Spatial Super-resolution ......................................................... 16 1.5 Literature Review on Angular Super-resolution ....................................................... 21 ix 1.6 Thesis Outline ............................................................................................................ 24 Chapter 2: Light Field Spatial Super-resolution using EPI ................................................ 28 2.1 Proposed Spatial Super-resolution Framework ......................................................... 30 2.1.1 Epipolar Image Plane (EPI) ................................................................................... 33 2.2 Super-resolution Model ............................................................................................. 34 2.2.1 Implementation Details ......................................................................................... 34 2.2.2 Training ................................................................................................................. 39 2.3 Experiments and Evaluations .................................................................................... 40 2.3.1 Experimental Evaluation ....................................................................................... 41 2.3.2 Subjective Evaluation ............................................................................................ 50 2.4 Conclusion ................................................................................................................. 52 Chapter 3: Light Field Super-resolution using 4D Spatial and Angular Information ..... 54 3.1 Proposed Light Field Spatial Super-Resolution Framework ..................................... 56 3.1.1 Downsampling Framework ................................................................................... 58 3.1.2 Super-resolution Framework ................................................................................. 60 3.1.2.1 Spatial and Angular Features Extraction ....................................................... 61 3.1.2.2 Super-resolution Generator ........................................................................... 63 3.1.2.3 Patch Discriminator ....................................................................................... 67 3.1.2.4 Loss Function ................................................................................................ 67 3.1.2.5 Training ......................................................................................................... 69 3.2 Performance Evaluation and Discussion ................................................................... 70 3.2.1 Ablation Study ....................................................................................................... 71 x 3.2.2 Experimental Evaluation and Discussion .............................................................. 74 3.3 Conclusion ................................................................................................................. 77 Chapter 4: Light Field Angular Super-resolution ................................................................ 78 4.1 View Synthesis Framework using EPI ...................................................................... 79 4.1.1 Epipolar Plane Image (EPI) ................................................................................... 80 4.1.2 Convolutional Neural Network (CNN) ................................................................. 81 4.1.3 Experiments and evaluation .................................................................................. 84 4.1.4 Conclusion ............................................................................................................. 87 4.2 View Synthesis Framework using Light Field 4D Spatial and Angular Information88 4.2.1 Proposed Angular Super-Resolution Framework .................................................. 90 4.2.1.1 Features Extraction ........................................................................................ 90 4.2.1.2 Generator Network Structure ........................................................................ 91 4.2.1.3 Patch Discriminator ....................................................................................... 93 4.2.1.4 Training ......................................................................................................... 93 4.2.2 Experiments and Evaluations ................................................................................ 95 4.2.3 Conclusion ............................................................................................................. 99 Chapter 5: Conclusion .......................................................................................................... 101 5.1 Summary of Contributions ...................................................................................... 101 5.2 Significance and Potential Applications of the Research ........................................ 104 5.3 Future Work ............................................................................................................. 106 Bibliography ...................................................................................................................... 108 xi List of Tables Table 2.1 Comparisons Against Different Approaches ................................................................. 42 Table 2.2 Comparisons of Four Different Scenes ......................................................................... 44 Table 3.1 Comparisons Against Different Model Settings ........................................................... 71 Table 3.2 Comparisons Against Different Approaches ................................................................. 74 Table 4.1 Quantative comparisons of different light field scenes ................................................. 85 Table 4.2 Quantative comparisons of different LF scenes ............................................................ 96 Table 4.3 Quantative comparisons of yellow flowers and pink flowers scenes ............................ 97 xii List of Figures Figure 1.1 Light field parameterization (a) Parameterization of 5D light field. (b) Parameterization of 4D light field. .................................................................................................. 3 Figure 1.2 Scene capturing (a) Scene captured by traditional camera. (b) Scene captured by light field camera [7]. ............................................................................................................................... 4 Figure 1.3 Conceptual schematic of Ng’s plenoptic camera design, which is composed of a main lens, microlens array, and a photosensor. ........................................................................................ 5 Figure 1.4 Plenoptic camera with microlens array focused at the image plane [11]. ...................... 6 Figure 1.5 Heterodyne light field camera [13]. ............................................................................... 7 Figure 1.6 Light Field Camera Array (a) 2D camera array arrangement. (b) 1D linear camera array arrangement. [14] ................................................................................................................... 8 Figure 1.7 Camera array multiview images [15]. ............................................................................ 8 Figure 1.8 Light field camera output (a) Raw LF image captured by plenoptic camera and a close up. .................................................................................................................................................. 11 Figure 1.9 Offset between the center of microlens and its image [16]. ......................................... 12 Figure 1.10 Representation of a discrete 4D light field value [18]. .............................................. 12 Figure 1.11 The relation between the physical microlens center and microlens image center [19]. ....................................................................................................................................................... 13 Figure 1.12 The six special points around the microlens image center to calculate the score map [19]. ............................................................................................................................................... 14 xiii Figure 1.13 Epipolar Image Plane (EPI) example (a) 4D Light Field. (b) 2D horizontal EPI for the first raw of the 4D light field sub-aperture images. (c) 2D vertical EPI for the first column of the 4D light field sub-aperture images. ......................................................................................... 16 Figure 1.14 (a) Original sub-aperture image. (b) Bicubic up-sampled sub-aperture image. (c) CNN up-sampled sub-aperture image. .......................................................................................... 17 Figure 1.15 Overview of angular super-resolution approach ........................................................ 21 Figure 1.16 Thesis Contributions. Contribution 1 and 2 are presented to solve the spatial resolution issue and Contribution 3 and 4 are proposed to solve the angular resolution issue. .... 24 Figure 2.1 Overview of the proposed approach, showing the input and output EPI volumes and our deep learning model for LF super-resolution. ......................................................................... 29 Figure 2.2 Overview of the proposed framework for each: (a) row (b) column. .......................... 32 Figure 2.3 (a) Super-resolution CNN structure. (b) overview of a residual unit. (c) residual unit layers. ............................................................................................................................................. 35 Figure 2.4 Overview of the reconstruction function of the super-resolution model. .................... 36 Figure 2.5 Comparison showing the ground truth central view with a cropped close-up, and its interpolated versions using different approaches of the: (a) reflective_22, (b) bikes_2 scenes. ... 46 Figure 2.6 Qualitative comparison of different methods using Red Flowers scene [22] (a) Close up of the high-resolution central view of the sub-aperture images. (b) EPIs generated from the high-resolution sub-aperture images. ............................................................................................ 47 Figure 2.7 Comparison showing the high-resolution central view of the sub-aperture images from (a) the Garden and (b) the Chairs scenes [22] using different approaches. ................................... 49 Figure 2.8 Subjective of paired comparison of our proposed method against: (a) LFCNN, (b) GMM, (c) GB and (d) D-LFSSR. .................................................................................................. 52 xiv Figure 3.1 Overview of the proposed approach at training phase. ................................................ 55 Figure 3.2 Overview of the proposed approach at testing phase. .................................................. 56 Figure 3.3 Overview of (a) the downsampling approach (b) KernelGAN framework. ................ 58 Figure 3.4 Light Field Camera output (a) Macro-pixel LF image captured by Plenoptic Camera. ....................................................................................................................................................... 61 Figure 3.5 An illustration of angular and spatial features extraction. To extract angular features, pixels under each microlens are convolved with the green kernel. To extract spatial features, pixels of the same color under all the microlenses are convolved with the brown kernel. ........... 62 Figure 3.6 An Illustration of Proposed Model. (a) Overview of Proposed Network Architecture. (b) Angular and Spatial Feature Extraction Sub-network. (c) Residual Dense Block (RDB). (d) Laplacian Up-sampling Sub-Network ........................................................................................... 64 Figure 3.7 SSIM of proposed network trained with L2, L1, and Charbonnier loss functions. ..... 73 Figure 3.8 Comparison showing the ground truth central view with a cropped close-up, and its interpolated versions using different approaches of the: (a) green_flowers and (b) car_bumper scenes. ............................................................................................................................................ 76 Figure 4.1 Overview of proposed view synthesis technique. ........................................................ 78 Figure 4.2 Overview of proposed view synthesis technique using EPI. ....................................... 79 Figure 4.3 Steps for generating the in between views for the first row of images. ....................... 80 Figure 4.4 Three steps to synthesize in between views. ................................................................ 81 Figure 4.5 (a) Deep learning model structure. (b) One residual unit. (c) Convolutional layers. ... 82 Figure 4.6 16x16 Chess scene captured by camera array. ............................................................. 86 Figure 4.7 Comparison of our method and other state-of-the-art view synthesis schemes against ground truth using “Chess”. .......................................................................................................... 86 xv Figure 4.8 An overview of proposed angular super-resolution approach. (a) Generator Network structure. (b) Residual Dense Block (RDB) structure. .................................................................. 89 Figure 4.9 Comparison of our method and other state-of-the-art view synthesis schemes against ground truth using “yellow_flowers” scene. ................................................................................. 98 Figure 4.10 Comparison of our method and other state-of-the-art view synthesis schemes against ground truth using “Pink_Flowers”. .............................................................................................. 99 xvi List of Abbreviations 2D Two Dimension 4D Four Dimension 5D Five Dimension 7D Seven Dimension AR Augmented Reality BN Batch Normalization CNN Convolutional Neural Network DFF Dense Feature Fusion D-LFSSR Dense Light Field Spatial Super Resolution DRCN Deeply-Recursive Convolutional Network DRRN Deep Residual DSLR Digital Single-lens Reflex EPI Epipolar Image Plane ESPCN Efficient Sub-Pixel Convolutional Neural Network GAN Generative Adversarial Networks GB Graph Based GMM Gaussian Mixture Model GT Ground Truth HR High Resolution LF Light Field xvii LF-ATO Light Field All-to-One LFCNN Light Field Convolutional Neural Network LF-DCNN Light Field Deep Convolutional Neural Network LR Low Resolution MP-LF Macro-pixel Light Field MSE Mean Square Error PSNR Peak Signal-to-Noise Ratio ReLU Rectified Linear Activation Unit RDB Residual Dense Block RDN Residual Dense Network RU Residual Unit SA-LF Sub-Aperture Light Field SSIM Structural Similarity Index SISR Single Image Super Resolution SR Super Resolution SRCNN Super Resolution Convolutional Neural Network Up-Net Up-sampling Network VMAF Video Multimethod Assessment Fusion VR Virtual Reality VGG Visual Geometry Group xviii Acknowledgements First of all, all praises go to Allah the most merciful for his guidance. I would like to start by expressing my utmost gratitude to my supervisor, Dr. Panos Nasiopoulos for not only being an outstanding supervisor and a constant source of guidance and inspiration, but also for being a role model in kindness, patience, dedication to research, and being supportive throughout all my years here at UBC. His encouragement and insightful feedback have been extremely valuable to me. I would also like to thank Dr. Mahsa T. Pourazad for her time, thorough feedback, support, and help during the different stages of this thesis. I would also like to thank my Ph.D. committee members, Dr. Victor Leung, and Dr. Matthew Yedlin, for investing time and energy to provide guidance and help throughout my graduate studies. I am incredibly grateful to my parents Salahuddin and Amal, who always believed in me and never stopped encouraging me. I am thankful for their love and their unwavering dedication to my education throughout the years. For all the sacrifices they made for me so that I can be who I am today. I am thankful to them for raising me in a loving and providing family, and for pushing and encouraging me to try harder. xix I express my sincere and wholehearted thanks to my husband, my love, Omar for encouraging me and believing in me. Without his support, I would not have been able to complete this work. I am thankful for his help in taking care of our daughters and sharing the housework during the most difficult time of working on this thesis. To my daughters Serene and Yasmeen, thank you for making me stronger, better, and more fulfilled than I could have ever imagined. Thank you for for giving me unlimited happiness and pleasure. I am thankful to my siblings, Hatem, Ebtihal, and Fouad, for their love and emotional support throughout the years I spent away from home. Finally, I would like to thank the Saudi government and Prince Sultan University for their funding and financial support. 1 Chapter 1: Introduction Light plays an important role in our daily lives while communicating with the world. It reflects off objects around us and travels in every direction, creating a continuous light field. As light rays either come directly from light sources or reflect towards our eyes, our visual system produces neural signals that form perception of the world around us. These signals inform us about the rays’ color, intensity, and direction, which in turn, the brain uses to perceive the objects. Such visual information plays a vital role in our lives. Over the past decades, researchers have been trying to capture and reproduce our visual experience. Such a perceptual representation would contain a description of all the information available to an observer at any point in space and time. This can be represented by the Light Field (LF) function referred to as the plenoptic function [1]. This is a seven-dimensional function that describes the amount of light flowing in every direction through every point in space [2]. A conventional camera sensor only captures a two-dimensional (2D) slice of this plenoptic function (light rays’ color and intensity, but none of the direction information) to create the images/videos we see, falling short of what the human visual system can perceive. This motivated researchers to capture additional plenoptic dimensions in order to extract more information and deliver a richer and more adaptive visual experience. Light field technology captures the plenoptic function as light travelling through space of rays, thus, it captures the lighting directional distribution along with its color and intensity. This emerging digital media format allows us to come closer to the quest of reproducing human perception by capturing light direction in addition to light color and intensity. As a result, this 2 technology is the foundation for a truly immersive experience, offering enhanced depth perception, better object recognition, and even post capture focusing, thus having the potential to revolutionize entertainment, training, education, virtual and augmented reality, gaming, autonomous driving, and digital health. 1.1 Plenoptic Function of Light The term light field was originally introduced in radiometry by Gershun [3]. The plenoptic function was then introduced by Adelson and Bergen in [1] to characterize the light field mathematically. In their work, the authors defined the total distribution of light as a function that describes the radiance of a ray of light L traveling over the seven dimensional (7D) space of rays as 𝐿(𝑥, 𝑦, 𝑧, 𝜃, 𝜑, 𝜆, 𝑡), where the variables (𝑥, 𝑦, 𝑧) represent the three-dimensional position at any point of a light ray, (𝜃, 𝜑) represent the angular coordinates encoding the direction of the ray, 𝜆 is the wavelength, and 𝑡 is the time. Assuming the plenoptic function does not vary with time and the light is monochromatic (𝑡 and 𝜆 can be omitted) [4], then the dimensionality of the function can be reduced to 5D, as illustrated in Figure 1.1 (a). The 5D plenoptic function was further simplified to 4D by Levoy and Hanrahan [2] and Gortler et al. [5] assuming that rays are travelling through free-space (un-occluded space). The authors in [2] and [5] justified that the reason for reducing the dimensionality is to make measurement feasible. This 4D light field is defined to be the radiance along rays in empty space as a function of position and direction, where rays are the fundamental light carrier [6]. There are different ways to parameterize this 4D set of rays. The most common one is the two-plane parameterization L(x, y, s, t) which is obtained by the intersection of the rays with two planes: (x, 3 y) represents the spatial coordinates (i.e., lens plane) and (s, t) represents the angular coordinates (i.e., sensor plane), as shown in Figure 1.1(b). In other words, (x, y) represents the spatial resolution of each image and (s, t) represents the angular resolution (number of images/views). 1.2 Light Field Capturing Light field acquisition schemes are implemented by the means of sampling the 4D plenoptic function to measure the light rays flowing in different directions. Unlike traditional cameras, which only capture the scene from a single viewpoint, light field cameras capture the same scene from different viewpoints. Figure 1.2 presents a comparison between a traditional camera and a light field camera [7]. When a scene is captured with a conventional camera, Figure 1.2 (a), only the light color and intensity are projected on the 2D image from one point of view. Alternatively, when a light field camera is used to capture the scene, as shown in Figure 1.2 (b), we can also preserve (a) (b) Figure 1.1 Light field parameterization (a) Parameterization of 5D light field. (b) Parameterization of 4D light field. 4 the directional information of the light, which enables the reconstruction of the scene from different viewpoints using the captured information. There are different approaches for capturing light field data, the two most common ways are plenoptic camera (i.e., LF camera) and camera array, both described in detail in the following subsections. 1.2.1 Plenoptic Camera With the introduction of digital photography and the theory behind a 4D plenoptic light field function, a significant number of new possibilities became available for designing light field (plenoptic) cameras. The first digital version of plenoptic camera was introduced to computer vision by Adelson and Wang [4], who prototyped a non-portable version of the digital plenoptic camera to infer object depths. In plenoptic cameras, the radiance is sampled by an array of microlenses densely positioned in the angular dimensions, but sparsely in the spatial dimensions. Ng et al. [8] furthered improved the plenoptic camera proposed by Adelson and Wang, and introduced a handheld camera in which a microlens array is placed in front of a 2D photosensor array, as shown in Figure 1.3. They implemented a camera that uses a lenslet (microlens) array placed in-between the main lens and the photosensor to capture full 4D spatial and angular (a) (b) Figure 1.2 Scene capturing (a) Scene captured by traditional camera. (b) Scene captured by light field camera [7]. 5 information. Each microlens covers a small array of the photosensor pixels to sample the rays’ directions reflected from every point captured in a single photographic exposure. As shown in Figure 1.3, rays of light from a single point on the subject are brought to a single convergence point on the focal plane of the microlens array. The microlens at that location separates these rays of light into the angular components that provide us with the function of light field, creating a focused image of the aperture of the main lens on the array of pixels underneath the microlens [9]. A camera model based on Ng’s proposed handheld light field camera design became commercially available by Lytro [10], primarily as a device for synthetic image formation, especially refocusing photographs. Lumsdaine and Georgiev developed a modified light field camera, called the focused plenoptic camera [11]. Similar to Ng’s plenoptic camera design [9], they placed the microlenses between the main lens and the photosensor. Unlike Ng’s design, where the microlenses are adjusted accurately to be exactly at one focal length from the sensor, their plenoptic camera focuses the microlenses on the image formed inside the camera by the main lens, not on the main lens, as shown in Figure Figure 1.3 Conceptual schematic of Ng’s plenoptic camera design, which is composed of a main lens, microlens array, and a photosensor. 6 1.4. The camera developed by Lumsdaine and Georgiev [11] became commercially available for industrial and scientific applications by Raytrix in 2010 [12]. Since then, Raytrix has been manufacturing high-end light field cameras. Unlike Lytro, Raytrix cameras are mainly targeted for the industrial sector. The plenoptic camera introduced by Adelson and Wang [4] and Ng et al. [8] is referred to as Plenoptic 1.0 camera, while the plenoptic camera developed by Lumsdaine and Georgiev [11] is referred to as Plenoptic 2.0 camera. Despite the theoretical effectiveness of the Plenoptic 2.0 architecture, this approach has not been widely adopted mainly due to its complicated optical system and the need for a sophisticated rendering process to generate high-quality LF images. Figure 1.4 Plenoptic camera with microlens array focused at the image plane [11]. 7 Instead of sampling individual rays on the sensor through microlenses, a mask-based design is used to sample linear combination of rays in Fourier space [13]. This capturing system, demonstrated in Fig. 1.5, is called a heterodyne light field camera. It involves the use of amplitude masks between the main lens and the sensor plane, as the mask shadows the incoming light and dapples the sensor to filter rays’ angular information. Any hand-held camera can be simply converted into a light field camera using this technology by inserting a low-cost film mask on top of the sensor. This mask could also be integrated to smartphone cameras. However, to our knowledge, this has not been implemented yet. It is worth noting that plenoptic cameras do not need frame synchronization or geometric and color calibration. However, one issue with the plenoptic camera design is that the spatial resolution (number of pixels in each image) is sacrificed to achieve a higher angular resolution (number of images/viewpoints). Figure 1.5 Heterodyne light field camera [13]. 8 1.2.2 Camera Array An optically equivalent configuration to the plenoptic camera is a physical array of traditional cameras. Such camera arrangement is flexible, as it can include a dense or sparse array of cameras, with linear or nonlinear arrangements, and with single (horizontal or vertical only) or full (both horizontal and vertical) parallax (see Figure 1.6) [14]. The multiview images [15] captured with the camera array setup, Figure 1.7, have high-spatial-resolution but the angular resolution, meaning the proximity between the neighboring views, is limited by the distance between the cameras. (a) (b) Figure 1.7 Camera array multiview images [15]. Figure 1.6 Light Field Camera Array (a) 2D camera array arrangement. (b) 1D linear camera array arrangement. [14] 9 Constructing a light field camera array is expensive, time consuming, requires substantial amount of engineering and is very impractical for the majority of applications. In addition, a camera array is typically very large and not portable, thus limiting its use within a more controlled environment. To manufacture low-cost camera array systems, several researchers investigated the use of cost-effective cameras and processors rather than using high quality ones. However, using low-cost cameras poses its own challenges. For instance, these cameras cannot be genlocked, which means that the captured images are not synchronized, somehow prohibiting rendering in real-time. Another issue with low-cost cameras is that they have defective color controls as they have varying color responses. This is especially important considering that different parts of the images are captured by different cameras. Large camera arrays present another challenge as the amount of data captured by a camera array is huge. To process and store the data, most existing methods use multiple computers connected to the cameras and apply compression on the raw data at each camera to reduce the amount of data. There have been some initiatives to integrate small camera array systems into smartphone/tablet products. One approach, developed by Pelican [16], involved an array of lenses with a sensor behind each lens. Their camera array is even smaller than a coin. A prototype of the camera was introduced on an Android tablet in 2013. However, the camera did not make its way into production, as there were limitations in the camera design such as image quality and processing speed. 10 1.3 Representation of Light Field Data Captured light field data can be represented in three different ways. The following subsections highlight the different LF data visualizations in detail. Each of these visualizations represents the light field as a 2D array of images, and each image represents the remaining two dimensions of variation. 1.3.1 Macro-pixel Representation A plenoptic camera captures LF raw data, which is the pixel values of the photosensor placed behind the microlens array. The LF raw data basically captures the lighting incident upon each microlens. It is assumed that the microlenses and the photosensor lie on the spatial plane (x, y) and the angular plane (s, t), respectively (see Figure 1.1 (b)). The photosensor of the plenoptic camera performs the 4D light field sampling of the axes x and y, where each image on the sensor represents the light arriving at that film point from different (s, t) positions. Thus, the raw image is composed of an (x, y) grid of circular images that form underneath each microlens. The raw image captured 11 by a plenoptic camera macroscopically, Figure 1.8 (a), appears like a typical image photographed by a conventional camera. However, looking more closely we can notice that the microlens array is clearly visible, where each circular image represents the light arriving at that sensor from a different angle. Such image representation is called Macro-pixel LF image (MP-LF). This MP-LF representation can then be used to get the sub-aperture LF image (SA-LF) shown in the bottom part of Figure 1.8 (b). 1.3.1.1 Macro-pixel Image Reconstruction Process Due to manufacturing defections, it is common to have a microlens array that does not perfectly align with the photosensor coordinates, Figure 1.9 [17]. Thus, the captured raw data need to first be calibrated in order to convert them to the Macro-pixel image representation effectively. The (a) (b) Figure 1.8 Light field camera output (a) Raw LF image captured by plenoptic camera and a close up. (b) Processed sub-aperture images and a close up to one sub-aperture image of size 625x434. 12 main goal of this calibration is to identify the center point locations of each microlens sub-image and rearrange them in a regular basis for a better sampling [17]. To reconstruct the 4D light field Macro-pixel representation, first we need to understand the relationship between the observable light field and the physical light field captured by plenoptic cameras. The output of plenoptic cameras is a discrete light field function 𝐿!"#$%&'&[𝑚, 𝑛, 𝑝, 𝑞] of the integer coordinates (m, n) which represent the microlens that light passed through to reach the sensor (i.e. correspond to the spatial coordinates (x, y)), and the (q, p) coordinates that represent which pixel on the photosensor is capturing this light (i.e. correspond to the angular coordinates Figure 1.10 Representation of a discrete 4D light field value [18]. Figure 1.9 Offset between the center of microlens and its image [16]. 13 (s, t)) [18]. As Figure 1.10 shows, a discrete light field 𝐿!"#$%&'&[𝑚, 𝑛, 𝑝, 𝑞] is then the total energy of light (blue) collected by a single pixel (p, q) (filled in red) through a single microlens (m, n) (filled in green) [18]. Each microlens aperture is ∆( × ∆) in size and each pixel is ∆* × ∆+in size. The spatial extent of the pixel causes it to capture a spread of angles in the scene (dark gray). As illustrated in Figure 1.11 [19], the relation between the physical microlens center and microlens image center is ℎ′ = (1 + !!"! )ℎ (1.1) where d is the distance between the main lens plane and the microlens array plane, 𝑑(, is the distance between microlens array and photosensor, ℎ and ℎ′ are the heights of microlens centre and microlens image centre with respect to the optical axis, respectively. Ideally, given the physical location of each microlens center, optical settings d and 𝑑(,, we can determine the corresponding microlens image centers intermediately by Equation 1.1. However, this is not possible in practice. A robust local centroiding estimator, which detects the position of each microlens image, is introduced in [19]. In their work, they measure the individual microlens Figure 1.11 The relation between the physical microlens center and microlens image center [19]. 14 image centers by examining the dark pixels among the microlens images, i.e., the dark gaps between microlens images. For example, as shown in Figure 1.12, the position of the darkest gaps with respect to the center of a microlens are 𝔭! = #𝑅, 𝑅√3( , 𝔭" = #0, 2𝑅√3( , 𝔭# = #−𝑅, 𝑅√3( , 𝔭$ = #−𝑅,− 𝑅√3( , 𝔭% = #0,−2𝑅√3( , 𝔭& = #𝑅,− 𝑅√3( where R is the radius of the hexagonal grid. For an arbitrary pixel at position 𝑥 = [𝑥, 𝑦]- of the light field raw image, the summation of the six special surrounding pixels denoted by 𝑃(𝑥) is used for detecting the center of the microlens image. Additionally, to reduce the effect of dark current, a Gaussian filter 𝐺. is applied. The local centroiding estimator is then defined as a score map 𝑃(𝑥), 𝑃(𝑥) = ∑ (𝐺.)(𝑥 + 𝑝")/"01 (1.2) where 𝑃(𝑥) reaches a minimum only when x is at the center of a microlens image. After identifying the centres of the microlens image array, the full color information in each pixel is recovered through demosaicing. This demosacing, which is the final step of the reconstruction Figure 1.12 The six special points around the microlens image center to calculate the score map [19]. 15 of the LF Macro-pixel image representation, plays an important role in restoring a high quality 4D light field. In contrast to 2D demosaicing, 4D demosaicing consider the coherence of both angular and spatial information. In order to infer the interest pixel value from the structure of its 4D neighbors, 4D kernel regression method is used [20], where a Gaussian function is used as the Kernel function and only the pixels which within the distance of 2 pixels are included in the sample dataset in each dimension. 1.3.2 Sub-aperture Representation The second representation of light field data provides pictures of the scene from an array of viewpoints spread over the extent of the lens aperture. Since the lens aperture has a finite area, these different views, referred to as sub-aperture representation, provide some parallax information about the world [8]. The sub-aperture representation is basically the different pictures of the same scene captured from different viewing angles. A camera array captures the light field data in the sub-aperture (i.e., Multiview) representation shown in Figure 1.7. For light field data captured by a plenoptic camera, the sub-aperture images can be formed by taking the same pixel underneath each microlens, as shown in top portion of Figure 1.8 (b). This creates an array of images, where each image is a picture of the scene from the corresponding (s, t) position. The bottom part of Figure 1.8 (b) shows all the sub-aperture images together and a single sub-aperture image enlarged on the right side. 16 1.3.3 Epipolar Image Plane Representation The light field data, captured by a plenoptic camera or a camera array, can also be presented in the form of an epipolar image plane (EPI). An EPI is obtained by stacking the light field images of each row or each column over each other, as shown in Fig. 1.13. Thus, fixing y and t, we can obtain a horizontal 2D slice E(y*,t*)(x, s) as shown in Fig. 1.13 (b). Similarly, a vertical 2D slice E(x*,s*)(y, t) can be obtained if x and s are fixed as shown in Fig. 1.13 (c). This is done for each row and column of images in light field. Thus, if we have a 7x7 light field images, and each image resolution is 400x600, we would generate 7x400 = 2800 horizontal EPIs and 7x600 = 4200 vertical EPIs. 1.4 Literature Review on Spatial Super-resolution Comparing both plenoptic camera and camera array capturing approaches, we can see that a camera array is typically very large and not portable, thus limiting its use within only a controlled environment. In addition, constructing a camera array is expensive, time consuming, and requires (a) (b) (c) Figure 1.13 Epipolar Image Plane (EPI) example (a) 4D Light Field. (b) 2D horizontal EPI for the first raw of the 4D light field sub-aperture images. (c) 2D vertical EPI for the first column of the 4D light field sub-aperture images. 17 substantial amount of engineering, making capturing light field data with plenoptic camera a better choice. The main issue with using a plenoptic camera to capture light field data, however, is that the spatial resolution of each image is sacrificed in order to achieve a higher angular resolution due to the limited sensor resolution. To mitigate this problem, various approaches have been presented to increase the light field spatial resolution of the sub-aperture images generated from the microlenses and form spatial high-resolution (HR) images. The latter are desirable and often very crucial in terms of human perception as in the case of remote surgery, for instance. The visual difference between a low-spatial-resolution sub-aperture image extracted from the raw light field image and a super-resolution (SR) version is shown in Fig. 1.14, where Fig 1.14 (a) shows the original sub-aperture image, Fig. 1.14 (b) the upscaled version in the original resolution, and Fig. 1.14 (c) a generated high-resolution sub-aperture image using the LF super-resolution framework. Some techniques that are designed to increase the spatial resolution of sub-aperture images have the limitation that they treat each image in the light field as individual entity, not taking advantage of (a) (b) (c) Figure 1.14 (a) Original sub-aperture image. (b) Bicubic up-sampled sub-aperture image. (c) CNN up-sampled sub-aperture image. 18 the redundancies available in both spatial and angular dimensions of the sub-aperture images [21]–[23]. Moreover, considering each sub-aperture image separately does not enforce disparity continuity and smooth transition among light field views. Farrugia et al. introduce a spatial super-resolution technique, where they first construct a dictionary of examples by extracting pairs of high- and low-resolution patch-volumes and then use a linear mapping function learning algorithm between the subspaces of these volumes [21]. Finally, each overlapping patch-volume of the low-resolution light field is then super-resolved by applying the learned mapping function. The deep learning-based approach presented in [22] and [23] first up-samples each sub-aperture image separately, and then uses a convolutional neural network to restore the high frequency details. Many other techniques have been proposed for generating spatial high-resolution (HR) views from several low-resolution (LR) views. In these cases, increased resolution is achieved by first estimating a depth map and then formulating the Super-Resolution (SR) problem as an optimization framework such as the Bayesian model [24], variational model [25], Gaussian mixture model (GMM) [26], and graph-based model [27], all with different constraints. These methods divide the images into patches and assume a constant global disparity for each patch. Although the disparity continuity in light field is taken into account, yet, the disparity range does not accurately reflect each object’s disparity, since a constant disparity value is considered for each patch. As a result, the disparity errors translate into significant artifacts along the object edges of the SR images. A hybrid approach involves coupling a single high resolution camera with a light field camera [28]–[30]. In this case, patches are extracted from each view of sub-aperture images, which are 19 matched to parts of the reference high resolution image captured by the Digital Single-Lens Reflex (DSLR) camera, and then are blended together using a weighted average. Even though the performance of the hybrid methods outperforms some of the existing methods, a major disadvantage is the use of a DSLR which is impractical for may applications, for instance endoscopy. Recent convolutional neural network (CNN) solutions to super-resolve LF spatial resolution demonstrated superior performance to the traditional methods in LF spatial SR. Specifically, in LFCNN [23], [31], each sub-aperture LF (SA-LF) image is first super-resolved separately using CNN that was developed for single-image SR, and then they combine every pair of images to enhance both spatial and angular resolution. However, such technique super-resolving each image separately does not take advantage of the redundancies available in both spatial and angular dimensions of the SA-LF images. Consequently, EPI-enhancement networks have been proposed to incorporate the angular information [32]–[34]. To enforce disparity continuity between subaperture images, a deep learning approach was introduced by Yeung in [35] and a combined Convolutional Neural Network CNN framework was proposed for the same purpose by Yaun in [33]. First, the spatial resolution of each sub-aperture image is enhanced separately utilizing a CNN. Then, the geometric structure of the epipolar plane image (EPI) is improved through another CNN model. However, using a CNN model in the first step to increase the spatial resolution of each subaperture image separately increases the computational complexity. Further, in their approach, they feed the horizontal EPI and the vertical EPI separately to the enhancement deep CNN and then they use these EPIs to obtain the horizontal and vertical images, respectively. The 20 final result is then achieved by taking the mean of these super-resolved horizontal and vertical images. Thus, the correlation between all the views is not considered. In addition, since an EPI is a 2D slice of the 4D LF, performing convolutions on EPIs can only incorporate angular information from the same horizontal or vertical views. Thus, such methods underuse of spatial and angular information limits their performance. To mitigate this problem, Zhang et al., [36] proposed resLF, a residual network by stacking SA-LF images along four different angular directions (i.e., horizontal, vertical, diagonal, and antidiagonal) as its inputs and finally combine the residuals in every direction to reconstruct higher-resolution light fields. Nevertheless, since resLF stacks SA-LF images in each direction to super-resolve one view, it has the same limitation as the EPI approaches, meaning it reduces 4D LF to 3D LF since it discards angular information in the other directions. Consequently, Yeung et al. [35] proposed LFSSR to super-resolve all LF views at a single forward pass by alternately convolving spatial and angular features. Jin et al., [37] also designed an all-to-one LF SR framework (LF-ATO), where each view of an LF image is first individually super-resolved by exploring the complementary information among views using combinatorial geometry embedding. Then, to preserve the parallax structure among reconstructed views, they used a regularization network. In addition, Wang et al., [38] developed a spatial-angular interactive network (Distg-Block) where spatial and angular features are separately extracted from LF images and then repetitively interacted for progressive incorporation of the spatial and angular information. Even though, these methods exploit the 4D spatial and angular information of the light field, however, they use a pre-defined downscaling operator to generate LR LF images, which are not close to real-world LR images. In addition, they do not use progressive and multi-scale up-sampling, instead they perform one up-sampling step, 21 which makes learning mapping functions for large scaling factors more difficult. Finally, existing methods optimize the networks with an L1 or L2 loss functions which corresponds to over-smoothed reconstructed HR images that are inconsistent to human visual perception on natural images. 1.5 Literature Review on Angular Super-resolution Light Field is a rapidly evolving technology, supporting capturing and displaying of visual information that approximates the human eye. The densely sampled LF images is an essential technical component for delivering an immersive LF experience with realistic disparity, smooth motion parallax between views, and natural refocusing on the different regions of the image. However, the enormous amount of data generated are a challenge, as they require significant increase in bandwidth. A possible solution may be to drop specific views at the transmitting end and effectively synthesize them at the receiver end or when they are needed (as seen in Figure 1.15), thus minimizing the amount of data needed to be transferred or stored, with minimal impact Figure 1.15 Overview of angular super-resolution approach 22 on the visual quality on the LF content. Light field view synthesis aims at reconstructing a densely sampled novel LF views from a sparse set of views. In other words, it is super-resolving LF data along the angular dimension. Various techniques have been proposed that address the need for angular SR by synthesizing virtual views using a set of sparsely sampled views in light field. These methods can be classified into two categories: depth-dependent methods and depth-independent methods. Depth-dependent view synthesis approaches are typically based on first estimating a depth map and then use this information to warp the existing views to generate the virtual view at target angular positions based on the estimated disparities [39]. Similar approach has been modeled using CNNs by Kalantrair et al. [40]. In their work, they break down the process into disparity and color estimation components, and then they use two sequential convolutional neural networks to model these two components and train both networks simultaneously by minimizing the error between the synthesized and ground truth images. Similarly, Navarro et al., [41] proposed a learning-based method to extract features and estimate dispersity. However, both methods fail to generate good results in the presence of occlusions. In addition, these approaches struggle against inputs with large baseline as the depth estimation component fails to capture long-distance correspondences. To deal with wide baseline images, Flynn et al., [42] proposed to use the variational optimization framework in conjunction with deep learning. Their model takes as input a sparse set of views, processes the input images using learned gradient descent to produce a volume of in-between views, and then uses a CNN to predict the novel views from the input volume. A drawback of this approach is the need to build the volume of in-between views, which is memory and time 23 consuming. Jin et al. [43] proposed a CNN to estimate 4D depth maps and utilized these depth maps to synthesize novel views by backward wrapping. However, the performance of such depth-estimation based approaches is heavily relied on the accuracy of estimated depth maps and they tend to be very sensitive to texture, reflection and occlusion. For depth-independent approaches, various learning-based schemes have also been proposed for LF angular super-resolution from a sparse set of views. Yoon et al. [23] proposed CNN for both spatial and angular super-resolution by first up-sampling each LF sub-aperture image individually and then synthesize novel views one by one based on neighboring input views. However, the performance of this approach is very limited, as the relations between input views are not explored. Wu et al. [44] proposed a CNN model to map blurred Epipolar Plane Images (EPIs) from low to high resolution. However, their “blur-restoration-deblur” approach requires heavy pre- and post-processing steps, making them impractical for the majority of LF applications. To leverage the 4D information in the light field content, Wang et al. [45] proposed a pseudo 4D CNN to increase the LF angular resolution. In their work, they up-sample stacked EPIs using 2D convolutions and then they apply 3D CNNs to restore high-frequency details of volumes in the row or column pattern. Even though these approaches exploit the information available in the 4D light field volume, however, the developed networks only process information within a single domain at each stage, and thus they do not take advantage of the deeper representation in the spatial and angular dimension. Yeung et al. [46] developed a LF reconstruction CNN by exploring the correlation between LF sub-aperture images through pseudo 4D filters with spatial and angular alternating convolution. Meng et al. [47] introduced a high-dimensional dense residual CNN with 4D convolutions to extract LF spatial and angular features and use this information to generate in- 24 between views. However, the performance of such approaches depends on the architecture, the loss function, and the training dataset. 1.6 Thesis Outline In this thesis, we present novel methods for increasing the spatial and angular resolutions of light field content while considering disparity consistency between views. More specifically, our contributions include (1) modifying the existing single-image-super-resolution SISR learning-based model to up-sample the spatial resolution of light field content using EPI representation, (2) the introduction of a novel learning-based model to up-scale light filed content progressively by exploiting the 4D spatial and angular information, (3) the development of a new method to generate new light filed content using EPI representation and finally (4) the introduction of a novel deep-leaning based scheme that generates new virtual views for light field content. Figure 1.16 presents a visualization of the presented contributions. Figure 1.16 Thesis Contributions. Contribution 1 and 2 are presented to solve the spatial resolution issue and Contribution 3 and 4 are proposed to solve the angular resolution issue. 25 In Chapter 2, we present contribution (1) where we propose a learning-based model for spatial super-resolution which considers simultaneously all the sub-aperture images and also takes advantage of the epipolar image plane (EPI) information present in light field to ensure smooth disparity between the generated views and in turn construct high spatial resolution light field sub-aperture images. Performance evaluations have shown that the proposed approach outperforms other state-of-the-art LF spatial super-resolution methods. In Chapter 3, we discuss contribution (2) where we investigate the common drawbacks of current LF super-resolution methods. Based on our findings, we develop a novel deep learning spatial SR approach that considers the spatial (i.e., information within each view) and angular information (i.e., information among other views) of the LF images and progressively reconstructs high-resolution LF images at different upscaling levels. Specifically, we introduced a spatial and angular features extraction approach that preserve local features information from current and preceding layers and then use global feature fusion to adaptively learn global features as well in a holistic way. The model then uses a Laplacian framework that progressively predicts high-frequency residuals in a coarse-to-fine manner to reconstruct high-resolution LF images. We optimize the proposed model using a robust Charbonnier loss function and achieve high-quality reconstruction. Additionally, we train the proposed approach on low-resolution LF images generated using a 4D LF downsampling framework to achieve high-quality reconstruction. Experimental results demonstrate the superiority of our proposed method over state-of-the-art light field super-resolution techniques. 26 In Chapter 4, we explain contribution (3) and (4) where we introduce two learning based LF angular super-resolution (i.e., view synthesis) approach that address the bandwidth challenge of the enormous amount of generated data. To this end, efficient transmission and/or storage of LF content is achieved by dropping specific views at the transmitting end and then synthesizing them at the receiver end, with minimal impact on visual quality. Our modified deep recursive residual network architecture uses EPI to ensure smooth disparity between the generated and original views. We consider plenoptic as well as camera array implementations as well as synthetic content which support different baseline settings. We also introduce a second deep learning-based angular super-resolution approach that utilizes a Generative Adversarial Network (GAN) by first extracting spatial and angular feature maps using a residual sub-network. Then, we reconstruct the novel LF views by up sampling the angular resolution of the extracted feature maps using sub-pixel operation. Thus, the proposed network preserves the intrinsic structure of the reconstructed densely sampled LF among the synthesized novel sub-aperture images. Performance evaluations have shown that our model outperforms state-of-the art angular super-resolution methods. It should be noted that the network architectures for contributions 1 and 2 are similar to those of contributions 3 and 4, respectively, while the aim is different. In contributions 1 and 2, we use the learning-based models to increase the pixel density (i.e., spatial resolution) of each image, while in contributions 3 and 4 we use the learning-based models to increase the number of LF views (i.e., angular resolution). The methods used to increase the spatial and angular resolutions in contributions 1 and 3, are also different. In addition, we use 25 RUs in contribution 1 and 20 RU in contribution 3. The reason 27 behind the increase of the number of RUs in contribution 1 is that we have more data for training with a variety of textures and reflections, thus, increasing the number of RU increases the network accuracy without reaching the point of saturation where the network could overfit. On the other hand, since we do not have much data for contribution 3, increasing the number of RU led to overfitting. Thus, our experiments suggested that the optimal number of RUs is 25 and 20 for contributions 1 and 3, respectively, as they yielded the highest accuracy without overfitting the data. Regarding contributions 2 and 4, the only similarity is the features extraction part of the deep learning models. In addition, for contribution 2, we have different sub-network architectures for the downsampling and up-sampling parts of the framework that is unique for the spatial super-resolution problem. For contribution 4, we use an up-sampling technique that works better for the angular super-resolution. Finally, in Chapter 5 we present the main conclusions of this thesis and the significance of our contributions as well as we discuss possible future work. 28 Chapter 2: Light Field Spatial Super-resolution using EPI Existing LF acquisition approaches that capture LF images either require a set-up of multiple cameras (camera array) [48], or the microlens based LF approach used in plenoptic cameras [10], [49]. A camera array is typically very large and not portable, thus limiting its use within only a controlled environment. Furthermore, the multiview images [15] captured with the camera array setup, have high-spatial-resolution but the angular resolution, meaning the proximity between the neighboring views, is limited by the distance between the cameras. In plenoptic cameras, although the sub-aperture views are close together, their spatial resolution is very low due to the size of the camera sensors. Comparing both plenoptic camera and camera array capturing approaches, we can see that with a plenoptic camera, the spatial resolution of each image is sacrificed in order to achieve a higher angular resolution due to the limited sensor resolution. To mitigate this problem, we propose a learning-based model for spatial super-resolution which considers simultaneously all the sub-aperture images and also takes advantage of the epipolar image plane (EPI) information to ensure smooth disparity between the generated views and in turn construct high spatial resolution light field sub-aperture images. In this chapter, we introduce a new approach, that uses a deep learning network to generate EPI volumes for the upscaled sub-aperture images in a way that takes advantage of the neighbouring similarities of the light field views. To this end, we feed the horizontal EPI to the deep learning 29 model, obtain the row super-resolved images, extract the vertical EPI from these row super-resolved images which is fed again to the deep learning model, and then extract the final super-resolved images. A visual representation of proposed approach that includes the input and out EPI volumes and our deep learning model is shown in Figure 2.1, where blue squares represent the Batch normalization layer, the green ones represent the ReLU layers, and the red ones represent the Weight layers. Performance evaluations against the state-of-the-art approaches presented in [35] and [33] showed that our method achieves superior performance, achieving 1.88 dB and 1.79 dB improvement in PSNR, respectively. Our contributions are as follows: • We base our approach to the DRRN architecture [50], which is modified to apply to light field content. To this end, we modify the number of recursive blocks, residual units, layers, filters, filter sizes, and learning rate to serve our light field data. Figure 2.1 Overview of the proposed approach, showing the input and output EPI volumes and our deep learning model for LF super-resolution. 30 • We take advantage of the full 4D light field volume by feeding the horizontal EPIs to the model, use the output to obtain up-sampled sub-apertures. Then, feed the vertical EPI of these up-sampled sub-apertures to obtain the high-quality super-resolved images. • We train the model using EPIs extracted from datasets obtained using different approaches including plenoptic camera, camera array, and synthetic data, thus, accommodating different disparities represented by the EPI structure. 2.1 Proposed Spatial Super-resolution Framework Our goal is to increase the light field content spatial resolution of microlens sub-aperture images. In the proposed approach, we take advantage of the redundancies in all the views by working with the EPI in two stages. Figure 2.2 illustrates the architecture of the proposed model. First, we bicubically upscale each low-resolution sub-aperture image to increase the pixel density of each image to match that of the target resolution, e.g., x2, x3 or x4, as shown in step 1 of Figure 2.2 (a). Since this step bicubically interpolates each sub-aperture image separately without considering the other views, the pixels of the interpolated images may be dispersed in a different way than the original image. To address this issue, we exploit the epipolar image plane (EPI) structure of light field to ensure smooth disparity between the generated views. To this end, the bicubically up-sampled sub-aperture images are stacked, and the horizontal EPI of each row is obtained. In the second step, we feed the EPIs of the bicubically up-sampled sub-aperture images to a spatial convolutional neural network, which enhances the EPI structure, recovering the high-frequency details of each sub-aperture images while adjusting the disparities between the views. We extract the CNN-ROW up-sampled sup-aperture images from the EPI of the CNN-ROW up-sampled images. Next, we stack these sub-aperture images to obtain the vertical EPI of each column, feed 31 them to the deep learning model and construct the final super-resolved light field sub-aperture images, i.e. CNN-COL up-sampled sub-aperture images, as shown in Figure 2.2 (b). It is worth noting that based on our experimental results, we came to the conclusion that changing the order of applying the deep learning model on the horizontal EPI and vertical EPI, i.e., horizontal EPI then vertical EPI vs vertical EPI then horizontal EPI, is inconsequential. 32 (a) (b) Figure 2.2 Overview of the proposed framework for each: (a) row (b) column. Bicubic Up-samplingDeep LearningHorizontal EPI representation of Bicubicup-sampled imagesHorizontal EPI representation of CNN-ROWup-sampled imagesBicubicup-sampled sub-aperture imagesCNN-ROWup-sampled sub-aperture imagesLow-resolution sub-aperture imagesStep 1 Step 2 33 There are many Deep Learning networks that have shown to work for image processing applications and more specifically upscaling. We have decided to adapt the Deep Recursive Residual Network (DRRN) model for the advantages in its use of a multi-path model and recursive learning [50]. Its proven performance in super-resolution and ease of training with limited datasets make it an attractive starting point for our implementation. Architectures used for single image super-resolution such as SRCNN [51] may also be attractive but showed lower performance due to the fact that this model does not consider self-similarity (i.e. repeated texture), which is a main property of the EPI structure that consists of lines with different slopes and patterns. DRCN [52] is another architecture that can be used for spatial super-resolution, however, it uses very deep network which requires enormous parameters, thus, demand more storage space and are less applicable to mobile systems. Thus, the network proposed for single image super-resolution in [50] was used as basis for our spatial SR network design. Our approach is described in detail in the following subsections. 2.1.1 Epipolar Image Plane (EPI) Light field is a four-dimensional (4D) function that defines the direction and the amount of light that goes through every point in space, commonly parameterized as L (x, y, s, t) in two planes (see Figure 1.1 (b)). An epipolar image plane (EPI) is obtained as explained in Chapter 1, section 1.3.3. As Fig. 1.13 (in Chapter 1, section 1.3.3) indicates, EPI consists of lines with various slopes when it is viewed from the top or the side. Here, each pixel of the first image is tracked throughout all the images, and this trajectory appears in the EPI as a continuous straight line. Each such line in an EPI is 34 related to the depth of the corresponding point in space, (i.e., the larger the slope, the larger is the disparity across the views). Thus, the lines in the EPI define the geometric structure of light field. Therefore, by taking advantage of the EPI structure, we can recover the texture details of the images while correctly estimating the disparity among light field images, which provides the necessary consistency that will result in highly accurate super-resolution results. The EPI in this thesis is considered to have a linear structure. However, when the camera arrangement is circular, the generated LF data would have a non-linear EPI structure. Our proposed approach may be applied to the nonlinear EPI structure, as well with the difference that we would have to to define a circular EPI image similar to the linear case by compiling the camera images following the order of the camera positions on the circle and re-train the proposed super-resolution model using the generated data. 2.2 Super-resolution Model In the first step of our spatial SR approach, we upscaled each sub-aperture image to a target resolution using bicubic interpolation. The EPIs of the bicubically interpolated sub-aperture images are fed to a spatial convolutional neural network, which is trained as explained in the next subsection to enhance the EPI structure, recovering the high-frequency details and adjusting the disparities between the views. 2.2.1 Implementation Details We based our implementation on the DRRN [50] architecture as it features global residual learning, which basically means carrying most of the input values to the end of the network, thus 35 considering the self-similarity between the input and output of the network since our goal is to benefit the EPI structure. Figure 2.3 (a) Super-resolution CNN structure. (b) overview of a residual unit. (c) residual unit layers. 36 For our implementation, we explored different combinations of recursive blocks and residual units, including combinations suggest in the DRRN paper [50], to test how these two parameters affect the performance of the model on our data. Based on our tests results, a modified network structure with one recursive block and 25 residual units (represented by red box in Fig. 2.3 (a)) followed by a convolutional layer achieves superior performance when trained using the EPI data. We further compared the proposed network structure with different filter sizes 3x3, 5x5, and 7x7. The results suggest that a better performance is accomplished when smaller filter size 3x3 is used. This could be explained by the fact that smaller filters collect as much local information as possible. However, for the last convolutional layer (represented by green box in Fig. 2.3 (a)), a larger filter size 5x5 produces better results in restoring the pattern structure of the EPI. The model takes as an input the EPI Ebicubic of the bicubically up-sampled sub-aperture images and outputs a residual image R. It basically learns a residual image R, which is the difference between the EPI of the bicubically up-sampled LR sub-aperture images “Ebicubic” and the EPI of the HR ground truth sub-aperture images “EHR”. The residual image is then added to the bicubically up-sampled EPI Ebicubic to estimate the HR EPI Eestimated_HR, as shown in Figure 2.4. Figure 2.4 Overview of the reconstruction function of the super-resolution model. 37 The recursive block contains a convolutional layer followed by 25 residual units. The first convolutional layer extracts batches from the EPI and represents each batch as a local feature map (or activation map). All the following 25 residual units share the same input from the identity branch of that convolutional layer, represented by a dashed line in Fig. 2.3 (a). Global Residual Learning is adopted in the identity branch. This branch accommodates the self-similarity (or correlation) between the input and output values by carrying rich image details until the end of the network. Furthermore, Local Residual Learning is achieved through the residual branch. The identity paths help in learning low-level-features (such as edges, lines, etc.), while the residual paths help in learning high-level-features (such as color gradients, luminance, patterns, etc.). Consequently, this multi-path residual-based model facilitates learning as the residual moves features forward all the way to the end of the network, and overfitting is avoided because of the identity path and avoiding gradient vanish. Each of the 25 residual units consists of two convolutional layers, as shown in Fig. 2.3 (b), where u is the number of residual units (i.e., u = 1, 2, …, 25), xu and xu-1 are the output and input of the uth residual unit, and x0 is the output of the first convolutional layer in the recursive block. Each one of these two convolutional layers contains a batch normalization (BN) layer, a Rectified Linear Units (ReLU) layer, and a weight layer, as shown in Fig. 2.3 (c). A batch normalization (BN) layer normalizes the output of a previous layer by adjusting and scaling the activation values. This batch normalization speeds up training. For our implementation, based on our data, we used a batch size of 128. A Rectified Linear Units (ReLU) layer is basically a non-linear activation function f(x) = 38 max(x,0) which passes all the positive values and zeros the negative ones. Finally, the Weight layer consists of the weights that are updated during backpropagation according to the gradient and learning rate. Each layer has 128 filters of size 3x3. The residual unit can be represented mathematically as follows: 𝒙𝒖 = 𝑭(𝒙𝒖3𝟏,𝑾) + 𝒙𝟎 2.1 where F denotes the function of the residual unit, represented in Fig. 2.3(b). It non-linearly maps the batches of feature maps to another set of feature maps using a weight set W, and x0 is the output of the first convolutional layer in the recursive block. The output of the final recursive block xu, i.e. u = 25, is then fed to a convolutional layer, represented by a green box in Fig. 2.3 (a), to construct the residual image R. This layer generates the residual image by aggregating the patch wise representation output from the recursive block. In other words, the representation of the feature maps out from the recursive block is reshaped and averaged to produce the final residual image. The residual image R is then added to input Ebicubic image to estimate the HR EPI Eestimated_HR, as shown in Fig. 2.3 (a). This can be formulated as: 𝑬𝒆𝒔𝒕𝒊𝒎𝒂𝒕𝒆𝒅_𝑯𝑹 = 𝑹 + 𝑬𝒃𝒊𝒄𝒖𝒃𝒊𝒄 = 𝑭𝑪𝑳(𝒙𝒖) + 𝒙 2.2 where FCL is the reconstruction function of the last convolutional layer (CL) in our CNN model, xu is the output of the recursive block and x is the input Ebicubic. 39 The loss function used is the mean square error (MSE) between the residual predicted by the network 𝑅D and the corresponding ground truth residual R and is defined as: 𝑳(𝜽) = 𝟏𝟐𝑵∑ G𝑹H𝒊 − 𝑹𝒊G𝟐𝑵𝒊0𝟏 2.3 where 𝜃 denotes the parameter set, and N is the number of training patches. 2.2.2 Training We trained our model using images captured by a plenoptic camera (Lytro) [53] as well as the Stanford Lytro Light Field Archive [54], which includes real-world light field each composed of 196 views arranged in 14x14 grid. We also use synthetic light field data [55], which include 24 synthetic light field scenes each composed of 81 views arranged in a 9x9 grid. In addition, we used half of images in the real-world datasets captured with camera array from the Stanford dataset [15]. Using light field datasets obtained with different approaches, in the training step, makes our model more robust, thus, accommodating different setups. We increased the number of training data by using two data augmentation techniques on the training set, horizontal flip and the addition of Gaussian noise. This augmentation helps to avoid overfitting and improves the training process overall. It must be noted that none of the data used for testing were used for training. We synthesized LR images by down-sampling and then up-sampling the original images and used the original images as ground truth. We then generated the EPI by stacking each row of the 40 bicubically up-sampled sub-aperture images and used the EPI of the original sub-aperture images as the ground truth. After generating the horizontal EPI of the CNN-ROW up-sampled images for each row (step 2 of Fig. 2.2 (a)), we reconstruct the CNN-ROW up-sampled sub-aperture images and use it to obtain the vertical EPI for each column of the sub-aperture images. We feed these vertical EPIs, to the network, to get the EPI of the CNN-COL up-sampled image, as shown in step 3 of Fig. 2.2 (b). Finally, we reconstructed the estimated HR sub-aperture images using these generated vertical EPIs of the CNN-COL up-sampled image. The weight values for each layer are randomly initialized from a Gaussian distribution with zero mean and standard deviation 0.001. The task of minimizing the loss function involves trying to adjust the weights so that the loss decreases. The network is trained using stochastic gradient descent with backpropagation [56]. In order to achieve higher test accuracy, we choose to first initialize the learning rate 𝜂 to 10-1 and then decrease it by a factor of 2 every 20 epochs. 2.3 Experiments and Evaluations The performance of our spatial SR model is compared with that of the spatial SR part of LFCNN [23] and D-LFSSR [35] which use a CNN model to increase the spatial resolution of light field images. We also compared our method against the GMM method presented in [26] and the graph-based (GB) method described in [27], since they both consider the geometric consistency of the sub-aperture images in their framework. We further compared our method with LF-DCNN [33] as they use CNN on EPI. The performance of comparative methods [23], [26], [27], and [35] were 41 obtained using the source codes provided by the authors. Since the LF-DCNN [33] code is not available online, we compared our method against the numbers presented in their paper. The real-world datasets captured with camera array from the Stanford dataset [15] and Plenoptic camera (Lytro Ilum) [53] and the HCI dataset [55] are used for the performance evaluations. We should note that these datasets were not used for training nor validation. 2.3.1 Experimental Evaluation For testing purposes, we first down-sample the test images and then apply our trained model to these low spatial resolution images to upscale them to their original resolution. We utilize the peak-signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) quality metrics for numerical comparison between the ground truth test images in their original resolution and the upscaled version of the test images using different approaches. We also assess the visual quality of the upscaled scenes using the Video Multimethod Assessment Fusion (VMAF) quality metric [57], which is known to yield the best correlation with human perception. VMAF is a perceptual video quality assessment algorithm developed by Netflix. It is a combination of several spatial and temporal metrics including Visual Information Fidelity (VIF), Detail Loss Metric (DLM), and Motion, which are fused using a machine-learning algorithm, specifically a Support Vector Machine (SVM) regressor. Since the LF-DCNN [17] code is not publicly available, we are using the results presented in their paper, where they report the scores of upscaling with a factor of x2 and x3. For a fare comparison, we follow the LF-DCNN [17] testing approach, so we down-sample the test images using bicubic interpolation before upscaling it back to the original resolution using different approaches, 42 including our proposed model. We then average the quality metrics over 18 scenes and the results are provided in Table 2.1, which compares the average PSNR and SSIM values on the test images for the central view of each scene. We observe that, for all of the approaches, upscaling with a factor of x2 yields better results than upscaling with a factor of x3. We can further observe that our method improves the PSNR results compared to LFCNN [23], GMM [26], GB [27] and D-LFSSR [35], and LF-DCNN [17] by an average of 3.55, 3.41, 2.66, 1.88, and 1.79 dB, and the SSIM results by an average of 0.018, 0.043, 0.017, 0.009, and 0.006, respectively, confirming its superior performance. To demonstrate the effectiveness of the proposed approach, we further compare the different methodologies using three specific scenes that contain different texture details and reflection regions. These scenes were captured using both a plenoptic camera (bikes_2 and reflective_22 [22]) and a camera array setup (Amethyst [4]). We first down-sample the aforementioned scenes with a factor of x2 and x4, using bilinear interpolation and then upscale it back to their original resolution. The results are provided in Table 2.2, which compare the PSNR, SSIM, and VMAF on Datasets Methods ILLUM (x2) ILLUM (x3) Stanford (x2) Stanford (x3) HCI (x2) HCI (x3) PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM LFCNN 37.56 0.967 32.01 0.893 37.42 0.971 31.75 0.873 35.85 0.927 31.17 0.862 GMM 36.32 0.938 32.34 0.897 37.55 0.974 31.98 0.903 37.4 0.942 33.46 0.835 GB 37.34 0.968 31.2 0.936 37 0.97 30.24 0.928 36.18 0.938 32.43 0.867 D-LFSSR 39.92 0.98 35.39 0.943 39.37 0.979 33.23 0.934 39.55 0.96 35.03 0.896 LF-DCNN 39.89 0.979 35.97 0.945 40.61 0.984 35.07 0.956 38.63 0.954 34.84 0.883 Proposed 40.28 0.985 36.73 0.974 44.45 0.995 38.94 0.969 39.76 0.968 36.02 0.904 Table 2.1 Comparisons Against Different Approaches 43 the central view of each scene (i.e., the image of row #7 and column #7 for each scene). As expected, we observe that the images in Table 2.2 that are upscaled with a factor of 2 produce better results than those upscaled with a factor of 4. 44 From Table 2.2, we notice that upscaling the Amethyst scene with a factor of x2 using our proposed approach yields 90.58 in VMAF, 42.86 dB in PSNR, and 0.9709 in SSIM while the second-best Methods Datasets LFCNN GMM GB D-LFSSR Proposed Amethyst (x2) VMAF 63.44 82.16 68.76 85.79 90.58 PSNR 35.12 37.22 36 37.37 42.86 SSIM 0.9075 0.9052 0.9201 0.9251 0.9709 Amethyst (x4) VMAF 56.95 70.27 67.67 79.20 88.24 PSNR 29.84 31.96 30.44 35.09 37.63 SSIM 0.8653 0.8601 0.8738 0.8824 0.9296 Reflective_22 (x2) VMAF 74.68 64.12 68.84 83.68 87.64 PSNR 36.54 36.66 34.92 38.11 38.52 SSIM 0.9544 0.9542 0.8957 0.9601 0.9663 Reflective_22 (x4) VMAF 45.98 14.05 52.87 59.33 75.33 PSNR 32.88 33.06 29.33 35.03 35.87 SSIM 0.8816 0.8938 0.8482 0.9453 0.9606 Bikes_2 (x2) VMAF 53.39 59.49 71.56 76.24 85.10 PSNR 38.81 39.09 36.09 38.62 41.2 SSIM 0.9461 0.9663 0.8403 0.8694 0.9792 Bikes_2 (x4) VMAF 25.01 50.17 41.06 63.45 81.91 PSNR 32.37 35.23 33.68 34.42 38.88 SSIM 0.8872 0.9036 0.8023 0.8117 0.9465 Table 2.2 Comparisons of Four Different Scenes 45 approach, i.e., D-LFSSR [35] results in 85.79 in VMAF, 37.37 dB in PSNR, and 0.9211 in SSIM values. Thus, our method improves the VMAF, PSNR, and SSIM values by 4.79, 5.49 dB, and 0.0498, respectively. When up-sampling with a factor of x4, our approach improves the VMAF, PSNR and SSIM values by 9.04, 2.54 dB and 0.0322 over D-LFSSR [35]. We can notice that the gain over the D-LFSSR decreases by half in terms of PSNR, while it increases by a factor 1.88 in terms of VMAF. This difference between the two up-scaling levels is probably due to the fact that PSNR relies strictly on numeric comparison and does not actually take into account human perception, as any slight change in luminance would impact the PSNR results. On the other hand, VMAF gives us the confidence that the quality is acceptable as it approximates human perception of video quality. Finally, for the scenes captured with a plenoptic camera, i.e., reflective_22 and bikes_2, we can observe that images upscaled using our method result in improved VMAF, PSNR, and SSIM compared to LFCNN [23], GMM [26], GB [27] and D-LFSSR [35], as shown in Table 2.2. These results are further confirmed in Figure 2.5, where images upscaled using other approaches look either blurred, pixelized, or contain other visual artifacts. 46 (a) (b) Figure 2.5 Comparison showing the ground truth central view with a cropped close-up, and its interpolated versions using different approaches of the: (a) reflective_22, (b) bikes_2 scenes. 47 A visual quality evaluation of our model using the Red Flower scene [53] can be seen in Figure 2.6, which shows the central view generated using the four different schemes. As we can see in Figure 2.6 (a), the stamen part of the flower, represented by the blue square, appears the sharpest using our proposed method. Furthermore, Figure 2.6 (b) provides the EPI extracted from the reconstructed HR images. The smooth line patterns in the EPI produced by our proposed method confirms that it preserves the geometric consistency which in turn yields sharp high-quality high-resolution images. We can conclude that compared to the other methods, our proposed approach generates the best visual quality. (a) (b) Figure 2.6 Qualitative comparison of different methods using Red Flowers scene [22] (a) Close up of the high-resolution central view of the sub-aperture images. (b) EPIs generated from the high-resolution sub-aperture images. 48 Fig. 2.7 presents the central view generated using the four different schemes for the Garden and Chair scenes [53]. We observe that our method yields high resolution images whose visual quality is significantly better than that of the other methods. The Chair scene further verifies the advantage of our method, i.e. the power outlet, represented by the red square, appears the sharpest compared to other methods. 49 (a) (b) Figure 2.7 Comparison showing the high-resolution central view of the sub-aperture images from (a) the Garden and (b) the Chairs scenes [22] using different approaches. 50 2.3.2 Subjective Evaluation We subjectively evaluated our method using 10 test scenes captured using Lytro Ilum in our lab [53]. We upsampled the sub-aperture images using the proposed approach by a factor of x2 and chose the central view of each of the following scenes: Yellow Flowers, Car Wheel, Garden, Car Light, Figure, Chairs, Shiny Car, Bikes, UBC Sign, and Flower. We subjectively evaluated the visual quality of the high-resolution sub-aperture light field images generated by our method against the LFCNN [23], GMM [26], GB [27] and D-LFSSR [35]. We could not compare our method subjectively against LF-DCNN [33] as the code is not available online. The subjective studies in this work are in accordance with the ethical standards of the UBC Ethics Board (H12-00308). The evaluation method was performed on a single display based on the Side-by-Side presentation [58]. We used this evaluation process for its high accuracy and reliability in constructing a scale of perceptual preference while minimizing the visual working memory limitations. One side was always the estimated HR sub-aperture images generated by our proposed method, while on the other side the estimated HR content generated by LFCNN, GMM, and GB methods. The two sides were shown to the observers in a random order. As described earlier, ten different scenes were used, i.e., three pairs for each content resulting in 3x10 = 30 comparisons. The stimuli were labeled A and B on the left side and the right side of the display, respectively. Subjects were asked to rate which image in a pair of A and B they preferred in quality/resolution. We also included the option same to avoid random preference selections. The order of scenes in each session of the test was randomized and we made sure that the same content is not shown 51 consecutively. The test took approximately 10 minutes. A training session was organized using 3 additional contents that were not part of the test dataset, to familiarize the participating subjects with the assessment procedure. A total of 18 adult subjects (12 males and 6 females) participated in our experiment. The subjects’ age range was from 20 to 33 years old. Prior to the tests, all the subjects were screened for color blindness using the Ishihara chart, and visual acuity using the Snellen charts. Those subjects that failed the pre-screening test did not participate in the test. After collecting the subjective results, outliers were determined according to the ITU-R BT.500-13 recommendation in [59]. One outlier was detected and was removed from the test results. Figure 2.8 illustrates the preference probabilities for each scene using each super-resolution method. Images generated by our proposed method were largely preferred over images produced by the other methods. Especially when comparing our method against LFCNN. We believe the main reason behind this is that LFCNN does not consider all the light field views as it generates each HR image separately. However, for Car Light and Shiny Car scenes, in Proposed-GB paired comparison, almost half of the times the proposed method and GB results were rated “same”, and the rest half the proposed was preferred over GB. One reason may be that these two images feature scenes with smooth surfaces, which means that they do not contain fine details or texture. 52 2.4 Conclusion In this Chapter, we have presented a new deep learning approach method for LF spatial SR using the EPI representation. Our proposed end-to-end deep learning model constructs high-resolution images and enforces the disparity continuity between the views to achieve high quality results. We (a) (b) (c) (d) Figure 2.8 Subjective of paired comparison of our proposed method against: (a) LFCNN, (b) GMM, (c) GB and (d) D-LFSSR. 0%10%20%30%40%50%60%70%80%90%100%Yellow FlowersCar WheelCar LightGardenChairsFigureShiny CarBikesUBC SignFlowerNumber of SubjectsProposed Vs. LFCNNLFCNN Same Proposed0%10%20%30%40%50%60%70%80%90%100%Yellow FlowersCar WheelCar LightGardenChairsFigureShiny CarBikesUBC SignFlowerNumber of SubjectsProposed Vs. GMMGMM Same Proposed0%10%20%30%40%50%60%70%80%90%100%Yellow FlowersCar WheelCar LightGardenChairsFigureShiny CarBikesUBC SignFlowerNumber of SubjectsProposed Vs. GBGB Same Proposed0%10%20%30%40%50%60%70%80%90%100%Yellow FlowersCar WheelCar LightGardenChairsFigureShiny CarBikesUBC SignFlowerNumber of SubjectsProposed Vs. D-LFSSRD-LFSSR Same Proposed 53 have demonstrated the effectiveness of our method by using synthetic data as well as real-world scenes captured with a plenoptic camera. We further compared our method with state-of-the-art techniques and the results confirm the superior performance of the proposed approach. 54 Chapter 3: Light Field Super-resolution using 4D Spatial and Angular Information In this chapter, we introduce a new approach that first uses a downsampling framework, based on the single-image downsampling approach presented in [60], by estimating downsampling kernels and noise distribution of the 4D GT LF images to generate 4D LR LF images that share common domain attributes with their HR counterparts. Then, we feed the 4D LR LF images to a generative adversarial network (GAN) framework to generate the 4D up-sampled LF images by extracting spatial and angular features using a residual network inspired by RDN [61] and progressively reconstructing the up-sampled LF images using a Laplacian pyramid framework based on LapSRN [62]. The proposed network super-resolves the LR LF images progressively at scale factor of 2 and 4. Therefore, we have two Generators sub-networks for each up-sampling level. Finally, we train the GAN framework to minimize three losses: pixel loss, perceptual loss, and adversarial loss using the Charbonnier loss function, which handles outliers better than the L1 and L2 loss functions and generates sharper images, thus, improving the performance based on our abolition study and the extensive quantitative and qualitative evaluations on benchmark datasets presented in section III. A visual representation of the proposed approach at the training phase is shown in Fig. 3.1 and at the testing phase in Fig. 3.2. Performance evaluations against the state of the art approaches presented in [36], [35], [37], and [38] showed that our method achieves superior performance in terms of PSNR, SSIM, and VMAF. Our proposed work differs from existing LF SR CNN-based frameworks that consider the 4D LF information in the following aspects: 55 • Instead of using a fixed downsampling operator (i.e., bilinear or bicubic) to construct LR and HR pairs for training, which generate LR images that do not belong to the same domain as the original image, we use a downsampling framework presented in [60], which is modified to apply to 4D light field content. • Our spatial and angular feature extraction uses hierarchical features from LR images, extracting local features using dense connected convolutional layers to learn more effective features from preceding and current layers. Then we use global feature fusion to learn global hierarchical features in a holistic way before performing the up sampling. • We use a SR GAN framework where the Generator network super-resolves LR LF images progressively at scale factors of 2 and 4. Thus, we have two Generators sub-networks for each up-sampling level and only one discriminator. Figure 3.1 Overview of the proposed approach at training phase. 56 The rest of this chapter is organized as follows: Section 3.1 presents an overview of the proposed LF super-resolution framework. Ablation study and experimental results are presented and discussed in Section 3.2. Finally, Section 3.3 provides the concluding remarks. 3.1 Proposed Light Field Spatial Super-Resolution Framework An overview of the proposed approach is presented in Fig. 3.1. First, we generate the LR sub-aperture light field (SA-LF) images from the GT SA-LF images through a downsampling framework by estimating downsampling kernels and noise distribution of the GT images. Then, we reorganize the pixels of the downsampled SA-LF images to get the downsampled Macro-pixel light field (MP-LF) images and feed them to a GAN framework to generate the super-resolved LF images. For the Generator, we first extract spatial and angular features from the MP-LF image. Then, we feed the extracted spatial and angular features to a residual sub-network to obtain spatial and angular feature maps. These feature maps are then fed to a Laplacian up-sampling sub-network, which generates the HR SA-LF images at multiple scales in one feed-forward pass, Figure 3.2 Overview of the proposed approach at testing phase. 57 through progressive reconstruction. Note that, in Fig. 3.1, the Generator has one sub-network for each up-sampling level, while we only have one Discriminator. In a sense, each level output from the Generator is passed once through the same Discriminator. For the GAN Discriminator, we use a patch Discriminator, which learns the image patch distribution by looking at multiple local patches of the image and judging whether each patch is real or not. In other words, the Generator is trained to produce an upscaled version of the LR LF image, such that the Discriminator cannot distinguish between the patch distribution of the upscaled image, and the patch distribution of the GT image. Thus, the Generator is trained to fool the Discriminator to believe that all the patches of the upscaled image were actually taken from the original one. We train the proposed model using a Charbonnier loss function and achieve high-quality reconstruction. Finally, we conduct ablation experiments to verify the effectiveness of the following: the generation of LR LF images using downsampling framework, the GAN framework, the patch discriminator, and the Charbonnier loss. The evaluation results demonstrate that our method outperforms the state-of-the-art methods. Our approach is described in detail in the following subsections. 58 3.1.1 Downsampling Framework Current LF SR approaches train a network through paired low and high-resolution images, which are generated using an ideal downsampling kernel, such as bicubic or bilinear. However, such LR images are not available in real-world applications. Thus, the aim here is instead to generate LR LF images that share a common domain with real-world images. We base our proposed downsampling framework on the work presented in [60]. As shown in Fig. 3.3, we adopt a similar approach using 4D kernels instead of 2D kernels and set appropriate parameters based on the 4D SA-LF images. (a) (b) Figure 3.3 Overview of (a) the downsampling approach (b) KernelGAN framework. 59 First, we generate sharper (i.e., HR) SA-LF images (ISA_HR) from the GT SA-LF images (ISA_GT) through bicubic downsampling to remove noise. This can be formulated as: 𝑰𝑺𝑨_𝑯𝑹 = (𝑰𝑺𝑨_𝑮𝑻 ∗ 𝒌𝒃𝒊𝒄𝒖𝒃𝒊𝒄) ↓𝒔 3.1 where kbicubic represents the ideal bicubic kernel and s the downsampling scale factor. Since the downsampling process acts as a filter that gets rid of high-frequency information, the noise distribution of the downsampled images differs from the GT images. Accordingly, we collect noise patches from the GT SA-LF image, as shown in Fig. 3.3 (a), and then we add the collected noise to the downsampled image to make sure it has a similar noise distribution to the GT image. Since image patches with richer content have a larger variance, we separate the GT image’s noise and content (i.e., to collect the noise) by the following constraint: σ(n)<v where n denotes the noise, v is the maximum value of variance and σ(·) denotes the function to calculate variance. Then, we estimate the downsampling kernel from the GT SA-LF images using KernelGAN [26], as shown in Fig. 3.3 (b), which estimates the kernels that best preserves the distribution of patches across scales of the GT image. KenelGAN Generator is trained to produce a downsampled version of the GT SA-LF image, so that the Discriminator cannot distinguish between the patch distribution of the downsampled LF-SA image and the patch distribution of the GT SA-LF image. Hence, the goal of the Discriminator is to learn the distribution of patches of the input GT SA-LF image and discriminate between real patches belonging to this distribution and fake patches generated by the Generator. Since KernelGAN is an image specific GAN that is trained on a single input image to learn its unique internal distribution of patches. Therefore, once the Generator is 60 trained, it constitutes the downscaling operation with the correct image-specific downsampling kernel. The estimated downsampling kernel should meet the following constraints: 𝒂𝒓𝒈 𝒎𝒊𝒏 G(𝑰𝑺𝑨_𝑮𝑻 ∗ 𝒌) ↓𝒔− 𝑰𝑺𝑨_𝑮𝑻 ↓𝒔 G𝟏 + |𝟏 − ∑𝒌𝒊| + |∑𝒌𝒊| + W𝟏 − 𝑫((𝑰𝑺𝐀_𝐆𝐓 ∗ 𝒌) ↓𝒔)W 3.2 where the first part of the formula minimizes the difference between (ISA_GT ∗ k)↓s, which is the downsampled LR image with kernel k, and ISA_GT↓s, which is the downsampled image with ideal kernel, to preserve important low-frequency information of the GT image. The second part of equation (3.2) constrains k to sum to 1, and the third part is to penalize non-zero values close to the boundaries of k. Finally, the last part, where D represents the discriminator, is to ensure the kernel is consistent with the GT image domain. Finally, to obtain the LR SA-LF image ISA_LR, we convolve the HR SA-LF image ISA_HR (that was obtained by the bicubic downsampling) with the estimated kernel k and add the collected noise n as follows: 𝑰𝑺𝑨_𝑳𝑹 = (𝑰𝑺𝑨_𝑯𝑹 ∗ 𝒌) + 𝒏 3.3 3.1.2 Super-resolution Framework Our goal is to increase the light field content spatial resolution. Nevertheless, to achieve high performance in LF spatial SR, we should exploit LF spatial information (i.e., information within each view) and angular information (i.e., information among other views). The spatial and angular features extraction is described in the next subsection. These features are then fed to a SR Generator and patch Discriminator, which are trained as explained in the following subsections to construct the HR LF images. 61 3.1.2.1 Spatial and Angular Features Extraction The 4D LF can be parameterized as L (S, T, X, Y), where S and T represent the angular information (i.e., number of views), while X and Y represent the spatial information (i.e., number of pixels in each view). Since SA-LF captured by plenoptic cameras are distributed in a square array, we set S = T = R in our method, where R denotes the angular resolution. The first step of our SR framework is to extract the spatial and angular features of the LF images. In our proposed approach, we use the MP-LF representation to extract spatial and angular features. Looking at the MP-LF representation at the top portion of Fig. 3.4 (a), we can notice that the pixels under each microlens, which are surrounded by black circles in the figure, represent one point in (a) (b) Figure 3.4 Light Field Camera output (a) Macro-pixel LF image captured by Plenoptic Camera. (b) Processed sub-aperture images and a close-up to one sub-aperture image of size 625x434. 62 the scene (i.e., angular information). Each pixel under each microlens represents one direction of view of that point, meaning each colored box under the black circle in Fig.3.4 (a) represents one pixel of a different view. Accordingly, if we put together the boxes of the same color, for example the orange one, we get one view as shown in Fig. 3.4 (b). In other words, the pixels under each microlens contain the angular information, while each pixel by itself contains the spatial information. Specifically, to extract the angular information from the MP-LF representation, we convolve a kernel (the green box in Figure 3.5) of size R x R, a stride of R, and a dilation of 1. The output angular features have the size of X x Y. On the other hand, we extract spatial information from the MP-LF representation by convolving a kernel (the brown box in Figure 3.5) of size 3x3, a stride of 1, and a dilation of R. We perform zero padding to ensure that the output spatial features Figure 3.5 An illustration of angular and spatial features extraction. To extract angular features, pixels under each microlens are convolved with the green kernel. To extract spatial features, pixels of the same color under all the microlenses are convolved with the brown kernel. 63 have the same spatial size RX × RY as the input MP-LF. A visualization of the angular and spatial features extraction process is shown in Fig. 3.5. 3.1.2.2 Super-resolution Generator To generate super-resolved LF images, we propose a super-resolution GAN model, presented in Fig. 3.6 (a), where the generator is constructed of two parts. The first part, shown in Fig. 3.6 (b), which learns the angular and spatial features of the MP-LF image representation, is inspired by the RDN model [61]. While the second part, shown in Fig. 3.6 (c), which up-samples the SA-LF image representation, is based on the Laplacian up-sampling for single image SR [62]. The entire Generator is a cascade of the two parts with the same structure at each level as shown in Fig. 3.6 (a). 64 (a) (d) Figure 3.6 An Illustration of Proposed Model. (a) Overview of Proposed Network Architecture. (b) Angular and Spatial Feature Extraction Sub-network. (c) Residual Dense Block (RDB). (d) Laplacian Up-sampling Sub-Network 65 (a) Learning Angular and Spatial Features The residual dense network (RDN) proposed by Zhang et al. in [61] mainly consists of four parts: shallow feature extraction net, residual dense blocks (RDBs), dense feature fusion (DFF), and finally the up-sampling net (UPNet). Different from RDN, we only use the RDB part and a modified version of the DFF part of the network. First, we use d RDBs to extract local dense features for angular and spatial information. For our model, we set d to 3. In a sense, we have two sets of 3 RDBs, one for angular features and another one for spatial features, as shown in Fig. 3.6 (b)). An illustration of the RDB structure is shown in Fig. 6(c), where the feature map output xd of the dth RDB can be obtained by: 𝒙𝒅 = 𝑭𝒏 (𝒙𝒏3𝟏) + 𝒙𝟎 = 𝑭𝒏 Y𝑭𝒏3𝟏 Z … \ 𝑭𝟏(𝑭𝟎)]^_ + 𝒙𝟎 3.4 where Fn denotes the function of the nth set of convolutional and rectified linear units (ReLU) layers, which non-linearly maps the batches of feature maps to another set of feature maps. It can be noted from equation (3.4) that the outputs of the preceding layer have direct connections to all subsequent layers as shown in Fig. 3.6 (c), which not only preserves the feed-forward nature, but also extracts local dense features. Each RDB in our model has n = 8 sets of convolution and ReLU layers, where each convolution has 64 filters with kernel size 3x3. Next, to exploit hierarchical features in a global way, we fuse the angular and spatial features processed by the RDBs. We first concatenate the angular feature maps produced by RDB 1, RDB 2, and RDB 3 using DFF as follows: 𝒙𝑨_𝑮𝑭 = 𝑭𝑮𝑭\[𝒙𝑨_𝟏, 𝒙𝑨_𝟐, 𝒙𝑨_𝟑]] + 𝒙𝑨_𝟎 3.5 66 where xA_0 denotes the angular features extracted from the LR MP-LF images, and [xA_1, xA_2, xA_3] refers to the concatenation of the angular feature maps produced by the RDBs, while FGF is a 1×1 convolutional layer and a ReLU layer that is used to adaptively fuse a range of features with different levels. Since the angular feature maps do not have the same size as the spatial feature maps, we up-sample the global angular feature map xA_GF using transposed convolution, concatenate it with the spatial features maps from the RDBs [xS_1, xS_2, xS_3] and add it to the spatial features extracted from the LR MP-LF images xS_0 as follows: 𝒙𝑮𝑭 = 𝑭𝑮𝑭\[𝒙𝑺_𝟏, 𝒙𝑺_𝟐, 𝒙𝑺_𝟑, (𝒙𝑨_𝑮𝑭) ↑ ]] + 𝒙𝑺_𝟎 3.6 (b) Up-sampling Feature Maps and Light Field Images Since the final angular and spatial feature map xGF is organized in a MP-LF pattern, we first reorganize the pixels to convert xGF into a SA-LF representation for up-sampling. In this part of the model, we up-sample the SA-LF images in a progressive scheme using Laplacian pyramid following the work presented in [62]. The up-sampling part of the proposed SR model consists of two branches: feature up-sampling and image reconstruction as seen in Fig. 3.6 (d). However, unlike [62], we do not have the embedding feature block for feature extraction, instead we use the feature extraction methodology explained in the previous section. For the features up-sampling branch, we use a transposed convolutional layer for up-sampling the angular and spatial feature map xGF by a scale of 2 (i.e., the output is xGF_2), we then apply a convolutional layer for predicting the sub-band residual image R2. At the second level, the 67 generator for the up-sampling scale of 4, extracts the angular and spatial features directly from the SA-LF images up-sampled by the previous pyramid level and then apply a convolutional layer for predicting the sub-band residual image R4. For the Generator’s image reconstruction branch at an up-sampling scale of 2, the input LR SA-LF image is up-sampled by a scale of 2 with a transposed convolutional layer. We then combine the up-sampled image (using element-wise summation) with the predicted residual image R2 to generate a high-resolution output image. The reconstructed HR SA-LF image at scale x2 is then used as an input for the image reconstruction branch at level 4 and combined with the predicted residual image R4 to generate HR SA_LF at scale x4. 3.1.2.3 Patch Discriminator A typical GAN Discriminator looks at the entire image and decides whether this image is real or fake and then outputs a single scaler value that ranges from 0 to 1. This formulation focuses on the entire image and consequently may neglect local texture details of the image. Therefore, in our SR GAN model we use a patch Discriminator [63], which outputs an array of elements representing local regions in the input image, where each element in this array ranges from 0 to 1. This means that the Discriminator looks at multiple local image patches and decides on each patch whether it is real or not. Hence, it enhances local texture details of the generated images. 3.1.2.4 Loss Function For training, several losses are applied including pixel loss, perceptual loss, and adversarial loss. Training with pixel loss Lpixel optimizes PSNR, yet it does not directly optimize the perceptual 68 quality, and hence generates images which might not be pleasing to the human eye. Hence, we also use perceptual loss Lper to match the high-level features in a generated image with a given HR output image. This is achieved by utilizing a pre-trained network, in this paper we use the inactive features of VGG-19 [64] to find the difference of feature outputs between predicted and output images as loss. We also use adversarial loss Ladv, which helps in fooling the Discriminator and generally produces images which have better perceptual quality, making them look more realistic. The total loss function is the weighted sum of these three losses: 𝑳𝒕𝒐𝒕𝒂𝒍 = 𝝀𝟏. 𝑳𝒑𝒊𝒙𝒆𝒍 + 𝝀𝟐. 𝑳𝒑𝒆𝒓 + 𝝀𝟑. 𝑳𝒂𝒅𝒗 3.7 where λ1, λ2, and λ3 are set empirically as 1, 0.01, and 0.05, respectively. To choose these loss weight values (i.e., λ1, λ2, and λ3), we trained the network using different combinations of weight values. By plotting the loss over training epochs, we noticed that a higher weight for pixel loss increases pixel loss. However, we observed that higher pixel loss weightage in the total loss resulted in better generated image compared to feature loss or adversarial loss. This means, it is better to have a higher pixel loss weight (more important) than feature loss weight or adversarial loss weight (not as important). In addition, the plot of the adversarial loss showed that there is a little difference between its effect and the feature loss effect on the total loss. To minimize the error between the generated up-sampled SA-LF images (ISR) and the GT SA-LF images (IHR), we use Charbonnier loss as follows: 𝑳𝑪𝒉𝒂𝒓 = 𝟏𝑵∑ c(𝑰𝑺𝑹 − 𝑰𝑯𝑹)𝟐 + 𝜺𝟐𝑵𝟏 3.8 where N denotes the number of training samples in each patch and epsilon is a fixed hyperparameter to ensure that the Charbonnier penalty is always non-zero. In our work, we set 69 epsilon to 1x10-3, empirically. We found that using the Charbonnier distance instead of the traditional L1 or L2 loss for pixel loss (Lpixel) and feature loss (Lper) as regularizers for the GAN training improves the learning behavior of the GAN model. The Charbonnier loss in pixel loss Lpixel provides regularization to ensure that the super-resolved frames do not depart by a great extent from the content in the corresponding ground truth high-resolution frame. Furthermore, the Charbonnier regularization in perceptual loss Lper, leverages the deep features learned by deep discriminative classifiers to compare the reconstructed frame from the ground truth frame. 3.1.2.5 Training We trained our model using real-world light field images from the Stanford dataset captured by a plenoptic camera (Lytro) [54] as well as data captured using our Lytro Ilum [53]. In addition, we used the synthetic light field dataset from [55]. We split the dataset into 80% for training and 20% for testing. Then we split the training set into 70% training and 30% validation. We increased the number of training data by using three data augmentation techniques on the training set: random horizontal flip, random rotation, and addition of Gaussian noise. This augmentation helps to avoid overfitting and improves the training process overall. We convert the LF images into YCbCr color space, and super-resolve only the Y channel. Our model was initialized using the Kaiming method [65] and optimized using the Adam approach [66]. We first generate the LR LF images through the downsampling framework. Then, we pass the LR images through the Generator, which gives us up-sampled (generated) images for each scale. The GAN framework uses Laplacian Pyramid based Generator internally, where the output of one scale is passed to the next level. 70 To update the Generator layers, for the generated up-sampled LF images at each scale, we compute the pixel loss (using Charbonnier loss function) by pixel-wise comparing it with the GT LF image at that scale. In this case, scale 4 loss will be backpropagated all the way back to level 2, i.e., we do not detach loss of scale 4 from scale 2. Separately, we also backpropagate the scale 2 loss. Similarly, we also compute the perceptual loss for each scale without detachment. For the perceptual loss, we pass both the generated up-sampled LF images and GT LF images to VGG-19 and compute the Charbonnier loss between the corresponding features. Note that this loss only updates the Generator layers. Finally, we pass the generated up-sampled LF images and GT LF images at each scale to the Discriminator to compute the adversarial loss. In this case, gradients for GT LF image outputs from the Discriminator are detached, while gradients of up-sampled LF images outputs are backpropagated. This loss only updates the Generator parameters and follows the above two losses in backpropagation. These three losses are collectively called total generator loss. To update the Discriminator layers, we pass the generated up-sampled LF images with detachment and the GT LF images at each scale to the Discriminator to compute the adversarial loss. In the case of detaching the Discriminator, gradients do not backpropagate to the Generator and thus only the discriminator parameters are updated. 3.2 Performance Evaluation and Discussion The performance of our proposed LF spatial SR model is compared with the following state-of-the-art LF super-resolution methods that are based deep learning: EPI [34], resLF [36], LFSSR 71 [35], LF-ATO [37], and Distg-Block [38]. We performed objective evaluations in terms of the peak-signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) quality metrics, comparing the GT test images in their original resolution and the upscaled version of the test images using the different approaches. We also evaluated the visual quality of the up-sampled scenes using the Video Multimethod Assessment Fusion (VMAF) quality metric which is known to yield the best correlation with human perception. For a fare comparison, we generated the LR LF images for testing using the proposed downsampling framework. We should note that the test datasets were not used for training nor validation. 3.2.1 Ablation Study To verify the effectiveness of the different settings of our network, we conduct ablation experiments on the Stanford LF dataset [15] to investigate the benefits of the downsampling framework, GAN framework, patch discriminator, and Charbonnier loss, respectively. • LR-Bicubic: we investigated the benefit of the downsampling framework by generating the LR LF images using bicubic downsampling, just like other LF SR methods [35]–[38], and we trained the proposed SR model on them. As shown in Table 3.1, training the network Model PSNR SSIM VMAF LR-Bicubic 36.18 0.938 77.53 Generator-Only 36.03 0.902 76.93 Regular-Discriminator 38.56 0.967 85.34 L2-Loss 35.85 0.927 79.20 L1-Loss 39.37 0.979 88.24 Proposed 40.61 0.984 89.64 Table 3.1 Comparisons Against Different Model Settings 72 on LR images generating through the bicubic downsampling achieves inferior PSNR, SSIM, and VMAF results compared to the proposed method, thus, demonstrating the effectiveness of the downsampling framework. • Generator-Only: to determine the efficiency of the GAN framework, we remove the discriminator and only train the network using the generator SR model. Results in Table 3.1 show that this setting produces inferior results even to the LR-Bicubic, a fact that confirms the importance of the GAN framework in producing high-quality upscaled images. • Regular-Discriminator: in this setting, we replace the patch discriminator with a regular discriminator. It can be observed in Table 3.1 that the network achieves lower reconstruction accuracy. This is due to the fact that the regular Discriminator is trained to determine whether the input image is real or fake by looking at the entire image rather than looking at each image patch. On the other hand, the patch Discriminator looks at multiple local regions in the image and decides the realness of each local region independently, which improves the visual quality of the generated images by focusing on more local texture details. 73 • L1-Loss, L2-Loss: to demonstrate the effectiveness of the Charbonnier loss, we train the network using L1 and L2 loss functions. Fig. 3.7 compares the training convergence curves of L1 loss, L2 loss, and Charbonnier loss over 60 epochs. It can be noted, from the figures, that the L1 loss is more robust than the L2 loss. This is because the L2 loss function is based on the mean squared error, which increases twice as much as the L1 loss. On the other hand, the Charbonnier loss function adds an epsilon variable compared to L1 loss, making it more robust than L1 loss. Thus, it can be observed that the learning curve of the Charbonnier loss is better than the other losses while the average SSIM values are higher. Therefore, we chose the Charbonnier loss to train our model. Figure 3.7 SSIM of proposed network trained with L2, L1, and Charbonnier loss functions. 74 3.2.2 Experimental Evaluation and Discussion Quantitative results in Table 3.2 compare the average PSNR, SSIM, and VMAF values on the test images for the central view of each scene. We obtain the quantitative results for n scenes of each dataset by first calculating the metric of the LF-SA images on each scene separately, then we obtain the score for each scene by averaging its scores over the number of LF-SA images, and Methods Datasets Bicubic resLF EPI LFSSR LF-ATO Distg-Block Proposed Illum (x2) VMAF 63.44 79.16 79.63 79.76 79.79 80.34 88.24 PSNR 30.12 35.32 35.56 35.37 36.17 36.59 37.86 SSIM 0.9475 0.9802 0.9815 0.9824 0.9831 0.9855 0.9898 Illum (x4) VMAF 59.95 71.97 71.99 72.17 72.20 72.93 85.24 PSNR 26.84 30.26 30.64 30.58 30.71 30.98 34.86 SSIM 0.8653 0.9427 0.9470 0.9447 0.9478 0.9528 0.9748 Stanford (x2) VMAF 65.48 80.32 80.62 80.24 81.72 81.22 88.92 PSNR 29.83 33.52 33.65 33.57 34.38 34.82 35.52 SSIM 0.9334 0.9705 0.9721 0.9724 0.9763 0.9784 0.9863 Stanford (x4) VMAF 57.98 73.05 74.65 73.87 73.33 74.43 84.33 PSNR 23.88 28.76 28.96 28.93 30.03 30.37 33.38 SSIM 0.8116 0.9076 0.9197 0.9182 0.9212 0.9214 0.9548 HCI (x2) VMAF 85.34 90.03 90.03 90.05 90.34 90.52 93.10 PSNR 37.81 43.42 43.64 43.89 44.20 44.89 44.93 SSIM 0.9761 0.9929 0.9944 0.9931 0.9942 0.9947 0.9954 HCI (x4) VMAF 70.21 83.17 83.53 83.62 83.95 84.20 88.91 PSNR 32.57 36.73 36.85 36.97 37.98 38.56 39.95 SSIM 0.9242 0.9672 0.9688 0.9684 0.9692 0.9732 0.9896 Table 3.2 Comparisons Against Different Approaches 75 finally we obtain the scores presented in Table 3.2 for each dataset by averaging the scores of all its n scenes. We observe that our method achieves the best results in terms of PSNR, SSIM, and VMAF compared to the state-of-the-art LF SR approaches, confirming its superior performance. We can further observe that, for all the approaches, upscaling with a factor of x2 yields better results than upscaling with a factor of x4. However, our method yields much better results when upscaling with a factor of x4 compared to the other approaches. This is because our approach uses progressive up-sampling, which makes learning mapping functions for large scaling factors easier. Moreover, we can notice that when up-sampling with a factor of x2, our approach achieves better results than LF-ATO [37], and Distg-Block [38] in terms of PSNR and SSIM. Yet, it achieves superior results in terms of VMAF. This difference is probably due to the fact that VMAF measures visual quality while PSNR and SSIM rely strictly on numeric comparison and do not actually take into account human perception, as any slight change in luminance would impact the PSNR and SSIM results. To demonstrate the effectiveness of the proposed approach, we further compare the different methodologies using two specific scenes that contain reflection regions and texture details. These scenes (green_flower and car_bumper [53]) were captured using a plenoptic camera. Qualitative results presented in Fig. 3.8 show the resulted images, reconfirming the superiority of our method, as sharpness, contrast and details are much closer to those of the original image. 76 (a) (b) Figure 3.8 Comparison showing the ground truth central view with a cropped close-up, and its interpolated versions using different approaches of the: (a) green_flowers and (b) car_bumper scenes. 77 3.3 Conclusion In this chapter, we have presented a novel Laplacian deep learning spatial SR approach that progressively reconstructs high-resolution LF images at different upscaling levels. We trained the proposed approach on low-resolution LF images that share a common domain with real-world images by generating them using a downsampling framework that uses kernel estimation and noise injection. We introduced a spatial and angular features extraction approach that preserves local features information from current and preceding layers and then uses global feature fusion to adaptively learn global features as well in a holistic way. The model then uses a Laplacian framework that progressively predicts high-frequency residuals in a coarse-to-fine manner to reconstruct high-resolution LF images. We trained the proposed model using a robust Charbonnier loss function and achieve high-quality reconstruction. Experimental results have demonstrated the superiority of our proposed LF spatial SR framework over state-of-the-art methods. 78 Chapter 4: Light Field Angular Super-resolution Light Field is a rapidly evolving technology, supporting capturing and displaying of visual information that approximates the human eye. However, the enormous amount of data generated are a challenge, as they require significant increase in bandwidth. A possible solution may be to drop specific views at the transmitting end and effectively synthesize them at the receiver end or when they are needed (as seen in Figure 4.1), thus minimizing the amount of data needed to be transferred or stored, with minimal impact on the visual quality on the LF content. Furthermore, view synthesis generates densely sampled novel views, which is an essential technical component for delivering an immersive LF experience with realistic disparity, smooth motion parallax between views, and natural refocusing on the different regions of the image. Light field view synthesis aims at reconstructing a densely sampled novel LF views from a sparse set of views. It is basically super-resolving LF data along the angular dimension. In this chapter, we focus on LF angular super-resolution (SR) by synthesizing virtual LF views for the 4D light field from a sparse set of input views using two novel approaches. First, we propose a deep recursive residual network to generates one in-between virtual view using the epipolar Figure 4.1 Overview of proposed view synthesis technique. 79 image plane (EPI) information, which would ensure having a smooth disparity between the generated views. Second, we propose a generative adversarial network approach, which generates up to five in-between virtual views, using LF spatial and angular information for an efficient angular SR with minimal impact on the visual quality on the generated LF content. 4.1 View Synthesis Framework using EPI Our first view synthesis approach is based on a deep learning model that is trained using the Epipolar Plane Images (EPI) of light field content. The overall framework is shown in Fig. 4.2. The transparent views are the ones dropped at the transmitting end, leaving only the other views to be transmitted or stored. The EPI of rows and columns are fed to our modified Deep Recursive Residual Network (DRRN) which produces the synthesized views at the receiver end. The following subsections give an overview of EPI, describe in detail the components of our method, and explain our choices and the training process of the deep learning model. Figure 4.2 Overview of proposed view synthesis technique using EPI. 80 4.1.1 Epipolar Plane Image (EPI) The most common parameterization of the 4D LF is the two-plane parameterization L(x, y, s, t), where (x, y) represents the spatial resolution of each image and (s, t) represents the angular resolution (number of views) of LF. Please refer back to Chapter 1, section 1.3.3 for the EPI extraction technique. In our proposed approach, during training we first drop every other view at the transmitting end, Fig. 4.3 (a). We then stack the images of the first row to obtain the “low-resolution” EPI, Fig. 4.3 (b). Next, we obtain the “up-sampled” EPI using bicubic interpolation, as seen in Fig. 4.3 (c), to obtain an EPI of the same size as the target one that uses all the existing original views. We feed the upscaled EPI to the deep learning model to attain the “full size” EPI, shown in Fig. 4.3 (d). We finally use this EPI to generate the in between views at the receiver end, Fig 4.3 (e). Figure 4.3 Steps for generating the in between views for the first row of images. 81 In general, the reconstruction process of synthesizing the in between views for the 4D light field, involves three steps which are illustrated in Fig. 4.4. In the first step, we generate a horizontal 2D EPI for each row of the inputs, as explained in Fig. 4.3. We repeat this step for every row of views in the scene. In the second step, we use the vertical columns of views to generate the up-sampled vertical 2D EPIs and use them to reconstruct the intermediate views in the LF columns. In the third step, we use the synthesized views from either the horizontal or the vertical EPIs to generate the remaining views, represented by green boxes in Fig. 4.4 (step 3). 4.1.2 Convolutional Neural Network (CNN) We base our deep learning model on the general Deep Recursive Residual Network (DRRN) architecture described in [50]. In order to determine the best combination of residual units and residual blocks need for our implementation and use of EPI structures, we trained a number of DRRN models with different combinations of recursive blocks, residual units, layers, and Figure 4.4 Three steps to synthesize in between views. 82 parameters. Our implementation learns a residual image R, which is the difference between the ground truth (High-resolution) EPI images “EHR” and the bicubically up-sampled EPI images “Ebicubic”. Our tests showed that the deep recursive residual CNN model with one recursive block and 20 residual units, (represented by the red box in Fig. 4.5 (a)) followed by a convolutional layer achieves the best performance for this application. A similar architecture was found to be efficient for our LF spatial super-resolution implementation which is described in detail in [67]. The recursive block contains a convolutional layer followed by 20 residual units. The first convolutional layer extracts patches from the EPI and represents each patch as a feature map. All Figure 4.5 (a) Deep learning model structure. (b) One residual unit. (c) Convolutional layers. 83 the 20 residual units share the same input from the identity branch of that convolutional layer, represented by a dashed line in Fig. 4.5 (a). Our model adopts global residual learning in the identity branch and recursive learning into the residual branch. Each of the residual units (i.e., u = 1, 2, …, 20) consists of two convolutional layers (Fig. 4.5 (b)), where 𝑥W and 𝑥W3X are the output and input of the u-th residual unit, and 𝑥1 is the output of the first convolutional layer in the recursive block. Each one of these two convolutional layers contains a batch normalization layer, a Rectified Linear Units layer, and a weight layer, as shown in Fig. 4.5 (c). For our implementation, we use a batch size of 128 for the batch normalization layer and each weight layer uses 128 filters of size 3x3. The residual unit can be represented mathematically as follows: 𝒙𝒖 = 𝑭(𝒙𝒖3𝟏,𝑾) + 𝒙𝟎 4.1 where F denotes the function that non-linearly maps the batches of feature maps to another set of feature maps using a weight set W. The mean square error (MSE) is used as a loss function. It minimizes the difference between the residual predicted by the network 𝑅D and the corresponding ground truth residual 𝑅. Thus, the loss function is defined as: 𝑳 = 𝟏𝟐𝑵∑ G𝑹H 𝒊 − 𝑹𝒊G𝟐𝑵𝒊0𝟏 4.2 where N is the number of batches. The weight values for each layer are randomly initialized from a Gaussian distribution with zero mean and standard deviation 0.001. They are then adjusted based on the loss function presented in equation (2). Next, we feed the output of the recursive block 𝑥W to a convolutional layer, illustrated by the green box in Fig. 4.5 (a), to construct the residual image 84 𝑅D. Finally, we add the residual image 𝑅D to the input Ebicubic image to estimate the original resolution EPI Eestimated_HR. This can be formulated as: 𝑬𝒆𝒔𝒕𝒊𝒎𝒂𝒕𝒆𝒅_𝑯𝑹 = 𝑹H + 𝑬𝒃𝒊𝒄𝒖𝒃𝒊𝒄 4.3 The network was trained using EPI images extracted from some of the scenes in the real-world Stanford dataset [15] captured with a camera array, as well as synthetic the light field dataset found in [68], which includes 24 different synthetic light field scenes. We further trained our model using light field data captured by a plenoptic camera (Lytro) [53] and Stanford Lytro Light Field Archive [54]. The purpose of having training datasets captured by different approaches is to accommodate for different baseline settings, i.e. disparity, between the input views. In addition, we applied data augmentation techniques on the training datasets to increase the number of training data and avoid overfitting. Since we are dropping the in between views (every other row/column) at the transmitting end, the resulting sparse views yield “low resolution” EPI images from the original dataset and the original number of views were used to obtain the “high resolution” EPI images as ground truth. It must be noted that none of the data used for training were used for testing. 4.1.3 Experiments and evaluation We evaluated our modified DNNR model on the real-world datasets captured with camera array from the (New) Stanford dataset [15], a plenoptic camera from the V-sense dataset [69] and the synthetic light field data from the HCI dataset [70]. We compared our method against state-of-the-art learning-based schemes for light field view synthesis, namely the Kalantari et al. (2016) [71], Wu et al. (2017) [44], Yeung et al. (2019) [35] and Wu et al. (2019) [72]. The performance of these comparative methods was obtained using the source codes available online by the authors. 85 We evaluated the methods numerically using the peak-signal-to-noise ratio (PSNR) and the structural similarity index (SSIM). The results of scenes captured using: plenoptic camera (Bikes and Flowers from the V-sense dataset [69]), camera array (Bunny and Buddha from [15]), and the average over 10 synthetic LF scenes from the HCI dataset [70] are provided in Table 4.1. We observe that our proposed approach achieves higher PSNR values with an average of (2.36, 1.51, 1.25, and 0.87) dB and higher SSIM values with an average of (0.0122, 0.1447, 0.0082, 0.0147) compared to Kalantari et al. (2016) [71], Wu et al. (2017) [44], Yeung et al. (2019) [35] and Wu et al. (2019) [72], respectively. Figure 4.7 presents a 16x16 real-world “Chess” scene captured by a camera array from the (New) Stanford dataset [15]. The shaded images represent the dropped views at the transmitter end which will be reconstructed at the receiver end. The central view, represented by the green box in Fig. 4.6, is reconstructed using the five different methods. This view has been chosen for comparison because it is the most difficult view to reconstruct since it is surrounded by a column full of virtual (generated) views. The visual quality evaluation of our model against the other four schemes using Scenes Methods Bikes Flowers Bunny Buddha Average over 10 LF PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM PSNR SSIM Kalantari et al. 32.37 0.935 36.28 0.960 43.04 0.983 42.47 0.985 38.58 0.944 Wu et al. 26.17 0.762 25.25 0.719 34.55 0.936 32.86 0.916 35.43 0.850 Yeung et al. 32.92 0.945 36.27 0.960 44.73 0.986 44.03 0.988 39.69 0.941 Wu et al. 30.84 0.924 35.60 0.962 41.42 0.981 42.91 0.986 37.61 0.942 Proposed 35.04 0.954 40.62 0.974 46.11 0.9889 45.77 0.995 40.94 0.970 Table 4.1 Quantative comparisons of different light field scenes 86 “Chess” scene is illustrated in Fig. 4.7. We observe that our scheme yields high resolution image with visual quality that is comparable to the Ground Truth and a significant improvement over the visual quality of the other methods. Figure 4.6 16x16 Chess scene captured by camera array. Figure 4.7 Comparison of our method and other state-of-the-art view synthesis schemes against ground truth using “Chess”. 87 4.1.4 Conclusion In this section, we introduced a learning-based LF view synthesis approach that addresses the bandwidth challenge of the enormous amount of generated data. To this end, efficient transmission and/or storage of LF content is achieved by dropping specific views at the transmitting end and then synthesizing them at the receiver end, with minimal impact on visual quality. Our modified DRRN architecture uses the EPI to ensure smooth disparity between the generated and original views. We consider plenoptic as well as camera array implementations as well as synthetic content which support different baseline settings. Experimental results demonstrate the effectiveness of our method over state-of-the-art techniques, yielding better visual quality LF synthesized views. This is a crucial requirement for delivering and storing LF content, making possible the use of this emerging technology in many applications including performing remote surgery and storing resulting LF data. 88 4.2 View Synthesis Framework using Light Field 4D Spatial and Angular Information In this section, we propose a novel angular super-resolution generative adversarial network (GAN) that is designed to synthesize novel sub-aperture images from a sparsely sampled set of 4D LF views, as shown in Fig. 4.8 (a). We feed the sparsely sampled 4D LF images to the GAN framework to generate the densely sampled 4D LF images by first extracting spatial and angular feature maps using a residual sub-network presented in our work to up-sample LF spatial resolution [73]. Then, we reconstruct the novel LF views by up-sampling the angular resolution of the extracted feature maps using sub-pixel operation based on ESPCN [74]. Thus, the proposed network preserves the intrinsic structure of the reconstructed densely sampled LF among the synthesized novel sub-aperture images. The proposed approach only takes four corner views (i.e., the outer four views) as an input and generates up to five in-between novel views. An overview of the proposed approach is shown in Fig. 4.8. Our proposed work differs from existing LF view synthesis approaches that consider the 4D LF information in the sense that our spatial and angular feature extraction uses hierarchical features from LR images, extracting local features using dense connected convolutional layers to learn more effective features from preceding and current layers. Then, we use global feature fusion to learn global hierarchical features in a holistic way before performing the angular resolution up-sampling. Performance evaluations against the state of the art approaches presented in [43], [45]–[47], [75], [76] showed that our method achieves superior performance in terms of PSNR, SSIM, and FVMA 89 (a) (b) Figure 4.8 An overview of proposed angular super-resolution approach. (a) Generator Network structure. (b) Residual Dense Block (RDB) structure. 90 4.2.1 Proposed Angular Super-Resolution Framework Our view synthesis approach is based on a GAN learning-based model that is trained using the spatial and angular information of the light field content. As illustrated in Fig.4.8, in our proposed network, we feed the Macro-pixel light field (MP-LF) images to the GAN framework to reconstruct novel virtual LF views. For the generator, we first extract spatial and angular features from the MP-LF image. Then, we feed the extracted spatial and angular features to a residual sub-network to obtain spatial and angular feature maps. Then, to reconstruct the novel LF views, we perform up-sampling on the angular dimension by applying sub-pixel convolution [74] to the MP-LF feature maps. Finally, we reorganise the pixels of the angularly up-sampled MP-LF image to get the sub-aperture light field (SA-LF) images with the novel synthesized LF views. For the discriminator, we use a patch discriminator, which learns the image patch distribution by looking at multiple local patches of the image and judge whether each patch is real or not. In other words, the Generator is trained to generate new in-between LF images, such that the Discriminator cannot distinguish between the patch distribution of the generated SA-LF images, and the patch distribution of the GT SA-LF images that were dropped out. The experimental results demonstrate that our method outperforms the state-of-the-art methods. Our approach is described in detail in the following subsections. 4.2.1.1 Features Extraction The 4D LF can be parameterized as L (X, Y, S, T), which is obtained by the intersection of the light rays with two planes: (X, Y) represents the spatial coordinates (i.e., lens plane) and (S, T) represents the angular coordinates (i.e., sensor plane), as shown in Fig. 1.1 (Chapter 1, section 1.1). 91 In other words, (X, Y) represents the spatial resolution of each image and (S, T) represents the angular resolution (number of views). In our proposed approach, we set S = T = R, where R denotes the angular resolution. The first step of our angular super-resolution framework is to extract the spatial and angular features of the LF images. In our proposed approach, we use the MP-LF representation to extract spatial and angular features. Thus, to extract the angular information, we convolve a kernel of size R×R, a stride of R, and a dilation of 1. The output angular features have the size of X×Y. On the other hand, we extract spatial information by convolving a kernel of size 3×3, a stride of 1, and a dilation of R. We perform zero padding to ensure that the output spatial features have the same spatial size RX×RY as the input MP-LF. 4.2.1.2 Generator Network Structure To generate novel LF images, we propose an angular super-resolution GAN model, presented in Fig. 4.8, where the generator first learns the angular and spatial features of the MP-LF image representation using a residual network presented in [73]. Then, sub-pixel operation is applied to the extracted features for the angular resolution up-sampling. First, we use d residual dense blocks RDBs to extract local dense features for angular and spatial information. For our model, we set d to 5 and 3 empirically for angular features and spatial features, respectively as shown in Fig. 4.8 (a)). The reason behind increasing the number of layers for the angular features is to increase the accuracy for leaning these features due to their importance in generating new views with the correct disparity. While increasing the number of layers for the 92 spatial features would lead to overfitting, thus, we kept at 3. An illustration of the RDB structure is shown in Fig. 4.9 (b), where the feature map output xd of the dth RDB can be obtained by: 𝒙𝒅 = 𝑭𝒏 (𝒙𝒏3𝟏) + 𝒙𝟎 = 𝑭𝒏 Y𝑭𝒏3𝟏 Z … \ 𝑭𝟏(𝑭𝟎)]^_ + 𝒙𝟎 4.4 where Fn denotes the function of the nth set of convolutional and rectified linear units (ReLU) layers. It can be noted from equation (4.4) that the outputs of the preceding layer have direct connections to all subsequent layers as shown in Fig. 4.8 (b), which not only preserves the feed-forward nature, but also extracts local dense feature. Each RDB in our model has n = 8 sets of convolutions and ReLU layers, where each convolution has 64 filters with kernel size 3x3. Next, to exploit hierarchical features in a global way, we fuse the angular and spatial features processed by the RDBs. We first concatenate of the angular feature maps produced each RDB as follow: 𝒙𝑨_𝑮𝑭 = 𝑭𝑮𝑭\[𝒙𝑨𝟏 , 𝒙𝑨𝟐 , 𝒙𝑨𝟑 , 𝒙𝑨𝟒 , 𝒙𝑨𝟓]] + 𝒙𝑨_𝟎 4.5 where xA_0 denotes the angular features extracted from the LR MP-LF images, and [xA1, …, xA5] refers to the concatenation of the angular feature maps produced by the RDBs, while FGF is a 1×1 convolutional layer and ReLU layer that is used to adaptively fuse a range of features with different levels. Since the angular feature maps do not have the same size as the spatial feature maps, we up-sample the global angular feature map xA_GF using transposed convolution, concatenate it with the spatial 93 features maps from the RDBs [xS_1, xS_2, xS_3] and add it to the spatial features extracted from the LR MP-LF image xS_0: 𝒙𝑮𝑭 = 𝑭𝑮𝑭\[𝒙𝑺_𝟏, 𝒙𝑺_𝟐, 𝒙𝑺_𝟑, (𝒙𝑨_𝑮𝑭) ↑ ]] + 𝒙𝑺_𝟎 4.6 where the generated feature map xGF size is RX×RY×C. Next, to apply the sub-pixel shuffle, we extract the angular features of xGF followed by 1×1 convolution with a fixed number of output channel R2A2C to generate an angular feature map xA of size X×Y×R2A2C, where X and Y are the spatial resolution, R is the original angular resolution (i.e., R=2) and A is the up-sampling angular resolution (i.e., A=/Y ). Finally, we apply the sub-pixel operation to generate a MP-LF image of size ARX×ARY. Then, we reorganize the image pixels to generate the SA-LF images with the novel LF views. 4.2.1.3 Patch Discriminator In our angular super-resolution GAN model, we use a patch Discriminator [63], which looks at multiple local regions in the image and decides the realness of each local region independently. Consequently, it improves the visual quality of the generated images by focusing on more local texture details. 4.2.1.4 Training For training, we apply three losses including pixel loss Lpixel, perceptual loss, Lper and adversarial loss Ladv to optimize the PSNR, to find the difference of feature outputs between predicted and 94 output images, and to produce images which have better perceptual quality, respectively. The total loss function is the weighted sum of these three losses: 𝑳𝒕𝒐𝒕𝒂𝒍 = 𝝀𝟏. 𝑳𝒑𝒊𝒙𝒆𝒍 + 𝝀𝟐. 𝑳𝒑𝒆𝒓 + 𝝀𝟑. 𝑳𝒂𝒅𝒗 (4) where λ1, λ2, and λ3 are set empirically as 0.5, 0.05, and 0.05, respectively. We ran experiments with different combinations of loss weight and came to the conclusion that the pixel loss affects the total loss more than the perceptual loss and adversarial loss, while the last two losses have the same effect on the total loss. To minimize the error between the generated novel SA-LF images (INew) and the GT SA-LF images (IGT), we use Charbonnier loss as follow: 𝑳𝑪𝒉𝒂𝒓 = 𝟏𝑵∑ c(𝑰𝑵𝒆𝒘 − 𝑰𝑮𝑻)𝟐 + 𝜺𝟐𝑵𝟏 (5) where N denotes the number of training samples in each patch and epsilon is a fixed hyperparameter to make sure the Charbonnier penalty is always non-zero. In our work, we set epsilon to 1x10-3, empirically. It should be noted that we tested training the network with L1-Loss and L2-Loss functions to verify the effectiveness of the Charbonnier loss. The experiments show that L1 loss is more robust than the L2 loss due to the fact that the L2 loss function is based on the mean squared error, which increases twice as much as the L1 loss. However, the Charbonnier loss function is even more robust than the L1 loss since it adds an epsilon variable compared to L1 loss. Therefore, in our work we train our network using the Charbonnier loss since its learning curve is better than the other losses while the average SSIM values are higher 95 The network was trained using SA-LF images of the scenes in the real-world Stanford dataset [15] captured with a camera array, as well as synthetic the light field dataset found in [68], which includes 24 different synthetic light field scenes. In addition, we further trained our model using light field data captured by a plenoptic camera (Lytro) [53] and Stanford Lytro Light Field Archive [54]. The purpose of having training datasets captured by different approaches is to accommodate for different baseline settings, i.e., disparity, between the input views. To increase the number of training data and avoid overfitting, we applied data augmentation techniques on the training datasets. For training, we only feed the generator the corner views and drop the in-between views (up to 5 in-between views), and we use the original in-between views as ground truth. It must be noted that none of the data used for training were used for testing. 4.2.2 Experiments and Evaluations sWe evaluated our novel GAN angular super-resolution model on the real-world datasets captured with camera array from the (New) Stanford dataset [15], a plenoptic camera from the V-sense dataset [69] and the synthetic light field data from the HCI dataset [70]. We compared our method against state-of-the-art learning-based schemes for light field view synthesis, namely the Kalantari et al. [76], Jin et al. [43], Wafa et al. [75], Wang et al. [45], Yeung et al. [46], and Meng et al. [47]. The performance of these comparative methods was obtained using the source codes available online by the authors. We generate LF with 7x7 angular resolution from LF with 2x2 angular resolution. Hence, we use the proposed approaches to generate five in-between views. 96 We evaluated the methods numerically using the peak-signal-to-noise ratio (PSNR) and the structural similarity index (SSIM). In addition, we evaluated the visual quality of the generated virtual views using the Video Multimethod Assessment Fusion (VMAF) quality metric [57]. The results of scenes captured using: plenoptic camera (the average over 5 scenes of the V-sense dataset [69]), camera array (the average over 5 scenes of the Stanford dataset [15]), and synthetic LF (the average over 5 scenes from the HCI dataset [70]) are provided in Table 4.2. The quantitative comparisons of averages PSNR, SSIM, and VMAF values confirms the superiority of our proposed approach over state-of-the-art schemes. To demonstrate the effectiveness of the proposed approach, we further compare the different methodologies using two specific scenes that contain different texture details. These scenes were captured using a plenoptic camera lytro illum (yellow_flowers and pink_flowers [53]). We first drop five in between views and only keep four corner views so the angular resolution would be 4x4. Then we use the seven methods to generate five in between views to have an angular Scene Method Stanford dataset [15] V-sense dataset [69] HCI dataset [70] PSNR SSIM VMAF PSNR SSIM VMAF PSNR SSIM VMAF Wang [45] 31.17 0.902 71.97 32.25 0.919 73.21 39.55 0.936 84.21 Wafa [75] 30.84 0.924 71.99 35.60 0.962 82.17 41.42 0.981 86.37 Kalantari [76] 32.37 0.935 72.17 36.28 0.960 83.55 43.04 0.983 89.51 Meng [47] 36.92 0.945 82.20 36.27 0.960 83.63 44.03 0.988 90.43 Yeung [46] 36.04 0.961 83.23 39.69 0.941 85.17 44.73 0.986 90.34 Jin [43] 38.51 0.942 87.03 40.62 0.974 86.07 45.77 0.991 91.43 Proposed 38.58 0.944 87.24 41.94 0.980 88.27 46.11 0.998 93.10 Table 4.2 Quantative comparisons of different LF scenes 97 resolution of 7x7. The results are provided in Table 4.3, which compares the PSNR, SSIM, and VMAF on the central view of each scene (i.e., the image of row #4 and column #4 for each scene). From Table 4.3, we notice that our method improves the VMAF, PSNR, and SSIM values over the other approaches. Scene Method Yellow_Flowers [53] Pink_Flowers [53] PSNR SSIM VMAF PSNR SSIM VMAF Wang [45] 35.17 0.907 75.07 36.54 0.929 81.21 Wafa [75] 35.84 0.909 75.93 37.60 0.942 83.74 Kalantari [40] 37.37 0.923 78.75 38.28 0.950 83.65 Meng [47] 37.29 0.920 85.20 38.72 0.950 84.73 Yeung [46] 38.04 0.962 85.34 39.56 0.961 88.05 Jin [43] 39.51 0.968 87.13 39.90 0.968 88.24 Proposed 39.98 0.974 88.14 40.07 0.978 89.56 Table 4.3 Quantitative comparisons of yellow flowers and pink flowers scenes 98 A visual quality evaluation of our model using the “yellow_flowers” scene [53] can be seen in Fig. 4.9, which shows the central view generated using the seven different schemes. As we can see in Fig. 4.9, the part of the flower, surrounded by the red square, appears the sharpest using our proposed method. Thus, we can conclude that compared to the other methods, our proposed approach generates the best visual quality. In addition, we evaluate our method using a 7x7 real-world “pink_flowers” scene captured by Lytro Illum [53]. The central view is reconstructed using the seven different methods. The visual quality evaluation of our model against the other six schemes using that central view of “pink_flowers” scene is illustrated in Figure 4.10. We observe Figure 4.9 Comparison of our method and other state-of-the-art view synthesis schemes against ground truth using “yellow_flowers” scene. 99 that our proposed approach generates a new image with visual quality that is comparable to the ground truth with a significant improvement over the visual quality of the other methods. 4.2.3 Conclusion In this section, we introduced a GAN model to up-sample the angular resolution of LF images that helps create an LF experience with realistic disparity while addressing the challenges of efficient transmission and storage of the enormous amount of generated LF content. These challenges are addressed using our up-sampling approach by dropping up to five in-between views and then synthesizing them when needed while preserving the visual quality of the views. Our GAN model uses the angular and spatial information in the LF data, which ensures a smooth disparity between the generated novel views and the original ones. Moreover, we train our proposed framework using LF captured by plenoptic and camera array schemes and synthetic content to support different Figure 4.10 Comparison of our method and other state-of-the-art view synthesis schemes against ground truth using “Pink_Flowers”. 100 baseline settings. Experimental results of the visual quality of the synthesized LF views confirm the superior performance of our proposed framework over state-of-the-art techniques. 101 Chapter 5: Conclusion 5.1 Summary of Contributions In this thesis, we propose methods and solutions to improve the limited spatial resolution of the light field cameras. Specifically, our contributions include two novel deep learning spatial super-resolution approaches that consider the spatial and angular information of light field images to generate high-resolution light field images at different upscaling levels. Another main issue when working with LF is the amount of data needed to create a truly immersive experience with realistic disparity and smooth motion parallax between views. To solve this bandwidth problem, we proposed two different approaches to drop specific views at the transmitting end and accurately synthesize them at the receiver end or when they are needed, thus minimizing the amount of data needed to be transferred or stored, with minimal impact on the visual quality on the LF content. Experimental results demonstrated the superiority of our proposed methods over state-of-the-art light field super-resolution and view synthesis techniques. In Chapter 2, we presented a novel method for light field spatial super-resolution using the full four-dimensional light field spatial and angular information. We proposed a learning-based model for spatial super-resolution which considers the epipolar image plane (EPI) information to ensure smooth disparity between the generated views and in turn construct high spatial resolution light field sub-aperture images. Our contributions for this chapter include: (1) modifying the single image super resolution DRRN architecture to apply to light field content; to this end, we modify the number of recursive blocks, residual units, layers, filters, filter sizes, and learning rate to serve our light field data; (2) taking advantage of the angular information available at the light field volume by feeding the horizontal EPIs to the model, use the output to obtain up-sampled sub- 102 apertures. Then, feed the vertical EPI of these up-sampled sub-apertures to obtain the high-quality super-resolved images; (3) training the model using EPIs extracted from datasets obtained using different approaches including plenoptic camera, camera array, and synthetic data, thus, accommodating different disparities represented by the EPI structure. In Chapter 3, we proposed a novel deep learning spatial super-resolution approach that considers the spatial and angular information of the light field images and progressively reconstructs high-resolution light field images at different upscaling levels. Specifically, we first extract spatial and angular feature maps from a light field image by adaptively learning local and global features. Then, at each upscaling level, our model takes the extracted coarse-resolution spatial and angular feature maps as input, predicts the high-frequency residuals, and uses transposed convolutions for up-sampling to a finer level. Additionally, we trained the proposed approach on low-resolution light field images generated using a 4D light field downsampling framework to achieve high-quality reconstruction. Experimental results demonstrated the superiority of our proposed method over state-of-the-art light field super-resolution techniques. Our contributions for this chapter are as follows: (1) instead of using a fixed downsampling operator (i.e., bilinear or bicubic) to construct LR and HR pairs for training, which generate LR images that do not belong to the same domain as the original image, we use a downsampling framework proposed for single image downsampling, which is modified to apply to 4D light field content; (2) our spatial and angular feature extraction uses hierarchical features from LR images, extracting local features using dense connected convolutional layers to learn more effective features from preceding and current layers. Then we use global feature fusion to learn global hierarchical features in a holistic way before performing the up sampling; (3) we use a SR GAN framework where the Generator network super- 103 resolves LR LF images progressively at scale factors of 2 and 4. Thus, we have two Generators sub-networks for each up-sampling level and only one discriminator. In Chapter 4, we first presented a deep learning network which is trained using the epipolar image plane (EPI) information to ensure smooth disparity between the generated views. In this approach, we drop every other light field view at the transmitting end, meaning we only drop one in-between view and then up-sample the LF images using the EPI representation at the receiver end to reconstruct the intermediate views. Our contribution includes modifying the DRRN architecture to use the epipolar image plane (EPI) to ensure smooth disparity between the synthesized and original views. Furthermore, we introduce another novel learning based LF angular super-resolution approach for efficient view synthesis of novel virtual images. This is achieved by taking four corner views and then generating up to five in-between views. Our generative adversarial network approach uses LF spatial and angular information to ensure smooth disparity between the generated and original views. We consider plenoptic, synthetic LF content and camera array implementations which support different baseline settings. Experimental results demonstrate the effectiveness of our method over state-of-the-art techniques, yielding better visual quality LF synthesized views. Our proposed work differs from existing LF view synthesis approaches that consider the 4D LF information in the sense that our feature extraction uses hierarchical methodology, which extracts local and global features from the spares set of LF images, facilitating an effective learning approach in a holistic way before performing the angular resolution up-sampling. 104 It is worth noting that the thesis contributions are applicable to both plenoptic 1.0 and plenoptic 2.0 cameras. Even though the rendering process for the two camera designs are different, yet once LF images are rendered correctly, the super-resolution methods in this thesis can be applied to increase the spatial and angular resolution. 5.2 Significance and Potential Applications of the Research Light Field is a rapidly evolving technology, supporting capturing and displaying of visual information that approximates the human eye, with applications ranging from entertainment to digital health. In addition, the amount of data captured by LF capturing techniques in the angular dimension (i.e., number of views) allow the user to move to a different perspective and see different information, just like real life. However, LF camera’s limited sensor resolution allows it only to capture images with low spatial resolution, intuitively, limiting the plethora of LF applications. Moreover, recording the scene from multiple points in space increases the storage and transmission requirements for such tremendous amount of data, which makes dropping some LF views and then synthesizing them when needed a possible solution to solve this problem. Accordingly, this thesis aims at improving the spatial (number of pixels) and angular (number of views) resolution of LF data to facilitate the reproduction of human visual experience in the widespread LF applications including but not limited to television displays, post-production, virtual and augmented reality (AR/VR), video gaming, security, autonomous driving, and digital health. Thus, the proposed spatial and angular SR approaches are the ideal solutions as they can generate high visual quality LF images, surpassing in performance every other existing approach, to enable the expected LF quality of experience. 105 To enable a more pleasing and immersive viewing experience in the home television displays and cinema applications, a range of post-production operations are adopted by content owners. Light field data facilitate countless post-production applications including generating virtual effects, de-occlusion [41], adding-depth [77], segmentation [78], etc. However, such post-processing applications require a large number of high-resolution images, which increases the storage and bandwidth requirements, making the proposed spatial and angular LF SR approaches the right solution. In medical imaging, light field allows doctors to clearly see the anatomy and to be an active participant in interacting with the subject they are viewing by zooming to details and changing the focus point to any location they desire [79]. However, low resolutions LF images captured by LF cameras prevent any meaningful magnification of the image/focus on desired objects due to significant loss of detail. Therefore, the proposed LF spatial SR frameworks address this issue offering higher spatial resolution LF images and visible details with the ability to focus on any point of the image. In the case of AR/VR, CREAL has revealed its first prototype of AR and VR head-mounted light field displays [80]. The company has created these prototypes to showcase its technology with the hopes that other companies will opt to incorporate it into their headsets. While most companies working on LF VR/AR displays focus on the compute and graphics requirements for virtual and augmented reality, storage is the bigger problem going forward. Hence, the proposed LF Angular 106 SR methods can be implemented to drop some LF views and synthesize them when needed offering new and exciting interactive experiences. 5.3 Future Work Regarding the LF spatial SR methods presented in Chapter 2 and Chapter 3, in the future we plan to further improve both methods by utilizing the additional depth of field information contained in light field images. In addition, the lack of evaluation metrics for light field image quality assessment is an issue worth considering. However, since current evaluation metrics for light field images quality are usually PSNR and SSIM, which do not consider the additional depth information in LF data, we plan to develop a LF objective quality metric to further enhance the super-resolution approaches. The new quality metric will attempt to model the Human Visual System (HVS) in order to assess perceptual quality of light field content. The HVS attempts to select salient areas to reduce cognitive processing efforts [35]. Visual saliency is detected by predicting salient areas through visual attention models (VAM). Thus, designing a VAM that best represents LF content is essential to understand how humans perceive LF. To this end, we will first have to identify which visual cues and in turn saliency maps we will have to use in the case of light field, including texture, focus, depth, and perspective. These saliency maps will be constructed from different types of features including (but not limited to): luminance, color, texture, motion, depth range, field of view, and visual comfort. The proposed visual attention model will be used to predict the saliency of LF content. The predictions will be evaluated through data collected from eye tracking experiments using existing 2D displays, Multiview displays and the Red Hydrogen smartphone [81], which is featuring a LF display. Finally, a dynamic fusion method will be used to combine the proposed saliency maps and form the corresponding quality 107 metric. Such quality metric will have a huge impact on improving any LF super-resolution Framework. Using our knowledge from Chapter 4, we will focus our future research efforts on identifying which of the captured views can be removed at the transmission end and be efficiently synthesized at the receiver end with the best possible visual quality. This process will first involve analysis of the level of redundancies between views and how they affect their synthesis from neighboring views. Then, a synthesis approach will be designed which will allow reconstruction of these views with minimum loss of visual quality. Possible need of using additional metadata will also be explored. 108 Bibliography [1] E. H. Adelson and J. R. Bergen, “The plenoptic function and the elements of early vision,” Comput. Model. Vis. Process., pp. 3–20, 1991. [2] M. Levoy and P. Hanrahan, “Light field rendering,” in Proceedings of the 23rd annual conference on Computer graphics and interactive techniques - SIGGRAPH ’96, 1996, pp. 31–42. [3] A. Gershun, “The light field,” J. Math. Phys., vol. 18, no. 1, pp. 51–151, 1939. [4] E. H. Adelson and J. Y. A. Wang, “Single Lens Stereo with a Plenoptic Camera,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 14, no. 2, pp. 99–106, 1992. [5] S. J. Gortler, R. Grzeszczuk, R. Szeliski, and M. F. Cohen, “The {L}umigraph,” Acm Siggraph 96, pp. 43–54, 1996. [6] M. Levoy, “Light fields and computational imaging,” Computer (Long. Beach. Calif)., vol. 39, no. 8, pp. 46–55, 2006. [7] H. Lu and Y. Li, Artificial Intelligence and Computer Vision, vol. 672. 2017. [8] R. Ng, “Digital light field photography,” Stanford University, 2006. [9] R. Ng, M. Levoy, G. Duval, M. Horowitz, and P. Hanrahan, “Light Field Photography with a Hand-held Plenoptic Camera,” Informational, pp. 1–11, 2005. [10] “Lytro.” [Online]. Available: https://www.lytro.com. [Accessed: 10-Jan-2017]. [11] A. Lumsdaine and T. Georgiev, “Full resolution lightfield rendering,” Indiana Univ. Adobe Syst. Tech. Rep, no. January, pp. 1–12, 2008. [12] Raytrix GmbH, “Raytrix 3D Light Field Camera Technology,” 2017. [Online]. Available: https://www.raytrix.de/. [Accessed: 04-May-2017]. [13] A. Veeraraghavan, R. Raskar, A. Agrawal, A. Mohan, and J. Tumblin, “Dappled 109 photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing,” ACM Trans. Graph., vol. 26, p. 69, 2007. [14] M. Levoy, “The stanford spherical gantry,” (2009-06). http://graphics. stanford,. edu/projects/gantry, 2002. [15] “The (New) Stanford Light Field Archive.” [Online]. Available: http://lightfield.stanford.edu/. [Accessed: 24-Jan-2019]. [16] K. Venkataraman et al., “PiCam: an ultra-thin high performance monolithic camera array,” ACM Trans. Graph., vol. 32, no. 6, pp. 1–13, 2013. [17] D. Cho, M. Lee, S. Kim, and Y. W. Tai, “Modeling the calibration pipeline of the lytro camera for high quality light-field image reconstruction,” in Proceedings of the IEEE International Conference on Computer Vision, 2013, pp. 3280–3287. [18] Z. Zhang and M. Levoy, “Wigner distributions and how they relate to the light field,” in 2009 IEEE International Conference on Computational Photography, ICCP 09, 2009. [19] S. Xu, Z. L. Zhou, and N. Devaney, “Multi-view image restoration from plenoptic raw images,” in Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2015, vol. 9009, pp. 3–15. [20] H. Takeda, S. Farsiu, and P. Milanfar, “Kernel regression for image processing and reconstruction,” IEEE Trans. Image Process., vol. 16, no. 2, pp. 349–366, 2007. [21] R. A. Farrugia, C. Galea, and C. Guillemot, “Super Resolution of Light Field Images Using Linear Subspace Projection of Patch-Volumes,” IEEE J. Sel. Top. Signal Process., vol. 11, no. 7, pp. 1058–1071, Oct. 2017. [22] C.-K. Liang and R. Ramamoorthi, “A Light Transport Framework for Lenslet Light Field Cameras,” ACM Trans. Graph., vol. 34, pp. 1–19, 2015. 110 [23] Y. Yoon, H.-G. Jeon, D. Yoo, J.-Y. Lee, and I. S. Kweon, “Light-Field Image Super-Resolution Using Convolutional Neural Network,” IEEE Signal Process. Lett., vol. 24, no. 6, pp. 848–852, Jun. 2017. [24] A. Levin, W. T. Freeman, and F. Durand, “Understanding Camera Trade-Offs through a Bayesian Analysis of Light Field Projections,” in European Conference on Computer Vision, 2008, pp. 88–101. [25] S. Wanner and B. Goldluecke, “Variational light field analysis for disparity estimation and super-resolution,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 36, no. 3, pp. 606–619, 2014. [26] K. Mitra and A. Veeraraghavan, “Light field denoising, light field superresolution and stereo camera based refocussing using a GMM light field patch prior,” in IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, 2012, pp. 22–28. [27] M. Rossi and P. Frossard, “Geometry-Consistent Light Field Super-Resolution Via Graph-Based Regularization,” IEEE Trans. Image Process., vol. 27, no. 9, pp. 4207–4218, Jan. 2018. [28] X. Wang, L. Li, and G. Hou, “High-resolution light field reconstruction using a hybrid imaging system,” Appl. Opt., vol. 55, no. 10, pp. 2580–2593, Apr. 2016. [29] M. Alam and B. Gunturk, “Hybrid stereo imaging including a light field and a regular camera,” in 24th Signal Processing and Communication Application Conference (SIU), 2016, pp. 1293–1296. [30] T.-C. Wang, J.-Y. Zhu, N. K. Kalantari, A. A. Efros, and R. Ramamoorthi, “Light field video capture using a learning-based hybrid imaging system,” ACM Trans. Graph., vol. 111 36, no. 4, pp. 1–13, 2017. [31] Y. Yoon, H. G. Jeon, L. Yoo, D., J. Y., and I. So Kweon, “Learning a deep convolutional network for light-field image super-resolution,” 2014, pp. 24–32. [32] S. Zhang, S. Chang, and Y. Lin, “End-to-end light field spatial super-resolution network using multiple epipolar geometry,” IEEE Trans. Image Process., vol. 30, pp. 5956–5968, 2021. [33] Y. Yuan, Z. Cao, and L. Su, “Light-Field image superresolution using a combined deep CNN based on EPI,” IEEE Signal Process. Lett., 2018. [34] A. Wafa, M. T. Pourazad, and P. Nasiopoulos, “A Deep Learning Based Spatial Super-Resolution Approach for Light Field Content,” IEEE Access, vol. 9, pp. 2080–2092, 2021. [35] H. W. F. Yeung, J. Hou, X. Chen, J. Chen, Z. Chen, and Y. Y. Chung, “Light Field Spatial Super-Resolution Using Deep Efficient Spatial-Angular Separable Convolution,” IEEE Trans. Image Process., 2019. [36] S. Zhang, Y. Lin, and H. Sheng, “Residual Networks for Light Field Image Super-Resolution,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11046–11055. [37] J. Jin, J. Hou, J. Chen, and S. Kwong, “Light Field Spatial Super-resolution via Deep Combinatorial Geometry Embedding and Structural Consistency Regularization,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 2257–2266, Apr. 2020. [38] Y. Wang et al., “Disentangling Light Fields for Super-Resolution and Disparity Estimation,” IEEE Trans. Pattern Anal. Mach. Intell., 2022. [39] S. Wanner and B. Goldluecke, “Spatial and Angular Variational Super-Resolution of 4D Light Fields,” in European Conference on Computer Vision, 2012, pp. 608–621. 112 [40] N. Kalantari, T. Wang, and R. Ramamoorthi, “Learning-based view synthesis for light field cameras,” ACM Trans., 2016. [41] J. Navarro and N. Sabater, “Learning occlusion-aware view synthesis for light fields,” Pattern Anal. Appl. 2021 243, vol. 24, no. 3, pp. 1319–1334, Feb. 2021. [42] J. Flynn et al., “DeepView: View Synthesis with Learned Gradient Descent,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2019-June, pp. 2362–2371, Jun. 2019. [43] S. Jin, J., Hou, J., Yuan, H. and Kwong, “Learning Light Field Angular Super-Resolution via a Geometry-Aware Network,” in AAAI, 2020, pp. 11141–11148. [44] G. Wu, M. Zhao, L. Wang, Q. Dai, T. Chai, and Y. Liu, “Light field reconstruction using deep convolutional network on EPI,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 6319–6327. [45] Y. Wang, F. Liu, Z. Wang, G. Hou, Z. Sun, and T. Tan, “End-to-End View Synthesis for Light Field Imaging with Pseudo 4DCNN,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 333–348. [46] H. W. F. Yeung, J. Hou, J. Chen, Y. Y. Chung, and X. Chen, “Fast light field reconstruction with deep coarse-to-fine modeling of spatial-angular clues,” in Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 137–152. [47] N. Meng, H. K. H. So, X. Sun, and E. Y. Lam, “High-Dimensional Dense Residual Convolutional Neural Network for Light Field Reconstruction,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 3, pp. 873–886, Mar. 2021. [48] N. Joshi, B. Wilburn, V. Vaish, M. Levoy, and M. Horowitz, “Automatic Color Calibration for Large Camera Arrays,” ACM SIGGRAPH, pp. 3–6, 2005. 113 [49] “Raytrix.” [Online]. Available: https://www.raytrix.de/. [Accessed: 12-Jan-2017]. [50] Y. Tai, J. Yang, and X. Liu, “Image Super-Resolution via Deep Recursive Residual Network,” in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2790–2798. [51] C. Dong, C. C. Loy, K. He, and X. Tang, “Learning a deep convolutional network for image super-resolution,” in European conference on computer vision, 2014, pp. 184–199. [52] J. Kim, J. Kwon Lee, and K. Mu Lee, “Deeply-recursive convolutional network for image super-resolution,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 1637–1645. [53] “UBC DML Light Field dataset captured with Lytro ILLUM,” 2019. [Online]. Available: http://dml.ece.ubc.ca/data/DML-LF/. [Accessed: 13-Jun-2019]. [54] “Stanford Lytro Light Field Archive.” [Online]. Available: http://lightfields.stanford.edu/LF2016.html. [Accessed: 24-Jan-2019]. [55] “4D Light Field Benchmark.” [Online]. Available: http://hci-lightfield.iwr.uni-heidelberg.de/. [Accessed: 24-Jan-2019]. [56] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, “Gradient-based learning applied to document recognition,” Proc. IEEE, vol. 86, no. 11, pp. 2278–2323, 1998. [57] Z. Li, A. Aaron, A. Katsavounidis, Ioannis Moorthy, and M. Manohara, “Toward A Practical Perceptual Video Quality Metric,” Netflix Blog, 2016. [58] M. Rerabek, P. Korshunov, P. Hanhart, and T. Ebrahimi, “Correlation of subjective scores and objective metrics for HDR video quality assessment,” ISO/IEC JTC1/SC29/WG11 MPEG2014/ m35273. Strasbourg, France, 2014. [59] B. T. Series, “Methodology for the subjective assessment of the quality of television 114 pictures BT Series Broadcasting service,” Recommendation ITU-R BT, vol. 3. Recommendation ITU-R BT.500-1 Vol.3, pp. 500–13, 2012. [60] X. Ji, Y. Cao, Y. Tai, C. Wang, J. Li, and F. Huang, “Real-world super-resolution via kernel estimation and noise injection,” IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. Work., vol. 2020-June, pp. 1914–1923, Jun. 2020. [61] Y. Zhang, Y. Tian, Y. Kong, B. Zhong, and Y. Fu, “Residual Dense Network for Image Super-Resolution,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., pp. 2472–2481, Dec. 2018. [62] W. S. Lai, J. Bin Huang, N. Ahuja, and M. H. Yang, “Fast and Accurate Image Super-Resolution with Deep Laplacian Pyramid Networks,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 11, pp. 2599–2613, Oct. 2017. [63] P. Isola, J. Y. Zhu, T. Zhou, and A. A. Efros, “Image-to-image translation with conditional adversarial networks,” Proc. - 30th IEEE Conf. Comput. Vis. Pattern Recognition, CVPR 2017, vol. 2017-January, pp. 5967–5976, Nov. 2017. [64] K. Simonyan and A. Zisserman, “Very Deep Convolutional Networks for Large-Scale Image Recognition,” 3rd Int. Conf. Learn. Represent. ICLR 2015 - Conf. Track Proc., Sep. 2014. [65] K. He, X. Zhang, S. Ren, and J. Sun, “Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification,” Proc. IEEE Int. Conf. Comput. Vis., Feb. 2015. [66] D. P. Kingma and J. L. Ba, “Adam: A Method for Stochastic Optimization,” in 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, 2014. 115 [67] A. Wafa, M. T. Pourazad, and P. Nasiopoulos, “A Deep Learning Based Spatial Super-Resolution Approach for Light Field Content,” submited for publication. . [68] Gordon Wetzstein, “MIT Synthetic Light Field Archive.” [Online]. Available: http://web.media.mit.edu/~gordonw/SyntheticLightFields/index.php. [Accessed: 24-Jan-2019]. [69] P. Matysiak, M. Grogan, M. Le Pendu, M. Alain, and A. Smolic, “A Pipeline for Lenslet Light Field Quality Enhancement,” in 25th IEEE International Conference on Image Processing (ICIP), 2018, pp. 639–643. [70] K. Honauer, O. Johannsen, D. Kondermann, and B. Goldluecke, “A Dataset and Evaluation Methodology for Depth Estimation on 4D Light Fields,” in Asian Conference on Computer Vision, 2016, pp. 19–34. [71] N. K. Kalantari, T. C. Wang, and R. Ramamoorthi, “Learning-based view synthesis for light field cameras,” ACM Trans. Graph., vol. 35, no. 6, p. 193, 2016. [72] G. Wu, Y. Liu, Q. Dai, and T. Chai, “Learning Sheared EPI Structure for Light Field Reconstruction,” IEEE Trans. Image Process., vol. 28, no. 7, pp. 3261–3273, 2019. [73] A. Wafa and P. Nasiopoulos, “GAN-based Light Field Spatial Super-Resolution Using Spatial and Angular Information,” submited Publ., 2022. [74] W. Shi et al., “Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2016-December, pp. 1874–1883, Sep. 2016. [75] A. Wafa, M. T. Pourazad, and P. Nasiopoulos, “LEARNING-BASED LIGHT FIELD VIEW SYNTHESIS FOR EFFICIENT TRANSMISSION AND STORAGE,” in Proceedings - International Conference on Image Processing, ICIP, 2021, vol. 2021- 116 September, pp. 354–358. [76] N. K. Kalantari, T. C. Wang, and R. Ramamoorthi, “Learning-based view synthesis for light field cameras,” ACM Trans. Graph., pp. 1–10, 2016. [77] Y. Zhang, W. Dai, M. Xu, J. Zou, X. Zhang, and H. Xiong, “Depth Estimation from Light Field Using Graph-Based Structure-Aware Analysis,” IEEE Trans. Circuits Syst. Video Technol., vol. 30, no. 11, pp. 4269–4283, Nov. 2020. [78] Q. Zhang, S. Wang, X. Wang, Z. Sun, S. Kwong, and J. Jiang, “A Multi-Task Collaborative Network for Light Field Salient Object Detection,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 5, pp. 1849–1861, May 2021. [79] B. R. Chen et al., “Utilizing Light-field Imaging Technology in Neurosurgery,” Cureus, vol. 10, no. 4, Apr. 2018. [80] “CREAL LIGHT-FIELD DISPLAY .” [Online]. Available: https://creal.com/. [Accessed: 27-Jun-2022]. [81] “RED Digital Cinema | 8K & 5K Professional Cameras.” [Online]. Available: https://www.red.com/. [Accessed: 25-Jun-2019].
Thesis/Dissertation
eng
Electrical and Computer Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Nasiopoulos, Panos
Light field spatial and angular super-resolution
Text