Applied Science, Faculty of
Electrical and Computer Engineering, Department of
DSpace
UBCV
Mehajabin, Nusrat
Doctor of Philosophy - PhD
University of British Columbia
Light field (LF) technology has transformed digital media by capturing scenes from multiple viewpoints, introducing a new level of visual immersion and interactivity. This thesis explores the complexities of LF technology, with a specific focus on compression techniques, quality metrics for assessing processed light fields, and refocusing strategies aligned with the principles of the human visual system. Efficient compression plays a crucial role in LF technology due to the inherent richness of data. Within this thesis, we introduce innovative pseudo-sequence-based prediction structures and coding orders for LF compression. The first structure, informed by interview-similarity, optimizes compression by leveraging reference adjacency and maximizing B-frame usage. This approach excels in bitrate efficiency, real-time decoding, and competitive encoding times, making it suitable for delay-tolerant applications such as broadcasting. The second compression technique employs a frame distance aware structure, enhancing random-access efficiency through diagonal references. This structure offers the shortest encoding time with competitive bitrate gains, making it suitable for real-time interactive applications. Existing LF refocusing methods often fall short of mimicking the natural behavior of the human visual system, resulting in refocused images that appear artificial. Therefore, we developed an innovative LF refocusing technique aimed at producing human perception consistent refocused LFs from camera arrays. By integrating view synthesis, depth estimation, and object segmentation, the proposed method addresses the challenges of depth-dependent refocusing, yielding authentic-looking, natural post-shoot refocusing for camera-array-based content. Traditional quality assessment metrics prove inadequate in capturing the multidimensional nature of LF, underscoring the necessity for novel quality metrics tailored to LF. Finally, we introduce the first LF quality metric tailored for sparsely sampled LFs. This novel full-reference LF quality assessment technique utilizes a volumetric LF representation's cross-section for holistic analysis employing a deep feature extractor. By simultaneously extracting spatial and angular features, this approach comprehensively captures LF quality. This thesis navigates the potential of LF by examining representation, compression, processing, and quality assessment. As LF continues to shape the future of digital media, the solutions presented herein facilitate its effective utilization.
EFFICIENT COMPRESSION, HUMAN-INSPIRED REFOCUSING, AND QUALITY ASSESSMENT OF LIGHT FIELD VIDEOS by Nusrat Mehajabin B.Sc., The University of Dhaka, 2013 M.S., The University of Dhaka, 2015 A THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF DOCTOR OF PHILOSOPHY in THE FACULTY OF GRADUATE AND POSTDOCTORAL STUDIES (Electrical and Computer Engineering) THE UNIVERSITY OF BRITISH COLUMBIA (Vancouver) July 2024 © Nusrat Mehajabin, 2024 ii The following individuals certify that they have read, and recommend to the Faculty of Graduate and Postdoctoral Studies for acceptance, the dissertation entitled: Efficient Compression, Human-Inspired Refocusing, and Quality Assessment of Light Field Videos submitted by Nusrat Mehajabin in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Electrical and Computer Engineering Examining Committee: Panos Nasiopoulos, Professor, Electrical and Computer Engineering, UBC Supervisor Jane Z. Wang, Professor, Electrical and Computer Engineering, UBC Supervisory Committee Member Shahriar Mirabbasi, Professor, Electrical and Computer Engineering, UBC Supervisory Committee Member John D. Madden, Professor, Electrical and Computer Engineering, UBC University Examiner Ryozo Nagamune, Professor, Mechanical Engineering, UBC University Examiner iii Abstract Light field (LF) technology has transformed digital media by capturing scenes from multiple viewpoints, introducing a new level of visual immersion and interactivity. This thesis explores the complexities of LF technology, with a specific focus on compression techniques, quality metrics for assessing processed light fields, and refocusing strategies aligned with the principles of the human visual system. Efficient compression plays a crucial role in LF technology due to the inherent richness of data. Within this thesis, we introduce innovative pseudo-sequence-based prediction structures and coding orders for LF compression. The first structure, informed by interview-similarity, optimizes compression by leveraging reference adjacency and maximizing B-frame usage. This approach excels in bitrate efficiency, real-time decoding, and competitive encoding times, making it suitable for delay-tolerant applications such as broadcasting. The second compression technique employs a frame distance aware structure, enhancing random-access efficiency through diagonal references. This structure offers the shortest encoding time with competitive bitrate gains, making it suitable for real-time interactive applications. Existing LF refocusing methods often fall short of mimicking the natural behavior of the human visual system, resulting in refocused images that appear artificial. Therefore, we developed an innovative LF refocusing technique aimed at producing human perception consistent refocused LFs from camera arrays. By integrating view synthesis, depth estimation, and object segmentation, the proposed method addresses the challenges of depth-dependent refocusing, yielding authentic-looking, natural post-shoot refocusing for camera-array-based content. Traditional quality assessment metrics prove inadequate in capturing the multidimensional nature of LF, underscoring the necessity for novel quality metrics tailored to LF. Finally, we introduce iv the first LF quality metric tailored for sparsely sampled LFs. This novel full-reference LF quality assessment technique utilizes a volumetric LF representation's cross-section for holistic analysis employing a deep feature extractor. By simultaneously extracting spatial and angular features, this approach comprehensively captures LF quality. This thesis navigates the potential of LF by examining representation, compression, processing, and quality assessment. As LF continues to shape the future of digital media, the solutions presented herein facilitate its effective utilization. v Lay Summary This thesis investigates the innovative world of light field (LF) technology, which has revolutionized digital media by capturing scenes from multiple angles, offering an unprecedented level of visual immersion. It centers on three main areas: efficient compression, human-like refocusing, and accurate quality assessment. We introduce two compression techniques. The first optimizes bitrate efficiency, making it ideal for transmission and broadcasting. The second technique can decode the compressed files quickly, which is desirable for real-time applications such as AR/VR content. Next, a novel LF refocusing method has been proposed which mimics human perception for natural-looking refocused images. Since traditional quality assessment methods are inadequate for LF, we proposed a new objective quality metric. Unlike other metrics, this is tailored for sparsely sampled LF data. By addressing these challenges, this thesis aims to unlock the full potential of LF technology, paving the way for its widespread adoption in digital media. vi Preface This thesis presents research conducted by Nusrat Mehajabin under the guidance of Dr. Panos Nasiopoulos. A list of publications resulting from the work presented in this thesis is provided below. A version of Chapter 2 is published in [P3] and two of the proposed improvements taken from the publication in [P1], [P2]. One contribution was made to the MPEG video compression standardization activities [P4] based on this chapter. Some results from the publications [P5] and [P12] are also presented in this chapter. The work presented in Chapter 2 was performed by Nusrat Mehajabin including algorithm design, implementation, performance evaluation, results analysis and verification-validation. For [P1], Nusrat Mehajabin was responsible for the concept, algorithm design, experiment design, verification, validation, and manuscript writing. S. R. Luo and H. W. Yu performed data collection, testing and validation, as well as provided editorial input. J. Khoury provided conceptual ideas, and J. Kaur assisted with data acquisition. M. T. Pourazad served as a supervisory author, offering guidance and oversight. Nusrat took charge of the concept, algorithm design, experiment design, verification, validation, and manuscript writing for [P2]. P. Nasiopoulos and M. Pourazad acted as supervisory authors, providing critical insights and guidance throughout the research. Nusrat was the primary investigator for [P3], handling the concept, algorithm design, experiment design, verification, validation, and manuscript writing. Dr. Panos Nasiopoulos and Dr. Mahsa T. Pourazad contributed as supervisory authors, offering essential oversight and direction. Joseph Khoury contributed to [P5]’s literature review and algorithm design. Nusrat Mehajabin conducted the performance evaluation and analysis for the publication. Dr. Panos Nasiopoulos, Dr. Mahsa T. Pourazad, and Dr. Victor C. Leung were supervisory authors. Nusrat Mehajabin was responsible for the concept, algorithm design, vii experiment design, implementation, and manuscript writing for [P12]. T. Bazzaza, H. R. Tohidypour, and Y. Wang contributed to validation, data sanitisation, testing, and manuscript review. P. Nasiopoulos provided supervisory guidance. Chapter 3’s content appears in the conference papers [P6] – [P8]. For [P6] Dr. Yixiao Wang designed and built the camera array. Dr. Wang, J. Song, M. Huang, and Babaghorbani contributed to the design of the calibration algorithm. The algorithm development, color correction and performance studies were performed by Nusrat. H. R. Tohidypour provided editorial input. V. C. Leung provided supervisory oversight. Penzhi Yan contributed to the design of the graphical user interface of [P7]. Nusrat Mehajabin spearheaded the research, handling the concept, algorithm design, experiment design, verification, validation, and manuscript writing. S. Kaur, J. Song, and Y. Wang supported with data collection, testing, and editorial input. M. T. Pourazad and P. Nasiopoulos provided supervisory guidance. Nusrat Mehajabin was responsible for the primary research tasks including the concept, algorithm design, experiment design, verification, validation, and manuscript writing for publication [P8]. Y. Wang and H. Tohidypour contributed through concept review, data validation and testing, along with editorial input. P. Nasiopoulos and M. Pourazad acted as supervisory authors. The content of Chapter 4 appears in the conference paper [P9] and the submitted journal paper [P11]. [P9] was led by Nusrat, contributing mainly by overseeing the concept, algorithm design, experiment design, verification, validation, and manuscript writing. D. Jin, R. Yao, and T. Dykstra assisted with algorithm implementation, data sanitisation and labelling, and rigorous testing. M. Pourazad and P. Nasiopoulos provided supervision and guidance. The research work of [P11] was mainly performed by Nusrat Mehajabin, including literature review, designing, and implementing viii the proposed methods, performing all evaluations, analyzing the results, and writing the manuscripts. P. Nasiopoulos and M. Pourazad provided supervisory oversight and guidance. The subjective studies in this work were covered under the UBC Ethics Board (H23-02928). List of Publications Based on Work Presented in this Thesis [P1] Mehajabin, N., Luo, S. R., Yu, H. W., Khoury, J., Kaur, J., & Pourazad, M. T. (2019, September). "An efficient random access light field video compression utilizing diagonal inter-view prediction." In 2019 IEEE International conference on image processing (ICIP) (pp. 3567-3570). IEEE. [P2] Mehajabin, N., Pourazad, M., & Nasiopoulos, P. (2020, January). "SSIM assisted Pseudo-sequence-based prediction structure for light field video compression." In 2020 IEEE International Conference on Consumer Electronics (ICCE) (pp. 1-2). IEEE. [P3] Mehajabin, N., Pourazad, M. T., & Nasiopoulos, P. (2021). "An efficient pseudo-sequence-based light field video coding utilizing view similarities for prediction structure." IEEE Transactions on Circuits and Systems for Video Technology, 32(4), 2356-2370. [P4] Mehajabin, N., Nasiopoulos, P., & Pourazad, M. (2019, October). "Light Field Video Compression for Random Access Efficiency." ISO/IEC JTC1/SC29/WG11, MPEG # m50607, Geneva, Switzerland. [P5] Khoury, J., Mehajabin, N., Pourazad, M. T., Nasiopoulos, P., & Leung, V. C. (2022). "An efficient three-dimensional prediction structure for coding light field video content using the MV-HEVC standard." International Journal of Multimedia Intelligence and Security, 4(1), 47-64. [P6] Wang, Y., Mehajabin, N., Tohidypour, H. R., Song, J., Huang, M., Babaghorbani, & Leung, V. C. (2023, May). "A Spatial Calibrated and Colour Corrected Light Field Outdoor Video ix Dataset from a 5x5 Dense Camera Array." In 2023 IEEE International Symposium on Circuits and Systems (ISCAS) (pp. 1-5). IEEE. [P7] Mehajabin, N., Yan, P., Kaur, S., Song, J., Pourazad, M. T., Wang, Y., ... & Nasiopoulos, P. (2022). "An Efficient Refocusing Scheme for Camera-Array Captured Light Field Video for Improved Visual Immersiveness." Hawaii International Conference on System and Sciences. (HICSS). [P8] Mehajabin, N., Wang, Y., Tohidypour, H., Nasiopoulos, P., & Pourazad, M. (2021, November). "An Object-Based Refocusing Scheme for Light Field Video Content." DIGITAL 2021, Advances on Societal Digital Transformation, Athens, Greece. [P9] Mehajabin, N., Jin, D., Yao, R., Dykstra, T., Pourazad, M., & Nasiopoulos, P. (2022, April). "Epipolar Shift Compensated Light Field Video Quality Metric." MMEDIA 2022, The Fourteenth International Conference on Advances in Multimedia, Barcelona, Spain. [P10] Mehajabin, N., Ma, Z., Wang, Y., Tohidypour, H. R., & Nasiopoulos, P. (2022, October). "Real-Time Deep Learning based Road Deterioration Detection for Smart Cities." In 2022 18th International Conference on Wireless and Mobile Computing, Networking and Communications (WiMob) (pp. 321-326). IEEE. [P11] Mehajabin, N., Nasiopoulos, P., & Pourazad, M. (Submitted). "A Deep Feature Consistency-based Sparse Light Field Quality Metric." Submitted. [P12] Mehajabin, N., Bazzaza, T., Tohidypour, H. R., Wang, Y., & Nasiopoulos, P. (2023, December). "Enhancing Light Field Video Compression Efficiency Through View Omission and Synthesis." The 20th ACM SIGGRAPH European Conference on Visual Media Production, London, UK. x Table of Contents Abstract ......................................................................................................................................... iii Lay Summary ................................................................................................................................ v Preface ........................................................................................................................................... vi Table of Contents .......................................................................................................................... x List of Tables .............................................................................................................................. xiii List of Figures ............................................................................................................................. xiv List of Abbreviations ................................................................................................................ xvii Acknowledgements ..................................................................................................................... xx Chapter 1: Introduction and Overview ...................................................................................... 1 1.1 Preliminaries and Definitions ..................................................................................... 5 1.1.1 LF Parametrization.................................................................................................. 5 1.1.2 LF Acquisition ........................................................................................................ 6 1.1.3 LF Representation ................................................................................................... 7 1.2 Overview of State-of-the-art LF Compression ......................................................... 11 1.2.1 PSB Light Field Video Coding ............................................................................. 11 1.3 Overview of State-of-the-art LF Refocusing ............................................................ 14 1.3.1 LF Refocusing ....................................................................................................... 14 1.3.2 LF View Synthesis ................................................................................................ 17 1.3.3 Sparse LF Refocusing ........................................................................................... 18 1.4 Overview of State-of-the-art LF Quality Metric ....................................................... 19 xi 1.5 Thesis Outline ........................................................................................................... 22 Chapter 2: Light Field Video Compression.............................................................................. 24 2.1 Proposed LF Video Compression Methods .............................................................. 25 2.1.1 UPS method .......................................................................................................... 25 2.1.2 Random-access Efficient UPS (RAE-UPS) method ............................................. 32 2.1.3 Coding Order ........................................................................................................ 36 2.1.4 Scalability to View Arrangement .......................................................................... 37 2.2 Performance Evaluation and Discussion .................................................................. 38 2.2.1 Compression Performance of UPS and RAE-UPS ............................................... 41 2.2.2 Computational Complexity of UPS and RAE-UPS .............................................. 50 2.2.3 Random-access performance of UPS and RAE-UPS ........................................... 53 2.2.4 Performance Summary.......................................................................................... 57 2.3 Conclusion ................................................................................................................ 58 Chapter 3: Human Visual System Consistent LF Refocusing ................................................ 59 3.1 Proposed LF Refocusing Method ............................................................................. 60 3.1.1 Sparse LF Refocusing ........................................................................................... 60 3.1.2 Sparse LF View Synthesis .................................................................................... 61 3.1.3 Depth Estimation and Object Segmentation ......................................................... 63 3.1.4 LF Refocusing ....................................................................................................... 65 3.2 Results and Discussion ............................................................................................. 66 3.2.1 Optimal number of views: .................................................................................... 67 3.2.2 Enhancement method ............................................................................................ 68 xii 3.3 Conclusion ................................................................................................................ 72 Chapter 4: LF Quality Assessment ........................................................................................... 74 4.1 Proposed LF Quality Metric ..................................................................................... 74 4.1.1 EPI Consistency Based LF Quality Metric ........................................................... 74 4.1.2 Discussion ............................................................................................................. 83 4.1.3 Deep Feature Consistent LF Quality Metric ......................................................... 84 4.2 Experiments and Evaluation ..................................................................................... 89 4.2.1 Data and LF Operations ........................................................................................ 90 4.2.2 Feature Extraction Model ..................................................................................... 91 4.2.3 Results and Discussion ......................................................................................... 91 4.3 Conclusion ................................................................................................................ 93 Chapter 5: Conclusion ................................................................................................................ 94 5.1 Summary of Contributions ........................................................................................ 94 5.2 Significance and Potential Applications ................................................................... 97 5.3 Future Research Directions ....................................................................................... 99 Bibliography .............................................................................................................................. 101 xiii List of Tables Table 1.1 Representations used in literature for LF quality assessment ....................................... 20 Table 2.1 Properties of the test sequences .................................................................................... 26 Table 2.2 Similarity among the views according to their positions .............................................. 27 Table 2.3 BD-rate gains compared to LF-MVC ........................................................................... 46 Table 2.4 BD-PSNR gains compared to LF-MVC ....................................................................... 47 Table 2.5 SSIM compared to reference ........................................................................................ 48 Table 2.6 Encoding time relative to proposed UPS ...................................................................... 52 Table 2.7 Decoding time relative to proposed UPS ...................................................................... 53 Table 2.8 Random-access complexities of various methods ........................................................ 54 Table 2.9 Performance Summary relative to LF-MVC ................................................................ 57 Table 4.1 LF spatial quality from EPI similarity .......................................................................... 80 Table 4.2 Kurtosis of gradient direction histogram ...................................................................... 82 Table 4.3 Properties of the datasets used. ..................................................................................... 89 Table 4.4 Performance Comparison of the proposed and other LF quality metrics ..................... 90 xiv List of Figures Figure 1.1: Light field camera design ............................................................................................. 1 Figure 1.2 4D Light Field captured with microlens based camera ................................................. 2 Figure 1.3: 2D camera array (5×5) ................................................................................................. 3 Figure 1.4 Lenslet LF representation .............................................................................................. 8 Figure 1.5 Lenslet LF representation with microlens view ............................................................ 9 Figure 1.6 Sup-aperture image (SAI) extraction from lenslet representation ................................. 9 Figure 1.7 Epipoloar Image (EPI) representation of LF ............................................................... 10 Figure 1.8 Existing pseudo sequence-based prediction structures................................................ 12 Figure 1.9. (a) Real focused imaged (b) synthetically refocused image ....................................... 15 Figure 1.10 Scope of the thesis with respect to general LF pipeline ............................................ 22 Figure 2.1 I-frame selection using similarity computation ........................................................... 29 Figure 2.2 Proposed and experimental PSB prediction structure for "ChessPieces" LF dataset .. 30 Figure 2.3 Proposed pseudo-sequence-based prediction structure utilizing diagonal references . 34 Figure 2.4 Scalability to different view arrangements .................................................................. 36 Figure 2.5 First frame of the test sequences (central/ I-frame) ..................................................... 40 Figure 2.6 Rate distortion curves for the LF video test sequences ............................................... 44 Figure 2.7 Different prediction structures and bitrates of 300 frames 7th view of “ChessPieces” LF video ........................................................................................................................................ 49 Figure 2.8 Temporal random-access complexity: to access all the views at any time instance using the proposed UPS method, for 5×5 LF video GOP-4. The green frames are already encoded views. The black arrows represent current predictions .................................................. 56 xv Figure 3.1 Refocusing when same object spans over multiple depth panes. Left – far, center – middle, and right - near ................................................................................................................. 59 Figure 3.2. Examples (n = 3) of the proposed sparse and dense view interpolation patterns ....... 61 Figure 3.3 Object segmentation for object base refocusing. (a) original and (b) annotated frame....................................................................................................................................................... 63 Figure 3.4 Detectron2 segmented frame ....................................................................................... 64 Figure 3.5. Comparison of shift-and-sum refocusing results: (a) original -no view synthesis, (b) one novel view synthesized between original views, (c) two novel views synthesized between original views, (d) three novel views synthesized between .......................................................... 65 Figure 3.6. Sparse vs dense view synthesis. (a), (c) and (e) sparse pattern with n=1, 2 and 3 respectively. (b), (d) and (f) dense pattern with n = 1, 2 and 3 respectively. ............................... 66 Figure 3.7. SSIM-based vs NCC-based enhancement method without blending weights. .......... 68 Figure 3.8. LF post refocusing in-focus region enhancement (a) no enhancement, (b) SSIM-based enhancement, (c) NCC-based enhancement and (d) NCC-based enhancement with blending weights. .......................................................................................................................... 69 Figure 3.9 (a) Original reference view, (b) Refocusing without enhancement, (c) Refocusing with object segmentation-based enhancement, (d), (e) and (f) are the enlargements of refocused objects of (a), (b) and (c), respectively ......................................................................................... 70 Figure 3.10 .(a) original reference view, (b) refocused on the painter at 3.2m distance and (c) refocused on the painter at 3.2m distance with enhancement ....................................................... 71 Figure 4.1. Horizontal and vertical EPIs of 'Painter' LF test sequence ......................................... 75 xvi Figure 4.2. Diagonal EPI for views I_{1,1} ,I_{2,2} ,I_{3,3} , and I_{4,4} . The three figures on the left show an overlay of each pair of views along the diagonal and the matching Scale Invariant Feature Transform (SIFT) features that were detected. ................................................. 76 Figure 4.3 Spatial quality measurement........................................................................................ 77 Figure 4.4 Angular quality measurement...................................................................................... 78 Figure 4.5 EPI similarity based LFQM outline ............................................................................ 79 Figure 4.6 Gradient histograms of the horizontal, vertical, and diagonal EPIs. The histograms show that the gradient direction is more concentrated at ±90° for higher QP values. .................. 81 Figure 4.7 Proposed volumetric LF representation using similarity-based stacking .................... 85 Figure 4.8 (a) a standard CNN with no efficient feature propagation and (b) a dense block concatenating learned features from previous layers .................................................................... 85 Figure 4.9 The framework of the proposed DFCS LF quality assessment metric ........................ 88 xvii List of Abbreviations 2D Two dimensions 3D Three dimensions 4D Four dimensions 7D Seven dimensions AR Augmented reality AVC Advanced video coding BD Bjontegaard delta CMOS Complementary metal-oxide-semiconductor CNN Convolutional neural network DB Decibel DFCS Depth feature consistent space DoF Degree of freedom DSLR Digital single lens reflex EPI Epipolar image plane FPS Frames per second FR Full reference FVS Free view synthesis GB Giga byte HD High definition HDR High dynamic range HEVC High efficiency video coding xviii HVS Human visual system IEEE Institute of electrical and electronic engineers ITU-T International telecommunication union – telecommunication JND Just noticeable difference Kbps Kilo bits per second LF Light field LR Low resolution MOS Mean opinon score MPEG Moving picture experts group MR Mixed reality MSE Mean square error MV-HEVC Multiview high efficiency video coding MVC Multiview coding NeRF Neural radiance field NR No reference PQ Perceptual quantizer PSNR Peak signal to noise ratio QA Quality assessment QM Quality metric QP Quantization parameter RAE Random access efficient xix SAI Subaperture image SDR Standard dynamic range SSIM Structural similarity index metric UHD Ultra high definition UPS Universal prediction structure USD United states dollar VMAF Video multimethod assessment fusion VR Virtual reality VVC Versatile video coding xx Acknowledgements I wish to begin with, by expressing my heartfelt appreciation to my supervisor, Dr. Panos Nasiopoulos. His remarkable guidance and unwavering inspiration have not only shaped my academic journey but also exemplified kindness, patience, and dedication to research. As a role model, he has consistently supported and uplifted me during my time at UBC, providing invaluable encouragement and insightful feedback that have truly enriched my growth. A special thank you goes to Dr. Mahsa T. Pourazad for her time, thorough feedback, and unwavering support across the various stages of this thesis. My sincere gratitude extends to my esteemed Ph.D. committee members, Dr. Jane Z. Wang and Dr. Shahriar Mirabbasi, who generously shared their time and wisdom, guiding me throughout my graduate studies. To my fellow colleagues at the UBC Digital Multimedia Lab – Abrar Wafa, Anahita Shojai, Doris Xiang, Dr. Hamid Reza Tohidypour, Dr. Yixiao Wang, Joseph Khoury, Lisa (Fangyuan) Chi, Maryam Azimi, Parnia Haji Faraji, Pedram Mohammadi, Stelios Ploumis, Tala Bazzaza, Thomas Zhang, and Zhenchao Ma – I extend my heartfelt thanks to you all. Our stimulating discussions, collaborations and friendship have greatly enriched my academic journey. I offer my deepest appreciation to my parents, Amina and Nazimuddin, whose unwavering belief and steadfast encouragement have been my foundation. Their sacrifices and dedication to my education have been truly humbling. My gratitude extends to my siblings, Nadia, for her unwavering support, love, understanding and kindness, and to Fahim for his encouragement. xxi Lastly, my sincere thanks to the Natural Sciences and Engineering Research Council of Canada (NSERC), the Electrical and Computer Engineering department, and the University of British Columbia for their financial support through research grants and tuition awards. This assistance has been pivotal in making my academic pursuit possible. 1 Chapter 1: Introduction and Overview Light field (LF) technology is the latest emerging innovation in digital media, seen as an “upgrade” to the way we capture and reproduce visual information. LF is defined as the light rays at every point in a space that travel in every direction and can be represented as a 4D-vector that describes the light as a function of position and direction. Unlike traditional cameras that only capture the scene from a single viewpoint or stereoscopic cameras (3D) that captures the scene from two viewpoints or 360˚ camera that captures the camera surroundings (different scenes from same viewpoint), LF camera captures the same scene from multiple viewpoints. LF acquisition schemes are implemented by the means of sampling the 4D plenoptic function to measure radiance, color and intensity values from multiple viewpoints [1]. To achieve this, LF (or else known as plenoptic) cameras have a microlens array after the main lens and just in front of the imaging sensor [1]. Such arrays consist of many microscopic lenses with tiny focal lengths and split up what would have become a 2D-pixel into individual light rays just before reaching the sensor. As Figure 1.1 shows, the light rays are separated by the micro-lens before they hit the image sensor at the back of the camera. The resulting raw image consists of a two-dimensional array of micro-lens images, which create a 4D light field as illustrated in Figure Figure 1.1: Light field camera design 2 1.2(a). Several views, each one containing a slightly shifted perspective from the other can be extracted as shown in Figure 1.2(b). Another way of capturing LF content is by a camera-array setup with homogeneous or heterogeneous cameras arranged on a rig (see Figure 1.3) [2]. The camera-array setup captures the scene from the perspective of each camera and the combination of the information from all the cameras creates the LF. In comparison to the microlens approach, a camera-array-based setup would have higher resolution but a longer baseline (distance between two lenses) and further apart views than the microlens-array [3]. Regardless of the capturing method, light field content is an abundance of captured information that enables immersiveness through the depth of field, refocusing at multiple levels, variable resolution object identification and all-around view perspective [4]. With that, however, comes a huge amount of information that is one of the challenges for enabling this technology. Presently this is a hot topic of research, while the wealth of visual information captured by light field technology is of profound importance to many emerging trends and markets in the digital economy, ranging from autonomous driving [5], highly immersive Virtual Reality (VR) and (a) (b) Figure 1.2 4D Light Field captured with microlens based camera 3 Augmented Reality (AR) [6], education/training, entertainment [7], remote medicine [8], to human-behavior assessment for health and wellness. The combined global digital entertainment, Augmented Reality, autonomous driving and health markets are expected to surpass $4T USD by 2024 and light field (LF) technology is in the center of all of them [9]–[11]. This is due to the fact that visual information is of profound importance to digital media, analytics, and artificial intelligence, whose intersection is expected to have the most disruptive impact on emerging trends and markets. The first mention of LF was back in 1908 by Lippman [12] who received a Nobel Prize for this contribution. The next spotlight on LF was cast by Levoy [3] in 1996. Despite the numerous possibilities and applications, LF did not receive significant attention until recently, mainly because of the technological challenges and the tremendous amount of data it generates. For example, the Lytro ILLUM generates raw output images of 7728 × 5360 pixels, which is almost 20 times the resolution of traditional high definition (HD) content, while the format of the content generated is dramatically different from the 2D images and videos [1]. Similarly, a camera-array setup such as the one shown in Figure 1.3 will produce 25 times more data than 4K UHD cameras. Due to the fact that the format and amount of generated video data are so different from legacy Figure 1.3: 2D camera array (5×5) 4 video streams, conventional compression standards are ill-suited for efficiently coding LF. In addition, legacy transmission and storage systems could not handle such a vast amount of data. Recent advances in capturing and displaying technologies have put light field in the center of attention for research and development sectors and forced international standard bodies such as JPEG and MPEG to start preparing calls for proposals that will address the need for efficient LF compression schemes [14]. However, all these efforts are at infancy stages and there is still no standard for LF video compression. Naturally, an efficient coding scheme for such a tremendous amount of data is a pivotal factor in enabling the technology and opening the door to new market opportunities. Although the most widely used video compression standard high efficiency video coding (HEVC) [13] has a Multiview extension [14] that can be used to compress LF video with modifications to the prediction structure and coding order, the outcome is far from satisfactory and more advanced and sophisticated schemes need to be developed to address the transmission and processing challenges of LF video content. One of the most attractive features of LF technology is on-demand digital refocusing, whose applications are crucial for many domain-specific markets, ranging from entertainment and training to autonomous driving and remote surgery. This feature allows us to focus on multiple levels and in turn objects, closely mimicking human perception [15]. This handy operation is performed by disparity estimations. For microlens-array-based cameras, the process is referred to as shift and sum [16]. Because of the close proximity of the microlenses and the narrow depth of field, the ‘shift and sum’ generates almost as good refocusing as human eyes. However, for camera-array-based set up the same process does not generate accurate results. This is mainly due to the large space between the views. A user would experience artifacts such as blockiness, non-uniform blur and blending, etc. for the defocused areas. To accommodate LF in different 5 applications beyond the short field of a microlens camera, we need to develop seamlessly robust and efficient digital refocusing techniques for camera array LF applications. Due to the numerous possibilities operations such as compression, calibration, and super-resolution started to receive attention from academia and industry alike. However, processing the LF data for these applications introduces artifacts and compromises the quality of the LF and consequently the performance of the application. There are several classical image quality metrics such as peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), video multimethod assessment fusion (VMAF) [17], etc. that can be adopted to evaluate the quality of LF. These metrics, however, are designed for 2D images where the data pertains to only color information of the scene (e.g., pixel intensities), while LF not only captures the spatial information, but it also captures the angular or geometric information about the scene from multiple viewpoints. The angular quality also needs to be quantified after processing as the performance or accuracy of the target application will depend on it. Although these metrics are convenient and can accurately estimate the quality of 2D images, they clearly cannot account for the relationship among the different views of LF. Therefore, the classical metrics are not well suited to evaluate the quality of LF. Naturally, there is a need for a metric that comprehensively assesses the quality of the processed LF. 1.1 Preliminaries and Definitions In this section we discuss a definitions and preliminaries essential for the rest of the thesis. 1.1.1 LF Parametrization Space is filled with light rays of various intensities. LF is defined as the light rays at every point in space that travel in every direction. It is represented by the 7D plenoptic function 6 𝐿(𝑥, 𝑦, 𝑧, 𝜃, 𝜑, 𝛾, 𝑡). Here, (𝑥, 𝑦, 𝑧) is the location, (𝜃, 𝜑) is the angle, 𝛾 is the wavelength and 𝑡 is the time of the light ray [5]. Researchers have reduced this 7D function to 4D for practical usage [18]. This reduced function is assumed to be time-invariant (reducing time dimension), monochromatic with constant radiance across a light ray. The 4D LF 𝐿(𝑢, 𝑣, 𝑠, 𝑡) is parameterized by the intersections of any light ray with two arbitrary planes 𝑠𝑡 and 𝑢𝑣. We can consider the 𝑠, 𝑡 dimensions as angular dimensions and 𝑢, 𝑣 as the spatial dimensions. In other words, samples of the 𝑠𝑡 plane are cameras that capture the light rays coming from the samples of the 𝑢𝑣 (perpendicular to the view direction) plane. Hence, each sample of the 𝑠𝑡 plane will have its own perspective of all the light rays coming from the 𝑢𝑣 plane [19]. The simplified LF representation as a 4D function is presented in eq. 1.1. 𝐿𝐹 = 𝐿(𝑢, 𝑣, 𝑠, 𝑡) (1.1) 1.1.2 LF Acquisition LF can be captured by multiple sensors/cameras. The cameras are distributed over a planar [2] or spherical [20] rig to simultaneously capture the scene. Here, the number of the cameras is the angular resolution (𝑠𝑡 plane) and the spatial resolution 𝑢 × 𝑣 is determined by the resolution of the image sensors of the cameras. Another popular way of capturing LF is by multiplexed sensors. An array of microlenses is placed in between the main lens and the imaging sensor of the camera [1], [21]. The resultant 2D image is a multiplexed image, where the 4D information is encoded. We can demultiplex [22] the 2D image by gathering the pixels in the same location of each micro-image (image from each microlens). Each decoded image is from a certain viewpoint and is called a subaperture image (SAI). Plenoptic cameras produce angularly dense LFs [23]. On the other hand, camera arrays [2] involve multiple cameras arranged on a rig to capture a scene. The distances between the lenses of the cameras (centimeters) are far greater than they are in plenoptic 7 cameras. This means the point of views are further apart too. Hence, camera array’s produce angularly sparse LFs. Here the number of cameras dictates the angular resolution/number of viewpoints of the LF. Each camera's photo-sensor is dedicated to capturing a single image, resulting in higher spatial resolution LFs. 1.1.3 LF Representation Regardless of the capture medium, tremendous amount of information is store to describe an LF. State-of-the-art LR representations are discussed below. 1.1.3.1 Lenslet LF could be represented using a lenslet representation when captured with microlens array. The camera captures LF raw data, which is the pixel values of the photosensor placed behind the microlens array and the main lens. The LF raw data basically captures the light incident upon each microlens. It is assumed that the microlenses and the photosensor lie on the spatial plane (𝑥, 𝑦) and the angular plane (𝑠, 𝑡), respectively. The photosensor of the plenoptic camera performs the 4D light field sampling of the axes x and y, where each image on the sensor represents the light arriving at that film point from different (s, t) positions. Thus, the raw image is composed of an (x, y) grid of circular images that form underneath each microlens. The raw image appears like a typical image photographed by a conventional camera. However, looking more closely we can notice that the microlens array is clearly visible, where each circular image represents the light arriving at that sensor from a different angle. Figure 1.4 shows this phenomenon. Such image representation is called lenslet LF image. Figure 1.5 shows a lenslet representation of an LF. This is an uncommon format and is usually stored with. lfp or .lfr extensions [24] and can be edited or processed with software designed for LF content processing. 8 Figure 1.4 Lenslet LF representation 1.1.3.2 Sub-aperture Images (SAI’s): Sub-aperture Images (SAIs) are the set of conventional images but taken from each viewpoint. The content from camera array is already in this format. Feed from each camera is considered a SAI. For microlens cameras the sub-aperture images can be formed by taking the same pixel underneath each microlens, as shown in left portion of Figure 1.6. This creates an array of images [22], where each image is a picture of the scene from the corresponding (s, t) position. The right part of Figure 1.6 shows the extracted sub-aperture images. Parallax is readily visible in such representation. 9 Figure 1.5 Lenslet LF representation with microlens view Figure 1.6 Sup-aperture image (SAI) extraction from lenslet representation 1.1.3.3 Epipolar Images: LF is described using a standard two-plane parameterization. Rays are defined using two parallel planes  and . The first plane Ω denotes image coordinates (x, y)  . The second plane Π contains the focal points (s, t) ∈ Π of all cameras. An entire 4D light field can thus be described by a function. 𝑅 (𝑠, 𝑡, 𝑥, 𝑦) → 𝐿(𝑠, 𝑡, 𝑢, 𝑣) (1.2) where 𝐿(𝑠, 𝑡, 𝑢, 𝑣) defines the intensity of the corresponding ray defined by the intersection (𝑢, 𝑣) with the image plane and (𝑠, 𝑡) with the focal plane, respectively. Furthermore, (𝑢, 𝑣) can 10 be treated as the spatial, and (𝑠, 𝑡) can be treated as the angular resolution of the LF. Hence, there will be 𝑠 × 𝑡 sub-aperture views each having the resolution of 𝑢 × 𝑣. Figure 1.7 Epipoloar Image (EPI) representation of LF These sub-aperture views are slightly shifted from each other depending on the distance between the cameras. These disparities among the views can be estimated on the 2D slices. The captured scene from both acquisition systems can be represented either by sub-aperture images (SAIs), the 2D images of the scene from each viewpoint or, by epipolar plane images (EPIs) [25]. To elaborate, if an LF is described using equation (1), (𝑢, 𝑣) can be treated as the spatial, and (𝑠, 𝑡) as the angular resolution of the LF. Hence, there will be 𝑠 × 𝑡 sub-aperture views, each having a resolution of 𝑢 × 𝑣. By setting 𝑡 to a fixed value 𝑡∗ and 𝑣 to a fixed value 𝑣∗ we can generate the EPIs as ∑ 𝐿𝑡∗𝑣∗ . ∑ 𝐿𝑡∗𝑣∗ from the 4D light field structure. Figure 1.7 shows typical horizontal (bottom right) and vertical (right) LF EPI. 11 𝑆𝑡∗𝑣∗: ∑ 𝐿𝑡∗𝑣∗ → 𝑅 (1.3) (𝑢, 𝑠) → 𝑆𝑡∗𝑣∗(𝑢, 𝑠) ∶= 𝐿(𝑠, 𝑡∗, 𝑣∗, 𝑢) (1.4) Other slices with different fixed coordinates such as 𝑠 and 𝑢, are defined analogously. 1.2 Overview of State-of-the-art LF Compression In this section, we discuss the literature of pseudo sequence-based prediction structure and coding order for LF videos. Most existing works in this domain focus on compression efficiency, random-access complexity and encoding time or performance of the techniques. 1.2.1 PSB Light Field Video Coding The LF capturing process creates multiple views from multiple vantage points of a scene with high inter-view and temporal similarities among the views. Consequently, the multi-view extension of HEVC (MV-HEVC) [14] is one option for compressing LF videos. The compression efficiency depends on how the different views are ordered for prediction and redundancy removal. As this order could be different from the actual view order of the LF content, it is known as pseudo-sequence based (PSB) prediction structure and coding order. This is a challenging issue, as there is a large number of possible prediction structures and coding orders. As a result, there have been several works that try to address this problem. Recent approaches for PSB prediction structure include the LF Multiview coding (LF-MVC) [26] which improves compression efficiency over multi-view coding (MVC) [27] by introducing a two-directional prediction structure. Another method proposed by Khoury et al. [28] places the I-frame at the center assuming the center view has the highest similarity with all other views. This improves compression compared to LF-MVC. Other approaches, such as the ones described in [29] and [30] introduce diagonal references to enhance random-access efficiency. Random-access efficiency relates to the number of frames that 12 are in the decoding path to access a random view. The lower this number the higher the random-access efficiency is [29]. The most recent work proposes a PSB prediction structure that aims at maximizing the number of B-frames [31]. 1.2.1.1 LF-MVC One of the noteworthy contributions in PSB LF video coding is LF-MVC [26]. This method uses a two-directional parallel inter-view prediction structure, shown in Figure 1.8(a). Due to the parallelism, the encoding process is faster. Unlike previous works such as transposed sequence [32], Zig-Zag [33], and Rotation Scan [34] that traverse views in a single direction, LF-MVC selects the top left view as the I-frame and propagates the prediction flow in horizontal and vertical directions. A 3×3 substructure is iterated to match the number of views. This prediction structure achieves a 26% bitrate gain over the conventional hierarchical B prediction structure of MVC. Although the repetition of the 3×3 substructure forms a closed loop, this does not decouple the substructure from other views. During reconstruction, many views inside the closed loops must wait for views from other loops to finish reconstruction before they could start. This increases the (a) LF-MVC [20] (b) Khoury’s method [22] (c) Full-scheme [25] PBBBBB BB B BBBBB BI PBPBPBBBBPBBBBB BB B BBBB B BI PBPBPBB B B PBBBBB BB B BBBB BBIB BBBB BBB BFigure 1.8 Existing pseudo sequence-based prediction structures 13 random-access complexity of the views. This structure only exploits either horizontal or vertical correlation among views. Although the authors have presented how their prediction structure can be modified for view arrangement other than odd × odd structures, they have not considered how the arrangement will adapt if the number of views increases. Therefore, this structure is not scalable. As the number of views increases, the number of B-frames situated further away from I-frame will grow dramatically compared to the rest of the structure. This in turn will increase the decoding time. 1.2.1.2 Khoury’s Method Liu et al. [35] have observed that the central view of an LF content bears higher similarity with the rest of the content than any other view and is a better choice for predicting other views. Khoury et al. drew from Liu’s observation and designed a prediction structure shown in Figure 1.8(b) for LF video compression using MV-HEVC [28]. This structure also forms the 3×3 substructure of closed loops. But unlike LF-MVC here the reconstruction of any view only seeks frames from within the loop. Therefore, the random-access complexity is confined to the loop only. Khoury’s method has achieved up to 38% bitrate reduction and 30% less random-access complexity compared to LF-MVC. However, the prediction structure relies on a fixed, symmetric reference frame placement strategy. Placing the P- frames further away from the I-frame is bitrate consuming. This is particularly concerning when the number of views increases beyond 5×5. The similarity to the distant views will decrease as a result bitrate will increase. Furthermore, the hierarchical B-frames are predicted only by using either horizontal or vertical references. Therefore, this structure cannot take full advantage of both horizontal and vertical correlation. And how the structure would change for view arrangements other than odd × odd has also not been discussed. 14 1.2.1.3 Full Scheme: B-frame maximization approach Avramelos et al. published one of the most recent works in PSB LF video compression techniques [31], known as the Full- scheme. This work presents a detailed overview of the existing prediction structures and proposes a prediction structure of their own, which is shown in Figure 1.8(c). In this unconventional prediction structure, they place the I-frame at the bottom right corner and the only P-frame of the structure at the top left corner. Next, the hierarchical B-frames are predicted in reverse spiral order. This structure is based on the theory that maximizing the number of bi-directional (B) frames minimizes the bitrate. Apart from horizontal and vertical references, they have also utilized diagonal references to form the spiral pattern. The compression efficiency gained by the Full-scheme is only visible in higher quality videos. As the structure has a significantly low number of keyframes and a spiral topology, the random-access complexity increases substantially compared to any of the discussed structures. To address the high random-access complexity, the authors have provided a random-access efficient structure. In this structure, the I-frame is placed at the center and every other view is a P-frame predicted from the I-frame. This prediction structure is aggressive to achieve minimum random-access complexity, but the repercussion is a loss of compression efficiency. 1.3 Overview of State-of-the-art LF Refocusing 1.3.1 LF Refocusing The basic LF refocusing algorithm relies on shifting the sub-aperture images according to the desired focus plane and the disparity of each sub-aperture image and then averaging the shifted images. Let us assume there are 𝑈 × 𝑉 cameras and each image 𝐼(𝑢,𝑣) captured by the camera {u, v) | u ∈ U, v ∈ V, has a 𝑆 × 𝑇 pixel resolution. During the refocusing process, one camera will 15 be firstly picked as the reference camera. The baseline disparity shift (𝛿𝑢, 𝛿𝑣) between the pixel (𝑠0, 𝑡0) | s ∈ S, t ∈ T in the reference view and the pixel (𝑠, 𝑡) | s ∈ S, t ∈ T in the other views is calculated at depth z1 with a far-field depth z0, where z0 is arbitrarily large to minimize the reprojection distortion. For each camera, the disparity shift is constant for all the pixels according to the assumption that all the cameras are coplanar. Then, the disparity shift at any depth 𝑧 could be calculated as 𝑑(𝑧) ∗ (𝛿𝑢, 𝛿𝑣) as follows: 𝑑(𝑧) = 1𝑧−1𝑧01𝑧1−1𝑧0 (1.5) The refocused image 𝐼𝑐′(𝑠, 𝑡, 𝑧) at depth 𝑧 is computed by the shift-and-sum algorithm as follows: 𝑠′ = 𝑠 + 𝑑(𝑧) 𝑡′ = 𝑡 + 𝑑(𝑧) 𝐼𝑐′(𝑠, 𝑡, 𝑧) = ∑ ∑ 𝐼(𝑢,𝑣)(𝑠′⋅δ𝑢,𝑡′⋅δ𝑣)𝑀−1𝑣=0𝑁−1𝑢=0𝑈𝑉 (1.6) Pixels in all the other views are shifted based on the disparity, then averaged to get the refocused image [36]. Note that eq. 1.6 also follows the coplanar assumption, which states that for each camera, the disparity shift (𝛿𝑢, 𝛿𝑣) is constant over the entire image. The aperture of an LF capture can be synthetically (in software post capture) adjusted to refocus on an arbitrary focal plane (depth) [37]. However, the main difference with real aperture capture (a) (b) Figure 1.9. (a) Real focused imaged (b) synthetically refocused image Spatial sampling Spatial samplingAngular samplingAngular samplingp1 p2 p’1 p’216 at that depth and a synthetic aperture is that the real one gathers all the light rays passing through the camera and the synthetic one acquires only a subset of those rays from all the cameras of the capture setup. Ideally an image 𝐼 is formed by integrating all the light rays 𝑅 entering the camera from the scene. However, if we have a subset of all the light rays coming from the scene the resultant image looks jagged. Let us analyze this problem for two points 𝑝1 and 𝑝2. Figure 1.9(a) illustrates the situation for real focused capture and Figure 1.9(b) shows what happens when we synthetically adjust the focal plane to refocus. For the real image both 𝑝1 and 𝑝2 receive angular samples of all three colors however, for the synthetic case 𝑝1′ receives green and 𝑝2′ gets red and blue angular samples. This sudden and abnormal behaviour in neighboring pixels causes aliasing artifacts in LF refocused images. This phenomenon effects the not-in-focus region severely as the samples are further scattered. The impact is graver if the content is from a camera array. It also effects the in-focus region however, due to the shifting of the shift-and-sum algorithm the samples become closer consequently reducing the aliasing. The aliasing reduction is less for sparse content and some blurriness remains even in refocused region. Later, when the averaging of the angular samples for the same spatial sample takes place, it leaves the pixel blurry rather than in sharp focus. Refocusing camera array content enables a wide array of applications as it lifts the restriction of specialty hardware e.g., microlens array. Any consumer camera could be used to form a camera array or even one camera/smartphone can capture the scene from multiple viewpoints. Furthermore, the sampling and storage requirements of lenslet LF images is too high and not feasible for many applications. One practical way to obtain photorealistic and human visual system consistent refocusing on sparse LF is to render novel views from the available ones to compensate for the shift and sum artifacts. 17 1.3.2 LF View Synthesis Image based rendering is a well-studied area of computer vision and graphics. A review of early methods can be found in Shum and Kang [79]. More recently, deep learning techniques introduced advanced level of flexibility and realism to the rendered content. This section reviews the view synthesis approaches with specific focus on LF novel view synthesis. Deep learning based view synthesis methods use different scene representations such as densely sampled LFs from lenslet camera, sparsely sampled LFs from camera gantries and some propose intermediate representations such as EPIs, Multiplane Images (MPIs), voxel grids, NeRF [38] etc. to aid their methods. We are particularly interested in novel view rendering from sparse LF. Traditional interpolation techniques might seem like an obvious solution however, large motion among the views, in other words large disparity among the sparse samples renders unrealistic views. 1.3.2.1 Neural Radiance Field (NeRF) NeRF [38] aims to recreate the 5D plenoptic function discussed in eq. 1.1 from sparse LF samples. This is remarkably successful in generating photorealistic novel views. At core NeRF is a fully connected neural network. It is trained to map every spatial location (opacity and color) of the input 2D images in a volumetric space. Although it is highly effective, the rendering is slow, complex scenes can take days to render, and it needs synthesizing device to run a complex architecture as the rendering occurs in a ray-by-ray fashion making this unsuitable for an LF refocusing application. 1.3.2.2 Free View Synthesis (FVS) FVS [39] can render novel views from freely distributed input images of a scene. It leverages a recurrent encoder-decoder deep learning architecture to reproject the features of the input images 18 to a geometric scaffold which is later used as a reference for depth related information. One lucrative feature of FSV is the proposed network does not need to be retrained or fine-tuned for different datasets. However, to render the novel view from the geometric scaffold, FSV uses a heuristic selection of relevant input images. Therefore, when the target viewpoint changes the selection of input images changes too. This transition creates temporal instability and creates visual artifacts. The severity of the artifact depends on how the drastically target viewpoint changes. As the refocusing application is for human consumption therefore, techniques that may create visual artifacts are not suitable. 1.3.2.3 Stereo Magnification with Multi-layer Images Stereo layers [40] based LF view synthesis offers fast rendering time, compatibility with graphics cards and driver available in most devices and photorealistic novel views. This approach uses two stereo images vertical or horizontal to predict the geometry of the scene. It is represented in a mesh form. Next, the alpha channel and the colors (texture) are inferred. The alpha or the transparency layer number impacts the generalization of the scene geometry. Previous methods either used too few layers e.g., 32 or way too many layers e.g., 128. Inadequate number of layers leads to not approximating the true geometry and excessive number of layers leads to overfitting of the network. The authors solved this issue by incorporating the number of layer selection into the stereo matching network. 1.3.3 Sparse LF Refocusing A stereo-image refocusing method was introduced in [41]. This method selectively blurs the image based on the estimated depth using the stereo image, to create a refocused effect. However, this method was not extended for more than two cameras. Similar problems can also be found in [42] and [15]; the former creates a refocused image using three images, whereas the latter generates the 19 entire LF from a single image. Recent method [37] also proposes to generate the out of focus region using depth-based anisotropic filters and in-focus region is produced by reconstruction based super resolution. Up to this point, all LF refocusing research assumed the new focal plane for refocusing will be parallel to the camera. Another recent work [4] considered the scenario where the viewer tilts their head and the desired refocus plane and viewpoint are no longer parallel to the camera. In summary, all the above-mentioned approaches are designed for refocusing to a new focus plane, thereby, bringing all the pixels that are a certain distance from the camera in-focus and rendering everything else as depth wise blur. However, when we observe a scene and focus on an object, every part of the scene having the same distance from our eyes does not come into focus for us. Instead, we see that object clearly and everything else is in depth wise blurred with respect to that object. 1.4 Overview of State-of-the-art LF Quality Metric Recently, researchers have designed some objective quality metrics specifically for LF [43]. Paudyal et al. proposed a reduced reference objective quality metric, where the authors compared the depth map distortion of the processed LF with the original LF’s depth map [44]. It is a reduced reference metric as the entire original LF is not needed (only the depth map is needed) to evaluate the quality of the distorted LF. The depth maps are significantly smaller in size compared to the LF therefore, it is suitable for applications that need the LF to be transmitted over communication channels. However, the performance of the metric is dependent on the depth map estimation method involved and is only as accurate as the estimated depth map. The metric is validated using only densely sampled LFs as the depth maps for sparse LFs are less accurate. Shan et al. designed a no-reference quality metric that utilizes 2D and 3D features of LFs to objectively quantify JPEG compression, white noise, and blurring artifacts [45]. It is a no-reference metric as no information 20 regarding the original LF is required to assess the distorted LF. This was possible as the authors trained a support vector regression model on distorted LFs and corresponding subjective scores. Again, the model was trained on dense LFs and hence applicable to only densely sampled LFs. Min et al. measure global spatial quality, local spatial quality, and angular quality separately and combines them for an overall LF quality metric [46]. The authors have reduced the LF to 3D hence their metric is suitable for LFs with horizontal parallax. Other recent works in LF quality metric domain such as FR LFI-QA [47] measures the gradient magnitude similarity between the reference and processed content. The accuracy of the metric depends on the granularity of the sampling of the gradient magnitude. Another quality metric called SDFM [48] extracts symmetry and depth features of LF for a full reference model which agrees with the human visual system (HVS). This model would be ideal when the LF is displayed to human users as the properties of HVS are Method Representation used Category Features explored Shan at el. [43] Individual SAIs No reference Visual, texture, depth Paudyal at el. [44] Stereo pairs Reduced reference Depth map distortion Fang at el. [47] EPIs Full reference Spatial, angular (Gradient magnitude) Shi at el. [25] Stereo pairs, EPIs No reference Cyclopean naturalness, gradient direction distribution LF-ASC [45] SAIs in raster order No reference Spatial, angular Min at el. [46] Individual SAIs Full reference Spatial (global and local), angular SDFM [48] SAIs and EPIs Full reference Symmetry, depth Table 1.1 Representations used in literature for LF quality assessment 21 preserved. However, the LFs produced for machine consumption (3D modeling, refocusing) might require different assessments. Table 1.1 presents a summary of the recent LF quality metrics and their noteworthy properties. One noteworthy difference between LF specific quality metrics discussed above and traditional image quality metrics is that most of the LF quality metrics consider spatial and angular features as the dominant quality defining features to accurately describe the LF quality in contrast to just pixel intensities. This is highly desirable in LF quality metrics. It is worth mentioning that all of these quality metrics are catered to densely sampled LFs, whereas LFs are generally sparsely sampled for ease of transmission and memory efficiency. Therefore, it is evident that there is a need for an LF quality metric that is suitable for sparsely sampled LFs making our proposed method the first LF quality metric specifically designed for sparsely sampled LFs. Furthermore, the recent quality metrics assess parts of the LF at a time and aggregate the results, therefore, do not consider the quality of the LF as a whole. In this paper, a full reference LF quality assessment technique is presented. The main contribution of the proposed quality metric is the usage of the cross-section of the novel volumetric LF representation. Specifically, it enables the feature extractor to look at the LF as a whole rather than in parts. Therefore, the model can extract both spatial angular features simultaneously. The rest of this article is arranged as follows. In the “Proposed LF Quality Metric” section we present our proposed deep feature consistent sparse (DFCS) LF quality assessment model. The “Performance Evaluation” section compares our proposed method with classical and recent LF quality metrics. We conclude our paper in the “Conclusion” section. 22 1.5 Thesis Outline In this thesis, we propose novel light field compression techniques capable of generating highly compressed, low random-access complexity LFs. We also design human perception consistent LF refocusing for sparse LF content. Finally, we propose a novel quality metric for comprehensively assessing the quality of processed LFs. The focus of the work is highlighted with respect to the end-to-end LF pipeline in Figure 1.10. Figure 1.10 Scope of the thesis with respect to general LF pipeline In Chapter 2, we propose a LF compression technique that takes advantage of the high efficiency video coding (HEVC) standard and compresses the content either in a compression efficient or random-access efficient manner based on application requirement. To maximize compression efficiency, we compress views using other views that has high similarity with it rather than following a fixed structure. In Chapter 3, we propose sparse LF refocusing technique that takes how human visual system focuses on observable world. This produces more visually alluring and authentic refocused content than was possible before. 23 In Chapter 4, we propose an LF quality metric that considers the low to high level features of the entire LF ray space of the original and processed LF to assess the quality.24 Chapter 2: Light Field Video Compression To address the need for an efficient compression technique for LF video, we propose a new and efficient LF video compression technique named Universal Prediction Structure (UPS), which unlike the existing techniques takes full advantage of the redundancies among all the surrounding LF views and uses them to generate a coding order and a prediction structure that lead to an improved overall compression. Major contributions of this work include: A novel SSIM based P and B-frame selection strategy that prioritize reference adjacency over B-frame maximization, significantly reducing the motion compensation residual and improving compression efficiency. A novel hierarchical B-frame prediction structure, which ensures that every bidirectional frame leverage both horizontal and vertical correlation, also improving compression. An efficient diagonal reference selection strategy for random-access efficient predictions structure. We evaluated our approach against four existing PSB prediction structures and coding orders on microlens array-based, camera array-based and synthetic LF video sequences. Performance evaluations show that our compression efficient method outperforms the best state-of-the-art method for microlens array-based content by 17%, and the best state-of-the-art method for synthetic and camera array content by 11% and 26% BD-rate (Bjøntegaard delta bitrate), respectively. The random-access efficient structure reduces access complexity by 20%. Moreover, the proposed methods have proven to better adapt to large number of views, different spatial view arrangement, and offer the best trade-off between BD-rate gain and computational complexity compared to any other method. 25 The remainder of this chapter is organized as follows. In Section 2.1, the proposed light field video compression system is described. Section 2.2 discusses the performance evaluation and the results, and Section 2.3 concludes the chapter. 2.1 Proposed LF Video Compression Methods Our main objective is a design a PSB prediction structures and coding orders that is based on computed similarities among the views in the LF sequence. Therefore, each view is predicted from views that are most similar to the view(s) being coded; hence, efficient compression is achieved. Given the matrix 𝕊, of subaperture videos/views of an LF video 𝕃 of 𝑚 × 𝑛 views, we derive the inter-view prediction structure for all the views. Here, 𝕊 = [𝑠1,1 ⋯ 𝑠1,𝑛⋮ ⋱ ⋮𝑠𝑚,1 ⋯ 𝑠𝑚,𝑛]. Each element 𝑠𝑖,𝑗 of the matrix represents a subaperture video/view here, 𝑖 = 1, … , 𝑚, 𝑗 = 1, … , 𝑛. Each view 𝑠𝑖,𝑗 , is assigned a reference list 𝑅𝑖,𝑗, that contains the view(s) from which 𝑠𝑖,𝑗 is predicted. The inter-view prediction structure ℝ is composed with the reference lists of all the views in 𝕃. We also determine the coding order ℂ of the views. That is, which video should be encoded first, which should be encoded second and so on. 2.1.1 UPS method Each view in an LF video is compressed either as a keyframe (I or P-frames) or a bi-directional (B) frame. Our strategy selects if a view should be an I, P or B-frame and generates its reference list. This choice of I, P and B-frames significantly affects the compression we achieve. We have calculated the average similarity of adjacent horizontal, vertical, diagonal views and the average similarity of non-adjacent (distant) views for every view in several LF video test 26 sequences. Properties of these test sequences are presented in Table 2.1 and the averaged similarity results over all frames are presented in Table 2.2. We observe that any adjacent (horizontal/ vertical) view is at least 2% and at most 10% more similar than diagonal or non-adjacent views. We also discover that horizontal views have more similarities among them than the vertical views have. This is particularly important for microlens array-based content, as for camera array-based content is dependent on the baseline among the cameras. We conclude that due to the intricate nature of LF video, choosing adjacent horizontal or vertical views as references will improve overall compression efficiency. Our next step is to develop a strategy for choosing IBP frames that take advantage of this characteristic of LF content. Typically, B-frames are highly compressed compared to I and P- Test sequence Capture device Number of views Baseline Frames, fps Resolution ChessPieces R8 Raytrix 5×5 4 mm 300, 30 1920×1080 Boxer-IrishMan-Gladiador R8 Raytrix 5×5 4 mm 300, 25 1920×1080 Car Lytro Illum 8×7 1 mm 100, 25 512×352 Toy Lytro Illum 7×8 1 mm 100, 25 480×320 Temple1 Blender 3×3 Pixel disparity [-22, +9] 50, 20 1024×432 Bamboo2 Blender 3×3 Pixel disparity [-8, +52] 50, 20 1024×432 TableSoccer GoPro Hero3+ Black Edition 3×5 y:x is 0.625 (cameras 60-100 mm apart) 250, 25 1920×1080 Newspresenter GoPro Hero3+ Black Edition 4×4 y:x is 0.6 (cameras 60-100 mm apart) 150, 25 1920×1080 Painter CMOSIS (CMV2000) 4×4 70 mm 300, 30 2048×1080 Birthday CMOSIS (CMV2000) 4×4 70 mm 300, 30 2048×1080 Table 2.1 Properties of the test sequences 27 frames, thus B-frame maximization will achieve higher compression efficiency. Though Avramelos et al. [22] also follow the B-frame maximization strategy, the uniqueness of our method lies in the priority we impose on the reference adjacency constraint over B-frame maximization. This diminishes the need for diagonal references and non-adjacent references. This prioritization of reference adjacency enables us to place P-frames adjacent to each other and take full advantage of high spatial similarity, which is novel to our method. Our similarity computation among the views of LF videos derives that a prediction structure with adjacent references is compression efficient for LF videos. In literature, Dricot et al. [49] have used adjacent frames for super multi-view content compression. To the best of our knowledge, this is the first work in LF video compression that adopts a similarity-based reference adjacency constraint for both P and B-frame prediction to enhance compression efficiency. While generating UPS we follow two constraints: Reference Adjacency: Any view should only be predicted from vertically and horizontally adjacent views. LF Test Sequence Horizontal Adjacent Vertical Adjacent Diagonal Adjacent Non-adjacent ChessPieces 97.7% 96.2% 95.1% 91.7% Boxer-IrishMan-Gladiador 97.1% 95.4% 93.8% 89.5% Car 98.2% 98.01% 95.02% 91.8% Toy 97.9% 97% 94.8% 90.5% Temple1 78.4% 71.4% 69.7% - Bamboo2 56.8% 62.5% 41.7% - TableSoccer 61.7% 62.6% 50.8% 52.2% News presenter 87.5% 88.9% 86.2% 84.2% Painter 64.8% 58.2% 43.6% 46.1% Birthday 62.3% 61.8% 51.02% 54.5% Table 2.2 Similarity among the views according to their positions 28 B-frame Maximization: Maximize the number of bi-directional frames. In order to satisfy these two constraints, first, we select the most compression efficient I-frame. Next, we repeatedly place P-frames satisfying the reference adjacency constraint, until a view with already encoded horizontal and vertical adjacent view is found. Finally, that view is predicted as a B-frame. We discuss the IBP frame selection strategies below: 2.1.1.1 I-frame Selection Overview To select the I-frame, we calculate the SSIM values among all the sub-aperture videos and select the view with highest average SSIM value as the I-frame, as it has the highest similarity with all the other frames and hence better prediction. For example, we compute similarity of each view 𝑠𝑖,𝑗, with all the other (𝑚 × 𝑛) − 1 views. This way, we can find which view has the most similarity with all the other views and we select that view as our I-frame. This is mainly due to the bitrate savings we will achieve if we choose such an I-frame. For example, P-frames will be predicted from an I-frame, so if we choose an I-frame that has high similarity with all the other views, it will surely have high similarity with the would-be P-frames, so the resultant P-frames will require a lower bitrate. For selecting I-frame, we first calculate the average SSIM ?̂? for every view with all the other views in the LF video: ?̂? = ∑ ∑∑ ∑ 𝑆𝑆𝐼𝑀 (𝑠𝑖,𝑗,𝑠𝑝,𝑞)𝑛𝑞=0𝑚𝑝=0𝑚×𝑛𝑛𝑗=0𝑚𝑖=0 (2.1) The function 𝑆𝑆𝐼𝑀 returns the average SSIM of view 𝑠𝑖,𝑗 with all the other views. The view with the maximum value in ?̂? is selected as the I-frame for our UPS method: 𝐼𝑓𝑟𝑎𝑚𝑒 = argmax𝑖,𝑗 ?̂? (2.2) Not surprisingly, for all the LF video content we tested, such a view is found around the center. For example, we present ?̂? for the test sequence “ChessPieces” in Figure 2.1. From the SSIM 29 values, we see that the center view 𝑠3,3 should be chosen as the I-frame. Since it has the highest average SSIM value. We present the prediction structure of “ChessPieces” in Figure 2.2(a). Similarity computation becomes essential when the view arrangement in the LF video is anything other than odd×odd. For example, for an even×even structure there are four central views which are all candidates for I-frame. Our method chooses the one with the highest average similarity as the I-frame. Similar situations occur when view arrangement is either odd×even or even×odd. The reference list of the I-frame is empty as it does not need any other views for encoding. 2.1.1.2 P-frame Prediction Structure P-frames are predicted from other keyframes (I and P-frames) and are bitrate expensive compared to B-frames. However, at the initial state of the prediction structure we cannot satisfy our reference adjacency rule, as there are not enough (at least 2) vertically or horizontally adjacent references for B-frames. Therefore, we place P-frames until a B-frame could be placed. The P-frame placement follows the reference adjacency rule. This is a major contribution of our work, as most existing work prioritizes B-frame maximization over reference adjacency; however, we show that reference adjacency in LF is more important for compression efficiency than maximizing the number of B-frames. (a) Average SSIM of “ChessPieces” (5×5) (b) SSIM values with respect to the I-frame Figure 2.1 I-frame selection using similarity computation 30 Placing P-frames right next to an I-frame or another P-frame might seem unconventional. In the general case of 2D video sequences, P-frames are placed further from an I-frame to benefit from the bidirectional residual option when the keyframe will predict an in-between B-frame. This is because the differences between frames are solely dependent on moving objects or camera movement. This is not true, however, for the case of the LF structure, where the frame sequence involves different camera views and not just temporally moving objects. This situation can be seen in the experimental pseudo sequence presented in Figure 2.2(b). For the “ChessPieces” LF video test sequence, our compression experiment showed that the encoder indeed assigns a lot of bits (700kbps per P-frame on average) to 4 P-frames which is further away (raster scan order views 3, 11, 15 and 23) and saves a lot of bits (72kps per B-frame on average) while predicting the 4 B-frames in between the I-frame and the P-frames (raster scan order views 8, 12, 14, and 18). As a result, the total bitrate of this LF video is 640kbps. However, when these P-frames are placed adjacent to other keyframes with higher similarity, as it is in our proposed structure, they tend to be much smaller in size. For example, the 8 P-frames of Figure 2.2(a) (raster scan order 3, 8, 11, 12, 14, 15, 18, and 23) yields 312kps on average. This is less than half the size of the experimental (a) Proposed prediction structure UPS (b) Experimental pseudo sequence to validate P-frame reference adjacency Figure 2.2 Proposed and experimental PSB prediction structure for "ChessPieces" LF dataset IPPPPB BB BBBBBB BBBBBBBPPPP IPPPPB BB BBBBBB BBBBBBBBBBB31 pseudo sequence for the same 8 views. Therefore, for LF content, our reference adjacency constraint outweighs the benefits of references jumping back and forth. Furthermore, the numerical results show, our proposed UPS method, when compared against the experimental pseudo sequence of Figure 2.2(b), improves compression efficiency by 3.5%, 3.56% and 7% for the microlens, synthetic and camera array test sequences, respectively. Our P-frame selection is performed as follows. We find the yet to be predicted adjacent view with highest similarity to an already predicted keyframe and make it a P-frame. Again, this similarity is measured using SSIM. For example, in Figure 2.1(b), we present the similarity (SSIM) of all other views with I-frame 𝑠3,3 (initially the only keyframe). The most similar adjacent view (view 𝑠3,2in this case) is chosen as a P-frame. This is a novelty of this work. To the best of our knowledge, no other method has placed P-frames based on their similarities with other keyframes. Our technique continues to place P-frames until we can place a B-frame satisfying both B-frame maximization and reference adjacency constraints. After exhausting the adjacent views of the I-frame we continue placing P-frames adjacent to existing P-frames maintaining the reference adjacency rule if needed. The reference list of a P-frame contains the view(s) it was predicted from. If the P-frame 𝑠𝑖,𝑗, is predicted from an I-frame, the reference list only consists of the I-frame, 𝑅𝑖,𝑗 = {𝐼 − 𝑓𝑟𝑎𝑚𝑒}. Otherwise, it consists of the adjacent P-frame it was predicted from and inherits its reference list. For example, in Figure 2.2(a), view 𝑠2,3, will have I-frame (view 𝑠3,3) as a reference picture. Therefore, 𝑅2,3 = {𝑠3,3}. However, view 𝑠1,3, will have the view 𝑠2,3 (a P frame) as a reference and inherit its reference list which consists of view 𝑠3,3. Therefore, 𝑅1,3 = {𝑠2,3, 𝑠3,3} 32 2.1.1.3 B-frame Prediction Structure B-frames are the most compression efficient frames. Traditionally, they are predicted from previous and future frames. However, due to the unique nature of LF content discussed above, an efficient prediction order should take benefit from both horizontally and vertically correlated frames. For B-frame maximization, whenever we encounter a yet to be predicted view that has both the horizontal and vertical adjacent views already predicted, we categorize it as a B-frame. We include the horizontal and vertical adjacent views in the reference list and inherit their reference lists. In other words, for view 𝑠𝑖,𝑗, if there already exists predicted views 𝑎𝑑𝑗ℎ(𝑠𝑖,𝑗) and 𝑎𝑑𝑗𝑣(𝑠𝑖,𝑗) where 𝑎𝑑𝑗ℎ(𝑠𝑖,𝑗) is 𝑠𝑖+1,𝑗 or 𝑠𝑖−1,𝑗 and 𝑎𝑑𝑗𝑣(𝑠𝑖,𝑗) is 𝑠𝑖,𝑗+1or 𝑠𝑖,𝑗−1 then the list of references is: 𝑅𝑖,𝑗 = {𝑎𝑑𝑗𝑣(𝑠𝑖,𝑗), 𝑎𝑑𝑗ℎ(𝑠𝑖,𝑗)} ∪ 𝑅𝑎𝑑𝑗𝑣(𝑠𝑖,𝑗) ∪ 𝑅𝑎𝑑𝑗ℎ(𝑠𝑖,𝑗) (2.3) For example, in Figure 2.2(a), the B-frame corresponding to view 𝑠2,2 is predicted from two P-frames corresponding to views 𝑠2,3 and 𝑠3,2, so the reference list for view 𝑠2,2 would be 𝑅2,2 ={𝑠2,3, 𝑠3,2,∪ 𝑅2,3 ∪ 𝑅3,2}. Both views 𝑠2,3 and 𝑠3,2’s reference lists consist of view 𝑠3,3 (I-frame) only. Therefore, 𝑅2,2 = {𝑠2,3, 𝑠3,2, 𝑠3,3}. Similarly, view 𝑠2,1 (another B-frame) is predicted from a P and a B-frame, views 𝑠3,1 and 𝑠2,2, respectively, and its reference list would be 𝑅2,1 ={𝑠3,1, 𝑠2,2, 𝑠2,3, 𝑠3,2, 𝑠3,3 }. We go back and forth between B-frame placement and P-frame placement until all the views have their respective prediction structures. 2.1.2 Random-access Efficient UPS (RAE-UPS) method UPS is designed for maximum compression efficiency however, depending on the application requirements, random-access efficiency might be more desirable than compression efficiency. 33 Therefore, we also propose RAE-UPS a random-access efficient universal prediction structure and coding order. We define random access complexity as the number of frames needed to be decoded to get the view of interest [50]. For example, if we need to access 4 frames to get to a certain view, the random access cost of that view is 4. We first extract the 25 views from light field content using the method presented in [35]. We arrange the views in a 2D hierarchical structure (5×5) according to their viewpoint or microlens position in the camera. Fig. 4 illustrates our proposed pseudo-sequence-based prediction structure. It inherits the I selection from UPS with modifications to the P-frame and B-frame prediction structure. Once we choose the central I-frame view, the views furthest from the I-frame on the center axes are predicted as P-frames. These are lower level frames (level 1, random access cost: 1), as only the I-frame is required to decode them. Consequently, they incur low random access complexity. The remaining views on the center axes come next in the coding order at level 2 and they are predicted as B-frames from the I-frame and P-frames nearest to them. This allows the prediction algorithm to exploit the high vertical and horizontal correlation for compression efficiency and also use only lower level frames (level 0 and 1) that yield low random- access cost of 2. For coding corner views, which are the furthest ones from the center view, the nearest P-frames are used for prediction to guarantee the best trade-off between random access and compression efficiency. Choosing any other pair of already coded frames as reference would increase random access complexity. This proposed prediction structure results in random access cost of 3 (they are at level 3) for corner views. Then, the remaining views on the horizontal outer boundary are predicted from the horizontal neighboring views and the vertical outer boundary views are predicted from the vertical neighbors. The random access cost for these views is 4. The P-frame place strategy is quite similar to Khoury’s method. However, for the 34 remaining center views of the four quadrants, unlike Khourys’ work that uses highly correlated vertical or horizontal neighboring views to enhance compression efficiency, we propose to use the less correlated diagonal neighboring views from the lower level of prediction hierarchy to achieve better random access complexity. This is the most aggressive approach for reducing random access complexity as it may impact compression efficiency. More specifically we propose to use the lowest level frame (I-frame) for all the four views and use a P-frame from the view’s neighborhood. The P-frames are chosen arbitrarily. The proposed prediction structure results in random access cost of 2 compared to 6 of Khoury’s method. Next, we expanded this idea to pick a computationally efficient P-frame instead of the arbitrary selection we made. In cases where there are more than two possible references, we seek out the two having higher SSIM with the view being predicted. For example, in Figure 2.3(a) views 7, 9, 17 and 19 have this choice to make. After this, we determine the prediction structure for the B-frames using horizontal, vertical and diagonal references from the already predicted I and P-frames. Although our calculations have shown that views are usually more similar to their (a) (b) (c) Figure 2.3 Proposed pseudo-sequence-based prediction structure utilizing diagonal references IPPPPB BB BBBBBB BBBBBBBPPPP35 horizontal and vertical neighbors, we utilize diagonal reference for better random-access efficiency. For the rest of the views, we use only horizontal and vertical references from already predicted views. This is done for improving compression efficiency, since diagonal references will yield higher bitrate. With this configuration the resultant pseudo sequence would look like Figure 2.3(b). Though this results in excellent random-access efficiency it is harsh on compression. To obtain a better trade-off between the two, we relax the reference adjacency constraint for the B-frames in our proposed UPS method and include diagonally adjacent views as candidate references. Now all horizontally, vertically, and diagonally adjacent already predicted views are candidate references. First, we choose the diagonally adjacent view with least random-access complexity (shortest reference list) as our first reference. Then we choose the most similar horizontal or vertical neighbor as the second reference. This is another major contribution of this work. As no other methods before has jointly considered compression efficiency by view similarity and random-access complexity with shortest reference list while choosing references. It is worth noting here is that the view being predicted does not lose all its horizontal or vertical correlation as one of the two references are chosen based on similarity. The RAE-UPS is illustrated in Figure 2.3 (c). For a view 𝑠𝑖,𝑗 the prediction structure chooses 𝑎𝑑𝑗𝑑(𝑠𝑖,𝑗) an already predicted immediate diagonal view with smallest number of references 𝑚𝑖𝑛 (|𝑅𝑎𝑑𝑗𝑑(𝑠𝑖,𝑗)|) where, 𝑎𝑑𝑗𝑑(𝑠𝑖,𝑗) = {𝑠𝑖+1,𝑗+1,𝑠𝑖+1,𝑗−1, 𝑠𝑖−1,𝑗+1, 𝑠𝑖−1,𝑗−1} as one reference picture and the other reference 𝑎𝑑𝑗ℎ𝑣(𝑠𝑖,𝑗) is either the horizontal or the vertical reference. This is chosen based on the SSIM similarity between the frame being predicted and its vertical and horizontal candidate references during similarity pre-calculation. The change in the reference list can be accommodated as in eq. 2.4 𝑎𝑑𝑗ℎ𝑣(𝑠𝑖,𝑗) = argmax (SSIM (𝑎𝑑𝑗𝑣(𝑠𝑖,𝑗), 𝑠𝑖,𝑗) , SSIM (𝑎𝑑𝑗ℎ(𝑠𝑖,𝑗), 𝑠𝑖,𝑗)) 36 𝑅𝑖,𝑗 = {𝑎𝑑𝑗𝑑(𝑠𝑖,𝑗), 𝑎𝑑𝑗ℎ𝑣 (𝑠𝑖,𝑗)} ∪ 𝑅𝑎𝑑𝑗𝑑(𝑠𝑖,𝑗)∪ 𝑅𝑎𝑑𝑗ℎ𝑣 (𝑠𝑖,𝑗) (2.4) 2.1.3 Coding Order We generate the coding order ℂ by scanning the reference lists of the views. The coding order ℂ determines in which order the views of the LF video will be encoded. I-frame must be coded first to facilitate the coding of other frames. Therefore, ℂ0 = 𝐼 − 𝑓𝑟𝑎𝑚𝑒. Let 𝑝 and 𝑞 be indices of ℂ. For any view 𝑠𝑖,𝑗 at ℂ𝑝, all the views in its reference list 𝑅𝑖,𝑗 should be at ℂ𝑞, where 𝑞 < 𝑝. In other words, all the reference videos should be coded before coding the video itself. The coding order is implemented by maintaining two lists of views. The former contains the coding order ℂ and the latter contains the views not in the coding order yet. Initially, ℂ only contains the I-frame, while the other list contains the rest of the views. We traverse the second list and peek at the reference list of each view 𝑠𝑖,𝑗, to see if all the views in the reference list 𝑅𝑖,𝑗 are in ℂ or not. If all the references are in ℂ, then we move the view 𝑠𝑖,𝑗 to ℂ. P-frames with only I-frame in their reference lists will be transferred to ℂ after the first traversal. We keep on traversing the second list until it is empty. Once it is empty, we have our coding order ℂ. As the coding order affects the decoding buffer size, it directly impacts the decoding time of the proposed UPS method. We evaluate the execution time of our proposed UPS method in section IV.A.2. The proposed UPS (a) Even by even (4×4) (b) Odd by even (5×6) Figure 2.4 Scalability to different view arrangements 37 prediction structure and coding order is summarized in the Algorithm 1 below for the reproducibility of this work. 2.1.4 Scalability to View Arrangement One of the major concerns in pseudo sequence-based LF video compression is its scalability to the number of views. Due to the simplicity and adaptability of our prediction structure and coding order, our approach is scalable to any number of views. To this end, we have illustrated our UPS and RAW-UPS only for 𝑛 × 𝑛 views where 𝑛 is odd. In Figure 2.4 we present other variations such as 𝑚 × 𝑚, where 𝑚 is even and 𝑛 × 𝑚, where 𝑛 is odd and 𝑚 is even. Figure 2.4(a) depicts UPS for an LF video with 4 × 4 views and Figure 2.4(b) represents the structure for 5 × 6 views. These structures could be easily extended to any number of views. Unlike other methods such as LF-MVC where the random-access along with encoding 38 and decoding time increase drastically; the computational complexity of UPS increases gradually as the distance from I-frame increases. Furthermore, the structure remains consistent compared to the other prediction structures, when dealing with view arrangements other than odd×odd. For example, LF-MVC losses its repeated pattern of 3×3 substructure of closed loops. RAE-UPS can be extended similarly. 2.2 Performance Evaluation and Discussion We evaluated the compression efficiency, computational complexity and random-access efficiency of our proposed UPS and RAE-UPS against four state-of-the-art approaches. Namely, LF-MVC, Khoury’s method, Full-scheme and SRAE. Details of the performance measures are discussed in the following subsections. The performance of different pseudo-sequences is evaluated on ten different test sequences (see properties in Table 2.1). Two of them [51] were captured using microlens of Raytrix 8 [21] (Figure 2.5(a) ChessPieces and Figure 2.5 (b) Boxer-IrishMan-Gladiator) the next two (Figure 2.5(c) Car and Figure 2.5(d) Toy) [52] were captured using Lytro Illum. The next two LF videos, Figure 2.5 (e) Temple1 and Figure 2.5(f) Bamboo2, were synthetic sequences [53] and the last four sequences, Figure 2.5(g) TableSoccer and Figure 2.5(h) Newspresenter, [54], Figure 2.5(i) Painter and Figure 2.5(j) Birthday [36] were captured using camera arrays. These represent a wide variety of capture medium, number of views, and baseline. The HEVC reference encoder software HTM 16.2 is used in our evaluations. The encoding profile used is multi-view main with GOP size 4. The intra period is set to 32 for 30 fps and 24 for 20 and 25 fps. The quantization parameter (QP) is varied from 25 to 45 at every 5 intervals. Input and internal bit depth are both 8 bits (chroma and luma), the coding unit size used is 64, lossy cost 39 function has been used and internal chroma format for all layers is 4:2:0. As we do not modify any part of the reference software to accommodate our proposed prediction structure and coding order, our method can be seamlessly integrated as a preprocessing step with the newest video 40 compression standard VVC (Versatile Video Coding) [55] which was finalized in July 2020. The PC used had a 2.1GHz Intel Broadwell CPU, 32GB RAM, with Linux Operating system, and 99.4% of CPU was utilized with 30-70% memory efficiency. (a) ChessPieces (b) Boxer-IrishMan-Gladiator (c) Car (d) Toy (e) Temple1 (f) Bamboo2 (g) TableSoccer (h) Newspresenter (i) Painter (j) Birthday Figure 2.5 First frame of the test sequences (central/ I-frame) 41 2.2.1 Compression Performance of UPS and RAE-UPS We measure BD-rate, BD-PSNR (Bjøntegaard delta Peak Signal to Noise Ratio) [56], [57], and SSIM (Structural Similarity Index) [58] to compare compression performance. The controlled parameter is the quality of the videos. The average YUV-PSNR and bitrate of all the views in an LF video sequence for different QPs are used to calculate the BD-rate and BD-PSNR. We calculated SSIM between the original and the reconstructed video and then averaged the results for QPs 25, 30 and 35. We chose the three lower QPs as they do not present noticeable visual distortion in the decoded videos of any of the methods. Recall that SSIM produces a value between 0 and 1, where 1 represents perfect perceptual quality compared to ground truth. 2.2.1.1 Microlens LF Video sequences Figure 2.6 (a)-(d) show the rate-distortion (RD) performance of our UPS and RAE-UPS methods against the existing methods for four microlens array-based test sequences. From these curves, it is evident that our proposed UPS (solid blue line) performs better than all the other methods. It is interesting to note that the performance of our method improves as the visual quality increases. Compared to Khoury’s method, the overall BD-rate gain is approximately 17%, while for higher quality videos (QP 25) we achieve an improvement of 22-24%. The visual quality is improved by at least 0.5db for all four test sequences. Compared to LF-MVC, Full-scheme and SRAE we witness an overall 50%, 44.27% and 20% BD-rate reduction or 1.26 dB, 1.23dB and 0.46dB improvement in BD-PSNR respectively. As for RAE-UPS, we observe that the compression improves over LF-MVC by approximately 40%, Full-scheme by 33%, SRAE method by 3.84% and Khoury’s method by 2.11%, and the bitrate is increased by 15% compared to our original UPS method. The details of BD-rate, BD-PSNR and SSIM are presented in Table 2.3 - Table 2.5. using 42 LF-MVC as the anchor. It should be mentioned that for higher number of views (Car 8×7 and Toy 7×8) UPS’s BD-rate gains are higher compared to others. That makes UPS the most appropriate prediction structure for large number of views. The reason behind this significant performance improvement of UPS is the utilization of the similarities among the views. Due to the small baseline, microlens array contents demonstrate high similarity among the views, which is even higher among the vertically and horizontally adjacent views. Our prediction structure leverages this feature. We observe an overall BD-rate gain of 47.8% and 44.32% for “ChessPieces” and “BoxerIrishMan-Gladiator” respectively against the Full-scheme. The main differences between the Full-scheme and the proposed UPS method are the number of P-frames and their position. This validates our prioritization of the reference adjacency constraint over B-frame maximization. The Full-scheme only employs one P-frame placed furthest from the I-frame. This makes an extremely large P-frame. B-frames are predicted from distant frames with a mixture of horizontal, vertical and diagonal references making them larger than they would be in UPS. The increased number of P-frames in UPS does not adversely affect compression, as these P-frames end up being represented by a significantly smaller number of bits due to the high similarity with their references. 2.2.1.2 Synthetic LF video sequences For the synthetic test sequences UPS ends up being the same as Khoury’s method. Note that the uniqueness of UPS only comes into play if the view arrangement is greater than 3×3 and here the view arrangement is exactly 3×3. For this reason, we compare our methods with LF-MVC, Full-scheme and SRAE. The rate-distortion curves for these sequences are presented in Figure 2.6(e)-(f) and the summary is in Table 2.3 - Table 2.5. 43 (a) ChessPieces 5×5 (b) Boxer-IrishMan-Gladiator 5×5 (c) Car 8×7 (d) Toy 7×8 44 We observe that the overall BD-rate gains for our methods are higher in all cases. Evaluation results show that UPS performs on average 43%, 18% and 35% better than LF-MVC, Full-scheme and SRAE respectively. The BD-PSNR gains are 1.5 dB, 0.58 and 0.4 dB, respectively. RAE-UPS shows significant compression gains over LF-MVC (34%), Full-Scheme (5%) and SRAE (10%). (e) Bamboo2 3×3 (f) Temple1 3×3 (g) TableSoccer 3×5 (h) Newspresenter 4×4 (i) Painter 4×4 (j) Birthday 4×4 Figure 2.6 Rate distortion curves for the LF video test sequences 45 Although SRAE has performed better than Full-scheme for microlens based test sequences, here it lags behind for a 3×3 synthetic LF video. The reason behind this behavior is the ratio of P and B-frames. P-frames are typically larger in size compared to B-frames. The Full-scheme places only one P frame (1:9) whereas SRAE places four such frames (4:9), and as a result the presence of the additional P frames increases the bitrate. Full-scheme is performing better by maximizing B-frames and by placing only one P-frame. SRAE not only places four such P-frames for a 3×3 LF video it makes the situation even worse by using more frequent diagonal reference frames. However, for larger view arrangement as in “ChessPieces”, the Full-scheme suffers the lack of a sufficient number of P-frames by creating large B-frames. Although the proposed UPS method also places four P-frames it compensates for them by producing B-frames with extremely low bitrates. These B-frames are so small in size because of the reference adjacency constraint. SSIM values presented in Table 2.3 - Table 2.5 show the proposed method achieves better performance than all the others; indicating that our method not only picks the best references for lower bitrate but also for better perceptual quality. Overall, for synthetic data and a smaller number of views, our methods performs better than the existing methods. 46 2.2.1.3 Camera array LF video sequences The compression performance for the camera array-based sequences is presented in Figure 2.6(g)-(j). We observe that UPS achieves average BD-rate gains of 51%, 29%, 38% and 66% over LF-MVC, Khoury’s, SRAE and Full-scheme. RAE-UPS consistently outperforms these methods by 30%, 7%, 10% and 41% respectively. Note that even with higher resolution (2048×1080) test sequences such as “Painter” and “Birthday”, UPS shows consistent performance. On the other hand, the performance of Khoury’s method drops as the resolution increases. The UPS method attains the best performance against the Full-scheme with at least 66% BD-rate gain. Objective SSIM evaluation shows the proposed UPS method has at least 2% better perceptual quality than Test Sequence BD-rate UPS (Proposed) RAE-UPS (Proposed) Khoury [22] Full [25] SRAE [24] 5×5 Chess-Pieces -47.39% -42.38% -37.14% 2.71% -38.45% 5×5 Boxer-Irishman-Gladiator -53.06% -46.78% -43.43% -1.10% -44.38% 8×7 Car -54.72% -38.45% -33.49% 5.84% -34.95% 7×8 Toy -57.76% -28.67% -20.69% 3.62% -24.49% 3×3 Temple1 -42.48% -28.56% -42.48% -22.30% -19.18% 3×3 Bamboo2 -46.06% -38.66% -46.06% -35.25% -16.27% 3×5 Table-Soccer -59.84% -36.98% -31.72% 41.76% -27.80% 4×4 News-presenter -51.17% -31.23% -26.19% 42 .55% -20.20% 4×4 Painter -52.45% -29.56% -23.52% 37.37% -22.17% 4×4 Birthday -60.51% -31.44% -25.12% 35.18% -27.54% Average -52.54% -35.27% -32.98% 7.54% -27.54% Table 2.3 BD-rate gains compared to LF-MVC 47 any other method. This confirms that our method is more suitable for LF content with a wider baseline than any other pseudo sequence. Test Sequence BD-PSNR (dB) UPS RAE-UPS Khoury [22] Full [25] SRAE [24] 5×5 Chess-Pieces 1.1 0.95 0.61 -0.05 0.86 5×5 Boxer-Irishman-Gladiator 1.27 1.1 0.75 0.02 1.04 8×7 Car 1.3 0.9 0.61 -0.08 0.83 7×8 Toy 1.42 0.75 0.52 -0.07 0.55 3×3 Temple1 1.86 0.68 1.86 0.45 0.43 3×3 Bamboo2 1.25 0.79 1.25 0.71 0.36 3×5 Table-Soccer 1.72 0.91 0.85 -0.92 0.63 4×4 News-presenter 1.77 0.91 0.75 -0.85 0.45 4×4 Painter 1.78 0.95 0.78 -0.75 0.55 4×4 Birthday 1.8 0.94 0.74 -0.71 0.64 Average 1.53 0.89 0.87 -0.23 0.63 Table 2.4 BD-PSNR gains compared to LF-MVC 48 Due to wider baseline, distant views of camera array content will have higher dissimilarity among each other than microlens array-based content. Therefore, predicting from adjacent reference is even more important. Test sequences “TableSoccer” and “Newspresenter” have shorter baseline vertically, so they experience higher vertical similarity. Therefore, B-frames utilizing vertical references will be bitrate efficient. All the B-frames in UPS have vertical references so, unlike other methods UPS takes full advantage of such situations. It is also worth noting that, for even×even view arrangement performance for other prediction structures drop, however, UPS performs consistently making it more adaptable to different view arrangements. Recall that for Test Sequence SSIM UPS RAE-UPS Khoury [22] Full [25] SRAE [24] LF-MVC [20] 5×5 Chess-Pieces 0.9386 0.938 0.9379 0.9073 0.9379 0.937 5×5 Boxer-Irishman-Gladiator 0.933 0.9329 0.9329 0.9006 0.9327 0.9306 8×7 Car 0.9377 0.9363 0.9363 0.9076 0.9354 0.9354 7×8 Toy 0.9342 0.933 0.9328 0.9086 0.9323 0.9326 3×3 Temple1 0.9476 0.9465 0.9468 0.9404 0.9428 0.9412 3×3 Bamboo2 0.9333 0.918 0.9099 0.9001 0.9061 0.9099 3×5 Table-Soccer 0.8963 0.8954 0.8914 0.8952 0.8957 0.8836 4×4 News-presenter 0.9759 0.9768 0.9768 0.9761 0.9766 0.9734 4×4 Painter 0.9096 0.8956 0.8914 0.8913 0.8957 0.8862 4×4 Birthday 0.91 0.895 0.8908 0.8821 0.9003 0.8904 Average 0.9316 0.9268 0.9247 0.9109 0.9256 0.922 Table 2.5 SSIM compared to reference 49 microlens array content, compared to LF-MVC, the compression gain achieved by UPS was on average 50%. For camera array-based content it is around 55%. In contrast, for Khoury’s method, the compression efficiency decreases from an average of 40% for microlens content to 26% for camera array content when compared to LF-MVC. Similar trends can be seen in SRAE and Full-scheme. The main reason for this relative difference in compression performance is that these methods do not have reference adjacency. As a result, they end up predicting large dissimilarities from reference views. 2.2.1.4 Compression Performance Analysis To understand the effect of our prediction structures on bitrate savings, we closely look at the 7th view (raster scan order) from the “ChessPieces” LF video in Figure 2.7. This view is an excellent (a) Similarity matrix of 7th view (b) UPS (Proposed) 432.71 kbps (c) LF-MVC [20] 523.81 kbps (d) Khoury [22] 478.24 kbps (e) Full-scheme [25] 518.75 kbps (f) SRAE [24] 595.54 kbps (g) RAE-UPS (Proposed) 534.45 kbps Figure 2.7 Different prediction structures and bitrates of 300 frames 7th view of “ChessPieces” LF video 16[0.994]17[0.996]18[0.993]19[0.989]20[0.983]21[0.992]22[0.994]23[0.991]24[0.986]25[0.981]6[0.996]7[1.0]8[0.998]9[0.994]10[0.988]11[0.995]12[0.998]13[0.996]14[0.991]15[0.986]1[0.995]2[0.998]3[0.997]4[0.994]5[0.989]B BB BB B50 candidate for analysis, as it is predicted using different references in all the discussed methods, including the proposed methods. Figure 2.7 (a) shows the average SSIM based similarity matrix of 300 frames of the 7th video; we observe that views 12, 8 and 2 are the most similar videos with view 7. Thus, if used as references, they would produce the smallest residue. Figure 2.7 (b) shows that our UPS method chooses views 8 and 12 as references and compresses the 7th video at 432.71 kbps. However, no other method choses the best references. For example, LF-MVC (Fig. 7(c)) choses only horizontal references, one of which is a lot less similar than other available adjacent views resulting in 523.81 kbps. It is interesting to note that although Khoury’s method (Figure 2.7 (d)) choses two of the highly similar views, it is not the most compression efficient prediction structure (478.24 kbps), as both of its references are vertical and it does not take advantage of horizontal correlation. The Full-scheme (Figure 2.7 (e)) takes one of each (horizontal and vertical), however, the chosen references are not the most compression efficient ones, also increasing the bitrate (518.75 kbps). Although diagonal references reduce random-access complexity, we can see their adverse effect on compression performance for methods SRAE (Figure 2.7 (f)) and RAE-UPS (Figure 2.7 (g)), yielding 595.54 kbps and 534.45 kbps, respectively. For generality purposes, we analyzed all the views of the LF test sequences used in this work and found that 91.2% of the time our UPS method choses the best references. 2.2.2 Computational Complexity of UPS and RAE-UPS Table 2.6 and Table 2.7 gives the execution time of the proposed coding orders and the state-of-the-art methods when QP is 25. Presented results are relative to the encoding and decoding time of the proposed UPS coding order. From this, we see that UPS needs 34% and 48% less execution time compared to LF-MVC and Full-scheme. However, it needs slightly more time to execute compared to Khoury’s (3%) method and SRAE (9%). The proposed RAE-UPS exhibits an overall 51 18.5% better execution time compared to the proposed UPS method, making it the fastest PSB prediction structure and coding order. To this end, our RAE-UPS yields a significant improvement in execution time and compression efficiency over all the existing methods (except for the proposed UPS in terms of compression) and as such, it is highly suitable for applications with fast encoding requirements. The computational complexity depends on how IBP-frames are computed in video compression. A P-frame is computed by subtracting the previous and current frames [26]. On the other hand, B-frames encoded from an I and a P-frame perform 3 such subtractions and a B-frame encoded from two P-frames would need 4 such subtractions. Thus, the encoding time of such B-frames are 3 times and 4 times the P-frame respectively [59]. Depending on the pseudo-sequence, the number of total operations performed varies significantly. For example, to encode the first frames of all the 25 views in “ChessPieces”, we calculated a total of 179 subtractions are performed by LF-MVC, 120 by Khoury, 281 by Full-scheme, 112 by SRAE, 132 by the proposed UPS and 109 by the proposed RAE-UPS method. Therefore, theoretically, RAR-UPS is the fastest and UPS should be faster than LF-MVC and Full-scheme and slower than Khoury’s and SRAE. The experimental results show on average UPS is only 3% slower than Khoury’s and 9% slower than SRAE. It is important to note that, for camera-array based content, our method is comparable to Khoury’s method in terms of speed. Similar trends were observed while decoding the LF videos. The UPS coding order saves on average 36% decoding time against LF-MVC and 43% against Full-scheme. Khoury’s method achieves similar performance as UPS and SRAE has 8% faster decoding than UPS. While RAE-UPS proves to be the fastest with 18% better decoding time than UPS. It is worth mentioning that the encoding speed of UPS is 0.0296fps (5×5 LF “ChessPieces” video with full HD (1920×1080) 52 resolution at QP 25 and using only one CPU) and the decoding speed is 129.3fps. For this video, stream UPS achieves a 52% BD-rate gain over LF-MVC. In comparison, SRAE, the existing fastest method, achieves a 27% BD-rate gain with 0.0326fps over LF-MVC. This indicates that UPS offers the best trade-off between computational complexity and BD-rate. Furthermore, recall that our prediction structure and coding order are preprocessing steps to MV-HEVC encoding, thus the above discussed performance can be significantly enhanced using software and hardware acceleration. For example, Hamid et al. proposed a software decoder implementation specific to MV-HEVC [60] that can decrease the decoding time by 25% and hardware decoders [61] can also speed up the decoding process. For encoding, hardware accelerations can decrease the encoding time by 30% [62]. Test Sequence Relative Encoding Time (other method/UPS) LF-MVC [20] Full [25] Khoury [22] SRAE [24] RAE-UPS (Proposed) ChessPieces 1.28 1.43 0.94 0.87 0.82 Boxer-Irishman-Gladiator 1.28 1.41 0.95 0.89 0.81 Car 1.36 1.6 0.92 0.89 0.78 Toy 1.4 1.6 0.92 0.91 0.77 Temple1 1.1 1.34 1 0.95 0.9 Bamboo2 1.21 1.25 1 0.92 0.9 TableSoccer 1.47 1.61 0.97 0.91 0.82 Newspresenter 1.4 1.44 0.97 0.9 0.81 Painter 1.48 1.54 1 0.9 0.8 Birthday 1.37 1.59 1.01 0.94 0.8 Average 1.34 1.48 0.97 0.91 0.821 Table 2.6 Encoding time relative to proposed UPS 53 Test Sequence Relative Decoding Time (other method/UPS) LF-MVC [20] Full [25] Khoury [22] SRAE [24] RAE-UPS (Proposed) ChessPieces 1.25 1.4 0.95 0.88 0.81 Boxer-Irishman-Gladiator 1.24 1.38 0.95 0.88 0.8 Car 1.53 1.62 0.97 0.92 0.81 Toy 1.58 1.64 0.96 0.91 0.79 Temple1 1.12 1.35 1 0.98 0.88 Bamboo2 1.18 1.33 1 0.97 0.88 TableSoccer 1.5 1.57 0.96 0.93 0.8 Newspresenter 1.41 1.4 0.94 0.92 0.83 Painter 1.5 1.6 0.98 0.92 0.81 Birthday 1.48 1.56 0.98 0.92 0.8 Average 1.36 1.43 0.97 0.92 0.821 Table 2.7 Decoding time relative to proposed UPS 2.2.3 Random-access performance of UPS and RAE-UPS In a PSB prediction structure, the reference list 𝑅𝑖,𝑗 of a view 𝑠𝑖,𝑗 has all the views that are needed to decode 𝑠𝑖,𝑗. Therefore, the spatial random-access complexity (complexity to decode a specific view of the LF video) of a view is the size of its reference list. We calculate the average (average of all the views at a time instance) and the worst possible (the view with the highest random-access at a time instance) random-access complexity of any view as follows: Average complexity, 𝜇𝑠 = 1𝑚×𝑛 ∑ |𝑅𝑖,𝑗|𝑚,𝑛𝑖=1,𝑗=1 (2.5) Worst-case complexity, Μ𝑠 = max𝑖=1,…,𝑚,𝑗=1,…,𝑛|𝑅𝑖,𝑗| (2.6) where 𝑚 × 𝑛 is the view arrangement, and |𝑅𝑖,𝑗| returns the number of views in the reference list 𝑅𝑖,𝑗. We calculated the average and worst-case random-access complexity for the test sequence “ChessPieces” using eq. 2.5 and 2.6 respectively. Our proposed RAE-UPS achieves the best random-access efficiency with mere 2.56 and 5 frames on average and worst-case respectively. 54 This is an impressive 47.9% improvement over LF-MVC, 57.9% improvement over the full method, 20% over Khoury’s and 33.33% over our UPS method. While average complexity is 3.84 frames and the worst-case complexity is 8 frames for UPS. UPS’s performance is better than LF-MVC (average 4.92 and worst-case 11) and full scheme (average 6.08 and worst-case 17). However, Khoury’s method achieves an average of 3.2 and a worst-case complexity of 6 frames. The SRAE demonstrates even better results. Random-access complexity for other sequences can be calculated similarly. The comparative performance is reported in Table 2.8. Some applications might need to access all the spatial views at a time instance rather than only one view. Therefore, we calculate the temporal random-access of the views as well. We categorize the IBP frames according to their random-access complexities. There are 5 types of frames in the spatio-temporal (horizontal, vertical and time) direction I, P, B2, B3 and B4 (B-frames with 2, 3 and 4 references respectively). Please refer to Figure 2.8. A P-frame is computed by subtracting the current and the previous frame. Let’s assume the random-access complexity for such a frame to be µP. For B2, B3 and B4 frames the above process is repeated 2, 3 and 4 times respectively Method Random-access Complexity Bitrate (kbps) Average spatial Worst-case spatial Temporal random-access QP25 ChessPieces LF-MVC [20] 4.92 11 Anchor 738.13 Khoury [22] 3.2 6 -28.5% 691.04 Full -scheme [25] 6.08 17 21.3% 651.25 SRAE [24] 2.72 6 -33.3% 671.42 UPS (proposed) 3.84 8 -20.6% 527.88 RAE-UPS (proposed) 2.56 5 -38.7% 612.90 Table 2.8 Random-access complexities of various methods 55 [59]. Therefore, µB2 ≈ 2 µP, µB3 ≈ 3 µP and µB4 ≈ 4 µP. Hence, the temporal random-access complexity can be derived as: 𝜇𝑡 = 𝑁𝑃𝜇𝑃 + 𝑁𝐵2𝜇𝐵2 + 𝑁𝐵3𝜇𝐵3 + 𝑁𝐵4𝜇𝐵4 (2.7) where 𝑁𝑃 is the number of P-frames, 𝑁𝐵2 is the number of B2-frames etc. For the proposed method UPS, we derive the values for (7) using a 5×5 LF video with GOP size 4 and find 𝑁𝑃 = 16, 𝑁𝐵2 =35, 𝑁𝐵3 = 24 and 𝑁𝐵4 = 48. Therefore, 𝜇𝑡−𝑈𝑃𝑆 = 16𝜇𝑃 + 35𝜇𝐵2 + 24𝜇𝐵3 + 48𝜇𝐵4 = 350𝜇𝑃 Similarly, we can compute the temporal random-access complexity for LF-MVC, 𝜇𝑡−𝐿𝐹−𝑀𝑉𝐶 =441 𝜇𝑃, for Full-scheme, 𝜇𝑡−𝐹𝑢𝑙𝑙−𝑠𝑐ℎ𝑒𝑚𝑒 = 535𝜇𝑃, for Khoury’s method, 𝜇𝑡−𝐾ℎ𝑜𝑢𝑟𝑦 = 315𝜇𝑃 for SRAE, 𝜇𝑡−𝑆𝑅𝐴𝐸 = 294𝜇𝑃 and for RAE-UPS 𝜇𝑡−𝑅𝐴𝐸−𝑈𝑃𝑆 = 270𝜇𝑃. 56 Detailed random-access complexity results are presented in Table 2.8. The numerical results show that the proposed UPS method incurs 20.6% and 34.5% less temporal random-access complexity than LF-MVC and Full-scheme, respectively. However, Khoury’s method and SRAE perform 10% and 16% better than UPS. We observe that RAE-UPS is the most random-access efficient method. It is approximately 38% less complex than LF-MVC. Introducing the diagonal references for B-frames has led to 18% less temporal random-access complexity compared to its predecessor (a) µP = 8, µB2=1 (b) µB2=10, µB3=4 (c) µP =8, µB3=8, µB4=4 (d) µB2=16, µB3=4, µB4=8 (e) µB2=8, µB3=8, µB4=8 (f) µB4=20 (g) µB4=8 Figure 2.8 Temporal random-access complexity: to access all the views at any time instance using the proposed UPS method, for 5×5 LF video GOP-4. The green frames are already encoded views. The black arrows represent current predictions 57 UPS. It beats the best random-access efficient method SRAE by 5%. Therefore, RAE-UPS is the best random-access efficient PSB prediction structure. 2.2.4 Performance Summary The overall performance (compression, computational complexity and random-access) summary is presented in Table 2.9. The fundamental difference between our proposed methods and other existing work is that we create the prediction structures based on computed similarity among the different views of LF videos. While other methods focus on creating maximum number of B-frames, two directional prediction or symmetrical structures. Therefore, our method always chooses reference pictures that create minimum residue compared to other methods while encoding. We observe that the proposed prediction structure and coding order UPS shows significant superiority over other pseudo sequences in terms of the rate-distortion performance. However, the encoding process is more time consuming than a few other. Our RAE-UPS, the random-access efficient variant of UPS, proves to be the fastest in terms of encoding time and randomly accessing frames of the light field video while achieving compression efficiency better than existing ones and very close to that of UPS. Method BD-rate BD-PSNR Computational Complexity Temporal random-access Khoury [22] -32.98% +0.87 0.73 -28.50% Full-scheme [25] +7.54% -0.23 1.11 +21.30% SRAE [24] -27.54% +0.63 0.69 -33.30% UPS (Proposed) -52.54% +1.53 0.75 -20.60% RAE-UPS (Proposed) -36.10% +0.91 0.63 -38.70% Table 2.9 Performance Summary relative to LF-MVC 58 2.3 Conclusion In this paper, we proposed two prediction structures and coding orders that efficiently compress LF video content using the existing HEVC standard and reduce the overall random-access complexity. Our universal prediction structure achieves this by utilizing horizontal and vertical correlation among the views and by better inter-view prediction among the views using adjacent reference placement. Diagonal references are utilized for improved random-access efficiency. As a result, we achieved up to 17% bitrate reduction and 20% random-access efficiency with reasonable computational complexity when compared against the best pseudo-sequence based LF video compression method to date. Our method does not introduce any changes to HEVC; therefore, it could be easily incorporated into the codec as a low-cost preprocessing step. As light field video technology is opening the doors to riveting, rapidly approaching life-like visual reality and promising more realistic AR and VR applications, it is paramount to find efficient ways of compressing and easily accessing the resulting large amounts of data. In this paper, we proposed a new method for optimizing the trade-off between compression and random access efficiency for light field videos. Our method utilizes diagonal references frames at lower level of prediction hierarchy structure, significantly improving the trade-off between compression efficiency and random access complexity when compared to the state-of-the-art LF-MVC and Khoury’s light field video compression methods. 59 Chapter 3: Human Visual System Consistent LF Refocusing In this chapter we propose a human perception based LF refocusing technique capable of producing high quality refocused LFs that is consistent with how we as humans observe the world. We employ best state of the art view synthesis, depth estimation, and a novel depth wise object segmentation technique to achieve HVS consistent refocusing. To overcome the depth dependent refocusing we determine the object of interest in the user defined region and extract the different depth planes of the object. By doing so, we make the refocusing process object dependent rather than being based on the depth of each pixel relative to the capture medium. Figure 3.1 Refocusing when same object spans over multiple depth panes. Left – far, center – middle, and right - near 60 The rest of this chapter is organized as follows: Section 3.1 provide the details of our proposed HVS consistent LF refocusing technique. Section 3.2 presents the performance evaluation. Finally, Section 3.3 concludes this chapter. 3.1 Proposed LF Refocusing Method In this section we define the sparse LF refocusing problem and discuss the three components of our proposed LF refocusing pipeline namely, view synthesis for sparse LFs, fast depth estimation, and the novel object segmentation based refocusing technique. 3.1.1 Sparse LF Refocusing One of the most lucrative features of LF is that it is possible estimate depth from LF more accurately than from 2D or 3D imaging [18]. Leveraging the depth information post shoot refocusing became possible. That is, even after capturing a photo/ video we could adjust the focus later from one region/object of the scene to another. This feature works seamlessly when applied to microlens camera captured LF as the baseline (distance between cameras/lenses) consequently the depth of field is small. However, for sparse camera array based LFs it proved to be more challenging. Researchers have used view synthesis to generate more views to convert the sparse LF into a dense one before refocusing. Depending on the view synthesis technique the results are more or less accurate. However, they do not look natural. For example, in Figure 3.1, if we refocus on the red circled flower of the top image, we achieve results similar to the three images on the bottom row, where the focus is either on the farther, center or near part of the flower. This is because the pixels belonging to the flower spans over three depth planes and each depth plane gives a different refocused image. However, a human observer when looking or focusing on that flower blends all three depth planes and brings the whole flower in focus. In this chapter we propose a technique that makes LF refocusing consistent with human visual perception. 61 3.1.2 Sparse LF View Synthesis Assuming that the cameras in an LF camera array are aligned with equal spacing horizontally and vertically, we investigated the refocusing performance for two different patterns for view synthesis, namely sparse and dense as shown in Figure 3.2. We define 𝑛 ∈ ℤ0+ as the view synthesis factor where 𝑛 = 1 is equivalent to one novel view in between two adjacent views. In the sparse pattern, n represents the number of synthesized views between every nearest pair of horizontal/vertical original views. In the dense pattern, we also synthesize n views between each pair of the horizontal/vertical synthesized views from the sparse pattern. We assume the camera array has an angular resolution of N×N ( 𝑁 > 1). Therefore, the total number of views is 2𝑛𝑁2 −2𝑛𝑁 + 𝑁2 for the sparse pattern and and (𝑛𝑁 − 𝑛 + 𝑁)2 for the dense pattern. The chosen deep learning network [40] leverages a pre-trained fully-convolutional encoder-decoder architecture (modeled after VGG-19 [63]) to synthesize the novel views. This network does not require any camera parameter information, and thus it can be generalized to any LF video Figure 3.2. Examples (n = 3) of the proposed sparse and dense view interpolation patterns 62 dataset. Given two input views captured by two horizontally/vertically aligned cameras and the distance between those two cameras, the network can synthesize as many novel views between the input views as needed. First, we use the network to interpolate views for the sparse pattern, for n =1, 2, and 3, then we use it again to fill the "empty" view locations to achieve the dense pattern. 3.1.2.1 Network Overview The network is a fully convolutional encoder-decoder architecture (see [40] Table 1 for detailed layer specification). The encoder is modeled after a VGG-19 variant [63]. In order to capture fine texture details the decoder consists of deconvolution layers with skip-connections from lower layers. To maintain spatial resolution and larger scene context it also has dilated convolutions [64] in the intermediate layers. Each layer is followed by a ReLU nonlinearity and layer normalization [65] however in the last layer the 𝑡𝑎𝑛ℎ activation function without layer normalization is applied. The system is implemented in TensorFlow [66]. It was trained using the ADAM solver [67] for 600K iterations with learning rate 0.0002, β1 = 0.9, β2 = 0.999, and batch size 1. The network was trained on images having a spatial resolution of 1024 × 576, but the model can be applied to arbitrary resolution at inference time in a fully convolutional manner. To tackle the large number of parameters produced by the fully connected layers the training images were cropped into 256×256 size patches. Training time was approximately one week on a Tesla P100 GPU with 32GB memory. To refocus using the synthesized views, we need to calculate the disparity shifts of the synthetic views. We linearly interpolate these shifts using the disparity shifts of the two adjacent original views. For example, if 𝑛 = 1 that is, we synthesize one view 𝐼𝑢,𝑣 between every two original views 𝐼𝑢−1,𝑣 and 𝐼𝑢+1,𝑣, then we use the disparity shifts of views𝐼𝑢−1,𝑣 and 𝐼𝑢+1,𝑣, to linearly interpolate 63 the disparity shift of the synthesized view 𝐼𝑢,𝑣 . Finally, we apply the shift and sum algorithm using the interpolated and original disparities to produce the refocused video. 3.1.3 Depth Estimation and Object Segmentation At first, we segment individual objects in the video. We used an object segmentation deep learning network Detectron2 [68] on LF videos [36] and segmented the objects. We used COCO Annotator [69] to annotate 1 frame per second for the LF videos and trained Detectron2 using these frames. (a) (b) Figure 3.3 Object segmentation for object base refocusing. (a) original and (b) annotated frame 64 Then, we used the trained network to segment all the other frames of the video. Figure 3.3 (a), (b) show the original frame and the COCO Annotator annotated frame. Figure 3.4 shows the Detectron2 segmented frame. After refocusing, we see that all the objects/pixels of the refocused plane are in-focus rather than the object of interest. We also observe that the refocused plane appears to be a little blurry, especially when it is close to the camera. The reason for this is that the synthesized views are not as accurate as the original views and as a result the average values for the in-focus pixels are not accurate either. To address these problems, we introduce an object segmentation-based enhancement method. First, we segment the object of interest from the reference view using the trained deep learning network. Next, we replace the pixels values of the segmented object at the refocused view with those of the reference view. Figure 3.4 Detectron2 segmented frame 65 3.1.4 LF Refocusing During the refocusing process, one camera will be picked as the reference camera. Then, the user selects any object on the reference view to refocus on. We estimate the depth of that object using Depth Estimation Reference Software (DERS) [70]. Next, we need the disparity shift of all the views from the reference view at the determined depth. The disparity shift for original views can be found in [36]. The synthesized views are calculated by linearly interpolating between the disparity shifts of the two adjacent original views. Finally, we apply the generalized shift and sum algorithm [4] using the interpolated and original disparities to produce the refocused video. Figure 3.5. Comparison of shift-and-sum refocusing results: (a) original -no view synthesis, (b) one novel view synthesized between original views, (c) two novel views synthesized between original views, (d) three novel views synthesized between (a) No view synthesis (b) 4x4 to 7x7 views(c) 4x4 to 10x10 views (d) 4x4 to 13x13 views66 3.2 Results and Discussion We use the Interdigital LF video dataset [36] to evaluate the proposed approach. The LF videos were captured by a synchronized 4×4 camera array at 30fps. The cameras are 70mm apart with 50°×37° field of view. Each LF video has 2048×1088 spatial resolution in raw 4:2:0 8bit YUV format and is 12.3 seconds long (i.e., 372 frames). To compare the two similarity methods used in our enhancement approach (SSIM and NCC), we experimented with sparse and dense view interpolation patterns with 𝑛 ∈ {0, 1, 2, 3, 5, 10}, at various depth planes - 𝑧 ∈{1.9, 2.0, 3.0, 3.2, 4.0}. For the controlled parameters i.e., kernel size, stride and blending tolerance (a) (b) (c) (d) (e) (f) Figure 3.6. Sparse vs dense view synthesis. (a), (c) and (e) sparse pattern with n=1, 2 and 3 respectively. (b), (d) and (f) dense pattern with n = 1, 2 and 3 respectively. 67 of the similarity computation we experimented with sliding kernel window sizes 10-90 at every 10 interval and have empirically is set 𝑘 = 60, for the stride Δ we examined values from 10 to 15 and finalized 12 to be optimum, and finally for the tolerance threshold we varied γ from 0.1 to 1m at every 0.1m and found 0.5m to be acceptable. A pretrained model [40] was used to render the synthesized views. This model was trained on YouTube videos which have multiple views of the same scene i.e., static scenes shots from a moving camera. For demonstration purposes of our method, we show the 90th frame of the “Painter” LF video sequence. Figure 3.5 present shift and sum using the synthesized views using the deep learning network. We used the videos from camera 5 and 6 and synthesized 3 novel views between the two. As these two cameras are placed side by side horizontally, we can observe the synthesized views have new content appearing to the very right from the left most synthesized view to the right most and content disappearing from the left side. This is exactly what we were expecting. The overall quality of the synthesized views looks good too. No visible holes, discontinuity or distortion were observed. 3.2.1 Optimal number of views: Using a Tesla P100 GPU with 32GB memory, the network takes roughly 4.5 minutes to generate the scene representation between two original views and 56 seconds to synthesize a novel view. For 16 original views we have 24 stereo pairs hence 24 unique scene representations. So, it takes 108 minutes to generate these scene representations. For n=3 we have 169 views and 153 among them are synthesized and 16 are original. The view synthesis would take approximately 143 minutes. Therefore, the whole view synthesis process takes about 251 minutes. We observed that as we increase the value of n for view synthesis, the refocusing results improve for both sparse and dense patterns. However, the refocusing quality for dense pattern improves drastically with 68 increasing value of n. The dense pattern achieves visually acceptable results with n = 3 (total 169 views with 16 original views). These results are presented in Figure 3.6 for side-by-side comparison of sparse and dense patterns for n = 1, 2, and 3. From the figure we can observe that, the sparse pattern still causes the ghosting artifacts in the not in-focus region even if we keep increasing n. Based on the above observations, from here on we fix n = 3 and only use the dense pattern to study SSIM-based and NCC-based enhancement methods. 3.2.2 Enhancement method Results of both enhancement methods refocused at a 2m distance are shown in Figure 3.8. We find that the NCC-based quality enhancement results are more visually pleasing and natural compared to SSIM based. The overall running time of the entire refocusing pipeline for any reference view is 252 minutes. It is worth mentioning that if we precompute the depth maps and synthesize the required novel views beforehand, the shift and sum refocusing, and the in-focus region enhancement can be done in near real-time. To study the effect of our blending approach, we use Figure 3.7. SSIM-based vs NCC-based enhancement method without blending weights. 69 the normalized similarity map ρc as blending weights instead of θc. Without the blending weights, we see that both methods introduce blurriness. However, the SSIM-based method introduces larger and more noticeable artifacts than the NCC-based method. In addition, the transition from in-focus to not in-focus region looks abrupt and unnatural. For example, for the books on the table we expect a smooth transition from the in-focus region to the not in-focus region. Figure 3.8 compares a refocused frame without any enhancement (Figure 3.8 (a)), SSIM and NCC-based enhancement without blending weights (Figure 3.8 (b) and (c) respectively) and NCC-based enhancement with blending weights (Figure 3.8 (d)). The blending weights from the last step provide us with a smoother transition as seen in Figure 3.8 (d). Based on our observation, we recommend using the NCC-based quality enhancement method. We used the Interdigital LF video dataset [36] to (a) (b) (c) (d) Figure 3.8. LF post refocusing in-focus region enhancement (a) no enhancement, (b) SSIM-based enhancement, (c) NCC-based enhancement and (d) NCC-based enhancement with blending weights. 70 evaluate the proposed approach. The LF videos were captured by a synchronized 4×4 camera array at 30fps. The cameras are 70mm apart with 50°×37° field of view. Each LF video has 2048×1088 spatial resolution in raw 4:2:0 8bit YUV format and is 12.3 seconds long (i.e., 372 frames). We demonstrate the original reference frame, refocused frames without and with our proposed object segmentation- based enhancement in Figure 3.9. We have experimented with different numbers of synthesized views n = {0, 1, 2, 3, 5, 10}, at various depths (in meters) from the camera: z = {1.9, 2.0, 3.0, 3.2, 4.0}. For demonstration purposes, we show the 75th frame of the “Painter” LF video sequence in Figure 3.9. We observed that as we increase the value of n for view synthesis, the refocusing results improve, with acceptable results achieved by n = 3 (total 169 views with 16 original views). Based on the above observations and the increased computational complexity of generating more views, (a) (b) (c) (d) (e) (f) Figure 3.9 (a) Original reference view, (b) Refocusing without enhancement, (c) Refocusing with object segmentation-based enhancement, (d), (e) and (f) are the enlargements of refocused objects of (a), (b) and (c), respectively 71 we decided to fix n = 3. Results of both without and with enhancement refocused at a 2m distance are shown in Figure 3.9 (b) and Figure 3.9 (c), respectively. From Figure 3.9, it is obvious that object segmentation-based enhancement results are more consistent with human visual perception, more visually pleasing and natural compared to the results where object segmentation has not been used. The object of interest here are the books on the stool and they have been brought in sharp focus with object segmentation-based enhancement, whereas the background experiences depth wise blur. Please refer to the zoomed results in Figure 3.9(f). When refocused without object segmentation-based enhancement, as in Figure 3.9 (b), we observe the in-focus region/object is not at sharp focus, but rather looks a little blurry and suffers from some aliasing artifacts. This is more evident in the zoomed in version of Figure 3.9 (e). This is because we used original and synthesized views for refocusing and the synthesized views are not accurate and as a result the refocusing is not accurate either. Therefore, an enhancement technique is essential. For both with and without enhancement, the background still has some aliasing artifacts. This can be mitigated through more accurate view synthesis or by using blurring filters [37]. Comparing the backgrounds of the original all in-focus Figure 3.9 (d) and the proposed method in Figure 3.9 (f), we can notice that the background patterns (floral foliage from the painting behind) in Figure 3.9 (d) are clearly (a) (b) (c) Figure 3.10 .(a) original reference view, (b) refocused on the painter at 3.2m distance and (c) refocused on the painter at 3.2m distance with enhancement 72 visible, whereas in Figure 3.9 (f) those patterns experience depth wise blur as they should in a human visual perception consistent refocused view. We also observe that the refocused region is blurrier if the object/region is closer to the camera. In Figure 3.10, we present the results of the same frame refocused on the painter at 3.2m distance from the camera. We notice the refocused region is blurry but free of any aliasing artifact. Even then, the object segmentation-based enhancement method brings the painter in sharp focus. The transition from in-focus to not in-focus region for Figure 3.10 (c) as well as for Figure 3.9 (f) looks slightly abrupt. For example, we do not see a smooth transition from the painter to background, while we would expect a smooth transition from the in-focus books on the table to the not in-focus region. This is a drawback of the proposed method. We leave this as future work, planning to explore the use of depth wise blending techniques for achieving better smoothing transition. 3.3 Conclusion This chapter presented an efficient refocusing scheme for camera array LF content, which uses a deep learning network to synthesize an appropriate number of new views and a similarity-based enhancement technique for improving the overall visual quality of the refocused frame. We found that a dense pattern of synthesized views yields better visual results for the out of focus regions, while the quality enhancement approach improves the visual quality of the in-focus regions by replacing the blurry pixels with corresponding pixels from the reference view using similarity detection and blending. As a result, our method achieves visually acceptable and natural-looking refocused LF videos. To the best of our knowledge, this is the first method designed to efficiently enhance the quality of the refocused region of camera array LF content, offering unprecedented immersiveness and an excellent infrastructure for producing top-tier mixed reality content. Having 73 said that one limitation of this work is the results are not reproducible in a short time frame. Our refocusing pipeline consists of a deep learning network. Depending on the spatial resolution of the LF content generating the required number of novel views might take a while. We have also developed an easy-to-use UI application [71]. This allows the user to experience the click and refocus functionality. This might be of special interest for the community. Our future work will focus on finetuning the object segmentation process and increasing the size of our training dataset to include as many as possible unique objects. 74 Chapter 4: LF Quality Assessment In this chapter, we introduce a novel full reference LF quality assessment technique. The core contribution of the proposed quality metric lies in the novel volumetric LF representation. It enables a feature extractor to look at the LF as a cohesive whole rather than in parts, pieces or views, unlike the state-of-the-art LF quality metrics. Therefore, the metric can discover even the most nuanced inconsistencies within the processed the LF. 4.1 Proposed LF Quality Metric To construct the quality metric, we first explored LF EPI representation to extract its features. Prior work predominantly leveraged horizontal and vertical EPIs, therefore, could measure loss of quality only in horizontal and vertical features of the LF. However, how the relationship between any two arbitrary views were impacted in the processed LF were never measured. We recognize the significance of measuring the relationship between arbitrary view pairs withing the LF. So, we first designed a diagonal EPI and incorporated it in the quality measure. This metric could directly measure quality of horizontal, vertical, and diagonal views of the LF. However, the challenge of unrelated features still loomed. To bridge this gap, we designed a deep learning-based feature extractor that traverses the whole LF and picks up low to high level features of the LF hence performs a complete quality evaluation. 4.1.1 EPI Consistency Based LF Quality Metric Traditionally, EPIs have been used to estimate the depth and synthesize novel views from LF. However, EPIs also has the potential to measure spatial quality by comparing the original and the processed LF’s EPIs. The gradients in LF EPIs due to disparity among views can be used to 75 measure the angular quality of the LF. For a resilient metric, we revisited the complete ray space multiple times in the form of horizontal, vertical, and diagonal EPIs. 4.1.1.1 Generate Horizontal, Vertical & Diagonal EPIs The horizontal and vertical EPIs are generated as described in Shi et. Al and Cui et. al. [25]. Each view in the light field image is denoted as 𝐼{𝑢,𝑣}(𝑠, 𝑡) where (𝑠, 𝑡) is the spatial coordinate and (𝑢, 𝑣) is the angular coordinate. Each horizontal EPI is denoted as 𝐸{𝑣∗,𝑡∗}(𝑠, 𝑢) where 𝑣∗ and 𝑡∗ are the fixed angular and spatial coordinates. Each row of the horizontal EPI contains the row of pixels from views 𝐼{𝑢∗,𝑣∗}(𝑠, 𝑡∗). Similarly, the vertical EPI is denoted as 𝐸{𝑢∗,𝑠∗}(𝑡, 𝑣). Similarly, each row of the vertical EPI contains the column of pixels from views 𝐼{𝑢∗,𝑣∗}(𝑠∗, 𝑡). The process is illustrated in Figure 4.1 Figure 4.1. Horizontal and vertical EPIs of 'Painter' LF test sequence Horizontal EPIVertical EPI76 The challenge of generating diagonal EPI lies in identifying corresponding pixels in the diagonally aligned views. To determine which row of pixels in each view belong to a particular diagonal EPI, the vertical and horizontal offsets are required. We call this shift compensation or diagonal translation. For every pair of views, we detect all matching Scale Invariant Feature Transform (SIFT) features [72]. The matching features are used to calculate the average vertical and horizontal offsets and are used to determine the row of pixels in the other view. To the best of our knowledge, this is the first work that uses SIFT to compensate for the shift in EPIs and create diagonal EPIs to measure angular consistency of LF content. Figure 4.2 shows an example of the matching SIFT features corresponding to each pair of views along the diagonal that contains 𝐼{1,1}, 𝐼{2,2}, 𝐼{3,3}, and 𝐼{4,4}. For the frame (all views) depicted in, Figure 4.2 we have generated 28×2048 diagonal EPIs. Figure 4.2. Diagonal EPI for views I_{1,1} ,I_{2,2} ,I_{3,3} , and I_{4,4} . The three figures on the left show an overlay of each pair of views along the diagonal and the matching Scale Invariant Feature Transform (SIFT) features that were detected. View(3,3)-Red vs. View (4,4)-GreenView(2,2)-Red vs. View (3,3)-GreenView(1,1)-Red vs. View (2,2)-GreenDiagonal EPI77 This process was repeated thrice, comprehensively covering neighboring views and ensuring a holistic assessment. 4.1.1.2 EPI Similarity To quantify spatial quality, we compare the average PSNR and SSIM between the horizontal, vertical, and diagonal EPIs of original and processed LFs. This differs from the traditional quality metrics for LF content where the average PSNR or SSIM is calculated between each view separately and averaged to report quality. Figure 4.3 presents spatial consistency measurement of EPIs. PSNR values ranges from 0 dB meaning no similarity to 999 dB meaning no distortion detected. SSIM values ranges from 0-1 meaning no similarity and exact match respectively. Although SSIM has been proven to bear more similarity with human perception compared to PSNR, our experiments conclude for EPI spatial consistency measure both metrics are equally Figure 4.3 Spatial quality measurement 78 effective. The reason being EPIs are unlike regular images and do not follow natural scene statistic or distribution. 4.1.1.3 EPI Gradient & Average Kurtosis We measure the angular distortion or deterioration using the pixel-wise gradient from horizontal, vertical, and diagonal EPIs. The gradient at each pixel is calculated using the Sobel gradient operator. The process of angular similarity measure is illustrated in Figure 4.4. A histogram of the gradients is generated, and we use the average kurtosis as a metric to describe the amount of distortion introduced from compression or any other type of distortion. Figure 4.4 Angular quality measurement 79 4.1.1.4 Shift Compensated Quality Metric Now we have two separate measures for the spatial and angular quality of LF. In the final step of our shift compensated LF quality metric we aggregate the quality measure to form a unified LF quality score. The following equation is used for the unification 𝐿𝐹𝑄𝑀 = 𝛼 × 𝑆𝐶 + (1 − 𝛼) × 𝐴𝐶 Here, SC and AC are spatial and angular consistency respectively. 𝛼 is a tuning parameter chosen based on LF operation performed. For example, 𝛼 = 0.74 yields best quality scores for compressed LF content. This is justified as compression has little impact on angular relationship among views. Figure 4.5 illustrates the framework of this EPI consistency based LF quality metric. 4.1.1.5 Performance Evaluation LF content can experience degradation from different operations, such as compression, calibration, super-resolution, and other operations. The efficacy of the proposed quality metrics was Figure 4.5 EPI similarity based LFQM outline 80 demonstrated by using LF videos compressed at different Quantization Parameter (QP) levels of 25, 30, 35, 40, and 45 using the MV-HEVC, Multi-view extension of High Efficiency Video Coding [14]. LF test sequences [36] originated from camera-array. We report the results for the ‘Painter’ test sequence in this thesis. Other test sequences show consistent performance. Table 4.1 contains the average PSNR and SSIM values between all horizontal, vertical, and diagonal EPIs from the compressed LF with respect to the original LF. The results indicate that the EPI similarity correlates with different amounts of compression. For example, when the LF is less compressed e.g., QP 25, the spatial quality measure is higher, meaning the quality loss is less however, for QP 45 the quality loss is greater. We can also see diagonal EPIs has lower average PSNR QP45 QP40 QP35 QP30 QP25 Horizontal EPI 30.64 33.14 35.64 37.89 39.95 Vertical EPI 30.91 33.43 35.90 38.07 40.04 Diagonal EPI 29.21 31.37 32.95 33.80 35.39 Average PSNR 30.26 32.65 34.83 36.59 38.46 SSIM QP45 QP40 QP35 QP30 QP25 Horizontal EPI 0.89 0.94 0.96 0.97 0.98 Vertical EPI 0.88 0.93 0.95 0.97 0.98 Diagonal EPI 0.85 0.90 0.92 0.94 0.96 Average SSIM 0.88 0.92 0.95 0.96 0.97 Table 4.1 LF spatial quality from EPI similarity 81 PSNR and SSIM compared to horizontal and vertical EPIs. This is because horizontal and vertical (a) (b) (c) Figure 4.6 Gradient histograms of the horizontal, vertical, and diagonal EPIs. The histograms show that the gradient direction is more concentrated at ±90° for higher QP values. 82 EPIs either only consider shifts in horizontal or in vertical direction respectively. However, both shifts are present in diagonal EPIs. Therefore, the diagonal EPI represent distortion both in horizontal and vertical pixels making a distorted diagonal EPI more dissimilar to the original undistorted EPI. Notably, the introduction of diagonal EPIs proved significant in measuring dissimilarities that were overlooked by existing metrics. The angular quality is measured using the gradient direction of the three EPIs. The pixel-wise gradient calculation is presented in Figure 4.6 in 10° bins (36 bins in total). We can adjust the number of bins depending on the level of accuracy desired. We have experimented with bin sizes 10, 20, 30, 40 and 50. We found that 36 is a granular enough number of bins and beyond that the quality measure plateaus. We have quantified the results of gradient direction using mean, kurtosis, entropy and skewness. Table 4.2 contains the average kurtosis values for all horizontal, vertical, and diagonal EPIs. The results show that with increasing QP levels or more compression, the kurtosis of the gradient histogram decreases. The gradient histograms show the count of gradients from all EPIs and are presented in bins of 10°. The angle is measured counterclockwise from the positive 𝑥 − 𝑎𝑥𝑖𝑠. For each 10° bin, the histograms show the gradient counts at each QP level. The histograms show that with increasing QP levels, the gradients become more concentrated at ±90°, which is likely a Reference QP45 QP40 QP35 QP30 QP25 Horizontal EPI 1.36 1.23 1.26 1.27 1.29 1.30 Vertical EPI 1.40 1.24 1.26 1.28 1.29 1.31 Diagonal EPI 1.37 1.24 1.26 1.28 1.29 1.31 Average Kurtosis 1.38 1.24 1.26 1.28 1.29 1.31 Table 4.2 Kurtosis of gradient direction histogram 83 result of the stepwise artifacts introduced in the EPI with more compression. Since the histogram is centered at 0°, it makes sense for the kurtosis to decrease with more compression since the tails of the histogram at ±90° is larger relative to the center. We form the angular consistency metric using average over the selected statistical measures (mean, kurtosis, entropy and skewness) of the pixel wise gradient map of the three EPIs. After measuring the spatial and angular consistency using the proposed method we compared the quality score with the subjective scores. 4.1.2 Discussion Even though the proposed EPI consistency based LF quality metric perform better than few state-of-the-art LF quality metrics, it still does not have good correlation with subjective quality scores. The main reason for this discrepancy is that the metric measures spatial and angular distortion in a pixel wise manner. For example, in the case of spatial consistency SSIM and PSNR are both measuring distortion for every pixel and reporting the average. Similarly, pixel wise gradient is used to report angular consistency. However, to a human observer the distortion measure is not visible at pixel level. It pertains to low- and high-level features such as structures, edges, etc. Another reason why the quality score does not have perfect correlation with subjective scores is that we are sweeping the LF ray space horizontally, vertically, and diagonally. Though it makes sure that every point (pixel) is measured against the corresponding pixels in the horizontally, vertically, and diagonally previous and next views, it however, measures all other corresponding pixels (for example a non-adjacent distant view) in a passive way. Therefore, to design an LF quality metric that has higher correlation with subjective scores we need to measure spatial and angular quality over the whole LF and not over discrete parts of it. These observations motivated the development of our subsequent Deep Feature Consistent LF quality metric. 84 4.1.3 Deep Feature Consistent LF Quality Metric Many of the existing techniques available at the time of writing this thesis either use SAI or EPI or a combination of the two representations to analyze the quality of the LF. Table 1.1 shows the representations used by the state-of-the-art techniques to evaluate LF quality. Though these are complete and efficient representation of LF, they are not well suited for quality assessment as they only represent part of the LF at a time. In the case of SAI, the LF is traversed view by view and for EPI either horizontal or vertical views are represented together. However, for an efficient and comprehensive quality assessment, we should be able to compare the subtlest details. This needs a representation that can offer the complete view of the LF at once. In addition to the above two, there is another way of representing LFs which is called the volumetric representation. The center view of the SAIs (13th view for a 5×5 view arrangement) is selected and the neighboring views are stacked one after the other in a spiral fashion. We take inspiration from our previous work [73] where we have established that any SAI is most similar to its horizontal or vertical neighbors and LF processing techniques can benefit from this property. Therefore, we modify the ordering of views in the volumetric representation so that it is better suited for LF quality assessment. The details are discussed in the following subsection. 4.1.3.1 Modified Volumetric Representation The LF representation for quality assessment should maintain the spatial and angular relationship among the views. For any LF view the adjacent horizontal or vertical views are the most similar/correlated views [73]. Stacking similar views together will enhance the inherent consistency or the lack of it (when too distorted) for the spatial and angular features. Therefore, we modify the existing volumetric LF representation to place similar SAIs together for a 85 comprehensive LF quality assessment. Given the matrix 𝕊, of SAIs of LF 𝐿 with 𝑚 × 𝑛 views, we design the volumetric LF representation for quality assessment. Here, 𝕊 = [𝑠1,1 ⋯ 𝑠1,𝑛⋮ ⋱ ⋮𝑠𝑚,1 ⋯ 𝑠𝑚,𝑛]. Each element 𝑠𝑖,𝑗 of the matrix represents a sub-aperture view here, 𝑖 =1, … , 𝑚, 𝑗 = 1, … , 𝑛. First, we find the view 𝐶 with the highest average spatial similarity with all the other (𝑚 × 𝑛) − 1 views in the LF. 𝐶 is calculated using equation (2). Here 𝑆(∙) is a similarity measure function. We (a) (b) Figure 4.8 (a) a standard CNN with no efficient feature propagation and (b) a dense block concatenating learned features from previous layers Figure 4.7 Proposed volumetric LF representation using similarity-based stacking 86 have used different similarity measures such as cross-correlation, and SSIM to find the view with the highest average similarity with all the other views and they yield the same result. 𝐶 = argmax (∑ ∑∑ ∑ 𝑆 (𝑠𝑖,𝑗,𝑠𝑝,𝑞)𝑛𝑞=0𝑚𝑝=0𝑚×𝑛𝑛𝑗=0𝑚𝑖=0 )𝑖,𝑗 (4.1) We take the view 𝐶 (typically at the center of the LF) and find the view that is most similar to 𝐶 and stack them together. The example deals with an odd-by-odd view arrangement where the search of 𝐶 is apparent. However, even with an even-by-even number of SAIs or any other arrangements, we have found no two SAIs have the same similarity score in practice. We also found the views closest to the center are prime candidates for 𝐶. In the unlikely event of two or more SAIs having the same highest similarity with all the other SAIs, we can make an arbitrary selection. We continue this process until all the views in the LF are stacked. In an ideal scenario, this process will lead to a spiral pattern. However, there might be cases where it creates a different pattern depending on the horizontal and vertical distances (baseline) among the cameras of the array. This is our LF representation for our proposed depth feature consistent sparse light field quality metric (DFCS LFQM). It is different from the existing volumetric representation as it takes the correlation among the views into consideration rather than following a spiral pattern. Figure 4.7 shows an example of our proposed volumetric LF representation. We compute the volumetric representation for the original or undistorted LF and follow the same order of stacking for the distorted LF. This ensures that the distortions are the only changes experienced by the feature extractor (discussed in the next subsection). 4.1.3.2 Spatial and Angular Feature Extraction Previous methods compared parts of the LF to determine quality, as a result, some relationships among different parts of the LF were not explored. For example, Fang at el. compared features of 87 horizontal and vertical EPIs [47]. The feature extractor will not be able to extract the relationship (feature) among the views that it has not seen together. We can see from Table 1.1 that most existing techniques except for LF-ASC do not provide the feature extractor with the whole picture. However, the proposed representation stacks all the views present in the LF so the feature extractor can see the whole LF and learn salient features. Because of its structure, each cross-section (horizontal or vertical) of the volumetric LF, has both spatial and angular features. The 2D cross-section has color information as well as geometric information from each view capturing the intricate details and characteristics of LF. Therefore, from the same features, we can assess the color and geometric consistency of the reference and distorted LF. We extract features from the horizontal (or vertical) cross-sections of both the original and distorted volumetric LF using DenseNet that we fine-tuned for the LF content. We choose DenseNet because it has strong feature propagation inside each dense block compared to other convolutional neural networks (CNNs). Moreover, it is light in terms of number of parameters compared to other famous CNNs making it an ideal feature extractor for real-time applications. The difference between a DenseNet and a traditional CNN is illustrated in Figure 4.8. DenseNet is composed of several dense blocks, one such block is shown in Figure 4.8 (b) and each dense block has several dense convolution layers, where the feature extraction occurs. Other CNNs, will pass on the features extracted from one convolution layer to the next and the next layer will convolve the features again as presented in Figure 4.8 (a). So, the features extracted are not 88 available beyond the layer they were extracted in. Therefore, to extract the features of interest we should accumulate features from most layers and there are many redundant features too. However, DenseNet preserves the feature maps generated after every convolution and using dense connections concatenates (places them one after the other) the feature maps inside each dense block. This property is presented in equations (3) and (4) for any layer 𝑙. Here 𝐹 is a composite that represents different operations such as convolution, pooling, activation, etc. 𝑓𝑙 = 𝐹(𝑓𝑙−1) (4.2) 𝑓𝑙 = 𝐹([𝑓0, 𝑓1, … , 𝑓𝑙−1]) (4.3) 4.1.3.3 Feature Consistency After extracting the salient features of the original and distorted LFs we compute the consistency of the features extracted. As mentioned previously because of the representation both spatial and angular features are now extracted as spatial features. We compute the mean squared error of the reference feature vector 𝑿 and distorted LF’s feature vector 𝒀 using equation (5), this gives us the error between the original and distorted cross-sections and hence represents the quality of the distorted LF. The proposed method is summarized in Figure 4.9. 𝑄𝑀 =1𝑛+1∑ (𝑥𝑖 − 𝑦𝑖)2𝑛𝑖=0 (4.4) Figure 4.9 The framework of the proposed DFCS LF quality assessment metric 89 Even though the volumetric representation may create different representations for different capture setups (baseline, number of cameras, arrangement), the feature consistency measure will ensure that the quality metric reflects the level of distortion experienced compared to the original LF and produce uniform assessment scores. Therefore, the volumetric representation is crucial to enable the deep feature extraction model to better learn the relationship among the views in the original LF and how they have changed in the distorted LF. 4.2 Experiments and Evaluation In this section, we compare the proposed method with multiple state-of-the-art LF quality metrics (i.e. PSNR, SSIM, VMAF [17], Shi [25], Fang [47], and SDFM [48]). Although PSNR, SSIM, and VMAF were developed for regular 2D image and video quality measurement, Shi [25] is a no-reference, Fang [47] and SDFM [48] are full-reference quality metrics developed specifically for LF. While choosing the existing LF quality metrics to compare with our proposed method we have considered variables such as the need of a reference, representation, and feature(s) explored (see Table 1.1), and the best performance from the discussed methods. Sequence Camera Views Baseline Resolution Chess Player GoPro 3×5 60-100 mm 1920×1080 Table Soccer Bar Automaton CMOSIS 4×4 70 mm 2048×1088 Painter Birthday Table 4.3 Properties of the datasets used. 90 4.2.1 Data and LF Operations We use two different publicly available and widely used camera array-based LF datasets from [36] and [54]. The properties of the dataset are presented in Table 4.3. The videos were recorded at 30 or 25 frames per second and each video has 150-300 frames. We compare three different types of widely used LF operations, namely, compression, calibration, and super-resolution to generate the distorted LF. For compression, we used high-efficiency video coding (HEVC) [13] with 4 degrees of compression (quantization parameters 20, 26, 32, and 40). For calibration, we use the original Criteria Distortion PSNR SSIM VMAF [17] NR-LFQA [25] Fang [47] SDFM [48] Proposed RMSE Compression 1.0020 0.9710 0.8500 0.7805 0.5814 0.5612 0.5500 0.053 0.038 0.027 0.003 0.019 0.016 0.018 Calibration 0.9840 0.9512 0.8774 0.7748 0.5970 0.5882 0.5571 0.039 0.015 0.016 0.023 0.083 0.015 0.013 Super resolution 0.9550 0.8470 0.7754 0.7641 0.7415 0.7541 0.6020 0.041 0.03 0.03 0.025 0.021 0.023 0.023 PLCC Compression 0.8452 0.9021 0.9110 0.8207 0.8451 0.9141 0.9314 0.043 0.03 0.027 0.013 0.019 0.003 0.012 Calibration 0.7890 0.8002 0.8102 0.8244 0.8492 0.9001 0.9435 0.038 0.031 0.035 0.03 0.021 0.019 0.016 Super resolution 0.7540 0.7781 0.7845 0.7840 0.8145 0.847 0.8842 0.041 0.05 0.032 0.026 0.033 0.021 0.014 SROCC Compression 0.8517 0.8800 0.7820 0.8273 0.8462 0.8714 0.8741 0.028 0.031 0.021 0.028 0.019 0.016 0.015 Calibration 0.7521 0.7584 0.7841 0.8140 0.8547 0.8512 0.8804 0.04 0.04 0.031 0.031 0.021 0.023 0.017 Super resolution 0.7064 0.7170 0.7621 0.8164 0.8471 0.8014 0.8431 0.034 0.034 0.031 0.027 0.02 0.021 0.023 Table 4.4 Performance Comparison of the proposed and other LF quality metrics 91 and calibrated frames provided by the authors of [36]. For super-resolution, we used the method proposed in [74] for upscaling the spatial resolution by ×2, ×3, and ×4 over the discussed datasets. The native resolutions of the LFs can be found in Table 4.3. Note we used the same computational environment to generate the distorted LFs. The machine used was provided by a state-of-the-art computing cluster Digital Research Alliance of Canada [75] and had Tesla V100 SXM2 GPU, with 32GB of HBM2 memory and 640 tensor cores available. 4.2.2 Feature Extraction Model The DenseNet121 [76] model was trained on the horizontal cross sections of the LFs. A total of 1050 original and 7350 distorted volumetric LFs were formed from the dataset. The training, validation, and testing split was 70, 15, and 15 with equal distribution of original, and three applied operations. The subjective ground truth score was obtained in the form of just noticeable differences (JNDs) using the method outlined in [77]. 20 (11 female and 9 male) subjects viewed 40 original 150 distorted LFs side by side. The learning rate was 0.001 and it was halved after every validation step, batch size was 64 and we have used Adam optimizer. Training time for the deep learning model was 120 hours with 84% accuracy at 85 epochs. This achieved 87% testing accuracy with the inference speed of 41.6 frames (not LFs) per second. However, we can speed up the process by using multiple GPUs to process different parts of the volumetric representation to reach the desired speed of the application. It is also worth mentioning that for 100% of the test cases, the quality score belonged to either the correct class or the next best one. 4.2.3 Results and Discussion Different metrics report results in different ranges. For example, PSNR uses 0-999 dB to present worst to best quality whereas, SSIM uses 0 to 1 to report no similarity to an exact match. We normalized the quality scores from different metrics using a logistic mapping function 92 recommended by the international telecommunication union (ITU) [78] for a fair comparison across the board. We use root mean square error (RMSE), Pearson linear correlation coefficient (PLCC), and Spearman rank order correlation coefficient (SROCC) to evaluate the reliability of the different quality metrics. A low RMSE score and high PLCC and SROCC scores indicate better performance. Table 4.4 illustrates the results obtained through our extensive experiments. Each data point in the table is the average of all the experiments performed for each distortion. To better present the results we highlight the top 3 performances in bold and report a 95% confidence interval. The quality scores from the classical methods e.g., PSNR, SSIM, and VMAF are averaged over the SAIs of the LF. For the LF quality metrics, the respective model-generated output was directly fitted to the JND using a logistic function. Then the PLCC, SROCC, and RMSE were computed. From the results, we can see the classical metrics such as PSNR and SSIM do not perform as well as metrics designed specifically for LF. VMAF however, has a better performance among the non-LF quality metrics and at times outperforms no-reference LF quality metrics. This is because sparse LF needs high-quality references to verify the quality. The performances of the full-reference metrics are comparable. However, the overall RMSE, PLCC, and SROCC values indicate our method outperforms full-reference LF quality assessment methods. This is because we have represented the LF in a manner where the deep neural network has the opportunity to experience all the views of the LF at once and extract the obvious as well as latent salient features. Another reason for the performance boost is the extraction of features from different stages of the DenseNet, which has the feature preservation characteristic. This enables us to include low-level simple features which are visible at the early stages to high-level complicated features which are found in the deeper layers. In terms of inference speed, our method outperforms most methods with a single GPU except for SDFM. This method has a 6% lower accuracy than our method with 93 a 12 fps higher prediction time. This is the best trade-off between speed and accuracy among the existing LF quality metrics. We have also compared the performance of the proposed volumetric representation against the existing one. For the LFs used in this paper, 43% of the representations defaulted to the existing volumetric representation. For the rest of the LFs, the new representation saw at least 26% better quality predictions than the existing representation. 4.3 Conclusion In this chapter, a full-reference LF quality assessment model, called the deep feature consistent sparse (DFCS) LF quality metric has been proposed. The superiority of the proposed DFCS is credited to unique LF representation and the usage of the deep LF features for the quality estimation. It fully explores the intricate characteristics of an LF and is thus effective for evaluating its perceived quality. Experimental results demonstrate that our proposed DFCS LF quality metric obtain better performance on perceptual quality evaluation of the LFs than multiple classical and state-of-the-art quality assessment methods. 94 Chapter 5: Conclusion In this thesis, we have extensively explored concepts within light field video content, with a specific focus on compression, refocusing capabilities, and quality assessment. The adoption of LF technology instigated substantial changes in the ways we capture, process, store and consume visual content. Throughout this dissertation, our objective has been to make LF applications easily adaptable and seamlessly integrated into various multimedia contexts. We achieve this objective in three stages. Firstly, optimizing compression efficiency and speeding up the decoding process by implementing a random-access efficient compression strategy. Secondly, we have investigated the feasibility of refocusing wide baseline LF content using advanced techniques such as view-synthesis, similarity-based enhancements and object segmentation. Finally, we designed an objective LF quality evaluation metric which is consistent with human visual perception. A summary of the contributions and future research directions are discussed below. 5.1 Summary of Contributions The detailed methods and results are discussed in chapter 2, 3, and 4. Here we summarize the major contributions this thesis made to the state-of-the-art and the key takeaways. In Chapter 2, we proposed two pseudo-sequence-based prediction structures and coding orders that efficiently compress LF video content using the existing HEVC standard. The first structure optimizes for encoding efficiency, and the second one reduces the overall random-access complexity of the LF. The first coding structure, called the 'universal prediction structure,' follows a heuristic algorithm. Our first contribution in the LF compression domain is the extensive analysis of LFs acquired through various capture methods. This makes the results of the analysis agnostic to the capture medium. The next contribution is the knowledge that stems from the analysis. 95 Our first discovery from the analysis was regarding the placement of I-frame. We realized that the central views in an LF have the highest average similarity with all the other views. Making them best suited to become I-frame in compression. Our second discovery was about how the neighboring views of LF are correlated with each other. Finally, by leveraging the discoveries from the analysis, our last contribution of this chapter is the design of two inter-view prediction structures for compressing LF videos. For optimizing encoding efficiency, we utilized the horizontal and vertical correlations among the views and adjacent reference placement. This method stands out by achieving a remarkable 17% bitrate reduction. For the second method, diagonal references are utilized for improved random-access efficiency. It offers a 20% enhancement in random-access efficiency while maintaining reasonable computational complexity. These techniques were designed so they seamlessly integrate into HEVC as a cost-effective preprocessing step, without necessitating any changes to the codec. The key takeaways from the research conducted for LF video compression are as follows: different topological structures can be used to compress LF video using existing standards such as HEVC. Not all of these structures will be compression efficient. A low-cost preprocessing of the LF can reveal which inter-view prediction structure will lead to the maximum bitrate saving. Our approach can also be customized to leverage other qualities such as ease of random-access. In chapter 3, we presented an efficient and human visual system consistent refocusing scheme meticulously designed for sparse LF content, which harnesses the power of a deep learning network to synthesize an optimal number of new views and is complemented by an object segmentation network to digitally focus on the whole object. To elevate the visual quality of the refocused frame, we incorporate a depth filter-based enhancement. We found that a dense collection of synthesized views yields better visual results for the out of focus regions, while the 96 quality enhancement approach improves the visual quality of the in-focus regions by replacing blurry pixels with corresponding pixels from the reference view. This fusion generates visually captivating and natural-looking refocused LF videos. To the best of our knowledge, this is the first method designed to efficiently enhance the visual quality of the refocused region of camera array LF content, offering unprecedented immersiveness, consistency with human visual perception and an excellent infrastructure for producing high-quality mixed reality content. The key takeaways from this chapter are as follows: due to the wider baseline there might be some parts of the scene that were not captured by any of the cameras in a camera array. Even with high quality view synthesis techniques it will not be possible to produce those missing parts. Therefore, there might always be some residual artifacts in the final refocused image. Advanced reconstruction techniques might come in handy in such situations. However, they are out of the scope of this thesis and have not been explored. If we are using object segmentation to refocus on specific objects, then the object segmentation method must be highly effective in recognizing the edges of the objects. It is common for an object and the area surrounding it to belong to different depth planes (they could also belong to far away depth planes). Therefore, if the object segmentation method used cannot separate the object from its surroundings with high precision, the resultant refocusing will lose its naturalness. Chapter 4 introduced a full-reference LF quality assessment framework. Addressing the limitations of existing LF quality metrics that often evaluates the LF in parts leading to a quality score that do not account for subtle inaccuracies. We designed an LF quality metric by augmenting an existing metric with diagonal EPI information to better align the objective assessment score to the subjective ones. Our main contribution was the use of SIFT-assisted diagonal EPI translation for scanning the diagonal ray space of the LF. Though it introduces previously unutilized LF 97 information in the metric, it still cannot accurately describe the entire LF. To mitigate this problem, we designed a new LF representation called the “similarity based volumetric LF representation”. It enables a feature extractor to view the LF as a whole and not in pieces. Therefore, an accurate assessment can be performed to verify if the intricate relationship among the views is preserved or not in the processed LF. We also choose a deep neural network that has the feature conservation property so that we can leverage even the most intricate features of the LF to evaluate the quality of the distorted LF. To verify and validate the performance of our proposed framework we tested three commonly used LF operations, compression, calibration, and spatial super-resolution. Experimental results show that our method outperforms classical as well as state-of-the-art LF image quality metrics. The key takeaways from this chapter are as follows: the quality measurement of LFs differ significantly from traditional 2D videos. Each view has spatial and angular relationship with all the other views and also has temporal relationship with itself. Therefore, transitive quality assessment techniques such as assessing all the horizontal views then assessing all the vertical views will lead to error propagation. In order to stop error propagation in final quality score, we should avoid transitive ways of quality assessment. Different operations lead to different kinds of distortions such as compression operations rarely lead to angular distortion in LFs and calibration almost always has some angular distortion. Therefore, if we are using a trained model for quality assessment of LFs first we need to make sure the model was trained for the specific distortion. 5.2 Significance and Potential Applications While computation photography such as LF will have disruptive impact on the capturing, post processing and display industries, it also opens the doors to new opportunities to revolutionize 98 digital multimedia creation and consumption. Visualization is beyond what was ever possible with traditional, 3D or even 360 content. However, the technology is still far from offering a truly immersive life like experience. This thesis aims to move one step closer to this ultimate goal by encoding the content in a consumer-friendly format, merging human perception with attractive features like post-shoot refocusing, and proposing an objective way of measuring the quality of processed LFs. LF compression methods will have wide spread applications in streaming, broadcasting high quality LFs to consumer homes. High compression and random-access efficiency of LF content is also desirable in immersive media such as virtual, augmented, mixed reality (VR, AR, MR) applications. Delay sensitive applications such as remote surgery, autonomous driving can take advantage of the high bitrate gain. Efficient post shoot refocusing that is closer to how we as humans perceive the world will contribute to the visual appeal and naturalness of the content. Such a feature empowers the creative freedom of the artists. It is also offering substantial potential in video gaming industry where refocusing on object of interest is a crucial operation. A refocusing pipeline solely trained on LF surgery videos will be able seamlessly refocus on objects of interest (organs, arteries, tumors, etc.) and that will help the medical professionals make even more informed decisions. Measuring the quality of the LF after processing is of utmost importance. As in near future LF will power mission-critical applications such as autonomous vehicles, machine vision based medical applications (diagnosis, surgery, training, etc.) coupled with entertainment industry (cinema, broadcasting, streaming, gaming, live streaming, holographic events, etc.), the development of a robust LF quality stand as a linchpin in ensuring service quality and, consequently, end-user experience. 99 5.3 Future Research Directions Future work may include the standardization of the proposed LF compression techniques. A non-formative (introducing no changes to the standard but as an add-on) technical contribution document should be prepared and submitted to moving picture coding experts’ group (MPEG) for review by the work group on immersive media coding MPEG immersive video (MIV). Due to the unavailability LF datasets with diverse view arrangements compression and random-access efficiency were validated on planer camera or lens arrangements. Investigating the appropriateness of the compression methods for other emerging LF capture setups is also a logical next step. The impact of certain views being corrupted or missing on the compression efficiency could be evaluated. The proposed sparse LF refocusing framework is real-time if the synthesized novel views are precomputed or readily available. However, fast refocusing is not possible when only the sparse LF is provided. Depending on the spatial resolution of the LF content (which is usually high for sparse LF) generating the required number of novel views might take a while. It might be worthwhile to explore GPU accelerators to fast track the view synthesis process. Even though the object segmentation-based approach brings the refocusing application closer to human perception, it will be interesting to add cues from visual attention models to include even finer properties of HVS to the refocusing pipeline. While camera array LFs offer a larger depth of field, the optimum number of depth planes for a given LF and camera setup could be computed for even better perceptual refocusing. Quality evaluation of other operations such as restoration, refocusing and view synthesis using our metric is another future direction that might bring new contributions in the LF quality assessment research domain. Though our refocusing method and quality metric is designed specifically for 100 sparse LF, it would be interesting to see their applicability in dense LF as well. As we have leveraged several deep learning models to design the methods in this thesis another possible topic would be to upgrade the model to continual learning models to automatically improve the accuracy as new data becomes available. 101 Bibliography [1] Ng, R., Levoy, M., Brédif, M., Duval, G., Horowitz, M., & Hanrahan, P. (2005). Light field photography with a hand-held plenoptic camera (Doctoral dissertation, Stanford university) [2] Wilburn, B., Joshi, N., Vaish, V., Talvala, E. V., Antunez, E., Barth, A., ... & Levoy, M. (2005). High performance imaging using large camera arrays. In ACM siggraph 2005 papers (pp. 765-776). [3] Levoy, M., & Hanrahan, P. (2023). Light field rendering. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2 (pp. 441-452 [4] Alain, M., Aenchbacher, W., & Smolic, A. (2019). Interactive light field tilt-shift refocus with generalized shift-and-sum. arXiv preprint arXiv:1910.04699. [5] Wu, G., Masia, B., Jarabo, A., Zhang, Y., Wang, L., Dai, Q., ... & Liu, Y. (2017). Light field image processing: An overview. IEEE Journal of Selected Topics in Signal Processing, 11(7), 926-954. [6] Yücer, K., Sorkine-Hornung, A., Wang, O., & Sorkine-Hornung, O. (2016). Efficient 3D object segmentation from densely sampled light fields with applications to 3D reconstruction. ACM Transactions on Graphics (TOG), 35(3), 1-15 [7] Zhang, C., Hou, G., Zhang, Z., Sun, Z., & Tan, T. (2018). Efficient auto-refocusing for light field camera. Pattern Recognition, 81, 176-189 [8] Wang, Y., Long, Y., Fan, S. H., & Dou, Q. (2022, September). Neural rendering for stereo 3d reconstruction of deformable tissues in robotic surgery. In International conference on medical image computing and computer-assisted intervention (pp. 431-441). Cham: Springer Nature Switzerland 102 [9] Global Market Insights, “Light Field Market Share 2018-2024 Industry Growth Forecast Report,” 2018. [10] B. Derviş, “Global Consumer Electronics to be a US$ 3 Trillion Market by 2020,” 2013. [11] Statista, “Value of the entertainment and media market in Canada 2014-2020,” 2014. [12] Lippmann, G. (1908). Epreuves reversibles donnant la sensation du relief. J. Phys. Theor. Appl., 7(1), 821-825 [13] Sullivan, G. J., Ohm, J. R., Han, W. J., & Wiegand, T. (2012). Overview of the high efficiency video coding (HEVC) standard. IEEE Transactions on circuits and systems for video technology, 22(12), 1649-1668 [14] Tech, G., Chen, Y., Müller, K., Ohm, J. R., Vetro, A., & Wang, Y. K. (2015). Overview of the multiview and 3D extensions of high efficiency video coding. IEEE Transactions on Circuits and Systems for Video Technology, 26(1), 35-49. [15] Bando, Y., & Nishita, T. (2007, October). Towards digital refocusing from a single photograph. In 15th Pacific Conference on Computer Graphics and Applications (PG'07) (pp. 363-372). IEEE. [16] Knight, T. J., Pitts, C., Ng, Y. R., Fishman, A., Romanenko, Y., Kalt, J., & Akeley, K. B. (2015). U.S. Patent No. 8,995,785. Washington, DC: U.S. Patent and Trademark Office. [17] Bampis, C. G., Li, Z., Katsavounidis, I., & Bovik, A. C. (2018). Recurrent and dynamic models for predicting streaming video quality of experience. IEEE Transactions on Image Processing, 27(7), 3316-3331. 103 [18] M. Levoy, “Light fields and computational imaging,” Computer (Long. Beach. Calif)., vol. 39, no. 8, pp. 46–55, Aug. 2006, doi: 10.1109/MC.2006.270. [19] J. Hou, J. Chen, and L. P. Chau, “Light Field Image Compression Based on Bi-Level View Compensation with Rate-Distortion Optimization,” IEEE Trans. Circuits Syst. Video Technol., vol. 29, no. 2, pp. 517–530, 2019, doi: 10.1109/TCSVT.2018.2802943. [20] M. Broxton et al., “Immersive light field video with a layered mesh representation,” ACM Trans. Graph., vol. 39, no. 4, pp. 1–15, 2020, doi: 10.1145/3386569.3392485. [21] Raytrix GmbH, “Raytrix 3D Light-Field Vision. 3D cameras. Light field cameras,” 2020. [Online]. Available: https://raytrix.de/products/. [Accessed: 13-Jan-2024]. [22] Dansereau, D. manual 2015, “Light field toolbox for matlab.” [23] “Technical report of the joint ad hoc group for digital representations of light/sound fields for immersive media applications,” 2021, Geneva, Switzerland. [24] L. D. Software, “the resources – LYTRO meltdown.” [Online]. Available: http://optics.miloush.net/lytro/TheResources.aspx. [Accessed: 12-Aug-2023]. [25] L. Shi, W. Zhou, Z. Chen, and J. Zhang, “No-Reference Light Field Image Quality Assessment Based on Spatial-Angular Measurement,” IEEE Trans. Circuits Syst. Video Technol., vol. 30, no. 11, pp. 4114–4128, 2020, doi: 10.1109/TCSVT.2019.2955011. [26] G. Wang, W. Xiang, M. Pickering, and C. W. Chen, “Light Field Multi-View Video Coding with Two-Directional Parallel Inter-View Prediction,” IEEE Trans. Image Process., vol. 25, no. 11, pp. 5104–5117, 2016, doi: 10.1109/TIP.2016.2603602. [27] A. Vetro, T. Wiegand, and G. J. Sullivan, “Overview of the Stereo and Multiview Video Coding Extensions of the H.264/MPEG-4 AVC Standard,” Proc. IEEE, vol. 99, no. 4, pp. 626–642, Apr. 2011, doi: 10.1109/JPROC.2010.2098830. 104 [28] J. Khoury, “A New Prediction Structure for Efficient MV-HEVC based Light Field Video Compression,” in International Conference on Computing, Networking and Communications (ICNC 2019), 2019, p. 4. [29] N. Mehajabin, S. R. Luo, H. Wei Yu, J. Khoury, J. Kaur, and M. T. Pourazad, “An Efficient Random Access Light Field Video Compression Utilizing Diagonal Inter-View Prediction,” in 2019 IEEE International Conference on Image Processing (ICIP), 2019, pp. 3567–3570, doi: 10.1109/ICIP.2019.8803668. [30] N. Mehajabin, M. Pourazad, and S. Member, “SSIM Assisted Pseudo-sequence-based Prediction Structure for Light Field Video Compression,” in IEEE International Conference on Consumer Electronics, 2020, no. c, pp. 5–6. [31] V. Avramelos, G. Van Wallendael, and P. Lambert, “Overview of MV-HEVC prediction structures for light field video,” in Applications of Digital Image Processing XLII, 2019, p. 51, doi: 10.1117/12.2529137. [32] U. Fecker and A. Kaup, “H.264/AVC-Compatible Coding of Dynamic of Light Fields using Transposed Picture Ordering,” Eusipco, p. 4, 2005. [33] C. Perra and P. Assuncao, “High efficiency coding of light field images based on tiling and pseudo-temporal data arrangement,” 2016 IEEE Int. Conf. Multimed. Expo Work. ICMEW 2016, pp. 1–4, 2016, doi: 10.1109/ICMEW.2016.7574671. [34] F. Dai, J. Zhang, Y. Ma, and Y. Zhang, “Lenselet image compression scheme based on subaperture images streaming,” Proc. - Int. Conf. Image Process. ICIP, vol. 2015-Decem, pp. 4733–4737, 2015, doi: 10.1109/ICIP.2015.7351705. 105 [35] D. Liu, L. Wang, L. Li, X. Zhiwei, F. Wu, and Z. Wenjun, “Pseudo-sequence-based light field image compression,” 2016 IEEE Int. Conf. Multimed. Expo Work. ICMEW 2016, pp. 3–6, 2016, doi: 10.1109/ICMEW.2016.7574674. [36] N. Sabater et al., “Dataset and Pipeline for Multi-view Light-Field Video,” IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit. Work., vol. 2017-July, pp. 1743–1753, 2017, doi: 10.1109/CVPRW.2017.221. [37] Y. Wang, J. Yang, Y. Guo, C. Xiao, and W. An, “Selective Light Field Refocusing for Camera Arrays Using Bokeh Rendering and Superresolution,” IEEE Signal Process. Lett., vol. 26, no. 1, pp. 204–208, 2019, doi: 10.1109/LSP.2018.2885213. [38] T. Li et al., “Neural 3D Video Synthesis from Multi-view Video,” Proc. IEEE Comput. Soc. Conf. Comput. Vis. Pattern Recognit., vol. 2022-June, pp. 5511–5521, 2022, doi: 10.1109/CVPR52688.2022.00544. [39] G. Riegler and V. Koltun, “Free View Synthesis,” Lect. Notes Comput. Sci. (including Subser. Lect. Notes Artif. Intell. Lect. Notes Bioinformatics), vol. 12364 LNCS, pp. 623–640, 2020, doi: 10.1007/978-3-030-58529-7_37. [40] T. Zhou, R. Tucker, J. Flynn, G. Fyffe, and N. Snavely, “Stereo magnification: Learning view synthesis using multiplane images,” ACM Trans. Graph., vol. 37, no. 4, 2018, doi: 10.1145/3197517.3201323. [41] B. Busam, M. Hog, S. McDonagh, and G. Slabaugh, “SteReFo: Efficient image refocusing with stereo vision,” Proc. - 2019 Int. Conf. Comput. Vis. Work. ICCVW 2019, pp. 3295–3304, 2019, doi: 10.1109/ICCVW.2019.00411. 106 [42] O. Cossairt, N. Matsuda, and M. Gupta, “Digital refocusing with incoherent holography,” 2014 IEEE Int. Conf. Comput. Photogr. ICCP 2014, 2014, doi: 10.1109/ICCPHOT.2014.6831819. [43] Q. Qu, X. Chen, V. Chung, and Z. Chen, “Light Field Image Quality Assessment With Auxiliary Learning Based on Depthwise and Anglewise Separable Convolutions,” IEEE Trans. Broadcast., vol. 67, no. 4, pp. 837–850, 2021, doi: 10.1109/TBC.2021.3099737. [44] P. Paudyal, F. Battisti, S. Member, M. Carli, and S. Member, “Reduced Reference Quality Assessment of Light Field Images,” IEEE Trans. Broadcast., vol. 65, no. 1, pp. 152–165, 2019. [45] L. Shan, P. An, C. Meng, X. Huang, C. Yang, and L. Shen, “A No-Reference Image Quality Assessment Metric by Multiple Characteristics of Light Field Images,” IEEE Access, vol. 7, pp. 127217–127229, 2019, doi: 10.1109/ACCESS.2019.2940093. [46] X. Min, J. Zhou, G. Zhai, P. Le Callet, X. Yang, and X. Guan, “A Metric for Light Field Reconstruction, Compression, and Display Quality Evaluation,” IEEE Trans. Image Process., vol. 29, pp. 3790–3804, 2020, doi: 10.1109/TIP.2020.2966081. [47] Y. Fang, K. Wei, J. Hou, W. Wen, and N. Imamoglu, “Light Filed Image Quality Assessment by Local and Global Features of Epipolar Plane Image,” 2018 IEEE 4th Int. Conf. Multimed. Big Data, BigMM 2018, 2018, doi: 10.1109/BigMM.2018.8499086. [48] Y. Tian, H. Zeng, J. Hou, J. Chen, J. Zhu, and K. K. Ma, “A Light Field Image Quality Assessment Model Based on Symmetry and Depth Features,” IEEE Trans. Circuits Syst. Video Technol., vol. 31, no. 5, pp. 2046–2050, 2021, doi: 10.1109/TCSVT.2020.2971256. 107 [49] A. Dricot, J. Jung, M. Cagnazzo, B. Pesquet, and F. Dufaux, “Full parallax super multi-view video coding,” 2014 IEEE Int. Conf. Image Process. ICIP 2014, pp. 135–139, 2014, doi: 10.1109/ICIP.2014.7025026. [50] U. Fecker, “Complexity Evaluation Of Random Access To Coded Multi-View Video Data Ulrich Fecker and Andr ´ e Kaup,” no. Eusipco, pp. 1–4, 2007. [51] L. Guillo et al., “Light field video dataset captured by a R8 Raytrix camera ( with disparity maps ) To cite this version : HAL Id : hal-01804578,” 2018. [52] B. Wang, Q. Peng, E. Wang, K. Han, and W. Xiang, “Region-of-interest compression and view synthesis for light field video streaming,” IEEE Access, vol. 7, pp. 41183–41192, 2019, doi: 10.1109/ACCESS.2019.2907572. [53] C. G. P. David, M. Le Pendu, “Sintel light field video dataset.” [Online]. Available: http://clim.inria.fr/Datasets/SyntheticVideoLF/index.html. [54] Ł. D. ˛ Abała et al., “Efficient Multi-image Correspondences for On-line Light Field Video Processing,” 2016. [55] B. Bross, J. Chen, J. R. Ohm, G. J. Sullivan, and Y. K. Wang, “Developments in International Video Coding Standardization After AVC, With an Overview of Versatile Video Coding (VVC),” Proc. IEEE, no. July, pp. 1–31, 2021, doi: 10.1109/JPROC.2020.3043399. [56] G. Bjontegaard, “Calculation of average PSNR differences between RD-curves.” VCEG-M33, ITU-T SG 16/Q6, 13th VCEG Meeting, Austin, TX, USA, 2001. [57] G. Bjøntegaard, “Improvements of the BD-PSNR Model.” VCEG-AI11, ITU-T SG16/Q6, 35th VCEG Meeting, Berlin, Germany, July, 2008. 108 [58] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image quality assessment: From error visibility to structural similarity,” IEEE Trans. Image Process., vol. 13, no. 4, pp. 600–612, 2004, doi: 10.1109/TIP.2003.819861. [59] M. Pilu, “On using raw MPEG motion vectors to determine global camera motion,” HP Lab. Tech. Rep., no. 97–102, pp. 1–17, 1997, doi: 10.1117/12.298354. [60] W. Hamidouche, M. Raulet, and O. Déforges, “4K real-time and parallel software video decoder for multilayer HEVC extensions,” IEEE Trans. Circuits Syst. Video Technol., vol. 26, no. 1, pp. 169–180, 2016, doi: 10.1109/TCSVT.2015.2478705. [61] M. Abeydeera, M. Karunaratne, G. Karunaratne, K. De Silva, and A. Pasqual, “4K real-time HEVC decoder on an FPGA,” IEEE Trans. Circuits Syst. Video Technol., vol. 26, no. 1, pp. 236–249, 2016, doi: 10.1109/TCSVT.2015.2469113. [62] T. S. Kim, C. E. Rhee, and H. J. Lee, “Merge mode estimation for a hardware-based HEVC encoder,” IEEE Trans. Circuits Syst. Video Technol., vol. 26, no. 1, pp. 195–209, 2016, doi: 10.1109/TCSVT.2015.2496820. [63] K. Simonyan and A. Zisserman, “Very deep convolutional networks for large-scale image recognition,” 3rd Int. Conf. Learn. Represent. ICLR 2015 - Conf. Track Proc., pp. 1–14, 2015. [64] L. C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille, “Semantic image segmentation with deep convolutional nets and fully connected CRFs,” 3rd Int. Conf. Learn. Represent. ICLR 2015 - Conf. Track Proc., vol. 40, no. 4, pp. 834–848, 2015. [65] J. L. Ba, J. R. Kiros, and G. E. Hinton, “Layer Normalization,” 2016. 109 [66] J. D. Martín Abadi, Paul Barham, Jianmin Chen, et al., “TensorFlow: A System for Large-Scale Machine Learning,” in 12th USENIX Symposium on Operating Systems Design and Implementation (OSDI ’16), 2016, doi: 10.1016/0076-6879(83)01039-3. [67] D. P. Kingma and J. L. Ba, “Adam: A method for stochastic optimization,” 3rd Int. Conf. Learn. Represent. ICLR 2015 - Conf. Track Proc., pp. 1–15, 2015. [68] Y. Wu, A. Kirillov, F. Massa, W.-Y. Lo, and R. Girshick, “Detectron2.” 2019. [69] J. Brooks, “COCO Annotator.” 2019. [70] S. Rogge et al., “MPEG-I Depth Estimation Reference Software,” in 2019 International Conference on 3D Immersion (IC3D), 2019, pp. 1–6, doi: 10.1109/IC3D48390.2019.8975995. [71] Mehajabin, N., Wang, Y., Tohidypour, H., Nasiopoulos, P., Pourazad, M., An Object-Based Refocusing Scheme for Light Field Video Content, DIGITAL 2021, Advances on Societal Digital Transformation, Athens, Greece, November 14, 2021 to November 18, 2021. [72] D. G. Lowe, “Object recognition from local scale-invariant features,” Proc. IEEE Int. Conf. Comput. Vis., vol. 2, pp. 1150–1157, 1999, doi: 10.1109/iccv.1999.790410. [73] N. Mehajabin, M. T. Pourazad, and P. Nasiopoulos, “An Efficient Pseudo-Sequence-Based Light Field Video Coding Utilizing View Similarities for Prediction Structure,” IEEE Trans. Circuits Syst. Video Technol., 2021. [74] A. Wafa, M. T. Pourazad, and P. Nasiopoulos, “A Deep Learning Based Spatial Super-Resolution Approach for Light Field Content,” IEEE Access, vol. 9, pp. 2080–2092, 2021, doi: 10.1109/ACCESS.2020.3046577. [75] “Digital Research Alliance of Canada.” 110 [76] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger, “Densely connected convolutional networks,” Proc. - 30th IEEE Conf. Comput. Vis. Pattern Recognition, CVPR 2017, vol. 2017-Janua, pp. 2261–2269, 2017, doi: 10.1109/CVPR.2017.243. [77] V. K. Adhikarla et al., “Towards a quality metric for dense light fields,” in Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017, vol. 2017-January, pp. 3720–3729, doi: 10.1109/CVPR.2017.396. [78] VQEG, “Final Report From the Video Quality Experts Group on the Validation of Objective Models of Video Quality Assessment,” 2004. [79] Shum, Heung-Yeung, Shing-Chow Chan, and Sing Bing Kang. Image-based rendering. Springer Science & Business Media, 2008.
Thesis/Dissertation
eng
Electrical and Computer Engineering
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Nasiopoulos, Panos
Efficient compression, human-inspired refocusing, and quality assessment of light field videos
Text