Science, Faculty of
Statistics, Department of
DSpace
UBCV
Ding, Xin
Doctor of Philosophy - PhD
University of British Columbia
Conditional generative adversarial networks (cGANs) are state-of-the-art models for synthesizing images dependent on some conditions. These conditions are usually categorical variables such as class labels. cGANs with class labels as conditions are also known as class-conditional GANs. Some modern class-conditional GANs such as BigGAN can even generate photo-realistic images. The success of cGANs has been shown in various applications. However, two weaknesses of cGANs still exist. First, image generation conditional on continuous, scalar variables (termed regression labels) has never been studied. Second, low-quality fake images still appear frequently in image synthesis with state-of-the-art cGANs, especially when training data are limited. This thesis aims to resolve the above two weaknesses of cGANs and explore the applications of cGANs in improving a lightweight model with the knowledge from a heavyweight model (i.e., knowledge distillation). First, existing empirical losses and label input mechanisms of cGANs are not suitable for regression labels, making cGANs fail to synthesize images conditional on regression labels. To solve this problem, this thesis proposes the continuous conditional generative adversarial network (CcGAN), including novel empirical losses and label input mechanisms. Moreover, even the state-of-the-art cGANs may produce low-quality images, so a subsampling method to drop these images is necessary. In this thesis, we propose a density ratio based subsampling framework for unconditional GANs. Then, we introduce its extension to the conditional image synthesis setting called cDRE-F-cSP+RS, which can effectively improve the image quality of both class-conditional GANs and CcGAN. Finally, we propose a unified knowledge distillation framework called cGAN-KD suitable for both image classification and regression (with a scalar response), where the synthetic data generated from class-conditional GANs and CcGAN are used to transfer knowledge from a teacher net to a student net, and cDRE-F-cSP+RS is applied to filter out bad-quality images. Compared with existing methods, cGAN-KD has many advantages, and it achieves state-of-the-art performance in both image classification and regression tasks.
Improved Conditional Generative Adversarial Networksfor Image Generation: Methods and Their Application inKnowledge DistillationbyXin DingB.Sc., Anhui University, 2014M.Sc., University of Illinois at Urbana-Champaign, 2016A THESIS SUBMITTED IN PARTIAL FULFILLMENTOF THE REQUIREMENTS FOR THE DEGREE OFDoctor of PhilosophyinTHE FACULTY OF GRADUATE AND POSTDOCTORALSTUDIES(Statistics)The University of British Columbia(Vancouver)August 2021© Xin Ding, 2021The following individuals certify that they have read, and recommend to the Fac-ulty of Graduate and Postdoctoral Studies for acceptance, the thesis entitled:Improved Conditional Generative Adversarial Networks for Image Gen-eration: Methods and Their Application in Knowledge Distillationsubmitted by Xin Ding in partial fulfillment of the requirements for the degree ofDoctor of Philosophy in Statistics.Examining Committee:William J. Welch, Department of StatisticsCo-supervisorZ. Jane Wang, Department of Electrical and Computer EngineeringCo-supervisorPurang Abolmaesumi, Department of Electrical and Computer EngineeringUniversity ExaminerEldad Haber, Department of Earth, Ocean and Atmospheric SciencesUniversity ExaminerAdditional Supervisory Committee Members:Benjamin Bloem-Reddy, Department of StatisticsSupervisory Committee MemberiiAbstractConditional generative adversarial networks (cGANs) are state-of-the-art mod-els for synthesizing images dependent on some conditions. These conditions areusually categorical variables such as class labels. cGANs with class labels as con-ditions are also known as class-conditional GANs. Some modern class-conditionalGANs such as BigGAN can even generate photo-realistic images. The successof cGANs has been shown in various applications. However, two weaknessesof cGANs still exist. First, image generation conditional on continuous, scalarvariables (termed regression labels) has never been studied. Second, low-qualityfake images still appear frequently in image synthesis with state-of-the-art cGANs,especially when training data are limited. This thesis aims to resolve the abovetwo weaknesses of cGANs and explore the applications of cGANs in improving alightweight model with the knowledge from a heavyweight model (i.e., knowledgedistillation).First, existing empirical losses and label input mechanisms of cGANs are notsuitable for regression labels, making cGANs fail to synthesize images conditionalon regression labels. To solve this problem, this thesis proposes the continuousconditional generative adversarial network (CcGAN), including novel empiricallosses and label input mechanisms.Moreover, even the state-of-the-art cGANs may produce low-quality images,so a subsampling method to drop these images is necessary. In this thesis, wepropose a density ratio based subsampling framework for unconditional GANs.Then, we introduce its extension to the conditional image synthesis setting calledcDRE-F-cSP+RS, which can effectively improve the image quality of both class-conditional GANs and CcGAN.iiiFinally, we propose a unified knowledge distillation framework called cGAN-KD suitable for both image classification and regression (with a scalar response),where the synthetic data generated from class-conditional GANs and CcGAN areused to transfer knowledge from a teacher net to a student net, and cDRE-F-cSP+RS is applied to filter out bad-quality images. Compared with existing meth-ods, cGAN-KD has many advantages, and it achieves state-of-the-art performancein both image classification and regression tasks.ivLay SummaryConditional generative adversarial networks (cGANs) are deep learning modelsaiming to synthesize images given some conditions. For example, we can train acGAN to produce novel and diverse dog images given the dog class as the condi-tion. However, existing cGANs cannot generate images given continuous, scalarconditions (termed regression labels) such as ages, and low-quality images oftenappear in the image generation. In this thesis, we first propose the continuous con-ditional generative adversarial network (CcGAN), suitable for regression labels.Then, we develop a subsampling framework to drop low-quality images generatedfrom cGANs. Finally, all these techniques are applied to generate high-qualityfake images, which are then used to improve the prediction accuracy of some smallmodels in image classification and regression tasks.vPrefaceThis thesis is an original, independent work written by Xin Ding under the super-vision of Dr. William J. Welch and Dr. Z. Jane Wang. It is based on a collection ofmanuscripts that have been published in or submitted to international journals orconferences.Chapter 2 is based on the following manuscripts:• X. Ding, Y. Wang, Z. Xu, W. J. Welch, and Z. J. Wang, “CcGAN:ContinuousConditional Generative Adversarial Networks for Image Generation,” in In-ternational Conference on Learning Representations, 2021.• X. Ding, Y. Wang, Z. Xu, W. J. Welch, and Z. J. Wang, “Continuous Condi-tional Generative Adversarial Networks for Image Generation: Novel Lossesand Label Input Mechanisms,” arXiv preprint arXiv:2011.07466, 2020.The author is responsible for the problem formulation, methodology design, theo-retical analysis, experimental design, data preprocessing, coding, and paper writ-ing. This work was conducted with the guidance and editorial input from Dr.William J. Welch and Dr. Z. Jane Wang. Yongwei Wang participated in themethodology design, experimental design, coding, and paper writing. YongweiWang generated the Cell-200 and RC-49 datasets used in the experiments. ZuhengXu derived some important lemmas and theorems in the theoretical analysis.Chapter 3 is based on the following manuscript:• X. Ding, Z. J. Wang, and W. J. Welch, “Subsampling Generative AdversarialNetworks: Density Ratio Estimation in Feature Space with Softplus Loss,”IEEE Transactions on Signal Processing, vol. 68, pp. 1910–1922, 2020.viThe author is responsible for the problem formulation, methodology design, the-oretical analysis, experimental design, coding, and paper writing. This work wasconducted with the guidance and editorial input from Dr. William J. Welch andDr. Z. Jane Wang. Zuheng Xu provided valuable suggestions to the theoreticalanalysis.Chapter 4 is based on the following manuscript:• X. Ding, Y. Wang, Z. J. Wang, and W. J. Welch, “Efficient Subsampling forGenerating High-quality Images from Conditional Generative AdversarialNetworks,” arXiv preprint arXiv:2103.11166, 2021.The author is responsible for the problem formulation, methodology design, the-oretical analysis, experimental design, coding, and paper writing. This work wasconducted with the guidance and editorial input from Dr. William J. Welch and Dr.Z. Jane Wang. Yongwei Wang participated in the experimental design and paperwriting.Chapter 5 is based on the following manuscript:• X. Ding, Y. Wang, Z. Xu, Z. J. Wang, and W. J. Welch, “Distilling and Trans-ferring Knowledge via cGAN-generated Samples for Image Classificationand Regression,” arXiv preprint arXiv:2104.03164, 2021.The author is responsible for the problem formulation, methodology design, the-oretical analysis, experimental design, coding, and paper writing. This work wasconducted with the guidance and editorial input from Dr. William J. Welch and Dr.Z. Jane Wang. Yongwei Wang participated in the methodology design and paperwriting. Zuheng Xu participated in the theoretical analysis.viiTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xixGlossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxviAcknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxixDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxx1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Generative Adversarial Networks . . . . . . . . . . . . . . . . . . 21.1.1 Vanilla Generative Adversarial Networks . . . . . . . . . 31.1.2 Conditional Generative Adversarial Networks . . . . . . . 71.1.3 Other Variants of Vanilla Generative Adversarial Networks 81.1.4 Existing Metrics for Evaluating Fake Images From GANs 101.1.5 Density Ratio Based Subsampling Methods for GANs . . 121.2 Density Ratio Estimation for Image Data . . . . . . . . . . . . . 15viii1.3 Knowledge Distillation in Image Classification and Regression . . 171.4 Research Contributions and Organization of Thesis . . . . . . . . 222 CcGAN: Continuous Conditional Generative Adversarial Networks 262.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262.2 From cGAN to CcGAN . . . . . . . . . . . . . . . . . . . . . . . 282.2.1 Solution to (P1): Reformulated Empirical Losses . . . . . 292.2.2 Solutions to (P2): Two Novel Label Input Mechanisms . . 352.3 Sliding Fre´chet Inception Distance . . . . . . . . . . . . . . . . . 392.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 402.4.1 Circular 2-D Gaussians . . . . . . . . . . . . . . . . . . . 412.4.2 RC-49 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 422.4.3 UTKFace . . . . . . . . . . . . . . . . . . . . . . . . . . 452.4.4 Cell-200 . . . . . . . . . . . . . . . . . . . . . . . . . . 522.4.5 Steering Angle . . . . . . . . . . . . . . . . . . . . . . . 522.4.6 Effectiveness of SFID . . . . . . . . . . . . . . . . . . . 572.5 Theoretical Analysis for HVDL and SVDL . . . . . . . . . . . . 612.5.1 Some Necessary Definitions and Notations . . . . . . . . 622.5.2 Some Necessary Assumptions . . . . . . . . . . . . . . . 642.5.3 Some Necessary Lemmas and Theorems . . . . . . . . . 652.5.4 Error Bounds of D Trained with HVDL and SVDL . . . . 742.5.5 Interpretation of Theorems 2.5.5 and 2.5.6 . . . . . . . . . 762.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 773 Subsampling Generative Adversarial Networks: Density Ratio Esti-mation in Feature Space with Softplus Loss . . . . . . . . . . . . . . 783.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 783.2 Preliminary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 813.2.1 Sampling-Importance Resampling . . . . . . . . . . . . . 813.2.2 Fitting Density Ratio Models Under Bregman Divergence 813.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 823.3.1 Softplus Loss Function for Density Ratio Estimation . . . 823.3.2 Error Bound . . . . . . . . . . . . . . . . . . . . . . . . 84ix3.3.3 Density Ratio Estimation in Feature Space . . . . . . . . 883.3.4 Application of DRE-F-SP in Subsampling GANs . . . . . 913.4 Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 943.4.1 Mixture of 25 2-D Gaussians . . . . . . . . . . . . . . . . 943.4.2 CIFAR-10 Dataset . . . . . . . . . . . . . . . . . . . . . 993.4.3 Reduced MNIST Dataset . . . . . . . . . . . . . . . . . . 1063.4.4 CelebA Dataset . . . . . . . . . . . . . . . . . . . . . . . 1103.5 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1164 Efficient Subsampling for Generating High-Quality Images from Con-ditional Generative Adversarial Networks . . . . . . . . . . . . . . . 1194.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1194.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1214.2.1 Conditional Density Ratio Estimation in Feature Space withConditional Softplus Loss . . . . . . . . . . . . . . . . . 1214.2.2 Error Bound . . . . . . . . . . . . . . . . . . . . . . . . 1244.2.3 cDRE-F-cSP+RS: A cDRE-based Rejection Sampling Schemefor Conditional GANs . . . . . . . . . . . . . . . . . . . 1274.2.4 A Filtering Scheme for Subsampling CcGANs . . . . . . 1274.3 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1294.3.1 CIFAR-10 . . . . . . . . . . . . . . . . . . . . . . . . . . 1304.3.2 Tiny-ImageNet . . . . . . . . . . . . . . . . . . . . . . . 1314.3.3 RC-49 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1354.3.4 UTKFace . . . . . . . . . . . . . . . . . . . . . . . . . . 1414.4 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1465 Distilling and Transferring Knowledge via cGAN-generated Sam-ples for Image Classification and Regression . . . . . . . . . . . . . 1475.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1475.2 Proposed Method . . . . . . . . . . . . . . . . . . . . . . . . . . 1515.2.1 Problem Formulation . . . . . . . . . . . . . . . . . . . . 1525.2.2 The Workflow of cGAN-KD . . . . . . . . . . . . . . . . 1525.2.3 M1: Drop Low-quality Fake Samples . . . . . . . . . . . 153x5.2.4 M2: Distill Knowledge via Label Adjustment . . . . . . . 1545.2.5 M3: Transfer Knowledge via Data Augmentation . . . . . 1555.3 Theoretical Anlaysis . . . . . . . . . . . . . . . . . . . . . . . . 1555.4 Advantages of cGAN-KD . . . . . . . . . . . . . . . . . . . . . . 1615.4.1 A Unified Knowledge Distillation Framework for ImageClassification and Regression . . . . . . . . . . . . . . . 1615.4.2 Compatibility with State-of-the-art KD Methods . . . . . 1615.4.3 Suitability for Limited Labeled Data Scenarios . . . . . . 1615.4.4 Architecture-invariance . . . . . . . . . . . . . . . . . . 1625.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1625.5.1 CIFAR-10 . . . . . . . . . . . . . . . . . . . . . . . . . . 1625.5.2 Tiny-ImageNet . . . . . . . . . . . . . . . . . . . . . . . 1665.5.3 RC-49 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1715.5.4 UTKFace . . . . . . . . . . . . . . . . . . . . . . . . . . 1725.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1736 Conclusion and Future Work . . . . . . . . . . . . . . . . . . . . . . 1776.1 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1776.2 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180A Supporting Materials . . . . . . . . . . . . . . . . . . . . . . . . . . 198A.1 GitHub Repositories . . . . . . . . . . . . . . . . . . . . . . . . 198A.2 More Details of the Circular 2-D Gaussians Experiment in Section2.4.1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199A.2.1 Network Architectures . . . . . . . . . . . . . . . . . . . 199A.2.2 Training Setups . . . . . . . . . . . . . . . . . . . . . . . 199A.2.3 Testing Setups . . . . . . . . . . . . . . . . . . . . . . . 200A.2.4 Varying Number of Gaussians for Training Data Generation 200A.3 More Details of the RC-49 Experiment in Section 2.4.2 . . . . . . 202A.3.1 Description of RC-49 . . . . . . . . . . . . . . . . . . . . 202A.3.2 Network Architectures . . . . . . . . . . . . . . . . . . . 202A.3.3 Training Setups . . . . . . . . . . . . . . . . . . . . . . . 203xiA.3.4 Testing Setups . . . . . . . . . . . . . . . . . . . . . . . 203A.3.5 Performance Measures . . . . . . . . . . . . . . . . . . . 203A.3.6 Extra Experiments . . . . . . . . . . . . . . . . . . . . . 205A.4 More Details of the Experiment on UTKFace in Section 2.4.3 . . . 206A.4.1 Description of the UTKFace Dataset . . . . . . . . . . . . 206A.4.2 Network Architectures . . . . . . . . . . . . . . . . . . . 208A.4.3 Training Setups . . . . . . . . . . . . . . . . . . . . . . . 209A.4.4 Performance Measures . . . . . . . . . . . . . . . . . . . 209A.4.5 Extra Experiments . . . . . . . . . . . . . . . . . . . . . 210A.5 More Details of the Experiment on Cell-200 in Section 2.4.4 . . . 211A.5.1 Description of Cell-200 . . . . . . . . . . . . . . . . . . . 211A.5.2 Network Architectures . . . . . . . . . . . . . . . . . . . 211A.5.3 Training Setups . . . . . . . . . . . . . . . . . . . . . . . 212A.5.4 Testing Setups . . . . . . . . . . . . . . . . . . . . . . . 214A.5.5 Performance Measures . . . . . . . . . . . . . . . . . . . 214A.5.6 Extra Experiments . . . . . . . . . . . . . . . . . . . . . 215A.6 More Details of the Experiment on the Steering Angle Dataset inSection 2.4.5 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216A.6.1 Description of Steering Angle . . . . . . . . . . . . . . . 216A.6.2 Network Architectures . . . . . . . . . . . . . . . . . . . 217A.6.3 Training Setups . . . . . . . . . . . . . . . . . . . . . . . 217A.6.4 Testing Setups . . . . . . . . . . . . . . . . . . . . . . . 218A.6.5 Performance Measures . . . . . . . . . . . . . . . . . . . 218A.6.6 Extra Experiments . . . . . . . . . . . . . . . . . . . . . 219A.7 More Details of the 2D-Gaussians Experiment in Section 3.4.1 . . 221A.7.1 Network Architectures . . . . . . . . . . . . . . . . . . . 221A.7.2 Training Setups . . . . . . . . . . . . . . . . . . . . . . . 221A.7.3 How Does the Loss Function Affect Density Ratio Estima-tion and the Subsequent Subsampling? . . . . . . . . . . 222A.8 More Details of the Experiment on CIFAR-10 in Section 3.4.2 . . 224A.8.1 Network Architectures . . . . . . . . . . . . . . . . . . . 224A.8.2 Training Setups . . . . . . . . . . . . . . . . . . . . . . . 225A.8.3 Performance Measures . . . . . . . . . . . . . . . . . . . 225xiiA.9 More Details of the Experiment on MNIST in Section 3.4.3 . . . . 226A.9.1 Network Architectures . . . . . . . . . . . . . . . . . . . 226A.9.2 Training Setups . . . . . . . . . . . . . . . . . . . . . . . 229A.9.3 Performance Measures . . . . . . . . . . . . . . . . . . . 233A.10 More Details of the Experiment on CelebA in Section 3.4.4 . . . . 234A.10.1 Network Architectures . . . . . . . . . . . . . . . . . . . 234A.10.2 Training Setups . . . . . . . . . . . . . . . . . . . . . . . 234A.10.3 Performance Measures . . . . . . . . . . . . . . . . . . . 234A.11 More Details of Experiments on CIFAR-10 in Section 4.3.1 . . . . 235A.11.1 Network architectures . . . . . . . . . . . . . . . . . . . 235A.11.2 Training setups . . . . . . . . . . . . . . . . . . . . . . . 235A.11.3 Testing setups . . . . . . . . . . . . . . . . . . . . . . . . 236A.12 More Details of Experiments on Tiny-ImageNet in Section 4.3.2 . 237A.12.1 Network architectures . . . . . . . . . . . . . . . . . . . 237A.12.2 Training setups . . . . . . . . . . . . . . . . . . . . . . . 237A.12.3 Testing setups . . . . . . . . . . . . . . . . . . . . . . . . 238A.13 More Details of Experiments on RC-49 in Section 4.3.3 . . . . . . 238A.13.1 Network architectures . . . . . . . . . . . . . . . . . . . 238A.13.2 Training setups . . . . . . . . . . . . . . . . . . . . . . . 238A.13.3 Testing setups . . . . . . . . . . . . . . . . . . . . . . . . 240A.14 More Details of Experiments on UTKFace in Section 4.3.4 . . . . 241A.14.1 Network architectures . . . . . . . . . . . . . . . . . . . 241A.14.2 Training setups . . . . . . . . . . . . . . . . . . . . . . . 241A.14.3 Testing setups . . . . . . . . . . . . . . . . . . . . . . . . 242A.15 More Details of Experiments on CIFAR-10 in Section 5.5.1 . . . . 242A.15.1 Experimental Setups . . . . . . . . . . . . . . . . . . . . 242A.15.2 Visual Quality versus the Error Between Predicted and As-signed Labels . . . . . . . . . . . . . . . . . . . . . . . . 243A.16 More Details of Experiments on Tiny-ImageNet in Section 5.5.2 . 244A.16.1 Experimental Setups . . . . . . . . . . . . . . . . . . . . 244A.17 More Details of Experiments on RC-49 in Section 5.5.3 . . . . . . 247A.17.1 Experimental Setups . . . . . . . . . . . . . . . . . . . . 247xiiiA.17.2 Visual Quality versus the Error Between Predicted and As-signed Labels . . . . . . . . . . . . . . . . . . . . . . . . 248A.18 More Details of Experiments on UTKFace in Section 5.5.4 . . . . 249A.18.1 Experimental Setups . . . . . . . . . . . . . . . . . . . . 249xivList of TablesTable 1.1 The network architectures for the generator and the discrimina-tor in a vanilla GAN. . . . . . . . . . . . . . . . . . . . . . . . 5Table 1.2 The network architectures for the generator and the discrimina-tor in an example cGAN. . . . . . . . . . . . . . . . . . . . . 9Table 2.1 Average quality of 36,000 fake samples from cGAN and CcGAN. 42Table 2.2 Average quality of 179,800 fake RC-49 images from cGAN andCcGAN. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45Table 2.3 Average quality of 60,000 fake UTKFace images from cGANand CcGAN. . . . . . . . . . . . . . . . . . . . . . . . . . . . 50Table 2.4 Average quality of 200,000 fake Cell-200 images from cGANand CcGAN. . . . . . . . . . . . . . . . . . . . . . . . . . . . 54Table 2.5 Average quality of 100,000 fake Steering Angle images fromcGAN and CcGAN. . . . . . . . . . . . . . . . . . . . . . . . 59Table 2.6 Evaluation results of SFID on RC-49 under different setups ofrSFID and number of cSFID’s. . . . . . . . . . . . . . . . . . . . 61Table 3.1 Average quality of 10,000 fake synthetic samples from differentsubsampling methods over three repetitions. . . . . . . . . . . 96Table 3.2 Ablation study on synthetic data. . . . . . . . . . . . . . . . . 98Table 3.3 Average quality of 50,000 fake CIFAR-10 images from differ-ent subsampling methods over three repetitions. . . . . . . . . 104Table 3.4 Ablation study 1 on CIFAR-10. . . . . . . . . . . . . . . . . . 106Table 3.5 Ablation study 2 on CIFAR-10. . . . . . . . . . . . . . . . . . 107xvTable 3.6 Average quality of 50,000 fake MNIST images from differentsubsampling methods over three repetitions. . . . . . . . . . . 108Table 3.7 Average quality of 50,000 fake CelebA images from differentsubsampling methods over three repetitions. . . . . . . . . . . 116Table 4.1 CIFAR-10 dataset: Quality of 50,000 fake images (5000 perclass) sampled from different sampling methods on three trainedGAN models. . . . . . . . . . . . . . . . . . . . . . . . . . . 131Table 4.2 Tiny-ImageNet dataset: Quality of one million fake images (5000per class) sampled from two sampling methods on a trained Big-GAN model. . . . . . . . . . . . . . . . . . . . . . . . . . . . 135Table 4.3 RC-49 dataset: Quality of 179,800 fake images (200 per an-gle) sampled from two sampling methods on three trained GANmodels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141Table 4.4 RC-49 dataset: Influence of the MAE quantile in the filteringalgorithm (Alg. 11) on the performance of cDRE-F-cSP+RS insubsampling SR-25. . . . . . . . . . . . . . . . . . . . . . . . 141Table 4.5 UTKFace dataset: Quality of 60,000 fake images (1000 per age)sampled from different sampling methods on the SU-200 Cc-GAN model. . . . . . . . . . . . . . . . . . . . . . . . . . . . 146Table 5.1 CIFAR-10: Test error rate comparisons among NOKD, BLKD[53], TAKD [92], and our methods (Mg = 105 and ρ = 0.9,0.6,0.7for three settings respectively). . . . . . . . . . . . . . . . . . 165Table 5.2 CIFAR-10: Test error rate comparisons between SSKD [159]and cGAN-KD+SSKD (Mg = 105 and ρ = 0.9,0.6,0.7 for threesettings respectively). . . . . . . . . . . . . . . . . . . . . . . 166Table 5.3 CIFAR-10: Test error rate comparisons between BLKD+UDA[118] and cGAN-KD+BLKD+UDA (Mg = 105 and ρ = 0.9,0.6,0.7for three settings respectively). . . . . . . . . . . . . . . . . . 167Table 5.4 CIFAR-10: Test error rates of the student in each setting-studentcombination before and after applying the best KD method. . . 168xviTable 5.5 Tiny-ImageNet: Test error rate comparisons among NOKD,BLKD [53], TAKD [92], and our methods (Mg = 105 and ρ =0.5). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 168Table 5.6 Tiny-ImageNet: Test error rate comparisons between SSKD[159] and cGAN-KD+SSKD (Mg = 105 and ρ = 0.5). . . . . . 169Table 5.7 Tiny-ImageNet: Test error rate comparisons between BLKD+UDA[118] and cGAN-KD+BLKD+UDA (Mg = 105 and ρ = 0.5). . 170Table 5.8 Tiny-ImageNet: Test error rates of each student before and afterapplying the best KD method. . . . . . . . . . . . . . . . . . . 171Table 5.9 RC-49: Test MAE comparisons between NOKD and cGAN-KD (Mg = 5× 104 and ρ = 0.5,0.8,0.9 for three settings re-spectively). . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173Table 5.10 UTKFace: Test MAE comparisons between NOKD and cGAN-KD (Mg = 8×104 and ρ = 0.7). . . . . . . . . . . . . . . . . 174Table A.1 Network architectures for the generator and discriminator ofcGAN in the simulation. . . . . . . . . . . . . . . . . . . . . . 199Table A.2 Network architectures for the generator and discriminator of ourproposed CcGAN in the simulation. . . . . . . . . . . . . . . 200Table A.3 Network architectures for the generator and discriminator in thesimulation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221Table A.4 Architecture of the 5-layer MLP for DRE in the simulation. . . 221Table A.5 Setups for training the 5-layer MLP under different loss func-tions in the simulation. . . . . . . . . . . . . . . . . . . . . . . 222Table A.6 Hyperparameter selection in Simulation. . . . . . . . . . . . . 222Table A.7 Network architectures for the generator and discriminator ofDCGAN and WGAN-GP in the experiment on CIFAR-10. . . . 229Table A.8 Architecture of the ResNet-34 for feature extraction for CIFAR-10. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230Table A.9 5-layer MLP for DRE in feature space for CIFAR-10. . . . . . 230Table A.10 2-layer CNN for DRE in pixel space [98] for CIFAR-10 . . . . 230Table A.11 6-layer CNN for DRE in pixel space [63] for CIFAR-10 . . . . 231Table A.12 Binary classifier for the BOC-based DRE [47] for CIFAR-10 . 231xviiTable A.13 Training setups for three types of GAN on CIFAR-10. . . . . . 231Table A.14 Training setups for DRE methods on CIFAR-10. . . . . . . . . 232Table A.15 Hyperparameter selection for CIFAR-10. . . . . . . . . . . . . 232Table A.16 Network architectures for the generator and the discriminator ofDCGAN in the experiment on the MNIST. . . . . . . . . . . . 233Table A.17 Hyperparameter selection for MNIST. . . . . . . . . . . . . . 233Table A.18 Hyperparameter selection for CelebA. . . . . . . . . . . . . . 234Table A.19 The 5-layer MLP for DRE in feature space for CIFAR-10. . . . 235Table A.20 The 5-layer MLP for cDRE in feature space for CIFAR-10. . . 236Table A.21 5-layer MLP for cDRE in feature space for Tiny-ImageNet. . . 237Table A.22 The architecture of the encoder in the sparse autoencoder forextracting features from 64×64 RGB images. . . . . . . . . . 239Table A.23 The architecture of the decoder in the sparse autoencoder forreconstructing 64×64 input images from extracted features. . . 239Table A.24 The architecture of the label prediction branch in the sparse au-toencoder for 64×64 images. . . . . . . . . . . . . . . . . . . 240Table A.25 CIFAR-10: Comparison on the model size, inference speed, andtest error rate between popular convolutional neural networks. . 244Table A.26 CIFAR-10: The grid search results for selecting the optimal ρof M1 in three settings. . . . . . . . . . . . . . . . . . . . . . 245Table A.27 Tiny-ImageNet: Comparison on the model size, inference speed,and test error rate between popular convolutional neural networks.246Table A.28 Tiny-ImageNet: The grid search results for selecting the opti-mal ρ of M1. . . . . . . . . . . . . . . . . . . . . . . . . . . . 246Table A.29 RC-49: Comparison on the model size, inference speed, andtest MAE between popular convolutional neural networks. . . . 249Table A.30 RC-49: The grid search results for selecting the optimal ρ of M1.250Table A.31 UTKFace: Comparison on the model size, inference speed, andtest MAE between popular convolutional neural networks. . . . 252Table A.32 UTKFace: The grid search results for selecting the optimal ρ ofM1. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 252xviiiList of FiguresFigure 1.1 The workflow of a vanilla GAN. . . . . . . . . . . . . . . . . 3Figure 1.2 The workflow of a vanilla cGAN. . . . . . . . . . . . . . . . 7Figure 1.3 Four fake images with high quality generated by BigGAN [14]. 13Figure 1.4 An unrealistic fake image generated by BigGAN [14]. . . . . 13Figure 1.5 The generic teacher-student framework for knowledge distilla-tion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18Figure 1.6 An example of the hard and soft labels of a dog image in animage classification task with three classes. . . . . . . . . . . 20Figure 1.7 The workflow of the baseline knowledge distillation (BLKD)[53]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Figure 1.8 The workflow of teacher assistant knowledge distillation (TAKD)[92]. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Figure 1.9 The connections between Chapters 2 to 5. . . . . . . . . . . . 25Figure 2.1 A visual illustration of HVE (Eq. (2.6)) and SVE (Eq. (2.7)). . 32Figure 2.2 The naive label input mechanism (NLI) for G of CcGAN. . . 37Figure 2.3 The naive label input mechanism (NLI) for D of CcGAN. . . 37Figure 2.4 The pre-trained CNN T1 + T2 for label embedding. The firstsubnetwork T1 consists of some convolutional layers (Conv.)and some linear layers. The second subnetwork T2 includesone linear layer. . . . . . . . . . . . . . . . . . . . . . . . . . 38Figure 2.5 The label embedding network is a multilayer perceptron (MLP). 38Figure 2.6 The improved label input mechanism (ILI) for G of CcGAN. . 38Figure 2.7 The improved label input mechanism (ILI) for D of CcGAN. . 38xixFigure 2.8 Visualization of Sliding Fre´chet Inception Distance (SFID). . 39Figure 2.9 Visual results for the Circular 2-D Gaussians simulation. . . . 43Figure 2.10 Three RC-49 example images for each of 6 angles. . . . . . . 46Figure 2.11 Line graphs of FID/NIQE/Diversity/Lable Score versus yawangle for RC-49. . . . . . . . . . . . . . . . . . . . . . . . . 47Figure 2.12 Line graphs of Intra-FID versus the sample size for each dis-tinct training angle of RC-49. . . . . . . . . . . . . . . . . . 48Figure 2.13 Three UTKFace example images for each of 6 ages. . . . . . 49Figure 2.14 Line graphs of FID/NIQE/Diversity/Lable Score versus Agefor UTKFace. . . . . . . . . . . . . . . . . . . . . . . . . . . 50Figure 2.15 Line graphs of Intra-FID versus the maximum sample size foreach distinct age in the training set of UTKFace. . . . . . . . 51Figure 2.16 Three Cell-200 images for each of 6 cell counts absent in thetraining data. . . . . . . . . . . . . . . . . . . . . . . . . . . 53Figure 2.17 Line graphs of FID Score versus Cell Count for Cell-200. . . . 54Figure 2.18 Line graphs of NIQE Score versus Cell Count for Cell-200. . 55Figure 2.19 Line graphs of Label Score versus Cell Count for Cell-200. . . 56Figure 2.20 Three Steering Angle images for each of 6 angles. . . . . . . 58Figure 2.21 Line graphs of FID/NIQE/Diversity/Lable Score versus SFIDCenter for the Steering Angle dataset. . . . . . . . . . . . . . 59Figure 3.1 The workflow of our proposed density ratio model ψ(h;β ) inthe feature space in Eq. (3.21). . . . . . . . . . . . . . . . . . 91Figure 3.2 Workflow of density ratio based subsampling for GANs. . . . 92Figure 3.3 Visual results of the 25 2-D Gaussians example. . . . . . . . . 97Figure 3.4 Training curves of a 5-layer MLP under the penalized uLSIFloss (1.16) and the penalized SP loss (3.10) when λ = 0.05 inthe 25 2-D Gaussians example. . . . . . . . . . . . . . . . . . 100Figure 3.5 The density ratio estimation and subsampling performance ofa 5-layer MLP trained under the penalized uLSIF loss (1.16)and the penalized SP loss (3.10) when λ = 0.05 in the 25 2-DGaussians example. . . . . . . . . . . . . . . . . . . . . . . . 101xxFigure 3.6 Fake CIFAR-10 images (car and horse) from DCGAN (PartI). For the “car” and “horse” classes, our proposed methodsperform better than three baselines. . . . . . . . . . . . . . . 105Figure 3.6 Fake CIFAR-10 images (car and horse) from DCGAN (PartII). For the “car” and “horse” classes, our proposed methodsperform better than three baselines. . . . . . . . . . . . . . . 108Figure 3.7 Fake CIFAR-10 images (car and horse) from WGAN-GP (PartI). For the “car” and “horse” classes, our proposed methodsperform better than three baselines. . . . . . . . . . . . . . . 109Figure 3.8 Fake CIFAR-10 images (car and horse) from WGAN-GP (PartII). For the “car” and “horse” classes, our proposed methodsperform better than three baselines. . . . . . . . . . . . . . . 110Figure 3.9 Fake CIFAR-10 images (car and horse) from MMD-GAN. Forthe “car” and “horse” classes, our proposed methods still per-form well when DRS and MH-GAN are not applicable. . . . . 111Figure 3.10 Inception Score (IS) and Frechet Inception Distance (FID) ver-sus the epoch of DCGAN training. . . . . . . . . . . . . . . . 112Figure 3.11 Fake MNIST images from DCGAN: Part 1. Many fake im-ages from No subsampling, DRS, and MH-GAN are not rec-ognizable, while most images from our proposed methods arerecognizable. . . . . . . . . . . . . . . . . . . . . . . . . . . 113Figure 3.11 Fake MNIST images from DCGAN: Part 2. Many fake im-ages from No subsampling, DRS, and MH-GAN are not rec-ognizable, while most images from our proposed methods arerecognizable. . . . . . . . . . . . . . . . . . . . . . . . . . . 114Figure 3.11 Fake MNIST images from DCGAN: Part 3. Many fake im-ages from No subsampling, DRS, and MH-GAN are not rec-ognizable, while most images from our proposed methods arerecognizable. . . . . . . . . . . . . . . . . . . . . . . . . . . 115Figure 3.12 Fake CelebA images from SNGAN (yellow outlines indicatesamples with obvious distortions and asymmetry) (Part I). Ourproposed methods have fewer unrealistic images (yellow out-lines) than baselines do. . . . . . . . . . . . . . . . . . . . . 117xxiFigure 3.12 Fake CelebA images from SNGAN (yellow outlines indicatesamples with obvious distortions and asymmetry) (Part II). Ourproposed methods have fewer unrealistic images (yellow out-lines) than baselines do. . . . . . . . . . . . . . . . . . . . . 118Figure 4.1 The architecture of the sparse autoencoder for feature extrac-tion in subsampling CcGANs. . . . . . . . . . . . . . . . . . 122Figure 4.2 The workflow of cDRE-F-cSP+RS has two modules: cDRE-F-cSP and rejection sampling. . . . . . . . . . . . . . . . . . 127Figure 4.3 CIFAR-10: FID versus class plots in sampling the BC-10kclass-conditional GAN model. . . . . . . . . . . . . . . . . . 132Figure 4.4 Example CIFAR-10 images for the “car” class in the BC-10ksetting. We observe numerous images (marked in red boxes)from the no-subsampling method, which we can hardly recog-nize as cars. By contrast, there appear much less such low-quality examples after subsampling by 10-DRE-F-SP+RS andthe proposed cDRE-F-cSP+RS methods. . . . . . . . . . . . . 133Figure 4.5 Example CIFAR-10 images for the “horse” class in the BC-10ksetting. We observe numerous images (marked in red boxes)from the no-subsampling method, which we can hardly recog-nize as cars. By contrast, there appear much less such low-quality examples after subsampling by 10-DRE-F-SP+RS andthe proposed cDRE-F-cSP+RS methods. . . . . . . . . . . . . 134Figure 4.6 Example Tiny-ImageNet images for the “goldfish” class. Asmarked by red boxes, such “goldfish” images from the no-subsampling method are of very low quality. Also, we can ob-serve some replicated samples for samples from the no-subsamplingmethod. Even for images that look like “goldfish”, such im-ages appear to look “darker” and less colorful than real im-ages. Compared to the no-subsampling method, the proposedmethod generates significantly improved image quality, i.e.,our image samples are both more diverse and look much morenatural those from the baseline method. . . . . . . . . . . . . 136xxiiFigure 4.7 Example Tiny-ImageNet images for the “tower” class. We canobserve fake image samples from the no-subsampling methodgenerally look less appealing than those from the proposedmethod. Moreover, for some images (marked in red boxes)from the no-subsampling method, the tower is actually miss-ing in those “tower” images. However, we can clearly see thetower in images that are sampled from our method. . . . . . . 137Figure 4.8 Example RC-49 images for angles from 4.5 degree to 85.5 de-gree (from top to bottom) in the SR-5 setting. We observethat the no-subsampling method generates many image sam-ples that suffer from the mode collapse issue. By contrast,we have largely alleviated this issue in the proposed cDRE-F-cSP+RS subsampling method. . . . . . . . . . . . . . . . . . 139Figure 4.9 Line graphs of Diversity/Label Score versus regression labelsin subsampling the SR-5 CcGAN model. . . . . . . . . . . . 140Figure 4.10 The distribution of the 49 chair types within 200 fake imagesgenerated from no-subsampling and cDRE-F-cSP+RS at eachof four randomly selected angles in the SR-5 setting (Part I). . 142Figure 4.10 The distribution of the 49 chair types within 200 fake imagesgenerated from no-subsampling and cDRE-F-cSP+RS at eachof four randomly selected angles in the SR-5 setting (Part II). . 143Figure 4.11 Example UTKFace images for ages from 3 to 57 (from topto bottom). As marked in red boxes in images from the no-subsampling method, we observe that, occasionally some faceimages indeed have clearly wrong ages. For example, in thesecond row, two marked face images at an age 9 look clearlymore mature than they should be. Such image samples giverise to the label inconsistency issue in CcGANs. By contrast,we can hardly observe such label inconsistency issue for im-ages from the proposed method. . . . . . . . . . . . . . . . . 145Figure 5.1 The workflow of cGAN-KD. . . . . . . . . . . . . . . . . . . 153Figure 5.2 Evolution of fake sample datasets and their distributions. . . . 156xxiiiFigure 5.3 CIFAR-10: An ablation study of the subsampling, filtering,and label adjustment modules in the C-10K setting. . . . . . . 169Figure 5.4 CIFAR-10: Analysis on the effect of number of processed fakeimages (i.e., Mg) in the C-10K setting. . . . . . . . . . . . . . 170Figure 5.5 RC-49: An ablation study of the subsampling, filtering, andlabel adjustment modules in the R-5 setting. . . . . . . . . . . 174Figure 5.6 RC-49: Analysis on the effect of number of processed fakeimages (i.e., Mg) in the R-5 setting. . . . . . . . . . . . . . . 175Figure 6.1 Example hand images with keypoint annotations from [129]. . 179Figure A.1 Line graphs of 2-Wasserstein Distance (log scale) versus thenumber of Gaussians for training data generation. . . . . . . . 201Figure A.2 Some example RC-49 fake images from the four CcGAN meth-ods. We fix the noise z but vary the label y. . . . . . . . . . . 205Figure A.3 Some example RC-49 fake images from a degenerated NLI-based CcGAN. . . . . . . . . . . . . . . . . . . . . . . . . . 207Figure A.4 Example RC-49 fake images from cGAN when we bin the yawangle range into different number of classes. . . . . . . . . . . 208Figure A.5 The histogram of the UTKFace dataset with ages varying from1 to 60. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 209Figure A.6 Some examples of generated UTKFace images from the fourCcGAN methods. We fix the noise z but vary the label y from3 to 57. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210Figure A.7 Some example UTKFace fake images from a degenerated NLI-based CcGAN. . . . . . . . . . . . . . . . . . . . . . . . . . 212Figure A.8 Example UTKFace fake images from cGAN when we bin theage range into different number of classes. . . . . . . . . . . . 213Figure A.9 Some examples of generated Cell-200 images from the fourCcGAN methods. We fix the noise z but vary the label y from10 to 200. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215Figure A.10 Example Cell-200 fake images from cGAN when we bin therange of cell count into different number of classes. . . . . . . 216xxivFigure A.11 The histogram of the Steering Angle dataset with steering an-gles varying from −80◦ to 80◦. . . . . . . . . . . . . . . . . . 217Figure A.12 Example Steering Angle images from 5 scenes, i.e., tree, tree+barrier,bush, bush+barrier, and winding mountain road (from left toright). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219Figure A.13 Some examples of generated Steering Angle images from thefour CcGAN methods. We fix the noise z but vary the label yfrom −71.8◦ to 72◦. . . . . . . . . . . . . . . . . . . . . . . 220Figure A.14 Example Steering Angle fake images from cGAN when we binthe range of steering angles into different number of classes. . 220Figure A.15 Average density ratio of 10,000 fake samples and percentageof high quality samples versus the epoch of DRE training whenλ = 0.05. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 224Figure A.16 The training curve of a 5-layer MLP under the penalized uLSIFloss (1.16) with λ = 0.05 and a constant learning rate 10−5. . 225Figure A.17 Average density ratio of 10,000 fake samples and percentageof high quality samples versus the epoch of DRE training whenλ = 0. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226Figure A.18 Average density ratio of high/low quality fake samples versusthe epoch of DRE training when λ = 0. . . . . . . . . . . . . 227Figure A.19 A residual block (ResBlock) in ResNet-34 for CIFAR-10. Pleasecheck our codes for detailed setups of each layer. . . . . . . . 228Figure A.20 CIFAR-10: A line graph of the average NIQE versus the quan-tile of the cross entropy losses between predicted and assignedclass labels in the C-50K setting. . . . . . . . . . . . . . . . . 247Figure A.21 RC-49: A line graph of the average NIQE versus the quantileof the MAE loss between predicted and assigned regressionlabels in the R-25 setting. . . . . . . . . . . . . . . . . . . . . 251xxvGlossaryBLKD baseline knowledge distillationBOC Bayes optimal classifierBR BregmanCcGAN continuous conditional generative adversarial networkcDRE conditional density ratio estimationcDRE-F-cSP conditional density ratio estimation in feature space withconditional Softplus losscGAN conditional generative adversarial networkcGAN-KD cGAN-based knowledge distillationcSP conditional SoftplusCNN convolutional neural networkDCGAN deep convolutional generative adversarial networkDRE density ratio estimationDRE-F-SP density ratio estimation in feature space with Softplus lossDRS discriminator rejection samplingERM empirical risk minimizationxxviFID Fre´chet inception distanceGAN generative adversarial networkHVDL hard vicinal discriminator lossKD knowledge distillationKDE kernel density estimationILI improved label inputIS Inception scoreMH Metropolis-Hastings algorithmMH-GAN Metropolis-Hastings GANMLP multilayer perceptronNLI naive label inputPDF probability density functionPMF probability mass functionRS rejection samplingSAGAN self-attention generative adversarial networkSFID sliding Fre´chet Inception distanceSGD stochastic gradient descentSIR sampling importance resamplingSNGAN spectral normalization generative adversarial networkSP SoftplusSSKD self-supervised learning as an auxiliary task for knowledge distillationSVDL soft vicinal discriminator lossxxviiTAKD teacher assistant knowledge distillationUDA unsupervised data augmentationuLSIF unconstrained least-squares importance fittingVRM vicinal risk minimizationWGAN Wasserstein generative adversarial networkxxviiiAcknowledgmentsFirst and foremost, I would like to express my innumerable thanks to my advisors,Dr. William J. Welch and Dr. Z. Jane Wang, for their continuous support, guid-ance, and encouragement during my doctoral research. This thesis would neverbe possible without their valuable input. I also appreciate them giving me enoughfreedom to explore research topics that attract me.I would also like to express thanks to Dr. Ben Bloem-Reddy for his valuabletime and excellent comments as the committee member.I am thankful to my wonderful collaborators, Yongwei Wang and Zuheng Xu,for their valuable efforts, suggestions, and comments.Last but not least, I am grateful to my parents for their endless love. I sincerelyappreciate them for everything they have done to support me and the best educationthey gave me.xxixDedicationThis dissertation is dedicated to my beloved mother and father.谨以此论文献给我挚爱的母亲和父亲。xxxChapter 1IntroductionGenerative adversarial networks (GANs) first introduced by [43] are well-knownand powerful deep generative models [119] for image synthesis and have beenapplied in various types of image-related tasks (e.g., image-to-image translation[168], super-resolution [70], text-to-image synthesis [75, 113], image editing [108],data augmentation for image classification [35, 56, 88, 124, 131, 136, 153, 156,161, 169], imputation of missing data [9], recoloring [77, 127], image inpainting[49, 82, 106, 157], 3D point cloud to image translation [8, 90], etc.). Conditionalgenerative adversarial networks (cGANs), as an essential family of GANs, syn-thesize images given some conditions in a generative way. These conditions areusually categorical variables such as class labels. cGANs with class labels as con-ditions are also known as class-conditional GANs. Although the success of cGANshas been demonstrated in various applications [6, 8, 9, 34, 75, 82, 90, 127, 149, 155,157], two weaknesses of cGANs still exist.1. Many cGANs have been proposed to generate images conditional on cate-gorical conditions (e.g., class labels); however, image generation conditionalon continuous, scalar variables (termed regression labels) has never beenstudied.2. To improve the quality of images generated from cGANs, most of the pre-vious works focus on improving the training algorithm, such as adoptinglarge scale training schemes [14], novel normalization methods [95], ad-1vanced GAN architectures [14, 59, 61, 112, 163], different loss functions [4,48, 76], and advanced regularization [97]. Although state-of-the-art cGANs(e.g., BigGAN [14]) are already able to generate photo-realistic images, low-quality fake images still appear frequently, especially when training data arelimited. Instead of improving the training procedure, we are more interestedin subsampling fake images generated from a trained cGAN (i.e., filteringout low-quality fake images). Unfortunately, very few works have been doneon this topic.In this thesis, we focus on the following new research directions related to theweaknesses of cGANs:1. Proposing a novel cGAN framework, termed continuous conditional gen-erative adversarial networks (CcGANs), which is able to generate imagesconditional on regression labels.2. Developing subsampling methods to filter out low-quality images gener-ated from a class-conditional GAN (conditioning on class labels) or CcGAN(conditioning on regression labels).3. Proposing a novel knowledge distillation (KD) framework suitable for bothimage classification and image regression (with a scalar response), where thesynthetic data generated from class-conditional GANs and CcGANs are usedto transfer knowledge from a teacher net to a student net and the proposedsubsampling methods are applied to filter out bad-quality images.1.1 Generative Adversarial NetworksAn image with resolution C×H ×W (number of channels × height × width)can be flattened and then modeled as a continuous random vector x with lengthC×H×W in a high dimensional spaceX . Assuming the image x follows an un-known distribution with pr(x) as probability density function (PDF), the first GANframework proposed by [43] (a.k.a. vanilla GANs) aims at estimating pr(x) interms of samples xr1,xr2, · · · ,xrNr which are randomly drawn from pr(x). (Through-out, the subscript or superscript r will denote “real” images.) Vanilla GANs allow2us to generate infinite images from the estimated distribution without explicitlyknowing pr(x) or its estimate. Vanilla GANs are then extended by [91] to the con-ditional scenario where a GAN, termed cGAN, is trained to estimate the imagePDF given some conditions. Vanilla GANs and cGANs are further improved bymany subsequent works [4, 10, 14, 48, 59–61, 76, 95, 99, 121, 138, 163, 166],where the superiority of GANs in generating images over other generative modelshas been demonstrated.1.1.1 Vanilla Generative Adversarial NetworksA vanilla GAN [43] consists of two components: a generator and a discrimina-tor. The generator and the discriminator are modeled by two neural networksG(z;θ) and D(x;φ), respectively, where θ and φ are network parameters. Thegenerator G(z;θ) takes as input a sample z ∈R128 from a simple prior q(z) (e.g.,N(0128×1, I128×128)) and outputs a fake image xg ∈X . (The dimension of z isoften set to 128 in the GAN literature.) Assume the fake image xg follows a dis-tribution with PDF pg(x). The discriminator D(x;φ) is a binary classifier whichtakes an image x ∈X as input and outputs the probability that x is real (i.e., fromthe true image distribution with PDF pr(x)). The workflow of a vanilla GAN isvisualized in Figure 1.1.real image xrnoise z G(z;θ)D(x;φ) Real/FakeDiscriminatorGeneratorFigure 1.1: The workflow of a vanilla GAN.These two neural networks are trained alternately with opposite objectives. Thediscriminator is trained to assign a high probability to a real images xr but a lowprobability to fake images xg. On the contrary, the training purpose of the generatorG(z;θ) is to make the discriminator assign a high probability to a fake image xg.The vanilla GAN loss defined by [43] for the generator and the discriminator are3shown as follows:L GAND (φ) =−Ex∼pr [logD(x;φ)]−Ez∼q [log(1−D(G(z;θ);φ))] ,L GANG (θ) =−Ez∼q [logD(G(z;θ);φ)] .(1.1)Theorem 1.1.1. Given a fixed generator G, the optimal D∗ which minimizesL GANDis shown as follows:D∗(x) =pr(x)pr(x)+ pg(x). (1.2)Proof. The proof for Theorem 1.1.1 can be found in [43].Based on Eqs. (1.1) and (1.2), we can find that a GAN model implicitly esti-mates the true data distribution’s PDF, i.e., pr(x), because if we plug the optimaldiscriminator D∗ (1.2) into theL GANG in (1.1), we can getargminGEz∼q [− logD∗(G(z))]=argminGEz∼q [log(1−D∗(G(z)))]=argminG{Ex∼pg [log(1−D∗(x))]+Ex∼pr [logD∗(x)]}(2nd term is irrelevant to G)=argminG{Ex∼pg[logpg(x)12 (pr(x)+ pg(x))]+Ex∼pr[logpr(x)12 (pr(x)+ pg(x))]−2log2}=argminG{2JS(pr(x)||pg(x))−2log2}.(1.3)From Eq. (1.3), we can see, given an optimal discriminator D∗, minimizingL GANGis equivalent to minimizing the Jensen–Shannon divergence (JS) between pr andpg. In other words, the objective of G is to implicitly estimate pr. [43] alsoproves that pg converges to pr if G is updated to optimize Eq. (1.3) and some othernecessary conditions hold.Two simple network architectures are used by [43] for the generator G and thediscriminator D respectively and shown in Table 1.1. Both G and D are 3-layerfully connected neural networks (see [1] for more details). Vanilla GANs and theirvariants are often trained with (minibatch) stochastic gradient descent (SGD) or itsvariants (e.g., Adam [64]). [43] proposes to use Alg. 1 to train vanilla GANs.4Table 1.1: The network architectures for the generator and the discriminatorin a vanilla GAN. The height, width, and number of channels of an imageare denoted by H, W , and C, respectively. The first and last rows are theinput and output of the two neural networks. Three fully-connected lay-ers are denoted by fc1 to fc3. The two numbers on the left and right sidesof each right arrow denote the input dimension and the output dimen-sion of each fully-connected layer. Three types of activation functionsare used by [43], i.e., ReLU, Sigmoid and Maxout [44]. The number oflinear pieces in each maxout unit is set to 5 (Please see [44] for moredetails about Maxout).(a) Generator G(z)Input: A Gaussian noise z ∈R128fc1: 100→ 8000; ReLUfc2: 8000→ 8000; Sigmoidfc3: 8000→ (C×H×W )Output: a fake image xg ∈ RC×H×W(b) Discriminator D(x)Input: A sample x ∈RC×H×Wfc1: (C×H×W )→ 1600; Maxoutfc2: 1600→ 1600; Maxoutfc3: 1600→ 1; SigmoidOutput: a probability that x is realSince vanilla GANs only estimate the marginal image distribution, it cannotgenerate images given some conditions. However, in practice, we often want tohave more control, such as generating images for a specified image class. More-over, vanilla GANs are only able to generate low-resolution images, and the train-ing process is usually very unstable [121]. To extend vanilla GANs to the condi-tional scenario, enhance the quality of generated images, and stabilize the trainingprocess, there is much work following [43].Remark 1.1.1. The generator G(z;θ) can be interpreted as a mapping which mapsa noise z ∈ Z to a point x in the pixel space X (may be an unrealistic image).Training of the generator can also be interpreted as finding a correct mapping froman arbitrary noise to a realistic image. A mapping is defined as correct if it mapsany given noise to a realistic image and any realistic image is mapped to by at leastone noise. Such a mapping may be incorrect at the beginning of the training, (itmay map a noise to an unrealistic image), but it gradually converges to a correctmapping if the generator and the discriminator are well trained. Please note thatthere may be more than one correct mapping.5Algorithm 1: A (minibatch) stochastic gradientdescent algorithm for vanilla GANs [43].1 for number of training iterations do2 Sample a minibatch of m noise samples{z(1),z(2), · · · ,z(m)} from the simple prior q(z);3 Sample a minibatch of m examples {x(1),x(2), · · · ,x(m)}from the true data distribution pr(x);4 Update the discriminator via the SGD or its variantswith the gradient ofL GAND in Eq.(1.1):∂∂φ{− 1mm∑i=1[logD(x(i);φ)+ log(1−D(G(z(i);θ);φ))]};5 Update the generator via the SGD or its variants withthe gradient ofL GANG in Eq.(1.1):∂∂θ{− 1mm∑i=1logD(G(z(i);θ);φ)};6 endRemark 1.1.2. By rearranging Eq.(1.2), we can get an important relation betweenthe density ratio r∗(x) = pr(x)/pg(x) and the optimal discriminator D∗ as follows:r∗(x) =pr(x)pg(x)=D∗(x)1−D∗(x) . (1.4)This important relation will be used in the following context. If we denote all layersin a discriminator D(x) before the final Sigmoid function (the Sigmoid function infc3 of the discriminator in Table 1.1) by D˜(x), then D(x) can be re-written asD(x) = σ(D˜(x)) =11+ e−D˜(x), (1.5)where the σ denotes a Sigmoid function. Thus, Eq. (1.4) can also be rewritten asr∗(x) =pr(x)pg(x)=D∗(x)1−D∗(x) = eD˜∗(x). (1.6)Remark 1.1.3. The adversarial training mechanism of GANs is also formulatedfrom the game theory perspective as a two-player zero-sum game (or a minimax6problem) in some papers [7, 11, 52]. In this case, the purpose of the GAN trainingis to reach a Nash equilibrium [89, 103].1.1.2 Conditional Generative Adversarial NetworksThe conditional version of GANs is first proposed by [91] and then developed by[14, 94, 95, 100, 163]. Such GANs, as an important variant of vanilla GANs, aretermed as conditional generative adversarial networks (cGANs) and allow us togenerate images in terms of some conditions. cGANs perform conditioning byfeeding conditions into both the generator and discriminator networks, which isvisualized in Fig 1.2. The generator network and the discriminator network of areal image xrnoise zG(z,y;θ)D(x,y;φ) Real/FakeDiscriminatorGeneratorfake label ygreal label yrFigure 1.2: The workflow of a vanilla cGAN.cGAN are denoted, respectively, by G(z,y;θ) and D(x,y;φ) in this thesis, where yrepresents the condition, and z, x, θ , φ , respectively, stand for a random noise, animage, the learnable parameters of the generator network, and the learnable param-eters of the discriminator network. Most existing cGANs are designed for categor-ical conditions such as class labels, and cGANs with class labels as conditions arealso known as class-conditional GANs. In this section, when discussing cGANs,we only refer to class-conditional GANs so y is discrete. The vanilla cGAN lossproposed by [91] to train G(z,y;θ) and D(x,y;φ) is shown as follows:L cGAND (φ) =−Ey∼pr(y)[Ex∼pr(x|y) [logD(x,y;φ)]]−Ey∼pg(y)[Ez∼q(z) [log(1−D(G(z,y;θ),y;φ))]],L cGANG (θ) =−Ey∼pg(y) [Ez∼q [logD(G(z,y;θ),y;φ)]] ,(1.7)7where x ∈X is an image, y ∈ Y is a label, pr(y) and pg(y) are respectively theprobability mass functions (PMFs) of the true and fake label marginal distributions,pr(x|y) is the conditional PDF of the true image distributions given y, and q(z) isthe PDF of N(0, I). In real practice, we usually assume the categorical PMFs pr(y)and pg(y) are the same. If the dataset is balanced (different classes have almostthe same number of samples), we also assume all C classes in this categoricaldistribution have equal probabilities (i.e., Pr(y = c) = 1/C, where c = 1,2, · · · ,C).Similar to vanilla GANs, cGANs aim at estimating pr(x|y). After the training ofcGANs, we are able to generate infinite images from the generator given a classlabel.We use an example to illustrate how vanilla cGANs [91] take class labels intoaccount in the image generation. Assume we have a cGAN whose generator anddiscriminator have architectures similar to those in Table 1.1. Before incorporatingclass labels into GANs, we often transform a class label y (an integer) to a one-hotvector with length C whose elements are either 1 or 0. For example, if imagesare from 5 classes and class labels are in the one-hot representation, class 1 isrepresented by (1,0,0,0,0), class 2 is represented by (0,1,0,0,0), etc. Note that thereare other transformations for class labels such as label embedding [2]. Then, whengenerating a fake image, this example cGAN concatenates the one-hot encodedclass label y to the end of a noise z from the prior distribution q(z) and feedsthem into the generator. Similarly, when distinguishing an image from real to fake,this example cGAN concatenates the one-hot encoded class label y to the end ofa flattened image x (i.e., flatten a C×H ×W image to a long vector) and feedsthem into the discriminator. The network architectures of the generator and thediscriminator in this example cGAN is shown in Table 1.2. This example providesone way to incorporate class labels into GANs. There are many other methods toincorporate class labels into GANs, and please see [14, 94, 95, 100, 163] for moredetails.1.1.3 Other Variants of Vanilla Generative Adversarial NetworksBesides cGANs, many other variants of vanilla GANs are proposed. They improvevanilla GANs from different perspectives. These perspectives include but not lim-8Table 1.2: The network architectures for the generator and the discriminatorin a example cGAN. Assume there are C classes and a class label y isencoded by the one-hot representation.(a) Generator G(z,y;θ)Input: A Gaussian noise z ∈R128 and a class label y.Concatenate the one-hot encoded y to the end of z.fc1: (100+C)→ 8000; ReLUfc2: 8000→ 8000; Sigmoidfc3: 8000→ (C×H×W )Output: a fake image xg ∈ RC×H×W with the class label y(b) Discriminator D(x,y;φ)Input: A sample x ∈RC×H×W and a class label y.Concatenate the one-hot encoded y to the end of flat(x).fc1: (C×H×W +C)→ 1600; Maxoutfc2: 1600→ 1600; Maxoutfc3: 1600→ 1; SigmoidOutput: a probability that x is realited to the points shown as follows:• Advanced architectures for the generator and the discriminatorFor example, deep convolutional generative adversarial networks (DCGANs)[112] model the generator and the discriminator with deep convolutionalneural networks, and this architecture is prevalent in the GAN literature.Self-attention generative adversarial networks (SAGANs) introduce the at-tention mechanism [23, 104, 143] from natural language processing to thenetworks of GANs.• Large scale training schemesFor example, BigGAN [14] (the state-of-the-art GAN model) shows that avery large batch size in the GAN training can lead to high-quality (evenphotorealistic) fake images.• Novel normalization methodsFor example, spectral normalization generative adversarial networks (SNGANs)9[95] propose to use spectral normalization to normalize outputs from someintermediate layers of GAN models, and this technique is also adopted bythe BigGAN [14].• Different loss functionsBesides the standard loss function (1.1) defined in [43], there are other lossfunctions for GANs. For example, [4] proposes the WGAN loss in termsof the Wasserstein distance, based on which [4] proposes Wasserstein gen-erative adversarial networks (WGANs)—a new GAN model. Note that thediscriminator in WGANs does not have the Sigmoid function (the last activa-tion function of the discriminator of a vanilla GAN in Table 1.2b) to convertclass scores to class probabilities. This discriminator is assumed to be 1-Lipschitz continuous, but this assumption may not be satisfied. Thus, [48]proposes a penalty term for the WGAN loss to encourage the discriminator tosatisfy this assumption. In terms of the penalized WGAN loss, [48] proposesthe WGAN-GP—another new GAN model. [76] proposes the MMD-GANloss based on the maximum mean discrepancy, and a GAN trained with thisloss is known as MMD-GAN. Note that the discriminator in a MMD-GANis quite different from that in a vanilla GAN or a WGAN/WGAN-GP since itoutputs a reconstructed image instead of a class probability or a class score.When GANs are trained with the (penalized) WGAN loss or the MMD-GANloss, we cannot get the optimal discriminator D∗ in the form of Eq.1.2. Inthis case, estimating the density ratio r∗(x) = pr(x)/pg(x) via Eq.1.4 maynot be applicable.1.1.4 Existing Metrics for Evaluating Fake Images From GANsTo evaluate the quality of fake images generated from GANs, three common met-rics, including Inception score (IS) [121], Fre´chet inception distance (FID) [52],and Intra-FID [94], are adopted in this thesis:• IS [121] is a popular evaluation metric for GANs which is defined based ona pre-trained CNN f (x) (say Inception-V3 [134]) used for classification. Tocompute IS, we should first train f (x) on the same dataset used for the GANtraining if this dataset has class labels. Otherwise, f (x) is often trained on a10large scale dataset (e.g., ImageNet [28]) that may contain images related tothe dataset for the GAN training. Then, given a fake image x, the pre-trainedf (x) outputs the probability that this image belongs to each class, whichdefines a conditional probability distribution over predicted classes. (Notethat the class corresponding to the highest probability is the predicted classlabel of x.) Assuming a class label y follows this conditional probabilitydistribution, we compute the conditional entropy of y given x denoted byH(y|x). Moreover, we also evaluate f (x) on a set of fake images to get theirpredicted class labels. The frequency of the predicted class labels definesa marginal probability distribution of the class label y, based on which wecompute the marginal entropy of y denoted by H(y). Finally, IS is defined interms of H(y|x) and H(y) as follows:IS = exp{H(y)−Ex[H(y|x)]}. (1.8)IS evaluates the quality of fake images from two perspectives: high classifi-ability and diversity. We assume high quality images are more classifiable,so we favor smaller Ex[H(y|x)]. On the other hand, high diversity meansthe GAN model can generate images from all potential classes not just froma few classes, so we expect high entropy in the predicted class labels (i.e.,larger H(y)). Therefore, the larger the IS is, the better quality the fake imageshave.• FID [52] is another popular evaluation metric for GAN models. FID mea-sures how similar a group of real images and a group of fake images are.Lower score means these fake images are more similar to the real imagesand have better quality. This similarity is computed in terms of the statis-tics of the high-level features extracted from these images. We usually usea pre-trained Inception-V3 [134] which is used for classification to extractsuch high-level features. After feeding this pre-trained Inception-V3 withan image, the high-level features of this image are defined as the output ofthe final average pooling layer. [52] assumes that the high-level feature vec-tor h extracted by this pre-trained Inception-V3 follows a multivariate nor-11mal distribution with mean µ and covariance Σ. In other words, we assumehr ∼ N(µ r,Σr) and hg ∼ N(µ g,Σg), where hr and hg are high level featuresfor real and fake images respectively. This assumption looks very strong, butempirical studies show that FID is often consistent with human judgmentsand works more robustly than IS does [52]. FID is defined as followsFID = ‖µ r−µ g‖22+Tr(Σr +Σg−2(ΣrΣg)12 ), (1.9)where µ r, µ g, Σg and Σr are estimated from samples. Note that FID is com-puted based on both real images and fake images while IS is only computedon fake images.• A conditional GAN (no matter the type of the condition) needs to be eval-uated from three perspectives [94]: (1) the visual quality, (2) the intra-labeldiversity (e.g., the diversity of images from the same image class), and (3) thelabel consistency (whether assigned labels of fake images are consistent withtheir true labels). Measuring the performance of cGANs from these threeperspectives is often conducted by using a popular overall metric, termed theIntra-FID [29, 94, 163]. Intra-FID computes FID separately at each of thedistinct labels and reports the average FID score. Therefore, we prefer smallIntra-FID. Please note that FID and IS cannot evaluate the label consistency,so Intra-FID is taken as the main metric when we evaluate cGANs.1.1.5 Density Ratio Based Subsampling Methods for GANsAlthough state-of-the-art GANs (e.g., BigGAN [14]) are able to generate very re-alistic, high-resolution fake images (e.g., Figure 1.3), some unrealistic images stillappear (e.g., Figure 1.4). Motivated by this finding, two subsampling methodsfor GANs, including discriminator rejection sampling (DRS) [10] and Metropolis-Hastings GAN (MH-GAN) [138], are proposed recently and demonstrated to beeffective in filtering out bad quality images generated from unconditional GANs.Both methods rely on accurately estimating the density ratio pr(x)/pg(x), wherepr(x) and pg(x), respectively, stand for the density functions of the true marginalimage distribution and fake marginal image distribution.12Figure 1.3: Four fake images with high quality generated by BigGAN [14].These fake images have 256×256 resolution.Figure 1.4: An unrealistic fake image generated by BigGAN [14]. This im-age is named “Dog ball” by the author of [14], since it has both thecharacteristics of a dog and a tennis ball.Discriminator rejection sampling (DRS) [10] filters out bad fake images byusing the rejection sampling and a discriminator-based density ratio estimation. Toestimate the density ratio pr(x)/pg(x) required in the rejection sampling, DRS [10]proposes to keep training the discriminator on some hold-out real images and thesame number of fake images by early stopping. Then, the trained discriminatoris assumed to be the optimal discriminator D∗(x) and a density ratio at x can becomputed by evaluating eD˜∗(x) in Eq.(1.6). As a key step in the rejection sampling,DRS [10] estimates M = maxx pr(x)/pg(x) by evaluating eD˜∗(x) on 10,000 fakeimages and this M may be replaced by a larger density ratio if we find one in thesubsequent sampling. In regular rejection sampling procedures, a proposed fakesample x′ is accepted with the probabilityp =pr(x′)Mpg(x′)=r(x′)M. (1.10)13To deal with too low acceptance probabilities when the target distribution is highdimensional, [10] uses a modified acceptance probabilityp = σ(Fˆ(x,M,ε,γ)),whereFˆ(x,M,ε,γ) = D˜∗(x)− logM− log(1− eD˜∗(x)−logM−ε)− γ,, F(x)− γ,(1.11)ε is a small constant (e.g., 10−14) for numerical stability and γ is a hyper-parameterto control the overall acceptance probability. As we mentioned in Section 1.1.1, aGAN is often trained with a SGD method and [10] sets the γ dynamically for eachbatch of fake images drawn from a trained GAN, to the 80-th or 95-th percentile ofthe F(x) in Eq.1.11 for each x in this batch.Metropolis-Hastings GAN (MH-GAN) [138] applies the Metropolis-Hastingsalgorithm to correct the sampling bias of an imperfect generator with informationfrom a calibrated discriminator D∗. To be more specific, MH-GAN constructsa Markov chain{x1,x2, · · ·} where xk is generated as follows: (1) Draw x′ fromthe proposal distribution p(x|xk−1) = pg(x) and u from Uniform(0,1); (2) Theacceptance probability p is defined asp = min(1,pr(x′)pr(xk−1)· pg(xk−1)pg(x′))= min(1,r∗(x′)r∗(xk−1)), (1.12)where r∗(x) and r∗(x′) are computed based on Eq.(1.4); (3) If u≤ p, then xk = x′;otherwise xk = xk−1. This generation-acceptance/rejection procedure is recursivelyrepeated K times and results in a Markov chain with a length of K. To produce in-dependent filtered images, MH-GAN builds one Markov chain per filtered image,and for each chain, only the last image xK is taken. MH-GAN also proposes acalibration method to refine the trained discriminator. It attaches either a logistic,isotonic or beta regression on top of D˜ and trains this regression model on nholdfake images and nhold hold-out real images to distinguish between fake and real.Then the calibrated discriminator is built via D∗(x) =Cal(D˜(x)), where Cal is the14trained regression model. In our experiment in Chapter 3, by default, we use thecalibrated discriminator to compute density ratios when implementing MH-GAN.This calibration technique can also be applied to WGANs (or similar GANs) tolet the calibrated discriminator output class probabilities rather than class scores.However, this calibration is not suitable for MMD-GAN because the “discrimina-tor” of MMD-GAN outputs a reconstructed image instead of class scores or classprobabilities.A major drawback of both methods is that they rely heavily on the optimal-ity of the discriminator to estimate the density ratio, but such optimality is hardto guarantee in practice. Moreover, both methods are designed for unconditionalGANs rather than cGANs.1.2 Density Ratio Estimation for Image DataDensity ratio estimation (DRE) for images, as a standalone topic (i.e., not limitedto the GAN setting), has been researched by [63, 98]. [63, 98] propose to model thetrue density ratio function r∗(x) = pr(x)/pg(x) by a convolutional neural network(CNN) denoted by r(x;α ), where α represents some learnable parameters. TheCNN r(x;α ) is trained on Nr real images xr1, . . . ,xrNr ∼ pr(x) and Ng fake imagesxg1, . . . ,xgNg ∼ pg(x) to map a given image to its density ratio and the estimateddensity ratio at x can be obtained by evaluating the fitted r(x;α ) at x. This typeof density ratio estimation method consists of two components: a neural networkr(x;α ) (used to model the true density ratio function r∗(x)) and a loss function.[98] proposes a CNN with only two convolutional layers to model the density ratiofunction and trains this CNN by the unconstrained least-squares importance fitting(uLSIF) loss defined asL̂uLSIF(α ) =12NgNg∑i=1r(xgi ;α )2− 1NrNr∑i=1r(xri ;α ). (1.13)We denote this DRE method by DRE-P-uLSIF, where P stands for working in thepixel space in contrast to the feature-based methods of Chapter 3. There are tworeasons, however, why uLSIF loss is not well-defined for training a neural networkto model the true density ratio function:15a) Due to the strong expression capacity of neural networks, training r(x;α )under the uLSIF loss may encourage r(x;α ) to memorize all training databy simply assigning almost zero density ratio to all fake images (no mat-ter realistic or not) but very large density ratio to all real images. In thiscase, if we use the SGD optimizer or its variants, the training loss may keepdecreasing without converging.b) To prevent r(x;α ) from simply “memorizing” training data, we may addextra constraints on r(x;α ). Since∫r∗(x)pg(x)dx =∫pr(x)dx = 1, a naturalconstraint on r(x;α ) is ∫r(x;α )pg(x)dx = 1. (1.14)An empirical approximation to this constraint is1NgNg∑i=1r(xgi ;α ) = 1. (1.15)We can apply this constraint by adding a penalty term to the uLSIF loss, i.e.,minα{L̂uLSIF(α )+λ Qˆ(α )}, (1.16)whereQˆ(α ) =(1NgNg∑i=1r(xgi ;α )−1)2. (1.17)However, due to the unbounded nature of the uLSIF loss (the range of L̂uLSIF(α )is (−∞,∞) given r(x;α )≥ 0), the penalty term λ Qˆ(α ) can not stop L̂uLSIF(α )from going to negative infinity during training, no matter how large λ is. Inthis case, the penalty term has no effect.Two new DRE methods are given by [63] in which a 6-layer-CNN is adaptedto model the true density ratio function. These two new DRE methods differ onlyin their two new training loss functions—DSKL and BARR—and they are denotedby DRE-P-DSKL and DRE-P-BARR, respectively. The two new loss functions16are defined as follows:L̂DSKL(α ) =− 1NrNr∑i=1logr(xri ;α )+1NgNg∑i=1logr(xgi ;α ), (1.18)L̂BARR(α ) =− 1NrNr∑i=1logr(xri ;α )+λ∣∣∣∣∣ 1Ng Ng∑i=1r(xgi ;α )−1∣∣∣∣∣ . (1.19)[63] suggests setting λ = 10 in Eq.(1.19). Unfortunately, these two new loss func-tions still suffer from the same problems besetting uLSIF, so they are still unsuit-able for density ratio estimation with neural networks.Different from [63, 98], [47] estimates the density ratio using a relationshipbetween the true density ratio r∗(x) and a Bayes optimal classifier (BOC) denotedby BOC(x):r∗(x) =pr(x)pg(x)= γBOC(x)1−BOC(x) , (1.20)where γ is a prior odds that an image is fake and BOC is a binary classifier whichdistinguishes between images from pr and pg. A CNN is trained by [47] on anequal number of real and fake samples. This trained CNN is used as the BOC andγ is assumed to be 1. Unfortunately, a Bayes optimal classifier is difficult to obtainin practice.1.3 Knowledge Distillation in Image Classification andRegressionIn deep learning, a heavyweight model is often a deep, overparameterized neuralnetwork or an ensemble of many deep neural networks. It usually has high accuracyin image classification and regression but also incurs some costs: (1) high mem-ory cost due to the large model size (i.e., many learnable parameters); (2) a lowinference speed (i.e., number of images processed by a model per second). Pleasenote that a neural network’s inference speed is related to both the number of learn-able parameters and the connection design. For example, DenseNet-121 [55] has adensely connected pattern (leads to a high computation cost) while VGG-11 [130]is a simple, straightforward network. Although DenseNet-121 has fewer learnableparameters than VGG-11 does, evaluating VGG-11 is about 10 times faster than17DenseNet-121. Unfortunately, in many application scenarios (e.g., deploying neu-ral networks on mobile devices), limited computational resources are available toevaluate a trained model, so we can only afford some lightweight models whichare fast and memory-efficient but less accurate. In these scenarios, we usually careabout the computational cost spent in the evaluation stage, so in the training stage,how to use a heavyweight model to improve the training of a lightweight modelattracts much attention recently.Knowledge distillation (KD), first proposed by [15] and then developed by[53], is a popular method to improve the performance of a lightweight model byutilizing the knowledge from a heavyweight model (knowledge is some informa-tion learned by the heavyweight model from the training data). The heavyweightand lightweight models are often known respectively as a teacher model and astudent model. A generic teacher-student framework for knowledge distillation isvisualized in Fig. 1.5. Since 2015, more than 100 KD models have been proposedKnowledgeDistill TransferTeacher ModelStudent ModelFigure 1.5: The generic teacher-student framework for knowledge distilla-tion.for different visual tasks (e.g., [45, 53, 79, 92, 118, 125, 147, 150, 159, 160, 165]).In Chapter 5 of this thesis, we compare our proposed KD method with the base-line knowledge distillation (BLKD) [53] and some recently proposed KD methods[92, 118, 159] in classification tasks [25, 36, 110]. We are also interested in apply-ing KD in image regression with a scalar response in addition to image classifica-tion. [165] proposes a KD method specially designed for estimating ages from theimages of human faces. However, this method may be inapplicable to other image18regression tasks with a scalar response because some components of the proposedframework are only designed for age estimation. To our best knowledge, there isno KD method general enough for all image regression tasks with scalar response.The baseline knowledge distillation (BLKD) [53] transfers knowledge fromthe teacher model to the student model by matching the logits between these twomodels, so it is also known as a logits-based KD method. Denote by l the logits ofan image x from a neural network, where l = [l1, . . . , lC]ᵀ is a C by 1 vector and C isthe number of classes. With the softmax function, we can calculate the probabilitythat the image x belonging to class i (termed soft label) as follows:pi =exp(li/T )∑ j exp(l j/T ), (1.21)where i = 1, . . . ,C and T is the temperature factor. A higher T leads to a moreuniform probability distribution over classes. On the contrary, the one-hot encodedclass label is also known as the hard label. An example of hard labels and softlabels is shown in Fig. 1.6. Usually, the soft label is more informative than thehard label because it can reflect the similarity between classes and the confidenceof prediction. The logits of the same image x from the teacher model ft and thestudent model fs are denoted by l t and l s respectively. Then, the correspondingsoft labels are denoted respectively by pt and ps. The student model fs is trainedto minimize the following loss function:Loverall = (1−λKD)Ls+λKDLKD, (1.22)where λKD ∈ [0,1] is a hyperparameter controlling the trade-off between two LsandLKD. Ls is the standard loss for classification andLKD encourages the knowl-edge transfer. To be specific,Ls is the cross entropy between the one-hot encodedclass label y and the soft label ps, i.e.,Ls =∑i{−yi log psi}. (1.23)19LKD is the cross entropy between pt and ps, i.e.,LKD =∑i{−pti log psi}. (1.24)The workflow of BLKD is visualized in Fig. 1.7.hard labelsoft label0 1 00.09 0.9 0.01cat dog carFigure 1.6: An example of the hard and soft labels of a dog image in an im-age classification task with three classes. The hard label is the one-hotencoded class label. The soft label is obtained by (1) evaluating a neuralnetwork on the dog image to get the logits, (2) and then converting log-its to illustrative probabilities with the softmax function. The soft labelis more informative than the hard label since it can reflect the similaritybetween classes and the confidence of prediction.The teacher assistant knowledge distillation (TAKD) [92] is a recent variantof BLKD. [92] finds that if the performance gap between a teacher model and astudent model is big, BLKD usually does not perform well. Therefore, TAKDintroduces an intermediate model, termed the teacher assistant (TA) model. TheTA model often performs better than the student model but worse than the teachermodel. BLKD is applied to the teacher-TA and TA-student pairs, respectively,where the knowledge is first transferred from the teacher model to the TA modeland then from the TA model to the student model.The self-supervised learning as an auxiliary task for knowledge distillation(SSKD) [159] is another recent KD method. Like BLKD, SSKD encourages thestudent model to mimic the teacher model’s classification performance on labeleddata. Additionally, SSKD also minimizes the difference between the student andteacher models’ performance on a self-supervised learning task [22] (a task to learna more informative representation of images in an unsupervised manner). However,20image 𝒙hardlabel 𝑦teacher 𝑓௧student 𝑓௦logits soft labelℒ௄஽ℒ௦𝒍௧𝒍௦𝒑௧𝒑௦softmaxsoftmaxFigure 1.7: The workflow of the baseline knowledge distillation (BLKD)[53]. The logits l = [l1, l2, . . . , lC] is the output of the teacher net ftor student net fs, where C is the number of classes. The i-th element ofthe soft label p is pi =exp(li/T )∑ j exp(l j/T ), where a larger T (termed tempera-ture) leads to a more uniform probability distribution over classes. Thetraining loss for fs is a linear combination of two componentsLKD andLs. LKD is defined to match the soft labels between ft and fs. Ls isdefined to match the soft label ps and the hard label y.teacher 𝑓௧teacher assistant 𝑓௧௔student 𝑓௦BLKD BLKDFigure 1.8: The workflow of teacher assistant knowledge distillation(TAKD) [92]. The intermediate model, termed the teacher assistantmodel, is used to fill the performance gap between the teacher modeland the student model. The teacher model first teaches the teacher as-sistant, and then the teacher assistant teaches the student model.we often need to carefully adjust the architectures of the student model and theteacher model to let them fit into the proposed algorithm of SSKD.In many applications of knowledge distillation, we often have limited data.Data augmentation is often used to alleviate this problem [158]. [118] proposesto incorporate unsupervised data augmentation (UDA) [158], a recently proposeddata augmentation method, into BLKD. The empirical study in [118] shows thatBLKD+UDA is a very effective knowledge distillation method.211.4 Research Contributions and Organization of ThesisThe rest contents of this thesis are organized as follows:• Chapter 2: We propose the concept of a continuous conditional generativeadversarial network (CcGAN), the first generative model for image gener-ation conditional on continuous, scalar conditions (termed regression la-bels). Existing cGANs are mainly designed for categorical conditions (e.g.,class labels). Conditioning on regression labels is mathematically distinctand raises two fundamental problems: (P1) since there may be very few(even zero) real images for some regression labels, minimizing existing em-pirical versions of cGAN losses (a.k.a. empirical cGAN losses) often failsin practice; and (P2) since regression labels are scalar and infinitely many,conventional label input mechanisms (e.g., combining a hidden map of thegenerator/discriminator with a one-hot encoded label) are not applicable.We solve these problems by: (S1) reformulating existing empirical cGANlosses to be appropriate for the continuous scenario; and (S2) proposing anaive label input (NLI) mechanism and an improved label input (ILI) mech-anism to incorporate regression labels into the generator and the discrimi-nator. The reformulation in (S1) leads to two novel empirical discriminatorlosses, termed the hard vicinal discriminator loss (HVDL) and the soft vici-nal discriminator loss (SVDL) respectively, and a novel empirical generatorloss. Hence, we propose four versions of CcGAN employing different pro-posed losses and label input mechanisms. The error bounds of the discrim-inator trained with HVDL and SVDL respectively, are derived under mildassumptions. To evaluate the performance of CcGANs, two new benchmarkdatasets (RC-49 and Cell-200) are created. A novel evaluation metric (Slid-ing Fre´chet Inception Distance) is also proposed to replace Intra-FID whenIntra-FID is not applicable. Our extensive experiments on several bench-mark datasets (i.e., the Circular 2-D Gaussians, RC-49, UTKFace, Cell-200,and Steering Angle datasets) support the following findings: the proposedCcGAN is able to generate diverse, high-quality samples from the imagedistribution conditional on a given regression label; and CcGAN substan-tially outperforms cGAN both visually and quantitatively.22• Chapter 3: In this chapter, we propose a novel Softplus (SP) loss for den-sity ratio estimation. Based on it, we develop a sample-based density ratioestimation method in a feature space learned by a specially designed andpre-trained ResNet-34, termed DRE-F-SP. We derive the error bound of adensity ratio model trained under the SP loss. Then, we propose three den-sity ratio based subsampling methods for GANs based on DRE-F-SP. Oursubsampling methods do not rely on the optimality of the discriminator andare suitable for all types of GANs. We empirically show our subsamplingapproach can substantially outperform DRS [10] and MH-GAN [138] on asynthetic dataset, CIFAR-10, MNIST and CelebA, using multiple GANs.• Chapter 4: Subsampling GANs to improve the overall image quality in theunconditional image synthesis setting has been widely studied in Chapter 3and other papers in the GAN literature. However, subsampling conditionalGANs such as class-conditional GANs and CcGANs has rarely been inves-tigated. In this chapter, to fill this gap, we first propose a novel conditionaldensity ratio estimation (cDRE) method, termed conditional density ratio es-timation in feature space with conditional Softplus loss (cDRE-F-cSP). WithcDRE-F-cSP, we estimate the conditional density ratio of an image in a fea-ture space learned by a pre-trained, specially designed ResNet-34 or sparseautoencoder. A novel Softplus loss is also proposed for cDRE. We then de-rive the error bound of a conditional density ratio model trained with thisSoftplus loss. Finally, based on cDRE-F-cSP, we propose a rejection sam-pling scheme, termed cDRE-F-cSP+RS, which is able to subsample bothclass-conditional GANs and CcGANs. An extra filtering scheme is alsodeveloped for CcGANs to increase the label consistency. Experiments onCIFAR-10, Tiny-ImageNet, RC-49 and UTKFace demonstrate the effective-ness of cDRE-F-cSP+RS in subsampling BigGAN and CcGANs in terms ofmultiple evaluation metrics.• Chapter 5: Knowledge distillation (KD) has been actively studied for imageclassification tasks in deep learning, aiming to improve the performance ofa lightweight (student) model based on the knowledge from a heavyweight(teacher) model. However, applying KD in image regression with a scalar23response variable has been rarely studied, and there exists no KD methodapplicable to both classification and regression tasks yet. Moreover, ex-isting KD methods often require a practitioner to carefully select or adjustthe teacher and student architectures, making these methods less scalable inpractice. Furthermore, although KD is usually conducted in scenarios withlimited labeled data, very few techniques are developed to alleviate suchdata insufficiency. To address the above concerns in a unified way, we pro-pose a comprehensive KD framework based on cGANs, termed cGAN-KD.Fundamentally different from existing KD methods, cGAN-KD distills andtransfers knowledge from a teacher model to a student model via cGAN-generated samples. This novel mechanism makes cGAN-KD suitable forboth classification and regression tasks, compatible with other KD meth-ods, and insensitive to the teacher and student architectures. Also, benefit-ing from recent advances in cGAN (e.g., CcGANs proposed in Chapter 2)and our specially designed subsampling (i.e., cDRE-F-cSP+RS proposed inChapter 4) and filtering procedures, the proposed cGAN-KD also performswell when labeled data are limited. An error bound for a student modeltrained in the cGAN-KD framework is derived in this work, providing atheory for why cGAN-KD is effective as well as guiding the practical im-plementation of cGAN-KD. Extensive experiments on CIFAR-10 and Tiny-ImageNet show that we can incorporate state-of-the-art KD methods intothe cGAN-KD framework to yield a new state of the art. Also, experimentson RC-49 and UTKFace demonstrate the effectiveness of cGAN-KD in im-age regression tasks, where existing KD methods are inapplicable. Notably,cGAN-KD still works well even if only 20% of the CIFAR-10 data and 10%of the RC-49 data are available for training.The connections between these chapters are summarized in Fig. 1.9. Finally,Chapter 6 concludes the thesis and discusses the future research.24Figure 1.9: The connections between Chapters 2 to 5. In Chapter 2, we pro-pose a conditional GAN called CcGAN to generate images conditionalon continuous scalars. In Chapter 3, we propose a method to subsampleunconditional GANs to improve the image quality. In Chapter 4, weextend the subsampling method in Chapter 3 to the conditional imagesynthesis setting, which can be used to subsample both class-conditionalGANs and CcGAN. In Chapter 5, we introduce a unified knowledge dis-tillation framework suitable for both image classification and regressiontasks, which applies to class-conditional GANs, CcGAN, and cDRE-F-cSP+RS.25Chapter 2CcGAN: Continuous ConditionalGenerative Adversarial Networks2.1 IntroductionConditional generative adversarial networks (cGANs), first proposed in [91], aimto estimate the distribution of images conditioning on some auxiliary information,especially class labels. Subsequent studies [14, 94, 100, 163] confirm the fea-sibility of generating diverse, high-quality (even photo-realistic), and class-labelconsistent fake images from class-conditional GANs. Unfortunately, these cGANsdo not work well for image generation with continuous, scalar conditions, termedregression labels, due to two problems:(P1) cGANs are often trained to minimize the empirical versions of their losses(a.k.a. empirical cGAN losses) on some training data, a principle also known asempirical risk minimization (ERM) [96, 123, 141]. The success of ERM relies on alarge sample size for each distinct condition. Unfortunately, we usually have onlya few real images for some regression labels. Moreover, since regression labels arecontinuous, some values may not even appear in the training set. Consequently, acGAN cannot accurately estimate the image distribution conditional on such miss-ing labels.(P2) In class-conditional image generation, class labels are often encoded by one-hot vectors or label embedding and then fed into the generator and discriminator by26hidden concatenation [91], an auxiliary classifier [100] or label projection [94]. Aprecondition for such label encoding is that the number of distinct labels (e.g., thenumber of classes) is finite and known. Unfortunately, in the continuous scenario,we may have infinitely many distinct regression labels.A naive approach to solve (P1)-(P2) is to “bin” the regression labels into aseries of disjoint intervals and still train a cGAN in the class-conditional manner(these interval are treated as independent classes) [101]. However, this approachhas four shortcomings: (1) our experiments in Section 2.4 show that this approachoften makes cGANs collapse; (2) we can only estimate the image distribution con-ditional on membership in an interval and not on the target label; (3) a large intervalwidth leads to high label inconsistency; (4) inter-class correlation is not considered(images in successive intervals have similar distributions).In machine learning, vicinal risk minimization (VRM) [18, 141] is an alterna-tive rule to ERM. VRM assumes that a sample point shares the same label withother samples in its vicinity. Motivated by VRM, in generative modeling condi-tional on regression labels where we estimate a conditional distribution p(x|y) (xis an image and y is a regression label), it is natural to assume that a small pertur-bation to y results in a negligible change to p(x|y). This assumption is consistentwith our perception of the world. For example, the image distribution of facialfeatures for a population of 15-year-old teenagers should be close to that of 16-year olds, the assumption underlying analysis of the UTKFace images described inSection 2.4.3.We therefore introduce the continuous conditional generative adversarial net-work (CcGAN) to tackle (P1) and (P2). To our best knowledge, this is the firstgenerative model for image generation conditional on regression labels. It is notedthat [116] and [115] train GANs in an unsupervised manner and synthesize un-labeled fake images for a subsequent image regression task. [102] proposes asemi-supervised GAN for dense crowd counting. CcGAN is fundamentally dif-ferent from these works since they do not estimate the conditional image distri-bution. Moreover, some recent works [60, 137, 166, 167] propose several novelschemes to train GANs when training data are limited, which seems to be relevantto (P1). However, they are also fundamentally different from CcGAN, since theyare designed for the unconditional and class-conditional scenarios rather than the27continuous scenario. Our contributions can be summarized as follows:• We propose in Section 2.2.1 a solution to address (P1), which consists of twonovel empirical discriminator losses, termed the hard vicinal discriminator loss(HVDL) and the soft vicinal discriminator loss (SVDL), and a novel empiricalgenerator loss. We take the vanilla cGAN loss as an example to show how toderive HVDL, SVDL, and the novel empirical generator loss by reformulatingexisting empirical cGAN losses.• In Section 2.2.2, we propose two novel label input mechanisms, consisting ofa naive label input (NLI) mechanism and an improved label input (ILI) mecha-nism, as solutions to address (P2).• We propose in Section 2.3 a novel evaluation metric, termed sliding Fre´chet In-ception distance (SFID), to evaluate the generative image modeling conditionalon regression labels when there are insufficient real images to compute Intra-FID[94].• In Section 2.4, we propose two new benchmark datasets, RC-49 and Cell-200,for generative image modeling conditional on regression labels, since very fewbenchmark datasets are suitable for the studied continuous scenario. We conductexperiments on the new datasets and three others to demonstrate that CcGAN notonly generates diverse, high-quality, and label consistent images, but also sub-stantially outperforms cGAN both visually and quantitatively. The effectivenessof SFID is also studied on the RC-49 dataset at the end of this section.• We derive in Section 2.5 the error bounds of a discriminator trained with HVDLand SVDL.2.2 From cGAN to CcGANIn this section, we introduce CcGAN, consisting of solutions to (P1) and (P2). Thecombinations of two vicinal discriminator losses (HVDL and SVDL) proposedin Section 2.2.1 and two novel label input mechanisms (NLI and ILI) proposed inSection 2.2.2 result in four CcGAN methods denoted by HVDL+NLI, SVDL+NLI,HVDL+ILI, and SVDL+ILI respectively.282.2.1 Solution to (P1): Reformulated Empirical LossesTheoretically, cGAN losses (e.g., the vanilla cGAN loss [91], the Wasserstein loss[4, 48], and the hinge loss [95]) are suitable for both class labels and regressionlabels; however, their empirical versions fail in the continuous scenario (i.e., (P1)).Our first solution (S1) focuses on reformulating these empirical cGAN losses forcontinuous labels. Without loss of generality, we only take the vanilla cGAN lossas an example to show such reformulation (the empirical versions of the Wasser-stein loss and the hinge loss can be reformulated similarly).The vanilla discriminator loss and generator loss [91] are defined as:L (D) =−Ey∼pr(y)[Ex∼pr(x|y) [log(D(x,y))]]−Ey∼pg(y)[Ex∼pg(x|y) [log(1−D(x,y))]]=−∫log(D(x,y))pr(x,y)dxdy−∫log(1−D(x,y))pg(x,y)dxdy, (2.1)L (G) =−Ey∼pg(y)[Ez∼q(z) [log(D(G(z,y),y))]]=−∫log(D(G(z,y),y))q(z)pg(y)dzdy, (2.2)where x ∈X is an image, y ∈ Y is a label, pr(y) and pg(y) are respectively PDF-s/PMFs of the true and fake label marginal distributions, pr(x|y) is conditionalPDF of the true image distributions conditional on y, pr(x,y) and pg(x,y) are re-spectively mixed densities (a mixed density is a product of a PDF and a PMF wheny is discrete) of the true and fake joint distributions of x and y, and q(z) is PDF ofN (0, I).Since the PDFs in the losses of Eqs. (2.1) and (2.2) are unknown, for class-conditional image generation, [91] follows ERM and minimizes the empirical losses:L̂ δ (D) =− 1NrC∑c=1Nrc∑j=1log(D(xrc, j,c))− 1NgC∑c=1Ngc∑j=1log(1−D(xgc, j,c)),(2.3)29L̂ δ (G) =− 1NgC∑c=1Ngc∑j=1log(D(G(zc, j,c),c)), (2.4)where C is the number of classes, Nr and Ng are respectively the number of real andfake images, Nrc and Ngc are respectively the number of real and fake images withlabel c, xrc, j and xgc, j are respectively the j-th real image and the j-th fake imagewith label c, and the zc, j are independently and identically sampled from q(z). Eq.(2.3) implies we estimate pr(x,y) and pg(x,y) by their empirical PDFs as follows:pˆδr (x,y) =1NrC∑c=1Nrc∑j=1δD(x− xrc, j)δ (y− c),pˆδg (x,y) =1NgC∑c=1Ngc∑j=1δD(x− xgc, j)δ (y− c),(2.5)where δD(·) is a Dirac delta function (Appendix A of [120]) centered at 0. How-ever, pˆδr (x,y) and pˆδg (x,y) in Eq. (2.5) are not good estimates in the continuousscenario because of (P1).To overcome (P1), we propose a novel estimate for each of pr(x,y) and pg(x,y),termed the hard vicinal estimate (HVE). We also provide an intuitive alternative toHVE, named the soft vicinal estimate (SVE). The HVEs of pr(x,y) and pg(x,y)are:pˆHVEr (x,y) =C1 ·[1NrNr∑j=1exp(−(y− yrj)22σ2ε)]·[1Nry,κNr∑i=11{|y−yri |≤κ}δD(x− xri )],pˆHVEg (x,y) =C2 ·[1NgNg∑j=1exp(−(y− ygj)22σ2ε)]·[1Ngy,κNg∑i=11{|y−ygi |≤κ}δD(x− xgi )],(2.6)where xri and xgi are respectively real image i and fake image i, yri and ygi are re-spectively the labels of xri and xgi , κ and σε are two positive hyper-parameters,C1 and C2 are two constants making these two estimates valid PDFs, Nry,κ is the30number of the yri satisfying |y− yri | ≤ κ , Ngy,κ is the number of the ygi satisfying|y−ygi | ≤ κ , and 1 is an indicator function with support in the subscript. The termsin the first square brackets of pˆHVEr and pˆHVEg imply we estimate pr(y) and pg(y)by kernel density estimation (KDE) [26, 50, 105, 128]. The terms in the secondsquare brackets are designed based on the assumption that a small perturbation toy results in negligible changes to pr(x|y) and pg(x|y). If this assumption holds, wecan use images with labels in a small vicinity of y to estimate pr(x|y) and pg(x|y).The SVEs of pr(x,y) and pg(x,y) are:pˆSVEr (x,y) =C3 ·[1NrNr∑j=1exp(−(y− yrj)22σ2ε)]·[∑Nri=1 wr(yri ,y)δD(x− xri )∑Nri=1 wr(yri ,y)],pˆSVEg (x,y) =C4 ·[1NgNg∑j=1exp(−(y− ygj)22σ2ε)]·[∑Ngi=1 wg(ygi ,y)δD(x− xgi )∑Ngi=1 wg(ygi ,y)],(2.7)where C3 and C4 are two constants making these two estimates valid PDFs,wr(yri ,y) = e−ν(yri−y)2 and wg(ygi ,y) = e−ν(ygi−y)2 , (2.8)and the hyper-parameter ν > 0. In Eq. (2.7), similar to the HVEs, we estimate pr(y)and pg(y) by KDEs. Instead of using samples in a hard vicinity, the SVEs use allrespective samples to estimate pr(x|y) and pg(x|y) but each sample is assignedwith a weight based on the distance of its label from y. Two diagrams in Fig. 2.1visualize the process of using hard/soft vicinal samples to estimate the Gaussiandistribution’s conditional density p(x|y), for univariate x.By plugging Eq. (2.6) and (2.7) into Eq. (2.1), we derive the hard vicinal dis-31(a) Hard Vicinity(b) Soft VicinityFigure 2.1: HVE (Eq. (2.6)) and SVE (Eq. (2.7)) estimates of p(x|y) (a uni-variate Gaussian conditional on y) using two samples in hard and softvicinities, respectively, of y. To estimate p(x|y) (the red Gaussian curve)only from samples drawn from p(x|y1) and p(x|y2) (the blue Gaussiancurves), estimation is based on the samples (red dots) in a hard vicinity(defined by y±κ) or a soft vicinity (defined by the weight decay curve)around y. The histograms in blue are samples in the hard or soft vicinity.The labels y1, y, and y2 on the x-axis denote the means of x conditionalon y1, y, and y2, respectively.32criminator loss (HVDL) and the soft vicinal discriminator loss (SVDL) as follows:L̂ HVDL(D)=− C5NrNr∑j=1Nr∑i=1Eεr∼N (0,σ2ε )[W1 log(D(xri ,yrj + εr))]− C6NgNg∑j=1Ng∑i=1Eεg∼N (0,σ2ε )[W2 log(1−D(xgi ,ygj + εg))],(2.9)L̂ SVDL(D)=− C7NrNr∑j=1Nr∑i=1Eεr∼N (0,σ2ε )[W3 log(D(xri ,yrj + εr))]− C8NgNg∑j=1Ng∑i=1Eεg∼N (0,σ2ε )[W4 log(1−D(xgi ,ygj + εg))],(2.10)where εr := y− yrj, εg := y− ygj ,W1 =1{|yrj+εr−yri |≤κ}Nryrj+εr,κ, W2 =1{|ygj+εg−ygi |≤κ}Ngygj+εg,κW3 =wr(yri ,yrj + εr)∑Nri=1 wr(yri ,yrj + εr), W4 =wg(ygi ,ygj + εg)∑Ngi=1 wg(ygi ,ygj + εg),and C5, C6, C7, and C8 are some constants.Generator training: The generator of CcGAN is trained by minimizing Eq.(2.11),L̂ ε(G) =− 1NgNg∑i=1Eεg∼N (0,σ2ε ) log(D(G(zi,ygi + εg),ygi + εg)). (2.11)How do HVDL, SVDL, and Eq. (2.11) overcome (P1)?(i) Given a label y as the condition, we use images in a hard/soft vicinity of y totrain the discriminator instead of just using images with label y. It enables us toestimate pr(x|y) when there are not enough real images with label y.(ii) From Eqs. (2.9) and (2.10), we can see that the KDEs in Eqs. (2.6) and (2.7)are adjusted by adding Gaussian noise to the labels. Moreover, in Eq. (2.11), we33add Gaussian noise to seen labels (assume the ygi are seen) to train the generatorto generate images at unseen labels. This enables estimation of pr(x|y′) when y′ isnot in the training set.Remark 2.2.1. Based on the kernel density estimation [26, 50, 105, 128] and theproperty of the Dirac delta function (Appendix A of [120]),∫δD(x − xri )dx =∫δD(x − xgi )dx = 1 and C1 = C2 = C3 = C4 = 1/σε . Therefore, C5 = C6 andC7 =C8, which implies these constants C1, . . . ,C8 can be ignored when minimizingL̂ HVDL(D) and L̂ SVDL(D).Remark 2.2.2. Algorithms are proposed in Algs. 2 and 3 for training CcGAN inpractice. If should be noted that, for computational efficiency, the normalizing con-stants Nryrj+εr,κ , Ngygj+εg,κ, ∑Nri=1 wr(yri ,yrj + εr), and ∑Ngi=1 wg(ygi ,ygj + εg) in Eq. (2.9)and (2.10) are excluded from the training and only used for theoretical analysis.Moreover, CcGAN does not require any specific network architecture, so it canuse modern GAN architectures such as SNGAN [95], SAGAN [163] and BigGAN[14].Remark 2.2.3 (A rule of thumb for hyper-parameter selection). In our experi-ments, we normalize labels to real numbers in [0,1] and the hyper-parameter selec-tion is conducted based on the normalized labels. To be more specific, the hyper-parameter σε is computed based on a rule-of-thumb formula for the bandwidthselection of KDE [128], i.e., σε = (4σˆ5yr/3Nr)1/5, where σˆyr is the sample standarddeviation of normalized labels in the training set. Let κbase = max(yr[2]− yr[1],yr[3]−yr[2], . . . ,yr[Nruy]−yr[Nruy−1]), where yr[l] is the l-th smallest normalized distinct real labeland Nruy is the number of normalized distinct labels in the training set. The κ is setas a multiple of κbase (i.e., κ = mκκbase) where the multiplier mκ stands for 50%of the minimum number of neighboring labels used for estimating pr(x|y) given alabel y. For example, mκ = 1 implies using 2 neighboring labels (one on the leftwhile the other one on the right). In our experiments, mκ is generally set as 1 or 2.In some extreme case when many distinct labels have too few real samples, we mayconsider increasing mκ . We also found ν = 1/κ2 works well in our experiments.34Algorithm 2: An algorithm for CcGAN training with the proposedHVDL.Data: Nr real image-label pairs Ωr = {(xr1,yr1), . . . ,(xrNr ,yrNr )}, Nruy ordered distinct labelsϒ= {yr[1], . . . ,yr[Nruy]} in the dataset, preset σε and κ , number of iterations K, thediscriminator batch size md , and the generator batch size mg.Result: Trained generator G.1 for k = 1 to K do2 Train D;3 Draw md labels Y d with replacement from ϒ;4 Create a set of target labels Y d,ε = {yi + ε|yi ∈ Y d ,ε ∈N (0,σ2), i = 1, . . . ,md} (Dtraining is conditional on these labels) ;5 Initialize Ωrd = φ ,Ωfd = φ ;6 for i = 1 to md do7 Randomly choose an image-label pair (x,y) ∈Ωr satisfying |y− yi− ε| ≤ κwhere yi + ε ∈ Y d,ε and let Ωrd =Ωrd ∪ (x,yi + ε);8 Randomly draw a label y′ from U(yi + ε−κ,yi + ε+κ) and generate a fakeimage x′ by evaluating G(z,y′), where z ∼N (0, I). Let Ω fd =Ω fd ∪ (x′,yi + ε);9 end10 Update D with samples in set Ωrd and Ωfd via gradient-based optimizers based onEq.(2.6);11 Train G;12 Draw mg labels Y g with replacement from ϒ;13 Create another set of target labels Y g,ε = {yi+ε|yi ∈Y g,ε ∈N (0,σ2ε ), i = 1, . . . ,mg}(G training is conditional on these labels) ;14 Generate mg fake images conditional on Y g,ε and put these image-label pairs in Ω fg ;15 Update G with samples in Ω fg via gradient-based optimizers based on Eq.(2.11) ;16 end2.2.2 Solutions to (P2): Two Novel Label Input MechanismsIn this section, we propose two solutions, consisting of a naive and an improvedlabel input mechanism, to solve (P2).• A naive label input (NLI) mechanism:We first propose a naive approach to incorporate the regression labels intothe cGANs. For G, we add the label y element-wise to the output of its firstlinear layer. For D, the label y is first projected to the latent space learnedby an extra linear layer. Then, we incorporate the embedded label into thediscriminator by label projection [94]. Figs. 2.2 and 2.3 visualize the naivelabel input mechanism.35Algorithm 3: An algorithm for CcGAN training with the proposedSVDL.Data: Nr real image-label pairs Ωr = {(xr1,yr1), . . . ,(xrNr ,yrNr )}, Nruy ordered distinct labelsϒ= {yr[1], . . . ,yr[Nruy]} in the dataset, preset σε and ν , number of iterations K, thediscriminator batch size md , and the generator batch size mg.Result: Trained generator G.1 for k = 1 to K do2 Train D;3 Draw md labels Y d with replacement from ϒ;4 Create a set of target labels Y d,ε = {yi + ε|yi ∈ Y d ,ε ∈N (0,σ2ε ), i = 1, . . . ,md} (Dtraining is conditional on these labels) ;5 Initialize Ωrd = φ ,Ωfd = φ ;6 for i = 1 to md do7 Randomly choose an image-label pair (x,y) ∈Ωr satisfying e−ν(y−yi−ε)2 > 10−3where yi + ε ∈ Y d,ε and let Ωrd =Ωrd ∪ (x,yi + ε). This step is used to excludereal images with too small weights.;8 Compute wri (y,yi + ε) = e−ν(yi+ε−y)2 ;9 Randomly draw a label y′ from U(yi + ε−√− log10−3ν ,yi + ε+√− log10−3ν ) andgenerate a fake image x′ by evaluating G(z,y′), where z ∼N (0, I). LetΩ fd =Ωfd ∪ (x′,yi + ε). ;10 Compute wgi (y′,yi + ε) = e−ν(yi+ε−y′)2 ;11 end12 Update D with samples in set Ωrd and Ωfd via gradient-based optimizers based onEq.(2.7);13 Train G;14 Draw mg labels Y g with replacement from ϒ;15 Create another set of target labels Y g,ε = {yi+ε|yi ∈Y g,ε ∈N (0,σ2ε ), i = 1, . . . ,mg}(G training is conditional on these labels) ;16 Generate mg fake images conditional on Y g,ε and put these image-label pairs in Ω fg ;17 Update G with samples in Ω fg via gradient-based optimizers based on Eq.(2.11) ;18 end• An improved label input (ILI) mechanism:Empirical studies in Section 2.4 show that CcGAN with the naive label inputmechanism already substantially outperforms cGAN. Nevertheless, it stillsuffers from severe label inconsistency on some datasets (e.g., Cell-200 andSteering Angle). To improve the label consistency of CcGAN, we proposean improved label input (ILI) mechanism. The ILI approach consists of apre-trained CNN and a label embedding network. The pre-trained CNN, asshown in Fig. 2.4, includes two subnetworks, T1 and T2, where T1 maps36Figure 2.2: The naive label inputmechanism (NLI) for G of Cc-GAN.Figure 2.3: The naive label inputmechanism (NLI) for D of Cc-GAN.an image x to a feature space and T2 maps the extracted feature h to a re-gression label y. The dimension of the feature space is set to 128 in ourexperiments. The label embedding network T3, as shown in Fig. 2.5, is amultilayer perceptron (MLP) [50] mapping a regression label y back to itshidden representation h in the feature space defined by T1. Assume that thereare m distinct regression labels in the training set, i.e., yu1,yu2, . . . ,yum, then thelabel embedding network T3 is trained by:minT31mEγ∼N (0,σ2γ )[(T2(T3(yui + γ))− (yui + γ))2], (2.12)where σγ is often a small value and is set at 0.2 in this chapter. Then, given aregression label y, we can evaluate T3(y) to get the hidden representation of ywhich will be incorporated into CcGAN as the condition (visualized in Fig.2.6 and Fig. 2.7). Specifically, for G, we input the embedded label by usingconditional batch normalization [27]. For D, similar to the naive approach,we input the embedded label into D by label projection [94].37Figure 2.4: The pre-trained CNN T1+T2 for label embedding. The first sub-network T1 consists of some convolutional layers (Conv.) and somelinear layers. The second subnetwork T2 includes one linear layer.Figure 2.5: The label embedding network is a multilayer perceptron (MLP).Figure 2.6: The improved label inputmechanism (ILI) for G of Cc-GAN.Figure 2.7: The improved label inputmechanism (ILI) for D of Cc-GAN.382.3 Sliding Fre´chet Inception DistanceA conditional GAN (no matter the type of the condition) needs to be evaluated fromthree perspectives [29]: (1) the visual quality, (2) the intra-label diversity, and (3)the label consistency (whether assigned labels of fake images are consistent withtheir true labels). Measuring the performance of cGANs from these three perspec-tives is often conducted by using a popular overall metric, termed the Intra-FID[29, 94, 163]. Intra-FID computes the Fre´chet inception distance (FID) [52] sep-arately at each of the distinct labels and reports the average FID score. Intra-FIDis also used in our experiments on RC-49, UTKFace, and Cell-200 in Section 2.4;however, Intra-FID is not reliable or is even inapplicable when we have very few(even zero) real images for some distinct regression labels, e.g., the experiment onthe Steering Angle dataset in Section 2.4.5. We therefore propose a novel met-ric, termed the Sliding Fre´chet Inception Distance (SFID), to replace Intra-FID inthis scenario. SFID computes FID within an interval sliding on the range of theregression label y, and then reports the average of these FIDs. Specifically, wefirst prespecify a finite set of SFID centers cSFID evenly over the range of y anda constant SFID radius rSFID. Then, based on the cSFID and rSFID, we can definemany joint SFID intervals of the form [cSFID− rSFID,cSFID+ rSFID]. For each SFIDinterval, we compute FID between real and fake images with labels within this in-terval. Finally, SFID reports the average of these FIDs. Usually, we also report thestandard deviation of these FIDs. We visualize the procedure for computing SFIDin Fig. 2.8. A pseudo code for computing SFID is shown in Alg. 4. Similar toIntra-FID, a small SFID is preferred.Figure 2.8: Sliding Fre´chet Inception Distance (SFID). We preset finite cen-ters (blue dots) on y’s range evenly and a radius rSFID. Given an interval[cSFID− rSFID,cSFID + rSFID], we compute FID between fake and realimages with labels in this interval. SFID is the average of these FIDs.39Algorithm 4: An algorithm to compute the Sliding Fre´chet InceptionDistance (SFID).Data: Real image-label pairs Dr = {(xr1,yr1),(xr2,yr2), · · · ,(xrNr ,yrNr )}; fake image-labelpairs Dg = {(xg1,yg1),(xg2,yg2), · · · ,(xgNg ,ygNg)}; preset SFID centers {yc1,yc2, · · · ,ycNc};preset window radius rSFID.Result: SFID.1 Initialize real and fake image sets in l-th sliding-window, i.e., Drl = φ and Dgl = φ ,l = 1,2, · · · ,Nc;2 Initialize a FID array FID(l) = ∞, l = 1,2, · · · ,Nc;3 for l = 1 to Nc do4 Update real image set at ycl : Drl =⋃|yri−ycl |≤rSFID{xri }, i = 1,2, · · · ,Nr;5 Update fake image set at ycl : Dgl =⋃|ygi−ycl |≤rSFID{xgi }, i = 1,2, · · · ,Ng;6 Compute current FID: FID(l) = FID(Drl ,Dgl ) ;7 end8 Compute SFID = 1Nc ∑Ncl=1 FID(l).2.4 ExperimentsIn this section, we study the effectiveness of CcGAN on five datasets where cGAN[91] cannot generate realistic samples. For a fair comparison, cGAN and CcGANuse the same network architecture (a customized fully-connected architecture forCircular 2-D Gaussians, a customized DCGAN [112] architecture for Cell-200 andthe SNGAN [95] architecture for the remaining datasets) except for the label inputmodules. The four CcGAN methods (i.e., HVDL+NLI, SVDL+NLI, HVDL+ILI,and SVDL+ILI) are tested in all our experiments below except Circular 2-D Gaus-sians. As Circular 2-D Gaussians is a simple, low-dimensional simulation, we usea hidden concatenation-like scheme to input the regression label into CcGAN in-stead of NLI and ILI. For stability, regression labels in RC-49, UTKFace, Cell-200,and Steering Angle are normalized to [0,1] during training.The proposed SFID (see Section 2.3) is only used in the experiment conductedon the Steering Angle dataset in Section 2.4.5, where there are not enough realimages to compute Intra-FID. The effectiveness of SFID is studied on the RC-49dataset in Section 2.4.6, where we can control the sample size of real images.402.4.1 Circular 2-D GaussiansWe first test on synthetic data generated from 120 2-D Gaussians with differentmeans.Experimental setup: The means of the 120 Gaussians are evenly arranged on aunit circle centered at the origin O of a 2-D space. The Gaussians share a commoncovariance matrix σ˜2I2×2, where σ˜ = 0.02. We generate 10 samples from eachGaussian for training. Fig. 2.9(a) shows 1,200 training samples (blue dots) fromthese Gaussians with their means (red dots) on a unit circle. The unit circle canbe seen as a clock where we take the mean at 12 o’clock (point A) as the baselinepoint. Given another mean on the circle (point B), the label y for samples gener-ated from the Gaussian with mean B is defined as the clockwise angle (in radians)between line segments OA and OB. E.g., the label for samples from the Gaussianat A is 0. Both cGAN and our proposed CcGAN are trained on this training set.When implementing cGAN, angles are treated as class labels (each Gaussian istreated as a class) and input into cGAN by the hidden concatenation [91], whilewhen implementing CcGAN, angles are treated as real numbers and incorporatedinto CcGAN by a hidden concatenation-like scheme. The two novel label inputmechanisms (NLI and ILI) are not implemented in this simple, low-dimensionalscenario. Please see Supp. A.2.1 for the net architectures of cGAN and CcGANand more details of the label input schemes. Both cGAN and CcGAN are trainedfor 6,000 iterations. We use the rule of thumb formulae in Remark 2.2.3 to se-lect the hyper-parameters of HVDL and SVDL, i.e., σε ≈ 0.074, κ ≈ 0.017 andν = 3600 (see Supp. A.2.2 for details).For testing, we choose 360 points evenly distributed on the unit circle as themeans of 360 Gaussians. For each Gaussian, we generate 100 samples, yielding atest set with 36,000 samples. It should be noted that, among these 360 Gaussians,at least 240 are not used for training. In other words, there are at least 240 labelsin the testing set which do not appear in the training set. For each test angle, wegenerate 100 fake samples from each trained GAN, yielding 36,000 fake samplesfrom each GAN in total. The quality of these fake samples is evaluated. We repeatthe whole experiment three times and report in Table 2.1 the average quality overthree repetitions.41Evaluation metrics and quantitative results: In the label-conditional scenario,each fake sample x with label y is compared with the mean (sin(y),cos(y)) ofa Gaussian on the unit circle with label y. A fake sample is defined as ”high-quality” if its Euclidean distance from x to (sin(y),cos(y)) is smaller than 4σ˜ =0.08. A mode (i.e., a Gaussian) is said to be recovered if at least one high-qualitysample is assigned to it. We also measure the quality of fake samples with la-bel y by computing the 2-Wasserstein Distance (W2) [109] between pr(x|y) =N ([sin(y),cos(y)]ᵀ, σ˜ I) and pg(x|y) =N (µgy ,Σgy), where we assume pg(x|y) isGaussian and its mean and covariance are estimated by the sample mean and sam-ple covariance of 100 fake samples with label y. In Table 2.1, we report the averagepercentage of high-quality fake samples and the average percentage of recoveredmodes over 3 repetitions. We also report the average W2 over 360 testing angles.We can see CcGAN substantially outperforms cGAN.Visual results: We select 12 angles which do not appear in the training set. Wethen use cGAN and CcGAN to generate 100 samples for each unobserved angle.Fig. 2.9 visually confirms the obervation from the numerical metrics: the fakesamples from the two CcGAN methods are more realistic.Table 2.1: Average quality of 36,000 fake samples from cGAN and CcGANover three repetitions with standard deviations after the “±” symbol. “↓”(“↑”) indicates lower (higher) values are preferred. Two novel label inputmechanisms are not tested in this simple, low-dimensional simulation.Method % High Quality ↑ % Recovered Modes ↑ 2-Wasserstein Dist. ↓cGAN (120 classes) 68.8±4.8 81.8±3.9 3.32×10−2±3.13×10−2CcGAN (HVDL) 99.3± 0.4 100.0± 0.0 3.03× 10−4± 5.05× 10−5CcGAN (SVDL) 99.6± 0.1 100.0± 0.0 2.56× 10−4± 8.95× 10−62.4.2 RC-49Since most benchmark datasets in the GAN literature do not have continuous,scalar regression labels, we propose a new benchmark dataset—RC-49, a syntheticdataset created by rendering 49 3-D chair models at different yaw angles. Each of49 chair models is rendered at 899 yaw angles ranging from 0.1 to 89.9 with stepsize 0.1. Therefore, RC-49 consists of 44,051 64× 64 rendered RGB images and42(a) 1200 training samples and 120means (red dots).(b) cGAN(c) CcGAN (HVDL) (d) CcGAN (SVDL)Figure 2.9: Visual results for the Circular 2-D Gaussians simulation. (a)shows 1,200 training samples from 120 Gaussians, with 10 samples perGaussian. In (b) to (d), each GAN generates 100 fake samples at eachof 12 means not appearing in the training set, where green and blue dotsstand for fake and real samples respectively.43899 distinct angles. Please see Supp. A.3 for more details of the data generation.Some example images are shown in Fig. 2.10.Experimental setup: Not all images are used for the cGAN and CcGAN training.A yaw angle is selected for training if its last digit is odd. Moreover, at eachselected angle, only 25 images are randomly chosen for training. Thus, the trainingset includes 11250 images and 450 distinct angles. The remaining images are heldout for evaluation.When training cGAN, we divide [0.1,89.9] into 150 equal intervals where eachinterval is treated as a class. When training CcGAN, we use the rule of thumb for-mulae in Remark 2.2.3 to select the three hyper-parameters of HVDL and SVDL,i.e., σε ≈ 0.047, κ ≈ 0.004 and ν = 50625. The two novel label input mecha-nisms for CcGAN (NLI and ILI) are implemented in this experiment. For ILI, wepre-train a modified ResNet-34 [51] with 3 linear layers after the average poolinglayer and we only keep the last linear layer for label embedding (i.e., the T1 +T2in Fig. 2.4). We use a five-layer MLP with 128 nodes in each layer to convertan angle into its hidden representation (i.e., the T3 in Fig. 2.5). Both cGAN andCcGAN are trained for 30,000 iterations with batch size 256. Afterwards, we eval-uate the trained GANs on all 899 angles by generating 200 fake images for eachangle. Please see Supp. A.3 for the network architectures and more details aboutthe training/testing setup.Quantitative and visual results: To evaluate (1) the visual quality, (2) the intra-label diversity, and (3) the label consistency of fake images, we study an overallmetric and three separate metrics here. (i) Intra-FID [94] is utilized as the overallmetric. It computes the FID [52] separately at each of the 899 evaluation anglesand reports the average FID score along with the standard deviation of these 899FIDs. (ii) Naturalness Image Quality Evaluator (NIQE) [93] measures the vi-sual quality only. (iii) Diversity is the average entropy of predicted chair types offake images over evaluation angles. (iv) Label Score is the average absolute errorbetween assigned angles and predicted angles. Please see Supp. A.3.5 for detailsof these metrics.We report in Table 2.2 the performances of each GAN. The example fake im-ages in Fig. 2.10 and line graphs in Fig. 2.11 support the quantitative results. cGANoften generates unrealistic, identical images for a target angle (i.e., low visual qual-44ity and low intra-label diversity). “Binning” [0.1,89.9] into other number of classes(e.g., 90 classes and 210 classes) is also tried but does not improve cGAN’s perfor-mance. In contrast, strikingly better visual quality and higher intra-label diversityof the four CcGAN methods are visually evident. Moreover, both ILI-based Cc-GANs outperform the two NLI-based CcGANs in terms of all four metrics, espe-cially the label score.Table 2.2: Average quality of 179,800 fake RC-49 images from cGAN andCcGAN with standard deviations after the “±” symbol. “↓” (“↑”) indi-cates lower (higher) values are preferred.Method Intra-FID ↓ NIQE ↓ Diversity ↑ Label Score ↓cGAN (150 classes) 1.720±0.384 2.731±0.162 0.779±0.199 4.815±5.152CcGAN (HVDL+NLI) 0.612±0.145 1.869±0.181 2.353±0.121 5.617±4.452CcGAN (SVDL+NLI) 0.515±0.181 1.853±0.159 2.610±0.113 4.982±4.439CcGAN (HVDL+ILI) 0.424± 0.081 1.805± 0.179 2.814± 0.052 1.816± 1.481CcGAN (SVDL+ILI) 0.389± 0.095 1.783± 0.173 2.949± 0.069 1.940± 1.489Extra experimental results: To test cGAN and CcGAN under more challengingscenarios, we vary the sample size for each distinct angle in the training set from45 to 5. We visualize the line graphs of Intra-FID versus the sample size for eachdistinct training angle in Fig. 2.12. From this figure, we can see the four CcGANmethods substantially outperform cGAN and ILI performs better than NLI no mat-ter what is the sample size for each distinct angle in the training set. The overalltrend in this figure also shows that smaller sample size reduces the performance ofboth cGAN and CcGAN.2.4.3 UTKFaceIn this section, we compare CcGAN and cGAN on UTKFace [164], a dataset con-sisting of RGB images of human faces which are labeled by age.Experimental setup: In this experiment, we only use images with age in [1,60].Some images with bad visual quality and watermarks are also discarded. Afterthe preprocessing, 14,760 images are left. The number of images for each ageranges from 50 to 1051. We resize all selected images to 64×64. Some exampleUTKFace images are shown in the first image array in Fig. 2.13.45Figure 2.10: Three RC-49 example images for each of 6 angles: real imagesand example fake images from cGAN and four proposed CcGANs, re-spectively. CcGANs produce chair images with higher visual qualityand more diversity.46Figure 2.11: Line graphs of FID/NIQE/Diversity/Lable Score versus yaw an-gle for RC-49. Figs. (a) to (c) show that four CcGAN methods con-sistently outperform cGAN across all angles. Fig. (d) shows that twoILI-based CcGANs outperform cGAN at most angles and the graph ofcGAN is very unstable. All graphs of CcGANs appear much smootherthan those of cGAN because of HVDL and SVDL. Moreover, in mostgraphs, we can clearly see ILI-based CcGANs perform better thanNLI-based CcGANs.When implementing cGAN, each age is treated as a class. For CcGAN we usethe rule of thumb formulae in Remark 2.2.3 to select the three hyper-parametersof HVDL and SVDL, i.e., σε ≈ 0.041, κ ≈ 0.017 and ν = 3600. Similar to theRC-49 experiment, we use NLI and ILI to incorporate ages into CcGAN. BothcGAN and CcGAN are trained for 40,000 iterations with batch size 512. In testing,we generate 1,000 fake images from each trained GAN for each age. Please seeSupp. A.4 for more details of the data preprocessing, network architectures andtraining/testing setup.47Figure 2.12: Line graphs of Intra-FID versus the sample size for each distincttraining angle of RC-49. The grey vertical dashed line stands for thesample size used in the main study of the RC-49 experiment. Four Cc-GAN methods substantially outperform cGAN and ILI performs betterthan NLI no matter what the sample size for each distinct angle in thetraining set. The overall trend in this figure shows that a smaller samplesize deteriorates the performance of both cGAN and CcGAN.Quantitative and visual results: Similar to the RC-49 experiment, we evaluatethe quality of fake images by Intra-FID, NIQE, Diversity (entropy of predictedraces), and Label Score. We report in Table 2.3 the average quality of 60,000 fakeimages. From this table, we can see the four CcGAN methods substantially out-perform cGAN and ILI performs better than NLI. We also show in Fig. 2.13 someexample fake images from cGAN and CcGAN and line graphs of FID/NIQE/Di-versity/Lable Score versus Age in Fig. 2.14. Analogous to the quantitative com-parisons, we can see that CcGAN performs much better than cGAN.Extra experimental results: The histogram in Fig. A.5 shows that the UTKFacedataset is highly imbalanced. To balance the training data and also test the perfor-48Figure 2.13: Three UTKFace example images for each of 6 ages: real imagesand example fake images from cGAN and four proposed CcGANs, re-spectively. CcGANs produce face images with higher visual qualityand more diversity.49Table 2.3: Average quality of 60,000 fake UTKFace images from cGAN andCcGAN with standard deviations after the “±” symbol. “↓” (“↑”) indi-cates lower (higher) values are preferred.Method Intra-FID ↓ NIQE ↓ Diversity ↑ Label Score ↓cGAN (60 classes) 4.516±0.965 2.315±0.306 0.254±0.353 11.087±8.119CcGAN (HVDL+NLI) 0.572±0.167 1.739±0.145 1.338± 0.178 9.782±7.166CcGAN (SVDL+NLI) 0.547±0.181 1.753±0.196 1.326± 0.198 10.739±8.340CcGAN (HVDL+ILI) 0.480± 0.145 1.709± 0.169 1.280±0.203 7.505± 5.857CcGAN (SVDL+ILI) 0.425± 0.157 1.725± 0.171 1.298±0.176 7.452± 6.022Figure 2.14: Line graphs of FID/NIQE/Diversity/Lable Score versus Age forUTKFace. The four CcGAN methods significantly outperform cGANin Figs. (a) to (c). All graphs of CcGANs also appear much smootherthan those of cGAN because of HVDL and SVDL. Fig. (d) shows thatthe ILI-based CcGANs have higher label consistency than the NLI-based CcGANs.50Figure 2.15: Line graphs of Intra-FID versus the maximum sample size foreach distinct age in the training set of UTKFace. Fig. 2.15 shows thatfour CcGAN methods perform much better than cGAN and ILI is bet-ter than NLI. Moreover, a smaller sample size deteriorates the perfor-mance of both cGAN and CcGAN.mance of cGAN and CcGAN under smaller sample sizes, we vary the maximumsample size for each distinct age in the training from 200 to 50. Note that, we do notrestrict the maximum sample size in the main study. Since we have a much smallersample size, we reduce the number of iterations for the GAN training from 40,000to 20,000 and slightly increase mκ in Remark 2.2.3 from 1 to 2 (we therefore usea wider hard/soft vicinity). We visualize the line graphs of Intra-FID versus themaximum sample size for each age of cGAN and CcGAN in Fig. 2.15. From thefigure, we can clearly see that a smaller sample size worsens the performance ofboth cGAN and CcGAN. Moreover, the Intra-FID scores of cGAN always stayat a high level and are much larger than those of the four CcGAN methods. TheILI-based CcGANs are also better than the NLI-based CcGANs.512.4.4 Cell-200In addition to RC-49, we propose another benchmark dataset—Cell-200, a datasetof synthetic fluorescence microscopy images with cell populations generated bySIMCEP [72]. Please see Supp. A.5.1 for more details about the data generation.Some example images are shown in Fig. 2.16.Experimental setup: The Cell-200 dataset consists of 200,000 64×64 grayscaleimages. The number of cells per image ranges from 1 to 200 and there are 1,000images for each cell count. However, only a subset of Cell-200 with only odd cellcounts and 10 images per count (1,000 training images in total) is used for theGAN training .When training cGAN, we divide [1,200] into 100 equal intervals where eachinterval is treated as a class. We use the rule of thumb formulae in Remark 2.2.3 toselect the three hyperparameters of HVDL and SVDL, i.e., σε ≈ 0.077, κ ≈ 0.020and ν = 2500. Both cGAN and CcGAN are trained for 5,000 iterations. After-wards, we evaluate the trained GANs on all 200 cell counts by generating 1,000fake images for each count. Please see Supp. A.5 for the network architectures andmore details about the training/testing setup.Quantitative and visual results: We evaluate the quality of fake images by Intra-FID, NIQE, and Label Score. Please note that the Diversity score is not available inthis experiment because there is no class label in Cell-200. We report in Table 2.4the average quality of 200,000 fake images from cGAN and CcGAN. We also showin Fig. 2.16 some example fake images from cGAN and CcGAN and line graphsof FID/NIQE/Label Score versus Cell Count in Figs. 2.17 to 2.19. Different fromthe experimental results on RC-49 and UTKFace, although NLI-based CcGANsoutperforms cGAN in terms of Intra-FID and NIQE, their label scores are veryhigh, which implies low label consistency. Fortunately, two ILI-based CcGANsstill perform very well and substantially outperform cGAN and the two NLI-basedCcGANs.2.4.5 Steering AngleIn this section, we demonstrate the effectiveness of the proposed CcGAN on theSteering Angle dataset, a subset of an autonomous driving dataset [20, 21]. The52Figure 2.16: Three Cell-200 images for each of 6 cell counts absent in thetraining data: real images and example fake images from cGAN andfour proposed CcGANs, respectively. cGAN has severe mode collapseproblem in this experiment. Two NLI-based CcGANs do not performwell enough but two ILI-based CcGANs produce images with highervisual quality, more diversity, and higher label consistency.53Table 2.4: Average quality of 200,000 fake Cell-200 images from cGAN andCcGAN with standard deviations after the “±” symbol. “↓” (“↑”) indi-cates lower (higher) values are preferred.Method Intra-FID ↓ NIQE ↓ Label Score ↓cGAN (100 classes) 90.255±64.595 2.130±2.440 66.748±51.711CcGAN (HVDL+NLI) 50.052±20.584 1.488±0.153 72.599±37.425CcGAN (SVDL+NLI) 56.078±19.334 1.8289±0.3856 83.367±49.577CcGAN (HVDL+ILI) 8.759± 6.652 1.283± 0.534 5.861± 4.900CcGAN (SVDL+ILI) 7.266± 2.305 1.220± 0.515 5.905± 5.020Figure 2.17: Line graphs of FID Score versus Cell Count for Cell-200.54Figure 2.18: Line graphs of NIQE Score versus Cell Count for Cell-200.complete autonomous driving dataset consists of 109,231 RGB images. Each im-age is taken by using a dash camera mounted on a car and, at the same moment, theangle of the steering wheel rotation of the same car (i.e., steering angle) is recordedby a device attached to the steering wheel. Thus, each image in this autonomousdriving dataset is paired with a steering angle ranging from −338.82◦ to 501.78◦.Experimental setup: To make the training and evaluation easier, we remove manyimages in this autonomous driving dataset where an image is removed due to atleast one of the following reasons:• The image is incorrectly labeled (e.g., some images show that the car wasturning left/right but the corresponding steering angles are zero).• The image has very bad visual quality due to overexposure or underexposure.• There is no reference object (e.g., double yellow lines or side roads) in theimage to let a human visually determine whether the car was turning left-/right.55Figure 2.19: Line graphs of Label Score versus Cell Count for Cell-200.• The corresponding steering angle is outside [−80◦,80◦].Eventually, there are 12,271 images left with 1,904 distinct steering angles in[−80◦,80◦]. These images are then resized to 64×64 and they form a subset of theautonomous driving dataset [20], termed Steering Angle in this chapter. Please notethat the Steering Angle dataset is highly imbalanced and a histogram of steeringangles is shown in Fig. A.11.When training cGAN, we divide [−80◦,80◦] into 210 equal intervals whereeach interval is taken as a class. When implementing CcGAN, the three hyper-parameters of HVDL and SVDL are selected by the rule of thumb formulae inRemark 2.2.3, i.e., σε ≈ 0.029, κ ≈ 0.032 and ν ≈ 1000.438. Both cGAN andCcGAN are trained for 20,000 iterations. To evaluate the candidate models, wechoose 2,000 evenly spaced angles in [−80◦,80◦] and generate 50 images fromeach candidate GAN model for each of these angles. Please see Supp. A.6 for thenetwork architectures and more details about the training/testing setup.Quantitative and visual results: To evaluate the quality of fake images, we use the56proposed Sliding Fre´chet Inception Distance (SFID) as the overall metric insteadof Intra-FID, since we have very few real images for many angles (e.g., angles closeto the two end points of [−80◦,80◦]). We preset 1,000 SFID centers in [−80◦,80◦]and let the SFID radius be 2◦. NIQE, Diversity (entropy of predicted types ofscenes), and Label Score are also reported. Please see Supp. A.6.5 for more detailsof these performance measures.We report in Table 2.5 the average quality of 100,000 fake images from eachcandidate method. Some example fake images are also shown in Fig. 2.20. We alsocompute FID, NIQE, Diversity, and Label Score in each SFID interval and plot theline graphs of FID/NIQE/Diversity/Label Score versus SFID Center in Fig. 2.21.Based on these quantitative and visual results, we can conclude:• The two ILI-based CcGAN methods are better than cGAN in terms of allfour metrics; however, the two NLI-based CcGAN methods have lower labelconsistency than cGAN.• Although the NIQE score and Label Score of cGAN are not grossly uncom-petitive, cGAN has a very low Diversity score and Fig. 2.21(c) shows thatthe Diversity scores are almost zero at some angles. Example fake images inFig. 2.20 also show that cGAN has the mode collapse problem [4, 17, 132](i.e., it always generates the same image for some angles).• The line graphs in Fig. 2.21 show that the performance of cGAN is veryunstable across all SFID intervals.• The two ILI-based CcGANs perform better than the two NLI-based CcGANsin terms of all metrics except NIQE.2.4.6 Effectiveness of SFIDIn this section, we study the effectiveness of SFID on RC-49. Since RC-49 has alarge enough sample size of real images, we can get a reliable Intra-FID. At thesame time, we can also deliberately reduce the sample size of real images to mimicthe scenario where a reliable Intra-FID is not applicable but SFID still works well.The experiment in this section can also be conducted on Cell-200 but is omitted in57Figure 2.20: Three Steering Angle images for each of 6 angles: real imagesand example fake images from cGAN and four proposed CcGANs,respectively. cGAN has severe mode collapse problem in this exper-iment. Two NLI-based CcGANs do not work well but two ILI-baesdCcGANs produce images with higher visual quality and more diver-sity.58Table 2.5: Average quality of 100,000 fake Steering Angle images fromcGAN and CcGAN with standard deviations after the “±” symbol. “↓”(“↑”) indicates lower (higher) values are preferred.Method SFID ↓ NIQE ↓ Diversity ↑ Label Score ↓cGAN (210 classes) 3.285±0.647 1.296±0.095 0.603±0.396 14.596±15.402CcGAN (HVDL+NLI) 1.969±0.676 1.093± 0.024 0.991±0.361 22.322±18.758CcGAN (SVDL+NLI) 1.866±0.649 1.098± 0.038 1.007±0.248 19.678±18.281CcGAN (HVDL+ILI) 1.635± 0.699 1.152±0.047 1.153± 0.153 10.868± 9.644CcGAN (SVDL+ILI) 1.546± 0.626 1.130±0.078 1.156± 0.189 10.933± 8.978Figure 2.21: Line graphs of FID/NIQE/Diversity/Lable Score versus SFIDCenter for the Steering Angle dataset. To plot these line graphs, wecompute these metrics within each SFID interval defined by the corre-sponding SFID center. Figs. (a) to (d) show that, although the NLI-based CcGANs do not have good label consistency, the ILI-based Cc-GANs substantially outperform cGAN in most SFID intervals. Allgraphs of CcGANs also appear much smoother than those of cGANbecause of HVDL and SVDL.59this thesis. We study 11 combinations of the SFID radius (rSFID) and the number ofSFID centers (# cSFID) in this experiment where we use SFID to evaluate cGAN andtwo CcGAN methods (i.e., HVDL+NLI and HVDL+ILI) pre-trained in Section2.4.2. In Setting 1, we let rSFID = 0 so SFID degenerates to Intra-FID. In the samesetting, we evaluate the three GANs on all 899 distinct angles and all real imagesin RC-49 are used to compute Intra-FID, so Setting 1 is taken as the oracle in thisexperiment. In Setting 2, we also let rSFID = 0 so SFID degenerates to Intra-FIDagain. In Setting 2, however, we only evaluate GANs on the 450 distinct angleswhich are seen in the training set of the experiment in Section 2.4.2. Moreover, tosimulate the scenario where we have very few real images to compute Intra-FID,we deliberately reduce the number of real images at each distinct angle from 49to 10. Therefore, in Setting 2, there are 10 real images for each angle seen in thetraining set and 0 real image for each angle unseen in the training set. Setting 2 istreated as the baseline in this experiment. Settings 3 to 11 are designed to show theeffectiveness of SFID so we let rSFID > 0. Similar to Setting 2, from Settings 3 to11, real images are available only for those 450 distinct angles seen in the trainingset and only 10 real images are available for each angle. We consider three valuesfor rSFID (i.e., 0.5, 1, and 2) and three values for the number of cSFID’s (i.e., 400,600, and 800).For all settings, we compute one FID in each SFID interval (in Settings 1 and2, the SFID interval degenerates to an angle) and report in Table 2.6 the meanof these FIDs along with their standard deviation after the “±” symbol. Setting1 is the oracle setting whose evaluation results can be seen as the ground truth,and we hope the evaluation results of SFID are close to Setting 1. In Setting 2,when we have very few real images (even zero) for each angle, Intra-FID over-estimates the average FID of each GAN (e.g., from 1.7201 to 1.9664 for cGAN)and underestimates the quantitative difference between cGAN and CcGANs (e.g.,from (1.7201− 0.6119)/1.7201 ≈ 64.4% to (1.9664− 1.2102)/1.9664 ≈ 38.5%for cGAN and HVDL+NLI). However, the performance of our proposed SFID inSettings 3 to 11 is very close to the oracle setting. If we compare within Settings3 to 11, we can see rSFID is inversely proportional to the SFID score while # cSFIDdoes not have obvious influence on SFID. From Table 2.6, we may conclude that aslong as rSFID is set at a moderate level, SFID is a valid proxy to the oracle Intra-FID60when there are insufficient real images to compute Intra-FID.Table 2.6: Evaluation results of SFID on RC-49 under different setups ofrSFID and number of cSFID’s. In the first two settings, SFID degener-ates to Intra-FID since rSFID = 0. In Setting 1, we evaluate GANs on all899 distinct angles and all real images are used to compute Intra-FID, soSetting 1 is the oracle setting. In Setting 2, we evaluate GANs on the450 angles seen in the training set and, for each angle, 10 real images areused to compute Intra-FID, so Setting 2 is treated as the baseline. Theperformance of our proposed SFID (Settings 3 to 11) is close to the ora-cle setting while Intra-FID (Setting 2) tends to overestimate the averageFID and underestimate the quantitative difference between cGAN andCcGANs.Settings rSFID # cSFID cGAN HVDL+NLI HVDL+ILI1 (Oracle) 0 899 1.7201±0.3843 0.6119±0.1452 0.4241±0.08112 (Baseline) 0 450 1.9664±0.4970 1.2102±0.2976 1.0318±0.30353 0.5 400 1.7642±0.4010 0.6748±0.1445 0.4257±0.08674 1 400 1.7258±0.3790 0.6182±0.1233 0.4121±0.07855 2 400 1.6845±0.3560 0.5814±0.1040 0.4012±0.07296 0.5 600 1.7641±0.3993 0.6747±0.1435 0.4247±0.08657 1 600 1.7266±0.3781 0.6176±0.1228 0.4120±0.07838 2 600 1.6854±0.3566 0.5816±0.1040 0.4014±0.07269 0.5 800 1.7649±0.3994 0.6748±0.1436 0.4254±0.086910 1 800 1.7257±0.3782 0.6175±0.1232 0.4120±0.078311 2 800 1.6855±0.3566 0.5817±0.1039 0.4014±0.07272.5 Theoretical Analysis for HVDL and SVDLIn this section, we theoretically analyze L̂ HVDL and L̂ SVDL’s properties by deriv-ing the error bounds of D trained with L̂ HVDL and L̂ SVDL. To illustrate the deriva-tion precisely, we make some definitions and notations in Section 2.5.1, which areonly valid locally in Section 2.5. Some necessary assumptions, lemmas, and theo-rems are also introduced or derived in Sections 2.5.2 and 2.5.3. The main theoremson the error bounds of D are derived in Section 2.5.4.612.5.1 Some Necessary Definitions and NotationsThis section summarizes some necessary definitions and notations used in thederivation. Please note that all these definitions and notations are valid in Sec-tion 2.5 only.• Unlike other contents of this thesis, we use different symbols to denote ran-dom variables/vectors and the fixed values that random variables/vectorsmay take. Specifically, a random image and a random label are representedrespectively by a bold capital X and a capital Y . A sequence of N randomimage-label pairs are represented by (X 1,Y1), . . . ,(X N ,YN). Please note thatsome subscripts or superscripts may apply to X and Y to provide some extrainformation. An observed (fixed) image and an observed (fixed) label aredenoted respectively by a bold lowercase x and a lowercase y. Moreover,without loss of generality, we assume Y,y ∈ [0,1], i.e., Y = [0,1].• Let p(x|Y = y) denote the conditional PDF of X given the occurrence of thevalue y of Y . p may have superscripts or subscripts to provide some extrainformation.• Let p(y′|Y = y) denote the conditional PDF of Y ′ given the occurrence of thevalue y of Y . p may have superscripts or subscripts to provide some extrainformation.• Let D stand for the Hypothesis Space of D. D is a set of functions thatcan be represented by D (a neural network with determined architecture butundetermined weights).• Let f (x,y) =− logD(x,y) andF =− logD .• Let pˆKDEr (y) and pˆKDEg (y) stand for the KDEs of pr(y) and pg(y) respectively.• For HVDL, denote respectively bypy,κr (x) :=∫pr(x|Y = y′)1{|y′−y|≤κ}pr(y′)∫1{|y′−y|≤κ}pr(y′)dy′dy′62andpy,κg (x) :=∫pg(x|Y = y′)1{|y′−y|≤κ}pg(y′)∫1{|y′−y|≤κ}pg(y′)dy′dy′the PDFs of the marginal distributions for real and fake images with labelsin [y−κ,y+κ].• For SVDL, given y and the weight functions (E.q. (2.8)), if the numberof real and fake images are infinite, the real and fake empirical densitiesconverges topy,wrr (x) :=∫pr(x|Y = y′)wr(y′,y)pr(y′)W r(y)dy′andpy,wgg (x) :=∫pg(x|Y = y′)wg(y′,y)pg(y′)W g(y)dy′respectively, whereW r(y) :=∫wr(y′,y)pr(y′)dy′,W g(y) :=∫wg(y′,y)pg(y′)dy′,and wr and wg are the weight functions defined in Eq. (2.8) as followswr(y′,y) = e−ν(y′−y)2 and wg(y′,y) = e−ν(y′−y)2 .We also letprw(y′|Y = y) := wr(y′,y)pr(y′)W r(y)andpgw(y′|Y = y) := wg(y′,y)pg(y′)W g(y).• The Ho¨lder Class defined below is a set of functions with bouned secondderivatives, which controls the variation of the function when parameterchanges.63Definition 2.5.1. (Ho¨lder Class) Define the Ho¨lder class of functions as:Σ(L) :={p : ∀t1, t2 ∈ Y ,∃L > 0,s.t. |p′(t1)− p′(t2)||t1− t2| ≤ L}. (2.13)• With some new notations above, we restate the theoretical discriminatorlossesL (D) as follows:L (D) =−EY∼pr(y)[EX∼pr(x|Y ) [log(D(X ,Y ))]]−EY∼pg(y)[EX∼pg(x|Y ) [log(1−D(X ,Y ))]]. (2.14)• Recall that, given a G, the optimal discriminator which minimizes L (D) isin the form ofD∗(x,y) =pr(x,y)pr(x,y)+ pg(x,y). (2.15)However, D∗ may not be covered by the hypothesis space D . Define D˜,D̂HVDL, and D̂SVDL as followsD˜ := argminD∈DL (D),D̂HVDL := argminD∈DL̂HVDL(D),D̂SVDL := argminD∈DL̂SVDL(D).Note that L (D˜)−L (D∗) should be a non-negative constant. In CcGAN,we minimize L̂ HVDL(D) or L̂ SVDL(D) with respect to D ∈ D , so we areinterested in the distance of D̂HVDL and D̂SVDL from D∗ respectively, i.e.,L (D̂HVDL)−L (D∗) andL (D̂SVDL)−L (D∗).2.5.2 Some Necessary AssumptionsIn this theoretical analysis, we work with the following assumptions:(A1) All D’s in D are measurable and uniformly bounded. LetU := max{ supD∈D[− logD] , supD∈D[− log(1−D)]}and U < ∞;64(A2) For ∀x ∈ X and y,y′ ∈ Y , ∃gr(x) > 0 and Mr > 0, s.t. |pr(x|Y = y′)−pr(x|Y = y)| ≤ gr(x)|y′− y| with∫gr(x)dx = Mr;(A3) For ∀x ∈ X and y,y′ ∈ Y , ∃gg(x) > 0 and Mg > 0, s.t. |pg(x|Y = y′)−pg(x|Y = y)| ≤ gg(x)|y′− y| with∫gg(x)dx = Mg;(A4) pr(y) ∈ Σ(Lr) and pg(y) ∈ Σ(Lg).2.5.3 Some Necessary Lemmas and TheoremsIn this section, we first introduce the Hoeffding’s inequality that is widely usedlater to derive some lemmas.Theorem 2.5.1 (Hoeffding’s inequality [123]). Let Z1, . . . ,Zm be a sequence ofi.i.d. random variables and let Z¯ = 1m ∑mi=1 Zi. Assume that E[Z¯] = µ and Pr(a ≤Zi ≤ b) = 1 for every i. Then, for any ε > 0Pr[∣∣∣∣∣ 1m m∑i=1 Zi−µ∣∣∣∣∣> ε]≤ 2exp(− 2mε2(b−a)2).Proof. Please see [123, Lemma B.6] for the proof.Remark 2.5.1. Let δ = 2exp(− 2mε2(b−a)2), then ε =√12m log( 2δ). Thus, we canget another form of the Hoeffding’s inequality. For ∀δ ∈ (0,1), with probability atleast 1−δ , we have ∣∣∣∣∣ 1m m∑i=1 Zi−µ∣∣∣∣∣≤√12mlog(2δ).Lemma 2.5.1 (Lemma for HVDL). Suppose that (A1)–(A2) and (A4) hold andlet (X 1,Y1), . . . ,(X N ,YN) be a sequence of i.i.d. random image-label pairs, then∀δ ∈ (0,1), with probability at least 1−δ ,supD∈D∣∣∣∣∣ 1Ny,κN∑i=11{|y−Yi|≤κ} [− logD(X i,y)]−EX∼p(x|Y=y) [− logD(X ,y)]∣∣∣∣∣≤U√12Ny,κlog(2δ)+κUM, (2.16)65for a fixed y. If image-label pairs are real, then N = Nr, Ny,κ = Nry,κ , p = pr, andM = Mr. Similarly, we have N = Ng, Ny,κ = Ngy,κ , p = pg, and M = Mg for fakeimage-label pairs.Proof. Triangle inequality yieldssupD∈D∣∣∣∣∣ 1Ny,κN∑i=11{|y−Yi|≤κ} [− logD(X i,y)]−EX∼pr(x|Y=y) [− logD(X ,y)]∣∣∣∣∣≤ supD∈D∣∣∣∣∣ 1Ny,κN∑i=11{|y−Yi|≤κ} [− logD(X i,y)]−EX∼py,κ (x) [− logD(X ,y)]∣∣∣∣∣+ supD∈D∣∣EX∼py,κ (x) [− logD(X ,y)]−EX∼p(x|Y=y) [− logD(X ,y)]∣∣We then bound the two terms of the RHS separately as follows:1. Real images with labels in [y−κ,y+κ] can be seen as independent samplesfrom py,κ(x). Then the first term can be bounded by applying Hoeffding’sinequality as follows: ∀δ ∈ (0,1), with at least probability 1−δ ,supD∈D∣∣∣∣∣ 1Ny,κN∑i=11{|y−Yi|≤κ}[U− logD(X i,y)U]−EX∼py,κ (x)[U− logD(X ,y)U]∣∣∣∣∣≤U√12Ny,κlog(2δ).(2.17)2. For the second term, we havesupD∈D∣∣EX∼py,κ (x) [− logD(X ,y)]−EX∼p(x|Y=y) [− logD(X ,y)]∣∣= supD∈D∣∣∣∣∫ [− logD(x,y)] · [py,κ(x)− p(x|Y = y)]dx∣∣∣∣≤ supD∈D∫|− logD(x,y)| · |py,κ(x)− p(x|Y = y)|dx≤U∫|py,κ(x)− p(x|Y = y)|dx. (2.18)Then, we focus on |py,κ(x)− p(x|Y = y)|. By the definition of py,κ(x) and66defining pκ(y′) =1{|y′−y|≤κ}p(y′)∫1{|y′−y|≤κ}p(y′)dy′, we have|py,κ(x)− p(x|Y = y)|=∣∣∣∣∫ p(x|Y = y′)pκ(y′)dy′− p(x|Y = y)∣∣∣∣≤∫ ∣∣p(x|Y = y′)− p(x|Y = y)∣∣ pκ(y′)dy′(by (A2), and let g = gr for real images and g = gg for fake images)≤∫g(x)|y′− y|pκ(y′)dy′≤κg(x).Thus, Eq. (2.18) is upper bounded as follows,supD∈D∣∣EX∼py,κ (x) [− logD(X ,y)]−EX∼p(x|Y=y) [− logD(X ,y)]∣∣≤U∫κg(x)dx(by (A2))=κUM. (2.19)By combining Eq. (2.17) and (2.19), we can get Eq. (2.16), which finishes theproof.Lemma 2.5.2 (Lemma for SVDL). Suppose that (A1), (A2) and (A4) hold andlet (X 1,Y1), . . . ,(X N ,YN) be a sequence of i.i.d. random image-label pairs, then∀δ ∈ (0,1), with probability at least 1−δ ,supD∈D∣∣∣∣∣ 1N ∑Ni=1 w(Yi,y) [− logD(X i,y)]1N ∑Ni=1 w(Yi,y)−EX∼p(x|Y=y) [− logD(X ,y)]∣∣∣∣∣≤ 2UW (y)√12Nlog(4δ)+UMEY ′∼pw(Y ′|Y=y)[|Y ′− y|] , (2.20)for a fixed y. If image-label pairs are real, then N = Nr, Ny,κ = Nry,κ , p = pr,pw = prw, w = wr, W =W r, and M = Mr. Similarly, we have N = Ng, Ny,κ = Ngy,κ ,67p = pg, pw = pgw, w = wg, W =W g, and M = Mg for fake image-label pairs.Proof. Triangle inequality yieldssupD∈D∣∣∣∣∣ 1N ∑Ni=1 w(Yi,y) [− logD(X i,y)]1N ∑Ni=1 w(Yi,y)−EX∼pr(x|Y=y) [− logD(X ,y)]∣∣∣∣∣(Recall f (x,y) =− logD(x,y) andF =− logD .)= supf∈F∣∣∣∣∣ 1N ∑Ni=1 w(Yi,y) f (X i,y)1N ∑Ni=1 w(Yi,y)−EX∼pr(x|Y=y) [ f (X ,y)]∣∣∣∣∣≤ supf∈F∣∣∣∣∣ 1N ∑Ni=1 w(Yi,y) f (X i,y)1N ∑Ni=1 w(Yi,y)−EX∼py,w(x) [ f (X ,y)]∣∣∣∣∣+ supf∈F∣∣EX∼py,w(x) [ f (X ,y)]−EX∼pr(x|Y=y) [ f (X ,y)]∣∣ . (2.21)(py,w = py,wrr for real images and py,w = py,wgg for fake images)We then derive bounds for both terms on the RHS of the inequality.1. For the first term, we can further split it into two parts,∣∣∣∣∣ 1N ∑Ni=1 w(Yi,y) f (X i,y)1N ∑Ni=1 w(Yi,y)−EX∼py,w(x) [ f (X ,y)]∣∣∣∣∣≤∣∣∣∣∣ 1N ∑Ni=1 w(Yi,y) f (X i,y)1N ∑Ni=1 w(Yi,y)−1N ∑Ni=1 w(Yi,y) f (X i,y)W (y)∣∣∣∣∣+∣∣∣∣∣ 1N ∑Ni=1 w(Yi,y) f (X i,y)W (y) −EX∼py,w(x) [ f (X ,y)]∣∣∣∣∣ (2.22)Focusing on the first part of RHS of Eq.(2.22). By (A1),∣∣∣∣∣ 1N ∑Ni=1 w(Yi,y) f (X i,y)1N ∑Ni=1 w(Yi,y)−1N ∑Ni=1 w(Yi,y) f (X i,y)W (y)∣∣∣∣∣≤U∣∣ 1N ∑Ni=1 w(Yi,y)−W (y)∣∣W (y)Note that ∀y,y′,w(y′,y) = e−ν |y−y′|2 ≤ 1 (since ν > 0) and hence given y,w(Y ′,y) is a random variable bounded by 1. Moreover, given y, W (y) is the68expectation of w(Y ′,y). Then, apply Hoeffding’s inequality to the numeratorof above, yielding that with probability at least 1−δ ′,∣∣∣∣∣ 1N N∑i=1 w(Yi,y)−W (y)∣∣∣∣∣≤√12Nlog(2δ ′).Thus, by the boundedness of f , with probability at least 1−δ ′,∣∣∣∣∣ 1N ∑Ni=1 w(Yi,y) f (X i,y)1N ∑Ni=1 w(Yi,y)−1N ∑Ni=1 w(Yi,y) f (X i,y)W (y)∣∣∣∣∣≤ UW (y)√12Nlog(2δ ′). (2.23)Then, consider the second part of RHS of Eq.(2.22). Recall that py,w(x) :=∫p(x|Y = y′)w(y′,y)p(y′)W (y) dy′. Thus,∣∣∣∣∣ 1N ∑Ni=1 w(Yi,y) f (X i,y)W (y) −EX∼py,w(x) [ f (X ,y)]∣∣∣∣∣=1W (y)∣∣∣∣∣ 1N N∑i=1 w(Yi,y) f (X i,y)−E(X ,Y ′)∼p(x,y′) [wr(Y ′,y) f (X i,y)]∣∣∣∣∣ ,where p(x,y′) = p(x|Y = y′)p(y′) denotes PDF of the joint distribution ofreal images and labels. Again, since w(Y ′,y) f (X i,y) is uniformly boundedby U under (A1), we can apply Hoeffding’s inequality. This implies thatwith probability at least 1−δ ′, the above can be upper bounded byUW (y)√12Nlog(2δ ′). (2.24)Combining Eq. (2.23) and (2.24) and by setting δ ′ = δ2 , we have with prob-ability at least 1−δ ,∣∣∣∣∣ 1N ∑Ni=1 w(Yi,y) f (X i,y)1N ∑Ni=1 w(Yi,y)−EX∼py,w(x) [ f (X ,y)]∣∣∣∣∣≤ 2UW (y)√12Nlog(4δ).69Since this holds for ∀ f ∈F , taking supremum over f , we havesupf∈F∣∣∣∣∣ 1N ∑Ni=1 w(Yi,y) f (X i,y)1N ∑Ni=1 w(Yi,y)−EX∼py,w(x) [ f (X ,y)]∣∣∣∣∣≤ 2UW (y)√12Nlog(4δ). (2.25)2. For the second term on the RHS of Eq. (2.21). By (A1) that | f | ≤U ,supf∈F∣∣EX∼py,w(x) [ f (X ,y)]−EX∼p(x|Y=y) [ f (X ,y)]∣∣(See Eq. (2.18))≤U∫|py,w(x)− p(x|Y = y)|dx.Note that by the definition ofpy,w(x) :=∫p(x|Y = y′)w(y′,y)p(y′)W (y)dy′andpw(y′|Y = y) := w(y′,y) pr (y′)W r(y),we have|py,w(x)− p(x|Y = y)|=∣∣∣∣∫ p(x|Y = y′)pw (y′|Y = y)dy′− p(x|Y = y)∣∣∣∣≤∫ ∣∣p(x|Y = y′)− p(x|Y = y)∣∣ pw (y′|Y = y)dy′.By (A.2) and y ∈ [0,1], the above is upper bounded byg(x)EY ′∼pw(y′|Y=y)[|y−Y ′|] .Thus,supf∈F∣∣EX∼py,w(x) [ f (X ,y)]−EX∼p(x|Y=y) [ f (X ,y)]∣∣70≤U∫g(x)EY ′∼pw(y′|Y=y)[|Y ′− y|]dx=UMEY ′∼pw(y′|Y=y)[|Y ′− y|] . (2.26)Therefore, combining both Eq.(2.25) and (2.26), with probability at least 1−δ ,supD∈D∣∣∣∣∣ 1N ∑Ni=1 w(Yi,y) [− logD(X i,y)]1N ∑Ni=1 w(Yi,y)−EX∼p(x|Y=y) [− logD(X ,y)]∣∣∣∣∣≤ 2UW (y)√12Nlog(4δ)+UMEY ′∼pw(y′|Y=y)[|Y ′− y|] .This finishes the proof.As introduced in Section 2.2, we use KDE for the PDF of the marginal labeldistribution with Gaussian kernel. The next theorem characterizes the differencebetween a pr(y), pg(y) and their KDE using N i.i.d. samples.Theorem 2.5.2. Let pˆKDEr (y) and pˆKDEg (y) stand for the KDE of pr(y) and pg(y)respectively. Under condition (A4), if the KDEs are based on N i.i.d. samples frompr/pg and a bandwidth σε , for all δ ∈ (0,1), with probability at least 1−δ ,supt∣∣pˆKDEr (y)− pr(y)∣∣≤√CKDE1,δ logNNσε+Lrσ2ε , (2.27)supt∣∣pˆKDEg (y)− pg(y)∣∣≤√CKDE2,δ logNNσε+Lgσ2ε , (2.28)for some constants CKDE1,δ ,CKDE2,δ depending on δ .Proof. By ([151]; P.12), for any p(t) ∈ Σ(L) (the Ho¨lder Class, see Definition2.5.1), with probability at least 1−δ ,supt∣∣pˆKDE(t)− p(t)∣∣≤√CKDEδ logNNσε+ cσ2ε , (2.29)for some constants CKDEδ and c, where C depends on δ and c = L∫K(s)|s|2ds.71Since in this work, K is chosen as Gaussian kernel, c = L∫K(s)|s|2ds = L.Based on above lemmas and theorems, we derive the following two theorems,which will be used in the derivation of the error bounds of D trained with HVDLand SVDL in Section 2.5.4.Theorem 2.5.3. Assume that (A1)-(A4) hold, then ∀δ ∈ (0,1), with probability atleast 1−δ ,supD∈D∣∣∣L̂ HVDL(D)−L (D)∣∣∣≤U√CKDE1,δ logNrNrσε+Lrσ2ε+U√CKDE2,δ logNgNgσε+Lgσ2ε+κU(Mr +Mg)+U√12log(8δ)(EY∼pˆKDEr (y)[√1NrY,κ]+EY∼pˆKDEg (y)[√1NgY,κ]), (2.30)for some constants CKDE1,δ ,CKDE2,δ depending on δ .Proof. Let (X r1,Yr1 ), . . . ,(XrNr ,YrNr) and (Xg1,Yg1 ), . . . ,(XgNg ,YgNg) denote respectivelyreal and fake i.i.d. random image-label pairs.We first decompose supD∈D∣∣∣L̂ HVDL(D)−L (D)∣∣∣ as followssupD∈D∣∣∣L̂ HVDL(D)−L (D)∣∣∣≤ supD∈D∣∣∣∣∫ [∫ [− logD(x,y)] pr(x|Y = y)dx](pr(y)− pˆKDEr (y))dy∣∣∣∣+ supD∈D∣∣∣∣∫ [∫ [− log(1−D(x,y))] pg(x|Y = y)dx](pg(y)− pˆKDEg (y))dy∣∣∣∣+ supD∈D∣∣∣∣∣∫ [ 1Nry,κNr∑i=11{|y−Y ri |≤κ} [− logD(X ri ,y)]−EX∼pr(x|Y=y) [− logD(X ,y)]]pˆKDEr (y)dy∣∣∣∣∣+ supD∈D∣∣∣∣∣∫ [ 1Ngy,κNr∑i=11{|y−Y gi |≤κ}[− log(1−D(X gi ,y))]−EX∼pg(x|Y=y) [− log(1−D(X ,y))]]pˆKDEg (y)dy∣∣∣∣∣ .These four terms in the RHS can be bounded separately as follows1. The first term can be bounded by using Theorem 2.5.2 and the boundness of72D. For the first term, ∀δ1 ∈ (0,1), with at least probability 1−δ1,supD∈D∣∣∣∣∫ [∫ [− logD(x,y)] pr(x|Y = y)dx](pr(y)− pˆKDEr (y))dy∣∣∣∣≤U√CKDE1,δ1 logNrNrσε+Lrσ2ε , (2.31)for some constants CKDE1,δ1 depending on δ1.2. Similarly, for the second term, ∀δ2 ∈ (0,1), with at least probability 1−δ2,supD∈D∣∣∣∣∫ [∫ [− log(1−D(x,y))] pg(x|Y = y)dx](pg(y)− pˆKDEg (y))dy∣∣∣∣≤U√CKDE2,δ2 logNgNrσε+Lgσ2ε ,(2.32)for some constants CKDE2,δ2 depending on δ2.3. The third term can be bounded by using Lemma 2.5.1. For the third term,∀δ3 ∈ (0,1), with at least probability 1−δ3,supD∈D∣∣∣∣∣∫ [ 1Nry,κNr∑i=11{|y−Y ri |≤κ} [− logD(X ri ,y)]−EX∼pr(x|Y=y) [− logD(X ,y)]]pˆKDEr (y)dy∣∣∣∣∣≤∫supD∈D∣∣∣∣∣ 1Nry,κNr∑i=11{|y−Y ri |≤κ} [− logD(X ri ,y)]−EX∼pr(x|Y=y) [− logD(X ,y)]∣∣∣∣∣ pˆKDEr (y)dy≤∫ [U√12Nry,κlog(2δ3)+κUMr]pˆKDEr (y)dyNote that Nry,κ =∑Nri=11{|y−Y ri |}, which is a random variable of Yi’s. The abovecan be expressed assupD∈D∣∣∣∣∣∫ [ 1Nry,κNr∑i=11{|y−Y ri |≤κ} [− logD(X ri ,y)]−EX∼pr(x|Y=y) [− logD(X ,y)]]pˆKDEr (y)dy∣∣∣∣∣≤ κUMr +U√12log(2δ3)EY∼pˆKDEr (y)[√1NrY,κ].(2.33)734. Similarly, for the fourth term, ∀δ4 ∈ (0,1), with at least probability 1−δ4,supD∈D∣∣∣∣∣∫ {∫ [ 1Ngy,κNr∑i=11{|y−Y gi |≤κ}[− log(1−D(X gi ,y))]−EX∼pg(x|Y=y) [− log(1−D(X ,y))]]dx}pˆKDEg (y)dy∣∣∣∣∣≤κUMg+U√12log(2δ4)EY∼pˆKDEg (y)[√1NgY,κ].(2.34)With δ1 = δ2 = δ3 = δ4 = δ4 , combining Eq. (2.31) - (2.34) leads to the upperbound in Theorem 2.5.3.Theorem 2.5.4. Assume that (A1)-(A4) hold, then ∀δ ∈ (0,1), with probability atleast 1−δ ,supD∈D∣∣∣L̂ SVDL(D)−L (D)∣∣∣≤U√CKDE1,δ logNrNrσε+Lrσ2ε+U√CKDE2,δ logNgNgσε+Lgσ2ε+4U√12log(16δ)(1√NrEY∼pˆKDEr (y)[1W r(Y )]+1√NgEY∼pˆKDEg (y)[1W g(Y )])+2U(MrEY∼pˆKDEr (y)[EY ′∼prw(y′|Y )∣∣Y ′−Y ∣∣]+MgEY∼pˆKDEg (y) [EY ′∼pgw(y′|Y ) ∣∣Y ′−Y ∣∣])(2.35)for some constant CKDE1,δ ,CKDE2,δ depending on δ .Proof. Similar to the decomposition for Theorem 2.5.3, we can decompose supD∈D|L̂ SVDL(D)−L (D)| into four terms which can be bounded by using Theorem2.5.2, the boundness of D, and Lemma 2.5.2. The detail is omitted because it isalmost identical to the one of Theorem 2.5.3.2.5.4 Error Bounds of D Trained with HVDL and SVDLBased on above theorems and lemmas, we derive the error bounds of D that istrained with HVDL and SVDL respectively. The error bound is characterized by74the distance of D̂HVDL and D̂SVDL from the optimal D∗ under the theoretical dis-criminator lossL (D), i.e.,L (D̂HVDL)−L (D∗) andL (D̂SVDL)−L (D∗) respec-tively. Please see Theorem 2.5.5 and 2.5.6 for details.Theorem 2.5.5 (Error bound of D trained with HVDL). Assume that (A1)-(A4)hold, then ∀δ ∈ (0,1), with probability at least 1−δ ,L (D̂HVDL)−L (D∗)≤ 2U√CKDE1,δ logNrNrσε+Lrσ2ε+2U√CKDE2,δ logNgNgσε+Lgσ2ε+2κU(Mr +Mg)+2U√12log(8δ)(EY∼pˆKDEr (y)[√1NrY,κ]+EY∼pˆKDEg (y)[√1NgY,κ])+L (D˜)−L (D∗), (2.36)for some constants CKDE1,δ ,CKDE2,δ depending on δ .Proof. We first decomposeL (D̂HVDL)−L (D∗) as followsL (D̂HVDL)−L (D∗)=L (D̂HVDL)− L̂ (D̂HVDL)+ L̂ (D̂HVDL)− L̂ (D˜)+ L̂ (D˜)−L (D˜)+L (D˜)−L (D∗)(by L̂ (D̂HVDL)− L̂ (D˜)≤ 0)≤2 supD∈D∣∣∣L̂ HVDL(D)−L (D)∣∣∣+L (D˜)−L (D∗)(by Theorem 2.5.3)≤2U√CKDE1,δ logNrNrσε+Lrσ2ε+2U√CKDE2,δ logNgNgσε+Lgσ2ε+2κU(Mr +Mg)+2U√12log(8δ)(EY∼pˆKDEr (y)[√1NrY,κ]+EY∼pˆKDEg (y)[√1NgY,κ])75+L (D˜)−L (D∗).Theorem 2.5.6 (Error bound of D trained with SVDL). Assume that (A1)-(A4)hold, then ∀δ ∈ (0,1), with probability at least 1−δ ,L (D̂SVDL)−L (D∗)≤ 2U√CKDE1,δ logNrNrσε+Lrσ2ε+2U√CKDE2,δ logNgNgσε+Lgσ2ε+4U√12log(16δ)(1√NrEY∼pˆKDEr (y)[1W r(Y )]+1√NgEY∼pˆKDEg (y)[1W g(Y )])+2U(MrEY∼pˆKDEr (y)[EY ′∼pˆrw(y′|Y )∣∣Y ′−Y ∣∣]+MgEY∼pˆKDEg (y)[EY ′∼pˆgw(y′|Y )∣∣Y ′−Y ∣∣])+L (D˜)−L (D∗), (2.37)for some constant CKDE1,δ , CKDE2,δ depending on δ .Proof. Similarly, based on Theorem 2.5.4, we can derive Theorem 2.5.6. Thedetailed proof is omitted.2.5.5 Interpretation of Theorems 2.5.5 and 2.5.6Both theorems imply HVDL and SVDL perform well if the output of D is not tooclose to 0 or 1 (i.e., favor small U). The first two terms in both upper boundscontrol the quality of KDE, which implies KDE works better if we have larger Nrand Ng and a smaller σε . The rest terms of the two bounds are different. In theHVDL case, we favor smaller κ , Mr, and Mg. However, we should avoid setting κfor a too small value because we prefer larger Nry,κ and Ngy,κ . In the SVDL case, weprefer small Mr and Mg but large W r(y) and W g(y). Large W r(y) and W g(y) imply76that the weight function decays slowly (i.e., small ν ; similar to large Nry,κ and Ngy,κin Eq.(2.30)). However, we should avoid setting ν too small because a small νleads to large EY ′∼pˆrw(y′|Y=y) |Y ′− y| and EY ′∼pˆgw(y′|Y=y) |Y ′− y| (i.e., y′’s which arefar away from y have large weights). In our experiments, we use some rule-of-thumb formulae to select κ and ν . As a future work, a refined hyper-parameterselection method should be proposed.2.6 ConclusionAs the first generative model, we propose the CcGAN in this thesis for generativeimage modeling conditional on regression labels. In CcGAN, two novel empiricaldiscriminator losses (HVDL and SVDL), a novel empirical generator loss and twonovel label input mechanisms (NLI and ILI) are proposed to overcome the twoproblems of existing cGANs. The error bounds of a discriminator trained underHVDL and SVDL are studied in this work. Two new benchmark datasets, RC-49and Cell-200, are also created for the continuous scenario. A new evaluation met-ric, termed SFID, is also proposed to replace Intra-FID when there are insufficientreal images. Finally we demonstrate the superiority of the proposed CcGAN tocGAN on the Circular 2-D Gaussians, RC-49, UTKFace, Cell-200, and SteeringAngle datasets.77Chapter 3Subsampling GenerativeAdversarial Networks: DensityRatio Estimation in FeatureSpace with Softplus Loss3.1 IntroductionGenerative adversarial networks (GANs) first introduced by [43] are well-knownand powerful generative models for image synthesis and have been applied to var-ious types of image-related tasks [39, 54, 111, 145, 148, 152]. The vanilla GANsproposed by [43] consist of two neural networks: a generator and a discriminator.The generator is trained to generate fake images to fool the discriminator while thediscriminator is trained to distinguish fake images from real ones. To enhance thequality of fake images generated from a vanilla GAN, many subsequent works haveaimed to improve its training, such as large-scale training (e.g., BigGAN [14]),novel normalization (e.g., SN-GAN [95]), advanced GAN architectures (e.g., SA-GAN [163]), and different loss functions (e.g., WGAN loss [4, 48] based on theWasserstein distance [144] and MMD-GAN loss [76] based on maximum meandiscrepancy [46]). Instead of improving the training procedure, we are more in-78terested in this article in post-processing fake images from a trained GAN, i.e.,subsampling fake images to filter out unrealistic images.Two density ratio based subsampling methods for GANs were proposed re-cently and demonstrated to be effective. Discriminator rejection sampling (DRS)[10] is based on rejection sampling (RS) to accept or reject a fake image generatedfrom a trained GAN, and Metropolis-Hastings GAN (MH-GAN) [138] utilizes theMetropolis-Hastings algorithm (MH) to sample from a trained GAN. Denote thetrue image distribution’s PDF by pr(x) and fake image distribution’s PDF by pg(x).The key step of these two subsampling methods is density ratio estimation (DRE)where the density ratio pr(x)/pg(x) is estimated. When GANs are trained withthe standard adversarial loss function defined in [43], given a fixed generator, theoptimal discriminator D∗(x) and the density ratio r∗(x) = pr(x)/pg(x) satisfy therelationshipr∗(x) =pr(x)pg(x)=D∗(x)1−D∗(x) . (3.1)This property is leveraged by DRS and MH-GAN to estimate the density ratiopr(x)/pg(x), and hence DRS and MH-GAN rely heavily on an assumption of op-timality of the discriminator. In practice, however, the quality of the discriminatoris difficult to guarantee in GAN training. Moreover, this property no longer holdsif a GAN is trained with other loss functions such as the WGAN loss [4, 48] or theMMD-GAN loss [76]. Thus, strictly speaking, DRS and MH-GAN are not suit-able for WGANs and MMD-GANs. To reduce the reliance of DRS and MH-GANon the quality of a trained discriminator and broaden their application to differentGANs, direct estimation of the density ratio from samples is needed.Previous research on density ratio estimation for images includes [47, 63, 98].[63, 98] propose use of a convolutional neural network (CNN) to model the truedensity ratio function. [98] models the density ratio function by a CNN with onlytwo convolutional layers and fits this shallow CNN under the unconstrained least-squares importance fitting (uLSIF) loss function. A deeper CNN structure thatcontains six convolutional layers along with two new loss functions (called DSKLand BARR, respectively) are proposed by [63]. However, the loss functions usedby [63, 98] to train CNNs are not bounded from below. Hence, if stochastic gradi-ent descent (SGD) or a variant is used for the optimization, the training loss keeps79decreasing without converging as long as the CNN has enough capacity. Ratherthan using a neural network to model the true density ratio function, [47] lever-ages the relationship between the true density ratio function and a Bayes optimalclassifier (BOC) to estimate density ratios. The BOC is learned from samples andused for classifying real and fake samples. However, this method suffers from thedifficulty of achieving the optimality of the BOC.In this chapter, we focus on improving density ratio based subsampling meth-ods for GANs [10, 138] by proposing a novel sample-based density ratio estima-tion (DRE) method. Our contributions can be summarized as follows:• We propose in Section 3.3.1 a novel loss function called Softplus (SP) lossfor density ratio estimation with neural networks.• We derive in Section 3.3.2 the rate of convergence of a density ratio modeltrained with the SP loss under Bregman divergence.• In Section 3.3.3 we further propose a density ratio estimation method: den-sity ratio estimation in feature space with Softplus loss (DRE-F-SP). Wemodel the true density ratio function by a 5-layer multilayer perceptron(MLP) in a feature space learned by a specially designed and pre-trainedResNet-34, and the MLP is trained under the SP loss.• Then, in Section 3.3.4, we incorporate the proposed DRE-F-SP into the RSand MH schemes of [10, 138]. We also apply the sampling importance re-sampling (SIR) scheme based on the DRE-F-SP because of the high effi-ciency of the SIR. These three subsampling methods for GANs are denotedby DRE-F-SP+RS, DRE-F-SP+MH, and DRE-F-SP+SIR, respectively.• Finally, in Section 3.4, we conduct experiments on a synthetic dataset andthe CIFAR-10 dataset to justify our proposed subsampling methods. The ex-periments show that they can substantially outperform DRS and MH-GAN.In addition to the main study, we also conduct an ablation study on boththe synthetic dataset and the CIFAR-10 dataset, respectively, to demonstratethat the novel SP loss is one source of the improvement. On CIFAR-10, weconduct a second ablation study to show the density ratio estimation in the80feature space is another source of the improvement. Moreover, our exper-iments show that our subsampling methods can improve different types ofGANs, e.g., DCGAN, WGAN-GP, and MMD-GAN. Extra experiments onMNIST and CelebA also demonstrate the superiority of our methods.3.2 Preliminary3.2.1 Sampling-Importance ResamplingWhen a target distribution with PDF pr(x) is difficult to sample directly, samplingimportance resampling (SIR) [12, 117] generates samples from an easier proposaldistribution with PDF pg(x) and then takes subsamples. Specifically, SIR generates{xg1, · · · ,xgNg}from pg and takes subsamples with replacement from them usingprobabilitywi =pr(xgi )/pg(xgi )∑Ngi=1 pr(xgi )/pg(xgi )for xgi . The probability wi is also known as the normalized importance weight forxgi . If Ng is large enough, resampling from{xg1, · · · ,xgNg}in this way approximatessamples generated from pr.3.2.2 Fitting Density Ratio Models Under Bregman DivergenceThe uLSIF loss (1.13) introduced in Section 1.2 is a special case of the Bregman(BR) divergence, based on which we propose a novel loss call Softplus loss inSection 3.3.1. BR divergence [13, 142], an extension of the squared Euclideandistance, measures the distance between two points t∗ and t in terms of a functionf as follows:BR′f (t∗|t) = f (t∗)− f (t)−O f (t)(t∗− t), (3.2)where f : Ω→R is a continuously differentiable and strictly convex function de-fined on a closed set Ω. Assume f is defined on Ω= [min(m1,m2),max(M1,M2)],where m1 = minr∗(x), m2 = minr(x;α ), M1 = maxr∗(x), M2 = maxr(x;α ), andr(x;α ) is a density ratio model with a learnable parameter α . The BR divergencedefined based on f is used by [133] to quantify the discrepancy between r∗(x) and81r(x;α ) as follows:BR′f (α ) =∫pg(x) [ f (r∗(x))− f (r(x;α ))−O f (r∗(x))(r∗(x)−r(x;α ))]dx= C+BR f (α ),(3.3)where C =∫pg(x) f (r∗(x))dx does not depend on r(x;α ) andBR f (α ) =∫pg(x)O f (r(x;α ))r(x;α )dx−∫pg(x) f (r(x;α ))dx−∫pr(x)O f (r(x;α ))dx.(3.4)An empirical approximation to BR f (α ) isB̂R f (α ) =1NgNg∑i=1O f (r(xgi ;α ))r(xgi ;α )− 1NgNg∑i=1f (r(xgi ;α ))−1NrNr∑i=1O f (r(xri ;α )).(3.5)With f appropriately chosen, B̂R f (α ) can be used as a loss function to fit r(x;α ).For example, the uLSIF loss (1.13) is a special case of Eq. (3.5) when f (t) =0.5(t−1)2.3.3 Method3.3.1 Softplus Loss Function for Density Ratio EstimationMotivated by the two shortcomings of uLSIF (1.13), DSKL (1.18) and BARR(1.19), we propose a novel loss function called Softplus (SP) loss for density ratioestimation with neural networks. The SP loss is a special case of BR f (α ) in Eq.(3.4) when f (t) is the softplus functionη(t) = ln(1+ et). (3.6)82The derivative of the softplus function η(t) is the sigmoid functionσ(t) =et1+ et. (3.7)The second derivative of η(t) is σ(t)(1−σ(t)) which is positive so the softplusfunction is strictly convex. Then, the SP loss and its empirical approximation aredefined asSP(α ) =∫σ(r(x;α ))r(x;α )pg(x)dx−∫η(r(x;α ))pg(x)dx−∫σ(r(x;α ))pr(x)dx,(3.8)andŜP(α ) =1NgNg∑i=1σ(r(xgi ;α ))r(xgi ;α )− 1NgNg∑i=1η(r(xgi ;α ))−1NrNr∑i=1σ(r(xri ;α ))=1NgNg∑i=1[σ(r(xgi ;α ))rˆ(xgi ;α )−η(r(xgi ;α ))]− 1NrNr∑i=1σ(r(xri ;α )),(3.9)where r(x;α ) is a neural network for the approximation of the true density ratiofunction.Theorem 3.3.1. The empirical SP loss (3.9) is bounded from below, i.e., ŜP(α )>− ln2−1.Proof. We defineg(t) = σ(t) · t−η(t) = et · t · (1+ et)−1− ln(1+ et), t ≥ 0.Since g′(t) = et · t · (1+ et)−2 > 0 when t ≥ 0, g(t) is monotonically increasing onits domain and mint g(t) = g(0) =− ln2. Moreover, σ(t) is lower bounded by -1.Therefore, the empirical SP loss has a lower bound, i.e., ŜP(α )>− ln2−1.Then we propose to train the density ratio model r(x;α ) by minimizing the83following penalized SP loss:minα{ŜP(α )+λ Qˆ(α )}, (3.10)whereQˆ(α ) =(1NgNg∑i=1r(xgi ;α )−1)2. (3.11)and λ (a hyper-parameter) controls the penalty strength. Now the penalty term maytake effect if a proper λ is chosen.Hyperparameter Selection. To select the optimal hyperparameter λ ∗, we eval-uate a trained density ratio model r(x;α ) on Nr real images which are used fortraining and Nv hold-out real images {xv1, · · · ,xvNv} separately. Then we have twosets of density ratios: {r(xr1;α ), · · · ,r(xrNr ;α )} and {r(xv1;α ), · · · ,r(xvNv ;α )}. Ifthe model does not overfit the training images, these two sets should have simi-lar distributions. We use the two-sample Kolmogorov-Smirnov (KS) test statistic[62] to quantify the divergence between these two distributions. The optimal hy-perparameter λ ∗ is selected to minimize this KS test statistic. Other metrics (e.g.,Kullback–Leibler (KL) divergence [140]) may also be useful for the hyperparame-ter selection.3.3.2 Error BoundIn this section, we derive the error bound of a density ratio model trained with ourproposed Softplus loss under the Bregman divergence in the GAN setting (i.e., Ngis large enough). LetR = {r ∈R : x 7→ r(x)} denote the set of potential functionsthat can be represented by the density ratio model r(x,α ) (i.e., the HypothesisSpace). Also let σ ◦R = {r ∈R : x 7→ σ(r(x))}, where σ is the Sigmoid function.Lemma 3.3.1. Let Rˆpr(x),Nr(R) and Rˆpr(x),Nr(σ ◦R) denote the empirical Remachercomplexities ofR and σ ◦R respectively, where Rˆpr(x),Nr(R) and Rˆpr(x),Nr(σ ◦R)are defined based on independent samples {x1, · · · ,xNr} from pr(x). The followinginequality holds:Rˆpr(x),Nr(σ ◦R)≤14Rˆpr(x),Nr(R),84whereRˆpr(x),Nr(R) = Eρ{supr∈R∣∣∣∣∣ 1Nr Nr∑i=1ρir(xri )∣∣∣∣∣},Rˆpr(x),Nr(σ ◦R) = Eρ{supr∈R∣∣∣∣∣ 1Nr Nr∑i=1ρiσ(r(xri ))∣∣∣∣∣},and ρ1, · · · ,ρNr are independent Remacher random variables whose distribution isP(ρi = 1) = P(ρi =−1) = 0.5.Proof. Since the Sigmoid function σ is 14 -Lipschitz continuous, the inequality canbe obtained by applying Talagrand’s Lemma (Lemma 4.2 in [96]).Theorem 3.3.2 (Rademacher Bound). If a hypothesis space F is a class of func-tions f such that 0≤ f (x)≤ 1, then for ∀δ ∈ (0,1) with probability at least 1−δ ,supf∈F∣∣∣∣∣Ex∼p(x)[ f (x)]− 1N N∑i=1 f (xi)∣∣∣∣∣≤ 2Rˆp(x),N(F )+√4Nlog(2δ), (3.12)where the xi’s are independently drawn from a distribution with PDF p(x) andRˆp(x),N(F ) is the empirical Rademacher complexity of the hypothesis space Fdefined on these N samples.Proof. The proof of Theorem 3.3.2 can be found in [67].Let BR f (r) be the Bregman divergence between the true density ratio functionr∗ in Eq. (3.1) and a function r in the hypothesis space R. Let B̂R f (r) be theempirical approximation of BR f (r). If f is replaced by the Softplus function η ,thenBR f (r) = Ex∼pg(x) [σ(r(x))r(x)−η(r(x))]−Ex∼pr(x) [σ(r(x))] , (3.13)B̂R f (r) =1NgNg∑i=1[σ(r(xgi ))r(xgi )−η(r(xgi ))]− 1NrNr∑i=1σ(r(xri )). (3.14)Note that the BR f (r) and B̂R f (r) are equivalent to Eq. (3.8) and Eq. (3.9) respec-tively. For simplicity, we only consider the Softplus loss without a penalty term.85Following the notation in Appendix D of [139], we define r˜ and rˆ asr˜ = argminr∈RBR f (r), rˆ = argminr∈RB̂R f (r).Note that BR f (r) reaches its minimum if and only if r = r∗ but r∗ may be not inR. If r∗ /∈ R, then BR f (r˜)−BR f (r∗) is a positive constant; otherwise BR f (r˜)−BR f (r∗)= 0. However, in practice, we can only optimize B̂R f (r) instead of BR f (r).Therefore, we are interested in the distance of rˆ from r∗ under the Bregman diver-gence, i.e., BR f (rˆ)−BR f (r∗).Before we introduce our main theorem for the rate of convergence, we needsome more notation. Denote by A the parameter space of the density ratio modelr(xri ;α ). Note that the hypothesis space R is determined by the parameter spaceA. Denote σ(r(x))r(x)−η(r(x)) by g(x;α ).Theorem 3.3.3. If (i) the f in the Bregman divergence is the Softplus function η in(3.6), (ii) Ng is large enough, (iii), A is compact, (iv) ∀g(x;α ) is continuous at α ,(v) ∀g(x;α ),∃ a function gu(x) that does not depend on α , s.t. |g(x;α )| ≤ gu(x),and (vi)Ex∼pggu(x)<∞, then ∀δ ∈ (0,1) and ∀δ ′ ∈ (0,δ ] with probability at least1−δ ,BR f (rˆ)−BR f (r∗)≤ 1Ng + Rˆpr(x),Nr(R)+2√4Nrlog(2δ ′)+BR f (r˜)−BR f (r∗).(3.15)Proof. Following Eq.(11) of Appendix D in [139], we first decompose BR f (rˆ)−BR f (r∗) as followsBR f (rˆ)−BR f (r∗)=BR f (rˆ)− B̂R f (rˆ)+ B̂R f (rˆ)− B̂R f (r˜)+ B̂R f (r˜)−BR f (r˜)+BR f (r˜)−BR f (r∗)≤BR f (rˆ)− B̂R f (rˆ)+ B̂R f (r˜)−BR f (r˜)+BR f (r˜)−BR f (r∗)≤∣∣∣BR f (rˆ)− B̂R f (rˆ)∣∣∣+ ∣∣∣B̂R f (r˜)−BR f (r˜)∣∣∣+BR f (r˜)−BR f (r∗)≤2 supr∈R|BR f (r)− B̂R f (r)|+BR f (r˜)−BR f (r∗).(3.16)86The second term in Eq. (3.16) is a constant so we just need to bound the first term,and if f = η , the first term can be further decomposed with an upper bound asfollowssupr∈R∣∣∣BR f (r)− B̂R f (r)∣∣∣≤ supr∈R∣∣∣∣∣Ex∼pg(x) [σ(r(x))r(x)−η(r(x))]− 1NgNg∑i=1[σ(r(xgi ))r(xgi )−η(r(xgi ))]∣∣∣∣∣+ supr∈R∣∣∣∣∣Ex∼pr(x) [σ(r(x))]− 1Nr Nr∑i=1σ(r(xri ))∣∣∣∣∣ .Since A is compact, g(x;α ) is continuous at α , |g(x;α )| ≤ gu(x), andEx∼pggu(x)<∞, based on the uniform law of large numbers [58, 126], for ∀ε > 0,limNg→∞P{supr∈R∣∣∣∣∣Ex∼pg(x) [σ(r(x))r(x)−η(r(x))]− 1NgNg∑i=1[σ(r(xgi ))r(xgi )−η(r(xgi ))]∣∣∣∣∣> ε}= 0.Since Ng is large enough, let ε = 1/2Ng, ∀δ1 ∈ (0,1) with probability at least1−δ1, whereuponsupr∈R∣∣∣∣∣Ex∼pg(x) [σ(r(x))r(x)−η(r(x))]− 1NgNg∑i=1[σ(r(xgi ))r(xgi )−η(r(xgi ))]∣∣∣∣∣≤ 12Ng .(3.17)Moreover, based on Theorem 3.3.2 and Lemma 3.3.1, ∀δ2 ∈ (0,1) with probability87at least 1−δ2,supr∈R∣∣∣∣∣Ex∼pr(x) [σ(r(x))]− 1Nr Nr∑i=1σ(r(xri ))∣∣∣∣∣≤2Rˆpr(x),Nr(σ ◦R)+√4Nrlog(2δ2)≤12Rˆpr(x),Nr(R)+√4Nrlog(2δ2).(3.18)With δ = max{δ1,δ2} and δ ′ = δ2, combining Eq. (3.17) and Eq. (3.18) leads tothe upper bound in Theorem 3.3.3.Remark 3.3.1. If f (t) = 0.5(t− 1)2, then BR f (r) is the uLSIF loss. In this case,[139] gives an upper bound for BR f (rˆ)− BR f (r∗) and at least one term in thisupper bound is proportional to a constant M ([139] assumes all elements in R arebounded by M). However, in real practice, M may be quite large so the upperbound provided by [139] may be too loose, which helps explain why the SP lossoutperforms the uLSIF loss in our experiments.Remark 3.3.2. The Rˆpr(x),Nr(R) term on the right hand side of the inequality(3.15) implies we should not use a density ratio model that is too complex. There-fore, we propose to estimate the density ratio by a simple multilayer perceptron inthe feature space learned by a pre-trained deep CNN in Section 3.3.3.3.3.3 Density Ratio Estimation in Feature SpaceIn this section, we propose a novel density ratio estimation method called densityratio estimation in feature space with Softplus loss (DRE-F-SP). Assume we haveNr real images xr1,xr2, · · · ,xrNr ∼ pr(x), and Ng fake images xg1,xg2, · · · ,xgNg ∼ pg(x).The PDFs pr(x) and pg(x) are both unknown. Rather than estimating density ratiosin the pixel space [63, 98] (the density ratio model r(x;α ) directly maps an imageto its density ratio) or using the property of a well-trained GAN model [10, 138],we model the true density ratio function via a small-scale multilayer perceptron(MLP) in a feature space learned by a pre-trained deep CNN. This deep CNNtakes an image as input and outputs a class label. The architecture of this CNN is88specially designed to let one of its hidden layers output a feature map h that hasthe same dimension as the input x. In our experiment, we build such a CNN byadding an extra fully connected layer which can output such feature map h on topof all convolutional layers of the ResNet-34 [51]. We train this specially designedResNet-34 on a set of labelled samples with the cross-entropy loss. Denote thefully connected layer which is used to output the feature map h and other layersbefore it in this pre-trained ResNet-34 as φ(x), then φ defines a mapping of a rawimage x in the pixel spaceX to a high-level feature h in the feature spaceH , i.e,h = φ(x). (3.19)In the remainder of this chapter, we simply call φ(x) ResNet-34 which is approx-imately invertible [41]. We define qr and qg as the distribution of real and fakefeatures respectively. Since X and H have the same dimension, the Jacobianmatrix ∂h/∂x is a square matrix and the relationship between the distributions ofx and h can be summarized as follows:pr(x) = qr(h) ·∥∥∥∥∂h∂x∥∥∥∥ , pg(x) = qg(h) ·∥∥∥∥∂h∂x∥∥∥∥ ,where∥∥∥ ∂h∂x∥∥∥ is the absolute value of the Jacobian determinant and assumed to bepositive. Then the true density ratio function r(x) can be equivalently expressed inthe features space viaψ∗(φ(x)) = ψ∗(h) =qr(h)qg(h)=qr(h) ·∥∥∥ ∂h∂x∥∥∥qg(h) ·∥∥∥ ∂h∂x∥∥∥ =pr(x)pg(x)= r(x), (3.20)where ψ∗(h) denotes the true density ratio function in the feature space. Notethat the Jacobian determinant is cancelled so we only need to model the densityratio function ψ∗(h) in the feature space. We propose to model ψ∗(h) by a 5-layermultilayer perceptron ψ(h;β ) with a learnable parameter β , and ψ(h;β ) is trained89by minimizing the following penalized SP loss:minβ{ŜP(β )+λ Qˆ(β )}=minβ{1NgNg∑i=1[σ(ψ(hgi ;β ))ψ(hgi ;β )−η(ψ(hgi ;β ))]− 1NrNr∑i=1σ(ψ(hri ;β )) +λ(1NgNg∑i=1ψ(hgi ;β )−1)2 .(3.21)Eq. (3.21) is adapted from Eq. (3.9) by replacing r(x;α ) with ψ(h;β ). Thenψ(h;β ) can be seen as a density ratio model in the feature space, and ψ(φ(x);β )can be seen as a density ratio model in the pixel space. Their workflows are visu-alized in Fig. 3.1. We implement DRE-F-SP by Alg. 5.Remark 3.3.3. The density ratio model in the feature space is not necessarily aMLP. It can be another small-scale neural network such as a CNN as long as itscomplexity is moderate.Algorithm 5: Density Ratio Estimation in the Feature Space Under Pe-nalized SP Loss (DRE-F-SP)Data: Nr real samples {xr1, · · · ,xrNr}, a generator G, a pre-trained CNN φ(x) (3.19), auntrained MLP ψ(h;β ) in (3.21) and a preset hyperparameter λResult: a trained density ratio model r(x) = ψ(φ(x);β ) = ψ(h;β )1 Initialize β ;2 for k = 1 to K do3 Sample a mini-batch of m real samples xr(1), · · · ,xr(m) from {xr1, · · · ,xrNr};4 Sample a mini-batch of m fake samples xg(1), · · · ,xg(m) from G;5 Update β via the SGD or its variants with the gradient of Eq. (3.21)∂∂β{1mm∑i=1[σ(ψ(φ(xg(i));β ))ψ(φ(xg(i));β )−η(ψ(φ(xg(i));β ))]− 1mm∑i=1σ(ψ(φ(xr(i));β ))+λ(1mm∑i=1ψ(φ(xg(i));β )−1)26 end90Figure 3.1: The workflow of our proposed density ratio model ψ(h;β ) inthe feature space in Eq. (3.21). The composition of ψ and φ leads tothe density ratio model ψ(φ(x);β ) in the pixel space. The pre-trainedResNet-34 φ(x) (3.19) takes an image x as input and outputs a featuremap h from the fully-connected layer fc 1. If we flatten the image x,then flat(x) and h are two CHW ×1 vectors, where C, H and W denotethe number of channels, height and width of the image x.3.3.4 Application of DRE-F-SP in Subsampling GANsFig. 3.2 describes the workflow of a density ratio based subsampling method forGANs. Each density ratio based subsampling method consists of two components:a DRE method and a sampler. DRE methods can be as proposed in [10, 47, 63, 98,138] or our DRE-F-SP. A sampler here is a density ratio based sampling schemesuch as the RS sampler in DRS, the MH sampler in MH-GAN and the SIR sam-pler in Section 3.2.1. Moreover, a neural network based DRE method can also bedecomposed into two components: a density ratio model and a loss function. Forexample, our DRE-F-SP uses the composition of a pre-trained ResNet-34 and a5-layer MLP as the density ratio model and trains the density ratio model with theSP loss.We propose three density ratio based subsampling methods for GANs, which91are called DRE-F-SP+RS, DRE-F-SP+MH, and DRE-F-SP+SIR, respectively. Thesethree methods utilize the same DRE method (i.e., DRE-F-SP) but three differentsamplers (i.e., RS, MH, and SIR). We provide three corresponding algorithms Alg.6–8 to implement them. In some scenarios, the RS sampler and MH sampler sufferfrom low acceptance rates, and consequently they may take a very long time. TheSIR sampler does not suffer from this problem, so it is more efficient than the RSand MH samplers, but the SIR sampler may perform poorly if we subsample froma small pool of fake images.BOC-based DRE [47]Discriminator-based DRE [10, 138]Samplesfrom GANSamplerRS sampler [10]MH samper [138]SIR sampler......Loss function2-layer CNN (in pixelspace) [98]6-layer CNN (in pixelspace) [63]Pre-trained ResNet-34+MLP (in feature space)(ours)DR modeluLSIF [98]DSKL [63]BARR [63]SP (ours)......DRE methodoror FilteredGANSamplesFigure 3.2: Workflow of density ratio based subsampling for GANs.Algorithm 6: DRE-F-SP+RSData: a generator G, a trained CNN φ(x) (3.19), a trained MLP ψ(h;β ) in (3.21)Result: images = {N filtered images from G}1 Generate N′ fake images from G;2 Estimate the density ratios of these N′ fake images by evaluating ψ(φ(x);β );3 M←max{N′ estimated density ratios};4 images← /0;5 while |images|< N do6 x← get a fake image from G;7 ratio← ψ(φ(x);β );8 M←max{M,ratio};9 p← ratio/M (based on Eq. (1.10));10 u← Uniform(0,1);11 if u≤ p then12 Append(x, images);13 end14 end92Algorithm 7: DRE-F-SP+MHData: a generator G, a trained CNN φ(x) (3.19), a trained MLP ψ(h;β ) in (3.21), realimagesResult: images = {N filtered images from G}1 images← /0;2 while |images|< N do3 x← a real image;4 for i = 1 to K do5 x′← get a fake image from G;6 u← Uniform(0,1);7 p = min(1, ψ(φ(x′);β )ψ(φ(x);β ))(based on Eq. (1.12));8 if u≤ p then9 x← x′;10 end11 end12 if x is not a real image then13 Append(x, images);14 end15 endAlgorithm 8: DRE-F-SP+SIRData: a generator G, a trained CNN φ(x) (3.19), a trained MLP ψ(h;β ) in (3.21)Result: images = {N filtered images from G}1 Generate a pool of N p samples {xg1, · · · ,xgN p} from G;2 Compute N p normalized importance weights {w1, · · · ,wN p} for these fake samples viawi =ψ(φ(xgi );β )∑ni=1ψ(φ(xgi );β );3 images← /0;4 while |images|< N do5 Sample an integer j from {1,2, · · · ,N p} where j is drawn with probabability w j;6 Append(xgj , images);7 end933.4 ExperimentIn this section, our main objective is to justify that DRE-F-SP+RS, DRE-F-SP+MH,and DRE-F-SP+SIR perform better than DRS and MH-GAN for subsampling GANs.Hence, we conduct experiments on a synthetic dataset and three real datasets—CIFAR-10 [66], MNIST [69], and CelebA [84]. We also conduct several ablationstudies to empirically demonstrate that the power of three proposed subsamplingmethods comes from the novel Softplus loss and the scheme of estimating densityratio in the feature space.3.4.1 Mixture of 25 2-D GaussiansWe first test the performance of our proposed subsampling methods on syntheticdata generated from a mixture of 25 two-dimensional Gaussians (the 25 mixturecomponents have equal weights). This mixture model is used as a toy example in[10, 138] and is very popular in the GAN literature.Experimental setup of the main study: The means of these 25 Gaussians arearranged on a 2-D grid µ ∈ {−2,−1,0,1,2}×{−2,−1,0,1,2} and the commoncovariance matrix is set to σ I2×2, where σ = 0.05. From this mixture model,we generate 50,000 training samples, 50,000 validation samples and 10,000 testsamples.Following [10, 138], we train a GAN model with the standard loss (1.1) onthe training set. Both the generator and discriminator in this GAN consist of fourfully connected layers with ReLU activation functions, and all hidden layers havesize 100. The last layer of the discriminator is a Sigmoid function, and the noisez ∈ R2 fed into the generator is drawn from a 2-D Gaussian with mean 0 andstandard deviation 1. We deliberately train the generator and discriminator for only50 epochs to prevent them from reaching optimality, so density ratio estimation interms of Eq.(1.4) is not reliable.When implementing DRS, we follow the setting in [10] and set γ dynamicallyfor each batch of fake samples drawn from the GAN to the 95th percentile of F(x)in Eq. (1.11) for each x in this batch. We also keep training the discriminatoron the validation set for another 20 epochs to further improve performance of theDRS. When implementing MH-GAN, we calibrate the trained discriminator on the94validation set with logistic regression and set the MCMC iteration K to 100 (moreiterations do not show significant improvement).In our proposed sampling method, at the density ratio estimation stage, we usea 5-layer MLP as the density ratio model r(x;α ) in Eq. (3.9) to directly map asample to its density ratio without a pre-trained CNN since our synthetic data arenot images; its architecture is shown in Supp. A.7. The 5-layer MLP is trainedwith our proposed penalized SP loss (3.10). To select the optimal λ , we generatea grid of values between 0 and 0.1 and select the one which minimizes the KS teststatistic on the validation set (shown in Table A.6 of the supplemental material).To show the superiority of our proposed SP loss, we also train the 5-layer MLPwith the uLSIF [98], DSKL, and BARR [63] losses. Following the setting in [63],the λ in BARR is set to 10. At the sampling stage, all three samplers—RS, MH,and SIR—are considered. The number of burn-in samples N′ for RS in Alg. 6 is50,000. The MCMC iterations K for MH in Alg. 7 is set to 100. The pool size Npfor SIR in Alg. 8 is set to 20,000.We subsample 10,000 fake samples from the trained GAN with each method,and the quality of these fake samples is evaluated. We repeat the whole experiment(i.e., data generation, GAN training, MLP training, subsampling) three times andreport in Table 3.1 the average quality of 10,000 fake samples from each subsam-pling method over the three repetitions.Experimental setup of an ablation study: To evaluate the effectiveness of the SPloss, we conduct an ablation study by training the 5-layer MLP in DRE-F-SP withother losses: uLSIF [98], DSKL [63], and BARR [63]. We subsample 10,000 fakesamples under different losses and three samplers and evaluate the quality of thesesamples. Similar to the main study, we repeat the whole setting three times andreport in Table 3.2 the average quality of 10,000 fake samples under each loss andeach sampler.Evaluation metrics: To measure performance, following [10, 138], we assigneach fake sample to its closest mixture component. A fake sample is defined as“high-quality” if its Euclidean distance to the mean of its mixture component issmaller than 4σ = 0.2. Also, we define that a mode (i.e., a mixture component)is recovered if at least one “high-quality” fake sample is assigned to it. For eachsampling method in the main study and the ablation study, we report in Tables 3.195and 3.2 the average percentage of high-quality samples and the average percentageof recovered modes.Quantitative results: From Table 3.1, we can see that three proposed samplingmethods almost perfectly correct the sampling bias of the imperfect generator andsignificantly outperform DRS and MH-GAN without trading off mode coveragefor quality. Table 3.2 shows that the power of three proposed sampling methodscomes from the novel SP loss.Table 3.1: Average quality of 10,000 fake synthetic samples from differentsubsampling methods over three repetitions. Higher % high-quality sam-ples and higher % recovered modes are better. Each setting is repeatedthree times, and we report the averaged % high-quality samples and av-eraged % recovered modes. The optimal λ ∗ in each round is shown inTable A.6.No Subsampling DRS [10] MH-GAN [138]% High Quality 69.8±15.4 96.3±1.3 89.7±5.3% Rec. Modes 100.0±0.0 100.0±0.0 100.0±0.0DRE-F-SP+RS DRE-F-SP+MH DRE-F-SP+SIR% High Quality 99.1± 0.5 99.2± 0.5 99.2± 0.4% Rec. Modes 100.0±0.0 100.0±0.0 100.0±0.0Visual results: We visualize the first-round results of the main study in Fig. 3.3. InFig. 3.3(b), we can see that many samples directly drawn from the generator locatebetween two neighboring modes. Figs. 3.3(c) and 3.3(d) show that DRS and MH-GAN can remove some “bad-quality” points, but many between-modes points stillexist. Figs. 3.3(e) to 3.3(g) show that fake samples from our proposed methods areclose to their assigned mixture components where between-modes samples onlyaccount for a small portion.The superiority of the Softplus loss: Besides the main study and the ablationstudy, we conduct an extra experiment to empirically show why the SP loss per-forms better than other loss functions such as the uLSIF loss. In this study, the5-layer MLP as a density ratio model is trained for 5000 epochs with the penalizeduLSIF loss (1.16) and our penalized SP loss (3.10) respectively when λ = 0.05.The initial learning rate is 10−3 and decayed every 1000 epochs with factor 0.196(a) Real Samples(b) No subsampling (c) DRS [10] (d) MH-GAN [138](e) DRE-F-SP+RS (f) DRE-F-SP+MH (g) DRE-F-SP+SIRFigure 3.3: Visual results of the 25 2-D Gaussians example. Each setting isrepeated for three times and we visualize the results of the first roundhere. In each figure, blue dots denote 10,000 real samples in the testset and green dots denote 10,000 fake samples from each method. TheGAN model is trained for only 50 epochs so the discriminator and gen-erator do not reach their optimality. Thus many samples generated bythis GAN shown in Fig. 3.3(b) are between-modes. DRS and MH-GANare effective but we still observe many between-modes samples in Figs.3.3(c) and 3.3(d). On the other hand, our proposed methods can nearlycorrect the bias in the generator and almost all generated samples inFigs. 3.3(e) to 3.3(g) are “high quality”.97Table 3.2: Ablation study on synthetic data. We train the 5-layer MLP withdifferent loss functions. Each setting is repeated three times, and wereport the averaged % high-quality samples and averaged % recoveredmodes.RSuLSIF [98] DSKL [63] BARR [63] SP% High Quality 89.7±2.4 66.4±7.6 71.9±14.9 99.1± 0.5% Rec. Modes 100.0±0.0 42.7±5.0 100.0±0.0 100.0±0.0MHuLSIF [98] DSKL [63] BARR [63] SP% High Quality 89.6±2.6 66.3±7.3 72.1±15.0 99.2± 0.5% Rec. Modes 98.7±1.9 38.7±5.0 100.0±0.0 100.0±0.0SIRuLSIF [98] DSKL [63] BARR [63] SP% High Quality 89.5±2.3 66.2±7.8 72.1±15.1 99.2± 0.4% Rec. Modes 98.7±1.9 40.0±5.7 100.0±0.0 100.0±0.0(see Supp. A.7.3 for details). From Figs. 3.4(a) and 3.4(b), we can see the SPloss converges after 1000 epochs while the uLSIF loss does not stop decreasinguntil the 3000th epoch and then starts fluctuating over a large range. The uLSIFloss stops decreasing because of a too small learning rate (Fig. A.16 of the supple-mental material shows the training curve of the 5-layer MLP under the uLSIF losswith a constant learning rate 10−5 where we can observe a constantly descendingtrend). We draw 10,000 fake samples from the trained GAN and evaluate the 5-layer MLP on the high and low quality fake samples separately. Figs. 3.5(a) and3.5(b) show the average density ratios on high/low quality samples and the per-centage high quality samples versus epoch. We estimate pg by a Gaussian mixturemodel [114] and pr is known so we can get the true density ratio function whichis used to compute the ground truth. From Fig. 3.5(a), when using the SP loss,the average density ratio of high quality samples is slightly above the ground truthand does not decrease over epochs. This implies the SP loss does not overfit thetraining data and the penalty term takes effect. Note that the SP loss may overfittraining data if λ = 0; see Supp.A.7.3 for details. In contrast, when using uLSIF98loss, the average density ratio of high quality samples decreases after around 900epochs and is always below the ground truth implying that the uLSIF loss overfitsthe training data and the penalty term does not effectively control its unbounded-ness. Fig. 3.5(b) shows that the uLSIF loss tends to overestimate the density ratiosof low quality samples while the SP loss performs optimally. The underestimationand overestimation of the uLSIF loss results in a small difference between high andlow quality samples from the density ratio perspective and makes it difficult for thesubsequent sampler to distinguish between high and low quality samples. Thesefindings explain why SP loss outperforms uLSIF loss when subsampling.3.4.2 CIFAR-10 DatasetIn this section, our main study is to empirically show the superiority of our ap-proach to DRS [10] and MH-GAN [138] in subsampling DCGAN [112], WGAN-GP [48], and MMD-GAN [76] trained on the CIFAR-10 [66] dataset. We alsoconduct two extra ablation studies to investigate the reason behind the efficacy ofour approach.Experimental setup of the main study: The CIFAR-10 dataset consists of 60,00032×32 RGB images which are classified into 10 classes. The dataset is split intoa training set of 50,000 images with 5000 per class and a validation set of 10,000images with 1000 per class. GANs are trained with network architectures andtraining setups shown in Supp. A.8.1.We use DRE-F-SP+RS, DRE-F-SP+MH, and DRE-F-SP+SIR to subsample50,000 fake images from a trained GAN. At density ratio estimation stage, we traina ResNet-34 [51] on the training set with a modified architecture shown in Supp.A.8.1 where we incorporate an extra fully connected layer to output a feature mapwith dimension (32×32×3)×1= 3072×1. A 5-layer MLP is used as the densityratio model ψ(y;β ) in Eq.(3.21) to map the extracted features of an image to itsdensity ratio, which is trained with the penalized SP loss (3.10) on the training setand fake images from the trained GAN. Detailed training setups of the ResNet-34and the 5-layer MLP are described in Supp. A.8.2. We conduct hyperparameterselection on a grid of values from 0 to 0.1 on the validation set (shown in TableA.15 of the supplemental material). At the sampling stage, the number of burn-in99(a) Penalized uLSIF loss (1.16)(b) Penalized Softplus loss (ours) (3.10)Figure 3.4: Training curves of a 5-layer MLP under the penalized uLSIF loss(1.16) and the penalized SP loss (3.10) when λ = 0.05 in the 25 2-DGaussians example.100(a) Average density ratio (solid lines) of high quality fake samples and %high quality samples (dotted lines) versus epoch of DRE training.(b) Average density ratio (solid lines) of low quality fake samples and %high quality samples (dotted lines) versus epoch of DRE training.Figure 3.5: The density ratio estimation and subsampling performance of a5-layer MLP trained under the penalized uLSIF loss (1.16) and the pe-nalized SP loss (3.10) when λ = 0.05 in the 25 2-D Gaussians example.The grey lines stand for the ground truth average density ratios.101samples N′ for RS in Alg. 6 is set to 50,000; the MCMC iterations K for MH inAlg. 7 is set to 640; the pool size Np for SIR in Alg. 8 is set to 100,000.We consider three competitors: no subsampling, DRS and MH-GAN. We useeach subsampling method to draw 50,000 fake images from a trained GAN. Nosubsampling refers to directly sampling from a generator. When implementingDRS, following the setting of [10] on ImageNet dataset, we set γ dynamicallyfor each batch of fake samples drawn from the GAN to the 80th percentile of theF(x) in Eq.(1.11) for each x in this batch. Continuing to train the discriminatoron the validation set does not improve the performance of DRS, so we do notconduct “keep training”. Since the discriminator of WGAN-GP outputs a classscore instead of a probability, we apply the calibration technique in MH-GAN tocalibrate the trained discriminator on the validation set with logistic regression tolet it output class probabilities. When implementing MH-GAN, following [138],the MCMC iteration K is set to 640. Note that, as we mentioned in Section 1.1.5,DRS and MH-GAN cannot be applied to MMD-GAN.In the main study, we subsample 50,000 fake images with each subsamplingmethod from each GAN three times. The average quality of 50,000 fake images ofeach method over three repetitions is reported in Table 3.3. Note that, in real dataanalysis, we only repeat subsampling three times, but train each GAN and eachdensity ratio model only once.Experimental setup of two ablation studies: The first ablation study aims atjustifying the effectiveness of our proposed DRE-F-SP in subsampling three typesof GANs. We consider four other density ratio estimation methods for imagesin the comparison: DRE-P-uLSIF [98], DRE-P-DSKL [63], DRE-P-BARR [63]and BOC [47]. The architectures of the 2-layer CNN for DRE-P-uLSIF and the6-layer CNN for DRE-P-DSKL and DRE-P-BARR are shown in Supp. A.8.1.When implementing BOC, we train a CNN as the Bayes optimal classifier withthe architecture proposed in [47] and shown in Supp. A.8.1 on 10,000 hold-outvalidation images and 10,000 fake images. We attach a RS sampler to these DREmethods and conduct the same three repetitions of the main study. We report inTable 3.4 the average quality of 50,000 fake images for different DRE methodsover three repetitions.The second ablation study focuses on researching the effect of different loss102functions on the final subsampling performance. We replace the SP loss in DRE-F-SP with other loss functions—uLSIF [98], DSKL [63] and BARR [63]—whileusing the same RS sampler and the same 5-layer MLP. The average quality of50,000 fake images for each loss over three repetitions is shown in Table 3.5.Evaluation metrics: We evaluate the quality of fake images from different sub-sampling methods by Inception Score (IS) [121] and Fre´chet Inception Distance(FID) [52]. Larger IS and smaller FID are better.Quantitative results: Table 3.3 shows the results of the main study and demon-strates our approaches significantly outperform other existing subsampling meth-ods and can also dramatically improve MMD-GAN, where DRS and MH-GAN arenot applicable.Table 3.4 shows the results of ablation study 1. Four existing DRE methodsfor images are applied in this case, but they are incapable of improving any GANmodel, let alone outperforming DRE-F-SP. This ablation study demonstrates thatthe effectiveness of the three proposed subsampling methods results from our pro-posed density ratio estimation method—DRE-F-SP.Table 3.5 shows the results of the ablation study 2 and demonstrates the novelSP loss plays a crucial role in the success of the density ratio estimation in thefeature space.Visual results: We also show in Figs. 3.6 to 3.9 some example images from eachsubsampling method in the main study. For each method, we draw images fromtwo classes (car and horse) with 50 images per class (first 5 rows correspond tocars and the rest correspond to horses). The improvement of our methods is moreobvious on images from WGAN-GP in Figs. 3.7 to 3.8 where our methods generatemore recognizable cars and horses.An extra experiment: [138] compares the authors’ MH-GAN with DRS and nosubsampling when the DCGAN is trained for only 60 epochs. To show how theepoch of DCGAN training influences the subsampling results, we fix all settings(e.g., λ = 0) but only vary the epoch of DCGAN training and visualize the resultsin Fig. 3.10. In terms of IS and FID, our proposed DRE-F-SP+RS is consistentlybetter than other methods at all epochs.103Table 3.3: Average quality of 50,000 fake CIFAR-10 images from differentsubsampling methods over three repetitions. We draw 50,000 fake im-ages by each method on which we compute the IS and FID. We repeatthis sampling three times and report the average IS and FID. Higher ISand lower FID are better. A grid search is conducted for DRE-F-SP to se-lect the hyperparameter, and the results under the optimal λ ∗ are shownin this table. We include the IS and FID of 50,000 training data and10,000 test data as a reference.Method IS (mean±std) FID (mean±std)- Real Data -50,000 Training Data 9.984 —10,000 Test Data 9.462 0.134- DCGAN -No Subsampling 6.261±0.003 3.006±0.011DRS [10] 6.385±0.004 2.930±0.008MH-GAN [138] 6.300±0.010 2.982±0.009DRE-F-SP+RS (λ ∗ = 0) 8.597± 0.011 1.664± 0.007DRE-F-SP+MH (λ ∗ = 0) 8.588± 0.007 1.669± 0.004DRE-F-SP+SIR (λ ∗ = 0) 8.572± 0.021 1.685± 0.027- WGAN-GP -No Subsampling 6.445±0.015 2.944±0.004DRS [10] 6.427±0.012 2.947±0.013MH-GAN [138] 6.428±0.021 2.948±0.014DRE-F-SP+RS (λ ∗ = 0.005) 8.625± 0.013 1.774± 0.011DRE-F-SP+MH (λ ∗ = 0.005) 8.606± 0.013 1.796± 0.014DRE-F-SP+SIR (λ ∗ = 0.005) 8.605± 0.043 1.826± 0.030- MMD-GAN -No Subsampling 5.508±0.016 3.682±0.007DRS [10] — —MH-GAN [138] — —DRE-F-SP+RS (λ ∗ = 0.006) 7.800± 0.012 2.471± 0.017DRE-F-SP+MH (λ ∗ = 0.006) 7.782± 0.008 2.469± 0.007DRE-F-SP+SIR (λ ∗ = 0.006) 7.740± 0.017 2.525± 0.045104(a) No subsampling (b) DRS(c) MH-GAN (d) DRE-F-SP+RSFigure 3.6: Fake CIFAR-10 images (car and horse) from DCGAN (Part I).For the “car” and “horse” classes, our proposed methods perform betterthan three baselines.105Table 3.4: Ablation study 1 on CIFAR-10. The average quality of 50,000fake CIFAR-10 images from subsampling methods with different DREmethods but the same RS sampler over three repetitions.Method IS (mean±std) FID (mean±std)- DCGAN -DRE-P-uLSIF [98] 6.340±0.005 2.773±0.004DRE-P-DSKL [63] 5.584±0.015 3.986±0.008DRE-P-BARR [63] 6.191±0.002 3.156±0.013BOC [47] 6.259±0.005 3.003±0.003DRE-F-SP 8.597± 0.011 1.664± 0.007- WGAN-GP -DRE-P-uLSIF [98] 6.418±0.008 2.897±0.007DRE-P-DSKL [63] 6.274±0.010 2.998±0.003DRE-P-BARR [63] 6.427±0.006 2.945±0.005BOC [47] 6.431±0.022 2.953±0.007DRE-F-SP 8.625± 0.013 1.774± 0.011- MMD-GAN -DRE-P-uLSIF [98] 5.427±0.007 3.776±0.004DRE-P-DSKL [63] 5.473±0.002 3.668±0.008DRE-P-BARR [63] 5.465±0.008 3.733±0.000BOC [47] 5.384±0.013 3.884±0.006DRE-F-SP 7.800± 0.012 2.471± 0.0173.4.3 Reduced MNIST DatasetIn this section, we repeat the main study of the CIFAR-10 experiment on theMNIST dataset [69]. The MNIST dataset has 70,000 28× 28 gray-scale hand-written digits from 10 classes (10 digits). The dataset is split into a training setof 60,000 images with 6000 per class and a test set of 10,000 images with 1000per class. Since MNIST is a very simple dataset, to increase the difficulty for boththe GAN training and the subsampling, we randomly select 5000 images from theoriginal training set to form a reduced training set.Experimental setup of the main study: In the MNIST setting, we comparethe performance of 6 candidate methods (i.e., No Subsampling, DRS, MH-GAN,DRE-F-SP+RS, DRE-F-SP+MH, and DRE-F-SP+SIR) in subsampling DCGAN106Table 3.5: Ablation study 2 on CIFAR-10. The average quality of 50,000fake CIFAR-10 images from subsampling methods with different lossfunctions but the same DR model and RS sampler over three repetitions.Method IS (mean±std) FID (mean±std)- DCGAN -uLSIF [98] 8.036±0.007 2.194±0.018DSKL [63] 6.619±0.010 2.736±0.004BARR [63] 6.910±0.005 2.582±0.006SP 8.597± 0.011 1.664± 0.007- WGAN-GP -uLSIF [98] 8.435±0.006 1.943±0.020DSKL [63] 5.941±0.003 3.623±0.005BARR [63] 6.966±0.009 2.528±0.010SP 8.625± 0.013 1.774± 0.011- MMD-GAN -uLSIF [98] 7.760±0.017 2.503±0.017DSKL [63] 5.590±0.005 3.765±0.003BARR [63] 5.763±0.008 3.488±0.005SP 7.800± 0.012 2.471± 0.017which is trained on the reduced training set. The detailed training setups are shownin Supp. A.9.2. We draw 50,000 fake images from each candidate method andcompare them with all 60,000 training images.Evaluation metrics: Similar to the CIFAR-10 experiment, we evaluate the qualityof fake images from different subsampling methods by IS and FID. Please seeSupp. A.9.3 for more details.Quantitative results: Table 3.6 shows the results of the experiment and demon-strates that our approaches significantly outperform two existing subsampling meth-ods.Visual results: We show some example MNIST images from different subsam-pling methods in Fig. 3.11. For each method, we draw images from all 10 classeswith 30 images per class.107(e) DRE-F-SP+MH (f) DRE-F-SP+SIRFigure 3.6: Fake CIFAR-10 images (car and horse) from DCGAN (Part II).For the “car” and “horse” classes, our proposed methods perform betterthan three baselines.Table 3.6: Average quality of 50,000 fake MNIST images from different sub-sampling methods over three repetitions. We draw 50,000 fake imagesby each method on which we compute the IS and FID. We repeat thissampling for three times and report the average IS and FID. Higher ISand lower FID are better. A grid search is conducted for DRE-F-SP toselect the hyperparameter, and the results under the optimal λ ∗ are shownin this table. We include the IS and FID of all 60,000 training data and10,000 test data as a reference.Method IS (mean±std) FID (mean±std)- Real Data -All 60,000 Training Data 9.984 -10,000 Test Data 9.462 0.134- DCGAN -No Subsampling 7.791±0.004 1.723±0.004DRS [10] 8.032±0.007 1.658±0.007MH-GAN [138] 8.461±0.001 1.269±0.007DRE-F-SP+RS (λ ∗ = 0.1) 9.676± 0.005 0.374± 0.007DRE-F-SP+MH (λ ∗ = 0.1) 9.677± 0.006 0.368± 0.008DRE-F-SP+SIR (λ ∗ = 0.1) 9.675± 0.003 0.380± 0.010108(a) No subsampling (b) DRS(c) MH-GAN (d) DRE-F-SP+RSFigure 3.7: Fake CIFAR-10 images (car and horse) from WGAN-GP (Part I).For the “car” and “horse” classes, our proposed methods perform betterthan three baselines.109(a) DRE-F-SP+MH (b) DRE-F-SP+SIRFigure 3.8: Fake CIFAR-10 images (car and horse) from WGAN-GP (PartII). For the “car” and “horse” classes, our proposed methods performbetter than three baselines.3.4.4 CelebA DatasetWe also conduct experiment on the CelebA dataset [84]. The CelebA dataset [84]has 202,599 218× 178 RGB face images. Each image has 40 binary attributes.We create 6 classes in terms of 3 binary attributes (“Wearing Lipstick”, “Smiling”,“Mouth Sligthly Open”); some minority classes are merged together to balancethe dataset. Then the dataset is randomly split into a training set of 192,599 imagesand a test set of 10,000 images. All images are resized to 64×64.Experimental setup of the main study: In this experiment, we test the perfor-mance of 6 candidate methods (i.e., No Subsampling, DRS, MH-GAN, DRE-F-SP+RS, DRE-F-SP+MH, and DRE-F-SP+SIR) in subsampling SNGAN [95].SNGAN is trained on the training set of CelebA. Please see Supp. A.10.2 for moredetialed training setups.Evaluation metrics: Similar to the CIFAR-10 and MNIST experiments, we evalu-ate the quality of fake images from different subsampling methods by IS and FID.Please see Supp. A.10.3 for more details. We draw 50,000 fake images from eachcandidate method and compare them with all training images.110(a) No subsampling (b) DRE-F-SP+RS(c) DRE-F-SP+MH (d) DRE-F-SP+SIRFigure 3.9: Fake CIFAR-10 images (car and horse) from MMD-GAN. Forthe “car” and “horse” classes, our proposed methods still perform wellwhen DRS and MH-GAN are not applicable.111(a) IS versus the epoch of GAN training(b) FID versus the epoch of GAN trainingFigure 3.10: Inception Score (IS) and Frechet Inception Distance (FID) ver-sus the epoch of DCGAN training. The grey vertical line indicates theepoch 60. DRS and MH-GAN perform well relative to the baseline ofNo Subsampling only at epoch 60 but the DCGAN is not well-trainedat that epoch. It is more reasonable to train the DCGAN well first(e.g., 500 epochs) and then apply any subsampling method to furtherimprove the image quality.112(a) No subsampling (b) DRSFigure 3.11: Fake MNIST images from DCGAN: Part 1. Many fake im-ages from No subsampling, DRS, and MH-GAN are not recognizable,while most images from our proposed methods are recognizable.113(c) MH-GAN (d) DRE-F-SP+RSFigure 3.11: Fake MNIST images from DCGAN: Part 2. Many fake im-ages from No subsampling, DRS, and MH-GAN are not recognizable,while most images from our proposed methods are recognizable.114(e) DRE-F-SP+MH (f) DRE-F-SP+SIRFigure 3.11: Fake MNIST images from DCGAN: Part 3. Many fake im-ages from No subsampling, DRS, and MH-GAN are not recognizable,while most images from our proposed methods are recognizable.115Quantitative results: Table 3.7 shows the results of the experiment and demon-strates that our approaches significantly outperform two existing subsampling meth-ods.Table 3.7: Average quality of 50,000 fake CelebA images from different sub-sampling methods over three repetitions. We draw 50,000 fake imagesby each method on which we compute the IS and FID. We repeat thissampling for three times and report the average IS and FID. Higher ISand lower FID are better. A grid search is conducted for DRE-F-SP toselect the hyperparameter, and the results under the optimal λ ∗ are shownin this table. We include the IS and FID of all training data and 10,000test data as a reference.Method IS (mean±std) FID (mean±std)- Real Data -All Training Data 3.321 -10,000 Test Data 3.319 1.572- SNGAN -No Subsampling 2.781±0.006 6.607±0.013DRS [10] 2.796±0.006 6.490±0.015MH-GAN [138] 2.766±0.005 6.649±0.019DRE-F-SP+RS (λ ∗ = 0.005) 2.865± 0.009 6.088± 0.007DRE-F-SP+MH (λ ∗ = 0.005) 2.870± 0.009 6.032± 0.019DRE-F-SP+SIR (λ ∗ = 0.005) 2.877± 0.021 6.116± 0.021Visual results: We show some example CelebA images from different subsam-pling methods in Figs. 3.12 and 3.12. We manually check each generated imageand outline it in yellow if we find obvious distortion or asymmetry in its face area(we ignore the background). We can see our proposed methods have fewer markedimages.3.5 ConclusionWe propose a novel subsampling framework (including DRE-F-SP+RS, DRE-F-SP+MH, and DRE-F-SP+SIR) for GANs to replace DRS and MH-GAN. In thisframework, a novel SP loss function is proposed for density ratio estimation, and itsrate of convergence is determined theoretically with respect to training size. Based116(a) No subsampling (28 yellow squares) (b) DRS (25 yellow squares)(c) MH-GAN (22 yellow squares) (d) DRE-F-SP+RS (13 yellow squares)Figure 3.12: Fake CelebA images from SNGAN (yellow outlines indicatesamples with obvious distortions and asymmetry) (Part I). Our pro-posed methods have fewer unrealistic images (yellow outlines) thanbaselines do.117(e) DRE-F-SP+MH (10 yellow squares) (f) DRE-F-SP+SIR (13 yellow squares)Figure 3.12: Fake CelebA images from SNGAN (yellow outlines indicatesamples with obvious distortions and asymmetry) (Part II). Our pro-posed methods have fewer unrealistic images (yellow outlines) thanbaselines do.on the SP loss, we further propose to do density ratio estimation in the featurespace learned by a specially designed ResNet-34. We demonstrate the efficiency ofthe overall framework on a 25 2-D Gaussians example and the CIFAR-10 dataset.Experimental results show that our proposed framework can dramatically improvedifferent types of GANs and substantially outperform DRS and MH-GAN. Ourapproach can also improve GANs (e.g., MMD-GAN), where DRS and MH-GANare not applicable.118Chapter 4Efficient Subsampling forGenerating High-Quality Imagesfrom Conditional GenerativeAdversarial Networks4.1 IntroductionAs an extension and an essential family of GANs, conditional generative adver-sarial networks (cGANs) [91] are able to estimate the image distribution givensome conditions. These conditions are usually categorical variables such as classlabels. cGANs with class labels are also known as class-conditional GANs [14, 94,100, 163]. In Chapter 2, we propose a new conditional GAN framework, contin-uous conditional generative adversarial network (CcGAN), which takes continu-ous, scalar variables (termed as regression labels) such as steering angles as con-ditions. Recent advances in unconditional GANs [59, 61] and conditional GANs[14, 30, 33] generally enable these models to generate high-quality images. Never-theless, low-quality images still appear frequently even with such advanced GANmodels during image generation. We would like to clarify that image quality dis-cussed in this chapter refers to not only the visual quality but also (1) image di-119versity for unconditional GANs; (2) intra-label diversity and label consistency (theconsistency of generated images with respect to the conditioning label) [29, 33] forconditional GANs.In Chapter 3, we propose a post hoc, subsampling framework for uncondi-tional GANs to filter out low-quality samples. This framework includes three sub-samplers, i.e., DRE-F-SP+RS, DRE-F-SP+MH, or DRE-F-SP+SIR. A naive ap-proach for subsampling class-conditional GANs is to apply DRE-F-SP+RS, DRE-F-SP+MH, or DRE-F-SP+SIR within each image class. This approach is practicalonly if there are a small number of image classes. Conversely, if there exist manyclasses, it may be infeasible in practice. For example, if we apply this approach tosubsample a BigGAN trained on Tiny-ImageNet [68] with 200 classes, we need tofit 200 density ratio models separately, which is often time-consuming and compu-tationally expensive. Moreover, since these 200 density ratio models may not sharethe same optimal training setups and it is generally impractical to tune the hyper-parameters for each model carefully, it is likely that some density ratio models arenot well-trained. Besides the high training costs, this approach may be invalid insubsampling CcGANs because the feature extraction in DRE-F-SP is inapplicableto images with regression labels only. Even if we find a suitable feature extrac-tion mechanism, this approach is still impractical unless there are very few distinctregression labels.To address such challenges and fill the research gap in subsampling condi-tional GANs, we propose a conditional density ratio based subsampling method.The proposed method is applicable to both class-conditional GANs and CcGANs.Moreover, we only need to fit one density ratio model in subsampling such condi-tional GANs. Our contributions can be summarized as follows:• In Section 4.2.1, we propose cDRE-F-cSP, which is able to estimate an im-age’s density ratio conditional on a class/regression label. We first introducea new feature extraction method for images with regression labels, since thefeature extraction mechanism in DRE-F-SP is not applicable to CcGANs.Then, we propose a novel conditional Softplus (cSP) loss, which enablesus to estimate density ratios conditional on different labels by fitting onlyone density ratio model. This density ratio model takes as input both the120high-level features and the class/regression label of an image and outputs thedensity ratio conditional on the given label.• We derive in Section 4.2.2 the error bound of the proposed conditional den-sity ratio model trained with the novel cSP loss.• In Section 4.2.3, we propose a rejection sampling scheme based on the es-timated conditional density ratios, termed cDRE-F-cSP+RS, to subsamplecGANs. An extra filtering scheme is proposed in Section 4.2.4 for CcGANsto increase the label consistency.• In Section 4.3, experiments on CIFAR-10, Tiny-ImageNet, RC-49 and UTK-Face demonstrate the effectiveness of the proposed cDRE-F-cSP+RS in sub-sampling BigGAN and CcGANs in terms of multiple metrics. Even withmuch less training cost, the proposed method performs comparable to (oreven better than) the state-of-the-art subsampling method.4.2 MethodAs discussed in Section 4.1, due to high training costs and the incomplete fea-ture extraction mechanism (only suitable for images with class labels), the un-conditional subsampling approach (i.e., DRE-F-SP+RS, DRE-F-SP+MH, DRE-F-SP+SIR) may be impractical for subsampling cGANs. Motivated by these issues,in this section, we propose a general and efficient subsampling method, which issuitable for both class-conditional GANs and CcGANs no matter the number ofdistinct class/regression labels.4.2.1 Conditional Density Ratio Estimation in Feature Space withConditional Softplus LossThis section introduces cDRE-F-cSP, a novel conditional density ratio estimation(cDRE) method. Assume we have Nr real image-label pairs (xr1,yr1), . . . ,(xrNr ,yrNr)and Ng fake image-label pairs (xg1,yg1), . . . ,(xgNg ,ygNg). Based on these samples,cDRE-F-cSP aims to estimate r∗(x|y) := pr(x|y)/pg(x|y), where pr(x|y) and pg(x|y)are respectively conditional PDFs of the real and fake image distributions given acondition y. Similar to DRE-F-SP, we conduct cDRE in a feature space learned by121a pre-trained neural network φ . For class-conditional GANs, we use the speciallydesigned ResNet-34 (also used by DRE-F-SP) to extract high-level features. ForCcGANs, since regression datasets (e.g., Cell-200 [30]) may not have class labels,we train a specially designed sparse autoencoder (AE) to extract features whose ar-chitecture is visualized in Fig. 4.1. The encoder with ReLU [42] as the final layeris treated as φ to extract sparse high-level features from images. The bottleneckdimension of the sparse autoencoder equals the dimension of the flattened inputimage. The decoding process is trained to not only reconstruct the input image butalso predict the regression label of the input image. The training loss of this sparseAE is the summation of three components: (1) the mean square error (MSE) be-tween the input image and the reconstructed image; (2) the MSE between the trueregression label and the predicted regression label; (3) the product of a positiveconstant λ ′ and the mean of all elements in h, where λ ′ controls the sparsity andis set as 10−3 in our experiment. The sparsity regularizer and the extra branch topredict regression labels are both used to avoid overfitting.Figure 4.1: The architecture of the sparse autoencoder for feature extractionin subsampling CcGANs. The bottleneck dimension equals the size ofthe flattened input image x. Since the encoder has ReLU as the lastlayer, the feature vector h is sparse. This autoencoder also has an extrabranch to predict the regression label of x.122Next, we propose a formulation of the true conditional density ratio functionin the feature space as followsψ∗(h|y) := qr(h|y)qg(h|y) =qr(h|y) · ‖∂h/∂x‖qg(h|y) · ‖∂h/∂x‖ =pr(x|y)pg(x|y) =: r∗(x|y), (4.1)where qr(h|y) and qg(h|y) are respectively the conditional PDFs of the real and fakedistributions of high-level features given a condition y. Based on Eq. (4.1) and thepre-trained neural network φ(x), we model the conditional density ratio functionψ∗(h|y) in the feature space by a 5-layer MLP denoted by ψ(h|y) with both h andits label y as input. To train ψ(h|y), we propose the conditional Softplus (cSP) lossas follows:Lc(ψ) = E(h,y)∼qg(h,y) [σ(ψ(h|y))−η(ψ(h|y))]−E(h,y)∼qr(h,y) [σ(ψ(h|y))] ,(4.2)where qg(h,y)= qg(h|y)p(y), qr(h,y)= qr(h|y)p(y), and p(y) is PDF of the marginallabel distribution. The empirical approximation to Eq. (4.2) isL̂c(ψ) =1NgNg∑i=1[σ(ψ(hgi |ygi ))ψ(hgi |ygi )−η(ψ(hgi |ygi ))]− 1NrNr∑i=1σ(ψ(hri |yri )).(4.3)Similar to DRE-F-SP, to prevent ψ(h|y) from overfitting the training data, a naturalconstraint applied to ψ(h|y) isEh∼qg(h|y)[ψ(h|y)] = 1. (4.4)If Eq. (4.4) holds, thenE(h,y)∼qg(h,y)[ψ(h|y)] = 1. (4.5)An empirical approximation to Eq. (4.5) is1NgNg∑i=1ψ(hgi |ygi ) = 1. (4.6)123Therefore, in practice, we minimize the penalized version of Eq. (4.3) as follows:minψ{L̂c(ψ)+λ Q̂c(ψ)}, (4.7)whereQ̂c(ψ) =(1NgNg∑i=1ψ(hgi |ygi )−1)2. (4.8)All experiments in this chapter use λ = 10−3. An algorithm shown in Alg. 9 isused to implement cDRE-F-cSP in practice.Algorithm 9: The optimization algorithm for the density ratio modeltraining in cDRE-F-cSP.Data: Nr real image-label pairs {(xr1,yr1), · · · ,(xrNr ,yrNr )}, a generator G, a pre-trainedResNet-34 or encoder φ(x) for feature extraction, a 5-layer MLP ψ(h|y) and apreset hyperparameter λ .Result: a trained conditional density ratio model r(x|y) = ψ(φ(x)|y) = ψ(h|y).1 Initialize ψ;2 for k = 1 to K do3 Sample a mini-batch of m real image-label pairs {(xr(1),yr(1)), · · · ,(xr(m),yr(m))} from{(xr1,yr1), · · · ,(xrNr ,yrNr )};4 Sample a mini-batch of m fake image-label pairs {(xg(1),yg(1)), · · · ,(xg(m),yg(m))} fromG;5 Update ψ via SGD or a variant with the gradient of Eq.(4.7), i.e.,5{L̂c(ψ)+λ Q̂c(ψ)}.6 end4.2.2 Error BoundIn this section, we derive the error bound of a density ratio model ψ(h|y) trainedwith the conditional Softplus loss L̂c(ψ). For simplicity, we ignore the penaltyterm in this analysis.Firstly, we introduce some notations. Let Ψ = {ψ : h → ψ(h|y)} denote thehypothesis space of the density ratio model ψ(h|y). We also define ψ˜ and ψˆ asfollows: ψ˜ = argminψ∈ΨLc(ψ) and ψˆ = argminψ∈Ψ L̂c(ψ). Please note that thehypothesis space Ψ may not cover the true density ratio function ψ∗. Therefore,Lc(ψ˜)−Lc(ψ∗)≥ 0. Denote by α all learnable parameters of ψ and assume α isin a parameter space A . Denote σ(ψ(h|y))ψ(h|y)−η(ψ(h|y)) by g(h|y;α ). Let124Rˆqr(h,y),Nr(Ψ) denote the empirical Rademacher complexity of Ψ, which is definedbased on independent feature-label pairs {(hr1,yr1), . . . ,(hrNr ,yrNr)} from qr(h,y).Then, we derive the error bound of the conditional density ratio estimate ψˆ underthe theoretical loss as follows:Theorem 4.2.1. If (i) Ng is large enough, (ii) A is compact, (iii) ∀g(h|y;α ) iscontinuous at α , (iv) ∀g(h|y;α ), ∃ a function gu(h|y) that does not depend on α ,s.t. |g(h|y;α )| ≤ gu(h|y), and (iv) E(h,y)∼qg(h,y)gu(h|y) < ∞, then ∀δ ∈ (0,1) and∀δ ′ ∈ (0,δ ] with probability at least 1−δ ,Lc(ψˆ)−Lc(ψ∗)≤ 1Ng + Rˆqr(h,y),Nr(Ψ)+2√4Nrlog(2δ ′)+Lc(ψ˜)−Lc(ψ∗).(4.9)Proof. We first decomposeLc(ψˆ)−Lc(ψ∗) as follows:Lc(ψˆ)−Lc(ψ∗)=Lc(ψˆ)− L̂c(ψˆ)+ L̂c(ψˆ)− L̂c(ψ˜)+ L̂c(ψ˜)−Lc(ψ˜)+Lc(ψ˜)−Lc(ψ∗)(Since L̂c(ψˆ)− L̂c(ψ˜)≤ 0)≤Lc(ψˆ)− L̂c(ψˆ)+ L̂c(ψ˜)−Lc(ψ˜)+Lc(ψ˜)−Lc(ψ∗)≤2 supψ∈Ψ∣∣∣L̂c(ψ)−Lc(ψ)∣∣∣+Lc(ψ˜)−Lc(ψ∗). (4.10)The second term in Eq. (4.10) is a constant which implies an inevitable error. Thefirst term can be bounded as follows:supψ∈Ψ∣∣∣L̂c(ψ)−Lc(ψ)∣∣∣≤ supψ∈Ψ∣∣∣∣E(h,y)∼qg(h,y) [σ(ψ(h|y))ψ(h|y)−η(ψ(h|y))]− 1NgNg∑i=1[σ(ψ(hgi |ygi ))ψ(hgi |ygi )−η(ψ(hgi |ygi ))]∣∣∣∣∣125+∣∣∣∣∣E(h,y)∼qr(h,y) [σ(ψ(h|y))]− 1Nr Nr∑i=1σ(ψ(hri |yri ))∣∣∣∣∣ . (4.11)Because of assumptions (i)-(iv), based on the uniform law of large number[126], for ∀ε > 0,limNg→∞P{supψ∈Ψ∣∣∣∣E(h,y)∼qg(h,y) [σ(ψ(h|y))ψ(h|y)−η(ψ(h|y))]− 1NgNg∑i=1[σ(ψ(hgi |ygi ))ψ(hgi |ygi )−η(ψ(hgi |ygi ))]∣∣∣∣∣}= 0.Based on this limit, we can derive an upper bound of the first term of Eq. (4.11)as follows. Since we can generate infinite fake images from a trained cGAN, Ng islarge enough. Let ε = 1/2Ng, ∀δ1 ∈ (0,1) with probability at least 1−δ1,supψ∈Ψ∣∣∣∣E(h,y)∼qg(h,y) [σ(ψ(h|y))ψ(h|y)−η(ψ(h|y))]− 1NgNg∑i=1[σ(ψ(hgi |ygi ))ψ(hgi |ygi )−η(ψ(hgi |ygi ))]∣∣∣∣∣≤ 12Ng .(4.12)The second term of Eq. (4.11) can be bounded based on Lemma 3.3.1 andTheorem 3.3.2 (the Rademacher bound [67]) in Chapter 3 as follows: ∀δ2 ∈ (0,1)with probability at least 1−δ2,∣∣∣∣∣E(h,y)∼qr(h,y) [σ(ψ(h|y))]− 1Nr Nr∑i=1σ(ψ(hri |yri ))∣∣∣∣∣≤2Rˆqr(h,y),Nr(σ ◦Ψ)+√4Nrlog(2δ2)≤12Rˆqr(h,y),Nr(Ψ)+√4Nrlog(2δ2). (4.13)Let δ =max{δ1,δ2} and δ ′ = δ , based on Eq. (4.12) and (4.13), we can derive Eq.(4.9).126Remark 4.2.1. Rˆqr(h,y),Nr(Ψ) on the right of Eq. (4.9) implies that we should notuse an overly complicated density ratio model. It supports our proposed cDRE-F-cSP because we just need a small neural network (e.g., a shallow MLP) to modelthe density ratio function in the feature space.4.2.3 cDRE-F-cSP+RS: A cDRE-based Rejection Sampling Schemefor Conditional GANsBased on the cDRE method proposed in Section 4.2.1, we develop a rejectionsampling scheme, termed cDRE-F-cSP+RS, to subsample conditional GANs. Theworkflow can be summarized in Fig. 4.2 and Alg. 10. This rejection samplingscheme is conducted for each distinct label y of interest. For example, on CIFAR-10 [66], we train only one density ratio model ψ(h|y), based on which we repeatthe rejection sampling scheme 10 times for 10 classes respectively.Figure 4.2: The workflow of cDRE-F-cSP+RS has two modules: cDRE-F-cSP and rejection sampling. M in the acceptance probability p equals tomaxh{qr(h)/qg(h)}, which can be estimated by evaluating ψ(h|y) onsome burn-in samples before subsampling.4.2.4 A Filtering Scheme for Subsampling CcGANsTo solve the problem of insufficient data, CcGANs use images with labels in thevicinity of y to estimate pr(x|y). When the CcGAN training is complete, the gen-127Algorithm 10: Subsampling fake images with label y by cDRE-F-cSP+RS.Data: a generator G, a trained ResNet-34 or encoder φ(x) for feature extraction, a trainedconditional density ratio model ψ(h|y).Result: images = {N filtered fake images with label y}1 Generate N′ burn-in fake images from G conditional on label y.;2 Estimate the density ratios of these N′ fake images conditional on y by evaluatingψ(φ(x)|y);3 M←max{N′ estimated density ratios};4 images← /0;5 while |images|< N do6 x← get a fake image with label y from G;7 ratio← ψ(φ(x)|y);8 M←max{M,ratio};9 p← ratio/M (i.e., the acceptance probability in RS);10 u← Uniform[0,1];11 if u≤ p then12 Append(x, images);13 end14 enderator can generate infinite images given a regression label y (termed the assignedlabel). However, the true labels of these generated images may be inconsistent withthe assigned label, and such label inconsistency is very likely to happen in the Cc-GAN sampling because of the vicinal training technique. Consequently, evaluatingthe density ratio model ψ(φ(x)|y) on images generated from CcGANs with the as-signed label y may be problematic because the true density ratio function r∗(x|y)may be ill-defined on these images. We propose a filtering scheme in both the den-sity ratio model training and the subsampling process to deal with this issue. Tobe specific, we first train a regression-oriented CNN (we use VGG-11 [130] in ourexperiments) on the same dataset used for the CcGAN training. This CNN is usedto predict labels of fake images generated from CcGANs, and these predicted la-bels are treated as the true labels of the generated images. Then, before evaluating5{L̂c(ψ)+λ Q̂c(ψ)}in the density ratio model training (Alg. 9), we predict thelabels of the m fake images and filter out those images if the mean absolute error(MAE) between their assigned labels and predicted labels is larger than a thresholdτ . Finally, before conducting rejection sampling in the subsampling process (Fig.4.2 and Alg. 10), we filter out fake images with MAE between assigned labels and128true labels larger than the threshold τ . A rule of thumb to select the filtering thresh-old τ is shown in Alg. 11. Besides making valid the evaluation of ψ(φ(x)|y) onimages generated from CcGANs, empirical study in Section 4.3.3 and 4.3.4 showsthat this filtering scheme can effectively increase the label consistency.Algorithm 11: A rule of thumb to select the filtering threshold τ .1 Sample N f images per label from the trained CcGAN without subsampling;2 Predict the labels of these fake images by the pre-trained VGG-11;3 Compute MAE between the predicted labels and assigned labels of the generated images;4 Sort these MAEs from smallest to largest and the 0.8 quantile of these MAEs is set as thefiltering threshold τ .4.3 ExperimentsIn this section, the objective is to experimentally demonstrate the effectiveness ofthe proposed subsampling scheme for both class-conditional GANs and CcGANs.We conduct experiments on four image datasets. In subsampling class-conditionalGANs, we utilize the CIFAR-10 [66] and Tiny-ImageNet [68] datasets. In subsam-pling CcGANs, we experiment on RC-49 [30, 33] and UTKFace [164] datasets.In the class-conditional GAN setting, the quality of fake images is evaluated bytwo metrics: Fre´chet inception distance (FID) [52] and Intra-FID [94]. Intra-FIDis an overall image quality metric, which computes the FID separately for eachclass and reports the average FID score. A lower Intra-FID/FID index indicates abetter quality of sampled fake images, or vice versa. In the CcGAN setting, wefollow CcGAN in Chapter 2 and adopt four image quality assessment metrics: (i)Intra-FID [94] is an overall image quality metric; (ii) Naturalness Image QualityEvaluator (NIQE) [93] evaluates the visual quality of fake images. Please notethat visual quality is only one aspect of image quality; (iii) Diversity measures thediversity of fake images; and (iv) Label Score (LS) evaluates label consistency.We prefer small Intra-FID, FID, NIQE, and Label Score but large Diversity.Please refer to Supp. A.11.3 and A.13.3 for the details of these metrics.1294.3.1 CIFAR-10We first evaluate the effectiveness of the proposed method in subsampling BigGAN[14] in the conditional image synthesis setting on the CIFAR-10 dataset [66].Experimental setup: CIFAR-10 consists of 60,000 (32× 32) RGB images uni-formly from 10 classes. The overall number of training samples is 50,000 (5000for each class), and the remaining 10,000 samples (1000 for each class) are fortest. The class-conditional GAN models employ the BigGAN architecture [14]. Toinvestigate the effectiveness of the proposed method on different GAN models, wetrain the BigGAN under three different settings by varying the number of trainingsamples. Thus we obtain three BigGAN models, denoted respectively as BC-50k,BC-20k, and BC-10k. Specifically, the BC-50k model is obtained by training Big-GAN using all 50,000 samples from the CIFAR-10 training set. The other twomodels are obtained using randomly chosen subsets of the training set with 20,000(BC-20k) and 10,000 (BC-10k) samples, respectively.Next, we implement the proposed cDRE-F-cSP+RS for each setting. Firstly,we train a customized ResNet-34 classifier on the training set for feature extrac-tion. Then, by employing the proposed loss in Eq. (4.7), a 5-layer MLP is trainedas the conditional density ratio model in the feature space learned by the ResNet-34. As a reference, we also implement DRE-F-SP+RS [31] in each setting (de-noted by 10-DRE-F-SP+RS), where ten unconditional density ratio models (ten5-layer MLPs) are carefully trained for the ten image classes separately. Thus, theproposed method needs only 10% of the training costs (e.g., storage space and pa-rameter tuning) spent by 10-DRE-F-SP+RS. Please refer to Section A.13 for thedetailed setups.Quantitative results: We quantitatively compare the image quality of fake im-ages subsampled using no subsampling, 10-DRE-F-SP+RS, and cDRE-F-cSP+RS.With each sampling method, we draw 50,000 fake images (5000 per class) fromeach of the three GAN models. Table 4.1 compares the results and shows thatsamples from either 10-DRE-F-SP+RS or cDRE-F-cSP+RS subsampling methodssignificantly outperform no subsampling. Moreover, the performance gain tendsto increase when GAN models become less well-trained after applying either ofthe two subsampling schemes. Finally, compared with 10-DRE-F-SP+RS, the pro-130posed method achieves comparable performances, however, it requires only 10%of the computational costs to train the density ratio models. Please note that itis impractical to adopt DRE-F-SP+RS [31] and other unconditional subsamplingmethods [10, 19, 83, 138] if the number of classes is large (e.g., 200 classes onTiny-ImageNet as shown in Section A.11).Visual results: We also show the example fake images generated from each sub-sampling method for some image classes (e.g., car and horse). E.g., in Figs. 4.4and 4.5, we show 100 examples images for each method for the “car” class and the“horse” class in the BC-10k setting. We also show 100 real images as reference inthe visualized image samples.In Figs. 4.4, we observe numerous images (marked in red boxes) from the no-subsampling method, which we can hardly recognize as cars. By contrast, thereappear much less such low-quality examples after subsampling by 10-DRE-F-SP+RS and the proposed cDRE-F-cSP+RS methods. We have similar observationsin Fig. 4.5.We also show in Fig. 4.3 the FID comparison for each class. For each class,cDRE-F-cSP+RS shows clear superiority over the no-subsampling scheme, and itachieves comparable performance with 10-DRE-F-SP+RS.Table 4.1: CIFAR-10 dataset: Quality of 50,000 fake images (5000 per class)sampled from different sampling methods on three trained GAN mod-els. cDRE-F-cSP+RS only requires 10% training costs of 10-DRE-F-SP+RS.Sampling methodsBC-50k BC-20k BC-10kIntra-FID FID Intra-FID FID Intra-FID FIDno-subsampling 0.915 0.415 1.636 0.973 3.271 2.60110-DRE-F-SP+RS 0.542 0.226 1.158 0.675 2.511 1.948cDRE-F-cSP+RS 0.598 0.240 1.267 0.700 2.621 1.9874.3.2 Tiny-ImageNetThis experiment further demonstrates the effectiveness of the proposed method insubsampling class-conditional GANs on the Tiny-ImageNet dataset [68].131Figure 4.3: CIFAR-10: FID versus class plots in sampling the BC-10k class-conditional GAN model. FID scores of cDRE-F-cSP+RS are smallerthan those of no-subsampling and comparable to those of 10-DRE-F-SP+RS over all classes.Experimental setup: The Tiny-ImageNet dataset [68] contains 200 image classesfor training. Each class has 500 images for training and 50 images for test. Allimages are RGB images of size 64× 64. In the Tiny-ImageNet dataset, we adoptthe BigGAN architecture and train the BigGAN model on all training images. Wealso adopt respectively a customized ResNet-34 and a 5-layer MLP for featureextraction and conditional density ratio estimation.Since it is impractical to train 200 unconditional density ratio models [31] onthis dataset, we compare the proposed subsampling scheme with the no-subsamplingmethod only. From each method, we sample one million (5000 per class) fake im-ages for image quality evaluation. Please refer to Supp. A.12 for details.Quantitative results: Table 4.2 shows image quality comparisons of fake images132(a) Real Images (b) no-subsampling(c) 10-DRE-F-SP+RS (d) cDRE-F-cSP+RSFigure 4.4: Example CIFAR-10 images for the “car” class in the BC-10k set-ting. We observe numerous images (marked in red boxes) from the no-subsampling method, which we can hardly recognize as cars. By con-trast, there appear much less such low-quality examples after subsam-pling by 10-DRE-F-SP+RS and the proposed cDRE-F-cSP+RS meth-ods.133(a) Real Images (b) no-subsampling(c) 10-DRE-F-SP+RS (d) cDRE-F-cSP+RSFigure 4.5: Example CIFAR-10 images for the “horse” class in the BC-10ksetting. We observe numerous images (marked in red boxes) fromthe no-subsampling method, which we can hardly recognize as cars.By contrast, there appear much less such low-quality examples aftersubsampling by 10-DRE-F-SP+RS and the proposed cDRE-F-cSP+RSmethods.134sampled using no-subsampling and the proposed methods on Tiny-ImageNet. Theproposed method enhances image quality for both metrics, particularly the FIDscore.Visual results: We visualize some example fake images generated from two sam-pling methods for some image classes (e.g., goldfish and tower) in Figs. 4.6 and4.7.In Fig. 4.6, as marked by red boxes, such “goldfish” images from the no-subsampling method are of very low quality. Also, we can observe some replicatedsamples (a.k.a mode collapse) for samples from the no-subsampling method. Evenfor images that look like “goldfish”, such images appear to look “darker” and lesscolorful than real images. Compared to the no-subsampling method, the proposedmethod generates significantly improved image quality, i.e., our image samples areboth more diverse and look much more natural those from the baseline method. InFig. 4.7, we can also observe fake image samples from the no-subsampling methodgenerally look less appealing than those from the proposed method. Moreover, forsome images (marked in red boxes) from the no-subsampling method, the tower isactually missing in those “tower” images. However, we can clearly see the towerin images that are sampled from our method.Table 4.2: Tiny-ImageNet dataset: Quality of one million fake images (5000per class) sampled from two sampling methods on a trained BigGANmodel.Sampling methods Intra-FID FIDno-subsampling 30.226 5.012cDRE-F-cSP+RS 29.361 2.0054.3.3 RC-49This experiment is conducted on RC-49 to show that the proposed subsamplingscheme also works in CcGANs.Experimental setup: RC-49 is a benchmark dataset in Chapter 2. It is made byrendering 49 3-D chair models individually. Each chair model is rendered at 899yaw angles from 0.1◦ to 89.9◦ with a stepsize of 0.1◦. This dataset contains 44,051135(a) Real Images(b) no-subsampling (c) cDRE-F-cSP+RSFigure 4.6: Example Tiny-ImageNet images for the “goldfish” class. Asmarked by red boxes, such “goldfish” images from the no-subsamplingmethod are of very low quality. Also, we can observe some replicatedsamples for samples from the no-subsampling method. Even for imagesthat look like “goldfish”, such images appear to look “darker” and lesscolorful than real images. Compared to the no-subsampling method, theproposed method generates significantly improved image quality, i.e.,our image samples are both more diverse and look much more naturalthose from the baseline method.136(a) Real Images(b) no-subsampling (c) cDRE-F-cSP+RSFigure 4.7: Example Tiny-ImageNet images for the “tower” class. We canobserve fake image samples from the no-subsampling method generallylook less appealing than those from the proposed method. Moreover, forsome images (marked in red boxes) from the no-subsampling method,the tower is actually missing in those “tower” images. However, we canclearly see the tower in images that are sampled from our method.137RGB images of size 64×64 with corresponding yaw angles as labels. We followthe implementation of CcGAN in 2 which adopts the SNGAN architecture [95]. OnRC-49, we train three CcGAN models with SVDL+ILI by varying the number oftraining samples (i.e., 25, 15, 5) for each label. Thus we obtain three CcGAN mod-els as SR-25, SR-15, and SR-5, respectively. To perform subsampling, we firstlytrain on RC-49 a sparse autoencoder with an extra branch for label prediction (seeFig. 4.1). The trained autoencoder is utilized as the feature extractor. We then traina 5-layer MLP to estimate conditional density ratios. In subsampling CcGANs, wealso adopt the proposed filtering scheme in Alg. 11. Detailed experimental setupscan be found in Supp A.13.Quantitative results: Table 4.3 reports quantitative evaluations of subsampledfake images from the no-subsampling and cDRE-F-cSP+RS on RC-49. Clearly,cDRE-F-cSP+RS consistently outperforms no-subsampling in all three settings interms of the overall image quality measured by Intra-FID. More specifically, bothsubsampling methods produce fake images with comparable visual quality as in-dicated by NIQE. The proposed subsampling method, however, generates fake im-ages with improved diversity and label consistency on RC-49. Moreover, as shownin Fig. 4.9, we evaluate the diversity/label consistency of both methods at eachangle. The proposed method consistently improves over no-subsampling, indicat-ing its effectiveness in generating more diverse and label consistent fake imagesuniformly over all angles in RC-49.The influence of the MAE quantile in Alg. 11 on the performance of cDRE-F-cSP+RS is shown in Table 4.4. We can see a smaller MAE quantile often leadsto a larger Intra-FID, a higher Diversity, a smaller Label Score, but an unchangedNIQE, which implies there is a trade-off between the image diversity and labelconsistency and Intra-FID is more sensitive to Diversity. This table confirms 0.8 isa good choice for the MAE quantile, trading slightly less image diversity for betterlabel consistency.Visual results: We show the example fake images generated from each subsam-pling method in Figs. 4.8. From such image examples, we observe that the no-subsampling method generates many image samples that suffer from the modecollapse issue. By contrast, we have largely alleviated this issue in the proposedcDRE-F-cSP+RS subsampling method. Therefore, we can safely conclude that our138method generates more diverse fake images than the baseline sampling method.(a) Real Images(b) no-subsampling (c) cDRE-F-cSP+RSFigure 4.8: Example RC-49 images for angles from 4.5 degree to 85.5 de-gree (from top to bottom) in the SR-5 setting. We observe that the no-subsampling method generates many image samples that suffer from themode collapse issue. By contrast, we have largely alleviated this issuein the proposed cDRE-F-cSP+RS subsampling method.The reason for the improvement of the Diversity score: It may be counter-intuitive that the Diversity score is increased when cDRE-F-cSP may reject some139(a) RC-49: Diversity vs Angle(b) RC-49: Label Score vs AngleFigure 4.9: Line graphs of Diversity/Label Score versus regression labelsin subsampling the SR-5 CcGAN model. The proposed subsamplingmethod consistently outperforms the no-subsampling method across allregression labels.140Table 4.3: RC-49 dataset: Quality of 179,800 fake images (200 per angle)sampled from two sampling methods on three trained GAN models.CcGANmodelsSamplingmethodsIntra-FID NIQE DiversityLabelScoreSR-25no-subsampling 0.386 1.759 2.952 1.930cDRE-F-cSP+RS 0.307 1.755 3.081 1.545SR-15no-subsampling 0.395 1.779 2.939 2.020cDRE-F-cSP+RS 0.309 1.794 3.052 1.559SR-5no-subsampling 0.465 1.860 2.778 2.515cDRE-F-cSP+RS 0.357 1.836 2.971 1.930Table 4.4: RC-49 dataset: Influence of the MAE quantile in the filtering algo-rithm (Alg. 11) on the performance of cDRE-F-cSP+RS in subsamplingSR-25.MAEquantileIntra-FID NIQE Diversity Label Score1.0 0.292 1.763 3.110 1.9960.9 0.308 1.755 3.089 1.7240.8 0.307 1.755 3.081 1.5450.7 0.353 1.756 3.035 1.371generated images. We show Figs. 4.10 and 4.10 to illustrate. In this figure, wevisualize the distribution of chair types (class labels) of 200 fake images generatedfrom no-subsampling and cDRE-F-cSP+RS at each of four randomly selected an-gles (i.e., 20◦, 30◦, 65◦, and 75◦). If we do not apply subsampling, some chair typesmay appear very frequently while others appear with very low frequencies. Thisimbalance among chair types reduces the diversity among the 200 generated im-ages. After applying cDRE-F-cSP+RS, we can see many blue spikes are shortened,which balances the chair types among generated images. Therefore, the Diversityscore is improved.4.3.4 UTKFaceThis experiment evaluates the performance of subsampling methods on UTKFace[164], another regression dataset used in CcGANs [30, 33].141(a) Angle 20 degree(b) Angle 30 degreeFigure 4.10: The distribution of the 49 chair types within 200 fake imagesgenerated from no-subsampling and cDRE-F-cSP+RS at each of fourrandomly selected angles in the SR-5 setting (Part I). After subsam-pling, the distribution of chairs types become more balance, whichincreases the entropy of 200 images’ class labels (i.e., the Diversity).142(c) Angle 65 degree(d) Angle 75 degreeFigure 4.10: The distribution of the 49 chair types within 200 fake imagesgenerated from no-subsampling and cDRE-F-cSP+RS at each of fourrandomly selected angles in the SR-5 setting (Part II). After subsam-pling, the distribution of chairs types become more balance, whichincreases the entropy of 200 images’ class labels (i.e., the Diversity).143Experimental setup: UTKFace is an RGB human face image dataset with agesas regression labels. This experiment utilizes the preprocessed UTKFace dataset[30, 33], which consists of 14,760 RGB images with ages in [1, 60]. The numberof images ranges from 50 to 1051 for different ages. All images are of size 64×64.Similar to the RC-49 experiments in Section 4.3.3, we adopt the SNGAN architec-ture [95] for CcGAN. In training CcGAN on UTKFace, the maximum number ofimages per age equals 200. Thus we obtain a trained CcGAN model as SU-200.Next, we train a sparse autoencoder on UTKFace as the feature extractor. We thentrain a 5-layer MLP for conditional density estimation with training images fromUTKFace and images generated by SU-200. Besides the proposed method, we alsoimplement 60-DRE-F-SP+RS, i.e., training 60 unconditional density ratio models[31] for 60 ages separately. By contrast, cDRE-F-cSP+RS only needs to train onedensity ratio model. Thus, cDRE-F-cSP+RS consumes about 1.7% of the trainingcosts spent by 60-DRE-F-SP+RS. A detailed experimental setup is in Supp. A.11.Quantitative results: In Table 4.5, we show performance comparisons of theno-subsampling method, 60-DRE-F-SP+RS, and cDRE-F-cSP+RS. In terms ofthe overall image quality comparison, both the proposed method and 60-DRE-F-SP+RS outperform the no-subsampling method measured by Intra-FID. Comparedwith 60-DRE-F-SP+RS, the proposed method shows a comparable overall perfor-mance. The three methods generate fake images with comparable visual quality.While 60-DRE-F-SP+RS has slightly worse diversity than no-subsampling, theproposed cDRE-F-cSP+RS shows substantial improvement in image diversity andlabel score over both no-subsampling and 60-DRE-F-SP+RS.Visual results: We show the example fake images generated from each subsam-pling method in Figs. 4.11. As marked in red boxes in images from the no-subsampling method, we observe that, occasionally some face images indeed haveclearly wrong ages. For example, in the second row, two marked face images at anage 9 look clearly more mature than they should be. Such image samples give riseto the label inconsistency issue in CcGANs. By contrast, we can hardly observesuch label inconsistency issue for images from the proposed method.144(a) Real Images (b) no-subsampling(c) 60-DRE-F-SP+RS (d) cDRE-F-cSP+RSFigure 4.11: Example UTKFace images for ages from 3 to 57 (from top tobottom). As marked in red boxes in images from the no-subsamplingmethod, we observe that, occasionally some face images indeed haveclearly wrong ages. For example, in the second row, two marked faceimages at an age 9 look clearly more mature than they should be. Suchimage samples give rise to the label inconsistency issue in CcGANs.By contrast, we can hardly observe such label inconsistency issue forimages from the proposed method.145Table 4.5: UTKFace dataset: Quality of 60,000 fake images (1000 perage) sampled from different sampling methods on the SU-200 CcGANmodel. cDRE-F-cSP+RS only requires 1.7% training costs of 60-DRE-F-SP+RS.SubsamplingmethodsIntra-FID NIQE DiversityLabelScoreno-subsampling 0.432 1.725 1.304 7.44460-DRE-F-SP+RS 0.423 1.722 1.298 7.300cDRE-F-cSP+RS 0.424 1.737 1.335 6.7634.4 ConclusionIn this chapter, we present a novel conditional subsampling scheme to improvethe image quality of fake images from conditional GANs. First, we propose novelconditional extensions of density ratio estimation (cDRE) in the feature space andthe conditional Softplus loss function (cSP). Then, we learn the conditional densityratio model through an MLP network. Also, we derive the error bound of a con-ditional density ratio model trained with the proposed cSP loss. A novel filteringscheme is also proposed in subsampling CcGANs to improve the label consistency.Finally, we validate the effectiveness of the proposed subsampling scheme with ex-tensive experiments in sampling multiple conditional GAN models on four imagedatasets with diverse evaluation metrics.146Chapter 5Distilling and TransferringKnowledge via cGAN-generatedSamples for Image Classificationand Regression5.1 IntroductionA heavyweight model is often a deep, overparameterized neural network or anensemble of multiple deep neural networks in deep learning. It usually has highprecision but also incurs some costs: (1) a high memory cost due to the large modelsize (i.e., many learnable parameters); (2) a low inference speed (i.e., number ofimages processed by a model per second). Note that a neural network’s inferencespeed is related to both the number of learnable parameters and the connection de-sign. For example, DenseNet-121 [55] has a densely connected pattern (leading toa high computation cost) while VGG11 [130] is a simple, straightforward network.Although DenseNet-121 has fewer learnable parameters than VGG11 does, evalu-ating DenseNet-121 is about 10 times slower than VGG11. Unfortunately, in manyapplication scenarios (e.g., deploying neural networks on mobile devices), limitedcomputational resources are available to evaluate a trained model, therefore we can147only afford some lightweight models that are fast and memory-efficient. However,the small model capacity and the limited amount of labeled data often preventthe lightweight model from achieving sufficiently high precision. Therefore, howto use an accurate heavyweight model to improve the performance of a lightweightmodel has been actively studied recently.Knowledge distillation (KD), first proposed by [15] and then developed by [53],is a popular method to improve the performance of a lightweight model by utiliz-ing the knowledge (some information learned from training data) distilled from anaccurate, heavyweight model [78, 80, 85, 86, 107, 146, 162]. The heavyweightand lightweight models in KD are often known respectively as a teacher modeland a student model. After [53] introduced the baseline knowledge distillation(BLKD) [53], many KD methods have been proposed for the image classificationtask [45, 147], but we will only briefly review in this chapter BLKD [53] and somerecently proposed methods [92, 118, 159]. BLKD transfers knowledge from theteacher model to the student model by matching the logits between these two mod-els (i.e., the raw predictions generated by a classification-oriented neural network,which is then passed to the softmax function). Teacher assistant knowledge distil-lation (TAKD) [92] is a recent improvement of BLKD. [92] finds that BLKD mayfail if the teacher and student models’ performance gap is too big. TAKD appliesBLKD to transfer knowledge from the teacher model to an intermediate model,termed the teacher assistant (TA) model, to fill this gap. The TA model often per-forms better than the student model but worse than the teacher model. Then, TAKDtransfers knowledge from the TA model to the student model by applying BLKDagain. However, our experiments in Section 5.5 show that TAKD is sometimesinferior to BLKD. Self-supervised learning as an auxiliary task for knowledge dis-tillation (SSKD) [159] is another recently proposed KD method, which introducesself-supervised learning as an auxiliary task into knowledge distillation to transferknowledge from the teacher model to the student model. However, the architec-tures of the teacher and student models sometimes need to be adjusted to fit into theSSKD framework, which may deteriorate the performance of the teacher model. Inmany scenarios where we need to apply KD, we often have limited labeled images.Thus, [118] incorporates the unsupervised data augmentation (UDA) [158] intoBLKD to improve the performance of the baseline distillation. Different from the148image classification task, the application of KD in image regression with a scalarresponse variable (e.g., angle and age) has rarely been studied. [165] proposes aKD method specially designed for estimating ages from human face images. How-ever, this method is not applicable to general image regression tasks with a scalarresponse because some techniques of the proposed framework are only applicablefor age estimation. To our best knowledge, there is no KD method general enoughfor all image regression tasks with a scalar response. Moreover, all the abovemethods are either designed for image classification or image regression; there isno unified KD framework that is suitable for both tasks.Generative adversarial networks (GANs) [14, 30, 33, 43, 59, 61, 91, 94, 95,100, 163] are state-of-the-art generative models for image synthesis. Some modernGAN models such as BigGAN [14] and StyleGAN [59, 61] are able to generatehigh-resolution, even photo-realistic images. Conditional generative adversarialnetworks (cGANs) are a type of GANs that can generate images in terms of someconditions. Most cGANs are designed for categorical conditions such as class la-bels [14, 91, 94, 95, 100, 163], and cGANs with class labels as conditions arealso known as class-conditional GANs. Recently, [30, 33] propose a new cGANframework, termed continuous conditional GANs (CcGANs). CcGANs can gener-ate images conditional on continuous, scalar variables (termed regression labels).In the scenario with limited training data, the performance of GANs often dete-riorates. To alleviate this problem for unconditional GANs and class-conditionalGANs, DiffAugment [166] proposes to conduct online transformation on imagesduring the GAN training. Our experiments show that it also applies to CcGANs.Besides the advances in the GAN theory, some papers [35, 88, 131, 153, 169]use GAN-generated data for data augmentation in image classification tasks withinsufficient training data. However, even state-of-the-art GANs may generate low-quality samples, which may negatively impact the classification task. Fortunately,some recently proposed subsampling methods [31, 32] may be applied to eliminatethese low-quality samples. Additionally, some works [81, 125, 150, 160] proposeto incorporate the adversarial loss of GANs into KD, but their performance is notstate-of-the-art.Motivated by the limitations of existing KD methods and the recent advances ofcGANs, we propose a general and flexible cGAN-based KD framework suitable for149both image classification and regression (with a scalar response). Our contributionscan be summarized as follows:• In Section 5.2, we introduce a novel KD framework termed cGAN-KD,which distills and transfers knowledge via cGAN-generated samples. As apreliminary, we propose to train BigGAN [14] for classification or CcGANs[30, 33] for regression. We also suggest incorporating DiffAugment [166]into the cGAN training when labeled data are limited. Fake image-label pairs(i.e., fake samples) generated from cGAN are then subsampled and filteredto drop low-quality samples. The knowledge distillation takes place whena pre-trained teacher model adjusts the labels of fake samples. Then, theseprocessed fake samples are used to augment the training set. Finally, thestudent model is trained on the augmented training set, where the knowledgetransfer is conducted implicitly.• Compared with existing KD methods, the proposed framework has severaladvantages, as summarized in Section 5.4. Notably, cGAN-KD is a uni-fied KD framework suitable for both classification and regression tasks. Itis compatible with state-of-the-art KD methods and particularly suitable forlimited labeled data scenarios. Moreover, cGAN-KD is insensitive to ar-chitectural differences between teacher and student networks as long as theteacher model has a high precision.• In Section 5.3, we derive the error bound of a student model trained in thecGAN-KD framework, which not only helps us understand how cGAN-KDtakes effect but also guides the implementation of cGAN-KD in practice.Such an analysis is often omitted in knowledge distillation papers. The errorbound suggests that we should generate as many processed fake samples aspossible and choose a teacher model with high precision.• In Section 5.5, extensive experiments on CIFAR-10, Tiny-ImageNet, RC-49, and UTKFace datasets demonstrate the effectiveness of cGAN-KD inthe image classification and regression tasks with limited training data. Inimage classification tasks, state-of-the-art KD methods are also improved ifincorporated into cGAN-KD. An ablation study on CIFAR-10 and RC-49 is150also conducted to show the necessity of the subsampling, filtering, and labeladjustment modules in the cGAN-KD framework. Another ablation studyshows that more processed fake images often lead to more stable knowledgedistillation performance.Although existing GAN-related KD methods [81, 125, 150, 160] seem relatedwith the proposed cGAN-KD in concept, our method is fundamentally differentfrom them, mainly due to three reasons: (1) Our approach is the first frame-work that utilizes cGAN-generated samples to distill and transfer knowledge, whileworks [81, 125, 150, 160] only incorporate adversarial losses into conventional KDmethods (e.g., [53]), and they cannot achieve the state-of-the-art performance. (2)Our KD framework is applicable to both classification and regression tasks, whileKD methods in [81, 125, 150, 160] can only apply to classification tasks. (3) Ourapproach is compatible with state-of-the-art KD methods (e.g., [92, 159]) and wecan generally boost their performances, while methods in [81, 125, 150, 160] donot have such a merit.It is also worth noting that cGAN-KD has a subsampling and filtering schemeto efficiently drop low-quality samples generated from cGANs, which does notexist in existing GAN-based data augmentation methods [35, 88, 131, 153, 169].5.2 Proposed MethodWhile many KD methods have been proposed for image classification [45, 147],there is only one KD method for image regression (scalar response) [165]. Un-fortunately, it is specially designed for age estimation, and is not applicable as ageneral KD method for image regression. Moreover, there is no KD framework forboth types of tasks.This section proposes a unified KD framework, termed cGAN-KD, which issuitable for both image classification and regression (with scalar response) tasks.The proposed framework can also fit into many state-of-the-art KD methods forimage classification to improve their performances. In addition, it is suitable fortasks with limited labeled data and insensitive to architecture differences betweenthe teacher and student models.1515.2.1 Problem FormulationBefore we introduce cGAN-KD, let us formulate the KD task mathematically asfollows. Assume we have a set of Nr image-label pairs, i.e.,Dr = {(xri ,yri )|i = 1, . . . ,Nr} ,which are randomly drawn from the true joint distribution for image and labelswith PDF pr(x,y). We also have a teacher model ft and a student model fs whichare trained on Dr. ft often has a smaller test error than fs does, i.e.,E(x,y)∼pr(x,y)L ( ft(x),y)≤ E(x,y)∼pr(x,y)L ( fs(x),y),whereL is either the cross entropy (CE) loss (i.e., ∑Cc=1{−yc log psc}) for classifi-cation or the square error (SE) loss (i.e., ( f (x)− y)2) for regression. The objectiveof KD is to reduce the test error of fs by using the knowledge learned by ft .5.2.2 The Workflow of cGAN-KDAAs a preliminary of cGAN-KD, we need to train a cGAN on Dr. For imageclassification, we suggest adopting state-of-the-art class-conditional GANs suchas BigGAN [14]. For image regression with a scalar response, we should useCcGANs [30, 33]. In the scenario with very few training data, we propose to applyDiffAugment [166] to stabilize the cGAN training. After the cGAN training, theproposed KD framework can be applied. In Fig. 5.1, we show the workflow ofcGAN-KD, which includes three important modules denoted respectively by M1,M2, and M3. First, we draw a set of Ng unprocessed fake image-label pairs fromthe trained cGAN, i.e.,Dg ={(xgi ,ygi )|i = 1, . . . ,Ng}.These fake samples are then subsampled and filtered by M1 to drop low-qualitysamples and form a subset of Dg, i.e.,D˜g ={(x˜gi , y˜gi )|i = 1, . . . ,Mg,Mg ≤ Ng}.152The next module M2 in the pipeline adjusts the labels of images in D˜g by a pre-trained teacher model ft and outputs a set of processed samples, i.e.,D̂g ={(x˜gi , yˆgi )|i = 1, . . . ,Mg,Mg ≤ Ng, yˆgi = ft(x˜gi )}.The processed samples D̂g are then used to augment the training set Dr. Finally,M3 trains the student model fs on the augmented training set Dr∪ D̂g. The studentmodel trained on Dr ∪ D̂g is expected to perform better than the one trained on Dr.More details of the three modules are described in Sections 5.2.3 to 5.2.5 and theevolution of fake sample datasets is shown in Fig. 5.2.Figure 5.1: The workflow of cGAN-KD. There are three important modulesdenoted respectively by M1, M2, and M3 in this framework.5.2.3 M1: Drop Low-quality Fake SamplesSince low-quality samples may reduce prediction accuracy if used to augment thetraining set, M1 is adopted to drop these samples, including two sequential sub-modules: a subsampling module and a filtering module.The subsampling module implements cDRE-F-SP+RS [32] which performsrejection sampling to accept or reject a fake image-label pair (xg,yg) in terms ofthe density ratio of xg conditioning on yg. [32] shows that cDRE-F-SP+RS caneffectively improve the overall image quality of both class-conditional GANs andCcGANs in the conditional image synthesis setting. Thus, the subsampling moduleis very suitable for dropping low-quality samples.The subsequent filtering module is another strategy to drop low-quality sam-ples. Assume that we generate a fake image xg from a trained cGAN conditionalon a label yg, then yg is called the assigned label of xg in this chapter. In the filtering153module, we use the pre-trained teacher model ft to predict the label of xg. Our ex-perimental study in Supp. A.15.2 and A.17.2 shows that a significant error (i.e., thecross entropy loss for classification or mean absolute error for regression) betweenthe assigned and predicted labels often implies poor visual quality. Based on thisobservation, we propose to drop fake samples with errors larger than a threshold,which is summarized in Alg. 12. The filtering threshold equals to the ρ-th quantileof fake samples’ errors and the optimal ρ is selected by a grid search algorithm(i.e., Alg. 13).Algorithm 12: An algorithm to implement the filtering module with ahyper-parameter ρ to drop low-quality samples.1 Sample Ng fake image-label pairs from a trained cGAN with cDRE-F-SP+RS;2 Predict the labels of these fake images by the pre-trained teacher model ft ;3 Compute the error (i.e., cross entropy for classification or MAE for regression) betweenthe assigned and predicted labels;4 Sort these errors from smallest to largest and the ρ-th quantile of these errors is set as thefiltering threshold;5 Remove fake image-label pairs with errors larger than the filtering threshold.Algorithm 13: A grid search algorithm to select the hyper-parameter ρin the filtering module.1 Set a grid of candidate values for ρ (e.g., 0.3 to 0.9 with a stepsize 0.1 in our experiments);2 Randomly split the training set Dr into a sub-training set and a validation set with a ratioof 4:1;3 for each ρ in the grid do4 Generate Mg fake image-label pairs based on Alg. 12;5 Augment the sub-training set with these fake samples;6 Train the student model fs from scratch on the augmented training set;7 Compute the validation error of fs on the validation set;8 end9 The optimal ρ in the grid minimizes the validation error. Please note that if we havemultiple student models (e.g., our experiments in Section 5.5), the optimal ρ minimizesthe average validation error of these student models.5.2.4 M2: Distill Knowledge via Label AdjustmentThe subsequent module M2 adjusts the labels of Mg fake samples in D˜g gener-ated from the previous module M1 via a pre-trained teacher model ft . Similar to154pseudo-labeling [3, 71] in semi-supervised learning, the label adjustment is con-ducted by replacing the i-th assigned label y˜gi in D˜g with the predicted label ft(x˜gi ).Please note that in classification, the predicted labels are hard labels as we de-scribed in Fig. 1.6. This adjustment distills the knowledge about the relation be-tween an image and its label from the trained ft and stores it in the adjusted datasetD̂g. Moreover, cGANs especially CcGANs may suffer from substantial label in-consistency, i.e., a fake image’s assigned label may diverge from its ground truthlabel. The label adjustment can alleviate this issue.5.2.5 M3: Transfer Knowledge via Data AugmentationThe adjusted samples in D̂g are also called the processed fake samples. They areused to augment the original training set Dr, i.e., to give Dr∪D̂g. To transfer knowl-edge distilled from the pre-trained ft , we train fs on the augmented dataset in M3.Please note that empirical studies in Section 5.5 show that as Mg increases, the testerror of fs often does not stop decreasing until Mg is larger than a certain thresholdand then starts fluctuating over a small range. Since it is hard to obtain the opti-mal Mg in practice and a hefty Mg usually does not cause significant adverse effecton precision, we suggest generating the maximum number of processed samplesallowed by the computational budget.Note that M3 makes our method fundamentally different from existing KDmethods [53, 92, 118, 159] because the distilled knowledge is transferred throughsamples instead of specially designed loss functions or network architectures. M3is also distinct from existing GAN-based data augmentation methods [35, 88, 131,153, 169], because they do not have the subsampling, filtering, and label adjustmentsteps.5.3 Theoretical AnlaysisIn this section, we derive the error bound of the student model fs, which theo-retically illustrates how the teacher model ft improves the precision of fs in thecGAN-KD framework. Before we move to the derivation, we first introduce somenotations. Denote by pg(x,y) PDF for the distribution of unprocessed fake sam-ples. Denote by p˜g(x,y) and pˆg(x,y) PDFs for the distributions of fake samples155after processing by M1 and M2 respectively. The evolution of fake samples’ dis-tributions and datasets is visualized in Fig. 5.2. Additionally, we denote the aug-mented training dataset as Daug, i.e., Daug := Dr ∪ D̂g. Then, we define theoreticaland empirical losses,V ( f ) := E(x,y)∼pr(x,y) [L ( f (x),y)] ,V̂ ( f ) :=1Nr +Mg ∑(xi,yi)∈DaugL ( f (xi),yi),whereL is either the CE loss for classification or the SE loss for regression. With-out loss of generality, we assume y ∈ [0,1] in regression tasks. Let f ∗ be theoptimal predictor which minimizes V ( f ). We denote by Fs the hypothesis spaceof fs. Note thatFs may not include f ∗. Then, we define f ◦s and fˆs asf ◦s = arg minfs∈FsV ( fs), fˆs = arg minfs∈FsV̂ ( fs).Figure 5.2: Evolution of fake sample datasets and their distributions.The error bound of fs in cGAN-KD is depicted by the distance of fs from f ∗,which is described in Theorem 5.3.1.Theorem 5.3.1 (Error Bound). Suppose thatA1. (i.i.d. samples) Dr i.i.d.∼ pr(x,y), D̂g i.i.d.∼ pˆg(x,y), and the augmented datasetis considered as i.i.d. samples from a mixture distribution, i.e.,Daugi.i.d.∼ θ pr(x,y)+(1−θ)pˆg(x,y) =: pθ (x,y), (5.1)where θ ∈ [0,1].156A2. (Measurability) fs is measurable for all fs ∈Fs.A3. (Distribution gap) There is a constant CM1 > 0 such thatTV (pr, pˆg) =CM1+Θ(E(x,y)∼pr(x,y) [L ( ft(x),y)]), (5.2)where TV denotes the total variation distance [40] between two probabilitydistributions p,q; and f (x) = Θ(g(x)) means f (x) = O(g(x)) and g(x) =O( f (x)).A4. (Boundedness) There exists a constant 0 < CL < +∞, such that ∀(x,y),L ( fs(x),y)≤CL .Then, ∀δ ∈ (0,1), with probability at least 1−δ ,V ( fˆs)−V ( f ∗)≤4CL R̂Nr+Mg(Fs)+2CL√4Nr +Mglog(2δ)+4CL (1−θ)(CM1+Θ(E(x,y)∼pr(x,y) [L ( ft(x),y)]))+(V ( f ◦s )−V ( f ∗)) , (5.3)where R̂Nr+Mg(Fs) stands for the empirical Rademacher complexity [96, Defini-tion 3.1] of Fs, which is defined on Nr +Mg samples independently drawn frompθ .Proof. We first decompose V ( fˆs)−V ( f ∗) as followsV ( fˆs)−V ( f ∗)=V ( fˆs)− V̂ ( fˆs)+ V̂ ( fˆs)− V̂ ( f ◦s )+ V̂ ( f ◦s )−V ( f ◦s )+V ( f ◦s )−V ( f ∗)(by V̂ ( fˆs)− V̂ ( f ◦s )≤ 0)≤V ( fˆs)− V̂ ( fˆs)+ V̂ ( f ◦s )−V ( f ◦s )+V ( f ◦s )−V ( f ∗)≤2 supfs∈Fs∣∣∣V̂ ( fs)−V ( fs)∣∣∣+V ( f ◦s )−V ( f ∗). (5.4)The second term V ( f ◦s )−V ( f ∗) in Eq. (5.4) is a non-negative number becausethe student model’s hypothesis space Fs may not cover the optimal predictor f ∗.157sup fs∈Fs∣∣∣V̂ ( fs)−V ( fs)∣∣∣ in the first term of Eq. (5.4) can be bounded as follows.Using the triangular inequality and A4 (i.e., boundedness ofL ) yields thatsupfs∈Fs∣∣∣V̂ ( fs)−V ( fs)∣∣∣= supfs∈Fs∣∣∣∣∣ 1Nr +Mg ∑(xi,yi)∈DaugL ( fs(xi),yi)−E(x,y)∼pr(x,y) [L ( fs(x),y)]∣∣∣∣∣(by the triangular inequality)≤ supfs∈Fs∣∣∣∣∣ 1Nr +Mg ∑(xi,yi)∈DaugL ( fs(xi),yi)−E(x,y)∼pθ (x,y) [L ( fs(x),y)]∣∣∣∣∣+ supfs∈Fs∣∣E(x,y)∼pθ (x,y) [L ( fs(x),y)]−E(x,y)∼pr(x,y) [L ( fs(x),y)]∣∣(by the Boundedness assumption A4)=CL supfs∈Fs∣∣∣∣∣ 1Nr +Mg ∑(xi,yi)∈Daug1CLL ( fs(xi),yi)−E(x,y)∼pθ (x,y)[1CLL ( fs(x),y)]∣∣∣∣∣ (5.5)+ supfs∈Fs∣∣E(x,y)∼pθ (x,y) [L ( fs(x),y)]−E(x,y)∼pr(x,y) [L ( fs(x),y)]∣∣ . (5.6)For Eq. (5.5), we apply the Rademacher bound (Theorem 3.3.2), yielding that withat least probability 1−δ ,CL supfs∈Fs∣∣∣∣∣ 1Nr +Mg ∑(xi,yi)∈Daug1CLL ( fs(xi),yi)−E(x,y)∼pθ (x,y)[1CLL ( fs(x),y)]∣∣∣∣∣≤2CL R̂Nr+Mg(Fs)+CL√4Nr +Mglog(2δ). (5.7)Before we bound Eq. (5.6), we first review the definition of the total variation158distance [40] between any two distributions P and Q, i.e.,TV (P,Q) :=12sup|g|≤1∣∣∣∣∫ gdP−∫ gdQ∣∣∣∣ .where g is a measurable function. Thus,TV (pr,θ pr +(1−θ)pˆg)=12sup|g|≤1∣∣E(x,y)∼pθ (x,y) [g(x,y)]−E(x,y)∼pr(x,y) [g(x,y)]∣∣=12sup|g|≤1{(1−θ)∣∣∣E(x,y)∼pˆg(x,y) [g(x,y)]−E(x,y)∼pr(x,y) [g(x,y)]∣∣∣}=(1−θ)TV (pr, pˆg). (5.8)Since fs is measurable (by A2) and L is continuous, L ( fs(x),y) is also measur-able. Let 1CLL ( fs(x),y) be g(x,y) in Eq. (5.8), then by A3 (i.e., the distributiongap between pr and pˆg), we havesupfs∈Fs∣∣E(x,y)∼pθ (x,y) [L ( fs(x),y)]−E(x,y)∼pr(x,y) [L ( fs(x),y)]∣∣=2CL (1−θ)(CM1+Θ(E(x,y)∼pr(x,y) [L ( ft(x),y)])). (5.9)Combining Eqs. (5.7) and (5.9), we can getsupfs∈Fs∣∣∣V̂ ( fs)−V ( fs)∣∣∣≤2CL R̂Nr+Mg(Fs)+CL√4Nr +Mglog(2δ)+2CL (1−θ)(CM1+Θ(E(x,y)∼pr(x,y) [L ( ft(x),y)])). (5.10)Finally, incorporating Eq. (5.10) into Eq. (5.4), we can get the inequality (i.e., Eq.(5.3)) in Theorem 5.3.1, which completes the proof.Remark 5.3.1 (Rationale for A3 and A4). In the cGAN-KD framework, processed159fake images are used to augment the training set, so the distribution gap betweenpr(x,y) and pˆg(x,y) (measured by the total variation distance) should have a signif-icant impact on the student model’s performance. Thus, in A3 of Theorem 5.3.1,we model the distribution gap by the summation of two components. The firstcomponent CM1 stands for the divergence caused by the trained cGAN and thesubsampling and filtering steps. The second component is controlled by the gen-eralization performance of ft—the expected loss of the trained teacher model overthe true data distribution.It is also worth discussing the rationale for A4. The two types of learningtasks considered in this work are the regression and classification, for which weuse the square loss ( fs(x)− y)2 and the cross entropy loss ∑Cc=1{−yc log psc} (Cis the number of classes, yc is the c-th element of the one-hot encoded true classlabel, and psc is the predicted probability that the input image belonging to the c-th class) respectively. Take the regression task first. In our experiments on theregression datasets, the last layer of fs(x) is the ReLU activation function [37, 38],so fs(x) ≥ 0. Since y ∈ (0,1), as long as fs(x) is not an unstable predictor, itshould not output arbitrarily large values, which implies fs(x) can be boundedby a positive constant. Therefore, the square loss is bounded and A4 is satisfied.For the classification task, a sufficient condition to A4 is that psc ≥ ε > 0 whenyc = 1, representing that our classifier cannot produce 0 probability for the truelabel, which is reasonable in practice.Remark 5.3.2 (Illustration of Theorem 5.3.1). The four terms on the right side ofEq. (5.3) show that the error of fs has four components. The first and last termsare only relevant to the nature of fs, so they are not influenced by ft . If fs does notoutput arbitrarily extreme predictions (as discussed in Remark 5.3.1), CL stays ata moderate level, implying the first term is also small. The last term is inevitablebecause Fs may not include f ∗. The second term diminishes if we set Mg large.For the third term, θ → 0 as Mg increases. Then, the third term is only controlledby the properties of L and the distribution gap. To reduce the distribution gap,we can either improve the cGAN model, subsampling, and filtering or choose ft tohave better generalization performance.Therefore, Theorem 5.3.1 implies when implementing cGAN-KD we should160(1) use state-of-the-art cGANs and subsampling methods, (2) set Mg large, and (3)choose a ft with highest precision as possible.5.4 Advantages of cGAN-KDCompared with existing KD methods, our proposed cGAN-KD framework hasmany advantages, which are summarized in this section.5.4.1 A Unified Knowledge Distillation Framework for ImageClassification and RegressionSince all necessary steps in the workflow of cGAN-KD are applicable to bothclassification and regression (scalar response), cGAN-KD is actually a unified KDframework. Moreover, the theoretical analysis of cGAN-KD (see Section 5.3) alsohas the same general formulation for both tasks.5.4.2 Compatibility with State-of-the-art KD MethodscGAN-KD distills and transfers knowledge via fake samples, and it does not re-quire extra loss functions or network architecture changes. Thus, cGAN-KD canbe combined with many state-of-the-art KD methods for image classification toimprove their performance. To embed a state-of-the-art KD method into cGAN-KD, we just need to train the student model on the augmented training set with thisKD method in M3 but keep other procedures in Fig. 5.1 unchanged.5.4.3 Suitability for Limited Labeled Data ScenariosThe performance of cGANs heavily deteriorates given a limited amount of labeledtraining data. Fortunately, DiffAugment [166] can effectively alleviate this prob-lem. Besides DiffAugment, the subsampling and filtering modules in the cGAN-KD framework can further mitigate this issue by removing low-quality fake sam-ples. Therefore, cGAN-KD is very suitable for scenarios with limited labeled data.1615.4.4 Architecture-invarianceAs shown by our experiments in Section 5.5 and some papers [92, 159], the archi-tecture differences between a teacher model and a student model may influence theperformance of some existing KD methods because these methods rely on logitsor intermediate layers to transfer knowledge. Other KD methods such as SSKD[159] even require some adjustments to the teacher and student models’ architec-tures. Conversely, since the proposed cGAN-KD framework distills and transfersknowledge via fake samples, there are no restrictions on the teacher and studentmodels’ architectures, making cGAN-KD more flexible than other KD methods.5.5 ExperimentsThis section aims to experimentally demonstrate the effectiveness of the proposedcGAN-KD framework in image classification and regression (scalar response) taskswhen limited training data are available. We conduct extensive experiments on fourimage datasets: CIFAR-10 [66] and Tiny-ImageNet [68] for image classification;and RC-49 [30, 33] and UTKFace [164] for image regression. Candidate baselineKD methods for classification tasks are NOKD (i.e., no KD method is applied),BLKD [53], TAKD [92], SSKD [159], BLKD+UDA [118], the proposed cGAN-KD framework (including incorporating other KD methods into cGAN-KD). Forregression tasks, candidate methods only include NOKD and cGAN-KD. Pleasenote that for image regression, as suggested by [30, 33], when training CcGANs,ft , and fs, regression labels are normalized to real numbers in [0,1]. Nevertheless,in the evaluation stage of ft and fs, we compute mean absolute error (MAE) onunnormalized regression labels. For detailed experimental setups, please refer tothe supplementary material.5.5.1 CIFAR-10We first evaluate the effectiveness of the proposed cGAN-KD framework on theCIFAR-10 dataset [66].Experimental setup: CIFAR-10 consists of 60,000 RGB images of resolution32× 32 uniformly spread across 10 classes. The overall number of training sam-ples is 50,000 (5000 for each class), and the remaining 10,000 samples (1000 for162each class) are for testing. To compare our proposed method with existing KDmethods when few labeled data are available, we design three settings denoted re-spectively by C-50K, C-20K, and C-10K with different numbers of training sam-ples. Specifically, all 50,000 training samples are available in C-50K. C-20K has20,000 randomly selected training samples (about 2000 per class). In C-10K, thesize of training samples is further reduced to 10,000 (about 1000 per class).Next, to select student and teacher models for this experiment, some popularclassifiers are trained from scratch in each setting, and their test errors are shownin Table A.25 in the supplementary material. We choose three teacher models (i.e.,MobileNet V2 [122], ResNet-18 [51], and DenseNet-121 [55]) with similarly highprecision and three student models (i.e., ShuffleNet V2 [87], efficientnet-b0 [135],and VGG11 [130]) with similarly low precision, based on their performances here.Note that although MobileNet V2 is a popular lightweight model, its performanceis surprisingly good and comparable to other two teachers on CIFAR-10. There-fore, MobileNet V2 is chosen as a teacher model in our experiment. To implementBLKD and TAKD, we set λKD = 0.5 and T = 5 following [118]. VGG13 is chosenas the TA model in TAKD. When implementing SSKD, MobileNet V2 performedpoorly after the necessary architecture adjustment. Additionally, efficientnet-b0and DenseNet-121 are not supported by the official implementation of SSKD, thuswe only consider one teacher model (ResNet-18) and two student models (VGG11and ShuffleNet V2). To implement the proposed framework, we train one Big-GAN model [14] for each setting. DiffAugment [166] is also incorporated into theBigGAN training in C-20K and C-10K due to the availability of limited trainingsamples. In all three settings, no matter the teacher-student combination, we useDenseNet-121 to do the filtering and label adjustment due to its highest averageprecision. We set Mg = 105 and the optimal ρ in Alg. 12 selected by Alg. 13 are0.9, 0.6, and 0.7 for the three settings respectively. In the experiments with SSKDand BLKD+UDA, we only consider the combination of cGAN-KD with these twoKD methods due to limited computational resources.An ablation study is designed to test the effectiveness of the subsampling, fil-tering, and label adjustment modules in the cGAN-KD framework in C-10K withMg = 105, aiming to show how cGAN-KD performs if these three modules areadded into the framework one by one. A second ablation study is conducted in the163C-10K setting to analyze the effect of Mg, where Mg varies from 0 to 1.5×105.Please refer to Supp. A.15 for more detailed setups.Quantitative results: The quantitative comparison results of the main study areshown in Tables 5.1, 5.2, 5.3, and 5.4 among different baseline methods. Pleasenote that since SSKD needs to modify network architectures, and BLKD+UDA in-corporates UDA into teacher training (the teacher model’s precision may change),their performances are uncomparable. For the same reason, it is also inappropri-ate to compare them with BLKD and TAKD. Therefore, these existing KD meth-ods can be classified into three groups: (1) BLKD and TAKD, (2) SSKD, and (3)BLKD+UDA. We compare the proposed framework with them separately in Ta-bles 5.1 - 5.3. In Table 5.1, we can see cGAN-KD related methods consistentlyoutperform NOKD, BLKD [53], and TAKD [92] under all three settings and allteacher-student combinations. BLKD [53] and TAKD [92] methods are improvedafter incorporation into the proposed cGAN-KD framework. We also observe thatcGAN-KD leads to higher performance gains when there are fewer training sam-ples. Tables 5.2 and 5.3 show that cGAN-KD can effectively improve the per-formance of SSKD [159] and BLKD+UDA [118]. In Table 5.4, we show the testerror rates of the student in each setting-student combination before and after ap-plying the best KD method. The best KD method minimizes the test error ratesin the cells of Tables 5.1 to 5.3 corresponding to the given setting-student pair.We can see that the cGAN-KD framework achieves the best performance in allsetting-student combinations.Additionally, we conduct ablation studies to evaluate the effect of our proposedM1, M2, M3, and the parameter Mg. The quantitative results of our ablation stud-ies are visualized respectively in Figs. 5.3 and 5.4. Fig. 5.3 reveals the necessity ofthe subsampling, filtering, and label adjustment modules because their interactionresults in the highest precision. Fig. 5.4 shows that more processed fake imagesstabilize the student models’ performance without significantly decreasing the pre-cision, which confirms the necessity of a large Mg.164Table 5.1: CIFAR-10: Test error rate comparisons among NOKD, BLKD[53], TAKD [92], and our methods (Mg = 105 and ρ = 0.9,0.6,0.7 forthree settings respectively). We always use DenseNet-121 to do the filter-ing and label adjustment for different teacher-student combinations. Thetest error rate is defined as the proportion of incorrectly classified testsamples. NOKD implies no KD method is applied. The test error ratesof teachers are shown in parentheses. Our methods outperform NOKD,BLKD, and TAKD in all settings, and this advantage is more significantwhen training data are scarcer. In almost all scenarios, BLKD and TAKDare improved after incorporation into the proposed cGAN-KD framework(i.e., cGAN-KD+BLKD and cGAN-KD+TAKD).Settings Teachers Students NOKD BLKD TAKD cGAN-KD cGAN-KD+ BLKDcGAN-KD+ TAKDC-10KMobileNet V2(13.53)VGG11 18.57 15.76 15.98 14.32 12.65 12.28ShuffleNet V2 17.50 14.03 14.96 13.90 12.53 12.69efficientnet-b0 15.71 14.49 15.40 13.48 12.48 12.71ResNet-18(16.07)VGG11 18.57 16.78 16.42 14.32 14.45 14.55ShuffleNet V2 17.50 15.54 15.27 13.90 14.29 14.26efficientnet-b0 15.71 14.99 15.68 13.48 14.79 13.89DenseNet-121(14.63)VGG11 18.57 15.95 16.43 14.32 13.81 13.62ShuffleNet V2 17.50 14.81 14.86 13.90 13.66 13.80efficientnet-b0 15.71 14.80 17.47 13.48 13.38 13.50C-20KMobileNet V2(9.28)VGG11 12.48 10.94 11.48 10.81 9.60 9.39ShuffleNet V2 11.33 10.10 10.01 11.00 9.63 9.19efficientnet-b0 12.46 10.11 9.99 11.02 9.28 9.16ResNet-18(9.26)VGG11 12.48 11.29 11.56 10.81 9.58 9.88ShuffleNet V2 11.33 10.28 10.39 11.00 9.83 9.86efficientnet-b0 12.46 10.55 10.38 11.02 10.10 9.75DenseNet-121(8.52)VGG11 12.48 10.88 11.73 10.81 9.63 9.70ShuffleNet V2 11.33 10.36 10.23 11.00 9.97 9.94efficientnet-b0 12.46 10.28 9.86 11.02 10.03 9.75C-50KMobileNet V2(5.92)VGG11 8.42 6.72 7.35 6.83 5.89 6.23ShuffleNet V2 7.18 6.30 6.56 6.84 6.11 5.88efficientnet-b0 7.10 6.45 6.42 6.85 6.11 6.05ResNet-18(4.88)VGG11 8.42 7.24 7.43 6.83 6.24 6.41ShuffleNet V2 7.18 6.44 6.69 6.84 6.34 6.25efficientnet-b0 7.10 6.72 6.59 6.85 6.21 6.29DenseNet-121(4.47)VGG11 8.42 7.40 7.27 6.83 6.37 6.51ShuffleNet V2 7.18 6.73 6.70 6.84 6.57 6.81efficientnet-b0 7.10 6.34 6.75 6.85 5.59 6.35165Table 5.2: CIFAR-10: Test error rate comparisons between SSKD [159] andcGAN-KD+SSKD (Mg = 105 and ρ = 0.9,0.6,0.7 for three settings re-spectively). We always use DenseNet-121 in Table 5.1 to do the filteringand label adjustment. In most scenarios, SSKD is improved after incor-poration into the cGAN-KD framework. The test error rates of teachersare shown in parentheses.Settings Teachers Students NOKD SSKDcGAN-KD+ SSKDC-10K ResNet-18 (11.87)VGG11 16.05 12.60 12.33ShuffleNet V2 18.19 12.30 12.40C-20K ResNet-18 (8.05)VGG11 11.17 8.80 8.53ShuffleNet V2 11.42 8.41 8.31C-50K ResNet-18 (5.06)VGG11 7.59 5.82 5.63ShuffleNet V2 7.03 5.38 5.275.5.2 Tiny-ImageNetThis experiment further demonstrates the effectiveness of cGAN-KD on the Tiny-ImageNet dataset [68].Experimental setup: Tiny-ImageNet contains 200 image classes with 500 imagesper class for training and 50 images per class for testing. Most images are RGBimages of size 64× 64, and a few grey-scale images are excluded in this experi-ment.Based on the test errors of some popular networks (refer to Fig. A.27), wechoose two teacher models (ResNet-50 [51] and DenseNet-121 [55]) and two stu-dent models (ShuffleNet V2 [87] and VGG11 [130]). When implementing TAKD[92], MobileNet V2 [122] is chosen as the TA model because its performanceis around the average of the performance of teachers and students. To generatefake samples, we adopt the BigGAN model [14] and DiffAugment [166]. SinceDenseNet-121 performs the best in this experiment, it is used for the filtering andlabel adjustment. Other experimental setups are similar to those of the CIFAR-10experiment except that all training images are used. Please refer to Supp. A.16 formore details.Quantitative results: The quantitative results are shown in Tables 5.5, 5.6, 5.7,and 5.8. Similar to the CIFAR-10 experiment, cGAN-KD is very effective and it166Table 5.3: CIFAR-10: Test error rate comparisons between BLKD+UDA[118] and cGAN-KD+BLKD+UDA (Mg = 105 and ρ = 0.9,0.6,0.7 forthree settings respectively). We always use DenseNet-121 in Table 5.1to do the filtering and label adjustment. In most scenarios (25 out of27), BLKD+UDA are improved after incorporation into the cGAN-KDframework, and this advantage is more significant when training data arescarcer. The test error rates of teachers are shown in parentheses.Settings Teachers Students NOKDBLKD+ UDAcGAN-KD+ BLKD+ UDAC-10KMobileNet V2 (13.43)VGG11 18.61 15.14 12.54ShuffleNet V2 16.76 12.97 12.23efficientnet-b0 22.45 14.46 12.92ResNet-18 (14.18)VGG11 18.61 14.79 13.40ShuffleNet V2 16.76 13.51 13.21efficientnet-b0 22.45 15.37 13.92DenseNet-121 (13.4)VGG11 18.61 14.86 12.80ShuffleNet V2 16.76 13.13 12.25efficientnet-b0 22.45 14.93 12.97C-20KMobileNet V2 (9.25)VGG11 12.28 9.57 9.19ShuffleNet V2 10.72 8.54 8.40efficientnet-b0 13.27 9.82 9.81ResNet-18 (9.87)VGG11 12.28 10.04 9.30ShuffleNet V2 10.72 9.09 9.04efficientnet-b0 13.27 10.15 9.85DenseNet-121 (8.54)VGG11 12.28 10.21 8.87ShuffleNet V2 10.72 9.64 8.60efficientnet-b0 13.27 11.56 9.32C-50KMobileNet V2 (5.40)VGG11 8.18 6.18 5.68ShuffleNet V2 7.11 5.12 5.49efficientnet-b0 8.62 6.32 6.18ResNet-18 (5.28)VGG11 8.18 6.17 5.63ShuffleNet V2 7.11 5.56 5.01efficientnet-b0 8.62 6.78 6.38DenseNet-121 (4.55)VGG11 8.18 6.56 5.32ShuffleNet V2 7.11 5.26 4.94efficientnet-b0 8.62 6.26 6.29167Table 5.4: CIFAR-10: Test error rates of the student in each setting-studentcombination before and after applying the best KD method. The bestKD method minimizes the test error rates in the cells of Tables 5.1 to 5.3corresponding to the given setting-student pair. The teachers adopted bythese KD methods are shown in parentheses. All best KD methods arethe proposed cGAN-KD framework. Note that SSKD modifies students’architectures and BLKD+UDA applies UDA in the student training, sothe test error rate before applying KD locates in an interval.Settings Students Best KD Methods (Teachers)Test Error RatesBefore KD After KDC-10KVGG11 cGAN-KD+TAKD (MobileNet V2) 16.05∼18.61 12.28ShuffleNet V2 cGAN-KD+BLKD+UDA (MobileNet V2) 16.76∼18.19 12.23efficientnet-b0 cGAN-KD+BLKD (MobileNet V2) 15.71∼22.45 12.48C-20KVGG11 cGAN-KD+SSKD (ResNet-18) 11.17∼12.48 8.53ShuffleNet V2 cGAN-KD+SSKD (ResNet-18) 10.72∼11.42 8.31efficientnet-b0 cGAN-KD+TAKD (MobileNet V2) 12.46∼13.27 9.16C-50KVGG11 cGAN-KD+BLKD+UDA (DenseNet-121) 7.59∼8.42 5.32ShuffleNet V2 cGAN-KD+BLKD+UDA (DenseNet-121) 7.03∼7.18 4.94efficientnet-b0 cGAN-KD+BLKD (DenseNet-121) 7.10∼8.62 5.59improves state-of-the-art KD methods in all scenarios.Table 5.5: Tiny-ImageNet: Test error rate comparisons among NOKD,BLKD [53], TAKD [92], and our methods (Mg = 105 and ρ = 0.5). Weuse DenseNet-121 to do the filtering and label adjustment no matter theteacher-student combination. The test error rate is defined as the propor-tion of incorrectly classified test samples. NOKD implies no KD methodis applied. The test error rates of teachers are shown in parentheses.Our methods outperform NOKD, BLKD, and TAKD in all settings. Inall scenarios, BLKD and TAKD are improved after incorporation into thecGAN-KD framework (i.e., cGAN-KD+BLKD and cGAN-KD+TAKD).Teachers Students NOKD BLKD TAKD cGAN-KD cGAN-KD+ BLKDcGAN-KD+ TAKDResNet-50(35.86)ShuffleNet V2 44.52 43.78 43.80 42.99 43.39 43.05VGG11 44.13 42.57 41.91 41.55 39.69 39.46DenseNet-121(35.22)ShuffleNet V2 44.52 43.97 44.03 42.99 43.52 43.67VGG11 44.13 42.22 42.03 41.55 38.95 39.49168Figure 5.3: CIFAR-10: An ablation study of the subsampling, filtering, andlabel adjustment modules in the C-10K setting. The interaction of thesubsampling, filtering, and label adjustment modules leads to the lowesterror rates for all three student models. Compared with filtering, labeladjustment does not make significant enhancement here because it isvery likely that most unprocessed fake images have correct assignedclass labels in this experiment with only 10 classes.Table 5.6: Tiny-ImageNet: Test error rate comparisons between SSKD [159]and cGAN-KD+SSKD (Mg = 105 and ρ = 0.5). We use DenseNet-121 inTable 5.5 to do the filtering and label adjustment. In both two scenarios,SSKD are improved after incorporation into the cGAN-KD framework.The test error rates of teachers are shown in parentheses.Teacher Students NOKD SSKDcGAN-KD+ SSKDResNet-50(37.6)ShuffleNet V2 46.92 39.44 38.82VGG11 43.05 36.70 36.37169Figure 5.4: CIFAR-10: Analysis on the effect of number of processed fakeimages (i.e., Mg) in the C-10K setting. As Mg is large enough, all threemodels’ test error rates stop decreasing and start fluctuating in a smallrange.Table 5.7: Tiny-ImageNet: Test error rate comparisons betweenBLKD+UDA [118] and cGAN-KD+BLKD+UDA (Mg = 105 andρ = 0.5). We use DenseNet-121 in Table 5.5 to do the filtering andlabel adjustment. In all scenarios, BLKD+UDA are improved afterincorporation into the cGAN-KD framework. The test error rates ofteachers are shown in parentheses.Teachers Students NOKDBLKD+ UDAcGAN-KD+ BLKD+ UDAResNet-50(36.07)ShuffleNet V2 43.06 37.47 36.16VGG11 45.67 35.27 34.89DenseNet-121(34.59)ShuffleNet V2 43.06 37.29 35.89VGG11 45.67 34.51 34.02170Table 5.8: Tiny-ImageNet: Test error rates of each student before and afterapplying the best KD method. The best KD method minimizes the testerror rates in the cells of Tables 5.5 to 5.7 corresponding to the givenstudent. The teachers adopted by both KD methods are shown in paren-theses. The proposed cGAN-KD framework achieves the best KD per-formance for both students. Note that SSKD modifies students’ architec-tures and BLKD+UDA applies UDA in the student training, so the testerror rate before applying KD locates in an interval.StudentsBest KD Methods(Teachers)Test Error RatesBefore KD After KDShuffleNet V2cGAN-KD+BLKD+UDA(DenseNet-121)43.06∼46.92 35.89VGG11cGAN-KD+BLKD+UDA(DenseNet-121)43.05∼45.67 34.025.5.3 RC-49This experiment is conducted on RC-49 [30, 33] to show that cGAN-KD also per-forms well in the image regression tasks with a scalar response variable.Experimental setup: The RC-49 dataset is made by rendering 49 3-D chair mod-els individually. Each chair model is rendered at 899 yaw angles from 0.1◦ to 89.9◦with a stepsize of 0.1◦. A yaw angle is selected for training if its last digit is odd,so only 450 angles are in the training set while the others are left for testing. Thisdataset contains 44,051 RGB images of size 64× 64 with corresponding yaw an-gles as labels. In this experiment, we design three settings denoted respectivelyby R-25, R-15, and R-5. The numbers in the setting names specify the number ofimages for each distinct angle in the training set. For example, in the R-25 setting,each of the 450 angles in the training set has 25 images, so there are 11,250 imagesavailable for training. In all three settings, all images not used for training are heldout for testing.Three student models (ShuffleNet V2 [87], MobileNet V2 [122], and efficientnet-b0 [135]) and one teacher model (VGG16 [130]) are selected in this experimentbased on the performance of some popular networks (refer to Table A.29). Sinceno general KD method exists for image regression tasks with a scalar response, we171only compare cGAN-KD with NOKD. For cGAN-KD, we adopt the SNGAN ar-chitecture [95] and train one CcGAN model (SVDL+ILI) [30] with DiffAugment[166] for each setting. We use VGG16 to do the filtering and label adjustment.Alg. 13 is applied to select the optimal ρ in the filtering module. 50,000 processedfake samples are generated to augment the training set in each setting. Detailedexperimental setups can be found in Supp. A.17.Similar to the CIFAR-10 experiment, an ablation study is conducted to showthe necessity of the subsampling, filtering, and label adjustment modules. Anotherablation study is conducted to show the effect of Mg.Quantitative results: The quantitative results are shown in Table 5.9. We can seethat cGAN-KD outperforms NOKD by a large margin and the performance en-hancement is even more significant if we have fewer training samples. The quan-titative results of the two ablation studies are visualized in Figs. 5.5 and 5.6. Theconclusion from these ablation studies is consistent with that from the CIFAR-10experiment, except that the label adjustment module’s effect is more substantial inregression than in classification. One explanation of this difference is that the labelinconsistency issue of CcGANs for regression is more severe than that of BigGANfor classification, and the label adjustment module can effectively alleviate thisproblem.5.5.4 UTKFaceThe last experiment evaluates the performance of cGAN-KD on UTKFace [164],another benchmark regression dataset.Experimental setup: UTKFace is an RGB human face image dataset with ages asregression labels. We use the processed UTKFace dataset [30, 33], which consistsof 14,760 RGB images with ages in [1, 60]. The number of images ranges from 50to 1051 for different ages, and all images are of size 64×64. Among these images,80% are randomly selected to create a training set, and the rest are held out fortesting.Similar to the RC-49 experiment, three student models (ShuffleNet V2 [87],MobileNet V2 [122], and efficientnet-b0 [135]) and one teacher model (VGG16[130]) are selected in this experiment based on the performance of some popular172Table 5.9: RC-49: Test MAE comparisons between NOKD and cGAN-KD(Mg = 5×104 and ρ = 0.5,0.8,0.9 for three settings respectively). TheMAE is evaluated on all samples that are not used for training. NOKDimplies no KD method is applied. The MAE of teachers are shown inparentheses. cGAN-KD outperforms NOKD with a large margin in allsettings, and this advantage is more substantial when training data arescarcer.Settings Teachers Students NOKD cGAN-KDR-5 VGG16 (0.49)ShuffleNet V2 1.95 0.95MobileNet V2 1.74 1.18efficientnet-b0 1.86 1.15R-15 VGG16 (0.31)ShuffleNet V2 0.76 0.40MobileNet V2 0.95 0.48efficientnet-b0 1.00 0.69R-25 VGG16 (0.20)ShuffleNet V2 0.52 0.34MobileNet V2 0.55 0.39efficientnet-b0 0.99 0.57networks (please refer to Table A.31). For cGAN-KD, we adopt the SAGAN ar-chitecture [163] and DiffAugment [166] in the CcGAN (SVDL+ILI) training [30].We apply VGG16 to conduct the filtering and label adjustment and the optimal ρis selected by following Alg. 13. 80,000 processed fake samples are generated toaugment the training set. Please refer to Supp. A.18 for detailed setups.Quantitative results: The quantitative results of this experiment are summarizedin Table 5.10. We observe that cGAN-KD performs substantially better than NOKD.Please also note that in the scenario with efficientnet-b0 as the student model, thetest MAE of cGAN-KD is even smaller than that of the teacher model. This phe-nomenon implies the CcGAN model may generate some human faces which arequite different from those in the training set. In other words, the CcGAN modelmay synthesize some new information.5.6 ConclusionAs the first attempt, this chapter proposes a unified knowledge distillation frame-work widely applicable for both classification and regression tasks even in the lim-173Figure 5.5: RC-49: An ablation study of the subsampling, filtering, and labeladjustment modules in the R-5 setting. The interaction of the subsam-pling, filtering, and label adjustment modules leads to the lowest MAEfor all three student models.Table 5.10: UTKFace: Test MAE comparisons between NOKD and cGAN-KD (Mg = 8× 104 and ρ = 0.7). NOKD implies no KD method isapplied. The MAE of teachers are shown in parentheses. cGAN-KDoutperforms NOKD with a large margin in all settings.Teachers Students NOKD cGAN-KDVGG16 (5.14)ShuffleNet V2 7.326 5.774MobileNet V2 7.335 5.507efficientnet-b0 6.143 4.766174Figure 5.6: RC-49: Analysis on the effect of number of processed fake im-ages (i.e., Mg) in the R-5 setting. As Mg is large enough, all three mod-els’ test error rates stop decreasing and start fluctuating in a small range.ited data scenario based on CcGANs and the subsampling methods proposed inChapters 2 and 4. Fundamentally different from existing knowledge distillationmethods, we propose distilling and transferring knowledge from the teacher modelto the student model through cGAN-generated samples, termed cGAN-KD. First,cGAN models are trained to generate a sufficient number of fake image samples.Then, high quality samples are obtained via subsampling and filtering procedures.Essentially, the knowledge is distilled by adjusting fake image labels utilizing theteacher model. Finally, the distilled knowledge is transferred to student models bytraining them on these knowledge-conveyed samples. The proposed novel knowl-edge distillation framework is particularly effective when labeled training data arelimited. Moreover, the proposed framework is architecture-agnostic and it is com-patible with existing state-of-the-art knowledge distillation models. We also derivethe error bound of a student model trained in the cGAN framework for theoreti-cal guidance. Extensive experiments demonstrate that the cGAN-KD incorporated175methods can achieve state-of-the-art knowledge distillation performances for bothclassification and regression tasks.176Chapter 6Conclusion and Future Work6.1 ConclusionAlthough cGANs are state-of-the-art generative models for image synthesis, theystill fail in some scenarios due to their weaknesses. This work is intended to im-prove cGANs by resolving these weaknesses.First, in Chapter 2, we propose CcGAN, the first cGAN framework that cangenerate images conditional on regression labels (e.g., ages and angles). In thischapter, we propose novel empirical cGANs losses and label input mechanisms,which are suitable for the continuous scenarios. The effectiveness and superiorityof CcGAN are demonstrated on four regression datasets, i.e., RC-49, UTKFace,Cell-200, and Steering Angle.Next, in Chapter 3, we propose DRE-F-SP to estimate images’ density ratios ina feature space learned by a specially designed and pretrained ResNet-34. Basedon DRE-F-SP and some standard samplers (e.g., RS, MH, and SIR), we introducea subsampling framework to filter out low-quality images generated from uncondi-tional GANs.In Chapter 4, we introduce cDRE-F-cSP+RS which extends the subsamplingframework in Chapter 3 to the conditional image synthesis setting. cDRE-F-cSP+RScan subsample both class-conditional GANs and CcGANs.Finally, in Chapter 5, we propose a novel KD framework called cGAN-KDthat distills and transfers knowledge from a teacher model to a student model via177cGAN-generated samples. This framework is suitable for both image classificationand regression (scalar response) tasks. CcGAN proposed in Chapter 2 is used togenerate fake samples in the regression tasks. cDRE-F-cSP+RS proposed in 4 isused to drop low-quality cGAN-generated samples. The superiority of cGAN-KDover existing KD methods is demonstrated on various datasets.6.2 Future WorkIn this thesis, CcGAN can only generate 64× 64 images. Thus, the first possiblefuture work is to improve the current CcGAN model to generate high-resolutionimages (e.g., 512×512 images). To achieve this goal, the network architectures ofBigGAN [14] may be adopted, and some state-of-the-art training techniques usedby BigGAN are also needed.Furthermore, the performance of CcGAN on highly imbalanced datasets (e.g.,the Steering Angle dataset) is not as good as that on balanced datasets (e.g., RC-49). Therefore, another future work is introducing a method to alleviate such dataimbalance in the CcGAN training. An intuitive idea is using adaptive hard/softvicinity. For example, if we have sufficient images with labels close to a regressionlabel y, then we should use a narrow hard vicinity for y; otherwise, we shouldchoose a wider one.Moreover, extending CcGAN to scenarios with multivariate continuous condi-tions is also an important direction. For example, in 3-D hand pose estimation [5],hand images are often labelled by 3-D coordinates of 21 hand keypoints (e.g., Fig.6.1). Thus, the label for each hand image is a vector with a length of 63. If CcGANis properly extended to this scenario, we can generate many hand images with thesame keypoints but different backgrounds and skin textures. Afterward, we canpropose a new knowledge distillation method for hand pose estimation based onthese synthetic hand images.Last but not least, exploring more applications of fake images generated fromconditional GANs is also necessary. In this thesis, we only research the applica-tion of cGAN-generated images in classification and regression tasks. Meanwhile,other visual recognition tasks (e.g., object detection, segmentation, tracking, etc.)may also benefit from cGAN-generated images.178Figure 6.1: Example hand images with keypoint annotations from [129].179Bibliography[1] Codes for ”generative adversarial nets”. https://github.com/goodfeli/adversarial/blob/master/cifar10 fully connected.yaml. → page 4[2] Z. Akata, F. Perronnin, Z. Harchaoui, and C. Schmid. Label-embedding forimage classification. IEEE transactions on pattern analysis and machineintelligence, 38(7):1425–1438, 2015. → pages 8, 199, 202[3] E. Arazo, D. Ortego, P. Albert, N. E. O’Connor, and K. McGuinness.Pseudo-labeling and confirmation bias in deep semi-supervised learning. In2020 International Joint Conference on Neural Networks (IJCNN), pages1–8, 2020. → page 155[4] M. Arjovsky, S. Chintala, and L. Bottou. Wasserstein generativeadversarial networks. In Proceedings of the 34th International Conferenceon Machine Learning, volume 70, pages 214–223, 2017. → pages2, 3, 10, 29, 57, 78, 79[5] A. Armagan, G. Garcia-Hernando, S. Baek, S. Hampali, M. Rad, Z. Zhang,S. Xie, M. Chen, B. Zhang, F. Xiong, et al. Measuring generalisation tounseen viewpoints, articulations, shapes and objects for 3d hand poseestimation under hand-object interaction. In Proceedings of the EuropeanConference on Computer Vision (ECCV), 2020. → page 178[6] H. Arnelid, E. L. Zec, and N. Mohammadiha. Recurrent conditionalgenerative adversarial networks for autonomous driving sensor modelling.In 2019 IEEE Intelligent Transportation Systems Conference (ITSC), pages1613–1618, 2019. → page 1[7] S. Arora, R. Ge, Y. Liang, T. Ma, and Y. Zhang. Generalization andequilibrium in generative adversarial nets (GANs). In InternationalConference on Machine Learning, pages 224–232, 2017. → page 7180[8] R. Atienza. A conditional generative adversarial network for renderingpoint clouds. In Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition Workshops, pages 10–17, 2019. → page 1[9] S. E. Awan, M. Bennamoun, F. Sohel, F. M. Sanfilippo, and G. Dwivedi.Imputation of missing data with class imbalance using conditionalgenerative adversarial networks. arXiv preprint arXiv:2012.00220, 2020.→ page 1[10] S. Azadi, C. Olsson, T. Darrell, I. Goodfellow, and A. Odena.Discriminator rejection sampling. In International Conference on LearningRepresentations, 2019. → pages3, 12, 13, 14, 23, 79, 80, 88, 91, 92, 94, 95, 96, 97, 99, 102, 104, 108, 116, 131[11] D. Berthelot, T. Schumm, and L. Metz. Began: Boundary equilibriumgenerative adversarial networks. arXiv preprint arXiv:1703.10717, 2017.→ page 7[12] M. Bolic, P. M. Djuric, and S. Hong. Resampling algorithms andarchitectures for distributed particle filters. IEEE Transactions on SignalProcessing, 53(7):2442–2450, 2005. → page 81[13] L. Bregman. The relaxation method of finding the common point of convexsets and its application to the solution of problems in convex programming.USSR Computational Mathematics and Mathematical Physics, 7(3):200 –217, 1967. → page 81[14] A. Brock, J. Donahue, and K. Simonyan. Large scale GAN training forhigh fidelity natural image synthesis. In International Conference onLearning Representations, 2019. → pagesxix, 1, 2, 3, 7, 8, 9, 10, 12, 13, 26, 34, 78, 119, 130, 149, 150, 152, 163, 166, 178[15] C. Buciluaˇ, R. Caruana, and A. Niculescu-Mizil. Model compression. InProceedings of the 12th ACM SIGKDD international conference onKnowledge discovery and data mining, pages 535–541, 2006. → pages18, 148[16] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li,S. Savarese, M. Savva, S. Song, H. Su, et al. Shapenet: An information-rich3d model repository. arXiv preprint arXiv:1512.03012, 2015. → page 202[17] C.-C. Chang, C. Hubert Lin, C.-R. Lee, D.-C. Juan, W. Wei, and H.-T.Chen. Escaping from collapsing modes in a constrained space. In181Proceedings of the European Conference on Computer Vision (ECCV),pages 204–219, 2018. → page 57[18] O. Chapelle, J. Weston, L. Bottou, and V. Vapnik. Vicinal riskminimization. In Advances in neural information processing systems,pages 416–422, 2001. → page 27[19] T. Che, R. Zhang, J. Sohl-Dickstein, H. Larochelle, L. Paull, Y. Cao, andY. Bengio. Your GAN is secretly an energy-based model and you shoulduse discriminator driven latent sampling. arXiv preprint arXiv:2003.06060,2020. → page 131[20] S. Chen. The Steering Angle dataset @ONLINE.https://github.com/SullyChen/driving-datasets, 2018. → pages52, 56, 216, 218[21] S. Chen. How a high school junior made a self-driving car? @ONLINE.https://towardsdatascience.com/how-a-high-school-junior-made-a-self-driving-car-705fa9b6e860, 2018. → pages 52, 216, 218[22] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework forcontrastive learning of visual representations. In International conferenceon machine learning, pages 1597–1607, 2020. → page 20[23] J. Cheng, L. Dong, and M. Lapata. Long short-term memory-networks formachine reading. In Proceedings of the 2016 Conference on EmpiricalMethods in Natural Language Processing, pages 551–561, 2016. → page 9[24] A. Coates, A. Ng, and H. Lee. An analysis of single-layer networks inunsupervised feature learning. In G. Gordon, D. Dunson, and M. Dudı´k,editors, Proceedings of the Fourteenth International Conference onArtificial Intelligence and Statistics, volume 15 of Proceedings of MachineLearning Research, pages 215–223, 2011. → pages 221, 222[25] U. J. Dang, M. P. Gallaugher, R. P. Browne, and P. D. McNicholas.Model-based clustering and classification using mixtures of multivariateskewed power exponential distributions. arXiv preprint arXiv:1907.01938,2019. → page 18[26] R. A. Davis, K.-S. Lii, and D. N. Politis. Remarks on some nonparametricestimates of a density function. In Selected Works of Murray Rosenblatt,pages 95–100. Springer, 2011. → pages 31, 34182[27] H. De Vries, F. Strub, J. Mary, H. Larochelle, O. Pietquin, and A. C.Courville. Modulating early visual processing by language. In Advances inNeural Information Processing Systems, pages 6594–6604, 2017. → pages37, 202[28] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: Alarge-scale hierarchical image database. In CVPR09, 2009. → pages11, 234[29] T. DeVries, A. Romero, L. Pineda, G. W. Taylor, and M. Drozdzal. On theevaluation of conditional GANs. arXiv preprint arXiv:1907.08175, 2019.→ pages 12, 39, 120[30] X. Ding, Y. Wang, Z. Xu, W. J. Welch, and Z. J. Wang. Continuousconditional generative adversarial networks for image generation: Novellosses and label input mechanisms. arXiv preprint arXiv:2011.07466,2020. → pages119, 122, 129, 141, 144, 149, 150, 152, 162, 171, 172, 173, 238, 240, 248[31] X. Ding, Z. J. Wang, and W. J. Welch. Subsampling generative adversarialnetworks: Density ratio estimation in feature space with Softplus loss.IEEE Transactions on Signal Processing, 68:1910–1922, 2020. → pages130, 131, 132, 144, 149, 235, 236[32] X. Ding, Y. Wang, Z. J. Wang, and W. J. Welch. Efficient subsampling forgenerating high-quality images from conditional generative adversarialnetworks. arXiv preprint arXiv:2103.11166, 2021. → pages149, 153, 243, 248[33] X. Ding, Y. Wang, Z. Xu, W. J. Welch, and Z. J. Wang. CcGAN:Continuous conditional generative adversarial networks for imagegeneration. In International Conference on Learning Representations,2021. URL https://openreview.net/forum?id=PrzjugOsDeE. → pages119, 120, 129, 141, 144, 149, 150, 152, 162, 171, 172, 238[34] M. Fabbri, G. Borghi, F. Lanzi, R. Vezzani, S. Calderara, and R. Cucchiara.Domain translation with conditional GANs: from depth to RGBface-to-face. In 2018 24th International Conference on PatternRecognition (ICPR), pages 1355–1360, 2018. → page 1[35] M. Frid-Adar, E. Klang, M. Amitai, J. Goldberger, and H. Greenspan.Synthetic data augmentation using GAN for improved liver lesion183classification. In 2018 IEEE 15th international symposium on biomedicalimaging (ISBI 2018), pages 289–293, 2018. → pages 1, 149, 151, 155[36] J. Friedman, T. Hastie, R. Tibshirani, et al. The elements of statisticallearning, volume 1. Springer series in statistics New York, 2001. → page18[37] K. Fukushima. Visual feature extraction by a multilayered network ofanalog threshold elements. IEEE Transactions on Systems Science andCybernetics, 5(4):322–333, 1969. → page 160[38] K. Fukushima and S. Miyake. Neocognitron: A self-organizing neuralnetwork model for a mechanism of visual pattern recognition. InCompetition and cooperation in neural nets, pages 267–285. 1982. → page160[39] Y. Gao, Y. Liu, Y. Wang, Z. Shi, and J. Yu. A universal intensitystandardization method based on a many-to-one weak-paired cyclegenerative adversarial network for magnetic resonance images. IEEETransactions on Medical Imaging, 38(9):2059–2069, 2019. → page 78[40] A. L. Gibbs and F. E. Su. On choosing and bounding probability metrics.International statistical review, 70(3):419–435, 2002. → pages 157, 159[41] A. C. Gilbert, Y. Zhang, K. Lee, Y. Zhang, and H. Lee. Towardsunderstanding the invertibility of convolutional neural networks. InProceedings of the 26th International Joint Conference on ArtificialIntelligence, pages 1703–1710, 2017. → page 89[42] X. Glorot, A. Bordes, and Y. Bengio. Deep sparse rectifier neural networks.In Proceedings of the fourteenth international conference on artificialintelligence and statistics, pages 315–323, 2011. → page 122[43] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley,S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. InAdvances in Neural Information Processing Systems 27, pages 2672–2680.2014. → pages 1, 2, 3, 4, 5, 6, 10, 78, 79, 149[44] I. J. Goodfellow, D. Warde-Farley, M. Mirza, A. Courville, and Y. Bengio.Maxout networks. In Proceedings of the 30th International Conference onMachine Learning, volume 28, pages III–1319, 2013. → page 5[45] J. Gou, B. Yu, S. J. Maybank, and D. Tao. Knowledge distillation: Asurvey. arXiv preprint arXiv:2006.05525, 2020. → pages 18, 148, 151184[46] A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Scho¨lkopf, and A. Smola. Akernel two-sample test. Journal of Machine Learning Research, 13(Mar):723–773, 2012. → page 78[47] A. Grover, J. Song, A. Kapoor, K. Tran, A. Agarwal, E. J. Horvitz, andS. Ermon. Bias correction of learned generative models usinglikelihood-free importance weighting. In Advances in Neural InformationProcessing Systems, pages 11058–11070, 2019. → pagesxvii, 17, 79, 80, 91, 92, 102, 106, 231[48] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville.Improved training of Wasserstein GANs. In Advances in NeuralInformation Processing Systems 30, pages 5767–5777. 2017. → pages2, 3, 10, 29, 78, 79, 99[49] W. Harvey, S. Naderiparizi, and F. Wood. Image completion via inferencein deep generative models. arXiv preprint arXiv:2102.12037, 2021. →page 1[50] T. Hastie, R. Tibshirani, and J. Friedman. The elements of statisticallearning: data mining, inference, and prediction. Springer Science &Business Media, 2009. → pages 31, 34, 37[51] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for imagerecognition. In Proceedings of the IEEE conference on computer visionand pattern recognition, pages 770–778, 2016. → pages44, 89, 99, 163, 166, 203[52] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter.GANs trained by a two time-scale update rule converge to a local nashequilibrium. In Advances in Neural Information Processing Systems, pages6626–6637, 2017. → pages 7, 10, 11, 12, 39, 44, 103, 129, 204, 236, 240[53] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neuralnetwork. NIPS Deep Learning Workshop, 2015. → pagesxvi, xvii, xix, 18, 19, 21, 148, 151, 155, 162, 164, 165, 168[54] C.-C. Hsu, C.-W. Lin, W.-T. Su, and G. Cheung. SiGAN: Siamesegenerative adversarial network for identity-preserving face hallucination.IEEE Transactions on Image Processing, 28(12):6225–6236, 2019. →page 78185[55] G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Denselyconnected convolutional networks. In Proceedings of the IEEE conferenceon computer vision and pattern recognition, pages 4700–4708, 2017. →pages 17, 147, 163, 166[56] H. Huang, C. Wang, P. S. Yu, and C.-D. Wang. Generative dual adversarialnetwork for generalized zero-shot learning. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, pages 801–810,2019. → page 1[57] S. Ioffe and C. Szegedy. Batch normalization: accelerating deep networktraining by reducing internal covariate shift. In International Conferenceon Machine Learning, pages 448–456, 2015. → page 221[58] R. I. Jennrich. Asymptotic properties of non-linear least squares estimators.The Annals of Mathematical Statistics, 40(2):633–643, 1969. → page 87[59] T. Karras, S. Laine, and T. Aila. A style-based generator architecture forgenerative adversarial networks. In Proceedings of the IEEE conference oncomputer vision and pattern recognition, pages 4401–4410, 2019. → pages2, 3, 119, 149[60] T. Karras, M. Aittala, J. Hellsten, S. Laine, J. Lehtinen, and T. Aila.Training generative adversarial networks with limited data. arXiv preprintarXiv:2006.06676, 2020. → page 27[61] T. Karras, S. Laine, M. Aittala, J. Hellsten, J. Lehtinen, and T. Aila.Analyzing and improving the image quality of StyleGAN. In Proceedingsof the IEEE/CVF Conference on Computer Vision and Pattern Recognition,pages 8110–8119, 2020. → pages 2, 3, 119, 149[62] M. Karson. Handbook of methods of applied statistics. volume i:Techniques of computation descriptive methods, and statistical inference.,1968. → page 84[63] H. Khan, L. Marcuse, and B. Yener. Deep density ratio estimation forchange point detection. arXiv preprint arXiv:1905.09876, 2019. → pagesxvii, 15, 16, 17, 79, 88, 91, 92, 95, 98, 102, 103, 106, 107, 231[64] D. P. Kingma and J. Ba. Adam: a method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014. → pages 4, 222, 235, 237186[65] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. InY. Bengio and Y. LeCun, editors, 3rd International Conference on LearningRepresentations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015,Conference Track Proceedings, 2015. → pages 199, 203, 209, 212, 217[66] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features fromtiny images. Technical report, Citeseer, 2009. → pages94, 99, 127, 129, 130, 162[67] J. Lafferty, H. Liu, and L. Wasserman. Concentration of measure. URLhttp://www.stat.cmu.edu/∼larry/=sml/Concentration.pdf. → pages 85, 126[68] Y. Le and X. Yang. Tiny ImageNet visual recognition challenge. 2015. →pages 120, 129, 131, 132, 162, 166[69] Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learningapplied to document recognition. Proceedings of the IEEE, 86(11):2278–2324, 1998. → pages 94, 106[70] C. Ledig, L. Theis, F. Husza´r, J. Caballero, A. Cunningham, A. Acosta,A. Aitken, A. Tejani, J. Totz, Z. Wang, et al. Photo-realistic single imagesuper-resolution using a generative adversarial network. In 2017 IEEEConference on Computer Vision and Pattern Recognition (CVPR), pages105–114, 2017. → page 1[71] D.-H. Lee et al. Pseudo-label: The simple and efficient semi-supervisedlearning method for deep neural networks. In Workshop on challenges inrepresentation learning, ICML, volume 3, 2013. → page 155[72] A. Lehmussola, P. Ruusuvuori, J. Selinummi, H. Huttunen, andO. Yli-Harja. Computational framework for simulating fluorescencemicroscope images with cell populations. IEEE transactions on medicalimaging, 26(7):1010–1016, 2007. → page 52[73] A. Lehmussola, P. Ruusuvuori, J. Selinummi, H. Huttunen, andO. Yli-Harja. Computational framework for simulating fluorescencemicroscope images with cell populations. IEEE transactions on medicalimaging, 26(7):1010–1016, 2007. → page 211[74] V. Lempitsky and A. Zisserman. Learning to count objects in images. InAdvances in neural information processing systems, pages 1324–1332,2010. → page 211187[75] B. Li, X. Qi, T. Lukasiewicz, and P. Torr. Controllable text-to-imagegeneration. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alche´-Buc,E. Fox, and R. Garnett, editors, Advances in Neural Information ProcessingSystems, volume 32, 2019. → page 1[76] C.-L. Li, W.-C. Chang, Y. Cheng, Y. Yang, and B. Poczos. MMD GAN:Towards deeper understanding of moment matching network. In Advancesin Neural Information Processing Systems 30, pages 2203–2213. 2017. →pages 2, 3, 10, 78, 79, 99, 224[77] H. Li, L. Zhang, X. Zhang, M. Zhang, G. Zhu, P. Shen, P. Li,M. Bennamoun, and S. A. A. Shah. Color vision deficiency datasets &recoloring evaluation using gans. Multimedia Tools and Applications, 79(37):27583–27614, 2020. → page 1[78] Q. Li, S. Jin, and J. Yan. Mimicking very efficient network for objectdetection. In Proceedings of the IEEE conference on computer vision andpattern recognition, pages 6356–6364, 2017. → page 148[79] T. Li, J. Li, Z. Liu, and C. Zhang. Few sample knowledge distillation forefficient network compression. In Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition, pages14639–14647, 2020. → page 18[80] Z. Li and D. Hoiem. Learning without forgetting. IEEE transactions onpattern analysis and machine intelligence, 40(12):2935–2947, 2017. →page 148[81] P. Liu, W. Liu, H. Ma, Z. Jiang, and M. Seok. KTAN: Knowledge transferadversarial network. In 2020 International Joint Conference on NeuralNetworks (IJCNN), pages 1–7, 2020. → pages 149, 151[82] R. Liu, X. Wang, H. Lu, Z. Wu, Q. Fan, S. Li, and X. Jin. SCCGAN: Styleand characters inpainting based on CGAN. Mobile Networks andApplications, 26(1):3–12, 2021. → page 1[83] Y. Liu, P. Kothari, and A. Alahi. Collaborative sampling in generativeadversarial networks. In Proceedings of the AAAI Conference on ArtificialIntelligence, volume 34, pages 4948–4956, 2020. → page 131[84] Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in thewild. In Proceedings of International Conference on Computer Vision(ICCV), December 2015. → pages 94, 110188[85] P. Luo, Z. Zhu, Z. Liu, X. Wang, and X. Tang. Face model compression bydistilling knowledge from neurons. In Proceedings of the AAAI Conferenceon Artificial Intelligence, volume 30, 2016. → page 148[86] Z. Luo, J.-T. Hsieh, L. Jiang, J. C. Niebles, and L. Fei-Fei. Graphdistillation for action detection with privileged modalities. In Proceedingsof the European Conference on Computer Vision (ECCV), pages 166–183,2018. → page 148[87] N. Ma, X. Zhang, H.-T. Zheng, and J. Sun. Shufflenet V2: Practicalguidelines for efficient cnn architecture design. In Proceedings of theEuropean conference on computer vision (ECCV), pages 116–131, 2018.→ pages 163, 166, 171, 172[88] G. Mariani, F. Scheidegger, R. Istrate, C. Bekas, and C. Malossi. BAGAN:Data augmentation with balancing GAN. arXiv preprint arXiv:1803.09655,2018. → pages 1, 149, 151, 155[89] E. Maskin. Nash equilibrium and welfare optimality. The Review ofEconomic Studies, 66(1):23–38, 1999. → page 7[90] S. Milz, M. Simon, K. Fischer, M. Po¨pperl, and H.-M. Gross. Points2pix:3d point-cloud to image translation using conditional gans. In GermanConference on Pattern Recognition, pages 387–400, 2019. → page 1[91] M. Mirza and S. Osindero. Conditional generative adversarial nets. arXivpreprint arXiv:1411.1784, 2014. → pages3, 7, 8, 26, 27, 29, 40, 41, 119, 149, 199[92] S. I. Mirzadeh, M. Farajtabar, A. Li, N. Levine, A. Matsukawa, andH. Ghasemzadeh. Improved knowledge distillation via teacher assistant. InProceedings of the AAAI Conference on Artificial Intelligence, volume 34,pages 5191–5198, 2020. → pagesxvi, xvii, xix, 18, 20, 21, 148, 151, 155, 162, 164, 165, 166, 168, 242[93] A. Mittal, R. Soundararajan, and A. C. Bovik. Making a “completelyblind” image quality analyzer. IEEE Signal processing letters, 20(3):209–212, 2012. → pages 44, 129, 204, 218, 240, 243, 247, 248, 251[94] T. Miyato and M. Koyama. cGANs with projection discriminator. InInternational Conference on Learning Representations, 2018. → pages7, 8, 10, 12, 26, 27, 28, 35, 37, 39, 44, 119, 129, 149, 202, 204, 236, 240189[95] T. Miyato, T. Kataoka, M. Koyama, and Y. Yoshida. Spectral normalizationfor generative adversarial networks. In International Conference onLearning Representations, 2018. → pages1, 3, 7, 8, 10, 29, 34, 40, 78, 110, 138, 144, 149, 172, 202, 234, 248[96] M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of machinelearning. MIT Press, 2018. → pages 26, 85, 157[97] A. Munk, W. Harvey, and F. Wood. Assisting the adversary to improve gantraining. arXiv preprint arXiv:2010.01274, 2020. → page 2[98] H. Nam and M. Sugiyama. Direct density ratio estimation withconvolutional neural networks with application in outlier detection. IEICETransactions on Information and Systems, 98(5):1073–1079, 2015. →pages xvii, 15, 17, 79, 88, 91, 92, 95, 98, 102, 103, 106, 107, 230[99] S. Nowozin, B. Cseke, and R. Tomioka. f-GAN: Training generative neuralsamplers using variational divergence minimization. In Advances in NeuralInformation Processing Systems 29, pages 271–279. 2016. → page 3[100] A. Odena, C. Olah, and J. Shlens. Conditional image synthesis withauxiliary classifier gans. In International conference on machine learning,pages 2642–2651, 2017. → pages 7, 8, 26, 27, 119, 149[101] G. Olmschenk. Semi-supervised Regression with Generative AdversarialNetworks Using Minimal Labeled Data. PhD thesis, 2019. → page 27[102] G. Olmschenk, J. Chen, H. Tang, and Z. Zhu. Dense crowd countingconvolutional neural networks with minimal data using semi-superviseddual-goal generative adversarial networks. In IEEE Conference onComputer Vision and Pattern Recognition: Learning with Imperfect DataWorkshop, 2019. → page 27[103] M. J. Osborne and A. Rubinstein. A course in game theory. MIT press,1994. → page 7[104] A. Parikh, O. Ta¨ckstro¨m, D. Das, and J. Uszkoreit. A decomposableattention model for natural language inference. In Proceedings of the 2016Conference on Empirical Methods in Natural Language Processing, pages2249–2255, 2016. → page 9[105] E. Parzen. On estimation of a probability density function and mode. Theannals of mathematical statistics, 33(3):1065–1076, 1962. → pages 31, 34190[106] D. Pathak, P. Kra¨henbu¨hl, J. Donahue, T. Darrell, and A. Efros. Contextencoders: Feature learning by inpainting. In Computer Vision and PatternRecognition (CVPR), 2016. → page 1[107] Z. Peng, Z. Li, J. Zhang, Y. Li, G.-J. Qi, and J. Tang. Few-shot imagerecognition with knowledge transfer. In Proceedings of the IEEE/CVFInternational Conference on Computer Vision, pages 441–449, 2019. →page 148[108] G. Perarnau, J. van de Weijer, B. Raducanu, and J. M. A´lvarez. Invertibleconditional GANs for image editing. In NIPS Workshop on AdversarialTraining, 2016. → page 1[109] G. Peyre´, M. Cuturi, et al. Computational optimal transport. Foundationsand Trends® in Machine Learning, 11(5-6):355–607, 2019. → page 42[110] A. Punzo, M. Blostein, and P. D. McNicholas. High-dimensionalunsupervised classification via parsimonious contaminated mixtures.Pattern Recognition, 98:107031, 2020. → page 18[111] T. M. Quan, T. Nguyen-Duc, and W.-K. Jeong. Compressed sensing MRIreconstruction using a generative adversarial network with a cyclic loss.IEEE Transactions on Medical Imaging, 37(6):1488–1497, 2018. → page78[112] A. Radford, L. Metz, and S. Chintala. Unsupervised representationlearning with deep convolutional generative adversarial networks. arXivpreprint arXiv:1511.06434, 2015. → pages 2, 9, 40, 99, 211[113] S. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele, and H. Lee.Generative adversarial text to image synthesis. In International Conferenceon Machine Learning, pages 1060–1069, 2016. → page 1[114] D. A. Reynolds. Gaussian mixture models. Encyclopedia of biometrics,741, 2009. → page 98[115] M. Rezagholiradeh and M. A. Haidar. Reg-GAN: Semi-supervisedlearning based on generative adversarial networks for regression. In 2018IEEE International Conference on Acoustics, Speech and SignalProcessing (ICASSP), pages 2806–2810, 2018. → page 27[116] M. Rezagholizadeh, M. A. Haidar, and D. Wu. Semi-supervised regressionwith generative adversarial networks, Nov. 22 2018. US Patent App.15/789,518. → page 27191[117] C. P. Robert, G. Casella, and G. Casella. Introducing Monte Carlo methodswith R, volume 18. 2010. → page 81[118] F. Ruffy and K. Chahal. The state of knowledge distillation forclassification. arXiv preprint arXiv:1912.10850, 2019. → pagesxvi, xvii, 18, 21, 148, 155, 162, 163, 164, 167, 170[119] L. Ruthotto and E. Haber. An introduction to deep generative modeling.arXiv preprint arXiv:2103.05180, 2021. → page 1[120] L. Salasnich. Quantum Physics of Light and Matter: A ModernIntroduction to Photons, Atoms and Many-Body Systems. Springer, 2014.→ pages 30, 34[121] T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, andX. Chen. Improved techniques for training GANs. In Advances in neuralinformation processing systems, pages 2234–2242, 2016. → pages3, 5, 10, 103[122] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen.MobileNet V2: Inverted residuals and linear bottlenecks. In Proceedings ofthe IEEE conference on computer vision and pattern recognition, pages4510–4520, 2018. → pages 163, 166, 171, 172[123] S. Shalev-Shwartz and S. Ben-David. Understanding machine learning:from theory to algorithms. Cambridge University Press, 2014. → pages26, 65[124] S. Shehnepoor, R. Togneri, W. Liu, and M. Bennamoun. ScoreGAN: Afraud review detector based on multi task learning of regulated GAN withdata augmentation. arXiv preprint arXiv:2006.06561, 2020. → page 1[125] Z. Shen, Z. He, and X. Xue. Meal: Multi-model ensemble via adversariallearning. In Proceedings of the AAAI Conference on Artificial Intelligence,volume 33, pages 4886–4893, 2019. → pages 18, 149, 151[126] X. Shi. Some useful asymptotic theory. URL https://www.ssc.wisc.edu/∼xshi/econ715/Lecture 2 some asymptotic theorems.pdf. → pages 87, 126[127] F. C. Silva, P. A. L. de Castro, H. R. Ju´nior, and E. C. Marujo. Mangan:Assisting colorization of manga characters concept art using conditionalGAN. In 2019 IEEE International Conference on Image Processing(ICIP), pages 3257–3261, 2019. → page 1192[128] B. W. Silverman. Density estimation for statistics and data analysis,volume 26. CRC press, 1986. → pages 31, 34[129] T. Simon, H. Joo, I. Matthews, and Y. Sheikh. Hand keypoint detection insingle images using multiview bootstrapping. In Proceedings of the IEEEconference on Computer Vision and Pattern Recognition, pages1145–1153, 2017. → pages xxiv, 179[130] K. Simonyan and A. Zisserman. Very deep convolutional networks forlarge-scale image recognition. arXiv preprint arXiv:1409.1556, 2014. →pages 17, 128, 147, 163, 166, 171, 172[131] L. Sixt, B. Wild, and T. Landgraf. RenderGAN: Generating realistic labeleddata. Frontiers in Robotics and AI, 5:66, 2018. → pages 1, 149, 151, 155[132] A. Srivastava, L. Valkov, C. Russell, M. U. Gutmann, and C. Sutton.VEEGAN: Reducing mode collapse in GANs using implicit variationallearning. In Advances in Neural Information Processing Systems, pages3308–3318, 2017. → page 57[133] M. Sugiyama, T. Suzuki, and T. Kanamori. Density ratio estimation inmachine learning. 1st edition, 2012. ISBN 0521190177, 9780521190176.→ page 81[134] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinkingthe Inception architecture for computer vision. In Proceedings of the IEEEconference on computer vision and pattern recognition, pages 2818–2826,2016. → pages 10, 11[135] M. Tan and Q. Le. Efficientnet: Rethinking model scaling for convolutionalneural networks. In International Conference on Machine Learning, pages6105–6114, 2019. → pages 163, 171, 172[136] B. Tong, M. Klinkigt, J. Chen, X. Cui, Q. Kong, T. Murakami, andY. Kobayashi. Adversarial zero-shot learning with semantic augmentation.In Thirty-Second AAAI Conference on Artificial Intelligence, 2018. → page1[137] N.-T. Tran, V.-H. Tran, N.-B. Nguyen, T.-K. Nguyen, and N.-M. Cheung.On data augmentation for GAN training, 2020. → page 27[138] R. Turner, J. Hung, E. Frank, Y. Saatchi, and J. Yosinski.Metropolis-Hastings generative adversarial networks. In Proceedings of the19336th International Con-ference on Machine Learning, volume 97, pages 6345–6353, 2019.→ pages3, 12, 14, 23, 79, 80, 88, 91, 92, 94, 95, 96, 97, 99, 102, 103, 104, 108, 116, 131[139] M. Uehara, I. Sato, M. Suzuki, K. Nakayama, and Y. Matsuo. Generativeadversarial nets from a density ratio estimation perspective. arXiv preprintarXiv:1610.02920, 2016. → pages 86, 88[140] T. Van Erven and P. Harremos. Re´nyi divergence and Kullback-Leiblerdivergence. IEEE Transactions on Information Theory, 60(7):3797–3820,2014. → page 84[141] V. Vapnik. The nature of statistical learning theory. Springer, 2nd edition,2000. → pages 26, 27[142] K. R. Varshney. Bayes risk error is a Bregman divergence. IEEETransactions on Signal Processing, 59(9):4470–4472, 2011. → page 81[143] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez,Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Proceedings ofthe 31st International Conference on Neural Information ProcessingSystems, pages 6000–6010, 2017. → page 9[144] C. Villani. Optimal transport: old and new, volume 338. 2008. → page 78[145] C. Wang, C. Xu, C. Wang, and D. Tao. Perceptual adversarial networks forimage-to-image transformation. IEEE Transactions on Image Processing,27(8):4066–4079, 2018. → page 78[146] J. Wang, L. Gou, W. Zhang, H. Yang, and H.-W. Shen. Deepvid: Deepvisual interpretation and diagnosis for image classifiers via knowledgedistillation. IEEE transactions on visualization and computer graphics, 25(6):2168–2180, 2019. → page 148[147] L. Wang and K.-J. Yoon. Knowledge distillation and student-teacherlearning for visual intelligence: A review and new outlooks. IEEETransactions on Pattern Analysis and Machine Intelligence, 2021. → pages18, 148, 151[148] Q. Wang, H. Fan, L. Zhu, and Y. Tang. Deeply supervised face completionwith multi-context generative adversarial network. IEEE Signal ProcessingLetters, 26(3):400–404, 2018. → page 78194[149] T.-C. Wang, M.-Y. Liu, J.-Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro.High-resolution image synthesis and semantic manipulation withconditional GANs. In Proceedings of the IEEE conference on computervision and pattern recognition, pages 8798–8807, 2018. → page 1[150] X. Wang, R. Zhang, Y. Sun, and J. Qi. KDGAN: Knowledge distillationwith generative adversarial networks. In NeurIPS, pages 783–794, 2018.→ pages 18, 149, 151[151] L. Wasserman. Density estimation @ONLINE.http://www.stat.cmu.edu/∼larry/=sml/densityestimation.pdf. → page 71[152] J. Wei, G. Lu, H. Liu, and J. Yan. Facial image inpainting with deepgenerative model and patch search using region weight. IEEE Access, 7:67456–67468, 2019. → page 78[153] E. Wu, K. Wu, D. Cox, and W. Lotter. Conditional infilling GANs for dataaugmentation in mammogram classification. In Image Analysis for MovingOrgan, Breast, and Thoracic Images, pages 98–106. 2018. → pages1, 149, 151, 155[154] Y. Wu and K. He. Group normalization. In Proceedings of the EuropeanConference on Computer Vision (ECCV), pages 3–19, 2018. → page 221[155] Y. Xia, N. Ravikumar, J. P. Greenwood, S. Neubauer, S. E. Petersen, andA. F. Frangi. Super-resolution of cardiac MR cine imaging usingconditional GANs and unsupervised transfer learning. Medical ImageAnalysis, page 102037, 2021. → page 1[156] Y. Xian, T. Lorenz, B. Schiele, and Z. Akata. Feature generating networksfor zero-shot learning. In 2018 IEEE/CVF Conference on Computer Visionand Pattern Recognition, pages 5542–5551, 2018. → page 1[157] P. Xiang, L. Wang, J. Cheng, B. Zhang, and J. Wu. A deep networkarchitecture for image inpainting. In 2017 3rd IEEE InternationalConference on Computer and Communications (ICCC), pages 1851–1856,2017. → page 1[158] Q. Xie, Z. Dai, E. Hovy, T. Luong, and Q. Le. Unsupervised dataaugmentation for consistency training. In Advances in Neural InformationProcessing Systems, volume 33, pages 6256–6268, 2020. → pages 21, 148195[159] G. Xu, Z. Liu, X. Li, and C. C. Loy. Knowledge distillation meetsself-supervision. In European Conference on Computer Vision, pages588–604, 2020. → pagesxvi, xvii, 18, 20, 148, 151, 155, 162, 164, 166, 169[160] Z. Xu, Y.-C. Hsu, and J. Huang. Training shallow and thin networks foracceleration via knowledge distillation with conditional adversarialnetworks. ICLR 2018 Workshop, 2018. → pages 18, 149, 151[161] Z. Ye, F. Lyu, L. Li, Q. Fu, J. Ren, and F. Hu. SR-GAN: Semanticrectifying generative adversarial network for zero-shot learning. In 2019IEEE International Conference on Multimedia and Expo (ICME), pages85–90, 2019. → page 1[162] C. Zhang and Y. Peng. Better and faster: knowledge transfer from multipleself-supervised learning tasks via graph distillation for video classification.In Proceedings of the 27th International Joint Conference on ArtificialIntelligence, pages 1135–1141, 2018. → page 148[163] H. Zhang, I. Goodfellow, D. Metaxas, and A. Odena. Self-attentiongenerative adversarial networks. In International Conference on MachineLearning, pages 7354–7363, 2019. → pages2, 3, 7, 8, 12, 26, 34, 39, 78, 119, 149, 173[164] Z. Zhang, Y. Song, and H. Qi. Age progression/regression by conditionaladversarial autoencoder. In Proceedings of the IEEE conference oncomputer vision and pattern recognition, pages 5810–5818, 2017. → pages45, 129, 141, 162, 172, 206[165] Q. Zhao, J. Dong, H. Yu, and S. Chen. Distilling ordinal relation and darkknowledge for facial age estimation. IEEE Transactions on NeuralNetworks and Learning Systems, 2020. → pages 18, 149, 151[166] S. Zhao, Z. Liu, J. Lin, J.-Y. Zhu, and S. Han. Differentiable augmentationfor data-efficient GAN training. Advances in Neural InformationProcessing Systems, 33, 2020. → pages3, 27, 149, 150, 152, 161, 163, 166, 172, 173, 237[167] Z. Zhao, Z. Zhang, T. Chen, S. Singh, and H. Zhang. Image augmentationsfor GAN training, 2020. → page 27[168] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-imagetranslation using cycle-consistent adversarial networks. In Proceedings of196the IEEE international conference on computer vision, pages 2223–2232,2017. → page 1[169] X. Zhu, Y. Liu, J. Li, T. Wan, and Z. Qin. Emotion classification with dataaugmentation using generative adversarial networks. In Pacific-AsiaConference on Knowledge Discovery and Data Mining, pages 349–360,2018. → pages 1, 149, 151, 155197Appendix ASupporting MaterialsA.1 GitHub RepositoriesThe codes for the experiments in this thesis can be found in the following GitHubrepositories:• CcGAN in Chapter 2:https://github.com/UBCDingXin/improved CcGAN• DRE-F-SP+RS/MH/SIR in Chapter 3:https://github.com/UBCDingXin/DDRE Sampling GANs• cDRE-F-cSP+RS in Chapter 4:https://github.com/UBCDingXin/cDRE-based Subsampling cGANS• cGAN-KD in Chapter 5:https://github.com/UBCDingXin/cGAN-KD198A.2 More Details of the Circular 2-D GaussiansExperiment in Section 2.4.1A.2.1 Network ArchitecturesPlease refer to Table A.1 and Table A.2 for the network architectures we adoptedfor cGAN and CcGAN in our Simulation experiments.Please note that when input labels into CcGAN, we first convert y to a pointon the circle (i.e., (sin(y),cos(y))) and concatenate the 2-D coordinate of this pointwith z or x. When input labels into cGAN, we use the hidden concatenation scheme[91].Table A.1: Network architectures for the generator and discriminator ofcGAN in the simulation. “fc” denotes a fully-connected layer. “BN”stands for batch normalization. The label y is treated as a class label andencoded by label-embedding [2] so its dimension equals to the numberof distinct angles in the training set (i.e., y ∈R120).(a) Generatorz ∈R2 ∼ N(0, I); y ∈R120concat(z, y) ∈R122fc→ 100; BN; ReLUfc→ 100; BN; ReLUfc→ 100; BN; ReLUfc→ 100; BN; ReLUfc→ 100; BN; ReLUfc→ 100; BN; ReLUfc→ 2(b) DiscriminatorA sample x ∈R2fc→ 100; ReLUfc→ 100; ReLUfc→ 100; ReLUfc→ 100; ReLUconcat(output of previous layer, y) ∈R220,where y ∈R120 is the label of x.fc→ 100; ReLUfc→ 1; SigmoidA.2.2 Training SetupsThe cGAN and CcGAN are trained for 6000 iterations on the training set withthe Adam [65] optimizer (with β1 = 0.5 and β2 = 0.999), a constant learning rate5× 10−5 and batch size 128. The rule of thumb formulae in Remark 2.2.3 areused to select the hyper-parameters for HVDL and SVDL, where we let mκ = 2.199Table A.2: Network architectures for the generator and discriminator of ourproposed CcGAN in the simulation. The label y is treated as a real scalarso its dimension is 1. We do not directly input y into the generator anddiscriminator. We first convert each y into the coordinate of the meanrepresented by this y, i.e., (sin(y),cos(y)). Then we insert this coordinateinto the networks.(a) Generatorz ∈R2 ∼ N(0, I); y ∈Rconcat(z, sin(y), cos(y)) ∈R4fc→ 100; BN; ReLUfc→ 100; BN; ReLUfc→ 100; BN; ReLUfc→ 100; BN; ReLUfc→ 100; BN; ReLUfc→ 100; BN; ReLUfc→ 2(b) DiscriminatorA sample x ∈R2 with label y ∈Rconcat(x, sin(y), cos(y)) ∈R4fc→ 100; ReLUfc→ 100; ReLUfc→ 100; ReLUfc→ 100; ReLUfc→ 100; ReLUfc→ 1; SigmoidThus, the three hyper-parameters in this experiments are set as follows: σε = 0.074,κ = 0.017, ν = 3600.A.2.3 Testing SetupsWhen evaluating the trained cGAN, if a test label y′ is unseen in the training set,we first find its closest, seen label y. Then, we generate samples from the trainedcGAN at y instead of at y′. On the contrary, generating samples from CcGAN atunseen labels is well-defined.A.2.4 Varying Number of Gaussians for Training Data GenerationIn this section, we study the influence of the number of Gaussians used for trainingdata generation on the performance of cGAN and CcGAN. We vary the number ofGaussians from 120 to 10 with step size 10 but keep other settings in Section 2.4.1unchanged and plot the line graphs of 2-Wasserstein Distance (log scale) versus thenumber of Gaussians in Fig. A.1. Reducing the number of Gaussians for trainingimplies a larger gap between any two consecutive distinct angles in the training200set. As the number of Gaussians decreases, the continuous scenario gradu-ally degenerates to the categorical scenario, therefore the assumption that asmall perturbation to y results in a negligible change to p(x|y) is no longersatisfied. Consequently, the 2-Wasserstein distances of the proposed two CcGANmethods gradually increase and eventually surpass the 2-Wasserstein distance ofcGAN when the number of Gaussians is small (e.g., less than 40). Note that reduc-ing the number of Gaussians in the training data generation will not improve theperformance of cGAN in the testing because many angles seen in the testing stage(we evaluate each method on 360 angles) do not appear in the training set.Figure A.1: Line graphs of 2-Wasserstein Distance (log scale) versus thenumber of Gaussians for training data generation. As the number ofGaussians decreases, the continuous scenario gradually degenerates tothe categorical scenario, therefore the assumption that a small pertur-bation to y results in a negligible change to p(x|y) is no longer satis-fied. Consequently, the 2-Wasserstein distances of two CcGAN meth-ods gradually increase and eventually surpass the 2-Wasserstein dis-tance of cGAN when the number of Gaussians is small (e.g., less than40).201A.3 More Details of the RC-49 Experiment in Section2.4.2A.3.1 Description of RC-49To generate RC-49, firstly we randomly select 49 3-D chair object models fromthe “Chair” category provided by ShapeNet [16]. Then we use Blender v2.79 1 torender these 3-D models. Specifically, during the rendering, we rotate each chairmodel along with the yaw axis for a degree between 0.1◦ and 89.9◦ (angle reso-lution as 0.1◦) where we use the scene image mode to compose our dataset. Therendered images are converted from the RGBA to RGB color model. In total, theRC-49 dataset consists of 44051 images of image size 64×64 in the PNG format.A.3.2 Network ArchitecturesThe RC-49 dataset is a more sophisticated dataset compared with the simulation,thus it requires networks with deeper layers. We employ the SNGAN architecture[95] in both cGAN and CcGAN consisting of residual blocks for the generatorand the discriminator. Moreover, for the generator in cGAN, the regression labelsare input into the network by the label embedding [2] and the conditional batchnormalization [27]. For the discriminator in cGAN, the regression labels are fedinto the network by the label embedding and the label projection [94]. For Cc-GAN, the regression labels are fed into networks by the two proposed label inputmethods (NLI and ILI) in Section 2.2.2. The pre-trained CNN T1 +T2 for ILI is amodified ResNet-34 with two extra linear layers before the final linear layer. Thelabel embedding network T3 is a 5-layer MLP with 128 nodes in each layer. Thedimension of the noise z is 128 for NLI-based CcGANs and 256 for ILI-based Cc-GANs. Please refer to our codes for more details about the network specificationsof cGAN and CcGAN.1https://www.blender.org/download/releases/2-79/202A.3.3 Training SetupsThe cGAN and CcGAN are trained for 30,000 iterations on the training set withthe Adam [65] optimizer (with β1 = 0.5 and β2 = 0.999), a constant learning rate10−4 and batch size 256.The rule of thumb formulae in Remark 2.2.3 are used to select the hyper-parameters for HVDL and SVDL, where we let mκ = 2. Thus, the three hyper-parameters in this experiments are set as follows: σε = 0.0473, κ = 0.004, ν =50625.The modified ResNet-34 (i.e., the T1+T2 in Fig. 2.4) for ILI is trained for 200epochs with the SGD optimizer, initial learning rate 0.1 (decay at epoch 60, 120,and 160 with factor 0.2), weight decay 10−4, and batch size 256. The 5-layer MLPfor the label embedding in ILI is trained for 500 epochs with the SGD optimizer,initial learning rate 0.1 (decay at epoch 100, 200, and 400 with factor 0.2), weightdecay 10−4, and batch size 256.Please see our codes for more details of the training setups.A.3.4 Testing SetupsThe RC-49 dataset consists of 899 distinct yaw angles and at each angle there are49 images (corresponding to 49 types of chairs). At the test stage, we ask thetrained cGAN or CcGAN to generate 200 fake images at each of these 899 yawangles. Please note that, among these 899 yaw angles, only 450 of them are seen atthe training stage so real images at the rest 449 angles are not used in the training.We evaluate the quality of the fake images from three perspectives, i.e., visualquality, intra-label diversity, and label consistency. One overall metric (Intra-FID)and three separate metrics (NIQE, Diversity, and Label Score) are used. Theirdetails are shown in Supp. A.3.5.A.3.5 Performance MeasuresBefore we conduct the evaluation in terms of the four metrics, we first train an au-toencoder (AE) , a regression-oriented ResNet-34 [51] and a classification-orientedResNet-34 [51] on all real images of RC-49. The bottleneck dimension of the AEis 512 and the AE is trained to reconstruct the real images in RC-49 with the MSE203loss. The regression-oriented ResNet-34 is trained to predict the yaw angle of agiven image. The classification-oriented ResNet-34 is trained to predict the chairtype of a given image. The autoencoder and both two ResNets are trained for 200epochs with a batch size of 256.• Intra-FID [94]: We take Intra-FID as the overall score to evaluate the qual-ity of fake images and we prefer the small Intra-FID score. At each evalu-ation angle, we compute the FID [52] between 49 real images and 200 fakeimages in terms of the bottleneck feature of the pre-trained AE. The Intra-FID score is the average FID over all 899 evaluation angles. Please notethat we also try to use the classification-oriented ResNet-34 to compute theIntra-FID but the Intra-FID scores vary in a very wide range and sometimesobviously contradict with the three separate metrics.• NIQE [93]: NIQE is used to evaluate the visual quality of fake images withthe real images as the reference and we prefer the small NIQE score. Wetrain one NIQE model with the 49 real images at each of the 899 angles sowe have 899 NIQE models. During evaluation, an average NIQE score iscomputed for each evaluation angle based on the NIQE model at that angle.Finally, we report the average and standard deviations of the 899 averageNIQE scores over the 899 yaw angels (i.e., “the mean/standard deviation of899 means”). Note that the NIQE is implemented by the NIQE library inMATLAB. The block size and the sharpness threshold are set to 8 and 0.1respectively in this and rest experiments.• Diversity: Diversity is used to evaluate the intra-label diversity and thelarger the better. In RC-49, there are 49 chair types. At each evaluationangle, we ask a pre-trained classification -oriented ResNet-34 to predict thechair types of the 200 fake images and an entropy is computed based on thesepredicted chair types. The diversity reported in Table 2.2 is the average ofthe 899 entropies over all evaluation angles.• Label Score: Label Score is used to evaluate the label consistency and thesmaller the better. We ask the pre-trained regression-oriented ResNet-34to predict the yaw angles of all fake images and the predicted angles are204then compared with the assigned angles. The Label Score is defined as theaverage absolute distance between the predicted angles and assigned anglesover all fake images, which is equivalent to the Mean Absolute Error (MAP).Note that, to plot the line graphs, we compute Label Score at each of the 899evaluation angles.A.3.6 Extra Experiments• InterpolationIn Fig. A.2, we present some interpolation results of the four CcGAN meth-ods (i.e., HVDL+NLI, SVDL+NLI, HVDL+ILI, and SVDL+ILI). For aninput pair (z,y), we fix the noise z but perform label-wise interpolations, i.e.,varying label y from 4.5 to 85.5. Clearly, all generated images are visuallyrealistic and we can see the chair distribution smoothly changes over continu-ous angles. Please note that, Fig. A.2 is meant to show the smooth change ofthe chair distribution instead of one single chair so the chair type may changeover angles. This confirms CcGAN is capable of capturing the underlyingconditional image distribution rather than simply memorizing training data.Figure A.2: Some example RC-49 fake images from the four CcGAN meth-ods. We fix the noise z but vary the label y.• Degenerated CcGANIn this experiment, we consider the extreme case of the proposed CcGAN(degenerated CcGAN), i.e., σ → 0 and κ → 0 or ν →+∞. Some examples205from a degenerated NLI-based CcGAN are shown in Fig. A.3. Since, at eachangle, the degenerated CcGAN only uses the images at this angle, it leadsto the mode collapse problem (e.g, the row in the yellow rectangle) and badvisual quality (e.g., images in the red rectangle) at some angles.Note that the degenerated CcGAN is still different from cGAN, since westill treat y as a continuous scalar instead of a class label here and we use theproposed label input method (e.g., NLI) to incorporate y into the generatorand the discriminator.• cGAN: different number of classesIn this experiment, we show that cGAN still fails even though we bin [0.1,89.9]into other number of classes. We experimented with three different bin set-ting – grouping labels into 90, 150, and 210 classes, respectively. Experi-mental results are shown in Fig. A.4 and we observe all three cGANs com-pletely fail.A.4 More Details of the Experiment on UTKFace inSection 2.4.3A.4.1 Description of the UTKFace DatasetThe UTKFace dataset is an age regression dataset [164], with human face imagescollected in the wild. We use a preprocessed version (cropped and aligned), withages spanning from 1 to 60. After the data cleaning (i.e., removing images ofvery low quality or with clearly wrong labels), the number of images left is 14760.These images are then resized to 64×64. The histogram of the UTKFace datasetafter data cleaning is shown in A.5.From Fig. A.5, we can see UTKFace dataset is very imbalanced so the samplesfrom the minority age groups are unlikely to be chosen at each iteration duringthe GAN training. Consequently, cGAN and CcGAN may not be well-trainedat these minority age groups. To increase the chance of drawing these minoritysamples during training, we randomly replicate samples in the minority age groupsto ensure that the sample size of each age is more than 200.206Figure A.3: Some example RC-49 fake images from a degenerated NLI-based CcGAN.207Figure A.4: Example RC-49 fake images from cGAN when we bin the yawangle range into different number of classes.A.4.2 Network ArchitecturesThe network architectures used in this experiment is similar to those in the RC-49experiment. Please refer to our codes for more details about the network specifica-tions.208Figure A.5: The histogram of the UTKFace dataset with ages varying from 1to 60.A.4.3 Training SetupsThe cGAN and CcGAN are trained for 40,000 iterations on the training set withthe Adam [65] optimizer (with β1 = 0.5 and β2 = 0.999), a constant learning rate10−4 and batch size 512. The rule of thumb formulae in Remark 2.2.3 are used toselect the hyper-parameters for HVDL and SVDL, where we let mκ = 1.Please see our codes for more details of the training setups.A.4.4 Performance MeasuresSimilar to the RC-49 experiment, we evaluate the quality of fake images by Intra-FID, NIQE, Diversity, and Label Score. We also train an AE (bottleneck dimensionis 512), a classification-oriented ResNet-34, and a regression-oriented ResNet-34on the UTKFace dataset. Please note that, the UTKFace dataset consists of face209images from 5 races based on which we train the classification-oriented ResNet-3.The AE and both two ResNets are trained for 200 epochs with a batch size 256.A.4.5 Extra Experiments• InterpolationTo perform label interpolation experiments, we keep the noise vector z fixedand vary label from age 3 to age 57 for the four CcGANs. The interpolationresults are illustrated in A.6. As age y increases, we observe the syntheticface gradually becomes older in appearance. This observation convincinglyshows that all four CcGANs do not simply memorize or overfit to the trainingset. Indeed, our CcGANs demonstrate continuous control over syntheticimages with respect to ages.Figure A.6: Some examples of generated UTKFace images from the four Cc-GAN methods. We fix the noise z but vary the label y from 3 to 57.• Degenerated CcGANWe consider the extreme cases of the proposed CcGANs on the UTKFacedataset. As shown in Fig. A.7, the degenerated NLI-based CcGANs fails togenerate facial images at some ages (e.g., 51 and 57) because of too smallsample sizes.• cGAN: different number of classesIn the last experiment, we bin samples into different number of classes basedon ground-truth labels, in order to increase the number of training samples210at each class. Then we train cGAN using samples from the binned classes.We experimented with two different bin setting, i.e., binning image sam-ples into 60 classes and 40 classes, respectively. The results are reportedin Fig.A.8. The results demonstrate cGANs consistently fail to generatediverse synthetic images with labels aligned with their conditional infor-mation. Moreover, the image quality is much worse than those from theproposed CcGANs. In conclusion, compared with existing cGANs, our pro-posed CcGANs have substantially better performance in terms of the imagequality and diversity.A.5 More Details of the Experiment on Cell-200 inSection 2.4.4A.5.1 Description of Cell-200Cell-200 is a synthetic image dataset, emulating the colonies of bacterial cells in theview of fluorescence microscope. This dataset contains cell populations with over-all number varying between 1 and 200, generated with [73]. For each cell popula-tion (e.g., 1 to 200), we generate 1000 different synthetic fluorescence microscopicimages, with diverse cell variations (e.g., shapes, locations, overlaps and blurringeffects). As in [74], we set nucleus radius as 5, and image size as 256× 256. Toalleviate computational burden, images in the Cell-200 dataset are then resized to64×64.A.5.2 Network ArchitecturesThe network architectures for cGAN and CcGAN in this experiment are adaptedfrom the famous DCGAN [112] architecture. The dimension of the noise z is 128for NLI-based CcGANs and 256 for ILI-based CcGANs. Please refer to our codesfor more details about the network specifications.211Figure A.7: Some example UTKFace fake images from a degenerated NLI-based CcGAN.A.5.3 Training SetupsThe cGAN and CcGAN are trained for 5000 iterations on the training set withthe Adam [65] optimizer (with β1 = 0.5 and β2 = 0.999) and a constant learningrate 10−4. The rule of thumb formalue in Remark 2.2.3 are used to select thehyperparameters for HVDL and SVDL, where we let mκ = 2.212Figure A.8: Example UTKFace fake images from cGAN when we bin theage range into different number of classes.For cGAN training, the cell count range [1,200] is split into 100 disjoint in-tervals, i.e., [1,3), [3,5), . . . , [197,199), [199,200]. In this case, cGAN estimatesimage distribution conditional on these intervals. In Supp. A.5.6, we also comparethe performance of cGAN under different splitting schemes.Please note that we use different batch sizes for cGAN and CcGAN in this ex-periment. The batch size for cGAN is 512. Differently, for all CcGAN methods inthis experiment, the batch size is 512 for the generator but 32 for the discriminator.The reason that we use different batch sizes for the generator and discriminator inCcGAN is based on some observations we got during training. In this experiment,213if the generator and the discriminator in CcGAN have the same batch size, the dis-criminator loss often decreases to almost zero very quickly while the generator lossstill maintains at a high level. Consequently, at each iteration, the discriminator caneasily distinguish the real and fake images while the generator cannot fool the dis-criminator and won’t improve in the next iteration, which implies a high imbalancebetween the generator update and the discriminator update. To balance the train-ing of the generator and the discriminator, we deliberately decrease the number ofimages seen by the discriminator at each iteration to slow down the update of thediscriminator so that the generator can catch up.Please see our codes for more details of the training setups.A.5.4 Testing SetupsWe evaluate the trained cGAN and four CcGAN methods on all 200 cell counts(half of them are blinded during training). Each method generates 1,000 imagesfor each cell count, so there are 200,000 fake images from each method.When evaluating the trained cGAN, if a test label y′ is unseen in the trainingset, we just need to find which interval (recall we split [1,200] is split into 100disjoint intervals) covers this label. Then, we generate samples from the trainedcGAN conditional on this interval instead of y′.A.5.5 Performance MeasuresSimilar to the previous two experiments, we evaluate the quality of fake imagesby Intra-FID, NIQE, and Label Score but Diversity. The Diversity score is notavailable here because we don’t have any class label in Cell-200. An AE with abottleneck dimension of 512 and a regression-oriented ResNet-34 are pre-trainedon the complete Cell-200 dataset (i.e., 200,000 images) to compute the Intra-FIDand Label Score respectively. The possibility of lacking class labels in regression-oriented datasets is another reason that we propose to use an AE to compute Intra-FID instead of a classification-oriented CNN. The AE is trained for 50 epochs witha batch size of 256. The regression-oriented ResNet-34 is trained for 200 epochswith a batch size of 256.214A.5.6 Extra Experiments• InterpolationTo perform the label interpolation, we keep the noise vector z fixed and varylabel from 10 to 200 for the four CcGANs. The interpolation results are illus-trated in Fig. A.9. As cell count y increases, we observe the cells in imagesbecome more crowded. This observation convincingly shows that all fourCcGANs do not simply memorize or overfit to the training set. Indeed, ourCcGANs demonstrate continuous control over synthetic images with respectto cell counts.Figure A.9: Some examples of generated Cell-200 images from the four Cc-GAN methods. We fix the noise z but vary the label y from 10 to 200.• cGAN: different number of classesIn this experiment, we experimented with two different bin setting – group-ing labels into 100 classes and 50 classes, respectively. Experimental resultsare shown in Fig. A.10. We observe both cGANs fail in this experiment.First of all, cGANs still suffer from the mode collapse problems. Besides,cell counts of generated images do not match those of their given labels.215Figure A.10: Example Cell-200 fake images from cGAN when we bin therange of cell count into different number of classes.A.6 More Details of the Experiment on the SteeringAngle Dataset in Section 2.4.5A.6.1 Description of Steering AngleThe Steering Angle dataset is a subset of an autonomous driving dataset [20, 21].Steering Angle consists of 12,271 RGB images with 1,904 distinct steering anglesranging from −80◦ to 80◦. We resize all images to 64×64. The histogram of thesteering angles in this dataset is show in Fig. A.11.216Figure A.11: The histogram of the Steering Angle dataset with steering an-gles varying from −80◦ to 80◦. At many angles, we only have 1 or 2images.A.6.2 Network ArchitecturesThe network architectures used in this experiment is similar to those in the RC-49and UTKFace experiments. Please refer to our codes for more details about thenetwork specifications.A.6.3 Training SetupsThe cGAN and CcGAN are trained for 20,000 iterations on the training set withthe Adam [65] optimizer (with β1 = 0.5 and β2 = 0.999) and a constant learningrate 10−4. The rule of thumb formalue in Remark 2.2.3 are used to select thehyperparameters for HVDL and SVDL, where we let mκ = 5.Please note that, similar to the Cell-200 experiment, we use different batch217sizes for the generator and discriminator in four CcGAN methods. The batch sizeis set to 64 and 512 respectively for the discriminator and generator in CcGAN.Please refer to Supp. A.5.3 for the reason.Please see our codes for more details of the training setups.A.6.4 Testing SetupsAt the testing stage, we first set 2,000 evenly spaced evaluation labels in [−80◦,80◦]and we ask each GAN model to generate 50 images conditional on each evaluationlabel.A.6.5 Performance MeasuresThe quality of generated images from each GAN is evaluated by SFID, NIQE,Diversity, and Label Score.• SFID: To computing SFID, we preset 1,000 SFID centers [−80◦,80◦] andlet rSFID = 2◦. These SFID centers and rSFID = 2 define 1,000 joint SFIDintervals. We compute one FID between real and fake images with labels inthis interval. We report the mean (i.e., SFID) and standard deviation of theseFIDs for each GAN.• NIQE [93]: Different from previous three experiments, we train only oneNIQE model by using all real images in the Steering Angle dataset since thisdataset is highly imbalanced. In the evaluation, we compute one NIQE scorefor each SFID interval. The reported NIQE score in Table 2.5 is the mean ofthese NIQE scores.• Diversity: The original autonomous driving dataset [20, 21] does not haveclass labels. To compute Diversity, we manually group the images in Steer-ing Angle into five categories according to their background objects or thetypes of the road in the images. The five groups are labeled respectively bytree, tree+barrier, bush, bush+barrier, and winding mountain road. Some ex-ample images for these five groups are show in Fig. A.12. Images in the treegroup all have trees in the background. Images in the tree+barrier group allhave trees and barriers in the background. Images in the bush group all have218bushes in the background. Images in the bush+barrier group all have bushesand barriers in the background. Images in the winding mountain road groupall correspond to the scenes of winding mountain roads. A classification-oriented ResNet-34 is trained to classify images from these five groups, andthen the Diversity score can be computed based on the entropies of predictedscenes in each SFID interval.Figure A.12: Example Steering Angle images from 5 scenes, i.e., tree,tree+barrier, bush, bush+barrier, and winding mountain road (fromleft to right).• Label Score: Similar to previous experiments, we pre-train a regression-oriented ResNet-34 to predict the angle for each fake image, and then com-putes Label Score.Please note that, when plotting the line graph of Label Score versus SFIDCenter in Fig 2.21, one Label Score is computed for each SFID interval.A.6.6 Extra Experiments• InterpolationIn this section, for each CcGAN method, we fix the noise vector z but varythe regression label y from −71.8◦ to 72◦. We can see the road in the imagegradually changes from a left turn to a right turn.• cGAN: different number of classesIn this experiment, we experimented with three different bin setting – group-ing labels into 90 classes, 150 classes, and 210 classes, respectively. Experi-mental results are shown in Fig. A.14. We observe that different bin settingscannot improve cGAN’s performance.219Figure A.13: Some examples of generated Steering Angle images from thefour CcGAN methods. We fix the noise z but vary the label y from−71.8◦ to 72◦.Figure A.14: Example Steering Angle fake images from cGAN when we binthe range of steering angles into different number of classes.220A.7 More Details of the 2D-Gaussians Experiment inSection 3.4.1A.7.1 Network ArchitecturesTable A.3: Network architectures for the generator and discriminator in thesimulation. “fc” denotes a fully-connected layer.(a) Generatorz ∈R128 ∼ N(0, I)fc→ 100; ReLUfc→ 100; ReLUfc→ 100; ReLUfc→ 2(b) DiscriminatorA sample x ∈R2fc→ 100; ReLUfc→ 100; ReLUfc→ 100; ReLUfc→ 1; SigmoidTable A.4: Architecture of the 5-layer MLP for DRE in the simulation. Weuse group normalization (GN) [154] in each hidden layer instead of batchnormalization [57] because we find batch normalization performs quitedifferently in the training stage and evaluation stage.A sample x ∈R2fc→ 2048; GN (4 groups); ReLU; dropout (p = 0.2)fc→ 1024; GN (4 groups); ReLU; dropout(p = 0.2)fc→ 512; GN (4 groups); ReLU; dropout(p = 0.2)fc→ 256; GN (4 groups); ReLU; dropout(p = 0.2)fc→ 128; GN (4 groups); ReLU; dropout(p = 0.2)fc→ 1; ReLUA.7.2 Training SetupsThe GAN model is trained for 50 epochs with the Adam [24] optimizer, a constantlearning rate 10−3 and batch size 512. DR models are trained with the setups inTable A.5, and the hyperparameter selections are in Table A.6.221Table A.5: Setups for training the 5-layer MLP under different loss functionsin the simulation.Loss uLISF DSKL BARR SPOptimizer Adam [64] Adam [64] Adam [64] Adam [64]Constant LR 1e-5 1e-5 1e-5 1e-3Epochs 400 400 400 400Batch Size 512 512 512 512Table A.6: Hyperparameter selection in Simulation. Two-SampleKolmogorov-Smirnov test statistic is shown for each λ at eachround.Round 1λ 0 0.005 0.01 0.05 0.1KS Stat. 0.00464 0.00388 0.00386 0.00496 0.0046Round 2λ 0 0.005 0.01 0.05 0.1KS Stat. 0.00896 0.00838 0.01044 0.0096 0.01068Round 3λ 0 0.005 0.01 0.05 0.1KS Stat. 0.00494 0.00508 0.00398 0.008 0.00588A.7.3 How Does the Loss Function Affect Density Ratio Estimationand the Subsequent Subsampling?We repeat the following steps 5 times and report the average result. For each one,we do the following:1. Train the GAN for 50 epochs. Train the density ratio model (MLP-5) withthe SP loss and uLSIF loss respectively for 5000 epochs. We use the Adam[24] optimizer. The initial learning rate for both loss functions is set to 10−3and decayed every 1000 epochs. Set 17 checkpoints during training from the20th epoch to the 5000th epoch. The hyperparameter λ is set to 0 or 0.05.2222. Use GMM to model pg. The optimal number of components is selected byminimizing BIC. Since we can draw a large number of fake samples from pg,the estimation of pg by GMM should be very accurate. In this experiment,we use 100,000 fake samples to fit the GMM. The true distribution pr is amixture of 25 Gaussian with parameters known to us. Therefore, we canknow the ground truth density ratio function r = pr/pg.3. At each checkpoint, evaluate the ground truth density ratio function r andtwo estimated density ratio functions rˆ (trained with SP and uLSIF respec-tively) on all 10000 fake samples, only high-quality fake samples, and onlylow-quality fake samples. Report the average density ratio of these fakesamples at each checkpoint.4. At each checkpoint, we use DRE-SP+SIR to draw 10,000 fake samples andreport the percentage of high quality samples.In Fig. 3.4 of the main article, we show the results when λ = 0.05. Notethat, in Figs. 3.4(a) and 3.4(b) we only report the training loss of the first roundof 5 repetitions; however, in Figs. 3.5(a) and 3.5(b), the average density ratios ofhigh/low quality samples and the percentage of high quality samples are averagedover 5 repetitions.In Fig. A.15, we also show the average density ratio over all 10,000 fake sam-ples and the percentage of high quality samples (averaged over 5 repetitions). Wecan see, when using the SP loss, the average density ratio over all 10,000 fake sam-ples is close to the ground truth. However, when using the uLSIF loss, the averagedensity ratio keeps decreasing (a sign of overfitting).In Fig. 3.4(a), the training loss of uLSIF stops decreasing after 3000 epochsbecause of a too small learning rate which seems to conflict with our argumentin Section 1.2. In Fig. A.16, however, we show the training curve for a constantlearning rate 10−5 and we can see a constantly decreasing trend.Besides λ = 0.05, we also set λ to 0 and show the corresponding results inFigure A.17 and A.18. From Fig. A.17, we can see both loss functions suffer fromoverfitting. However, the overfitting problem for the uLSIF loss is more obviousthan the SP loss, because Fig. A.18 shows that the difference between high quality223Figure A.15: Average density ratio of 10,000 fake samples and percentage ofhigh quality samples versus the epoch of DRE training when λ = 0.05.The grey horizontal line stands for the ground truth average densityratio.and low quality samples in terms of the average density ratios is bigger when usingSP loss. Therefore, even with the overfitting problem, the SP loss still leads to amuch better subsampling performance than the uLSIF loss does.A.8 More Details of the Experiment on CIFAR-10 inSection 3.4.2A.8.1 Network ArchitecturesWe implement DCGAN and WGAN-GP with generator and discriminator shownin Table A.7. For MMD-GAN, we directly use codes in https://github.com/OctoberChang/MMD-GAN. Please see [76] for more details about MMD-GAN.224Figure A.16: The training curve of a 5-layer MLP under the penalized uLSIFloss (1.16) with λ = 0.05 and a constant learning rate 10−5.A.8.2 Training SetupsIn the CIFAR-10 setting, three types of GAN are trained on the 50,000 trainingimages with setups in Table A.13. The modified ResNet-34 for feature extractionis trained on the training set for 200 epochs with the SGD optimizer, initial learningrate 0.1 (decay at epoch 100 and 150 with factor 0.1), weight decay 10−4, and batchsize 256. The training setups for different DRE methods are shown in Table A.14.A.8.3 Performance MeasuresTo evaluate the quality of fake images by IS and FID, we train the Inception-V3on 50,000 CIFAR-10 training images. FID is computed based on the final averagepooling features from the pre-trained Inception-V3. We compute the FID between50,000 fake images and 50,000 training images.225Figure A.17: Average density ratio of 10,000 fake samples and percentage ofhigh quality samples versus the epoch of DRE training when λ = 0.The grey horizontal line stands for the ground truth average densityratio.A.9 More Details of the Experiment on MNIST inSection 3.4.3A.9.1 Network ArchitecturesWe implement DCGAN with the generator and the discriminator shown in TableA.16. The ResNet-34 for feature extraction and the MLP-5 for DRE are identical tothe CIFAR-10 experiment except a few setups (e.g., the number of input channelsof the first convolutional layer of ResNet-34) and please see our codes for moredetails.226(a) Average density ratio of high quality fake samples versus the epoch of DRE train-ing when λ = 0.(b) Average density ratio of low quality fake samples versus the epoch of DRE trainingwhen λ = 0.Figure A.18: Average density ratio of high/low quality fake samples versusthe epoch of DRE training when λ = 0. The grey horizontal linestands for the ground truth.227Figure A.19: A residual block (ResBlock) in ResNet-34 for CIFAR-10.Please check our codes for detailed setups of each layer.228Table A.7: Network architectures for the generator and discriminator of DC-GAN and WGAN-GP in the experiment on CIFAR-10. The slopes ofall LeakyReLU are set to 0.2. We denote stride and padding by s and prespectively.(a) Generatorz ∈R128 ∼ N(0, I)fc→ 4×4×512deconv, 4×4, s = 2, p = 1, 256; BN; ReLUdeconv, 4×4, s = 2, p = 1, 128; BN; ReLUdeconv, 4×4, s = 2, p = 1, 64; BN; ReLUconv, 3×3, s = 1, p = 1, 3; Tanh(b) DiscriminatorRGB image x ∈R3×32×32conv, 3×3, s = 1, p = 1, 64; LeakyReLUconv, 4×4, s = 2, p = 1, 64; LeakyReLUconv, 3×3, s = 1, p = 1, 128; LeakyReLUconv, 4×4, s = 2, p = 1, 128; LeakyReLUconv, 3×3, s = 1, p = 1, 256; LeakyReLUconv, 4×4, s = 2, p = 1, 256; LeakyReLUconv, 3×3, s = 1, p = 1, 512; LeakyReLUfc→ 1sigmoid (for DCGAN only)A.9.2 Training SetupsIn the MNIST setting, DCGAN is trained on the reduced training set with theAdam optimizer, constant learning rate 10−4, batch size 256, and 500 epochs. Themodified ResNet-34 for feature extraction is trained on the reduced training set for200 epochs with the SGD optimizer, initial learning rate 0.01 (decay at epoch 100with factor 0.1), weight decay 10−4, and batch size 512. To implement DRE-F-SP,we use the MLP-5 as in the CIFAR-10 setting, and it is trained with the Adamoptimizer, initial learning rate 10−4 (decay at the epoch 400 with factor 0.1), batchsize 512, and 500 epochs. The optimal hyperparameter is selected based on TableA.17229Table A.8: Architecture of the ResNet-34 for feature extraction for CIFAR-10. “down” refers to down sampling. We add an extra fully-connectedlayer fc1 to output features for density ratio estimation in feature space.×n represents n consecutive such blocks.RGB image x ∈R3×32×32conv, 3x3, stride=1, p=1, 64; BN; ReLU{ResBlock, 64} ×3ResBlock, down, 128{ResBlock, 128}×3ResBlock, down, 256{ResBlock, 256} ×5ResBlock, down, 512{ResBlock, 512} ×2Avg. pooling, 4×4, s = 4fc1→ 32×32×3 = 3072fc2→ 10Table A.9: 5-layer MLP for DRE in feature space for CIFAR-10.extracted feature y ∈R3072fc→ 2048, GN (4 groups), ReLU, Dropout(p = 0.4)fc→ 1024, GN (4 groups), ReLU, Dropout(p = 0.4)fc→ 512, GN (4 groups), ReLU, Dropout(p = 0.4)fc→ 256, GN (4 groups), ReLU, Dropout(p = 0.4)fc→ 128, GN (4 groups), ReLU, Dropout(p = 0.4)fc→ 1, ReLUTable A.10: 2-layer CNN for DRE in pixel space [98] for CIFAR-10RGB image x ∈R3×32×32conv, 9×9, s = 1, 6Avg. pooling, 2×2, s = 2Sigmoidconv, 9×9, s = 1, 12Avg. pooling, 2×2, s = 2Sigmoidfc→ 1; ReLU230Table A.11: 6-layer CNN for DRE in pixel space [63] for CIFAR-10RGB image x ∈R3×32×32conv, 3×3, s = 1, 60Max pooling, 2×2, s = 1conv, 3×3, s = 1, 50conv, 3×3, s = 1, 40Max pooling, 2×2, s = 1conv, 3×3, s = 1, 20Max pooling, 2×2, s = 1conv, 2×2, s = 1, 10conv, 2×2, s = 1, 5fc→ 250, ReLU, Dropout(p = 0.25)fc→ 1; ReLUTable A.12: Binary classifier for the BOC-based DRE [47] for CIFAR-10RGB image x ∈R3×32×32conv, 3×3, s = 1, p = 1, 64; ReLU; BNMax pooling, 2×2, s = 2conv, 3×3, s = 1, p = 1, 64; ReLU; BNMax pooling, 2×2, s = 2conv, 3×3, s = 1, p = 1, 64; ReLU; BNMax pooling, 2×2, s = 2conv, 3×3, s = 1, p = 1, 64; ReLU; BNMax pooling, 2×2, s = 2fc→ 1; SigmoidTable A.13: Training setups for three types of GAN on CIFAR-10.DCGAN WGAN-GP MMD-GANOptimizer Adam Adam AdamConstant learning rate 2E-04 2E-04 5E-05Epochs 500 2000 4000Batch Size 256 256 256231Table A.14: Training setups for DRE methods on CIFAR-10. Correspondingsubsampling results are shown in Tables 3.3 to 3.5 of the main article.DRE-P-uLSIF DRE-P-DSKL DRE-P-BARR BOCOptimizer Adam Adam Adam AdamInitia LR 1E-4 1E-4 1E-4 1E-3LR Decay No No No NoEpochs 200 200 200 100Batch Size 512 512 512 100DRE-F-uLSIF DRE-F-DSKL DRE-F-BARR DRE-F-SPOptimizer Adam Adam Adam AdamInitia LR 1E-5 1E-5 1E-5 1E-4LR Decay No No No epoch 100(×0.1)Epochs 200 200 200 200Batch Size 512 512 512 512Table A.15: Hyperparameter selection for CIFAR-10. Two-SampleKolmogorov-Smirnov test statistic is shown for each λ and each GAN.DCGANλ 0 0.005 0.01 0.05 0.1KS Stat. 1.380E-01 1.390E-01 1.386E-01 1.389E-01 1.394E-01WGAN-GPλ 0 0.005 0.01 0.05 0.1KS Stat. 1.131E-01 1.118E-01 1.138E-01 1.165E-01 1.204E-01MMD-GANλ 0 0.005 0.006 0.008 0.01KS Stat. 1.220E-01 1.231E-01 1.209E-01 1.233E-01 1.209E-01232Table A.16: Network architectures for the generator and the discrimina-tor of DCGAN in the experiment on the MNIST. The slopes of allLeakyReLU are set to 0.2. We denote stride and padding by s and prespectively.(a) Generatorz ∈R128 ∼ N(0, I)dense→ 7×7×256deconv, 4×4, s = 2, p = 1, 128; BN; ReLUdeconv, 4×4, s = 2, p = 1, 64; BN; ReLUconv, 3×3, s = 1, p = 1, 3; Tanh(b) DiscriminatorRGB image x ∈R1×28×28conv, 3×3, s = 1, p = 1, 64; LeakyReLUconv, 4×4, s = 2, p = 1, 64; LeakyReLUconv, 3×3, s = 1, p = 1, 128; LeakyReLUconv, 4×4, s = 2, p = 1, 128; LeakyReLUconv, 3×3, s = 1, p = 1, 256; LeakyReLUfc→ 1; SigmoidTable A.17: Hyperparameter selection for MNIST. Two-SampleKolmogorov-Smirnov test statistic is shown for each λ .DCGANλ 0 0.001 0.01 0.1KS Stat. 1.203E-01 1.182E-01 1.168E-01 1.115E-01A.9.3 Performance MeasuresTo evaluate the quality of fake images by IS and FID, we train the Inception-V3 onall 60,000 MNIST training images (not the reduced training set). FID is computedbased on the final average pooling features from the pre-trained Inception-V3. Wecompute the FID between 50,000 fake images and all 60,000 training images.233A.10 More Details of the Experiment on CelebA inSection 3.4.4A.10.1 Network ArchitecturesWe implement SNGAN [95] based on codes from https://github.com/christiancosgrove/pytorch-spectral-normalization-gan and https://github.com/pfnet-research/sngan projection. The ResNet-34 for feature extraction and the MLP-5 for DREare identical to the CIFAR-10 experiment except a few setups (e.g., the stride sizeof the first residual block of ResNet-34) and please see our codes for more details.A.10.2 Training SetupsThe SNGAN is trained on the training set with the Adam optimizer, constant learn-ing rate 10−4 for the generator, constant learning rate 4×10−4 for the discrimina-tor, batch size 256, and 100 epochs. The modified ResNet-34 for feature extractionis trained on the training set for 100 epochs with the SGD optimizer, initial learningrate 0.001 (decay at epoch 30 and 70 with factor 0.1), weight decay 10−4, and batchsize 256. The MLP-5 is trained on the training set with the Adam optimizer, initiallearning rate 10−4 (decay at epoch 30 and 70), batch size 512, and 100 epochs. Theoptimal hyperparameter is selected based on Table A.18.Table A.18: Hyperparameter selection for CelebA. Two-SampleKolmogorov-Smirnov test statistic is shown for each λ .SNGANλ 0 0.005 0.01 0.05 0.1KS Stat. 0.2643 0.2622 0.2646 0.2643 0.2645A.10.3 Performance MeasuresTo evaluate the quality of fake images by IS and FID, we use the Inception-V3which is pre-trained on the ImageNet [28]. FID is computed based on the finalaverage pooling features from the pre-trained Inception-V3. The computation isbased on codes from https://github.com/sbarratt/inception-score-pytorch and234https://github.com/mseitzer/pytorch-fid. We compute the FID between 50,000 fakeimages and all 192,599 training images.A.11 More Details of Experiments on CIFAR-10 inSection 4.3.1In this section, we introduce the training and test setups for our experiments onCIFAR-10.A.11.1 Network architecturesFor feature extraction, we use the specially designed ResNet-34 in [31]. Similarto [31], we use an MLP-5 with structure shown in Table A.19 as the unconditionaldensity ratio model to implement DRE-F-SP. Another MLP-5 with structure shownin Table A.20 is used as the conditional density ratio model to implement cDRE-F-cSP.Table A.19: The 5-layer MLP for DRE in feature space for CIFAR-10.Extracted feature h ∈R3072fc→ 2048, GN (8 groups), ReLU, Dropout(p = 0.5)fc→ 1024, GN (8 groups), ReLU, Dropout(p = 0.5)fc→ 512, GN (8 groups), ReLU, Dropout(p = 0.5)fc→ 256, GN (8 groups), ReLU, Dropout(p = 0.5)fc→ 128, GN (8 groups), ReLU, Dropout(p = 0.5)fc→ 1, ReLUA.11.2 Training setupsFor each experimental setting described in Section 4.3.1, we train ResNet-34,MLPs, and BigGAN with setups as follows. The specially designed ResNet-34for feature extraction is trained on the corresponding training set for 200 epochswith the SGD optimizer, initial learning rate 0.1 (decay at epoch 100 and 150 withfactor 0.1), weight decay 10−4, and batch size 128. Two MLP-5 models are trainedon the training set with the Adam optimizer [64], initial learning rate 10−4 (decay235Table A.20: The 5-layer MLP for cDRE in feature space for CIFAR-10. Theembedded class label is appended to the extracted feature h.Input: extracted feature h ∈R3072and embedded class label y ∈R10Concatenate [h,y] ∈R3082fc→ 2048, GN (4 groups), ReLU, Dropout(p = 0.5)fc→ 1024, GN (4 groups), ReLU, Dropout(p = 0.5)fc→ 512, GN (4 groups), ReLU, Dropout(p = 0.5)fc→ 256, GN (4 groups), ReLU, Dropout(p = 0.5)fc→ 128, GN (4 groups), ReLU, Dropout(p = 0.5)fc→ 1, ReLUat epoch 150 and 250), batch size 128, 350 epochs, and λ = 10−3. The implemen-tation of BigGAN is mainly based on https://github.com/ajbrock/BigGAN-PyTorch.The BigGAN model in each setting is trained for 2000 epochs with batch size 512.Following [31], to evaluate fake images, we pre-train an Inception V3 on CIFAR-10 using all 50000 training images. zA.11.3 Testing setupsAt the testing stage, we generate 50,000 images (5000 per class) from each subsam-pling method. We adopt two popular metrics—Fre´chet inception distance (FID)[52] and Intra-FID [94]—to evaluate the image quality of the fake images. Intra-FID is an overall metric for the evaluation of conditional GANs, which evaluatecGANs from three perspectives: (1) visual quality, (2) intra-label diversity, and (3)label consistency. On the other hand, FID can only evaluate the visual quality andthe marginal diversity (since it is not computed within each class). Both Intra-FIDand FID are computed based on the feature map with dimension 2048 of the lastaverage pooling layer of the pre-trained Inception V3.236A.12 More Details of Experiments on Tiny-ImageNet inSection 4.3.2In this section, we introduce the detailed training and testing setups of our experi-ments on Tiny-ImageNet.A.12.1 Network architecturesSimilar to the CIFAR-10 experiment, a specially designed ResNet-34 is trainedto extract high-level features from images. This ResNet-34 is almost identical tothe one used in the CIFAR-10 experiment. The MLP-5 with structure shown inTable A.21 is used as the conditional density ratio model to implement cDRE-F-cSP on the Tiny-ImageNet dataset.Table A.21: 5-layer MLP for cDRE in feature space for Tiny-ImageNet. Theembedded class label is appended to the extracted feature h.Input: extracted feature h ∈R12,288and embedded class label y ∈R200Concatenate [h,y] ∈R12,488fc→ 2048, GN (4 groups), ReLU, Dropout(p = 0.5)fc→ 1024, GN (4 groups), ReLU, Dropout(p = 0.5)fc→ 512, GN (4 groups), ReLU, Dropout(p = 0.5)fc→ 256, GN (4 groups), ReLU, Dropout(p = 0.5)fc→ 128, GN (4 groups), ReLU, Dropout(p = 0.5)fc→ 1, ReLUA.12.2 Training setupsThe modified ResNet-34 for feature extraction is trained for 200 epochs with theSGD optimizer, initial learning rate 0.1 (decay at epoch 100 and 150 with factor0.1), weight decay 10−4, and batch size 128. A 5-layer MLP is trained as theconditional density ratio model in cDRE-F-cSP+RS with Adam optimizer [64],initial learning rate 10−4 (decay at epoch 150 and 250), batch size 128, 350 epochs,and λ = 10−3. The BigGAN model is trained with 2000 epochs and batch size128. We also adopt DiffAugment [166] (a data augmentation method for the GAN237training with limited data) to improve the performance of BigGAN. Similar tothe CIFAR-10 experiment, we train an Inception V3 net on Tiny-ImageNet fromscratch for evaluation.A.12.3 Testing setupsIn the testing phase, we generate one million images (5000 per class) from theno-sbusampling and the proposed methods, respectively. Similar to CIFAR-10 ex-periments, we adopt FID and Intra-FID as two evaluation metrics. Both metrics arealso computed based on the feature map with dimension 2048 of the last averagepooling layer of a pretrained Inception V3.A.13 More Details of Experiments on RC-49 in Section4.3.3In this section, we introduce the detailed training and testing setups of our experi-ments on RC-49.A.13.1 Network architecturesFor the feature extraction model, we adopt a specially designed sparse antoencoderwith architecture shown in Table A.22. We also employ an MLP-5 as the condi-tional density ratio model, which uses the same architecture as shown in Table A.21(i.e., encoder branch), Table A.23 (i.e., decoder branch), and Table A.24 (i.e., labelprediction branch). We adopt the SNGAN architecture for CcGAN which is alsoused by [30].The models for the computation of Intra-FID, NIQE, Diversity, and LabelScore are consistent with those used by [30, 33]. Please refer to the official im-plementation of CcGANs at https://github.com/UBCDingXin/improved CcGANfor more details.A.13.2 Training setupsThe sparse autoencoder for feature extraction is trained for 200 epochs with theSGD optimizer, initial learning rate 0.01 (decay every 50 epochs with a factor 0.1),238Table A.22: The architecture of the encoder in the sparse autoencoder for ex-tracting features from 64× 64 RGB images. In convolutional (Conv)operations, ch denotes the number of channels, k/s/p denote kernelsize, stride and number of padding, respectively.Input: an RGB image x ∈R64×64×3.Conv (ch→ 64, k4/s2/p1), BN, ReLUConv (ch→ 64, k3/s1/p1), BN, ReLUConv (ch→ 128, k4/s2/p1), BN, ReLUConv (ch→ 128, k3/s1/p1), BN, ReLUConv (ch→ 256, k4/s2/p1), BN, ReLUConv (ch→ 256, k3/s1/p1), BN, ReLUConv (ch→ 512, k4/s2/p1), BN, ReLUConv (ch→ 768, k3/s1/p1), ReLUOutput: extracted sparse features h ∈R4×4×768Table A.23: The architecture of the decoder in the sparse autoencoder forreconstructing 64 × 64 input images from extracted features. Intransposed-convolutional (ConvT) operations, ch denotes the numberof channels, k/s/p denote kernel size, stride and number of padding,respectively.Input: extracted sparse features h ∈R4×4×768from Table A.22.ConvT (ch→ 512, k4/s2/p1), BN, ReLUConvT (ch→ 512, k3/s1/p1), BN, ReLUConvT (ch→ 256, k4/s2/p1), BN, ReLUConvT (ch→ 256, k3/s1/p1), BN, ReLUConvT (ch→ 128, k4/s2/p1), BN, ReLUConvT (ch→ 128, k3/s1/p1), BN, ReLUConvT (ch→ 64, k4/s2/p1), BN, ReLUConvT (ch→ 3, k3/s1/p1), TanhOutput: a reconstructed image x ∈R64×64×3239Table A.24: The architecture of the label prediction branch in the sparse au-toencoder for 64×64 images.Input: extracted sparse features h ∈R4×4×768from Table A.22.fc→ 2048, BN, ReLUfc→ 1024, BN, ReLUfc→ 512, BN, ReLUfc→ 128, BN, ReLUfc→ 1, ReLUOutput: the predicted label yˆweight decay 10−4, and batch size 128. Similarly, the VGG-11 network in thefiltering scheme is trained for 200 epochs with the SGD optimizer, initial learningrate 0.01 (decay at epoch 50 and 120 with factor 0.1), weight decay 10−4, and batchsize 128. The CcGAN (SVDL+ILI) training setup is consistent with work [30].A.13.3 Testing setupsIn the testing phase, we generate 179,800 fake images (200 per angle) from theno-subsampling and the proposed cDRE-F-cSP+RS methods, respectively. Thisexperiment adopts four popular evaluation metrics—(i) Intra-FID [94] is an overallimage quality metric; (ii) Naturalness Image Quality Evaluator (NIQE) [93] eval-uates the visual quality of fake images. Please note again that the visual qualityis only one aspect of image quality; (iii) Diversity measures the diversity of fakeimages; and (iv) Label Score (LS) evaluates label consistency. Quantitatively, weprefer smaller Intra-FID, FID, NIQE and Label Score indices but larger Diversityvalues.Specifically, the four metrics are computed as follows. (i) For the Intra-FIDindex, at each of the 899 angles (0.1◦− 89.9◦), we compute the FID [52] valuebetween 49 real images and 200 fake images in terms of the bottleneck feature ofthe pre-trained autoencoder. The Intra-FID score is the average FID over all 899evaluation angles. (ii) For the NIQE index, firstly we fit an NIQE model with the 49real rendered chair images at each of the 899 angles which gives 899 NIQE models.We then compute an average NIQE score for each evaluated angle using the NIQE240model at that angle. Finally, we report the average of the 899 average NIQE scoresover the 899 yaw angles. The block size and the sharpness threshold are set to 8and 0.1 respectively in this experiments. We employ the built-in NIQE library inMATLAB. (iii) For the Diversity index, at each evaluation angle, firstly we use apretrained classification-oriented ResNet-34 to predict the chair types (49 types intotal) of these 200 fake images. Then, an entropy value can be computed basedon the chair type predictions at this angle. Finally, the Diversity index is definedas the average of the entropies at all 899 angles. (iv) For the Label Score index,at each evaluation angle, firstly we ask a pretrained regression-oriented ResNet-34to predict the yaw angles of all fake image samples and the predicted angles arethen compared with the assigned angles. The Label Score value is defined as theaverage absolute distance between the predicted angles and assigned angles overall fake images, which is equivalent to the Mean Absolute Error (MAP).A.14 More Details of Experiments on UTKFace inSection 4.3.4This section describes the details of training and testing setups of our experimentson the UTKFace dataset.A.14.1 Network architecturesThe network architectures used in this experiment is similar to those in the RC-49experiment. Please refer to Section A.13 for details.A.14.2 Training setupsSimilar to the RC-49 experiment, we evaluate the quality of fake images by Intra-FID, NIQE, Diversity, and Label Score. We also train a sparse autoencoder (bot-tleneck dimension is 512), a classification-oriented ResNet-34, and a regression-oriented ResNet-34 on the UTKFace dataset. Please note that, the UTKFace datasetconsists of face images from 5 races based on which we train the classification-oriented ResNet-34. The autoencoder and both two ResNets are trained for 200epochs with a batch size 256.241A.14.3 Testing setupsIn the testing stage, we generate 60,000 fake images (1000 per age) from each sub-sampling method. Similar to experiments on RC-49, we adopt Intra-FID, NIQE,Diversity and Label Score as quantitative measures in Section A.13.3. Analogueto experiments on RC-49, four metrics can be computed accordingly. Please notethat the metric values are average over 60 ages.A.15 More Details of Experiments on CIFAR-10 inSection 5.5.1A.15.1 Experimental SetupsFor each experimental setting described in Section 5.5.1, we implement all KDmethods as follows.All classifiers in this experiment except those related to SSKD are trained for350 epochs with the SGD optimizer, initial learning rate 0.1 (decays at epoch 150and 250 with factor 0.1), weight decay 10−4, and batch size 128.To determine teacher and student models, we first train some popular classifiersfrom scratch and their test errors are shown in Table A.25. The performance ofShuffleNet V2, efficentnet-b0, and VGG11 is the worst in three settings, so theyare chosen as the student models. Oppositely, MobileNet V2, ResNet-18, andDenseNet-121 are chosen as teacher models. Please note that although VGG11 hasmuch more parameters than DenseNet-121, it has highest inference speed amongthese classifiers. Therefore, VGG11 is taken as a lightweight model in this paper.When implementing TAKD, we borrow some codes from its official imple-mentation at https://github.com/imirzadeh/Teacher-Assistant-Knowledge-Distillation. In TAKD, the precision of a good TA model is usually the av-erage of those of the teacher and student models [92]. Therefore, VGG13 ischosen as the TA model based on Table A.25. SSKD is implemented based onhttps://github.com/xuguodong03/SSKD. We use the default training setups ofSSKD at https://github.com/xuguodong03/SSKD/blob/master/command.sh.BLKD+UDA is implemented based on https://github.com/karanchahal/distiller.The setups of cGAN-KD is shown as follows. The implementation of BigGAN242is mainly based on https://github.com/ajbrock/BigGAN-PyTorch. The BigGANmodel is trained for 2000, 2000, and 6000 epochs in C-50K, C-20K, and C-10Krespectively with batch size 512. The DiffAugment is enabled in C-20K and C-10K, and we use its official implementation at https://github.com/mit-han-lab/data-efficient-gans. The strongest transformation combination (Color + Translation+ Cutout) is used in the training. cDRE-F-cSP+RS is fitted by using the setupsand codes in https://github.com/UBCDingXin/cDRE-based Subsampling cGANS. Different from [32], we use a specially designed DenseNet-121 instead ofResNet-34 to extract features for density ratio estimation. The grid search resultsfor selecting the optimal ρ in M1 is shown in Table A.26. In each setting, theoptimal ρ minimizes the average validation error of three student models.In each setting, a student model trained by each KD method is evaluated onthe 10,000 hold-out test samples of CIFAR-10. The performance of these studentmodels is reflected by test error rates, i.e., the proportion of incorrectly classifiedtest samples.Please refer to our codes for more detailed experimental setup.A.15.2 Visual Quality versus the Error Between Predicted andAssigned LabelsWe also conduct an additional study in C-50K to research the relation betweenthe visual quality of fake images and the ρ-th quantile of cross entropy betweenassigned and predicted labels, controlling the filtering threshold in M1. In thisstudy, we first generate 500,000 unprocessed fake images-label pairs. We evaluatea pre-trained DenseNet-121 (a teacher model) on these fake images to get theirpredicted labels. Then, we compute the cross entropy loss between the assignedand predicted labels of these unprocessed fake images. Given a ρ , we find thefake images with cross entropy losses smaller than the ρ-th quantile. Next, wecompute the average Naturalness Image Quality Evaluator (NIQE) [93] score ofthe selected images, which evaluates the visual quality. The smaller the NIQEscore is, the better the visual quality will be. Finally, in Fig. A.20, we plot the linegraph of Avg. NIQE versus Quantile of Cross Entropy.From Fig. A.20, we can see a significant difference between predicted andassigned labels often implies bad visual quality. This finding shows the filtering243Table A.25: CIFAR-10: Comparison on the model size, inference speed, andtest error rate between popular convolutional neural networks. Modelsize is evaluated by counting the number of learnable parameters in thenetwork. Inference speed is defined as the number of images processedby a network per second. To compute the inference speed, we evaluateeach network on 10,000 images with batch size 64 by using one RTX2080 Ti. Based on the test error rates, ShuffleNet V2, efficientnet-b0, and VGG11 are selected as student models while MobileNet V2,ResNet-18, and DenseNet-121 are selected as teacher models.Networks # Params InferenceSpeed (FPS)Test Error RateC-50K C-20K C-10KShuffleNet V2 1,263,854 11,169 7.18 11.33 17.50MobileNet V2 2,296,922 14,762 5.92 9.28 13.53efficientnet-b0 3,599,686 7,505 7.10 12.46 15.71VGG11 9,231,114 21,850 8.42 12.48 18.57VGG13 9,416,010 16,457 6.63 10.73 15.63VGG16 14,728,266 12,712 6.57 10.75 16.15ResNet-18 11,173,962 9,883 4.88 9.26 16.07ResNet-50 23,520,842 2,693 4.88 9.72 16.81DenseNet-121 6,956,298 1,999 4.47 8.52 14.63module’s necessity in the cGAN-KD framework.A.16 More Details of Experiments on Tiny-ImageNet inSection 5.5.2A.16.1 Experimental SetupsFor each experimental setting described in Section 5.5.2, we implement all KDmethods as follows.All classifiers in this experiment except those related to SSKD are trained for350 epochs with the SGD optimizer, initial learning rate 0.1 (decays at epoch 150and 250 with factor 0.1), weight decay 10−4, and batch size 128.To determine teacher and student models, we first train some popular classifiersfrom scratch and their test errors are shown in Table A.27. The performance of244Table A.26: CIFAR-10: The grid search results for selecting the optimal ρof M1 in three settings. The optimal ρ in each setting is selected byminimizing the average validation error rate of three student models.The values of ρ range from 0.3 to 0.9. In C-10K, we have very fewfake samples after filtering with ρ = 0.3, so we skip 0.3.C-50KNetworks 0.3 0.4 0.5 0.6 0.7 0.8 0.9VGG11 6.84 6.40 6.90 6.83 6.78 6.49 6.65ShuffleNet V2 6.91 6.93 6.80 6.95 6.54 6.62 6.35efficientnet-b0 6.76 6.43 6.55 6.46 6.07 6.53 6.13Average 6.84 6.59 6.75 6.75 6.46 6.55 6.38C-20KNetworks 0.3 0.4 0.5 0.6 0.7 0.8 0.9VGG11 10.60 11.38 10.33 10.12 9.85 9.97 11.32ShuffleNet V2 10.72 10.05 10.00 9.30 9.60 9.97 9.92efficientnet-b0 10.85 10.85 10.08 9.42 9.45 9.53 10.17Average 10.72 10.76 10.14 9.61 9.63 9.82 10.47C-10KNetworks 0.3 0.4 0.5 0.6 0.7 0.8 0.9VGG11 NA 12.95 11.45 11.65 11.20 11.10 12.85ShuffleNet V2 NA 12.65 11.85 12.00 10.95 11.80 12.70efficientnet-b0 NA 12.45 12.15 12.70 11.70 11.50 12.20Average NA 12.68 11.82 12.12 11.28 11.47 12.58ShuffleNet V2 and VGG11 is the worst, so they are chosen as the student models.Oppositely, ResNet-50 and DenseNet-121 are chosen as teacher models.We train a BigGAN model with 400,000 iterations and DiffAugment is alsoused to stabilize the GAN training. A specially designed DenseNet-121 is used toextract features in the implementation of cDRE-F-cSP+RS. The grid search resultsfor the ρ selection is shown in Table A.28.Please refer to our codes for more detailed training and testing setups.245Table A.27: Tiny-ImageNet: Comparison on the model size, inference speed,and test error rate between popular convolutional neural networks.Model size is evaluated by counting the number of learnable param-eters in the network. Inference speed is defined as the number of im-ages processed by a network per second. To compute the inferencespeed, we evaluate each network on 10,000 images with batch size 64by using one RTX 2080 Ti. Based on the test error rates, ShuffleNetV2 and VGG11 are selected as student models while ResNet-50 andDenseNet-121 are selected as teacher models.Networks # Params InferenceSpeed (FPS)Test ErrorRateShuffleNet V2 1,458,604 9,371 44.52MobileNet V2 2,539,880 6,625 39.99efficientnet-b0 3,660,676 4,331 39.21VGG11 9,635,784 10,110 44.13VGG13 9,820,680 5,931 42.94VGG16 15,132,936 4,977 42.29ResNet-18 11,275,656 8,334 40.46ResNet-50 23,910,152 2,580 35.86DenseNet-121 7,151,048 973 35.22Table A.28: Tiny-ImageNet: The grid search results for selecting the optimalρ of M1. The optimal ρ in each setting is selected by minimizing theaverage validation error rate of two student models. The values of ρrange from 0.3 to 0.9.Networks 0.3 0.4 0.5 0.6 0.7 0.8 0.9ShuffleNet V2 42.24 41.81 41.60 41.31 42.45 42.74 42.79VGG11 40.42 40.33 40.02 40.33 41.62 42.07 42.12Average 41.33 41.07 40.81 40.82 42.03 42.40 42.46246Figure A.20: CIFAR-10: A line graph of the average NIQE versus the quan-tile of the cross entropy losses between predicted and assigned classlabels in the C-50K setting. Given the ρ-th quantile of the cross en-tropy losses of 500,000 unprocessed fake images-label pairs, we findthe fake images with the cross entropy losses smaller than or equal tothe ρ-th quantile. Then, we compute the average NIQE [93] score ofthe selected images.A.17 More Details of Experiments on RC-49 in Section5.5.3A.17.1 Experimental SetupsIn this experiment, all networks are trained for 350 epochs with the SGD optimizer,initial learning 0.01 (decays at epoch 150 and 250 with factor 0.1), weight decay10−4, and batch size 128.247To determine teacher and student models, we first train some popular networksfrom scratch and their test errors are shown in Table A.29. The performance ofShuffleNet V2, MobileNet V2, and efficentnet-b0 is the worst in three settings, sothey are chosen as the student models. Oppositely, VGG16 are chosen as teachermodels.We use the SNGAN architecture [95], SVDL+ILI, and hinge loss in the Cc-GAN training. The CcGAN model is trained for 30,000 iterations with batch size256. The rest setups are consistent with the official implementation of CcGANs[30]. We follow [32] to implement cDRE-F-cSP+RS. Specifically, the sparse au-toencoder for feature extraction is trained for 200 epochs with the Adam optimizer,initial learning rate 0.01 (decays every 50 epochs with factor 0.1), sparsity penalty10−3, and batch size 128. A 5-layer multilayer percentron is trained as the condi-tional density ratio model for 350 epochs with the Adam optimizer, initial learningrate 10−4 (decays at epoch 100 and 200), penalty 10−3, and batch size 128. TheVGG16 is used for the filtering and label adjustment. The grid search results areshown in Table A.30, where the optimal ρ in each setting minimizes the averagevalidation MAE of three student models.Please refer to our codes for more detailed training and testing setups.A.17.2 Visual Quality versus the Error Between Predicted andAssigned LabelsWe also conduct an additional study in R-25 to research the relation between thevisual quality of fake images and the ρ-th quantile of MAE between assigned andpredicted labels. In this study, we first generate 179,800 unprocessed fake images-label pairs. We evaluate a pre-trained VGG16 (the teacher model) on these fakeimages to get their predicted labels. Then, we compute MAE between the assignedand predicted labels of these unprocessed fake images. Given a ρ , we find the fakeimages with MAEs smaller than the ρ-th quantile. Next, we compute the averageNIQE [93] score of the selected images, which evaluates the visual quality. Thesmaller the NIQE score is, the better the visual quality will be. Finally, in Fig.A.21, we plot the line graph of Avg. NIQE versus Quantile of MAE.From Fig. A.21, we can see a significant difference between predicted andassigned labels often implies bad visual quality. This finding shows the filtering248Table A.29: RC-49: Comparison on the model size, inference speed, and testMAE between popular convolutional neural networks. Model size isevaluated by counting the number of learnable parameters in the net-work. Inference speed is defined as the number of images processedby a network per second. To compute the inference speed, we eval-uate each network on 10,000 images with batch size 64 by using oneRTX 2080 Ti. Based on the test error rates, ShuffleNet V2, MobileNetV2, and efficientnet-b0 are selected as student models while VGG16 isselected as the only teacher model.Networks # Params InferenceSpeed (FPS)MAER-25 R-15 R-5ShuffleNet V2 1,401,957 6,692 0.521 0.76 1.95MobileNet V2 2,404,993 7,962 0.547 0.95 1.74efficientnet-b0 4,188,669 5,029 0.985 1.00 1.86VGG11 9,505,409 10,015 0.275 0.35 0.73VGG13 9,690,305 5,906 0.257 0.32 0.64VGG16 15,002,561 4,934 0.203 0.31 0.49ResNet-18 11,255,873 8,214 0.351 0.41 0.98ResNet-50 23,779,777 2,623 0.365 0.38 0.99DenseNet-121 7,094,401 965 0.310 0.43 0.91module’s necessity in the cGAN-KD framework.A.18 More Details of Experiments on UTKFace inSection 5.5.4A.18.1 Experimental SetupsMost experiment setup are consistent with the RC-49 experiment except that weadopt the SAGAN architecture for CcGANs and all data in the training set are usedfor training. The comparison of some popular networks on UTKFace is shown inTable A.31. The grid search results for selecting the optimal ρ is shown in TableA.32.Please refer to our codes for more detailed training and testing setups.249Table A.30: RC-49: The grid search results for selecting the optimal ρ of M1.The optimal ρ in each setting is selected by minimizing the averagevalidation MAE of three student models. The values of ρ range from0.3 to 0.9.R-25Networks 0.3 0.4 0.5 0.6 0.7 0.8 0.9ShuffleNet V2 0.42 0.42 0.38 0.47 0.42 0.47 0.43MobileNet V2 0.50 0.49 0.44 0.43 0.47 0.45 0.45efficientnet-b0 0.63 0.67 0.65 0.73 0.62 0.77 0.71Average 0.52 0.53 0.49 0.55 0.50 0.56 0.53R-15Networks 0.3 0.4 0.5 0.6 0.7 0.8 0.9ShuffleNet V2 0.46 0.46 0.56 0.47 0.58 0.48 0.57MobileNet V2 0.50 0.63 0.55 0.55 0.56 0.60 0.73efficientnet-b0 0.88 0.73 0.76 0.88 0.84 0.74 0.87Average 0.61 0.61 0.62 0.63 0.66 0.61 0.73R-5Networks 0.3 0.4 0.5 0.6 0.7 0.8 0.9ShuffleNet V2 1.07 1.09 1.07 1.14 1.13 1.14 1.02MobileNet V2 1.16 1.16 1.11 1.02 1.17 1.10 1.17efficientnet-b0 1.09 1.18 1.18 1.16 1.19 1.21 1.12Average 1.11 1.14 1.12 1.11 1.16 1.15 1.10250Figure A.21: RC-49: A line graph of the average NIQE versus the quantileof the MAE loss between predicted and assigned regression labels inthe R-25 setting. Given the ρ-th quantile of the cross entropy losses of179,800 unprocessed fake images-label pairs, we find the fake imageswith the MAE losses smaller than or equal to the ρ-th quantile. Then,we compute the average NIQE [93] score of the selected images.251Table A.31: UTKFace: Comparison on the model size, inference speed, andtest MAE between popular convolutional neural networks. Model sizeis evaluated by counting the number of learnable parameters in the net-work. Inference speed is defined as the number of images processedby a network per second. To compute the inference speed, we eval-uate each network on 10,000 images with batch size 64 by using oneRTX 2080 Ti. Based on the test error rates, ShuffleNet V2, MobileNetV2, and efficientnet-b0 are selected as student models while VGG16 isselected as the only teacher model.Networks # Params InferenceSpeed (FPS) MAEShuffleNet V2 1,401,957 6,692 7.326MobileNet V2 2,404,993 7,962 7.335efficientnet-b0 4,188,669 5,029 6.143VGG11 9,505,409 10,015 5.196VGG13 9,690,305 5,906 5.338VGG16 15,002,561 4,934 5.136ResNet-18 11,255,873 8,214 5.873ResNet-50 23,779,777 2,623 5.627DenseNet-121 7,094,401 965 5.493Table A.32: UTKFace: The grid search results for selecting the optimal ρ ofM1. The optimal ρ is selected by minimizing the average validationMAE of three student models. The values of ρ range from 0.3 to 0.9.Method 0.3 0.4 0.5 0.6 0.7 0.8 0.9ShuffleNet V2 5.37 5.57 5.58 5.68 5.34 5.50 5.63MobileNet V2 5.49 5.38 5.32 5.39 5.48 5.37 5.39efficientnet-b0 4.80 4.69 4.58 4.59 4.63 4.63 4.62Average 5.22 5.21 5.16 5.22 5.15 5.17 5.21252
Thesis/Dissertation
eng
Statistics
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Welch, William J.
Wang, Z. Jane
Improved conditional generative adversarial networks for image generation : methods and their application in knowledge distillation
Text