Science, Faculty of
Computer Science, Department of
DSpace
UBCV
He, Xingzhe
Doctor of Philosophy - PhD
University of British Columbia
Object correspondence and structure play critical roles in image generation, 3D reconstruction and animation. In recent years, supervised algorithms have dramatically improved the accuracy of the learned correspondence. However, these approaches are expensive due to manual annotation and do not generalize well to new domains. We propose several methods in this dissertation on unsupervised structure learning from casually recorded images and videos. To be specific, we propose a Generative Adversarial Network (GAN)-based unsupervised keypoint detector and extend it for object part segmentation. Furthermore, we introduce a representation for unsupervised keypoints relationship estimation. We later adapted this technique for few-shot keypoint learning, depth prediction, and occlusion handling. In addition, we propose a dataset generation approach for diffusion model personalization to implicitly learn the object structure and appearance. The overarching goal of this dissertation is to make progress in the field of unsupervised object correspondence and structure learning. Our proposed methods outperform existing unsupervised methods on the established keypoint estimation and part segmentation benchmarks and paves the way for structure-conditioned generative models on more diverse datasets.
Structured Representation Learning byControlling Generative ModelsbyXingzhe HeB.Sc., Xi’an Jiaotong-Liverpool University, 2017B.Sc., The University of Liverpool, 2017M.Sc., Rutgers, The State University of New Jersey, 2019A THESIS SUBMITTED IN PARTIAL FULFILLMENT OFTHE REQUIREMENTS FOR THE DEGREE OFDOCTOR OF PHILOSOPHYinThe Faculty of Graduate and Postdoctoral Studies(Computer Science)THE UNIVERSITY OF BRITISH COLUMBIA(Vancouver)March 2024© Xingzhe He 2024The following individuals certify that they have read, and recommend to the Faculty ofGraduate and Postdoctoral Studies for acceptance, the thesis entitled:Structured Representation Learning by Controlling Generative Modelssubmitted by Xingzhe He in partial fulfillment of the requirements forthe degree of Doctor of Philosophyin Computer Science .Examining Committee:Helge Rhodin, Assistant Professor, Department of Computer Science, UBCSupervisorLeonid Sigal, Associate Professor, Department of Computer Science, UBCSupervisory Committee MembersKwang Moo Yi, Assistant Professor, Department of Computer Science, UBCSupervisory Committee MembersDavid Poole, Professor, Department of Computer Science, UBCUniversity ExaminerTrevor Campbell, Associate Professor, Department of Statistics, UBCUniversity ExaminerAndrea Vedaldi, Professor, Department of Engineering Science, University of OxfordExternal ExamineriiAbstractObject correspondence and structure play critical roles in image generation, 3D reconstruc-tion and animation. In recent years, supervised algorithms have dramatically improved theaccuracy of the learned correspondence. However, these approaches are expensive due tomanual annotation and do not generalize well to new domains. We propose several meth-ods in this dissertation on unsupervised structure learning from casually recorded imagesand videos. To be specific, we propose a Generative Adversarial Network (GAN)-basedunsupervised keypoint detector and extend it for object part segmentation. Furthermore,we introduce a representation for unsupervised keypoints relationship estimation. We lateradapted this technique for few-shot keypoint learning, depth prediction, and occlusionhandling. In addition, we propose a dataset generation approach for diffusion model per-sonalization to implicitly learn the object structure and appearance. The overarching goalof this dissertation is to make progress in the field of unsupervised object correspondenceand structure learning. Our proposed methods outperform existing unsupervised methodson the established keypoint estimation and part segmentation benchmarks and paves theway for structure-conditioned generative models on more diverse datasets.iiiLay SummaryLearning object correspondence and structures is one of the core intelligences humanshave. It helps humans understand and interact with the world. The correspondenceis also important to many computer vision applications, such as image generation, and3D reconstruction. However, the modern machine learning methods require expensiveannotation on large amount of images. Furthermore, they struggle to adapt to newsituations. Manually labeling correspondence, is more challenging and time-consumingthan other machine learning tasks, such as classification. The dissertation aims to developmethods that automatically learn object correspondence from images on the internet. Thechallenge lies in dealing with variations in appearance, pose, and lighting without explicitguidance. We propose an analysis-by-synthesis method for object correspondence learningwithout using any annotations. We further extend the method to consume a few annotatedexamples for customized structure learning. In addition, we propose a novel method toimplicit learn the object identity.ivPrefaceThe written content of this dissertation is the original work of Xingzhe He under the gratefulguidance by Helge Rhodin, Bastian Wandt, Gaurav Bharaj, and Pablo Garrido. Hardwareresources were generously provided by Compute Canada and UBC ARC Sockeye. Thisdissertation is based on the following publications, for all of which, the contributions ofeach author are listed.Xingzhe He, Bastian Wandt, and Helge Rhodin. Latentkeypointgan: Controlling gansvia latent keypoints. In Proceedings of the IEEE/CVF Conference on Computer Visionand Pattern Recognition Workshop, 2022 [93]. This work is covered in Chapter 3. Theinitial idea emerged from prior work in the group using autoencoders. I was responsible fordeveloping the concrete algorithm and conducting experiments, in which Helge Rhodin andBastian Wandt provided valuable discussion. I wrote the paper draft and Helge Rhodincontributed a significant part in polishing it.Xingzhe He, Bastian Wandt, and Helge Rhodin. Ganseg: Learning to segment byunsupervised hierarchical image generation. In Proceedings of the IEEE/CVF Conferenceon Computer Vision and Pattern Recognition, pages 1225–1235, 2022 [91]. This work iscovered in Chapter 4. This second project was developed with much larger independence.The initial idea is an extension of [93] proposed by me. I contributed the majority of idea,algorithm design, paper writing, and all experiments. Helge Rhodin and I contributedequally in writing and Bastian Wandt gave valuable suggestions in paper writing andrebuttal.Xingzhe He, Bastian Wandt, and Helge Rhodin. Autolink: Self-supervised learning ofhuman skeletons and object outlines by linking keypoints. In Advances in Neural InformationProcessing Systems, 2022 [89]. This work is covered in Chapter 5. I proposed the initial idea,which deviates significantly from the previous pipelines/frameworks. While the purpose isthe same, the method is significantly simplified. I contributed the majority of algorithmdesign, paper writing, and all experiments. All authors took part in writing.Xingzhe He, Gaurav Bharaj, David Ferman, Helge Rhodin, and Pablo Garrido. Few-shot geometry-aware keypoint localization. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, 2023 [86]. This work is covered in Chapter 6.Pablo Garrido proposed to extend [89] to make the outputs human interpretability andtransferable. I proposed the few-shot learning scheme and the spatial constrains that makealgorithm robust. I conducted the experiments with the help from co-authors in setting upcomputation environment. Pablo Garrido and I wrote a significant portion of the publishedvPrefacearticle and contributed equally in writing. Gaurav Bharaj designed the story line of thepaper and formats of all figures. Helge Rhodin gave valuable suggestions in the writing.Xingzhe He, Zhiwen Cao, Nicholas Kolkin, Lantao Yu, Helge Rhodin, and RatheeshKalarot. A Data Perspective on Enhanced Identity Preservation for Diffusion Personal-ization. In Submission[86]. This work is covered in Chapter 7. Zhiwen Cao proposed todo research on the topic of diffusion personalization. I proposed the learning scheme andthe data generation method that make algorithm robust. I conducted the experiments andwrote the published article. Zhiwen Cao and Nicholas Kolkin participated actively in theidea formulation. All co-authors gave valuable suggestions in the writing.viTable of ContentsAbstract . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . iiiLay Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ivPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vTable of Contents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . viiList of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiList of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiiAcknowledgements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xivDedication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52.1 Self-supervised Structure Learning . . . . . . . . . . . . . . . . . . . . . . . 52.2 Semi-supervised Structure Learning . . . . . . . . . . . . . . . . . . . . . . 62.3 Self-supervised Spacial Image Editing . . . . . . . . . . . . . . . . . . . . . 73 LatentKeypointGAN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113.3 Algorithm Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133.4 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.4.1 Keypoint Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.4.2 Spatial Embedding Layer . . . . . . . . . . . . . . . . . . . . . . . . 163.4.3 Image Generator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163.4.4 Loss Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.5 Results and Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173.5.1 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18viiTable of Contents3.5.2 LatentKeypointGAN variants . . . . . . . . . . . . . . . . . . . . . . 183.5.3 Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183.5.4 Benchmarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.5.5 Interactive Editing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203.5.6 Editing Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213.5.7 Disentangled Representations . . . . . . . . . . . . . . . . . . . . . . 233.5.8 Generalization to Diverse Datasets . . . . . . . . . . . . . . . . . . . 243.5.9 Ablation Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243.5.10 Keypoint Consistency and Interpretability . . . . . . . . . . . . . . . 253.6 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . 283.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284 GANSeg . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324.3.1 Assumptions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 324.3.2 Theoretical Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 324.3.3 Level 1: Point Generation and Part Scale . . . . . . . . . . . . . . . 354.3.4 Level 2: From Points to Masks . . . . . . . . . . . . . . . . . . . . . 374.3.5 Level 3: Mask-conditioned Image Generation . . . . . . . . . . . . . 394.3.6 Training Objective . . . . . . . . . . . . . . . . . . . . . . . . . . . . 434.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 454.4.1 Datasets and Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . 454.4.2 Baselines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.4.3 Qualitative analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 474.4.4 Part Center Consistency . . . . . . . . . . . . . . . . . . . . . . . . . 484.4.5 Mask Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484.4.6 Image Quality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484.4.7 Embedding Visualization . . . . . . . . . . . . . . . . . . . . . . . . 494.4.8 Ablation Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494.5 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . 534.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 555 AutoLink . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 565.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595.3.1 Assumptions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595.3.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 595.3.3 Image Structure Representation . . . . . . . . . . . . . . . . . . . . . 59viiiTable of Contents5.3.4 Image Reconstruction . . . . . . . . . . . . . . . . . . . . . . . . . . 605.3.5 Applications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 615.3.6 Network Architecture . . . . . . . . . . . . . . . . . . . . . . . . . . 625.3.7 Implementation Details . . . . . . . . . . . . . . . . . . . . . . . . . 625.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635.4.1 Datasets and Evaluation Metrics . . . . . . . . . . . . . . . . . . . . 635.4.2 Qualitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . 665.4.3 Quantitative Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . 665.4.4 Ablation Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 675.5 Limitations and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . 705.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 716 Few-shot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 736.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 736.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 766.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 776.3.1 Assumptions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 786.3.2 Keypoint and Uncertainty Detection . . . . . . . . . . . . . . . . . . 786.3.3 2D Few-shot Supervision . . . . . . . . . . . . . . . . . . . . . . . . . 786.3.4 2D Geometric Constraints . . . . . . . . . . . . . . . . . . . . . . . . 786.3.5 3D Aware Geometry Constraints and Uplifting . . . . . . . . . . . . 796.3.6 Geometry-aware Image Reconstruction . . . . . . . . . . . . . . . . . 796.3.7 Coefficients and Few-shot Examples Chosen . . . . . . . . . . . . . . 816.3.8 Implementation details . . . . . . . . . . . . . . . . . . . . . . . . . . 816.4 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 816.4.1 Datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 826.4.2 Qualitative Comparisons . . . . . . . . . . . . . . . . . . . . . . . . . 846.4.3 Quantitative Comparisons . . . . . . . . . . . . . . . . . . . . . . . . 846.4.4 Comparison with DatasetGAN . . . . . . . . . . . . . . . . . . . . . 846.4.5 Comparison on LSP Dataset . . . . . . . . . . . . . . . . . . . . . . 846.5 Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 846.6 Limitations. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 886.7 Conclusion & Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 907 DataPerspective . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 917.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 917.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 937.3 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 947.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 997.4.1 Result Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . 997.4.2 Model Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102ixTable of Contents7.5 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1047.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1048 Conclusion and Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . 1068.1 Ethics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1068.2 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1078.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1088.3.1 Utilizing Pre-trained Models . . . . . . . . . . . . . . . . . . . . . . 1088.3.2 Semi-supervised Learning via Dataset Generation . . . . . . . . . . . 1098.3.3 Combination with Human Annotation . . . . . . . . . . . . . . . . . 1098.3.4 Scene-Level Keypoints for Intuitive Editing . . . . . . . . . . . . . . 1108.3.5 Self/Weakly/Semi-supervised 3D Reconstruction . . . . . . . . . . . 110Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112xList of Tables3.1 Feature table comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123.2 Survey results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233.3 Portrait image generation and editing quality . . . . . . . . . . . . . . . . . 243.4 Quantitative ablation on keypoint localization . . . . . . . . . . . . . . . . . 274.1 Landmark detection on CelebA . . . . . . . . . . . . . . . . . . . . . . . . . 424.2 Foreground-Background segmentation . . . . . . . . . . . . . . . . . . . . . 434.3 Part segmentation on Taichi . . . . . . . . . . . . . . . . . . . . . . . . . . . 434.4 Penalty coefficients . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444.5 Quantitative ablation tests on number of parts . . . . . . . . . . . . . . . . 514.6 Ablation tests on number of points per part . . . . . . . . . . . . . . . . . . 524.7 Quantitative ablation tests on number of parts . . . . . . . . . . . . . . . . 535.1 Landmark detection on CelebA . . . . . . . . . . . . . . . . . . . . . . . . . 655.2 Landmark detection on Human Body . . . . . . . . . . . . . . . . . . . . . . 665.3 Landmark detection on CUB Birds . . . . . . . . . . . . . . . . . . . . . . . 675.4 Ablation tests on the numbers of keypoints and edge thickness . . . . . . . 705.5 Ablation tests on masking ratio and patch size . . . . . . . . . . . . . . . . 705.6 Ablation tests on variants of edge heatmap generation . . . . . . . . . . . . 716.1 Quantitative comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 856.2 NME(%) on Datagen 3D faces . . . . . . . . . . . . . . . . . . . . . . . . . 866.3 Quantitative comparison with DatasetGAN . . . . . . . . . . . . . . . . . . 866.4 Quantitative comparison on LSP . . . . . . . . . . . . . . . . . . . . . . . . 866.5 Ablation tests on WFLW and ATRW . . . . . . . . . . . . . . . . . . . . . . 897.1 Best #iterations of datasets in DreamBench. . . . . . . . . . . . . . . . . . 987.2 Evaluation on the DreamBench. . . . . . . . . . . . . . . . . . . . . . . . . . 1007.3 Name change. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100xiList of Figures3.1 Teaser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153.3 Location and scale editing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193.4 Our Correlation-based part Disentanglement (CPD) metric . . . . . . . . . 213.5 Supported editing features . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223.6 Editing on Bedroom . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.7 Editing on BBC Pose . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263.8 Out-of-distribution positions . . . . . . . . . . . . . . . . . . . . . . . . . . . 274.1 Teaser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304.2 Intuition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334.3 Hierarchical generation overview . . . . . . . . . . . . . . . . . . . . . . . . 354.4 Level 1 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364.5 Level 2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 374.6 Level 3 - Part I . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 384.7 Level 4 - Part II . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394.8 Detection examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 404.9 Mask quality comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . 414.10 SPADE ResBlock. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 444.11 Integrated network architecture . . . . . . . . . . . . . . . . . . . . . . . . . 464.12 Embedding visualization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 494.13 Ablation tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 504.14 Ablation test on estimated σ versus fixed σ . . . . . . . . . . . . . . . . . . 514.15 Ablation test on replacing the GAN with an auto-encoder . . . . . . . . . . 524.17 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 534.16 Ablation test on the number of parts. . . . . . . . . . . . . . . . . . . . . . 545.1 Teaser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 575.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 605.3 Network architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 635.4 Qualitative comparison on detected keypoints . . . . . . . . . . . . . . . . . 645.5 Ablation tests on the number of keypoints and edge thickness . . . . . . . . 685.6 Ablation tests on masking strategy . . . . . . . . . . . . . . . . . . . . . . . 68xiiList of Figures5.7 Examples of different numbers of detected keypoints . . . . . . . . . . . . . 695.8 Ablation tests and failure cases . . . . . . . . . . . . . . . . . . . . . . . . . 716.1 Teaser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 746.2 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 776.3 Qualitative results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 826.4 Qualitative comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 836.5 Comparison with DatasetGAN . . . . . . . . . . . . . . . . . . . . . . . . . 876.6 Detection on 2D and 3D-projected face landmarks . . . . . . . . . . . . . . 896.7 Effects of incertainty and failure cases . . . . . . . . . . . . . . . . . . . . . 907.1 Teaser. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 927.2 Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 957.3 CLIP-T score on different subject names. . . . . . . . . . . . . . . . . . . . 997.4 Overfitting prevention. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1017.5 Rephrasing prompts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1027.6 Influence of adaption to living entities. . . . . . . . . . . . . . . . . . . . . . 1037.7 Color modification. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104xiiiAcknowledgementsFirst, of course, I wish to sincerely convey my profound gratitude to my Ph.D. advisor,Professor Helge Rhodin. Words seem inadequate to capture the depth of my appreciation.Without his invaluable guidance, I would not be standing here today, embarking on myPh.D. journey. Notably, during the Ph.D. application process, I inadvertently sent hima seemingly blank email (technically white text, given my use of dark mode in Chrome).Despite this unconventional communication, he responded, ultimately paving the way formy opportunity to pursue a Ph.D.I extend my sincere thanks to Professor Bo Zhu for the first ticket into the realm ofmachine learning research, and giving me the invaluable opportunity of an internship atDartmouth College, fueled solely by my passion for machine learning.I am indebted to Professor Corina Constantinescu and Professor Wei Zhu for my firstexposure to coding and statistics. It is the every first place where I got the passion ofmachine learning.I also want to thank Professor Bastian Wandt for his valuable insights and suggestionsthat significantly contributed to my first three research papers during my Ph.D. journey.My gratitude extends to my internship advisors, namely Dr. Pablo Garrido, Dr. GauravBharaj, Mr. David Ferman, Dr. Zhiwen Cao, Dr. Nicholas Kolkin, Dr. Lantao Yu, andDr. Ratheesh Kalarot. Their guidance opened doors for me to immerse myself in appliedresearch within the industry.I acknowledge the invaluable feedback from my committee members, Professor LeonidSigal and Professor Kwang Moo Yi, and the additional examiners, Professor David Poole,Professor Trevor Campbell, and Professor Andrea Vedaldi, who provided valuable insightsfor my dissertation.I express my gratitude to my current and former labmates, including Olivia Perryman,Daniel Ajisafe, Shih-Yang Su, Chunjin Song, Luis Bolanos, Ahmad Khajehnejad, AndrewEvans, Frank Yu, Paritosh Parmar, and Blaise Appolinary, for engaging in fruitful researchdiscussions.Special appreciation goes to Professor Shiying Xiong, whose relentless commitment toour research in Professor Bo Zhu’s lab allowed us to successfully complete projects andpublish them, despite that I am fully occupied by my Ph.D. life.xivDedicationFirst, of course, to my parents. Without them, no me.More importantly, I want to extend a special thanks to my wife, Jingyi He. Herunwavering support sustained me through countless nights sacrificed for conference deadlines.Finally, to anyone reading this, thank you very much for your time and attention.xvChapter 1IntroductionHumans have a remarkable ability to understand the correspondence between objects ofthe same category by simply observing them [72]. This constitutes a fundamental aspect ofintelligent behavior, allowing us to comprehend, utilize, and innovate with tools [72]. Objectcorrespondence and structure also play a critical role in digitizing real-world observations incomputer vision applications, such as image generation and 3D reconstruction. Additionally,these aspects are significant in fields like computer graphics and the film industry, whereanimating virtual characters and mapping the movement of real-world objects are essential.In recent years, researchers have been developing supervised algorithms that utilizeannotated object correspondence and structures. These methods have significantly improvedthe accuracy of learned correspondence and have diverse applications, such as pose transferand 3D reconstruction. Pose transfer involves using the learned correspondence to guidecharacters into novel poses that have not been performed by the characters before [11, 45, 159,180, 182, 192, 262, 279]. The advancement of pose transfer techniques holds the potentialto revolutionize the film industry by seamlessly transferring realistic and expressive posesbetween characters, thereby enhancing the efficiency of animation production and elevatingthe overall quality of visual storytelling. In 3D reconstruction, correspondences are typicallyused as a loss function for improved 3D shape estimation [47, 156, 172, 220, 237, 318] or asinput signals [269–271].However, these approaches are expensive due to the requirement for manual annotation,and they do not generalize well to new domains. The annotation of spatial correspondence,as highlighted in [164], is considerably more costly and challenging than other classicalmachine learning tasks, such as image classification [245] and text description [251, 252],for which existing annotated data is two or three orders of magnitude larger than thatavailable for object correspondence. In computer vision, object correspondence is typicallydelineated by keypoints and part masks that represent distinctive and consistent featuresof the object, such as eyes on a face and hands on humans. Annotating correspondencedemands more clicks, additional time for shape understanding, and sometimes even expertknowledge. More detailed annotations are necessary whenever additional information isrequired. In addition to 2D correspondence, 3D annotations usually require spatial sensors[106, 188] and are typically impossible to annotate on images as a postprocess. Whilehumans intuitively understand spatial structure, supervised algorithms necessitate explicitlabels and lack the capacity to intelligently learn from unlabeled examples.The goal of this dissertation is to advance the field of unsupervised object correspondence1Chapter 1. Introductionand structure learning, focusing on algorithms that emulate human intelligence in extractingstructure from unstructured images. Our aim is to develop accurate and robust methodscapable of autonomously learning object correspondence and structure by observing casuallyrecorded images and videos, which are abundant on the internet. The objective is to extractobject structure information, such as keypoints and part masks, from these unstructuredand unconstrained online images and videos without the need for explicit external su-pervision. We have developed algorithms for different representations of correspondence,including keypoints, skeletons, masks, and implicit representations. Keypoints offer a sparserepresentation, providing limited control but allowing users to intuitively edit the image.Skeletons provide linkage information, which is more valuable in downstream tasks, such aspose transfer. Masks offer dense and detailed spatial information and are highly desirablefor unsupervised learning due to its difficulty in annotation.However, acquiring such artificial intelligence through the learning of correspondenceand structure without explicit external supervision is a challenging task. First, objectsin casually recorded images and videos can exhibit a wide range of appearances andlighting conditions, making it difficult to establish correspondence without understandingthe semantic meaning of every part of the object. Second, other scene elements may occludethe object, significantly interfering with performance. The model needs to comprehendnot only the object itself but also its relationship and interaction with other elements inthe images. Lastly, objects may be occluded by themselves due to a large viewpoint range,requiring the model to understand the objects in 3D. While these challenges have not beenfully addressed in supervised methods, they demand additional efforts and attention in thecontext of self-supervision.Previous self-supervised methods [103, 112, 173, 257, 288, 324, 341, 366] are primarilybased on the assumption that the image transforms with the same transformation appliedto the spatial representation, such as keypoints and masks. Modeling transformationequivariance allows them to capture the basic spatial structure of the image. However, if themodel is trained without the assistance of foreground masks, they may mistakenly modelthe background instead of focusing on the foreground. They may also face challenges indealing with large pose variation. Additionally, these methods employ multiple losses thatrequire careful balancing. In contrast, our approach aims to achieve simple yet effectivemethods using a single loss, which simplifies and stabilizes the training process.In this dissertation, we propose three unsupervised methods for learning correspondenceusing solely static images from the internet. Furthermore, we introduce a few-shot approachthat combines our unsupervised methods with annotated examples, reducing the necessaryamount of annotation while achieving performance comparable to that of supervised methods.This method also learns 3D information from only a few 2D annotations. Additionally, wepropose a novel approach to fine-tune diffusion models, implicitly learning the shape andappearance of objects and applying them in newly generated images.Our proposed methods consistently advance the state-of-the-art performance in estab-lished benchmarks for keypoint estimation, part segmentation, and identity preservation.2ContributionsNotably, our methods demonstrate robustness against extreme poses and unseen identi-ties. By successfully disentangling shape and appearance in an unsupervised manner, ourapproaches empower downstream tasks such as pose transfer and conditional generativemodels. We believe that our method represents a step towards general artificial intelligenceas it learns structure from unlabeled images. Despite focusing only on geometric structure,it paves the way for future work to learn long-range effects from video observation.The contributions outlined in this dissertation have the potential to facilitate a variety ofpositive applications. However, it is crucial to acknowledge that they also carry the risk ofbeing exploited for malicious activities. The examination of these risks and considerationsregarding ethical implementation will be addressed in Chapter 8.ContributionsThe main contributions of Chapter 3 ([93] published as best paper at CRV 2023) include:• A Generative Adversarial Network (GAN) that learns to generate object keypointsand images simultaneously from only image sets.• A dataset-generation framework for unsupervised keypoint learning.• Intuitive Editing by dragging, adding and removing keypoints.The main contributions of Chapter 4 ([91] published at CVPR 2022) include:• A GAN that learns to generate object part masks and images simultaneously fromonly image sets.• A hierarchical generation process that utilizes the unsupervised keypoints.• The first part segmentation method that does not need foreground masks or saliencymaps.The main contributions of Chapter 5 ([89] published as spotlight at NeurIPS 2022) include:• A simple yet effective framework for self-supervised keypoint learning from only imagesets.• A learnable skeleton representation for robust keypoint learning.The main contributions of Chapter 6 ([86] published at CVPR 2023) include:• Utilizing existing self-supervised methods in few-shot keypoint learning.• Keypoint uncertainty estimation along with 2D and 3D aware geometry constraintsfor better keypoint localization.3Contributions• Learning 3D keypoints from only a few 2D annotations.• Flexible modeling of complex deformable objects and geometric parts, such as mouthinterior.The main contributions of Chapter 7 ([86] in submission) include:• A simple yet effective method to fine-tune text-to-image models for a specific subject,which preserves details such as logos and text.• A dataset-generation approach that can be applied on different pre-trained text-to-image models.4Chapter 2Related WorkIn the following, we concentrate on the most relevant self-supervised structure learningmethods and generative models. To be more specific, we will introduce self-supervisedstructure learning, semi-supervised structure learning, and image editing via self-supervisedstructure. We leave the work that is only partially related to each individual chapter.2.1 Self-supervised Structure LearningSelf-supervised Keypoint Detection The most common idea for discovering keypointsin an unsupervised manner is to rely on the notion that keypoints move as the imagechanges. Various constraints have been used to enforce that the keypoints follow a knowntransformation, including view changes in a multi-view recording [234, 235, 275] or thenatural motion in videos [53, 59, 114, 137, 143, 196, 257]. When only single images areavailable, artificial image deformation is applied, either from randomized [112, 173, 288, 366]or learned [324, 341] transformations within a pre-defined deformation space. However,the learned keypoints may model the background [257, 366] and struggle with large posevariation [103] as image deformations do not separate foreground from background, which isusually tuned for each dataset and bound to be small. Most models leverage multi-branchnetwork architectures to encode the structure and appearance separately and utilize multiplelosses that need to be balanced. Recently, researchers found pre-trained diffusion modelshave strong semantic meaning [94, 201, 282], and thus can be used for self-supervisedkeypoint detection. Unlike these methods that build upon the assumption that keypointsfollow the same transformation applied to images, we propose two different approaches: 1)using Generative Adversarial Networks (GAN) to generate keypoint-image pairs in Chapter4; and 2) reconstructing masked images through linked keypoints in Chapter 5.Self-supervised Part Segmentation This task aims at pixel-level masks for multipleparts of an object, including separating the foreground from the background without maskannotation. Collins et al. [41] use matrix factorization to find similar parts in images, butit requires test images at training time, making it computationally prohibitive. Hung et al.[103] draw lessons from unsupervised keypoint detection and extend them to predict thepart segmentation masks of an object using various loss functions that preserve the geometryand semantic consistency of the masks. However, it needs off-the-shelf saliency maps orground truth background masks. Liu et al. [169] alleviate the need for the background mask,52.2. Semi-supervised Structure Learningbut they use a central prior to constrain the object mask to the center of the image, whichcan introduce a constraining bias. For example, in portrait images, the hair is often notmasked. Temporal information can be used [68, 261] to achieve better segmentation results.In comparison to all of these approaches, our models in Chapter 4 use less information(single images without video or saliency map) yet outperform them on most establishedmetrics and datasets.2.2 Semi-supervised Structure LearningSemi-supervised Keypoint Localization We draw a line between semi-supervisedkeypoint localization, where hundreds or thousands of image annotations is needed, andfew-shot keypoint localization, where the number is typically restricted to dozens. Qian etal. [227] transfer the style of labeled images to augment the face appearance distributionof the training set. Dong and Yang [52] propose a teacher network to select pseudo labelsgenerated by student networks. The methods mentioned above handle limited head posevariations. Wang et al. [304] extend the pseudo-label idea to the general image distributionby exploiting reinforcement learning. However, training a reinforcement learning model canbe unstable and computationally consuming. Additional information, such as classificationlabels [99, 296], multi-view constraints [62], and video frames [223], can aid semi-supervisedlearning but is not always available for all datasets. Mathis et al. [186] fine-tune a poseestimation network [104] on hundreds of labeled images for constrained animal pose detectiontasks under simple laboratory conditions. Similar to our method, Moskvyak et al. [197]directly learn from unlabeled images and a few annotated examples in an end-to-end manner.They impose equivariance constraints between images and keypoints, ensuring that featuresextracted at keypoints remain invariant to linear transformations. These semi-supervisedmethods still require a labeled dataset one or two orders of magnitude higher than that ofour approach in Chapter 6 and cannot generate 3D keypoints like ours.Few-shot Keypoint Localization Most existing few-shot keypoint localization methodsfocus on specific domains, primarily faces [21, 316], or medical X-ray images [32, 351, 354,373]. Browatzki et al. [21] pre-train an auto-encoder on millions of faces and regressthe feature maps to heatmaps to generate keypoints. Wei et al. [316] fine-tune a pre-trained face landmark detector with a few examples of a new template to generate customkeypoint locations. However, a large clean dataset or a well-trained keypoint detector maynot be available for all object categories, limiting the scope of their applications. In themedical domain, X-ray images often share common appearances and viewpoints. Thus,state-of-the-art methods constrain 2D keypoint deviations [32] and the features extracted atkeypoint locations [351] or resort to unsupervised registration [354] to adapt the keypointsfrom the few-shot examples to the unlabeled images. Although these approaches showeffectiveness for X-ray images, they may not be applicable to images with larger viewpoint62.3. Self-supervised Spacial Image Editingand appearance variations. Instead of targeting a single domain, our proposed methodin Chapter 6 addresses more general object distributions. Furthermore, our method inChapter 6 lifts to 3D keypoints from only a few 2D annotations.Semi-supervised Part Segmentation Generative models, such as GANs [126, 128] anddiffusion models [96], learn generators with rich semantic spatial information. Consequently,such generators are widely used as powerful semantic priors for semi-supervised partsegmentation [79, 149, 292, 347, 368]. As demonstrated in [368], segmentation masks canbe easily adapted for keypoints using one-hot binary masks or heatmap representations. Asthese methods involve labeling generated images, which are susceptible to artifacts and noise,user annotations may be inaccurate, resulting in imprecise estimated labels [368]. Moreover,methods trained on generative image-label pairs are confined to the data distribution andbiases of the underlying image generator model [212, 285]. While these approaches canachieve comparable performance to ours in Chapter 6 in terms of 2D localization accuracy,they cannot provide any 3D information.2.3 Self-supervised Spacial Image EditingPose Transfer by Self-supervised Keypoint One of the most common applications ofself-supervised keypoints is pose transfer. Since most self-supervised keypoint methods followthe “analysis by synthesis” approach [53, 59, 112, 114, 137, 143, 173, 196, 257, 288, 324, 341,366], they typically incorporate a generator in the model. The generator utilizes the learnedkeypoints and appearance information to reconstruct the original image. Consequently, itbecomes possible to naturally generate a novel pose and obtain a new image with the originalappearance and the new pose by adjusting the keypoints. Particularly, if each keypointcontains unique appearance information, users can also modify the keypoint appearancefeature to achieve local editing [366]. However, these approaches have limitations. Theycannot add or remove keypoints for more flexible control, and they struggle to handle largepose variants due to the manually designed transformations during training. Our methodsin Chapters 3 and 5 address these shortcomings.Spatial Editing with Features from Pre-trained Models Pre-trained models canprovide useful feature maps for editing, such as StyleGANs [127, 128], and DINO [25, 211].Collins et al. [42] propose clustering regions in the feature maps of StyleGAN with similaractivation and editing images by exchanging clusters. However, this requires humansupervision to select suitable clusters and can lead to artifacts since the network was nottrained for this editing task. In a similar vein, Kwon et al. [146] manipulate attention mapsto edit the pose and local shape but do not demonstrate local appearance editing, such asexchanging lip color while keeping the eyes unchanged. Furthermore, researchers proposedragging a point in the feature map for intuitive spatial editing [166, 198, 215, 256]. In72.3. Self-supervised Spacial Image Editingaddition to using generative models, Tumanyan et al. [294] utilize DINO [25] feature mapsfor editing. Instead of using existing feature maps, Alharbi et al. [5] and Kim et al. [136]train spatially-variable maps with structured noise generators, improving feature editingat inference. However, these methods usually require ’painting’ source and target featureregions by hand for every instance and cannot add, remove, or move objects by modifyingsparse keypoints, as our methods in Chapter 3 can.Personalization of Pre-trained Diffusion Models While one way to learn objectstructure is to understand explicit structures, such as keypoints and masks, an alternativeapproach is to do so implicitly by fine-tuning a pre-trained generative model, such asdiffusion models [97, 225, 238], to overfit a single object. After fine-tuning on a few imagesof the same object, the model can generate novel images of the object. However, due tothe limited number of given images, the model tends to either underfit the object detailsor overfit to the training images, losing the ability to generate novel images. Addressingthe problem of underfitting the object, Hao et al. [81] propose injecting a reference imagefeature map into attention modules to preserve identity. Chen et al. [28], on the other hand,embark on the task of disentangling subjects from backgrounds, employing a CLIP imageencoder for encoding backgrounds [229]. To combat the issue of overfitting, researchershave explored diverse strategies, fine-tuning different decomposition parts of the model[78, 100, 145, 228, 266, 287, 299]. To sidestep the time-consuming process of fine-tuningthe diffusion model, several encoder-based methods have emerged, involving the training ofan additional encoder for all subjects [12, 67, 150, 244, 255]. These encoder-based methodsoften yield increased diversity in generated content, albeit with some trade-offs in identitypreservation compared to fine-tuning-based approaches. Overall, these approaches cannotpreserve the details of the subject like our method in Chapter 7. Besides, they may belimited to a single architecture or require careful hyperparameter tuning to transfer toanother diffusion model, while our dataset-generation approach can be applied to differentarchitectures.8Chapter 3LatentKeypointGAN3.1 IntroductionIt is a long-standing goal to build generative models that produce photo-realistic imagesdriven by intuitive user input.While photo-realism is already reached for well-constrained domains, such as portraits,it remains challenging to make this image generation process interpretable and editable.Desired is a latent space that disentangles an image into parts and their appearances, whichwould allow a user to re-combine and re-imagine a generated portrait interactively andartistically.There are promising methods [112, 173, 288, 366] that use spatial image transformations,such as the thin-plate spline (TPS) method, to create pairs of real and deformed imagesand impose an equivariance loss to discover keypoints and object appearance embeddingsas the bottleneck of an autoencoder. Thereby, one can edit the image by moving, thekeypoints or modifying the appearance embedding. Yet, generated images lack fine detail.On the other hand, GAN-based editing approaches attain impressive image quality but lacksome editing capabilities. Methods such as StyleGAN [2, 42, 127, 129] and SPADE [218]enable the mixing of appearance properties from different faces and to synthesize naturalimages by ’painting’ source and target regions in the feature space. While powerful, theirediting on feature maps makes it difficult to faithfully reposition image parts spatially as theinfluence region of the mask has to be drawn manually with pixel-level accuracy. Our goalis user-friendly control via automatically learned keypoints providing handles analogousto how character rigs are keyframed in classical animation, thereby overcoming manualdrawing and applying to domains without semantic labels.Inspired by the control of autoencoder-based techniques and by the improved imagequality of GANs, we introduce keypoint locations and associated feature embeddings aslatent variables in the generator network of a GAN. Thereby, the location and appearance ofimage parts is separated and can be controlled. Figure 3.1 shows how LatentKeypointGANenables editing the output image by changing the keypoint position, adding or removingpoints, and exchanging associated appearance embeddings locally while maintaining a highimage quality that approaches that of existing GANs.We target an unsupervised setting in that the position, extend, and appearance ofparts (image regions that share appearance and belong together) is learned from unlabelledexample images. It eases the application to new domains, where large image collections93.1. Introduction(b) Keypoints(a) Original (d) Exchanging(c) Moving (e) Removing (f) AddingFigure 3.1: Teaser. GANs can generate phot-realistic images (a) but lack local editingcapability. LatentKeypointGAN generates images with associated keypoints (a-b),which enables local editing by moving keypoints (c), exchanging appearance (d), removingindividual parts (e), and adding one or more parts (f). Our improvements are on theunsupervised learning of an interpretable latent space that disentangles pose and appearance,which makes it easy to use and applicable to diverse domains, including portraits (top row),indoor rooms (bottom row), and persons (see results section).are available but exact segmentation masks or part labels are missing. By using the GANobjective in favor of image quality, we cannot rely on the equivariance constraints that areestablished for autoencoders setups. Instead, we introduce new auxiliary objective functionsand control the flow of information in the network to re-instantiate equivariance propertiesand to disentangle pose and appearance.LatentKeypointGAN is designed as a two-stage GAN architecture that is trained end-to-end. In the first step, a generator network turns the input values sampled from a normaldistribution into 2D keypoint locations and their associated encoding. We ensure withsuitable neural network pathways that some of the encodings are correlated while othersremain independent. These generated keypoints are then mapped to spatial heatmaps ofincreasing resolution. The heatmaps define the position of the keypoints and their supportsets the influence range of their respective encodings. In the second step, a SPADE-like[218] image generator turns these spatial encodings into a complete and realistic image.Although entirely unsupervised, the learned keypoints meaningfully align with the imagelandmarks, such as a keypoint linked to the nose when generating images of faces, enablingthe desired editing. As a byproduct, we can learn a separate keypoint detector on generatedimage-keypoint pairs for unsupervised keypoint detection, which we utilize to quantifylocalization accuracy.We summarize our contributions below:103.2. Related Work1. Development of a GAN-based framework for handle-based image manipulation requiresless user input than existing techniques and that succeeds on more diverse domains;2. Design of a keypoint generator that models dependent and independent factorsexplicitly;3. A new GAN-based methodology for keypoint detection that contests establishedautoencoder methods;4. Empirical study comparing different editing methods in terms of perceptual quality;5. A new metric to compare part disentanglement across existing models.Ethics - Risks. The keypoints generated by the proposed GAN have the potential to beexploited for deep fakes, surveillance, and other manipulative activities. Furthermore, alongwith the keypoints, the GAN may preserve the appearance and unintentionally disclosedirect identity information. A more detailed discussion of the potential risks is presented inSection 8.1, within the context of the other methods introduced in this dissertation.3.2 Related WorkIn the following, we discuss variants of generative models that learn to synthesize imagesfrom a collection of examples, focusing on methods providing control on the image content.Table 3.1 summarizes the editing features of the most-related methods. The summary ofunsupervised keypoint learning is provided in Chapter 2.GANs [73] are trained to generate images from a distribution that resembles thetraining distributions. Recent approaches attain photo-realism for portrait images. [124]train a progressive growth of the GAN’s generator and discriminator. After that, [127, 129]integrate multiplicative neural network layers to build StyleGAN and StyleGAN2. StyleGANgains some control on the generated high-resolution, high-quality faces by adopting ideasfrom style transfer [69, 101] and exchanging features of hidden layers between differentsamples. We build upon these network architectures and equip them with additionalstructure.Editing entire images with GANs. More recently, efforts have been made onexploring the latent space of a pre-trained StyleGAN for image editing [111, 254]. Toallow editing real-world images, various encoders [2, 3, 74, 236, 327, 374] have been trainedto project images into the latent space of StyleGANs. These methods provide controlover the image synthesis process, such as for changing age, pose, gender, illumination andenable rig-like controls over semantic face parameters [50, 71, 284, 286] by conditioningon parametric face models [18, 156]. Compared with these methods, our model focuses ondetailed and local semantic controls. Instead of changing the face as a whole, our methodis able to change a local patch without an unwanted impact on other regions. Furthermore,113.2. Related WorkFeature Zhangetal.[366]Lorenzetal.[173]Karrasetal.[129]Collinsetal.[42]Alharbietal.[5]Kimetal.[136]Wangetal.[310]Parketal.[218]Zhuetal.[376]OursAppearance transfer (global) H# Appearance transfer (local, part-based) # H# Removing and adding parts # # # # Moving parts spatially # # # # # # # Image quality w/o edits # # H# H# H#Image quality after editing # # H# H# H# H#Training w/o part annotation (unsupervised) # # # Inference w/o manual feature region ’painting’ # # Table 3.1: Feature table comparison.We compare our method to state-of-the-art generative image editing methods / H# / # :full / partial / no support. For our task of intuitive editing, our method is a better fit thanexisting methods, though sacrificing some image quality over GANs with less control—aninevitable trade-off.our keypoints provide control handles for animation without manual rigging, making iteasily applicable to different objects and image domains.Local editing with unconditional GANs. Instead of exchanging entire feature mapsfor style editing, local modifications are possible but require care to maintain consistentand artifact-free synthesis. [42] propose to cluster regions in the feature maps of StyleGANthat have similar activation and to edit images by exchanging clusters. But this requireshuman supervision to select suitable clusters and can lead to artifacts since the networkwas not trained on this editing task. In a similar vein, [146] manipulate attention maps toedit the pose and local shape, but they do not demonstrate local appearance editing, suchas exchanging the lip color while keeping the eyes as is. Instead of using existing featuremaps, [5] and [136] train spatially-variable maps by structured noise generators, whichimproves feature editing at inference. However, this requires to ’paint’ source and targetfeature regions by hand for every instance. By contrast, our model provides a higher-levelcontrol to explicitly change the pose by modifying keypoint locations that are consistentacross images. While enabling very useful forms of editing, all of the methods above requiresome form of a pixel-level selection of regions at test time or manual cluster selection outof hundreds of candidates after training. As a result, they provide limited spatial editingcontrol. Neither of them demonstrates moving parts relative to each other or adding andremoving parts. Most existing methods that achieve part addition and removal either123.3. Algorithm Overviewuse available segmentation masks [130, 175, 176, 224, 231] or require separate datasetswith and without that part (e.g., faces with and without glasses) [293, 360]. More recentworks [204, 206] learn to rotate, add, or move entire objects in an unsupervised manner,but their setting does not work on modeling parts. In sum, our keypoint-based editing isboth more intuitive and enables additional editing capabilities.Conditioned image synthesis methods synthesize images that resemble a givenreference input, including segmentation masks [109, 218, 375], scene graphs [119, 158],surface normals [311], part labels [35], and human pose [192]. They achieve local editing bymodifying the explicit part labels [35, 119, 158] or the latent codes for a mask segment [192,218]. The closest to ours are the mask-conditioned image synthesis methods. [218] pioneeredusing spatially-adaptive denormalization (SPADE) to transfer segmentation masks to images,which we borrow and adapt to be conditioned on landmark position. To control individualaspects of faces, such as changing eye or nose shape, recent works [276, 277, 376] furthermodify SPADE to allow local editing on images. The recently proposed DatasetGAN [369]and its application EditGAN [165] significantly reduce the required number of annotatedexamples to less than a hundred. Compared with these methods, our model does not takeany kind of supervision or other conditions at training time. It is trained in a totallyunsupervised manner and therefore eases the application to new domains for which nolabels exist. Still, our method allows the landmarks to learn a meaningful location andsemantic embedding that can be controlled at test time.Multi-stage trained GANs. Researchers sometimes divide the training of GANs intodifferent stages, where different intermediate representations are usually learned for betterperformance or better controllability. The representations include surface normals [311],scene layouts [119, 158], or segmentation masks [192]. While they require explicit supervisionof the ground truth annotation, Karras et al. [124, 127] proposed using smaller images asthe intermediate representations. They progressively increase the image resolution in eachstage to generate high-resolution high-quality images, which is also adopted in our networktraining.Unsupervised landmark discovery methods aim to detect the landmarks fromimages without manual labels. Most works train two-branch autoencoders, where shapeand appearance are disentangled by training on pairs of images where one of the two ismatching while the other factor varies [112, 173, 288, 366]. By contrast, we show thatour generator can all-together disentangle the appearance and keypoint location, controlthe images locally, and provides an improved image quality. For more information on theprevious work, please refer to Chapter 2 Section 2.1.3.3 Algorithm OverviewAssumptions. We assume 1) the dataset contains more than thousands of images ofobjects from the same category, e.g., only human faces or dogs; 2) the objects are roughly133.4. Methodcropped in the center.Overview. Given a set of example images, our aim is a GAN that starting from randomnoise generates new images of the same type, e.g., indoor or portrait, and provides controlover meaningful image parts by parametrizing the image in terms of keypoint locations pand their embeddings w. We operate in the unsupervised setting, these parameters arelatent variables that are inferred from the example images without requiring labels suchas segmentation masks. Instead of direct supervision, we impose the following high-levelobjectives, either as loss functions or by channeling the information flow with tailored neuralnetwork layers.GAN objective. A discriminator network is trained to distinguish real from fake images.The generator is trained to fool this discriminator. In our setting, the generated imagesmust look realistic after editing or mixing the position and embedding of keypoints.Disentangling pose and appearance. To enable editing of parts independently, weequip every keypoint with a 2D position and a Dembed dimensional feature embedding—either component can be edited independently.Part disentanglement. The two eyes in a face naturally form separate parts, but wehave to be able to model their differences (left-right orientation) and their symmetry (sameeye color or glasses). We therefore split keypoint embeddings into shared and independentcomponents that are generated from independent noise vectors—independent user controlat inference time.Locality objective. Keypoints should have a local support, influencing only nearbypixels. We use Gaussian heatmaps as intermediate representation for turning keypoints pand their embedding w into feature maps with a limited range of influence, with its smoothform enabling variations in scale—a local but learnable support.Translation invariance of parts. Pixels around a keypoint—a part of the object—shouldmove together under editing. This is imposed by using translation-invariant convolutionallayers on the localized heatmaps.Background separation. Most images can be separated into foreground and background.To model this, we introduce a separate background ’part’ that controls those image regionsnot covered by keypoints and associated parts.143.4. MethodKeypoints coordinatesHeatmapsConststyle map MLPMLPMLPbroadcast multiplicationtoRGB toRGB toRGBKeypoint Generator Spatial embedding layer Image generatorSPADE Block SPADE Block concatFigure 3.2: Overview. Starting from noise z, LatentKeypointGAN generates keypointcoordinates, k and their embeddings w. Cruicial is how they are turned into feature mapsthat are localized around the keypoints, forming conditional maps for the image generationvia SPADE block at different resolutions. At inference time, the position and embedding ofkeypoints can be edited by the user to control the position and appearance of parts.3.4 MethodOur proposed architecture operates in two stages. Figure 3.2 shows the entire architecture.First, the keypoint generator, K, defines the spatial arrangement of parts and their em-bedding explicitly as a point cloud {pj}Kj=1 and associated point-wise feature embeddings{wj}Kj=1. Its responsibility is to learn the spatial arrangement and part-specific appearance.In the next stage, a spatial embedding layer, S, turns these sparse estimates into densefeature maps that are amendable for processing by the convolutional image generator G.The generator up-scales features into a high-resolution image. At inference time, the latentkeypoints allow one to author the keypoint location and appearance interactively. In thefollowing, we explain how the previously introduced objectives are implemented.3.4.1 Keypoint GeneratorThe keypoint generator K learns the embeddings and spatial arrangement of image parts,such as eyes, nose, and mouth for describing a portrait image. It takes three equallyshaped Gaussian noise vectors as input zkp pose, zkp app, zbg emb ∼ N (0Dnoise , IDnoise×Dnoise),where Dnoise is the noise dimension. Each vector is passed through a three-layer MLPto respectively generate the K keypoint coordinates kj ∈ [−1, 1]2, j = 1, ...,K, a globalstyle vector wglobal ∈ RDembed , and a background embedding wbg. Here K is a pre-defined hyperparameter. Crucial for the desired pose-appearance-disentanglement andpart-disentanglement is that the noise for all three factors is independent and how the153.4. Methodkeypoint embedding wj is combined with a global embedding. We found that learning aconstant factor during training that is multiplied with the varying input noise works best.Formally we write,wj = wglobal ⊗wjconst, (3.1)with ⊗ the elementwise product. The constant embedding wjconst ∈ RDembed is designed toencode the keypoint semantics, e.g., left or right eye. They are updated during the trainingbut fixed during inference. The global style vector wglobal ∈ RDembed can be regarded aslearning the correlation of parts while the global noise ensures that a different appearanceis drawn for every generated image. We show in the evaluation that this factorization intodependent and independent factors is nontrivial and improves on straightforward variantsthat directly use an MLP.3.4.2 Spatial Embedding LayerWith keypoint coordinates and embeddings generated, we now turn these point-wiseestimates into an image discretized over pixels. To be able to apply a convolutionalgenerator, which provides the highest image generation quality, we generate feature mapsSj ∈ RDembed×H×W for each keypoint j. These are low-resolution prototypes with theembedded features describing the part appearance and the position of the features localizedto the corresponding keypoint position. Thereby the subsequent convolutional generatorcombines nearby part features into a crisp image, maintaining the desired locality andtranslation invariance. Style maps are generated by multiplying the keypoint embeddingwj ∈ RDembed with a Gaussian heatmap Hj ∈ RH×W that defines the local support. For apixel pSj(p) = Hj(p)wj ,where Hj(p) = exp(−∥p− kj∥22/τ) (3.2)has Gaussian shape, is centered at the keypoint location kj , and τ controls the influencerange. We also define a heatmapHbg for the background as the negative of all keypoint maps,Hbg(p) = 1−maxj{Hj(p)}Kj=1, thereby ensuring the desired background separation. Thebackground heatmap is multiplied with the independent noise vector wbg generated fromzbg emb instead of keypoint embedding, but treated equally otherwise. Then we concatenateall K+1 style maps to S ∈ RDembed×H×W×K+1. How these are used as conditional variablesat different levels of the image generator is explained in the next section.3.4.3 Image GeneratorOur image generator G follows the progressively growing architecture of StyleGAN [127] andcombines it with the idea of spatial normalization from SPADE [218], which was designedto generate images conditioned on segmentation masks. Our generator starts from a learned4 × 4 × 512 constant matrix and keeps applying convolutions and upsampling to obtain163.5. Results and Validationfeature maps of increasing resolution. The generator increases resolution layer-by-layer withmultiple adaptive normalization layers, requiring differently-sized feature maps. We applyEquation 3.2 multiple times to generate spatial embeddings of the right size that replacethe manually annotated segmentation masks in SPADE.To improve the image quality under editing operations, we replaced the BatchNorm[105] layers in [127] with spatial adaptive normalization and removed the residual links [85]because we found it detrimental in combination with progressive training.3.4.4 Loss FunctionsAdversarial losses. We found it crucial for LatentKeypointGAN to use the non-saturatingloss [73],L(G)GAN = Ez∼N log(exp(−D(G(z))) + 1) (3.3)for the generator, and logistic loss,L(D)GAN =Ez∼N log(exp(D(G(z))) + 1)+Ex∼pdata log(exp(−D(x)) + 1),(3.4)for the discriminator, with gradient penalty [193] applied only on real data,L(D)gp = Ex∼pdata∇D(x). (3.5)Other losses such as hinge loss [218] failed.Background loss. To further disentangle the background and keypoints, and tostabilize their location, we introduce a background penalty,L(G)bg = Ez1,z2 [(1−Hbg1 )⊗ G(z1)− (1−Hbg2 )⊗ G(z2)],(3.6)where z1 and z2 share the same keypoint location and appearance input noise, and only differat the background noise input. The H1 and H2 are the background heatmaps generatedby z1 and z2. With this penalty, we expect the keypoint location and appearance do notchange with the background. The final loss for the discriminator and the generator are,respectively,L(D) = L(D)GAN + λgpL(D)gp,L(G) = L(G)GAN + λbgL(G)bg.(3.7)3.5 Results and ValidationWe compare our results on the established benchmarks to the most related methods in avariety of ways, showing that we introduce new editing capabilities while maintaining highimage quality.173.5. Results and Validation3.5.1 BaselinesGANs have the highest image quality. We compare against the four most related onesproviding editing (Tab. 3.3 bottom half), validating that we extend their editing capabili-ties while maintaining a high image quality after editing. Image translation methodscondition on a segmentation mask label, also at test time. To enable fair comparisons, wetrain a self-supervised keypoint detector (see below) to condition on test images whenevercomparing to them. We compare against three methods (Tab. 3.3 top half), demonstratingequal image editing quality and similar disentanglement while not using any labels. Au-toencoders have been tailored for keypoint localization accuracy [59, 288, 366] and donot report FID metrics. We therefore only compare their quality in the user study andfocus the quantitative comparison on localization accuracy.3.5.2 LatentKeypointGAN variantsUnless specified otherwise, we set the scale parameter τ = 0.01 and use 10 keypoints. Todetermine localization accuracy and consistency, we learn an independent self-supervisedkeypoint detector. A standard ResNet detector [329] supervised on 200,000 image-keypoint pairs generated on-the-fly by LatentKeypointGAN. We also tuned our architectureLatentKeypointGAN-tuned for precise localization by enforcing stronger translationequivariance.3.5.3 MetricsWe use the most widely used metrics, plus a new one on disentanglement and a user studyon perceived editing quality.Image quality. We use the Fre´chet inception distance (FID) [95] that compares thedistribution of generated images with the distribution of real training images in the featurespace of a pre-trained model. Scores are computed by using the Pytorch FID calculator [253]over N = 50k generated and natural images unless specified otherwise by the benchmark.Editing quality-global. We use the FIDlerp [136], computed as the FID after editing.N composite images are generated from random pairs of 500 generated images by linearlyinterpolating the latent variables (for ours embeddings and keypoint locations) of a pair,with interpolation points uniform at random.Editing quality-local. While FIDlerp captures global editing, it does not apply tochanging individual parts as incompatible parts would change the image distribution,distorting FID scores. Therefore, we compare local editing quality with a comparative userstudy.183.5. Results and ValidationMove eyesEdit hair locationScalingKeypoint exchangeFigure 3.3: Location and scale editing. The first column is the source and the lastthe target. The images in-between are the result of the following operations. First row:pushing the eye keypoint distance from 0.8x to 1.2x. Note that the marked eye keypointsin this row are slightly shifted upward for better visualization. Second row: interpolatingthe hair keypoint to move the fringe from right to left. Third row: scaling the keypointlocation and, therefore, the face from 1.15x to 0.85x. Fourth row: interpolating allkeypoint locations, to rotate the head to the target orientation.Disentanglement. Previous disentangling methods analyzed latent space trajecto-ries [127], which does not generalize across network architectures, or use the differencemagnitude between an original and edited image over manually annotated segmentationmasks [42], which is not applicable to mask-free methods, including our keypoint approach.To this end, we propose a new correlation part disentanglement (CPD) that generalizesbetter. We generate 2000 images and randomly pair them. For a model with K parts, we193.5. Results and Validationthen create K variants by exchanging part embeddings one by one. We take the differencebefore and after editing and compute the spatial correlation of parts over all 1000 pairs.Figure 3.4 shows how the off-diagonal entries of the resulting correlation matrix quantifyhow much two different parts overlap. Our CPD score captures this as one minus theaverage of the off-diagonal elements, which ranges from 1 (perfect disentanglement) to 0(no control).3.5.4 BenchmarksWe use the official test splits from five different datasets. In the following, we detail theslightly varying protocol variants that are established for each. For portrait editing we use:a resolution of 512×512 on FFHQ [127] to compare to GAN approaches; 256×256 on CelebA-HQ [376] to match FID comparison with the imge translation methods, including SEAN;and 128 × 128 on CelebA [171] for the autoencoder methods. For editing experimentson FFHQ and detection experiments on CelebA, we augment by randomly cropping thetraining images to a size of 70-100%. For both human pose experiments on BBCPose [27]and the indoor domain on LSUN Bedroom [355] the resolution is 128× 128.3.5.5 Interactive EditingOur key advancement is the conditioning of a GAN on keypoint locations. Figure 3.3demonstrates how this enables moving of eyes and hair, and scaling of the entire face byscaling all keypoint positions. Surprisingly, due to the strong disentanglement of parts, thegenerator is not limited to the number of keypoints it is trained with. The teaser shows thatremoving and adding parts is possible by adding and removing an arbitrary number of peaksin the Gaussian heatmap and associated embedding masks. Figure 3.5 compared editingcapabilities to related GANs. None of these spatial operations have been demonstrated byexisting GANs, as these focus on global and local editing via feature maps, but not on therepositioning and addition of parts as their parametrization on part position is not explicit.[374] comes closest to our editing results, they demonstrate global and local editing, andwe used their public implementation to test adding and removing. However, they requiresegmentation masks at training time, and parts cannot be moved manually; the underlyingmasks would rather need to be repainted in a consistent manner, which is not applicable tointeractive animation. Notably, the autoencoder methods offer the same editing as we dobut at much lower image generation quality (see next section).203.5. Results and ValidationOriginalApp SourceEye Editing Difference MapMouth Editing Difference MapNose Editing Difference MapCorrelation MatrixFigure 3.4: Our Correlation-based part Disentanglement (CPD) metric is computedas the sum over part correlations. From a pair of images (left) the appearance of parts isexchanged one by one, here for eyes, nose, and mouth (center). The pairwise correlation ofthe resulting differences maps (right) forms the entries of the correlation matrix on whichCPD is computed.3.5.6 Editing QualityOur goal is to minimize the inevitable drop in image quality when enabling different levels ofediting. For global editing, our FIDlerp is close to that of all but the very recent approachby [136], which however does not supports the desired positional editing.213.5. Results and ValidationDiagGANStyleMapGANGANLocalEditingSEANOursOriginal Global Local Removing Adding Movingn / an / an / an / a n / a n / an / an / a n / an / an / aFigure 3.5: Supported editing features While all of the tested methods enable globalediting, only some offer local editing of part appearance, and non of the GAN-based methods(top three) demonstrated adding new parts, removing, nor animating parts via controlhandles, as we provide.User study. Because there is no established metric for local editing and the au-toencoder methods do not provide FID scores, we additionally conducted a comparativeuser study to [366] (best autoencoder) on CelebA and [376] (most disentangled prior work)on CelebA-HQ. Participants were asked to ”Choose the image (A or B) with higher facequality” and ”regardless of image quality” choose the pair that better preserves ”facialfeatures”, ”identity”, and ”outline of the face and its parts”. The results are summerized inTable 3.2. Compared to [366], who supports the same keypoint editing as we do, our image223.5. Results and ValidationAspect Method to compare In favour of ours In favour of others Equal qualityEditing image quality [366] 92.17% 0.87% 6.96%Editing image quality SEAN [376] 94.78% 2.61% 2.61%Part disentanglement [366] 67.83% 5.22% 26.95%Part disentanglement SEAN [376] 28.69% 21.74% 49.57%Identity preservation [366] 55.65% 14.78% 29.57%Shape preservation SEAN [376] 33.91% 46.96% 19.13%Table 3.2: Survey results. We our solution to [366] (autoencoder based) and [376] (maskbased) at hand of four different aspects: editing image quality, part disentanglement, shapepreservation while changing expression, and shape preservation while changing appearance.quality is preferred drastically, by 92.17%. Also compared to, [376] which reaches a betterFID score before editing and better preserve the outlines with explicit segmentation masks(ours preferred 33.91%, vs. 46.96%), our face quality is preferred (by 94.78%) after localediting mouth/eyes, and the identity preservation is nearly the same (49.57% voted equalquality). We conclude that our edits are preferred because their masks and embeddings canbecome incompatible when exchanged while our keypoints are trained to be disentangled.For image generation, our results are in the same ballpark as the supervised imagetranslation approaches (Table 3.3 FID, top half), yet above those unsupervised ones buildingupon StyleGAN (bottom half). This loosely quantifies the cost of imposing additionaldisentanglement constraints as very similar training strategies and generator architecturesare used.3.5.7 Disentangled RepresentationsThe ability to move parts depends strongly on their disentanglement. We measure this asthe overlap of the image regions controlled by pairs of parts using the CPD score explainedin Figure 3.4. Table 3.3 compares CPD scores to all related methods that had trained modelsavailable. To compare to GANLocalEditing [42] with 8 parts, we grouped parts for SEANand ours into semantically equivalent groups and, when comparing to GANLocalEditing,removed the background as not parametrized by them. Our method scores the best acrossall unsupervised methods (0.63 CPD vs. 0.45, 0.39, 0.35) and matches that of the supervisedSEAN on their labeled dataset (0.7 CPD vs. 0.7), despite them having the advantageof conditioning on sharp segmentation masks while we only condition on self-supervisedkeypoints. This demonstrates the improvement in part disentanglement brought about byour contributions.233.5. Results and ValidationMethod Conditioned on FID ↓ FIDlerp ↓ CPD ↑Pix2PixHD [310] masks 23.69⋆ - -SPADE [218] masks 22.43⋆ - -SEAN [376] masks & image 17.66 30.29† 0.70Ours (+self-sup. detector, see Sec. 3.5.10) image 21.71 32.93 0.70StyleGAN2 [129] unsupervised 4.97 30.30† -Alharbi et al. [5] unsupervised - 27.96† -StyleMapGAN [136] unsupervised 4.72 9.97 -GANLocalEditing (ϵ = 5) unsupervised - - 0.45Collins et al. [42] (ϵ = 50) unsupervised - - 0.39(ϵ = 500) unsupervised - - 0.35Ours unsupervised 16.26 31.80 0.63Table 3.3: Portrait image generation and editing quality compared to supervisedimage translation on CelebA-HQ (top half) and to GANs on FFHQ (bottom). Our imagequality scores before (FID column) and after editing (FIDlerp) are close to the methodsin each category, only outperformed by those providing fewer editing options or usingsupervision. This editing is enabled by the improved disentanglement (CPD column). ⋆ istrained by [376] and † by [136] using official implementations.3.5.8 Generalization to Diverse DatasetsLSUN Bedroom. In Figure 3.6, we explore the editing ability of entire scenes on theLSUN bedroom dataset. No previous unsupervised keypoint-based model has tried thisdifficult task before. We successfully interpolate the local appearance by changing thecorresponding keypoint embeddings and translating the local key parts (window, bed) bymoving the corresponding keypoints. Only [42] can change the appearance of parts, butthey cannot move, remove or add individual parts; operations we support.BBC Pose. Figure 3.7 explores the editing of entire persons. Although artifacts remaindue to the detailed background and motion blur in the datasets, pose and appearance canstill be exchanged.3.5.9 Ablation TestsWe demonstrate the importance of our contributions by (1) removing the background; (2)removing the global style vector; (3) using additive global style vector instead of multi-plicative ones; (4) using contrastive learned keypoint embeddings instead of multiplicativeones; (5) removing the keypoint embedding; (6) removing keypoints. Table 3.4 showsthat all additions increase one or more of the keypoint localization (L2 error normalizedby interocular distance), FID, or CPD scores. To speedup experimentation, the FID is243.5. Results and Validationcalculated on FFHQ at resolution 256× 256, and CPD is calculated on all 10 keypointswith the background included. We also include examples in Figure 3.8, where keypointsare moved to out-of-distribution positions. Some of the parts become invisible or distortedin unrealistic positions but the remaining parts are intact and interact in a plausible waywith nearby parts. For instance, the nose can be moved up as one part and connects tothe eyebrows without distorting them. This highlights the strong disentanglement of partsand supports the numerical analysis in Table 3.3. Notably only the relative positioning ofparts influences the image, without having any bias to the absolute image center position,as shown by moving all keypoints in the teaser Figure 3.1.3.5.10 Keypoint Consistency and InterpretabilityTo be interpretable, the learned keypoints must correspond to the same semantic imageregion across different images. We test this with the detector trained on our generatedexamples (see variants paragraph), which can only succeed when parts are placed consistently.For evaluation, we follow [288] and subsequent methods: As the order and semantics ofunsupervised keypoints are undefined, a linear regressor from the predicted keypoints to the5 ground truth keypoints is learned on the MAFL training set [370]. All images are resizedto 128× 128 and the training and test set of the MAFL subset from CelebA are excludedwhen training the LatentKeypointGAN. The test error is the Euclidean distance to theground truth on the MAFL test set, normalized by the inter-ocular distance in percent.High accuracy can therefore only be obtained when the learned disentangled keypointsmove consistently with the human-annotated ones.LatentKeypointGAN strikes a low error of 5.9%, which lies between the original 8.0%by [288] and the most recent 2.8% by [59], validating the interpretability and consistentlyof the learned keypoints. Furthermore, LatentKeypointGAN-tuned variant reaches 3.3%,thereby contesting existing unsupervised keypoint detection approaches with an alternativemethodology using GANs. We would like to point out that linear regression can onlyindicate goodness of the linear interpretation of ground truth keypoints by the unsupervisedkeypoints, and it cannot represent higher order relationship, such as quadratic relationship.253.5. Results and ValidationOriginal Difference w = 0.0 w = 0.33 w = 0.66 w = 1.0Change individual keypoint embeddings Change individual keypoint locationsOriginal w = 0.0 w = 0.25 w = 0.50 Target (w=1)w = 0.75Removing bed Removing light Removing window Adding another bed Adding another bed Adding another drawingFigure 3.6: Editing on Bedroom by (top-left) interpolating the keypoint embeddingsof curtain and window and (top-right) moving bed and light; and (bottom) removing andadding objectsAppearance Pose BackgroundFigure 3.7: Editing on BBC Pose. The first row shows the source image and the secondrow the editing results. Left: the human appearance is swapped with the small targetimage. Center: changing the position to the one in the overlay. Right: changing thebackground (inset shows the difference).263.5. Results and ValidationMethod relative L2 error % ↓ FID ↓ CPD ↑adding global style vector 5.29% 42.02 0.72w/o keypoint - 34.69 -w/o keypoint embedding 22.81% 32.41 -w/o global style vector 6.76% 28.75 -contrastive keypoint embedding 7.53% 28.47 0.76w/o background 6.43% 25.67 0.71full model 5.85% 23.50 0.67Table 3.4: Quantitative ablation on keypoint localization compared to supervisedimage translation on CelebA-HQ (top half) and to GANs on FFHQ (bottom). Our imagequality scores before (FID column) and after editing L2 error normalized by interoculardistance), image quality (FID), and Correlation-based Part Disentanglement (CPD), withall contributions improving either one or more of the metrics.original keypoints edit 1) edit 2) edit 3)Figure 3.8: Out-of-distribution positions. In the first row, 1) the eyes disappear whenthe keypoints are moved outside the facial region; 2) the mouth is maintained even if thekeypoints are far from the original; 3) notably, the nose retains the shape when movedto the forehead while not influencing the neighboring eyebrows. In the second row, 1,2):moving one of the bed keypoints first enlarges the bed and then breaks it into two pieces;3) the window disappears when the keypoint moves to the ceiling.273.6. Limitations and Future Work3.6 Limitations and Future WorkOur model brings a new trade-off between the disentanglement required for editing, generalityto diverse domains, and the image quality.While successful in disentangling facial details and objects in the bedroom dataset, thehair in portraits can mix with the background and, on the other hand, locally encodedfeatures can lead to asymmetric faces, such as a pair of glasses with differently styledsides. If a different level of entanglement is desired, one could attempt to tune τ for eachapplication domain or even each keypoint. For BBC Pose, the keypoints are consistentacross images with the same pose, which permits pose transfer, but are too inconsistentfor keypoint detection, which could be overcome by linking keypoints hierarchically witha skeleton. While the face orientation in portrait images can be controlled by movingkeypoints, we found that orientation changes on the bedroom images are not reliable. Webelieve that it will be necessary to learn a 3D representation and datasets with less-biasedviewpoints.3.7 ConclusionWe present a GAN-based framework that is internally conditioned on keypoints and theirappearance encoding. By learning disentangled representations from scratch instead ofstarting from a pre-trained StyleGAN, we provide an interpretable hidden space that enablesintuitive spatial editing via control handles. This LatentKeypointGAN also facilitates thegeneration of image-keypoint pairs, thereby providing a new methodology for unsupervisedkeypoint detection that is typically addressed with autoencoders.28Chapter 4GANSeg4.1 IntroductionThis chapter tackles the problem of unsupervised part segmentation. Discovering objectparts in images is a fundamental problem in computer vision as parts provide an intermediaterepresentation that is robust to object appearance and pose variation [103, 261]. Manyhigh-level tasks benefit from part representations, such as 3D reconstruction [157, 377],pose estimation [135, 237], and image editing [93, 376]. Keypoints and part segmentationmaps are among the most commonly used forms. However, their supervised training[83, 240, 314] requires pixel-level annotations for every new application domain since labelshardly generalize to other object categories and the number of parts and their granularityvary across tasks.On the side of keypoint detection, several unsupervised detectors exist [112, 366] butsegmentation methods are still in their infancy [103, 169]. Segmenting parts without pixel-level annotation is difficult because it requires disentangling parts from other parts and theforeground from the background. Existing unsupervised1 methods mainly follow the samestrategy as applied for unsupervised keypoint detection [288]. Real images are transformedby an affine map or a thin plate spline to find those parts that are equivariant under theknown deformation. For precise reconstruction they require additional information, suchas saliency maps [103] or assume the objects to be consistently centered [169], which isconstraining. For example, when applied to face datasets [171], the neck and shoulders areoften ignored although part of almost every image.Our goal is to improve the unsupervised part segmentation task. We propose to firsttrain a generative adversarial network (GAN) [73] to generate images that are internallyconditioned on latent masks. This GAN formulation alleviates the dependency on imagepairs and pre-defined image transformation in existing autoencoder networks. In thisway, the network learns the part distribution from the dataset instead of from pre-definedimage transformations. Subsequently, we use the generator to synthesize virtually infinitemask-image pairs for training a segmentation network. Figure 4.1 provides an overview ofour model.The key question we address is how to design a GAN that generates images with partsegmentation masks that are meaningful, i.e., group pixels into regions that typically move1Most existing literature refers to unsupervised when training on single images without annotation, andself-supervised when training on auxiliary tasks using multi-views or videos. We follow this convention.294.1. IntroductionHierarchical Image GenerationPoints Part Masks Foreground ImagePart Segmentation on Real ImagesTrain Part SegmentationFigure 4.1: Teaser. A segmentation network (right) is trained on mask-image pairsgenerated by a new hierarchical image generator (left; from points, to mask, over foregroundto the image). It is unsupervised and applies to faces, persons, birds, and flowers.together and have shared appearance across images. We start from a backbone architecturethat is borrowed from supervised segmentation networks [30, 240] and the GAN strategy isinspired by its recent application to unsupervised keypoint detection [93]. Our innovationis the hierarchical generation of the image via multiple abstraction levels, including the useof masks. In the first level, we use Gaussian noise to generate part appearance embeddingsand a set of 2D latent points. Unlike [93] which continues straight from points to imagegeneration, we first group points to define the position and the scale of each part. In thesecond abstraction level, we use a part-relative positional encoding to generate 2D featuremaps and then generate the mask with a CNN. In the third level, the foreground image isgenerated from a combination of feature maps with corresponding appearance embeddings.Independently, a background image is generated with a randomized position to disentangleforeground and background. Finally the foreground is blended with the background. Thegenerated masks are used here a second time to define the blend weight.The key to the success of our GAN framework are several design choices that preservetranslational equivariance of parts [125], which are not applicable to the traditional autoen-coder approaches. As a result, by not knowing the absolute location, the convolutional304.2. Related Worknetwork is forced to condition the image purely on the spatial extent of the masks; movingthe part mask will move the image part. This is a crucial inductive bias in our unsupervisedlearning. Our contributions are threefold:1. An unsupervised GAN approach to generate mask-image pairs for training partsegmentation;2. A novel hierarchical image generator, which encourages the segmentation of meaningfulparts;3. Alleviating prior assumptions on saliency maps and object position.Ethics - Risks. The proposed GAN has the potential to be misused for deep fakes,surveillance, and other manipulative activities. In comparison with LatentKeypointGANdiscussed in Chapter 3, it offers more intricate control, thus widening the scope of potentialabuse. Additionally, the GAN may retain the appearance and inadvertently reveal directidentity information. Further elaboration on the potential risks is provided in Section 8.1,within the context of the other methods presented in this dissertation.4.2 Related WorkThe high-level relation to unsupervised structured learning is given in Chapter 2. In thefollowing, we focus on unsupervised foreground and part segmentation.Unsupervised Foreground Segmentation aims to segment foreground objects fromthe background in an unsupervised manner. Gupta et al. [75] learn a patch level maskof foreground to benefit representation learning, which, however, is only coarse. Singhet al. [265] use a multi-stage GAN to disentangle the foreground shape and texturebut they only focus on editing. Bielski et al. [17] propose to re-position the generatedforeground to disentangle it from the background. Yang et al. [344] propose ContextualInformation Separation (CIS), a general objective to segment the foreground by maximizingthe error of inpainting the mask and its complement. It was first applied to optical flow andsubsequently to color images by [131, 132, 249, 343]. Chen et al. [31] achieved unsupervisedforeground segmentation by resampling the foreground appearance to disentangle foregroundand background. Voynov et al. [300] and Yang et al. [346] introduce comparison-basedsegmentation methods using pre-trained GANs, achieving better results than generation-based methods. As pointed out by [17, 31], such unsupervised methods can easily get intotrivial solutions, where the background includes the whole foreground. To counteract, weintroduce two losses that overcome trivial solutions. Besides, we found that preservingtranslation equivariance in the network architecture can naturally mitigate trivial solutions.This is another important reason why we chose to use a GAN, as we will explain in Section 4.3and supplementary. In comparison to the above methods, this yields comparable results314.3. Methodeven if they are specialized for foreground separation while we provide more fine-grainedpart segmentation.Unsupervised Part Segmentation aims at pixel-level masks for multiple parts of anobject, including separating foreground from background [41, 103, 169]. The summary ofunsupervised part segmentation is provided in Chapter 2. In comparison to all of previousapproaches, our model uses less information (single images without video or saliency map)yet outperform these on half of the most established metrics and datasets, as we evaluatein our experiments, Section 4.4.4.3 MethodWe train a Generative Adversarial Network (GAN) [73] to generate points, part masks,foreground, background, and the image in a hierarchy. Figure 4.3 gives an overview ofour method. In a second stage, we generate mask-image pairs to train the Deeplab V3[30] segmentation network, thereby enabling unsupervised part segmentation. Our corenetwork architecture design principle is to build a hierarchy that preserves the translationequivariance of its part representations at each of the following three stages.4.3.1 Assumptions.We make the assumption that 1) the dataset comprises over a thousand images featuringobjects belonging to the same category, such as exclusively human faces or dogs; and 2) theobjects are approximately cropped and centered in the images.4.3.2 Theoretical AnalysisIn the following, we explain in detail how and why the proposed GAN architecture providestranslation equivariance. On top of the translation equivariance of convolutions, we addressequivariances in the point- and part-conditioned image generation.Intuition. Consider a greyscale image, with a single point on it. Assume the image getsdarker to the point. How should the image change if we move the point one pixel step tothe right? We illustrate it in Figure 4.2 (Left). The left part of the image w.r.t. the pointgets brighter and the right part gets darker. More specifically, if the point moves one pixelto the right, the pixels on the grid take the value of their left neighbor. The continuous case(1D for simplicity) is shown in Figure 4.2 (Middle). We assume that the point x controlsthe translation of the function f . If we move the x to x+∆t, the function f is also shiftedby ∆t. Then for each fixed position p, the value changes from f(p) to f(p−∆t). We desirea network architecture that fulfills this equivariance in the vicinity of each point.324.3. MethodMove Right(a) (b)Figure 4.2: Intuition.. (a) We move the points to the right by one pixel. The left part ofthe point (red box) becomes brighter and the right part of the point (blue box) becomesdarker. (b) Continuous 1D case. (c) Rotation of parts can be achieved by translating points.Formulation. Formally, let I ∈ RH×W denote a feature map (or an image), x ∈ [−1, 1]×[−1, 1] denote the point location, and p ∈ [−1, 1]× [−1, 1] denote the pixel location. WehaveI(p−∆t,x) = I(p,x+∆t). (4.1)If we divide both sides by ∆t (both axis independently), and let ∆t → 0, we get theequation for the motion of a single point−∂I(p,x)∂p=∂I(p,x)∂x. (4.2)This equation is the first integral of a homogeneous linear partial differential equation[332, 333]. Its solution is in the form of I(p− x), where I can be any function. In otherwords, whenever p (or x) shows up, it must be in a combination with x (or p) as p− x orx− p.If there exist multiple points, each pixel may be influenced by multiple points. AssumingK points x1, ...,xK ∈ [0, 1]2 givesIk(p−∆t,x1, ...,xK) = Ik(p,x1, ...,xk +∆t, ...,xK), (4.3)where k = 1, ...,K.We assume additivity of Ik, which models the additivity of the generated feature mapsin our convolutional generator well,I =K∑k=1Ik, (4.4)and writeI(p−∆t,x1, ...,xK) =K∑k=1Ik(p,x1, ...,xk +∆t, ...,xK). (4.5)334.3. MethodAs before, we divide both sides by ∆t (both axis independently), and let ∆t→ 0. Due tothe additivity assuption in Equation 4.5, we can write the movement of pixel value as thesummation of the resulting changes of keypoints movement,∂I(p,X)∂p= −K∑k=1∂Ik(p,X)∂xk. (4.6)where X = {x1, ...,xK} for simplicity. Similarly, the solution to this equation is in the formof I(p− x1, ...,p− xk). Therefore, in theory, if we build a neural network that takes thedifference between the points and the pixels as input, i.e., (p− x1, ...,p− xk), this equationis automatically satisfied. Hence, unlike recent trends of adding absolute coordinates tonetworks [39, 54, 108, 340], we need to remove the absolute coordinate and point informationfrom the model.Background Handling. We generate foreground and background independently asforeground and background are not additive. The background exists on all pixels of theimage, just with parts occluded by the foreground. Thus the movement of the backgroundwill not affect the foreground pixels. Mathematically, if only a non-empty subset of the Kpoints influence the pixel p, Equation 4.6 still holds because ∂I(p,x1,...,xK)∂xk = 0 if p is notinfluenced by the point xk. However, if p belongs to the background (no point affects thepixel p), the RHS becomes zero while the LHS does not (unless the background has only asingle color). Therefore, we need to separate foreground and background and blend themat the end, to be able to assume additivity only on the foreground where it is meaningful.Convolution. We use convolutions in our network since they are translation equivariant.In formulation, a neighbor of p has value I(p− x1 +∆p, ...,p− xk +∆p) where ∆p is astep to the neighbor. Note that I(p− x1 +∆p, ...,p− xk +∆p) itself can be written as afunction of (p− x1, ...,p− xK), which makes our desired Equation 4.6 hold.Why to use a GAN? A more precise question is, ”What is the advantange usingupsamling-layer-stacked GAN architecture than U-Shaped Auto-encoder architecture”.Unlike most previous work which uses auto-encoders, we first use a GAN to generate points,masks, and images, and train a segmentation network to obtain segments. A core reasonfor us is that GANs can effectively prevent leaking absolute position [125]. As pointed outby [6, 108, 134, 340], a convolution (kernel size larger than 1) with zero padding implicitlyencodes absolute grid position. The more layers and downsampling layers, the larger isthe region around the boundary that is affected by leaking absolute position [6]. However,if we were to remove the zero-padding, using only valid convolution [291], encoder willperform much worse at the boundary of images [199]. GANs, however, usually do nothave downsampling but multiple upsampling layers. If we maintain a fixed margin aroundthe feature map and crop after each upsampling [125], we can effectively prevent leaking344.3. Methodabsolute position and work with cropped and uncropped datasets. Moreover, autoencodersrequire to learn the encoder and decoder together, which implies a larger memory footprintand they generally lack behind in image generation quality.Points Positions Part Appearance BG Appearance Part Positions BG Position Embeddings Background Foreground Masks Image BackgroundGeneratorPointGeneratorMask GeneratorForeground GeneratorBlendLevel 1 Level 2 Level 3 Figure 4.3: Hierarchical generation overview. Level 1 (red): The Point Generatorconverts two Gaussian noise vectors to part position and appearance embedding. Level 2(orange): The Mask Generator turns part positions and embeddings into masks defining thepart support. Level 3 (green and blue): The Foreground Generator uses the quantitiesfrom the previous level to generate a foreground image that is finally blended with theindependently generated background.4.3.3 Level 1: Point Generation and Part ScaleIn the first level, we utilize independent noise vectors to generate the locations and ap-pearances of K parts. We found training to be most stable by first predicting nper ×Kpoints separated into K groups with nper points. The part location and scale are computedfrom the mean and standard deviation of the corresponding nper points, which regularizestraining. Figure 4.4 gives an overview of the underlying Point Generator module. Ittakes two noise vectors as input zpoint, zapp ∼ N (0Dnoise , IDnoise×Dnoise), where Dnoise is thenoise dimension. We use a 3-layer multi-layer perceptron (MLP) to map zpoint to nper ×K354.3. MethodMLPMLPPoints Group AveragePart Positions Embeddings Points Positions Part Appearance Figure 4.4: Level 1. The Point Generator uses two Gaussian noise vectors to generate partlocation, and part appearance embeddings, respectively.points {x1k, ...,xnperk }Kk=1. Then we calculate the part locations {x1, ...,xK} and part scales{σ1, ..., σK},xk =1npernper∑i=1xik, σk =√∑nperi ∥xik − xk∥2nper − 1 ,with {x1k, ...,xnperk }Kk=1 = MLPpoint(zpoint), (4.7)where k = 1, ...,K.We use another 3-layer MLP to map zapp to a part appearance vector wdynamic ∈ RDemb .Following [93], we define a constant embedding vector wconstk ∈ RDemb for each part. Wethen perform an elementwise multiplication between wdynamic and wconstk to obtain the finalpart embedding wk ∈ RDemb . That is,wdynamic = MLPapp(zapp) (4.8)wk = wdynamic ⊗wconstk , (4.9)where ⊗ is the elementwise product. It is important that the noise source for appearanceand position is independent to prevent the appearance from interfering with the locationinformation.364.3. MethodGaussain Heatmaps BroadcastPositional Encoding on Relative PositionsSPADE ResBlockPoints Part Positions Embeddings SPADE ResBlockSPADE ResBlockSPADE ResBlockMasks Figure 4.5: Level 2. The Mask Generator uses points, part locations, part scales, and partappearance embeddings to generate masks. Note that it is different from the architecturein Chaper 3, as it does not take background into consideration and produces masks insteadof images.4.3.4 Level 2: From Points to MasksIn the second level, the mask generation, we use Gaussian heatmaps to model the localindependence and positional encoding [218] to generate masks relative to the predictedpart location. We encode relative instead of absolute position between the points and theimage pixels to keep long-distance relations and to prevent leaking the absolute coordinateinformation, which would violate the translation equivariance. To further preserve thetranslation equivariance, we initialize the positional encoding in a larger grid than the realimage range and crop to a fixed margin size after each 2x upsampling [125], which preventsconvolutional layers from passing on boundary information (see Section 4.3.5 for additionaldetails).These operations are implemented with the Mask Generator illustrated in Figure 4.5.It takes the nper ×K points {x1k, ...,xnperk }Kk=1, part locations xk and part scales σK , andpart embeddings wk as input. We generate a Gaussian heatmap for each part using themean and standard deviation of each part defined in Equation 4.7. The embedding wkis then multiplied with every pixel of the corresponding heatmap, generating a spatiallylocalized embedding map. We assume the additivity of feature maps. All K part-specific374.3. MethodBroadcastPositional Encoding on Relative PositionsSPADE ResBlockEmbeddings SPADE ResBlockSPADE ResBlockSPADE ResBlockForeground Masks Part Positions Figure 4.6: Level 3 - Part I. The foreground Generator uses part locations, part appearanceembeddings, and masks.embeddings are summed to form a single feature map Wmask ∈ RDemb×H×W . Formally wewrite,Hk(p) = exp(−∥p− xk∥22/σ2k) ,Wmask(p) =K∑k=1Hk(p)wk.(4.10)Note that we use σ2k instead of 2σ2k to obtain sharper heatmaps, making it easier for thenetwork to generate sharp masks.The generated embedding map Wmask will subsequently be used to generate masks,together with the mask starting tensor M(0) ∈ RDemb×H×W . To avoid leaking absoluteposition information, we do not use a constant tensor [127] or a linearly mapped noise [218].Instead, we use low frequency positional encoding [278] of the difference between the pixelposition and the nper ×K points. That is,M(0)(p) = [ sin(πFC([p− x11, ...,p− xnperK ])),cos(πFC([p− x11, ...,p− xnperK ]))],(4.11)where FC stands for a fully connected layer without activation function followed (a linearprojection).384.3. MethodMLP Uniformly Random Background Position Positional Encoding on Relative PositionsAdaIN ConvBlockAdaIN ConvBlockAdaIN ConvBlockAdaIN ConvBlockBackground BG Appearance Figure 4.7: Level 4 - Part II. The Background Generator uses a Gaussian noise vector,and a random position to generate translation-invariant background.With the mask starting tensor M(0) and mask embedding map Wmask defined, wegenerate masks M = [Mbg,M1, ...,MK ] ∈ R(K+1)×H×W with SPADE ResBlocks [218],M(i) = SPADE ResBlock(M(i−1),Wmask)M = softmax(M(Tmask)),(4.12)where i = 1, ..., Tmask, Tmask is the number of blocks, and an additional channel is reservedfor the background.Figure 4.10 (top) illustrates the SPADE ResBlock [218]. SPADE takes two featuremaps as input. We denote the two feature maps as Finput ∈ RDinput emb×H×W andFstyle ∈ RDstyle emb×H×W . We first use BatchNorm [105] to normalize Finput followedby two convolutions to map Fstyle to the new mean β ∈ RDinput emb×H×W and new standarddeviation γ ∈ RDinput emb×H×W for the normalized Finput. Although we use BatchNormto normalize Finput in batch and channel dimension, we apply the generated β and γ todenormalize every individual element.In theory, the Batch Normalization [105] may leak absolute position information andbreak the translation equivariance. However, in practice, experiments have shown SPADEhas strong local disentanglement [93, 218, 376].4.3.5 Level 3: Mask-conditioned Image GenerationIn the third level, we generate the foreground and the background separately and blendthem linearly by reusing the masks from the previous level. The Foreground Generatoris illustrated in Figure 4.6. It takes the K + 1 masks M, K part locations xk, and K partappearance embedding wk as input. Similar to the procedure that generates masks, we394.3. MethodFigure 4.8: Detection examples showing faithful part segmentation on the tested datasets,with varying object size and complexity.first broadcast the embedding with the corresponding mask to generate the foregroundembedding map Wfg ∈ RDemb×H×W ,Wfg(p) =K∑k=1Mk(p)wk. (4.13)We then use K part locations to generate the foreground starting tensor F(0) with lowfrequency positional encoding similar to Equation 4.11. Finally we use SPADE ResBlocksto generate the Foreground feature map F ∈ RDemb×H×W ,F(i) = SPADE ResBlock(F(i−1),Wfg)F = F(Tfg),, (4.14)where i = 1, ..., Tfg and Tfg is the number of SPADE ResBlocks. Independent of this, theBackground Generator takes two noise vectors as input zbg app ∼ N (0Dnoise , IDnoise×Dnoise),404.3. MethodDEF [41] SCOPS [103] OursFigure 4.9: Mask quality comparison. We qualitatively compare the mask on wildCelebA (first row) and Taichi (second row). Our masks have better quality than othermethods due to our hierarchical generator.ubg pos ∼ U([−1, 1]2). We first use a 3-layer MLP to map zbg app to a background appearancevector wbg ∈ RDemb ,wbg = MLPbg app(zbg app). (4.15)Positional encoding is used on the difference between the background center ubg pos and thepixel position, to generate the background starting tensor B(0), similar to Equation 4.11.Finally we use AdaIN ConvBlocks [101] to generate the background feature mapB ∈ RDemb×H×W ,B(i) = AdaIN ConvBlock(B(i−1),Wbg),B = B(Tbg),(4.16)where i = 1, ..., Tbg and Tbg is the number of AdaIN ConvBlocks.Figure 4.10 (bottom) illustrates the AdaIN ConvBlock [101]. AdaIN takes a feature mapand a style vector as input. We denote the feature map as Finput ∈ RDinput emb×H×W and thestyle vector as vstyle ∈ RDstyle emb . Unlike the original paper [101], which uses InstanceNorm[297], we found it beneficial in our case to use BatchNorm [105] to normalize Finput. Weuse two fully connected layers to map vstyle to the new mean β ∈ RDinput emb and newstandard deviation γ ∈ RDinput emb for the normalized Finput. As for the SPADE block, weuse BatchNorm to normalize Finput in batch and channel dimension. For denormalization,β and γ are generated for each channel and broadcast to the spatial size of the feature map.Combining Foreground and Background. Recall that we generate the backgroundmask Mbg ∈ RH×W along with the part masks. This is used to combine our Foreground414.3. MethodMethod Type Aligned (K=10) Wild (K=4) Wild (K=8)Thewlis et al. [288] Landmark 7.95% - 31.30% ⋆Zhang et al. [366] Landmark 3.46% - 40.82% ⋆LatentKeypointGAN [93] Landmark 5.85% 25.81% 21.90%Lorenz et al. [173] Landmark 3.24% 15.49% † 11.41% †IMM [112] Landmark 3.19% 19.42% † 8.74% †DEF [41] by [103] Part - - 31.30% ⋆SCOPS [103] (w/o saliency) Part - 46.62% 22.11%SCOPS [103] (w/ saliency) Part - 21.76% 15.01%Liu et al. [169] Part - 15.39% 12.26%Huang et al. [102] (w/ detailed label) Part - - 8.40%Ours Part 3.98% 12.26% 6.18%Table 4.1: Landmark detection on CelebA. The metric is the landmark regression(without bias) error in terms of mean L2 distance normalized by inter-ocular distance (loweris better). Although the pair-based methods work better on aligned CelebA, they do notgeneralize well. Our generation-based methods are more robust on Wild CelebA. The sign⋆ means reported by [103] and † by [169].and Background together. The final image is generated by feeding the feature map into atwo-layer CNN. That is,I = Conv((1−Mbg)⊗ F +Mbg ⊗B), (4.17)where ⊗ is the pixel-wise product.Upsampling and Cropping. For simplicity, we used size H ×W for all above featuremaps. For mask, foreground and background generation, the starting tensor has a 10-pixel-wide margin at each border, same as [125], so that the central feature map will not beinterfering with the boundary. For example, instead of generating a H0 ×W0 grid in range[−1, 1]2, we generate a (H0 + 20)× (W0 + 20) grid in range[−1− 20/H0, 1 + 20/H0]× [−1− 20/W0, 1 + 20/W0]. (4.18)We use this grid to calculate the starting tensors for masks, foreground, and background.After each SPADE ResBlock and each AdaIN ConvBlock, we use 2x upsampling on thefeature maps. The margin becomes 20-pixel wide. To bound the otherwise increasingboundary width, we subsequently crop the feature map to keep the 10-pixel margin. TheGaussian heatmaps are calculated on a grid with a 10-pixel-wide-margin, separately foreach resolution.Integrated Network Architecture We provide a complete overview of the networkarchitecture in Figure 4.11. It combines the individual modules introduced above.424.3. MethodMethod Type CUB FlowersGrabCut [242] Foreground Segmentation 0.360 0.692PerturbGAN [17] Foreground Segmentation 0.380 -ReDO [31] Foreground Segmentation 0.426 0.764Savarese et al. [249] Foreground Segmentation 0.551 0.789SCOPS [103] w/ GT BG by us Part Segmentation 0.329 † 0.544 †Ours Part Segmentation 0.629 0.739Table 4.2: Foreground-Background segmentation. The metric on CUB and Flowers isthe IoU of foreground (higher is better). We use K = 8 for both ours and SCOPS [103].The sign † means being trained by us using their official implementation.Method Type MAE ↓ IoU ↑DFF [41] no train/test split 494.48 † -SCOPS [103] images with saliency maps 411.38 † 0.5485 †Siarohin et al. [261] trained on videos 389.78 0.7686Ours trained on single images 417.17 0.8538Table 4.3: Part segmentation on Taichi. The sign † means being reported by [261]. Allthe results are shown in the case of K = 10.4.3.6 Training ObjectiveOur hierarchical GAN is trained end-to-end on image collections using the following lossfunctions.Geometric Concentration Loss Pixels from the same segment are usually connectedand concentrated around its center as assumed by [103]. We enforce the mask to be in anarea around its center with the geometric concentration lossLcon(G) =K∑k=1∑pMk(p)∑p′ Mk(p′)∥p− xk∥22. (4.19)Note that the background mask is not constrained by L(G)con. We found that this lossalone can cause parts to collapse since it encourages a small mask area. To mitigate thisproblem, we introduce the Area Loss below.Adversarial Loss. We denote G as the generator and D as the discriminator. We usethe non-saturating loss [73],LGAN(G) = Ez∼N log(exp(−D(G(z))) + 1) (4.20)434.3. MethodBatchNormConvConvSPADE ResBlockFeature MapSPADELeakyReLUConvSPADELeakyReLUConvSPADEBatchNormAdaIN ConvBlockFeature VectorAdaINLeakyReLUConvAdaINLinearLinearBroadcastBroadcastFigure 4.10: SPADE ResBlock.. (top) and AdaIN ConvBlock (bottom).for the generator, and logistic loss,LGAN(D) =Ez∼N log(exp(D(G(z))) + 1)+Ex∼pdata log(exp(−D(x)) + 1)(4.21)for the discriminator, with gradient penalty [193] applied only on real data,Lgp(D) = Ex∼pdata∇D(x). (4.22)Area Loss. We force the mask area to be larger than the area of the Gaussian heatmap,which is not predefined but predicted from the generated points in level 1. If the part isCelebA Taichi CUB Flowerλcon 10 30 10 30λarea 1 1 1 1Table 4.4: Penalty coefficients. .444.4. Experimentsvisible, this loss encourages the visibility of the mask. Otherwise, the area loss encouragesa smaller (close to a zero area) Gaussian heatmap.Larea(G) =K∑k=1max(0,∑pHk(p)−∑pMk(p)). (4.23)We will empirically show that this loss makes the masks more consistent.The final loss for the discriminator isL(D) = LGAN(D) + λgpLgp(D), (4.24)and the final loss for the generator isL(G) = LGAN(G) + λconLcon(G) + λareaLarea(G). (4.25)Hyperparameters We use the Adam optimizer [140] with a learning rate of 0.0001 forthe generator G and 0.0004 for the discriminator D, both with β1 = 0.5, β2 = 0.9. Weset the gradient penalty coefficient λgp = 10 for the discriminator. In every experiment,we update the generator 30,000 times. The learning rate for DeepLab is 0.0003, withβ1 = 0.9, β2 = 0.999. DeepLab is trained for 10,000 iterations. The feature map resolutionsare (322, 322, 642, 1282), respectively. We set nper = 4 for all experiments. The penaltycoefficients for each experiment is listed in Table 4.4. For Taichi, we set the vertical positionof the background to be always in the center and only sample the horizontal positionuniformly.4.4 ExperimentsFollowing related work [103, 169, 261], we analyze the improvement of our method in termsof the positioning of parts and the mask coverage on the established benchmarks. Weprovide results for a wide variety of images containing faces, animals, flowers, and humans.4.4.1 Datasets and MetricsOur evaluation follows the dataset-specific protocols and metrics as established in priorwork:CelebA-in-the-wild [171] shows celebrity faces in unconstrained conditions and is usedto estimate the consistency of part positions and the mask center of mass. We follow [103]removing images with a face-covering fewer than 30% of the pixel area from the MAFLtrain and test sets. As in [103, 169], we train a linear regression model without bias fromthe part centers to the ground truth keypoints. The error metric is the landmark regression454.4. ExperimentsPoint LocationPart AppearanceBackground AppearanceMLPMLPMLPPointsGroup AveragePart Location Uniformly Random Background PositionPositional Encoding on Relative PositionsStartingTensorfor BGEmbeddingsAdaIN ConvBlockAdaIN ConvBlockAdaIN ConvBlockAdaIN ConvBlockBackgroundForegroundGaussain HeatmapsBroadcastStartingTensorfor MaskPositional Encoding on Relative PositionsSPADE ResBlockSPADE ResBlockSPADE ResBlockSPADE ResBlockMasksSPADE ResBlockSPADE ResBlockSPADE ResBlockSPADE ResBlockBroadcastImageStartingTensorfor FGLevel 1 Level 2 Level 3BlendFigure 4.11: Integrated network architecture. for better understanding of our dataflow.464.4. Experimentserror in terms of mean L2 distance normalized by inter-ocular distance. Note that linearregression can only model linear mapping from unsupervised keypoints to ground truthkeypoints. Higher order relationship cannot be shown in this metric. The splits are 45609images for GAN training, 5379 with keypoint labels for regression, and 283 for testing.CelebA-aligned [171] contains 200k faces, each centered such that eyes align. Following[288], we use three subsets: CelebA training set without MAFL (160k images), MAFLtraining set (19k images), MAFL test set (1k images). The error metric is the same as onCelebA-in-the-wild.CUB-2011 [302] consists of 11,788 images of birds. We follow [31] to use 10,000 imagesfor training, 1,000 for testing, and the remaining 778 for validation. We use this dataset toanalyze segmentation coverage accuracy. We aggregate part segments to form the foregroundmask and calculate the Intersection Over Union (IoU) between the predicted foregroundmasks and the ground truth foreground masks [103, 169]. To this end, we calculate theforeground mask as the sum of our part masks.Flowers [207] consists of 8,189 images of flowers. The ground truth masks are obtainedby an automated method built specifically for flowers [207]. We follow [31] to use 6,149images for training, 1,020 for testing, and 1020 for validation. The metric is IoU of theforeground mask.Taichi [171] contains 3049 training videos and 285 test videos of people performingTai-Chi. We train our GAN on the training images (not videos) using 5000 images forfitting the regression model and 300 other images for testing. For a fair comparison, we usethe same 5300 images as [261]. The metric, mean average error (MAE), is calculated as thesum of the L2 distances between 18 regressed keypoints and their ground truth. We alsocalculate IoU based on the provided foreground masks.4.4.2 BaselinesWe compare to the following unsupervised methods, most requiring stronger assumptions:DEF [41] uses test images at training time, which has a high computational cost. SCOPS[103] uses saliency maps. Liu et al. [169] requires a strong prior on the mask center beingclose to the image center. Siarohin et al. [261] trained on videos exploiting temporalinformation.4.4.3 Qualitative analysisWe show detection examples in Figure 4.8, and qualitatively compare our predicted maskswith the baselines in Figure 4.9. Our mask has better coverage with the foreground object,474.4. Experimentsless fragmented parts, and better consistency, with the same part masks mapping to thesame body parts across different images. For example, the shoulder exists in almost allimages in wild CelebA [171], but SCOPS [103] cannot consistently discover this apparentpart, likely because their employed saliency map focuses on the face.4.4.4 Part Center ConsistencyTable 4.1 shows the results for keypoint detection. Our method outperforms other partsegmentation methods on the part consistency metric on the challenging CelebA-in-the-wild.The keypoint-based methods that perform better on aligned face images do not generalizeto real-world examples. The results on Taichi in Table 4.3 confirm that our model iscomparable in consistency (MAE metric) even though others are trained with additionalinformation. In addition, we evaluate our model on CUB using the protocol of [173]. Ourestimation error 3.23% (L2 error normalized by edge length of the image) is lower (better)than Zhang et al. [366] (5.36%) and Lorenz et al. [173] (3.91%).4.4.5 Mask CoverageSince the datasets used in [103, 169] are too small for training a GAN (see limitations) we onlyuse the sufficiently large CUB-2011 [171] and Flowers [207] dataset. To nevertheless comparewith SCOPS [103] we retrain their model. We use the ground truth masks and the requiredsaliency masks on our train/val/test split with their default parameters. Furthermore, wecompare our results with the unsupervised foreground-background segmentation methodsthat already report on these datasets. Note that these methods focus on foreground-background segmentation instead of part segmentation. Their approaches are tailored forthis task while it is rather a byproduct for us.As shown in Table 4.2, our model outperforms the unsupervised part segmentationmethods with better mask coverage, and is comparable with the dedicated foregroundsegmentation methods. On the most challenging Taichi dataset, we outperform all methodsin Table 4.3 in terms of mask coverage (foreground IOU) even though they use additionalinformation, such as saliency maps and videos.Figure 4.13 shows a qualitative comparison when removing different parts from our fullmodel. Instead of showing the segmentation on real images, we show the generated masksfrom our GAN. These masks explain our model more directly.4.4.6 Image QualityTo quantify image quality, we report the FID score [253] for the models used in the mainpaper: 21.42 for CelebA-in-the-wild, 116.62 for Taichi, 38.74 for CUB, and 38.41 for Flowers.Note that FID numbers are not comparable across datasets. The FID is calculated by50,000 generated images and the corresponding original dataset as established by [127].Note that image quality is not our primary goal but it is still important since it influences484.4. Experimentshow well the learned detector will generalize. The better the quality, the smaller the domaingap between generated and real images.4.4.7 Embedding VisualizationWe generated 320 images for each dataset and provide the T-SNE visualization of theirembedding vectors in Figure 4.12, including the background as a part. The embeddingsfrom different parts are well separated in all datasets, indicating that our model learnsunique embeddings for each part. Since overlapping part embeddings would mean that thesame image features could be generated by multiple parts, the separation explains the goodconsistency of parts across images.CelebA-in-the-wild Taichi CUB FlowerFigure 4.12: Embedding visualization..4.4.8 Ablation TestsDisabling Points. We remove the point generation from the network and in turn also theGaussian heatmaps and SPADE blocks. Instead, we use AdaIN [101], which is a spatiallyinvariant version of SPADE. We use the mean of the part embeddings as the embedding forthe feature map of AdaIN and replace the starting tensor with a learned constant tensor asin StyleGAN [127].Disabling Separated Background Generation. We integrate the background genera-tor into the foreground generator and treat the background as one of the parts.Disabling Losses. We remove losses and show examples from each dataset in Figure 4.13.Without the Geometric Concentration Loss, masks become fragmented. Without theArea Loss some parts vanish. Surprisingly, even without points and separate backgroundgeneration, the model can discover some commonly shared parts in the images exceptfor human bodies, as shown in the third row in Figure 4.13. We hypothesize it is due toour spatially variant network design with translation equivariance encouraged at all levels.Overall, it is evident that our hierarchical architecture is crucial for mask generation.494.4. ExperimentsFull Modelw/o Areaw/o Conw/o BGw/o PTsw/o PTs & BGw/o *Figure 4.13: Ablation tests. From top to bottom: models disabling 1) points, 2) separatedbackground, 3) points and separated background, 4) geometric concentration loss 5) arealoss, and 6) nothing (full model). For visualization purposes we show the background ingrey-scale. All contributions improve across datasets.504.4. ExperimentsFigure 4.14: Ablation test on estimated σ versus fixed σ. Examples are given forfour different datasets.Replacing the GAN with an Auto-encoder We tried two kinds of auto-encoders toreplace our Point Generator (Level 1 of the hierarchy) and Mask Generator (Level 2). Inthe first one, we train a U-Net [240] to obtain points {x1k, ...xnperk }Kk=1, and two separateResNet-18 [85] to extract part appearance vector wdynamic, and background appearancevector wbg along with background position ubg pos, to feed into our Mask Generator that isleft unchanged. In the second version, we directly use DeepLab V3 to generate masks M,and then feed it to our Foreground and Background generator. We use the mean squaredimage reconstruction error to train this auto-encoder. We observe that all of the testedauto-encoders give trivial solutions, as shown in Figure 4.15. Hence, we decided to use apure GAN setup.Using Fixed Standard Deviation Instead of estimating from points, we directly seta fixed standard deviation σ for each part. The fixed standard deviation is calculated bythe average of the σk from our pre-trained model. We sample 5000 images. The averagefor all datasets are 0.00725 (CelebA-in-the-wild), 0.010 (Taichi), 0.0016 (CUB), and 0.0065(Flower). The quantitative results are shown in Table 4.5 and the qualitative results areshown in Figure 4.14. The results show that fixing σ harms the performance severely. It isimportant to have various σ for each part and each object.Method CelebA ↓ CUB ↑ Flowers ↑ Taichi (MAE) ↓ Taichi (IoU) ↑fixed σ 26.32% 0.467 0.697 657.61 0.7452original 6.18% 0.629 0.739 417.17 0.8538Table 4.5: Quantitative ablation tests on number of parts. The metric for eachdataset in this table follows the main paper.Number of Points per Part We test on the influence of the number of points per partnper. We found that the network always collapses if nper ≤ 2, and collapses occasionally if514.4. ExperimentsFigure 4.15: Ablation test on replacing the GAN with an auto-encoder. Detectedpoints and masks are overlayed on the corresponding images. In this experiment, pointsmove to degenerate positions and scales. Masks are inaccurate.nper CelebA ↓ CUB ↑ Flowers ↑ Taichi (MAE) ↓ Taichi (IoU) ↑3 6.08% 0.644 0.696 594.73 0.58634 6.18% 0.629 0.739 417.17 0.85386 8.84% 0.682 0.715 434.47 0.83368 13.02% 0.641 0.744 467.69 0.7792Table 4.6: Ablation tests on number of points per part. The number of parts andthe metric for each dataset in this table follow the main paper. We use nper = 4 in themain paper.nper = 3. Table 4.6 shows that the optimal nper differs among different datasets. We choosenper = 4 in our main paper for consistency.Number of Parts We test how the number of parts affects segmentation. Althoughthe Area Loss is vital, its coefficient is not sensitive to the numbers of parts K. Thus forsimplicity, we set λarea = 1 as in the main paper. However, we found that small number ofparts with large Concentration Loss coefficient causes the trivial or sub-optimal solutions.Therefore, we use the formulaλcon(K) = CconK (4.26)The constant Ccon varies between datasets. In our experiments, we set 1.25 for Celeba-in-the-wild, 3 for Taichi, 1.25 for CUB, and 3.75 for Flower.We show quantitative results in Table 4.7 and qualitative results in Figure 4.16. Wefound our model starts to degenerate when K = 16. Even though the IoU on CUB is higher,524.5. Limitations and Future WorkMethod CelebA ↓ CUB ↑ Flowers ↑ Taichi (MAE) ↓ Taichi (IoU) ↑K=1 56.23% 0.631 0.570 748.30 0.7895K=4 12.26% 0.607 0.767 652.87 0.8176K=8 6.18% 0.629 0.739 420.95 0.8481K=12 6.17% 0.663 0.708 450.51 0.7939K=16 6.71% 0.692 0.712 441.04 0.7475K=32 7.00% 0.380 0.530 1605.84 0.1758Table 4.7: Quantitative ablation tests on number of parts. The metric for eachdataset in this table follows the main paper.the mask degenerates, which is still a state-of-the-art foreground/background segmentationmodel but not a good part model. If K = 32, the model fails to distinguish the foregroundfrom the background.4.5 Limitations and Future WorkOurs Siarohin et al. [261]Figure 4.17: Limitations The method fails for poses underrepresented in the training set(1st and 2nd col) and under occlusions (left eye of the face is marked as blue, which isusually on the ear), which we have in common with other methods such as [261] (4th col).In some rare cases the generator fails to generate good body shape on Taichi (5th col).Our GAN training requires a larger dataset (>5000 images), which does not apply to someexisting benchmarks but is acceptable for domains with large unlabelled image collections.Figure 4.17 shows representative failure cases that are typical also for related work. Insome rare cases and only for the Taichi dataset, the generator fails to generate good shape.Nevertheless, the detection remains accurate as there is no such unusual shape in the realdata. When a part is occluded in the image the associated mask will still cover a region534.5. Limitations and Future WorkFigure 4.16: Ablation test on the number of parts.. Seven examples are tested foreach dataset.544.6. Conclusionnearby, such as the left arm moving to the back instead of being occluded. Sometimes,front/back for bird and left/right for humans are flipped. The comparison to [261] showsthat this is common for 2D methods and demands for extensions to 3D and occlusionhandling.4.6 ConclusionWe presented a GAN-based approach for learning part segmentation from unlabelled imagecollections. Crucial is our hierarchical generator design that synthesizes images in a coarse-to-fine manner, with independence and invariance built in. It forms a viable alternative toexisting autoencoder techniques and opens up a path for learning part-based 3D modelsfrom 2D images in the future.55Chapter 5AutoLink5.1 IntroductionObject structure representations are widely used in modern computer graphics and computervision techniques, including keypoints for image generation [180, 182, 262] and skeletons for3D reconstruction [63, 110, 222, 271, 358]. However, the structure is usually supervised onlarge annotated datasets [8, 9, 163, 189] or via hand-crafted parametric models [18, 156,172, 213, 220, 336]. Neither approach generalizes well to new domains and both requireadditional manual annotation whenever more detail is needed [237].Our goal is to reconstruct the keypoint locations of an object by learning from anunlabelled image collection, thereby sidestepping the generalization problem. Our key ideais to leverage that the same object shares the same topology by introducing an explicit graphthat links the same pairs of keypoints in all instances. By contrast, existing self-supervisedkeypoint learning methods model objects as a set of independent parts. Their consistencyover different instances of the same object is encouraged by either enforcing parts to followhand-crafted image transformations [103, 112, 169, 173, 288, 366] or by adding implicit biasin the network architecture that encodes such spatial equivariances [91, 93]. Only [114, 250]use an explicit skeleton representation, but both require predefined topology, rely on videoinput, and [114] is trained in a CycleGAN setting that still requires manually labeledexamples.We propose a simple yet effective method to learn both the keypoints and their linkswithout supervision in terms of a sparse graph. The graph acts as a bottleneck thatcan only store structural information. As a result, it forms a constraint that associatesobservations across training images. We enforce the same topology across instances of thesame class by learning a shared graph with a single set of edge weights. In the absence oflabels, we train AutoLink with only an autoencoder reconstruction objective. Since thegraph bottleneck should not model appearance, we additionally feed the decoder with theinput image after masking the majority of its pixels. In turn, it is important that theheavily masked image contains appearance without leaking structural information. This isthe case as inpainting methods are unable to infer the original image precisely from onlysparse pixel colors [359, 372] unless conditioning on structure representations, such as edgemaps [80, 147, 152, 202, 334]. Therefore, by forcing the autoencoder to reconstruct theoriginal image, the detector converges to generate representative structures of images.We demonstrate on 4 benchmarks that the trained detector has a significantly improved565.2. Related Work(a) detected keypoints and visualized graph representationautoencoder that transfers poseconditional GAN(b) applicationsFigure 5.1: Teaser. (a) AutoLink applies to diverse collections of images and automaticallyyields keypoints linked to a graph without ground truth. It recovers animal and humanposes and object shapes in settings where previous methods struggle, including clutteredbackgrounds, structured stripe textures, articulated fingers, and detailed faces.keypoint localization accuracy and on 6 additional datasets that it applies to a broaderset of images spanning portraits, persons, animals, hands, and flowers, which we attributeto the explicit modeling of links in the graph. Figure 5.1 shows the diverse set of imagedomains it applies to, including challenging textures and uncontrolled background, howboth skeleton representations as well as object outlines are learned by varying the numberof keypoints, and exemplifies applications to controlled image generation.Ethics - Risks. The detected keypoints and edges hold the potential for exploitationin deep fakes, serving as driving signals for generative models. They may also be utilizedfor surveillance, discerning habits and unique poses that can identify an individual. A morecomprehensive discussion of the potential risks is provided in Section 8.1, within the contextof the other methods introduced in this dissertation.5.2 Related WorkThe high-level relation to unsupervised structured learning is given in Chapter 2. In thefollowing, we focus on different structure representation, including skeleton and sketch, and575.2. Related Workstructure-enhanced image inpainting.Self-supervised Keypoints Detection. We give a detailed introduction on self-supervisedkeypoints detection in Chapter 2 Section 2.1. While most models leverage multi-branchnetwork architectures to encode the structure and appearance separately and utilize multiplelosses that need to be balanced [112, 173, 288, 366], we do not apply artificial transformations,but use a single branch, and a single loss which eases and stabilizes training.Skeleton Representations. Bone maps representing the keypoints connectivity as affinityfields [24] or via explicit offsets [217] are used in supervised human, animal, and objectpose estimation. We use a similar edge map representation but learn both the locationand linking from scratch without annotations. In the weakly-supervised setting, Jakab etal. [114] exploit CycleGAN [375] to translate between image and edge maps. The graphconnectivity is predefined to the human skeleton and edges are supervised by a large datasetof unpaired ground truth object edges, which can come from a different dataset but aremanually annotated. Schmidtke et al. [250] overcome the manual labeling by deforming atemplate skeleton. However, they both require the known connectivity of the keypoints andvideos for training while ours learns both the keypoints and connectivity from a collectionof single images. Noguchi et al. [208] generate a skeleton heuristically by linking the centersof part-wise learned Signed Distance Fields [185]. However, they require videos withoutthe background of the same object, and the learned skeleton does not generalize to otherobjects of the same class.Object Sketch Learning. Sketches are made of strokes drawn by a pen. It is a conciseand abstract representation, which can be used in object recognition [339, 357] and imageretrieval [306, 338, 356]. There are two common sketch representations used in neuralmodels [337]: black-white raster images [248, 307, 353], often used for image-to-imagetranslation [109, 122, 216, 375] and sequences of points (pen coordinates) [60, 76, 247],which is usually used by recurrent generation models [23, 34, 48, 70, 76, 121]. This graphrepresentation is similar to ours. However, instead of learning to mimic human drawings,ours directly predicts both the keypoints and their connectivity on real natural images.Structure-enhanced Image Inpainting. When key parts are missing in an image,e.g., eyes on faces or arms of humans, it is hard for inpainting networks to imagine thecontent accurately from scratch. Therefore, additional structural cues are detected toguide the subsequent image generation. The cues can be supervised segmentation masks[80, 160, 268, 371], foreground contours [334], and landmarks [147, 345, 364], or automati-cally extracted edges [22, 116, 152, 202, 335] and low-frequency image components [233, 308].Our reconstruction objective can be seen as such two-stage inpainting, but self-supervisedand with the image edges replaced with the learned graph edge representation.585.3. Method5.3 Method5.3.1 Assumptions.We posit the following conditions: 1) the dataset encompasses at least thousands of imagesshowcasing objects of a single category, such as exclusively human faces or dogs; and 2) theobjects are generally centered within the images. The first condition is to ensure the objectshave the same topology so that learning skeletons are reasonable. The second condition isused to avoid modeling the background.5.3.2 OverviewWe leverage that the objects in the dataset share the same topology and can be representedas a graph that connects keypoints by a shared set of edges. To learn the keypoints andedges, we design an autoencoder that aims to accurately reconstruct the input image, withthe graph as the intermediate representation. To encode the input image into a graph,we detect the keypoints and create an edge heatmap based on learnable edge weights. Tomostly obtain appearance information, we mask out the majority of the image, whichrandomizes the structure information and reduces the remainder to a very low level. Theedge heatmap is combined with the masked image to reconstruct the original image. Sincethe missing structure is important to reconstructing the original image, the network isforced to learn the structure of the object in a self-supervised manner. Figure 5.2 shows anoverview of our method.Formally, given an image I ∈ RH×W×3 with height H and width W we aim to learna set of keypoints {ki}Ki=1, where ki ∈ [−1, 1] × [−1, 1] ⊂ R2 is the normalized keypointcoordinate, and K is the number of keypoints. We use a ResNet with upsampling [329]to detect keypoints. Afterward, we draw a differentiable edge [194] between each pair ofkeypoints (details below). The edge map S ∈ RH×W is concatenated along the channeldimension with the randomly masked image Im ∈ RH×W×3 and fed into a UNet [240] toobtain the reconstructed image I′.5.3.3 Image Structure RepresentationIn this section, we introduce the generation of keypoints and the edge map from the image.Let H ∈ RH×W×K be the K heatmaps generated by a ResNet with upsampling [329] fromthe image I. The keypoint ki is calculated by the differentiable soft-argmax function,ki =∑pexp(H(p))∑p(expH(p))p, (5.1)where p ∈ [−1, 1]× [−1, 1] is the normalized pixel coordinates.Given two keypoints ki,kj , we draw a differentiable edge map Sij , where values are 1 onthe edge linked by the two keypoints and decrease exponentially based on the distance to595.3. Methodimage keypointsrandommaskingmasked imagedraw edges reconstructed imageedge mapDecoderDetectorlearned graph edge weightssingle-image reconstruction shared by all instances in the same dataset Figure 5.2: Overview. Given an image, we detect keypoints and draw differentiable edgesbetween keypoints according to the learned graph edge weights that is visualized as a colormatrix. The method is self-supervised in that the latent edge map and keypoints are learnedby reconstructing the masked input images. Note that keypoints are image specific andedge maps are shared.the line. Formally, the edge map Sij is a Gaussian extended along the line [194], defined asSij(p) = exp(d2ij(p)/σ2), (5.2)where σ is a hyperparameter controlling the thickness of the edge, and dij(p) is the L2distance between the pixel p and the edge drawn by keypoints ki and kj ,dij(p) =∥p− ki∥2 if t ≤ 0,∥p− (ki + tkj)∥2 if 0 < t < 1,∥p− kj∥2 if t ≥ 1,where t =(p− ki) · (kj − ki)∥ki − kj∥22. (5.3)We assign a weight wij > 0 to each edge, which is enforced to be positive by SoftPlus[56]. This weight is learned during training and shared across all object instances in adataset. Finally, we take the maximum at each pixel of the heatmaps to obtain the finaledge map S ∈ RH×W ,S(p) = maxijwijSij(p). (5.4)Taking the maximum at each pixel avoids the entanglement of the edge weights and theconvolution kernel weights, which is further explained in Section 5.4.4.5.3.4 Image ReconstructionThe masked image Im is generated by first uniformly dividing the image I into a 16× 16grid, and randomly masking out 80% of the grid cells, similar to [82]. We concatenate the605.3. Methodmasked image with the edge map and feed it into a UNet decoder [240] to reconstruct theoriginal image,I′ = Decoder(αIm ⊕ S) (5.5)where ⊕ means concatenation along the channel dimension and α is a learnable parameterthat compensates for the change of the edge weight magnitude during training. α isinitialized to 1. We found this parameter to be helpful in training stability. Different to[82], we condition on an edge map. Different to [114], we have no ground truth for the edgemap. Our edge map is an unobserved latent variable. Thus we only minimize the differenceof the original image and the reconstructed image by the perceptual loss [118],L = 1NN∑i=1∥Γ(Ii)− Γ(I ′i)∥22 (5.6)where N is the number of examples and Γ is the feature extractor. The perceptual loss isbelieved to measure the structure similarity [55, 69, 118], and leads to more robust training[112, 114].5.3.5 ApplicationsIn this section, we briefly describe how we created the two applications, conditionalGenerative Adversarial Networks and pose transfer networks, based on the learned graphrepresentation, as shown in the teaser. Note that, although we apply the graph representationto videos for pose transfer, it is only learned from the collections of single images.Conditional Generative Adversarial NetworkThe conditional GAN is a simplified StyleGAN2 [129], where the spatial noise injection isremoved and the starting tensor is replaced by the feature map generated from the edgemap. For simplicity, we do not use EqualLinear [127] or Path Regularization [127].Formally speaking, given an edge map S ∈ RH×W , where H,W are the spatial size, weuse bicubic interpolation to downsample it to 32×32 and feed it into a two-layer convolutionnetwork to generate a feature map F ∈ R8×8×512. In parallel, we generate the embeddingvector w ∈ R256 from a noise vector z ∈ R256 by a three linear-layer MLP. The feature mapF is fed into a residual convolution-based generator where the kernel weights are modulatedby the embedding vector w [129]. The final image x ∈ RH×W×3 is generated by a singleconvolution layer.We denote G as the generator and D as the discriminator. We use the non-saturatingloss [73],LGAN(G) = Ez∼N log(exp(−D(G(z))) + 1) (5.7)for the generator, and logistic loss,LGAN(D) = Ez∼N log(exp(D(G(z))) + 1) + Ex∼pdata log(exp(−D(x)) + 1) (5.8)615.4. Experimentsfor the discriminator, with gradient penalty [193] applied only on real data,Lgp(D) = Ex∼pdata∇D(x). (5.9)Pose Transfer NetworkWe train the pose transfer network on videos. We randomly sample two frames I1, I2 ∈RH×W×3, where H,W are the spatial size. We use keypoint detector on the frame I1 andgenerate the edge map S1 ∈ RH×W . The edge map is fed into a UNet [240] to reconstruct I1.The smallest feature map in the UNet is concatenated by a feature map of the appearanceinformation, which is generated by frame I2 to provide the appearance information. Theloss is a combination of Mean Squared Error, VGG perceptual loss [118] and the GAN lossin Eq 5.7, 5.8, and 5.9.5.3.6 Network ArchitectureFigure 5.3 shows the architectures we used in the main paper. The keypoint detector is aResNet with upsampling [329], which is a simple baseline used in human pose estimation.The decoder and the pose transfer network are UNets [240]. The conditional GAN is asimplified StyleGAN2 [129], where the spatial noise injection is removed and the startingtensor is replaced by the feature map generated from the edge map. In Figure 5.3, wedenote Conv for 3x3 convolution [148], BN for Batch Normalization [105], LReLU for LeakyReLU [184], Up for 2x bilinear upsampling, and Down for 2x bilinear downsampling.5.3.7 Implementation DetailsWe use the Adam optimizer [140] with a learning rate of 10−4 with β1 = 0.9, β2 = 0.99.The batch size is 64. We train for 20k iterations. It takes 3 hours to train on a singleV100 GPU. All images are resized to 128× 128. The learning rate for the edge weights ismultiplied by 512 due to the small gradient of SoftPlus [56] when the value is close to 0. Toshow the robustness of our model, we report all experiments on the sampling strategy ofmasking 80% of the 16× 16 patches. We perform experiments with the same edge thicknessof σ2 = 5e− 5 for all benchmark datasets. We train 10 times and report the mean and thestandard deviation of the evaluation metrics. Although it already outperforms other work inmost experiments, we also tune thicknesses to each individual dataset, as others did for theirhyperparameters, which further improves the results. The only other hyperparameter is thenumber of keypoints, which we set to that of the established benchmarks for quantitativecomparisons, ranging from 4 to 32 points.625.4. Experiments Conv-Down-BN-LReLU-Conv-BN (64, 64, 64) Conv-Down-BN-LReLU-Conv-BN (32, 32, 128) Conv-Down-BN-LReLU-Conv-BN (16, 16, 256) Conv-Down-BN-LReLU-Conv-BN (8, 8, 512)Up-Conv-BN-LReLU (16, 16, 256)Up-Conv-BN-LReLU-Conv (32, 32, n_kp) Conv-BN-LReLU-Conv-BN-Down-LReLU (64, 64, 128) Conv-LReLU (128, 128, 64) Conv-BN-LReLU-Conv-BN-Down-LReLU (32, 32, 256) Conv-BN-LReLU-Conv-BN-Down-LReLU (16, 16, 512) Conv-BN-LReLU-Conv-BN-Down-LReLU (8, 8, 512)Up-Conv-BN-LReLU-Conv-BN-LReLU (16, 16, 512)Up-Conv-BN-LReLU-Conv-BN-LReLU (32, 32, 256)Up-Conv-BN-LReLU-Conv-BN-LReLU (64, 64, 128)Up-Conv-BN-LReLU-Conv-BN-LReLU (128, 128, 64)Conv (128, 128, 3)Bicubic Interpolation (32, 32, 1) Conv-LReLU-Down (16, 16, 128) Conv-LReLU-Down (8, 8, 512) WeightMod-LReLU-WeightMod-LReLU-Up (16, 16, 512) WeightMod-LReLU-WeightMod-LReLU-Up (32, 32, 512) WeightMod-LReLU-WeightMod-LReLU-Up (64, 64, 256) WeightMod-LReLU-WeightMod-LReLU-Up (128, 128, 128) WeightMod-LReLU-WeightMod-LReLU (128, 128, 64)Conv (128, 128, 3) Linear-LReLU (256) Linear-LReLU (256) Linear-LReLU (256)Noise (256)Embedding (256) Conv-LReLU-Conv-Down-LReLU (64, 64, 128) Conv-LReLU (128, 128, 64) Conv-LReLU-Conv-Down-LReLU (32, 32, 256) Conv-LReLU-Conv-Down-LReLU (16, 16, 512) Conv-LReLU-Conv-Down-LReLU (8, 8, 512)Up-Conv-LReLU-Conv-LReLU (16, 16, 512)Up-Conv-LReLU-Conv-LReLU (32, 32, 256)Up-Conv-LReLU-Conv-LReLU (64, 64, 128)Up-Conv-LReLU-Conv-LReLU (128, 128, 64)Conv (128, 128, 3)(a) Keypoint Detector (b) Image Decoder (c) Conditional GAN (d) Video Transfer Decoder Conv-LReLU-Conv-Down-LReLU (64, 64, 128) Conv-LReLU-Conv-Down-LReLU (32, 32, 256) Conv-LReLU-Conv-Down-LReLU (16, 16, 512) Conv-LReLU-Conv-Down-LReLU (8, 8, 512)concatenateFigure 5.3: Network architectures. From left to right: detector (encoder), decoder,conditional GAN, conditional autoencoder. The shortcuts in (a) and (c) are addition whilethe shortcuts in (b) and (d) are concatenation.5.4 ExperimentsIn this section, we compare our results to the related methods, showing that our model issimple yet effective. Besides, we perform a number of ablation studies on hyperparametersand algorithm variants, exhibiting the robustness of our model and justifying the necessityof every model component. For visualization purposes, we scale the edge weights by dividingthe maximum value to obtain visible edges. For DeepFashion, before dividing the maximumvalue, we further add 0.01 to the edge weights larger than 0.0001. Although the modelsare trained on different thicknesses, we draw them in the same thickness σ2 = 5× 10−4 forpleasing visualization.5.4.1 Datasets and Evaluation MetricsCelebA-aligned [171] contains 200k celebrity faces aligned in center. We follow [288]splitting it into three subsets: CelebA training set without MAFL (160k images), MAFLtraining set (19k), MAFL test set (1k). We train our network on the CelebA trainingset without MAFL. To quantitatively evaluate the consistency of our predicted keypoints,we follow [288] training a linear regression without bias from our detected keypoints tothe ground truth keypoints on the MAFL training set and reporting the mean L2 errornormalized by inter-ocular distance on the MAFL test set.635.4. ExperimentsLatentKeypointGAN[93]GANSeg[91]OursOurs(edges)Figure 5.4: Qualitative comparison on detected keypoints. Our model is more robuston wild face poses, and depicts more details on human bodies compared to [93] and [91].For example, the feet poses in the middle four images are clearly detected.CelebA-in-the-wild [171] contains celebrity faces in unconstrained conditions. We follow[103] and first split it into three subsets as for CelebA-aligned, and then remove the imageswhere a face covers less than 30% of the area, which results in 45,609 images for modeltraining, 5,379 with keypoint labels for regression, and 283 for testing. The evaluationmetric is the same as CelebA-aligned.Human3.6m [106] contains human activity videos in static backgrounds. We follow[366] considering six activities (direction, discussion, posing, waiting, greeting, walking),and using subjects 1, 5, 6, 7, 8, 9 for training and 11 for testing. This results in 796,648images for training and 87,975 images for testing. The evaluation metric is the regressed(without bias) mean L2 error normalized by the image size. We remove the backgroundas in [173, 366] to make a fair comparison to others. To underline the robustness againststructured backgrounds we also report the numbers including background.DeepFashion [170] contains 53k in-shop clothes images. We follow [173] only keeping thefull-body images. We use 10604 images for training and 1179 images for testing as in [19].We use the keypoints generated by AlphaPose [61] as the ground truth. The evaluationmetric is Percentage of Correct Keypoints of d=6 pixels in resolution 256× 256.Taichi [260] contains 3049 training videos and 285 test videos of people performingTai-Chi, with the various appearance of foreground and background. We follow [261] using5000 and 300 images (not contained in training data) for training a linear regression andfor testing, respectively. The evaluation metric, mean average error (MAE), is calculated asthe sum of the L2 error in resolution 256× 256.645.4. ExperimentsTable 5.1: Landmark detection on CelebA. The metric is the landmark regression(without bias) error in terms of L2 distance normalized by inter-ocular distance (lower isbetter). While all methods perform well on aligned CelebA, ours is more robust on WildCelebA. The sign ⋆ means being reported by [103] and † means being reported by [169].Method Type Aligned (K=10) Wild (K=4) Wild (K=8)DFF [41] by [103] Part Segmentation - - 31.30% ⋆SCOPS [103] (w/o saliency) Part Segmentation - 46.62% 22.11%SCOPS [103] (w/ saliency) Part Segmentation - 21.76% 15.01%Liu et al. [169] Part Segmentation - 15.39% 12.26%Huang et al. [102] (w/ detailed label) Part Segmentation - - 8.40%GANSeg [91] Part Segmentation 3.98% 12.26% 6.18%Thewlis et al. [288] Landmark 7.95% - 31.30% ⋆Zhang et al. [366] Landmark 3.46% - 40.82% ⋆LatentKeypointGAN [93] Landmark 5.85% 25.81% 21.90%Lorenz et al. [173] Landmark 3.24% 15.49% † 11.41% †IMM [112] Landmark 3.19% 19.42% † 8.74% †LatentKeypointGAN-tuned [93] Landmark 3.31% 12.10% 5.63%Ours (general) Landmark 3.92±0.69% 7.72±0.47% 5.66±0.29%Ours (thickness-tuned) Landmark 3.54% 6.11% 5.24%CUB-200-2011 [302] consists of 11,788 images of birds. We follow two establishedprotocols [37, 173] to evaluate our method: 1) Images are cropped based on the birdlandmarks, aligned to face to the left [173], and seabirds are removed; 2) Birds are croppedbased on the given bounding box and the train/val/test split of [37] is used. In both cases,the evaluation metric is the regressed (without bias) mean L2 error normalized by thecropped image size.Flower [207], 11k Hands [4], Horses [375], and Zebras [375] are used for qualitativeexperiments. VoxCeleb2 [40] and AFHQ [36] are used for pose transfer and conditionalimage generation, respectively. Horses and Zebras are extracted from the CycleGAN dataset[375] by removing the images with multiple horses and aligning them to face left. Notethat the horses and zebra are trained separately, yet the model learns similar structures.The train/test split of Flower follows [31]. All the other datasets follow the train/test splitspecified by the dataset.Limitation of current metric . While using linear regression from unsupervisedkeypoints to ground truth keypoints is the most common way for evaluation, we want topoint out that higher order relationship is not considered in this metric.655.4. ExperimentsTable 5.2: Landmark detection on Human Body. Our model outperforms all the otherunsupervised baselines. The metric for each dataset follows the corresponding descriptionin the text. The sign † means being reported by [261] and the sign ⋆ means being reportedby [19]. The number of keypoints is K = 16 for Human3.6m and DeepFashion and K = 10for Taichi.Method Supervision Human3.6m ↓ DeepFashion ↑ Taichi ↓Jakab et al. [114] video & unpaired ground truth 2.73 - -Newell et al. [114] paired ground truth 2.16 - -DFF [41] testing dataset - - 494.48 †SCOPS [103] saliency maps - - 411.38 †Siarohin et al. [261] videos - - 389.78Zhang et al. [365] videos - - 343.67Zhang et al. [366] videos 4.14 - -Schmidtke et al. [250] video & T-pose template 3.31 - -Sun et al. [272] videos 2.53±0.06 - -Thewlis et al. [288] unsupervised 7.51 - -Zhang et al. [366] unsupervised 4.91 - -LatentKeypointGAN [93] unsupervised - 49% 437.69Lorenz et al. [173] unsupervised 2.79 57% ⋆ -GANSeg [91] unsupervised - 59% 417.17Ours (general) unsupervised 2.81±0.07 65±1.2% 337.50±25.08Ours (thickness-tuned) unsupervised 2.76 66% 316.105.4.2 Qualitative AnalysisWe qualitatively compare our detected keypoints with other methods and show the examplesof the learned edges in Figure 5.4. For visualization purposes, we scale the edge weights toobtain visible edges. We use the same number of keypoints as the previous method [91] for afair comparison, which are 8 for CelebA-in-the-Wild, 16 for DeepFashion, and 10 for Taichi.We will discuss more on the choice of the number of keypoints in Ablation Study 5.4.4.As shown in Figure 5.4, our model not only detects consistent keypoints but also learnsreasonable edges, such as human skeletons in DeepFashion and Taichi. For example, thefeet are clearly connected with the corresponding knees, and there is no edge between theleft and right hands.5.4.3 Quantitative AnalysisWe compare the keypoint detection results with other methods in Table 5.1 (CelebA),Table 5.2 (Human3.6, DeepFashion, Taichi), and Table 5.3 (CUB). Our simple modeloutperforms all other unsupervised methods in all benchmarks, except [103] on three CUBsubsets, which however requires saliency maps, and for the most constrained setting CelebA-aligned and CUB-aligned where all methods perform well and the results are comparable.The results on CelebA in Table 5.1 confirms that our model is more robust to poses in the665.4. ExperimentsTable 5.3: Landmark detection on CUB Birds. Our model outperforms most otherbaselines and achieves comparable results with the ones using ground truth segmentationmasks. The metric is the landmark regression (without bias) error of L2 distance normalizedby the image size (lower is better). A star ⋆ means being reported by [37], † means beingreported by [103], and ‡ means tested by us with their official code; all other numbers aretaken from the respective papers. The number of keypoints is K = 10 for CUB-aligned andK = 4 for CUB-001, CUB-002, CUB-003, and CUB-all.Method Supervision CUB-aligned ↓ CUB-001 ↓ CUB-002 ↓ CUB-003 ↓ CUB-all ↓SCOPS [103] GT silhouette - 18.3 ⋆ 17.7 ⋆ 17.0 ⋆ 12.6 ⋆Choudhury et al. [37] GT silhouette - 11.3 15.0 10.6 9.2DFF [41] testing dataset - 22.4† 21.6† 22.0† -SCOPS [103] saliency maps - 18.5 18.8 21.1 -Lorenz et al. [173] unsupervised 3.91 - - - -ULD [288, 366] unsupervised - 30.1† 29.4† 28.2† -Zhang et al. [366] unsupervised 5.36 26.9‡ 27.6‡ 27.1‡ 22.4‡LatentKeypointGAN [93] unsupervised 5.21‡ 22.6‡ 29.1‡ 21.2‡ 14.7‡GANSeg [91] unsupervised 3.23 22.1‡ 22.3‡ 21.5‡ 12.1‡Ours (general) unsupervised 4.15 ± 0.24 20.6 ± 0.54 20.3 ± 0.96 19.7 ± 0.91 11.6 ± 0.33Ours (thickness-tuned) unsupervised 3.51 20.2 19.2 18.5 11.3wild. Since self-supervised part segmentation methods are usually more robust on wildfaces [91, 103, 169], we also include them for comparison, demonstrating the robustnessof our model over existing baselines. The results on Human3.6m and DeepFashion inTable 5.2 show the capability of our model to detect keypoints on human bodies of eithersimilar or diverse appearances. The result on Taichi in Table 5.2 demonstrates the generalapplicability of our model to human bodies of diverse poses in complex backgrounds.5.4.4 Ablation TestsIn this section, we analyze the hyperparameters and demonstrate the robustness of ourmodel. We also discuss the possible variants of our model and show the superiority of ourdesign.Number of Keypoints & Thickness . We show ablation test results on the differentnumbers of keypoints and edge thicknesses in Figure 5.5. Our model shows very strongrobustness to edge thickness. The accuracy remains state-of-the-art while the thickness ofthe edges varies by multiple orders of magnitude. On the other hand, with the increasingnumber of keypoints the accuracy increases. This is expected since more keypoints are ableto capture structure in more detail, as shown in Figure 5.7. Yet, some other methods failfor a large number of keypoints [91].Table 5.4 shows the complete results on the different numbers of keypoints and edgethickness. While a larger number of keypoints gives better details and thus higher accuracy,675.4. ExperimentsFigure 5.5: Ablation tests on the number of keypoints and edge thickness. Whilethe model shows better performance with more keypoints, it is robust to the edge thickness.Figure 5.6: Ablation tests on masking strategy. Overall, the performance increases asthe mask ratio increases. Too small or too large patch sizes can decrease the performance.Empirically, masking 80% of the 16× 16 patches is a golden rule.the performance is robust to the thickness. Table 5.5 shows the results on differentmasking ratios and mask patch sizes. A too small masking ratio significantly decreases theperformance since the structure can be directly extracted from the masked image with alow masking ratio.Masking Strategy . In our standard setting, the image is divided into 16× 16 patchesand 80% of the patches are randomly masked. We investigate how the patch size andmasking ratio affect the model performance. Figure 5.6 shows that a too low maskingratio enables the network to directly infer the structure from the masked image which isundesired in our case. In these cases, the network would not choose to infer a set of compactkeypoints from the original image. Figure 5.6 illustrates that the patch size cannot be toosmall (4× 4) or too large (64× 64). Although in some cases, such as CelebA-in-the-Wild, adifferent masking strategy gives better results (4.14 vs 5.24), we choose to report a unifiedstrategy that we mask 80% of 16× 16 patches for simplicity and conciseness.685.4. ExperimentsFigure 5.7: Examples of different numbers of detected keypoints. With very fewkeypoints, the model only models a very basic shape, such as the box on the face, andsometimes cannot fully capture the structure. For example, with K=4, only legs are modeledon humans. With abundant keypoints, it is able to model details.Variants of Edge Heatmap Generation . Besides the heatmap generation describedin Method 5.3, we test four more ways to generate the edge maps: 1) we define the thicknessas a globally learnable parameter; 2) we learn each edge thickness independently as aparameter; 3) we treat each edge heatmap as an independent channel of the feature map,instead of making them a single channel heatmap with Equation 5.4; 4) instead of usingthe edge heatmap, we generate Gaussian heatmaps for the keypoints and use Equation 5.4to combine them in a single channel heatmap. The results are listed in Table 5.6. Overall,these variants have worse performance. Although in some cases, the model has slightlybetter results on specific datasets, the performance boost does not hold in general. Weobserve that with only keypoints without edges, the model may degenerate, as shown inFigure 5.8a. Interestingly, assigning each edge a different channel performs worse thansimply combining all edges into a single channel. We believe it is caused by entangling theedge weights with the convolution kernel weights. As visualized in Figure 5.8b, there existdummy edges that do not model the object structure. In addition, we tried to remove thelearnable α in Equation 5.5, fixing α = 1, but the overall performance decreases as shown695.5. Limitations and Future WorkCelebA-wild ↓ Human3.6m ↓ DeepFashion ↑ Taichi ↓σ2 K=2 K=4 K=8 K=16 K=32 K=2 K=4 K=8 K=16 K=32 K=2 K=4 K=8 K=16 K=32 K=2 K=4 K=8 K=16 K=321.0e-5 60.9 52.5 42.6 34.9 31.9 5.67 5.13 3.37 2.90 2.81 17.8 48.1 58.1 62.6 63.8 667 657 622 592 4582.5e-5 50.5 38.8 29.8 10.7 6.11 5.64 5.15 3.50 3.02 2.87 29.5 48.9 57.1 65.2 66.2 665 637 611 506 3515.0e-5 49.4 8.06 5.41 4.88 4.65 6.10 5.03 3.76 2.76 2.91 22.5 50.4 58.7 65.8 66.6 654 550 338 338 2877.5e-5 58.3 7.71 5.62 4.92 4.65 5.49 5.08 3.19 2.89 3.00 43.5 51.1 59.3 65.7 69.8 650 516 383 301 2971.0e-4 54.5 7.44 5.71 4.93 4.64 5.56 5.09 3.25 2.96 2.96 44.5 49.0 58.1 64.1 67.6 647 512 385 329 2802.5e-4 11.3 6.68 5.57 5.05 4.42 5.49 5.03 3.36 3.01 3.32 42.9 48.7 56.8 64.9 67.6 624 526 391 321 2845.0e-4 11.4 6.56 5.77 5.01 4.43 5.49 5.10 3.45 2.94 3.16 42.0 47.5 58.0 65.4 67.2 612 531 381 307 2757.5e-4 10.6 6.11 5.81 4.86 4.39 5.57 5.09 3.84 3.31 3.06 35.6 49.6 58.2 65.7 66.4 604 479 363 296 2891.0e-3 13.3 6.84 5.70 4.43 4.50 5.49 5.04 3.42 3.47 3.18 40.4 51.0 57.1 64.6 68.8 608 462 342 306 2752.5e-3 15.8 6.70 5.24 4.72 4.49 5.48 5.04 3.40 3.37 3.01 43.6 51.6 55.8 61.2 66.7 609 442 326 289 2865.0e-3 12.6 6.29 5.53 4.69 4.48 5.52 5.03 3.59 3.36 3.12 44.7 50.3 56.7 62.3 68.3 598 510 333 323 3027.5e-3 11.6 6.16 5.61 4.76 4.41 5.54 5.06 3.50 3.77 2.99 39.4 48.6 57.7 62.6 65.5 604 514 325 328 3401.0e-2 11.2 6.22 6.66 4.89 4.47 5.45 5.02 3.39 3.05 2.94 44.6 50.3 56.5 60.9 65.1 593 515 327 327 341Table 5.4: Ablation tests on the numbers of keypoints and edge thickness. Weremove the % sign in metrics for simplicity. The best one for each number of keypoints ismarked in bold.CelebA-wild ↓ Human3.6m ↓ DeepFashion ↑ Taichi ↓ratio 4x4 8x8 16x16 32x32 64x64 4x4 8x8 16x16 32x32 64x64 4x4 8x8 16x16 32x32 64x64 4x4 8x8 16x16 32x32 64x6410% 17.9 16.4 49.2 49.3 48.3 3.22 3.25 3.55 5.28 6.39 58.3 39.7 40.5 52.1 51.0 640 638 630 659 64220% 19.7 29.4 44.6 43.3 39.8 3.06 3.52 3.21 4.29 5.99 62.1 55.5 43.4 39.6 49.3 522 605 614 638 61330% 6.56 8.46 45.4 41.3 46.8 3.08 3.08 3.44 3.80 6.24 62.8 58.9 62.8 42.4 41.8 509 570 627 642 62740% 8.31 6.71 7.58 6.31 6.81 3.15 2.91 3.38 3.40 4.19 62.5 59.2 63.4 41.2 40.0 500 428 634 640 65150% 8.06 6.39 6.23 6.58 5.99 3.73 2.99 2.94 3.72 4.24 61.3 65.4 64.3 41.1 38.9 444 452 481 640 66760% 7.69 6.22 5.52 6.98 5.56 3.11 3.13 2.95 3.35 4.87 62.8 62.0 61.7 41.4 39.9 483 396 418 654 64370% 7.03 6.38 5.44 5.11 4.43 2.95 3.09 2.87 3.28 4.85 60.7 63.2 63.1 59.7 41.5 447 362 376 523 65680% 6.95 6.68 5.24 4.73 4.65 3.63 3.47 2.76 2.97 4.08 58.3 61.7 65.8 61.6 39.6 501 371 316 347 64290% 7.24 7.15 5.77 5.62 4.14 2.99 3.18 2.95 3.64 3.75 60.9 63.1 66.4 62.7 40.2 626 388 330 346 526Table 5.5: Ablation tests on masking ratio and patch size. We remove the % signin metrics for simplicity. The best one for each number of keypoints is marked in bold.in Table 5.6.Does Texture Matter? We trained two networks on horses and zebras separately. Asshown in Figure 5.1a, the horse and the zebra share similar shape structures but only oneis textured. The striped texture not having a significant impact on the learned structureshows that our model primarily learns the structure instead of texture features.What if the model is trained on images with a structured background? Wetested on Human3.6m with a background, where all images are taken in a single room. Theerror is 5.02. As shown in Figure 5.8c, our model captures the entrance in the background.It is expected since we assume the foreground object is structured. If we apply spectralclustering [203] on the learned graph, the keypoints are clearly divided into two clusters,one for the room and one for the person.5.5 Limitations and Future WorkIf the background is highly structured, the keypoints will appear on the background. Yet,we showed an avenue for future work, as already a simple graph clustering could separate705.6. ConclusionTable 5.6: Ablation tests on variants of edge heatmap generation. The originaldesign is proved to be the most robust one. Although in some cases it is not optimal, thedifference is almost trivial.Model CelebA in The Wild ↓ Human3.6m ↓ DeepFashion ↑ Taichi ↓original model 5.24% 2.76% 65.8% 316fixed α 6.39% 2.87% 66.0% 374shared learnable thickness 6.12% 3.25% 49.1% 425independent learnable thickness 5.94% 3.73% 50.2% 311edge-specific heatmap 5.65% 3.83% 65.1% 407only using keypoints without edges 6.55% 3.58% 52.9% 722(a) w/o edges (b) w/o shared edge channel (c) effect of structured BG (d) failure casesFigure 5.8: Ablation tests and failure cases. (a) If we do not model the edges, themodel may degenerate. (b) If we give different edges a different channel in the featuremap, the model would give dummy edges. (c) If the model is trained on the dataset withstructured background, the background would be modeled. However, the keypoints canbe separated into two sets by spectral clustering. (d) failure case: left) the model cannotmodel the occlusion well; right) the model has left and right ambiguity.the object from the background on the Human3.6M dataset. Similar to the previous 2Dself-supervised methods [91, 173, 261], our model cannot model occlusion well. We show inFigure 5.8d left that the occluded right arm becomes the back when the person turns to theleft. In addition, as for all other methods, the model cannot distinguish the left and rightsides of the objects as shown in Figure 5.8d middle and right. We believe it is necessary tomodel the structure in 3D to solve these problems.5.6 ConclusionWe presented a simple approach for learning a spatial graph representation from unlabelledimage collections by reconstructing masked images. The crucial part is our learnable graphdesign that models the relationship between different keypoints. It is simpler than existing715.6. Conclusionalternatives and opens up a path for image understanding, image editing, and learning 3Dmodels from 2D images.72Chapter 6Few-shot6.1 IntroductionKeypoint localization is a long-standing problem in computer vision with applicationsin classification [38, 46], image generation [181, 263], character animation [258, 259], 3Dmodeling [64, 221], and anti-spoofing [49], among others. Traditional supervised keypointlocalization approaches require a large dataset of annotated images with balanced datadistributions to train robust models that generalize to unseen observations [84, 315, 319].However, annotating keypoints in images and videos is expensive, and usually requires severalannotators with domain expertise [178, 305, 312]. Manual annotations can be inaccuratedue to low resolution imagery [26] and temporal variations in illumination and appearance[117, 303], or even subjective, especially in presence of external occlusions [144, 325] andimage blur effects [273, 361]. Besides, modeling self-occluded object parts is proven to be anambiguous task since 3D consistent keypoint annotations are needed [362]. As a consequence,supervised approaches are prone to learning suboptimal models from noisy training data.Unsupervised keypoint detection methods can predict consistent keypoint structures [88,90, 92, 113, 174, 367], but they lack human interpretability or may render insufficient, e.g.,for editing tasks that require more detailed manipulation of object parts. These methods canbe extended to few-shot scenarios either by learning a mapping that regresses automaticallydetected keypoints to human labeled annotations [289], which requires thousands of examples[113], or by attaching few-shot annotated examples in the training batch as weak supervision[197]. However, we show that the latter approach cannot produce competitive predictionswhen applied to a state-of-the-art unsupervised methods for keypoint localization, e.g., [90].Recent advances in semi-supervised keypoint localization has shown significant progressin the field. Still, most existing methods are mainly designed for a single object categorysuch as faces [21, 227, 316] and X-rays [32, 351, 354, 373], or require hundreds or thousandsof annotated examples to achieve competitive performance [197, 223, 304]. Our approach onthe other hand only needs a few dozens of examples. In an orthogonal direction, Honari et al.[99] utilize classification labels and equivariance transforms to assist keypoint localization,though classification labels are not always available. Generative image labeling has alsoshown great promise [274, 326, 368]. DatasetGAN [368] regresses Gaussian heatmaps fromthe StyleGAN latent space [126] to detect keypoints, but requires annotating StyleGAN-generated images prone to artifacts and noise, thus adding extra burden to the annotationprocess. Besides generative approaches are limited to the biases of the underlying data736.1. IntroductionImageKeypoints Uncertainty StructureFigure 6.1: Teaser. All results are obtained by 10-shot learning except Tigers where20 examples are used. The left/right/middle keypoints are marked in red/blue/green.Using only a few shots, the model learns semantically consistent and human interpretablekeypoints. Uncertainty modeling help us identify occlusions and ambiguous boundaries, asshown in the mouth, eye, and car example.746.1. Introductiondistribution [212, 285], thus decreasing overall keypoint localization performance.Current limitations then create the need for an approach that can leverage a smalleryet semantically consistent corpus of human labeled annotations while generalizing to amuch larger unlabelled image set.In this paper, we present a novel formulation that learns to localize semanticallyconsistent keypoint definitions, even for occluded regions, for various object categories withcomplex geometry using only a few user labeled images. We use as input a few example-baseduser-labeled 2D images with predefined keypoint definitions and their linkages to learn tolocalize keypoints. Unlike unsupervised methods, the few-shots images selected by the useract as semantic shape constraints for human interpretable keypoint localization. To enablegeneralization to the target data distribution, we extend our approach via self-supervisionusing a larger unlabeled dataset. In addition, we introduce 3D geometry-aware constraintsthat allow us to model depth and uplift 2D keypoints in 3D with view point consistency,thus achieving more accurate 2D localization.Quantitative and qualitative experiments demonstrate that our proposed approachcompetes or outperforms state-of-the-art methods on few-shot keypoint localization ofhuman faces, eyes, animals, and cars using a few user-defined semantic examples. We alsoshow the capabilities of our keypoint localization approach on a novel data distribution,specifically mouth interior that has not been attempted with previous few-shot localizationapproaches. Thus, our novel general purposed formulation paves the way for semanticallyconditional generative modeling with a few user-labeled examples, which we hope will enablea broader set of downstream applications including, fast dataset labeling, and in-the-wildmodeling and tracking of complex objects, among others.Our key contributions are summarized as follows:1. A novel formulation for few-shot 3D geometry-aware keypoint localization that workson diverse data distributions.2. We introduce keypoint uncertainty estimation along with 2D and 3D aware geometryconstraints for better keypoint localization.3. Our approach enables flexible modeling of complex deformable objects and geometricparts, such as mouth interior, faces, eyes, cars, and animals via a few user examples withconsistent semantic definitions.Ethics - Risks. The potential misuse of estimated keypoints and edges poses a threat inthe realm of deep fakes, where they can serve as the input signal for generative models or beexploited in undesired surveillance applications. The creation of an unauthorized animatedtalking face can be accomplished with just a single frontal facial image. Furthermore, sinceit only requires around 10 annotated images to train a new keypoint detector, uncommonfacial deformation might be abused for unwanted editing. In addition, since people can useit for their own designed keypoint annotations, it is easier to hack the systems that requiresspecific keypoints. For a thorough exploration of potential risks, please refer to Section 8.1,considering the broader context of the methods introduced in this dissertation.756.2. Related Work6.2 Related WorkThe high-level relation to semi-supervised and few-shot structured learning is given inChapter 2. In the following, we focus on supervised keypoint localization and generativeimage labeling.Supervised keypoint localization Supervised methods learn semantically consistentkeypoint localization with outstanding generalization capabilities usually by leveraging alarge corpus of in-the-wild human labeled images for standard object categories [164] ordomain-specific classes, such as faces [141, 325, 362], eyes [142], teeth [305], human bodies[10, 107, 117, 190], animals [155], and vehicles [232, 328], among others. Annotating imagesand videos is not only expensive but also prone to labeling errors [98, 187, 239] due to imagedownsampling [26], object occlusions [144, 325], image blur [273, 361], and harsh appearanceand lighting variations especially in video datasets [117, 303]. As such, supervised modelsrender inaccurate for downstream tasks, often requiring statistical uncertainty modeling[65, 151] or robust regressors [51] to achieve state-of-the-art performance. A recent lineof work resorts to problem-specific high-quality synthetic datasets, mainly human bodies[219], faces [317], eyes [200, 320] and teeth [319] to produce perfect semantic annotations,even for partially occluded object parts. Despite efforts, creating a synthetic dataset withreal-world data distribution remains a challenging and very laborious task, and methodstrained on them still achieve near-competitive performance [200, 317, 319]. On the otherhand, our approach can learn a general model that preserves semantic definitions fromlimited annotations and user constraints while still generalizing well to unseen distributionvia self-supervision.Semi-supervised keypoint localization We give a detailed introduction on semi-supervised keypoints detection in Chapter 2 Section 2.2. These semi-supervised methodsrequire a labeled dataset that is one or two orders of magnitude higher than that of ourapproach [52, 104, 197, 227, 304], and yet cannot generate 3D keypoints like ours.Few-shot keypoint localization We give a detailed introduction on few-shot keypointsdetection in Chapter 2 Section 2.2. Compared with the existing methods, [21, 21, 32, 316,351, 354, 373], which mostly target on a single domain, such as faces, or X-ray images, ourproposed method addresses general object distribution.Generative image labeling Generative models, such as GANs [126, 128] and diffusionmodels [96], learn generators with rich semantic spatial information. Such generators canbe used as powerful semantic priors to solve downstream tasks, e.g., semantic segmentation[79, 149, 292, 347, 368], video object detection [274], and salient object detection [326] froma few annotated examples. As shown in [84, 368], segmentation masks can be easily adapted766.3. MethodInput ImageMaskedImageGeometry Aware Keypoints, UncertaintyEdge MapsRandom MaskingReconstructed Image2D Few-shot Supervision(Pre-defined Keyframes)2D Geometry Constraints(Equivariance)3D Aware Geometry Constraints(Reprojection Error)GTPrediction1 Transformed keypointsKeypoints on transformed image PredictionClosest Transformed Match in BatchDetector Decoder1 1[ R | t ]Figure 6.2: Overview. Given an image, we detect the keypoints and their uncertainty.They are used to generate an edge map, which is concatenated with a randomly maskedimage to reconstruct the original image. The keypoints are forced to be semanticallymeaningful by few-shot supervision and consistent by reconstruction. In addition, the 2Dand 3D geometric constraints increase the robustness of keypoints.for keypoints using one-hot binary masks or heatmaps representations. As these methodsrequire labeling generated images that are prone to artifacts and noise, user annotationsmight be inaccurate, and so are the estimated labels [368]. Besides, methods trained ongenerative image-label pairs are limited to the data distribution and biases of the underlyingimage generator model [212, 285]. We on the contrary show our approach can adapt betterto diverse data distribution using a few manual labels.6.3 MethodOur goal is to learn semantically meaningful and consistent keypoints by using only dozensof annotated examples combined with thousands of unlabeled images. Our starting point issupervised learning on the dozens of labeled examples. This is used to define the semanticmeaning of keypoints, but alone would horrendously overfit and is hence augmented withself-supervision objectives that encourage meaningful and consistent detection on theadditional unlabeled set. We achieve geometry aware keypoint localization via a multi-tasklearning strategy that first detects keypoints with edge and uncertainty maps, and thendecodes these maps with random masked images to synthesize photo-realistic images, asshown in Figure 6.2. The edge linkages are provided by the user. We train both detectionand reconstruction stages in an end-to-end manner to ensure that the predicted edges and776.3. Methoduncertainty maps derived from keypoints encode the right semantic object shape definitionto synthesize a photo-realistic object.6.3.1 Assumptions.We make the following assumptions. 1) The dataset includes a few, typically 10-20,annotated images and over thousands of unannotated images; 2) the images feature objectswithin the same category, such as exclusively human faces or dogs; and 3) the objects areapproximately cropped and centered in the images.6.3.2 Keypoint and Uncertainty DetectionIn this section, we introduce how we obtain the keypoints and uncertainty from the images.Given an image I ∈ RH×W×3, we use a ResNet with upsamplings [330] to predict Kheatmaps Hi ∈ RH×W , and K uncertainty maps Vi ∈ RH×W , where i = 1...,K. The 2Dkeypoints ki ∈ R ∈ [−1, 1]2 are generated as the arg-softmax of the heatmap, and theuncertainty is calculated as the sum of the map Vi weighted by the heatmap, as follows,ki =∑pw(p)p, vi =∑pw(p)Vi(p),where w(p) =exp(Hi(p))∑p exp(Hi(p)),(6.1)where p ∈ [−1, 1]2 is the normalized pixel coordinate.6.3.3 2D Few-shot SupervisionTo leverage the user-provided few-shot 2D keypoint annotations, during each trainingiteration, we randomly select several image-keypoint pairs, along with a batch of imageswithout any annotations. We concatenate them and penalize deviations of the detectedkeypoints k′ from the ground truth keypoints k on those with annotations,Lfew shot = 1|A|∑i∈A∥ki − k′i∥1, (6.2)where A is the set of annotated examples.6.3.4 2D Geometric ConstraintsWe force the keypoints to be equivariant to the 2D image transformations, which benefitsthe robustness as suggested in the unsupervised keypoint detection methods [113, 174].Denote T as a 2D transformation, which is a combination of affine transformations, flippingand color jitter, and k(I) as the keypoints detected on image I. We force the transformed786.3. Methodkeypoints T (k(I)) to be close to the keypoints k(T (I)) detected on the transformed imageT (I),L2d geo = 1NN∑i=1∥k(T (Ii))− T (k(Ii))∥1. (6.3)The keypoints that are out of image boundary after the transformation are ignored. Wenotice that this equivariance loss significantly harms the performance in the first few it-erations during training. Therefore, we linearly increase the range of image transformationbased on the number of iterations to stabilize training.6.3.5 3D Aware Geometry Constraints and UpliftingIn the domain of medical X-ray image, it is proven to be effective to constrain predictedkeypoints to be close to few-shot examples in 2D [32]. However, in general, the objects inimages usually have large viewpoint variation. Therefore, it is problematic to simply enforcethe keypoints to be similar in 2D. We propose to constrain the similarity in 3D. To obtaina 3D keypoint representation, we use the detector to additionally generate K depth mapsDi ∈ RH×W , i = 1, ...,K and calculate the depth as the weighted sum of the depth maps,di =∑pw(p)Di(p), (6.4)where w(p) is the weight defined in Equation 6.1. The 3D keypoints are defined as theconcatenation of the 2D keypoints and the corresponding depths, k3Di = (ki,di).For the keypoints k(I1) detected on image I1, we select another set of keypoints k(I2)detected on image I2. We estimate a similarity transformation η between the two sets ofkeypoints [298], and force the transformed keypoints η(k3D(I2)) of the second example tobe close to the keypoints k3D(I1) of the first example,L3d geo = 1NN∑i=1minI2∥k3D(Ii)− η(k3D(I2)) ∥1. (6.5)To prevent overfitting in the beginning, in the first 200 iterations, we randomly pairthe images in the batch. Afterwards, we pair each image with the one in the same batchthat minimizes the mean L2 distances as shown in Equation 6.5. In practice, we not onlyenforce it on the whole object keypoints, but also on part keypoints to allow articulation.6.3.6 Geometry-aware Image ReconstructionFinally, we exploit that the keypoints should be semantically meaningful enough to recon-struct the original image from a largely masked image [90]. Similar to [90], we introducean objective to reconstruct the original image from the edge map, with appearance pro-vided by a masked image (90% of pixels removed). We take the keypoint uncertainty into796.3. Methodconsideration instead of simply assuming all keypoints are certain as in [90]. Given twokeypoints ki,kj defined as ”linked” by users, we draw a differentiable edge map Sij , whichis a Gaussian extended along the line [90, 195]. The values decrease exponentially based onthe distance to the line, and are smaller for the uncertain keypoints. Formally, the edgemap Sij of keypoints (ki,kj) is defined asSij(p) = exp(vij(p)d2ij(p)/σ2), (6.6)σ2 = 1/1000 exp θ, (6.7)where σ is a learnable parameter controlling the thickness of the edge, θ is learnable andinitialized to 1, vij(p) is the uncertainty at the pixel p, and dij(p) is the L2 distance betweenthe pixel p and the edge drawn by keypoints ki and kj ,dij(p) =∥p− ki∥2 if t ≤ 0,∥p− (ki + tkj)∥2 if 0 < t < 1,∥p− kj∥2 if t ≥ 1,vij(p) =sigmoid(vi) if t ≤ 0,sigmoid((1− t)vi + tvj) if 0 < t < 1,sigmoid(vj) if t ≥ 1,where t =(p− ki) · (kj − ki)∥ki − kj∥22.(6.8)We assign a learnable weight α to edges, which is enforced to be positive by SoftPlus[57]. This weight is learned during training and shared across all edges and all objectinstances in a dataset. Finally, we take the maximum at each pixel of the heatmaps toobtain the edge map S ∈ RH×W ,S(p) = αmaxijSij(p). (6.9)Taking the maximum at each pixel avoids the entanglement of the uncertainty and theconvolution kernel weights [90]. The edge map weight α in Equation 6.9 is formulated asα = SoftPlus(γ), (6.10)where γ is learnable and initialized to -4.The edge map is concatenated with the masked image and fed into a UNet [241] toreconstruct the original image. We minimize the L1 loss and ViT perceptual loss [294]between the reconstructed images I′ and the original images I,Lrecon = 1NN∑i=1∥Ii − I′i∥1 + ∥Γ(Ii)− Γ(I′i)∥1 (6.11)where Γ is the feature extractor.806.4. Results6.3.7 Coefficients and Few-shot Examples ChosenThe coefficient for Lfew shot,Lrecon,L2d geo,L3d geo are (1, 1, 1, 0.1), which is decided by thevalidation sets. There is no other hyperparameters. Note that the validation sets are notused to choose the best model during training but only used to find the loss coefficients,which are shared by all datasets. We believe that using the validation sets for modelselection in practice is not available in few-shot learning. The few-shot examples are chosenby the centers of k-means clustering on the features of the 3nd last layer of VGG [264].6.3.8 Implementation detailsOptimization All images are resized to 128×128. We use the Adam optimizer [139] witha learning rate of 10−4 with β1 = 0.9, β2 = 0.99. The batch size is 16 for unlabeled imagesand min(16,n examples) for few-shot examples. We train for 20k iterations. The gradientsare stopped in the similarity transformation estimation in the 3D geometric constraint sothat the shape instead of transformation itself would be optimized. The ViT perceptualloss is based on the last attention keys and the the global context vector.Reconstruction of masked images For the reconstruction, we divide the image into16× 16 patches and randomly mask 90%. The random seeds for all packages are fixed to 0.2D geometric constraint The image transformation in 2D geometric constraint is acombination of random rotation (−60 ∼ 60), translation (−10% ∼ 10%), scaling (0.9 ∼ 1),flipping (p=0.5), and color jitter (brightness, contrast, saturation, hue from -50% to 50%).The augmentation ranges are multiplied by 0 and 1 at iteration 0 and 20k, respectively.This coefficient increases linearly from 0 to 1 during the training.Facial landmark smoothing For face dataset only, we encourage the cosine of the angleof the two neighboring landmarks to be 0. This penalty has weight 0.02, and is only forjaws and noses.6.4 ResultsWe test our model on 6 diverse datasets, including rigid, soft, and articulated objects,which may contain various appearance and severe occlusion. Notably the mouth interioris extremely challenging due to the large occlusions and had not been attempted before.We compare with three baselines, a supervised model [330], a semi-supervised model [197]and an unsupervised method [90]. We adapt the last one to few-shot learning by ourtraining strategy in Section 6.3.3. Unless otherwise stated, all results are obtained by10-shot learning except Tigers where 20 examples are used.816.4. ResultsImageKeypoints Uncertainty Structure Image Keypoints Uncertainty Structure Image Keypoints Uncertainty StructureFigure 6.3: Qualitative results. With only few shots, the model learns semanticallyconsistent and meaningful keypoints.6.4.1 DatasetsMEAD Part0 [309] contains high resolution audio-visual clips of 12 actors. We use thefirst actor and crop around the mouth. We annotate 10 images with the lips annotationfrom [65] and manual annotation on 4 upper teeth and 5 bottom teeth. This dataset is toqualitatively test the ability of modeling occlusion.SynthesEyes [320] contains 11382 synthesized images of eyes. We use the first 4 malesand 4 females as our training set and use the last male and female as the test set. Thisdataset is to quantitatively evaluate the ability of modeling occlusion. The evaluationmetric is the mean point-wise L2 errors normalized by the distances of eye corners.CUB [301] contains birds of 200 categories. We use the first 100 categories for training(5864 images), the middle 50 categories for validation (2958 images), and the last 50categories for testing (2966 images). This dataset is to quantitatively evaluate the ability ofdetecting keypoints on objects of highly various appearance. The evaluation metric is thepercentage of keypoints within a range of 10% largest bounding box length centered by theGT keypoints.CarFusion [232] contains various car images captured in Pittsburgh, PA. We usethe ones captured in Fifth street Part 2 (2794 images) as training set and Part 1 (1597images) as testing set. This dataset is to test the the ability of modeling rigid object andself-occlusion. The evaluation metric is the percentage of keypoints within a range of 10%826.4. ResultsGTOursMoskvyak et al. [45]ImagesAutoLink [17]Xiao et al. [82]Figure 6.4: Qualitative comparison. We qualitatively compare our results with thebaselines and the ground truth. The invisible keypoints are not shown in the ground truthif they are not provided. Our performance is significantly better than others. In the difficultcases, such as tigers and cars, our model still generates good shapes while others fail.largest bounding box length centered by the GT keypoints.WFLW [325] contains 10k faces with a 7500/2500 train/test split. This dataset ismore challenging than most face datasets as it contains a significant portion of occlusion,make-up, and extreme poses. This dataset is to test the ability of modeling soft bodies.We follow [21], where the evaluation metric is the mean point-wise L2 errors normalized byinter-ocular distances.ARTW [155] contains 5159 tiger images captured in multiple wild zoos in unconstrainedsettings, with 3610/516/1033 for train/val/test split. This dataset is to test the ability ofmodeling articulated bodies. The evaluation metric is the percentage of keypoints within arange of 10% image size centered by the GT keypoints.Datagen/Faces [283] contains 10k images of 100 diverse identities with ground truth3D facial landmarks, including the occluded ones. We use the first 7500 image for trainingand the rest 2500 images for testing. We use this dataset to quantitatively measure howgood the 3D landmarks that our 3D geometry constraint yields are.836.5. Analysis6.4.2 Qualitative ComparisonsWe show the qualitative results in Figure 6.3 and comparison in Figure 6.4. When thenumber of annotations are limited, the baselines tend to overfit and fail to detect on thedifficult cases. For example, only our model successfully detects the keypoints of the busin the last column of Figure 6.4.6.4.3 Quantitative ComparisonsWe quantitatively compare our model on 5 benchmarks, WFLW, SynthesEyes, CUB, ATRW,and CarFusion. Table 6.1 shows that our model significantly outperforms other methodswhen only 10-50 annotated examples are available. Furthermore, our model is more robustto the number of annotated examples than other methods. For instance, on ATRW, theaccuracy of the baselines decrease (58%/61%/40%) if only 50 out of 3610 examples areused, while our performance only decreases 16%.6.4.4 Comparison with DatasetGANWe additionally compare with DatasetGAN [369]. We use the pre-trained StyleGANgenerator on FFHQ [126] of resolution 256× 256. We sample 10000 images and choose 10images by the centers of k-means clustering on the features of the 3rd last layer of VGG [264].The keypoints are annotated by DLIB [138], which is originally used for FFHQ alignment.To train our model, we use the first 60000 images for training and the last 10000 images fortesting. To make a fair comparison, we train our model in two different sets of few-shotexamples: 1) pick from the dataset as in the main paper; 2) use the generated examplessame as DatasetGAN. The results are summarized in Table 6.3. Our model outperformsDatasetGAN in all different number of annotated examples. Note that their StyleGANmodel is trained on all 70000 images in the dataset while our unlabeled dataset only containsthe first 60000 images. It is noticeable that our model trained on the generated examples isnot as good as those trained on real images. This demonstrates that the artifacts and noisein the generated images has significant impact on the keypoint localization.6.4.5 Comparison on LSP DatasetFor completeness, we also show the results on LSP [120] dataset in Table 6.4. The metricand train/test split follow [197]. Our accuracy in the few-shot scenario is significantly higherthan all the other baselines.6.5 AnalysisIn this section, we test the effects and the necessity of our designed modules. Quantitativecomparison is implemented on WFLW and ATRW, with the same setting and the same846.5. AnalysisNME (%) on WFLW dataset ↓Method Training set size1 10 20 50 5% 20% 100%SA [99] - - - - - 6.00† 4.39Xiao et al. [330] 43.0 21.9 19.3 17.6 10.6 7.08 5.62Moskvyak et al. [197] 137 133 76.6 21.9 10.27 6.84 6.65AutoLink [90] 14.9 13.5 13.3 11.2 7.68 7.31 6.353FabRec [21] 15.8† 9.66† - 8.39† 7.68† 6.51† 5.62ours 12.4 9.19 8.62 7.90 6.22 5.61 5.38NME (%) on SynthesEyes dataset ↓Method Training set size1 10 20 50 5% 20% 100%Xiao et al. [330] 25.1 15.9 10.6 8.11 4.07 3.12 2.65Moskvyak et al. [197] 91.1 86.8 45.8 18.5 4.52 3.24 2.49AutoLink [90] 26.4 14.2 8.78 7.80 4.28 3.32 2.86ours 24.0 6.93 6.83 5.50 3.69 3.13 2.96PCK@0.1 (%) on CUB-200-2011 dataset ↑Method Training set size1 10 20 50 5% 20% 100%Xiao et al. [330] 6.01 31.2 35.3 42.8 60.5 73.9 90.5Moskvyak et al. [197] 7.38 20.9 28.3 63.2 91.1† 92.4† 93.8AutoLink [90] 26.2 35.1 41.2 51.8 67.2 75.9 87.6ours 16.3 70.7 73.1 75.1 84.2 88.3 90.1PCK@0.1 (%) on ATRW dataset ↑Method Training set size1 10 20 50 5% 20% 100%Xiao et al. [330] 13.2 21.5 22.5 23.9 51.6 86.1 96.1Moskvyak et al. [197] 3.38 17.9 27.5 57.1 92.6† 94.5† 95.3AutoLink [90] 20.8 22.1 33.8 37.4 77.8 89.4 97.1ours 14.4 36.3 78.8 81.4 92.7 96.5 96.9PCK@0.1 (%) on CarFusion dataset ↑Method Training set size1 10 20 50 5% 20% 100%Xiao et al. [330] 11.7 23.4 30.0 37.5 51.1 77.9 89.7Moskvyak et al. [197] 3.84 5.38 22.4 34.7 64.7 87.3 92.5AutoLink [90] 14.5 31.9 42.3 62.2 69.9 83.5 90.8ours 13.8 66.7 68.8 75.5 81.6 93.5 93.8Table 6.1: Quantitative comparison. In the few-shot scenario, where only 10-50 annotatedexamples are available, our model significantly outperforms the baselines. The sign † meansthe number is reported in another set of examples used in their papers.856.5. AnalysisMethods Training set size1 10 20 50 5% 20% 100%Xiao et al. [330] 36.5 22.5 22.1 17.8 8.33 5.37 4.19Full 15.6 11.2 9.25 8.46 7.07 6.44 5.99Table 6.2: NME(%) on Datagen 3D faces. We compare with the supervision baseline.Note that the supervision baseline uses ground truth 3D keypoints while ours only needs2D keypoints.NME (%) on FFHQ dataset ↓Method Training set size1 10 20 50DatasetGAN [368] 18.4 8.04 7.24 5.50ours (trained with generated examples) 14.2 7.08 6.37 5.22ours 11.3 5.87 5.89 4.59Table 6.3: Quantitative comparison with DatasetGAN.PCK@0.1 (%) on LSP datasetMethod Training set size1 10 20 50 5% 20% 100%Xiao et al. [330] 15.3 22.1 23.9 26.5 24.0 47.8 71.1AutoLink [90] 19.7 22.3 26.9 35.4 35.3 50.0 75.0Moskvyak et al. [197] 3.89 7.13 15.1 27.7 67.0 71.9 74.3ours 14.2 49.8 53.2 58.1 64.5 70.4 77.9Table 6.4: Quantitative comparison on LSP.866.5. AnalysisGT OursImages Ours † DatasetGAN. [89]Figure 6.5: Comparison with DatasetGAN. Our model generates better shapes that arecloser to the ground truth than DatasetGAN. The third row and fourth row with symbol †are obtained by training on the real annotated images and the synthetic annotated images,respectively.876.6. Limitations.few-shot examples in Section 6.4.Image Reconstruction. As shown in Table 6.5, without the image Reconstructionconstraint, the error increases dramatically for low numbers of annotated examples. It isbelieved that reconstructing from edge maps aligns the edges with the object edges [90],which stabilize the keypoints in the few-shot learning. It is noticeable that with morethan 20% annotated examples, the image reconstruction harms the performance, probablybecause the reconstruction task drifts the gradient from accurate keypoint supervisiongradient.Geometric Constraint. While both 2D and 3D geometric constraints increaseaccuracy, their insights differ slightly. The 2D constraint is at the image level, whichis accurate. It works as data augmentation on learned keypoints, which is expected toincrease the robustness. It is interesting that on Tigers (10-shots), it has better performancewithout 2D geometric constraint. We believe it is due to the symmetric shape of the tigers,which is explained more in limitations later. On the other hand, the 3D constraint is anapproximation, which enforces similarity between two different examples. It is expected toprevent the model from generating extreme outliers. Its effects on articulated objects, suchas tigers, is not as significant as it is on soft and rigid objects, such as faces.Uncertainty. Uncertainty models occluded and ambiguous edges in image reconstruc-tion by making the uncertain edges lighter. Without it, the model may be confused in thereconstruction stage about whether the edge should be drawn if it is occluded. As a result,it harms performance. Figure 6.7(a) shows an example of a largely occluded face, wherethe model without uncertainty gives an average face. On the ATRW tigers, the difference isnot obvious because the keypoints are visible in the most cases.How good are the 3D keypoints? We train on the Datagen/Faces [283] using only2D landmarks and compare the 3D landmarks quality with the supervised baseline [330],where depth is additionally learned. We qualitatively evaluate the learned 3D landmarks inFigure 6.6 and quantitatively in Table 6.2. Figure 6.6 also illustrates the difference betweenthe jaw landmarks learned from 2D facial landmark dataset and that from 3D-projectedfacial landmark dataset. The signal provided by the latter gives better 3D jaw shapebecause the landmarks are not deformed as those in the 2D facial dataset.Few-shot Example Selection. We tested replacing the KMeans with random selection.We run the experiments three times and report the average in Table 6.5. When only dozensof annotated examples are available, it is important to pick the most representative ones,especially for articulated objects, such as ATRW tigers.6.6 Limitations.There are three limitations. First, if the object keypoints are highly symmetric, theremay be left-right or front-back ambiguity. Second, if the object is highly articulated, e.g.,humans in LSP dataset [120], the estimated keypoints are less accurate. We visualize both886.6. Limitations.Datagen (3D)WFLW (2D)Figure 6.6: Detection on 2D and 3D-projected face landmarks. If the model istrained on the 2D face dataset WFLW, the learned jaw landmarks are not 3D consistent.However, if it is trained on the synthetic 3D-projected landmarks, the jaw contours are 3Dconsistent, though the projection is still 2D facial contour.NME (%) on WFLW dataset ↓Variants Training set size1 10 20 50 5% 20% 100%- image reconstruction 27.4 24.8 20.2 12.4 6.49 5.32 4.69- 2D geometry 16.7 14.2 13.7 12.6 9.24 7.81 6.83- 3D geometry 15.5 12.4 12.1 11.3 7.67 6.47 6.37- uncertainty 13.5 12.6 11.8 10.4 7.28 6.75 6.42- kmeans selection 24.2 14.5 12.6 10.8 6.81 5.54 -Full 12.4 9.19 8.62 7.90 6.22 5.61 5.38PCK@0.1 (%) on ATRW dataset ↑Variants Training set size1 10 20 50 5% 20% 100%- image reconstruction 20.2 22.1 22.8 56.5 91.0 96.8 97.6- 2D geometry 24.7 40.6 39.3 39.6 87.4 93.7 96.1- 3D geometry 16.7 36.2 75.8 81.2 92.1 96.1 96.4- uncertainty 20.8 31.9 78.7 81.2 92.3 95.9 95.4- kmeans selection 8.2 10.1 15.1 40.7 90.9 96.1 -Full 14.4 36.3 78.8 81.4 92.7 96.5 96.9Table 6.5: Ablation tests on WFLW and ATRW. Each our design plays an importantrole in the few-shot scenario.896.7. Conclusion & Discussionw/o uncertaintyGTImagew/ uncertaintyGTPred(a) Effects of Uncertainty (b) Failure casesFigure 6.7: Effects of incertainty and failure cases. (a) Effects of Uncertainty.Modeling uncertainty improves the keypoints on objects with occlusion. Note the differenceon nose and eyebrow. (b) Failure Cases. The model fails if the objects are highlysymmetric or articulated.cases in Figure 6.7(b). However, neither problem is observed when we increase the datasetto hundreds of annotations (instead of dozens). The third limitation is that we entangleocclusion with ambiguous boundaries, and model both of them using a single variable,uncertainty.6.7 Conclusion & DiscussionWe presented a few-shot keypoint localization method that is formed by keypoint detectionwith uncertainty, 2D/3D geometric constraints, and image reconstruction. These compo-nents prevent the detector from overfitting to the few-shot examples and utilize unlabelledimages. Our experiment results demonstrate that, with only dozens of annotations, ourmodel works on various datasets, including rigid, soft, articulated objects, and the verydifficult mouth interior which has not been tried before. In the few-shot scenario, our modelsignificantly outperforms the baselines and works on those datasets where others fail with10 or 20 annotated examples. It opens the path for conditional generative modeling andimage editing with a few annotated examples. Our future work focuses on leveraging 3Daware image synthesis for better generalization to extreme poses, solving the symmetricproblem, and testing on more different object categories .90Chapter 7DataPerspective7.1 IntroductionTo tackle this issue, previous research has explored several promising approaches. Text-conditioned image editing methods [20, 133] enable image manipulation based on text inputwhile retaining the subject’s identity. However, these methods struggle when tasked withgenerating entirely new images in different contexts. On the other hand, image compositiontechniques [43, 162, 321, 342] can transplant a given subject into alternative backgroundsbut cannot create the subject within a novel scene from text alone. Recently, researchershave shown promise by fine-tuning large-scale diffusion models to associate an identifiertoken with the desired subject. This not only preserve subject identity but also exhibitimpressive generalization across various textual inputs [66, 243]. Nevertheless, these meth-ods still face challenges in reproducing fine details of the subject and are susceptible tooverfitting to the limited training data.One straightforward method to enhance detail preservation is training longer. However,this raises the challenge of mitigating overfitting of the background and the subject pose.Previous studies have suggested various strategies, such as fine-tuning specific modelcomponents [78, 100, 145, 228, 266, 287, 299]. In contrast, our contribution is a data-centricapproach that leaves the model architecture unchanged.Recent text-to-image diffusion models have made significant strides in creating realistic,diverse, and precise images from text inputs [225, 238, 246]. However, their capability toaccurately represent specific subjects, like unique backpack designs or individual dog breeds,is constrained by the textual descriptions provided. This limitation has spurred substantialinterest and investigation in the research community, particularly in the quest to generatefresh and contextually adaptable images of the same subject matter.We introduce a novel technique for creating a regularization dataset tailored to counter-act overfitting to the training images. Fundamental to our method is the observation thatthe fine-tuning not only associates the subject to the identifier token but also to the samesimple sentence structure used for fine-tuning, limiting the ability to use more complexprompts at test time. To this end, we combine LLMs and text-to-image diffusion modelsto automatically generate prompts that are diverse in content and differ slightly in sentencestructure to subsequently generate corresponding images that show the subject class invery diverse context. Using these synthetic examples for regularization yields a more stablefine-tuning process that can continue longer without overfitting to the subject pose or the917.1. IntroductionDreamBoothOurs (SDXL) Ours (SD) SuTIa <new> backpack in Grand CanyonText InversionDreamBoothOurs (SDXL) Ours (SD) SuTIa <new> monster toy flying a kite in the desertText InversionInput ImageExampleInput ImageExampleDreamBoothOurs (SDXL) Ours (SD) SuTIa <new> drink can in supermarketText InversionInput ImageExampleFigure 7.1: Teaser.. Demonstration of our method applied to both SDXL and SD, comparedwith several prior works. Note that our results preserves very fine details of the subjectwithout overfitting to the training images and losing text alignment.927.2. Related Workbackground and significantly enhances both identity preservation and text alignment. Asdemonstrated in Figure 7.1, our approach excels in preserving subject-specific details whilemaintaining robust generalization across diverse textual contexts.This paper validates the effectiveness of our approach through extensive experimen-tation on the DreamBench dataset [243], which contains a broad spectrum of subjects,including complex items like drink cans and backpacks, as well as living animals withdiverse poses and movements. Our experiments encompass a range of tasks, includingsubject re-contextualization, attribute modification, accessorization, and style transfer.Quantitative assessments using CLIP-I/CLIP-T and DINO scores, coupled with compar-isons against DreamBooth, confirm our method’s ability to proficiently preserve subjectidentity, including intricate details like logos, while demonstrating robust generalizationacross diverse textual inputs. Additionally, we highlight limitations in the original CLIP-Tscore used in DreamBooth [243] and propose an enhanced version of CLIP-T tailored tooffer more precise and informative indications of text-image alignment.We summarize our contributions in the following aspects:1. A data-oriented approach to tackle overfitting of background and subject pose duringdiffusion model fine-tuning.2. A modification to the CLIP-T score to enhance its capacity for indicating text alignment.3. A systematic exploration of diverse methods for generating a regularization dataset, withthe intention of providing valuable insights to the broader research community.Ethics - Risks. Enabling the generation of realistic images from textual prompts anda designated subject poses a potential risk for misuse in the creation of deep fakes. Incontrast to the methods outlined in Chapters 3, 4, 5, and 6, this approach avoids explicitobject structure leakage but produces high-quality fake images with minimal input. For anin-depth examination of potential risks, kindly refer to Section 8.1, taking into account thebroader context of the methods presented in this dissertation.7.2 Related WorkThe high-level relation to implicit structured learning is given in Chapter 2. In the following,we focus on diffusion-based text-to-image models, and the methods curating data insteadof architectures.Diffusion Based Text-to-Image Models have recently demonstrated remarkableprogress, primarily through the utilization of diffusion models [97]. These achievementsare particularly pronounced when employing large models trained on extensive datasets[15, 205, 225, 230, 238, 246]. Our method extends the pre-trained diffusion models toincorporate personalized concepts. We apply this method to well-established models, suchas StableDiffusion [238] and StableDiffusionXL [225]. Note that our method is fundamentallyrooted in data, making it potentially applicable to a wide range of different diffusion modelarchitectures.937.3. MethodText-to-Image Personalization aims to imbue pre-trained diffusion models with theability to produce novel images of specific subjects [66, 243]. We give a detailed introductionon text-to-image personalization in Chapter 2 Section 2.3. While most following methodsdemonstrate on multi-subject generation [145], memory-saving [78, 228, 266, 287, 299] andacceleration [12, 67, 150, 244, 255], our approach focuses on maximizing quality at the costof training time, improving both identity preservation and text alignment relative to priorwork. We leave the topic of accelerated training as an avenue for future exploration.Involving More Data is our core idea. The most similar approach is proposed inDreamBooth [243]. They propose generating a prior dataset using prompt “a [class noun]”,e.g., “a backpack”, to alleviate overfitting. However, our experiments reveal that thisgenerated dataset lacks the strength to effectively counter overfitting. [145] extract the realtext-image pairs that similar to the training examples from existing dataset, but they stillneed to restrict the number of training iterations to mitigate overfitting risks. [33] speed upcustomization with a feed forward network trained on many datasets of custom objects,prioritizing speed over quality improvements. [179] propose to generate masks, boundingboxes, and their corresponding tags to train a better adapter and thus a better compositionmethod. Our work is primarily centered on enhancing generation quality, with the aspectof speed left for future exploration.7.3 MethodOur main goal is to fine-tune pre-trained diffusion models using a limited set of subject-specific training examples, typically around 4 to 6 images. This fine-tuning process is aimedat enabling these models to generate new images of the same subject. In another word, wewant to overfit the subject appearance and shape. However, we also want to prevent themodel from overfitting to background or the subject pose in the training images. How tooverfit the subject appearance and shape without overfitting to the background and subjectpose in the training examples is a challenging task. One existing approach to address this isby creating a regularization dataset of text-image pairs, employing prompts in the format“a [class noun]” (e.g., “a backpack”). This dataset serves as a counterbalance to the trainingexamples, which often involve specialized identifier word indicating the subject, such as “asks backpack” [243]. During training, both the training examples and the regularizationexamples are fed into the diffusion model. However, we discovered that the regularizationeffect is limited as the generated images and their prompts lack diversity. In contrast toprevious methods using a simplistic prompt for the real training examples, we first introducedindividual and more concrete prompts, such as “a backpack on a rock”. Surprisingly, thisadjustment did not yield significant improvements, and the model still exhibited rapidoverfitting. It became effective only when combined with a strong regularization datasetusing prompts that describe foreground and background in more detail. Thus, our focusshifted towards automatically constructing an appropriate regularization dataset to address947.3. Methodthis form of overfitting. The overview of our method is presented in Figure 7.2. Importantly,our approach retains the model architecture while concentrating on the development of anovel regularization dataset to combat overfitting effectively. This means that it can beseamlessly combined with any personalization algorithm using a prior preservation dataset.1. a <new> backpack2. a <new> backpack3. a <new> backpackOriginal training examples Step1: Add background prompts for example images1. a contoured orchid woven backpack on a rock2. a semicircular turquoise canvas backpack hanging on a tree branch3. a rhomboid coral crinkled backpack on a women’s back1. a cone-shaped eggshell scaly backpack amidst the aromas of a lively outdoor market2. a spiky green knitted backpack in the heart of a bustling culinary street…Step2: Sample different subjects with the same class noun and same backgroundStep3: Sample different subjects with the same class noun and different background1. a <new> backpack on a rock2. a <new> backpack hanging on a tree3. a <new> backpack on a women’s backFigure 7.2: Overview.. In contrast to prior approaches, we introduce specific promptsto the training examples and create a regularization dataset with a wider range of imagesguided by these prompts. To further boost diversity of regularization dataset, we generateadditional prompts and images using the same prompt formats.Enhancing Training with Prompts . The inclusion of prompts in training instancesplays a pivotal role in our approach. While employing a uniform prompt format such as “a[identifier] [class noun]” for all training examples may seem straightforward, it inadvertentlyincreases the risk of overfitting to the training data. To address this challenge, we developthis idea further by incorporating background descriptions into each image, as depictedin the upper-right corner of Figure 7.2. Additionally, we replace the provided class namewith more specific ones, which will be discussed further in Section 7.4.1. It’s important957.3. Methodto note that merely adding the background prompt in isolation does not yield the desiredbenefits. Instead, our contribution is on constructing a dedicated regularization dataset,which proves to be essential in mitigating overfitting, as elaborated in subsequent sections.Generating against training prompts . We present a novel approach for generatinga regularization dataset, which builds upon the original example prompt. For instance,let’s consider the initial prompt “a backpack on a rock”. We enhance this prompt byintroducing structural components in the format of “a [shape] [color] [texture] [class noun][background]”, where [shape], [color], and [texture] represent randomly selected adjectivesdrawn from a pool of 100 options for each attribute. These adjectives are then combinedto create prompts for image generation, resulting in prompts such as “a contoured orchidwoven backpack on a rock” and “a semicircular turquoise canvas backpack hanging on a treebranch”, as demonstrated in the lower-left corner of Figure 7.2. The pools of the adjectivesare generated automatically by Large Language Models [210]. To be more specific, fornon-live objects, we use the following prompts to generate words describing shape, color,textures in ChatGPT:• shape: give me 100 adjective words describing the shape of an object• color: give me 100 adjective words describing the color of an object• texture: give me 100 adjective words describing the texture of an objectAfter removing duplicated ones, there are 85 shapes, 93 colors, and 96 textures.Amplifying Diversity with Structured Prompts . To increase the diversity of ourregularization dataset, we randomly generate 500 additional background phrases and 100style phrases. We then combine these phrases in a random manner with the object-relatedprompts, resulting in structured prompts in the format “a [style] [shape] [color] [texture][class noun] [background]”. For instance, this approach generates prompts like “a photo ofa cone-shaped eggshell scaly backpack amidst the aromas of a lively outdoor market” and“a children’s storybook illustration of a trapezoidal coral embossed backpack against thecanvas of a city skyline”, as showcased in the lower-right corner of Figure 7.2. This strategysubstantially enhances image diversity compared to the simplistic use of the prompt “a[class noun]”. To generate random style and background prompts, we use• style: give me 100 image style descriptions, such as “a photo of”, and “a painting of”.• background: give me 500 phrases that describe the background, such as “on thetable”, as diverse as possible.After removing duplicated ones, there are 99 styles and 455 backgrounds left.967.3. MethodAdaption for Living Entities . For living entities, we use descriptors like body, skin/fur,and emotion instead of shape, color, and texture. Additionally, we introduce motion into thebackgrounds, transitioning from static scenes like “in an urban city” to dynamic contextssuch as “walking in an urban city”. To be more specific, for live objects, we use the followingprompts to generate words describing shape, color, textures, and motion in ChatGPT:• body: give me 100 adjective words describing the body of an animal• skin: give me 100 adjective words describing the skin or fur of an animal• emotion: give me 100 adjective words describing the emotion of an animal• motion: give me 1000 different short concise sentences that contains a special token“$concept” which stands for a specific animal, which can be a dog, a cat or a human.For example: “a $concept sitting in a temple”, “a $concept walking in a supermarket”.Keep “a $concept” in the sentences.After removing duplicated ones, there are 89 bodies, 86 skins/furs, 75 emotions, and744 motions.Dropout . In our strategy to enrich the dataset, we include three descriptive wordsfor each object, which increases diversity. However, this approach can lead to a potentialissue: overfitting to sentence structure. The model might learn that one adjective wordcorresponds to a specific subject, while three adjective words signify a different subject,impacting its ability to generalize. To mitigate this, we introduce randomness by randomlyexcluding some words during training. For example, we may have a prompt like “a [shape][color] backpack on a rock” excluding a texture descriptor. This dropout technique adds anelement of unpredictability, strengthening the model’s adaptability and reducing its relianceon fixed sentence formats.Cropping . [287] point out that the model is prone to overfit to the image layout whenattempting to learn personalized concepts from a limited set of examples. Similarly, [145]observe that employing random cropping enhances convergence speed and yields improvedresults. To enhance our model’s performance, we incorporate random cropping with avariable ratio, ranging from 0.75 to 1. It’s worth noting that for SDXL, we incorporatecropping coordinates as described by [225], where we increase the original image size toachieve a cropped image size of 1024x1024 pixels.Implementation Details We create 2000 regularization images for each subject, dis-tributed as follows: 20% from training prompts, 60% photorealistic images, and 20% styledimages. We opt for the identifier word “olis” instead of the more commonly used “sks”.This choice is based on the fact that “olis” corresponds to the least frequently utilized token977.3. Methodin the model’s vocabulary [1]. Each training batch contains one example from training setand one example from regularization set. For SD, we fine-tune the entire model with alearning rate of 2e-6 and perform inference using 200 steps of DDIM [267]. For SDXL, whichhas a larger model size, we employ a LoRA with a rank of 32 for both the text encoders andUNet. We also train the text embeddings. We set learning rate to 1e-4. We use 50 steps ofDDIM for inference. While it’s worth noting that different datasets exhibit varying degreesof diversity, we find that employing different early stopping points can enhance diversity.For simplicity, we use 4000/8000 iterations for SD/SDXL in this paper. Additional detailsregarding the optimal iteration range can be found in Table 7.1.subject name best #iterations on SD best #iterations on SDXLbackpack 6000-8000 8000-10000backpack dog 2000-3000 4000-6000bear plushie 2000-4000 4000-6000berry bowl 6000-8000 8000-10000can 6000-8000 8000-10000candle 4000-6000 8000-10000cat 1000-3000 1000-3000cat2 6000-8000 8000-10000clock 6000-8000 8000-10000colorful sneaker 4000-6000 6000-8000dog 1000-3000 1000-3000dog2 2000-4000 4000-6000dog3 2000-4000 8000-10000dog5 3000-4000 6000-8000dog6 3000-4000 6000-8000dog7 3000-4000 6000-8000dog8 1000-3000 1000-3000duck toy 3000-4000 3000-4000fancy boot 3000-4000 6000-8000grey sloth plushie 3000-4000 6000-8000monster toy 3000-4000 8000-10000pink sunglasses 3000-4000 4000-6000poop emoji 3000-4000 4000-6000rc car 3000-4000 4000-6000red cartoon 6000-8000 8000-10000robot toy 3000-4000 6000-8000shiny sneaker 3000-4000 6000-8000teapot 6000-8000 8000-10000vase 6000-8000 8000-10000wolf plushie 3000-4000 4000-6000Table 7.1: Best #iterations of datasets in DreamBench.. The variation mainly comesfrom the diversity of the dataset itself.987.4. Experiments7.4 ExperimentsWe employ the DreamBench dataset [243], featuring 30 subjects ranging from backpacksand stuffed animals to dogs and cats. The dataset encompasses 25 distinct testing promptsfor inanimate objects and 25 for living entities. Following the original paper, we generatefour images per prompt, yielding a total of 3000 images for evaluation. Importantly, thetesting prompts used in our experimental results are included in our regularization dataset.Our baseline model is DreamBooth (SD backbone) [243]. For a fair comparison, we takethe following steps: 1) Employ SDXL to generate 2000 high-quality regularization images;2) Use “olis” as the identifier word; 3) Set the learning rate to 2e-6 for 1500 iterations; and4) Apply 200 steps of DDIM for inference.toy: 0.268poop emoji toy: 0.3667toy: 0.278duck toy: 0.325toy: 0.245teddy bear toy: 0.285toy: 0.245teddy bear toy: 0.276Prompt: a {} in the grass next to a tree Prompt: a {} sitting on a sofa Figure 7.3: CLIP-T score on different subject names.. We randomly generate animage with the prompt “a teddy bear toy sitting beside a river”. Subsequently, we evaluateprompts “a in the grass next to a tree” and “a sitting on a sofa”. When employing thevague prompt “toy”, both the ground truth and the mismatched example are classifiedas mismatches (<0.3). In contrast, when using specific class names, the ground truthis categorized as a match (>0.3), while the mismatched examples remain classified asmismatches.7.4.1 Result ComparisonThe qualitative comparison is depicted in Figure 7.1. Our approach outperforms alternativemethods in preserving intricate subject-specific details, ensuring identity retention. Thisis evident in the preservation of fine elements like logos on drink cans, unique icons onbackpacks, and fine hair textures on toy monsters. While other methods have their strengths,they may struggle with preserving these details, emphasizing the superior subject fidelity ofour approach.997.4. ExperimentsMethods Backbone DINO ↑ CLIP-I ↑ CLIP-T (vague class) ↑ CLIP-T (object) ↑Real Image (on training set) - 0.774 0.885 0.298 0.311Textual Inversion [66] SD 0.569 0.780 0.255 -Textual Inversion (our impl) SD 0.611 0.772 0.267 0.289DreamBooth [243] SD 0.668 0.803 0.305 -DreamBooth (our impl) SD 0.682 0.808 0.301 0.306Ours SD 0.704 0.824 0.293 0.309Ours SDXL 0.744 0.842 0.297 0.312Table 7.2: Evaluation on the DreamBench..In our quantitative assessment, we employ DINO [25] and CLIP-I [229], followingDreamBooth, to evaluate subject fidelity. We argue that the CLIP-T used in DreamBoothmay yield inaccurate measurements of text-image alignment when vague class names areemployed. As shown in Figure 7.3, there is a notable discrepancy between the utilization ofvague class names, such as “toy”, and more specific object names, such as “duck toy”, onthe ground truth images. Notably, the CLIP-T score appears to be significantly influencedby the nomenclature chosen for the object, thereby potentially undermining its accuracy asan indicator of text-image alignment. To delve deeper into this matter, we calculate theCLIP-T score on the original images and the manually added prompts. Table 7.2 presentsthat, when using vague class names, the CLIP-T score for ground truth text-image pairsfalls even below the conventional threshold of 0.3, typically considered as the thresholdfor assessing text-image pair compatibility [252]. To rectify this issue, we replace vagueclass names with highly specific object names, resulting in a substantial improvement in theCLIP-T score for ground truth text-image pairs. We show the name change in Table 7.3.subject name original class modified classbear plushie stuffed animal bear plushieberry bowl bowl berry bowlcan can drink canclock clock alarm clockduck toy toy duck toygrey sloth plushie stuffed animal sloth plushiemonster toy toy monster toypoop emoji toy poop emoji toyrc car toy racing car toyred cartoon cartoon 2d cartoon devilrobot toy toy robot toywolf plushie stuffed animal wolf plushieTable 7.3: Name change.. We change the name for a more reasonable CLIP-T metric andbetter performance.1007.4. ExperimentsA <new> backpack in Grand Canyona van Gogh painting of a <new> drink can on the beacha <new> dog jumping in a muddy puddlew/o reg dataset#iters=6000 Input imageexamplesOursw/o format#iters=10000#iters=4000#iters=2000#iters=1000w/o reg datasetOursw/o formatw/o reg datasetOursw/o formatIncreasing training iterations#iters=6000 Input imageexamples#iters=10000#iters=4000#iters=2000#iters=1000#iters=6000 Input imageexamples#iters=10000#iters=4000#iters=2000#iters=1000Figure 7.4: Overfitting prevention.. Our regularization dataset prevents overfitting.1017.4. ExperimentsDreamBoothOurs (SDXL) Ours (SD)a <new> dog playing football in a playgroundInput ImageExample DreamBoothOurs (SDXL) Ours (SD)In a playground, a <new> dog playing footballDreamBoothOurs (SDXL) Ours (SD)a <new> toy sitting in a sea of rosesInput ImageExample DreamBoothOurs (SDXL) Ours (SD)in a sea of roses, a <new> toy sittingFigure 7.5: Rephrasing prompts.. Even when using rephrased prompts, our methodmaintains subject identity, a quality DreamBooth lacks. Notably, SDXL consistentlygenerates images in response to rephrased prompts with minimal deviation from theoriginals.7.4.2 Model AnalysisHow important is the formatted prompts? We conducted an ablation test withtwo variations: 1) without using a regularization dataset, and 2) using a regularizationdataset with simple prompts in the format “a [class noun]”. The results are illustrated inFigure 7.4. These findings emphasize the importance of both the regularization datasetand the utilization of well-structured prompts. Notably, when employing a regularizationdataset without well-structured prompts, there is an initial increase in diversity during earlytraining phases compared to scenarios without any regularization dataset. However, overextended training iterations, a noticeable decline in model performance becomes evident.We attribute this decline to the incongruity between the overly simplistic prompt format “a[class noun]” and the complexities of the generated images. This observation underscoresthe pivotal role of the well-structured prompts in enhancing model performance, thushighlighting their significance in our approach.Does the model overfit to the formatted prompts? To assess the impact ofprompt phrasing variations, we rephrase the prompts and compare them to the originals, asillustrated in Figure 7.5. While our model consistently employs formatted prompts startingwith “a <new>[class noun]” during training, it demonstrates resilience against overfittingto this specific format. In contrast, DreamBooth displays a notable departure from subject1027.4. ExperimentsOurs (SDXL) Ours (SDXL)a <new> dog eating sushiInput ImageExamplea <new1> dog playing guitarInput ImageExample Ours (SD)Ours (SDXL) Ours (SDXL)Ours (SD) w/o ada (SDXL) w/o ada (SDXL) w/o ada (SD)Ours (SDXL) Ours (SDXL) Ours (SD)Figure 7.6: Influence of adaption to living entities.. Without adaptation, the modelmight overlook motion in the prompts and focus solely on assembling objects within theimages.identity, confirming our assertion that the simplistic use of unvaried prompts in the format“a [class noun]” for constructing a regularization dataset can lead to overfitting to sentencestructure. Additionally, it is noteworthy that SDXL exhibits a higher degree of consistencyin generating images in response to rephrased prompts, with minimal deviation from theoriginals, thereby effectively preserving the underlying image structure.Does the adaption to living entities help? The key difference between inanimateobjects and living things is motion. Training without adaptation tends to overlook motionwhile still generating objects separately, as shown in Figure 7.6. However, it’s importantto note that adaptation makes generating motion easier, but the model can still producemotion even without it.Is the random dropout of adjective words necessary? When altering attributeslike color, we found that using random dropout is essential. Without it, the model tends toexcessively preserve the original identity, making color changes challenging. With randomdropout, the task becomes easier while still preserving identity, as shown in Figure 7.7. It’sworth noting that for color modification, using “a <new>[color] [class noun]” works betterthan “a [color] <new>[class noun]”.1037.5. Limitationspurpleblue greena <new> {} monster toyInput ImageExample purpleblue greenSDXL SDXLSDXL SD SD SDpurpleblue greena <new> {} robot toyInput ImageExample purpleblue greenSDXL SDXLSDXL SD SD SDFigure 7.7: Color modification.. Our method can alter the color of the subject. It isimportant to mention that when modifying the color, using “a <new>[color] [class noun]”is more effective than “a [color] <new>[class noun]”.7.5 LimitationsDespite its impressive quality, our method has two limitations: 2) It needs to generate aregularization dataset of 2000 images for each category, which could extend the overalltraining time; 2) On a single A100 GPU, it requires approximately 1.5 hours for SD and3.5 hours for SDXL to complete 4000 and 8000 training iterations, respectively. We leaveacceleration to future research. Currently, we assume that the real images of the targetobject are annotated manually. In the future, caption generator could be used, such asBLIP [153, 154].7.6 ConclusionWe introduced a data-centric approach to enhance diffusion model personalization, using astructured regularization dataset. By incorporating formatted prompts and their generatedimages, this method effectively reduces overfitting, allowing for up to a 5x longer trainingduration. This results in significant improvements in image quality, identity preservation,and diversity of generated samples aligned with input text prompts. Although this approachrequires extended training times and increased memory resources, we foresee potential1047.6. Conclusionsolutions in the future, as our method sets the stage for forthcoming data-centric approaches.Notably, our method complements existing work by focusing on data augmentation ratherthan model architecture adjustments, making it potentially compatible for integration withprevious research.105Chapter 8Conclusion and Future Work8.1 EthicsIt is important to note that while the proposed methods enable a range of new applicationscenarios, they can also be abused for deep fakes, surveillance, and other manipulativeactivities. The methods in Chapters 3, 4, 5, and 6 extract object structures from onlineimages, which can be used as the driving signal for generative models, such as face swapping.Fake animations can be generated with only a single image. The method in Chapter 7could be used to generate unwanted images for a specific person given a few selfies. In thefollowing, we consider a range of important scenarios in which such image and video editingand structure detection could be abused. It is essential to keep in mind that this list willnever be exhaustive, and there is potential for the development of even more sophisticatedexploits in the future.Deep fake applications include not only slanderous and discredited video generation butalso identity spoofing. The stolen appearance can be used to decrypt phones, or even carsor houses. In particular, hackers may attack self-driving cars, targeting lives instead of justassets. On the other hand, the keypoints themselves may partially encode personal identity,including the shape. If a sequence of keypoints is extracted from videos, personal habitsmay also be learned.The methods in Chapters 3, 4, 5, and 6 could also be exploited for unwanted surveillanceapplications. Since these methods require little to no annotation, they are more applicablefor in-the-wild images than traditional supervised methods.Furthermore, as the methods in Chapters 3, 4, and 5 require no annotation, and themethods in Chapters 6 and 7 require only a few annotations, they significantly lower thebar and cost for criminal usage. Any person can use a personal laptop to conduct crimes,including deep fakes, identity theft, and property hacks.In addition to informing the general public about the capabilities of the most recentgenerative image models, there should be regulations on the market and the usage ofthese techniques to prevent unwanted abuse. Furthermore, it is important to research fakedetection techniques [209, 290] for the rapidly advancing image generation technology.1068.2. Conclusion8.2 ConclusionWhile learning structure information from unstructured images is a natural ability for humanbeings, it is much more challenging than learning from annotations. In this dissertation, wemake contributions to self-supervised structure learning from images without or with only afew annotations. We believe that our method is a step towards general artificial intelligence.In Chapter 3, we introduce LatentKeypointGAN, a framework based on GenerativeAdversarial Networks (GANs). This model is internally conditioned on keypoints andtheir appearance encoding. Instead of starting with a pre-trained StyleGAN, we constructdisentangled representations from scratch, resulting in an interpretable hidden space thatallows for intuitive spatial editing using control handles. LatentKeypointGAN also supportsthe creation of image-keypoint pairs, offering a novel approach to unsupervised keypointdetection, traditionally tackled using autoencoders.In Chapter 4, we introduced GANSeg, an approach based on GANs that focuses onlearning part segmentation from unlabeled image collections. Our pivotal hierarchicalgenerator design synthesizes images step by step, incorporating independence and invariancewithin the process. This design provides a promising alternative to current autoencodermethods and sets the stage for future exploration into learning part-based 3D models from2D images.In Chapter 5, we introduced AutoLink, a method aimed at learning a spatial graphrepresentation from unlabeled image collections through the reconstruction of maskedimages. The key component is our adaptable graph design, which captures the connectionsbetween various keypoints. This approach is less complex than current options and laysthe groundwork for advancements in image comprehension, image manipulation, and theacquisition of 3D models from 2D images.In Chapter 6, we introduced a few-shot keypoint localization method that incorporateskeypoint detection with uncertainty, 2D/3D geometric constraints, and image reconstruction.These elements prevent the detector from overfitting to limited examples and make useof unlabeled images. Our experimental results show that, with only a few annotations,our model performs effectively across various datasets, including complex scenarios likesoft, articulated objects, as well as the challenging mouth interior, which has not beenattempted previously. In the few-shot setting, our model significantly outperforms standardapproaches and succeeds on datasets where others struggle with just 10 or 20 annotatedexamples. This approach paves the way for conditional generative modeling and imageediting using minimal annotated examples.In Chapter 7, we introduced a novel way to improve personalized diffusion modelsusing data-centric methods. We achieved this by using a structured regularization datasetand incorporating formatted prompts along with their generated images. This approacheffectively reduces overfitting, allowing for longer training periods—up to five times thatof current methods. As a result, we observed significant enhancements in image quality,preservation of identity, and diversity of generated samples based on input text prompts.1078.3. Future WorkWhile this method requires more time and memory resources for training, we believe thereare potential future solutions to address these challenges. This approach complementsexisting research by focusing on data augmentation rather than making adjustments to themodel architecture, making it compatible with previous studies.Our methods make advances toward self-supervised structure learning, but many chal-lenges remain to be solved, and many new directions can be explored. In the following, weoutline promising research directions for future explorations.8.3 Future WorkOur previous research has primarily centered on unsupervised object correspondence andstructure learning. Our future plan will focus on leveraging existing resources, such aslarge-scale pre-trained models, human annotations, and motion information, to enhance ourlearning outcomes. By combining these with self-supervised learning, we hope to achieveeven better performance.8.3.1 Utilizing Pre-trained ModelsLarge-scale self-supervised models such as DINO [25], have shown promising learning abilityin object saliency segmentation. The extracted deep features can be further clustered forfiner scene segmentation [77] and object part segmentation [7]. These models are typicallytrained on massive datasets, leading to robust learned representations. However, similar tomost existing unsupervised methods [37, 91, 103], the learned masks do not fit the objectboundary well and struggle distinguishing the left and right. However, they can serve as auseful prior for more fine-grained structure learning and transfer learning.Besides the discriminative self-supervised models, generative models. such as StyleGANs[126, 129], also show strong ability in spatial feature disentanglement. Their features can beclustered for part segmentation [42, 214]. While they provide finer object boundaries, theyneed to be trained on large-scale dataset where objects are roughly aligned in the center.Diffusion Models [97] extend the generative models to generating high-quality high-resolution in-the-wild images on very diverse object categories. Researchers has show thatthe pre-trained diffusion models have spatial feature disentanglement and correspondencein the middle feature maps [94, 201, 282, 295]. However, how to best utilize the spatialdisentanglement is still an open question.In addition to directly using the features in pre-trained model for unsupervised learning,utilizing features in semi-supervised learning could be a simpler and more effective way.The problem of previous self-supervised methods still exists in the ones using pre-traineddiffusion models. For example, neither of them could solve left/right ambiguity in symmetricobejcts [94, 282]. From our work on few-shot learning [86], perhaps a few human annotationis a simpler way to solve this problem. This pose a new problem of how to utilize thepre-trained features for semi-supervised learning.1088.3. Future Work8.3.2 Semi-supervised Learning via Dataset GenerationHow to best utilize a pre-trained large-scale model for semi-supervised learning is a long termtopic in this area. Previous methods try to train an adapter to transform the feature mapsto the desired representation, such as segmentation masks [369] and keypoint heatmaps[21, 369].Recently, a dataset-generation approach emerges [14, 16, 29, 87, 177, 243]. To extractthe information from large-scale pre-trained models, instead of designing new networkmodules, they guide the models to generate desired data, such as text-image pairs andmask-image pairs. These methods shows superior performance than manual designedmodules. Furthermore, they are shown to be more model-agnostic and thus may be easierto apply to new base models than manual designed modules. We believe it can be animportant future direction.In particular, text-to-image diffusion models have shown much higher generation qualitythan other generative models [225, 238, 246]. We believe it will be the base model of datasetgeneration methods. Recent methods try to embed the data annotation in text, such asbounding box coordinates [14] and object identity token [87, 243]. This simple way showssurprisingly superior performance.An promising future direction can be, whether more different representations can beinserted into text embedding? For example, keypoints may be directly inserted into text asfloating number tuple. By fine-tuning on a few annotated examples, people may use textto generate much more diverse examples than the original dataset. The same idea can beapplied to domain transfer problems.Another possible way to involve existing annotation can be ControlNet [363]. Insteadof inserting annotations into text, we can directly train additive modules, with spatialfeature maps as input, to adapt the text-to-image diffusion models to consume annotations.Afterwards, we can still use text to generate more diverse training set.A possible problem we might encounter is overfitting. With limited training annotations,fine-tuning diffusion models can easily overfit. We might be able to combine existingsemi-supervised methods with the text-to-image diffusion models. For example, we firstuse traditional semi-supervised methods to generate a reasonable amount of annotations,and fine-tune diffusion models afterwards. A second option can be that we adapt existingsemi-supervised modules to be compatible within text-to-image diffusion models to preventoverfitting in fine-tuning stage.8.3.3 Combination with Human AnnotationHuman annotations have been widely used in supervised learning [107, 164], but theirintegration with unsupervised learning has yet to be fully explored. A prior step has beenmade in representation pre-training [75, 123, 331], and semi-supervised learning [86, 197].However, they all focus on recognition tasks, such as segmentation and keypoint learning.1098.3. Future WorkOur plan is to merge the existing human annotated keypoints with our self-supervisedkeypoints for improved generative tasks, such as motion transfer.While human annotated keypoints are typically sparse and inadequate for capturingdetailed motion, the combination of both types of keypoints could lead to better results.For instance, in the case of human motion, only keypoints at skeleton joints are provided,neglecting the importance of cloth shape. It is proven that explicit keypoints can preservemotion better [260]. Therefore, using unsupervised keypoints for clothes is promising.Existing human-in-cloth models assume the cloth is fitting the body [44, 183], and cannothandle cloth with fat shape, such as large and long sleeve. By learning auxiliary clothkeypoints in an unsupervised manner, we aim to capture this aspect for more realisticmotion transfer, and more detailed conditional image generation.Our previous work [86] has shown that with the help of human annotation duringtraining, the accuracy of keypoints can be much higher than the ones without. A directextension of this work can be predicting both human annotated and unsupervised keypointsto reconstruction, and supervise the former ones with ground truth.8.3.4 Scene-Level Keypoints for Intuitive EditingUsing point on the image to move, replace, add, and remove objects is one of the mostintuitive and straightforward way for edition [215]. However, most current work focuses onediting a single foreground object in the image, lacking the ability to edit the whole scenes[53, 59, 112, 114, 137, 143, 173, 196, 257, 288, 324, 341, 366]. Recently, DragGAN [215]and DragDiffusions [198, 256] enables scene level intuitive editing by dragging an arbitraryselected pixel. However, they still lack the ability to add, remove, or appearance the embed-ding of a specific point or part, which is possible in our previous work LatentKeypointGAN[93]. Therefore, we would like to explore the scene-level intuitive editing by manipulatingarbitrary pixels or points in the image.The main challenge lies in this direction is to define a representation that fits all needs.Operating on arbitrary pixels does not allow addition and removal of objects, while operatingon keypoints does not work on arbitrary images. Our current initial idea is to divide theimage into coarse grid and modify the grid to edit the image.8.3.5 Self/Weakly/Semi-supervised 3D ReconstructionAs 3D annotations are substantially harder to obtain than 2D annotations, it is moredesired in self-supervised structure learning. Previous methods make multiple assumptionsabout the data distribution [58, 115, 322, 348–350] and object properties, such as symmetry[323, 352], and have multiple losses to balance. Furthermore, their generated shape is notdetailed enough, and are limited to objects instead of scenes. Recent attempts focuses oncombining pre-trained diffusion models with large 3D data [167, 168, 191, 281], or directlygenerating 3D representation from text-to-image diffusion models [161, 226, 280, 313].1108.3. Future WorkHowever, the former still require a large 3D dataset, lacking its ability to extend to differentcategories, and the latter may generate 3D shapes with multiple front faces, known as Janusproblem [13].To solve Janus problem, Armandpour et al. [13] propose to use negative text prompt toavoid generating front face on the side, but it only relieves Janus problem instead of solvingit. Nevertheless, we believe it shows a promising direction and the key to Janus problemmay lie in the text-image alignment. Even the training image distribution is biased towardshaving more front than side views of the object, models with strong text-image alignmentcan still generate correct views from the input text prompts [16].A possible solution can be fine-tune the pre-trained diffusion models on a few objectswith multiviews, which forms a semi-supervised learning. A second solution can be fine-tunethe models on text-image pairs with precise spatial description of the foreground objects,which forms weakly-supervised learning. In addition, training a self-supervised method togive roughly correct view information may help in providing more precise text and thusalso helps in 3D reconstruction.111Bibliography[1] 2kpr. dreambooth-tokens, 2022.[2] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed imagesinto the stylegan latent space? In Proceedings of the IEEE international conferenceon computer vision, pages 4432–4441, 2019.[3] Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan++: How to editthe embedded images? In Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition, pages 8296–8305, 2020.[4] Mahmoud Afifi. 11k hands: gender recognition and biometric identification using alarge dataset of hand images. Multimedia Tools and Applications, 2019.[5] Yazeed Alharbi and Peter Wonka. Disentangled image generation through structurednoise injection. In Proceedings of the IEEE/CVF Conference on Computer Visionand Pattern Recognition, pages 5134–5142, 2020.[6] Bilal Alsallakh, Narine Kokhlikyan, Vivek Miglani, Jun Yuan, and Orion Reblitz-Richardson. Mind the pad – {cnn}s can develop blind spots. In InternationalConference on Learning Representations, 2021.[7] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as densevisual descriptors. ECCVW What is Motion For?, 2022.[8] M. Andriluka, U. Iqbal, E. Insafutdinov, L. Pishchulin, A. Milan, J. Gall, and SchieleB. PoseTrack: A benchmark for human pose estimation and tracking. In CVPR,2018.[9] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele. 2d humanpose estimation: New benchmark and state of the art analysis. In IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR), June 2014.[10] Mykhaylo Andriluka, Leonid Pishchulin, Peter V. Gehler, and Bernt Schiele. 2dhuman pose estimation: New benchmark and state of the art analysis. In CVPR,pages 3686–3693. IEEE Computer Society, 2014.112Bibliography[11] Relja Arandjelovic´ and Andrew Zisserman. Object discovery with a copy-pasting gan.arXiv preprint arXiv:1905.11369, 2019.[12] Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir,and Amit H Bermano. Domain-agnostic tuning-encoder for fast personalization oftext-to-image models. arXiv preprint arXiv:2307.06925, 2023.[13] Mohammadreza Armandpour, Huangjie Zheng, Ali Sadeghian, Amir Sadeghian, andMingyuan Zhou. Re-imagine the negative prompt algorithm: Transform 2d diffusioninto 3d, alleviate janus problem and beyond. arXiv preprint arXiv:2304.04968, 2023.[14] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David JFleet. Synthetic data from diffusion models improves imagenet classification. arXivpreprint arXiv:2304.08466, 2023.[15] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, KarstenKreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprintarXiv:2211.01324, 2022.[16] James Betker, Gabriel Goh, Li Jing, † TimBrooks, Jianfeng Wang, Linjie Li, †LongOuyang, † JuntangZhuang, † JoyceLee, † YufeiGuo, † WesamManassra, † Pra-fullaDhariwal, † CaseyChu, † YunxinJiao, and Aditya Ramesh. Improving imagegeneration with better captions. 2023.[17] Adam Bielski and Paolo Favaro. Emergence of object segmentation in perturbedgenerative models. Advances in Neural Information Processing Systems, 32:7256–7266, 2019.[18] Volker Blanz and Thomas Vetter. A morphable model for the synthesis of 3d faces.In Proceedings of the 26th annual conference on Computer graphics and interactivetechniques, pages 187–194, 1999.[19] Sandro Braun. unsupervised-disentangling. https://github.com/theRealSuperMario/unsupervised-disentangling/tree/reproducing_baselines, 2020.[20] Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning tofollow image editing instructions. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pages 18392–18402, 2023.[21] Bjo¨rn Browatzki and Christian Wallraven. 3fabrec: Fast few-shot face alignment byreconstruction. In CVPR, pages 6109–6119. Computer Vision Foundation / IEEE,2020.113Bibliography[22] Chenjie Cao and Yanwei Fu. Learning a sketch tensor space for image inpaintingof man-made scenes. In Proceedings of the IEEE/CVF International Conference onComputer Vision, pages 14509–14518, 2021.[23] Nan Cao, Xin Yan, Yang Shi, and Chaoran Chen. Ai-sketcher : A deep generativemodel for producing high-quality sketches. Proceedings of the AAAI Conference onArtificial Intelligence, 33(01):2564–2571, Jul. 2019.[24] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi-person 2dpose estimation using part affinity fields. In Proceedings of the IEEE conference oncomputer vision and pattern recognition, pages 7291–7299, 2017.[25] Mathilde Caron, Hugo Touvron, Ishan Misra, Herve´ Je´gou, Julien Mairal, PiotrBojanowski, and Armand Joulin. Emerging properties in self-supervised visiontransformers. In Proceedings of the IEEE/CVF international conference on computervision, pages 9650–9660, 2021.[26] Prashanth Chandran, Derek Bradley, Markus H. Gross, and Thabo Beeler. Attention-driven cropping for very high resolution facial landmark detection. In CVPR, pages5860–5869. Computer Vision Foundation / IEEE, 2020.[27] James Charles, Tomas Pfister, Derek Magee, David Hogg, and Andrew Zisserman.Domain adaptation for upper body pose tracking in signed tv broadcasts. The BritishMachine Vision Conference, 2013.[28] Hong Chen, Yipeng Zhang, Xin Wang, Xuguang Duan, Yuwei Zhou, and Wenwu Zhu.Disenbooth: Disentangled parameter-efficient tuning for subject-driven text-to-imagegeneration. arXiv preprint arXiv:2305.03374, 2023.[29] Kai Chen, Enze Xie, Zhe Chen, Lanqing Hong, Zhenguo Li, and Dit-Yan Yeung.Integrating geometric control into text-to-image diffusion models for high-qualitydetection data generation via text prompt. arXiv preprint arXiv:2306.04607, 2023.[30] Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Re-thinking atrous convolution for semantic image segmentation. arXiv preprintarXiv:1706.05587, 2017.[31] Mickae¨l Chen, Thierry Artie`res, and Ludovic Denoyer. Unsupervised object segmenta-tion by redrawing. In Advances in Neural Information Processing Systems 32 (NIPS2019), pages 12705–12716, 2019.[32] Runnan Chen, Yuexin Ma, Lingjie Liu, Nenglun Chen, Zhiming Cui, Guodong Wei,and Wenping Wang. Semi-supervised anatomical landmark detection via shape-regulated self-training. Neurocomputing, 471:335–345, 2022.114Bibliography[33] Wenhu Chen, Hexiang Hu, Yandong Li, Nataniel Rui, Xuhui Jia, Ming-Wei Chang,and William W Cohen. Subject-driven text-to-image generation via apprenticeshiplearning. arXiv preprint arXiv:2304.00186, 2023.[34] Yajing Chen, Shikui Tu, Yuqi Yi, and Lei Xu. Sketch-pix2seq: a model to generatesketches of multiple categories. arXiv preprint arXiv:1709.04121, 2017.[35] Yen-Chi Cheng, Hsin-Ying Lee, Min Sun, and Ming-Hsuan Yang. Controllable imagesynthesis via segvae. In European Conference on Computer Vision, 2020.[36] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverseimage synthesis for multiple domains. In Proceedings of the IEEE/CVF conferenceon computer vision and pattern recognition, pages 8188–8197, 2020.[37] Subhabrata Choudhury, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Un-supervised part discovery from contrastive reconstruction. In A. Beygelzimer,Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in NeuralInformation Processing Systems, 2021.[38] Vasileios Choutas, Philippe Weinzaepfel, Je´roˆme Revaud, and Cordelia Schmid.Potion: Pose motion representation for action recognition. In CVPR, pages 7024–7033. Computer Vision Foundation / IEEE, 2018.[39] Xiangxiang Chu, Zhi Tian, Bo Zhang, Xinlong Wang, Xiaolin Wei, Huaxia Xia,and Chunhua Shen. Conditional positional encodings for vision transformers. arXivpreprint arXiv:2102.10882, 2021.[40] J. S. Chung, A. Nagrani, and A. Zisserman. Voxceleb2: Deep speaker recognition. InINTERSPEECH, 2018.[41] Edo Collins, Radhakrishna Achanta, and Sabine Susstrunk. Deep feature factorizationfor concept discovery. In Proceedings of the European Conference on ComputerVision (ECCV), pages 336–352, 2018.[42] Edo Collins, Raja Bala, Bob Price, and Sabine Susstrunk. Editing in style: Uncov-ering the local semantics of gans. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pages 5771–5780, 2020.[43] Wenyan Cong, Jianfu Zhang, Li Niu, Liu Liu, Zhixin Ling, Weiyuan Li, and LiqingZhang. Dovenet: Deep image harmonization via domain verification. In Proceedingsof the IEEE/CVF conference on computer vision and pattern recognition, pages8394–8403, 2020.[44] Enric Corona, Albert Pumarola, Guillem Alenya, Gerard Pons-Moll, and FrancescMoreno-Noguer. Smplicit: Topology-aware generative model for clothed people. In115BibliographyProceedings of the IEEE/CVF conference on computer vision and pattern recognition,pages 11875–11885, 2021.[45] Eric Crawford and Joelle Pineau. Spatially invariant unsupervised object detectionwith convolutional neural networks. In Proceedings of the AAAI Conference onArtificial Intelligence, pages 3412–3420, 2019.[46] G. Csurka, C. Bray, C. Dance, and L. Fan. Visual categorization with bags ofkeypoints. Workshop on Stat. Learn. Comput. Vis., ECCV, Vol. 1:1–22, 2004.[47] Radek Daneˇcˇek, Michael J. Black, and Timo Bolkart. Emoca: Emotion drivenmonocular face capture and animation. In Proceedings of the IEEE/CVF Conferenceon Computer Vision and Pattern Recognition (CVPR), pages 20311–20322, June2022.[48] Ayan Das, Yongxin Yang, Timothy Hospedales, Tao Xiang, and Yi-Zhe Song.Be´ziersketch: A generative model for scalable vector sketches. In European Conferenceon Computer Vision, pages 632–647. Springer, 2020.[49] Debayan Deb and Anil K. Jain. Look locally infer globally: A generalizable faceanti-spoofing approach. IEEE Trans. Inf. Forensics Secur., 16:1143–1157, 2021.[50] Yu Deng, Jiaolong Yang, Dong Chen, Fang Wen, and Xin Tong. Disentangled andcontrollable face image generation via 3d imitative-contrastive learning. In IEEEComputer Vision and Pattern Recognition, 2020.[51] Piotr Dolla´r, Peter Welinder, and Pietro Perona. Cascaded pose regression. In CVPR,pages 1078–1085. IEEE Computer Society, 2010.[52] Xuanyi Dong and Yi Yang. Teacher supervises students how to learn from partiallylabeled images for facial landmark detection. In ICCV, pages 783–792. IEEE, 2019.[53] Xuanyi Dong, Shoou-I Yu, Xinshuo Weng, Shih-En Wei, Yi Yang, and Yaser Sheikh.Supervision-by-registration: An unsupervised approach to improve the precision offacial landmark detectors. In Proceedings of the IEEE Conference on ComputerVision and Pattern Recognition, pages 360–368, 2018.[54] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, XiaohuaZhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold,Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for imagerecognition at scale. In International Conference on Learning Representations, 2020.[55] Alexey Dosovitskiy and Thomas Brox. Generating images with perceptual similaritymetrics based on deep networks. Advances in neural information processing systems,29, 2016.116Bibliography[56] Charles Dugas, Yoshua Bengio, Franc¸ois Be´lisle, Claude Nadeau, and Rene´ Garcia.Incorporating second-order functional knowledge for better option pricing. In T. Leen,T. Dietterich, and V. Tresp, editors, Advances in Neural Information ProcessingSystems, volume 13. MIT Press, 2000.[57] Charles Dugas, Yoshua Bengio, Franc¸ois Be´lisle, Claude Nadeau, and Rene´ Garcia.Incorporating second-order functional knowledge for better option pricing. In NeurIPS,pages 472–478. MIT Press, 2000.[58] Shivam Duggal and Deepak Pathak. Topologically-aware deformation fields for single-view 3d reconstruction. In Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition, pages 1536–1546, 2022.[59] Aysegul Dundar, Kevin J Shih, Animesh Garg, Robert Pottorf, Andrew Tao, andBryan Catanzaro. Unsupervised disentanglement of pose, appearance and backgroundfrom images and videos. arXiv preprint arXiv:2001.09518, 2020.[60] Mathias Eitz, James Hays, and Marc Alexa. How do humans sketch objects? ACMTrans. Graph. (Proc. SIGGRAPH), 31(4):44:1–44:10, 2012.[61] Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. RMPE: Regional multi-personpose estimation. In ICCV, 2017.[62] Qi Feng, Kun He, He Wen, Cem Keskin, and Yuting Ye. Active learning withpseudo-labels for multi-view 3d pose estimation. arXiv preprint arXiv:2112.13709,2021.[63] Yao Feng, Fan Wu, Xiaohu Shao, Yanfeng Wang, and Xi Zhou. Joint 3d face recon-struction and dense alignment with position map regression network. In Proceedingsof the European conference on computer vision (ECCV), pages 534–551, 2018.[64] Yao Feng, Fan Wu, Xiaohu Shao, Yanfeng Wang, and Xi Zhou. Joint 3d facereconstruction and dense alignment with position map regression network. In ECCV,volume 11218 of Lecture Notes in Computer Science, pages 557–574. Springer, 2018.[65] David Ferman and Gaurav Bharaj. Multi-domain multi-definition landmark localiza-tion for small datasets. CoRR, abs/2203.10358, 2022.[66] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik,and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-imagegeneration using textual inversion. arXiv preprint arXiv:2208.01618, 2022.[67] Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and DanielCohen-Or. Designing an encoder for fast personalization of text-to-image models.arXiv preprint arXiv:2302.12228, 2023.117Bibliography[68] Qingzhe Gao, Bin Wang, Libin Liu, and Baoquan Chen. Unsupervised co-partsegmentation through assembly. arXiv preprint arXiv:2106.05897, 2021.[69] Leon A Gatys, Alexander S Ecker, and Matthias Bethge. Image style transfer usingconvolutional neural networks. In Proceedings of the IEEE conference on computervision and pattern recognition, pages 2414–2423, 2016.[70] Songwei Ge, Vedanuj Goswami, Larry Zitnick, and Devi Parikh. Creative sketchgeneration. In International Conference on Learning Representations, 2021.[71] Partha Ghosh, Pravir Singh Gupta, Roy Uziel, Anurag Ranjan, Michael Black, andTimo Bolkart. Gif: Generative interpretable faces. arXiv preprint arXiv:2009.00149,2020.[72] James J. Gibson. The Ecological Approach to Visual Perception. Houghton Mifflin,Boston, 1979.[73] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley,Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. InAdvances in neural information processing systems, pages 2672–2680, 2014.[74] Shanyan Guan, Ying Tai, Bingbing Ni, Feida Zhu, Feiyue Huang, and Xiaokang Yang.Collaborative learning for faster stylegan embedding. arXiv preprint arXiv:2007.01758,2020.[75] Kamal Gupta, Saurabh Singh, and Abhinav Shrivastava. Patchvae: Learning lo-cal latent codes for recognition. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pages 4746–4755, 2020.[76] David Ha and Douglas Eck. A neural representation of sketch drawings. InInternational Conference on Learning Representations, 2018.[77] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T.Freeman. Unsupervised semantic segmentation by distilling feature correspondences.In International Conference on Learning Representations, 2022.[78] Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and FengYang. Svdiff: Compact parameter space for diffusion fine-tuning. arXiv preprintarXiv:2303.11305, 2023.[79] Mengya Han, Heliang Zheng, Chaoyue Wang, Yong Luo, Han Hu, and Bo Du.Leveraging gan priors for few-shot part segmentation. In Proceedings of the 30thACM International Conference on Multimedia, pages 1339–1347, 2022.118Bibliography[80] Xintong Han, Zuxuan Wu, Weilin Huang, Matthew R. Scott, and Larry S. Davis.Finet: Compatible and diverse fashion image inpainting. In Proceedings of theIEEE/CVF International Conference on Computer Vision (ICCV), October 2019.[81] Shaozhe Hao, Kai Han, Shihao Zhao, and Kwan-Yee K Wong. Vico: Detail-preserving visual condition for personalized text-to-image generation. arXiv preprintarXiv:2306.00971, 2023.[82] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dolla´r, and Ross Girshick.Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.06377,2021.[83] Kaiming He, Georgia Gkioxari, Piotr Dolla´r, and Ross Girshick. Mask r-cnn. InProceedings of the IEEE international conference on computer vision, pages 2961–2969, 2017.[84] Kaiming He, Georgia Gkioxari, Piotr Dolla´r, and Ross B. Girshick. Mask R-CNN. InICCV, pages 2980–2988. IEEE Computer Society, 2017.[85] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning forimage recognition. In Proceedings of the IEEE conference on computer vision andpattern recognition, pages 770–778, 2016.[86] Xingzhe He, Gaurav Bharaj, David Ferman, Helge Rhodin, and Pablo Garrido.Few-shot geometry-aware keypoint localization. preprint, 2023.[87] Xingzhe He, Zhiwen Cao, Nicholas Kolkin, Lantao Yu, Helge Rhodin, and RatheeshKalarot. A data perspective on enhanced identity preservation for diffusion personal-ization. arXiv preprint arXiv:2311.04315, 2023.[88] Xingzhe He, Bastian Wandt, and Helge Rhodin. Latentkeypointgan: Controlling gansvia latent keypoints. arXiv preprint arXiv:2103.15812, 2021.[89] Xingzhe He, Bastian Wandt, and Helge Rhodin. Autolink: Self-supervised learningof human skeletons and object outlines by linking keypoints. In Advances in NeuralInformation Processing Systems, 2022.[90] Xingzhe He, Bastian Wandt, and Helge Rhodin. Autolink: Self-supervised learning ofhuman skeletons and object outlines by linking keypoints. In NeurIPS, 2022.[91] Xingzhe He, Bastian Wandt, and Helge Rhodin. Ganseg: Learning to segmentby unsupervised hierarchical image generation. In Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition, pages 1225–1235, 2022.119Bibliography[92] Xingzhe He, Bastian Wandt, and Helge Rhodin. Ganseg: Learning to segmentby unsupervised hierarchical image generation. In Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition, pages 1225–1235, 2022.[93] Xingzhe He, Bastian Wandt, and Helge Rhodin. Latentkeypointgan: Controllinggans via latent keypoints. In Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition Workshop, 2022.[94] Eric Hedlin, Gopal Sharma, Shweta Mahajan, Hossam Isack, Abhishek Kar, AndreaTagliasacchi, and Kwang Moo Yi. Unsupervised semantic correspondence using stablediffusion. arXiv preprint arXiv:2305.15581, 2023.[95] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and SeppHochreiter. Gans trained by a two time-scale update rule converge to a local nashequilibrium. In Advances in neural information processing systems, pages 6626–6637,2017.[96] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.NeurIPS, 33:6840–6851, 2020.[97] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models.Advances in Neural Information Processing Systems, 33:6840–6851, 2020.[98] Derek Hoiem, Yodsawalai Chodpathumwan, and Qieyun Dai. Diagnosing error inobject detectors. In Andrew W. Fitzgibbon, Svetlana Lazebnik, Pietro Perona,Yoichi Sato, and Cordelia Schmid, editors, ECCV, volume 7574 of Lecture Notes inComputer Science, pages 340–353. Springer, 2012.[99] Sina Honari, Pavlo Molchanov, Stephen Tyree, Pascal Vincent, Christopher J. Pal,and Jan Kautz. Improving landmark localization with semi-supervised learning. InCVPR, pages 1546–1555. Computer Vision Foundation / IEEE, 2018.[100] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, SheanWang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large languagemodels. arXiv preprint arXiv:2106.09685, 2021.[101] Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptiveinstance normalization. In Proceedings of the IEEE International Conference onComputer Vision, pages 1501–1510, 2017.[102] Zixuan Huang and Yin Li. Interpretable and accurate fine-grained recognition viaregion grouping. In Proceedings of the IEEE/CVF Conference on Computer Visionand Pattern Recognition, pages 8662–8672, 2020.120Bibliography[103] Wei-Chih Hung, Varun Jampani, Sifei Liu, Pavlo Molchanov, Ming-Hsuan Yang,and Jan Kautz. Scops: Self-supervised co-part segmentation. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 869–878,2019.[104] Eldar Insafutdinov, Leonid Pishchulin, Bjoern Andres, Mykhaylo Andriluka, andBernt Schiele. Deepercut: A deeper, stronger, and faster multi-person pose estimationmodel. In ECCV, volume 9910 of Lecture Notes in Computer Science, pages 34–50.Springer, 2016.[105] Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep networktraining by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.[106] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m:Large scale datasets and predictive methods for 3d human sensing in natural environ-ments. IEEE Transactions on Pattern Analysis and Machine Intelligence, 36(7):1325–1339, 2014.[107] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu. Human3.6m:Large scale datasets and predictive methods for 3d human sensing in natural environ-ments. IEEE TPAMI, 36(7):1325–1339, 2014.[108] Md Amirul Islam, Sen Jia, and Neil DB Bruce. How much position information doconvolutional neural networks encode? arXiv preprint arXiv:2001.08248, 2020.[109] Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image trans-lation with conditional adversarial networks. In Proceedings of the IEEE conferenceon computer vision and pattern recognition, pages 1125–1134, 2017.[110] Aaron S Jackson, Adrian Bulat, Vasileios Argyriou, and Georgios Tzimiropoulos.Large pose 3d face reconstruction from a single image via direct volumetric cnnregression. In Proceedings of the IEEE international conference on computer vision,pages 1031–1039, 2017.[111] Ali Jahanian*, Lucy Chai*, and Phillip Isola. On the ”steerability” of generativeadversarial networks. In International Conference on Learning Representations, 2020.[112] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervisedlearning of object landmarks through conditional image generation. In Advances inneural information processing systems, pages 4016–4027, 2018.[113] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Unsupervisedlearning of object landmarks through conditional image generation. Advances inneural information processing systems, 31, 2018.121Bibliography[114] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi. Self-supervisedlearning of interpretable keypoints from unlabelled videos. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8787–8797, 2020.[115] Tomas Jakab, Ruining Li, Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi.Farm3D: Learning articulated 3d animals by distilling 2d diffusion. arXiv preprintarXiv:2304.10535, 2023.[116] Yong Shi Jie Yang, Zhiquan Qi. Learning to incorporate structure knowledge forimage inpainting. In Proceedings of the AAAI Conference on Artificial Intelligence,pages 12605–12612, 2020.[117] Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang,and Ping Luo. Whole-body human pose estimation in the wild. In ECCV, volume12354 of Lecture Notes in Computer Science, pages 196–214. Springer, 2020.[118] Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time styletransfer and super-resolution. In European conference on computer vision, pages694–711. Springer, 2016.[119] Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs.In Proceedings of the IEEE conference on computer vision and pattern recognition,pages 1219–1228, 2018.[120] Sam Johnson and Mark Everingham. Clustered pose and nonlinear appearance modelsfor human pose estimation. In Proceedings of the British Machine Vision Conference,pages 12.1–12.11. BMVA Press, 2010. doi:10.5244/C.24.12.[121] Kurmanbek Kaiyrbekov and Metin Sezgin. Deep stroke-based sketched symbol recon-struction and segmentation. IEEE computer graphics and applications, 40(1):112–126,2019.[122] Moritz Kampelmuhler and Axel Pinz. Synthesizing human-like sketches from naturalimages using a conditional convolutional decoder. In Proceedings of the IEEE/CVFWinter Conference on Applications of Computer Vision, pages 3203–3211, 2020.[123] Guoliang Kang, Yunchao Wei, Yi Yang, Yueting Zhuang, and Alexander Haupt-mann. Pixel-level cycle association: A new perspective for domain adaptive semanticsegmentation. Advances in Neural Information Processing Systems, 33:3569–3580,2020.[124] Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growingof gans for improved quality, stability, and variation. In International Conference onLearning Representations, 2018.122Bibliography[125] Tero Karras, Miika Aittala, Samuli Laine, Erik Ha¨rko¨nen, Janne Hellsten, JaakkoLehtinen, and Timo Aila. Alias-free generative adversarial networks. arXiv preprintarXiv:2106.12423, 2021.[126] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecturefor generative adversarial networks. In Proceedings of the IEEE/CVF conference oncomputer vision and pattern recognition, pages 4401–4410, 2019.[127] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture forgenerative adversarial networks. In Proceedings of the IEEE conference on computervision and pattern recognition, pages 4401–4410, 2019.[128] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, andTimo Aila. Analyzing and improving the image quality of stylegan. In CVPR, pages8110–8119, 2020.[129] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, andTimo Aila. Analyzing and improving the image quality of stylegan. In Proceedingsof the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages8110–8119, 2020.[130] Yoni Kasten, Dolev Ofri, Oliver Wang, and Tali Dekel. Layered neural atlases forconsistent video editing. ACM Transactions on Graphics (TOG), 40(6):1–12, 2021.[131] Isinsu Katircioglu, Helge Rhodin, Victor Constantin, Jorg Sporri, Mathieu Salzmann,and Pascal Fua. Self-supervised human detection and segmentation via backgroundinpainting. IEEE Transactions on Pattern Analysis & Machine Intelligence, 01:1–1,2021.[132] Isinsu Katircioglu, Helge Rhodin, Jorg Sporri, Mathieu Salzmann, and Pascal Fua.Human detection and segmentation via multi-view consensus. In Proceedings of theIEEE/CVF International Conference on Computer Vision, pages 2855–2864, 2021.[133] Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, InbarMosseri, and Michal Irani. Imagic: Text-based real image editing with diffusionmodels. In Proceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition, pages 6007–6017, 2023.[134] Osman Semih Kayhan and Jan C van Gemert. On translation invariance in cnns:Convolutional layers can exploit absolute spatial location. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14274–14285, 2020.123Bibliography[135] Martin Kiefel and Peter Vincent Gehler. Human pose estimation with fields of parts.In David Fleet, Tomas Pajdla, Bernt Schiele, and Tinne Tuytelaars, editors, ComputerVision – ECCV 2014, pages 331–346, Cham, 2014. Springer International Publishing.[136] Hyunsu Kim, Yunjey Choi, Junho Kim, Sungjoo Yoo, and Youngjung Uh. Exploitingspatial dimensions of latent in gan for real-time image editing. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 852–861,2021.[137] Yunji Kim, Seonghyeon Nam, In Cho, and Seon Joo Kim. Unsupervised keypoint learn-ing for guiding class-conditional video prediction. In Advances in Neural InformationProcessing Systems, pages 3814–3824, 2019.[138] Davis E. King. Dlib-ml: A machine learning toolkit. Journal of Machine LearningResearch, 10:1755–1758, 2009.[139] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization.arXiv preprint arXiv:1412.6980, 2014.[140] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. InInternational Conference on Learning Representations, 2015.[141] Martin Ko¨stinger, Paul Wohlhart, Peter M. Roth, and Horst Bischof. Annotatedfacial landmarks in the wild: A large-scale, real-world database for facial landmarklocalization. In ICCV Workshops, pages 2144–2151. IEEE Computer Society, 2011.[142] Rakshit Sunil Kothari, Zhizhuo Yang, Christopher Kanan, Reynold Bailey, Jeff B. Pelz,and Gabriel J. Diaz. Gaze-in-wild: A dataset for studying eye and head coordinationin everyday activities. CoRR, abs/1905.13146, 2019.[143] Tejas D Kulkarni, Ankush Gupta, Catalin Ionescu, Sebastian Borgeaud, MalcolmReynolds, Andrew Zisserman, and Volodymyr Mnih. Unsupervised learning of objectkeypoints for perception and control. In Advances in neural information processingsystems, pages 10724–10734, 2019.[144] Abhinav Kumar, Tim K. Marks, Wenxuan Mou, Ye Wang, Michael Jones, AnoopCherian, Toshiaki Koike-Akino, Xiaoming Liu, and Chen Feng. Luvli face alignment:Estimating landmarks’ location, uncertainty, and visibility likelihood. In CVPR,pages 8233–8243. Computer Vision Foundation / IEEE, 2020.[145] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-YanZhu. Multi-concept customization of text-to-image diffusion. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1931–1941, 2023.124Bibliography[146] Gihyun Kwon and Jong Chul Ye. Diagonal attention and style-based gan for content-style disentanglement in image generation and translation. In Proceedings of theIEEE international conference on computer vision, pages 13980–13989, 2021.[147] Avisek Lahiri, Arnav Kumar Jain, Sanskar Agrawal, Pabitra Mitra, and Prabir KumarBiswas. Prior guided gan based semantic inpainting. In Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition (CVPR), June 2020.[148] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, andL. D. Jackel. Backpropagation applied to handwritten zip code recognition. NeuralComputation, 1(4):541–551, 1989.[149] Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semanticsegmentation with generative models: Semi-supervised learning and strong out-of-domain generalization. In Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition, pages 8300–8311, 2021.[150] Dongxu Li, Junnan Li, and Steven CH Hoi. Blip-diffusion: Pre-trained subjectrepresentation for controllable text-to-image generation and editing. arXiv preprintarXiv:2305.14720, 2023.[151] Jiefeng Li, Siyuan Bian, Ailing Zeng, Can Wang, Bo Pang, Wentao Liu, and CewuLu. Human pose regression with residual log-likelihood estimation. In ICCV, pages11005–11014. IEEE, 2021.[152] Jingyuan Li, Fengxiang He, Lefei Zhang, Bo Du, and Dacheng Tao. Progressive recon-struction of visual structure for image inpainting. In Proceedings of the IEEE/CVFInternational Conference on Computer Vision (ICCV), October 2019.[153] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrappinglanguage-image pre-training with frozen image encoders and large language models.arXiv preprint arXiv:2301.12597, 2023.[154] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. InInternational Conference on Machine Learning, pages 12888–12900. PMLR, 2022.[155] Shuyuan Li, Jianguo Li, Hanlin Tang, Rui Qian, and Weiyao Lin. ATRW: A benchmarkfor amur tiger re-identification in the wild. In ICM, pages 2590–2598. ACM, 2020.[156] Tianye Li, Timo Bolkart, Michael. J. Black, Hao Li, and Javier Romero. Learning amodel of facial shape and expression from 4D scans. ACM Transactions on Graphics,(Proc. SIGGRAPH Asia), 36(6):194:1–194:17, 2017.125Bibliography[157] Xueting Li, Sifei Liu, Kihwan Kim, Shalini De Mello, Varun Jampani, Ming-HsuanYang, and Jan Kautz. Self-supervised single-view 3d reconstruction via semanticconsistency. In European Conference on Computer Vision, pages 677–693. Springer,2020.[158] Yijun Li, Lu Jiang, and Ming-Hsuan Yang. Controllable and progressive imageextrapolation. In Proceedings of the IEEE/CVF Winter Conference on Applicationsof Computer Vision, pages 2140–2149, 2021.[159] Yining Li, Chen Huang, and Chen Change Loy. Dense intrinsic appearance flow forhuman pose transfer. In Proceedings of the IEEE/CVF Conference on ComputerVision and Pattern Recognition (CVPR), June 2019.[160] Liang Liao, Jing Xiao, Zheng Wang, Chia-Wen Lin, and Shin’ichi Satoh. Guidanceand evaluation: Semantic-aware image inpainting for mixed scenes. In EuropeanConference on Computer Vision, pages 683–700. Springer, 2020.[161] Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, XunHuang, Karsten Kreis, Sanja Fidler, Ming-Yu Liu, and Tsung-Yi Lin. Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conferenceon Computer Vision and Pattern Recognition, pages 300–309, 2023.[162] Chen-Hsuan Lin, Ersin Yumer, Oliver Wang, Eli Shechtman, and Simon Lucey. St-gan: Spatial transformer generative adversarial networks for image compositing. InProceedings of the IEEE Conference on Computer Vision and Pattern Recognition,pages 9455–9464, 2018.[163] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, DevaRamanan, Piotr Dolla´r, and C Lawrence Zitnick. Microsoft coco: Common objects incontext. In European conference on computer vision, pages 740–755. Springer, 2014.[164] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, DevaRamanan, Piotr Dolla´r, and C. Lawrence Zitnick. Microsoft COCO: common objectsin context. In ECCV, volume 8693 of Lecture Notes in Computer Science, pages740–755. Springer, 2014.[165] Huan Ling, Karsten Kreis, Daiqing Li, Seung Wook Kim, Antonio Torralba, andSanja Fidler. Editgan: High-precision semantic image editing. In Advances in NeuralInformation Processing Systems (NeurIPS), 2021.[166] Pengyang Ling, Lin Chen, Pan Zhang, Huaian Chen, and Yi Jin. Freedrag: Pointtracking is not you need for interactive point-based image editing. arXiv preprintarXiv:2307.04684, 2023.126Bibliography[167] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Zexiang Xu, Hao Su, et al. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization.arXiv preprint arXiv:2306.16928, 2023.[168] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, andCarl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object. In Proceedings of theIEEE/CVF International Conference on Computer Vision, pages 9298–9309, 2023.[169] Shilong Liu, Lei Zhang, Xiao Yang, Hang Su, and Jun Zhu. Unsupervised partsegmentation through disentangling appearance and shape. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8355–8364, 2021.[170] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang. Deepfashion:Powering robust clothes recognition and retrieval with rich annotations. In Proceedingsof IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June2016.[171] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributesin the wild. In Proceedings of International Conference on Computer Vision (ICCV),December 2015.[172] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J.Black. SMPL: A skinned multi-person linear model. ACM Trans. Graphics (Proc.SIGGRAPH Asia), 34(6):248:1–248:16, October 2015.[173] Dominik Lorenz, Leonard Bereska, Timo Milbich, and Bjorn Ommer. Unsupervisedpart-based disentangling of object shape and appearance. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, pages 10955–10964, 2019.[174] Dominik Lorenz, Leonard Bereska, Timo Milbich, and Bjorn Ommer. Unsupervisedpart-based disentangling of object shape and appearance. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10955–10964, 2019.[175] Erika Lu, Forrester Cole, Tali Dekel, Weidi Xie, Andrew Zisserman, David Salesin,William T Freeman, and Michael Rubinstein. Layered neural rendering for retimingpeople in video. In SIGGRAPH Asia, 2020.[176] Erika Lu, Forrester Cole, Tali Dekel, Andrew Zisserman, William T Freeman, andMichael Rubinstein. Omnimatte: Associating objects and their effects in video.In Proceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition, pages 4507–4515, 2021.127Bibliography[177] Yingzhou Lu, Huazheng Wang, and Wenqi Wei. Machine learning for synthetic datageneration: a review. arXiv preprint arXiv:2302.04062, 2023.[178] Yuhang Lu, Weijian Li, Kang Zheng, Yirui Wang, Adam P. Harrison, Chihung Lin,Song Wang, Jing Xiao, Le Lu, Chang-Fu Kuo, and Shun Miao. Learning to segmentanatomical structures accurately from one exemplar. In MICCAI, volume 12261 ofLecture Notes in Computer Science, pages 678–688. Springer, 2020.[179] Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject-diffusion: Open domainpersonalized text-to-image generation without test-time fine-tuning. arXiv preprintarXiv:2307.11410, 2023.[180] Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool.Pose guided person image generation. Advances in neural information processingsystems, 30, 2017.[181] Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, and MarioFritz. Disentangled person image generation. In CVPR, pages 99–108. ComputerVision Foundation / IEEE, 2018.[182] Liqian Ma, Qianru Sun, Stamatios Georgoulis, Luc Van Gool, Bernt Schiele, andMario Fritz. Disentangled person image generation. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, pages 99–108, 2018.[183] Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, SiyuTang, and Michael J. Black. Learning to dress 3d people in generative clothing. InComputer Vision and Pattern Recognition (CVPR), June 2020.[184] Andrew L Maas, Awni Y Hannun, and Andrew Y Ng. Rectifier nonlinearities improveneural network acoustic models. In in ICML Workshop on Deep Learning for Audio,Speech and Language Processing. Citeseer, 2013.[185] R. Malladi, J.A. Sethian, and B.C. Vemuri. Shape modeling with front propagation: alevel set approach. IEEE Transactions on Pattern Analysis and Machine Intelligence,17(2):158–175, 1995.[186] Alexander Mathis, Pranav Mamidanna, Kevin M. Cury, Taiga Abe, Venkatesh N.Murthy, M. Mathis, and Matthias Bethge. Deeplabcut: markerless pose estimationof user-defined body parts with deep learning. Nature Neuroscience, 21:1281–1289,2018.[187] Alexander Mathis, Steffen Schneider, Jessy Lauer, and Mackenzie W. Mathis. Aprimer on motion capture with deep learning: Principles, pitfalls, and perspectives.Neuron, 108(1):44–65, 2020.128Bibliography[188] Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko,Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation in thewild using improved cnn supervision. In 3D Vision (3DV), 2017 Fifth InternationalConference on. IEEE, 2017.[189] Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko,Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation in thewild using improved cnn supervision. In 3D Vision (3DV), 2017 Fifth InternationalConference on. IEEE, 2017.[190] Dushyant Mehta, Helge Rhodin, Dan Casas, Pascal Fua, Oleksandr Sotnychenko,Weipeng Xu, and Christian Theobalt. Monocular 3d human pose estimation in thewild using improved CNN supervision. In 3DV, pages 506–516. IEEE ComputerSociety, 2017.[191] Luke Melas-Kyriazi, Iro Laina, Christian Rupprecht, and Andrea Vedaldi. Realfusion:360deg reconstruction of any object from a single image. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8446–8455, 2023.[192] Yifang Men, Yiming Mao, Yuning Jiang, Wei-Ying Ma, and Zhouhui Lian. Control-lable person image synthesis with attribute-decomposed gan. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June2020.[193] Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods forgans do actually converge? In International conference on machine learning, pages3481–3490. PMLR, 2018.[194] Daniela Mihai and Jonathon Hare. Differentiable drawing and sketching. arXivpreprint arXiv:2103.16194, 2021.[195] Daniela Mihai and Jonathon S. Hare. Differentiable drawing and sketching. CoRR,abs/2103.16194, 2021.[196] Matthias Minderer, Chen Sun, Ruben Villegas, Forrester Cole, Kevin P Murphy, andHonglak Lee. Unsupervised learning of object structure and dynamics from videos.In Advances in Neural Information Processing Systems, pages 92–102, 2019.[197] Olga Moskvyak, Fre´de´ric Maire, Feras Dayoub, and Mahsa Baktashmotlagh. Semi-supervised keypoint localization. In ICLR. OpenReview.net, 2021.[198] Chong Mou, Xintao Wang, Jiechong Song, Ying Shan, and Jian Zhang. Dragon-diffusion: Enabling drag-style manipulation on diffusion models. arXiv preprintarXiv:2307.02421, 2023.129Bibliography[199] Rito Murase, Masanori Suganuma, and Takayuki Okatani. How can cnns use imageposition for segmentation? arXiv preprint arXiv:2005.03463, 2020.[200] Nitinraj Nair, Rakshit Sunil Kothari, Aayush K. Chaudhary, Zhizhuo Yang, Gabriel J.Diaz, Jeff B. Pelz, and Reynold J. Bailey. Rit-eyes: Rendering of near-eye images foreye-tracking applications. In ACM SAP, pages 5:1–5:9. ACM, 2020.[201] Jisu Nam, Gyuseong Lee, Sunwoo Kim, Hyeonsu Kim, Hyoungwon Cho, Seyeon Kim,and Seungryong Kim. Diffmatch: Diffusion model for dense matching. arXiv preprintarXiv:2305.19094, 2023.[202] Kamyar Nazeri, Eric Ng, Tony Joseph, Faisal Qureshi, and Mehran Ebrahimi. Edge-connect: Structure guided image inpainting using edge prediction. In The IEEEInternational Conference on Computer Vision (ICCV) Workshops, Oct 2019.[203] Andrew Y. Ng, Michael I. Jordan, and Yair Weiss. On spectral clustering: Analysisand an algorithm. In Proceedings of the 14th International Conference on NeuralInformation Processing Systems: Natural and Synthetic, NIPS’01, page 849–856,Cambridge, MA, USA, 2001. MIT Press.[204] Thu Nguyen-Phuoc, Christian Richardt, Long Mai, Yong-Liang Yang, and NiloyMitra. Blockgan: Learning 3d object-aware scene representations from unlabelledimages. In Advances in Neural Information Processing Systems 33, Nov 2020.[205] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, BobMcGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image gener-ation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741,2021.[206] Michael Niemeyer and Andreas Geiger. Giraffe: Representing scenes as compositionalgenerative neural feature fields. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pages 11453–11464, 2021.[207] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification overa large number of classes. In Indian Conference on Computer Vision, Graphics andImage Processing, Dec 2008.[208] Atsuhiro Noguchi, Umar Iqbal, Jonathan Tremblay, Tatsuya Harada, and OrazioGallo. Watch it move: Unsupervised discovery of 3D joints for re-posing of articulatedobjects. In Proceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition, 2022.[209] Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards universal fake image detectorsthat generalize across generative models. In Proceedings of the IEEE/CVF Conferenceon Computer Vision and Pattern Recognition, pages 24480–24489, 2023.130Bibliography[210] OpenAI. Chatgpt, 2023.[211] Maxime Oquab, Timothe´e Darcet, The´o Moutakanni, Huy Vo, Marc Szafraniec, VasilKhalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby,et al. Dinov2: Learning robust visual features without supervision. arXiv preprintarXiv:2304.07193, 2023.[212] Roy Or-El, Xuan Luo, Mengyi Shan, Eli Shechtman, Jeong Joon Park, and IraKemelmacher-Shlizerman. Stylesdf: High-resolution 3d-consistent image and geometrygeneration. In CVPR, pages 13493–13503. IEEE, 2022.[213] Ahmed AA Osman, Timo Bolkart, and Michael J Black. Star: Sparse trainedarticulated human body regressor. In European Conference on Computer Vision,pages 598–613. Springer, 2020.[214] Daniil Pakhomov, Sanchit Hira, Narayani Wagle, Kemar E Green, and Nassir Navab.Segmentation in style: Unsupervised semantic image segmentation with stylegan andclip. arXiv preprint arXiv:2107.12518, 2021.[215] Xingang Pan, Ayush Tewari, Thomas Leimku¨hler, Lingjie Liu, Abhimitra Meka, andChristian Theobalt. Drag your gan: Interactive point-based manipulation on thegenerative image manifold. In ACM SIGGRAPH 2023 Conference Proceedings, pages1–11, 2023.[216] Kaiyue Pang, Da Li, Jifei Song, Yi-Zhe Song, Tao Xiang, and Timothy M Hospedales.Deep factorised inverse-sketching. In Proceedings of the European Conference onComputer Vision (ECCV), pages 36–52, 2018.[217] George Papandreou, Tyler Zhu, Liang-Chieh Chen, Spyros Gidaris, Jonathan Tompson,and Kevin Murphy. Personlab: Person pose estimation and instance segmentationwith a bottom-up, part-based, geometric embedding model. In Proceedings of theEuropean conference on computer vision (ECCV), pages 269–286, 2018.[218] Taesung Park, Ming-Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic im-age synthesis with spatially-adaptive normalization. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, pages 2337–2346, 2019.[219] Priyanka Patel, Chun-Hao P. Huang, Joachim Tesch, David T. Hoffmann, ShashankTripathi, and Michael J. Black. AGORA: avatars in geography optimized for regressionanalysis. In CVPR, pages 13468–13478. Computer Vision Foundation / IEEE, 2021.[220] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A.Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3D hands,face, and body from a single image. In Proceedings IEEE Conf. on Computer Visionand Pattern Recognition (CVPR), pages 10975–10985, 2019.131Bibliography[221] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A.Osman, Dimitrios Tzionas, and Michael J. Black. Expressive body capture: 3d hands,face, and body from a single image. In CVPR, pages 10975–10985. Computer VisionFoundation / IEEE, 2019.[222] Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed AAOsman, Dimitrios Tzionas, and Michael J Black. Expressive body capture: 3d hands,face, and body from a single image. In Proceedings of the IEEE/CVF conference oncomputer vision and pattern recognition, pages 10975–10985, 2019.[223] Dario Pavllo, Christoph Feichtenhofer, David Grangier, and Michael Auli. 3d humanpose estimation in video with temporal convolutions and semi-supervised train-ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition, pages 7753–7762, 2019.[224] Patrick Pe´rez, Michel Gangnet, and Andrew Blake. Poisson image editing. ACMTrans. Graph., 22(3):313–318, jul 2003.[225] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, JonasMu¨ller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models forhigh-resolution image synthesis. arXiv preprint arXiv:2307.01952, 2023.[226] Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022.[227] Shengju Qian, Keqiang Sun, Wayne Wu, Chen Qian, and Jiaya Jia. Aggregationvia separation: Boosting facial landmark detector with semi-supervised style transla-tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision,pages 10153–10163, 2019.[228] Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang,Adrian Weller, and Bernhard Scho¨lkopf. Controlling text-to-image diffusion byorthogonal finetuning. arXiv preprint arXiv:2306.07280, 2023.[229] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, SandhiniAgarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learn-ing transferable visual models from natural language supervision. In Internationalconference on machine learning, pages 8748–8763. PMLR, 2021.[230] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.Hierarchical text-conditional image generation with clip latents. arXiv preprintarXiv:2204.06125, 1(2):3, 2022.132Bibliography[231] Alex Rav-Acha, Pushmeet Kohli, Carsten Rother, and Andrew Fitzgibbon. Unwrapmosaics: A new representation for video editing. ACM Trans. Graph., 27(3):1–11,aug 2008.[232] N. Dinesh Reddy, Minh Vo, and Srinivasa G. Narasimhan. Carfusion: Combiningpoint tracking and part detection for dynamic 3d reconstruction of vehicles. In CVPR,pages 1906–1915. Computer Vision Foundation / IEEE, 2018.[233] Yurui Ren, Xiaoming Yu, Ruonan Zhang, Thomas H Li, Shan Liu, and Ge Li.Structureflow: Image inpainting via structure-aware appearance flow. In Proceedingsof the IEEE/CVF International Conference on Computer Vision, pages 181–190,2019.[234] Helge Rhodin, Victor Constantin, Isinsu Katircioglu, Mathieu Salzmann, and PascalFua. Neural scene decomposition for multi-person motion capture. In Proceedings ofthe IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR),June 2019.[235] Helge Rhodin, Mathieu Salzmann, and Pascal Fua. Unsupervised geometry-aware rep-resentation for 3d human pose estimation. In Proceedings of the European Conferenceon Computer Vision (ECCV), pages 750–767, 2018.[236] Elad Richardson, Yuval Alaluf, Or Patashnik, Yotam Nitzan, Yaniv Azar, Stav Shapiro,and Daniel Cohen-Or. Encoding in style: a stylegan encoder for image-to-imagetranslation. arXiv preprint arXiv:2008.00951, 2020.[237] Iasonas Kokkinos Riza Alp Gu¨ler, Natalia Neverova. Densepose: Dense human poseestimation in the wild. In The IEEE Conference on Computer Vision and PatternRecognition (CVPR), 2018.[238] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjo¨rnOmmer. High-resolution image synthesis with latent diffusion models. In Proceedingsof the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages10684–10695, 2022.[239] Matteo Ruggero Ronchi and Pietro Perona. Benchmarking and error diagnosis inmulti-instance pose estimation. In ICCV, pages 369–378. IEEE Computer Society,2017.[240] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networksfor biomedical image segmentation. In International Conference on Medical imagecomputing and computer-assisted intervention, pages 234–241. Springer, 2015.133Bibliography[241] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networksfor biomedical image segmentation. In MICCAI, volume 9351 of Lecture Notes inComputer Science, pages 234–241. Springer, 2015.[242] Carsten Rother, Vladimir Kolmogorov, and Andrew Blake. ”grabcut” interactiveforeground extraction using iterated graph cuts. ACM transactions on graphics(TOG), 23(3):309–314, 2004.[243] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, andKfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF Conference on Computer Visionand Pattern Recognition, pages 22500–22510, 2023.[244] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Wei Wei, Tingbo Hou, Yael Pritch, NealWadhwa, Michael Rubinstein, and Kfir Aberman. Hyperdreambooth: Hypernetworksfor fast personalization of text-to-image models. arXiv preprint arXiv:2307.06949,2023.[245] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma,Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenetlarge scale visual recognition challenge. International journal of computer vision,115:211–252, 2015.[246] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily LDenton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, TimSalimans, et al. Photorealistic text-to-image diffusion models with deep languageunderstanding. Advances in Neural Information Processing Systems, 35:36479–36494,2022.[247] Patsorn Sangkloy, Nathan Burnell, Cusuh Ham, and James Hays. The sketchydatabase: Learning to retrieve badly drawn bunnies. ACM Transactions on Graphics(proceedings of SIGGRAPH), 2016.[248] Kazuma Sasaki, Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. Learningto Restore Deteriorated Line Drawing. The Visual Computer (Proc. of ComputerGraphics International 2018), 34(6-8):1077–1085, 2018.[249] Pedro Savarese, Sunnie SY Kim, Michael Maire, Greg Shakhnarovich, and DavidMcAllester. Information-theoretic segmentation by inpainting error maximization.In Proceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition, pages 4029–4039, 2021.[250] Luca Schmidtke, Athanasios Vlontzos, Simon Ellershaw, Anna Lukens, Tomoki Arichi,and Bernhard Kainz. Unsupervised human pose estimation through transforming134Bibliographyshape templates. In Proceedings of the IEEE/CVF Conference on Computer Visionand Pattern Recognition, pages 2484–2494, 2021.[251] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, RossWightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, MitchellWortsman, et al. Laion-5b: An open large-scale dataset for training next generationimage-text models. Advances in Neural Information Processing Systems, 35:25278–25294, 2022.[252] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk,Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki.Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprintarXiv:2111.02114, 2021.[253] Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. https://github.com/mseitzer/pytorch-fid, August 2020. Version 0.1.1.[254] Yujun Shen, Jinjin Gu, Xiaoou Tang, and Bolei Zhou. Interpreting the latent spaceof gans for semantic face editing. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pages 9243–9252, 2020.[255] Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304.03411,2023.[256] Yujun Shi, Chuhui Xue, Jiachun Pan, Wenqing Zhang, Vincent YF Tan, and SongBai. Dragdiffusion: Harnessing diffusion models for interactive point-based imageediting. arXiv preprint arXiv:2306.14435, 2023.[257] Aliaksandr Siarohin, Ste´phane Lathuilie`re, Sergey Tulyakov, Elisa Ricci, and NicuSebe. Animating arbitrary objects via deep motion transfer. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages 2377–2386,2019.[258] Aliaksandr Siarohin, Ste´phane Lathuilie`re, Sergey Tulyakov, Elisa Ricci, and NicuSebe. Animating arbitrary objects via deep motion transfer. In CVPR, pages 2377–2386. Computer Vision Foundation / IEEE, 2019.[259] Aliaksandr Siarohin, Ste´phane Lathuilie`re, Sergey Tulyakov, Elisa Ricci, and NicuSebe. First order motion model for image animation. In NeurIPS, pages 7135–7145,2019.[260] Aliaksandr Siarohin, Ste´phane Lathuilie`re, Sergey Tulyakov, Elisa Ricci, and NicuSebe. First order motion model for image animation. In Conference on NeuralInformation Processing Systems (NeurIPS), December 2019.135Bibliography[261] Aliaksandr Siarohin, Subhankar Roy, Ste´phane Lathuilie`re, Sergey Tulyakov, ElisaRicci, and Nicu Sebe. Motion-supervised co-part segmentation. In 2020 25thInternational Conference on Pattern Recognition (ICPR), pages 9650–9657. IEEE,2021.[262] Aliaksandr Siarohin, Enver Sangineto, Ste´phane Lathuiliere, and Nicu Sebe. De-formable gans for pose-based human image generation. In Proceedings of the IEEEConference on Computer Vision and Pattern Recognition, pages 3408–3416, 2018.[263] Aliaksandr Siarohin, Enver Sangineto, Ste´phane Lathuilie`re, and Nicu Sebe. De-formable gans for pose-based human image generation. In CVPR, pages 3408–3416.Computer Vision Foundation / IEEE, 2018.[264] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks forlarge-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.[265] Krishna Kumar Singh, Utkarsh Ojha, and Yong Jae Lee. Finegan: Unsupervised hierar-chical disentanglement for fine-grained object generation and discovery. In Proceedingsof the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages6490–6499, 2019.[266] Kihyuk Sohn, Nataniel Ruiz, Kimin Lee, Daniel Castro Chin, Irina Blok, HuiwenChang, Jarred Barber, Lu Jiang, Glenn Entis, Yuanzhen Li, et al. Styledrop: Text-to-image generation in any style. arXiv preprint arXiv:2306.00983, 2023.[267] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models.arXiv preprint arXiv:2010.02502, 2020.[268] Yuhang Song, Chao Yang, Yeji Shen, Peng Wang, Qin Huang, and C-C Jay Kuo.Spg-net: Segmentation prediction and guidance network for image inpainting. arXivpreprint arXiv:1805.03356, 2018.[269] Shih-Yang Su, Timur Bagautdinov, and Helge Rhodin. Danbo: Disentangled articu-lated neural body representations via graph neural networks. In European Conferenceon Computer Vision, 2022.[270] Shih-Yang Su, Timur Bagautdinov, and Helge Rhodin. Npc: Neural point charactersfrom video. In ICCV, 2023.[271] Shih-Yang Su, Frank Yu, Michael Zollho¨fer, and Helge Rhodin. A-nerf: Articulatedneural radiance fields for learning human shape, appearance, and pose. In Advancesin Neural Information Processing Systems, 2021.[272] Jennifer J Sun, Serim Ryou, Roni H Goldshmid, Brandon Weissbourd, John O Dabiri,David J Anderson, Ann Kennedy, Yisong Yue, and Pietro Perona. Self-supervised136Bibliographykeypoint discovery in behavioral videos. In Proceedings of the IEEE/CVF Conferenceon Computer Vision and Pattern Recognition, pages 2171–2180, 2022.[273] Keqiang Sun, Wayne Wu, Tinghao Liu, Shuo Yang, Quan Wang, Qiang Zhou,Zuochang Ye, and Chen Qian. FAB: A robust facial landmark detection frame-work for motion-blurred videos. In ICCV, pages 5461–5470. IEEE, 2019.[274] Vadim Sushko, Dan Zhang, Juergen Gall, and Anna Khoreva. One-shot synthesis ofimages and segmentation masks. arXiv preprint arXiv:2209.07547, 2022.[275] Supasorn Suwajanakorn, Noah Snavely, Jonathan J Tompson, and MohammadNorouzi. Discovery of latent 3d keypoints via end-to-end geometric reasoning. InAdvances in neural information processing systems, pages 2059–2070, 2018.[276] Zhentao Tan, Menglei Chai, Dongdong Chen, Jing Liao, Qi Chu, Bin Liu, GangHua, and Nenghai Yu. Diverse semantic image synthesis via probability distributionmodeling. In Proceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition, pages 7962–7971, 2021.[277] Zhentao Tan, Dongdong Chen, Qi Chu, Menglei Chai, Jing Liao, Mingming He,Lu Yuan, Gang Hua, and Nenghai Yu. Efficient semantic image synthesis viaclass-adaptive normalization. IEEE Transactions on Pattern Analysis and MachineIntelligence, 2021.[278] Matthew Tancik, Pratul P Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, NithinRaghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng.Fourier features let networks learn high frequency functions in low dimensionaldomains. arXiv preprint arXiv:2006.10739, 2020.[279] Hao Tang, Dan Xu, Gaowen Liu, Wei Wang, Nicu Sebe, and Yan Yan. Cycle in cyclegenerative adversarial networks for keypoint-guided image generation. In Proceedingsof the 27th ACM International Conference on Multimedia, MM ’19, page 2052–2060,New York, NY, USA, 2019. Association for Computing Machinery.[280] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaus-sian: Generative gaussian splatting for efficient 3d content creation. arXiv preprintarXiv:2309.16653, 2023.[281] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang, Ran Yi, Lizhuang Ma, and DongChen. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior.arXiv preprint arXiv:2303.14184, 2023.[282] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariha-ran. Emergent correspondence from image diffusion. arXiv preprint arXiv:2306.03881,2023.137Bibliography[283] Datagen Research Team. Facial landmark detection using synthetic data. https://datagen.tech/ai/facial-landmark-detection-using-synthetic-data/, 2022.[Online; accessed 28-Oct-2022].[284] Ayush Tewari, Mohamed Elgharib, Florian Bernard, Hans-Peter Seidel, Patrick Pe´rez,Michael Zollho¨fer, Christian Theobalt, et al. Pie: Portrait image embedding forsemantic control. arXiv preprint arXiv:2009.09485, 2020.[285] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-PeterSeidel, Patrick Pe´rez, Michael Zollho¨fer, and Christian Theobalt. Stylerig: Riggingstylegan for 3d control over portrait images. In CVPR, pages 6141–6150. ComputerVision Foundation / IEEE, 2020.[286] Ayush Tewari, Mohamed Elgharib, Gaurav Bharaj, Florian Bernard, Hans-PeterSeidel, Patrick Pe´rez, Michael Zo¨llhofer, and Christian Theobalt. Stylerig: Riggingstylegan for 3d control over portrait images, cvpr 2020. In IEEE Conference onComputer Vision and Pattern Recognition (CVPR). IEEE, june 2020.[287] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editingfor text-to-image personalization. In ACM SIGGRAPH 2023 Conference Proceedings,pages 1–11, 2023.[288] James Thewlis, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of objectlandmarks by factorized spatial embeddings. In Proceedings of the IEEE internationalconference on computer vision, pages 5916–5925, 2017.[289] James Thewlis, Hakan Bilen, and Andrea Vedaldi. Unsupervised learning of objectlandmarks by factorized spatial embeddings. In ICCV, pages 3229–3238. IEEEComputer Society, 2017.[290] Ruben Tolosana, Ruben Vera-Rodriguez, Julian Fierrez, Aythami Morales, and JavierOrtega-Garcia. Deepfakes and beyond: A survey of face manipulation and fakedetection. Information Fusion, 64:131–148, 2020.[291] Carlo Tomasi. Compsci 527 — computer vision, 2022.[292] Nontawat Tritrong, Pitchaporn Rewatbowornwong, and Supasorn Suwajanakorn.Repurposing gans for one-shot semantic part segmentation. In Proceedings of theIEEE/CVF conference on computer vision and pattern recognition, pages 4475–4485,2021.[293] Yi-Hsuan Tsai, Xiaohui Shen, Zhe Lin, Kalyan Sunkavalli, Xin Lu, and Ming-HsuanYang. Deep image harmonization. In Proceedings of the IEEE Conference onComputer Vision and Pattern Recognition, pages 3789–3797, 2017.138Bibliography[294] Narek Tumanyan, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Splicing vit featuresfor semantic appearance transfer. In Proceedings of the IEEE/CVF Conference onComputer Vision and Pattern Recognition, pages 10748–10757, 2022.[295] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusionfeatures for text-driven image-to-image translation. arXiv preprint arXiv:2211.12572,2022.[296] Norimichi Ukita and Yusuke Uematsu. Semi- and weakly-supervised human poseestimation. Comput. Vis. Image Underst., 170:67–78, 2018.[297] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization:The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.[298] Shinji Umeyama. Least-squares estimation of transformation parameters between twopoint patterns. IEEE TPAMI, 13(4):376–380, 1991.[299] Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman. p+: Extendedtextual conditioning in text-to-image generation. arXiv preprint arXiv:2303.09522,2023.[300] Andrey Voynov, Stanislav Morozov, and Artem Babenko. Object segmentationwithout labels with large-scale generative models. In International Conference onMachine Learning, pages 10596–10606. PMLR, 2021.[301] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-ucsd birds 200.Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.[302] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSDBirds-200-2011 Dataset. Technical Report CNS-TR-2011-001, California Institute ofTechnology, 2011.[303] Bastian Wandt, Marco Rudolph, Petrissa Zell, Helge Rhodin, and Bodo Rosenhahn.Canonpose: Self-supervised monocular 3d human pose estimation in the wild. InCVPR, pages 13294–13304. Computer Vision Foundation / IEEE, 2021.[304] Can Wang, Sheng Jin, Yingda Guan, Wentao Liu, Chen Qian, Ping Luo, andWanli Ouyang. Pseudo-labeled auto-curriculum learning for semi-supervised keypointlocalization. In ICLR. OpenReview.net, 2022.[305] Ching-Wei Wang, Cheng-Ta Huang, Jia-Hong Lee, Chung-Hsing Li, Sheng-Wei Chang,Ming-Jhih Siao, Tat-Ming Lai, Bulat Ibragimov, Tomaz Vrtovec, Olaf Ronneberger,Philipp Fischer, Timothy F. Cootes, and Claudia Lindner. A benchmark for com-parison of dental radiography analysis algorithms. Medical Image Anal., 31:63–76,2016.139Bibliography[306] Fang Wang, Le Kang, and Yi Li. Sketch-based 3d shape retrieval using convolutionalneural networks. In Proceedings of the IEEE conference on computer vision andpattern recognition, pages 1875–1883, 2015.[307] Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robustglobal representations by penalizing local predictive power. In Advances in NeuralInformation Processing Systems, pages 10506–10518, 2019.[308] Jin Wang, Chen Wang, Qingming Huang, Yunhui Shi, Jian-Feng Cai, Qing Zhu,and Baocai Yin. Image Inpainting Based on Multi-Frequency Probabilistic InferenceModel, page 1–9. Association for Computing Machinery, New York, NY, USA, 2020.[309] Kaisiyuan Wang, Qianyi Wu, Linsen Song, Zhuoqian Yang, Wayne Wu, Chen Qian,Ran He, Yu Qiao, and Chen Change Loy. MEAD: A large-scale audio-visual datasetfor emotional talking-face generation. In ECCV, volume 12366 of Lecture Notes inComputer Science, pages 700–717. Springer, 2020.[310] Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and BryanCatanzaro. High-resolution image synthesis and semantic manipulation with condi-tional gans. In Proceedings of the IEEE conference on computer vision and patternrecognition, pages 8798–8807, 2018.[311] Xiaolong Wang and Abhinav Gupta. Generative image modeling using style andstructure adversarial networks. In European conference on computer vision, pages318–335. Springer, 2016.[312] Yirui Wang, Le Lu, Chi-Tung Cheng, Dakai Jin, Adam P. Harrison, Jing Xiao,Chien-Hung Liao, and Shun Miao. Weakly supervised universal fracture detectionin pelvic x-rays. In MICCAI, volume 11769 of Lecture Notes in Computer Science,pages 459–467. Springer, 2019.[313] Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and JunZhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variationalscore distillation. arXiv preprint arXiv:2305.16213, 2023.[314] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutionalpose machines. In Proceedings of the IEEE conference on Computer Vision andPattern Recognition, pages 4724–4732, 2016.[315] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh. Convolutionalpose machines. In CVPR, pages 4724–4732. IEEE Computer Society, 2016.[316] Zhen Wei, Bingkun Liu, Weinong Wang, and Yu-Wing Tai. Few-shot model adaptationfor customized facial landmark detection, segmentation, stylization and shadowremoval. CoRR, abs/2104.09457, 2021.140Bibliography[317] Erroll Wood, Tadas Baltrusaitis, Charlie Hewitt, Sebastian Dziadzio, Thomas J.Cashman, and Jamie Shotton. Fake it till you make it: face analysis in the wild usingsynthetic data alone. In ICCV, pages 3661–3671. IEEE, 2021.[318] Erroll Wood, Tadas Baltrusˇaitis, Charlie Hewitt, Matthew Johnson, Jingjing Shen,Nikola Milosavljevic´, Daniel Wilde, Stephan Garbin, Toby Sharp, Ivan Stojiljkovic´,et al. 3d face reconstruction with dense landmarks. In European Conference onComputer Vision, pages 160–177. Springer, 2022.[319] Erroll Wood, Tadas Baltrusaitis, Charlie Hewitt, Matthew Johnson, Jingjing Shen,Nikola Milosavljevic, Daniel Wilde, Stephan J. Garbin, Toby Sharp, Ivan Stojiljkovic,Tom Cashman, and Julien Valentin. 3d face reconstruction with dense landmarks.CoRR, abs/2204.02776, 2022.[320] Erroll Wood, Tadas Baltrusaitis, Xucong Zhang, Yusuke Sugano, Peter Robinson, andAndreas Bulling. Rendering of eyes for eye-shape registration and gaze estimation. InICCV, pages 3756–3764. IEEE Computer Society, 2015.[321] Huikai Wu, Shuai Zheng, Junge Zhang, and Kaiqi Huang. Gp-gan: Towards real-istic high-resolution image blending. In Proceedings of the 27th ACM internationalconference on multimedia, pages 2487–2495, 2019.[322] Shangzhe Wu, Ruining Li, Tomas Jakab, Christian Rupprecht, and Andrea Vedaldi.Magicpony: Learning articulated 3d animals in the wild. In Proceedings of theIEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8792–8802, 2023.[323] Shangzhe Wu, Christian Rupprecht, and Andrea Vedaldi. Unsupervised learning ofprobably symmetric deformable 3d objects from images in the wild. In Proceedingsof the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages1–10, 2020.[324] Wayne Wu, Kaidi Cao, Cheng Li, Chen Qian, and Chen Change Loy. Transgaga:Geometry-aware unsupervised image-to-image translation. In Proceedings of theIEEE Conference on Computer Vision and Pattern Recognition, pages 8012–8021,2019.[325] Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou. Look atboundary: A boundary-aware face alignment algorithm. In CVPR, pages 2129–2138.Computer Vision Foundation / IEEE, 2018.[326] Zhenyu Wu, Lin Wang, Wei Wang, Tengfei Shi, Chenglizhao Chen, Aimin Hao, andShuo Li. Synthetic data supervised salient object detection. In Proceedings of the30th ACM International Conference on Multimedia, pages 5557–5565, 2022.141Bibliography[327] Jonas Wulff and Antonio Torralba. Improving inversion and generation diversity instylegan using a gaussianized latent space. arXiv preprint arXiv:2009.06529, 2020.[328] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond PASCAL: A benchmarkfor 3d object detection in the wild. In WACV, pages 75–82. IEEE Computer Society,2014.[329] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimationand tracking. In European Conference on Computer Vision (ECCV), 2018.[330] Bin Xiao, Haiping Wu, and Yichen Wei. Simple baselines for human pose estimationand tracking. In ECCV, volume 11210 of Lecture Notes in Computer Science, pages472–487. Springer, 2018.[331] Zhenda Xie, Yutong Lin, Zheng Zhang, Yue Cao, Stephen Lin, and Han Hu. Propagateyourself: Exploring pixel-level consistency for unsupervised visual representationlearning. In Proceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition, pages 16684–16693, 2021.[332] Shiying Xiong and Yue Yang. The boundary-constraint method for constructingvortex-surface fields. Journal of Computational Physics, 339:31–45, 2017.[333] Shiying Xiong and Yue Yang. Identifying the tangle of vortex tubes in homogeneousisotropic turbulence. Journal of Fluid Mechanics, 874:952–978, 2019.[334] Wei Xiong, Jiahui Yu, Zhe Lin, Jimei Yang, Xin Lu, Connelly Barnes, and Jiebo Luo.Foreground-aware image inpainting. In Proceedings of the IEEE/CVF Conferenceon Computer Vision and Pattern Recognition, pages 5840–5848, 2019.[335] H. Xu, X. Su, M. Wang, X. Hao, and G. Gao. An edge information and maskshrinking based image inpainting approach. In 2020 IEEE International Conferenceon Multimedia and Expo (ICME), pages 1–6, Los Alamitos, CA, USA, jul 2020. IEEEComputer Society.[336] Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, Bill Freeman, Rahul Sukthankar,and Cristian Sminchisescu. Ghum & ghuml: Generative 3d human shape andarticulated pose models. In IEEE/CVF Conference on Computer Vision and PatternRecognition (Oral), pages 6184–6193, 2020.[337] Peng Xu. Deep learning for free-hand sketch: A survey. IEEE transactions on patternanalysis and machine intelligence, PP, 2022.[338] Peng Xu, Yongye Huang, Tongtong Yuan, Kaiyue Pang, Yi-Zhe Song, Tao Xiang,Timothy M Hospedales, Zhanyu Ma, and Jun Guo. Sketchmate: Deep hashingfor million-scale human sketch retrieval. In Proceedings of the IEEE conference oncomputer vision and pattern recognition, pages 8090–8098, 2018.142Bibliography[339] Peng Xu, Chaitanya K Joshi, and Xavier Bresson. Multigraph transformer forfree-hand sketch recognition. IEEE Transactions on Neural Networks and LearningSystems, 2021.[340] Rui Xu, Xintao Wang, Kai Chen, Bolei Zhou, and Chen Change Loy. Positionalencoding as spatial inductive bias in gans. arXiv preprint arXiv:2012.05217, 2020.[341] Yinghao Xu, Ceyuan Yang, Ziwei Liu, Bo Dai, and Bolei Zhou. Unsupervised landmarklearning from unpaired data. arXiv preprint arXiv:2007.01053, 2020.[342] Binxin Yang, Shuyang Gu, Bo Zhang, Ting Zhang, Xuejin Chen, Xiaoyan Sun,Dong Chen, and Fang Wen. Paint by example: Exemplar-based image editing withdiffusion models. In Proceedings of the IEEE/CVF Conference on Computer Visionand Pattern Recognition, pages 18381–18391, 2023.[343] Yanchao Yang, Brian Lai, and Stefano Soatto. Dystab: Unsupervised object segmenta-tion via dynamic-static bootstrapping. In Proceedings of the IEEE/CVF Conferenceon Computer Vision and Pattern Recognition, pages 2826–2836, 2021.[344] Yanchao Yang, Antonio Loquercio, Davide Scaramuzza, and Stefano Soatto. Unsuper-vised moving object detection via contextual information separation. In Proceedingsof the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages879–888, 2019.[345] Yang Yang and Xiaojie Guo. Generative landmark guided face inpainting. InChinese Conference on Pattern Recognition and Computer Vision (PRCV), pages14–26. Springer, 2020.[346] Yu Yang, Hakan Bilen, Qiran Zou, Wing Yin Cheung, and Xiangyang Ji. Unsupervisedforeground-background segmentation with equivariant layered gans. arXiv preprintarXiv:2104.00483, 2021.[347] Yu Yang, Xiaotian Cheng, Hakan Bilen, and Xiangyang Ji. Learning to annotatepart segmentation with gradient matching. In International Conference on LearningRepresentations, 2022.[348] Chun-Han Yao, Wei-Chih Hung, Yuanzhen Li, Michael Rubinstein, Ming-HsuanYang, and Varun Jampani. Lassie: Learning articulated shapes from sparse imageensemble via 3d part discovery. Advances in Neural Information Processing Systems,35:15296–15308, 2022.[349] Chun-Han Yao, Wei-Chih Hung, Yuanzhen Li, Michael Rubinstein, Ming-HsuanYang, and Varun Jampani. Hi-lassie: High-fidelity articulated shape and skeletondiscovery from sparse image ensemble. In Proceedings of the IEEE/CVF Conferenceon Computer Vision and Pattern Recognition, pages 4853–4862, 2023.143Bibliography[350] Chun-Han Yao, Amit Raj, Wei-Chih Hung, Yuanzhen Li, Michael Rubinstein, Ming-Hsuan Yang, and Varun Jampani. Artic3d: Learning robust articulated 3d shapesfrom noisy web image collections. arXiv preprint arXiv:2306.04619, 2023.[351] Qingsong Yao, Quan Quan, Li Xiao, and S Kevin Zhou. One-shot medical land-mark detection. In International Conference on Medical Image Computing andComputer-Assisted Intervention, pages 177–188. Springer, 2021.[352] Yuan Yao, Nico Schertler, Enrique Rosales, Helge Rhodin, Leonid Sigal, and AllaSheffer. Front2back: Single view 3d shape reconstruction via front to back predic-tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition, pages 531–540, 2020.[353] Ran Yi, Yong-Jin Liu, Yu-Kun Lai, and Paul L Rosin. APDrawingGAN: Generatingartistic portrait drawings from face photos with hierarchical gans. In IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR ’19), pages 10743–10752, 2019.[354] Zihao Yin, Ping Gong, Chunyu Wang, Yizhou Yu, and Yizhou Wang. One-shot medicallandmark localization by edge-guided transform and noisy landmark refinement. arXivpreprint arXiv:2208.00453, 2022.[355] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and JianxiongXiao. Lsun: Construction of a large-scale image dataset using deep learning withhumans in the loop. arXiv preprint arXiv:1506.03365, 2015.[356] Qian Yu, Feng Liu, Yi-Zhe Song, Tao Xiang, Timothy M. Hospedales, andChen Change Loy. Sketch me that shoe. In IEEE Conference on Computer Visionand Pattern Recognition (CVPR), 2016.[357] Qian Yu, Yongxin Yang, Yi-Zhe Song, Tao Xiang, and Timothy M. Hospedales.Sketch-a-net that beats humans. In BMVC, 2015.[358] Tao Yu, Kaiwen Guo, Feng Xu, Yuan Dong, Zhaoqi Su, Jianhui Zhao, Jianguo Li,Qionghai Dai, and Yebin Liu. Bodyfusion: Real-time capture of human motion andsurface geometry using a single depth camera. In 2017 IEEE International Conferenceon Computer Vision (ICCV), pages 910–919, 2017.[359] Yingchen Yu, Fangneng Zhan, Rongliang WU, Jianxiong Pan, Kaiwen Cui, Shi-jian Lu, Feiying Ma, Xuansong Xie, and Chunyan Miao. Diverse Image Inpaintingwith Bidirectional and Autoregressive Transformers, page 69–78. Association forComputing Machinery, New York, NY, USA, 2021.[360] Fangneng Zhan, Hongyuan Zhu, and Shijian Lu. Spatial fusion gan for imagesynthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision andPattern Recognition, pages 3653–3662, 2019.144Bibliography[361] Chao Zhang, Xuequan Lu, and Takuya Akashi. Blur-countering keypoint detectionvia eigenvalue asymmetry. IEEE Access, 8:159077–159088, 2020.[362] Hongwen Zhang, Qi Li, and Zhenan Sun. Joint voxel and coordinate regression foraccurate 3d facial landmark localization. In ICPR, pages 2202–2208. IEEE ComputerSociety, 2018.[363] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-imagediffusion models. arXiv preprint arXiv:2302.05543, 2023.[364] Xian Zhang, Canghong Shi, Xin Wang, Xi Wu, Xiaojie Li, Jiancheng Lv, and ImranMumtaz. Face inpainting based on gan by facial prediction and fusion as guidanceinformation. Applied Soft Computing, 111:107626, 2021.[365] Yanping Zhang, Qiaokang Liang, Kunlin Zou, Zhengwei Li, Wei Sun, and YaonanWang. Self-supervised part segmentation via motion imitation. Image and VisionComputing, 120:104393, 2022.[366] Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, and Honglak Lee. Unsu-pervised discovery of object landmarks as structural representations. In Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition, pages 2694–2703,2018.[367] Yuting Zhang, Yijie Guo, Yixin Jin, Yijun Luo, Zhiyuan He, and Honglak Lee. Unsu-pervised discovery of object landmarks as structural representations. In Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition, pages 2694–2703,2018.[368] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, AdelaBarriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled datafactory with minimal human effort. In CVPR, pages 10145–10155. Computer VisionFoundation / IEEE, 2021.[369] Yuxuan Zhang, Huan Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, AdelaBarriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled datafactory with minimal human effort. In Conference on Computer Vision and PatternRecognition (CVPR), 2021.[370] Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmarkdetection by deep multi-task learning. In European Conference on Computer Vision2014, 09 2014.[371] Zibo Zhao, Wen Liu, Yanyu Xu, Xianing Chen, Weixin Luo, Lei Jin, Bohui Zhu,Tong Liu, Binqiang Zhao, and Shenghua Gao. Prior based human completion. In145BibliographyProceedings of the IEEE/CVF International Conference on Computer Vision, pages7947–7957, 2021.[372] Chuanxia Zheng, Tat-Jen Cham, and Jianfei Cai. Pluralistic image completion.In Proceedings of the IEEE/CVF Conference on Computer Vision and PatternRecognition, pages 1438–1447, 2019.[373] Xiao-Yun Zhou, Bolin Lai, Weijian Li, Yirui Wang, Kang Zheng, Fakai Wang, ChihungLin, Le Lu, Lingyun Huang, Mei Han, Guotong Xie, Jing Xiao, Chang-Fu Kuo,Adam P. Harrison, and Shun Miao. Scalable semi-supervised landmark localizationfor x-ray images using few-shot deep adaptive graph. In DGM4MICCAI, volume13003 of Lecture Notes in Computer Science, pages 145–153. Springer, 2021.[374] Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion forreal image editing. arXiv preprint arXiv:2004.00049, 2020.[375] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of theIEEE international conference on computer vision, pages 2223–2232, 2017.[376] Peihao Zhu, Rameen Abdal, Yipeng Qin, and Peter Wonka. Sean: Image synthesiswith semantic region-adaptive normalization. In Proceedings of the IEEE/CVFConference on Computer Vision and Pattern Recognition, pages 5104–5113, 2020.[377] Silvia Zuffi and Michael J. Black. The stitched puppet: A graphical model of 3dhuman shape and pose. In 2015 IEEE Conference on Computer Vision and PatternRecognition (CVPR), pages 3537–3546, 2015.146
Thesis/Dissertation
eng
Computer Science
Vancouver : University of British Columbia Library
Attribution-NonCommercial-NoDerivatives 4.0 International
Graduate
Rhodin, Helge
Structured representation learning by controlling generative models
Text