- Library Home /
- Search Collections /
- Open Collections /
- Browse Collections /
- UBC Theses and Dissertations /
- Person in context synthesis with compositional structural...
Open Collections
UBC Theses and Dissertations
UBC Theses and Dissertations
Person in context synthesis with compositional structural space Yin, Weidong
Abstract
Despite significant progress, controlled generation of complex images with interacting people remains difficult. Existing layout to image generation methods fall short of synthesizing realistic person instances, while pose-guided generation approaches focus on a single person and assume simple or known backgrounds. To tackle these limitations, we propose a new problem, Person in Context Synthesis, which aims to synthesize diverse person instance(s) in consistent contexts, with user control over both. The context is specified by the bounding box object layout which lacks shape information, while pose of the person(s) by keypoints which are sparse annotations. To handle the stark difference in input structures, we proposed two separate neural branches to attentively composite the respective (context/person) inputs into shared compositional structural space, which encodes shape, location and appearance information for both context and person structures in a disentangled manner. This structural space is then decoded to the image space using a multi-level feature modulation strategy, and learned in a self-supervised manner from image collections and their corresponding inputs. Extensive experiments on two large-scale datasets (COCO-Stuff and Visual Genome) demonstrate that our framework outperforms state-of-the-art methods with respect to synthesis quality.
Item Metadata
Title |
Person in context synthesis with compositional structural space
|
Creator | |
Publisher |
University of British Columbia
|
Date Issued |
2021
|
Description |
Despite significant progress, controlled generation of complex images with interacting people remains difficult. Existing layout to image generation methods fall short of synthesizing realistic person instances, while pose-guided generation approaches focus on a single person and assume simple or known backgrounds. To tackle these limitations, we propose a new problem, Person in Context Synthesis, which aims to synthesize diverse person instance(s) in consistent contexts, with user control over both. The context is specified by the bounding box object layout which lacks shape information, while pose of the person(s) by keypoints which are sparse annotations. To handle the stark difference in input structures, we proposed two separate neural branches to attentively composite the respective (context/person) inputs into shared compositional structural space, which encodes shape, location and appearance information for both context and person structures in a disentangled manner. This structural space is then decoded to the image space using a multi-level feature modulation strategy, and learned in a self-supervised manner from image collections and their corresponding inputs. Extensive experiments on two large-scale datasets (COCO-Stuff and Visual Genome) demonstrate that our framework outperforms state-of-the-art methods with respect to synthesis quality.
|
Genre | |
Type | |
Language |
eng
|
Date Available |
2021-01-27
|
Provider |
Vancouver : University of British Columbia Library
|
Rights |
Attribution-NonCommercial-NoDerivatives 4.0 International
|
DOI |
10.14288/1.0395749
|
URI | |
Degree | |
Program | |
Affiliation | |
Degree Grantor |
University of British Columbia
|
Graduation Date |
2021-05
|
Campus | |
Scholarly Level |
Graduate
|
Rights URI | |
Aggregated Source Repository |
DSpace
|
Item Media
Item Citations and Data
Rights
Attribution-NonCommercial-NoDerivatives 4.0 International