Open Collections

Liu, Michael F.

University of British Columbia

The task of generative modeling – building models which directly capture and represent distributions, has seen a surge of popularity in recent years. The massive-scale foundation models which underpin the advances in the field have incredible performance, but come at the cost of large computational resource requirements and low privacy. The attention mechanism used in large language models requires quadratic time with respect to the length of the input sequence, in contrast with the remainder of the network requiring linear time. We propose a Simhash-based attention scheme which approximates full attention while providing tuneable parameters which trades off runtime for accuracy. Preliminary results suggest that O(n log n) time is sufficient for a good approximation of full attention. Furthermore, our method naturally admits an implementation using O(n) memory, matching the memory cost of other space-efficient attention mechanisms. It has been shown that large-scale diffusion models can be better at memorizing its training data than GANs, the previous state-of-the-art in image generative modeling. This poses significant privacy risks. We combine previous advances in differentially private machine learning and apply it to latent diffusion models (LDMs). Specifically, we use DP-SGD to train a subset of parameters in the attention modules of the model. We make comparisons with existing works using Fréchet inception distance (FID), a perceptual distance metric, and validation-set classification accuracies of downstream classifier models trained on synthetic data.

Text

2024-09-03

Attribution 4.0 International

http://hdl.handle.net/2429/89134

Computer Science

University of British Columbia

UBCV

http://creativecommons.org/licenses/by/4.0/