Open Collections

Distributional Robustness and Regularization in Statistical Learning

Banff International Research Station for Mathematical Innovation and Discovery

A central problem in statistical learning is to design prediction algorithms that not only perform well on training data, but also perform well on new and unseen, but similar, data. We approach this problem by formulating a distributionally robust stochastic optimization (DRSO) problem, which seeks a solution that minimizes the worst-case expected loss over a family of distributions that are close to the empirical distribution as measured by Wasserstein distance. We establish a connection between such Wasserstein DRSO and regularization. Specifically, we identify a broad class of loss functions, for which the Wasserstein DRSO is asymptotically equivalent to a regularization problem with a gradient-norm penalty. Such relation provides a new interpretation for approaches that use regularization, including a variety of statistical learning problems and discrete choice models. The connection also suggests a principled way to regularize high-dimensional, non-convex problems, which is demonstrated with the training of Wasserstein generative adversarial networks (WGANs) in deep learning. This is joint work with Rui Gao and Xi Chen.

Mathematics; Operations research, mathematical programming; Probability theory and stochastic processes; Operation research

video/mp4

Author affiliation: Georgia Institute of Technology

2018-09-03

Attribution-NonCommercial-NoDerivatives 4.0 International

http://hdl.handle.net/2429/67071

Unreviewed

http://creativecommons.org/licenses/by-nc-nd/4.0/