Open Collections

Robustness Against Heterogeneity in Big Data

Banff International Research Station for Mathematical Innovation and Discovery

A major challenge in the world of Big Data is heterogeneity. This often results from the aggregation of smaller data sets into larger ones. Such aggregation creates heterogeneity because different experimenters typically make different design choices. Even when attempts are made at common designs, environmental or operator effects still often create heterogeneity. Thus motivates moving away from the classical conceptual model of Gaussian distributed data, in the direction of Gaussian mixtures. But classical mixture estimation methods are usually useless in Big Data contexts, because there are far too many parameters to efficiently estimate. Thus there is a strong need for statistical procedures which are robust against mixture distributions without the need for explicit estimation. Some early ideas in this important new direction are discussed.

Mathematics; Statistics

video/mp4

Author affiliation: University of North Carolina

2016-05-17

Attribution-NonCommercial-NoDerivatives 4.0 International

http://hdl.handle.net/2429/58151

Unreviewed

http://creativecommons.org/licenses/by-nc-nd/4.0/