HKS Authors

See citation below for complete author information.

Daniel Paul Professor of the Practice of Government and Technology, HKS and FAS

Abstract

The vast majority of social science research uses small (megabyte- or gigabyte-scale) datasets. These fixed- scale datasets are commonly downloaded to the researcher's computer where the analysis is performed. The data can be shared, archived, and cited with well-established technologies, such as the Dataverse Project, to support the published results. The trend toward big data-including large-scale streaming data-is starting to transform research and has the potential to impact policymaking as well as our understanding of the social, economic, and political problems that affect human societies. However, big data research poses new challenges to the execution of the analysis, archiving and reuse of the data, and reproduction of the results. Downloading these datasets to a researcher's computer is impractical, leading to analyses taking place in the cloud, and requiring unusual expertise, collaboration, and tool development. The increased amount of information in these large datasets is an advantage, but at the same time it poses an increased risk of revealing personally identifiable sensitive information. In this article, we discuss solutions to these new challenges so that the social sciences can realize the potential of big data.

Citation

Crosas, Mercè, Gary King, James Honaker, and Latanya Sweeney. "Automating Open Science for Big Data." The ANNALS of the American Academy of Political and Social Science 659 (May 2015).