In this paper, we consider the statistical foundations for empirical tests of disparate impact. We begin by considering a recent, popular proposal in the economics literature that seeks to assess disparate impact via a comparison of error rates for the majority and the minority group. We show that this approach suffers from what is colloquially known as “the problem of inframarginality”, in turn putting it in direct conflict with legal understandings of discrimination. We then proceed to analyze two alternative proposals that quantify disparate impact either in terms of risk-adjusted disparities or by comparing existing disparities to those under a statistically optimized decision policy. Both approaches have differing, context-specific strengths and weaknesses, and we discuss how they relate to the individual elements in the legal test for disparate impact. To demonstrate feasibility, we assess disparate impact in a large dataset of 2.2 million pedestrian stop-and-frisk decisions recorded by the New York City Police Department between 2008 and 2011. We find strong evidence for disparate impact and propose both complex and simple policy alternatives that are as efficient while exerting fewer disparities.
Grossman, Joshua, Julian Nyarko, and Sharad Goel. "Reconciling Legal and Empirical Conceptions of Disparate Impact." .