Resolving the Disparate Impact of Uncertainty: Affirmative Action vs. Affirmative Information
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Resolving the Disparate Impact of Uncertainty:
Affirmative Action vs. Affirmative Information
Claire Lazar Reich 1
MIT Economics & Statistics
clazar@mit.edu
arXiv:2102.10019v1 [stat.ML] 19 Feb 2021
Abstract
Algorithmic risk assessments hold the promise of greatly advancing accurate
decision-making, but in practice, multiple real-world examples have been shown to
distribute errors disproportionately across demographic groups. In this paper, we
characterize why error disparities arise in the first place. We show that predictive
uncertainty often leads classifiers to systematically disadvantage groups with lower-
mean outcomes, assigning them smaller true and false positive rates than their
higher-mean counterparts. This can occur even when prediction is group-blind.
We prove that to avoid these error imbalances, individuals in lower-mean groups
must either be over-represented among positive classifications or be assigned more
accurate predictions than those in higher-mean groups. We focus on the latter
condition as a solution to bridge error rate divides and show that data acquisition
for low-mean groups can increase access to opportunity. We call the strategy
“affirmative information” and compare it to traditional affirmative action in the
classification task of identifying creditworthy borrowers.
1 Introduction
Algorithmic risk scores are increasingly helping lenders, physicians, and judges make high-impact
decisions in the presence of uncertainty. Yet even the most advanced algorithms cannot eliminate the
uncertainty surrounding these decisions [9], and in recent years there has been growing awareness of
real-world cases where the burden of prediction error falls more heavily on some populations than
others [2, 8, 14]. In light of these issues, a growing branch of research in fair prediction has produced
methods to modify algorithms so that they yield equal error rates across subgroups [10, 16, 13].
In this paper, we show why error rate disparities across groups arise in the first place and characterize
necessary conditions to correct them. We consider screening decisions in which applicants are either
granted or denied an opportunity, such as a loan, based on predictions of their eventual outcomes,
such as loan repayment. We provide empirical and theoretical evidence showing that applicants from
lower-mean groups are often more likely than those from higher-mean groups to be misclassified as
negative and mistakenly denied a loan. Conversely, those from higher-mean groups are relatively
more likely to be misclassified as positive. Based on these error patterns, individuals from lower-mean
groups have lower probability of accessing loans conditional on their true repayment ability. The
imbalance can persist even when the predictions are group-blind, and it tends to be exacerbated when
they are imprecise. We therefore call this phenomenon “the disparate impact of uncertainty.”
We prove that to eliminate group error disparities in the presence of uncertainty, at least one of the
following must be true: (1) members of the lower-mean groups must be over-represented among
1
I am grateful to my advisors Anna Mikusheva, Iván Werning, and David Autor for sharing invaluable
insights at each stage of the project. I also thank Tom Brennan, Suhas Vijaykumar, Victor Chernozhukov, Ben
Bernanke, Ben Deaner, David Hughes, David Sontag, and Frank Schilbach for helpful and inspiring discussions.
This work was supported by the National Science Foundation Graduate Research Fellowship under Grant No.
1122374.positive classifications, or (2) their predictions must be more accurate than those of higher-mean
groups. We pose the enforcement of the latter condition as a promising alternative to affirmative
action: rather than lowering the admissions criteria for lower-mean groups, it is possible to increase
access to opportunity by acquiring data about lower-mean applicants that aligns predictions closer
to their outcomes. We call this strategy “affirmative information” and compare it to traditional
affirmative action in an empirical credit lending application. Such a strategy agrees with the insight
in [5] that data collection is an integral part of fair prediction that deserves greater attention as an
alternative to imposing model constraints.
In contrast to our proposal of data enrichment, many algorithms in use today seek to achieve fairness
through the omission of data. We show that contrary to intuition, omitting group identifiers and
sensitive features from data generally does not resolve the disparate impact of uncertainty. In fact, by
increasing predictive uncertainty, data omission may exacerbate disparities. We therefore support the
growing evidence in economics and computer science showing that blind algorithms are a misguided
approach to fair prediction. Within this literature, [12, 7, 15] produce theoretical results demonstrating
that fairness considerations are best imposed after estimating the most accurate predictions, rather
than before, and [11, 13] demonstrate this principle empirically in real-world settings.
While we focus on the fairness condition of error rate equality in this paper, it is important to
acknowledge that there are many additional components of algorithmic fairness [3]. Among them
is accuracy on the individual level [6]. To avoid compromising this criterion, we do not propose
intentionally reducing the accuracy of predictions for higher-mean groups in order to eliminate error
disparities. Rather, we raise the option of basing predictions on all immediately available data for all
applicants, as advocated in [11], supplemented with further data acquisition efforts to better identify
the qualified individuals from lower-mean protected groups. We believe approaches like these can
maximize the gains of algorithmic decision-making while ensuring they are broadly shared.
The paper proceeds in four sections. We empirically illustrate the disparate impact of uncertainty in
Section 2. In Section 3 we present general theoretical results that characterize this disparate impact
and identify conditions necessary to overcome it. To provide intuition for these results, we restructure
and analyze a canonical economic model of statistical discrimination [1]. Section 4 empirically
compares two methods that aim to overcome the disparate impact of uncertainty: affirmative action
and affirmative information. We show that in settings with selective admissions standards, affirmative
information can work like affirmative action to raise the probability that creditworthy individuals in
low-mean groups are granted loans. Unlike affirmative action, it can simultaneously yield higher
return to the decision-maker and create natural incentives to admit more applicants.
2 Motivation: a statistical phenomenon about blind prediction
Motivating this study is the observation that predictive algorithms can systematically favor individuals
from high-mean groups even when they do not discriminate by group. We illustrate this fact
empirically, showing that a credit-lending algorithm based only on financial features will lead to
greater true and false positive rates for high educated applicants than for less educated applicants.
Since it is uncontroversial to include financial characteristics in credit lending predictions, this
exercise will suggest that the disparities stem from unobservable variation in outcomes rather than
observable variation. Later sections lay out the theory that explains this finding.
We consider a lender that uses risk scores to classify loan applications into approvals and denials. We
suppose the designer of these risk scores wants to provide high-quality predictions that are fair to
demographic subgroups, one highly educated H (more than a high school degree) and another less
educated L (at most a high school degree). The lender’s notion of fairness is to avoid discrimination
by purging the dataset of sensitive characteristics, including education group. Therefore the dataset is
restricted only to financial features deemed relevant for loan repayment.
We simulate this scenario using the U.S. Census Survey of Income and Program Participation and
predict the outcome of paying rent, mortgage, and utilities in 2016 using survey responses from two
years prior. We construct risk scores with a logistic regression on just three predictors: total assets,
total debt, and monthly income.
The risk scores designed in this experiment can be converted to binary classifications at a number of
cutoffs. The results are illustrated in Figure 1. In panel (a), we see that less-educated individuals are
2(a) (b)
Figure 1: (a) Members of the higher-mean group are more likely to be classified as creditworthy. (b) Members
of the higher-mean group are more likely to be classified as creditworthy, even when we condition on their true
label. While the predictions are group-blind and there was no disparate treatment, there is disparate impact.
less likely to be classified as creditworthy at all cutoffs. This is not necessarily unfair, as perhaps the
scores have picked up on legitimate differences in repayment abilities. However, in panel (b), we
show that less-educated individuals are also less likely to be classified as creditworthy even when
we condition on their eventual repayment outcome. That is, they have higher true and false positive
rates than less educated applicants. In practice, this means that a creditworthy applicant from H has a
greater probability of being granted a loan than one from L; similarly, a non-creditworthy applicant
from H also has a greater probability of being granted a loan than one from L.
In this application we blinded the algorithm to education level and other sensitive characteristics
in an effort to avoid disparate treatment. However, there is disparate impact nonetheless. We next
show why a blind algorithm can systematically favor individuals from high-mean groups and what is
required to avoid it if not data omission.
3 Theory
3.1 Characterizing the disparate impact of uncertainty
In this section we describe conditions that are necessary for classifiers to yield equal true and false
positive rates to groups with different mean outcomes.
We start by briefly introducing our notation and basic definitions. Let Y ∈ {0, 1} be a binary outcome
variable and G ∈ {L, H} be a protected attribute differentiating two groups. Each group has a
unique fixed base rate µG ≡ E[Y |G] , where 0 < µL < µH < 1. We consider binary predictions ŷ of
Y formulated with incomplete information about applicants, so that there is at least some predictive
uncertainty:
Definition 1. Predictive uncertainty is present if each group’s true and false positive rates lie in the
open interval (0, 1), in particular TPRG = P[ŷ = 1|Y = 1, G], FPRG = P [ŷ = 1|Y = 0, G] ∈
(0, 1). At least one positive and one negative individual from each group will be misclassified.
We will formulate conditions under which ŷ assigns positive classifications systematically more often
to members of the high-mean group H, conditional on the true type Y .
First, we observe that in order to achieve balance in both error rates, the TPR and FPR, members
of L must be over-represented among the positive classifications. In particular, L must comprise a
strictly greater portion of the positive classifications ŷ = 1 than it does positive types Y = 1.
Proposition 1. If ŷ yields TPRL ≥ TPRH and FPRL ≥ FPRH then it must be that L members are
over-represented among positive classifications. That is, P[G = L|ŷ = 1] > P[G = L|Y = 1].
Proof in appendix.
3Over-representation is typically achieved in settings where admission standards are low for all groups,
or in settings where those standards are lowered specifically for members of L. We can think of this
as including traditional affirmative action.
Relaxing the constraint of equal FPRs in Proposition 1 enables an alternative solution: improve the
accuracy of L predictions, and through better targeting, increase their group TPR to match that of the
higher-mean group. The accuracy gain may simultaneously lower FPRL further below FPRH , but
in many settings there is no need to equate group FPRs anyway. For example, in credit lending, we
may wish for a creditworthy borrower to have equal opportunity to receive a loan regardless of group
(achieving equal TPRs). Yet equating group probabilities for non-creditworthy borrowers to secure
loans may be unimportant (equal FPRs) [10].
The following theorem establishes that in order to achieve a TPR for the L group at least as large as
for the H group, it is required that ŷ over-represents L members among positive classifications or
assigns more accurate classifications for the L members. By accuracy we refer to the groups’ positive
and negative predictive values, PPVG ≡ P[Y = 1|ŷ = 1, G] and NPVG ≡ P[Y = 0|ŷ = 0, G].
Theorem 1. In the presence of algorithmic uncertainty and base rate differences, one of the following
must be true in order for ŷ to achieve TPRL ≥ TPRH :
(i) Members of the L group are over-represented among positive classifications. That is,
P[G = L|ŷ = 1] > P[G = L|Y = 1]
(ii) The classifications of the L group should be more informative of the true outcomes than
those of the H group. In particular NPVL > NPVH and PPVL ≥ PPVH .
Proof in appendix.
The theorem shows that if individuals in L are not over-represented among positive classifications,
then uncertainty in prediction imparts a disparate impact. Given the same level of accuracy for the
two groups, the L group will be guaranteed to face a lower TPR than the H group.
It may seem that omitting sensitive features from a dataset would be a plausible solution to provide
equal opportunity to members of different groups, and that strategy is indeed the predominant fairness
approach used in today’s screening algorithms. However, omitting data does not resolve the tension
posed by Theorem 1.
In particular, the omission of data does not necessarily lead individuals from low-mean groups to
be over-represented among positive classifications. The reason is that variation in outcomes across
groups may still be at least partially explained by the predictors allowed in the model. In credit
lending, for example, individuals from a lower-mean group may have lower levels of income, higher
existing debt, or fewer assets. A model based on these variables assigns classifications according to the
variation in them, and that may prevent L from being over-represented among positive classifications.
If the L group’s classifications are no more accurate than those of H, then creditworthy individuals
from L will be denied at higher rates than those from H.
We construct a model to illustrate this phenomenon in the next section, where we assume that variation
in group outcomes is explained by the predictors underlying a risk score. This will allow us to isolate
the disparate effects of uncertainty and demonstrate the improvements from raising the accuracy of L
predictions.
3.2 Modeling the mechanism of the disparate impact
In this section, we present a model to explore why the presence of uncertainty imparts a disparate
impact on groups with different mean outcomes. We consider a lender who uses data to predict
applicants’ repayment ability. The signals they derive are constructed without regard to group
membership, but the financial data they are based on does explain the group variation in outcomes.1
As a result, we show that predictive uncertainty contracts each group’s scores toward the group
mean of ability. Scores for lower-mean groups will contract toward a lower mean, and in general the
conditional distribution of scores given true ability will vary by group, causing systematic differences
in true positive rates to emerge.
1
For example, suppose that low income is negatively correlated with repayment ability, and members of a
disadvantaged group tend to have lower incomes.
4Consider a setting where loan applicants each have some repayment ability A, and applicants with
A > 0 are “creditworthy” and would repay a loan if granted one. A lender seeks to estimate A using
financial data X available about the applicants. At no point does the lender observe protected group
membership G. However, the data X does explain the variation in repayment ability across those
groups. In particular we assume A and G are mean independent given X 2 , so E[A|X, G] = E[A|X].
The lender estimates E[A|X] in the model
A = E[A|X] + ε, (1)
where ε is a residual that is uncorrelated with any function of X. For simplicity we assume the
lender successfully constructs the Bayes optimal risk score S = E[A|X], the best prediction given
the available data. We can prove that those scores continue to serve the decision-maker as the best
prediction regardless of whether group membership is observed:
Proposition 2. The lender’s risk scores S = E[A|X] will satisfy E[A|S, G] = S. That is, an
applicants’ expected ability given their score will equal their score, regardless of group membership.
Proof in appendix.
We know from Proposition 2 that the lender’s best available prediction of an applicant’s true ability is
given to her by the score. If she imposes a cutoff rule to classify individuals, then she will choose the
same cutoff for each group regardless of whether she observes group affiliation. The decision-rule is
Accept ≡ 1{S > cutoff}. (2)
The choice of cutoff depends on the lender’s relative valuation of false positive and false negative
classifications. In a setting where false negatives are more costly, the cutoff will be lower, and in
a setting where false positives are more costly, the cutoff will be higher [13]. We call low-cutoff
settings “lemon-dropping markets” and high-cutoff settings “cherry-picking markets,” following the
terminology in [4].
As the scores guide the decision-maker’s classifications, we can abstract away from the data under-
lying them. Our model will be seen to reframe the canonical Aigner and Cain (1977) model for
statistical discrimination [1], treating the risk score as a “signal” that a decision-maker receives about
an applicants’ ability. At the end of the section, we clarify how our model compares to the original in
Aigner and Cain (1977).
We now can express an applicant’s repayment ability as a sum of two parts, the signal S estimated
with available data, and an unobservable error ε. For a member of group G we write
AG = SG + εG . (3)
By Proposition 2, εG is mean-zero. Invoking a similar assumption as Aigner and Cain (1977), we
suppose that SG and εG are bivariate normal. Since SG and εG are uncorrelated, this implies they
are independent. Our model is therefore specified by three random variables: SG ∼ N (µG , σS2 G ),
εG ∼ N (0, σε2G ), and AG ∼ N (µG , σS2 G + σε2G ).
While we know that the score satisfies E[AG |SG ] = SG for both groups, we will now show that
E[SG |AG ] 6= AG . The latter represents the expected score that a decision-maker will face for an
applicant with some fixed true ability, and we will see that it varies by group. Individuals in low-mean
groups will tend to display worse observable scores, even conditioned on their true ability.
To compute the expected score, we use the fact that when variables X1 and X2 are bivariate
normally distributed, then the distribution of X1 given X2 = x2 has conditional expectation
1 ,X2 )
E[X1 ] + cov(X
σ2
(x2 − E[X2 ]). In our model, then, for both G ∈ {L, H},
X2
cov(SG , AG )
E[SG |AG = a] = E[SG ] + 2 (a − E[AG ])
σA G
cov(SG , SG + εG )
= µG + (a − µG )
σS2 G + σε2G
σS2 G
= (1 − γG )µG + γG a, where γG ≡
σS2 G + σε2G
2
This is implied by the stronger condition (A ⊥ G)|X.
5Therefore, the expected signal is a weighted average of the true ability and the group mean. The
weight γG on the true ability reflects the portion of AG ’s total variance that is captured by the
observable score SG , and the weight 1 − γG on the mean reflects the unexplained variance due to the
presence of uncertainty. As γG increases, the score is expected to strongly track true ability. As γG
decreases, however, the scores are expected to bunch more tightly about the group mean.
An immediate consequence is that an individual from L with true ability a can generally be expected
to have a different score than an individual from H with ability a:
E[S|A = a, L] = (1 − γL )µL + γL a, (4)
E[S|A = a, H] = (1 − γH )µH + γH a. (5)
We next consider two cases, one demonstrating the disparate impact of uncertainty and the second
correcting for it. The first case equates the observed variance in ability (γL = γH ), while the second
explores the effect of increasing the observed variance for the L group (γL > γH ). In each case, the
best unbiased estimate of ability for each group is given by the risk score. However, in the latter case,
the estimate is made more precise for L.
3.2.1 Equally informative signal
First consider a setting with mean differences and equal variances: µL < µH and γ L = γ H . We
depict expected scores conditional on true underlying ability in Figure 2a. For each level of ability,
individuals of the H group are expected to have higher observable scores. The gap between their
expected scores and those of the L group is seen to increase in the differences between their means
(µH − µL ) and the extent of the uncertainty, (1 − γ).
Since decision-makers use the same cutoff rule (2) for members of both groups, it seems natural
that the systematic disparities among scores would propagate into systematic disparities among
classifications, that is, the ultimate lending decisions. We formally show this by deriving the
conditional distribution of S given creditworthiness, which we defined as A > 0 early in the section.
3
. The distributions are plotted in Figure 2b and 2c with two examples of decision-maker cutoffs 4 .
Because the true positive rate is the portion of creditworthy individuals ultimately granted a loan, it is
straightforward to visualize it on the plot: each group’s true positive rate is given by the portion under
its curve that is shaded, corresponding to those applicants with a score above the cutoff. We observe
that the true positive rates are higher for the H group in both lemon-dropping and cherry-picking
markets, but that the effect is most stark when the decision-maker cherry-picks applicants at the top
of the distribution (Figure 2c).
3.2.2 More informative L signal
Next consider the case where the score captures a greater portion of the variance in ability for the L
group than for the H group, that is when γ L > γ H . Then imbalances in the true positive rate will
be alleviated. The expected scores conditional on true underlying ability are plotted in Figure 2d,
where we see that at low levels of underlying ability, L members have lower expected scores than
H members, whereas they have higher expected scores at higher levels of ability. Meanwhile, the
distributions of the scores of creditworthy individuals are plotted in Figures 2e and 2f 5 . For the
particular signal variance depicted, the shaded portion under each groups’ curve is approximately the
same, signifying that the groups’ true positive rates are approximately the same.
We further consider the effect of signal strength on true positive rate imbalances in Figure 3. There,
we vary the relative signal strength γL /γH of the L group versus the H group on the x-axis, and
plot the resulting ratio of group true positive rates for various mean disparities across groups. In
cherry-picking markets, all mean disparities require L to have a greater signal strength to achieve a
true positive rate at least as large as that of the H group. The effect is present but relatively muted
in lemon-dropping markets, in which so many applicants are accepted from both groups that both
groups’ true positive rates are generally high.
3
The conditional distribution of S|A > 0 is normal. Derivation details for its mean and variance are provided
in the appendix.
4
The plots in Figures 2b and 2c are constructed using parameter values (µH , σS2 H , σU 2
H
) = (0, 1, 1) and
2 2
(µL , σSL , σUL ) = (−1, 1, 1). The lemon-dropping and cherry-picking cutoffs depicted are −1 and +1,
respectively.
5
The plots in Figures 2e and 2f are constructed by changing the L variances to (σS2 L , σU2
L
) = (1.6, 0.4).
6Case I: Equal Variances Case II: Unequal Variances
(a) (d)
(b) (e)
(c) (f)
Figure 2: Distributions of scores in the parametric model. Plots in left column illustrate the case with equal
variance scores, and plots in right column are based on higher-variance scores for the L group. In (a), we
see that expected scores conditional on true underlying ability are systematically higher for H members. In
(b) and (c), we plot the distribution of scores for creditworthy applicants (A > 0) in a lemon-dropping and
cherry-picking market, respectively. The group-specific true positive rates are given by the portion under each
curve that is shaded. Members of H have higher true positive rates, and the difference is particularly stark in the
cherry-picking example. The imbalance is alleviated when the L scores become more informative, as seen in
(d)-(f).
7smaller μL
smaller μL
Figure 3: Ratios of true positive rates as a function of relative signal strengths. The mean of group H is held
fixed at 0, while the mean of L is set to -0.5, -1.0, and -1.5. Bigger disparities in group means are associated
with bigger disparities in true positive rates. In cherry-picking markets, L group always needs strictly stronger
signal than H group to achieve the same true positive rate as the H group.
3.2.3 Comparison to Aigner and Cain (1977)
The canonical Aigner and Cain (1977) model of statistical discrimination showed that when a
rational decision-maker is presented with applicants’ noisy unbiased signals for ability, S = A + ε,
then there is an incentive to weigh applicants’ group membership G alongside those signals. The
decision-maker will assign equal outcomes to applicants with equal expected ability, but as a result
of their consideration of group membership, outcomes for applicants with equal true ability will
systematically differ by group.
Our model shows how the disparate impact documented in Aigner and Cain (1977) can persist even
when a decision-maker is blind to group membership. Group means enter our model not through the
decision-maker’s interpretation of S, but rather through the construction of S itself.
We first adjust the meaning of the signal S in the Aigner and Cain (1977) model. Rather than treating
S as a noisy measurement of ability, like an exam, we instead suppose that it is the best estimate of
ability given observables. Therefore it becomes natural for us to consider a risk score S = E[A|X]
estimated from the model A = S + ε, instead of Aigner and Cain’s S = A + ε. 6
We further suppose that the signal S is constructed with observables that fully explain the group
variation in ability. As a consequence, we prove that the best estimate for ability A is given by the
score S regardless of whether group G is observed; unlike in Aigner and Cain (1977), the rational
decision-maker has no incentive to update the ability estimate given group information. Barring the
decision-maker in our model from considering group membership does not alleviate disparities and in
fact has no effect. If both groups’ signals are equally informative, then the differences in true positive
rates identified by Aigner and Cain (1977) will persist.
We show that reducing uncertainty, however, does alleviate the disparities. If both groups’ signals are
made more informative, then each groups’ scores will more closely track true ability rather than bunch
around group means and disparities will narrow. Meanwhile, if care is taken to improve the precision
of signals for the low-mean group in particular, then the disparity can be completely eliminated.
6
Our signal is distributed as a mean-preserving contraction of the true ability, whereas the signal in Aigner
and Cain (1977) is distributed as a mean-preserving spread of the ability.
8(a) (b) (c)
Figure 4: Comparisons between affirmative information and affirmative action. (a) Affirmative information
corresponds to higher portions of loans repaid at every cutoff. (b) Before the lender responds to intervention,
affirmative action yields greater TPRs for the lower-mean group. (c) Next we model TPRs after the lender’s
endogenous response to interventions. When the original blind prediction is replaced by affirmative action, the
TPR of the higher-mean group falls and the TPR of the lower-mean group increases modestly. Meanwhile, when
blind prediction is replaced by affirmative information, the TPR of the higher-mean group is roughly unchanged
and the TPR of the lower-mean group increases. For all but the lowest values of k, affirmative information is
associated with higher TPRs for both groups than affirmative action.
4 Empirical Application
We return to our empirical example from Section 2, where we demonstrated how a set of credit risk
scores can lead to disparate outcomes across education groups. Despite the fact that the scores were
constructed without group identifiers, they yielded lower true and false positive rates to less-educated
applicants at every cutoff. In this section, we compare two strategies for raising the true positive rate
of the less-educated group: affirmative action and affirmative information.
In our setting, we use the U.S. Census Survey of Income and Program Participation (SIPP), a
nationally-representative panel survey of the civilian population. We label our outcome of “creditwor-
thiness” as the reported ability to pay rent, mortgage, and utilities in 2016, and use survey response
from two years prior to predict that outcome, as in [13]. Of the overall sample, we call the 55% who
studied beyond a high school education group H and those with at most a high school degree group
L. Mean outcomes vary by group: about 93% of H members report to have repaid their obligations,
compared with 89% of the L members. Our risk scores are trained with a logistic regression on a
subset of the data and then tested on a separate holdout set.
We suppose a lender is presented with these scores and uses them to assign loan denials and approvals,
without observing individuals’ group membership. The lender approves loans for anyone whose
score corresponds to a positive expected return, and therefore imposes a cutoff rule that depends on
their relative cost of false positive and false negative errors. When false positives are more costly,
the cutoff is greater. As in Section 2, at every cutoff, blind prediction based only on total debt, total
assets, and monthly income yields classifications with smaller TPR for the L group.
We first consider how affirmative action can correct the error disparity by over-representing individuals
from L among the positive classifications. For each of the lender’s intended cutoffs, we lower the
corresponding cutoff for L members until their TPR reaches the level of the H applicants. In Figure
4a, we can see that this scheme reduces the portion of loans subsequently repaid to the lender.
However, in Figure 4b, we can see that it yields group TPRs that are approximately equal.
As an alternative, we consider an affirmative information scheme that aims to raise the TPR of
individuals in L by improving accuracy for the L group. We suppose the decision-maker has taken
effort to gather additional data about the L applicants, and to simulate that we train new L scores
using the much richer set of variables available in SIPP, spanning household financial details as well
as more sensitive features such as food security, family structure, and participation in social programs.
Due to the large number of variables, we use LASSO to estimate risk scores. The MSE for L scores
drops from .098 in the Blind Prediction scheme to .092 in the affirmative information scheme. Unlike
9affirmative action, we impose the same cutoffs for each groups’ set of scores: the original scores for
H and the data-enriched scores for L.
The affirmative information scheme corresponds to a higher portion of loans repaid than affirmative
action at all cutoffs, as seen in Figure 4a. Meanwhile, its immediate effect on the L group TPR
depends on the cutoff. In Figure 4b we see that at many values, affirmative information raises the
TPR above its original value, though not as much as affirmative action.
However, this comparison does not account for how a profit-maximizing lender would respond to the
interventions. One can imagine, for example, that requiring the lender to reduce an optimal cutoff for
a substantial portion of the population would incentivize them to re-optimize and raise cutoffs overall.
To identify the lender response to affirmative action and affirmative information, we consider a profit
function that generates cutoff decision rules: TP − kFP, where TP is the number of creditworthy
applicants given loans, FP is the number of defaulters given loans, and k is a constant representing
the relative cost of false positive classifications. We simulate the lender response by considering
different values of k in the profit function. For each value, we use the training data scores and labels to
determine the decision-maker’s optimal cutoffs in the three schemes. Blind prediction and affirmative
information are each associated with one optimal cutoff per k, while affirmative action is associated
with two optimal group-specific cutoffs per k among all pairs that yield equal TPRs. Then for each
optimal decision-rule, we compute the corresponding group TPRs in the holdout data and plot them
in Figure 4c.
Figure 4c illustrates the lasting impacts on TPRs from each intervention. We see that unlike in panel
(b), affirmative action does not raise the TPR of L applicants all the way to the original levels of
the H applicants. Instead it leads the two groups’ TPRs to meet in the middle, modestly raising the
TPR of L while reducing that of H. In addition, it declines the fastest, eliminating any potential for
profit and closing the market starting at k ≈ 45. By comparison, the affirmative information scheme
leaves the H TPR essentially unchanged, while raising the TPR of L applicants above the level of
affirmative action for all but the lowest values of k.
There are two noteworthy caveats to affirmative information. The first is that affirmative information
requires data acquisition, which is generally costly. The second is that it may be less effective in
lemon-dropping settings in which cutoffs are well below the mean, as seen in Figure 4c at the lowest
values of k. In particular, increasing accuracy in low-cutoff settings can have a counter effect of
reducing the over-representation of L members among positive classifications. This can be explained
by considering the effect of data enrichment on the spread of L scores. When decision cutoffs
are high, increasing the spread has two effects that move the TPR in the same direction: accuracy
increases, and more individuals are pushed above the cutoff than before. Yet when decision cutoffs
are low, there are two opposing forces acting on the TPR: though accuracy increases, more individuals
are pushed below the cutoff than before.
Overall, this empirical example has demonstrated the potential advantages of affirmative information
as a tool to broaden access to opportunity. Particularly in settings with selective admissions criteria,
employing affirmative information rather than affirmative action can yield TPRs that are higher for
both groups. In addition, depending on the cost of the data acquisition, affirmative information may
yield relatively higher payoffs to the lender. Targeted data enrichment can therefore create natural
incentives to broaden access to opportunities.
5 Conclusion
In the presence of uncertainty, we have shown that classifiers tend to distribute Type I and Type II
errors unevenly among groups whose mean outcomes differ. They tend to assign more false negatives
to individuals in lower-mean groups, and more false positives to those in higher-mean groups. As a
result, individuals in lower-mean groups are less likely to be classified as positive conditional on their
true labels. We show both empirically and in a theoretical model that this phenomenon can persist
even if sensitive data are removed from the prediction process.
Instead of data omission, we show that one of two conditions is required to eliminate differences in
groups’ true positive rates: either applicants from lower-mean groups are over-represented among
positive classifications, or their classifications are more accurate than those of higher-mean groups.
The latter condition suggests that data enrichment for individuals in lower-mean protected groups is a
10promising alternative to data omission. We call the strategy “affirmative information” and suggest
that it can improve both efficiency and equity.
References
[1] Dennis J Aigner and Glen G Cain. Statistical theories of discrimination in labor markets. Ilr
Review, 30(2):175–187, 1977.
[2] Julia Angwin, Jeff Larson, Surya Mattu, and Lauren Kirchner. Machine bias. ProPublica, May,
23:2016, 2016.
[3] Solon Barocas, Moritz Hardt, and Arvind Narayanan. Fairness and Machine Learning. fairml-
book.org, 2019. http://www.fairmlbook.org.
[4] Vojtěch Bartoš, Michal Bauer, Julie Chytilová, and Filip Matějka. Attention discrimination:
Theory and field experiments with monitoring information acquisition. American Economic
Review, 106(6):1437–75, 2016.
[5] Irene Chen, Fredrik D Johansson, and David Sontag. Why is my classifier discriminatory? In
Advances in Neural Information Processing Systems, pages 3539–3550, 2018.
[6] Cynthia Dwork, Moritz Hardt, Toniann Pitassi, Omer Reingold, and Richard Zemel. Fairness
through awareness. In Proceedings of the 3rd innovations in theoretical computer science
conference, pages 214–226, 2012.
[7] Cynthia Dwork, Nicole Immorlica, Adam Tauman Kalai, and Max Leiserson. Decoupled classi-
fiers for group-fair and efficient machine learning. In Conference on Fairness, Accountability
and Transparency, pages 119–133, 2018.
[8] Virginia Eubanks. A child abuse prediction model fails poor families. Wired Magazine, 2018.
[9] Lex Fridman and Michael I. Jordan. Machine learning, recommender systems, and the future of
AI. Lex Fridman Podcast, 2020.
[10] Moritz Hardt, Eric Price, and Nati Srebro. Equality of opportunity in supervised learning. In
Advances in neural information processing systems, pages 3315–3323, 2016.
[11] Jon Kleinberg, Jens Ludwig, Sendhil Mullainathan, and Ashesh Rambachan. Algorithmic
fairness. In AEA papers and proceedings, volume 108, pages 22–27, 2018.
[12] Jon Kleinberg and Sendhil Mullainathan. Simplicity creates inequity: implications for fairness,
stereotypes, and interpretability. In Proceedings of the 2019 ACM Conference on Economics
and Computation, pages 807–808, 2019.
[13] Claire Lazar Reich and Suhas Vijaykumar. A possibility in algorithmic fairness: calibrated
scores for fair classifications. arXiv e-prints, arXiv:2002.07676v2, 2020.
[14] Ziad Obermeyer, Brian Powers, Christine Vogeli, and Sendhil Mullainathan. Dissecting racial
bias in an algorithm used to manage the health of populations. Science, 366(6464):447–453,
2019.
[15] Ashesh Rambachan, Jon Kleinberg, Sendhil Mullainathan, and Jens Ludwig. An economic
approach to regulating algorithms. Technical report, National Bureau of Economic Research,
2020.
[16] Muhammad Bilal Zafar, Isabel Valera, Manuel Gomez Rodriguez, and Krishna P Gummadi.
Fairness beyond disparate treatment & disparate impact: Learning classification without dis-
parate mistreatment. In Proceedings of the 26th international conference on world wide web,
pages 1171–1180, 2017.
116 Appendix
Proof of Proposition 1
Proof. Proof by contradiction. Suppose that L members are not over-represented among positive
classifications, so P[G = L|ŷ = 1] ≤ P[G = L|Y = 1]. Also denote the group-specific positive
classification rates as µ̂G = P[ŷ = 1|G]. Then, we can show that
µ̂H µH
≥ (6)
µ̂L µL
and
µ̂H > µ̂L (7)
In particular, we can prove (6) by observing the equivalence of the following statements
P[G = L|ŷ = 1] ≤ P[G = L|Y = 1]
P[ŷ = 1|G = L]P[G = L] P[Y = 1|G = L]P[G = L]
≤
P[ŷ = 1] P[Y = 1]
P[ŷ = 1|G = L] P[Y = 1|G = L]
≤
P[ŷ = 1] P[Y = 1]
µ̂L µL
≤
µ̂L P[G = L] + µ̂H P[G = H] µL P[G = L] + µH P[G = H]
µ̂L P[G = L] + µ̂H P[G = H] µL P[G = L] + µH P[G = H]
≥
µ̂L µL
µ̂H P[G = H] µH P[G = H]
P[G = L] + ≥ P[G = L] +
µ̂L µL
µ̂H P[G = H] µH P[G = H]
≥
µ̂L µL
µ̂H µH
≥
µ̂L µL
And from this and the assumption that µH > µL the inequality (7) also follows.
We will use (6) and (7) to arrive at a contradiction when the true and false positive rates are at
least as big for the L group as for the H group. First, using Bayes’ rule, we rewrite the following
group-specific error rates in terms of the group-specific positive predictive values P P VA , rates of
positive classification µ̂A , and base rates µA .
P[Y = 1|G, ŷ = 1]P[ŷ = 1|G] µ̂G
T P RG = P[ŷ = 1|G, Y = 1] = = P P VG (8)
P[Y = 1|G] µG
P[Y = 0|G, ŷ = 1]P[ŷ = 1|G] µ̂G
F P RG = P[ŷ = 1|G, Y = 0] = = (1 − P P VG ) (9)
P[Y = 0|G] 1 − µG
(10)
Then for the F P RL to be at least as big as F P RH , it must be that
µ̂L µ̂H
(1 − P P VL ) ≥ (1 − P P VH ) (11)
1 − µL 1 − µH
1 − P P VL µ̂H (1 − µL )
≥ (12)
1 − P P VH µ̂L (1 − µH )
12According to (7), the RHS of (12) is strictly greater than 1. Therefore it must be that
P P VL < P P VH . (13)
Meanwhile, for T P RL to be at least as big as T P RH ,
µ̂L µ̂H
P P VL ≥ P P VH (14)
µL µH
P P VL µ̂H µL
≥ (15)
P P VH µH µ̂L
µ̂H
P P VL µ̂L
≥ µH (16)
P P VH µL
Due to (6), the RHS is weakly greater than 1. Therefore, it must be that P P VL ≥ P P VH which
contradicts with (13).
Proof of Theorem 1
Proof. Consider a classifier ŷ for which T P RL ≥ T P RH . As seen in the proof of Proposition 1,
T P RG = P P VG µ̂µG
G
, so we can rewrite the inequality as
µ̂L µ̂H
P P VL ≥ P P VH (17)
µL µH
First we suppose condition (i) does not hold and show that condition (ii) then must hold for (17) to be
true. We will use our result in the proof of Proposition 1 where we showed that if L members are not
over-represented among positive classifications, it necessarily follows that (6) and (7).
Notice that (17) can be rewritten as P P VL µµHL ≥ P P VH µ̂µ̂HL , which combined with (6) implies that
P P VL ≥ P P VH .
Meanwhile to prove the N P V inequality of (ii), notice that if T P RL ≥ T P RH then it follows the
groups’ false negative rates must satisfy F N RL ≤ F N RH . Rewriting the group-A F N R using
Bayes’ rule gives F N RG = P[ŷ = 0|G, Y = 1] = P[Y =1|G,ŷ=0]P[ŷ=0|G]
P[Y =1|G] = (1 − N P VG ) 1−µ̂G
µG . So
we get the inequality
1 − µ̂L 1 − µ̂H
(1 − N P VL ) ≤ (1 − N P VH ) (18)
µL µH
1 − N P VL (1 − µ̂H ) µL
≤ (19)
1 − N P VH (1 − µ̂L ) µH
Using (7) and the assumption µL < µH , the RHS is strictly less than 1. Therefore on the LHS it must
be that N P VL > N P VH . Therefore we have shown that if (i) does not hold then (ii) must hold.
Next we show that if (ii) does not hold then (i) must for T P RL ≥ T P RH to be true. We consider
the cases P P VL < P P VH and N P VL ≤ N P VH separately:
µ̂H µ̂L
Case I: Suppose P P VL < P P VH . Then according to (17), it must be that µH < µL , or equivalently
µ̂H µH
µ̂L < µL . This necessarily implies that (i) holds and the L group is over-represented among positive
classifications, P[G = L|ŷ = 1] > P[G = L|Y = 1]. To see why, note that in the proof for
Proposition 1, we showed that the negation P[G = L|ŷ = 1] ≤ P[G = L|Y = 1] is equivalent to (6),
the negation of µ̂µ̂HL < µµHL .
Case II: Suppose N P VL ≤ N P VH . Then according to (19),
13(1 − µ̂H ) µL
≥1 (20)
(1 − µ̂L ) µH
(1 − µ̂H ) µH
≥ (21)
(1 − µ̂L ) µL
Since µH > µL , the RHS is strictly greater than 1 implying (1−µ̂ H)
(1−µ̂L ) > 1 and so µ̂H < µ̂L . This
clearly means that L members are over-represented among positive classifications and (i) holds. One
way to see this is to note that µ̂H < µ̂L implies µ̂µ̂HL < µµHL , which is the negation of (6), a condition
equivalent to P[G = L|ŷ = 1] ≤ P[G = L|Y = 1] as seen in the proof of Proposition 1.
Proof of Proposition 2
Proof. We will use a lemma in Mitchell et al. (2019), based on the law of iterated expectations: any
three random variables W, V, Z satisfy E[W |V, Z] = E[W |E[W |V, Z], Z].
Applied to our setting, we have
E[A|X, G] = E[A|E[A|X, G], G] by lemma (22)
= E[A|E[A|X], G] by mean independence E[A|X, G] = E[A|X] (23)
= E[A|S, G] by definition of S (24)
(25)
From our conditional mean independence assumption, we also know that the LHS is equal to
E[A|X] = S. Combining with the RHS, we have S = E[A|S, G].
Derivation of S|A > 0
We can derive the mean and variance for the distribution of X|Y > c where (X, Y ) are bivariate
normal. To find the mean, use the law of iterated expectations:
E[X|Y > c] = E[E[X|Y ]|Y > c] (26)
The inner expectation E[X|Y = y] is:
y − µy
E[X|Y = y] = µx + cov(X, Y )( ) (27)
σy2
Take the expectation of this given Y > c:
E[Y |Y > c] − µy
E[E[X|Y ]|Y > c] = µx + cov(X, Y )( ) (28)
σy2
c−µy
φ( σy )
(µy + σy ( c−µ )) − µy
1−Φ( σy y )
= µx + cov(X, Y ) (29)
σy2
c−µy
φ( σy )
= µx + cov(X, Y ) c−µ (30)
σy (1 − Φ( σy y ))
c−µy
ρσx φ( σy )
= µx + c−µy (31)
1− Φ( σy )
14And we get
c−µy
ρσx φ( σy )
E[X|Y > c] = µx + c−µ (32)
1− Φ( σy y )
The variance can be computed similarly, using Var(X|Y > c) = E[X 2 |Y > c] − E[X|Y > c]2 .
Solving and simplifying with Mathematica gives:
c−µy c−µy c−µy
ρ2 φ( σy )[σy φ( σy ) − (c − µy )(1 − Φ( σy ))]
V ar[X|Y > c] = σx2 (1 − c−µy 2 ) (33)
σy (1 − Φ( σy ))
15You can also read
