Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities - arXiv.org
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Fairness for Unobserved Characteristics:
Insights from Technological Impacts on Queer Communities
Nenad Tomasev,1 Kevin R. McKee,1 Jackie Kay,1 Shakir Mohamed1
1
DeepMind, London, UK
nenadt@, kevinrmckee@, kayj@, shakir@google.com
arXiv:2102.04257v2 [cs.CY] 9 Feb 2021
Abstract Algorithms have moral consequences for queer commu-
nities, too. However, algorithmic fairness for queer individ-
Advances in algorithmic fairness have largely omitted sexual uals and communities remains critically underexplored. In
orientation and gender identity. We explore queer concerns in
privacy, censorship, language, online safety, health, and em-
part, this stems from the unique challenges posed by study-
ployment to study the positive and negative effects of artificial ing sexual orientation and gender identity. Most definitions
intelligence on queer communities. These issues underscore of algorithmic fairness share a basis in norms of egalitari-
the need for new directions in fairness research that take into anism (Barocas, Hardt, and Narayanan 2019; Binns 2018).2
account a multiplicity of considerations, from privacy preser- For example, classification parity approaches to fairness aim
vation, context sensitivity and process fairness, to an aware- to equalise predictive performance measures across groups,
ness of sociotechnical impact and the increasingly important whereas anti-classification parity approaches rely on the
role of inclusive and participatory research processes. Most omission of protected attributes from the decision making
current approaches for algorithmic fairness assume that the process to ensure different groups receive equivalent treat-
target characteristics for fairness—frequently, race and legal ment (Corbett-Davies and Goel 2018). An inherent assump-
gender—can be observed or recorded. Sexual orientation and
gender identity are prototypical instances of unobserved char-
tion of these approaches is that the protected characteristics
acteristics, which are frequently missing, unknown or funda- are known and available within datasets. Sexual orientation
mentally unmeasurable. This paper highlights the importance and gender identity are prototypical examples of unobserved
of developing new approaches for algorithmic fairness that characteristics, presenting challenging obstacles for fairness
break away from the prevailing assumption of observed char- research (Andrus et al. 2020; Jacobs and Wallach 2019).
acteristics. This paper explores the need for queer fairness by review-
ing the experiences of technological impacts on queer com-
munities. For our discussion, we define ‘queer’ as ‘possess-
Introduction ing non-normative sexual identity, gender identity, and/or
As the field of algorithmic fairness has matured, the ways sexual characteristics’. We consider this to include les-
in which machine learning researchers and developers op- bian, gay, bisexual, pansexual, transgender, and asexual
erationalise approaches for fairness have expanded in scope identities—among others.3
and applicability. Fairness researchers have made important The focus on queer communities is important for several
advances and demonstrated how the risks of algorithmic reasons. Given the historical oppression and contemporary
systems are imbalanced across different characteristics of challenges faced by queer communities, there is a substantial
the people who are analysed and affected by classifiers and risk that artificial intelligence (AI) systems will be designed
decision-making systems (Barocas, Hardt, and Narayanan and deployed unfairly for queer individuals. Compounding
2019; Fudenberg and Levine 2012). Progress has been par- this risk, sensitive information for queer people is usually
ticularly strong with respect to race and legal gender.1 Fair- not available to those developing AI systems, rendering the
ness studies have helped to draw attention to racial bias in re- resulting unfairness unmeasurable from the perspective of
cidivism prediction (Angwin et al. 2016), expose racial and
gender bias in facial recognition (Buolamwini and Gebru 2
It is worth noting that certain algorithmic domains supplement
2018), reduce gender bias in language processing (Boluk- egalitarian concerns with additional ethical values and principles.
basi et al. 2016; Park, Shin, and Fung 2018), and increase For example, fairness assessments of healthcare applications typi-
the accuracy and equity of decision making for child protec- cally incorporate beneficence and non-malfeasance, two principles
tive services (Chouldechova et al. 2018). central to medical ethics (Beauchamp and Childress 2001).
3
Throughout this paper, we use ‘queer’ and ‘LGBTQ+’ in-
1
Throughout this paper, we distinguish between ‘legal gender’ terchangeably. The heterogeneity of queer communities—and the
(the gender recorded on an individual’s legal documents, often as- complexity of the issues they face—preclude this work from being
signed to them at birth by the government, physicians or their par- an exhaustive review of queer identity. As a result, there are likely
ents) and ‘gender identity’ (an individual’s personal feelings and perspectives that were not included in this manuscript, but that have
convictions about their gender; Brook n.d.). an important place in broader discussions of queer fairness.standard group fairness metrics. Despite these issues, fair- Promise Advances in privacy-preserving machine learn-
ness research with respect to queer communities is an un- ing (Nasr, Shokri, and Houmansadr 2018; Bonawitz et al.
derstudied area. Ultimately, the experiences of queer com- 2017; Jayaraman and Evans 2019) present the possibil-
munities can reveal insights for algorithmic fairness that are ity that the queer community might benefit from AI sys-
transferable to a broader range of characteristics, including tems while minimising the risk of information leakage. Re-
disability, class, religion, and race. searchers have proposed adversarial filters (Zhang et al.
This paper aims to connect ongoing efforts to strengthen 2020; Liu, Zhang, and Yu 2017) to obfuscate sensitive infor-
queer communities in AI research (Agnew et al. 2018; Ag- mation in images and speech shared online while reducing
new, Bilenko, and Gontijo Lopes 2019; John et al. 2020) the risks of re-identification.
and sociotechnical decision making (Out in Tech n.d.; Les- Still, challenges remain (Srivastava et al. 2019). More re-
bians Who Tech n.d.; Intertech LGBT+ Diversity Forum search is needed to ensure the robustness of the adversar-
n.d.; LGBT Technology Institute n.d.) with recent advances ial approaches. The knowledge gap between the privacy and
in fairness research, including promising approaches to pro- machine-learning research communities must be bridged for
tecting unobserved characteristics. This work additionally these approaches to achieve the desired effects. This will en-
advocates for the expanded inclusion of queer voices in fair- sure that the appropriate types of protections are included
ness and ethics research, as well as the broader development in the ongoing development of AI solutions (Al-Rubaie and
of AI systems. We make three contributions in this paper: Chang 2019).
1. Expand on the promise of AI in empowering queer com- Risks A multitude of privacy risks arise for queer people
munities and supporting LGBTQ+ rights and freedoms. from the applications of AI systems. We focus in particu-
2. Emphasise the potential harms and unique challenges lar on the categorisation of identity from sensitive data, the
raised by the sensitive and unmeasurable aspects of iden- ethical risk of surveillance, and invasions of queer spaces.
tity data for queer people. In 2017, Stanford researchers attempted to build an AI
‘gaydar’, a computer vision model capable of guessing a per-
3. Based on use cases from the queer experience, establish son’s sexual orientation from images (Wang and Kosinski
requirements for algorithmic fairness on unobserved char- 2018). The resulting algorithm, a logistic regression model
acteristics. trained on 35,326 facial images, achieved a high reported ac-
curacy in identifying self-reported sexual orientation across
Considerations for Queer Fairness both sexes. The results of this study have since been ques-
tioned, largely on the basis of a number of methodological
To emphasise the need for in-depth study of the impact of
and conceptual flaws that discredit the performance of the
AI on queer communities around the world, we explore sev-
system (Gelman, Marrson, and Simpson 2018). Other algo-
eral case studies of how AI systems interact with sexual
rithms designed to predict sexual orientation have suffered
orientation and gender identity. Each of these case studies
similar methodological and conceptual deficiencies. A re-
highlights both potential benefits and risks of AI applica-
cently released app claimed to be able to quantify the ev-
tions for queer communities. In reviewing these cases, we
idence of one’s non-heterosexual orientation based on ge-
hope to motivate the development of technological solutions
netic data, for example (Bellenson 2019), largely obfuscat-
that are inclusive and beneficial to everyone. Importantly,
ing the limited ability of genetic information to predict sex-
these case studies will demonstrate cross-cutting challenges
ual orientation (Ganna et al. 2019).
and concerns raised by unobserved and missing characteris-
tics, such as preserving privacy, supporting feature imputa- Though these specific efforts have been flawed, it is plau-
tion, context-sensitivity, exposing coded inequity, participa- sible that in the near future algorithms could achieve high ac-
tory engagement, and sequential and fair processes. curacy, depending on the data sources involved. Behavioural
data recorded online present particular risks to the privacy
of sexual orientation and gender identity: after all, the more
Privacy time people spend online, the greater their digital footprint.
Sexual orientation and gender identity are highly private AI ‘gaydars’ relying on an individual’s recorded interests
aspects of personal identity. Outing queer individuals—by and interactions could pose a serious danger to the privacy
sharing or exposing their sexual orientation or gender iden- of queer people. In fact, as a result of the long-running
tity without their prior consent—can not only lead to emo- perception of the queer community as a profitable ‘con-
tional distress, but also risk serious physical and social sumer group’ by business and advertisers alike, prior efforts
harms, especially in regions where queerness is openly dis- have used online data to map ‘queer interests’ in order to
criminated against (Wang et al. 2019), criminalised (DeJong boost sales and increase profits (Sender 2018). In at least
and Long 2014) or persecuted (Scicchitano 2019). Privacy one instance, researchers have attempted to use basic social
violations can thus have major consequences for queer in- media information to reconstruct the sexual orientation of
dividuals, including infringement upon their basic human users (Bhattasali and Maiti 2015).
rights (Bosia, McEvoy, and Rahman 2020; Amnesty Inter- The ethical implications of developing such systems for
national Canada 2015), denial of employment and education queer communities are far-reaching, with the potential of
opportunities, ill-treatment, torture, sexual assault, rape, and causing serious harms to affected individuals. Prediction al-
extrajudicial killings. gorithms could be deployed at scale by malicious actors, par-ticularly in nations where homosexuality and gender non- cal and virtual spaces is a basic human right. Multiple groups
conformity are punishable offences. In fact, in many such and institutions around the world violate this right through
nations, authorities already use technology to entrap or lo- censorship of queer content.
cate queer individuals through social media and LGBTQ+ This censorship is often justified by its supporters as ‘pre-
dating apps (e.g., Culzac 2014). Systems predicting sex- serving decency’ and ‘protecting the youth’, but in reality
ual orientation may also exacerbate the pre-existing privacy leads to the erasure of queer identity. Laws against ‘materials
risks of participating in queer digital spaces. There have promoting homosexuality’ were established in the late 1980s
been recorded cases of coordinated campaigns for outing in the United Kingdom and repealed as late as 2003 (Bur-
queer people, resulting in lives being ruined, or lost due ridge 2004). Nations that are considered major world pow-
to suicide (Embury-Dennis 2020). These malicious outing ers have laws banning the portrayal of same-sex romances in
campaigns have until now been executed at smaller scales. television shows (e.g., China; Lu and Hunt 2016), the men-
However, recent developments in AI greatly amplify the po- tion of homosexuality or transgender identities in public ed-
tential scale of such incidents, endangering larger commu- ucation (e.g., state-level laws in the United States; Hoshall
nities of queer people in certain parts of the world. Facial 2012), or any distribution of LGBT-related material to mi-
recognition technology (Voulodimos et al. 2018) could be nors (e.g., Russia; Kondakov 2019). Not only do such laws
employed by malicious actors to rapidly identify individu- isolate queer people from their communities—particularly
als sharing their pictures online, whether publicly or in di- queer youth—they implicitly shame queerness as indecent
rect messages. Facial recognition could similarly be used behaviour, setting a precedent for further marginalisation
to automatically identify people in captured recordings of and undermining of human rights. Many queer content pro-
protests, in queer nightclubs or community spaces, and other ducers in such nations have argued that their online content
in-person social events. These possibilities highlight the po- is being restricted and removed at the detriment of queer ex-
tential dangers of AI for state-deployed surveillance technol- pression and sex positivity, as well as at the cost of their
ogy. Chatbots have similarly been deployed to elicit private income (York 2015).
information on dating apps, compromising users’ device in-
tegrity and privacy (McCormick 2015). Existing bots are Promise AI systems may be effectively used to mitigate
scripted, and therefore can usually be distinguished from hu- censorship of queer content. Machine learning has been used
man users after longer exchanges. Nonetheless, strong lan- to analyse and reverse-engineer patterns of censorship. A
guage models (Brown et al. 2020) threaten to exacerbate the study of 20 million tweets from Turkey employed machine
such privacy risks, given their ability to quickly adjust to the learning to show that the vast majority of censored tweets
style of communication based on a limited number of ex- contained political content (Tanash et al. 2015). A statistical
amples. These language models amplify existing concerns analysis of Weibo posts and Chinese-language tweets un-
around the collection of private information and the com- covered a set of charged political keywords present in posts
promising of safe online spaces. with anomalously high deletion rates (Bamman, O’Connor,
In addition to the direct risks to privacy, algorithms in- and Smith 2012). Further study of censorship could be key
tended to predict sexual orientation and gender identity also to drawing the international community’s attention to hu-
perpetuate concerning ideas and beliefs about queerness. man rights violations. It could also potentially be used to
Systems using genetic information as the primary input, for empower affected individuals to circumvent these unfair re-
example, threaten to reinforce biological essentialist views strictions. However, a large-scale study of deleted queer con-
of sexual orientation and echo tenets of eugenics—a his- tent in countries which censor such content has yet to be
torical framework that leveraged science and technology to conducted.
justify individual and structural violence against people per- Risk Although we believe machine learning can be used
ceived as inferior (Ordover 2003; Wolbring 2001). More to combat censorship, tools for detecting queer digital con-
broadly, the design of predictive algorithms can lead to erro- tent can be abused to enforce censorship laws or heteronor-
neous beliefs that biology, appearance or behaviour are the mative cultural attitudes. As social network sites, search en-
essential features of sexual orientation and gender identity, gines and other media platforms adopt algorithms to moder-
rather than imperfectly correlated causes, effects or covari- ate content at scale, the risk for unfair or biased censorship of
ates of queerness. queer content increases, and governing entities are empow-
In sum, sexual orientation and gender identity are associ- ered to erase queer identities from the digital sphere (Cobbe
ated with key privacy concerns. Non-consensual outing and 2019). Automated content moderation systems are at risk of
attempts to infer protected characteristics from other data censoring queer expression even when the intention is be-
thus pose ethical issues and risks to physical safety. In order nign, such as protecting users from verbal abuse. To help
to ensure queer algorithmic fairness, it will be important to combat censorship restrictions and design fair content mod-
develop methods that can improve fairness for marginalised eration systems, ML fairness researchers could investigate
groups without having direct access to group membership how to detect and analyse anomalous omission of informa-
information. tion related to queer identity (or other protected characteris-
tics) in natural language and video data.
Censorship Censorship often goes hand-in-hand with the distortion of
Although queer identity is essentially unmeasurable, we be- facts. Recent advances in generative models have made the
lieve that its unrestricted outward expression in both physi- fabrication of digital content trivial, given enough data andcomputational power (Chesney and Citron 2019). Malicious entation and gender identity (Strengers et al. 2020).
and dehumanising misinformation about the queer commu-
nity has been used as justification for abuse and suppres- Risks Biases, stereotypes and abusive speech are persis-
sion throughout history, tracing back to medieval interpre- tently present in top-performing language models, as a re-
tations of ancient religious texts (Dynes 2014). Technolog- sult of their presence in the vast quantities of training
ical and political solutions to the threat of misinformation data that are needed for model development (Costa-jussà
are important for protecting queer expression—as well as 2019). Formal frameworks for measuring and ensuring fair-
global democracy. The AI community has begun to develop ness (Hendrycks et al. 2020a,b; Sheng et al. 2020) in lan-
methods to verify authentic data through, for example, open guage are still in nascent stages of development. Thus, for
datasets and benchmarks for detecting synthetic images and AI systems to avoid reinforcing harmful stereotypes and per-
video (Rossler et al. 2019). petuating harm to marginalised groups, research on inclusive
While the goal of fairness for privacy is preventing the im- language requires more attention. For language systems to
putation of sensitive data, the goal of fairness for censorship be fair, they must be capable of reflecting the contextual na-
is to reveal the unfair prevention of expression. This dual- ture of human discourse.
ity could surface important technical connections between
these fields. In terms of social impact, many people around Fighting Online Abuse
the world outside of the queer community are negatively af- The ability to safely participate in online platforms is crit-
fected by censorship. Further research in fairness for censor- ical for marginalised groups to form a community and find
ship could have far-reaching benefit across technical fields, support (Liu 2020). However, this is often challenging due
social groups and borders. to pervasive online abuse (Jane 2020). Queer people are
frequently targets of internet hate speech, harassment and
Language trolling. This abuse may be directed at the community as
Language encodes and represents our way of thinking and a whole or at specific individuals who express their queer
communicating about the world. There is a long history identity online. Adolescents are particularly vulnerable to
of oppressive language being weaponised against the queer cyberbullying and the associated adverse effects, including
community (Nadal et al. 2011; Thurlow 2001), highlighting depression and suicidal ideation (Abreu and Kenny 2018).
the need for developing fair and inclusive language models. Automated systems for moderation of online abuse are a
Inclusive language (Weinberg 2009) extends beyond the possible solution that can protect the psychological safety
mere avoidance of derogatory terms, as there are many ways of the queer community at a global scale.
in which harmful stereotypes can surface. For example, the
phrase ‘That’s so gay’ (Chonody, Rutledge, and Smith 2012) Promise AI systems could potentially be used to help hu-
equates queerness with badness. Using the term ‘sexual pref- man moderators flag abusive online content and commu-
erence’ rather than ‘sexual orientation’ can imply that sex- nication directed at members of marginalised groups, in-
ual orientation is a volitional choice, rather than an intrin- cluding the queer community (Saha et al. 2019; Schmidt
sic part of one’s identity. Assuming one’s gender identity, and Wiegand 2017). A proof of concept for this applica-
without asking, is harmful to the trans community as it tion was developed in the Troll Patrol project (Delisle et al.
risks misgendering people. This can manifest in the careless 2019; Amnesty International n.d.), a collaboration between
use of assumed pronouns, without knowledge of an individ- Amnesty International and Element AI’s former AI for Good
ual’s identification and requested pronouns. Reinforcing bi- team. The Troll Patrol project investigated the application of
nary and traditional gender expression stereotypes, regard- natural language processing methods for quantifying abuse
less of intent, can have adverse consequences. The use of against women on Twitter. The project revealed concerning
gender-neutral pronouns has been shown to result in lower patterns of online abuse and highlighted the technological
bias against women and LGBTQ+ people (Tavits and Pérez challenges required to develop online abuse detection sys-
2019). To further complicate the matter, words which orig- tems. Recently, similar systems have been applied to tweets
inated in a derogatory context, such as the label ‘queer’ it- directed at the LGBTQ+ community. Machine learning and
self, are often reclaimed by the community in an act of re- sentiment analysis were leveraged to predict homophobia
sistance. This historical precedent suggests that AI systems in Portuguese tweets, resulting in 89.4% accuracy (Pereira
must be able to adapt to the evolution of natural language 2018). Deep learning has also been used to evaluate the level
and avoid censoring language based solely on its adjacency of public support and perception of LGBTQ+ rights follow-
to the queer community. ing the Supreme Court of India’s verdict regarding the de-
criminalisation of homosexuality (Khatua et al. 2019).
Promise Natural language processing applications perme- The ways in which abusive comments are expressed when
ate the field of AI. These applications include use cases of targeted at the trans community pose some idiosyncratic
general interest like machine translation, speech recognition, research challenges. In order to protect the psychological
sentiment analysis, question answering, chatbots and hate safety of trans people, it is necessary for automated online
speech detection systems. There is an opportunity to de- abuse detection systems to properly recognise acts of mis-
velop language-based AI systems inclusively—to overcome gendering or ‘deadnaming’. These systems have a simulta-
human biases and establish inclusive norms that would fa- neous responsibility to ensure that deadnames and other sen-
cilitate respectful communication with regards to sexual ori- sitive information are kept private to the user. It is thereforeessential for the queer community to play an active role in riers they face in accessing healthcare (Hsieh and Ruther
informing the development of such systems. 2016).
Risks Systems developed with the purpose of automat- Promise To this day, the prevalence of HIV among the
ically identifying toxic speech could introduce harms by queer community remains a major challenge. Introducing
failing to recognise the context in which speech occurs. systems that both reduce the transmission risk and improve
Mock impoliteness, for example, helps queer people cope care delivery for HIV+ patients will play a critical role in
with hostility; the communication style of drag queens improving health outcomes for queer individuals.
in particular is often tailored to be provocative. A recent Machine learning presents key opportunities to augment
study (Gomes, Antonialli, and Dias Oliva 2019) demon- medical treatment decisions (Bisaso et al. 2017). For ex-
strated that an existing toxicity detection system would rou- ample, AI may be productively applied to identify the pa-
tinely consider drag queens to be as offensive as white tients most likely to benefit from pre-exposure prophylaxis
supremacists in their online presence. The system further for HIV. A research team recently developed such a sys-
specifically associated high levels of toxicity with words like tem, which correctly identified 38.6% of future cases of
‘gay’, ‘queer’ and ‘lesbian’. HIV (Marcus et al. 2019). The researchers noted substan-
Another risk in the context of combating online abuse tial challenges: model sensitivity on the validation set was
is unintentionally disregarding entire groups through igno- 46.4% for men and 0% for women, highlighting the im-
rance of intersectional issues. Queer people of colour ex- portance of intersectionality for fair outcomes in healthcare.
perience disproportionate exposure to online (and offline) Machine learning has also been used to predict early viro-
abuse (Balsam et al. 2011), even within the queer commu- logical suppression (Bisaso et al. 2018), adherence to anti-
nity itself. Neglecting intersectionality can lead to dispro- retroviral therapy (Semerdjian et al. 2018), and individual
portionate harms for such subcommunities. risk of complications such as chronic kidney disease (Roth
To mitigate these concerns, it is important for the re- et al. 2020) or antiretroviral therapy-induced mitochondrial
search community to employ an inclusive and participatory toxicity (Lee et al. 2019).
approach (Martin Jr. et al. 2020) when compiling training
Risks Recent advances in AI in healthcare may lead to
datasets for abusive speech detection. For example, there are
widespread increases in welfare. Yet there is a risk that ben-
homophobic and transphobic slurs with a racialised connota-
efits will be unequally distributed—and an additional risk
tion that should be included in training data for abuse detec-
that queer people’s needs will not be properly met by the
tion systems. Furthermore, methodological improvements
design of current systems. Information about sexual orienta-
may help advance progress. Introducing fairness constraints
tion and gender identity is frequently absent from research
to model training has demonstrably helped mitigate the bias
datasets. To mitigate the privacy risk for patients and pre-
of cyber-bullying detection systems (Gencoglu 2020). Ad-
vent reidentification, HIV status and substance abuse are
versarial training can similarly assist by demoting the con-
also routinely omitted from published data. While such prac-
founds associated with texts of marginalised groups (Xia,
tices may be necessary, it is worth recognising the important
Field, and Tsvetkov 2020).
downstream consequences they have for AI system devel-
opment in healthcare. It can become impossible to assess
Health fairness and model performance across the omitted dimen-
The drive towards equitable outcomes in healthcare entails sions. Moreover, the unobserved data increase the likelihood
a set of unique challenges for marginalised communities. of reduced predictive performance (since the features are
Queer communities have been disproportionately affected dropped), which itself results in worse health outcomes. The
by HIV (Singh et al. 2018), suffer a higher incidence of coupled risk of a decrease in performance and an inability
sexually-transmitted infections, and are afflicted by elevated to measure it could drastically limit the benefits from AI in
rates of substance abuse (Wallace and Santacruz 2017). healthcare for the queer community, relative to cisgendered
Compounding these issues, queer individuals frequently ex- heterosexual patients. To prevent the amplification of exist-
perience difficulties accessing appropriate care (Bize et al. ing inequities, there is a critical need for targeted fairness
2011; Human Rights Watch 2018). Healthcare professionals research examining the impacts of AI systems in healthcare
often lack appropriate training to best respond to the needs for queer people.
of LGBTQ+ patients (Schneider, Silenzio, and Erickson- To help assess the quality of care provided to LGBTQ+
Schroth 2019). Even in situations where clinicians do have patients, there have been efforts aimed at approximately
the proper training, patients may be reluctant to reveal their identifying sexual orientation (Bjarnadottir et al. 2019) and
sexual orientation and gender identity, given past experi- gender identity (Ehrenfeld et al. 2019) from clinical notes
ences with discrimination and stigmatisation. within electronic health record systems. While well inten-
In recent months, the COVID-19 pandemic has amplified tioned, these machine learning models offer no guaran-
health inequalities (Bowleg 2020; van Dorn, Cooney, and tee that they will only identify patients who have explic-
Sabin 2020). Initial studies during the pandemic have found itly disclosed their identities to their healthcare providers.
that LGBTQ+ patients are experiencing poorer self-reported These models thus introduce the risk that patients will be
health compared to cisgendered heterosexual peers (O’Neill outed without their consent. Similar risks arise from mod-
2020). The health burden of COVID-19 may be especially els developed to rapidly identify HIV-related social media
severe for queer people of colour, given the substantial bar- data (Young, Yu, and Wang 2017).The risk presented by AI healthcare systems could po- port workers) or new data sources (including social me-
tentially intensify during medical gender transitions. There dia and engagement data). The Trevor Project, a prominent
are known adverse effects associated with transition treat- American organisation providing crisis intervention and sui-
ment (Moore, Wisniewski, and Dobs 2003). The active cide prevention services to LGBTQ+ youth (The Trevor
involvement of medical professionals with experience in Project n.d.), is one organisation working on such an ini-
cross-sex hormonal therapy is vital for ensuring the safety tiative. In partnership with Google.org and its research fel-
of trans people undergoing hormone therapy or surgery. lows, The Trevor Project developed an AI system to iden-
Since cisgendered individuals provide the majority of tify and prioritise community members at high risk while si-
anonymised patient data used to develop AI systems for multaneously increasing outreach to new contacts. The sys-
personalised healthcare, there will be comparatively fewer tem was designed to relate different types of intake-form
cases of trans patients experiencing many medical condi- responses to downstream diagnosis risk levels. A separate
tions. This scarcity could have an adverse impact on model group of researchers developed a language processing sys-
performance—there will be an insufficient accounting for tem (Liang et al. 2019) to identify help-seeking conversa-
the interactions between the hormonal treatment, its adverse tions on LGBTQ+ support forums, with the aim of helping
effects and potential comorbidities, and other health issues at-risk individuals manage and overcome their issues.
potentially experienced by trans patients. In other healthcare contexts, reinforcement learning has
Framing fairness as a purely technical problem that can recently demonstrated potential in steering behavioural in-
be addressed by the mere inclusion of more data or com- terventions (Yom-Tov et al. 2017) and improving health out-
putational adjustments is ethically problematic, especially comes. Reinforcement learning represents a natural frame-
in high-stakes domains like healthcare (McCradden et al. work for personalised health interventions, since it can be
2020). Selection bias and confounding in retrospective data set up to maximise long-term physical and mental well-
make causal inference particularly hard in this domain. being (Tabatabaei, Hoogendoorn, and van Halteren 2018).
Counterfactual reasoning may prove key for safely planning If equipped with natural language capabilities, such systems
interventions aimed at improving health outcomes (Prosperi might be able to act as personalised mental health assistants
et al. 2020). It is critical for fairness researchers to engage empowered to support mental health and escalate situations
deeply with both clinicians and patients to ensure that their to human experts in concerning situations.
needs are met and AI systems in healthcare are developed
and deployed safely and fairly. Risks Substantial risks accompany these applications.
Overall, research on any intervention-directed systems
Mental Health should be undertaken in partnership with trained mental
health professionals and organisations, given the consider-
Queer people are more susceptible to mental health prob-
able risks associated with misdiagnosing mental illness (cf.
lems than their heterosexual and cisgender peers, largely
Suite et al. 2007) and exacerbating the vulnerability of those
as a consequence of the chronically high levels of stress
experiencing distress.
associated with prejudice, stigmatisation and discrimina-
tion (Meyer 1995, 2003; Mays and Cochran 2001; Tebbe The automation of intervention decisions and mental
and Moradi 2016). As a result, queer communities experi- health diagnoses poses a marked risk for the trans com-
ence substantial levels of anxiety, depression and suicidal munity. In most countries, patients must be diagnosed with
ideation (Mental Health Foundation 2020). Compounding gender dysphoria—an extensive process with lengthy wait
these issues, queer people often find it more difficult to ask times—before receiving treatments such as hormone therapy
for help and articulate their distress (McDermott 2015) and or surgery (e.g., National Health Service n.d.). During this
face systemic barriers to treatment (Romanelli and Hudson process, many transgender individuals experience mistrust
2017). A recent LGBTQ+ mental health survey highlighted and invalidation of their identities from medical profession-
the shocking extent of issues permeating queer communi- als who withhold treatment based on rigid or discriminatory
ties (The Trevor Project 2020): 40% of LGBTQ+ respon- view of gender (Ashley 2019). Automating the diagnosis of
dents seriously considered attempting suicide in the past gender dysphoria may recapitulate these biases and deprive
twelve months, with more than half of transgender and non- many transgender patients of access to care.
binary youth having seriously considered suicide; 68% of Mental health information is private and sensitive. While
LGBTQ+ youth reported symptoms of generalised anxiety AI systems have the potential to aid mental health workers
disorder in the past two weeks, including more than three in identifying at-risk individuals and those who would most
in four transgender and nonbinary youth; 48% of LGBTQ+ likely benefit from intervention, such models may be mis-
youth reported engaging in self-harm in the past twelve used in ways that expose the very people they were designed
months, including over 60% of transgender and nonbinary to support. Such systems could also lead queer communities
youth. to be shut out from employment opportunities or to receive
higher health insurance premiums. Furthermore, reinforce-
Promise AI systems have the potential to help address the ment learning systems for behavioural interventions will
alarming prevalence of suicide in the queer community. Nat- present risks to patients unless many open problems in the
ural language processing could be leveraged to predict sui- field can be resolved, such as safe exploration (Hans et al.
cide risk based on traditional data sources (such as ques- 2008) and reward specification (Krakovna et al. 2020). The
tionnaires and recorded interactions with mental health sup- development of safe intervention systems that support themental health of the queer community is likely also contin- measured and recorded. Many widely available datasets thus
gent on furthering frameworks for sequential fairness (Hei- focus on immutable characteristics (such as ethnic group)
dari and Krause 2018), to fully account for challenges in or characteristics which are recorded and regulated by gov-
measuring and promoting queer ML fairness. ernments (such as legal gender, monetary income or profes-
sion).
Employment In contrast, characteristics like sexual orientation and gen-
Queer people often face discrimination both during the hir- der identity are frequently unobserved (Andrus et al. 2020;
ing process (resulting in reduced job opportunities) and once Crocker, Major, and Steele 1998; Jacobs and Wallach 2019).
hired and employed (interfering with engagement, devel- Multiple factors contribute to this lack of data. In some
opment and well-being; Sears and Mallory 2011). Non- cases, the plan for data collection fails to incorporate ques-
discrimination laws and practices have had a disparate im- tions on sexual orientation and gender identity—potentially
pact across different communities. Employment nondis- because the data collector did not consider or realise that
crimination acts in the United States have led to an aver- they are important attributes to record (Herek et al. 1991).
age increase in the hourly wages of gay men by 2.7% and a As a result, researchers may inherit datasets where assess-
decrease in employment of lesbian women by 1.7% (Burn ment of sexual orientation and gender identity is logistically
2018), suggesting that the impact of AI on employment excluded. In other situations, regardless of the surveyor’s
should be examined through an intersectional lens. intent, the collection of certain personal data may threaten
an individual’s privacy or their safety. Many countries have
Promise To effectively develop AI systems for hiring, re- legislation that actively discriminates against LGBTQ+ peo-
searchers must first attempt to formalise a model of the hir- ple (Human Rights Watch n.d.). Even in nations with hard-
ing process. Formalising such models may make it easier to won protections for the queer community, cultural bias per-
inspect current practices and identify opportunities for re- sists. To shield individuals from this bias and protect their
moving existing biases. Incorporating AI into employment privacy, governments may instate legal protections for sen-
decision processes could potentially prove beneficial if un- sitive data, including sexual orientation (European Commis-
biased systems are developed (Houser 2019), though this sion n.d.). As a result, such data may be ethically or legally
seems difficult at the present moment and carries serious precluded for researchers. Finally, as recognised by discur-
risks. sive theories of gender and sexuality, sexual orientation and
gender identity are fluid cultural constructs that may change
Risks Machine learning-based decision making systems over time and across social contexts (Butler 2011). Attempts
(e.g., candidate prioritisation systems) developed using his- to categorise, label, and record such information may be
torical data could assign lower scores to queer candidates, inherently ill-posed (Hamidi, Scheuerman, and Branham
purely based on historical biases. Prior research has demon- 2018). Thus, some characteristics are unobserved because
strated that resumes containing items associated with queer- they are fundamentally unmeasurable. These inconsisten-
ness are scored significantly lower by human graders than cies in awareness and measurability yield discrepancies and
the same resumes with such items removed (LeCroy and tension in how fairness is applied across different contexts
Rodefer 2019). These patterns can be trivially learned and (Bogen, Rieke, and Ahmed 2020).
reproduced by resume-parsing machine learning models. Race and ethnicity are not immune to these challenges.
A combination of tools aimed at social media scraping, Race and ethnicity may be subject to legal observability is-
linguistic analysis, and an analysis of interests and activi- sues in settings where race-based discrimination is a sen-
ties could indirectly infringe of candidates’ privacy by out- sitive issue (e.g., hiring). Additionally, the definition of
ing them to their prospective employers without their prior racial and ethnic groups has fluctuated across time and place
consent. The interest in these tools stems from the commu- (Hanna et al. 2020). This is exemplified by the construction
nity’s emphasis on big data approaches, not all of which will of Hispanic identity in the United States and its inclusion
have been scientifically verified from the perspective of im- on the National Census, as well as the exclusion of mul-
pact on marginalised groups. tiracial individuals from many censuses until relatively re-
Both hiring and subsequent employment are multi-stage cently (Mora 2014). Though we choose to focus our analysis
processes of considerable complexity, wherein technical AI on queer identity, we note that the observability and measur-
tools may be used across multiple stages. Researchers will ability of race are also important topics (e.g., Scheuerman
not design and develop truly fair AI systems by merely fo- et al. 2020).
cusing on metrics of subsystems in the process, abstracting
away the social context of their application and their inter- Areas for Future Research
dependence. It is instead necessary to see these as sociotech-
The field of algorithmic fairness in machine learning is
nical systems and evaluate them as such (Selbst et al. 2019).
rapidly expanding. To date, however, most studies have over-
looked the implications of their work for queer people. To
Sources of Unobserved Characteristics include sexual orientation and gender identity in fairness
Most algorithmic fairness studies have made progress research, it will be necessary to explore new technical ap-
because of their focus on observed characteristics— proaches and evaluative frameworks. To prevent the risk of
commonly, race and legal gender. To be included in train- AI systems harming the queer community—as well as other
ing or evaluation data for an algorithm, an attribute must be marginalised groups whose defining features are similarlyunobserved and unmeasurable—fairness research must be protected characteristics (e.g., zip code as a proxy for race).
expanded. This approach aims to achieve intersectional fairness by op-
timising group fairness between all computationally identi-
Expanding Fairness for Queer Identities fiable groups (Kearns et al. 2018; Kim, Reingold, and Roth-
Machine learning models cannot be considered fair unless blum 2018). Distributionally robust optimisation represents
they explicitly factor in and account for fairness towards the an alternative method for preventing disparity amplification,
LGBTQ+ community. To minimise the risks and harms to bounding the worst-case risk over groups with unknown
queer people worldwide and avoid contributing to ongoing group membership by optimising the worst-case risk over
erasures of queer identity, researchers must propose solu- an appropriate risk region (Hashimoto et al. 2018). These
tions that explicitly account for fairness with respect to the methods have helped establish a link between robustness and
queer community. fairness, and have drawn attention to the synergistic bene-
The intersectional nature of sexual orientation and gen- fits of considering the relationship between fairness and ML
der identity (Parent, DeBlaere, and Moradi 2013) emerges generalisation (Creager, Jacobsen, and Zemel 2020). Other
as a recurring theme in our discussions of online abuse, adversarial approaches have also been proposed for improv-
health and employment. These identities cannot be under- ing counterfactual fairness, and by operating in continuous
stood without incorporating notions of economic and racial settings, have been shown to be a better fit for protected char-
justice. Deployed AI systems may pose divergent risks to acteristics that are hard to enumerate (Grari, Lamprier, and
different queer subcommunities; AI risks may vary between Detyniecki 2020).
gay, bisexual, lesbian, transgender and other groups. It is Fairness mitigation methods have been shown to be vul-
therefore important to apply an appropriate level of granular- nerable to membership inference attacks where the infor-
ity to the analysis of fairness for algorithmic issues. Policies mation leak increases disproportionately for underprivileged
can simultaneously improve the position of certain queer subgroups (Chang and Shokri 2020). This further highlights
groups while adversely affecting others—highlighting the the tension between privacy and fairness, a common theme
need for an intersectional analysis of queer fairness. when considering the impact of AI systems of queer com-
Demographic parity has been the focus of numerous ML munities. It is important to recognise the need for fairness
fairness studies and seems to closely match people’s con- solutions to respect and maintain the privacy of queer in-
ceptions of fairness (Srivastava, Heidari, and Krause 2019). dividuals and to be implemented in a way that minimises
However, this idea is very hard to promote in the context the associated reidentifiability risks. Differentially private
of queer ML fairness. Substantial challenges are posed by fair machine learning (Jagielski et al. 2019) could poten-
the sensitivity of group membership information and its ab- tially provide such guarantees, simultaneously meeting the
sence from most research datasets, as well as the associated requirements of fairness, privacy and accuracy.
outing risks associated with attempts to automatically de- Putting a greater emphasis on model explainability may
rive such information from existing data (Bjarnadottir et al. prove crucial for ensuring ethical and fair AI applications
2019; Ehrenfeld et al. 2019; Young, Yu, and Wang 2017). in cases when fairness metrics are hard or impossible to re-
Consensually provided self-identification data, if and when liably compute for queer communities. Understanding how
available, may only capture a fraction of the community. AI systems operate may help identify harmful biases that
The resulting biased estimates of queer fairness may in- are likely to have adverse downstream consequences, even if
volve high levels of uncertainty (Ethayarajh 2020), though these consequences are hard to quantify accurately. Even in
it may be possible to utilise unlabeled data for tightening cases when queer fairness can be explicitly measured, there
the bounds (Ji, Smyth, and Steyvers 2020). While it is possi- is value in identifying which input features contribute the
ble to root the analysis in proxy groups (Gupta et al. 2018), most to unfair model outcomes (Begley et al. 2020), in order
there is a risk of incorporating harmful stereotypes in proxy to better inform mitigation strategies.
group definitions, potentially resulting in harms of represen- It is important to acknowledge the unquestionable cisnor-
tation (Abbasi et al. 2019). Consequently, most ML fairness mativity of sex and gender categories traditionally used in
solutions developed with a specific notion of demographic the AI research literature. The assumption of fixed, binary
parity in mind may be inappropriate for ensuring queer ML genders fails to include and properly account for non-binary
fairness. identities and trans people (Keyes 2018). Incorporating such
Individual (Dwork et al. 2012; Jung et al. 2019), counter- biases in the early stages of AI system design poses a sub-
factual (Kusner et al. 2017), and contrastive (Chakraborti, stantial risk of harm to queer people. Moving forward, more
Patra, and Noble 2020) fairness present alternative defini- attention should be directed to address this lacuna.
tions and measurement frameworks that may prove useful Creating more-equitable AI systems will prove impossi-
for improving ML fairness for queer communities. However, ble without listening to those who are at greatest risk. There-
more research is needed to overcome implementational chal- fore, it is crucial for the AI community to involve more
lenges for these frameworks and facilitate their adoption. queer voices in the development of AI systems, ML fair-
A small body of work aims to address fairness for pro- ness, and ethics research (Poulsen, Fosch-Villaronga, and
tected groups when the collection of protected attributes is Søraa 2020). For example, the inclusion of queer perspec-
legally precluded (e.g., by privacy and other regulation). Ad- tives might have prevented the development of natural lan-
versarially Re-weighted Learning (Lahoti et al. 2020) aims guage systems that inadvertently censor content which is
to address this issue by relying on measurable covariates of wrongly flagged as abusive or inappropriate simply due to itsadjacency to queer culture, such as in the example of scoring incorporate a large number of proxy features. This would
drag queen language as toxic. Researchers should make ef- be a marked divergence from the proposed measures in
forts to provide a safe space for LGBTQ+ individuals to ex- most published work. Counterfactual and contrastive fair-
press their opinions and share their experiences. Queer in AI ness metrics come with their own set of practical implemen-
workshops have recently been organised at the Neural Infor- tation challenges.
mation Processing Systems conference (Agnew et al. 2018; On the other hand, the approaches aimed at providing
Agnew, Bilenko, and Gontijo Lopes 2019) and the Interna- worst-case fairness guarantees for groups with unknown
tional Conference on Machine Learning (John et al. 2020), group membership (Hashimoto et al. 2018; Lahoti et al.
providing a valuable opportunity for queer AI researchers to 2020) apply by definition to any marginalised group. They
network in a safe environment and discuss research at the are also specifically tailored to address the situation of un-
intersection of AI and queer identity. observed protected characteristics. Therefore, fairness solu-
tions required to address queer ML fairness are likely to be
Fairness for Other Unobserved Characteristics applicable to other groups as well.
The queer community is not the only marginalised group Fairness challenges are institutionally and contextually
for which group membership may be unobserved (Crocker, grounded, and it is important to go beyond purely com-
Major, and Steele 1998). Religion, disability status, and putational approaches to fully assess the sociotechnical as-
class are additional examples where fairness is often chal- pects of the technology being deployed. The complexity of
lenged by observability (Kattari, Olzman, and Hanna 2018; these issues preclude any single group from tackling them
Sanchez and Schlossberg 2001). Critically, they may also in their entirety, and a resolution would ultimately require
benefit from developments or solutions within queer fairness an ecosystem involving a multitude of partnering organisa-
research. For example, in nations where individuals of cer- tions, jointly monitoring, measuring and reporting fairness
tain religious groups are persecuted or subjected to surveil- of such systems (Veale and Binns 2017).
lance, privacy is an essential prerequisite for safety. Persecu- These issues are only a small sample of the common chal-
tion targeting religious communities may also include cen- lenges faced by groups with typically unobserved charac-
sorship or manipulation of information (Cook 2017). Even teristics. We invite future work to explore the impact of AI
in nations where religious freedoms are legally protected, re- from the perspective of such groups. It is important to ac-
ligious minorities may be subjected to online abuse such as knowledge that people with different identities have distinct
hate speech or fear-mongering stereotypes (Awan 2014). experiences of marginalisation, stigmatisation and discrimi-
Although the nature of the discrimination is different, nation. However, recognising common patterns of injustice
people with disabilities are also a frequent target of deroga- will likely enable the development of techniques that can
tory language on the internet, and are more likely to be ha- transfer across communities and enhance fairness for multi-
rassed, stalked or trolled online, often to the detriment of ple groups. In this way, shared ethical and technical design
their mental health (Sherry 2019). Youth with disabilities principles for AI fairness will hopefully result in a more eq-
more frequently suffer from adverse mental health due to uitable future.
bullying, and people of all ages with physical disabilities
are at higher risk for depression (King et al. 2018; Turner
and Noh 1988). Therefore, individuals with disabilities may Conclusion
benefit from insights on the interaction of unobserved char-
acteristics and mental health. Lower-income and lower-class The queer community has surmounted numerous historical
individuals also suffer from worse mental health, particu- challenges and continues to resist oppression in physical and
larly in countries with high economic inequality (Liu and Ali digital spaces around the world. Advances in artificial intel-
2008). Fairness for class and socioeconomic status is also an ligence represent both a potential aid to this resistance and
important consideration for employment, where class bias in a risk of exacerbating existing inequalities. This risk should
hiring limits employee diversity and may prevent economic motivate researchers to design and develop AI systems with
mobility (Kraus et al. 2019). fairness for queer identities in mind. Systems that attempt
Any particular dataset or AI application may instanti- to label sexual orientation and gender identity, even for the
ate observability difficulties with respect to multiple demo- purpose of fairness, raise technical and ethical challenges
graphics. This may frequently be the case for disability sta- regarding observability and measurability.
tus and class, for example. Individual fairness—a set of ap- A new discourse on queer fairness has the potential to
proaches based on the notion of treating similar individu- identify moral and practical considerations shared across
als similarly (Dwork et al. 2012; Jung et al. 2019)—could queer communities, as well as concerns specific to partic-
potentially promote fairness across multiple demographics. ular subpopulations in particular places. By further develop-
These approaches entail a handful of challenges, however. ing techniques supporting fairness for unobserved character-
The unobserved group memberships cannot be incorporated istics, the machine learning community can support queer
in the similarity measure. As a result, the similarity mea- communities and other marginalised groups. Broadly, the
sure used for assessing individual fairness must be designed present work—surveying the ways in which AI may ame-
carefully. To optimise fairness across multiple demograph- liorate or exacerbate issues faced by queer communities—
ics and better capture the similarity between people on a emphasises the need for machine learning practitioners to
fine-grained level, similarity measures will likely need to design systems with fairness and dignity in mind.Disclaimer Beauchamp, T. L.; and Childress, J. F. 2001. Principles of
Any opinions presented in this paper represent the personal Biomedical Ethics. Oxford University Press, USA.
views of the authors and do not necessarily reflect the official Begley, T.; Schwedes, T.; Frye, C.; and Feige, I. 2020.
policies or positions of their organisations. Explainability for fair machine learning. arXiv preprint
arXiv:2010.07389 1–15.
References Bellenson, J. 2019. 122 Shades of Gray. https://www.
Abbasi, M.; Friedler, S. A.; Scheidegger, C.; and Venkata- geneplaza.com/app-store/72/preview. Accessed: 2020-09-
subramanian, S. 2019. Fairness in representation: Quantify- 10.
ing stereotyping as a representational harm. In Proceedings Bhattasali, N.; and Maiti, E. 2015. Machine ‘gaydar’: Using
of the 2019 SIAM International Conference on Data Mining, Facebook profiles to predict sexual orientation.
801–809. SIAM.
Binns, R. 2018. Fairness in machine learning: Lessons from
Abreu, R. L.; and Kenny, M. C. 2018. Cyberbullying and political philosophy. In Conference on Fairness, Account-
LGBTQ youth: A systematic literature review and recom- ability and Transparency, 149–159.
mendations for prevention and intervention. Journal of
Bisaso, K. R.; Anguzu, G. T.; Karungi, S. A.; Kiragga, A.;
Child & Adolescent Trauma 11(1): 81–97.
and Castelnuovo, B. 2017. A survey of machine learning
Agnew, W.; Bengio, S.; Keyes, O.; Mitchell, M.; Mitchell, applications in HIV clinical research and care. Computers
S.; Prabhakaran, V.; Vazquez, D.; and Gontijo Lopes, R. in Biology and Medicine 91: 366–371.
2018. Queer in AI 2018 Workshop. https://queerai.github.io/
Bisaso, K. R.; Karungi, S. A.; Kiragga, A.; Mukonzo, J. K.;
QueerInAI/QinAIatNeurIPS.html. Accessed: 2020-09-10.
and Castelnuovo, B. 2018. A comparative study of logis-
Agnew, W.; Bilenko, N.; and Gontijo Lopes, R. 2019. Queer tic regression based machine learning techniques for predic-
in AI 2019 Workshop. https://sites.google.com/corp/view/ tion of early virological suppression in antiretroviral initiat-
queer-in-ai/neurips-2019/. Accessed: 2020-09-10. ing HIV patients. BMC Medical Informatics and Decision
Al-Rubaie, M.; and Chang, J. M. 2019. Privacy-preserving Making 18(1): 1–10.
machine learning: Threats and solutions. IEEE Security & Bize, R.; Volkmar, E.; Berrut, S.; Medico, D.; Balthasar, H.;
Privacy 17(2): 49–58. Bodenmann, P.; and Makadon, H. J. 2011. Access to qual-
Amnesty International. n.d. Troll Patrol. https://decoders. ity primary care for LGBT people. Revue Medicale Suisse
amnesty.org/projects/troll-patrol. Accessed: 2020-09-10. 7(307): 1712–1717.
Amnesty International Canada. 2015. LGBTI Rights. https: Bjarnadottir, R. I.; Bockting, W.; Yoon, S.; and Dowding,
//www.amnesty.ca/our-work/issues/lgbti-rights. Accessed: D. W. 2019. Nurse documentation of sexual orientation and
2020-09-10. gender identity in home healthcare: A text mining study.
CIN: Computers, Informatics, Nursing 37(4): 213–221.
Andrus, M.; Spitzer, E.; Brown, J.; and Xiang, A. 2020.
‘What we can’t measure, we can’t understand’: Challenges Bogen, M.; Rieke, A.; and Ahmed, S. 2020. Awareness in
to demographic data procurement in the pursuit of fairness. practice: Tensions in access to sensitive attribute data for an-
arXiv preprint arXiv:2011.02282 1–12. tidiscrimination. In Proceedings of the 2020 Conference on
Fairness, Accountability, and Transparency, 492–500.
Angwin, J.; Larson, J.; Mattu, S.; and Kirchner, L. 2016.
Machine bias: There’s software used across the coun- Bolukbasi, T.; Chang, K.-W.; Zou, J. Y.; Saligrama, V.; and
try to predict future criminals. And it’s biased against Kalai, A. T. 2016. Man is to computer programmer as
blacks. https://www.propublica.org/article/machine-bias- woman is to homemaker? Debiasing word embeddings. In
risk-assessments-in-criminal-sentencing. Advances in Neural Information Processing Systems, 4349–
4357.
Ashley, F. 2019. Gatekeeping hormone replacement therapy
Bonawitz, K.; Ivanov, V.; Kreuter, B.; Marcedone, A.;
for transgender patients is dehumanising. Journal of Medi-
McMahan, H. B.; Patel, S.; Ramage, D.; Segal, A.; and
cal Ethics 45(7): 480–482.
Seth, K. 2017. Practical secure aggregation for privacy-
Awan, I. 2014. Islamophobia and Twitter: A typology of on- preserving machine learning. In Proceedings of the 2017
line hate against Muslims on social media. Policy & Internet ACM SIGSAC Conference on Computer and Communica-
6(2): 133–150. tions Security, 1175–1191.
Balsam, K. F.; Molina, Y.; Beadnell, B.; Simoni, J.; and Wal- Bosia, M. J.; McEvoy, S. M.; and Rahman, M. 2020. The
ters, K. 2011. Measuring multiple minority stress: the LGBT Oxford Handbook of Global LGBT and Sexual Diversity Pol-
People of Color Microaggressions Scale. Cultural Diversity itics. Oxford University Press.
and Ethnic Minority Psychology 17(2): 163. Bowleg, L. 2020. We’re not all in this together: On COVID-
Bamman, D.; O’Connor, B.; and Smith, N. 2012. Censor- 19, intersectionality, and structural inequality. American
ship and deletion practices in Chinese social media. First Journal of Public Health 110(7): 917.
Monday . Brook. n.d. Gender: A few definitions. https://
Barocas, S.; Hardt, M.; and Narayanan, A. 2019. Fairness www.brook.org.uk/your-life/gender-a-few-definitions/. Ac-
and Machine Learning. http://www.fairmlbook.org. cessed: 2020-10-01.You can also read
