Does Propaganda Unconsciously Persuade? Testing Implicit Persuasion in the U.S. and Hong Kong - Rory Truex
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Does Propaganda Unconsciously Persuade?
Testing Implicit Persuasion in the U.S. and Hong Kong
[This draft: February 13, 2020]
Kevin Arceneaux† Rory Truex‡
Abstract
Propaganda is the dissemination of biased or misleading information in favor of a
particular political cause, party, or actor. There remains substantial debate as to
how propaganda works, and whether exposure actually persuades target audiences.
The paper evaluates a specific form of propaganda– self-aggrandizement– whereby
political actors inflate their own abilities and achievements. Drawing on prevailing
theories of persuasion, we hypothesize that this form of propaganda works through
nondeliberative cognition and is effective at creating associations between the actor
and positive attributes. A series of experiments in the U.S. and Hong Kong provide
little evidence in favor of this “implicit persuasion” argument. We conclude by
discussing the merits of an Open Science approach for social science research and
the implications of these findings for extant theories of persuasion.
Keywords: propaganda, implicit attitudes, persuasion, Donald J. Trump, Chinese
Communist Party
†
Thomas J. Freaney, Jr. Professor of Political Science. kevin.arceneaux@temple.edu.
‡
Assistant Professor of Politics and International Affairs, Princeton University. rtruex@princeton.edu.
This material is based upon work supported by the Department of Politics, the Woodrow Wilson School
of Public and International Affairs, and the University Center for Human Values at Princeton Univer-
sity. Thanks to Christopher Achen, Greg Distelhorst, Alisha Holland, Daniel Kahneman, Jennifer Pan,
Margaret Roberts, Eldar Shafir, Yiqing Xu, and Yuhua Wang for helpful comments at various stages of
the project.1
Power is in tearing human minds to pieces and putting them together again in new shapes
of your own choosing.
- George Orwell, 1984
1 Introduction
Propaganda is the dissemination of biased or misleading information in favor of a particular
political cause, party, or actor. Although existing research focuses on authoritarian systems
where the state has control over news media (Huang 2018), propaganda also exists in democratic
systems, where political actors often “spin” reality through press releases and advertisements in
ways that best suit their political goals (Farnsworth 2015).1 Joseph Goebels, the Nazi Germany
Minister of Propaganda, operated under the assumption that “propaganda becomes ineffective
the moment we are aware of it” (Sennett 2014). Central to this assumption is the idea that
propaganda works on an implicit level — that it persuades people outside of their conscious
awareness. It is this notion that we investigate here.
Existing research yields mixed results on the question of whether propaganda is in fact effec-
tive at shifting people’s attitudes. Some studies find that biased messages persuade their targets
to support the sponsor of the message (Peisakhin and Rozenas 2018; Yanagizawa-Drott 2014),
while others show that they burnish the perception that the sponsor is powerful (Huang 2015;
Wedeen 2015). In each of these accounts, individuals are described as consciously processing the
piece of propaganda and deciding whether to believe it or not. Yet whether a citizen consciously
agrees with a propaganda message depends on the citizen, the context, and the nature of the
message itself. This project aims to shift the focus of the scholarly debate about the effective-
ness of propaganda to consider the role of nondeliberative cognition. Drawing on research in
neuropsychology, we hypothesize that propaganda works not because it convinces citizens of the
arguments it contains —though sometimes it might. It works because it shifts people’s implicit
attitudes, or the positive and negative associations that they connect to political actors at a
largely subconscious level (Greenwald et al. 2002). By repeatedly pairing causes, parties, and
actors with positive attributes and accomplishments, propaganda should theoretically create
positive implicit attitudes, which in turn should inform explicit attitudes and behavior (Green-
wald et al. 2009; Ryan 2017). More important, this dynamic should unfold regardless of whether
a person disagrees with the propaganda message or knows it to be false.
We test this implicit persuasion hypothesis through a series of three experiments in the
1
Some political scientists further make a distinction between “hard” and “soft” propaganda, the former
being particularly “crude and heavy handed” and the latter “subtle and sleek” Huang (2018).2
U.S. and a conceptual replication of these experiments in Hong Kong. In the U.S. experiments,
respondents that received the propaganda treatment viewed a short video clip of Donald J.
Trump engaging in self-aggrandizing behavior, repeatedly pairing himself with positive attributes
like “best,” “incredible,” and so forth. All respondents then completed a short Single Category
Implicit Association Test (SCIAT) that measured implicit attitudes towards Trump (Karpinski
and Steinman 2006; Truex and Tavana 2019). This design was replicated three times over the
span of two years. In the China experiment, respondents in Hong Kong were randomly assigned
to view a short Chinese Communist Party (CCP) propaganda video, “Who Am I?,” which
highlights several virtues of the regime. They then completed a similar SCIAT that measured
implicit attitudes towards the CCP.
Overall, our results provide evidence against the hypothesis that implicit attitudes shift in
response to short term propaganda exposure. They also illustrate the value of embracing an
Open Science approach. The results of the first study conducted in the U.S. in 2017 were in
line with the implicit persuasion hypothesis. Respondents exposed to the Trump propaganda
treatment had more positive implicit attitudes towards Trump, and this effect was strongest for
Democrats. But, this was an exploratory study, so we decided to preregister our hypotheses (OSF
link [omitted for review]), and this finding did not replicate in the other two U.S. studies. We also
did not find a similar effect in the Hong Kong study. This sequence to the results underscores
the importance of replication (Dunning 2016; King 1995; Simons 2014) and preregistration for
predictive studies (Chambers 2019). Had we stopped at the first study, the paper would make
a very different argument, and featured a more dramatic finding, yet this would have been
the wrong scientific approach to take. As a result, our narrative is not a clean one, but we
agree with the editors for Nature Human Behaviour, who write, “The pressure to produce such
clean narratives, however, represents a significant threat to validity and runs counter to the
reality of what science looks like” (Tell it like it is 2020). In the Discussion section, we consider
possible reasons for our repeated null results and suggest directions for additional research on
nondeliberative persuasion and propaganda.
2 Implicit Persuasion
Political propaganda attempts to persuade people to change their attitudes, positively or neg-
atively, toward some political entity or personality. Theories of propaganda tend to presume
that the process by which people change their attitudes is a conscious one. Initial research on
the persuasive effects of propaganda focused on the degree to which messages changed people’s3 consciously held beliefs (Hovland, Lumsdaine and Sheffield 1949). Yet beliefs can be difficult to change, and subsequent research found that political messages can also change attitudes by sim- ply causing individuals to weight some considerations more heavily than others (Nelson, Oxley and Clawson 1997). Chong and Druckman (2007) formalize the process by which political messages change the weight that people attach to attitude-relevant considerations. First, a message makes a particular consideration (e.g., “the president is the smartest and most capable leader”) available in people’s memories for retrieval when they are asked to form an opinion (Eagly and Chaiken 1993). In their model, however, available considerations can only influence people’s attitudes to the extent they are consciously accessible in people’s minds, with those that are at the top of people’s minds being more influential over the attitudes that they express (Fazio 1995; Zaller et al. 1992). Finally, when political messages lead individuals to consciously evaluate considerations to form an opinion, people deliberate about the applicability of the accessible considerations at the top of their mind to arrive at an attitude (e.g., is intelligence a useful trait for a president to have?). If attitude change only arises through conscious deliberation, heavy handed propaganda should be unlikely to change many minds, because people can see through it. Intriguingly, though, Chong and Druckman (2007) note that individuals need to be sufficiently motivated to consciously weigh the applicability of considerations and that available and accessible con- siderations can influence people’s attitudes without engaging in deliberative thought (Higgins 1996). Existing research on implicit attitudes suggests how. An attitude is an evaluation of an object– a person, person, place, idea, and so forth. An explicit attitude is one that a person is consciously aware of and can report (e.g., “I like flowers”; “I dislike insects”). An implicit atti- tude reflects the constellation of positive and negative attributes that people connect to attitude objects (e.g., flowers are pretty, smell good, cause allergies, etc.). The degree to which people are consciously aware of the attributes they connect to attitude objects varies across individ- uals and across objects. As a result, people’s implicit attitudes need not track perfectly with their explicit attitudes. In the United States, for instance, researchers have found a disconnect between explicit and implicit attitudes that white individuals express and hold with respect to African Americans (Greenwald et al. 2009). Many white citizens express positive attitudes toward African Americans, while harboring negative stereotypes on a nonconscious level. Previous research suggests that implicit attitudes are malleable and can be readily shifted by brief stimuli. For instance, Foroni and Mayr (2005) ask research participants to consider a
4
fictional world, where due to radiation effects, flowers are poisonous, and insects are a critical
source of food. This simple scenario was enough to improve their subjects’ implicit preferences
for insects, and reduce their implicit preference for flowers. A meta-analysis of 492 studies finds
that various approaches to shifting implicit attitudes can effectively do so (Forscher et al. 2019).
Studies of the effect of direct persuasive appeals on implicit attitudes focus predominately on
consumer goods. Of the seven studies that touch on politics, one considered the effect of an
indirect appeal on implicit attitudes, and the findings were somewhat complex. Among a small
college student sample, religious appeals in speeches increased implicit support for Bill Clinton
and George W. Bush, but only among participants who identified as Christians (Albertson 2011).
Taken together, these findings suggest that propagandistic messages might also shift people’s
implicit attitudes. We are unaware of empirical research testing this question, but theorists have
suggested that propaganda influences individuals outside of their conscious awareness (Ellul
1966; Stanley 2015). In his study of Nazi propaganda, Klemperer (1998) — a German man of
Jewish faith who survived the Third Reich — described a scene of young people discussing the
concept of “heroism” just after the war. He observed “. . . how the young people in all innocence,
and despite a sincere effort to fill in the gaps and eliminate the errors in their neglected education,
cling to Nazi Thought processes. They don’t realize they are doing it” (quoted in Stanley, 2015,
2).
This study extends the study of implicit attitudes to the study of political propaganda, which
is a domain where citizens have more developed explicit attitudes and generally show resistance
to counter-attitudinal information (Geddes and Zaller 1989, Lodge and Taber 2013). The treat-
ments and experiments here focus on a form of “hard” propaganda — self-aggrandizement —
whereby an actor inflates his accomplishments and abilities. In Russia, President Vladimir Putin
regularly plays a pop-song anthem at his rallies, “A Man Like Putin,” where two women lament
that their boyfriends are not enough like him (Cassiday and Johnson 2010):
One like Putin, full of strength.
One like Putin, who won’t be a drunk.
One like Putin, who wouldn’t hurt me.
One like Putin, who won’t run away!
Or consider a typical passage from Ideological and Moral Cultivation and the Fundamentals of
Law, a required political education textbook for all Chinese university students (p. 57):5
“Without the Communist Party, there can be no New China.” This is the truth
clearly demonstrated by China’s history and reality. The history of the Commu-
nist Party of China is a history of striving for national independence and people’s
liberation, and for the great rejuvenation of the Chinese nation.... You should not
only understand the glorious history of the Communist Party of China, but also the
wisdom and greatness of the Communist Party of China, and appreciate the spirit
of the Communist Party of China.
Here the CCP is describing its own “wisdom” and “greatness,” and claiming a “glorious history”
that will lead to the “rejuvenation” of the Chinese people. The propaganda text draws linguistic
links between the regime and its virtues and achievements. This strategy is quite common in
authoritarian systems, as dictators seek to foster cults of personality and devout followings in
the population (Wedeen 2015). Democratic leaders can also engage in this sort of behavior,
though at greater risk of being exposed by opposition leaders or the media.
The hypothesis evaluated here is that propaganda can shift attitudes in the intended di-
rection, even among hostile targets. The process through which this occurs is automatic, af-
fective, and non-deliberative– the workings of implicit/associational thinking rather than ex-
plicit/deliberative thinking (Lodge and Taber 2013; Ryan 2017). Propaganda works by rein-
forcing associations in memory between the subject and positive attributes.
H1 : Citizens exposed to self-aggrandizing propaganda will display more positive
implicit attitudes towards the target actor.
H2 : The positive effect of self-aggrandizing propaganda on implicit attitudes will
be largest for citizens that are most critical of the target actor.
Combined, these hypotheses suggest a cognitive model where citizens are passive recipients
of propaganda messages. Propaganda infiltrates the subconscious mind, even if the recipient
deliberatively rejects its validity. A propaganda treatment should have the greatest effect on
citizens that are the most critical of the politician or government in question. Citizens that
are already supportive of the actor likely already have positive implicit attitudes, meaning that
additional propaganda exposure is unlikely to create new cognitive links.
3 Research Overview
The four experiments presented here measure the effect of self-aggrandizing propaganda on
implicit attitudes towards Donald J. Trump (among a sample of American respondents) and6
the Chinese Communist Party (among a sample of Hong Kong residents). Both Trump and
the CCP routinely engage in misleading self-promotion, albeit in different forms and political
contexts. We present the research in the sequence it was conducted.
Study 1: U.S.-Trump-IAT (7/2017)
Research Design
The first experiment was conducted in July 2017 among a sample of about 500 American respon-
dents solicited through the Prolific platform. Respondents first answered a series of demographic
and attitudinal questions before being randomly assigned (50/50) to the treatment or control
conditions. Respondents in the treatment condition viewed a one minute video with a montage
of clips of Donald J. Trump boasting about himself on the campaign trail. Respondents in the
control condition viewed nothing. The text of the treatment video is below:
I went to an Ivy League School. I’m very highly educated. I know words. I have the
best words. [break] I have the best people. [break] Even the really dishonest press
says Trump’s people are the most incredible. [break] When it comes to great steaks,
I’ve just raised the stakes. Trump steaks are the world’s greatest steaks and I mean
that in every sense of the word. [break] The thing they most want, you know what
they want? The top ten things? Anything Trump. You believe it? My carpets, my
ties... They love me. They love me! [break] So just to finish off, for the polls, they
said best in the military, and foreign policy... Lot of people think I’ll be the best in
jobs by far. I will be the greatest jobs President that God ever created... Best in the
economy... By far, best in leadership. [break] I will be the greatest jobs President
that God ever created.
Trump’s rhetorical strategy links himself to a number of achievements and positive attributes,
words like “best,” “incredible”, “love”, “leadership”, and even “God.”
Following this treatment, respondents completed a Single Category Implicit Association Test
(SCIAT), which involved sorting images of Donald J. Trump and positive and negative words
into categories as quickly as possible. The standard IAT procedure has two target attitude
objects (white vs. black, old vs. young, Pepsi vs. Coke, etc.) and measures their differential
association with a single attribute (Greenwald, McGhee and Schwartz 1998). The resulting
measure places respondents on a bipolar scale– i.e. an implicit bias against old people relative
to young people. However, many attitude objects do not have natural reference points, which
makes a standard IAT inappropriate. In response to this issue, Karpinski and Steinman (2006)
developed the Single Category IAT, which can measure the strength of associations for a single7
attitude object. This study implements their basic procedure using Trump as the attitude object
of interest, similar to the protocol in Truex and Tavana (2019).
Table 1: U.S.-Trump-IAT Block Ordering
Block Trials Function Left-key Right-key
1 20 Practice Good Bad
2a 48 Practice Good + Trump Bad
3a 48 Test Good + Trump Bad
4b 48 Practice Good Bad + Trump
5b 48 Test Good Bad + Trump
Note: Blocks with a common subscript experienced as one contin-
uous block. Table amended from Karpinski and Steinman (2006).
Table 1 provides an overview of the IAT. Each individual item presented is considered a single
trial. In the second two blocks of trials (one practice with 48 and test of 48), respondents place
“good” words and Trump images in the same group by pressing the “E” key on their keyboards.
“Bad” words are sorted into a separate category using the “I” key. In the second set of the
trials, “bad” and Trump images are sorted into the same “I” group, and “good” is sorted by
itself using the “E” key. As is standard practice, the order was reversed for half the participants
(4b and 5b administered before 2a and 3a ) to avoid biases induced by fatigue, learning, and so
forth. Respondents were told to complete each sorting trial as quickly as possible. Respondents
that pressed the wrong key (an error response) saw a large red “X” and had to click on the
correct answer before proceeding to the next trial.
The GorillaTM platform records the time (in milliseconds) it takes a respondent to complete
each trial, which in aggregate provides a measure of her implicit association between the target
object (Trump) and the different word sets (“good” and “bad”). The primary output is the
standardized difference in average reaction times across the two test blocks. This project will
use the SCIAT formula employed by Karpinski and Steinman (2006), adapted from Greenwald,
Nosek and Banaji (2003):
X̄iB − X̄iG
trump.dscorei = (d-score)
SDXi
Here, the “d-score” for individual i, trump.dscorei , is calculated by subtracting the mean re-
action time for all non-practice trials with the Trump-Good grouping (Block 3a ) X̄iG from the
mean reaction time for the non-practice trials with the Trump-Bad grouping (Block 5b ) X̄iB ,
and dividing by the standard deviation for all response times within both blocks SDXi . Large8
d-scores indicate greater positive implicit associations with Trump.
The experiment also contained measures of explicit attitudes (trump.approval ) and future
voting intentions (trump.vote). The protocol was administered to 500 American respondents
on July 15, 2017 through the Prolific participant recruiting platform. More information on the
sample, questionnaire, and randomization is included in the Appendix.
Results
The core results of the U.S.-Trump-IAT experiment are shown in Table 2. The table shows the
estimates of the ATE for the propaganda treatment across three different dependent variables.
The variable of primary interest is the Trump SCIAT d-score (trump.dscore), where positive
values indicate positive implicit attitudes towards Donald J. Trump. We observe a statistically
and substantively significant treatment effect for the full sample.
The table assesses H2 and the possibility of a heterogenous treatment effect by disaggre-
gating the results by participants’ vote in the 2016 election. The effect of the the Trump self-
aggrandizing video clip proves isolated to individuals who did not vote for him in the election. In
isolation, this result suggests that even individuals that hold negative explicit attitudes towards
a subject can have their implicit attitudes shifted in a positive direction through propaganda.
Table 2: Results from Study 1: U.S.-Trump-IAT (7/2017)
Outcome (Y ) Full Sample (n=544) Trump Voters (n=134) ! Trump Voters (n=410)
Y¯T Y¯C ATE p Y¯T Y¯C CATE p Y¯T Y¯C CATE p
trump.dscore 0.008 -0.113 0.121 0.001 0.126 0.201 -0.075 0.856 -0.043 -0.195 0.151 0.000
trump.approval 1.909 1.642 0.267 0.002 3.149 3.019 0.130 0.185 1.399 1.279 0.120 0.040
trump.2020 0.211 0.115 0.096 0.002 0.676 0.472 0.205 0.012 0.024 0.024 -0.000 0.983
Note: Table shows the difference of means for three dependent variables: implicit attitudes
(trump.dscore), explicit attitudes (trump.approval), and voter intention (trump.2020). Results are
shown for the full sample and subgroup analysis for Trump voters and respondents that did not vote
for Trump.
The null effect of the treatment for Trump supporters may be due to a ceiling effect.
Such individuals have very positive implicit attitudes towards Trump to begin with (mean
of trump.dscore = .144), and they have higher exposure to news outlets and messages that9
highlight his accomplishments.2 The short video clip treatment likely did not represent a novel
stimulus, nor was there as much “room” to move the IAT outcome measure.
The treatment in Study 1 also had a significant effect on explicit attitudes (trump.approval )
and vote intention (trump.2020 ). About 21.1% of respondents in the treatment group expressed
the intention to vote for Trump, compared to 11.5% of respondents in the control condition.
This effect appears to be primarily driven by Trump voters– the opposite of what we observed
with the implicit measure.3
Because Study 1 was an exploratory study, we cannot take these p-values at face value
(Nosek et al. 2018). Therefore, we sought to replicate these findings in predictive studies. We
report these results next.
Study 2: Hong Kong-CCP-IAT (2/2018)
Research Design
The results from Study 1 provide evidence in favor of the “implicit persuasion” hypothesis.
American respondents randomly exposed to a short video of Donald J. Trump boasting about
his accomplishments had more positive implicit associations towards him, and this effect was
strongest for citizens who did not support him in the 2016 election. At the time, we found
this to be a compelling and exciting result, and we wanted to see if the argument traveled to a
completely different social and political context. We decided to conduct a conceptual replication
in China, given the pervasiveness of self-aggrandizing propaganda and the scholarly interest in
that subject (Huang 2015, 2018; Stockmann and Gallagher 2011).
The China version of the experiment exposed a set of Chinese respondents to a short pro-
paganda clip recently promoted by the Chinese Communist Party (CCP). The advertisement,
titled “Who Am I?”, was created and disseminated to celebrate the 95th Anniversary of the
CCP in 2016. The ad shows a series of Chinese citizens hard at work at different professions
(teacher, doctor, custodian, etc.), with a voiceover describing their attributes:
Who am I? What kind of person am I? Maybe you’ve never even thought about it.
2
Viewership of FOX News programming proves highly positively correlated with implicit attitudes towards
Donald J. Trump.
3
One concern with Study 1 is that treatment assignment appears to be correlated with a number of im-
portant covariates, despite randomization through the GorillaTM platform. We observe that individuals
that received the treatment are more likely to be Trump voters, to have lower education, to be married,
and to be Republican. Such variables are obvious correlates of implicit and explicit attitudes, and thus
threaten the inference. The findings throughout are robust to the inclusion of covariates (f emale, age,
lowed, race.black, race.hispanic, race.asian, income.high, republican, trump.vote2016) to account for
the slight imbalances in treatment assignment.10
I am the one who leaves last. I’m the one who is there working the earliest. I’m the
one who does the least for myself. I am the one who follows the rules until the end.
I’m the one who is the first to act on things. I’m the one who is most concerned
about others. I am the Chinese Communist Party (CCP). I will be with you forever.
It closes with a group shot of the citizens, identifying them as members of the CCP. The final
screen marks the 95th Anniversary of the CCP.
The content of this CCP propaganda advertisement is quite different from the montage from
the Trump 2016 campaign. The general message is that CCP members are common people who
are particularly virtuous– hardworking, selfless, law abiding, compassionate. This is typical of
political discourse in contemporary China.
As with the Trump-IAT, respondents were randomly assigned to view the one minute pro-
paganda video treatment or the control condition, where no video was shown. After the ma-
nipulation, respondents completed a Single Category IAT (SCIAT) measuring attitudes towards
the CCP, with the design and scoring equivalent to that described in the previous section.4
Respondents also gave their explicit assessments of the CCP (ccp.trust) and the desirability
of independence for Hong Kong (independence). All question wordings and translations are
available in the Appendix.
Because of political sensitivities, it was not possible nor ethical to attempt this survey in
mainland China. Surveys conducted by foreigners on social topics formally require government
oversight, and it is unlikely a survey measuring implicit attitudes towards the CCP itself would
receive approval given the current political climate under Xi Jinping. For this reason, the survey
was administered online to residents of Hong Kong, to a sample of about 1500 respondents
solicited through Qualtrics. Hong Kong is under the sovereign control of the government in
mainland China but retains a more open political climate under the “one country, two systems”
policy. It is common for Hong Kong residents to be surveyed on their political attitudes, and
political discourse there operates largely unencumbered, or at least it did at the time of the
administration of the survey (February 2018).
From a research design perspective, the sample in the Hong Kong-CCP-IAT represents an-
other “least likely” population for the treatment to have its hypothesized effects (Gerring 2007).
Hong Kong residents have access to the full range of political discourse on China, and they are
4
It is harder to identify images of the CCP, as images might contain different leaders or symbols that
would activate other concepts in memory. To avoid this issue, the characters for “Chinese Communist
Party” were used instead of images. This sort of text-text IAT is also standard in the field.11
routinely exposed to counter-CCP arguments that would make them more aware of the propa-
ganda state than the average Chinese citizen in the mainland. We also know from the recent
pro-democracy protests– which occurred over a year after our survey– that latent support for
the central government in Beijing is quite low.
Results
The Hong Kong results raise doubts about the validity of the implicit persuasion dynamic.
The CCP propaganda video had a small positive effect on implicit attitudes towards the Party
(ccp.dscore), but this effect did not reach conventional levels of significance. The Bayes’ Factor in
direction of the null hypothesis is 9.525, which offers “moderate” evidence for the null hypothesis
(Schönbrodt and Wagenmakers 2018).5 The magnitude of the effect is also substantially smaller
than what we observed in Study 1 (0.121 vs. 0.018). Table 3 shows these results, as well the
effects disaggregated by respondents who reported a pro-democracy/localist political orientation,
and those who did not. In line with H2 , we observe the effect on the d-score is slightly larger
for citizens that are hostile to the target actor, but again this effect is not significant.
Table 3: Results from Study 2: Hong Kong-CCP-IAT (2/2018)
Outcome (Y ) Full Sample (n=1534) Pro-democracy (n=709) ! Pro-democracy (n=824)
Y¯T Y¯C ATE p Y¯T Y¯C CATE p Y¯T Y¯C CATE p
ccp.dscore -0.029 -0.046 0.018 0.166 -0.085 -0.108 0.024 0.196 0.011 0.007 0.004 0.427
ccptrust 3.584 3.224 0.359 0.004 2.757 2.178 0.578 0.000 4.314 4.253 0.061 0.359
independence 2.449 2.437 0.012 0.429 2.801 2.885 -0.083 0.807 2.144 1.991 0.153 0.046
Note: Table shows the difference of means for three dependent variables: implicit attitudes
(ccp.dscore) and explicit attitudes (ccptrust) and (independence). Results are shown for the full
sample and subgroup analysis for respondents who report a pro-democracy/localist political orienta-
tion and those who did not.
Interestingly, the propaganda treatment did appear to move explicit attitudes. Respondents
in the treatment group reported significantly greater trust in the CCP, and this effect was indeed
largest for citizens reporting a pro-democracy/localist political orientation.
5
We calculated the Bayes’ Factor using JASP version 0.9.2 for an independent sample t-test. We specified
uninformative priors via the Cauchy distribution with a prior width=0.71. Reasonable variation in the
prior width (i.e., from 0.25 to 1.5) yield Bayes’ Factors in direction of the null hypothesis that range
from 3 to 19.12
Study 3: U.S.-Trump-IAT (6/2019) and Study 4: U.S.-Trump-IAT (8/2019)
Research Design
At this stage in the research process, there were two primary possibilities for the set of results we
observed. The first is that the implicit persuasion dynamics do not generalize across all political
contexts, and that there was some meaningful difference in the information environment and
experimental design in the U.S. and Hong Kong experiments that produced a significant result
in Study 1 and a null result in Study 2. The second is that the effects found in our original
study of Donald J. Trump were the product of Type I error (Maxwell, Lau and Howard 2015).
The sampling draw produced data that rejected the null hypothesis of no effect, even though
the null hypothesis is true, and our theory of implicit persuasion is wrong.
To adjudicate these possibilities, we replicated the research design from Study 1 on two
new U.S. samples, one sample of 500 respondents solicited through Prolific (Study 3), and a
second more representative sample of 1000 respondents solicited through Qualtrics (Study 4).
The treatment, IAT protocol, and other questions remained the same as in Study 1. Study 3
was conducted in June 2019, and Study 4 was conducted in August 2019.
Figure 1: Effect of Propaganda Treatment on Trump IAT D-score
Outcome: Implicit Attitudes (dscore)
Study 1 − U.S.−Trump (7/2017) Study 3 − U.S.−Trump (6/2019) Study 4 − U.S.−Trump (8/2019)
Estimate of Propaganda Treatment Effect
0.2
0.1
Covariates
0.0
None
−0.1
−0.2
Full Not Trump Trump Full Not Trump Trump Full Not Trump Trump
Sample Supporters Supporters Sample Supporters Supporters Sample Supporters Supporters
Note: Figure shows estimates of ATE for three U.S. studies of the Trump propaganda treatment. The
outcome variable is the Trump IAT score (trump.dscore). Each panel shows results for the full sample and
then explores subgroup effects for Trump supporters and respondents who did not vote for him in 2016.
Results13
Figure 1 presents the results of Studies 3 and 4 as compared to our initial study from 2017. The
figures show the ATE from the propaganda treatment on the full sample, and then the CATEs
for the two subgroups of interest: people who voted for Trump in 2016 and people who did not.
The figure also shows the estimates with and without covariate adjustment. We see quickly that
the core results from Study 1 do not replicate. In Study 3, the propaganda treatment had a
small but insignificant effect in the full sample (Bayes’ Factor for null hypothesis = 9.3), and
in Study 4 the effect is negative but insignificant (Bayes’ Factor for null hypothesis = 4.1).6
Subgroup analyses also do not suggest any meaningful differences. Figures A1a and A1b in the
Appendix show the results using the trump.approval and trump.vote2020 outcome variables,
where the treatment effects were also insignificant in Studies 3 and 4.
Note that the three U.S. studies do not constitute a perfect replication because they occur at
different points in time. This is the downside of studying a political leader that is still in power.
One possible explanation for our findings is that Trump’s declining political fortunes from 2017
through 2019 made the propaganda treatment resonate less with the population. This strikes
us as unlikely, primarily because by the time we initiated Study 1, Trump had been in office for
several months, and his approval ratings had already dropped to the levels they have hovered
around for the last two years of his presidency.
4 Discussion
After three years of research and several replications, this project has yielded a somewhat un-
satisfying result. Our initial theoretical intuition was that propaganda works through a non-
deliberative mechanism, shifting implicit attitudes through the repeated pairing of an actor
and positive attributes. Though initial results confirmed our hypotheses, several conceptual
replications suggest that the effect is not robust or reliably observable across contexts.
The sequence of our research and findings underlines the importance of replication in the
study of political science and psychology (Dunning 2016; King 1995; Simons 2014). It also illus-
trates what happens with an Open Science approach. It necessarily opens the possibility that the
research we produce will not generate a “clean narrative” in which study predictions always turn
out to be supported by the evidence. A compelling audit study of the “file drawer” in the social
sciences shows that preregistered research that does not yield positive evidence for predictive
6
As with Study 2, we used JASP version 0.9.2 to calculate the Bayes’ Factors and we used the Cauchy
distribution to specify an uninformative prior with a width = 0.71. Reasonable variation in the prior
width (0.25 to 1.5) does not change the interpretation of the evidence in the direction of the null
hypothesis.14 hypotheses is less likely to be published (Franco, Malhotra and Simonovits 2014). As Chambers (2019) illustrates in the domain of psychological research and the editors of Nature Human Be- haviour recently criticized (Tell it like it is 2020), the pressure to write papers with clean results undermines good scientific practice. It encourages researchers to present exploratory findings as predictive hypotheses and to ignore studies that “did not work.” We resist the temptation to do so here, because we want the scientific record to be clear even if our narrative is less so. Despite initial evidence for an intriguing hypothesis, the trail of evidence suggests that something less splashy is at work. It can be difficult to make sense of this sort of null result, and there are a range of explanations for the pattern of findings we observed. Perhaps if we had chosen a set of actors other than Trump and the CCP, the experiments would have produced large, significant treatment effects. Or perhaps we should have used different treatments? Or tried somehow to simulate long term propaganda exposure rather than use short video clips? Or done the study in a lab rather than online? Or used a different measure of implicit attitudes? It is possible the implicit persuasion hypothesis holds, we have just not found evidence for it because of the features of our research design and case selection. The only way to adjudicate that possibility is further replication, a task we leave for future research. The findings in this set of studies do lead us to question our initial theoretical intuition, that propaganda messages could easily penetrate the subconscious mind even for citizens who hold explicit attitudes hostile to the sponsor. The set of null results suggest the opposite might be true. Citizens may be able to resist propaganda and indoctrination if they engage in effortful thinking and deliberation, or if they already have well formed explicit attitudes. The subconscious linkages fostered by propaganda language may fail to form if the citizen consciously rejects those messages as false. In doing so, our findings place boundary conditions on extant theories of attitude formation in political psychology that place a great deal of emphasis on the influence that nonconscious and automatic process exert on political attitudes. For instance, (Lodge and Taber, 2013) argue that “. . . conscious deliberation is the wake behind the boat, while automatically stimulated affective and cognitive processes control the rudder” (p. 48). Perhaps this statement is true for many political attitudes, but it is not true in the instance that we study here.
15 References Albertson, Bethany L. 2011. “Religious Appeals and Implicit Attitudes.” Political Psychology 32(1):109–130. Cassiday, Julie A and Emily D Johnson. 2010. “Putin, Putiniana and the question of a post- Soviet cult of personality.” Slavonic and East European Review pp. 681–707. Chambers, Chris. 2019. The seven deadly sins of psychology: A manifesto for reforming the culture of scientific practice. Princeton University Press. Chong, Dennis and James N Druckman. 2007. “Framing Public Opinion in Competitive Democ- racies.” American Political Science Review 101(4):637–655. Dunning, Thad. 2016. “Transparency, Replication, and Cumulative Learning: What Experi- ments Alone Cannot Achieve.” Annual Review of Political Science 19:S1–S23. Eagly, Alice H and Shelly Chaiken. 1993. The Psychology of Attitudes. Harcourt Brace Jo- vanovich College Publishers. Ellul, Jacques. 1966. Propaganda. Knopf New York. Farnsworth, Stephen J. 2015. Spinner in Chief: How Presidents Sell Their Policies and Them- selves. New York: Routledge. Fazio, Russell H. 1995. “Attitudes as Object-evaluation Associations: Determinants, Conse- quences, and Correlates of Attitude Accessibility.” Attitude Strength: Antecedents and Con- sequences 4:247–282. Foroni, Francesco and Ulrich Mayr. 2005. “The Power of a Story: New, Automatic Associations from a Single Reading of a Short Scenario.” Psychonomic Bulletin & Review 12(1):139–144. Forscher, Patrick S, Calvin K Lai, Jordan R Axt, Charles R Ebersole, Michelle Herman, Patri- cia G Devine and Brian A Nosek. 2019. “A Meta-analysis of Procedures to Change Implicit Measures.” Journal of Personality and Social Psychology 117(3):522–559. Franco, Annie, Neil Malhotra and Gabor Simonovits. 2014. “Publication bias in the social sciences: Unlocking the file drawer.” Science 345(6203):1502–1505.
16 Geddes, Barbara and John Zaller. 1989. “Sources of Popular Support for Authoritarian Regimes.” American Journal of Political Science 33(2):319–347. Gerring, John. 2007. “Is There a (Viable) Crucial-case Method?” Comparative Political Studies 40(3):231–253. Greenwald, Anthony G., Brian A. Nosek and Mahzarin R. Banaji. 2003. “Understanding and Us- ing the Implicit Association Test: I. An Improved Scoring Algorithm.” Journal of Personality and Social Psychology 85(2):197. Greenwald, Anthony G., Debbie E. McGhee and Jordan LK Schwartz. 1998. “Measuring Individ- ual Differences in Implicit Cognition: The Implicit Association Test.” Journal of Personality and Social Psychology 74(6):1464. Greenwald, Anthony G, Mahzarin R. Banaji, Laurie A Rudman, Shelly D Farnham, Brian A Nosek and Deborah S Mellott. 2002. “A Unified Theory of Implicit Attitudes, Stereotypes, Self-esteem, and Self-concept.” Psychological Review 109(1):3–25. Greenwald, Anthony G, T Andrew Poehlman, Eric Luis Uhlmann and Mahzarin R Banaji. 2009. “Understanding and using the Implicit Association Test: III. Meta-analysis of predictive validity.” Journal of personality and social psychology 97(1):17. Higgins, E Tory. 1996. “Activation: Accessibility, and Salience.” Social Psychology: Handbook of Basic Principles pp. 133–168. Hovland, Carl I, Arthur A Lumsdaine and Fred D Sheffield. 1949. “Experiments on Mass Communication.”. Huang, Haifeng. 2015. “Propaganda as Signaling.” Comparative Politics 47(4):419–444. Huang, Haifeng. 2018. “The Pathology of Hard Propaganda.” The Journal of Politics 80(3):1034–1038. Karpinski, Andrew and Ross B. Steinman. 2006. “The Single Category Implicit Association Test as a Measure of Implicit Social Cognition.” Journal of Personality and Social Psychology 91(1):16. King, Gary. 1995. “Replication, Replication.” PS: Political Science & Politics 28(3):444–452.
17 Klemperer, Victor. 1998. I Will Bear Witness: The Diaries of Victor Klemperer. Random House. Lodge, Milton and Charles S Taber. 2013. The Rationalizing Voter. Cambridge University Press. Maxwell, Scott E, Michael Y Lau and George S Howard. 2015. “Is Psychology Suffering From a Replication Crisis? What Does “Failure to Replicate” Really Mean?” American Psychologist 70(6):487. Nelson, Thomas E, Zoe M Oxley and Rosalee A Clawson. 1997. “Toward a Psychology of Framing Effects.” Political Behavior 19(3):221–246. Nosek, Brian A, Charles R Ebersole, Alexander C DeHaven and David T Mellor. 2018. “The preregistration revolution.” Proceedings of the National Academy of Sciences 115(11):2600– 2606. Peisakhin, Leonid and Arturas Rozenas. 2018. “Electoral Effects of Biased Media: Russian Television in Ukraine.” American Journal of Political Science 62(3):535–550. Ryan, Timothy J. 2017. “How Do Indifferent Voters decide? The Political Importance of Implicit Attitudes.” American Journal of Political Science 61(4):892–907. Schönbrodt, Felix D and Eric-Jan Wagenmakers. 2018. “Bayes factor design analysis: Planning for compelling evidence.” pp. 1–15. Sennett, Alan. 2014. “Film Propaganda: Triumph of the Will as a Case Study.” Framework: The Journal of Cinema and Media 55(1):45–65. Simons, Daniel J. 2014. “The Value of Direct Replication.” Perspectives on Psychological Science 9(1):76–80. Stanley, Jason. 2015. How Propaganda Works. Princeton University Press. Stockmann, Daniela and Mary E Gallagher. 2011. “Remote Control: How the Media Sustain Authoritarian Rule in China.” Comparative Political Studies 44(4):436–467. Tell it like it is. 2020. Nature Human Behaviour pp. 1–1. Truex, Rory and Daniel L Tavana. 2019. “Implicit Attitudes Toward an Authoritarian Regime.” The Journal of Politics 81(3):1014–1027.
18 Wedeen, Lisa. 2015. Ambiguities of Domination: Politics, Rhetoric, and Symbols in Contempo- rary Syria. University of Chicago Press. Yanagizawa-Drott, David. 2014. “Propaganda and Conflict: Evidence from the Rwandan Geno- cide.” The Quarterly Journal of Economics 129(4):1947–1994. Zaller, John R et al. 1992. The Nature and Origins of Mass Opinion. Cambridge University Press.
19
A Appendix
The Appendix includes the following materials:
A.1 Sample Information and Balance Statistics
Table A1: Sample Information (U.S. Studies)
Table A2: Sample Information (Hong Kong Study)
Figure A1: Balance Statistics for U.S. Studies
Figure A2: Balance Statistics for Hong Kong Study
A.2 Additional Analyses
Figure A3a: Effect of Propaganda Treatment on Approval of Trump
Figure A3b: Effect of Propaganda Treatment on Intention to Vote for Trump in 2020
A.3 Questionnaires
Questionnaire for U.S.-Trump-IAT (Study 1, 2, 4)
Questionnaire for HK-CCP-IAT (Study 3)20
A.1 Sample Information and Balance Statistics
Table A1: Sample Information (U.S. Studies)
Study 1 Study 3 Study 4
Trump US (7/2017) Trump US (6/2019) Trump US (8/2019)
Political
vote.trump.gen2016 0.242 0.146 0.388
vote.clinton.gen2016 0.437 0.450 0.318
vote.gen2016 0.769 0.694 0.765
party.rep 0.169 0.159 0.358
party.dem 0.427 0.490 0.359
Demographic
lowed 0.450 0.502 0.611
income.high 0.045 0.049 0.060
female 0.447 0.442 0.667
relationship.married 0.348 0.327 0.558
race.white 0.738 0.708 0.816
race.black 0.072 0.065 0.101
race.asian 0.076 0.091 0.028
race.hispanic 0.038 0.072 0.032
age 33.6 34.1 53.9
Date 7/2017 6/2019 8/2019
Sample Prolific Prolific Qualtrics
N 524 852 1062
Note: Table compares means for demographic and professional covariates across the dif-
ferent samples for the U.S. studies. All variables other than age are indicator variables.
Respondents with lowed = 1 are those that have not completed a college education. Re-
spondents with income.high = 1 are those with household income greater than $120,000
per year. All other variable names are self-explanatory.21
Table A2: Sample Information (Hong Kong Study)
Study 2
Hong Kong CCP (2/2018)
Political
vote.reg2016 0.826
polinc.opp 0.462
Demographic
lowed 0.302
work.unemp 0.016
work.student 0.071
income.high 0.390
female 0.549
mainland.frac 0.044
abroad.frac 0.022
age 35.6
Date 2/2018
Sample Qualtrics
N 1533
Note: Table compares means for demographic and profes-
sional covariates across the different samples for the Hong
Kong study. All variables other than age, mainland.frac,
and abroad.frac are indicator variables. Respondents with
lowed = 1 are those that have not completed a college edu-
cation. Respondents with income.high = 1 are those with
household income greater than 50,000 HKD per year. Re-
spondents with polinc.opp = 1 are those that identified
themselves as supporters of localist or radical/moderate
democracy parties. All other variable names are self-
explanatory.22
Figure A1: Balance Statistics for U.S. Studies
vote.trump.gen2016
vote.gen2016
vote.clinton.gen2016
relationship.married
race.white
Covariate
Study 1
race.black
Study 2
race.asian
Study 4
party.rep
party.dem
lowed
income.high
female
−0.2 0.0 0.2
Standardized Difference in Means (treatment − control)23
Figure A2: Balance Statistics for Hong Kong Study
work.unemp
work.student
vote.reg2016
polinc.opp
occupy.num
Covariate
mainlandfrac Study 3
lowed
income.high
female
age
abroadfrac
−0.2 0.0 0.2
Standardized Difference in Means (treatment − control)
Note: Figure shows the standardized difference in means between the treatment and control groups across
several covariates of interest for the Hong Kong study.24
A.2 Additional Analyses
Figure A3a: Effect of Propaganda Treatment on Approval of Trump
Outcome: Explicit Attitudes (trump.approval)
Study 1 − U.S.−Trump (7/2017) Study 3 − U.S.−Trump (6/2019) Study 4 − U.S.−Trump (8/2019)
0.50
Estimate of Propaganda Treatment Effect
0.25
0.00
Covariates
None
−0.25
−0.50
Full Not Trump Trump Full Not Trump Trump Full Not Trump Trump
Sample Supporters Supporters Sample Supporters Supporters Sample Supporters Supporters
Note: Figure shows estimates of ATE for three U.S. studies of the Trump propaganda treatment. The
outcome variable is the Trump explicit attitude measure trump.approval. Each panel shows results for the
full sample and then explores subgroup effects for Trump supporters and respondents who did not vote for
him in 2016.25
Figure A3b: Effect of Propaganda Treatment on Intention to Vote for Trump in 2020
Outcome: Vote Intention (trump.2020)
Study 1 − U.S.−Trump (7/2017) Study 3 − U.S.−Trump (6/2019) Study 4 − U.S.−Trump (8/2019)
0.4
Estimate of Propaganda Treatment Effect
0.2
Covariates
None
0.0
−0.2
Full Not Trump Trump Full Not Trump Trump Full Not Trump Trump
Sample Supporters Supporters Sample Supporters Supporters Sample Supporters Supporters
Note: Figure shows estimates of ATE for three U.S. studies of the Trump propaganda treatment. The out-
come variable is the respondents intention to vote for Trump in the next election trump.vote2020. Each panel
shows results for the full sample and then explores subgroup effects for Trump supporters and respondents
who did not vote for him in 2016.26
A.3 Questionnaires
Questionnaire for U.S.-Trump-IAT (Study 1, 2, 4)
Questionnaire for HK-CCP-IAT (Study 3)TRUMP - SINGLE CATEGORY IMPLICIT ASSOCIATION TEST
Questionnaire
INTRODUCTION AND CONSENT
This survey is about your political attitudes. It is part of an academic research project. The
survey is being conducted by academic researchers at XXXXXXXXX.
Your participation is completely voluntary. If you agree to participate, you will answer some
questions about yourself and complete a short timed categorization exercise. The questions
should take about 10 minutes to answer. If you complete the survey, you will receive a small
payment agreed on through Prolific..
If you agree to participate, you may refuse to answer any of the questions. Your participation
in this study will be confidential. Any identifying information will be accessible only to the
researchers and will never appear in any sort of report that might be published or shared.
If you have any problems or questions involving the research, you may contact the Principal
Investigator:
XXXXXXXXXXXXXXXXXXX
If you have questions regarding your rights as a research subject, or if problems arise which
you do not feel you can discuss with the Investigator, please contact the Institutional Review
Board at:
XXXXXXXXXXXXXXXXXX
By clicking Continue below, you are agreeing to participate in the survey.
SECTION I: DEMOGRAPHICS
First, please answer some questions about your personal background.
D1. Are you male or female?
Male
Female
No answerD2. In what state do you live? Alabama Alaska Arizona Arkansas California Colorado Connecticut Delaware Florida Georgia Hawaii Idaho Illinois Indiana Iowa Kansas Kentucky Louisiana Maine Maryland Massachusetts Michigan Minnesota Mississippi Missouri Montana Nebraska Nevada New Hampshire New Jersey New Mexico New York North Carolina North Dakota Ohio Oklahoma Oregon Pennsylvania Rhode Island South Carolina South Dakota Tennessee Texas Utah Vermont
Virginia Washington West Virginia Wisconsin Wyoming D3. In what year were you born? _______ Don’t know D4. What is the highest level of education you have completed? Did not graduate from high school High school graduate Some college, but no degree (yet) 2-year college degree 4-year college degree Postgraduate degree (MA, MBA, MD, JD, PhD, etc.) No answer D5. What racial or ethnic group best describes you? White Black or African-American Hispanic or Latino Asian or Asian-American Native American Middle Eastern Mixed Race Other No answer D6. What is your marital status? Married, living with spouse Separated Divorced Widowed Single, never married Domestic partnership No answer D7. Which of the following best describes your current employment status? Working full time now Working part time now Temporarily laid off
Unemployed
Retired
Permanently disabled
Taking care of home or family
Student
Other
No answer
D8. Thinking back over the last year, what was your family’s annual income?
Less than $10,000
$10,000-19,999
$20,000-29,999
$30,000-39,999
$40,000-49,999
$50,000-59,999
$60,000-69,999
$70,000-79,999
$80,000-89,999
$90,000-99,999
$100,000-119,999
$120,000-149,999
$150,000 or more
Prefer not to say
SECTION 2: POLITICAL ATTITUDES AND PARTICIPATION
P1. Which of the following television programs do you watch regularly? Please check any that
you watch at least once a month.
20/20; 60 Minutes; ABC News Nightline; ABC World News Tonight; America Live; America
This Morning; America's Newsroom; American Idol; Anderson Cooper; The Big Bang Theory;
CBS Evening News; CBS This Morning; Chris Matthew s Show; Colbert Report; Daily Show
with Jon Stewart; Dancing with the Stars; Dateline NBC; Doctors; The Ellen DeGeneres
Show; Face the Nation; The Five; Fox Report; Frontline; Good Morning America; Hannity;
Huckabee; Insider; Jimmy Kimmel Live; Key & Peele; The Late Late Show with Craig
Ferguson; Late Show with David Letterman; Meet the Press; NBC Nightly News; NCIS;
O'Reilly Factor; On the Record with Greta Van Susteren; Person of Interest; Rock Center with
Brian Williams; Saturday Night Live; Special Report with Bret Baier; Tavis Smiley; Sunday
Morning; The View; This Week; Today Show; The Voice; The Talk. [Reduce list down]
P2. Are you registered to vote?
Yes
NoNo answer P3a. Generally speaking do you usually think of yourself as a: Republican Democrat Independent Other: Please indicate: P3b. Would you call yourself a: [display if P3a = Republican] Strong Republican Not very strong Republican P3c. Would you call yourself a: [display if P3a = Democrat] Strong Democrat Not very strong Democrat P3d. Do you think of yourself as closer to the… [display if P3a = Independent or Other] Republican Party Democratic Party Neither party P4. On each of the next several pages you will be presented with a statement. Please rate how much you agree or disagree with the statement on each page. P4a. When I speak about the [P3a piped text] Party, I usually say “we” instead of “they”: Strongly agree Agree Neither agree or disagree Disagree Strongly disagree P4b. I am interested in what other people think about the [P3a piped text] Party. Strongly agree Agree Neither agree or disagree Disagree Strongly disagree P4c. When people criticize the [P3a piped text] Party, it feels like a personal insult. Strongly agree
Agree Neither agree or disagree Disagree Strongly disagree P4c. I have a lot in common with other supporters of the [P3a piped text] Party. Strongly agree Agree Neither agree or disagree Disagree Strongly disagree P4d. If the [P3a piped text] Party does badly in opinion polls, my day is ruined. Strongly agree Agree Neither agree or disagree Disagree Strongly disagree P4e. When I meet someone who supports the [P3a piped text] Party, I feel connected with this person. Strongly agree Agree Neither agree or disagree Disagree Strongly disagree P4f. When I speak about the [P3a piped text] Party, I refer to them as “my party”. Strongly agree Agree Neither agree or disagree Disagree Strongly disagree P4g. When people praise the [P3a piped text] Party, it makes me feel good. Strongly agree Agree Neither agree or disagree Disagree Strongly disagree P5. Did you vote in the 2016 general election?
You can also read
