Did Ohio's Vaccine Lottery Increase Vaccination Rates? A Pre-Registered, Synthetic Control Study - OSF

Pre-Print, July 2021

1 Did Ohio’s Vaccine Lottery Increase Vaccination
2 Rates? A Pre-Registered, Synthetic Control Study

3 David Lang∗ Lief Esbenshade† Robb Willer‡

 ∗ Graduate School of Education, Stanford University. Email: dnlang86@stanford.edu.
 † Graduate School of Education, Stanford University. Email: liefesbenshade@stanford.edu.
 ‡ Stanford University,. Email: willer@stanford.edu.

 1

Pre-Print, July 2021 VACCINE LOTTERY

4 Abstract

5 As the supply of COVID-19 vaccines in the US now exceeds demand, identifying
6 novel strategies to overcome vaccine hesitancy has become critically important for
7 containing the COVID-19 pandemic. To address persistent vaccine hesitancy, several
8 states have begun offering lottery tickets as incentives for vaccination. Several aspects
9 of lotteries suggest they are an attractive incentive for vaccine hesitant subpopulations,
10 but because of the recency of these programs, little is yet known about their effec-
11 tiveness. In a pre-registered study, we estimate the effects of Ohio’s lottery program
12 Vax-a-Million on vaccination rates by comparing it to a "synthetic control" made up of
13 other, comparable states. We hypothesized that this policy would increase state vacci-
14 nation rates relative to a weighted composite of Kansas, Wisconsin, Georgia, Delaware,
15 Virginia, Connecticut, Iowa, Hawaii, and Alaska. However, we found no evidence that
16 the lottery had a statistically significant impact on the percentage of adults fully vacci-
17 nated. Our point estimate found the percentage fully vaccinated in Ohio after the final
18 lottery drawing was 0.9% percentage points lower than in the synthetic comparison
19 group. Since Ohio’s announcement, many additional states have also launched vaccine
20 incentive lotteries. In exploratory analyses using the same methodology, we find that
21 the average effect of lotteries in these 17 states is also not statistically significant.
22 Keywords: COVID-19, vaccination, synthetic control, lotteries, vaccine hesitancy,
23 vaccine confidence

 2

Pre-Print, July 2021 VACCINE LOTTERY

24 1 Introduction
25 The COVID-19 pandemic is the largest public health crisis in recent history. With the
26 discovery and mass production of efficacious vaccines against the virus, motivating uptake
27 of the vaccines has emerged as a critical challenge. To this end, a number of incentives
28 and encouragement strategies have been implemented in the U.S., ranging from beer to
29 saving bonds1, 2 . Because of the rapid pace of the pandemic and pandemic response efforts,
30 however, little is yet known about the effectiveness of these programs and whether they can
31 help contain the virus in the United States3 .
32 Lotteries are a potentially powerful and low cost way to encourage vaccination that
33 have been shown to induce behavioral change both in financial4 and health behaviors5 .
34 Particularly in the context of vaccination, lotteries may be an effective incentive in that they
35 are uniquely attractive to risk-preferring individuals. In this context, risk-preferring means
36 an individual would pay more for a lottery ticket than its expected value equivalent. Prior
37 work finds risk-preference is positively correlated with vaccine skepticism,6 suggesting that
38 the vaccine-hesitant may be uniquely high in risk-preference, and thus uniquely responsive
39 to lottery incentive programs.
40 Beyond risk preference, other factors may offer some insight as to why probabilistic
41 incentives such as lotteries may be more useful than certain incentives. Individuals who
42 systematically overestimate low-probability events such as adverse vaccine reactions may
43 also systematically overestimate the probabilities of winning a lottery.7 Lastly, prior work
44 finds that low-income and minority communities are less likely to get vaccinated but also
45 more likely to have high participation rates in lotteries8–10 . For these reasons, lotteries may
46 be able to encourage individuals to get vaccinated where other strategies have failed.
47 Yet despite all of these appealing factors, there are serious concerns about whether and
48 how individuals should be compensated for obtaining COVID-19 vaccinations. Efforts to
49 promote uptake of the Human Papillomavirus (HPV) vaccine illustrate the uncertainty and
50 challenges associated with incentivizing vaccination. One randomized controlled trial in
51 the United Kingdom found that financial compensation boosted HPV vaccination rates in
52 participants by nearly ten percentage points11 . However, when community health service
53 providers in the Netherlands offered raffles for iPods in exchange for receiving the HPV
54 vaccine, these communities subsequently had lower vaccination rates and the vaccination
55 initiative received negative media coverage12 .
56 Vaccine promotion efforts have also faced political challenges,13, 14 including opposition
57 based on the cost of compensating vaccine recipients.15 Similar sentiments have been
58 shared with regards to COVID-19 vaccination efforts, including concerns that payment
 Copyright: © 2021. The authors license this article under the terms of the Creative Commons Attribution
 3.0 License.

 3

Pre-Print, July 2021 VACCINE LOTTERY

59 to individuals erodes intrinsic motivation, is coercive, and could reduce confidence in
60 the vaccine’s safety16 . Even work that suggests compensating vaccination recipients is
61 beneficial notes that that the optimal compensation strategy for staggered treatments - such
62 as the Moderna and Pfizer vaccines - are not obvious17 .
63 These concerns speak to a larger issue of how to manage potential tradeoffs between
64 short-term and long-term goals in social policy, and how to measure success in such efforts18 .
65 Certain policy actions may increase vaccination rates in the short-term, but also make
66 motivating the remaining population to vaccinate more difficult. For instance, assume that
67 both intrinsic and extrinsic factors are motivations for an individual to get vaccinated. The
68 expiration of a lottery could reduce individuals’ extrinsic motivation while simultaneously
69 eroding individuals’ intrinsic motivation with a perceived financial incentive.
70 In this paper, we present a pre-registered analysis of a program intervention that offers
71 vaccine recipients a chance to participate in a million dollar lottery if they had received their
72 first vaccination prior to any of several successive weekly drawings. Despite early results
73 suggesting Ohioans increased their first-doses in response to this lottery announcement,
74 assessing this program is difficult without a clearly articulated goal and counterfactual19 .
75 Developments in the social sciences over the last decade have made it clear that "researcher
76 degrees of freedom" are a significant problem leading to over-identification of treatment
77 effects when none exist.20 To avoid this concern, we pre-register our analysis plan, including
78 code for data processing, outcome selection, and model weights is a form of ‘blinding’21
79 , ensuring that our analysis was neither intentionally nor unintentionally biased towards
80 finding a specific result.22
81 We hypothesized that because lotteries offer incentives that may be uniquely motivating
82 to many unvaccinated individuals, we would see a relative increase in vaccination rates
83 in Ohio following the "Vax-a-million" announcement compared to states that were similar
84 before the lottery announcement. However, our analysis found no evidence that Ohio’s
85 vaccination rate increased any faster than in its comparison states. While absence of
86 evidence is not the same as evidence of absence, this result should caution other states that
87 are considering using lottery incentives to increase vaccine turnout. These findings are
88 particularly important as they contrast with early results suggesting that Ohio’s lottery was
89 effective at boosting vaccination rates in the short term,19 and those early results were used
90 by the White House to encourage other states to adopt lotteries of their own.23 Details of
91 the policy intervention and our specific causal inference strategy follow.

92 2 Intervention Details and Outcome Data
93 The focal program we study is called Vax-a-Million1. The intervention was announced
94 on May 12th, 2021 by Ohio Governor Mike Dewine. Starting on May 26th, a weekly
95 lottery drawing was conducted through June 23rd 2021. All Ohio residents who were 18
 1https://www.ohiovaxamillion.com/index.html

 4

Pre-Print, July 2021 VACCINE LOTTERY

 Figure 1: Vaccination Rates by State

96 years or older and entered to participate in the lottery were eligible to receive a one-million
97 dollar prize if they had received their first dose by the date of the drawing. Individuals
98 under 18 were eligible for college scholarships. To put the odds of this prize drawing into
99 perspective, if all eligible Ohio residents participated in a drawing, their odds of winning
100 a million dollars was substantially greater than traditional state-funded alternatives such as
101 Mega Millions 2.
102 The focal outcome of our study comes from Our World in Data’s COVID-19 vaccination
103 database, which uses numbers published by the US Center for Disease Control (CDC)24 .
104 The measure counts the percentage of individuals that are fully vaccinated in each US
105 state. We chose this to be our outcome measure because it is aligned with the public policy
106 goals. Notably, this measure requires that among individuals who receive either the Pfizer
107 or Moderna vaccines, they must receive two doses to count as fully vaccinated.
108 We plot fully vaccinated rates by state and week for the over 18 population in Figure 1.
109 At the time of the lottery announcement, Ohio was in the middle of the distribution, ranking
110 as the 25th most vaccinated state with 37.4% of the population being fully vaccinated. On
111 the day after the final lottery drawing, Ohio had slipped three positions to the 28th most
112 vaccinated state with 43.7% percent of the population fully vaccinated.
113 All data is aggregated to the week level (ISO 8601 standard), starting on Monday and
 2https://www.ohiolottery.com/games/drawgames/megamillions.aspx lists a probability of 1 in 12,607,306.
 Our calculations (see https://development.ohio.gov/files/research/P5032.pdf) suggest that there are 9.1 million
 eligible residents in Ohio yielding a minimum 1 in 1.8 million chance if an individual participated in all 5
 drawings.

 5

Pre-Print, July 2021 VACCINE LOTTERY

114 ending on Sunday. All subsequent plots and analyses are recentered and denominated in
115 weeks relative to the lottery announcement to facilitate communication.

116 3 Design
117 We use a synthetic control methodology to create the counterfactual vaccination outcome
118 for Ohio. This technique is useful for cases where a single-aggregated unit such as a
119 state or country receives a treatment25, 26 . One can then create a synthetic version of that
120 state by constructing a convex combination of other states (the donor pool) using either
121 pre-treatment outcomes and/or other relevant covariates. Researchers have recently used
122 synthetic control methods to estimate the effectiveness of California’s shelter in place orders
123 at the beginning of the COVID-19 pandemic,27 to estimate the impact of a super-spreading
124 event on COVID-19 case rates,28 and to estimate the effects of lockdowns on air pollution
125 and health in Wuhan, China.29
126 A particular novelty of this method is that it allows researchers to specify a counterfactual
127 without any knowledge of post-treatment data, making it well-suited for preregistration30 .
128 By identifying the specific weighting of states, it provides a clear and articulated counter-
129 factual of what would happen if no interventions occurred. In light of concerns regarding
130 cherry picking with synthetic control methodologies,31 we pre-registered the weights for
131 the synthetic comparison group using data from January 12th to May 9th. We defined the
132 pre-treatment period through the end of the last full week before the lottery announcement
133 on May 12th. We stopped data collection and calculated results after the last lottery was
134 run on June 23rd. All code used in this paper including, but not limited to, downloading
135 raw data, data processing, descriptive analyses, power tests, and synthetic control analysis
136 are publicly available on Github at https://github.com/williamlief/synth_vax/. Our initial
137 code and analysis was posted to the Open Science Foundation (OSF) repository on May
138 24th https://osf.io/cypbr/. On June 15th, we articulated additional sensitivity analyses that
139 specified we would also rerun our analysis excluding states that subsequently conducted
140 their own lottery after Ohio’s lottery announcement. This alternative specification and
141 associated results are included in the appendix.
142 While best practice on the role of covariates in synthetic control is still evolving, using
143 covariates in addition to outcome data for each pre-treatment period obviates their need and
144 shrinks their variable importance weights to zero32 . We believe using the full path of our
145 pre-treatment outcome is a parsimonious specification and one that can be readily facilitated
146 without additional control variables. All the code used to generate our synthetic controls
147 comes from the tidysynth package in R33 . We construct our synthetic control using the the
148 following expression:

 Õ  +1
 Õ 2
 1 − (1)
 =1 =2

 6

Pre-Print, July 2021 VACCINE LOTTERY

149 1 corresponds to our vector of pretreatment outcomes, vaccination rates before the lot-
150 tery, for the state of Ohio. corresponds to the pretreatment outcomes and the associated
151 indices of other states in the donor pool. corresponds to the unit weights, the associ-
152 ated weighting of each state in our synthetic construction. corresponds to a variable
153 importance weight of the pretreatment outcomes that we are matching on. We minimize
154 this expression subject to the constraints that both our unit weights and variable weights are
155 non-negative and sum to unity. We trained our synthetic control model on the 17 weeks
156 preceding the vaccination announcement. We used data from all 50 states in addition to the
157 District of Columbia in the donor pool. After optimizing expression 1 based on the past 17
158 weeks of vaccination data, we generated the synthetic control version of Ohio with exact
159 weights shown in Table 1. Synthetic Ohio is a composite of Kansas, Wisconsin, Georgia,
160 Delaware, Virginia, Connecticut, Iowa, Hawaii, and Alaska.

 Table 1: Synthetic Ohio Weights. Weights used to construct the synthetic counterfactual to
 Ohio. States not listed had weights less than 0.001. These weights were pre-registered on
 May 24th, 2021.

 unit weights

 AK 0.009
 CT 0.060
 DE 0.128
 GA 0.160
 HI 0.035
 IA 0.039
 KS 0.319
 VA 0.080
 WI 0.170

161 Synthetic Ohio and actual Ohio match quite well in terms of their cumulative vaccination
162 rate during the pre-treatment period, differing by at most a half-percent in any given week.
163 In Table 2 below, we show the value of our pre-treatment outcomes for Ohio, synthetic
164 Ohio, and the average across our donor pool, in the weeks leading up to the vaccination
165 announcement. In all cases, the error between Ohio and synthetic Ohio was at most 0.57
166 percentage points. This result suggests that difference between synthetic Ohio and actual
167 Ohio in the pre-treatment period is relatively small.

 7

Pre-Print, July 2021 VACCINE LOTTERY

 Table 2: Balance Table.

 pretreatment outcome Ohio Synthetic Ohio Difference Donor Pool

 lagged_vaccinations_week17 0.120 0.398 −0.278 0.556
 lagged_vaccinations_week16 0.610 0.839 −0.229 1.083
 lagged_vaccinations_week15 1.400 1.472 −0.072 1.871
 lagged_vaccinations_week14 2.440 2.451 −0.011 2.961
 lagged_vaccinations_week13 3.830 3.745 0.085 4.346
 lagged_vaccinations_week12 5.560 5.673 −0.113 6.157
 lagged_vaccinations_week11 7.670 7.651 0.019 7.972
 lagged_vaccinations_week10 9.440 9.538 −0.098 9.809
 lagged_vaccinations_week09 11.870 11.777 0.093 12.057
 lagged_vaccinations_week08 13.860 13.823 0.037 14.081
 lagged_vaccinations_week07 16.130 16.042 0.088 16.373
 lagged_vaccinations_week06 18.780 18.748 0.032 19.368
 lagged_vaccinations_week05 21.610 22.181 −0.571 22.757
 lagged_vaccinations_week04 26.320 26.101 0.219 26.153
 lagged_vaccinations_week03 30.110 29.802 0.308 29.197
 lagged_vaccinations_week02 33.230 33.076 0.154 32.036
 lagged_vaccinations_week01 35.620 35.822 −0.202 34.709

168 4 Results
169 We present results for the Synthetic Ohio and Actual Ohio in Figure 2. At the time of the
170 final lottery drawing, the vaccination rate for actual Ohio was 43.7% and the vaccination rate
171 for synthetic Ohio was 44.6%. This represents a difference of approximately 0.9 percentage
172 points.
173 We pre-registered our inference technique as the MSPE (mean squared predicted error)
174 ratio between the post-treatment and pre-treatment periods (See Appendix A for a placebo
175 analysis and Appendix B for a power analysis). Intuitively, if a policy exhibits a substantial
176 change in a focal outcome, the synthetic control will have relatively poor out of sample
177 fit in the treatment period for the treated state. We use a permutation test to compare and
178 rank the MSPE ratio in Ohio against the MSPE ratio calculated for each other state. In this
179 permutation test, we create a synthetic counterfactual for every state in the donor pool to
180 compare how much worse the out of sample fit becomes in the absence of the treatment. We
181 present the result for in Table 3. This MSPE ratio is 15.7, suggesting the treatment period
182 error is substantially larger than the pre-treatment period error. However, when we compute
183 our p-value using the state’s MSPE ranking in descending order, we see that the associated

 8

Pre-Print, July 2021 VACCINE LOTTERY

Figure 2: Trends in Vaccination Rates (Top). Difference in Vaccination rates between Ohio
and Synthetic Ohio (Bottom). Negative values show that Ohio has a lower total vaccination
rate than the synthetic comparison.
 9

Pre-Print, July 2021 VACCINE LOTTERY

184 p-value is 29/51, yielding an approximate p-value of 0.57. Thus, we cannot reject the
185 hypothesis that Ohio’s state lottery had no impact on statewide cumulative vaccination rates
186 - however, this is not proof that the lottery had no effect. To provide more interpretable
187 metrics, we also compute the average difference between synthetic control and actual values
188 for Ohio during this period as well as the end-of-period difference between the synthetic
189 and actual state by the time of the lottery’s final drawing. Additional measures indicate
190 point estimates that are negative and that Ohio has vaccination rates about 0.9% percentage
191 points lower than its counterfactual outcome.

 Table 3: Outcome Table

 Measure MSPE-Ratio Average Difference Last Period Difference

 Value 15.7 −0.81 −0.90
 Rank 29 31 29
 p-value 0.57 0.61 0.57

192 We include results for an alternative specification in Appendix C, where we re-estimate
193 our synthetic control model excluding all states that subsequently adopted a lottery outside
194 of Ohio during our specified analysis period. In total, this yielded 35 states as part of the
195 "donor pool" for the synthetic control. Despite these changes, the results do not materially
196 change and all three metrics provide similar negative point estimates on the associated
197 impact of vaccination rates.

198 5 Discussion
199 Contrary to our expectations, our work did not detect statistically significant effects and
200 we are unable to identify a material impact on Ohio’s vaccination rates from the lottery
201 program. We failed to find evidence that the lottery had a substantially different impact
202 relative to the other suite of public health initiatives that were occurring in other states in
203 the same time period. Our results corroborate other recent findings that also cast doubt on
204 the efficacy of lottery sweepstakes at increasing COVID-19 vaccination rates.34 We note
205 that county-level analysis has found some positive impact on starting-vaccination rates.35
206 We caution however that our results are only referring to state-level average vaccination
207 rates for Ohio. Other states and lotteries may have substantially different outcomes due to
208 the characteristics of the lotteries and other factors.
209 We encourage other researchers to look at this issue with more granular data and to ex-
210 amine heterogeneity in incentive effects for specific sub-populations with lower vaccination
211 rates.
212 As more states have adopted lottery incentives, future research should use methods
213 that allow for multiple treated units. New methods of multi treatment synthetic control

 10

Pre-Print, July 2021 VACCINE LOTTERY

214 models may be appropriate for this context.36, 37 We present an exploratory analysis of the
215 multi-state adoption in an Appendix D and do not find evidence for significant effects of the
216 lotteries on vaccination rates.
217 As the pace of vaccination continues to slow and the US has missed President Biden’s
218 goal of 70% vaccination by July 4th with only 67% of the adult population having received
219 at least their first dose,38 it is important that policymakers receive rapid feedback about the
220 effectiveness of their efforts. We proffer that our work acts as proof of concept that social
221 science methods can be used both in prospective and policy-relevant settings in real-time.
222 All analysis and interpretation of results was completed by June 25, 2021, two days after the
223 final drawing was conducted. We have made a pre-print of these results available on July
224 5th, less than 2 months after the policy was announced on May 13. We offer the following
225 closing thoughts on how policymakers and researchers may better facilitate such policy
226 evaluation.
227 First, to the extent that policy is guided by outcomes and efficacy, our work highlights the
228 importance of balancing and articulating long-term and short-term goals. The subsequent
229 adoption of vaccine lotteries in other states may have been influenced by the rise in first-
230 doses in the weeks following Ohio’s lottery announcement19 . Already news reports are
231 questioning the long term effectiveness of lottery incentives,39 but 15 more states have taken
232 up their own vaccination lotteries. Establishing a clear outcome and counterfactual before
233 an intervention is deployed can add credibility to its efficacy, and support policymakers’
234 subsequent decision-making.
235 Second, the ease and alacrity with which this analysis was conducted was due largely to
236 the fact that researchers and public officials offered a tremendous level of data transparency.
237 We as researchers had no privileged access. The fact that the intervention was well-defined
238 and conducted over a short period facilitated our analysis. Instrumenting high frequency
239 data with clearly defined policy changes can help facilitate assessment of such actions.
240 Lastly, we wish to highlight the value of synthetic control methods as a tool for prospec-
241 tive policy analysis for researchers. Of the nearly 80,000 registrations on the Open Science
242 Foundation repository, only seven use synthetic controls3. Synthetic control methods allow
243 researchers to generate a specific and transparent counterfactual set of outcomes before
244 any post-treatment data is generated. With these pre-defined weights, comparing treatment
245 outcomes between a synthetic and actual state is no more complicated than computing a
246 weighted average. Given the technique’s alignment with pre-registration, simple explana-
247 tion, and its broad utilization, we believe more researchers should consider pre-registering
248 their work on timely policy matters using this technique.40
 3https://web.archive.org/save/https://osf.io/registries/discover?provider=OSF%20Registries&q=
 %22synthetic%20control%22

 11

Pre-Print, July 2021 VACCINE LOTTERY

249 6 Acknowledgments
250 The authors received no specific funding for this work. The authors’ graduate studies are
251 supported by a grant from the Institute of Education Sciences (Award No. R305B140009).
252 The funders have/had no role in study design, data collection and analysis, decision to
253 publish or preparation of the manuscript. The authors would like to thank Ben Domingue
254 for support in this project. They would also like to thank Klint Kanopka and Jonas Mueller
255 for providing feedback on early drafts of this work.

256 7 Competing interests
257 The authors declare no competing interests.

258 References
259
 1 Zosia Kmietowicz. Sixty seconds on... vaccine bribes. British Medical Journal, 2021.

260
 2 Rachel Treisman. West Virginia Giving $100 To Young People Who Get Vaccinated :
261 Coronavirus Updates : NPR, 2021.

262
 3 Apoorva Mandvalli. Reaching ‘Herd Immunity’ Is Unlikely in the U.S., Experts Now
263 Believe - The New York Times, 2021.

264
 4 Paul Gertler, Sean Higgins, Aisling Scott, and Enrique Seira. The long-term effects of
265 temporary incentives to save: Evidence from a prize-linked savings field experiment.
266 Poverty Action Lab Working Paper, 2018.

267
 5 Koen Van Der Swaluw, Mattijs S Lambooij, Jolanda JP Mathijssen, Maarten Schipper,
268 Marcel Zeelenberg, Stef Berkhout, Johan J Polder, and Henriëtte M Prast. Commitment
269 lotteries promote physical activity among overweight adults—a cluster randomized trial.
270 Annals of Behavioral Medicine, 52(4):342–351, 2018.

271
 6 Sophie Massin, Bruno Ventelou, Antoine Nebout, Pierre Verger, and Céline Pulcini.
272 Cross-sectional survey: risk-averse french general practitioners are more favorable toward
273 influenza vaccination. Vaccine, 33(5):610–614, 2015.

274
 7 Colin F Camerer and Howard Kunreuther. Decision processes for low probability events:
275 Policy implications. Journal of policy analysis and management, 8(4):565–592, 1989.

276
 8 Donald I Price and E Shawn Novak. The tax incidence of three texas lottery games:
277 Regressivity, race, and education. National Tax Journal, pages 741–751, 1999.

278
 9 Mohammad S Razai, Tasnime Osama, Douglas G J McKechnie, and Azeem Majeed.
279 Covid-19 vaccine hesitancy among ethnic minority groups. BMJ, 372, 2021.

 12

Pre-Print, July 2021 VACCINE LOTTERY

280
 10 Patricia
 Soares, João Victor Rocha, Marta Moniz, Ana Gama, Pedro Almeida Laires,
281 Ana Rita Pedro, Sónia Dias, Andreia Leite, and Carla Nunes. Factors associated with
282 covid-19 vaccine hesitancy. Vaccines, 9(3):300, 2021.

283
 11 Eleni Mantzari, Florian Vogt, and Theresa M Marteau. Financial incentives for increasing
284 uptake of hpv vaccinations: a randomized controlled trial. Health Psychology, 34(2):160,
285 2015.

286
 12 Marc Rondy, Alies Van Lier, Jan Van de Kassteele, Laura Rust, and Hester De Melker.
287 Determinants for hpv vaccine uptake in the netherlands: a multilevel study. Vaccine,
288 28(9):2070–2075, 2010.

289
 13 KarlieA Intlekofer, Michael J Cunningham, and Arthur L Caplan. The hpv vaccine
290 controversy. AMA Journal of Ethics, 14(1):39–49, 2012.

291
 14 Gillian
 Haber, Robert M Malow, and Gregory D Zimet. The hpv vaccine mandate
292 controversy. Journal of pediatric and adolescent gynecology, 20(6):325–331, 2007.

293
 15 Tyler Buchanon.With first drawing days away, ohio legislator seeks to stop vax-a-million
294 lottery. Ohio Capital Journal, 2021.

295
 16 Emily A Largent and Franklin G Miller. Problems with paying people to be vaccinated
296 against covid-19. JAMA, 325(6):534–535, 2021.

297
 17 StephenT Higgins, Elias M Klemperer, and Sulamunn RM Coleman. Looking to the
298 empirical literature on the potential for financial incentives to enhance adherence with
299 covid-19 vaccination. Preventive Medicine, 145:106421, 2021.

300
 18 Susan Athey and Michael Luca. Economists (and economics) in tech companies. Journal
301 of Economic Perspectives, 33(1):209–30, 2019.

302
 19 OhioDepartment of Public Health. Ohio Vax-a-Million Details Announced | Ohio
303 Department of Health, 2021.

304
 20 Joseph P Simmons, Leif D Nelson, and Uri Simonsohn. False-positive psychology:
305 Undisclosed flexibility in data collection and analysis allows presenting anything as
306 significant. Psychological science, 22(11):1359–1366, 2011.

307
 21 Marcus R Munafò, Brian A Nosek, Dorothy VM Bishop, Katherine S Button, Christo-
308 pher D Chambers, Nathalie Percie Du Sert, Uri Simonsohn, Eric-Jan Wagenmakers,
309 Jennifer J Ware, and John PA Ioannidis. A manifesto for reproducible science. Nature
310 human behaviour, 1(1):1–9, 2017.

 13

Pre-Print, July 2021 VACCINE LOTTERY

311
 22 Andrew Gelman and Eric Loken. The garden of forking paths: Why multiple comparisons
312 can be a problem, even when there is no “fishing expedition” or “p-hacking” and the
313 research hypothesis was posited ahead of time. Department of Statistics, Columbia
314 University, 348, 2013.

315
 23 Press
 briefing by white house covid-19 response team and public health officials 2021.
316 May 2021.

317
 24 Edouard Mathieu, Hannah Ritchie, Esteban Ortiz-Ospina, Max Roser, Joe Hasell,
318 Cameron Appel, Charlie Giattino, and Lucas Rodés-Guirao. A global database of covid-
319 19 vaccinations. Nature Human Behaviour, pages 1–7, 2021.

320
 25 AlbertoAbadie and Javier Gardeazabal. The economic costs of conflict: A case study of
321 the basque country. American economic review, 93(1):113–132, 2003.

322
 26 AlbertoAbadie, Alexis Diamond, and Jens Hainmueller. Synthetic control methods for
323 comparative case studies: Estimating the effect of california’s tobacco control program.
324 Journal of the American statistical Association, 105(490):493–505, 2010.

325
 27 Andrew I Friedson, Drew McNichols, Joseph J Sabia, and Dhaval Dave. Did california’s
326 shelter-in-place order work? early coronavirus-related public health effects. Technical
327 report, National Bureau of Economic Research, 2020.

328
 28 Dhaval Dave, Drew McNichols, and Joseph J Sabia. The contagion externality of a
329 superspreading event: The sturgis motorcycle rally and covid-19. Southern economic
330 journal, 87(3):769–807, 2021.

331
 29 Matthew A Cole, Robert JR Elliott, and Bowen Liu. The impact of the wuhan covid-
332 19 lockdown on air pollution and health: a machine learning and augmented synthetic
333 control approach. Environmental and Resource Economics, 76(4):553–580, 2020.

334
 30 Scott Cunningham. Causal inference: The mixtape. Yale University Press, 2018.

335
 31 Bruno Ferman, Cristine Pinto, and Vitor Possebom. Cherry picking with synthetic
336 controls. Journal of Policy Analysis and Management, 39(2):510–532, 2020.

337
 32 Ashok Kaul, Stefan Klößner, Gregor Pfeifer, and Manuel Schieler. Synthetic control
338 methods: Never use all pre-intervention outcomes together with covariates. Research
339 Papers in Economics Archive, 2015.

340
 33 Eric Dunford. edunford/tidysynth: A tidy implementation of the synthetic control method
341 in R, 2021.

342
 34 Allan
 J. Walkey, Anica Law, and Nicholas A. Bosch. Lottery-Based Incentive in Ohio
343 and COVID-19 Vaccination Rates. JAMA, 07 2021.

 14

Pre-Print, July 2021 VACCINE LOTTERY

344
 35 Margaret E Brehm, Paul A Brehm, and Martin Saavedra. The Ohio Vaccine Lottery and
345 Starting Vaccination Rates. Technical report, 2021.

346
 36 Eli Ben-Michael, Avi Feller, and Jesse Rothstein. Synthetic controls with staggered
347 adoption. NBER Working Papers, 2021.

348
 37 Eli Ben-Michael, Avi Feller, and Jesse Rothstein. The augmented synthetic control
349 method. Journal of the American Statistical Association, (just-accepted):1–34, 2021.

350
 38 Martha Bebinger and Blake Farmer. As COVID Vaccinations Slow, Parts Of The U.S.
351 Remain Far Behind 70% Goal . NPR, 2021.

352
 39 Andrew Welsh-Huggins. Ohio ends incentive lottery with mixed vaccination results. The
353 Washington Post, 2021.

354
 40 Susan Athey and Guido W Imbens. The state of applied econometrics: Causality and
355 policy evaluation. Journal of Economic Perspectives, 31(2):3–32, 2017.

356
 41 Scott Cunningham. Causal inference: The mixtape. Yale University Press, 2021.

357
 42 Andrew Goodman-Bacon. Difference-in-differences with variation in treatment timing.
358 Technical report, National Bureau of Economic Research, 2018.

359
 43 Andrew Goodman-Bacon and Jan Marcus. Using difference-in-differences to identify
360 causal effects of covid-19 policies. In Survey Research Methods, volume 14, pages
361 153–158. Southampton: European Survey Research Association, 2020.

362 8 Appendix
363 A Placebo Analysis Plan
364 In the pre-registration we also presented a ’placebo’ test that created a false treatment period
365 of April 5th to May 9th. In this artificial treatment period, we saw little difference between
366 Ohio and synthetic Ohio.
367 As a demonstration of the tool and to show that these synthetic estimations have rea-
368 sonable out of sample fit, we tested whether or not we would detect any effect when no
369 announcement was made. We chose weeks -17 to -6 to estimate our synthetic Ohio and
370 evaluated on the five weeks before the actual lottery announcement. Once generated we
371 plotted the difference between synthetic Ohio’s vaccination rate and actual Ohio’s vacci-
372 nation rate (Figure 3). This placebo analysis suggested that it generated reasonable out of
373 sample fit, with an error of less than a percent in any of the out-of-sample periods.
374 The exact inference strategy we used to compute statistical significance is a permutation
375 test. We computed the ratio of the Mean Squared Predictive Error (MSPE) between our

 15

Pre-Print, July 2021 VACCINE LOTTERY

 Table 4: Vaccination Growth Rates by State, Pre Ohio Lottery Announcement

 State Mean SD State Mean SD

 AK 2.141 1.004 MT 2.074 0.844
 AL 1.601 0.66 NC 1.966 0.856
 AR 1.672 0.737 ND 2.005 0.863
 AZ 1.938 0.781 NE 2.259 1.022
 CA 2.188 1.129 NH 2.058 1.221
 CO 2.337 1.124 NJ 2.552 1.268
 CT 2.774 1.492 NM 2.534 0.816
 DC 2.176 1.58 NV 1.935 0.833
 DE 2.249 1.238 NY 2.455 1.397
 FL 2.063 0.849 OH 2.191 1.059
 GA 1.702 0.994 OK 1.851 0.85
 HI 2.503 1.252 OR 2.237 0.939
 IA 2.341 1.203 PA 2.241 1.147
 ID 1.772 0.745 RI 2.631 1.48
 IL 2.076 1.024 SC 1.824 0.872
 IN 1.855 0.672 SD 2.348 0.921
 KS 2.102 1.048 TN 1.688 0.661
 KY 2.097 0.974 TX 1.848 0.936
 LA 1.697 0.778 UT 1.685 0.887
 MA 2.678 1.376 VA 2.334 1.129
 MD 2.449 1.211 VT 2.671 1.341
 ME 2.825 1.517 WA 2.296 0.994
 MI 2.219 0.957 WI 2.409 1.111
 MN 2.385 1.036 WV 1.858 0.616
 MO 1.847 0.818 WY 1.756 0.873
 MS 1.5 0.746

376 pre-treatment and post treatment data using the synthetic representation of each state. We
377 then sorted them in descending order based on the ratio of MSPE and used the associated
378 rank for each state as it’s associated p-value (See Figure 4). In the case of our placebo
379 analysis, synthetic Ohio had a rank of 40 out of 51 units and an associated p-value of 0.784,
380 indicating that we fail to reject null effects.
381 For our actual analysis, we repeated these exact same steps using the 17 weeks prior to
382 the lottery announcement as our pre-treatment period and the six weeks following the lottery

 16

Pre-Print, July 2021 VACCINE LOTTERY

 Figure 3: Placebo difference in vaccination rates between Ohio and Synthetic Ohio, with a
 false treatment date set 5 weeks prior to the true treatment date. Positive values show Ohio
 with a higher percentage of the population fully vaccinated than the synthetic comparison.

383 announcement as our post-treatment period. Failure to reject the null effect hypothesis was
384 not interpreted as proof of null effects.
385 To describe the net effect of the program, we took the point estimate from the last
386 period’s difference between actual Ohio and synthetic Ohio. In the case of Figure 3, our
387 point estimate would suggest the lottery program increased participation by 0.6% . We also
388 computed the average difference between synthetic and actual Ohio across the treatment
389 period. This information can be quite descriptive if, for instance, a program had no effect
390 in the long run but encouraged some individuals to get vaccinated several weeks earlier.

 17

Pre-Print, July 2021 VACCINE LOTTERY

 Figure 4: Placebo Test

391 B Power Analysis
392 We conducted power analyses with different potential effect sizes. We generated subsequent
393 outcomes assuming that states would continue to grow at their weekly rate as sampled from
394 historical mean and standard deviation (See Table 4). We truncate these distributions such
395 that vaccination rates cannot decrease from week to week. We then assumed that the effect
396 of the lottery would have an increase between zero and two percentage points per week.
397 We computed 200 bootstrap simulations and conducted our permutation tests. Based
398 on Figure 5, we were reasonably powered to detect effect sizes on the order of 1.75%
399 percentage points or larger using a p-value cutoff of 0.10, this correspond to the top 5 states.
400 The associated power with this cutoff is 0.97. This effect is roughly on the order of the
401 absolute effect associated with compensating individuals to receive the HPV vaccine, which

 18

Pre-Print, July 2021 VACCINE LOTTERY

 Figure 5: Effect Size and Power

402 saw between a 9.8% to 13.2% percentage point increase in first-time vaccination rates11 . To
403 put this in context of statewide vaccination rates at the time of the lottery announcement,
404 Ohio is currently at 37%. Such an effect would make it the second-most vaccinated state in
405 the country just behind Maine at 48%.

406 C Alternative Specification
407 We present results and construction of synthetic controls excluding all states that adopted
408 vaccine lotteries after Ohio.4 See Table 5 for a list of state vaccine lottery announcements .
409 As Delaware was part of our original synthetic Ohio, we re-estimate excluding that state and
 4These modification plans were added to the OSF pre-registration repository on June 15, 2020. This was
 before the final two lottery drawings occurred.

 19

Pre-Print, July 2021 VACCINE LOTTERY

 Table 5: State Lottery Announcement Dates as of July 2nd, 2021
 State Lottery Announcement Date
 OH 5/13/21
 MD 5/20/21
 NY 5/20/21
 OR 5/21/21
 AR 5/25/21
 CO 5/25/21
 DE 5/25/21
 CA 5/28/21
 NM 6/1/21
 WV 6/1/21
 WA 6/3/21
 KY 6/4/21
 NC 6/10/21
 MA 6/15/21
 ME 6/16/21
 NV 6/16/21
 LA 6/17/21
 MI 6/30/21

410 all other states that subsequently announced lotteries. The new composition of the synthetic
411 comparison can be seen below in Table 6. The most notable change is the inclusion of
412 Pennsylvania in this version of Synthetic Ohio.
413 The quality of the match between Actual Ohio and this version of Synthetic Ohio exhibits
414 marginally worse fit. In total the error in this period is at most 0.6% in any given week (see
415 Table 7).
416 We present difference between Actual and Synthetic Ohio in Figure 6. Through the
417 entire treatment period, vaccination rates for Ohio are below our synthetic counterfactual.
418 At the end of the period, this difference was approximately 1.3%.
419 We present the same set of outcome measures as in our main analysis in Table 8 below.
420 The associated p-value for our pre-registered metric is 14/35 or approximately 0.40. Related
421 measures such as the average difference or end of period differences are also negative but
422 not statistically significant.

 20

Pre-Print, July 2021 VACCINE LOTTERY

Table 6: Synthetic Ohio weights, excluding states that also adopted lotteries from the donor
pool.

 Unit Original Weights Alternative Weights

 AK 0.009 0.000
 CT 0.060 0.029
 DE 0.128 Excluded
 GA 0.160 0.168
 HI 0.035 0.061
 IA 0.039 0.066
 KS 0.319 0.256
 PA 0.000 0.056
 VA 0.080 0.173
 WI 0.170 0.192

 Table 7: Balance Table (Alternative Specification)

 Pretreatment Outcome Ohio Synthetic Ohio Difference Donor Pool

 lagged_vaccinations_week17 0.120 0.362 −0.242 0.557
 lagged_vaccinations_week16 0.610 0.774 −0.164 1.089
 lagged_vaccinations_week15 1.400 1.435 −0.035 1.879
 lagged_vaccinations_week14 2.440 2.411 0.029 2.944
 lagged_vaccinations_week13 3.830 3.757 0.073 4.316
 lagged_vaccinations_week12 5.560 5.709 −0.149 6.142
 lagged_vaccinations_week11 7.670 7.692 −0.022 7.884
 lagged_vaccinations_week10 9.440 9.498 −0.058 9.722
 lagged_vaccinations_week09 11.870 11.796 0.074 12.008
 lagged_vaccinations_week08 13.860 13.837 0.023 14.014
 lagged_vaccinations_week07 16.130 16.056 0.074 16.274
 lagged_vaccinations_week06 18.780 18.804 −0.024 19.213
 lagged_vaccinations_week05 21.610 22.219 −0.609 22.566
 lagged_vaccinations_week04 26.320 26.083 0.237 25.918
 lagged_vaccinations_week03 30.110 29.780 0.330 28.878
 lagged_vaccinations_week02 33.230 33.021 0.209 31.631
 lagged_vaccinations_week01 35.620 35.827 −0.207 34.158

 21

Pre-Print, July 2021 VACCINE LOTTERY

 Figure 6: Alternate specification difference in Vaccination rates between Ohio and Synthetic
 Ohio, all states that adopted lotteries after Ohio have been excluded. Negative values show
 that Ohio has a lower total vaccination rate than the synthetic comparison

 Table 8: Outcome Table (Alternative Specification)

 Measure MSPE-Ratio Average Difference Last Period Difference

 Value 27.1 −1.14 −1.27
 Rank 14 24 26
 p-value 0.40 0.69 0.74

423 D Exploratory analysis of multiple lottery announcements
424 Given that 17 states to date have followed Ohio’s lead by announcing lotteries of their own
425 (see Table 5), we also estimate models that explicitly allow for multiple treated units with
426 differential treatment timing. These analyses were not included in our pre-registration plan

 22

Pre-Print, July 2021 VACCINE LOTTERY

 Figure 7: Vaccination Rates by State with Lottery Adoption Highlighted

427 and are therefore included here as exploratory. We present the descriptive trends of total
428 vaccination rates across states in Figure 7. In this figure we can see that both states with high
429 vaccination rates like Massachusetts and Maine, and states with low vaccination rates like
430 Louisiana and Arkansas have all adopted lottery incentives. Most states appear to maintain
431 a roughly constant relative ranking after adopting the lottery incentive.
432 We first use the augmented synthetic control method to estimate the average treatment
433 effect across states that adopt lotteries36, 37 . This method is a natural extension of the
434 pre-registered plan. We preserve the same outcome - the percent of the population fully
435 vaccinated - for this analysis.
436 This approach does have several key distinctions from the traditional synthetic control
437 approach for the single-state case. First, it provides more flexibility in terms of the possible
438 search space for generating the synthetic control, allowing weights to be negative and a
439 unit-intercept term. Second, it adds regularization to the construction of the match, both
440 to adjust for overfitting and to help ensure unique solutions to the optimization. Third,
441 it allows flexibility to balance the quality of match for an individual treated state and the
442 composite average of all treated states. The results of this analysis are presented in figure 8.
443 The size of the confidence interval expands over time as we have fewer observed states with
444 4 or more observed post-treatment weeks. We now observe a small, but still statistically
445 insignificant, average increase of 0.3 percentage points per week in the fully vaccinated rate
446 in states that adopt lotteries relative to the synthetic counterfactual. In the figure the drop
447 in the final time period is due to the fact that Ohio was the first lottery adopter and therefore
448 the estimate in the final period is based only on the effect in Ohio.
449 Given the rapid development of methodology in causal inference methods for staggered

 23

Pre-Print, July 2021 VACCINE LOTTERY

 Figure 8: Augmented Synthetic Comparison with Multiple Adopting States

450 treatment panel series data41 we also estimate a difference in difference model as a robustness
451 check of the findings from the synthetic control models as defined in equation 2. In
452 this model we control for state and year fixed effects and assume that the differences in
453 vaccination rates between the states that adopted lotteries and those that did not would
454 remain constant over time in the absence of the lottery adoption. is our treatment
455 indicator, equal to one in weeks after a state has announced a lottery incentive and zero
456 otherwise.

 = + + + (2)
457 Recent developments have shown that difference in difference methods with staggered
458 treatment timing can have biased effect estimates in the presence of time dynamic treatment
459 effects.42 We therefore supplement the traditional difference in difference model with an
460 event-study model (see equation 3) to explicitly examine treatment timing dynamic effects.43
461 In this model, we separately estimate for each week, with all estimates relative to the
462 announcement week.

 −1
 Õ 
 Õ
 = + + + + (3)
 =− =0

463 As an additional robustness check, we also analyze the weekly number of vaccinations
464 administered (per million). We include this analysis to address concerns that cross state
465 changes in the percent fully vaccinated may vary arbitrarily in the short term due to differing
466 local availability of the different vaccines and their different lag time between rounds (with
467 Johnson & Johnson notably requiring only a single dose). Our results are presented in table

 24

Pre-Print, July 2021 VACCINE LOTTERY

468 9. For our primary outcome, the percent of population fully vaccinated, we estimate a 95%
469 confidence interval for the effectiveness of the lottery incentive of -0.7 to +5.4 percentage
470 points. The dynamic treatment timing effects show consistently sized and generally non-
471 significant effect estimates in both the pre-announcement and post-announcement weeks.
472 We see some suggestive evidence that the number of people per million receiving a vaccine
473 dose each week may have increased (p-value = 0.07). However, this does not appear to be
474 converting into a clear increase in the percent of the population fully vaccinated.

 25

Pre-Print, July 2021 VACCINE LOTTERY

 Table 9: Difference in Difference Estimates

Dependent Variables: Vaccination Rate Weekly Vaccinations (per million)
Model: (1) (2) (3) (4)
Variables
Average Treatment Effect 2.4 2,267.5∗
 (1.6) (1,238.6)
Lottery State × Relative Week = -4 -1.1 1,432.4
 (0.82) (2,681.8)
Lottery State × Relative Week = -3 -0.75 2,137.1
 (0.61) (2,085.9)
Lottery State × Relative Week = -2 -0.46 1,981.2
 (0.39) (1,812.1)
Lottery State × Relative Week = -1 -0.20 2,263.6
 (0.18) (1,634.2)
Lottery State × Relative Week = 1 0.28 2,417.3
 (0.22) (2,093.2)
Lottery State × Relative Week = 2 0.54 1,109.0
 (0.38) (1,730.4)
Lottery State × Relative Week = 3 0.63 3,399.2
 (1.2) (2,976.3)
Lottery State × Relative Week = 4 1.3 2,216.4
 (1.4) (2,412.8)
Fixed-effects
week Yes Yes Yes Yes
state Yes Yes Yes Yes
Fit statistics
Observations 1,275 1,275 1,275 1,275
R2 0.96 0.96 0.79 0.80
Within R2 0.02 0.04 0.002 0.02

Clustered (state) standard-errors in parentheses
Signif. Codes: ***: 0.01, **: 0.05, *: 0.1
 Notes: Weekly effect estimates relative to the announcement week. Effects for relative
weeks less than -4 or greater than 4 omitted from table for legibility. Models include state
 clustered standard errors.

 26