LOCKDOWN LEARNING: CHANGES IN ONLINE STUDY ACTIVITY AND PERFORMANCE OF DUTCH SECONDARY SCHOOL STUDENTS DURING THE COVID-19 PANDEMIC - PSYARXIV ...
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Lockdown Learning:
Changes in Online Study Activity and Performance of Dutch
Secondary School Students during the COVID-19 Pandemic
Maarten van der Velde Florian Sense Rinske Spijkers
University of Groningen University of Groningen Noordhoff Publishers
Groningen, The Netherlands Groningen, The Netherlands Groningen, The Netherlands
m.a.van.der.velde@rug.nl
Martijn Meeter Hedderik van Rijn
Vrije Universiteit Amsterdam University of Groningen
Amsterdam, The Netherlands Groningen, The Netherlands
ABSTRACT children [29]. Indeed, initial reports have suggested learning
The COVID-19 pandemic caused lockdowns and sudden school decrements [10, 17] and adverse mental health effects [5, 26,
closures around the world in spring 2020, significantly impact- 34] in primary school students, compared to preceding years.
ing the education of students. Here, we investigate how the In many cases, the shift to distance learning also reinforced ex-
switch to distance learning affected study activity and perfor- isting inequalities: students from disadvantaged backgrounds
mance in an online retrieval practice tool used for language were found to suffer greater learning setbacks than their more
learning in Dutch secondary education. We report insights advantaged peers [1, 2, 8, 9, 10], and there were reports of dis-
from a rich data set consisting of over 116 million retrieval prac- parities in access to resources and preparedness of instruction
tice trials completed by more than 133 thousand students over materials required for distance learning between educational
the course of two consecutive school years. Our findings show levels and schools [6, 24, 30].
that usage of the tool increased substantially at the start of
lockdown, with the bulk of study activity occurring on weekday While many studies have reported on pandemic-related learn-
mornings. In general, students’ progress through the material ing losses in primary education, relatively little is known about
was largely unaffected by lockdown, although students from the impact of the pandemic on older students. Surveys of sec-
the highest educational track were somewhat more likely to ondary school students have suggested that these students
be on or ahead of schedule than students from lower tracks, spent significantly less time on school work during lockdown [3,
compared to the previous year. Performance on individual 13], and had difficulty concentrating at home [7]. In higher
study trials was generally stable, but accuracy and response education, preliminary results point to a similar drop in mo-
time on open answer questions went up, perhaps as a result tivation and effort, but indicate that academic performance
of students being more focused at home. These encouraging was unaffected or even somewhat improved [12, 15, 23]. These
findings contribute to a growing literature on the educational findings suggest that learning losses may vary with student
ramifications of distance learning during lockdown. age—perhaps as a function of students’ developing ability to
engage in self-regulated learning [25].
Keywords
Distance learning, learning analytics, COVID-19, technology- One factor that likely contributed to variation in the extent to
enhanced learning which students were affected by school closures is the availabil-
ity of digital educational materials and online learning tools [11,
14, 16]. German secondary school students reported spending
1. INTRODUCTION more time on school work if their school offered more digital
The COVID-19 pandemic has led to school closures around learning materials [3]. A study of French university students
the world. As schools shifted to distance learning in early showed that, in a course redesigned around appropriate online
2020, teachers were forced to swiftly revise their teaching meth- tools, distance learning achieved similar learning outcomes
ods [14, 20, 24]. The rapid transition has caused widespread to a course taught in person [15]. Offering online adaptive
concern about compromised learning and mental health in practice software as part of the curriculum could even benefit
students, as a study of Dutch primary schools showed that
including such software in mathematics education led to stu-
dents performing better, not worse, than they would otherwise
have [22]. Dutch educators surveyed before and during the
lockdown reported seeing the benefits of digital learning tools,
remarking on their efficiency and ability to offer a personalised
learning experience in particular [31]. The development of
suitable digital materials and learning tools may thus mitigate
(some of) the impact of school closures on learning.Number of students Number of trials Split multiple choice / open answer
Course 18/19 19/20 18/19 19/20 18/19 19/20
English 58,422 61,962 38,334,387 43,072,404 99.98% / 00.02% 99.99% / 00.01%
French 21,551 26,292 14,582,897 20,276,843 56.39% / 43.61% 58.68% / 41.32%
Total 75,695 83,496 52,917,284 63,349,247
Table 1: Available data per course and school year. Some students used the tool for both English and French and/or in more than
one school year, causing the total number of students to be slightly less than the sum of the separate counts.
To explore this further, we studied the use of online learning translation pourquoi. Data recorded in each trial included the
tools in secondary education in the Netherlands during the response accuracy and response time, as well as information
spring 2020 school closures. The Netherlands was relatively about the question format, the studied item, and the corre-
well-prepared for distance learning in terms of technological sponding textbook chapter. Over the course of two school years
infrastructure [10]; the regular curriculum already included and across both courses, a total of 133,450 students completed
digital components, and, by-and-large, students had access 116,266,531 trials. Table 1 summarises the data in the sample.
to the necessary tools at home [7]. The online learning tool
that we studied enables self-regulated, autonomous rehearsal Approval to analyse the anonymised activity data was granted
of foreign-language vocabulary. This tool was accessible to a by the Ethics Committee Psychology of the University of
large number of students in different year groups (12-16 years Groningen (study code: PSY-1920-S-0397).
old) and educational tracks (pre-vocational, general secondary,
and pre-university), and was already widely used before the 3. RESULTS
lockdown. This enabled us to compare usage and performance
We addressed the two research questions outlined in the In-
during the school closure period to measurements from earlier
troduction by first looking at usage of the retrieval practice
in the same school year and from the same period in the year
tool over time, and then investigating how study performance
before. As such, the data collected from this tool provides valu-
changed during lockdown.
able insight into the effects of lockdown on Dutch secondary
students’ learning.
All analyses were conducted in R (version 3.6.3; [27]). Regres-
sion models were fitted using the lme4 (version 1.1-21; [4]) and
In this paper, we use the collected data to address two research
lmerTest (version 3.1-0; [18]) packages. The analysis code is
questions:
available at https://osf.io/t25fe/.
1. Did usage of the retrieval practice tool change during 3.1 Study activity
the lockdown period? To identify changes in usage of the retrieval practice tool, we
looked at the frequency of study trials over time, as well as
2. Did retrieval practice performance change during the the time of day at which students were most active.
lockdown period?
3.1.1 Study frequency
2. DATA SET Figure 1 shows French-language study activity during both
We recorded the online retrieval practice activity of a large school years in terms of the total number of practice trials com-
sample of secondary education students in the Netherlands pleted per week, stratified by year group and educational track.
during two consecutive school years (18/19 and 19/20; both It also shows the percentage change in trial count during the
school years lasting from 1 August to 31 July). The sample whole lockdown period, marked in grey, relative to the same
includes students from each of the three educational tracks in period in the previous school year. Across all strata, there
Dutch secondary education—pre-vocational (vmbo), general was a notable increase in study activity during the lockdown
secondary (havo) and pre-university (vwo)—and from year period; in the most extreme cases, the number of completed
groups 1 (age: 12; corresponds to grade 7) through 4 (age: 16; trials grew almost nine-fold. Usage increased more strongly in
corresponds to grade 10). higher year groups, where baseline usage was much lower. The
increased usage persisted after schools started to reopen in
Students in the sample used SlimStampen, an online, adaptive June 2020. Similar patterns were found in English-language
retrieval practice tool made available to them through Noord- study activity (see Figure S1)1 .
hoff Publishers as part of the foreign language learning cur-
riculum for English and French. The workings of this tool are
described in detail in [28, 32, 33]. The tool enabled students to 3.1.2 Study timing
rehearse the course material through retrieval practice sessions, The effects of distance learning during lockdown were also
both upon their teacher’s instruction and of their own volition. visible in the time of day at which students were actively using
Sessions consisted of a sequence of trials in which students re- the retrieval practice tool. Figure 2A shows how study activity
hearsed a set of foreign vocabulary items by answering retrieval 1
Wherever analyses are split by year group and educational
prompts in various formats. For example, a student practis- track, we only show the French results in the main article.
ing French vocabulary might see the Dutch prompt waarom Results for English-language study are included in the
(English: why) and be asked to retrieve and type its French supplement.Year 1 Year 2 Year 3/4
Change: Change: Change:
300,000 188% 452% 846%
Pre−vocational
(vmbo)
200,000
100,000
0
Change: Change: Change:
General secondary
Trials per week
300,000 224% 588% 870%
School year
(havo)
200,000 18/19
19/20
100,000
0
Change: Change: Change:
300,000 222% 404% 778%
Pre−university
(vwo)
200,000
100,000
0
Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun
Figure 1: Comparison of French-language study activity during a regular school year (18/19) and the school year affected by
COVID-19 (19/20), stratified by year group (columns) and education track (rows). Activity is measured through the number
of trials completed per week. The shaded area marks the school closure period in the 19/20 school year. The change in activity
during this period is shown as a percentage in each plot.
A English French
Monday
Tuesday
Wednesday 0.03
18/19
Thursday
Friday
Saturday 0.02
Sunday
Monday 0.01
Tuesday
Wednesday
19/20
Thursday 0.00
Friday
Saturday
Sunday
0 3 6 9 12 15 18 21 0 3 6 9 12 15 18 21
Time of day (hour)
B English French
0.01
Monday
Tuesday
Change
Wednesday 0.00
Thursday
Friday
Saturday −0.01
Sunday
0 3 6 9 12 15 18 21 0 3 6 9 12 15 18 21
Time of day (hour)
Figure 2: A: Distribution of study trials over the week during the school closure period (16 March–2 June) in both school years. The
colour of each square shows how many trials were recorded in a learning session starting within a particular hour on a particular week-
day, as a proportion of the entire week. B: Change in distribution of study trials from the 18/19 school year to the 19/20 school year.was distributed over the week during the school closure period, no open answer questions (see Table 1), so only multiple choice
as well as during the same period in the preceding school performance is reported for English. We saw similar changes
year. The year-on-year change is depicted in Figure 2B, which in trial-level performance across educational tracks and year
shows that study activity during lockdown shifted towards groups; here we report performance at the population level.
the weekday mornings, with less activity being recorded on
weekday afternoons and on Sundays. Figure 4A shows mean response accuracy over the course of
the school year. Accuracy on multiple choice questions was
consistently high in both courses, a pattern that continued
3.2 Study performance during the lockdown period in the 19/20 school year. Accuracy
We assessed the effects of distance learning on study perfor-
on open answer questions, however, did change over the course
mance in two ways. Firstly, we tracked students’ progression
of the 19/20 school year, increasing by about six percentage
through the study materials in aggregate to identify changes in
points from the period before the lockdown (M = 77.47%,
study pace. Secondly, to identify more immediate behavioural
SD = 17.77%) to the lockdown period itself (M = 83.64%,
effects, we looked at response accuracy and response time at
SD = 15.25%) and remaining higher after schools had started
the level of individual trials.
reopening (M = 81.98%, SD = 17.94%). These patterns were
confirmed by a binomial generalised linear mixed-effects model
3.2.1 Progress through materials fitted to students’ daily accuracy scores (see Table S1). Due to
Figure 3 visualises students’ progress through the textbook the size of the data set, the model found significant changes in
chapters of the French course over the school year. Figure S2 accuracy over time for both question types, but there was only
shows the same for English. The time plots show the weekly a large effect size for open answer questions in the lockdown
share of trials that correspond to each chapter per year group and post-lockdown periods of the 19/20 school year.
and track in both school years. Gaps indicate weeks in which
no trials were recorded. There was generally a smooth pro- There were similar trends in the response time measured on
gression over time from one chapter to the next; a trend that correct answers, as Figure 4B shows. While response times on
appeared to continue during the lockdown. To measure the multiple choice questions remained stable across both school
effect of distance learning on progress, we performed pairwise years, response times on French open answer questions did
comparisons of the chapter distribution (i.e., the proportion change, increasing by about 0.25 s from the pre-lockdown
of trials associated with each chapter) during the lockdown period (M = 2.42 s, SD = 4.49 s) to the lockdown (M = 2.67
period in the 19/20 school year and the same period in the s, SD = 2.65 s) and the period that followed (M = 2.68 s,
preceding school year using a chi-square test of homogeneity. SD = 2.50 s). These changes were confirmed by a generalised
This test was done separately for each combination of year linear mixed-effects model (see Table S2). Although we can-
group and education level. In all cases, we found that there not be certain about the cause, the higher response time and
was a significant year-on-year change (all p < .001). However, accuracy may indicate that students experienced less time
the observed changes in trial share were largely within the pressure or were more focused on the task at home than at
bounds of typical fluctuations between school years, as a com- school, typing out their responses more carefully.
parison to changes outside the lockdown period indicated. We
calculated the year-on-year change in trial share per chapter
over the period preceding the lockdown, using a sliding window
4. DISCUSSION
Our results show that the shift to distance learning during the
of equal duration to the lockdown period. The change plots
COVID-19 lockdown of spring 2020 coincided with an increase
in Figure 3 show the spread of these typical changes as two
in usage of an online retrieval practice tool by students in
grey bands, extending to 1 SD and 2 SD, respectively. The
Dutch secondary education, with activity increasing on week-
changes in trial share during the lockdown period, shown as
day mornings in particular. In general, we found little evidence
points, fall mostly within this range. Larger changes outside
of major study delays as a result of the lockdown; students
the typical range sometimes pointed to students being behind
in the highest educational track were likely to be on or even
the previous year’s schedule (i.e., one chapter’s trial share was
somewhat ahead of the previous year’s schedule, and students
higher than the year before while that of a subsequent chapter
in lower tracks were also roughly on schedule. Students’ trial-
was lower), and sometimes to students being ahead of schedule.
to-trial learning performance was, if anything, better during
The tendency to be behind or ahead appeared to differ some-
the lockdown period than the year before. In particular, we
what between educational tracks: for both French and English,
saw higher accuracy and higher response times on open answer
pre-university (vwo) students were ahead of schedule in five
questions, perhaps indicating that students experienced less
year groups and on schedule in one group; general secondary
time pressure or were better able to focus at home.
(havo) students were ahead in two year groups, on schedule in
three, and behind in one; pre-vocational (vmbo) students were
The online learning data reported here offer a detailed look at
ahead of schedule in two year groups, on schedule in three,
day-to-day changes in behaviour of a large and diverse group
and behind schedule in two groups.
of students. Of course, observing students’ activity in a single
online learning tool provides only a limited view into their
3.2.2 Trial-level performance learning, and students who do not use the tool (anymore) fly
Trial-level performance was measured through response ac- completely under the radar. A more complete analysis of the
curacy and response time. These variables were analysed effects of lockdown should also consider other components of
separately for multiple choice questions, which only required the curriculum and students’ environment. As previous work
the student to select the answer from a set of options, and has shown, there are likely to be differences among school
open answer questions, which required the student to type the subjects, students, and schools in how much learning fell be-
answer. The English-language study data contained virtually hind during lockdown [10, 19, 21, 23]. Identifying the causesYear 1
Pre−vocational (vmbo) General secondary (havo) Pre−university (vwo)
100%
Share of trials
18/19
Chapter
50%
0% 1 5
100% 2 6
19/20
50% 3 7
0%
Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun
Change
20 pp
0 pp
−20 pp
1 2 3 5 6 7 1 2 3 5 6 7 1 2 3 5 6 7
Year 2
Pre−vocational (vmbo) General secondary (havo) Pre−university (vwo)
100%
Share of trials
18/19
Chapter
50%
0% 1 5
100% 2 6
19/20
50% 3 7
0%
Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun
Change
20 pp
0 pp
−20 pp
1 2 3 5 6 7 1 2 3 5 6 7 1 2 3 5 6 7
Year 3/4
Pre−vocational (vmbo) General secondary (havo) Pre−university (vwo) Chapter
100%
Share of trials
18/19
50% 1 7
0% 2 9
100% 3 12
19/20
50% 5 13
0%
6
Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun
Change
20 pp
0 pp
−20 pp
1 2 3 5 6 7 12 13 1 2 3 5 6 7 1 2 3 5 6 7
Figure 3: Progress through the French materials over time: the share of study trials pertaining to each textbook chapter as a
percentage of all trials completed in a week. The school closure period in the 19/20 school year and the corresponding period in
the previous year are marked by a dashed border. The change in trial share during this period is shown, per chapter, below each
set of plots, where the dark-grey and light-grey bands indicate the variability between the two school years in the period preceding
the lockdown (1 SD and 2 SD, respectively).A English French
100%
Accuracy
90%
School year
80%
18/19
70% 19/20
Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun
B English French Question type
Response time
3.5 s Multiple
choice
3.0 s Open
answer
2.5 s
2.0 s
Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun
Figure 4: Comparison of study performance over time during a regular school year (18/19) and the school year affected by
COVID-19 (19/20). A: Mean response accuracy (±1 SE). B: Mean response time (±1 SE) on correct answers.
of such differences can be helpful in combating the negative signers, particularly when regular classroom teaching becomes
consequences of lockdown effectively. Analyses like the current impossible.
one contribute to this goal.
Reported decrements in learning during the school closures 5. ACKNOWLEDGEMENTS
could be caused by both lower quantity of learning, and lower The online retrieval practice tool discussed in this manuscript
quality. Surveys have already shown that secondary school is licensed to Noordhoff Publishers by the University of Gronin-
students, on average, spent much less time on school-related gen.
work during the lockdown than they would normally do [3, 13].
In contrast, we found an increase in usage of the retrieval prac- 6. REFERENCES
tice tool. However, learning activity increased during what [1] M. Alvi and M. Gupta. Learning in times of lockdown:
would normally be students’ regular school hours, suggesting How Covid-19 is affecting education and food security
that retrieval practice may have replaced other school work, in India. Food Security, 12(4):793–796, Aug. 2020.
rather than being added on top of it. Our results suggest that [2] A. Andrew, S. Cattan, M. Costa Dias, C. Farquharson,
the quality of learning remained constant or improved slightly, L. Kraftman, S. Krutikova, A. Phimister, and A. Sevilla.
as indicated by normal or above-normal progress through the Learning during the lockdown: Real-time data
curriculum and stable or increasing response accuracy. It on children’s experiences during home learning. Briefing
should be noted, however, that the retrieval practice tool we Note BN288, Institute for Fiscal Studies, May 2020.
studied addresses the learning of vocabulary, which is only
[3] S. Anger, H. Dietrich,
part of learning a foreign language. It may be that students
A. Patzina, M. Sandner, A. Lerche, S. Bernhard,
focused on vocabulary learning to the detriment of other as-
and C. Toussaint. School closings during the COVID-19
pects that could not be studied as easily from home, in effect
pandemic: Findings from German high school students.
leading to a narrowing of the curriculum.
Report, Institute for Employment Research, May 2020.
There were slight differences between educational tracks in [4] D. Bates, M. Mächler, B. Bolker,
pre-lockdown usage of the online retrieval practice tool, but and S. Walker. Fitting Linear Mixed-Effects Models
usage increased fairly uniformly during the lockdown. We Using lme4. Journal of Statistical Software, 67(1), 2015.
therefore did not see evidence in our sample of disparities in [5] G. Bignardi, E. S. Dalmaijer, A. L. Anwyl-Irvine,
technological preparedness between educational tracks that T. A. Smith, R. Siugzdaite, S. Uh, and D. E. Astle.
others have reported [6]. Longitudinal increases in childhood depression symptoms
during the COVID-19 lockdown. Archives of Disease in
This work contributes to a growing literature aiming to under- Childhood, pages archdischild–2020–320372, Dec. 2020.
stand the positive and negative impact of distance learning on [6] T. Bol. Inequality in homeschooling
students and to identify the factors that may amplify positive during the Corona crisis in the Netherlands. First results
effects or mitigate negative effects. The analytics provided by from the LISS Panel. Preprint, SocArXiv, Apr. 2020.
online learning tools such as the one discussed here have the [7] M. de Haas, R. Faber, and M. Hamersma. How COVID-
potential to offer immediate and detailed insight into students’ 19 and the Dutch ‘intelligent lockdown’ change activities,
learning, which can be helpful for teachers and curriculum de- work and travel behaviour: Evidence from longitudinal
data in the Netherlands. Transportation Research
Interdisciplinary Perspectives, 6:100150, July 2020.[8] H. Dietrich, A. Patzina, and [24] G. Mohan, S. McCoy,
A. Lerche. Social inequality in the homeschooling efforts E. Carroll, G. Mihut, S. Lyons, and C. Mac Domhnaill.
of German high school students during a school closing Learning for all? Second-level education in
period. European Societies, pages 1–22, Oct. 2020. Ireland during COVID-19. Report 92, ESRI, June 2020.
[9] O. Doyle. COVID-19: Exacerbating [25] S. G. Paris and R. S. Newman.
Educational Inequalities? Report, UCD Geary Development Aspects of Self-Regulated Learning.
Institute for Public Policy, Dublin, Ireland, Apr. 2020. Educational Psychologist, 25(1):87–102, Jan. 1990.
[10] P. Engzell, A. Frey, [26] S. Pearcey,
and M. D. Verhagen. Learning Inequality During J. Raw, P. Waite, and C. Creswell. Changes in children
the Covid-19 Pandemic. Preprint, SocArXiv, Oct. 2020. and young people’s mental health symptoms from March
[11] F. Ferri, P. Grifoni, to October 2020. Report 6, Co-SPACE, Nov. 2020.
and T. Guzzo. Online Learning and Emergency [27] R Core Team. R:
Remote Teaching: Opportunities and Challenges A Language and Environment for Statistical Computing.
in Emergency Situations. Societies, 10(4):86, Nov. 2020. R Foundation for Statistical Computing, 2020.
[12] T. Gonzalez, M. de la Rubia, K. Hincz, M. C. [28] F. Sense, F. Behrens, R. R. Meijer,
Lopez, L. Subirats, S. Fort, and G. M. Sacha. Influence and H. van Rijn. An Individual’s Rate of Forgetting
of COVID-19 confinement in students’ performance Is Stable Over Time but Differs Across Materials.
in higher education. Preprint, EdArXiv, Apr. 2020. Topics in Cognitive Science, 8(1):305–321, 2016.
[13] M. Grätz and O. Lipps. Large loss in studying time during [29] UNICEF. Averting a lost COVID generation:
the closure of schools in Switzerland in 2020. Research in A six-point plan to respond, recover and reimagine
Social Stratification and Mobility, 71:100554, Feb. 2021. a post-pandemic world for every child. Brief, 2020.
[14] T. Hall, C. Connolly, S. Ó Grádaigh, K. Burden, [30] H. G. van de Werfhorst, E. Kessenich, and S. Geven. The
M. Kearney, S. Schuck, J. Bottema, G. Cazemier, Digital Divide in Online Education. Inequality in Digital
W. Hustinx, M. Evens, T. Koenraad, E. Makridou, Preparedness of Students and Schools before the Start of
and P. Kosmas. Education in precarious times: A the COVID-19 Pandemic. Preprint, SocArXiv, Aug. 2020.
comparative study across six countries to identify design [31] I. van der Spoel,
priorities for mobile learning in a pandemic. Information O. Noroozi, E. Schuurink, and S. van Ginkel. Teachers’
and Learning Sciences, 121(5/6):433–442, June 2020. online teaching expectations and experiences during
[15] S. Jacques, A. Ouahabi, and T. Lequeu. the Covid19-pandemic in the Netherlands. European
Remote Knowledge Acquisition and Assessment Journal of Teacher Education, 43(4):623–638, Aug. 2020.
During the COVID-19 Pandemic. International Journal [32] M. van der Velde, F. Sense,
of Engineering Pedagogy (iJEP), 10(6):120, Dec. 2020. J. P. Borst, and H. van Rijn. Alleviating the Cold
[16] F. Klapproth, L. Federkeil, F. Heinschke, and T. Jung- Start Problem in Adaptive Learning using Data-Driven
mann. Teachers experiences of stress and their coping Difficulty Estimates. Preprint, PsyArXiv, June 2020.
strategies during COVID - 19 induced distance teaching. [33] H. van Rijn, L. van Maanen, and M. van Woudenberg.
Journal of Pedagogical Research, 4(4):444–452, Aug. 2020. Passing the test: Improving learning gains by
[17] M. Kuhfeld, B. Tarasawa, A. Johnson, E. Ruzek, and balancing spacing and testing effects. In Proceedings of
K. Lewis. Initial findings on students’ reading and math the 9th International Conference on Cognitive Modeling,
achievement and growth. Brief, NWEA, Nov. 2020. pages 110–115, Manchester, United Kingdom, 2009.
[18] A. Kuznetsova, P. B. Brockhoff, and R. H. B. Christensen. [34] N. Wright, J. Hill, H. Sharp, and A. Pickles. Impact of
lmerTest Package: Tests in Linear Mixed Effects COVID-19 on Young Adolescent Mental Health: Compar-
Models. Journal of Statistical Software, 82(13), 2017. ison of Depression, Anxiety and Behaviour Problems in 12
[19] K. Lek, R. Feskens, Year Olds Immediately Before and During the Pandemic
and J. Keuning. Het effect van afstandsonderwijs in a UK Birth Cohort. Preprint, SSRN, Oct. 2020.
op leerresultaten in het PO. Report, Cito, Nov. 2020.
[20] L. M. L. Lorente,
A. A. Arrabal, and C. Pulido-Montes. The Right to
Education and ICT during COVID-19: An International
Perspective. Sustainability, 12(21):9091, Oct. 2020.
[21] J. E. Maldonado and K. D. Witte.
The Effect of School Closures on Standardised Student
Test Outcomes. Discussion Paper DPS20.17, KU Leuven
– Faculty of Economics and Business, Sept. 2020.
[22] M. Meeter. Primary school mathematics during
Covid-19: No evidence of learning gaps in adaptive
practicing results. Preprint, PsyArXiv, Jan. 2021.
[23] M. Meeter, T. Bele, C. den Hartogh,
T. C. Bakker, R. E. de Vries, and S. Plak. College
students’ motivation and study results after COVID-19
stay-at-home orders. Preprint, PsyArXiv, Oct. 2020.SUPPLEMENT
Year 1 Year 2 Year 3 Year 4
1,000,000
Change: Change: Change: Change:
142% 164% 158% 138%
Pre−vocational
750,000
(vmbo)
500,000
250,000
0
1,000,000
Change: Change: Change:
General secondary
Trials per week
158% 138% 198%
750,000
School year
(havo)
500,000 18/19
19/20
250,000
0
1,000,000
Change: Change: Change:
146% 208% 350%
Pre−university
750,000
(vwo)
500,000
250,000
0
Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun
Figure S1: Comparison of English-language study activity during a regular school year (18/19) and the school year affected by
COVID-19 (19/20), stratified by year group (columns) and education level (rows). Activity is measured through the number of
trials completed per week. The shaded area marks the school closure period in the 19/20 school year. The change in activity
during this period is shown as a percentage in each plot.
Effect b SE z p
Intercept (Period: pre-lockdown, School year: 19/20, Question type: open answer) 1.631 0.166 9.83Year 1
Pre−vocational (vmbo) General secondary (havo) Pre−university (vwo)
Chapter
Share of trials
100%
18/19
50% 1 5
0% 2 6
100%
19/20
50% 3 7
0% 4 8
Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun
Change
10 pp
0 pp
−10 pp
1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8 1 2 3 4 5 6 7 8
Year 2
Pre−vocational (vmbo) General secondary (havo) Pre−university (vwo) Chapter
Share of trials
100%
18/19
1 6
50%
0% 2 7
100% 3 8
19/20
50%
4 O
0%
Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun 5
Change
10 pp
0 pp
−10 pp
1 2 3 4 5 6 7 8 O 1 2 3 4 5 6 7 8 O 1 2 3 4 5 6 7 8
Year 3
Pre−vocational (vmbo) General secondary (havo) Pre−university (vwo) Chapter
Share of trials
100%
18/19
1 6
50%
0% 2 7
100% 3 8
19/20
50%
4 O
0%
Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun 5
Change
10 pp
0 pp
−10 pp
1 2 3 4 5 6 O 1 2 3 4 5 6 7 8 O 1 2 3 4 5 6 7 8
Year 4
Pre−vocational (vmbo) General secondary (havo) Pre−university (vwo)
Share of trials
100% Chapter
18/19
50%
0% 1 4
100% 2 5
19/20
50%
3 O
0%
Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun Oct Dec Feb Apr Jun
Change
10 pp
0 pp
−10 pp
1 2 3 4 5 O
Figure S2: Progress through the English materials over time: the share of study trials pertaining to each textbook chapter as a
percentage of all trials completed in a week. The school closure period in the 19/20 school year and the corresponding period in the
previous year are marked by a dashed border. The change in trial share during this period is shown below each set of plots, where
the dark-grey and light-grey bands indicate the variability between the two school years in the period preceding the lockdown (1
SD and 2 SD, respectively).Effect b SE z p Intercept (Period: pre-lockdown, School year: 19/20, Question type: open answer) 2133.210 13.019 163.85
You can also read
