TOWARD DATA-DRIVEN EDUCATION SYSTEMS - Insights into using information to measure results and manage change - Brookings Institution
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
FEBRUARY 2018
TOWARD DATA-DRIVEN
EDUCATION SYSTEMS
Insights into using information to
measure results and manage change
Samantha Custer
Elizabeth M. King
Tamar Manuelyan Atinc
Lindsay Read
Tanya Sethi
AIDDATA
A Research Lab at William & MarySamantha Custer is director of policy analysis at AidData Elizabeth M. King is a nonresident senior fellow at the Center for Universal Education at Brookings Tamar Manuelyan Atinc is a nonresident senior fellow at the Center for Universal Education at Brookings Lindsay Read was a research analyst at the Center for Universal Education at Brookings Tanya Sethi is a senior research analyst at AidData Acknowledgments The authors would like to thank Takaaki Masaki, Matthew DiLorenzo, Rebecca Latourell, John Custer, and David Batcheck for their invaluable contributions to this report. The authors appreciate the comments from peer reviewers who read an early version of the report and thus helped refine our thinking, including: Husein Abdul-Hamid, Shaida Badiee, Albert Motivans, and Eric Swanson. The Brookings Institution is a nonprofit organization devoted to independent research and policy solutions. Its mission is to conduct high-quality, independent research and, based on that research, to provide innovative, practical recommendations for policymakers and the public. AidData is a research lab at William & Mary that equips policymakers and practitioners with better evidence to improve how sustainable development investments are targeted, monitored, and evaluated. Its work crosses sectors and disciplines, serving the unique needs of both the policy and academic communities, as well as acting as a bridge between the two. AidData uses rigorous methods, cutting-edge tools, and granular data to answer the question: who is doing what, where, for whom, and to what effect? Brookings gratefully acknowledges the program support provided to the Center for Universal Education by the William and Flora Hewlett Foundation. Brookings and AidData recognize that the value they provide is in the absolute commitment to quality, independence, and impact. Activities supported by their donors reflect this commitment. i TOWARD DATA-DRIVEN EDUCATION SYSTEMS
Table of Contents
1. Introduction: The role of information in education 1
From information to impact: A theory of change 2
2. Do investments in education data match use? 7
A growing store of data and evidence 7
Evidence of information use 9
3. Identifying data needs: What do education decision-makers want? 17
About the data 18
What is the role of data and information in decisions? 19
Is the current supply of data meeting the demands of education decision-makers? 23
How can education data be more useful in decision-making? 27
4. Conclusion and key findings 32
References 35
Appendix A: Composition of survey participants and additional figures 38
Appendix B: Types of education data 45
Appendix C: Available datasets on selected education indicators 47
Appendix D: Survey methodology and questionnaires 49
Figures, Tables, and Boxes
Figure 1. Aid to statistics: Trends in volume and as a share of ODA, 2006-15, commitments 3
Figure 2. Data and evidence: From generation to use and impact 4
Figure 3. Number of impact evaluations in education, by year 9
Figure 4. Challenges for education management and information systems 11
Figure 5. Impact evaluations in education, by country 15
Figure 6. What is the most important factor influencing decisions in various education activities? 21
Figure 7. For what purposes do education decision-makers use information? 22
Figure 8. For what purposes do education decision-makers use information, by stakeholder group? 23
Figure 9. What types of data do education decision-makers use? 24
Figure 10. How granular is the information education leaders use? 25
Figure 11. What types of data are most essential for education decision-makers? 26
Figure 12. What makes some sources of data and analysis more helpful to education decision-makers? 30
Figure 13. What improvements can make information more helpful to education decision-makers? 31
Figure 14. Which solutions are the most important to enhance the value of data in education? 31
Table 1. How different education decision-makers use information? 5
Table 2. Stated use of EMIS initiatives, by country 10
Table 3. Areas of decision-making which used national student assessments, by country 13
Table 4. Types of uses of impact evaluations and systematic reviews (associated with 3ie) 14
Table 5. Education decisions included in the 2017 Snap Poll, by domain 19
Table 6. Data needs in the education sector, by decision-making domain 27
Box 1. Types of large-scale assessments of student learning 8
Box 2. Comparing the surveys 18
ii1. Introduction: The
role of information in
education
Today, 650 million children around the sources to education still fall short of ensur-
globe are at risk of being left behind as they ing that all children are learning. Meanwhile,
fail to learn basic skills. Inequitable access to relatively resource-poor education systems
education is part of the problem, but even in Latvia and Vietnam, for example, punch
when children are in school, they may not be above their weight in achieving greater gains
learning. In Uganda, for instance, barely half for students than their peers with similar in-
of grade 6 children read at a grade 2 level come levels (World Bank, 2018).
(Uwezo, 2016). In India, just one in four chil-
dren enrolled in grade 5 can read a simple ———
sentence or complete simple division prob-
lems (ASER Centre, 2017). Parents, teachers, policymakers, and school
administrators need better tools to diagnose
These challenges are widespread. Accord- where and why learning gaps exist, and
ing to the International Commission on assess what strategies they can employ to
Financing Global Education Opportunity turn things around. High-quality data and
(Education Commission; 2016), only one in evidence are essential for both tasks.
ten children in low-income countries (four
in ten in middle-income countries) are on Numerous governments, organizations, and
track to gain basic secondary-level skills by companies have responded to this chal-
2030. Moreover, the obstacles to learning lenge and are generating copious amounts
disproportionately affect marginalized of data and analysis to support education
populations—children in poor households decision-making around the world. None-
or rural areas (especially girls), children with theless, large gaps remain, as data manage-
disabilities, and children affected by conflict ment processes at the school and national
and violence. level are often under-funded, ad hoc, and of
variable quality and timeliness.
It is clear that the status quo is not good
enough, but what should be done dif- While continued investments in data cre-
ferently? While struggling schools would ation and management are necessary, the
certainly benefit from better facilities and ultimate value of information is not in its
more teachers, research underscores that in- production, but its use. Herein lies one of the
put-oriented solutions are likely insufficient. biggest challenges of translating information
Many countries that dedicate substantial re- into actionable insights: those that produce
education data are often far removed from
1 TOWARD DATA-DRIVEN EDUCATION SYSTEMSthose that make crucial decisions about ed- the path from information generation to use
ucation policies, programs, and investments. (i.e., how education systems transition from
With limited insight on what decision-mak- being data-rich to data-driven). In Chapter 2,
ers use and need, the likelihood of discon- we synthesize what past studies reveal about
nect between supply and demand is high. how data have influenced education policy,
programs, and practice, paying particular
Yet, there has been surprisingly little system- attention to the motivations and incentives
atic research on the types of information that appear to play a role in both the produc-
education decision-makers in developing tion and use of education data. In Chapter 3,
countries value most—and why. Much of the we present the findings from two surveys of
available evidence on the use of education education stakeholders conducted in 2017,
data in developing countries relies upon in- with the specific aim of identifying what
dividual case studies. These qualitative snap- data they use, how data are used, and how
shots offer deep insights on use patterns and data can be more useful for policy decisions
challenges in a single context, but make it and actions. Chapter 4 concludes with sev-
difficult to draw broader conclusions. eral implications for the future of education
data investments.
In this report, we offer a unique contribution
to this body of knowledge by analyzing the
results of two surveys of education policy- From information to
makers in low- and middle-income countries
that asked about their use of data in deci- impact: A theory of change
sion-making. Survey participants include Data has emerged at the forefront of the
senior- and mid-level government officials, global development agenda. Indeed, the
in-country staff of development partner or- United Nations (U.N.) issued a Report of the
ganizations, and domestic civil society lead- High-Level Panel of Eminent Persons on the
ers, among others (see Appendix A for more Post-2015 Development Agenda calling for a
information). Respondents do not include “data revolution.”
local-level officials, school administrators, or
teachers. Recent landmark reports echo this revolu-
tionary zeal for more and better data in the
This report aims to help the global education education sector. For example, the Education
community take stock of what information Commission’s Learning Generation report
decision-makers use to measure results argues that “setting clear priorities and high
and manage change. We define informa- standards, collecting reliable performance
tion broadly, including raw statistical and data to track system and student progress,
administrative data, quantitative and qual- and using data to drive accountability are
itative analysis, learning assessments, and consistent features of the world’s most im-
the results of program evaluations. Drawing proved education systems” (2016, p. 52). The
upon our review of the literature and the two 2016 Global Education Monitoring report
surveys of end users in developing countries, champions the generation and use of edu-
we offer practical recommendations to help cation data, particularly learning metrics, to
those who fund and produce education realize the promise of education for all (UN-
data to be more responsive to what deci- ESCO, 2017). The first World Development
sion-makers want and need. Report on education, Learning to Realize
Education’s Promise, reiterates the need to
In the remainder of this chapter, we articu- measure learning to catalyze action: “Lack of
late our working theory of change that charts data on learning means that governments
1. Introduction: The role of information in education 2FIGURE 1. Aid to statistics: Trends in volume and as a share of ODA, 2006-15, commitments
Aid to statistics (left axis) Share of aid to statistics in total ODA (right axis)
800 0.45
millions USD, 2014 constant prices
700 0.40
0.35
600
as a % of ODA
0.30
500
0.25
400
0.20
300
0.15
200
0.10
100 0.05
2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
Source: OECD (2017). Development Co-operation Report 2017: Data for Development.
can ignore or obscure the poor quality of Ultimately, these investments in data cre-
education, especially for disadvantaged ation must be matched by an equal (or
groups” (World Bank, 2018, p. 91). greater) emphasis on increasing the use of
evidence by decision-makers to allocate re-
The international response to the call for a sources, plan programs, and evaluate results.
data revolution has been positive. ODA sup- The path from data generation to use, how-
port for statistics has been increasing over ever, is not simple, automatic, or quick. The
the last decade, more than doubling from seemingly straightforward story of informa-
2006 and reaching $541 million in 2015 (Fig- tion supply, demand, and use is complicated
ure 1). While this long-term positive trend is by users’ norms (how they prefer to make
encouraging, ODA support for data still rep- decisions), relationships (who they know
resents only a miniscule 0.3 percent of total and trust), and capacities (their confidence
ODA and only a handful of bilateral agencies and capability to turn data into actionable
account for nearly four-fifths of the aid for insights). The process of moving from data
statistics. Moreover, year-to-year fluctuations generation to use and ultimately to an im-
indicate less than predictable support, as pact on education outcomes must also take
Figure 1 suggests.1 into account different institutional operating
environments (i.e., political context) that may
incentivize or dampen efforts to make deci-
sions based upon evidence.
1 Open Data Watch (2016) reports an impressive rise in global
investments in statistical capacity between 2015 and 2016— Figure 2 illustrates the complex chain from
from $264 million to $328 million—but, when comparing
only the donors for which data are available in both years,
data generation to use and impact. Each
the annual estimated contribution decreased by 10 percent.
3 TOWARD DATA-DRIVEN EDUCATION SYSTEMSlink in this chain involves different tasks and (Coburn, Honig, and Stein, 2009). Only then
several actors. can they use the data to inform specific de-
cisions regarding how to allocate resources,
Advances in technology and connectivity set policies and standards, or make course
have largely democratized the generation corrections.
of education data, particularly in front-
line service delivery contexts such as local The ultimate objective of evidence-based
schools. Multiple levels of government (local, policymaking in the education sector is to
provincial, national) are involved in collecting fuel progress toward three outcomes: im-
data on education inputs and outcomes. proved student learning, increased equity,
These officials collect, verify, curate, store, and stronger accountability relationships
analyze, and communicate information. among policymakers, school administrators,
Country-specific data are also generated teachers, parents, and students. Scholars
and analyzed by others outside government, suggest two avenues through which the use
such as researchers in academic institutions, of data can lead to these desired outcomes:
non-governmental organizations, interna- (1) improving the quality of decisions made
tional development agencies, and even and (2) strengthening the mechanisms
parents and teachers. available to monitor progress and motivate
responsiveness (See Best et al., 2013; Kel-
To move from generation to impact, de- laghan et al., 2009; Jacob, 2017; UNESCO,
cision-makers must first take notice of 2013; World Bank, 2018).
available data, interpret it, and link it to the
roles that they play in the education system
FIGURE 2. Data and evidence: From generation to use and impact
Use
»» Design, implement & adapt reforms and policies
»» Target resources based on need or return
»» Mobilize public and political support
»» Set standards and priorities
»» Hold actors accountable for student learning
Institutional Context
➤
➤
»» Role and decision-making capabilities
»» Power relationships
»» Data culture in bureaucracy & civil society
»» Capacity and resources
Generation Impact
»» Monitor & collect »» Improved student learning
»» Research, analyze & evaluate »» Increased equity
»» Curate & communicate »» Stronger accountability
relationships
➤
1. Introduction: The role of information in education 41. Better decision-making. opportunities for learning and adaptation by
Rather than making decisions based upon school actors and can restore confidence in
gut instinct, personal priorities, or anecdotal education service delivery as goals are met.
evidence, decision-makers can use disag-
gregated data to pinpoint problems and Unfortunately, not all education data are
assess the merits of possible solutions. In this used in these ways. Whether or not policy-
respect, use of empirical data and analysis makers embrace evidence-based practice is
can help decision-makers tackle difficult largely shaped by their conception of what
questions of how to bolster learning, reduce is valid evidence, their technical capacity to
wasteful spending, and target resources ef- understand available data and analysis, and
ficiently to areas of greatest need or highest their own “cost-benefit calculus” regarding
return. Empirical data and analysis can also the effort needed to make decisions based
arm policymakers with information they can upon evidence rather than other factors. The
use to counter vested interests and make likelihood that data are effectively used in the
an evidence-based case to mobilize public decision-making process is highly influenced
support for difficult reforms. by the extent to which data availability is ac-
companied by an institution-wide culture of
2. Stronger monitoring and accountability. open communication (or information shar-
The regular collection of data and infor- ing), appreciation of data, and accountability
mation allows for more consistent assess- for results.
ments of the functioning of the education
system—students, teachers, schools, and Education systems are complex and multi-
policies—based on objective performance tiered. Staff have different responsibili-
indicators and targets. Such assessments ties—from upstream policy formulation to
help all stakeholders, including parents downstream classroom instruction—each
and the public, stay up-to-date on how the with their own incentive structures and
education system is performing (Read and data needs. In promoting a culture of evi-
Atinc, 2017). These instruments also open up dence-based decision-making, leaders must
TABLE 1. How different education decision-makers use information
Ministry of Education (MoE)
Local Governments School Administrators
and decentralized units
The MoE and education sub-min- Local planning units use data to School leaders use data to track
istries use education data for allocate resources, identify and progress toward system targets,
policy design, strategic planning, support low-performing schools, formulate school action plans,
and decision-making. monitor the implementation of guide school-level practices, and
education programs, and gener- evaluate and support teachers
The MoE diagnoses strengths ate comparisons across schools. and staff.
and weaknesses of the system,
measures and ensures equity Depending on the level of
within the system, monitors the autonomy, some sub-national
distribution of resources, and governments are able to plan
holds the system accountable for and execute action plans and
making progress toward defined allocate financing based on local
standards and objectives. needs.
5 TOWARD DATA-DRIVEN EDUCATION SYSTEMSensure that staff at all levels not only have
access to data that is immediately relevant
to their needs, but that the information they
create feeds into the decisions of others (see
Table 1).
For instance, higher-level administrators
responsible for student testing and account-
ability are likely to focus on the reliability of
tests, whereas local district officials responsi-
ble for meeting national standards are more
likely to use those tests to measure achieve-
ment gaps across schools. Civil society
groups may use test scores as a barometer
of how well their neighborhood schools are
performing, while employers are primarily
interested how student performance signals
the quality of the new graduates available to
them.
A strategy for data generation and use must
reflect these differences in the perspectives
and roles of various stakeholders. There are in-
stances when local concerns are misaligned
with national priorities or standards. For ex-
ample, school-level mechanisms to monitor
teacher performance may not connect to
up-stream decisions about compensation
and in-service training. Similarly, reforms tar-
geted at strengthening social accountability
may be limited in cases where public feed-
back is not integrated into decision-making
processes (Read and Atinc, 2017).
———
This chapter articulated a theory of change
for how data, in the hands of motivated deci-
sion-makers, can lead to improved education
outcomes. In Chapter 2, we turn from the
theory of how data informs education sector
decisions to examining what this looks like in
practice based upon available evidence.
1. Introduction: The role of information in education 62. Do investments
in education data
match use?
The global education conversation has shift- frameworks, compacts, and task forces to
ed focus in recent years from raising enroll- overcome large data gaps. The World Bank
ments to improving learning outcomes. This alone has financed over 200 projects related
refocusing of education priorities has spurred to EMIS in a total of 89 countries between
a notable rise in national and international 1998 and 2013 (Abdul-Hamid et al., 2017).
investments to measure student learning.
However, it is less clear whether these data Moreover, global and local initiatives have
investments are substantively informing the emerged to facilitate the production of
design, delivery, and monitoring of educa- learning data. Consequently, participation
tion programs. in international, regional, and citizen-led
learning assessments has grown over the
In this chapter, we assess the current state past two decades in low- and middle-in-
of investments in data generation, particu- come countries (Box 1). In 2015, for example,
larly efforts to strengthen education man- 72 countries participated in the Program for
agement and information systems (EMIS), International Student Assessment (PISA), up
large-scale student assessments, and impact from 42 in 2001, with an additional seven
evaluations of policies and programs. We countries involved in PISA for Development
also review the existing evidence of whether (Lockheed, 2015). Similarly, participation in
and how education sector decision-makers the Trends in International Mathematics and
use these information sources. Science Study (TIMSS) increased from 26 to
51 countries for the 4th-grade test between
2003 and 2015 (NCES, 2017). Regional initia-
A growing store of data and tives on student assessments for countries in
Africa and in Latin America and the Caribbe-
evidence an have also increased their coverage.
A learning-focused education system must
capture accurate, timely, and comparable More countries than ever before are also
data that link inputs (e.g., school resources implementing their own large-scale national
and financing) to outputs (e.g., school en- assessments of learning. According to the
rollment and attendance) and outcomes UNESCO Institute of Statistics (UIS) Learn-
(e.g., performance assessments and other ing Assessment Capacity Index, 127 of 235
quality indicators, see Appendix B: Types of countries (54 percent) conducted a national
education data). Demand for more and bet- assessment between the years of 2010 and
ter education data has given rise to various 2015. These national assessment systems are
7 TOWARD DATA-DRIVEN EDUCATION SYSTEMSBOX 1. Types of large-scale assessments of student learning
National and sub-national National and sub-national learning assessments regularly track and assess
learning assessments whether students are mastering the national curriculum, in which areas
students are stronger or weaker, whether certain population groups are
lagging behind and by how much, and which factors are associated with
better student achievement. Assessments are census-based or capture
representative samples of students across countries or provinces.
International and regional Globally and regionally benchmarked assessments allow comparable
learning assessments assessments of performance across countries, which can provide a check
on information that emerges from national assessments. Examples of
international assessments include the Program for International Student
Assessment (PISA), Trends in International Mathematics and Science Study
(TIMSS), and Progress in International Reading Literacy Study (PIRLS).
Regional assessments include the Southern and Eastern Africa Consortium
for Monitoring Education Quality (SACMEQ), the Programme for the Analysis
of Education Systems (PASEC) in francophone West and Central Africa, and
the Latin American Laboratory for Assessment of the Quality of Education
(LLECE).
Citizen-led learning Citizen-led assessments measure learning outcomes for children both
assessments in and out of school. Such assessments, led by civil society organizations
such as the ASER Center in India and Uwezo in East Africa, involve parents
and community stakeholders to yield learning metrics on both access and
quality of education systems. Citizen-led assessments are of particular im-
portance in settings where official assessments are of questionable quality.
Source: Adapted from the World Development Report (2018).
thought to be more relevant than interna- nity initiatives that have made a quantifiable
tional assessments in designing pedagogical difference in learning outcomes. In addition,
reforms and changing the way resources are within the last five years, a number of critical
allocated to schools and classrooms (Clarke, meta-reviews have further clarified what
2012). works to improve student learning.3
Policymakers and practitioners also have ———
access to a larger body of research than ever
before about the determinants of increased There is no question that education informa-
learning and equity within education sys- tion is becoming more abundant—but is it
tems. For instance, the volume of impact being used by those making consequential
evaluations completed has increased three- decisions about where to devote scarce
fold since 2005 (Figure 3).2 These studies resources and how to design programs in
pinpoint reforms, investments, and commu- order to maximize student learning?
3 Murnane and Ganimian (2014); Andrabi, Das, and Khwaja.
2 These rigorous methods of impact evaluation, experimental 2014; Krishnaratne, White, and Carpenter (2013); Conn (2014);
or quasi-experimental, require that there is a clearly Blimpo and Evans (2011); McEwan (2015); Snilstveit, et al.
identified counterfactual in order to estimate the impact of (2016); Evans and Popova (2016); Glewwe and Muralidharan
programs or policies. (2015); Kremer, Brannen, and Glennerster (2013).
2. Do investments in education data match use? 8FIGURE 3. Number of impact evaluations in education, by year
114 115
98 102
84
82
71
57
48
37
32
2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015
Source: 3ie Impact Evaluation Repository. www.3ieimpact.org/en/evidence/impact-evaluations
Collecting, processing, and communicating er appreciation of the role or state of edu-
data requires substantial resources. There- cation among society writ large) and some
fore, it is imperative that we assess whether instrumental (e.g., improving specific edu-
data produced are indeed accessible and cation outcomes through motivating better
valuable to key decision-makers. The next decisions).4 Given the difficulty of assessing
section examines what the existing evidence the more intrinsic or symbolic role of infor-
can tell us about the current state of use of mation, here we employ an instrumentalist
data in education decisions. approach, restricting our focus on the use of
data to inform specific policies.
Evidence of information use EMIS
An Education Management Information
When data advocates promote evi- System (EMIS) produces data that could be
dence-based decisions in education systems, of tremendous value for the design, imple-
they rarely specify who are the intended mentation, and monitoring of education
users, for what purpose, and what kinds of programs. Table 2 summarizes the stated
data are needed. The implicit assumption uses of ten EMIS initiatives, including facili-
is: by everyone, for everything, and any data. tating information exchange between levels
However, the reality is more sobering. There
is little indication that decision-makers are
systematically using education data and
4 Boswell (2014) distinguishes between two uses of infor-
analysis to inform their policies or decisions. mation: legitimizing and substantiating. Policymakers can
use information to bolster their credibility in taking sound,
rational decisions, or deploy information to back up claims
There are many potential effects of evidence or preferences. For more information, see https://christinabo-
on stakeholders—some intrinsic (e.g., a great- swell.wordpress.com/2014/06/23/why-real-policy-impact-is-
more-difficult-to-evidence-than-symbolic-knowledge-use/.
9 TOWARD DATA-DRIVEN EDUCATION SYSTEMSof government, planning and budgeting, report these data on a periodic basis, using
performance monitoring, and reporting. standard forms and guidelines from the cen-
tral education ministry.
The EMIS centrally organizes information
from multiple levels of the education sys- Countries are increasingly adopting web-
tem, collecting and managing critical data based dissemination of EMIS data, which is
points such as student enrollments, number making education systems more open and
of teachers, and class size (see Appendix B transparent. However, the utility of this infor-
for more information on type of education mation is only as good as the strength of the
data). Schools or local governments usually underlying data.
TABLE 2. Stated use of EMIS initiatives, by country
Information
sharing
between
government Integrating Identifying
levels & hard-to-reach Budgeting School infrastructure Performance Reporting on
Country agencies areas Planning activities management needs monitoring system
Bosnia &
Herzegovina ✓ ✓
Cambodia
✓ ✓
Eritrea
✓ ✓ ✓
Guatemala
✓ ✓
Honduras
✓ ✓
Lebanon
✓ ✓ ✓
Lithuania
✓ ✓ ✓
Malaysia
✓
Nigeria
✓
Philippines
✓ ✓
Source: Abdul-Hamid, Saraogi and Mintz (2017)
2. Do investments in education data match use? 10Unfortunately, in many countries, the EMIS A number of case studies illustrate these chal-
is not fully functional, which inhibits effec- lenges in practice. A study of EMIS capacity
tive monitoring of education policies and in the Philippines, Ghana, and Mozambique,
programs (Abdul-Hamid, Saraogi and Mintz, for example, concludes that the countries
2017). In particular, education administrators were not generating adequate information
must tackle several challenges ranging from to monitor learning outcomes, inequality,
data quality to leadership and capacity, be- and cost-effectiveness (DeStefano, 2011).
fore these EMIS are “fit-for-purpose” (Figure Ghana, often cited as a regional leader for its
4). data capabilities, still faces major constraints
of duplicative data systems, limited quality
FIGURE 4. Challenges for education management and information systems
Lack of data utilization for
decision-making 4
Data
Untimely production and
Challenges 6
dissemination of data
Lack of reliable and quality data 8
MIS not functional due to
technical problems 6
System
Challenges
System capacity issues 13
Lack of training for data usage 2
Coordination issues 3
Operational Funding issues 3
Challenges
Long-term sustainability 5
MIS not implemented 6
Implementation delays 6
Leadership changes 2
Leadership
Challenges Lack of data culture 2
Lack of clear vision and support 6
NUMBER OF EMIS ACTIVITES
Note: MIS = Management Information System
Source: Abdul-Hamid, Saraogi and Mintz (2017)
11 TOWARD DATA-DRIVEN EDUCATION SYSTEMSassurance procedures, and over-reliance on Similarly, Early Grade Reading Assessment
paper-based and manual data entry pro- (EGRA) results triggered debate about the
cesses (Spratt et al., 2011). quality of education in Senegal, but little
change in policy (Mejia, 2011). Also similarly,
In India, a survey of frontline bureaucrats citizen-led learning assessments in India,
reveals that ambiguity about the purpose Pakistan, and a few African countries have in-
of school monitoring data can severely un- creased public awareness of poorly perform-
dercut the usefulness of information being ing schools, but have not spurred concrete
collected (Bhatty, 2016). “Information [on action to improve learning (R4D, 2015).
school quality] that is so assiduously collect-
ed through ever-more sophisticated formats Comparatively, national learning assess-
is neither analyzed nor followed-up [because ments have had more traction with policy-
it is poorly defined]. Instead, the files remain makers in developing countries. In Jordan
locked up in school, block, or district cup- and Uruguay, the national assessment results
boards” (11). were used to help teachers improve their
teaching (Obeidat and Dawani, 2014; Ravela,
LEARNING ASSESSMENTS 2005). In Bhutan, national assessment results
Sponsors of student learning assessments spurred a revision of the mathematics sylla-
emphasize their value for policymaking and bus (Kellaghan et al., 2009). In Kenya, they
agenda setting, but the evidence of such use triggered an effort to ensure that all schools
is scarce and uneven. Open data on student had sufficient desks and textbooks (ibid).
performance has been a boon to researchers
seeking to explain differences in education Table 3 summarizes evidence cited in re-
outcomes at regional and international lev- ports about the use of national assessment
els. However, the link between assessment data in decision-making. These data appear
results and educational reforms in countries primarily to support countries’ monitoring
is tenuous at best (Kellaghan, 2009). and evaluation systems, curriculum reform,
and allocation of school inputs, though the
International assessments appear to have the intensity of use is not known. They are less
greatest visibility in higher income countries. often utilized in budgeting and to shape
Countries like Germany and Norway have teacher policies.
responded to the release of their PISA re-
sults with a revision of curriculum standards PROGRAM EVALUATION AND
(Breakspear, 2012) and the introduction of a OTHER RESEARCH
national quality assessment system (Baird et While school-level administrative data from
al., 2011), respectively. a country’s EMIS and student learning as-
sessments can support real-time monitoring,
The challenge of translating awareness of in- program evaluations and other research may
ternational assessments into action is more be more helpful in assessing what is and is
acute in middle- and low-income countries. not working and why.
In Colombia, Indonesia, Jordan, and Turkey,
the release of PISA and TIMSS results have It makes sense that the growing number of
been associated with a subsequent uptick program evaluations in the education sector
in discussion of education reform (Lockheed, would prompt decision-makers to mine
2015). However, it is unclear whether this these data for lessons to inform program
heightened awareness has provoked action. selection, design, and implementation. In
fact, many organizations make their evalua-
tions accessible on their websites and easily
2. Do investments in education data match use? 12TABLE 3. Areas of decision-making which used national student assessments, by country
Professional Teacher
Curriculum Support to Student Staff develop- compensa- School Strategy /
Country reform teachers support deployment ment tion Budgeting inputs M&E
Bhutan
✓
Bolivia
✓
Burkina
Faso ✓
Ethiopia
✓
Gambia,
The ✓ ✓ ✓
Guinea
✓
India
✓ ✓ ✓
Jordan
✓ ✓
Kenya
✓
Nepal
✓
Sri Lanka
✓
Uganda
✓
Uruguay
✓
Vietnam
✓ ✓ ✓
Zimbabwe
✓ ✓ ✓
Source: Authors’ analysis based on information provided in Abdul-Hamid, Abu-Lebdeh, and Patrinos (2011), Kellaghan, Greaney, and Murray (2009); Obeidat
and Dawani (2014); Ravela (2005); Senghor (2014), Tobin, et al. (2015), UNESCO (2013).
13 TOWARD DATA-DRIVEN EDUCATION SYSTEMSdigestible through short policy briefs for this challenges or determine sector priorities,
purpose. In addition to individual program even when the information was readily avail-
evaluations, they also provide meta-reviews able.
that sift through hundreds or thousands of
published studies to distill relevant insights. Similarly, an earlier assessment of 46 sector
plans and joint sector reviews found that
Table 4 draws upon data from one of these these documents rarely discussed learning
organizations—the International Initiative for outcomes or cited empirical evidence (e.g.,
Impact Evaluation (3ie)—which tracks how education production functions, random-
their evaluation studies are being used by ized trials, meta-analyses, surveys) in articu-
decision-makers. While not limited to the lating their approach to improving learning
education sector, the data shows that impact outcomes (GPE, 2012). While education
evaluations are influencing decision-making sector plans are only one possible outlet,
across sectors, particularly with regard to among many, for decision-makers to make
informing course corrections, as well as dis- use of available program evaluations or other
cussion and design of policies and programs. research, this apparent disconnect between
evaluations and forward-looking planning
Nonetheless, two reviews by the Global Part- warrants further scrutiny.
nership for Education (GPE) suggest that
research and analysis have limited influence While we will probe this question in greater
on the education sector plans of the coun- depth in the next chapter using the survey
tries it supports. According to Bernard (2015), results, here we will make three observa-
only 18 of 42 country plans (43 percent) used tions. First, education decision-makers will
an education sector analysis to inform their only use evaluation data that is relevant to
policies and even fewer cited rigorous anal- them. This might be easier said than done.
ysis to identify root causes of performance The 3ie repository is a case in point: over
half of the 855 impact evaluations in the
education-sector pertain to just 10 countries
TABLE 4. Types of uses of impact evaluations and (Figure 5). Education stakeholders interested
systematic reviews (associated with 3ie) in other geographic areas are out of luck.
Second, evaluation studies tend to be
Use Percent
funder- and researcher-driven, and thus may
Change policy or program design 27.8 not cover the specific programs or topics of
interest to a broader set of education stake-
Inform discussion of policies & programs 25.8 holders.
Inform design of other programs 23.7
Finally, published evaluation studies typically
Inform global policy discussions 11.3 focus on programs that have shown some
impact, but education decision-makers are
Take successful programs to scale 8.2
interested in learning from not only program
Close programs that do not work 3.1 successes, but also their failures to avoid
common pitfalls.
Improve the culture of evaluation use & 0
strengthen the enabling environment
Beyond evaluation studies that focus on spe-
Total 100 cific policies or programs, other initiatives,
Source: Executive Director’s Report to the Eighteenth Meeting of the 3ie Board
such as the Research on Improving Systems
of Commissioners, London, November 7, 2017, page 9. of Education (RISE) program and the World
2. Do investments in education data match use? 14Bank’s Systems Approach to Better Educa- FIGURE 5. Impact evaluations in education,
tion Results (SABER), aim to produce a rich by country6
body of analytical work that could improve
our understanding of the underpinnings of
100%
progress in education outcomes in develop-
ing countries. China
SABER, in particular, offers system-level
diagnostics on the state of education in de-
India
veloping countries. The diagnostic toolkit en-
ables educators and policymakers to assess
education policies and practices in light of
Mexico
global standards and best practices.
There is some indication that SABER diag- Kenya
nostics are being used to influence country
reforms and dialogue with development Chile
agencies.5 For example, one case study doc-
uments steps that Jordan has taken (based Brazil
upon background information from SABER) Colombia
to strengthen its student assessment sys- South Africa
tems through linking student assessments
Peru
with teacher training and support, as well as 50% Pakistan
disseminating assessment results (Obeidat
Uganda
and Dawani, 2014).
Remaining Countries
0%
Source: 3ie Impact Evaluation Repository.
www.3ieimpact.org/en/evidence/impact-evaluations/
5 See country briefs on the program website: 6 China (81), India (68), Mexico (62), Kenya (61), Chile (33), Brazil
http://saber.worldbank.org/index.cfm?indx=6&sub=5. (32), Colombia (29), South Africa (28), Peru (26), Pakistan/
Uganda (both, 24).
15 TOWARD DATA-DRIVEN EDUCATION SYSTEMS———
In this chapter, we distilled insights from the
available literature to understand how deci-
sion-makers use three sources of evidence—
data from a country’s EMIS, student learning
assessments, and program evaluations—to
inform education policy and practice. The
scorecard is mixed at best: investments in
data generation appear to be outstripping
the use of this information to strengthen
education systems, aside from a few bright
spot examples.
However, there is still much that we do not
know about the data decision-makers use
and the evidence that they want to be more
effective in their jobs. In Chapter 3, we ana-
lyze the results of two recent surveys of edu-
cation decision-makers in order to shed light
on these questions from the perspective of
one important user base: national-level deci-
sion-makers and influencers involved in set-
ting and informing education policy across
public, private, and civil society spheres in
low- and middle-income countries world-
wide.
2. Do investments in education data match use? 163. Identifying data
needs: What do
education decision-
makers want?
To move from data generation to policy and use data in their work, as well as what
impact, we need better intelligence on the would be most helpful to them in the fu-
barriers to evidence use and the types of ture. In this chapter, we analyze these novel
information that decision-makers want. In datasets to answer three key questions: what
the previous chapter, we examined available data are in demand, by whom, and why?
evidence of education data investments and
use, but were left with more questions than We will discuss these findings in more detail
answers. in the remainder of this chapter. The follow-
ing section first provides more background
In 2017, AidData fielded two surveys of na- on the survey data to set the findings in
tional-level policymakers and practitioners context.
in low- and middle-income countries who
shared their experiences of how they source
Top-line findings
• Education decision-makers seldom view evidence as the decisive factor
when weighing the merits of policy decisions, but it does appear to play
a supporting role.
• Education decision-makers consume data from various sources and of
diverse types in their work, with demand outstripping supply when it
comes to program evaluation data.
• Education decision-makers want data to be timely, actionable, disag-
gregated, and locally relevant. To this end, they prioritize strengthening
their countries’ EMIS.
17 TOWARD DATA-DRIVEN EDUCATION SYSTEMSBOX 2. Comparing the surveys
Listening to Leaders Survey. The 2017 Listening to Leaders Survey was sent via email to
policymakers and practitioners knowledgeable about, or directly involved in, development policy
initiatives at any point between 2010 and 2015, in 126 low- and middle-income countries. We
were successfully able to send a survey invitation to roughly 47,000 of the 58,000 individuals
who met our inclusion criteria (about 80 percent). Of the 47,000 people who received an
invitation, 3,500 (7.4 percent) participated in the survey.
Education Snap Poll. We identified a subset of the broader Listening to Leaders sampling frame
of individuals who held positions of relevance to the education sector to use as the sampling
frame for the 2017 Education Snap Poll. Of the 2,000 individuals who were sent the survey
invitation, 180 leaders (9 percent) responded.
Both surveys captured the views of five stakeholder groups. The profile of education sector
respondents of both data sources is broadly similar: around 40 percent are host government
officials and most are from sub-Saharan Africa. Compared to the snap poll, the share of in-
country development partner respondents was smaller in the 2017 Listening to Leaders Survey;
and the corresponding share of CSO/NGO respondents was larger.
For more information on the composition of the survey respondents please see Table A1 and A2
in Appendix A.
About the data to examine the various roles that education
stakeholders perform and their specific data
AidData’s 2017 Listening to Leaders (LtL) Sur- needs.
vey captured the views of nearly 3,500 survey
participants in 126 low- and middle-income The respondents of the two surveys included
countries from 22 policy domains, including representatives from five stakeholder groups:
education. Their insights shed light on the government officials, development partner
broader picture of data and evidence use, as organizations, civil society groups and NGOs,
well as how the education sector is different the private sector, and independent experts.
from other social sectors.7
Given the relatively small sample size for the
Specific to this study, AidData and the Brook- education poll, we primarily draw insights
ings Institution fielded a more targeted regarding the survey respondents overall,
survey of decision-makers in 126 countries. though in some cases we mention differ-
Approximately 180 leaders from 78 countries ences among stakeholder groups. The ma-
responded to the 2017 Education Snap jority of the education poll respondents have
Poll. The poll provides a unique opportunity roles that support policymakers who make
decisions related to these domains. Some
7 The health and education sectors, for example, have a similar also make final decisions related to various
service delivery model, but their investment strategies hint at education activities.
very different views on the value of data. Only three percent
of official development assistance for education is spent on
global public goods such as data and research, compared
with 20 percent for health (Schäferhoff and Burnett, 2016).
3. Identifying data needs: What do education decision-makers want? 18In interpreting the results, it is important to
TABLE 5. Education decisions included in
recognize that the focus in both surveys is the 2017 Snap Poll, by domain
very much on national-level decision-mak-
ers. As such, these data give insight into
what some user groups care about, but not Organization Designing and
the needs and concerns of other groups (e.g., of instruction implementing support
parents, teachers, school-level administra- activities for students
tors, local officials).
Testing, assessing, and/or
credentialing students
What is the role of data and Personnel Hiring and deploying
information in decisions? management teachers or principals
Rather than studying data use in the ab-
Developing careers and
stract, survey respondents offered their
assessing performance of
insights on the types of decisions they make teachers and/or principals
in the education sector and the role infor-
mation plays—among other factors—in that Determining
process. Government officials, for example, compensation for
may allocate resources, determine quality teachers/principals
standards, and hold school administrators
accountable for meeting national targets.8 Resource Budgeting and allocating
Civil society leaders may advocate for more management financial resources for
effective government-run schools or directly education
administer their own programs that are sub-
Ensuring provision of
ject to national standards.
school inputs
Using their survey responses, we can gain
Planning and Designing and defining
insight into the extent to which education
structures programs of study and
data or analysis is a driver of these decisions course content
in practice. Specifically, we asked partici-
pants in the 2017 Education Snap Poll about Creating or closing/
the role of information versus other factors abolishing schools or
in driving ten common education decisions, grades
adapted from the OECD’s Education Sector
at a Glance (2012). We further categorized Planning and developing
strategies
these decisions into four decision-making
domains: (1) organization of instruction; (2)
Source: Decision-making domains adapted from OECD Education
personnel management; (3) resource man- at a Glance (2012)
agement; and (4) planning and structures
(see Table 5).
Leveraging responses from the 2017 Listening
to Leaders Survey, we can also pinpoint the
primary purposes for which decision-makers
and influencers use education data or anal-
8 School administrators supervise teachers, implement school
budgets, and report on student enrollment and progression. ysis in their work. Survey respondents could
It should be noted that while these front-line implementers identify several possible use cases, including:
are an important group of education data users, our survey
results are primarily capturing use patterns of national-level program design, program implementation,
leaders.
advocacy and agenda-setting, capacity
19 TOWARD DATA-DRIVEN EDUCATION SYSTEMSbuilding and technical assistance, monitor- and political support. It should be noted that
ing and evaluation, research and analysis, or this does not necessarily mean that these
external communications. decision-makers view this as the ideal situa-
tion, merely the status quo. That said, these
In the remainder of this section, we analyze results are consistent with prior studies, such
how decision-makers responded to these as that by Bruns and Schneider (2016) which
two questions on the role of information writ show that political considerations have
large and the particular purposes for which stymied education reforms in several Latin
they use education data or analysis. American countries, even when empirical
evidence justifies reforms.
———
There could plausibly be a mutually rein-
Finding 1: Having enough forcing relationship between government
information is seldom the decisive capacity and the perceived importance of
factor in making most education information in decision-making. Shortage
decisions; instead, decision- of staff in national statistical organizations
and ministries and limited capacity for using
makers point to having sufficient
and analyzing data have been reported to be
government capacity. among the most critical constraints to data
In determining whether to enact education use in Honduras, Timor-Leste and Senegal
policies, modify programs or allocate resourc- (Custer & Sethi, 2017).
es, decision-makers assess the anticipated
benefits and costs of the options before Is the availability of sufficient information
them. In the process, they weigh many fac- more critical to some decisions than others?
tors of which data is only one consideration. Leaders place a somewhat higher premi-
um on having enough information when
In the 2017 Education Snap Poll, we asked it comes to decisions such as creating or
policymakers and practitioners to identify abolishing schools or grades, and testing,
the most important factor in making or influ- assessing or credentialing students (Figure
encing ten common education decisions.9 6).10 One possible explanation for this could
Survey participants identified sufficient be that leaders feel that they need stronger
government capacity to implement [policy justification (via an evidence base) for these
or programmatic] changes as the decisive decisions as they could become easily po-
factor for most decisions in the education liticized. Teachers or parents may strongly
sector. disagree with a school closure, for example,
and mobilize dissenting voices.11
So, where does data or analysis fit into the
decision-making process? In Figure 6, we see
that leaders view having sufficient informa-
tion as less consequential in how decisions
10 This result is based on a rather small number of responses,
are made than technical capacity, financing, and therefore should be interpreted with caution, especially
in generalizing the findings to the education sector as a
whole.
11 In addition to the pre-defined activities, respondents could
9 Respondents first selected all the activities they were person- select “other” and write-in responses. These were mapped
ally involved with and then identified the most important to the decision domains as much as possible, but we
factor in making or influencing those decisions. See Figure combined the 15 responses that could not be mapped and
6 for the response options. While respondents could have created a fifth category “other”. Several of these responses
interpreted “sufficient government capacity” in a number of relate to working with education data, statistics, or analysis.
ways, we think it reasonable to interpret this as capacity to It is unsurprising that information (along with public and
implement programs or policies. See Appendix D for the full political support) is the most important factor for this group
Education Snap Poll questionnaire. of respondents.
3. Identifying data needs: What do education decision-makers want? 20FIGURE 6. What is the most important factor influencing decisions in various education activities?
Designing student
activities [27]
Organization of
instruction
Testing or credentialing
students [20]
Hiring teachers or
principals [11]
Personnel
Assessing teacher
management
performance [17]
Determining compensation
for teachers/principals [11]
Budgeting for education
resources [41]
Resource
management
Ensuring provision of
school inputs [17]
Designing programs and
content [31]
Planning and Creating or abolishing
structures schools/grades [10]
Planning and developing
strategies [74]
Enough Sufficient Public Sufficient Another
information financial and political government factor
resources support capacity
Notes: Of the ten activities listed on the left side of this figure, each respondent first selected the activities that s/he was involved in, and then the most
important factor influencing the decisions pertaining to each activity selected. For each activity, the distribution of responses is visualized from left to right.
The total number of responses for each activity is noted in parentheses.
Source: 2017 Education Snap Poll
Finding 2: Education decision- to Leaders Survey across all sectors, Masaki et
makers employ evidence in a al. (2017) observed that, on average, “leaders
supporting role throughout the use evidence more to conduct retrospective
assessments of past performance than to
policymaking process, for both
inform future policy and programs.”
retrospective assessment and
forward-looking activities. Notably, however, decision-makers in the
Information may not be the decisive factor education sector are more likely to use data
in education decisions, but it is still one of and analysis for forward-looking purposes,
the variables that decision-makers consider. such as design and implementation of
When analyzing the results of the Listening policies or programs, compared to the use
21 TOWARD DATA-DRIVEN EDUCATION SYSTEMSFIGURE 7. For what purposes do education decision-makers use information?
Education All sectors
77%
Monitoring and evaluation
72%
76%
Design
63%
76%
Research and analysis
73%
71%
Implementation
65%
71%
Advocacy and agenda-setting
61%
71%
Capacity building/TA
62%
51%
External communications
43%
0% PERCENTAGE OF RESPONDENTS 80%
Note: This figure shows the percentage of respondents in (a) the education sector and (b) all sectors who use evidence for different purposes (n=99 and
n=1769, respectively). Note that the percentages do not add up to 100 because respondents were able to select all applicable response options.
Source: 2017 Listening to Leaders Survey.
patterns observed overall for all sectors. As Nonetheless, education decision-makers are
shown in Figure 7, the majority of education not necessarily monolithic, and we see some
sector decision-makers (over 70 percent) important distinctions between stakeholder
that report using data or analysis, do so fairly groups in how they report using information
consistently throughout the policymaking in their work. Interestingly, considering their
process.12 This appears to reinforce the earlier oversight of vast public-sector education pro-
finding that evidence can play a supporting grams, government officials were less likely
role, even when it is not the major driver of than other stakeholder groups to use data
education decisions. and analysis for program implementation or
monitoring and evaluation (see Figure 8). This
finding may partly reflect the composition
of the survey, which includes national-level
12 The high incidence of use throughout the policymaking
officials, rather than local government repre-
process is also visible in the health sector. In contrast, the sentatives or school administrators.
governance sector used data primarily for two purposes: M&E
(77 percent) and research and analysis (73 percent). Figures
available upon request.
3. Identifying data needs: What do education decision-makers want? 22You can also read
