Using Student Achievement Data to Support Instructional Decision Making - IES PRACTICE GUIDE WHAT WORKS CLEARINGHOUSE

Page created by Geraldine Perry
Using Student Achievement Data to Support Instructional Decision Making - IES PRACTICE GUIDE WHAT WORKS CLEARINGHOUSE

Using Student Achievement Data to
Support Instructional Decision Making

NCEE 2009-4067
Using Student Achievement Data to Support Instructional Decision Making - IES PRACTICE GUIDE WHAT WORKS CLEARINGHOUSE
The Institute of Education Sciences (IES) publishes practice guides in education
to bring the best available evidence and expertise to bear on the types of challenges
that cannot currently be addressed by a single intervention or program. Authors of
practice guides seldom conduct the types of systematic literature searches that are
the backbone of a meta-analysis, although they take advantage of such work when
it is already published. Instead, authors use their expertise to identify the most im-
portant research with respect to their recommendations and conduct a search of
recent publications to ensure that the research supporting the recommendations
is up-to-date.

Unique to IES-sponsored practice guides is that they are subjected to rigorous exter-
nal peer review through the same office that is responsible for independent reviews
of other IES publications. A critical task for peer reviewers of a practice guide is to
determine whether the evidence cited in support of particular recommendations is
up-to-date and that studies of similar or better quality that point in a different di-
rection have not been ignored. Because practice guides depend on the expertise of
their authors and their group decision making, the content of a practice guide is not
and should not be viewed as a set of recommendations that in every case depends
on and flows inevitably from scientific research.

The goal of this practice guide is to formulate specific and coherent evidence-based
recommendations for use by educators and education administrators to create the
organizational conditions necessary to make decisions using student achievement
data in classrooms, schools, and districts. The guide provides practical, clear in-
formation on critical topics related to data-based decision making and is based on
the best available evidence as judged by the panel. Recommendations presented in
this guide should not be construed to imply that no further research is warranted
on the effectiveness of particular strategies for data-based decision making.
Using Student Achievement Data to Support Instructional Decision Making - IES PRACTICE GUIDE WHAT WORKS CLEARINGHOUSE

            Using Student Achievement
            Data to Support Instructional
            Decision Making
            September 2009
            Laura Hamilton (Chair)
            RAND Corporation

            Richard Halverson
            University of Wisconsin–Madison

            Sharnell S. Jackson
            Chicago Public Schools

            Ellen Mandinach
            CNA Education

            Jonathan A. Supovitz
            University of Pennsylvania

            Jeffrey C. Wayman
            The University of Texas   at   Austin

            Cassandra Pickens
            Emily Sama Martin
            Mathematica Policy Research

            Jennifer L. Steele
            RAND Corporation

NCEE 2009-4067
Using Student Achievement Data to Support Instructional Decision Making - IES PRACTICE GUIDE WHAT WORKS CLEARINGHOUSE
This report was prepared for the National Center for Education Evaluation and Re-
gional Assistance, Institute of Education Sciences, under Contract ED-07-CO-0062
by the What Works Clearinghouse, operated by Mathematica Policy Research.

The opinions and positions expressed in this practice guide are the authors’ and do
not necessarily represent the opinions and positions of the Institute of Education Sci-
ences or the U.S. Department of Education. This practice guide should be reviewed
and applied according to the specific needs of the educators and education agency
using it, and with the full realization that it represents the judgments of the review
panel regarding what constitutes sensible practice, based on the research available
at the time of publication. This practice guide should be used as a tool to assist in
decision making rather than as a “cookbook.” Any references within the document to
specific education products are illustrative and do not imply endorsement of these
products to the exclusion of other products that are not referenced.

U.S. Department of Education
Arne Duncan

Institute of Education Sciences
John Q. Easton

National Center for Education Evaluation and Regional Assistance
John Q. Easton
Acting Commissioner

September 2009

This report is in the public domain. While permission to reprint this publication is
not necessary, the citation should be:

Hamilton, L., Halverson, R., Jackson, S., Mandinach, E., Supovitz, J., & Wayman, J.
(2009). Using student achievement data to support instructional decision making
(NCEE 2009-4067). Washington, DC: National Center for Education Evaluation and
Regional Assistance, Institute of Education Sciences, U.S. Department of Education.
Retrieved from

What Works Clearinghouse Practice Guide citations begin with the panel chair,
followed by the names of the panelists listed in alphabetical order.

This report is available on the IES website at and http://ies.

Alternative formats
On request, this publication can be made available in alternative formats, such
as Braille, large print, audiotape, or computer diskette. For more information,
call the Alternative Format Center at 202–205–8113.
Using Student Achievement Data to
Support Instructional Decision Making
Introduction                                                                1

 The What Works Clearinghouse standards and their relevance to this guide   4

Overview                                                                    5

Scope of the practice guide                                                 6

 Status of the research                                                     6

 Summary of the recommendations                                             7

Checklist for carrying out the recommendations                              9

Recommendation 1. Make data part of an ongoing cycle of
instructional improvement                                                   10

Recommendation 2. Teach students to examine their own data and
set learning goals                                                          19

Recommendation 3. Establish a clear vision for schoolwide data use          27

Recommendation 4. Provide supports that foster a data-driven culture
within the school                                                           33

Recommendation 5. Develop and maintain a districtwide data system           39

Glossary of terms as used in this report                                    46

Appendix A. Postscript from the Institute of Education Sciences             49

Appendix B. About the authors                                               52

Appendix C. Disclosure of potential conflicts of interest                   54

Appendix D. Technical information on the studies                            55

References                                                                  66

                                            ( iii )

List of tables
Table 1.   Institute of Education Sciences levels of evidence for practice guides   3

Table 2. Recommendations and corresponding levels of evidence                       8

Table 3. Suggested professional development and training opportunities              37

Table 4. Sample stakeholder perspectives on data system use                         40

Table 5. Considerations for built and purchased data systems                        44

Table D1. Studies cited in recommendation 2 that meet WWC standards
with or without reservations                                                        57

Table D2. Scheduling approaches for teacher collaboration                           61

List of figures
Figure 1. Data use cycle                                                            10

Figure 2. Example of classroom running records performance
at King Elementary School                                                           13

List of examples
Example 1. Examining student data to understand learning                            17

Example 2. Example of a rubric for evaluating five-paragraph essays                 21

Example 3. Example of a student’s worksheet for reflecting on
strengths and weaknesses                                                            23

Example 4. Example of a student’s worksheet for learning from
math mistakes                                                                       24

Example 5. Teaching students to examine data and goals                              25

Example 6. Examples of a written plan for achieving school-level goals              30

                                          ( iv )
Introduction                                     and single subject designs to examine
                                                 whether data use leads to increases in
As educators face increasing pressure            student achievement. Among the studies
from federal, state, and local accountabil-      ultimately relevant to the panel’s recom-
ity policies to improve student achieve-         mendations, only six meet the causal va-
ment, the use of data has become more            lidity standards of the What Works Clear-
central to how many educators evaluate           inghouse (WWC) and were related to the
their practices and monitor students’ aca-       panel’s recommendations.2
demic progress.1 Despite this trend, ques-
tions about how educators should use data        To indicate the strength of evidence sup-
to make instructional decisions remain           porting each recommendation, the panel
mostly unanswered. In response, this             relied on the WWC standards for determin-
guide provides a framework for using stu-        ing levels of evidence, described below and
dent achievement data to support instruc-        in Table 1. It is important for the reader to
tional decision making. These decisions          remember that the level of evidence rating
include, but are not limited to, how to          is not a judgment by the panel on how ef-
adapt lessons or assignments in response         fective each of these recommended prac-
to students’ needs, alter classroom goals        tices will be when implemented, nor is it
or objectives, or modify student-grouping        a judgment of what prior research has to
arrangements. The guide also provides            say about the effectiveness of these prac-
recommendations for creating the orga-           tices. The level of evidence ratings reflect
nizational and technological conditions          the panel’s judgment of the validity of
that foster effective data use. Each rec-        the existing literature to support a causal
ommendation describes action steps for           claim that when these practices have been
implementation, as well as suggestions           implemented in the past, positive effects
for addressing obstacles that may impede         on student academic outcomes were ob-
progress. In adopting this framework, edu-       served. They do not reflect judgments of
cators will be best served by implement-         the relative strength of these positive ef-
ing the recommendations in this guide            fects or the relative importance of the in-
together rather than individually.               dividual recommendations.

The recommendations reflect both the ex-         A strong rating refers to consistent and
pertise of the panelists and the findings        generalizable evidence that an inter-
from several types of studies, including         vention strategy or program improves
studies that use causal designs to examine       outcomes.3
the effectiveness of data use interventions,
case studies of schools and districts that       A moderate rating refers either to evidence
have made data-use a priority, and obser-        from studies that allow strong causal con-
vations from other experts in the field. The     clusions but cannot be generalized with
research base for this guide was identi-         assurance to the population on which a
fied through a comprehensive search for          recommendation is focused (perhaps be-
studies evaluating academically oriented         cause the findings have not been widely
data-based decision-making interventions
and practices. An initial search for litera-     2. Reviews of studies for this practice guide ap-
ture related to data use to support instruc-     plied WWC Version 1.0 standards. See Version 1.0
                                                 standards at
tional decision making in the past 20 years
yielded more than 490 citations. Of these,
                                                 3. Following WWC guidelines, improved out-
64 used experimental, quasi-experimental,        comes are indicated by either a positive, statisti-
                                                 cally significant effect or a positive, substantively
1. Knapp et al. (2006).                          important effect size (i.e., greater than 0.25).


replicated) or to evidence from studies that        that researchers have not yet studied a
are generalizable but have more causal              practice or that there is weak or conflicting
ambiguity than that offered by experi-              evidence of effectiveness. Policy interest in
mental designs (e.g., statistical models of         topics of current study thus can arise be-
correlational data or group comparison de-          fore a research base has accumulated on
signs for which equivalence of the groups           which recommendations can be based.
at pretest is uncertain).
                                                    Under these circumstances, the panel ex-
A low rating refers to evidence either from         amined the research it identified on the
studies such as case studies and descrip-           topic and combined findings from that
tive studies that do not meet the stan-             research with its professional expertise
dards for moderate or strong evidence or            and judgments to arrive at recommenda-
from expert opinion based on reasonable             tions. However, that a recommendation
extrapolations from research and theory.            has a low level of evidence should not be
A low level of evidence rating indicates            interpreted as indicating that the panel
that the panel did not identify a body of           believes the recommendation is unimport-
research demonstrating effects of imple-            ant. The panel has decided that all five rec-
menting the recommended practice on                 ommendations are important and, in fact,
student achievement. The lack of a body of          encourages educators to implement all of
valid evidence may simply mean that the             them to the extent that state and district
recommended practices are not feasible or           resources and capacity allow.
are difficult to study in a rigorous, experi-
mental fashion.4 In other cases, it means

4. For more information, see the WWC Frequently
Asked Questions page for practice guides, http://


Table 1. Institute of Education Sciences levels of evidence for practice guides
                  In general, characterization of the evidence for a recommendation as strong requires both
                  studies with high internal validity (i.e., studies whose designs can support causal conclu-
                  sions) and studies with high external validity (i.e., studies that in total include enough of
                  the range of participants and settings on which the recommendation is focused to sup-
                  port the conclusion that the results can be generalized to those participants and settings).
                  Strong evidence for this practice guide is operationalized as
                  • A systematic review of research that generally meets WWC standards (see http://ies.
             and supports the effectiveness of a program, practice, or approach
    Strong            with no contradictory evidence of similar quality; OR
                  • Several well-designed, randomized controlled trials or well-designed quasi-experi-
                      ments that generally meet WWC standards and support the effectiveness of a program,
                      practice, or approach, with no contradictory evidence of similar quality; OR
                  • One large, well-designed, randomized controlled, multisite trial that meets WWC stan-
                      dards and supports the effectiveness of a program, practice, or approach, with no
                      contradictory evidence of similar quality; OR
                  • For assessments, evidence of reliability and validity that meets the Standards for
                      Educational and Psychological Testing.a

                  In general, characterization of the evidence for a recommendation as moderate requires
                  studies with high internal validity but moderate external validity or studies with high
                  external validity but moderate internal validity. In other words, moderate evidence is
                  derived from studies that support strong causal conclusions but generalization is uncer-
                  tain or studies that support the generality of a relationship but the causality is uncertain.
                  Moderate evidence for this practice guide is operationalized as
                  • Experiments or quasi-experiments generally meeting WWC standards and supporting
                      the effectiveness of a program, practice, or approach with small sample sizes and/
                      or other conditions of implementation or analysis that limit generalizability and no
                      contrary evidence; OR
   Moderate       • Comparison group studies that do not demonstrate equivalence of groups at pretest
                      and, therefore, do not meet WWC standards but that (1) consistently show enhanced
                      outcomes for participants experiencing a particular program, practice, or approach
                      and (2) have no major flaws related to internal validity other than lack of demonstrated
                      equivalence at pretest (e.g., only one teacher or one class per condition, unequal
                      amounts of instructional time, highly biased outcome measures); OR
                  • Correlational research with strong statistical controls for selection bias and for dis-
                      cerning influence of endogenous factors and no contrary evidence; OR
                  • For assessments, evidence of reliability that meets the Standards for Educational and
                      Psychological Testingb but with evidence of validity from samples not adequately rep-
                      resentative of the population on which the recommendation is focused.

                  In general, characterization of the evidence for a recommendation as low means that the
                  recommendation is based on expert opinion derived from strong findings or theories in
      Low         related areas and/or expert opinion buttressed by direct evidence that does not rise to
                  the moderate or strong level. Low evidence is operationalized as evidence not meeting
                  the standards for the moderate or strong level.

a. American Educational Research Association, American Psychological Association, and National Council on
   Measurement in Education (1999).­­­
b. Ibid.


The What Works Clearinghouse                   Following the recommendations and sug-
standards and their relevance to               gestions for carrying out the recommen-
this guide                                     dations, Appendix D presents more in-
                                               formation on the research evidence that
In terms of the levels of evidence indi-       supports each recommendation.
cated in Table 1, the panel relied on WWC
evidence standards to assess the quality       The panel would like to thank Cassandra
of evidence supporting educational pro-        Pickens, Emily Sama Martin, Dr. Jennifer
grams and practices. The WWC evaluates         L. Steele, and Mathematica and RAND staff
evidence for the causal validity of instruc-   members who participated in the panel
tional programs and practices according        meetings, characterized the research find-
to WWC standards. Information about            ings, and drafted the guide. We also appre-
these standards is available at http://ies.    ciate the help of the many WWC reviewers              who contributed their time and expertise
standards.pdf. The technical quality of        to the review process, and Sarah Wissel for
each study is rated and placed into one of     her support of the intricate logistics of the
three categories:                              project. In addition, we would like to thank
                                               Scott Cody, Kristin Hallgren, Dr. Shannon
•   Meets Evidence Standards for random-       Monahan, and Dr. Mark Dynarski for their
    ized controlled trials and regression      oversight and guidance during the devel-
    discontinuity studies that provide the     opment of the practice guide.
    strongest evidence of causal validity.
                                                                      Dr. Laura Hamilton
•   Meets Evidence Standards with Res-                            Dr. Richard Halverson
    ervations for all quasi-experimental                   Ms. Sharnell S. Jackson, Ed.M.
    studies with no design flaws and ran-                            Dr. Ellen Mandinach
    domized controlled trials that have                         Dr. Jonathan A. Supovitz
    problems with randomization, attri-                            Dr. Jeffrey C. Wayman
    tion, or disruption.

•   Does Not Meet Evidence Screens for
    studies that do not provide strong evi-
    dence of causal validity.

Using Student                                     progress is a logical way to monitor con-
Achievement Data to                               tinuous improvement and tailor instruc-
                                                  tion to the needs of each student. Armed
Support Instructional                             with data and the means to harness the
Decision Making                                   information data can provide, educators
                                                  can make instructional changes aimed at
                                                  improving student achievement, such as:
                                                  •   prioritizing instructional time;8
Recent changes in accountability and test-
ing policies have provided educators with         •   targeting additional individual instruc-
access to an abundance of student-level               tion for students who are struggling
data, and the availability of such data has           with particular topics;9
led many to want to strengthen the role of
data for guiding instruction and improving        •   more easily identifying individual stu-
student learning. The U.S. Department of              dents’ strengths and instructional in-
Education recently echoed this desire, call-          terventions that can help students
ing upon schools to use assessment data to            continue to progress;10
respond to students’ academic strengths
and needs.5 In addition, spurred in part          •   gauging the instructional effectiveness
by federal legislation and funding, states            of classroom lessons;11
and districts are increasingly focused on
building longitudinal data systems.6              •   refining instructional methods;12 and

Although accountability trends explain            •   examining schoolwide data to consider
why more data are available in schools,               whether and how to adapt the curricu-
the question of what to do with the data re-          lum based on information about stu-
mains primarily unanswered. Data provide              dents’ strengths and weaknesses.13
a way to assess what students are learn-
ing and the extent to which students are
making progress toward goals. However,
making sense of data requires concepts,           8. Brunner et al. (2005).
theories, and interpretative frames of ref-       9. Brunner et al. (2005); Supovitz and Klein
                                                  (2003); Wayman and Stringfield (2006).
erence.7 Using data systematically to ask
                                                  10. Brunner et al. (2005); Forman (2007); Wayman
questions and obtain insight about student
                                                  and Stringfield (2006).
                                                  11. Halverson, Prichett, and Watson (2007);
5. American Recovery and Reinvestment Act         Supovitz and Klein (2003).
of 2009; U.S. Department of Education (2009);     12. Halverson, Prichett, and Watson (2007);
Obama (2009).                                     Fiarman (2007).
6. Aarons (2009).                                 13. Marsh, Pane, and Hamilton (2006); Kerr
7. Knapp et al. (2006).                           et al. (2006).

Scope of the                                               these are administered consistently
practice guide                                             and routinely to provide information
                                                           that can be compared across class-
                                                           rooms or schools.
The purpose of this practice guide is to
help K–12 teachers and administrators use              Annual and interim assessments vary con-
student achievement data to make instruc-              siderably in their reliability and level of
tional decisions intended to raise student             detail, and no single assessment can tell
achievement. The panel believes that the               educators all they need to know to make
responsibility for effective data use lies             well-informed instructional decisions. For
with district leaders, school administrators,          this reason, the guide emphasizes the use of
and classroom teachers and has crafted the             multiple data sources and suggests ways to
recommendations accordingly.                           use different types of common assessment
                                                       data to support and inform decision mak-
This guide focuses on how schools can make             ing. The panel recognizes the value of class-
use of common assessment data to improve               room-specific data sources, such as tests or
teaching and learning. For the purpose of              other student work, and the guide provides
this guide, the panel defined common as-               suggestions for how these data can be used
sessments as those that are administered               to inform instructional decisions.
in a routine, consistent manner by a state,
district, or school to measure students’ aca-          The use of data for school management
demic achievement.14 These include                     purposes, rewarding teacher performance,
                                                       and determining appropriate ways to
•   annual statewide accountability tests              schedule the school day is beyond the
    such as those required by No Child                 scope of this guide. Schools typically col-
    Left Behind;                                       lect data on students’ attendance, behav-
                                                       ior, activities, coursework, and grades, as
•   commercially produced tests—includ-                well as a range of administrative data con-
    ing interim assessments, benchmark                 cerning staffing, scheduling, and financ-
    assessments, or early-grade reading                ing. Some schools even collect perceptual
    assessments—administered at mul-                   data, such as information from surveys or
    tiple points throughout the school                 focus groups with students, teachers, par-
    year to provide feedback on student                ents, or community members. Although
    learning;                                          many of these data have been used to
                                                       help inform instructional decision making,
•   end-of-course tests administered                   there is a growing interest among educa-
    across schools or districts; and                   tors and policy advocates in drawing on
                                                       these data sources to increase operational
•   interim tests developed by districts               efficiency inside and outside of the class-
    or schools, such as quarterly writing              room. This guide does not suggest how
    or mathematics prompts, as long as                 districts should use these data sources to
                                                       implement data-informed management
                                                       practices, but this omission should not be
14. The panel recognizes that some schools do
not fall under a district umbrella or are not part     construed as a suggestion that such data
of a district. For the purposes of this guide, dis-    are not valuable for decision making.
trict is used to describe schools in partnership,
which could be either a school district or a collab-   Status of the research
orative organization of schools. Technical terms
related to assessments, data, and data-based de-
cision making are defined in a glossary at the end     Overall, the panel believes that the ex-
of the recommendations.                                isting research on using data to make
Scope of the practice guide

instructional decisions does not yet pro-          research that proves the practices do im-
vide conclusive evidence of what works to          prove student achievement.
improve student achievement. There are a
number of reasons for the lack of compel-          Summary of the recommendations
ling evidence. First, rigorous experimental
studies of some data-use practices are dif-        The recommendations in this guide create
ficult or infeasible to carry out. For exam-       a framework for effectively using data to
ple, it would be impractical to structure a        make instructional decisions. This frame-
rigorous study investigating the effects of        work should include a data system that
implementing a districtwide data system            incorporates data from various sources,
(recommendation 5) because it is difficult         a data team in schools to encourage the
to establish an appropriate comparison             use and interpretation of data, collabora-
that reflects what would have happened in          tive discussion sessions among teachers
the absence of that system. Second, data-          about data use and student achievement,
based decision making is closely tied to           and instruction for students about how to
educational technology. As new technolo-           use their own achievement data to set and
gies are developed, there is often a lag           monitor educational goals. A central mes-
before rigorous research can identify the          sage of this practice guide is that effective
impacts of those technologies. As a result,        data practices are interdependent among
there is limited evidence on the effective-        the classroom, school, and district levels.
ness of the state-of-the-art in data-based         Educators should become familiar with all
decision making. Finally, studies of data-         five recommendations and collaborate with
use practices generally look at a bundle of        other school and district staff to implement
elements, including training teachers on           the recommendations concurrently, to the
data use, data interpretation, and utiliz-         extent that state and district resources and
ing the software programs associated with          capacity allow. However, readers who are
data analysis and storage. Studies typi-           interested in implementing data-driven
cally do not look at individual elements,          recommendations in the classroom should
making it difficult to isolate a specific ele-     focus on recommendations 1 and 2. Read-
ment’s contribution to effective use of data       ers who wish to implement data-driven
to make instructional decisions designed           decision making at the school level should
to improve student achievement.                    focus on recommendations 3 and 4. Read-
                                                   ers who wish to bolster district data sys-
This guide includes five recommendations           tems to support data-driven decision mak-
that the panel believes are a priority to im-      ing should focus on recommendation 5.
plement. However, given the status of the          Finally, readers interested in technical in-
research, the panel does not have compel-          formation about studies that the panel used
ling evidence that these recommendations           to support its recommendations will find
lead to improved student outcomes. As a            such information in Appendix D.
result, all of the recommendations are sup-
ported by low levels of evidence. While the        To account for the context of each school
evidence is low, the recommendations re-           and district, this guide offers recommen-
flect the panel’s best advice—informed by          dations that can be adjusted to fit their
experience and research—on how teachers            unique circumstances. Examples in this
and administrators can use data to make            guide are intended to offer suggestions
instructional decisions that raise student         based on the experiences of schools and
achievement. In other words, while this            the expert opinion of the panel, but they
panel of experts believes these practices          should not be construed as the best or only
will lead to improved student achieve-             ways to implement the guide’s recommen-
ment, the panel cannot point to rigorous           dations. The recommendations, described
Scope of the practice guide

Table 2. Recommendations and corresponding levels of evidence

 Recommendation                                                            Level of evidence

 1. Make data part of an ongoing cycle of instructional improvement               Low

 2. Teach students to examine their own data and set learning goals               Low

 3. Establish a clear vision for schoolwide data use                              Low

 4. Provide supports that foster a data-driven culture within the school          Low

 5. Develop and maintain a districtwide data system                               Low

Source: Authors’ compilation based on analysis described in text.

here briefly, also are listed with their levels     on the organizational and technological
of evidence in Table 2.                             conditions that support data use. Recom-
                                                    mendation 3 suggests that school leaders
Recommendations 1 and 2 emphasize the               establish a comprehensive plan for data
use of data to inform classroom-level in-           use that takes into account multiple per-
structional decisions. Recommendation 1             spectives. It also emphasizes the need to
suggests that teachers use data from multi-         establish organizational structures and
ple sources to set goals, make curricular and       practices that support the implementation
instructional choices, and allocate instruc-        of that plan.
tional time. It describes the data sources
best suited for different types of instruc-         The panel believes that effective data use
tional decisions and suggests that the use          depends on supporting educators who are
of data be part of a cycle of instructional         using and interpreting data. Recommenda-
inquiry aimed at ongoing instructional im-          tion 4 offers suggestions about how schools
provement. Building on the use of data to           and districts can prepare educators to use
drive classroom-based instructional deci-           data effectively by emphasizing the impor-
sions, recommendation 2 provides guidance           tance of collaborative data use. These col-
about how teachers can instruct students in         laboration efforts can create or strengthen
using their own assessment data to develop          shared expectations and common practices
personal achievement goals and guide learn-         regarding data use throughout a school.
ing. Teachers then can use these goals to
better understand factors that may motivate         Recommendation 5 points out that effec-
student performance and can adjust their            tive, sustainable data use requires a se-
instruction accordingly.                            cure and reliable data-management system
                                                    at the district level. It provides detailed
The panel believes that effective data use          suggestions about how districts or other
at the classroom level is more likely to            educational entities, such as multidistrict
emerge when it is supported by a data-              collaboratives or charter management or-
informed school and district culture. Rec-          ganizations, should develop and maintain
ommendations 3, 4, and 5, therefore, focus          a high-quality data system.
Checklist for carrying out the                   Recommendation 4. Provide supports
recommendations                                  that foster a data-driven culture within
                                                 the school
Recommendation 1. Make data part
of an ongoing cycle of instructional                 Designate a school-based facilitator
improvement                                      who meets with teacher teams to discuss
   Collect and prepare a variety of data
about student learning.                              Dedicate structured time for staff
   Interpret data and develop hypotheses
about how to improve student learning.             Provide targeted professional devel-
                                                 opment regularly.
    Modify instruction to test hypotheses
and increase student learning.                   Recommendation 5. Develop and
                                                 maintain a districtwide data system
Recommendation 2. Teach students
to examine their own data and set                    Involve a variety of stakeholders in
learning goals                                   selecting a data system.

     Explain expectations and assessment           Clearly articulate system require-
criteria.                                        ments relative to user needs.

    Provide feedback to students that               Determine whether to build or buy
is timely, specific, well formatted, and         the data system.
                                                     Plan and stage the implementation of
   Provide tools that help students learn       the data system.
from feedback.

    Use students’ data analyses to guide
instructional changes.

Recommendation 3. Establish a clear
vision for schoolwide data use

    Establish a schoolwide data team that
sets the tone for ongoing data use.

   Define critical teaching and learning

    Develop a written plan that articulates
activities, roles, and responsibilities.

    Provide ongoing data leadership.

Recommendation 1.                                    Figure 1. Data use cycle
Make data part of
an ongoing cycle
                                                            Collect and
of instructional                                         prepare a variety                    Interpret data
                                                          of data about                        and develop
improvement                                              student learning                   hypotheses about
                                                                                             how to improve
                                                                                             student learning

Teachers should adopt a systematic
process for using data in order to bring
evidence to bear on their instructional                                      Modify
decisions and improve their ability to                                instruction to test
meet students’ learning needs. The                                      hypotheses and
                                                                       increase student
process of using data to improve                                            learning
instruction, the panel believes, can be
understood as cyclical (see Figure 1).
It includes a step for collecting and                Because the data-use process is
preparing data about student learning                cyclical, teachers actually can begin at
from a variety of relevant sources,                  any point shown in Figure 1—that is,
including annual, interim, and classroom             with a hypothesis they want to test,
assessment data.15 After preparing                   an instructional modification they
data for examination, teachers                       want to evaluate, or a set of student
should interpret the data and develop                performance data they want to use
hypotheses about factors contributing                to inform their decisions. However,
to students’ performance and the                     the panel has observed that teachers
specific actions they can take to meet               are sometimes asked to use existing
students’ needs. Teachers then should                student assessment data without
test these hypotheses by implementing                receiving clear guidance on how to
changes to their instructional practice.             do so. Consequently, some teachers
Finally, they should restart the cycle by            may find it useful to begin with the
collecting and interpreting new student              collection and preparation of data
performance data to evaluate their own               from a variety of sources, and this
instructional changes.16                             guide presents that as the first step
                                                     in the process. Also, although the
                                                     steps represent the ongoing nature
                                                     of the cycle, teachers may find that
                                                     they need a considerable amount of
15. Halverson, Prichett, and Watson (2007), Her-     data collection and interpretation to
man and Gribbons (2001), Huffman and Kalnin          form strong hypotheses about how
(2003), and Fiarman (2007) outline these com-        to change their instruction.
ponents (in varied order) in their case studies
of how the inquiry process was implemented in
some school and district settings. Similarly, Ab-    Level of evidence: Low
bott (2008) discusses using data to assess, plan,
implement, and evaluate instructional changes as     The panel drew on a group of qualitative
part of a larger framework schools should use to     and descriptive studies to formulate this rec-
achieve accountability. Further detail under each
component is based on panelist expertise.            ommendation, using the studies as sources
16. Abbott (2008); Brunner et al. (2005); Halv-
                                                     of examples for how an inquiry cycle for
erson, Prichett, and Watson (2007); Kerr et al.      data use can be implemented in an educa-
(2006); Liddle (2000); Mandinach et al. (2005).      tional setting. No literature was located that
                                                ( 10 )
Recommendation 1. Make data part of an ongoing cycle of instructional improvement

assesses the impact on student achievement        Each assessment type has advantages and
of using an inquiry cycle, or individual steps    limitations (e.g., high-stakes accountability
within that cycle, as a framework for data        tests may be subject to score inflation and
analysis, however, and the panel determined       may lead to perverse incentives).18 There-
that the level of evidence to support this        fore, the panel believes that multiple data
recommendation is low.                            sources are important because no single
                                                  assessment provides all the information
Brief summary of evidence to                      teachers need to make informed instruc-
support the recommendation                        tional decisions. For instance, as teachers
                                                  begin the data-use process for the first time
The panel considers the inquiry cycle of          or begin a new school year, the accessibil-
gathering data, developing and testing hy-        ity and high-stakes importance of students’
potheses, and modifying instruction to be         statewide, annual assessment results pro-
fundamental when using assessment data            vide a rationale for looking closely at these
to guide instruction. Although no causal          data. Moreover, these annual assessment
evidence is available to support the effective-   data can be useful for understanding broad
ness of this cycle, the panel draws on studies    areas of relative strengths and weaknesses
that did not use rigorous designs for exam-       among students, for identifying students or
ples of the three-point cycle of inquiry—the      groups of students who may need particu-
underlying principle of this recommenda-          lar support,19 and for setting schoolwide,20
tion—and provides some detail on the con-         classroom, grade-level, or department-level
text for those examples in Appendix D.            goals for students’ annual performance.

How to carry out this                             However, teachers also should recognize
recommendation                                    that significant time may have passed
                                                  between the administration of these an-
1. Collect and prepare a variety of data about    nual assessments and the beginning of
student learning.                                 the school year, and students’ knowledge
                                                  and skills may have changed during that
To gain a robust understanding of stu-            time. It is important to gather additional
dents’ learning needs, teachers need to           information at the beginning of the year to
collect data from a variety of sources.           supplement statewide test results. In addi-
Such sources include but are not limited          tion, the panel cautions that overreliance
to annual state assessments, district and         on a single data source, such as a high-
school assessments, curriculum-based as-          stakes accountability test, can lead to the
sessments, chapter tests, and classroom           overalignment of instructional practices
projects. In most cases, teachers and their       with that test (sometimes called “teaching
schools already are gathering these kinds         to the test”), resulting in false gains that
of data, so carrying out data collection de-      are not reflected on other assessments of
pends on considering the strengths, limita-       the same content.21
tions, and timing of each data type and on
preparing data in a format that can reveal
                                                  Kalnin (2003); Lachat and Smith (2005); Supo-
patterns in student achievement. More-
                                                  vitz (2006).
over, by focusing on specific questions
                                                  18. Koretz (2003); Koretz and Barron (1998).
about student achievement, educators can
                                                  19. Halverson, Prichett, and Watson (2007); Her-
prioritize which types of data to gather to       man and Gribbons (2001); Lachat and Smith
inform their instructional decisions.17           (2005); Supovitz and Klein (2003); Wayman and
                                                  Stringfield (2006).
17. Bigger (2006); Cromey and Hanson (2000);      20. Halverson, Prichett, and Watson (2007).
Herman and Gribbons (2001); Huffman and           21. Hamilton (2003); Koretz and Barron (1998).

                                              ( 11 )
Recommendation 1. Make data part of an ongoing cycle of instructional improvement

To gain deeper insight into students’ needs       improved after a unit spent reading and
and to measure changes in students’ skills        analyzing expository writing.
during the academic year, teachers also
can collect and prepare data from interim         Finally, it is important to collect and prepare
assessments that are administered consis-         classroom performance data for examina-
tently across a district or school at regular     tion, including examples and grades from
intervals throughout the year (see the box        students’ unit tests, projects, classwork, and
below).22 As with annual assessments, in-         homework. The panel recommends using
terim assessment results generally have           these classroom-level data sources, in con-
the advantage of being comparable across          junction with widely accessible nonachieve-
classrooms, but the frequency of their ad-        ment data such as attendance records and
ministration means that teachers can use          cumulative files,23 to interpret annual and
the data to evaluate their own instructional      interim assessment results (see the box on
strategies and to track the progress of their     page 13). An important advantage of these
current students in a single school year. For     data sources is that in most cases, they can
instance, data from a districtwide interim        be gathered quickly to provide teachers with
assessment could help illuminate whether          immediate feedback about student learning.
the students who were struggling to con-          Depending on the assignment in question,
vert fractions to decimals improved after         they also can provide rich, detailed exam-
receiving targeted small group instruction,       ples of students’ academic performance,
or whether students’ expository essays            thereby complementing the results of an-
                                                  nual or interim tests. For example, if state
                                                  and interim assessments show that students
  Characteristics of interim
                                                  have difficulty writing about literature, then
  assessments                                     examination of students’ analytic essays,
  • Administered routinely (e.g., each            book reports, or reading-response journals
    semester, quarter, or month)                  can illuminate how students are accustomed
    throughout a school year                      to writing about what they read and can sug-
                                                  gest areas in which students need additional
  • Administered in a consistent                  guidance.24 An important disadvantage of
    manner across a particular grade              classroom-level data is that the assignments,
    level and/or content area within              conditions, and scores are not generally
    a school or district                          comparable across classrooms. However,
                                                  when teachers come together to examine
  • May be commercial or developed                students’ work, this variability also can be
    in-house                                      an advantage, since it can reveal discrepan-
                                                  cies in expectations and content coverage
  • May be administered on paper                  that teachers can take steps to remedy.
    or on a computer
                                                  As teachers prepare annual, interim,
  • May be scored by a computer                   and classroom-level data for analysis,
                                                  they should represent the information in
    or a person

                                                  23. The following studies provide examples of
                                                  available data sources: Owings and Follo (1992);
22. Standards for testing in educational envi-    Halverson, Prichett, and Watson (2007); Jones
ronments are discussed in more detail in Amer-    and Krouse (1988); Supovitz and Klein (2003);
ican Educational Research Association (AERA),     Supovitz and Weathers (2004); Wayman and
American Psychological Association (APA), and     Stringfield (2006).
National Council on Measurement in Education      24. This example is drawn and adapted from a
(NCME) (1999).                                    case study by Fiarman (2007).

                                             ( 12 )
Recommendation 1. Make data part of an ongoing cycle of instructional improvement

  Examples of classroom and                   progress on the interim math assessments
  other data                                  throughout the year. On the graph, she
                                              might create separate lines for students
  • Curriculum-based unit tests               from each performance quartile on the
                                              previous year’s state mathematics assess-
  • Class projects                            ment (see Figure 2). Such a graph would
                                              allow her to compare the growth trajec-
  • Classwork and homework                    tories for each group, although she would
                                              need to be certain that each quartile group
  • Attendance records                        contained numerous students, thereby en-
                                              suring that results were not driven by one
  • Records from parent meetings              or two outliers. (Some data systems will
    and phone calls                           include features that make graphing easier
                                              and more automatic. See recommendation
  • Classroom behavior charts                 5 for more information on data systems.)

  • Individualized educational plans          In general, preparing state and district data
    (IEPs)                                    for analysis will be easier for teachers who
                                              have access to the kind of districtwide data
  • Prior data from students’ cumula-         systems described in recommendation 5,
    tive folders                              although these teachers still will need to
                                              maintain useful records of classroom-level
                                              data. Online gradebooks that allow teach-
aggregate forms that address their own        ers to prepare aggregate statistics by class-
questions and highlight patterns of in-       room, content area, or assignment type can
terest. For instance, if a teacher wanted     be useful for identifying patterns in stu-
to use four waves of interim test data to     dents’ classroom-level performance and for
learn whether students who started the        identifying students whose classwork per-
year with weaker mathematics skills were      formance is inconsistent with their perfor-
narrowing the gap with their peers, she       mance on annual or interim assessments.
could make a line graph tracking students’

Figure 2. Example of classroom running records performance at King Elementary School

Source: Supovitz and Klein (2003).

                                          ( 13 )
Recommendation 1. Make data part of an ongoing cycle of instructional improvement

2. Interpret data and develop hypotheses            source of the discrepancy. In all cases, they
about how to improve student learning.              should use classroom and other data to
                                                    shed light on the particular aspects of the
Working independently or in teams, teach-           skill with which students need extra help.
ers should interpret the data they have
collected and prepared. In interpreting             As they triangulate data from multiple
the data, one generally useful objective            sources, teachers should develop hypoth-
is to identify each class’s overall areas           eses about ways to improve the achieve-
of relative strengths and weaknesses so             ment patterns they see in the data. As the
that teachers can allocate instructional            box on page 15 explains, good hypoth-
time and resources to the content that is           eses emerge from existing data, identify
most pressing. Another useful objective is          instructional or curricular changes likely
to identify students’ individual strengths          to improve student learning, and can be
and weaknesses so that teachers can adapt           tested using future assessment data. For
their assignments, instructional methods,           example, existing data can reveal places in
and feedback in ways that address those             which the school’s curriculum is not well
individual needs. For instance, teachers            aligned with state standards. In those situ-
may wish to adapt students’ class project           ations, teachers might reasonably hypoth-
assignments in ways that draw on stu-               esize that reorganizing the curriculum to
dents’ individual strengths while encour-           address previously neglected material will
aging them to work on areas for growth.             improve students’ mastery of the standards.
                                                    In other cases, teachers may hypothesize
To gain deeper insight into students’ learn-        that they need to teach the same content in
ing needs, teachers should examine evi-             different ways. Taking into account how they
dence from the multiple data sources they           and their colleagues have previously taught
prepared in action step 1.25 “Triangulation”        particular skills can help teachers choose
is the process of using multiple data sources       among plausible hypotheses. For instance,
to address a particular question or problem         teachers may find that students have diffi-
and using evidence from each source to              culty identifying the main idea of texts they
illuminate or temper evidence from the              read. This weak student performance may
other sources. It also can be thought of as         lead teachers to hypothesize that the skill
using each data source to test and confirm          should be taught differently. In talking to
evidence from the other sources in order            other teachers, they might choose a differ-
to arrive at well-justified conclusions about       ent teaching strategy, such as a discussion
students’ learning needs. When multiple             format in which students not only identify
data sources (e.g., results from the annual         the main idea of a text but also debate its
state assessment and district interim as-           evidence and merits.
sessment) show similar areas of student
strength and weakness (as in Example 1),            To foster such sharing of effective practices
teachers can be more confident in their             among teachers, the panel recommends
decisions about which skills to focus on.           that teachers interpret data collaboratively
In contrast, when one test shows students           in grade-level or department-specific teams.
struggling in a particular skill and another        In this way, teachers can begin to adopt
test shows them performing well in that             some common instructional and assess-
skill, teachers need to look closely at the         ment practices as well as common expec-
items on both tests to try to identify the          tations for student performance.26 Col-
                                                    laboration also allows teachers to develop
25. Halverson, Prichett, and Watson (2007); Her-
man and Gribbons (2001); Lachat and Smith           26. Fiarman (2007); Halverson, Prichett, and Wat-
(2005); Wayman and Stringfield (2006).              son (2007); Halverson et al. (2007).

                                               ( 14 )
Recommendation 1. Make data part of an ongoing cycle of instructional improvement

a collective understanding of the needs of        3. Modify instruction to test hypotheses and
individual students in their school, so that      increase student learning.
they can work as an organization to provide
support for all students.                         After forming hypotheses about students’
                                                  learning needs, teachers must test their
                                                  hypotheses by carrying out the instruc-
                                                  tional changes that they believe are likely
  Forming testable hypotheses                     to raise student achievement. The kinds
                                                  of changes they choose to implement may
  Situation: Based on data from your 3rd-
                                                  include—but are not limited to—one or
  grade class’s assignments and assess-
                                                  more of the following:
  ments, it appears that more than half
  of the students struggle with subtrac-
                                                  •    allocating more time for topics with
  tion. As their teacher, you ask yourself
                                                       which students are struggling;
  how they can better master subtraction
  skills. To answer this question, you hy-        •    reordering the curriculum to shore up
  pothesize that the students’ subtraction             essential skills with which students are
  skills might improve if they were taught             struggling;
  to use the “trade first” method for sub-
  traction, in which students do their re-        •    designating particular students to re-
  grouping from the tens to ones column                ceive additional help with particu-
  at the beginning, rather than at the end,            lar skills (i.e., grouping or regrouping
  of the problem. You determine that this              students);
  hypothesis can be tested by (1) working
  with these students in a group to teach         •    attempting new ways of teaching dif-
  them the trade first method and (2) ex-              ficult or complex concepts, especially
  amining changes in their subtraction                 based on best practices identified by
  scores on the interim assessment.                    teaching colleagues;

  Characteristics of testable                     •    better aligning performance expecta-
  hypotheses                                           tions among classrooms or between
                                                       grade levels; and/or
  • Identify a promising interven-
    tion or instructional modification            •    better aligning curricular emphasis
    (teaching the trade first method for               among grade levels.
    subtraction) and an effect that you
    expect to see (improvement in                 If the instructional modification was not
    the subtraction skills of struggling          developed collaboratively, teachers may
    students)                                     nonetheless find it useful to seek feedback
                                                  from peers before implementing it. This
  • Ensure that the effect can be mea-            is particularly true if teachers have cho-
    sured (students’ subtraction scores           sen to enact a large instructional change,
    on the interim assessment after               such as a comprehensive new approach
    they learn the trade first strategy)          to algebra instruction or a reorganization
                                                  of the mathematics curriculum sequence.
  • Identify the comparison data (stu-            Because curricular decisions are some-
    dents’ subtraction scores on the in-          times made at the school or district level,
    terim assessment before they were             teachers may even want to make a case for
    taught the strategy)                          curriculum reorganization with school or
                                                  district leaders ahead of time.
                                              ( 15 )
Recommendation 1. Make data part of an ongoing cycle of instructional improvement

The time it takes teachers to carry out their    should give themselves and their students
instructional changes will depend in part        time to adapt to it.28
on the complexity of the changes. If teach-
ers are delivering a discrete lesson plan or     Potential roadblocks and solutions
a series of lessons, then the change usually
can be carried out quickly. Larger interven-     Roadblock 1.1. Teachers have so much
tions take longer to roll out than smaller       data that they are not sure where they
ones. For instance, a teacher whose inter-       should focus their attention in order to raise
vention involves introducing more collab-        student achievement.
orative learning into the classroom may
need time to teach her students to work          Suggested Approach. Teachers can nar-
efficiently in small group settings.             row the range of data needed to solve a
                                                 particular problem by asking specific ques-
During or shortly after carrying out an in-      tions and concretely identifying the data
structional intervention, teachers should        that will answer those questions. In ad-
take notes on how students responded and         dition, administrators can guide this pro-
how they as teachers might modify deliv-         cess by setting schoolwide goals that help
ery of the intervention in future classes.       clarify the kinds of data teachers should be
These notes may not only help teachers           examining and by asking questions about
reflect on their own practice but also pre-      how classroom practices are advancing
pare them to share their experiences and         those goals. For instance, if administrators
insights with other teachers.                    have asked teachers to devote particular
                                                 effort to raising students’ reading achieve-
To evaluate the effectiveness of the in-         ment, teachers may decide to focus atten-
structional intervention, teachers should        tion on evidence from state, interim, and
return to action step 1 by collecting and        classroom assessments about students’
preparing a variety of data about student        reading needs. Teachers should then tri-
learning. For instance, they can gather          angulate data from multiple sources (as
classroom-level data, such as students’          described earlier) to develop hypotheses
classwork and homework, to quickly eval-         about instructional changes likely to raise
uate student performance after the inter-        student achievement. Note that recommen-
vention.27 Teachers can use data from later      dation 3 describes how administrators, data
interim assessments, such as a quarterly         facilitators, and other staff can help teach-
district test, to confirm or challenge their     ers use data in ways that are clearly aligned
immediate, classroom-level evidence.             with the school’s medium- and long-term
                                                 student achievement goals. Also, recom-
Finally, after triangulating data and con-       mendation 4 describes how professional
sidering the extent to which student learn-      development and peer collaboration can
ing did or did not improve in response           help teachers become more adept at data
to the intervention, teachers can decide         preparation and triangulation.
whether to keep pursuing the approach
in its current form, modify or extend the        Roadblock 1.2. Some teachers work in a
approach, or try a different approach alto-      grade level or subject area (such as early
gether. It is important to bear in mind that     elementary and advanced high school
not all instructional changes bear fruit im-     grades) or teach certain subjects (such as
mediately, so before discarding an instruc-      social studies, music, science, or physical
tional intervention as ineffective, teachers     education) for which student achievement
                                                 data are not readily available.

27. Forman (2007).                               28. Elmore (2003).

                                            ( 16 )
Recommendation 1. Make data part of an ongoing cycle of instructional improvement

Example 1. Examining student data to understand learning
 Consider this hypothetical example . . . When the 4th- and 5th-grade
 teachers at Riverview Elementary School met after school in Septem-
 ber for their first data meeting of the year, the data facilitator, Mr.
 Bradley, shared selected data about how students had performed on         Action Step 1
 the previous year’s standards-based state accountability test. Teach-
 ers quickly saw that in both grades, students’ proficiency rates were
 higher in language arts than in mathematics, so they decided to look
 more closely at particular mathematics skills. Examining the results
 on each math content strand, the teachers found that although stu-
 dents were performing adequately in arithmetic, they struggled with
 geometry skills concerning shapes and measurement. This news was          Action Step 2
 surprising because, consistent with state standards, teachers taught
 shapes and measurement in both the 4th and 5th grades.
 Because students had already taken their first district-based interim
 assessment of the school year, the teachers also were able to use
 the district’s data system to look at how students had performed in       Action Step 1
 geometry on that assessment. Studying one graph, Ms. Irving, a 4th-
 grade teacher, observed that the content strand with which students
 struggled most was measuring perimeters of polygons. Since calculat-
 ing perimeters was a matter of adding, and students had performed
 well on the addition strands of both the annual and interim tests, the
 teachers were perplexed. They decided to collect new data on students’    Action Step 2
 geometry skills using questions from the supplemental workbooks of
 their standards-based math curriculum.
 When teachers brought their students’ workbook responses to the next
 data meeting, they gathered in small groups to examine the students’
 work and generate hypotheses. As they shared the classwork exam-
 ples, they noticed a pattern. Students performed well on simple pe-
 rimeter problems when the shapes were drawn for them, but on word
 problems that required them to combine shapes before adding, they         Action Step 2
 largely faltered. The teachers hypothesized that students’ difficulties
 were not with calculating perimeters, but with considering when and
 how to combine polygons in response to real-world problems. They
 further hypothesized that students would benefit from opportunities
 to apply basic geometry skills to novel situations.
 Working together in grade-level teams, the teachers devised tasks for
 their students that would require them to use manipulatives and on-
 line interactive simulations to solve perimeter problems about floor      Action Step 3
 plans and land use. The teachers agreed to deliver these lessons in
 their classrooms and report back on how the students responded.
 At the next data meeting, teachers brought implementation notes and
 samples of student work from the hands-on perimeter lessons. Most
                                                                           Action Step 1
 reported that students were engaged in the lessons but needed addi-
 tional practice. After readministering similar lessons two weeks later,
 most teachers found that their students were getting the hang of the
 task. On the next interim assessment, teachers were pleased to learn
 that the percentage of perimeter and area questions answered correctly    Action Step 2
 had increased from 40 percent to 70 percent across the two grades.

                                           ( 17 )
You can also read