Developing new Drumcondra tests in paper and digital formats: Challenges and opportunities
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Developing new
Drumcondra tests
in paper and digital formats:
Challenges and
opportunities
Overview
1. Why develop new tests?
2. Considerations during test development
3. Standardising tests on two modes
- paper booklets
- computer-based DOTS
4. Demonstration of how the DOTS works
5. Questions
Why develop new
tests?
• “Norm drift”
• National improvement in standards
• Revision of curriculum; recent research
• Engagement: child-friendly design
• Inclusion and diversity
• Possibilities of digital testing
“Norm drift”
• Phenomenon whereby tests become easier
over time:
• Familiarity with test content
• Most pupils score above the mean
• The bell curve is skewed
• DES, 2016: report on standardised test
results for 2011-12 and 2012-13
• In mathematics and English reading,
performance was well to the right of the
bell curve
National improvement in
standards
• National Assessments, 2014
• Mathematics and English reading, 2nd and 6th class
• First NA since 1980 in which statistically significant
improvements were observed
• Targets for 2020 (as set out in the National Strategy to
Improve Literacy and Numeracy, 2011) exceeded in NA
2014
• TIMSS 2015
• Significant improvements in mathematics (4th class)
• PIRLS 2016
• Significant improvements in reading (4th class)
• Irish pupils performed equally well on paper PIRLS and
on new test of digital literacy (ePIRLS)
National Assessment – English Reading (2009 vs. 2014) – Percent Correct Scores % Correct Scores and effect sizes (d) NA ’09 NA ’14 d Vocabulary 63.8 70.4 0.31 Comprehension 65.5 69.5 0.21 Overall Reading 65.0 69.8 0.26
English Reading – Sixth Class, Performance by
Proficiency Levels (NA 2009, 2014) – Percents of
Pupils
50
45
43.9
40.0
40
35.0 35.0
35
30.0 30.0 30.0 31.3
30
24.8 2009
25
2020 Target
20
2014
15
10
5
0
Level 1 and Below Level 2 Levels 3-4
7
Mathematics – Second Class - Performance by Content Area
and Overall (Percent Correct Scores)
NA ’09 NA ’14 d
Number & Algebra 58.7 64.6 0.31
Shape & Space 72.7 68.8 0.18
Measures 49.0 54.5 0.24
Data 56.1 56.6 0.09
Overall 57.0 62.0 0.28
8
Mathematics – Second Class - Performance by Process a
(Percent Correct Scores)
NA ’09 NA ’14 d
Understand & Recall 73.5 75.1 0.31
Implement 58.2 61.3 0.18
Reason 64.4 63.6 0.24
Integrate & Connect 56.6 68.9 0.09
Apply & Problem
48.6 53.7 0.28
Solve
9
Relative strengths and weaknesses in TIMSS
2015 Maths in Ireland (Fourth Class)
Overall Content Process
% Correct Geometric
Scores Number Shapes & Data Display Knowing Applying Reasoning
Measures
Ireland 59 58 57 67 67 57 48
TIMSS Avg. 50 49 50 57 56 48 44
Singapore 74 77 70 78 81 74 65Revision of curriculum & recent research
• First major revision of primary school curriculum since 1999
is underway at NCCA
• New Language curriculum (replacing 1999 English and Irish
curricula):
• 2015: language curriculum for Junior Infants – Second
class published
• 2018: draft specification for Third – Sixth class
published; open for consultation
• New Mathematics curriculum:
• 2017: draft specification for Junior Infants – Second
class published; open for consultation
• Further impetus to revise tests – but also a challenge• NCCA reports (Dooley et al., 2014; Dunphy et al., 2014):
• Recommendation:
The appropriate use of screening/diagnostic tests
should be emphasised as should limitations of the
use of standardised tests for young children.
• Currently, Drumcondra tests for First class are
standardised tests. The new tests will be screening tests
instead:
• Less pressure
• Somewhat easier
• Not strictly timed
• The plan is for a diagnostic test to follow.Engagement
• Recent emphasis on importance of engagement,
motivation, positive disposition
• Not something standardised tests can measure...
• But the tests will not accurately represent the
achievement of children who are disengaged
• Aim: that children have as positive an experience as
possible of taking the test
• Design decisions: colour, font, layout,
illustrations
• Online system: intuitive navigation
• ContentInclusion and diversity
• Population of Ireland, and of schools in Ireland, has
changed notably over the past 15 years
• 2017: highest net inward migration to Ireland since 2008
• Increasing awareness of the need to include and engage
children who differ in terms of cultural background,
ethnicity, disability status, family structure, gender, etc.
• Draft Language curriculum: aims to support teachers to
‘broaden children’s understanding of the world
through a rich variety of language experiences and
through fostering an awareness and appreciation of
other languages and cultures in an enriching learning
environment’ (p. 1)Possibilities of digital testing
• Drumcondra Online Testing System (DOTS):
• At post-primary level, 3 tests currently available:
• Drumcondra Reasoning Test
• English reading test (Second year)
• Mathematics test (Second year)
• New primary tests being developed on two modes -
paper and computer:
• IT infrastructure is not yet present for all
primary schools
• But some particular advantages to digital testing
(e.g., instant reports)
• Therefore, two modes; schools can choose
• Scores will be equated across the modesConsiderations during test
development
• Responding to a changing curriculum
• Maths: balance of strands
• Reading: item types, comprehension processes,
text variety
• Role of technology / mode effectsResponding to a changing curriculum • Advisory Committee – with representation from PDST, NCCA, DES, subject experts • NCCA background documents • Awareness that even once the new curriculum has been published in its final form, it will take some time before it is embedded in schools
Mathematics - Challenges • Timing of the development of the new tests (curriculum change, age of starting school higher for many pupils) • Links to the new curriculum specifications • Dealing with limitations on upper limits of numbers that can be used (computation, money etc.) • Dealing with the ‘literacy’ issue. • Test read aloud to pupils in First class • Section of the test (first twenty items) read aloud in Second class. • Potential of CBT (adaptive testing, multiple item formats) not yet fully realised.
DPMT content - overview
Level Section A Break Section B Total
No. items 22 23 45
1
Time (mins) - 10 - -
No. items 20 30
2
Time (mins) 28 42 75
No. items 25 25 50
3-6
Time (mins) 35 5 35 75DPMT content - strand coverage
Level Number Algebra Measures Shape & Data
Space
No. items 17 6 12 6 4
1 % of items 38 13 27 13 9
No. items 18 6 14 7 5
2 % of items 36 12 28 14 10
No. items 17 6 13 8 6
3 % of items 34 12 26 16 12
No. items 16 7 13 8 6
4 % of items 32 14 26 16 12
No. items 15 7 13 8 7
5&6 % of items 30 14 26 16 14Reading
• Reading tests assess an expanded variety of content and processes:
• Second class: some ‘rogue word’ items, assessing pupils’ sentence comprehension
and grasp of syntax
• Fourth, Fifth, and Sixth classes: vocabulary questions are divided into two sections:
synonyms and antonyms
• Comprehension texts:
• Balance of literary and informational content
• Items reflect a variety of comprehension processes:
• Retrieval
• Inference
• Interpretation
• Evaluation
(Reflects approach used in international assessments)Reading-screening
test
• Vocabulary assessed using picture items and synonym items
• Comprehension assessed using picture sentences, ‘rogue word’
items, and a short passage with six questions
• Word analysis subtest• Role of technology /
mode effects
• Reduces familiarity with content (therefore
reduces ‘norm drift’)
• Positive attitudes from pupils (therefore
reduces effect of disengagement)
• Pilot results: some mode effects
• These can be dealt with during scaling process, so
that results are comparable across the modes
• Maths: need to encourage use of roughwork sheets
for computer-based testsDOTS demo
You can also read
