Developing the Enabling Context for Student Assessment in Chile - World Bank Documents
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Public Disclosure Authorized
S A B E R – S Y S T E M S A P P R O A C H F O R B E T T E R E D U C AT I O N R E S U LT S
STUDENT ASSESSMENT
2
69015
Public Disclosure Authorized
Developing the
Enabling Context for
Student Assessment
Public Disclosure Authorized
in Chile
Updated Edition
María-José Ramírez
Public Disclosure AuthorizedNote to the Reader As of August 27, 2012, the following updates were made to the original YHUVLRQRIWKLVSDSHUEDVHGRQWKH8QLYHUVLGDGGH&KLOH·VRIÀFLDOEXGJHW information obtained through the Access to Information Act (Ley de 7UDQVSDUHQFLD 3DJHVDQG7KHRIÀFLDOH[DPLQDWLRQEXUHDXEXGJHWZDV changed from US$10 million to US$6.6 million. An explanation was DGGHGUHJDUGLQJZKDWLVFRYHUHGE\WKLVRIÀFLDOEXGJHWDQGZKDWLV transferred as indirect cost to the university that hosts the examination bureau. 3DJHThe original text “The examination fee allowed Universidad de Chile to collect US$13.6 million, 36 percent more than the DEMRE yearly budget” was changed to “The examination fee allowed Universidad de Chile to collect US$14.9 million, more than double the DEMRE yearly budget.” 3DJH The original text “The budget for the examination bureau also seems to have remained relatively stable in the last decade” was changed to “The yearly budget for the examination bureau increased by US$3.8 million in the last decade.” An explanation was added comparing this increase with the increase in the amount of money collected via examination fees. 3DJH The original text “The examination budget has remained relatively stable since the 1990s” was changed to “The budget of the examination bureau more than doubled between 2000 and 2011.” 3DJH The original text “The examination budget has remained relatively stable since the 1990s, despite the fact that the number of VWXGHQWVSD\LQJWRWDNHWKHH[DPLQDWLRQVKDVJURZQPRUHWKDQ percent in the last decade” was changed to “Between 2000 and 2011, the yearly amount of money collected from the examination fee grew by around US$9.6 million, whereas the examination yearly budget increased by only US$3.8 million.”
WORKINGPAPERNO.2
DevelopingtheEnablingContext
forStudentAssessmentinChile
UpdatedEdition
MaríaJoséRamírez
©2012TheInternationalBankforReconstructionandDevelopment/TheWorldBank
1818HStreetNW
WashingtonDC20433
Telephone:2024731000
Internet:www.worldbank.org
1234 15141312
ThisworkisaproductofthestaffofTheWorldBankwithexternalcontributions.Thefindings,
interpretations,andconclusionsexpressedinthisworkdonotnecessarilyreflecttheviewsof
TheWorldBank,itsBoardofExecutiveDirectors,orthegovernmentstheyrepresent.
TheWorldBankdoesnotguaranteetheaccuracyofthedataincludedinthiswork.The
boundaries,colors,denominations,andotherinformationshownonanymapinthisworkdo
notimplyanyjudgmentonthepartofTheWorldBankconcerningthelegalstatusofany
territoryortheendorsementoracceptanceofsuchboundaries.
RightsandPermissions
Thematerialinthisworkissubjecttocopyright.BecauseTheWorldBankencourages
disseminationofitsknowledge,thisworkmaybereproduced,inwholeorinpart,for
noncommercialpurposesaslongasfullattributiontothisworkisgiven.
Anyqueriesonrightsandlicenses,includingsubsidiaryrights,shouldbeaddressedtothe
OfficeofthePublisher,TheWorldBank,1818HStreetNW,Washington,DC20433,USA;fax:
2025222422;email:pubrights@worldbank.org.
Coverdesign:PatriciaHord.GraphikDesign,Alexandria,VA
Contents
AbouttheSeries...............................................................................................................v
AbouttheAuthor..........................................................................................................vii
Acknowledgments..........................................................................................................ix
Abstract............................................................................................................................xi
ExecutiveSummary.....................................................................................................xiii
Introduction......................................................................................................................1
LargeScaleAssessment..................................................................................................2
TheUniversityEntranceExaminations........................................................................8
ClassroomAssessment..................................................................................................13
LessonsLearned.............................................................................................................18
References.......................................................................................................................22
Table
Table1.Keyfeaturesoftheenablingcontext,mechanismsofchange,and
driversforchangeofChile’sstudentassessmentsystem................................18
DevelopingtheEnablingContextforStudentAssessmentinChile iiiAbouttheSeries
Building strong education systems that promote learning is fundamental to
development and economic growth. Over the past few years, as developing
countries have succeeded in building more classrooms, and getting millions
more children into school, the education community has begun to actively
embrace the vision of measurable learning for all children in school. However,
learning depends not only on resources invested in the school system, but also
on the quality of the policies and institutions that enable their use and on how
wellthepoliciesareimplemented.
In 2011, the World Bank Group launched Education Sector Strategy 2020:
Learning for All, which outlines an agenda for achieving “Learning for All” in
the developing world over the next decade. To support implementation of the
strategy,theWorldBankcommencedamultiyearprogramtosupportcountries
in systematically examining and strengthening the performance of their
education systems. This evidencebased initiative, called SABER (Systems
Approach for Better Education Results), is building a toolkit of diagnostics for
examining education systems and their component policy domains against
global standards, best practices, and in comparison with the policies and
practices of countries around the world. By leveraging this global knowledge,
SABERfillsagapintheavailabilityofdataandevidenceonwhatmattersmost
toimprovethequalityofeducationandachievementofbetterresults.
SABERStudentAssessment,oneofthesystemsexaminedwithintheSABER
program, has developed tools to analyze and benchmark student assessment
policies and systems around the world, with the goal of promoting stronger
assessmentsystemsthatcontribute toimprovededucationqualityandlearning
for all. To help explore the state of knowledge in the area, the SABERStudent
Assessment team invited leading academics, assessment experts, and
practitioners from developing and industrialized countries to come together to
discussassessmentissuesrelevantforimprovingeducationqualityandlearning
outcomes. The papers and case studies on student assessment in this series are
the result of those conversations and the underlying research. Prior to
publication, all of the papers benefited from a rigorous review process, which
included comments from World Bank staff, academics, development
practitioners,andcountryassessmentexperts.
AllSABERStudentAssessmentpapersinthisseriesweremadepossibleby
supportfromtheRussiaEducationAidforDevelopmentTrustFund(READTF).
READTFisacollaborationbetweentheRussianFederationandtheWorldBank
that supports the improvement of student learning outcomes in lowincome
countriesthroughthedevelopmentofrobuststudentassessmentsystems.
DevelopingtheEnablingContextforStudentAssessmentinChile vTheSABERworkingpaperserieswasproducedunderthegeneralguidance
of Elizabeth King, Education Director, and Robin Horn, Education Manager in
theHumanDevelopmentNetworkoftheWorldBank.TheStudentAssessment
papersintheserieswereproducedunderthetechnicalleadershipofMarguerite
Clarke, Senior Education Specialist and SABERStudent Assessment Team
CoordinatorintheHumanDevelopmentNetworkoftheWorldBank.Papersin
thisseriesrepresenttheindependentviewsoftheauthors.
vi MaríaJoséRamírezAbouttheAuthor
MaríaJosé Ramírez is an Education Specialist in the Human Development
Network at the World Bank. She has been working on the SABERStudent
Assessmentinitiative,developingtoolsforevaluating the qualityofassessment
systems. Before joining the Bank, she was involved in key reforms of the
assessment system in Chile, with responsibilities in both national and
internationalassessments.IntheChileanMinistryofEducation,sheheadedthe
dataanalysisunitoftheassessmentprogram(SIMCE)(2005–06)andworkedasa
nationalcoordinatorforTIMSS(1998–2000).ShealsowastheCollegeDirectorof
Universidad Diego Portales, Chile (2007–09). She led universitylevel projects
related to quality assurance, institutional analysis, accreditation, and academic
evaluation. In the United States, she was a research assistant in the TIMSS and
PIRLSInternationalStudyCenter(2000–04).ShereceivedherPhDineducational
research,measurement,andevaluationfromBostonCollege(2004)andwasthe
recipientofaFulbrightscholarship(2000)andthe2005awardforbestempirical
dissertationfromtheInternationalAssociationfortheEvaluationofEducational
Achievement (IEA). Her work and publications focus on student assessments,
educationquality,andcomparativeeducation.
DevelopingtheEnablingContextforStudentAssessmentinChile viiAcknowledgments
Many people provided inputs and suggestions for this paper. Thanks in
particular go to the READ Trust Fund team, particularly Marguerite Clarke, as
well as Julia Liberman, Robin Horn, Elizabeth King, Olav Christensen, Emily
Gardner, Manorama Gotur, Emine Kildirgici, Diana Manevskaya, Cassia
Miranda, and Fahma Nur. In Chile, special thanks go to all those who
contributed with key information and insights: Violeta Arancibia, Juan Bravo,
Leonor Cariola, and Francisco Meneses, Ministry of Education; Matko Koljatic,
Lorena Meckes, Verónica Santelices, and Sandy Taut, Universidad Católica de
Chile; Jose Joaquin Brunner, Universidad Diego Portales; Jacqueline Gysling,
Alejandra Mizala, Pilar Romaguera, and Iván Silva, Universidad de Chile; and
Jose Manuel Robles, Universidad del Desarrollo. Thanks also to the peer
reviewersMarlaineLockheed,AlejandroGanimian,andJeffMarshall;andtothe
useful comments of Jamil Salmi, World Bank. Finally, thanks to the Russian
government for their support for this work under the READ Trust Fund
program.
DevelopingtheEnablingContextforStudentAssessmentinChile ixAbstract
Developing a strong assessment system is a priority for many countries. But
whereshouldtheystart?ThispaperdescribesthejourneyexperiencedbyChile
to develop the enabling context (policies, institutions, and human and fiscal
resources) for its largescale assessment program, its university entrance
examinations,andclassroomassessmentactivities.Countriesaimingtodevelop
their assessment system may draw valuable lessons from this case study, most
importantly the following: (i) developing the right enabling context is a long
term process that builds on gradual changes; (ii) stakeholders should be
represented in the governance of the assessment unit; (iii) while different
institutional arrangements are possible for an assessment unit, it is important
that the unit is accountable and autonomous to make technical decisions; (iv)
countriesneedtodeveloplocalcapacityinassessment;and(v)countriesneedto
anticipate the pressure the expansion of the education system will put in the
assessmentsystem,especiallyintheuniversityentranceexaminations.Countries
thatcanbenefitfromtheselessonswillbeinabetterpositiontocontinueonthe
journey of strengthening their enabling context for an effective assessment
system.
DevelopingtheEnablingContextforStudentAssessmentinChile xiExecutiveSummary
Countriesaimingtodeveloptheirstudentassessmentsystemshouldfindsome
interesting lessons in this case study. Chile was a pioneer among developing
countries in strengthening its student assessment system, which encompasses
three main types of assessment activities: the national largescale assessment
program, the university entrance examinations, and classroom assessment
activities by teachers and students. This achievement required putting in place
therightenablingcontext;thatis,developingtherightpoliciesandinstitutions,
andensuringtherightfiscalandhumanresourcesforassessment.
InChile,thedevelopmentofeachassessmenttypefolloweddivergentpaths.
The national largescale assessment started as a pilot program housed in a
university.Theuniversityentranceexaminationprogramwasdevelopedbyone
university, and then voluntarily adopted by the others. While classroom
assessment has been a common practice since the foundation of the school
systeminthenineteenthcentury,thecountrystillhaslittleinstitutionalcapacity
tosupportit.
The enabling context for each type of assessment varies considerably in
Chile.Boththelargescaleassessmentandtheuniversityentranceexaminations
have been in place for several decades. These programs are highly
institutionalized thanks to clear regulations, organizational structures, stable
funding,andtrainedstaff.Atthesametime,theenablingcontextforclassroom
assessmentisweak.Whilesomepolicydocumentsandguidelinesareavailable
online,implementationoftheseguidelinesremainsweak.
Keydriversthathaveallowedforthedevelopmentoftheassessmentsystem
in Chile include strong political and technical leadership from the Ministry of
Educationanduniversities,politicalandeconomicstability,socialpressures,and
curricularreforms.
Importantlessonscanbeextractedfromthiscasestudy.Developingastrong
assessmentsystemtakestime,effort,commitment,andresources.Thereisnoone
right way to develop a sustainable assessment system. Nevertheless, some
important ingredients include having a longterm vision, planning for gradual
implementation of changes, ensuring representation of stakeholders in the
governance of the assessments, creating local capacity, and securing enough
funding.Countriesthatmanagetoputatleastsomeoftheseingredientsinplace
will be in a better position to develop their assessment system and, most
importantly,tocontributetoeducationquality.
DevelopingtheEnablingContextforStudentAssessmentinChile xiii
DevelopingtheEnablingContext
forStudentAssessmentinChile
MaríaJoséRamírez
Introduction
Studentassessmentplaysakeyroleinpromotingeducationquality,whichisa
strategicpriorityforagrowingnumberofcountries.Becauseeducationismainly
about learning, assessing student learning has gained attention in the political
arena.Assessmentisnowconsideredakeypolicyareaofanyeducationsystem.
A growing number of countries are interested in developing strong
assessment systems that contribute to improved education quality. Assessment
contributestoqualitybyclearlysignalingwhatstudentsareexpectedtolearn,by
monitoring and certifying student learning, by tracking students into different
education paths, and by informing teaching and learning. How can a country
startbuildinganeffectiveassessmentsystem?Wheredoyoustartwhenthereare
noinstitutions,policies,andfundingfortheassessment?Howdoyoucreatean
assessmentculturewithteachersthatare“assessmentliterate”?
Because of the challenges involved in developing an assessment system,
governments are interested in learning from the experience of other countries.
ThepurposeofthiscasestudyistotrackthedevelopmentofChile’sassessment
system, and to offer lessons to other countries aiming to develop their own
assessment.Chileisaninterestingcasebecauseitsstudentassessmentsystemis
relatively strong compared to other developing countries. This study analyzed
thedevelopmentofthreetypesofassessmentactivities:largescaleassessments,
universityentranceexaminations,andclassroomassessments.Thefocusisonthe
conditions or enabling context that allowed for the development of the
assessment,encompassingpolicies,institutions,andhumanandfiscalresources.1
In learning from the Chilean experience, countries should consider the
broader context in which the assessment operates. Chile is an upper middle
income country that still needs to reduce poverty and inequality rates. The
country’s return to democracy in 1990 provided the necessary political stability
toallowforsocialandeconomicgrowth.ChileisaleaderinLatinAmericaforits
economicdevelopment:thegrossnationalincomeGNIpercapitawasUS$10,120
in2010,economicgrowthaveraged4.1percentofgrossdomesticproduct(GDP)
between 1991 and 2005, and the Human Development Index is “very high.”
Poverty was reduced from 20 to 15 percent between 2000 and 2009. However,
1
Forageneraldiscussionoftheenablingcontextforstudentassessment,seeClarke(2012).
DevelopingtheEnablingContextforStudentAssessmentinChile 1
Chile is lagging in the distribution of the wealth: its Gini coefficient (52) is the
worst among Organisation for Economic Cooperation and Development
(OECD)countries.2
The education system in Chile is relatively strong compared to Latin
America, but weak when compared to richer countries. In primary education,
coverage is universal; in secondary education,it expanded quickly, reaching 96
percentin2006.Intertiaryeducation,theexpansionwasalsostrong,reaching40
percent in 2012. Investment in education is high when considering public and
privateexpenditures(7.1percentofGDP),butitislowwhenconsideringpublic
expendituresonly(4.2percentofGDP).Theresultsofnationalandinternational
assessmentsshowthatChileanstudentsareslowlyreachinghigherperformance
levels,butoverallperformanceremainslowcomparedtoricherOECDcountries,
andisunevenlydistributedalongsocialclasses.3
The road to develop a student assessment system is not an easy one.
Transitingthisroadtakestimeandeffort.Themainlessonfromthiscasestudyis
that countries aiming to put in place their assessment system first must ensure
the necessary conditions or enabling context for the system to operate. These
conditions encompass the policy framework that supports the assessment, the
right institutions and organizational structures, together with the appropriate
human and fiscal resources. The absence of any of these conditions may put at
riskthedevelopmentandsustainabilityoftheassessmentsystem.
Thepaperisorganizedasfollow.Afterthisintroduction,thesecondsection
providesageneraldescriptionofthenationallargescaleassessmentprogram,its
enabling context, changes to the enabling context, and drivers that allowed for
thosechanges.Thethirdsectionprovidessimilarinformationfortheuniversity
entrance examinations, while the fourth section does the same for classroom
assessment.Thelastsectionpresentsthemainlessons.
LargeScaleAssessment
This section reviews the evolution of the enabling context of Chile’s national
largescale assessment program, SIMCE (Sistema de Medición de Calidad de la
Educación).ThefirstsubsectionpresentsanoverviewoftheSIMCE,describing
its main purposes, mandate, and technical features. The second subsection
describes the enabling context in which SIMCE was operating as of 2012,
providing information about the laws, institutional arrangements, and human
2
AlldatagatheredfromTheWorldBank(www.worldbank.org),UnitedNationsDevelopmentProgram
(http://hdr.undp.org/en/statistics), The Central Bank of Chile (www.bcentral.cl), Chile’s Ministry of
SocialDevelopment(www.ministeriodesarrollosocial.gob.cl),andfromtheHumanDevelopmentReport
oftheUnitedNationsDevelopmentProgram(www.undp.org).
3
AlldatagatheredfromWorldBankEdStats,OECD’sEducationataGlancereportandPISAprogram
(www.oecd.org), Chile’s Ministry of Education (www.mineduc.cl) and Ministry of Social Development
(www.ministeriodesarrollosocial.gob.cl),andChile’snationalassessmentprogram(www.simce.cl).
2 MaríaJoséRamírezand fiscal resources that support the program. The third subsection analyzes
howtheselaws,institutions,andhumanandfiscalresourceshavechangedover
theyears,whilethefourthsubsectionidentifiesthedriversthatallowedforthese
changestotakeplace.
GeneralDescription
Chile’slargescaleassessmentprogram(SIMCE)hasbeenoperatingonayearly
basis since 1988. The program aims to improve the quality and equity of
education by reporting about student performance and their learning
environment.
Two somewhat opposed rationales explain how SIMCE is expected to
contribute to education. The first one states that SIMCE would improve
efficiencybypushingschoolstocompeteforeducationquality,parents’choices,
andfunding(Majluf1987).SIMCEwouldinformparentsaboutschooltestscores
(an indicator of quality), so that they could choose the best school for their
children, no matter if public or private. The assessment was developed in the
contextofanationalvoucherpolicywherethestatepaysasubsidytotheschools
basedonstudentenrollmentandattendance.
Thesecondrationalearguesthattheassessmentprogramwouldcontribute
to effectiveness by monitoring student performance and supporting pedagogy.
Effectiveness would be improved by monitoring how many students were
reaching the learning standards set by the national curriculum. Pedagogical
support would be provided by disseminating guidelines and reports to school
supervisorsandteachers(Himmel1996).
The assessment program is highly institutionalized. SIMCE has been
operating in the Ministry of Education since 1992. The office has the necessary
autonomy to make decisions based on technical criteria, and to report results
despite political costs for the government. This has been possible thanks to the
integrityofthedifferentgovernmentadministrations.
Since 1996, the assessment program has also been responsible for
coordinating international largescale assessments. The SIMCE office has
coordinated, or is coordinating, the following assessments: ICILS 2013; ICCS/
CIVICS 1999, 2000, 2008; LLECE 1997, 2006, 2013; PISA 2000, 2006, 2009, 2012;
andTIMSS1999,2003,2011.4
4
Chilefirstparticipatedinaninternationalassessmentin1968–1972—theSixStudySurvey,administered
by the International Association for the Evaluation of Educational Achievement (IEA). Participation in
international assessments was resumed in 1997 with the Latin American Laboratory for Assessment of
the Quality of Education (LLECE), administered by the United Nations Educational, Scientific and
Cultural Organization (UNESCO). Chile has also participated in IEA’s International Civic and
Citizenship Education Study (ICCS); IEA’s International Computer and Information Literacy Study
(ICILS);UNESCO’sLaboratorioLatinoAmericanodeEvaluacióndelaCalidaddelaEducación(LLECE);
OECD’s Program for International Student Assessment (PISA); and IEA’s Trends in International
MathematicsandScienceStudy(TIMSS).
DevelopingtheEnablingContextforStudentAssessmentinChile 3
The largescale assessment program has high technical quality and
credibility.Rigorousproceduresarefollowedinallthestepsofboththenational
and international assessments. The technical standards of international
assessment exercises have always been met. Assessment results are
systematically used to inform decision making and are widely disseminated in
the media. SIMCE results are seen as the most credible indicator of education
qualityandequityinthecountry(Ramírez,2012).
The national assessment is a technically sophisticated program. SIMCE
provides information about student performance in core curricular areas:
mathematics, language (Spanish), and sciences (natural and social). The
questions are mainly in multiplechoice format, but openended questions and
essays have been gradually introduced. All schools and students in the target
grades (4, 8, and 10) participate in the assessment.5 In 2010, nearly 500,000
students from near 9,000 schools answered the SIMCE tests. Students, their
parents,andteachersalsoansweredbackgroundquestionnaires.Around22,500
test supervisors and external administrators were directly involved in the field
operation.
EnablingContext
This subsection describes the main features of the enabling context of Chile’s
largescale assessment program. These features encompass the policy
framework, public support for the assessment, institutional arrangements, and
thehumanandfiscalresourcesallocatedtotheprogram.
InChilethelargescaleassessmentismandatedbylaw.Thelawstatesthat
regular assessments of student learning should be carried out with the aim to
improve education quality and equity. The law ensures the availability of
resources to run both the national and international assessments. All schools
must participate in the national assessment and their results must be widely
disseminated.6
The assessment program has wide public support. SIMCE is the most
credibleindicatorofeducationqualityinthecountry.Itislargelybacked,orat
least accepted, by students, teachers, parents, the academia, the media,
politicians, and the government. The main criticism against SIMCE is that it
stigmatizes poor public schools. Claims about corruption or irregularities are
veryisolated.
Chilehasaclearinstitutionalarrangementsupportingtheassessment.Asof
2012,SIMCEislocatedintheCurriculumandEvaluationUnitoftheMinistryof
Education. It is accountable to the Minister of Education, and to the National
5
Sincetheearly2000s,newareas(writing,English,computerskills,andphysicaleducation)andgrades
(2,6,and11)havebeenaddedtotheassessmentonasampleorcensusbase.
6
SeeGovernmentofChile(2009,2011):the2009GeneralLawofEducation(LeyGeneraldeEducación—
LGE)andthe2011QualityAssuranceLaw(SistemaNacionaldeAseguramientodelaCalidaddelaEducación
Parvularia,BásicayMediaysufiscalización).
4 MaríaJoséRamírezEducation Council, an autonomous public agency that oversees education
quality in the country. SIMCE’s organizational structure includes the following
teams:subjectareas,dataprocessing,communication,standardsettings,quality
control,specialneedsandadaptations,andinformationtechnologies.
Thereisaspecialunitthatcoordinatesinternationalassessments.However,
most of the tasks related to the international assessments are executed by the
sameteamsinchargeofthenationalassessment(forexample,dataprocessing).
OtherfunctionsaredelegatedtootherteamsoftheMinistryofEducation(most
notably,humanresources).
SIMCE has a qualified staff to run the assessment. The staff has a suitable
technicalprofileandisrecruitedbasedonmerit.Asof2012,SIMCEhasastaffof
around 70 people, including subject matter specialists, teachers, psychologists,
textbookspecialists,economists,engineers,andbudgetandadministrativestaff.
Intermsofcontractualstatus,20percentofthestaffbenefitsfromacontractasa
public employee; the other 80 percent has to renew their contracts on a yearly
basis.Mostofthestaffhasabachelor’sdegree(Licenciatura),nearlyonefourth
hasamaster’sdegree,andtwopeoplehaveadoctorate.Mostofthestaffisinthe
early stages of their professional career and got onthejob assessment training.
There is enough staff stability to allow for capacity building and to train new
personnel.
Chileinvests0.26percentoftheprimaryandsecondaryeducationbudgetin
its largescale assessment program. The SIMCE yearly budget is public, and in
2012 it reached US$26 million, out of which US$2 million was devoted to
international assessments. The costs of the national assessment per student
reached US$20 in primary and US$15 in secondary schools (which seems
relativelylowcomparedtootherLatinAmericancountries).In2004,thebiggest
budget lines were personnel (28 percent of the total budget), printing (20
percent),andfieldwork(15percent).7
MechanismsofChange
Thissubsectionanalyzesthemechanismsofchangethatallowedforthecurrent
levelofdevelopmentofthelargescaleassessmentprograminChile.Itexamines
how the legal framework, institutional arrangements, infrastructure, staff, and
budgetevolvedovertime.Thesechangesallowedforasuccessfulrevampingof
theassessmentofficearoundyear2000.
Thelegalframeworksupportingtheassessmenthasbeeninplaceformore
than 20 years, and was recently updated. The largescale assessment was first
mandated by law in 1990. This law ensured the transfer of the assessment into
the Ministry of Education, and a stable budget to operate on a regular basis.
However,thelawlackedlegitimacysinceitwaspassedattheendofthemilitary
7
As for 2012, the budget from primary and secondary education reached near US$10,000 million. All
databasedonChile’s2012budgetlaw(http://www.dipres.gob.cl);onpersonalcommunicationswiththe
SIMCEoffice,andonWolff(2007).
DevelopingtheEnablingContextforStudentAssessmentinChile 5
regime.In2009and2011,withChilenowunderademocraticgovernment,two
related laws came to replace the first one. These new laws mandated that both
the national and international assessments should be in charge of a quality
assuranceagencyindependentfromtheMinistryofEducation,andaccountable
to a Superintendence of Education (this institutional change has yet to be
implemented)(seeGovernmentofChile2009,2011).
Different institutional arrangements have supported the largescale
assessment. Initially, the program was commissioned by the Ministry of
EducationtoUniversidadCatólicadeChile,theleadingprivateuniversityinthe
country.From1982–84,theassessmentwasnomorethanapilotsurveyhosted
in a university (Majluf, 1987). From 1988–92, the program was run by a team
from both the university and the Ministry of Education; and was gradually
transferredintotheMinistryofEducation.Duringthe1990s,SIMCEwashoused
under the office of the UnderSecretary of Education. In year 2000, it was
absorbed into a new Curriculum and Evaluation Unit in the Ministry of
Education.In2013,SIMCEisexpectedtobetransferredintotheaforementioned
qualityassuranceagency.
TherevampingofSIMCEcametogetherwithnewinfrastructuretocarryout
theassessment.In2000,SIMCEmovedintoanewbuilding,strategicallylocated
neartheMinister’soffice.Thenewofficesprovidedtheadditionalspaceneeded
for contracting more staff. SIMCE got new servers and data processing
equipment.
Theassessmentstaffhaschangedinparallelwiththedifferentinstitutional
arrangements of the program. In the 1980s, the university staff in charge of the
pilot assessment program was a highly qualified multidisciplinary team of 10
professors (most of whom had graduate degrees in engineering, education,
psychology, and sociology from prestigious universities in the United States)
(Himmel1996;Majluf1987).Inthe1990s,whenSIMCEwastransferredintothe
Ministry of Education, the staff in charge of the assessment was a team of
approximately 20 fulltime public employees. Most were school teachers who
hadlefttheclassroomsmanyyearsago.Manywereinthelateststagesoftheir
career,justbeforeretiring.
In the 2000s, when SIMCE was transferred into the Curriculum and
EvaluationUnitoftheMinistryofEducation,thestaffwasgreatlyincreasedand
professionalized.Thehiringofacriticalnumberofnewstaff—threetostartwith,
including a new SIMCE director—allowed for putting needed changes into
motion.
Theassessmentstaffmorethantripledin15years.Thepermanentstaffgrew
from approximately 20 in 1996, to 40 in 2000, and up to 70 in 2010. Young
professionals from different fields came to reinforce the technical capacities of
the existing team. The technical leadership was provided by new senior staff,
some of whom had completed graduate studies at prestigious universities in
EuropeandtheUnitedStates.However,theyallhadtolearnbydoingthebasic
6 MaríaJoséRamírezassessmentrelated tasks. Having both the old and the new staff working
togetherwasachallengingtask.
Theyearlybudgetfortheassessmentprogramhasincreasedmorethan10
foldduringthelasttwodecades.Duringthe1990s,theyearlybudgetwasUS$2
million; in 2001 it was US$2.5 million; in 2011, US$22.7 million; and in 2012
US$26million.8
DriversforChange
The following drivers allowed for changes in the legal framework, institutions,
andhumanandfiscalresourcesforrunningthelargescaleassessmentprogram
inChile:
x Leadership.LeadershipfromboththeMinistryofEducationandacademia
allowed for launching and sustaining the assessment program. This
leadership was also required for revamping the assessment office and
ensuringincreasedresources.
x Education as a national priority. This created the need for an assessment
program that could provide better information about the quality and
equity of education. SIMCE became the main indicator for judging the
effectivenessofeducationreformprograms.
x Political stability. This provided the context for longterm commitments
and sustainable reforms. An ad hoc technicalpolitical commission
(Comisión SIMCE) was created in 2003 to agree on the development
agendaofSIMCE.
x Economicgrowth.Economicgrowthofthecountryallowedforthegradual
increaseoftheassessmentbudgetandthehiringofmorestaff.
x Curricularreform.ThiscreatedtheneedtoaligntheSIMCEtestswiththe
newcurriculum.Theinstitutionalsolutionfordoingsowasthecreation
oftheCurriculumandEvaluationUnitoftheMinistryofEducation.This
needforalignmenttriggeredtherevampingoftheassessmentoffice.
x Social pressures. In 2006, student protests against poor education quality
translated into a stronger legal framework and a new institutional
arrangementfortheassessment.Inthenearfuture,schoolswillbemore
closelymonitoredbyaqualityassuranceagency.
x Participation in international largescale assessments. TIMSS and PISA
provided highquality, handson training on all key aspects of the
assessment. This training directly benefited the staff in the international
assessment unit as well as the staff working in the national assessment
program. The new knowledge and technology was quickly transferred
fromtheinternationaltothenationalassessmentprogram.
8
AlldatabasedonChile’snationalbudget(http://www.dipres.gob.cl)andHimmel(1996).
DevelopingtheEnablingContextforStudentAssessmentinChile 7
TheUniversityEntranceExaminations
This section reviews the evolution of the enabling context of Chile’s university
entrance examination program (Prueba de Selección Universitaria, PSU). The
firstsubsectionpresentsageneraloverviewofthePSU.Thefollowingsubsection
analyzes the enabling context of the PSU as of 2012. The last two subsections
examine reforms to the enabling context and the drivers that allowed (or
blocked)thesereforms.
GeneralDescription
The university entrance examination program is part of the educational
landscapeinChile.Ithasbeeninplacesince1967anditisthemaingatekeeper
for distributing educational opportunities. It is the higheststakestest that most
Chileanstudentswilltakeintheirlifetime.Dependingontheirresults,students
will gain access to a wide quality range of university programs, which in turn
will greatly affect their ability to get a good job and a good salary.9 The
examinationreceiveswidemediacoverage,especiallyduringtheadministration
anduniversityapplicationprocess.
The examination program was created by the leading public university in
the country, Universidad de Chile. This university made the examinations
availabletoallotheruniversities,whileretainingthecopyrights.Therefore,what
wasenvisionedastheuniversityentranceexaminationsforoneinstitutiongrew
tobecometheuniversityentranceexaminationsforthewholecountry.
Since the creation of the university entrance examinations, there have been
twotestbatteries.Thefirstbattery(PruebadeAptitudAcadémica,PAA)wasa
paper and pencil, multiplechoice test that included two compulsory “aptitude
tests” in mathematics and language; five optional subjectspecific tests in
(advanced) mathematics, social sciences, biology, chemistry, and physics; and
one compulsory subjectspecific test in history of Chile. In 2003, a new battery
based on a recentlyintroduced secondary school curriculum (Prueba de
SelecciónUniversitaria,PSU)cametoreplacethePAAexamination.10PSUisalso
apaperandpencil,multiplechoicebatterythatincludestwocompulsorytestsin
mathematicsandlanguageandtwooptionaltests(studentshavetotakeatleast
one)insocialsciencesandnaturalsciences.
Theexaminationadministrationinvolvesavastandcomplexoperation.The
majorityofsecondaryschoolgraduates(78percentin2009)takethetest,making
itsadministrationamajoreventinthecountry.In2011,231,000studentssatfor
9
SeeFuturoLaboralathttp://www.futurolaboral.cl/.
10
TheshiftfromPAAtoPSUreflectedanewvisionoflearning.ThePAAexaminationwasbasedonthe
assumptionthattheskills(oraptitudes)measuredinthetestswereinnate,andhencewerenotaffected
by (lack of or poor) education. The PSU model, on the other hand, recognizes that both contents and
skillsareaffectedbyeducation.Hence,itisfairerthatthetestmeasuresthecontentsandskillsthatall
studentshaveachancetolearnatschoolthroughthenationalcurriculum.
8 MaríaJoséRamírezthe examination in 169 testing centers. The tests are administered by external
administratorsunderstrictstandardizationandsecurityconditions.
Atighttimelineallowsforonlytwomonthsbetweenthetimestudentstake
the examinations and when they apply to university. The PSU is administered
once a year at the end of the school year (usually midDecember) at the same
timeinthewholecountry.ExaminationresultsarereleasedmidJanuaryandthe
university selection process concludes at the end of that month. This tight
timelineisnecessaryforallowingtheacademicyeartostartinMarch.
Results are confidential, and can be accessed online using the students’
nationalidentificationnumber.Theresultsincludetheexaminationscoresfrom
eachtest,andascorebasedonthestudents’secondaryschoolgrades,11bothof
which are also used to apply to university. Students apply to the university
programs of their preference using a weighted score. University programs
weightthescoresfromthedifferenttestsandfromtheschoolgradesdepending
ontherelativeimportancegiventoeachone.
Chilehastwouniversityapplicationprocesses,whichrunonparalleltracks.
Thecentralizedapplicationservesaroundhalfoftheuniversitiesinthecountries
(33/60).Studentscanapplytotheuniversitiesandrelatedcareeroptionsthrough
anautomatedsystem,anduniversitiesselectstudentsbasedontheirscoresand
places available. For the other universities, students have to apply to each
institutionseparately,makingthewholeprocesslessefficientandcumbersome.12
EnablingContext
This subsection describes the main features of the enabling context of the
university entrance examination program in Chile as of 2012. This includes its
governance and institutional framework, human resources and funding, policy
regulations,andpublicsupport.
Thebureauinchargeoftheexamination,theDepartamentodeEvaluación,
MediciónyRegistroEducacional(DEMRE),isasemiautonomousdepartmentof
Universidad de Chile. DEMRE is accountable to two institutions: the provost
office of Universidad de Chile (Vicerrectoría de Asuntos Académicos) and the
Council of Rectors. The provost office is responsible for overseeing all issues
related to the yearly budget of DEMRE, its personnel, administration,
equipment,andinfrastructure.
TheCouncilofRectorsoperatesasanautonomousexaminationboard.Ithas
an examination committee (Consejo Directivo para las Pruebas de Selección
UniversitariayActividadesdeAdmisión)thataddressesallstrategicissues(both
technical and political) regarding the PSU. The committee is supported by an
11
Secondary school grades (which are on a scale of 1–7) are transformed into the same scale of the
examinationtests(whichareonastandardizedscalewithM=500andSD=110).
12
ThecentralizedapplicationprocessbenefitsallthetraditionaluniversitiesexistinginChilebefore1981.
In2012,forthefirsttimenewerprivateuniversitieswereallowedparticipateinthecentralapplication
process.Eightnewuniversitiesjoinedthecentralprocessthatyear.
DevelopingtheEnablingContextforStudentAssessmentinChile 9
advisoryboard(ComitéTécnicoAsesor)thatliaisesbetweenthecommitteeand
theexaminationbureau,DEMRE.
The governance of the examinations has legitimacy problems. The Council
ofRectorsdoesnotrepresentmostoftheuniversitiesinthecountry(35outof60
universities are excluded). Instead, it only represents the “traditional”
universitiesexistinginChilebefore1981.13Thecouncilisformallyheadedbythe
MinistryofEducation,but,inpractice,UniversidaddeChileisitsmostpowerful
member.
DEMREisresponsibleforexecutingtheexaminationpoliciesagreedbythe
Council of Rectors. It is in charge of all core examinations areas, including
developing and administering the examinations, data processing, publishing
examination results, and student applications to universities. The bureau has
seven units: test construction, student registration, data processing, logistics,
communications,researchanddevelopment,andmanagementandbudget.
DEMRE has stable and professional staff to carry out its duties. It has 75
public employees, most of whomwork full time (both with term and nonterm
contracts).ManyofthemhavebeenworkingatDEMREformorethan10years.
Thestaffincludesteachersandsubjectmatterspecialists,statisticians,computer
specialists, media specialists, and administrative and support staff. Around 10
percent of the staff has a master’s or doctorate degree; in some cases, these
degrees are in areas directly related to student assessment (for example,
psychometrics).
Themainsource of professional development for the examination staff has
been onthejob training, and short training courses. Around 30 out of the 75
DEMRE staff have benefited from annual intensive twoweek courses to learn
aboutkeyassessmenttopics(forexample,testconstruction,scoring,scaling,and
logistics).DEMREcontractedthesecourseswithaprestigiousassessmentcenter
intheUnitedStates(EducationalTestingServices,ETS).
The examination bureau has a stable budget that is negotiated on a yearly
basiswithitshostinstitution,UniversidaddeChile.ThebudgetreachedUS$6.6
million in 2011. The budget covers all direct operational costs of DEMRE,
includingpersonnel,testdesign,administration,dataprocessing,dissemination,
students’ enrollment in the examination, and their applications to universities.
Thebudgetdoesnotcoverthecostsofsomecoretasksthatareabsorbedbythe
university, such as writing test questions, management, and administration by
the university provost. In 2011 and 2012, the budget from the Ministry of
Educationallocated US$582,000for an international and independent review of
theexamination.14
13
In 1981, a law was passed allowing the creation of new private universities. Thirtyfive private
universities exist today in Chile based on this law. These universities are not part of the Council of
Rectors.
14
All data based on Chile’s national budget (http://www.dipres.gob.cl), on the examination bureau
webpage (www.demre.cl),onUniversidad de Chile’s official budget, andon personal communications
withthebureau.
10 MaríaJoséRamírezStudents have to pay a fee to enroll in the PSU examination, but state
subsidiesare available for thosewho cannot affordit. In2011, 272,000students
enrolledinthetest,ofwhich150,000benefitedfromtheUS$8millionavailablein
state subsidies to pay the examination fee (around US$50 per student). The
examination fee allowed Universidad de Chile to collect US$14.9 million, more
thandoubleoftheDEMREyearlybudget.Moreinformationwouldbeneededto
knowiftheexaminationfeereflectstherealcostsofthePSU.
In Chile, there is no national law that recognizes the university entrance
examination. Instead, every year the Council of Rectors publishes the PSU
regulations. Universities have autonomy to decide their policies, and they
recognizethePSUexaminationintheiradmissionregulations.
Publicsupportfortheuniversityentranceexaminationshasbeensuffering.
WhilethePSUisfreeofcorruption,agrowingnumberofstakeholdersperceive
itasunfair.Lowerincomestudentsgetsystematicallylowerscoresthanhigher
income students, and therefore are less likely to get into a quality university.
Supporthasalsosufferedasaconsequenceofthegovernanceoftheexamination
board(OECDandWorldBank2009,chapter5).
MechanismsofChange
Whiletheuniversityentranceexaminationprogramgrewtonationalscaleafter
being launched, its governance and institutions, personnel, and the policy
framework supporting it do not seem to have changed accordingly. Only the
examinationsbudgetseemstohavebeenadjustedtothenewcontext.
Becauseofhistoricalreasons,theexaminationbureauhasalwaysremaineda
unitwithinUniversidaddeChile,althoughwithdifferentnames,functions,and
organizational dependency. The major change occurred in 1996, when the
examinationbureauandtheadmissionofficeoftheuniversitywereseparatedto
avoid conflicts of interest. The current institutional arrangement seems out of
linewiththefactthattheexaminationisnowanationalpublicgood.
Thestaffingoftheexaminationbureauhasremainedrelativelystablesince
the1990s,despitethegrowingnumberofstudentstakingtheexaminationsand
the greater technical sophistication of the tests. Changes in technology have
allowed for greater efficiency. For instance, whereas before, the DEMRE
personnel had to manually register all students, today students enroll for the
examinationonline.Accordingly,theexistingpersonnelhavebeenreallocatedto
newfunctions.
The policy framework that supports the examination (PSU regulations and
universityadmissionpolicies)hasalsoremainedessentiallythesame.Themain
variation is that all new universities have adopted the PSU in their admission
policies.
The yearly budget for the examination office increased significantly in the
lastdecade;nevertheless,thisincreasedidnotmatchtheincreaseintheamount
ofmoneycollectedviaexaminationfees.Theyearlybudgetfortheexamination
DevelopingtheEnablingContextforStudentAssessmentinChile 11You can also read
