Evaluating E-learning A Guide to the Evaluation of E-learning - Graham Attwell (ed.) Evaluate Europe Handbook Series Volume 2
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Graham Attwell (ed.)
Evaluating E-learning
A Guide to the
Evaluation of E-learning
Evaluate Europe Handbook Series Volume 2Graham Attwell (ed.) Evaluating e-learning A guide to the evaluation of e-learning Evaluate Europe Handbook Series Volume 2 ISSN 1610-0875 2006 This work is licenced under the Creative Commons Attribution-NonCommercial-ShareA- like License. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc- sa/2.0/de/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA. Design and layout: Dirk Stieglitz, Bremen, Germany Printer: Perspektiven-Offset-Druck, Bremen, Germany This brochure is financed by the European Commission within the Leonardo da Vinci-Programme. Disclaimer: The views expressed in this report are those of the authors and do not necessarily reflect those of the European Commission. Neither the European Commission nor any person acting on behalf of the Commission is responsible for the use that might be made of the following information.
Contents Preface . ..................................................................................................... 5 1. Introduction – why do we need new models and tools for the evaluation of e-learning.............................................................. 7 2. Evaluating e-learning – what does the literature tell us?................ 9 3. A Framework for the evaluation of e-learning . ............................ 14 4. Models and theories of evaluation . ................................................ 17 5. Models and tools for the evaluation of e-learning – an overview...................................................................25 6. The SPEAK model and tool ............................................................. 27 7. Tool for the evaluation of the effectiveness of e-learning programmes in small- and medium-sized enterprises (SMEs)........ 30 8. Models and tools for evaluation of e-learning in higher vocational education . ............................................................... 32 9. Policy model and tool ........................................................................38 10. A management oriented approach to the evaluation of e-learning . ...................................................................... 42 11. Individual learning model and tool............................................... 45
Preface
Evaluating e-learning
A guide to the evaluation
of e-learning
W
ho produced this guide?
This guide has been produced as a
report on the work of the Models and
Instruments for the evaluation of e-learning
and ICT supported learning (E-VAL) project.
The project took place between 2002 and
2005 and was sponsored by the European
Commission Leonardo da Vinci programme.
The project was coordinated by Pontydysgu
from Wales.
The following organisations and individuals
took part in the project:
Who produced this guide? • Centre for Research and Education Develop-
ment (CRED), Wales
E-VAL project Jenny Hughes – jenhughes@mac.com
• Interactive Institute, Sweden
Who is this guide for and how can it be used? Cecilia.Katzeff – cecilia.katzeff@tii.se
Laila Abdallah
• Institute for Future Studies, Austria
Friedrich Scheuermann – office@friedrich-
scheuermann.net
Klaus Reich – klaus.reich@futurestudies.org
• IVLOS, University of Utrecht, Netherlands
Wilfried Admiraal – W.F.Admiraal@ivlos.
uu.nl
Ineke Lam – J.I.Lam@ivlos.uu.nl
Yvonne de Jong – Y.C.deJong@ivlos.uu.nl
• Nexus, Ireland
Brian Dillon – brian@nexus.ie
• Pontydysgu, Wales
Graham Attwell – graham10@mac.com
Claire Middleton
• University of Tampere, Finland
Tuula.Heiskanen – tuula.heiskanen@uta.fi
Riitta Kuusinen
Who is this guide for and how can it be used? sections of the guide and vive versa, those
This guide is for anyone interested in the whose focus is more on research, to examine
evaluation of e-learning. the different models and tools which have been
More specifically it may of interest and developed. It is our strongly held belief that
value to teachers and trainers involved in one problem in e-learning, and especially the
e-learning, to education, pedagogy and edu- evaluation of e-learning, is the lack of iteration
cational technology students and researchers, between theory and practice.
to policy makers and planners and to institu- The guide does not claim to be the final
tional managers. answer to the issue of evaluating e-learning.
Whilst the Models and Instruments for the We claim only to have made some modest
evaluation of e-learning and ICT supported progress towards solving a number of impor-
learning (E-VAL) project was primarily fo- tant issues. We hope that others will build on
cused on research, the project resulted in the our work in developing new insights and new
development of a number of new models and models and tools in the forthcoming period.
tools, which were tested in the final stages of The project partners are continuing to develop
the project. the different models and tools described in
This guide provides an overview of the the guide and the evaluate-europe web site
research results of the project including the will provide up to date information on those
outcomes of an extensive literature review developments.
and an overview of different models and ap- The guide itself is only a summary of what
proaches to the evaluation of e-learning and a was produced for the project. Access to more
short review of the different models and tools detailed papers and to the full models and tools
developed through the project. themselves is provided through the project
Thus, it may be of interest to those involved web site and different sections of the report
in the more theoretical research approaches index links to section to that site.
to e-learning and to those who are looking
for ways of evaluating e-learning programmes Contents of the guide
and courses. 1. Introduction – why do we need new models and
The primary focus for the Leonardo da tools for the evaluation of e-learning
Vinci programme, which funded the project 2. Evaluating e-learning – what does the literature
is vocational education and training. But we tell us?
are aware that vocational learning takes place 3. A Framework for the evaluation of e-learning
in a variety of different contexts and institu- 4. Models and theories of evaluation
tional settings. The use of new technologies is 5. Models and tools for the evaluation
contributing to the breakdown of the divides of e-learning – an overview
between different sectors of education and 6. The SPEAK Model and Tool
leading to a widening of the contexts in which 7. Tool for the evaluation of the effectiveness
learning takes place. The models and tools of e-learning programmes in small- and medium-
outlined in this guide have been tested in Small sized enterprises (SMEs)
and Medium enterprises and in community 8. Models and tools for evaluation of e-learning in
education, as well as in the more traditional higher vocational education
vocational college and university settings. 9. Policy model and tool
The format of this guide has designed to 10. A management oriented approach to the evalua-
be flexible, to allow users to dip in and out of tion of e-learning
different sections. Nevertheless, we would urge 11. Individual learning model and tool
those whose primary interests are the practical
and applied evaluation of e-learning to at least
quickly look at the more research oriented
Section 1
Introduction – why do we
need new models and
tools for the evaluation
of e-learning
T
he development of e-learning products
and the provision of e-learning oppor-
tunities is one of the most rapidly ex-
panding areas of education and training.
Whether this is through an intranet, the
internet, multimedia, interactive TV or com-
puter based training, the growth of e-learning
is accelerating. However, what is known about
these innovative approaches to training has
been limited by the shortage of scientifically
credible evaluation. Is e-learning effective? In
what contexts? For what groups of learners?
How do different learners respond? Are there
marked differences between different ICT
platforms? Does the socio-cultural environ-
ment make a difference? Considering the costs
of implementing ICT based training, is there
a positive return on investment? What are
the perceptions of VET professionals? What
problems has it created for them?
E-learning is also one of the areas that
attracts the most research and development
Development of e-learning products is one funding. If this investment is to be maxim-
of the most rapidly expanding areas of edu- ised, it is imperative that we generate robust
cation and training models for the evaluation of e-learning and
tools which are flexible in use but consistent
Is e-learning effective? in results.
“Although recent attention has increased
E-learning is also one of the areas that e-learning evaluation, the current re-
attracts the most research and development search base for evaluating e-learning
funding is inadequate … Due to the initial cost
of implementing e-learning programs,
it is important to conduct evaluation
studies.”
(American Society for Training and
Development, 2001).
The Capitalisation report on the Leonardo
da Vinci 1 programme, one of the biggest
sponsors of innovative e-learning projects
in European VET, also identified the lack
of systematic evaluation as being the major
weakness in e-learning projects.
However, whilst some have been desperately
seeking answers to the question ‘What works
and what doesn’t work?’ and looking for ways
of improving the quality of e-learning, the
response by a large sector of the community
of e-learning developers and practitioners has
been a growing preoccupation with software
and platforms. There has been only limited
attention to pedagogy and learning. The devel-
opment of models and tools for the evaluation
of e-learning can help in improving the quality
of e-learning and in informing and shaping
future development in policy and practice.
Section 2
Evaluating e-learning –
what does the
literature tell us?
T
he Evaluation of e-learning project under-
took a collaborative review of literature
on the subject. Over one hundred reviews
have been added to the project database and
may be accessed through the project web site.
This is what we found out.
Categorising the literature
In general there is an overwhelming prepon-
derance on the evaluation of technology, usu-
ally through ethnographic research methods
(questionnaires to participants).
However, it is possible to identify identified
ten main categories of literature.
Case studies of specific e-training programmes
For the most part these are descriptive rather
than analytic or predictive, predominantly
American, mainly located in a Higher Edu-
cation rather than vocational training envi-
ronment and focused on the ‘virtual class-
room’ model. They also tend to be restricted
Categorising the literature to particular subject areas, in particular IT,
languages and engineering disciplines. (This
Comparisons with traditional learning is not necessarily to say that e-learning is
restricted to these areas, rather that they are
Return on Investment (ROI) reports over-represented in evaluation reports.)
Studies on the contribution of evaluation to Comparisons with traditional learning
metadata There are some (but surprisingly few) system-
atic studies that compare e-learning effective-
Issues and gaps ness with traditional learning and which are
empirically robust. Those that exist are mainly
Pedagogic and curricular approaches small-scale studies, often using a matched
pairs design and are frequently of very specific
Programme and policy evaluation instances of e-learning in which the e-learn-
ing methodologies are idiosyncratic and the
conclusions cannot be generalised.
Tools and instruments for evaluation of e-learning ment through traditional pedagogies. An
There is an abundance of literature detail- additional problem is that the designers of
ing tools for the evaluation of e-learning. these benchmarking systems are often locked
However, these are mainly divided into two in to a particular model of e-learning which
types. Firstly there are many on-line data limits their transferability.
gathering instruments for assessing, typically,
the user interface characteristics of software Product evaluation
(e.g. student perception questionnaires) or By far the greatest number of `hits’ on evalua-
secondly, there are devices to record and tion of e-learning are reports describing (and
analyse usage by duration and frequency of extolling the virtues of) particular education
log-in, pages accessed, user profile etc. Many software. The vast majority of these reports
of these are sophisticated in their design and are commissioned or published by the soft-
ingenuity but lack guidance on interpretation ware developers. This is not to question the
and analysis. usefulness of these reports or necessarily to
doubt their validity but evaluation of ‘de-
Return on Investment (ROI) reports contextualised’ software is not an acceptable
There are surprisingly few ROI reports, con- substitute for the rigorous evaluation of e-
sidering the huge investments into e-learning learning systems.
at all levels. The majority of those that exist
draw mainly from industry based examples Performance evaluation
and are written from an HRD perspective. The Scrivens (2000) in the USA, uses the term
conclusion is inevitably that the investment ‘performance evaluation’ for what would, in
was cost-effective and represented value-for- European terms, be called student assessment.
money but often the savings are defined in Whilst it is true that an examination of student
efficiency rather than effectiveness with no performance is a powerful indicator of the ef-
long-term impact analysis that takes account fectiveness of e-learning, it is by no means the
of unintended outcomes and consequences. only one. Moreover, a survey of reports on per-
It is also difficult to compare figures across formance evaluation in the context of e-learning
reports because the distinctions between net were mainly concerned with on-line tools and
and gross costs, capital and revenue costs, instruments for examining knowledge-based
displacement of existing funds, costs over time learner performance and could therefore be
etc. are often blurred or missing. Many ROI categorised under that heading.
type evaluation reports appear to be justify-
ing investment rather than evaluating it and Handbooks for the evaluation of e-learning
more geared to an audience of shareholders There is an increasing number of handbooks
rather than researchers. for e-learning which focus primarily on evalu-
ation. The evaluation methods and tools differ
Benchmarking models widely. What they do have in common is that
There have been several attempts to generate they recognise the importance of evaluation
sets of criteria for quality assuring e-learning. and many propose that evaluation should be
However, these tend to be skewed towards an integral part of any e-learning initiatives or
proposing quality standards for e-learning development. In this regard, they tend toward a
systems and software which often disregard management model of evaluation; the primary
key variables in the wider learning environ- aim of the evaluation is to provide feedback
ment or are based on criteria associated with to influence e-learning implementation and
evaluating traditional learning processes (and future development.
which disregard the technology) or criteria Many of the handbooks appear to have been
associated with measuring learner achieve- produced by education advisors and advisory
10services and agencies. They are frequently quite lihood that learning technologists focus on
basic and lack a theoretical approach. A sur- the development of tools for evaluation, whist
prising number cite The Evaluation Cookbook those coming from evaluation studies are
and appear to have borrowed many of their more likely to consider how traditional models
ideas from this influential publication. More of evaluation can be applied to e-learning.
worrying is the technology centred approach Psychologists are more likely to undertake
that many of the guides espouse. comparative studies. What is the importance
of this? It suggests that we need teams of
Meta-studies evaluators or of researchers from different
There are a number of meta-studies of the disciplines to develop cross-disciplinary ap-
evaluation of e-learning, all of them based on proaches to the evaluation of e-learning
US literature. These are attempting to answer
the question of the effectiveness of e-learning Issues and gaps
by combining or bringing together the results First, the positive things which have emerged
of a series of different studies to provide a form the literature review.
larger sample base. Although, obviously, the There appears to be a growing realisation
methodology is open to some question, these of the importance of evaluation. This seems
studies are interesting and offer a new ap- to be linked to concerns that e-learning is
proach to the issue, not succeeding in the way that had been ex-
pected. Evaluation is needed to gain a better
Studies on the contribution of evaluation understanding of the problems and issues
to metadata regarding e-learning.
This cannot be called a category of literature There is also an increasing focus on evalu-
on evaluation as such, as this literature only ation methodologies and a realisation that
includes evaluation as one of the factors to the evaluation of e-learning is complex and
be taken into consideration in developing requires the development of new models and
metadata. But, it is very important for the approaches.
future of e-learning evaluation and will be
explored further in the notes below. Evaluation and assessment
However there remains confusion between
Disciplinary backgrounds evaluation and assessment. To some extent
It was notable that there are considerable this can be explained by different understand-
differences in the disciplinary background ings of evaluation in different countries and
of the authors and in the journals in which by the linguistic confusions between the two
they were published or conferences they were processes. However, I think it goes further
presented at. These include: than this. There appears to be an over reliance
• Education and training journals and con- on assessment or achievement as the basis
ferences for evaluation. That is not to say that learner
• Journals and conferences relating to the achievement is not an evaluation factor. But
use of Information and Communication the assumption that if something has not
Technologies for learning been assessed it has not been learnt is surely
• Specialist evaluation journals and confer- wrong. Furthermore, such an approach misses
ences informal learning and learning not included
• Journals and conferences around the psy- in the assessments.
chology of learning.
The disciplinary background of the research- Pedagogic and curricular approaches
ers/authors tends to determine their approach Two big gaps stand out in the literature re-
to evaluation. For instance there is more like- viewed. The first is the complete lack of any
11evaluation – or attempts to evaluate – ped- Perhaps the lack of such studies just reflects
agogic approaches of e-learning. This is in a time delay, before evaluation catches up
contradiction to the increasing emphasis of with technical and e-learning development.
e-learning researchers and developers on the Yet I think it reflects the need for accom-
pedagogy of e-learning. panying evaluation where evaluation takes
Evaluation approaches still tend to focus on place alongside technical development and
the functionality of the learning technologies innovation. Secondly, it suggests to me that
and not on the learning which is facilitated a discourse is needed between technical de-
by the functionality. If pedagogy is such an velopers and innovators and evaluators with
important factor in e-learning, then an impor- a better understanding or both of what the
tant role for evaluation is to assist researchers other is doing.
through providing an understanding of the
impact of different pedagogic approaches. Programme and policy evaluation
At the same time, in the European literature The second gap in the literature review is
there is often an assumption that construc- programme and policy evaluation. Every
tivism is the proven and best pedagogy for country and most large institutions have
e-learning. Furthermore, it is often hard to active policy driven programmes to develop
know what the authors mean or understand e-learning. However, there are few programme
by constructivism. In the way the term is level evaluation reports, and still less models
being used constructivism is only a means or theoretical approaches for evaluating e-
of describing in one category all the varie- learning policy. This is both surprising and
ties of creative, perceptive and innovative worrying. Without evaluation, how do we
approaches to facilitating the acquisition of know which policy approaches are working
new knowledge. and which are not? I remain suspicious that
Linked to this lack of clear focus on peda- e-learning is merely seen by policy makers as a
gogy is a similar gap in the evaluation of cur- ‘good thing’ and that money is being invested
ricular approaches. Why is this important? with little understanding of where or why.
In the technical developers world a debate
has been raging for the last three years over Metadata
something called learning objects. Learning As I mentioned earlier there are a growing
objects have been seen as one of the main number of references to evaluation in the
answers to the problem of sustainable con- technical literature around metadata. I think
tent creation. A learning object is any digital, this is a significant development. Metadata
reproducible and addressable resource used is simply data about data. However, it is cru-
to perform learning activities or learning cial in allowing computers to know what
support activities, made available for others exists on other computers and providing a
to use. The problem is that is order to develop machine-readable description of learning
reusability, objects need to be granular. This resources. At first it had been assumed that
means that e-learning developers are pro- creators of learning resources would provide
moting modularity as a curricular process, the metadata according to some kind of agreed
driven by technical need. Many critics are common schema or standard. More recently,
concerned that modularisation and granular is a realisation that in education many dif-
learning object will challenge the coherence of ferent people have an interest and role in
learning programmes. Since this is such a key providing the metadata associated with any
debate in fort e-learning development, surely given object – developers, teachers, trainers,
evaluation should focus on what is happening curriculum developers, technical developers,
and whether modular programmes built from librarians and archivist, students and trainees
learning object can work. and evaluators. Clearly, it would be absurd
12to expect material developers or creators to
add an objective description of the quality of
their own learning materials and evaluation
may play a critical role in describing quality.
Technically the debate is around distributed
metadata and how all the different data which
becomes naturally associated with an object
or learning materials in the course of their
development, deployment and use, can be
found and aggregated.
Once more, I think this growing debate
shows a necessity for evaluators to work along-
side technical developers and at a more theo-
retical level for a discourse around ideas.
13Section 3
A framework for the
evaluation of e-learning
F
rom a baseline of practice of attempting
to evaluate many e-learning programmes,
one of the biggest problems has proved
to be handling the number of variables which
potentially impact on the effectiveness of the
programme and deciding what constitutes
dependent, independent and irrelevant vari-
ables in a given situation.
Literature reviews and the study of existing
evaluation practice, suggests that many evalua-
tion tools and schema tend to disregard – con-
sciously or otherwise many of these variables.
Much of existing practice is overly focused
on the technology – and on learner reaction
to the use of technology. Socio-economic
factors such as class or gender are seldom
considered and even learning environment
variables such as the subject environment are
all too often ignored.
Not only does this result in limitations
in the data available on the use of ICT in
learning but the limited recognition of the
different variables can distort analysis of the
weaknesses (and strengths) in current e-learn-
ing provision.
The evaluation of e-learning project has
What framework? developed a more comprehensive framework.
Over several e-learning evaluation projects,
What factors are being disregarded or five major clusters of variables have emerged;
edited out of the framework? individual learner variables, environmental
variables, technology variables contextual vari-
How might this framework be used? ables and pedagogic variables. Each of these
can be disaggregated into more precise groups
and further disaggregated until individual
variables can be identified and isolated.
Of course we recognise that no single evalu-
ation model or tool, much less evaluation
study, can address every variable. But, we
believe in approaching and designing any
evaluation it is important to be conscious of
14what factors are being disregarded or edited may be context determined) for mapping and
out of the framework. coding existing work into the effectiveness,
Individual learner variables include efficiency and economy of e-learning irrespec-
• physical characteristics (e.g. age, sex, physi- tive of whether this is an evaluation or an
cal abilities) independent research study. Methodologies
• learning history, (negative/positive experi- are cross-referenced against the variables
ence, level of attainment, duration, recency being studied and major areas of omission
etc.) can be identified that in turn will suggest a
• learner attitude (positive/negative) future research agenda.
• learner motivation (high/low) Secondly we are using the clusters of vari-
• familiarity with the technology ables can be sued for proposing and testing
Learning environment variables include hypotheses. Any one cluster can act as the
• the immediate (physical) learning environ- dependent variable; the other four then oper-
ment ate as independent variables. For example, at
• the organisational or institutional environ- the micro level, part of the Eval project has
ment tested the hypothesis that the effectiveness of
• the subject environment different e-learning pedagogies will depend
Contextual variables include on particular individual learning histories.
• socio-economic factors (e.g. class, gen- Another survey explored whether the effec-
der,) tiveness of particular technologies depends on
• the political context (e.g. who is fund- gender. At a macro level we are also interested
ing/paying for the e-learning and for what in whether the presence (or absence) of some
reason?) individual variables or clusters of variables are
• cultural background (e.g. how highly is more significant than others in determining
learning/e-learning valued?) the effectiveness of e-learning and, if so, can
• geographic location (e.g. country, language, they be weighted in some way? Is the profile of
urban/rural) the learner more significant than the nature of
Technology variables include the learning environment? Is the effectiveness
• hardware of the technological solution outweighed or
• software, enhanced by particular environmental vari-
• connectivity, ables? Which is more important – getting the
• the media software right or the learner support right?
• mode of delivery, Can we use statistical techniques such as fac-
Pedagogic variables include tor analysis to see which variables ‘cluster’
• Level and nature of learner support sys- together and impact on each other?
tems We were not able to test every variable in
• accessibility issues. the limited time and resources available to us
• Methodologies through he project. However, the research we
• Flexibility were able to undertake proved the value of the
• Learner autonomy framework as a tool for research and confirmed
• Selection and recruitment the validity of the framework design.
• Assessment and examination Thirdly, we have found it a useful framework
• Accreditation and certification for evaluating and researching the effective-
ness of specific e-learning projects and pro-
How might this framework be used ? grammes. The evaluation of e-learning, and
Firstly the framework can be used to develop a research into the evaluation of e-learning, has
robust classification system with clearly identi- been dominated by descriptive ethnographic
fied levels of aggregation, (which themselves studies, rather than interpretation and analyses
15and there is a predominance of ethno-meth-
odological approaches, in particular, heavily
contextualised case studies. The relatively small
number of empirical studies has focussed on a
limited number of variables. The best of these
have controlled for variables other than those
under study; the worst have simply discounted
them. As the databank of research results is
built up, particularly as the different variables
are ‘weighted’, it becomes easier to identify the
irrelevant variables and allow for the impact
of others. It also allows predictions to be made
which can short circuit the search for an ap-
propriate evaluation methodology.
16Section 4
Models and theories
of evaluation
E
volution of evaluation theory
Evaluation as a formal activity that
we would recognise, has existed for a
surprisingly long time. One of the earliest
recorded was the evaluation of the effective-
ness of lime-juice in preventing scurvy in
sailors – commissioned by the British navy
in the 18th century! The French make even
earlier claim and say that the Norman armies
conducted an evaluation of the relative ef-
fectiveness of the crossbow and the longbow.
Unfortunately, on the basis of the evaluation
findings, the management decision was to go
for the crossbow and the rest, as they say, is
history!
However, evaluation has only become a rec-
ognised area of academic study since about the
1960’s. It is probably true to say that evaluation
started as a field of practice and the theory was
derived from it. As it evolved, so ideological
disputes developed alongside disagreements
Evolution of evaluation theory on definitions, terminology, ethics and so on.
FitzPatrick, Sanders and Worthern in 2004
Philosophical /ideological difference identified nearly 60 different models in the
30 years between 1960 and 1990 alone. This
Differences based on defining value or proliferation of models was bewildering for
worth the practitioner, especially as many of these
models and the tools they generated had no
Differences according to discipline or field of obvious theoretical perspective.
application Why is this a problem? Why should practi-
tioners need a theoretical framework? Simply,
Differences in practice a ‘good’ theory will set out the assumptions
that it is making and on which its logic is
A classification schemata for evaluation predicated. Different theories make different
approaches assumption and generate models that will
be based on different pre-conceptions and
definitions of evaluation, which in turn lead
to very different practices.
17Deriving a taxonomy of evaluation approaches and conclusions. Thus, the evaluation pro-
Many researchers have tried to make sense cedures are ‘internal’ to each evaluator and
of this huge diversity of models and theories are not explicitly understood or reproducible
and to find some way of classifying them. by others.
However, even they could not agree so now Until 20 years ago, objectivism in evaluation
we have the problem of trying to classify the was a goal to be aspired to. However, the same
classification systems! criticisms levelled at the usefulness of positiv-
All this is by way of saying that what follows ism in the social sciences in general were also
is only one framework for distinguishing be- applied to objectivism in evaluation.
tween different theories of evaluation and you Campbell (1984) summed it up:
may well come across others. This framework, “twenty years ago positivism dominated
which we find comprehensive and useable, was the philosophy of science...Today the tide
devised by FitzPatrick, Sanders and Worthen has completely turned among the theo-
(2004) who we have quoted at length. rists of science in philosophy, sociology,
and elsewhere. Logical positivism is al-
Philosophical/ideological differences most universally rejected.”
Approaches to evaluation may differ funda- This point of view has been upheld by many
mentally because their underpinning philoso- writers on evaluation and even if it is not
phy or ideological base is different. That is, universally subscribed to, probably represents
different evaluation theories will be based on a general trend. The major argument is that
different assumptions about the way the world unlike traditional scientific research, evalu-
works and so the models and practices based ation has to deal with complex phenomena
on those theories will be different as well. By in real world settings, take into account a
and large, we can locate them on a continuum multiplicity of stakeholders, unstable and
from objectivist to subjectivist. unpredictable systems and requires a high
Objectivism is equivalent to the empirical level of human interactivity.
tradition in scientific research (positivism) The other criticism is that objectivism de-
and focuses on data collection and analysis pends for its validity on its ‘scientific’ meth-
techniques that produce results that are re- odology and is only credible if you happen to
producible and verifiable by other evaluators value that methodology. We would argue that
and to generate conclusions that are evidence objectivism conceals hidden values and biases
based and which can be ‘scientifically’ jus- of which many evaluators are unaware – even
tified. So the evaluation is ‘external’ to the the choice of data collection techniques and
evaluator who is simply someone technically instruments is not value-neutral but this is not
competent and proficient in the application recognised or else ignored by many so-called
of procedures. objective evaluations.
Subjectivism is based on: Despite the reaction of the theorists, how-
“... an appeal to experience rather than ever, the message does not seem to have filtered
to scientific method. Knowledge [of the through to the client base and the majority of
evaluator] is conceived as being largely evaluation consumers, particularly in educa-
tacit rather than scientific.” tion (and the public sector in general), are still
(House 1980 in FitzPatrick, Sanders wedded to the idea of objective evaluation and
and Worthen 2004) ‘finding the facts’.
The validity of a subjectivist evaluation de- The major criticism is that subjectivist evalu-
pends on the relevance of the evaluators’ ation often leads to contradictory conclusions
background, their experience and expertise, that cannot be reconciled because the processes
the keenness of their perceptions and their which led to the conclusions is largely inside the
insightfulness in generating interpretations evaluators head and so cannot be replicated.
18Differences based on defining value or worth there are other major divides based on meth-
We can also distinguish between different odological differences that are not necessarily
theoretical approaches depending on how rooted in a particular philosophical approach.
they define value and make judgements, rather For example, many evaluators (both theoreti-
than on their philosophical differences. This cians and practitioners) and also many clients
time the continuum extends from ‘utilitarian’ tend to view qualitative and quantitative ap-
to ‘intuitionist-pluralist’. proaches as different paradigms. We do not
‘Utilitarianism’ is a philosophy based on subscribe to this view, believing that this is
maximising happiness in society. Utilitarian not a fundamental divide but simply a way of
approaches to evaluation are based on the describing evaluation approaches by types of
premise that the best programmes are those data that are used. Nevertheless, we recognise
that achieve the greatest good for the great- this as an important distinction for others and
est number. The evaluator will try and assess one that impacts on the overall evaluation
overall impact in terms of total group gains methodology and the tools used.
by using average outcome scores against the
criteria selected to determine worth. Again, Differences according to discipline or field
governments and the public sector tend to of application
be adherents of this type of evaluation as Evaluation is a relatively young field and still
it lends itself to large-scale comparisons of draws heavily on methodologies adapted from
programmes and mass aggregation of data. anthropology, sociology, psychology, philoso-
Managers and public programme administra- phy, economics and mathematics. One of the
tors tend to be the main audiences. consequences is that evaluation approaches
According to FitzPatrick et al, the intui- can be grouped around their parent discipline
tionist-pluralist approach is at the other end so we tend to find ‘mathematical approaches’
of the spectrum and is based on the premise or ‘sociological approaches’. More recently the
that value depends on the impact of a pro- search for new models has widened its net and
gramme on each individual and the ‘greatest evaluation theorists such as Smith (1981) are
good’ is that which maximises the benefits for trawling newer disciplines such as investiga-
all stakeholders. This evaluation focus will be tive journalism, photography, storytellling,
on the distribution of gains (for example by philosophical analysis, forensic pathology
cultural or sub-cultural demographic groups and literary criticism for new ideas.
such as ethnicity or gender or age) or distri- Evaluation theory has also developed in a
bution of benefit across stakeholders (e.g. social context and practitioners work in dif-
learners, administrators, delivery agencies, ferent cultures, different sectors, with different
funding bodies, the public). There can be no target groups and different audiences. Conse-
common index of ‘good’ but rather a plural- quently, different approaches and models have
ity of criteria and judges. The evaluator is no tended to emerge based on these factors. For
longer an impartial ‘averager’ but a portrayer of example, ‘education programme’ evaluation
different values and needs. The merit or worth has developed along a different trajectory
of any programme depends on the values and than, for example, the health services. Whilst
perspectives of whoever is judging it and all many writers would argue that this is not a
stakeholders are legitimate judges. true theoretical divide, ‘theory-in-practice’
is a powerful determinant of evaluation ap-
Methodological differences proach and also stakeholders perceptions and
Although there is a strong correlation between expectations of the evaluation process.
an evaluator’s ideological approach and the
methodology and techniques they will use
(because of necessity one drives the other),
19Differences in practice process of evaluation. Others claim that not
The above distinctions are all based (loosely) only is this unnecessary but can, on occasions,
on theoretical divisions. However, FitzPatrick be unhelpful.
et al also point out that differences in approach
can be practice-driven. A classification system
Firstly, evaluators disagree about whether The above analysis is interesting and helps
evaluators should simply provide information understand the major theoretical divides in
so that decision makers can make the value evaluation. However, it does not get us far in
judgements. Others, would say that the evalu- terms of systematically examining the varia-
ator’s report to decision makers is incomplete tion between particular evaluation approaches
if it does not contain a value judgement. because although those approaches could be
Secondly, evaluators differ in their percep- positioned on each of the above ‘dimensions’,
tion of their own role and their place in the their location would vary from one dimension
evaluation process. Who has authority and to another. The next section tries to provide
responsibility for the evaluation and to whom some answers.
should the evaluator be accountable and an- Many evaluation theorists have attempted
swerable? If one evaluator sees his role as a this but we are going to stick with the solu-
‘critical friend’ and another as ‘inspector’ or tion put forward by Fitzpatrick, Sanders and
‘judge’, then this will obviously influence the Worthen (1983). We are proposing to use their
way they conduct an evaluation and also the work – with some modifications – partly in
conclusions they draw. the interests of consistency (having referenced
Thirdly, evaluators will be limited by their them heavily so far) and partly because they
prior experience both in evaluation and also set out very clearly the thinking and rationale
by their own discipline or professional back- underpinning their classification system.
ground. Evaluation skills and knowledge are For the purist it is a less than satisfactory
cumulative. Previous exposure to frequently taxonomy as the approaches do not neces-
recurring problems will affect the way an sarily differ from one another along the same
evaluator works. On the one hand it will prob- dimension. However, they are pragmatic as
ably mean the evaluator is quicker to detect they conveniently represent the major clusters
problems, to identify issues of concern and of models and approaches in use today.
make more insightful judgements. On the
other hand, it will also mean that the evaluator’s A classification schemata for evaluation approaches
perceptions in a new situation are unlikely to FitzPatrick et al identify 5 major clusters of
be ‘neutral’. evaluation approaches:
Fourthly, evaluators have different views • Objectives oriented approaches
about what skills and expertise they should • Management oriented approaches
possess. Evaluators are frequently chosen on • Consumer oriented approaches
the basis of their expertise or practitioner • Expertise oriented approaches
base in the field being evaluated rather than • Participant oriented approaches
on the basis of their skills and experience as However, to this we propose to add Van der
an evaluator. This is gradually changing but Knapp’s ‘learning oriented approach’.
as evaluation is becoming increasingly profes- These 6 categories fall more or less along
sionalised and recognised as a specialist area a continuum from utilitarian to intuition-
in its own right, so professional evaluators ist-pluralist so there is some logical basis in
are becoming specialised within the area. addition to its convenience and accessibility
Some evaluators would argue that specialist (see figure 1).
knowledge of the field being evaluated is a
pre-requisite for the credibility of the whole
20Objectivist subjectivist
Rationalist – positivist
Utilitarian intuitionist-pluralist
Naturalistic &
Objectives Management Consumer Learning Expert
participant
oriented oriented oriented oriented oriented
oriented
A Continuum of Evaluation Models
Figure 1: A Continuum of Evaluation Models
Objectives orientated evaluation approaches needs of managers, policy-makers, adminis-
Objectives-orientated evaluation is based on trators and practitioners.
the idea that the purposes, goals or targets of Developers of this approach have tradition-
a project are determined at the start and the ally relied on a systems approach to evaluation
evaluation process should establish whether in which decisions are made about inputs,
these have actually been achieved – and, if not, processes and outputs based on logic models
why not. It is very similar to another approach and cybernetic theory. However, more recent
known as ‘a systems approach’ to evaluation developments have highlighted different levels
and both are very popular with public sector of decision and decision makers and have fo-
agencies who are concerned with justifying cussed on who will use the evaluation results,
expenditure and performance measurement. how they will use them and what aspect(s) of
It is sometimes called ‘goal-driven’ evaluation, the system they are making decisions about.
in contrast with other approaches, which are Not surprisingly, it is the model preferred
called ‘goal-free’. by many managers and management com-
There are many examples of objectives mittees but the downside is that the needs of
orientated models; the earliest is probably other stakeholders are ignored.
Tyler’s and more recently, Provus’s Discrep- Stufflebeam’s CIPP model is one of the
ancy Model. most popular in management-orientated
The disadvantages are that this sort of ap- evaluation.
proach can miss important outcomes if they
were not included in the original objectives Consumer orientated approaches
nor does it challenge the value of the objec- Consumer orientated approaches to evaluation
tives themselves adopt the perspective of the end user of what-
Management orientated evaluation approaches ever service or product is being provided. For
The management-orientated approach to this reason they tend to be summative, rather
evaluation is meant to serve decision mak- than formative and are concerned primarily
ers. Its rationale is that evaluation information with product evaluation. Consumer-orientated
is an essential part of good decision making evaluation relies heavily on criteria referenced
and that the evaluator can be most effective evaluation techniques such as benchmarking or
by focussing the evaluation products on the kite marking and is understandably popular with
standards agencies and ‘watchdog’ organisations.
21Michael Scrivens ‘Key Evaluation Checklist’ is included it because it is an approach that we
probably the best-known example. personally use more than any other.
The major disadvantage of a consumer-ori- The operating principle is that the purpose
entated approach is that it is a ‘backward- of evaluation is to contribute to some form of
mapping’ approach and does not help make collective or organisational learning. Different
predictions about future impacts. It also tends models within this overall approach are based
to play down the nature of human interaction on different theories and types of learning
with the products being evaluated. including ‘corrective’ or behavioural learn-
ing, cognitive learning and social learning.
Expertise orientated approaches The outputs and processes of the evaluation
Expertise orientated evaluation is based on the form the inputs of the learning.
notion of ‘connoisseurship’ and criticism and The pioneer of work in this field was Peter
relies on the subjective professional judgement Van der Knaap. More recently we have ex-
and expert knowledge of the evaluator. This is tended the approach to include evaluation
the oldest form of evaluation and is still very as a contributor to knowledge creation in an
popular despite its limitations. organisation.
Expertise-orientated evaluation may be for- Learning-orientated evaluation approaches
mal or informal, based on individual expertise are still not widespread but are beginning to
or, more usually, on the collective expertise of gather momentum in the social agency sector,
a panel. The opinions of multiple experts is in education establishments and in voluntary
popularly believed to minimise bias, though organisations.
this does not always follow! It relies far less The main limitations of this approach is that
on external tools and instruments than other it does not lend itself to ‘mass surveys’ as it
forms of evaluation and more on the experi- relies heavily on personal interaction between
ence and wisdom of the evaluator. the evaluator and the project team and the
Many public systems are based on expertise evaluator’s understanding of the learning needs
orientated evaluation – for example the jury of the organisation. Also, within this overall
system, school inspection system, licensing approach there are very disparate models,
agencies, review boards, the refereeing system some requiring a high level of commitment
for academic journals, national commissions to the process, which may be lacking.
and enquiries and so on.
Many organisations expect this type of Participant-orientated evaluation approaches
evaluation if they employ an external evalu- An increasingly popular approach that dif-
ator and the notion of evaluation by ‘peer fers fundamentally from all the others as it
review’ is still the dominant model in most takes the needs of project participants as its
professional associations. starting point. This is not to say that the other
The disadvantages are obviously the high reli- approaches ignore participant needs but that
ance on the assumed expertise of the evaluator for the most part benefits for participants
and a lack of explicit and published standards. represent the end point of the evaluation and
Also, the credibility of results is affected by not the beginning.
the status of the evaluator but equally the Participants are not simply the direct ben-
credibility of the evaluator is often affected eficiary target group of a project but will
by the results. also include other stakeholders and potential
beneficiaries. Thus, an educational project for
Learning-orientated evaluation approaches women returners would include the learners
This is a relatively new group of approaches themselves, the project staff, the management
and not one that was included in FitzPatrick team and the funders but may also include
et al’s classification. Nevertheless we have the wider community, the learners families,
22the schools attended by the learners’ children, Boyle, P. G., Planning Better Programs, New
childcare agencies or whatever. York: MacGraw-Hill Book Company,
Participant-orientated evaluation does 1981.
not usually follow a formal plan drawn up C.L.Taylor and Carl E. Beeman2 (1992) Evalu-
in advance; rather it looks for patterns in ation for Accountability: An Overview,
the data as the evaluation progresses. Data is University of Florida
gathered in a variety of ways, using a range Describing Innovation as a prerequisite for
of techniques and culled from many different innovation, Hughes and Attwell 2000,
sources. Understandings grow from observa- www.evlauate-europe.net
tion and bottom up investigation rather than Donald Clarke (1997) Instructional system de-
rational deductive processes. The evaluator’s velopment: evaluation phase, http://www.
role is to represent multiple realities and values nwlink.com/~donclark/hrd/sat6.html, ac-
rather than singular perspectives. cessed November 4, 2003
Participant-orientated evaluation includes Israel, Glenn D. and C. L. Taylor. 1990. „Can
many sub-groups that share all or some of the Response-Order Bias Evaluations?“ Evalu-
above characteristics including Responsive ation and Program Planning. 13(4):365-
Evaluation, Naturalistic Evaluation, Utilisa- 371
tion Focussed evaluation and Empowerment Mendenhall, W., and R.L. Scheaffer. 1973.
Evaluation. Of all the models, probably the best Mathematical statistics with applications.
known and one of the most useful is Stakes Duxbury Press, North Scituate, MA.
Countenance Framework. Patton, M. Q. (1986) Utilization-focused evalu-
Criticisms of this approach are many; bu- ation. Newbury Park, CA: Sage.
reaucrats tend to hate it because of its lack of Project Star, User‘s Guide to Evaluation for
‘objectivity’ and because the outputs of the National Service Programs, http://www.
evaluation are unpredictable. It is difficult to projectstar.org/star/Generic/usersguide.
cost and control. Without a very experienced htm, accessed November 4, 2003
evaluator to facilitate the process, it can de- Rossi, P. H., & Freeman, H. E. (1989). Evalu-
generate from an ‘organic’ approach to one ation: A systematic approach. Newbury
which is chaotic and lacking in focus. Also, Park, CA: Sage Publications.
there may be concentration on process at the Ryans, D. G. (1960). Characteristics of teach-
expense of outputs. ers. Washington, DC: American Council
on Education
References
Administration on Children, Youth and Fami- Scrivens M. (1991), The evaluation thesaurus,
liesThe Program Manager‘s Guide to Evalu- Sage
ation, http://www.acf.dhhs.gov/programs/ Weiss The Interface Between Evaluation and
core/pubs_reports/foreword_pmguide. Public Policy, in Evaluation: The Interna-
html, accessed 4 November, 2003 tional Journal of Theory, Research, and
Allum, N.C Bauer, M.W., Gasgell, G. and. Practice (1999)
(2000) Quality, Quantity and Knowledge
interests, Avoiding confusion in Bauer
and Gasgell, Qualitative research, Lon-
mdon, Sag
Boone, E. J., Developing Programs in Adult
Education, Englewood Cliffs, New Jersey:
Prentice-Hall Inc., 1985.
23Section 5
New models and tools
for the evaluation of
e-learning – an overview
T
he remainder of this guide is given over
to a description of the different models
and tools developed through the project.
We do not suggest this is a comprehensive
selection – but it does illustrate the different
models and tools which can be deployed. Cen-
tral to our project’s findings are that evaluating
e-learning is no different than evaluating any
other form of learning – but that there are
many variables and that the models and tools
must take account of the different aims of the
evaluations and contexts in which e-learning
is taking place.
The following models and tools have been
developed through the project and are de-
scribed in more detail in the following sec-
tion.
Models and tools for evaluation of e-Learning in
higher vocational education
The learner-benchmarking tool developed
for the evaluation of e-learning in higher
vocational education is a consumer tool in
order to evaluate the use of Virtual Learn-
Models and tools for evaluation of e-Learn- ing Environments (VLE). It is based on an
ing in higher vocational education on-line questionnaire including items on the
appraisal of the VLE used by the teachers in
Models and tools for evaluating e-learning their courses.
policy The Learner Evaluation tool is focused on
the guidance or scaffolding teachers provide
Tool for the evaluation of the effectiveness within the VLE and is based on constructivist
of e-learning programmes in small and principles.
medium sized enterprises (SMEs)
A management oriented approach to the evaluation
Evalact of e-learning
The management oriented approach to the
The SPEAK model and tool evaluation of e-learning is a model and tool
designed to provide formative feedback for
24decision making in developing and imple- The SPEAK system has been designed to in-
menting an e-learning programme. It is based corporate a number of inter-related functions
on the CIPP model. There are three variants within a logical framework.
of the tool for a) Educational institutions; • Strategic Planning: producing information
b) small and medium enterprises; d) Large to assist both staff and management to make
companies informed decisions.
• Evaluation: ongoing internal review; cross
Models and tools for evaluating e-learning policy project periodic assessments; with a facility
The models and tools for evaluating e-learn- for external inputs.
ing policy are based on the framework for • Knowledge: networking based on sharing
the evaluation of e-learning. The five differ- information between institutes, countries,
ent groups of variables are further broken regions and themes.
down to provide differentiating factors or The SPEAK model has been developed as an
criteria against which a policy can be evalu- electronic tool distributed by a CD ROM.
ated. The tool is designed for uses in policy Data can be aggregated and queries through
evaluation at any level – national, regional or a server application.
institutional.
Tool for the evaluation of the effectiveness of
e-learning programmes in small and medium sized
enterprises (SMEs)
The model and tool for the evaluation of the
effectiveness of e-Learning programs in SMEs
has been designed to provide an easy-to-
use instrument to carry out a retrospective
evaluation of an e-learning program. This
retrospective analysis should enable enter-
prises to detect weaknesses and strengths
of their learning program with regard to its
organisational, pedagogic and technological
implications.
Evalact
Evalact is an electronic tool, designed to allow
facilitate the evaluation of individual learning.
It allows the creation of on-line questionnaires
and provides rich feedback and statistical
evidence for evaluators. It also allows the
comparison of different evaluation instances
and provides graphical interfaces for assisting
in the analysis of evaluation data.
The SPEAK model and tool
SPEAK is a model and tool for the self evalu-
ation of learning in a group and community
context. It is designed both to facilitate group
discussion and self evaluation and to provide
accumulative data for programme evaluation.
25You can also read
