Peer Reviewing E-Learning: Opportunities, Challenges, and Solutions
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Educational Strategies
Peer Reviewing E-Learning: Opportunities,
Challenges, and Solutions
Jorge G. Ruiz, MD, Chris Candler, MD, and Thomas A. Teasdale, DrPH
Abstract
Peer review is the foundation of of print materials, describe peer review navigation, interactivity, delivery, ease of
academic publication and a necessary issues regarding e-learning materials, updating, distribution, and access. Four
step in the scrutiny of any scholarly work. propose approaches to address the approaches are offered to ease the
Simply defined, peer review is the challenges of peer review of e-learning burden and improve the quality of
attentive, unbiased assessment of any materials, and outline directions for e-learning peer review: develop peer
scholarly work that is submitted for refinement of the e-learning peer review review training, embrace multidisciplinary
formal scrutiny. process. At its core, the peer review of peer review, develop guidelines, and
e-learning materials should not differ provide incentives and compensation.
Although medical school faculty substantially from that of traditional The authors conclude with suggestions
increasingly use technology in clinical manuscripts. However, e-learning
about peer review research.
teaching, e-learning materials are often introduces new demands that impel
not subjected to a rigorous peer review reviewers to consider aspects that are Acad Med. 2007; 82:503–507.
process. The authors contrast peer unique to educational technology,
review of e-learning materials with that including pedagogy, format, usability,
Peer review is an accepted and essential peer review has not been equally applied and reliable evaluation process.8,9 For
step in the publication process of print to educational scholarship.2 e-learning to be fully and widely
biomedical literature.1 Thoughtful recognized as evidence of scholarship,
scrutiny by unbiased peers is the Clinician educators are increasingly using peer review must become standard
hallmark of traditional peer review that technology to complement and enhance practice. Previous approaches to the
enables authors to transform scholarly clinical teaching and learning activities. evaluation of digital materials have
work into scholarship. In addition, peer We use e-learning to refer to the use of focused primarily on content and
review fulfills an essential quality-control Internet technologies to deliver a broad presentation and not on the scholarly and
requirement, because it ensures that array of approaches that enhance pedagogical aspects of e-learning.10 Alur
published materials meet accepted learners’ knowledge and performance.3–5 and colleagues10 used strict inclusion
standards. Print-based scientific journals Common synonyms for e-learning are criteria, looking for the adherence of
have relied on some form of peer review Web-based learning, online learning, medical teaching Web sites to instructional
to assess the scholarship of research since distributed learning, computer-assisted methods reflecting learning principles
the 19th century. However, the process of instruction, or Internet-based learning.
such as critical thinking, independent
Distance learning and computer-assisted
learning, evidence-based learning, and
instruction are two well-known
feedback. They found that the Web sites
e-learning modalities included in our
focused mainly on content, with fewer
Dr. Ruiz is assistant professor of clinical medicine, definition. Distance learning takes place
than 20% of sites using the four teaching
Division of Gerontology and Geriatric Medicine, at locations remote from the point of
University of Miami Miller School of Medicine, methods. Knight et al9 developed and
instruction. Computer-assisted instruction,
Miami, Florida, associate director for education/ implemented a peer review instrument
evaluation, Geriatric Research, Education and Clinical
on the other hand, uses stand-alone
that focused mainly on the format,
Center (GRECC), VA Medical Center, Miami, Florida, multimedia packages for learning and
presentation, feedback, and evaluation of
and senior investigator, Stein Gerontological teaching.6 Examples of e-learning include
Institute, Miami, Florida. Web-based curricula. This peer review
tutorials, case-based learning, simulations,
Dr. Candler is co-director and editor, and game-based learning modules. model is an attempt to identify and
MedEdPORTAL, and director of educational E-learning is fast becoming an established differentiate teaching Web sites according
technology, Association of American Medical
approach in medical education. to a set of criteria that reflects the learning
Colleges, Washington, DC.
paradigm and, in this way, to assist
Dr. Teasdale is associate professor of geriatric Some years ago, the development of medical educators creating e-learning
medicine, The Donald W. Reynolds Department of materials to achieve recognition for their
Geriatric Medicine, University of Oklahoma College
computer-based education materials was
of Medicine & Veterans Affairs South Central Health not considered evidence of scholarship work, with the ultimate goal of achieving
Care Network Mental Illness Research, Education in U.S. medical schools.7 That stance high-quality e-learning. Based on Boyer’s
and Clinical Center (MIRECC), Oklahoma City has eroded over the past decade, but work on the domains of scholarship,
VAMC, Oklahoma City, Oklahoma.
indiscriminate acceptance as scholarship Glassick and colleagues11 proposed six
Correspondence should be addressed to Dr. Ruiz, is also not justified. Historically, criteria to guide assessment of scholarly
GRECC (11GRC), Miami VAMC, 1201 NW 16th
Street, Miami, FL; telephone: (305) 5757-3388; fax: e-learning materials have not been work beyond the traditional paradigms of
(305) 575-3365; e-mail: (jruiz2@med.miami.edu). rigorously subjected to a formal, valid, teaching, research, and service. These
Academic Medicine, Vol. 82, No. 5 / May 2007 503Educational Strategies
criteria (clear goals, adequate preparation, E-learning materials introduce a government, industry, and higher
appropriate methods, significant results, new learning paradigm that allows education are beginning to adopt these
effective presentation, and reflective implementation of instructional and other guidelines, the typical peer
critique) can be especially helpful to strategies that could not readily be reviewer is not familiar with these
peer reviewers as they decide whether accomplished with print. The design concepts and will need to acquire new
educational materials, including e-learning of e-learning systems informed by skills to assess this new genre of
materials, meet the accepted standards educational theory and research can educational materials.
of scholarship. The Association of enhance the learning experience.
American Medical Colleges (AAMC) Examples of successful e-learning Navigation: Linearity versus branching
MedEdPORTAL project12 recently materials include multimedia case-based
Subject matter experts who create print
incorporated the Glassick criteria in its simulations and computer games, which
media usually present their content in a
peer review form for e-learning materials. allow medical learners access to a varied
sequential and predictable fashion, which
clinical case mix free from temporal and
In this paper, we discuss similarities allows learners to easily follow one
spatial restrictions.15 This and other
and differences between peer review of prescribed path that the author has set for
challenges may add to the complexity of
e-learning and print materials, focusing them in advance. There are abundant
the review for faculty not familiar with
specifically on delivery rather than examples of e-learning materials that
advances in e-learning methodologies.
content. We describe peer review simply deliver an automated linear
concepts, content, and practice with presentation. These can be highly
Format effective; whether they are depends on
regard to e-learning materials. We also
propose four approaches to handling Print media is limited to text and static the target audience, learning outcomes,
various challenges of peer reviewing graphical elements such as pictures and and topic. However, e-learning provides
e-learning materials. figures. E-learning materials, in their new opportunities that can exploit the
most versatile and popular format, hypermedia features and nonlinear
go beyond these formats to include structure that have made the World Wide
Differences hyperlinked text and multimedia Web so popular.
At its core, the peer review of e-learning elements that are only possible with
materials should not differ substantially computers. However, the addition of In the context of navigation, moving
from the review of traditional scholarship. media elements introduces important from the linear presentation of material
To do otherwise is to confer an unequal challenges to reviewers accustomed to to a branching paradigm opens a new
standing among academic products classic print materials. Specifically, the world of opportunities for medical
submitted for educational purposes. media elements must be evaluated for learners. Such flexibility affords new
For both traditional manuscripts and their appropriateness, educational value, opportunities to explore content, but
e-learning materials, the reviewer must and overall effectiveness. For example, a it may also overwhelm and confuse
examine the submission’s strengths and video depicting the “flash” seen within a learners. Although little is formally
weaknesses in terms of the veracity, syringe confirming successful phlebotomy known about the educational benefits
relevance, and effectiveness of content. can be powerful, but a video illustrating and pitfalls of branching navigation, the
The information contained in the various types of needle gauges offers peer review of e-learning materials may
educational materials must be correct, nothing more than what may be nonetheless consider the educational
up to date, and matched to learning illustrated in a simple photograph. benefits of such navigation. The parallel
outcomes and to the appropriate level of for print-based peer review is that
learners.13 However, e-learning requires Usability reviewers often consider the overall
consideration of additional dimensions. organization of content, including the
Print materials have been widely used for
For example, is the appearance conducive table of contents, embedded figures and
centuries, so their usability is time tested.
to education? Are multimedia elements tables, and references.
The history of e-learning, on the other
used effectively? Is it easy to “navigate” hand, is comparatively short, and
through the online material? Is the usability factors are continuously Interactivity
interactivity appropriate for the level of evolving. In human– computer One of the main instructional and
the learner? Are special computer skills, interaction and computer science, pedagogical advantages of e-learning
hardware, or software required? What usability refers to the quality of a user’s lies in its interactivity.3,15 The concept
special considerations are relevant experience when interacting with a of interactivity is still controversial,
regarding reproducibility? These and product or system such as a Web site or lacking even an accepted definition.18
other questions place new demands on software application.16 In contrast with Yacci18 outlines four criteria defining
reviewers. In this section, we suggest the the simple physical format of print, interactivity: instructional interactivity is
following aspects: pedagogy, format, e-learning input devices include a message loop, occurs from the learner’s
usability, navigation, interactivity, keyboards, mice, touch screens, stylus point of view, has two outputs (content
delivery, and currency. sticks, and verbal commands. learning and affective benefits), and
contains mutually coherent messages. An
Pedagogy Usability criteria for Web-based example will illustrate the concept. The
Mortimore refers to pedagogy as “any materials, which constitute the bulk medical student (learner perspective)
conscious action by one person designed of e-learning, have been devised17 but interacts with a Web-based virtual patient
to enhance learning in another.”14 are not yet widely used. Although by asking her about the characteristics of
504 Academic Medicine, Vol. 82, No. 5 / May 2007Educational Strategies
her abdominal pain, to which the virtual content without considerable cost and distance learning, asynchronous
patient responds (message loop). The time. Creators of print-based educational modalities, and online assessment.
virtual patient answers that the pain is content face much greater distribution Studies by Callaham and colleagues19,20
“stabbing” in nature (mutually coherent challenges than do their counterparts suggest that conventional face-to-face
message). The learner acquires knowledge who create e-learning products. workshops are ineffective at improving
regarding the virtual patient’s symptoms Information technologies allow the general quality of peer review efforts.
and becomes satisfied by it (content e-learning materials to be updated much The proposed educational program must
learning and affective outputs). According more easily and efficiently than print. include a variety of delivery methods,
to these criteria, traditional print Updating e-learning materials may entail including e-learning methods, mentoring
materials are low in interactivity. On costs similar to, or even greater than, of junior peer reviewers performing
the other hand, multimedia e-learning print materials, but the relative ease of actual reviews, and annotated examples
offers the potential for higher levels of redistributing updated e-learning content of well-conducted reviews of e-learning.
interactivity. Virtual patients and Web- offsets those costs. Consequently, authors These should all be posted for public
based simulations are dramatic examples should be encouraged to refine and access. Abbreviated training approaches
of highly interactive educational enhance their work using new information may offer busy educators opportunities
materials. from ongoing evaluation, new biomedical to acquire a basic set of knowledge and
discoveries, or changes in the underlying skills needed to conduct a peer review of
E-learning reviewers must be aware of the technology. This presents a unique e-learning materials.
current state of the art for interactivity challenge for the traditional model of
potential in order to provide valid peer review, which has focused on the Include multidisciplinary peer review
assessments of products under review. evaluation of relatively static documents.
Many journals already make use of
Moreover, the reviewer must be Editors of e-learning publications should
statisticians and methodologists to assist
sufficiently comfortable with the recommend that their published authors
editors in evaluating the methods
concepts to recognize when an attempt to maintain and openly display a revision
sections of submitted manuscripts. The
push the interactivity envelope results in history that indicates all significant
rationale behind this practice is the
a less useful educational approach than changes to content after the item was
recognition that many health care
might be achieved otherwise. The published. A publication that undergoes
professionals have insufficient facility
opposite can be true when interactivity is substantial modification could be
with statistical techniques. In support of
underutilized. A particularly important submitted as a new version for
this notion, a review of the literature
issue is for reviewers to discriminate publication, a model similar to
revealed that physicians with master
between good or effective versus progressive book editions. Furthermore,
degrees in public health and epidemiology
substandard or ineffective interactivity. any item that contains time-sensitive
were far more effective peer reviewers
information should be reviewed again
than were physicians without such
Delivery after a reasonable period of time. A
additional expertise.21 The process of
potential challenge for editorial staff
Dissemination is a condition of peer reviewing e-learning materials might
and peer reviewers is the likelihood of
scholarship. Emerging technologies also benefit from this multidisciplinary
increased workload resulting from new
surrounding e-learning represent new approach. MedEdPORTAL12 is already
versions of submitted e-learning
categories for the distribution of using a combination of reviewers
materials.
educational content. E-learning materials from varied disciplines, some with a
can be delivered through a wide range of background in e-learning, to assist in
media, including the Internet, CD-ROMs, their peer review process of selected
Approaches to Facilitate
DVDs, and PDAs. We do not include educational materials. We suggest that a
E-Learning Peer Review
videotape among e-learning media wide array of other professionals be
because of its linear and noninteractive We have described seven domains in considered to compose the pool of
presentation. Online repositories and which peer reviewer of e-learning available peer reviewers. When e-learning
digital libraries are particularly well materials differs from review of materials are submitted for review,
suited for the delivery of e-learning conventional print scholarship. We now appropriate members of this pool who
products. The peer reviewer would be suggest four approaches that have the have expertise in instructional design,
expected to recognize the subtle differences potential to ease the burden and improve usability issues, software development,
in how the content is organized and the quality of e-learning peer review. educational technology, and Web design
delivered through varied media to should be included, as appropriate, to
determine whether the materials are Develop peer review training ensure that peer review parameters
potentially effective and efficient. For specific to e-learning materials are
The increasing level of complexity that
example, experts in the field recognize appropriately appraised.
is required to peer review e-learning
that e-learning materials designed for
materials makes it imperative for experts
Web-based delivery may not be equally Develop guidelines
to design and develop training programs
well delivered via a DVD.
for prospective peer reviewers. This The formulation of technical specifications
training should cover the basics of and standards in e-learning for software
Currency innovative educational technologies, technologies facilitates the development,
Once a book or journal article is including e-learning content such as deployment, maintenance, and operation
published, it is difficult to update its tutorials, simulations, virtual patients, of e-learning systems. With regard to
Academic Medicine, Vol. 82, No. 5 / May 2007 505Educational Strategies
peer review, specifications and process and decrease the burden on of scholarship do warrant a rigorous
standards are nonexistent. The World editors, reviewers, and clerical staff. New process. Reviewers should ideally be
Association of Medical Editors22 and incentives to perform peer review must offered instruction regarding the fair and
other organizations provide valuable be established. However, in this era of efficient evaluation of the relevant
advice and suggestions regarding the increasing clinical and educational domains. In addition, the formidable
responsibilities of editors, the specific responsibilities, shrinking reimbursements, challenges to the establishment of an
role of reviewers, the development and faculty shortages, this issue becomes effective peer review process for e-learning
of a peer review policy, and editorial especially problematic for the editors and materials requires a comprehensive
decision making. These principles apply administrators of digital repositories. approach that develops peer review
to not only established print-based training, embraces multidisciplinary
journals but also e-learning publication peer review, develops guidelines,
venues. In particular, the AAMC The Work Ahead addresses incentives and compensation,
MedEdPORTAL group has developed Research in peer review of print materials and encourages peer review research.
guidelines and a standardized peer does not reveal sufficient solid evidence
review instrument using the consensus for deciding criteria that can help editors
of experts in the field—an important make consistent decisions regarding Acknowledgments
step to ensure a coherent and rational publication. Jefferson et al24 note that Dr. Ruiz would like to thank Dr. Bernard Roos
peer review process.12 editorial peer review lacks agreement for his mentorship. Dr. Ruiz would also like to
about aims and methods employed. A acknowledge the support from the D. W.
Provide incentives and compensation systematic review of the effects of Reynolds Foundation, the Miami VAMC
editorial review also has failed to show Geriatric Research, Education and Clinical
Discussion of incentives and Center, and the State of Florida Agency for
compensation for peer review activities any conclusive evidence for the
Health Care Administration (Florida’s
is common within academic and effectiveness of peer review of print Teaching Nursing Home Program). Dr.
professional circles. This dialogue is as materials.24 To better address the Teasdale would like to acknowledge the
relevant for reviews of e-learning challenges of peer reviewing e-learning support from the D. W. Reynolds Foundation
materials as it is for print media. materials, a consensus must be sought and the VA South Central Health Care
regarding minimum guidelines for peer Network Mental Illness Research, Education
However, editors of e-learning peer
and Clinical Center (MIRECC.)
review have to compete with the long- review, and reviewers must be evaluated
established prestige and tradition of on their expertise in those skills before
minimal compensation for print peer the process will be genuinely valued in
References
review. It is conceivable that without academic circles.25 The concealment of
1 Rennie D. Editorial peer review: its
an initial push, the establishment of author or reviewer identity, use of
development and rationale. In: Godlee F,
expertise in peer review of e-learning publication and statistical checklists, Jefferson T, eds. Peer Review in Health
materials may not occur or may fail effect of training, modes of communication Sciences. 2nd ed. London, UK: BMJ
to gather momentum. Monetary between editors and reviewers, and Publishing Group; 2003:1–13.
compensation occurs, but it is rare.23 reviewer bias are issues related to the 2 Fincher RM, Work JA. The scholarship of
Although the satisfaction and prestige review of conventional print scholarship teaching in health science schools. J Vet Med
that are probably not dissimilar from that Educ. 2005;32:1–4.
associated with the peer review activity
of e-learning materials. Researching some 3 Rosenberg M. E-Learning: Strategies for
are real, other forms of valued
Delivering Knowledge in the Digital Age.
compensation must be developed to of the issues inherent to the efficiency
New York, NY: McGraw Hill; 2001.
ensure that the general pool of qualified and effectiveness of the peer review
4 Harden RM, Hart IR. An international virtual
reviewers remains large and motivated. process and its impact on users, content medical school (IVIMEDS): the future for
For example, training courses in peer producers, and other stakeholders should medical education? Med Teach. 2002;24:
review could provide CME/CEU credits. be an important goal. The results of such 261–267.
Formal letters of commendation could be research, incorporating quantitative or 5 Ruiz JG, Mintzer MJ, Leipzig RM. The impact
generated from journal editors to deans qualitative data, can improve the quality of e-learning in medical education. Acad
and outcomes of this process with regard Med. 2006;81:207–212.
and department chairs as recognition
and reinforcement of an individual’s to both e-learning and traditional print 6 Ward JP, Gordon J, Field MJ, Lehmann HP.
materials. Communication and information technology
peer review activities for evidence of in medical education. Lancet. 2001;357:
scholarship. A new metric might be the 792–796.
number of e-learning products (or
Summary 7 Bader SA. Recognition of computer-based
manuscripts) reviewed annually by a materials in the promotion guidelines of U.S.
given reviewer. In this paper, we strongly advocate the medical schools. Acad Med. 1993; 68(10
development and adoption of peer review suppl):S16–S18.
In addition, editors may find it necessary guidelines for e-learning materials. 8 Glenn J. A consumer-oriented model for
to devote additional resources for Certain aspects of e-learning review are evaluating computer-assisted instructional
administrative support of associate comparable to print-based peer review, materials for medical education. Acad Med.
1996;71:251–255.
editors and peer reviewers. For example, but several new considerations must be
investments in the automation of addressed. Although we acknowledge that 9 Knight CL, Sakowski HA, Houghton BL, Laya
MB, DeWitt DE. Developing a peer review
the peer review process through the not all e-learning materials should be process for web-based curricula: minting a
acquisition of peer-review-management subjected to formal peer review, those new coin of the realm. J Gen Intern Med.
software can improve the efficiency of the that are under consideration as evidence 2004;19:594–598.
506 Academic Medicine, Vol. 82, No. 5 / May 2007Educational Strategies 10 Alur P, Fatima K, Joseph R. Medical teaching 15 Clark D. Pedagogy and E-Learning. Brighton, reviewer quality and performance. Ann websites: do they reflect the learning UK: Epic; 2006. Emerg Med. 1998;32:318–322. paradigm? Med Teach. 2002;24:422–424. 16 U.S. Department of Health and Human 21 Black N, van Rooyen S, Godlee F, Smith R, 11 Glassick C, Huber M, Maeroff G. Scholarship Services. Usability Basics. Available at: (http:// Evans S. What makes a good reviewer and a Assessed: Evaluation of the Professoriate. San www.usability.gov). Accessed January 9, 2007. good review for a general medical journal? Francisco, CA: Jossey-Bass; 1997. 17 Nielsen J. Designing Web Usability: The JAMA. 1998;280:231–233. 12 Association of American Medical Colleges. Practice of Simplicity. Indianapolis, Ind: New 22 World Association of Medical Editors. A MedEdPORTAL. Available at: (www.aamc.org/ Riders Press; 1999. Syllabus for Prospective and Newly Appointed mededportal). Accessed January 9, 2007. 18 Yacci M. Interactivity demystified: a Editors. Available at: (http://www.wame.org/ 13 Berry E, Parker-Jones C, Jones RG, et al. structural definition for distance education resources/editors-syllabus#reviewers). Accessed Systematic assessment of World Wide Web and intelligent computer-based instruction. January 16, 2007. materials for medical education: online, Educ Technol. 2000;40:5–16. 23 Eysenbach G. Peer-review and publication of cooperative peer review. J Am Med Inform 19 Callaham ML, Schriger DL. Effect of research protocols and proposals: a role for Assoc. 1998;5:382–389. structured workshop training on subsequent open access journals. J Med Internet Res. 14 Watkins C, Mortimore P. Pedagogy: what do performance of journal peer reviewers. Ann 2004;6:e37. we know? In: Mortimore P, ed. Understanding Emerg Med. 2002;40:323–328. 24 Jefferson T, Alderson P, Wager E, Davidoff F. Pedagogy and Its Impact on Learning. London, 20 Callaham ML, Wears RL, Waeckerle JF. Effect Effects of editorial peer review: a systematic UK: Paul Chapman; 1999:1–19. of attendance at a training session on peer review. JAMA. 2002;287:2784–2786. Academic Medicine, Vol. 82, No. 5 / May 2007 507
You can also read
