The Delphi Method: An Approach for Facilitating Evidence Based Practice in Athletic Training
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Athletic Training Education Journal; 2008;3(4):135-142 135
©by the National Athletic Trainers’ Association, Inc.
www.nataej.org
The Delphi Method: An Approach for Facilitating Evidence
Based Practice in Athletic Training
Michelle A. Sandrey , PhD, ATC; Sean M. Bulger, EdD
West Virginia University, Morgantown, WV
O bjective: The growing im portance of evidence based Data Synthesis: Textual support for the use of the Delphi
practice in athletic training is necessitating academ ics and Method in athletic training and a brief review of the literature
clinicians to be able to m ake judgm ents about the quality or pertaining to the: objectives; advantages; lim itations
lack of the body of research evidence and peer-reviewed com m only associated with the use of the Delphi Method; and
standards pertaining to clinical questions. To assist in the research protocol.
ju d g m e n t p roc es s , c ons e n s u s m e th o d s , n a m e ly Conclusions/Recom mendations: The Delphi Method in
brainstorm ing, nom inal group technique and the Delphi athletic training has been used to fulfill two objectives; the
m ethod can be used. The purpose of this paper is to review need for evidence based practice and the need to establish
the literature related to the Delphi Method and its potential policies and procedures when none are in existence or it is
im plications for evidence-based practice and peer-reviewed difficult for one individual to m ake a decision. The Delphi
standards in athletic training. Method and other consensus developm ent m ethods should
Data Sources: W e searched PubMed and MEDLINE (1978- not be viewed as a scientific m ethod for creating new
2007), CINAHL (1993-2006), Dissertation Abstracts (1979- knowledge, but rather as processes for m aking the best use
2006) and Google Scholar (1983-2007) using the term s of available inform ation, be that scientific data or the collective
“Delphi m ethod,” “m odified Delphi technique,” “consensus wisdom of participants.
m ethods,” “Delphi technique,” and com bined search term s of Key W ords: Delphi survey technique, group decision m aking,
“Delphi m ethod AND allied health, AND m edicine AND consensus, experts, nom inal group
dentistry, AND nursing.
he growing importance of evidence based practice in athletic scientific data or the collective wisdom of participants.3
T training necessitates that academics and clinicians are able to
make judgments about the quality of the body of research
evidence pertaining to clinical questions. Evidence based practice
Accordingly, the Delphi Method is best suited for situations where
evidence based practice is dependent on clinical expertise or expert
opinion.
integrates best available external clinical evidence from systematic Ziglio 5 described the Delphi Method as "a structured process
research with individual clinical expertise. 1 This is described as the for collecting and distilling knowledge from a group of experts by
proficiency and judgment acquired through clinical experience. means of a series of questionnaires interspersed with controlled
Since evidence based practice is in the infancy stage with athletic opinion feedback". The Delphi Method has been characterized as
training, it is difficult to make effective decisions in situations a systematic, effective, and comprehensive technique for
where there is contradictory or insufficient information. Therefore, administering a group communication process that enables a
use of consensus methods, namely brainstorming, nominal group collection of knowledgeable individuals to reach a consensus of
technique and the Delphi Method can be used.2 The Delphi Method opinion in circumstances when the available information is
and other consensus development methods should not be viewed as incomplete or multi-disciplinary in nature.6 ,7,8 The Delphi Method
a scientific method for creating new knowledge, but rather as has also been recommended for use when the complexity or
processes for making the best use of available information, be that ambiguity associated with a particular problem exceeds the
intellectual capabilities of a single decision-maker. 6 ,8 -1 0 The
underlying assumption of the Delphi Method is that the informed,
Dr. Sandrey is an Assoc. Professor and
Graduate ATEP Director at WVU collective judgment of a group of experts is more accurate and
msandrey@mail.wvu.edu reliable than individual judgment 5 ,6 ,1 1 within dynamic environments
where effective decision-making is dependent on the "knowledge,
Dr. Bulger is an Asst. Professor in the resourcefulness, expertise and creativity [of] different types of
College of Physical Activity & Sport people”. 1 0 T he purpose of this paper is to review the literature
Sciences at WVU related to the Delphi Method and its potential implications for
Sean.Bulger@mail.wvu.edu evidence-based practice and peer-reviewed standards in athletic
training. The following sections provide textual support for the use
of the Delphi Method in athletic training and a brief review of the
literature pertaining to the: 1) objectives; 2) advantages; 3)136 Sandrey - The Delphi Method
limitations commonly associated with the use of the Delphi in situations where: 1) the logistical constraints make repeated
Method; and 4) research protocol. multiple group meetings infeasible; 2) the heterogeneity of the
participants must be maintained to ensure validity of results; 3) the
Delphi Method Objectives individuals needed to contribute have diverse backgrounds and no
The Delphi Method was initially developed by Norman Dalkey established history of communication; 4) the group process must
and Olaf Helmer at the RAND Corporation during the 1950's and incorporate too many individuals for a face-to-face group exchange;
1960's for the purposes of military technology forecasting,1 0 -1 2 and 5) the disagreements among individuals are "so severe or
information gathering,8 and group decision-making 13 . The Delphi politically unpalatable that the communication process must be
Method has since matured and proven to be a highly adaptable refereed and/or anonymity assured."
research methodology that has been used in numerous industry The research objectives that are commonly associated with the
sectors including health care, education, business management, Delphi Method remain connected to the rationale that underlies this
information technology, military science, engineering, and group decision-making process. The three fundamental objectives
transportation. 1 4 More specifically, a literature search in the related of the Delphi Method have been summarized as the 1) development
areas of Allied Health, Medicine, Nursing and Dentistry indicated of a range of responses to a problematic issue; 2) the ranking of a
researchers in these fields have employed the Delphi Method to range of responses in order to provide an indication of significance;
answer a variety of research questions (see Table 1). and the 3) establishment of consensus regarding a range of
responses. 8 Similarly, Stahl and Stahl 13 identified the following
Table 1. Use of the Delphi Technique in Allied Health, possible objectives of a Delphi investigation: 1) identifying and
Medicine, Nursing, and Denistry investigating underlying assumptions that contribute to divergent
judgments or opinions; 2) ascertaining information that may help to
Database Years Number
generate a consensus of opinion from a selected panel of experts; 3)
CINAHL - English, Allied Health 1993-2006 13 establishing relationships between expert judgments in the form of
rankings on a topic that pertains to a number of disciplines; and 4)
CINAHL - No Specification 1993-2006 42 educating the respondent group to the diverse and multi-disciplinary
nature of the topic in question.
Dissertation Abstracts (General) 1979-2006 334
Google Scholar 1983-2007 259
Advantages of the Delphi Method
The inherent flexibility of the Delphi Method affords
Medline 1978-2007 266 researchers numerous advantages when identifying the research
question, planning the research design, collecting and analyzing
PubMed 1978-2007 107 data, and documenting the research process.1 4 The distinct
characteristics of the Delphi Method contribute to its usefulness as
The Delphi Method is considered to be most useful in a research instrument in policy decision-making and long-range
situations when precise analytical study is not possible due to the forecasting. 1 1 The most significant benefit of the Delphi Method
uncertain or ambiguous nature of the research problem. 5 ,1 5 ,1 6 Ziglio 5 concerns participant motivation. W hen conducted properly, the
reported that an individual may select from the following two Delphi Method enables the research participants to assume
courses of action when the problem-solving process is dependent ownership of a particular problem and its accompanying solution. 1 0
upon inadequate information and incomplete theory: This sense of personal ownership translates into a more effective
and efficient resolution to the problem.
The first option is to wait (perhaps indefinitely) until we Additional advantages of the Delphi Method include the
have an adequate theory based on tested scientific improvement in the accuracy of the decision-making process due to
knowledge enabling us to address the problem concerned. the use of controlled-feedback and anonymity; elimination of the
Of course, this option is not feasible if the problem needs geographical and logistical impediments inherent to face-to-face
urgent attention and action. Furthermore, many social and group meetings; establishment of consensus based on the group's
health problems are not amenable to solution by pure systematic evaluation, reflection, and reevaluation of the pertinent
positivistic or 'scientific' methods.17 The second alternative issues, and statistical description of the group responses. 1 6 Ziglio 5
is to make the most of what is, admittedly, an reviewed the relevant literature and argued that the strengths of the
unsatisfactory situation, and to try to obtain the relevant Delphi Method pertain directly to its ability to focus attention on the
intuitive insights of experts and use their informed most relevant issues, provide a framework for group
judgment as systematically as possible. According to communication among individuals with divergent backgrounds and
Dalkey1 8 the rationale and use of the Delphi Method geographical locations, minimize the psychological and
represents a systematic effort within the second professional barriers to communication that are inherent to face-to-
alternative. face meetings, provide an equal opportunity to respond for all the
participants, and produce a detailed record of the decision-making
The Delphi Method is predicated on the underlying assumption process and the resultant informed judgment.
that the informed judgment from a group of experts is likely to be
more reliable and accurate than the judgment of a single individual Limitations of the Delphi Method
or group of non-experts 5 ,1 0 Murry and Hammons 1 6 reported that the Despite the proposed benefits of this group decision-making
Delphi M ethod could be implemented as a valid research technique process, critics have raised concerns related to the sampling andAthletic Training Education Journal 2008; 3(4) 137
data analysis techniques associated with the Delphi Method.1 9 regarding the importance of each individual panel members’
Clayton 6 stated that while most of these criticisms regarding the contributions to the research process.
scientific rigor of the Delphi Method have been addressed in the In numerous Delphi studies, the concept of consensus has been
literature, it is essential that the researcher acknowledge and vaguely defined and there is little agreement among researchers
account for the following limitations in his or her research design: concerning the statistical determination of group consensus.2 2 The
Delphi M ethod has also been criticized regarding the manner by
1. The personal backgrounds and experiences of the which consensus is achieved. 2 Critics have argued that the
panel members are generally beyond the control of consensus is weakened due to limited participant discussion and
the researcher. lack of opportunities for participants to elaborate on their opinions
2. The panel members’ personal and professional or ideas. In light of these methodological limitations, researchers
responsibilities may limit the amount of time and must be careful not to overstate the significance of their findings.2
effort that each individual can invest in the decision- The determination of group consensus does not imply in any way
making process. that the correct answer or judgment has been reached in relation to
3. The process by which the panel arrives at consensus the research question or problem. The ultimate value of the Delphi
remains largely unknown. It is uncertain whether the Method pertains to its use as a means for structuring group
panel members alter their decision-making process as interaction and generating possible solutions to complex theoretical
a result of careful reconsideration or the pressure to issues or problems. W hen used in this manner the Delphi Method
conform. can be an excellent way to generate initial thoughts and ideas on a
4. The results of a Delphi Method cannot be generalized topic that might require future investigations.
beyond the specific panel of experts that participated The results of a Delphi investigation are specific to the panel
in the study. Additionally, the strength of the findings of experts. The results are not necessarily repeatable with other
depends largely on the backgrounds and perceptions groups of similarly qualified members due to the considerable
of the panel members. variation in individual backgrounds that exist. Even the most well
planned Delphi study may not yield “an exhaustive nor all-inclusive
Murry and Hammons 1 6 suggested that the questions formulated set of ideas”.6 W hile a Delphi study may produce thought-
by the researcher could influence the panel members’ responses and provoking results, “the value of the information is for the individual
that the researcher can never be completely certain that the reader to decide and is limited due to the constraints imposed by the
participants’ fully comprehend the purpose of the study. panel selection, as well as by the backgrounds, experiences and
Furthermore, the researcher may never be able to evaluate each biases of each member”.6 In response to this limitation, many
panel member’s full expertise due to the absence of face-to-face researchers recommend the need for further study to better
meetings. As with other forms of survey research, participant generalize the results to a wider population. 1 4 These verification
motivation and non-response rate or sample attrition remain primary studies can focus on a number of objectives including refining and
concerns as well. McKenna 2 0 recommended the use of face-to-face verifying results, extending the results with a similar or different
interviews during the first round of a study to help increase sample, and/or investigating a set of related research questions.
response rates throughout the Delphi process. That type of direct
interaction among the researchers and panel members, although Delphi Method Protocol
time consuming, may help to mediate additional questions In order to understand the protocol that should be followed
regarding the Delphi panel members’ qualifications and their when incorporating the Delphi Method, the following sections will
understanding of the research process. be included: the Delphi panel, questionnaire design and
Jeffery and Hache 8 suggested that the Delphi Method is not an administration, asynchronous and anonymous participation,
effective research tool for routine decision-making due to the time- supervision and feedback, and group consensus.
consuming nature of the communication process. The time delays The Delphi M ethod has been characterized as a highly flexible
between rounds may prove to be problematic if the research problem-solving process that provides "an opportunity for
involves nonprofessional or young respondents. The multiple structured communication, by which expert panel members could
rounds of questionnaire circulation that are employed increase the provide feedback, revise judgments, and contribute to the
required time for completion of the study. Advances in computer development of agreed-upon practices - all with complete
networking and electronic mail may help to eliminate this anonymity". 2 3 Keeney et al. 2 4 provided a description of a general
restriction and allow for more expedient questionnaire circulation, administrative protocol in their critical review of the Delphi
simultaneous data collection and transcription, and enhanced group Method. The Delphi process begins with an initial questionnaire
interaction among Delphi panelists.1 4 ,2 1 The Delphi Method, unlike (round one) that uses open-ended questions to generate a list of
other survey research protocol, also requires a continued ideas or concepts related to the research question. Members of the
commitment from expert panelists over multiple rounds of research team analyze, collate, and compile an inclusive list of
questionnaire circulation.2 0 This level of commitment can be responses for resubmission to the expert panel in the form of a
difficult to achieve unless specific measures are taken throughout second round questionnaire. 2 2 In all subsequent rounds of the
the research process to recruit and retain Delphi panel members. A Delphi process, the panel members are provided with feedback
number of strategies can be used to minimize this potential concern: regarding their individual responses and those of the other panelists.
pre-notification post-cards, phone calls, and/or e-mails to Descriptive group statistics such as mean, mode, and standard
prospective panel members; written contracts and incentives to deviation are shared between successive rounds and the panel
ensure panel member compliance; and continual reminders members are asked to reconsider and change their individual ratings138 Sandrey - The Delphi Method
Figure 1. Graphic Representation of a Sample Delphi Study Timeline
if deemed necessary. This multi-stage process of questionnaire Delphi Panel
circulation and controlled opinion feedback is continued until The selection of panel members is considered to be a critical
consensus is established. Most Delphi studies require between two component of the Delphi process that is directly related to the focus
and four rounds to gather a consensus of judgment, opinion, or or objectives of the investigation. 1 1 ,1 3 Unlike other survey research
choice. See Figure 1 for a graphic representation of a sample methods that rely on randomized sampling techniques, the Delphi
Delphi study time line. Method involves the purposeful sampling of a small group of
W hile the Delphi Method has been endorsed for its adaptability participants upon whose expert opinions the results of the study are
and numerous variations are described in the literature, researchers based. 1 4 The identification and recruitment of these panelists is the
must adhere to several basic guidelines in order to maintain an most important step in the Delphi process and critics have raised
adequate measure of scientific rigor and to support the credibility methodological concerns regarding a number of areas including the
of their research findings.2 4 The following characteristics are definition of expertise, the potential for researcher bias in panel
recognized as critical components of the Delphi Method: 1) the selection, and the possible overstating of results due to the use of
reliance on the informed judgment of a carefully selected panel of the expert label.2 4
experts; 2) utilization of multiple surveys; 3) asynchronous and In the interest of avoiding these methodological pitfalls,
anonymous participation of the Delphi panel members; 4) researchers should adhere to a stringent protocol for identifying the
establishment of consensus through the reporting of feedback pool of available experts. Delphi panel members are usually
regarding the responses from the previous round; and 5) provision identified through literature searches and recommendation from
of direction and control to the research process by a Delphi other recognized experts in the field.1 1 It is strongly recommended,
manager. 5 ,8 ,1 0 ,1 1 ,1 3 however, that the researcher adhere to a specific set of inclusion
criteria, rather than mere personal preference when selectingAthletic Training Education Journal 2008; 3(4) 139
prospective experts to serve on the Delphi panel. 5 ,2 2 Skulmoski et Delphi Method because it enables researchers to focus and guide
al. 1 4 reported that the following criteria for establishing expertise the communication process as it pertains to a wide array of
have been recommended in the literature: 1) knowledge and problems, disciplines, levels of expertise, and so forth. The
practical experience regarding the area under investigation; 2) modified Delphi Method is one design variation that is particularly
ability and willingness to participate in the study; 3) adequate time noteworthy. During a modified Delphi investigation, the researcher
to contribute to the Delphi panel; and 4) effective communication provides the panel members with an initial list of statements to be
skills. In compliance with these guidelines for panel selection a critiqued, and eliminates the traditional open-ended questionnaire
researcher must invest quite a bit of time in identifying experts that which is generally used during the first round of surveying. 1 6 The
meet these criteria. modified Delphi Method expedites the investigative process and
W hile considerable variation of opinion exists regarding the enables the researcher to maintain strict control over the range and
ideal size for a Delphi panel, the literature suggests that the panel scope of the issues that are being discussed.
should include at least 10 members 2 5 but little improvement in
results can be expected as the panel increases beyond 25-30 Asynchronous and Anonymous Participation
members.2 6 ,2 7 Furthermore, it is not unusual for Delphi panels to The asynchronous and anonymous participation of the panel
include individuals with varying degrees of expertise in a particular members represents a defining characteristic of the Delphi Method.
area. 1 3 A D elphi panel, for example, could consist of 15-20 The asynchronous nature of the Delphi M ethod refers to each panel
individuals from a specific homogenous population and 5-10 member's right to individually select the occasions when they will
individuals from a heterogeneous population with a different level participate in the group communication process.5 The panel
of expertise and social/professional stratification 6 who have members may elect to only respond to those items of the problem
displayed a high level of knowledge and practical engagement with that they feel best qualified to address.
the problems that are being studied 5 . The use of anonymous, sequential questionnaires is intended to
facilitate the equal participation and involvement of all panel
Questionnaire Design and Administration members. Anonymity eliminates numerous communication barriers
The Delphi Method involves the circulation of three or four that are inherent to face-to-face meetings, such as conflicting
questionnaires consisting of a number of items regarding a specific personalities, organizational hierarchy, political factors,
topic of interest. 1 3 Statements regarding the topic are generated presentation skills, and strong individual wills.1 0 The removal of
based on the available literature and the initial opinions of a these inhibiting variables enables each member of the group to
carefully selected panel of experts. Members of the panel are asked contribute fully and truthfully with the knowledge that their ideas
to respond to each statement in questionnaire form in accordance will be evaluated on the basis of merit rather than personality,
with their own personal knowledge and perceptions. The results of reputation, or position.
each round of anonymous questioning are summarized and shared W hile the value of anonymous participation remains central to
with the intention that panel members reconsider those responses the Delphi Method's success, it has been suggested that this
that deviate significantly from the group's overall mean ranking. anonymity “can cause participants to feel isolated and make it
Ziglio 5 stated that "during this interactive process, which can be difficult for them to judge how best to formulate their ideas so that
repeated as many times as are judged appropriate in the others will understand them”. 2 8 A number of solutions have been
circumstances, issues can be clarified, areas of agreement and recommended to prevent this communication barrier from occurring
disagreement can be identified, and an understanding of the throughout the duration of a Delphi investigation. Team building
priorities can be developed". techniques, straw model construction, and reduced participant
The Delphi process traditionally incorporates two primary anonymity have all been advocated as alternative methods for
investigative phases. 5 The exploration phase characterizes the first reducing participant isolation and enhancing in-depth conservation
one or two rounds of questionnaire distribution where the issues among panel members. For example, biographical sketches of each
being investigated are explored by the participating panel members. panel member can be shared prior to the initiation of the group
The evaluation phase involves all subsequent rounds of communication process in order to provide the participants with a
investigation for the purpose of evaluating the issues identified in deeper understanding of the individuals with whom they will be
the previous exploration phase. Throughout this interactive process, interacting.
it is important to remember that various features of the research
protocol, such as questionnaire design, the use of measurement Supervision and Feedback
scales, and the provision of feedback can influence the The provision of researcher-directed statistical group responses
communication among panel members and the eventual outcome of and feedback regarding each round of questioning represents
the study. Accordingly, the survey instrument and administration another defining characteristic of the Delphi Method. 1 6 Delphi panel
must be subjected to rigorous pre-testing and the procedures for the members are required "to consider, to rank and/or rate, to edit, and
provision of feedback should be carefully specified.2 4 to comment upon the responses developed during round one”. 1 6 The
The Delphi Method's ultimate purpose is to facilitate in-depth researcher then tabulates the results of each round of questioning
conversation among a group of experts by providing them with and provides the panel members with the frequency distributions,
opportunities to develop a more complete understanding of their means, and standard deviations for each item on the questionnaire.
peer's respective opinions, assessments, and forecast assumptions Additionally, a complete list of panel member comments could be
regarding a problematic issue.5 The precise methodology employed provided with successive iterations.6 This cyclical process of
during a Delphi investigation may vary depending on the focus of questioning, reconsideration, and feedback is continued until a
the research. 2 3 This high degree of flexibility is inherent to the convergence of panel member responses takes place.1 6 It is140 Sandrey - The Delphi Method
recommended that the researcher maintains and documents his or Table 2. Use of the Delphi Method in Athletic Training
her supervisory decision-making by using a journal to capture key Study Delphi Focus Rounds Outcome
information about the Delphi process.1 4 The use of this type of
research journal is an important step in documenting Erickson & Contributors to initial 3 Quantitative
methodological rigor. Martin30 success on BOC variables for
Given the methodological complexity of a Delphi study with exam perceived by predicting
its multiple rounds of survey design, data collection, data analysis, candidate sponsors exam success
and information sharing it is critical that the researcher possess
Development of 3 Standards and
strong administrative skills.2 The numerous administrative Weidner & standards and criteria criteria for
responsibilities of the researcher include developing a coding Henning31 for selection, training, selection,
system for tracking respondents across multiple rounds, creating file and evaluation of training, and
systems for participant responses, and creating mailing databases ACIs evaluation of
and labels. W ith the small number of participants included in most ACIs
Delphi studies, it is critical that the researcher administer the Delphi
Kutz32 Determine leadership 2 Leadership
process in a manner that facilitates continued participation across
(Dissertation) competencies development
multiple rounds. Poor response rates and the potential biases that necessary for AT in AT
they introduce are common in the final rounds of Delphi studies, so practice and inclusion
it is imperative that researchers operate in an efficient and effective in different levels of
manner that encourages participation until the process is ATEPs
completed.2 4
Andrews33 Development of 2 Eating
(Thesis) eating disorder Disorder
Group Consensus policies and policies
As previously discussed, the Delphi Method is useful for procedures for an
eliciting and combining expert judgments rather than factual NCAA Division I
information. 1 1 In many instances, the primary objective of the conference
Delphi investigation is to establish some measure of consensus
regarding the panel members' opinions.8 The reliance on small, non- Burmeister34 Development of 2 Eating
representative samples associated with most Delphi investigations (Thesis) eating disorder Disorder
policies and policies
prohibits the utilization of inferential statistics. 1 1 Gordon 1 1 theorized
procedures for an
that the true value of the Delphi Method concerns the generation of NCAA Division III
ideas rather than the determination of statistically significant results conference
that can be generalized to a larger population.
Unfortunately, the literature provides little direction concerning Perez35 Development of a 2 Lower Quarter
the determination of consensus because there is no agreement (Thesis) Lower Quarter screening tool
regarding the designation of a minimum percentage of response Screening Tool
needed to demonstrate consensus. 1 6 Brooks 2 6 suggested a Smrzley36 Development of an 2 Upper Quarter
generalized definition of consensus as the "gathering of individual (Thesis) Upper Quarter screening tool
evaluations around a median response, with minimal divergence". Screening Tool for
Bulger & Housner 2 9 for example, conducted a Modified Delphi Baseball
study to generate a consensus of expert judgment regarding the very
important theoretical and applied exercise science competencies Delphi Method in athletic training. Since evidence based practice
that pre-service physical educators need to learn within the is in the developmental stages with athletic training, the problem
undergraduate curriculum. The Delphi panel members were asked solving process of the Delphi Method will be useful when there
to rate each competency in terms of theoretical importance and may be difficulty in making effective decisions in situations where
pedagogical relevance using a 5-point Likert scale. Each item had there is contradictory or insufficient information. Furthermore, use
to meet the following criteria for consensus in order to be included of the Delphi M ethod is essential when the complexity or ambiguity
in the final list of recommended competencies. T he item received associated with a particular problem exceeds the intellectual
a mean rating of at least 4 or higher in the areas of importance and capabilities of a single decision-maker. 6 ,8 -1 0 This is important when
relevance, and the item received at least 75% of all individual the available information is relatively incomplete or multi-
ratings at the 4 level or higher. Any item that failed to meet this disciplinary in nature.6 -8
criterion was considered not to be critical in nature. The Delphi Method in athletic training has been used to fulfill
two objectives; the need for evidence based practice and the need
Application of Delphi Method in Athletic Training to establish policies and procedures when none are in existence or
A search of the literature using databases (Table 1) indicated it is difficult for one individual to make a decision. Two of the
that there were very few studies in athletic training that have used theses (Perez 3 5 and Smrzley3 6 ) established a thorough screening tool
the Delphi Method (Table 2). Only two published studies 3 0 ,3 1 were where none were in existence or incomplete information was
found. The other remaining studies were either a dissertation 3 2 or available. After a thorough literature search, questionnaires were
theses 3 3 -3 6 . Increased expectation for evidence-based practice in the developed that included sections with evidence based and best
academic and clinical setting may stimulate additional use of the clinical practice information. Perez 3 5 used a panel of lower quarterAthletic Training Education Journal 2008; 3(4) 141
experts (n=8) that have published or presented in this area, whereas Table 3. Additional Research Studies in Athletic Training Using
Smrzley3 6 used professional baseball athletic trainers and upper the Delphi Method
quarter researchers (n=19). Their final products both reflected a Academic Clinical
melding of best clinical practice and evidence-based for the upper
and lower quarter. Educational Trends in AT Preparation of Entry-level ACIs
W hat constitutes eating disorder policies and procedures are
scarce in the literature and very few institutions have shared or Standards and Guidelines Advanced Clinical Certifications
published their policies and procedures. Since policies and Technology in Athletic Development of Policies and
procedures are based on resources available at each institution, and Training Procedures
institutions in each conference are fairly comparable, each
institution can serve as a panel of experts. Andrews 3 3 used both Orthopedic and Medical Best Clinical Practice and Evidence-
physicians and athletic trainers (n=28) to develop policies and Issues based in Rehabilitation
procedures for a major Division I conference, whereas Burmeister 3 4 Core Stabilization
Sensiromotor Facilitation
used only athletic trainers (n=8) in a Division III conference. Their
Plyometrics
final product produced an eating disorders policies and procedures Gluteus Medius Strengthening
document applicable only to their conferences. Functional Rehabilitation
Kutz, 3 2 W eidner and Henning, 3 1 and Erickson and Martin 3 0 Wrist and Hand
utilized the Delphi Method to develop competencies, standards and Tendinopathies
criteria and qualitative variables, respectively where athletic
training “research-based and peer-reviewed standards” are not in Development of Injury Evaluation
existence 3 1 . In post-professional athletic training programs, Competencies at the Patellofemoral
Master’s and Doctoral Cold Urticaria
preparation for leadership roles is stressed. However, the athletic
Level Complex Regional Pain Syndrom
training literature is weak in identifying leadership competencies or Lumbar/SIJ
content important for practice or for inclusion in athletic training Rotator Cuff
education programs (ATEPs). Using the Delphi Method, Kutz 3 2
used a randomly selected sample of 161 faculty and athletic training Burnout and Stress Screening Instruments
practitioners. Thirty-five leadership content areas were rated as
Curriculum Development Clinical Development and
important and were organized by three constructs; Managerial
Assessment
Leadership and Knowledge Management, Leadership Theories, and
Leadership Issues, Trends, and Policies. Inclusion of leadership
competencies increased in importance from the entry-level to the flexible problem-solving process. This provides an opportunity for
doctorate level. W eidner and Henning 3 1 used all program directors expert panel members to provide feedback by rating items and
of 2003 entry-level (n=183) CAAHEP-approved athletic training adding additional write-in comments to be used in the final
programs to develop standards and criteria for Clinical Instructor development of evidence-based and best clinical practice,
Educators (CIEs) to use as a selection, training and evaluation of competencies, and standards and criteria. Since athletic training
Approved Clinical Instructors (ACIs). Of those randomly selected, research-based and peer-reviewed standards are not in existence as
16 panelists developed 7 standards each with 3 to 13 criteria that there are for medical and other allied health fields, a research
were necessary, clear and appropriate for the selection, training and method needs to be implemented. Therefore, use of consensus
evaluation of ACIs. In contrast to developing competencies and methods, namely brainstorming, nominal group technique and the
standards, Erickson and Martin 3 0 sought to determine the factors Delphi Method can be used. The Delphi Method in athletic training
that athletic training educators perceived as contributing to first- has been used to fulfill two objectives; the need for evidence based
time success on the Board of Certification (BOC) exam. Using a practice and the need to establish policies and procedures when
sample of CAAHEP-accredited ATEPs approved before 1995 none are in existence or it is difficult for one individual to make a
(n=35), 23 items were retained as contributing to passing all 3 decision. The Delphi Method and other consensus development
sections of the BOC on the first attempt. They concluded that good methods should not be viewed as a scientific method for creating
grades are not the only contributing factors and that ATEPs should new knowledge, but rather as processes for making the best use of
address these variables in their didactic and clinical instruction. available information, be that scientific data or the collective
The field of athletic training is multi-factorial and wisdom of participants.
multidisciplinary. On an every day basis, the clinical athletic
trainer is being inundated with new research and clinical References
techniques. With limited free time to read every research study and 1. Sackett DL, Rosenburg WMC, Gray JAM, Haynes RB, Richardson
clinical technique, the Delphi Method will allow clinicians and WS. Evidence-based medicine: what it is and what it isn’t. Br Med
researchers to identify and prioritize the most relevant emergent J. 1996;312:71–72.
2. Hasson F, Keeney S, McKenna H. Research guidelines for the Delphi
issues and trends in athletic training. Included in Table 3 are
technique. J Adv Nurs. 2000;32:1008-1015.
additional research studies in athletic training that could be 3. Murphy MK, Black N, Lamping DL, McKee CM, Sanderson CFB,
conducted using the Delphi Method. Askham J, Marteau T. Consensus development methods and their use
in clinical guideline development. Health Technol Assess.
Summary 1998;2(3):i-iv, 1-88.
The Delphi Method has been characterized as a highly 4. Powell C. The Delphi technique: myths and realities. J Adv Nurs.142 Sandrey - The Delphi Method
2003;41(4):376-382. Social Policy and Public Health. Bristol, PA: Jessica Kingsley
5. Ziglio E. The Delphi Method and its contribution to decision-making. Publishers;1996:34-55.
In Adler M, Ziglio E, eds. Gazing into the Oracle: The Delphi 29. Bulger SM, Housner LD. Modified Delphi investigation of exercise
Method and its Application to Social Policy and Public Health. science in physical education teacher education. J Teach Phys Edc.
Bristol, PA: Jessica Kingsley Publisher;1996:3-33. 2007;26:57-80.
6. Clayton MJ. Delphi: A technique to harness expert opinion for 30. Erickson MA, Martin M. Contributors to initial success on the
critical decision-making tasks in education [Electronic version]. Edc National Athletic Trainers’ Association Boatd of Certification
Psyc. 1997;17:373-387. examination as perceived by candidates sponsors: a Delphi study. J
7. Gillespie AM, Lowe JB. Use of the Delphi process for initial Athl Train. 2000;35(2):134-138.
investigation of community education on bowel cancer: A study 31. Weidner TG, Henning JM. Development of standards and criteria for
[Electronic version]. J Wellness Perspectives. 1995;12(1):35-42. the selection, training, and evaluation of athletic training approved
8. Jeffery G, Hache GA. A group-based Delphi application: Defining clinical instructors. J Athl Train. 2004;39(4):335-343.
rural career counseling needs. Measurement Evaluation Counseling 32. Kutz MR. Importance of leadership competencies and content for
Devlp. 1995;28(1):45-61. athletic training education and practice: A Delphi technique and
9. Chiappetta EL, Fillman DA. Clarifying the place of essential topics national survey. Dissertation, Lynn University;2006.
and unifying principles in high school biology [Electronic version]. 33. Andrews B. Establishment of policies and procedures for eating
School Sci Math. 1998;98(1):12-19. disorders at a Division I institution. Thesis, West Virginia
10. Sahakian CE. The Delphi Method. Skokie, IL: The Corporate University;2001.
Partnering Institute;1997. 34. Burmeister E. Establishment of policies and procedures for eating
11. Gordon TJ. The methods of futures research [Electronic version]. disorders at a Division III institution. Thesis, West Virginia
Annals Am Academy Political Soc Sci. 1992;522:25-36. Univesity;2007.
12. Pollard R, Tomlin ME. The use of expert teachers to improve 35. Perez B. A lower quarter screening tool as determined by the Delphi
education [Electronic version]. Edc. 1995;116(1):3-9. Technique. Thesis, West Virginia Universiy;2007.
13. Stahl NN, Stahl RJ. (1991). We can agree after all! Achieving 36. Smrzley E. An upper quarter screening tool as determined by the
consensus for a critical thinking component of a gifted program using Delphi technique. Thesis, West Virginia University;2007.
the Delphi Technique [Electronic version]. Roeper Rev. 1991;
14(2):79-89.
14. Skulmoski GJ, Hartman FT, Krahn J. The Delphi Method for
graduate research. J Info Tech Edc. 2007;6:1-21.
15. Baldwin-Morgan AA. The impact of expert system audit tools on
auditing firms in the year 2001: A Delphi investigation [Electronic
version]. J Info Systems. 1993;7(1):16-33.
16. Murry JW Jr, Hammons JO. Delphi: A versatile methodology for
conducting qualitative research. Rev Higher Edc. 1995;18, 423-436.
17. Goldschmidt P. 'Scientific inquiry or political critique? Remarks on
Delphi assessment, expert opinion, forecasting and group process by
H. Sackman.' Technological Forecasting Social Change. 1975;7:195-
213.
18. Dalkey NC. Delphi. Santa Monica, CA: The RAND
Corporation;1967:3704.
19. Sackman, H. Delphi Critique: Expert Opinion, Forecasting and
Group Process. Lexington, MA: D.C. Heath; 1974.
20. McKenna HP. The Delphi technique: A worthwhile approach for
nursing? J Adv Nurs. 1994;19:1221-1225.
21. Turoff M, Hiltz SR. Computer-based Delphi processes. In Adler M,
Ziglio E, eds. Gazing into the Oracle: The Delphi Method and its
Application to Social Policy and Public Health. Bristol, PA: Jessica
Kingsley Publishers;1996:56-85.
22. Williams PL, Webb C. The Delphi technique: A methodological
discussion. J Adv Nurs. 1994;19:180-186.
23. Flippo RF. Points of agreement: A display of professional unity in
our field [Electronic version]. Reading Teacher. 1998;52(1):30-41.
24. Keeney S, Hasson F, McKenna HP. A critical review of the Delphi
technique as a research methodology for nursing. Int J Nurs Stud.
2001;38:195-200.
25. Parentè, FJ, Anderson-Parentè J. Delphi inquiry systems. In Wright
G, Ayton P eds. Judgmental Forecasting. New York, NY: John
Wiley & Sons;1987:129-156.
26. Brooks KW. Delphi technique: Expanding applications. No Central
Assoc Quarterly, 1979;53: 377-385.
27. Delbecq AL,Van de Ven A, Gustafson D. Group Techniques for
Program Planning: A Guide to Nominal Group and Delphi
Processes. Glenview, IL: Scott, Foresman, and Company;1975.
28. Rotondi A, Gustafson D. Theoretical, methodological, and practical
issues arising out of the Delphi Method. In Adler M, Ziglio E eds.
Gazing into the Oracle: The Delphi Method and its Application toYou can also read
