Science-Based Substance Abuse Prevention: A Guide - PRACTICES
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
1
GUIDE TO
SCIENCE-BASED
PRACTICE S
Science-Based
Substance Abuse Prevention:
A Guide
Substance Abuse and Mental Health Services Administration
Center for Substance Abuse Prevention
Division of Knowledge Development and EvaluationAcknowledgments
This document was produced under the guidance of Stephen E. Gardner, D.S.W., project officer, Paul J. Brounstein,
Ph.D., and Deborah B. Stone, Ph.D., Division of Knowledge Development and Evaluation, Substance Abuse and
Mental Health Services Administration, Center for Substance Abuse Prevention (SAMHSA/CSAP), through contract
#282-98-0023 to ROW Sciences, Inc., Carol Winner, project director.
Science-Based Substance Abuse Prevention: A Guide is based on A Guide to Science-Based Practices in Substance
Abuse Prevention, a seminal monograph developed and written by Paul J. Brounstein, Janine M. Zweig, and Stephen
E. Gardner, with substantial contributions from Maria Carmona, Paul Florin, Roy Gabriel, and Kathy Stewart.
Special thanks go to a dedicated review panel composed of Eric Einspruch, Roy Gabriel, and Katherine Laws,
RMC Research Corporation; Carol Hays, Illinois State Incentive Grant Program; Patricia Post, Central Regional Center
for the Application of Prevention Technologies; and Mary Joyce Prudden, CSAP, who offered invaluable suggestions
for improving the document.
The Department of Health and Human Services (DHHS) has reviewed and approved policy-related information in
this document but has not verified the accuracy of data or analyses presented in the document. The opinions
expressed herein are the views of the authors and do not necessarily reflect the official position of SAMHSA or
DHHS.
DHHS Publication No. (SMA)01-3505
Printed 2001
For single copies of this document, contact SAMHSA’s National Clearinghouse for Alcohol and Drug Information
(NCADI), P.O. Box 2345, Rockville, MD 20847-2345; 1-800-729-6686, 301-468-2600, or TDD 1-800-487-4889; or
visit the Web site at www.samhsa.gov.
ii Science-Based Substance Abuse PreventionForeword
Although recent reports show a leveling or decrease in substance use among our nation’s youth, drug abuse remains
a problem in our country. There were 14.8 million current users of illicit drugs in 1999. This figure represents 6.7 per-
cent of the population 12 years and older. The 1999 National Household Survey also found increases in illicit drug
use among adults ages 18–25. Although the rates for those 26-34 years old and 35 years and older have not changed
significantly since 1994, overall statistics indicate that there is still work to be done in preventing substance abuse.
The Substance Abuse and Mental Services Health Administration’s (SAMHSA) Center for Substance Abuse
Prevention (CSAP) developed this series of products in response to the ongoing substance abuse problems. The three
components in this series support CSAP’s mission to provide resources that are based on science, with measurable
outcomes, and designed to help community and state leaders formulate targeted programs.
CSAP is committed to sponsoring, accumulating, and integrating knowledge regarding scientifically defensible and
effective prevention practices. The primary foci of each document in this series is CSAP grantees, constituent organi-
zations, and the communities these groups serve.
We are pleased to release this guide to CSAP’s conceptual framework and methodology for designing and assess-
ing scientifically defensible programs for substance abuse prevention. The results reflect the findings of 10 years of
CSAP-funded demonstration programs, as well as other advances in the design and evaluation of successful preven-
tion strategies. These findings together lay the foundation for a new, empirically based approach to prevention pro-
gramming.
The booklet highlights the risk and protective factors that help determine an individual’s vulnerability to substance
abuse. It also examines CSAP’s qualitative and quantitative strategies for evaluating existing substance abuse preven-
tion programs and developing scientifically defensible best practices.
This booklet is one in a series of products developed to help key stakeholders structure and assess scientifically
defensible programs. It is designed to serve practitioners and others involved in the development, implementation,
and evaluation of substance abuse prevention programs as we work together on innovative and effective solutions
that respond to the unique needs of our individual communities.
Joseph H. Autry III, M.D. Ruth Sanchez-Way, Ph.D.
(Acting) Administrator Director
Substance Abuse and Mental Health Services Center for Substance Abuse Prevention
Administration Substance Abuse and Mental Health Services
Administration
A Guide iiiTable of Contents
Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .ii
Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .iii
Science-Based Substance Abuse Prevention : A Guide . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .1
The CSAP Conceptual Framework: Risk and Protective Factors . . . . . . . . . . . . . . . . . . . . . . . . . .2
Issues in Defining Scientifically Defensible Knowledge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .4
Application of Specific Principles and Criteria to Research Findings . . . . . . . . . . . . . . . . . . . . . .6
The CSAP Approach to Identifying Scientifically Defensible Prevention Interventions . . . . . . . . . .8
Resources for Identifying Scientifically Defensible Prevention Principles and Programs . . . . . . .10
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .12
Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15
A Guide vScience-Based
Substance Abuse Prevention:
A Guide
The Substance Abuse and Mental Health Services This booklet, Science-Based Substance Abuse
Administration’s Center for Substance Abuse Prevention: A Guide, is intended to assist preven-
Prevention (CSAP) is responsible for identifying tion practitioners and others involved in the
and disseminating scientifically defensible knowl- design, implementation, and evaluation of sub-
edge about proven prevention models and princi- stance abuse prevention programs. This document
ples to the substance abuse prevention field. provides the following:
Because there is a wide range of opinion about ■ Describes the evolution of the conceptual model
appropriate methodological criteria, standards, that CSAP uses to provide a unifying framework
and expectations—ranging in rigor from requiring for substance abuse prevention.
experimental studies to observing systematically ■ Discusses the principles that provide the founda-
to accepting clinical judgments—determining what tion for understanding and defining research
constitutes that knowledge can be challenging. findings as scientifically defensible.
Over the past 10 years, CSAP demonstration
■ Illustrates how CSAP applies specific principles
programs have mirrored the field’s growth in
and criteria to research studies, program evalua-
understanding what works in substance use pre-
tions, and scholarly efforts to identify scientifical-
vention. Initially, programs implemented and test-
ly defensible findings, prevention principles, and
ed prevention approaches based on practitioner
prevention models.
observation. The realization, for example, that
The Guide also provides specifics about the
most people had little knowledge about the harm-
CSAP methodology for selecting model programs
ful effects of alcohol and illicit drugs led to the
and identifying scientifically defensible principles.
development of public information campaigns
It is one in a series of three products that together
and school-based curricula designed to replace
assist prevention practitioners, evaluators, state
myths with facts. With increasing sophistication,
and local program administrators, policymakers,
this knowledge-centered focus enlarged to include
and funders in designing and assessing scientifical-
innovative interventions based on theories derived
ly defensible programs. The other two products
from research in related fields (e.g., Bandura’s
are:
social learning theory). More recently, data have
■ Promising and Proven Substance Abuse
become available from substance abuse preven-
tion-specific program evaluations that provide Prevention Programs, a comprehensive compila-
empirical evidence about those strategies that suc- tion of both proven and promising interventions
ceeded in changing knowledge, attitudes, and in an easy-to-scan grid organized by risk factor
behavior. Those findings, many of which stem and domain; and
from CSAP-funded demonstration programs, have ■ Principles of Substance Abuse Prevention, a nar-
laid the foundation for a new empirically based rative and selected listing of proven and cited
approach to prevention programming—one that interventions organized by domain.
CSAP is challenging the field to use (Hansen and State and federal agencies, local governments,
McNeal, 1996; Pandina, 1998). and private foundations have become increasingly
interested in funding substance use prevention
A Guide 1Guide to Science-Based Practices
programs with measurable outcomes. The new young adulthood (Bry & Krinsley, 1990; Newcomb
emphasis on performance means that prospective & Felix-Ortiz, 1992). Risk factors include biologi-
funding applicants must demonstrate that the pro- cal, psychological/behavioral, and social/environ-
grams they propose are both outcome-oriented mental characteristics such as a family history of
and likely to achieve the outcomes predicted. substance use, depression or antisocial personality
Using scientifically defensible interventions can disorder, or residence in neighborhoods where
help prevention practitioners respond to demands substance use is tolerated. Researchers have also
for accountability and improve their capacity to found that the more the risks in a child’s life can
provide effective services. be reduced—for example, by effectively treating
CSAP’s demonstration programs have been the mental health disorders, improving parents’ family-
testing ground for many of the most promising management skills, and stepping up enforcement
ideas for intervening with those at high risk for of laws related to the sales of illicit drugs to
substance use. As a part of its knowledge develop- minors or to drinking and driving—the less vulner-
ment and application role, CSAP has accepted the able that child will be to subsequent health and
responsibility for assessing those interventions and social problems (Hawkins, Catalono, & Miller,
disseminating results to the field for replication 1992).
and adaptation. In 1999, CSAP issued a primer on
effective programs titled Understanding Substance Protective Factors and Resilience
Abuse Prevention, Toward the 21st Century: A Protective factors, such as solid family bonds and
Primer on Effective Programs that described eight the capacity to succeed in school, help safeguard
model programs and briefly summarized the youth from substance use. Research has also
methodology guiding their selection. Given the demonstrated that exposure to even a substantial
increased emphasis on incorporating scientifically number of risk factors in a child’s life does not
defensible principles and interventions in both necessarily mean that substance use or other prob-
new and existing programs, it is important that lem behaviors will inevitably follow. Many chil-
prevention practitioners understand the conceptu- dren and youth growing up in presumably
al framework and criteria used in rating preven- high-risk families and environments emerge rela-
tion interventions and attesting to their tively problem-free. The reason for this, according
effectiveness. to many researchers, is the presence of protective
factors that reduce the likelihood that a substance
The CSAP Conceptual Framework: Risk use disorder will develop (Hawkins et al., 1992;
and Protective Factors Mrazek & Haggerty, 1994). The research on pro-
Theory and theoretical frameworks in the sub- tective factors explores the positive characteristics
stance use prevention field have been evolving and circumstances in a person’s life and seeks
over time, often through induction based on opportunities to strengthen and sustain them as a
applied empirical research. Among the most preventive device. Among these resilient children,
important developments in substance abuse pre- protective factors appear to balance and buffer the
vention theory and programming in recent years negative impact of existing risk factors (Anthony &
has been a focus on risk/protective factors as a Cohler, 1987; Hawkins et al., 1992; Mrazek &
unifying descriptive and predictive framework. Haggerty, 1994; Wolin & Wolin, 1995). From a
substance abuse prevention perspective, protective
Risk Factors factors function as mediating variables that can be
targeted to prevent, postpone, or reduce the
Put simply, one often tested and supported
impact of use.
hypothesis derived from this framework is that the
Taken together, the concepts of risk and
more risk factors a child or youth experiences, the
resilience enhance understanding of how and why
more likely it is that she or he will experience sub-
youth initiate or refrain from substance use.
stance use and related problems in adolescence or
Although not all risk and protective factors are
2 Science-Based Substance Abuse Preventionamenable to change—genetic susceptibility to sub- Domain Subcategory of Risk
stance use, for example—research demonstrates
that their influence can often be assuaged or ■ Individual biological and psychological dispositions,
attitudes, values, knowledge, skills, problem
enhanced. behaviors
■ Peer norms, activities, bonding
Domains
■ Family function, management, bonding
Risk and protective factors exist at every level at
■ School/work bonding, climate, policy, performance
which an individual interacts with others and the
■ Community bonding, norms, resources,
society around him or her. Clearly, the individual awareness/mobilization
brings a set of qualities or characteristics to each ■ Society/ norms, policy/sanctions
interaction, and these factors act as a filter, color- environmental
ing the nature and tone of these interactions—
Research has also revealed that domains are not
whether positive or negative. One useful way to
static in their impact, but interact with each other
look at this interplay is to organize interactions by
and change over time. As an individual develops,
the six life or activity domains in which they
his or her perceptions and interactions with family,
chiefly occur. Based on more than 30 years of
peers, schools, work, and community alter (Botvin
study, researchers have delineated specific subcat-
et al., 1995; Donaldson, Graham, & Hansen,
egories of risk within each domain. They include
1994; Hawkins et al., 1992; Kumpfer, Molgaard, &
the following:
Spoth, 1996). CSAP graphically depicts this more
intricate set of relationships through its Web of
Figure 1
Web of Influence
Teen School Substance Teenage Violent
Pregnancy Dropout Use Suicide Crime
Individual Risk and
Protective Factors
Society/
Family Community** Risk Peer Association
Environment* School Risk and
Risk and Protective and Protective Risk and Protective
Related Risk and Protective Factors
Factors Factors Factors
Protective Factors
*Society/Environment: Refers to the total complex of external social, cultural, and economic conditions affecting a community or an individual.
**Community: Refers to the specific geographic location where an individual resides and to the conditions within that particular area.
A Guide 3Guide to Science-Based Practices
vide a conceptual and organizational scheme for
Institute of Medicine Prevention Categories identifying risk groups and targeting outcomes.
Universal programs (e.g., mass media, school-
based health curricula): Target the general popula- Issues in Defining Scientifically
tion. Defensible Knowledge
Selective programs (e.g., mentoring programs Scientific inquiry stems from the need to under-
aimed at children with school performance or stand the world at large. The strength of science
behavioral problems): Target those at higher-than- and the scientific method is that it makes use of
average risk for substance abuse. strictly defined, standardized procedures to deter-
Indicated programs (e.g., parenting programs for mine how events are causally related. As science
parents with substance abuse problems): Target those improves its methods, levels of certainty about the
already using or engaged in other high-risk behaviors nature and extent of cause-and-effect relationships
(such as delinquency) to prevent chronic use. increase and more is understood about the
resources and effort required to achieve specific
changes in existing relationships. Using the scien-
Influence model (figure 1). The Web model illus-
tific method more systematically to identify knowl-
trates the complex series of interactions that
edge also fosters recognition of the diversity of
occurs between the individual and the six external
approaches involved in implementing prevention
domains that can result in substance use and other
programs and extracting data.
problem behaviors.
Different Ways of Knowing
Institute of Medicine Prevention Classifications
A key CSAP objective is to ensure that the inter-
Just as the Web of Influence can be used to illus-
ventions it recommends to the field are out-
trate relationships among risk and protective fac-
growths of rigorous evaluation and are described
tors across domains, the Institute of Medicine’s
in credible terms, comprehensible to the multiple
(IOM) prevention program classification system is
target audiences involved in substance abuse pre-
useful in understanding the differing objectives of
vention. Emerging out of the cultural, geographic,
various interventions and matching them to the
and strategic diversity of high-risk population pro-
needs of a targeted population (Kumpfer et al.,
gramming is a recognition of the different ways in
1997).
which people know or understand that a program
The IOM system classifies prevention interven-
is having an impact on youth, families, and com-
tions according to the population they affect
munities.
(Gordon, 1987). Universal interventions target
Like good medicine, the practice of prevention
general population groups without reference to
is both an art and a science. In assessing preven-
those at particular risk. All members of a commu-
tion programs as a whole and attempting to under-
nity benefit from a universal prevention effort, not
stand whether the strategies and interventions
just specific individuals or groups within a com-
used are exerting an impact and how they affect
munity. Selective interventions target those who
human behavior, it is critical to consider both
are at greater-than-average risk for substance use.
quantitative and qualitative evidence. Quantitative
Targeted individuals are identified on the basis of
data supply the raw material for the extensive sta-
the nature and number of risk factors for substance
tistical analyses that lend scientific credence to
use to which they may be exposed. Indicated
program results. Qualitative data provide the rich,
interventions are aimed at individuals who may
descriptive information needed to explain the
already display signs of substance use or abuse
effects of program interventions.
and are designed to prevent the onset of regular or
heavy substance use. Together, the Web of
Influence and the IOM classification system pro-
4 Science-Based Substance Abuse PreventionData Types and Research Strategies ine qualitative data from prevention practice cases
Although much discussion of knowledge focuses in their efforts to assess interventions such as
on the results of quantitative outcome evaluations, tobacco prevention/cessation and family-oriented
qualitative information can also be extremely use- prevention.
ful even if it is not always amenable to strict out-
come evaluation. Qualitative data may describe
Data Collection Techniques
program process or identify contextual mediating Figure 2 identifies the numerous data collection
variables that affect outcome results. Such process techniques that are used to gain knowledge in the
information embellishes findings from programs, substance abuse field. The techniques are mapped
providing an enhanced understanding of program onto a pyramid to provide some idea of the pro-
results. When researchers and the field, in general, portion of total information available through par-
ignore qualitative data, valuable information can ticular techniques that have been sufficiently
be lost. implemented to yield credible findings.
Reviews of qualitative information can produce Paradoxically, as indicated in the figure, the
credible findings and recommendations. For more traditionally accepted and more quantitative
example, expert consensus panels convened by scientific approaches represent a small proportion
many government agencies (e.g., Center for of data collection efforts, yet the information
Substance Abuse Treatment, Food and Drug derived from such studies constitutes a significant
Administration, National Institute on Alcohol portion of the formal knowledge base. Although
Abuse and Alcoholism) and private organizations the tendency is to ascribe more validity to quanti-
review and use both qualitative and quantitative tative efforts, CSAP recognizes that important con-
data to reach conclusions and formulate recom- tributions to the prevention knowledge base are
mendations affecting the health and well-being of also made by qualitative studies. Independent of
the nation as a whole. CSAP has developed how quantitative or qualitative they are, when
Prevention Enhancement Protocol Systems (PEPS), findings are supported by sound scientific princi-
which convene expert consensus panels to identi- ples, they can be used legitimately and effectively
fy what is known about various dimensions of to construct and implement prevention interven-
substance use prevention. PEPS panels also exam- tions.
Figure 2
Application of Specific Principles and Criteria to Research Findings
Replications and Multisite Studies
Single Site Experimental and
Quasi-experimental Studies
Process Evaluations
Focus Groups
Expert Panels
Key Informant Activities
Participant and Program Staff Observations
A Guide 5Guide to Science-Based Practices
Application of Specific Principles and Criteria 7 and 8 took the form of summary eval-
uations. Criterion 7 captured how well the total
Criteria to Research Findings
design and execution ruled out alternative
Research studies and the findings they produce hypotheses, while criterion 8 reflected overall
vary in credibility. The level of observed credibili - confidence in results inspired by methodological
ty of research findings regarding the causes of aspects of study design and execution. Criterion 9
intervention program effects hinges on whether addressed the utility of study results. Considered
the methods employed provide at least a reason- together, the nine criteria provide one scheme for
able means of assessing change over time attribut- evaluating all aspects of a quantitative or qualita-
able only to the program. This simple criterion has tive intervention.
a number of components that can be applied to
assess the credibility of research. The development
of the High Risk Populations (HRP) Databank, a
previous CSAP endeavor that critically reviewed Program Review Criteria
the HRP Demonstration Grants, provides an exam-
1. Theory: the degree to which findings are
ple.
grounded in sound theory, reflect clearly
The HRP Findings Bank was an evaluation-
stated hypotheses, and are operationally
oriented information system with a comprehen- relevant
sive, unifying framework. The Findings Bank
2. Sampling strategy and implementation: the
consisted of three primary components:
quality of sampling design and implemen-
■ Descriptive information (e.g., location, number tation and strength of evidence concerning
and type of sites, target group demographics) sample quality (e.g., data on attrition)
about CSAP’s high-risk populations grants; 3. Measures: operational relevance, psycho-
■ Compilation of interventions and outcomes pro- metric quality of measures used in the eval-
duced by the grants; and uation, and quality of supporting evidence
■ Compilation of interventions and outcomes pro- 4. Data collection: quality of data collection
duced by similar, non-CSAP projects. implementation (e.g., amount of missing
data)
The Findings Bank allowed users to relate findings
from demonstration programs to outcome mea- 5. Fidelity of interventions: evidence of high-
sures and to the data that support the findings. fidelity implementation of program, as
designed, and sufficiency of dosage (e.g.,
Nine criteria were used by groups of trained,
duration, intensity, frequency) to effect pos-
expert evaluators to assess the rigor of grantee
itive change
program evaluations. Six criteria were used to
6. Analysis: appropriateness and adequacy of
rate various aspects of methodological rigor (1
statistical techniques used in analysis
through 4, 6, and 7), one was used to rate fidelity
(5), one for utility (9), and one for overall integrity 7. Plausible threats to validity: degree to
which evaluation design and implementa-
and credibility (8).
tion address and eliminate reasonable alter-
Viewed in an alternative fashion, criteria 1
native hypotheses about program effects
through 3 rated the design or planning of the
and warrant strong causal attributions
study. They encompassed theoretical bases, sam-
8. Integrity: overall level of confidence in
ple design, and the outcome measures chosen.
project findings based on the research
These three criteria pertained to the planning of
design and implementation
both quantitative and qualitative research.
9. Utility: strength of findings and strength of
Criteria 4 through 6 related to execution of the
evaluation to determine if findings were
study; in other words, how it was implemented.
consistent with respect to expectations or
Criterion 5 was a hybrid, addressing both fidelity predictions from theory
of program/intervention implementation and the
design issue of dosage of program/intervention.
6 Science-Based Substance Abuse PreventionWhen a few problems were encountered—for
Confidence Scale example, attrition was modest, intervention imple-
mentation was solid, and analysis was accept-
5 = strong confidence able—an integrity rating of 4 (i.e., confident) was
4 = confident given.
3 = some confidence An integrity rating of 3 reflected some confi-
2 = little confidence dence in resultant data. This rating was often used
when program characteristics were strong enough
1 = no confidence
to inspire some confidence, but because of imper-
fect implementation of the program, moderate
attrition rates, data analyses that were not compre-
Application of Criteria hensive, uncorrected differences between treat-
For the review of CSAP’s HRP demonstrations, ment and comparison samples, or secular events
pairs of trained evaluators rated each of the nine that contaminated samples, confidence was not
criteria on a 5-point scale. Of particular impor- sufficient to assure reviewers that the results were
tance was the rating of integrity, on which review- wholly attributable to the program intervention.
ers were required to come to consensus. This If reviewers had little confidence in an experi-
rating reflected how much confidence reviewers mental study in which there was high and differ-
had that the intervention alone was responsible for ential attrition, the integrity rating received by
yielding the findings observed. Confidence was such a study was 2—little confidence. If attrition
derived from the quality of the intervention imple- was egregiously pronounced, the study rating may
mentation as well as the design of the evaluation have dropped to 1—no confidence.
study and how well the evaluation was conduct- Because well-designed studies can be imple-
ed. For example, when problems in intervention mented poorly, well-implemented programs can
implementation and research design and execu- be evaluated poorly, and findings can be overstat-
tion were minimal, reviewers assigned integrity ed, it is important to use criteria capable of expos-
ratings reflecting strong confidence in the findings ing these problems. Although overstated findings
(i.e., 5). make no positive contribution to the knowledge
Figure 3
Sample Application of Scientific Criteria to an Intervention
Summary Matrix for Grant ZZZ
NA 1 2 3 4 5
1. Theory X
2. Sampling Strategy and X
Implementation
3. Measures X
4. Data Collection X
5. Fidelity of Intervention X
6. Analysis X
7. Plausible Threats to Validity X
8. Integrity X
9. Utility X
A Guide 7Guide to Science-Based Practices
base, from the perspective of the prevention pro- mixed confidence, this intervention was considered
gram designer, policymaker, and funder, negative promising and could be cited as corroborative evi-
findings from well-implemented, rigorously evalu- dence of the effectiveness of grant ZZZ in altering
ated interventions have enormous value because youth’s knowledge about the harmful effects of
they identify interventions that do not appear to substance use.
work in naturalistic settings. When objectively In addition to applying the criteria to the HRP
reported, they save programs from expending demonstration grants, which include both the High
resources in the application of ineffective Risk Youth (HRY) programs and programs for
approaches. Pregnant and Postpartum Women and their Infants,
The last rating of utility captured data patterns. CSAP is using them in a slightly modified form to
Ideally, the field would want to adopt or adapt extract findings from the Community Partnership
interventions that yield consistent changes across Program. Likewise, CSAP’s National Registry of
domains affected by the intervention. For exam- Effective Prevention Programs (NREPP)—an ongo-
ple, program X could demonstrate a dramatic ing repository of scientifically defensible guidance
change on a specific targeted behavior such as for the substance abuse field—has incorporated the
marijuana use. In response, prevention practition- criteria in the 15 dimensions used in its consensus
ers in a community where marijuana use was process when assessing the quality of a particular
increasing might rush to adopt program X without prevention program. These 15 criteria are an
considering changes across other outcome expansion of the original nine criteria described on
domains such as skyrocketing underage use of page 6 of this guide. For a full review of these cri-
alcohol (drug substitution) or plummeting self- teria, please refer to the NREPP Web site at
efficacy. Reviewers used the utility rating to gauge www.preventionregistry.org. NREPP has evolved
the pattern of outcomes and thereby avoid focus- from the HRP Findings Bank and is available to
ing on isolated positive or negative outcomes. review programs funded from any source, not only
from CSAP.
Hypothetical Review Illustrating Application of
Criteria The CSAP Approach To Identifying
Figure 3 depicts a hypothetical review. In the fig- Scientifically Defensible Prevention
ure, program ZZZ has an overall rating of 3. The Interventions
program was rated well on many of the criteria;
The CSAP HRP Databank review used a qualita-
however, reviewers did not think that program
tive meta-analytic technique, one of two types of
control and treatment groups were comparable
meta-analytic techniques currently favored to
prior to the intervention, hence the evaluation of
organize information and extract defensible princi-
2 for sampling strategy and implementation.
ples and practices. In the past, literature reviews
Furthermore, the statistical analyses testing differ-
were frequently analyzed to determine whether
ences between the groups after the intervention
substance abuse prevention programs worked.
did not attempt to control for what might have
Meta-analysis offers a major refinement on that
been meaningful pretest differences. For that rea-
approach.
son, analyses received a 2. In addition, the age
In a literature review, researchers scrutinize and
group differences observed on outcome measures
critique original papers, determine the merit of
were large and not predicted by theory. However,
specific items, and then integrate the findings in
this intervention, along with a number of others,
discussions with each other or alone.
showed differences in youth’s knowledge of the
Occasionally this thinking results in the proposi-
harmful effects of substance use as a result of par-
tion of a critical experiment that, if performed,
ticipating in classroom-based drug education, an
would shed light on the true phenomenon under
outcome that, given the measurement protocols,
scrutiny. More often, the result is a simple summa-
could not be attributed to any event or occurrence
ry of the field, with the authors’ conclusions sup-
other than the intervention. Therefore, despite
8 Science-Based Substance Abuse Preventionported by the amount and consistency of the data tive meta-analytic techniques focus on specific
assembled. effects, they are extremely useful in identifying
Literature reviews make important contributions consistently moderate or large outcomes, and then
to understanding the state of the art in the field in relating these observations to specific program
and help identify gaps in the knowledge base that characteristics. Quantitative techniques are also
should be filled. However, because literature useful in deriving prevention principles and identi-
reviews are, for the most part, essentially narrative fying specific prevention interventions that are most
and subjective, use few samples, and lack scientif- closely related to consistently favorable outcomes.
ic rigor, they cannot respond adequately to ques- Like any other procedure, meta-analysis is not
tions about the efficacy of substance abuse without potential hazards (e.g., relying solely on
prevention programs (Tobler, 1994, p. 343). journal publications, averaging equally over differ-
entially important measures). Recent quantitative
Meta-Analysis meta-analytic efforts, including Tobler’s and
To address concerns about program effects, CSAP Stratton’s (1997) analysis of the effectiveness of
turned to meta-analysis, a conceptual approach school-based substance use prevention programs
that reviews the results of the full range of primary and CSAP’s National Center for the Advancement
research, including both published and unpub- of Prevention meta-analyses of the Correlates of
lished, experimental and quasi-experimental stud- Marijuana Use and the Correlates of Alcohol and
ies of programs that succeeded and programs that Tobacco Use, have advanced the knowledge base
failed. Meta-analysis uses qualitative as well as considerably.
quantitative methods to produce aggregated Qualitative meta-analysis is based on programs
results from multiple programs that can be used to as a whole. As an example, in the CSAP HRP
study relationships and test hypotheses (Cook et Databank review, trained, expert reviewers used
al., 1992; Light & Pillemer, 1984; Tobler, 1994, standard instruments to evaluate individual pro-
p. 357). Because meta-analysis allows researchers grams on the basis of source documents in the
to use results from small studies, from studies that form of project final reports, as well as articles
are quasi-experimental in design, and from studies published in journals, and come to consensus
with incomplete information in some areas, it about their credibility. Inclusion of source materi-
offers important advantages when evaluating sub- als overcame biases that may have been inherent
stance abuse prevention programs (Tobler, 1994, in using journal articles only.
p. 350). In applying meta-analysis, CSAP makes a Qualitative techniques are not limited to main
distinction between the two types. While both effects models of causation but are robust across
forms rely on qualitative judgments as well as interactions and nonlinear models since judg-
quantifying information, they differ in the extent to ments of effectiveness are based on the integration
which they rely on translating initially observed of implementation, evaluation, and findings within
outcomes into hard estimates of effect sizes. the context of the program. While ratings may be
Quantitative meta-analysis systematically codes attached to program efforts (e.g., integrity, utility),
the results from each study for every variable that they are composites weighted by the judgment of
may influence program outcomes and converts expert evaluators and not the product of decon-
those results to a standardized score or effect size structed program characteristics averaged across
(Tobler, 1994, p. 345). The end product of the multiple programs or interventions. Systematizing
analysis consists of a single best quantified esti- procedures and categorizing outcomes across pro-
mate of effect for a specific intervention or the grams via qualitative meta-analytic technique not
impact of a single contextual factor. Standardi - only offers decided advantages to traditional litera-
zation enables meta-analysis to compare programs ture review efforts, but also ensures that both qual-
with different sample sizes, and if effect sizes are itative and quantitative information is scrutinized
consistent and positive, to aggregate them and so that researchers, policymakers, and program
achieve statistical significance. Because quantita-
A Guide 9Guide to Science-Based Practices
designers have access to needed outcome and Qualitative Meta-Analyses
process data. CSAP culled several well-implemented critical
reviews to identify credible scientifically defensi-
Distinctions Between Quantitative and ble principles and programs, including the follow-
Qualitative Techniques ing CSAP-sponsored efforts:
Like quantitative meta-analytic techniques, quali-
■ Environmental Strategies for Substance Abuse
tative procedures use rules for inclusion of infor-
Prevention: Analysis of the Effectiveness of
mation. In general, these rules mirror the nine
Policies to Reduce Alcohol, Tobacco, and Illicit
criteria described previously in reference to the
Drug Problems;
HRP Databank. No matter who conducts the
■ The Role of Education in Substance Abuse
review, research protocols are screened for quality
Prevention (Implementation Guide);
of design and implementation. What may differ is
■ The Role of Information Dissemination and Mass
the set of procedures, level of data disaggregation,
population of studies eligible for inclusion, and Media in Youth-Oriented Prevention
rigor of the review itself. (Implementation Guide);
While both quantitative and qualitative tech- ■ The Role of Problem Identification and Referral
niques have important roles to play, their purposes in Youth-Oriented Prevention (Implementation
are different. The more quantitative techniques are Guide); and
useful in identifying interventions and contextual ■ A Review of Alternative Activities and Alternative
factors that influence effectiveness, whereas quali- Programs in Youth-Oriented Prevention (CSAP
tative techniques are most useful in identifying Technical Report 13, 1996).
effective programs and models of intervention. As a
result, qualitative techniques can be used to identi- Expert Panels
fy both successful program models and prevention In addition to information derived from critical lit-
principles. Both quantitative and qualitative tech- erature reviews, CSAP also incorporated informa-
niques can be supported by field observation and tion derived from expert consensus procedures. As
careful review of complete program documenta- part of the consensus process, consensus panel
tion (e.g., process analyses). Also, both are valu- members assessed implementation for fidelity and
able in developing and deriving principles reviewed research for rigor. Outcomes and review-
regarding successful program implementation. er confidence in those outcomes were rated using
relatively objective, standard methods. Expert con-
Resources for Identifying Scientifically sensus resources included the following:
Defensible Prevention Principles and ■ National Structured Evaluation (NSE): The NSE
Programs was one of the first expert consensus reviews of
CSAP uses both qualitative and quantitative meta- prevention programs. Using standard procedures
analyses, as well as findings from critical and measures, the NSE reviewed and rated the
overviews of published research and expert con- level of methodological rigor of substance use
sensus procedures, to identify scientifically defensi- prevention programs implemented through
ble prevention principles and programs. Currently, 1991. The NSE effort included federally funded
those principles and programs are described in two programs as well as programs sponsored by uni-
CSAP publications, Principles of Substance Abuse versities, foundations, and state or local govern-
Prevention and Promising and Proven Substance ments. When the analysis was complete, 10
Abuse Prevention Programs. Following are descrip- percent of the studies reviewed met or exceeded
tions of the specific methods CSAP employed to the criteria set for moderately rigorous studies
identify the scientifically defensible principles and (i.e., a rating of 3).
programs in both documents. ■ William B. Hansen & L. A. Rose (1997): In his
work, Dr. Hansen continued to use techniques
10 Science-Based Substance Abuse Preventionemployed by the NSE to identify and review sub- tion of program effectiveness. Like the NSE,
stance abuse prevention program evaluations approximately 10 percent of program evalua-
and field studies that reflected effective program- tions met the criteria set for moderate rigor. Of
ming. Data from individual projects were used to those, an estimated 2 percent of studies reviewed
derive more general principles concerning pro- met the more stringent standards that CSAP
gram effectiveness. developed for identifying well-implemented,
■ National Institute on Drug Abuse (NIDA, 1997): solidly evaluated, effective model programs.
NIDA produced a review document titled Drug
Abuse Prevention for At-Risk Individuals that cit- Quantitative Meta-Analyses
ed NIDA-sponsored substance abuse prevention The most stringent coding and rating protocols can
programs viewed as effective and offered general be followed when there are a wealth of studies with
recommendations about key elements of effec- concordant data. As the discipline of substance
tive interventions. abuse prevention continues to evolve, quantitative
■ Lawrence W. Sherman et al. (1997): Dr. Sherman efforts like these will become more commonplace.
and colleagues conducted a methodologically CSAP used data from the following quantitative
rigorous review and rating of U.S. Department of meta-analyses to identify current scientifically
Justice programs modeled after the NSE defensible principles and programs, as follows:
approach. The authors also evaluated grant and ■ Tobler’s meta-analyses of school-based preven-
funding mechanisms for adequacy. Principles tion programs (1986, 1992).
pertaining to effective strategies were derived ■ Tobler & Stratton, 1997.
and presented as key findings. ■ CSAP meta-analysis: CSAP initiated a series of
■ PEPS: Over the past several years, CSAP has large-scale meta-analyses, each focusing on a
sponsored the PEPS, a knowledge development specific topic area relevant to substance use and
effort that convenes a panel of experts to review substance use prevention (e.g., Correlates of
the literature in a specific area using a strict evi- Marijuana Use Among Youth, Correlates of
dentiary procedure to evaluate and either Alcohol and Tobacco Use Among Youth and
include or exclude individual research findings. Meta-Analysis of the Effectiveness of School-
Findings are then assessed for consistency of Based Programs).
valence and magnitude to arrive at a defensible CSAP accepts data from these qualitative and
conclusion. quantitative sources as credible, relying on the
■ National Evaluation of the Community review processes implemented and the expertise
Partnership Prevention Grant Program (2000): of the review panels convened to extract impor-
Extensive cross-site and other evaluations of the tant research findings. With these data as building
CSAP Community Partnership Program have blocks, it is possible to generalize across agencies
yielded considerable information about commu- and the time periods in which the data were col-
nitywide prevention efforts and have identified lected to identify replicable model programs and
specific prevention strategies that produce mea- scientifically defensible principles and interven-
surable reductions in substance use. tions.
■ The CSAP HRP Databank review: This review of CSAP has developed a series of comprehensive
the HRP demonstration grants program was data matrices (Promising and Proven Substance
designed to identify credible evidence of pro- Abuse Prevention Programs) that organizes inter-
gram effectiveness. An expert consensus process ventions by risk and protective factors, age, strate-
evaluated each final report filed through gy, IOM classification, original program name, and
December 1995 on the criteria defining program source citation. This inclusive effort includes pro-
credibility. In addition, they rated utility of study grams supported by government agencies, includ-
information for each program and weighed the ing CSAP, National Institute on Drug Abuse
consistency of results in determining the evalua- (NIDA), National Institute on Alcohol Abuse and
A Guide 11Guide to Science-Based Practices
Alcoholism, the U.S. Department of Education, References
and the U.S. Department of Justice, that have
undergone strenuous reviews (such as the CSAP Anthony, E. J., & Cohler, B. J. (1987). The invulner-
model programs) or have been cited by experts in able child. New York: Guilford Press.
the field as examples of particular intervention
strategies. Interventions entered into the matrix Bandura, A. Social Learning Theory. Englewood
vary in the levels of rigor assigned by reviewers. Cliffs, NJ: Prentice Hall, 1977.
For example, the CSAP model programs incorpo-
rated in the matrix represent the highest level of Botvin, G. J., Baker, E., Dusenbury, L. D., Botvin,
rigor and credibility. Interventions from other CSAP E. M., & Diaz, T. (1995). Long-term followup
programs, such as the Community Partnerships, results of a randomized drug abuse prevention
which may have more moderate levels of credibili- trial in a white middle-class population. Journal
ty, are included because careful evaluation attests of the American Medical Association, 273(14),
to their ability to produce positive outcomes. 1106–1112.
Interventions from other federal agency programs
are included because their sponsors have identified Bry, B., & Krinsley, K. (1990). Adolescent substance
them as noteworthy through their own evaluative abuse. In E. Feindler & G. Kalfus (Eds.),
processes. The matrix is a work in progress that Adolescent behavior therapy handbook. New
will continue to expand as findings from other York: Springer.
promising interventions become available. As the
information in the data matrix increases, the inter- Center for Substance Abuse Prevention. (2000).
ventions cited will provide a broader range of Prevention works through community partner-
interventions and more specific guidance for ships: Findings from SAMHSA/CSAP’s national
selecting those most likely to prove effective with evaluation. Rockville, MD: CSAP.
particular populations and in particular domains.
Cook, T., Cooper, H., Cordray, D., Hartmann, H.,
Also available is a second, simpler compilation
Hedges, L., Light, R., Louis, T., & Mostellar, F.,
of scientifically defensible principles called
(Eds.) (1992). Meta-analysis for explanation:
Principles of Substance Abuse Prevention. As its
A casebook. New York: Russell Sage Foundation.
name implies, this brief booklet organizes princi-
ples and interventions by domain and links them to Donaldson, S. I., Graham, J. W., & Hansen, W.B.
the prevention strategies identified in the Substance (1994). Testing the generalizability of interven-
Abuse Prevention and Treatment block grant. ing mechanism theories: Understanding the
Service providers can refer to Principles for ideas effects of a school-based prevention program
about what works in substance use prevention, to for potential high school dropouts and drug
identify proven interventions in a particular abusers. International Journal of Addictions
domain, and to justify the use of one or more inter- 25(7), 773–801.
ventions in a program. Because each principle and
intervention is cited, it is relatively easy to locate Gordon, R. (1987). An operational classification
the full article for more detailed information. of disease prevention. In J. A. Steinberg & M. M.
Evaluators, grant reviewers, and policymakers may Silverman (Eds.), Preventing mental disorders.
also find it useful as a quick overview of the cur- Rockville, MD: U.S. Department of Health and
rent state of the art in prevention programming and Human Services.
as a tool useful in ascertaining whether a program
is employing scientifically defensible principles Hansen, W.B., and McNeal, R.B. The law of maxi-
and interventions. mum expected potential effect: Constraints
placed on program effectiveness by mediator
relationships. Health Education Research
11(4):501-507, 1996.
12 Science-Based Substance Abuse PreventionHansen, W. B., & Rose, L. A. (1997). Issues in Pandina, R.J. Risk and protective factor models in
classification in meta-analysis in substance abuse adolescent drug use: Putting them to work for
prevention research. In W. J. Bukoski (Ed.), Meta- prevention. In S.L.David, (Ed.), National
Analysis of Drug Abuse Prevention Programs. Conference on Drug Abuse Prevention
Research Monograph No. 170. Rockville, MD: Research: Presentations, papers, and recommen-
National Institute on Drug Abuse, 183–201. dations. National Institute on Drug Abuse,
Rockville, MD: 17-26, 1998.
Hawkins, J. D., Catalano, R. F., & Miller, J. Y. (1992).
Risk and protective factors for alcohol and other Sherman, L. W., Gottfredson, D. C., MacKenzie,
drug problems in adolescence and early adult- D. L., Eck, J., Reuter, P., & Bushway, S. W.
hood: Implications for substance abuse preven- (1997). Preventing crime: What works, what
tion. Psychological Bulletin 112(1), 64–105. doesn’t, what’s promising. Research in Brief,
Washington, DC: National Institute of Justice
Kumpfer, K. L., & Baxley, G. B. (1997). Drug (NCJ 171676).
abuse prevention: What works? Rockville, MD:
National Institute on Drug Abuse, National Tobler, N. (1986). Meta-analysis of 143 adolescent
Institutes of Health. drug prevention programs: Quantitative out-
comes results of program participants compared
Kumpfer, K. L., Molgaard, V., & Spoth, R. (1996). to a control or comparison group. Journal of
The Strengthening Families Program for the Drug Issues 16(4):537–567.
prevention of delinquency and drug use. In
R. Peters & R. McMahon, (Eds.), Preventing Tobler, N. (1992). Drug prevention programs can
childhood problems, substance abuse, and work: Research findings. Journal of Addictive
delinquency. Thousand Oaks, CA: Sage Disorders 11(3):1–28.
Publications, 241–267.
Tobler, N. (1994). Meta-analytical issues for pre-
Light, R., & Pillemer, D. (1984). Summing up: vention intervention research. In L. Seitz & L.
The science of reviewing research. Cambridge, Collins (Eds.), Advances in data analysis for pre-
MA: Harvard. vention intervention research. Research
Monograph No. 142. Rockville, MD: National
Mrazek, P. J., & Haggerty, R. J. (Eds.) (1994). Institute on Drug Abuse, 342–403.
Reducing the Risk for Mental Disorders:
Frontiers for Preventive Intervention Research. Tobler, N. S. (1997). Meta-analysis of adolescent
Washington, DC: National Academy. drug prevention programs: Results of the 1993
meta-analysis In W. J. Bukoski (Ed.), Meta-
National Institute on Drug Abuse. (1997). Drug Analysis of drug abuse prevention programs.
abuse prevention for at-risk individuals (NIH Research Monograph No. 170. Rockville, MD:
Publication No. 97-4115). Rockville, MD: National Institute on Drug Abuse, 5–68.
The Author.
Tobler, N. S., & Stratton, H. H. (1997).
Newcomb, M.D., & Felix-Ortiz, M. (1992). Multiple Effectiveness of school-based drug prevention
protective and risk factors for drug use and abuse: programs: A meta-analysis of the research.
Cross-sectional and prospective findings. Journal Journal of Primary Prevention 18(1), 71–128.
of Pers Soc Psychology 63(2), 280–296.
Wolin, S., & Wolin, S. (1995). Resilience among
youth growing up in substance abusing families.
Pediatric Clinics of North America 42(2),
415–429.
A Guide 13Glossary
Antisocial and other problem behaviors: Can describe behavior-related problems (e.g., poor conduct and impulsive-
ness), behavior-related disorders (e.g., attention deficit hyperactivity disorder), or both.
Approach: A set of prevention strategies that typifies a program and can be used in an intervention setting without
adopting the entire program.
Assignment: The process by which researchers place study subjects in an intervention, control, or comparison group.
Experimental design studies randomly assign study subjects to both intervention and control conditions. In quasi-
experimental studies, study subjects are nonrandomly assigned to intervention and comparison conditions. Random
assignment increases the likelihood that the intervention and control groups are equal or comparable and have simi-
lar characteristics.
Attrition: An unplanned reduction in the size of a study sample caused by participants dropping out of the evaluation
(e.g., they moved away from the study location).
Behavior-related disorder: A specific behavioral problem that occurs in persistent patterns and characteristic clusters
and causes clinically significant impairment.
Behavior-related problem: A behavioral problem that is isolated or intermittent, is not part of a persistent behavior
pattern, and varies in severity and seriousness of its consequences.
Community: A group of individuals who share cultural and social experiences within a common geographic or politi-
cal jurisdiction.
Community-based approach: A prevention approach that focuses on the problems or needs of an entire community,
be it a large city, small town, school, worksite, or public place.
Community readiness: The degree of support for or resistance to identifying substance use and abuse as significant
social problems in a community. Stages of community readiness for prevention provide an appropriate framework for
understanding prevention readiness at the community and state levels.
Community tolerance: Community norms that view problematic behavior as socially acceptable or actively encour-
age it.
Conduct disorder: A behavior-related disorder that has a repetitive and persistent pattern of violating the basic rights
of others or major age-appropriate societal norms or rules. The disorder can include aggression to people and ani-
mals, destruction of property, deceitfulness or theft, and serious violation of rules.
Construct: An attribute, usually unobservable (e.g. educational attainment or socioeconomic status), that is represent-
ed by an observable measure.
Control group: In experimental evaluation design, a group of participants that is essentially similar to the intervention
group but is not exposed to the intervention. Participants are designated to be part of either a control or an interven-
tion group through random assignment.
Credibility of findings: Derives from the quality of intervention implementation plus the methodological rigor of the
research. When both are high, findings are attributable to the intervention and therefore have high credibility.
Data: Information collected according to a methodology using specific research methods and instruments.
Data analysis: The process of examining systematically collected information.
Design: An outline or plan of the procedures to be followed in scientific experimentation and research studies to
reach valid conclusions.
A Guide 15You can also read
