Quality Improvement in Services for Adults with Intellectual and Developmental Disabilities: Guiding Principles
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Quality Improvement in Services
for Adults with Intellectual and
Developmental Disabilities:
Guiding Principles
Volume 20, Number 2, 2014 Abstract
Services and supports for persons with intellectual and develop-
mental disabilities (IDD) worldwide have transitioned – or cur-
Authors rently are – towards a community-based approach that enhanc-
es social inclusion. This paper presents principles that could
Virginie Cobigo,1 guide quality improvement monitoring in community services
Lynn Martin,2 for adults with IDD. The principles were developed through a
Rosemary Lysaght,3 series of activities aiming to better understand relevant service
Yona Lunsky,4 outcomes, such as social inclusion and choice; consultations
Robert Hickey,5 with various stakeholder groups including persons with IDD;
Hélène Ouellette-Kuntz6,7 and review of available data about the users of services and
supports for Ontarians with IDD. The five guiding principles
1
School of Psychology are: (1) activities are embedded within a continuous quality
and Centre for Research
on Educational and improvement cycle; (2) activities focus on personal outcomes
Community Services, that are important to persons with IDD; (3) activities capture
University of Ottawa, the multidimensional nature of the outcomes being measured;
Ottawa, ON (4) knowledge is developed from multiple perspectives; and
2
Department of Health (5) knowledge is meaningful, actionable, and informs policy and
Sciences, service improvements. The paper also presents a possible way
Lakehead University, forward in operationalizing a quality improvement monitoring
Thunder Bay, ON system related to services for adults with IDD.
3
Department of
Rehabilitation Therapy,
Queen’s University, International, national and regional bodies have called for
Kingston, ON demonstrated effectiveness and for improved quality in the
4
Centre for Addiction services and supports provided to persons with disabilities.
and Mental Health, The World Health Organization (World Health Organization,
Toronto, ON 2010) recommends a community-based approach to provid-
5
School of Policy Studies, ing supports that enhance the social inclusion and overall
Queen’s University, quality of life of persons with disabilities. The United Nations
Kingston, ON Convention on the Rights of Persons with Disabilities (UN
6
Department of General Assembly, 2006) reaffirms their fundamental rights,
Public Health Sciences, including the rights to social inclusion and individual choice.
Queen’s University, The World Report on Disability (World Health Organization,
Kingston, ON 2011) identifies persons with intellectual and developmental
7
Ongwanada, disabilities (IDD) as particularly vulnerable to social exclu-
Kingston, ON sion and poor quality of life. For instance, compared to per-
sons with other types of disabilities, persons with IDD may
Correspondence not have all their support needs met in community-based
settings. Furthermore, the general population may not be
virginie.cobigo@uottawa.ca sufficiently aware of their abilities, which limits opportuni-
ties for social connectedness and meaningful contributions.
Keywords
In Ontario (the most populated Canadian province at more
program evaluation,
than 13 million inhabitants), the Services and Supports to Promote
community services,
the Social Inclusion of Persons with Developmental Disabilities Act
social inclusion,
(SIPDDA) (2008) governs services and supports provided to
performance monitoring
adults with IDD. SIPDDA is part of a transformation agenda
that involved the closure of the last three residential institu-
© Ontario Association on
Developmental Disabilities90 Cobigo et al.
tions in Ontario in 2009. It aims to build on the United States, 25 states and four sub-state
transition to community-based supports to fur- regions are contributing information on an
ther promote the social inclusion of adults with annual basis to the National Core Indicators
IDD, and demonstrates Ontario’s commitment (NCI) (Human Services Research Institute &
to modernizing and improving the quality of The National Association of State Directors
services provided to this population. Ontario’s of Developmental Disabilities Services, 2013).
Ministry of Community and Social Services The program is a partnership between the
(MCSS, 2013) has developed an accompany- Human Services Research Institute (HSRI)
ing set of regulations and Quality Assurance in Cambridge Massachusetts and the United
Measures to help agencies plan for and pro- States National Association of State Directors
vide high-quality services and supports. MCSS of Developmental Disabilit ies Ser vices
funds more than 300 community agencies that (NASDDDS). Multiple data sources are used
provide services to adults with IDD, referred to including interviews with consumers, family
as “developmental services.” Developmental ser- self-administered surveys, and collated agency
vices support adults with IDD to live, work and or government-held data. The valid and reli-
participate in a wide range of activities in their able consumer survey gathers about 50% of
communities. In addition, MCSS helps adults the annual data on 114 indicators across five
with disabilities through its social assistance domains: (1) individual outcomes, (2) health,
programs. For example, the Ontario Disability welfare and rights, (3) system performance,
Support Program (ODSP) provides income and/ (4) staff stability, and (5) family indicators.
or employment supports to adults with disa-
bilities, including those with IDD. In 2013, the In the Netherlands, the Quality Qube ®
province’s budget announced an additional approach was developed (Buntinx Training &
investment of $42.5 million to enhance services Consultancy, 2012) to promote quality improve-
provided to persons with IDD and their families ment measures rather than performance indica-
– especially those with complex needs, as well as tors. One of its fundamental assumptions is that
reduce service delivery wait times (MCSS, 2013). the data collected are useful when they inform
In 2013, the province spent a total of $1.7 billion the achievement of expected outcomes from the
on developmental services (MCSS, 2013). Given service users’ perspective. The approach recog-
this significant investment, there is interest in nizes that comparisons and rankings alone do
ensuring that the supports and services pro- not lead to actions that can have positive out-
vided are of high quality, and are meeting the comes for service users; a quality improvement
needs of persons with IDD and their families. cycle has to be adopted. Quality Qube ® indica-
tors are categorized into five domains: (1) inde-
Many service agencies are committed to eval- pendence, (2) social participation, (3) well-be-
uating the quality of their services through ing, (4) facilitators, and (5) relationships with
accreditation processes and consumer satis- the service.
faction surveys. However, many stakeholders
expressed concerns about service gaps. Since Both the NCI and Quality Qube ® include the
2009, provincial and national media have perspectives of various stakeholders (services
reported stories of families waiting for services, users, families and staff members) when assess-
and families who can no longer cope with their ing the quality of services. Both have the capac-
situation. The Ontario Ombudsman, an inde- ity to report developed knowledge at different
pendent watchdog who investigates complaints levels (i.e., team, agency, jurisdiction) to inform
from the public about Ontario government ser- actions. Comparisons with other agencies or
vices, has enquired on hundreds of complaints other jurisdictions are also rendered possi-
from families of adults with IDD (report forth- ble through standardized data collection. The
coming). In 2013, the legislature convened a capacity to monitor the quality of developmen-
select committee (elected provincial represent- tal services allowing for comparisons across
ative) to investigate concerns and make recom- agencies or regions, and across stakeholder per-
mendations for future services across the prov- spectives has yet to be developed in Ontario.
ince (Legislative Assembly of Ontario, 2014).
This paper presents five principles that should
Outside of Ontario, some jurisdictions have guide efforts to assess and monitor the quality
already invested in monitoring and quali- of services for persons with IDD. These princi-
ty improvement systems. For example, in the ples have been developed in consultation with
JODDGuiding Principles for Quality Improvement
91
various stakeholders, including persons with Initiatives from other jurisdictions included the
IDD. Although this knowledge has been devel- National Core Indicators in the USA (Human
oped for and with Ontarian stakeholders, it Services Research Institute & the National
applies to many other jurisdictions. Association of State Directors of Developmental
Disabilities Services, 2013), as well as the Quality
Qube® from the Netherlands (Buntinx Training
Methods & Consultancy, 2012). These strategies were
chosen because they provide two examples of
This paper reports on the work completed by
quality monitoring of IDD services, which both
the Multidimensional Assessment of Providers
and System (MAPS) research program that facilitate the use of standardized data collection
aimed to inform the Ontario developmental ser- and multi-level reporting leading to action. In
vices system in its efforts to monitor the quali- addition, both are led by senior academics with
ty of supports and services provided to adults long-standing reputations in the IDD field.
with IDD (Martin & Ouellette-Kuntz, 2014).
Third, data routinely collected by MCSS and
Several activities led to the identification of developmental services agencies (in 2010–2013)
the principles of quality improvement present- were examined to determine whether they
ed in this paper. First, a better understanding could be used to support quality improve-
of social inclusion and choice was developed ment practices. These data included: service
through reviews of the literature and an inter- satisfaction surveys, information requested by
national consultation process (see Cobigo & accreditation bodies, information collected on
Stuart, 2010; Cobigo, Ouellette-Kuntz, Lysaght, persons with IDD who receive income supports
& Martin, 2012; Lysaght, Cobigo, & Hamilton, from the Ontario Disability Support Program
2012; Mahar, Cobigo, & Stuart, 2013; Mahar, (ODSP), and information on persons request-
Cobigo, & Stuart, 2014; Martin & Cobigo, 2011; ing services at one of the province’s nine
Webber & Cobigo, 2014). Reviewed literature Developmental Services Ontario (DSO) offices.
was not specific to the social inclusion and We described what data were collected, as well
choice of persons with IDD, but rather explored as what processes were used at the time to ana-
these concepts from the perspective of mem- lyse and report developed knowledge.
bers of society with and without disabilities.
The consultation process involved a survey of Fourth, findings from projects within the over-
various stakeholders, as well as presentations all program of research (i.e., literature reviews,
and workshops at international scientific con- surveys, interviews, and case studies related
ferences, including the European and world to productivity options, staff practices, parents
congresses of the International Association applying for services, and person-directed
for the Scientific Study of Intellectual and planning) informed the development of indi-
Developmental Disabilities (Cobigo, Mahar, cators and data collection methods that could
& Stuart, 2012; Cobigo & Webber, 2012), the be used to assess and monitor quality of sup-
world congress of the World Association for ports and services and inform quality improve-
Social Psychiatry (Cobigo & Webber, 2012), and ment efforts.1 Some of these projects used data
the congress of the Association internationale
routinely collected in Ontario. Most of them
de recherche sur le handicap mental (Cobigo,
included original data collection. These projects
Lysaght, & Ouellette-Kuntz, 2010).
have informed data collection feasibility in the
developmental services system in Ontario.
Second, initiatives in Ontario and elsewhere
evaluating or monitoring the quality of servic- Finally, persons with IDD and their families,
es provided to adults with IDD were reviewed. service providers, and government represent-
1 See Cobigo et al., 2013; Hickey, 2011; Lunsky, Tint, Robinson, Vickar, & Ouellette-Kuntz, 2012; Lunsky, Robinson, Tint,
Vickar, & Ouellette-Kuntz, 2013; Lysaght, Ouellette-Kuntz, Cobigo, & Petner-Arrey, 2013; Lysaght, Cobigo, Petner-Arrey, &
Ouellette-Kuntz, 2014; Martin, Ashworth, & Ouellette-Kuntz, 2012; Martin, Ouellette-Kuntz, Cobigo, & Ashworth, 2012a;
Martin, Ouellette-Kuntz, Cobigo, & Ashworth, 2012b; Martin, Ouellette-Kuntz, & Cobigo, 2013; Martin, Ouellette-Kuntz,
Petner-Arrey, & Walker, 2013; Martin & Ouellette-Kuntz, 2014; Ouellette-Kuntz et al., 2014; Ouellette-Kuntz et al., 2011;
Ouellette-Kuntz et al., 2012; Ouellette-Kuntz et al., 2013; Ouellette-Kuntz, Lunsky, Blinkhorn, Robinson, & Tint, 2013a;
Ouellette-Kuntz, Lunsky, Blinkhorn, Robinson, & Tint, 2013c; Ouellette-Kuntz, Lunsky, Blinkhorn, Robinson, & Tint, 2013d;
Saaltink & Ouellette-Kuntz, 2014 (all reports available upon request)
v.20 n.292 Cobigo et al.
atives were consulted on the best approach to services system cannot always be aggregated.
monitor the quality of services for Ontarians Agencies collect a broad range of information
with IDD. The consultation structure involved about the people receiving supports. Such
regular meetings with Consumer Consultations information may be related to individual ser-
groups (i.e., about six persons with IDD in each vice plans, operational planning or compliance
group), and Local Advisory Committees that with Quality Assurance Measures. However,
included at least one family member of indi- this information is not systematically aggre
viduals with IDD, service agency directors or gated because agencies lack capacity at the
delegates, and representatives of MCSS (about local level or because variances in indicators
10 individuals per group, most of whom were make broader aggregation at the regional or
service agency representatives) (Martin & provincial level impossible. The capacity of
Ouellette-Kuntz, 2014). These structures and the system to better understand what could be
meetings were held in three locations across the done for quality improvement would increase
province (i.e., Kingston, Toronto, and Thunder if standardized data collection existed across
Bay). In addition, annual meetings took place agencies. Similarly, some information collect-
with representatives of MCSS, including branch ed by the application entities (DSOs) is col-
directors, managers, policy analysts at the cor- lected in formats that make aggregation infea-
porate level, and representatives from region- sible. For example, the application package
al offices. Consultation groups were asked to Application for Developmental Services and
reflect on the following questions: What should Supports (ADSS) for developmental services is
we know to improve Ontario’s Developmental standardized across the province, but many of
Services system? What should we know to its questions are open-ended. The complexity
make sure we are doing the “best possible and resource intensity of analysing open-end-
job”? These groups were invited to advise the ed questions has made the aggregation of data
research program on the interpretation of its difficult (Ouellette-Kuntz et al., 2013c). This
findings and knowledge dissemination plan. limitation could be easily overcome with a list
Meeting minutes were recorded and shared of items within which to select those that best
with committee members. represent individuals’ responses. However,
because the ADSS is mainly used for person-
This paper integrates knowledge learned from al service planning, it is important to maintain
these activities and proposes guiding principles the capacity to record individuals’ responses
for quality improvement monitoring of services verbatim.
for adults with IDD in Ontario’s developmental
services system. Second, the data collected is not currently
linked between social assistance programs and
service agencies, which limits the capacity to
Results estimate the number of persons supported by
the developmental services system and only
In Ontario, extensive data are collected on the provides fragmented information on their sup-
services and supports provided to adults with port needs. Agreements between data holders
IDD. Nevertheless, consultations with repre- could be negotiated to facilitate data sharing
sentatives of service agencies and MCSS have and still respect the protection of personal
emphasized the limitations of the data collect- information (Lunsky, Klein-Geltink, & Yates,
ed; although these data have potential, in their 2013). Examples of such agreements exist, and
current form they cannot shed light on where often require the involvement of a third party
and how to improve service quality. that has the knowledge and capacity to link
datasets (e.g., the Manitoba Centre for Health
Lessons Learned on Data Collection in Policy holds and links data from multiple sec-
the Developmental Services System tors in the province of Manitoba; the Institute
for Clinical Evaluative Sciences is a data reposi-
There are five areas in which data collection tory for provincial health administrative data in
could be improved to maximize its ability to Ontario; and the Canadian Institute of Health
inform quality improvement efforts. First, the Information acts as a nation-wide repository for
data currently collected by the developmental linked administrative and clinical health data).
JODDGuiding Principles for Quality Improvement
93
Third, no comprehensive baseline information Therefore, in spite of the wealth of data that cur-
is available on the impact of services in order to rently exists within the developmental services
monitor change over time. Support needs assess- system, much work is needed to ensure that
ments are conducted for individuals request- these data are relevant to persons with IDD and
ing services. Although informative for service include their perspectives; are collected in a con-
planning purposes at the individual level, this sistent way across agencies; and are collected in
information is not currently used to monitor ser- a way that easily permits data linking, aggrega-
vice impact at an agency, regional or provincial tion, and analysis. To do this, stakeholders rein-
level. This information gap makes difficult any forced the need for clear partnerships between
evaluation of the effectiveness of services and researchers, service providers, database hold-
supports. The projects conducted within this ers, and service users. They expressed a desire
program of research (and reported elsewhere) to focus on data collection efforts that could
provide some baseline information to this end. inform service improvements needed to pro-
mote social inclusion, and would represent more
Fourth, most data collection activities do not than a “checklist” approach to measurement.
focus on issues persons with IDD and their Stakeholders also discussed the importance of
families consider most relevant. Data collec- focussing data collection efforts on elements that
tion focuses mostly on compliance to processes. are meaningful and actionable by the agencies
Information on many key personal outcomes – collecting them, and that go beyond measuring
i.e., the impact of services and supports on the issues related to compliance to process stan-
lives of persons with IDD and their families, as dards. To support stakeholders in their efforts to
well as on critical service needs – is missing or improve service quality monitoring, we propose
cannot be aggregated. For example, the sense of the following five guiding principles.
burden and perceived distress of caregivers are
critical indicators of the system’s capacity to care Guiding Principles to Quality
for persons with IDD that are currently only Improvement Monitoring
assessed by some agencies (Ouellette-Kuntz,
Lunsky, Blinkhorn, Robinson, & Tint, 2013b).
Research and consultation with Ontarian
stakeholders have permitted the identification
Last, and arguably most importantly, current
of five principles that should guide the assess-
data collection efforts often miss the perspective
ment and monitoring of supports and services
of persons with IDD, since many of the ques-
to inform quality improvement efforts in IDD
tionnaires and surveys used rely on parents and
services.
caregivers as proxy respondents. The perspective
of family members and significant others in the
lives of persons with IDD is important to inform Principle 1: Activities Are Embedded Within
quality improvement efforts, but their opinions a Continuous Quality Improvement Cycle
should not substitute those of persons with IDD,
since discrepancies between their opinions are In the context of IDD services, continuous qual-
not unusual (Foxx, Faw, Taylor, Davis, & Fulia, ity improvement requires that stakeholders
1993; Golubovic & Skrbic, 2013; Reid, Everson, & engage in a cycle of ongoing data collection and
Green, 1999; Stancliffe, 1995). Techniques exist to analysis to develop knowledge about actions
support the participation of persons with IDD in needed to improve the quality of services with
interviews and questionnaires (Finlay & Lyons, regards to desired outcomes for service users
2001; Finlay & Lyons, 2002; Lloyd, Gatherer, & (van Loon et al., 2013). The cycle leads to the
Kalsy, 2006) and should be considered when translation of knowledge into actions, and
planning data collection. Among others, the monitoring of their impact on outcomes for ser-
following tips apply: (1) Avoid using complex vice users. From the lessons learned through
multiple-choice questions; (2) Minimize acquies- the research program, a graphic representation
cence and be sensitive to the tendency to please of this cycle has been developed and is repre-
interviewers or carers; (3) Abstract concepts sented in Figure 1.
should be avoided, as well as negative questions;
and (4) The level of literacy should be consistent A continuous quality improvement approach
with the cognitive abilities of interviewees. is distinct from Quality Assurance Measures
v.20 n.294 Cobigo et al.
DATA COLLECTION
Meaningful, Actionable, Multidimensional & Multiperspective
SYSTEM ORGANIZATIONAL PERSONAL
Willingness to act promptly
STRATEGIES OUTPUTS OUTCOMES
RESPONSIVENESS
Legislation and Resources and Improved
policies, processes used social inclusion
community services to provide services for service users
and supports, and supports
anti-stigma
initiatives,
and system
monitoring
REPORTING ANALYSIS AND
INTERPRETATION
Translation into actions
Translation
into knowledge
Figure 1. Continuous quality improvement in the developmental services system
which focus on compliance to process stan- Principle 2: Activities Focus on Personal
dards (Schalock & Verdugo, 2012). The lat- Outcomes That Are Important to Persons
ter, while important to ensure safeguards
with IDD
against harm and abuse, lacks a collaborative
approach that could lead to quality improve-
ment, and does not focus on outcomes for the Decision-making should ideally be done based
service users. Stakeholders engaged in con- on empirical evidence of the effectiveness and
tinuous quality improvement take owner- efficacy of services and programs. However,
ship of the process, identify improvements in empirical knowledge of best practices for sup-
their own field of action, and are committed porting adults with IDD and improving their
to respond (Bryson & Patton, 2010; Cousins & social inclusion remains scarce. In the absence
Whitmore, 1998; O’Sullivan, 2012). A collabo- of such supporting evidence, there is a need for
rative approach is recommended to enhance quality improvement activities to focus on the
multidirectional communication between ser- outcomes that are important to service users,
vice providers, service users, policy makers, rather than solely on those relevant to the needs
data analysts and researchers, and to ensure of policy makers and service providers (Schalock
their meaningful contribution (O’Sullivan, & Verdugo, 2012; van Loon et al., 2013).
2012). Contrasting stakeholders’ perspectives
and identifying common priorities for action Without knowledge of the outcomes deemed
should be central to quality improvement prac- important by service users, it is difficult to
tices. In a continuous quality improvement determine what services and policies are need-
cycle, all stakeholders should be responsive to ed, and how to prioritize what actions are
the knowledge developed and put it into action. required to improve social inclusion (van Loon
Actions include service and support improve- et al., 2013). Social inclusion outcomes include
ments, as well as improvements to data collec- experiencing valued and meaningful social
tion and analysis processes. roles, as well as feelings of trust, reciprocity
and belonging shared among members of a
JODDGuiding Principles for Quality Improvement
95
community (Cobigo, 2012; Cobigo et al., 2012; Principle 4: Knowledge is Developed from
Lemay, 2006; Sherwin, 2010). Monitoring the Multiple Perspectives
quality of services for Ontarians with IDD,
therefore, should appraise the extent to which Because of the diverse systems in which they
services improve these outcomes (see Table 1 belong, stakeholders’ perspectives on the qual-
for a list of relevant personal outcomes). ity of services and the priorities for action are
likely to differ. It is important that all points of
Principle 3: Activities Capture the view are considered to support decision-mak-
Complexity and Multidimensional Nature ing when planning service quality improve-
of the Outcomes Being Measured ment (Buntinx & Benjamins, 2010; Buntinx,
2008). The perspective of service users and their
families is crucial, and they are the best respon-
A conceptual framework is important in
dents when measuring service users’ outcomes.
transforming data into meaningful knowl-
The perspective of service agencies and their
edge (Buntinx & Schalock, 2010; Schalock & staff, as well as the perspective of the funding
Verdugo, 2012; van Loon et al., 2013). It provides entity (i.e., the entity funding services, which is
an understanding of the complex interactions the MCSS in Ontario), must also be considered.
between input, processes and outcomes, and
therefore goes beyond the “checklist” approach Social inclusion is the outcome of complex
of quality assurance measures and accredita- interactions between personal and environ-
tion bodies. From the MAPS research program, mental factors (Cobigo, 2012; Cobigo et al.,
a theoretical framework of social inclusion, 2012; Cummins & Lau, 2003). A linear rela-
developed in consultation with persons with tionship between services or policies and
IDD and their families, providers of develop- improve social inclusion is difficult to draw;
mental services, policy makers and research- factors independent of the service system will
ers, has emerged (Cobigo, 2012; Cobigo et al., influence observed outcomes. An ecological
2012). Therefore, the information to be collect- approach is thus more appropriate, where the
ed should relate to the various dimensions of individual is at the centre of a microsystem (i.e.,
social inclusion, if quality improvement efforts family and friends), a mesosystem (i.e., connec-
are to be successful in promoting social inclu- tions between elements of the microsystem),
sion. an exosystem (i.e., services and benefits), and
a macrosystem (i.e., values, attitudes and laws)
In addition, gathered information should focus (Buntinx & Schalock, 2010).
on areas where the developmental services sys-
tem can act (i.e., organizational outputs and Principle 5: Knowledge Is Meaningful,
system-level strategies). Organizational outputs Actionable, and Informs Policy and Service
include efficiency measures, staff-related mea- Improvements
sures, and program and service options avail-
able to service users. A focus on the quality of Continuous quality improvement means that
outputs from the perspective of users is sug- the information collected should lead to knowl-
gested (Schalock & Verdugo, 2012; Zeithaml, edge and actions on how to improve outcomes
Parasuraman, & Berry, 1990) (see Table 1 for a for the individuals supported (Schalock &
list of the characteristics of outputs appraised Verdugo, 2012). To do so, stakeholders must
positively by users). System-level strategies to develop a management information system
improve the social inclusion of adults with IDD that allows for the collection, analysis and
include: legislation and policy enactment; com- reporting of meaningful, timely and useful
munity-based and person-centred services and data from various perspectives (Poister, 2010).
supports; anti-stigma and anti-discrimination Stakeholders must also be responsive to the
initiatives, and system monitoring (Cobigo & knowledge developed so that it leads to action.
Stuart, 2010). Ongoing exchange among policy makers, ser-
vice providers, service users and researchers is
essential if one is to collect relevant data, devel-
v.20 n.296 Cobigo et al.
Table 1. Quality Improvement in IDD Services: Domains Proposed by the Multidimensional
Assessment of Providers and System.
FOCUS DOMAIN DEFINITION
PERSONAL Valued Being socially included means experiencing a range
OUTCOMES social roles of social roles within a variety of contexts. Social roles
Benefits for are valued when they respect the individual and the
service users community’s expectations, choices and needs.
Competency To perform a given social role, the person has to have
acquired the required skills to be competent in that role.
Reciprocity Complementary social roles; social roles only exist
when reciprocated by others.
Belonging A sense of belonging is a subjective feeling of value
and respect that is built from reciprocal relationships
and shared experiences.
Choice Social roles are the result of an individual’s choices.
Satisfaction Social inclusion relies on the satisfaction of all
members of the community that their social roles are
valued, meaningful and respectful of choices.
ORGANIZATIONAL Tangibles Quality of physical facilities, equipment and people.
OUTPUTS Reliability Ability to perform the promised service dependably
Resources and and effectively.
processes used Responsiveness Willingness to help consumers and provide prompt
by agencies to services.
provide services
Assurance Knowledge and courtesy of employees and their ability
to convey trust and confidence.
Empathy Caring and individualized attention provided to
consumers.
Availability Presence of needed services and supports.
Affordability Ability to procure the services and supports.
Awareness Degree to which the service user is aware of potential
services or supports.
Accessibility Ability to access the services and supports (e.g.,
barrier-free environment).
Extensiveness Variety of services and supports.
Appropriateness Degree of correspondence between needs of the
consumer and availability of relevant services or
supports.
SYSTEM Legislation and Human rights discourse and legislation that protect
STRATEGIES policy the rights and freedoms of persons with IDD.
Ways in which Supports and Community-based and person-centered supports and
agencies and MCSS services services.
enhance personal Anti-stigma Efforts to eliminate social barriers and promote
outcomes and Initiatives adequate and equitable access to public goods and
organizational services.
outputs
System Developing evidence of the effectiveness of actions.
monitoring
JODDGuiding Principles for Quality Improvement
97
op knowledge of where improvement is need- Building Capacity for a Province‑Wide
ed, and transform this knowledge into action. Quality Improvement System
Social inclusion outcomes are complex, and
An important factor that emerged from our
causal links between actions from the devel-
review of evidence-based practices is the criti-
opmental services system and improved social
cal role of organizational readiness (Hodges &
inclusion cannot be ascertained. Activities are
Hernandez, 1999) to manage data collection,
intended to gather evidence that the devel-
analyse results, and translate findings into
opmental services system is moving towards
action. Key to building capacity is the develop-
targeted actions, and the social inclusion of
ment of partnerships between all stakeholders
persons with IDD is improving over time.
who play an important role in quality improve-
However, it is unlikely that a specific action
ment including persons with IDD and their fam-
will be the unique cause of observed changes.
ilies, services agencies, funding entities, respon-
In addition, factors external to the develop-
sible ministries, and researchers (see Table 1).
mental services system have an impact on the
These partnerships will open discussions rela-
social inclusion of adults with IDD (Hartley,
tive to obtaining legal agreements for data shar-
Finkenflugel, Kuipers, & Thomas, 2009). For
ing and linkages; understanding technical issues
example, well executed supported employment
for data sharing and linkages; and developing
programs are likely to improve access to pro-
the infrastructure for an efficient information
ductivity options for adults with IDD, although
management system, including data collection,
these also increase as a result of support from
analysis, interpretation, and reporting of mean-
family and friends, as well as changes in the
ingful and actionable information.
attitudes of employers. It is thus important to
observe trends in both service users’ outcomes
In addition to having common goals and objec-
and organisational outputs when monitoring
tives, these partnerships must be based on
the quality of the services. Trend analysis and
mutual trust and a commitment to improving
interpretation require multiple data points (i.e.,
the quality of services. Differences in perspec-
repeated measures at the population/system
tives, interests and power must be acknowl-
level), an understanding of random variability
edged. Persons with IDD and their families
in population/system level measures, and con-
must believe that their views and contributions
sideration of external factors contributing to
are valued, and trust that the information they
observed changes (Poister, 2010). Most impor-
provide will be used to improve the quality of
tantly, trend data are informative if analyzed
services. Further, there needs to be a clear com-
and understood in light of program and policy
mitment among all stakeholders to act based
changes (Rosenberg, 1997).
on the knowledge gained through quality
improvement activities, which means making
relevant changes to services and policies.
Discussion
Movement toward a continuous quality Over the three years of this program of
improvement approach to the assessment and research, active and effective collaborations
monitoring of developmental services and sup- have been developed and fostered with persons
ports demands a culture that is supportive of with IDD (through Consumer Consultations),
knowledge-informed policy and practice. To families and developmental services agencies
create that culture in Ontario, it is suggest- (through Local Advisory Committees and
ed that three goals be pursued concurrently: individual research projects), as well as with
(1) building capacity; (2) improving data col- representatives from various levels of MCSS
lection and interpretation; and (3) exchanging (through Local Advisory Committees, consul-
knowledge. tations, and individual research projects). This
type of partnership model and stakeholder
involvement is strongly suggested for advanc-
ing quality improvement activities.
v.20 n.298 Cobigo et al.
Improving Data Collection Exchanging Knowledge
and Interpretation
While key stakeholder groups have been
The quality of the data on which decisions to engaged throughout the program of research,
create or change policy and practice is key. there is a need for further collaboration and
Moving forward, several steps are recommend- exchange to build a consensus on how to
ed to improve data collection and interpreta- improve the quality of supports provided to
tion. The first step is to better understand how adults with IDD in Ontario and, more widely,
data currently collected could be interpreted to promote a culture of knowledge-informed
and used for quality improvement. As previ- policy and practice. In particular, electronic
ously mentioned, the system collects a range forms of dissemination (e.g., websites, news-
of data that could be integrated into a mean- letters); presentations, webinars, videoconfer-
ingful framework, and would provide addi- ences; as well as workshops with stakeholders
tional information if linked and aggregated. interested in more detailed information about
Modifications to current data collection and how to assess and monitor the quality of their
management is also required. Adopting new services are needed.
data collection processes would be an import-
ant step to allow agencies and their funders To some extent, this has already started with
to capture indicators of outcomes for service the MAPS team leading a variety of activities to
users. Finally, developing efficient systems for engage with multiple stakeholder groups. For
generating reports meaningful to stakeholder example, over 60 participants attended a webi-
groups is a crucial next step for data to be used nar from about 30 sites across the province in
in a timely and relevant manner. February 2014, a panel presentation was held
at the Ontario Association for Developmental
Consultations with stakeholders, exploration Disabilities (OADD) annual conference in April
of current data collection, as well as scientif- 2014, and persons with IDD, service providers
ic knowledge has led to the development of a and representatives of MCSS participated in
set of 150 indicators (available upon request; workshops in June 2014. These activities help
Table 1 presents the list of domains and sub support the sharing of knowledge, ideas, and
domains within which indicators are orga- experiences related to improving the quality
nized) related to personal outcomes, organiza- of services and supports, as well as to (1) build
tional outputs, and system-level strategies used capacity within the developmental services
to improve the social inclusion of Ontarians system through connection and exchange;
with IDD. Indicators appear to have face validi- (2) build on existing relationships with per-
ty (i.e., deemed relevant by stakeholder groups, sons with IDD, their families, agencies, and the
including persons with IDD) and to be con- representatives from the entity funding servi-
ceptually sound (i.e., supported by a review ces; (3) promote the involvement of new stake-
of the scientific knowledge). A necessary next holders; and (4) foster interest in promoting the
step is to appraise their validity and reliability social inclusion of persons with IDD.
through pilot testing among service agencies,
and relevant data holders. The proposed list of
indicators is not meant to be a rigid set of indi- Conclusion
cators that must be used in its entirety. Rather,
Over the past five years, Ontario has demon-
it is intended to serve as a flexible framework
strated leadership in the field of IDD through
guiding a phased approach to quality improve-
the adoption of the Services and Supports to
ment activities. Implementation should reflect
Promote the Social Inclusion of Persons with
a partnership including service users, service
Developmental Disabilities Act, 2008 and its relat-
providers, policy makers, data analysts, and
ed transformation agenda. The commitment to
researchers.
social inclusion is evident by a desire to under-
stand fully what social inclusion means and
how it should be measured. As services and
supports are transformed, the system’s pro-
cesses for data collection, use, and interpreta-
JODDGuiding Principles for Quality Improvement
99
tion must be adapted. Having already invested Rouette, Suzanne Robinson, Robyn Saaltink,
in the MAPS program of research, MCSS has Ami Tint, Sharon Walker, Colleen Webber, and
taken an important first step towards know- Andrea Winkelman-Gleed.
ledge-informed policy and practice. Next steps
require a collaborative approach whereby all MAPS is thankful for the contribution of: Valerie
stakeholder groups – i.e., policy-makers, ser- Bradley, President of Human Services Research
vice providers, service users and their families, Institute in Cambridge, Massachusetts, USA;
as well as researchers – engage in developing Wil Buntinx, Honorary Associate Research
a quality improvement monitoring system, and Professor in the Department of Health Services
become responsive to the developed knowledge. Research, Maastricht University and Research
Associate at the Governor Kremers Center
for Intellectual Disability, The Netherlands;
Key Messages From This Article Heather Stuart, Professor, Department of
Public Health Sciences, Queen’s University at
Persons with disabilities: It is important to Kingston and Bell Canada Mental Health and
know how your services and supports help you Anti-Stigma Research Chair; Yves Lachapelle,
to have the life you want. Professor, Département de psychoéducation,
Université du Québec à Trois-Rivières & Chaire
Professionals: Agencies collect a lot of infor- de recherche sur les technologies de soutien à
mation on the supports they provide, but a l’autodétermination des personnes présentant
standardized way of collecting this data across une déficience intellectuelle ou un trouble
all agencies is needed to maximize its use and envahissant du développement; Roy I. Brown,
inform quality improvement. It is also impor- Emeritus Professor, University of Calgary,
tant to involve persons with IDD and their fam- Alberta, Canada.
ilies in order to obtain meaningful information
on how to improve services and supports.
References
Policymakers: To improve the quality of the
services and supports provided to adults with Bryson, J. M., & Patton, M. Q. (2010). Analyzing
IDD, it is crucial to collect data on their impact and engaging stakeholders. In J. S. Wholey,
in a consistent and useable manner. H. P. Hatry, & K. E. Newcomer (ed.),
Handbook of Practical Program Evaluation
(3rd ed., pp. 30–54). San Francisco, CA:
Acknowledgements Jossey-Bass.
Buntinx Training & Consultancy. (2012).
The study was undertaken as part of the International consulting in disability
Multidimensional Assessment of Providers support services, creating value from
and Systems (MAPS). MAPS was supported knowledge. Retrieved from http://buntinx.
by a research grant from the Government of org/english
Ontario’s Ministry of Community and Social Buntinx, W. H. E. (2008). Quality of life based
Services (2010–2013). The views expressed in evaluation of support service performance.
this report are not necessarily the views of all Journal of Intellectual Disability Research, 52,
MAPS partners, researchers, collaborators or 794.
those of the Ministry. Buntinx, W. H. E., & Schalock, R. L. (2010).
Models of disability, quality of life, and
The MAPS team would like to thank mem- individualized supports: Implications
bers of its Consumer Consultation Groups for professional practice in intellectual
and Local Advisory Committees in Kingston, disability. Journal of Policy and Practice
Toronto, and Thunder Bay for their invaluable in Intellectual Disabilities, 7, 283–294.
contributions, as well as research assistants and doi:10.1111/j.1741-1130.2010.00278.x
Buntinx, W. H. E., & Benjamins, C. (2010).
associates, fellows and students. The following
Content analysis of client, family and staff
individuals have particularly contributed to
perceptions of service quality. Journal of
the present paper: Ashleigh Blinkhorn, Philip
Applied Research in Intellectual Disabilities,
Burge, Jamie Hagen, Maureen McDonald,
23, 486.
Alyson Mahar, Jami-Lynn Petner-Arrey, Julie
v.20 n.2100 Cobigo et al.
Cobigo, V. (2012). To be or not to be included? Foxx, R. M., Faw, G. D., Taylor, S., Davis, P. K.,
that is no longer the question. reaching a & Fulia, R. (1993). Would I be able to…?
consensus on the definition of social inclusion. Teaching clients to assess the availability
Kingston, Ontario, Canada: Department of of their community lifestyle preferences.
Public Health Sciences, Queen’s University. American Journal on Mental Retardation, 98,
Multidimensional Assessment of Providers 235–248.
and Systems (see www.mapsresearch.ca). Golubovic, S., & Skrbic, R. (2013). Agreement
Cobigo, V., Lysaght, R., & Ouellette-Kuntz, H. in quality of life assessment between
(2010). Understanding and defining social adolescents with intellectual disability and
inclusion. Journal of Applied Research in their parents. Research in Developmental
Disabilities, 34, 1863–1869.
Intellectual Disabilities, 23, 508.
Hartley, S., Finkenflugel, H., Kuipers, P., &
Cobigo, V., Mahar, A., & Stuart, H. (2012).
Thomas, M. (2009). Community-based
Belonging: What does it mean to belong.
rehabilitation: Opportunity and challenge.
Journal of Intellectual Disability Research, 56, The Lancet, 374, 1803–1804.
806. Hickey, R. (2011). The role and practices of
Cobigo, V., Ouellette-Kuntz, H., Lysaght, direct support workers in promoting greater
R., & Martin, L. (2012). Shifting our social inclusion, choice and independence: An
conceptualization of social inclusion. exploratory review of the literature. Kingston,
Stigma Research and Action, 2, 75–84. Ontario, Canada: Department of Public
Cobigo, V., Ouellette-Kuntz, H., Hickey, R., Health Sciences, Queen’s University.
Lunsky, Y., Lysaght, R., & Martin, L. (2013). Multidimensional Assessment of Providers
Multidimensional assessment of providers and Systems (see www.mapsresearch.ca).
and systems: Informing quality improvement Hodges, S. P., & Hernandez, M. (1999). How
practices in Ontario’s developmental services organizational culture influences outcome
system. Kingston, Ontario, Canada: information utilization. Evaluation and
Department of Public Health Sciences, Program Planning, 22, 183–197.
Queen’s University. Multidimensional Human Services Research Institute, & The
Assessment of Providers and Systems (see National Association of State Directors
www.mapsresearch.ca) of Developmental Disabilities Services.
Cobigo, V., & Stuart, H. (2010). Social inclusion
(2013). NCI. Retrieved from http://www.
and mental health. Current Opinion in
nationalcoreindicators.org/
Psychiatry, 23, 453–457.
Legislative Assembly of Ontario. (2014). Select
Cobigo, V., & Webber, C. (2012). Measuring our
success in promoting choices. Journal of Committee On Developmental Services Interim
Intellectual Disability Research, 56, 723. Report. Toronto, ON: Legislative Assembly
Cousins, J. B., & Whitmore, E. (1998). Framing of Ontario.
participatory evaluation. In Elizabeth Lemay, R. (2006). Social role valorization
Whitmore (ed.), New directions for program insights into the social integration
evaluation (80th ed., pp. 5–23) doi:10.1002/ conundrum. Mental Retardation, 44, 1–12.
ev.1114 Lloyd, V., Gatherer, A., & Kalsy, S. (2006).
Cummins, R. A., & Lau, A. L. D. (2003). Conducting qualitative interview research
Community integration or community with people with expressive language
exposure? A review and discussion in difficulties. Qualitative Health Research, 16,
relation to people with an intellectual 1386–1404.
disability. Journal of Applied Research in Lunsky, Y., Klein-Geltink, J. E., & Yates, E. A.
Intellectual Disabilities, 16, 145–157. (2013). Atlas on the primary care of adults with
Finlay, W. M. L., & Lyons, E. (2001). developmental disabilities in Ontario. Toronto,
Methodological issues when interviewing ON: Institute for Clinical Evaluative
and using self-report questionnaires Sciences (ICES) and Centre for Addiction
with people with mental retardation. and Mental Health.
Psychological Assessment, 13, 319–335. Lunsky, Y., Robinson, S., Tint, A., Vickar, M.,
Finlay, W. M. L., & Lyons, E. (2002). & Ouellette-Kuntz, H. (2013). System-wide
Acquiescence in interviews with people information about family caregivers of
who have mental retardation. Mental adults with intellectual/developmental
Retardation, 40, 14–29. disabilities – a scoping review of the
literature. Journal of Policy and Practice in
Intellectual Disabilities, 11, 8–14.
JODDGuiding Principles for Quality Improvement
101
Lunsky, Y., Tint, A., Robinson, S., Vickar, M., Martin, L., Ouellette-Kuntz, H., & Cobigo,
& Ouellette-Kuntz, H. (2012). System-wide V. (2013). Supporting person-directed
information about family caregivers of adults living through planning. Kingston,
with intellectual/developmental disabilities – A Ontario, Canada: Department of Public
scoping review of the literature. Kingston, Health Sciences, Queen’s University.
Ontario, Canada: Department of Public Multidimensional Assessment of Providers
Health Sciences, Queen’s University. and Systems (see www.mapsresearch.ca).
Multidimensional Assessment of Providers Martin, L., Ouellette-Kuntz, H., Cobigo,
and Systems (see www.mapsresearch.ca). V., & Ashworth, M. (2012b). Survey of
ww.mapsresearch.ca) planning practices in Ontario. Kingston,
Lysaght, R., Cobigo, V., & Hamilton, K. (2012). Ontario, Canada: Department of Public
Inclusion as a focus of employment related Health Sciences, Queen’s University.
research in intellectual disability: A Multidimensional Assessment of Providers
scoping review. Disability and Rehabilitation, and Systems (see www.mapsresearch.ca).
34, 1339–1350. Martin, L., Ouellette-Kuntz, H., Petner-Arrey,
Lysaght, R., Cobigo, V., Petner-Arrey, J., J., & Walker, S. (2013). Understanding
& Ouellette-Kuntz, H. (2014). Work planning experiences in Ontario through
case studies of planning teams. Kingston,
preparation and participation in
Ontario, Canada: Department of Public
Ontario for people with intellectual and
Health Sciences, Queen’s University.
developmental disabilities – A cross-
Multidimensional Assessment of Providers
region analysis. Journal on Developmental
and Systems (see www.mapsresearch.ca).
Disabilities, 20(2), 34–43.
MCSS. (2013). News release: Enhancing support
Lysaght, R., Ouellette-Kuntz, H., Cobigo, V.,
for adults with a developmental disability
& Petner-Arrey, J. (2013). Maximizing social
– Ontario government reducing waitlist
inclusion, choice and independence through
productivity options – A multi-phase study. pressures, improving services for those at high
Kingston, Ontario, Canada: Department of risk. Retrieved from http://news.ontario.
Public Health Sciences, Queen’s University. ca/mcss/en/2013/07/enhancing-support-
Multidimensional Assessment of Providers for-adults-with-a-developmental-disability.
and Systems (see www.mapsresearch.ca). html
Mahar, A., Cobigo, V., & Stuart, H. (2013). O’Sullivan, R. G. (2012). Collaborative
Conceptualizing belonging. Disability and evaluation within a framework of
Rehabilitation, 35, 1026–32. doi:10.3109/0963 stakeholder-oriented evaluation
8288.2012.717584 approaches. Evaluation and Program
Mahar, A., Cobigo, V., & Stuart, H. (2014). Planning, 35, 518–522.
Comments on measuring belonging as a Ouellette-Kuntz, H., Blinkhorn, A., Rouette,
service outcome. Journal on Developmental J., Blinkhorn, M., Lunsky, Y., & Weiss, J.
Disabilities, 20(2), 20–33. (2014). Parent resilience – an important
Martin, L., Ashworth, M., & Ouellette-Kuntz, indicator when planning services for
H. (2012). Planning practices in Ontario’s adults with intellectual and developmental
developmental services agencies. Journal disabilities. Journal on Developmental
on Developmental Disabilities, 18, 68–72. Disabilities, 20(2), 55–66.
Martin, L., & Cobigo, V. (2011). Definitions Ouellette-Kuntz, H., Cobigo, V., Hickey, R.,
matter in understanding social inclusion. Lysaght, R., Lunsky, Y., & Martin, L. (2011).
Journal of Policy and Practice in Intellectual Multidimensional Assessment of Providers and
Disabilities, 8, 276–282. Systems – Progress Report Year 1. Kingston,
Martin, L., & Ouellette-Kuntz, H. (2014). Ontario, Canada: Department of Public
Quality improvement in Ontario’s Health Sciences, Queen’s University.
developmental services system. Journal on Multidimensional Assessment of Providers
Developmental Disabilities, 20(2), iv–vii. and Systems (see www.mapsresearch.ca).
Martin, L., Ouellette-Kuntz, H., Cobigo, C.,
& Ashworth, M. (2012a). Measuring the
planning process: Review of existing measures.
Kingston, Ontario, Canada: Department of
Public Health Sciences, Queen’s University.
Multidimensional Assessment of Providers
and Systems (see www.mapsresearch.ca).
v.20 n.2102 Cobigo et al.
Ouellette-Kuntz, H., Cobigo, V., Hickey, R., Schalock, R. L., & Verdugo, M. A. (2012).
Lysaght, R., Lunsky, Y., & Martin, L. (2012). A leadership guide for today’s disabilities
Multidimensional Assessment of Providers and organizations: Overcoming challenges and
Systems – Progress Report Year 2. Kingston, making change happen. Baltimore, MD: Paul
Ontario, Canada: Department of Public H. Brookes Publishing Co.
Health Sciences, Queen’s University. Sherwin, J. (2010). Leadership for social
Multidimensional Assessment of Providers inclusion in the lives of people with
and Systems (see www.mapsresearch.ca). disabilities. International Journal of
Ouellette-Kuntz, H., Cobigo, V., Hickey, R., Leadership in Public Services, 6 supp., 84–93.
Lysaght, R., Lunsky, Y., & Martin, L. Stancliffe, R. J. (1995). Assessing opportunities
(2013a). Multidimensional Assessment of for choice-making: A comparison of self
Providers and Systems. MCSS-Queen’s and staff reports. American Journal on
Research Agreement 2010–2013. Year 3 Mental Retardation, 99, 418–429.
(2012–2013) and Final Report. Kingston, Thompson, J. R., Bryant, B. R., Campbell, E. M.,
Ontario, Canada: Department of Public Craig, E. M., Hughes, C. M., Rotholz, D. A.,
Health Sciences, Queen’s University. … Wehmeyer, M. (2004). Supports Intensity
Multidimensional Assessment of Providers Scale (SIS). Washington, DC: American
and Systems (see www.mapsresearch.ca). Association on Mental Retardation.
Ouellette-Kuntz, H., Lunsky, Y., Blinkhorn, United Nations Ad Hoc Committee on a
A., Robinson, S., & Tint, A. (2013b). Parents Comprehensive and Integral International
seeking adult developmental services for Convention on the Protection and
their children: A provincial study [Phase 1]. Promotion of the Rights and Dignity of
Kingston, Ontario, Canada: Department of Persons with Disabilities (2006). Convention
Public Health Sciences, Queen’s University. on the rights of persons with disabilities.
Multidimensional Assessment of Providers Retrieved from http://www.un.org/
and Systems (see www.mapsresearch.ca). disabilities/convention/conventionfull.
Poister, T. H. (2010). Performance shtml
measurement: Monitoring program van Loon, J. H. M., Bonham, G. S., Peterson, D.
outcomes. In J. S. Wholey, H. P. Hatry, & D., Schalock, R. L., Claes, C., & Decramer,
K. E. Newcomer (ed.), Handbook of practical A. E. M. (2013). The use of evidence-based
program evaluation (3rd ed., pp. 100–124). San outcomes in systems and organizations
Francisco, CA: Jossey-Bass. providing services and supports to
Reid, D. H., Everson, J. M., & Green, C. persons with intellectual disability.
W. (1999). A systematic evaluation of Evaluation and Program Planning, 36, 80–87.
preferences identified through person- Webber, C., & Cobigo, V. (2014). What should
centered planning for people with service providers know when measuring
profound multiple disabilities. Journal of how they impact consumers’ freedom
Applied Behavior Analysis, 32, 467–477. to make choice? Journal on Developmental
Rosenberg, D. (1997). Trend analysis and Disabilities, 20(2), 8–19.
interpretation. Rockville, MD: Division of World Health Organization. (2010). Community-
Science, Education and Analysis, Maternal based rehabilitation guidelines. Geneva,
and Child Health Bureau, US Department Switzerland: WHO Press.
of Health and Human Services. World Health Organization. (2011). World report
Saaltink, R., & Ouellette-Kuntz, H. (2014). on disability. New York, NY: World Health
What it takes to obtain services for Organization, World Bank.
adults with IDD in Ontario: An analysis Zeithaml, V. A., Parasuraman, A., & Berry, L.
of parents’ narratives. Journal on L. (1990). Delivering quality service: Balancing
Developmental Disabilities, 20(2), 44–54. Customer Perceptions and Expectations. New
Services and Supports to Promote the Social York, NY: Simon and Schuster.
Inclusion of Persons with Developmental
Disabilities Act, S.O. c.14. (2008). Retrieved
from http://www.e-laws.gov.on.ca/html/
statutes/english/elaws_statutes_08s14_e.
htm
JODDYou can also read
