Quality Improvement in Services for Adults with Intellectual and Developmental Disabilities: Guiding Principles

Quality Improvement in Services
                                        for Adults with Intellectual and
                                          Developmental Disabilities:
                                               Guiding Principles
Volume 20, Number 2, 2014                                    Abstract
                                      Services and supports for persons with intellectual and develop-
                                      mental disabilities (IDD) worldwide have transitioned – or cur-
  Authors                             rently are – towards a community-based approach that enhanc-
                                      es social inclusion. This paper presents principles that could
  Virginie Cobigo,1                   guide quality improvement monitoring in community services
  Lynn Martin,2                       for adults with IDD. The principles were developed through a
  Rosemary Lysaght,3                  series of activities aiming to better understand relevant service
  Yona Lunsky,4                       outcomes, such as social inclusion and choice; consultations
  Robert Hickey,5                     with various stakeholder groups including persons with IDD;
  Hélène Ouellette-Kuntz6,7           and review of available data about the users of services and
                                      supports for Ontarians with IDD. The five guiding principles
      School of Psychology            are: (1) activities are embedded within a continuous quality
      and Centre for Research
      on Educational and              improvement cycle; (2) activities focus on personal outcomes
      Community Services,             that are important to persons with IDD; (3) activities capture
      University of Ottawa,           the multidimensional nature of the outcomes being measured;
      Ottawa, ON                      (4) knowledge is developed from multiple perspectives; and
      Department of Health            (5) knowledge is meaningful, actionable, and informs policy and
      Sciences,                       service improvements. The paper also presents a possible way
      Lakehead University,            forward in operationalizing a quality improvement monitoring
      Thunder Bay, ON                 system related to services for adults with IDD.
      Department of
      Rehabilitation Therapy,
      Queen’s University,         International, national and regional bodies have called for
      Kingston, ON                demonstrated effectiveness and for improved quality in the
      Centre for Addiction        services and supports provided to persons with disabilities.
      and Mental Health,          The World Health Organization (World Health Organization,
      Toronto, ON                 2010) recommends a community-based approach to provid-
      School of Policy Studies,   ing supports that enhance the social inclusion and overall
      Queen’s University,         quality of life of persons with disabilities. The United Nations
      Kingston, ON                Convention on the Rights of Persons with Disabilities (UN
      Department of               General Assembly, 2006) reaffirms their fundamental rights,
      Public Health Sciences,     including the rights to social inclusion and individual choice.
      Queen’s University,         The World Report on Disability (World Health Organization,
      Kingston, ON                2011) identifies persons with intellectual and developmental
      Ongwanada,                  disabilities (IDD) as particularly vulnerable to social exclu-
      Kingston, ON                sion and poor quality of life. For instance, compared to per-
                                  sons with other types of disabilities, persons with IDD may
  Correspondence                  not have all their support needs met in community-based
                                  settings. Furthermore, the general population may not be
  virginie.cobigo@uottawa.ca      sufficiently aware of their abilities, which limits opportuni-
                                  ties for social connectedness and meaningful contributions.
                                  In Ontario (the most populated Canadian province at more
  program evaluation,
                                  than 13 million inhabitants), the Services and Supports to Promote
  community services,
                                  the Social Inclusion of Persons with Developmental Disabilities Act
  social inclusion,
                                  (SIPDDA) (2008) governs services and supports provided to
  performance monitoring
                                  adults with IDD. SIPDDA is part of a transformation agenda
                                  that involved the closure of the last three residential institu-
 © Ontario Association on
    Developmental Disabilities
90      Cobigo et al.

tions in Ontario in 2009. It aims to build on the     United States, 25 states and four sub-state
transition to community-based supports to fur-        regions are contributing information on an
ther promote the social inclusion of adults with      annual basis to the National Core Indicators
IDD, and demonstrates Ontario’s commitment            (NCI) (Human Services Research Institute &
to modernizing and improving the quality of           The National Association of State Directors
services provided to this population. Ontario’s       of Developmental Disabilities Services, 2013).
Ministry of Community and Social Services             The program is a partnership between the
(MCSS, 2013) has developed an accompany-              Human Services Research Institute (HSRI)
ing set of regulations and Quality Assurance          in Cambridge Massachusetts and the United
Measures to help agencies plan for and pro-           States National Association of State Directors
vide high-quality services and supports. MCSS         of Developmental Disabilit ies Ser vices
funds more than 300 community agencies that           (NASDDDS). Multiple data sources are used
provide services to adults with IDD, referred to      including interviews with consumers, family
as “developmental services.” Developmental ser-       self-administered surveys, and collated agency
vices support adults with IDD to live, work and       or government-held data. The valid and reli-
participate in a wide range of activities in their    able consumer survey gathers about 50% of
communities. In addition, MCSS helps adults           the annual data on 114 indicators across five
with disabilities through its social assistance       domains: (1) individual outcomes, (2) health,
programs. For example, the Ontario Disability         welfare and rights, (3) system performance,
Support Program (ODSP) provides income and/           (4) staff stability, and (5) family indicators.
or employment supports to adults with disa-
bilities, including those with IDD. In 2013, the      In the Netherlands, the Quality Qube ®
province’s budget announced an additional             approach was developed (Buntinx Training &
investment of $42.5 million to enhance services       Consultancy, 2012) to promote quality improve-
provided to persons with IDD and their families       ment measures rather than performance indica-
– especially those with complex needs, as well as     tors. One of its fundamental assumptions is that
reduce service delivery wait times (MCSS, 2013).      the data collected are useful when they inform
In 2013, the province spent a total of $1.7 billion   the achievement of expected outcomes from the
on developmental services (MCSS, 2013). Given         service users’ perspective. The approach recog-
this significant investment, there is interest in     nizes that comparisons and rankings alone do
ensuring that the supports and services pro-          not lead to actions that can have positive out-
vided are of high quality, and are meeting the        comes for service users; a quality improvement
needs of persons with IDD and their families.         cycle has to be adopted. Quality Qube ® indica-
                                                      tors are categorized into five domains: (1) inde-
Many service agencies are committed to eval-          pendence, (2) social participation, (3) well-be-
uating the quality of their services through          ing, (4) facilitators, and (5) relationships with
accreditation processes and consumer satis-           the service.
faction surveys. However, many stakeholders
expressed concerns about service gaps. Since          Both the NCI and Quality Qube ® include the
2009, provincial and national media have              perspectives of various stakeholders (services
reported stories of families waiting for services,    users, families and staff members) when assess-
and families who can no longer cope with their        ing the quality of services. Both have the capac-
situation. The Ontario Ombudsman, an inde-            ity to report developed knowledge at different
pendent watchdog who investigates complaints          levels (i.e., team, agency, jurisdiction) to inform
from the public about Ontario government ser-         actions. Comparisons with other agencies or
vices, has enquired on hundreds of complaints         other jurisdictions are also rendered possi-
from families of adults with IDD (report forth-       ble through standardized data collection. The
coming). In 2013, the legislature convened a          capacity to monitor the quality of developmen-
select committee (elected provincial represent-       tal services allowing for comparisons across
ative) to investigate concerns and make recom-        agencies or regions, and across stakeholder per-
mendations for future services across the prov-       spectives has yet to be developed in Ontario.
ince (Legislative Assembly of Ontario, 2014).
                                                      This paper presents five principles that should
Outside of Ontario, some jurisdictions have           guide efforts to assess and monitor the quality
already invested in monitoring and quali-             of services for persons with IDD. These princi-
ty improvement systems. For example, in the           ples have been developed in consultation with

Guiding Principles for Quality Improvement
various stakeholders, including persons with                       Initiatives from other jurisdictions included the
IDD. Although this knowledge has been devel-                       National Core Indicators in the USA (Human
oped for and with Ontarian stakeholders, it                        Services Research Institute & the National
applies to many other jurisdictions.                               Association of State Directors of Developmental
                                                                   Disabilities Services, 2013), as well as the Quality
                                                                   Qube® from the Netherlands (Buntinx Training
                      Methods                                      & Consultancy, 2012). These strategies were
                                                                   chosen because they provide two examples of
This paper reports on the work completed by
                                                                   quality monitoring of IDD services, which both
the Multidimensional Assessment of Providers
and System (MAPS) research program that                            facilitate the use of standardized data collection
aimed to inform the Ontario developmental ser-                     and multi-level reporting leading to action. In
vices system in its efforts to monitor the quali-                  addition, both are led by senior academics with
ty of supports and services provided to adults                     long-standing reputations in the IDD field.
with IDD (Martin & Ouellette-Kuntz, 2014).
                                                                   Third, data routinely collected by MCSS and
Several activities led to the identification of                    developmental services agencies (in 2010–2013)
the principles of quality improvement present-                     were examined to determine whether they
ed in this paper. First, a better understanding                    could be used to support quality improve-
of social inclusion and choice was developed                       ment practices. These data included: service
through reviews of the literature and an inter-                    satisfaction surveys, information requested by
national consultation process (see Cobigo &                        accreditation bodies, information collected on
Stuart, 2010; Cobigo, Ouellette-Kuntz, Lysaght,                    persons with IDD who receive income supports
& Martin, 2012; Lysaght, Cobigo, & Hamilton,                       from the Ontario Disability Support Program
2012; Mahar, Cobigo, & Stuart, 2013; Mahar,                        (ODSP), and information on persons request-
Cobigo, & Stuart, 2014; Martin & Cobigo, 2011;                     ing services at one of the province’s nine
Webber & Cobigo, 2014). Reviewed literature                        Developmental Services Ontario (DSO) offices.
was not specific to the social inclusion and                       We described what data were collected, as well
choice of persons with IDD, but rather explored                    as what processes were used at the time to ana-
these concepts from the perspective of mem-                        lyse and report developed knowledge.
bers of society with and without disabilities.
The consultation process involved a survey of                      Fourth, findings from projects within the over-
various stakeholders, as well as presentations                     all program of research (i.e., literature reviews,
and workshops at international scientific con-                     surveys, interviews, and case studies related
ferences, including the European and world                         to productivity options, staff practices, pa­rents
congresses of the International Association                        applying for services, and person-directed
for the Scientific Study of Intellectual and                       planning) informed the development of indi-
Developmental Disabilities (Cobigo, Mahar,                         cators and data collection methods that could
& Stuart, 2012; Cobigo & Webber, 2012), the                        be used to assess and monitor quality of sup-
world congress of the World Association for                        ports and services and inform quality improve-
Social Psychiatry (Cobigo & Webber, 2012), and                     ment efforts.1 Some of these projects used data
the congress of the Association internationale
                                                                   routinely collected in Ontario. Most of them
de recherche sur le handicap mental (Cobigo,
                                                                   included original data collection. These projects
Lysaght, & Ouellette-Kuntz, 2010).
                                                                   have informed data collection feasibility in the
                                                                   developmental services system in Ontario.
Second, initiatives in Ontario and elsewhere
evaluating or monitoring the quality of servic-                    Finally, persons with IDD and their families,
es provided to adults with IDD were reviewed.                      service providers, and government represent-

1   See Cobigo et al., 2013; Hickey, 2011; Lunsky, Tint, Robinson, Vickar, & Ouellette-Kuntz, 2012; Lunsky, Robinson, Tint,
    Vickar, & Ouellette-Kuntz, 2013; Lysaght, Ouellette-Kuntz, Cobigo, & Petner-Arrey, 2013; Lysaght, Cobigo, Petner-Arrey, &
    Ouellette-Kuntz, 2014; Martin, Ashworth, & Ouellette-Kuntz, 2012; Martin, Ouellette-Kuntz, Cobigo, & Ashworth, 2012a;
    Martin, Ouellette-Kuntz, Cobigo, & Ashworth, 2012b; Martin, Ouellette-Kuntz, & Cobigo, 2013; Martin, Ouellette-Kuntz,
    Petner-Arrey, & Walker, 2013; Martin & Ouellette-Kuntz, 2014; Ouellette-Kuntz et al., 2014; Ouellette-Kuntz et al., 2011;
    Ouellette-Kuntz et al., 2012; Ouellette-Kuntz et al., 2013; Ouellette-Kuntz, Lunsky, Blinkhorn, Robinson, & Tint, 2013a;
    Ouellette-Kuntz, Lunsky, Blinkhorn, Robinson, & Tint, 2013c; Ouellette-Kuntz, Lunsky, Blinkhorn, Robinson, & Tint, 2013d;
    Saaltink & Ouellette-Kuntz, 2014 (all reports available upon request)

v.20 n.2
92      Cobigo et al.

atives were consulted on the best approach to      services system cannot always be aggregated.
monitor the quality of services for Ontarians      Agencies collect a broad range of information
with IDD. The consultation structure involved      about the people receiving supports. Such
regular meetings with Consumer Consultations       information may be related to individual ser-
groups (i.e., about six persons with IDD in each   vice plans, operational planning or compliance
group), and Local Advisory Committees that         with Quality Assurance Measures. However,
included at least one family member of indi-       this information is not systematically aggre­
viduals with IDD, service agency directors or      gated because agencies lack capacity at the
delegates, and representatives of MCSS (about      local level or because variances in indicators
10 individuals per group, most of whom were        make broader aggregation at the regional or
service agency representatives) (Martin &          provincial level impossible. The capacity of
Ouellette-Kuntz, 2014). These structures and       the system to better understand what could be
meetings were held in three locations across the   done for quality improvement would increase
province (i.e., Kingston, Toronto, and Thunder     if standardized data collection existed across
Bay). In addition, annual meetings took place      agencies. Similarly, some information collect-
with representatives of MCSS, including branch     ed by the application entities (DSOs) is col-
directors, managers, policy analysts at the cor-   lected in formats that make aggregation infea-
porate level, and representatives from region-     sible. For example, the application package
al offices. Consultation groups were asked to      Application for Developmental Services and
reflect on the following questions: What should    Supports (ADSS) for developmental services is
we know to improve Ontario’s Developmental         standardized across the province, but many of
Services system? What should we know to            its questions are open-ended. The complexity
make sure we are doing the “best possible          and resource intensity of analysing open-end-
job”? These groups were invited to advise the      ed questions has made the aggregation of data
research program on the interpretation of its      difficult (Ouellette-Kuntz et al., 2013c). This
findings and knowledge dissemination plan.         limitation could be easily overcome with a list
Meeting minutes were recorded and shared           of items within which to select those that best
with committee members.                            represent individuals’ responses. However,
                                                   because the ADSS is mainly used for person-
This paper integrates knowledge learned from       al service planning, it is important to maintain
these activities and proposes guiding principles   the capacity to record individuals’ responses
for quality improvement monitoring of services     verbatim.
for adults with IDD in Ontario’s developmental
services system.                                   Second, the data collected is not currently
                                                   linked between social assistance programs and
                                                   service agencies, which limits the capacity to
                   Results                         estimate the number of persons supported by
                                                   the developmental services system and only
In Ontario, extensive data are collected on the    provides fragmented information on their sup-
services and supports provided to adults with      port needs. Agreements between data holders
IDD. Nevertheless, consultations with repre-       could be negotiated to facilitate data sharing
sentatives of service agencies and MCSS have       and still respect the protection of personal
emphasized the limitations of the data collect-    information (Lunsky, Klein-Geltink, & Yates,
ed; although these data have potential, in their   2013). Examples of such agreements exist, and
current form they cannot shed light on where       often require the involvement of a third party
and how to improve service quality.                that has the knowledge and capacity to link
                                                   datasets (e.g., the Manitoba Centre for Health
Lessons Learned on Data Collection in              Policy holds and links data from multiple sec-
the Developmental Services System                  tors in the province of Manitoba; the Institute
                                                   for Clinical Evaluative Sciences is a data reposi-
There are five areas in which data collection      tory for provincial health administrative data in
could be improved to maximize its ability to       Ontario; and the Canadian Institute of Health
inform quality improvement efforts. First, the     Information acts as a nation-wide repository for
data currently collected by the developmental      linked administrative and clinical health data).

Guiding Principles for Quality Improvement
Third, no comprehensive baseline information             Therefore, in spite of the wealth of data that cur-
is available on the impact of services in order to       rently exists within the developmental services
monitor change over time. Support needs assess-          system, much work is needed to ensure that
ments are conducted for individuals request-             these data are relevant to persons with IDD and
ing services. Although informative for service           include their perspectives; are collected in a con-
planning purposes at the individual level, this          sistent way across agencies; and are collected in
information is not currently used to monitor ser-        a way that easily permits data linking, aggrega-
vice impact at an agency, regional or provincial         tion, and analysis. To do this, stakeholders rein-
level. This information gap makes difficult any          forced the need for clear partnerships between
evaluation of the effectiveness of services and          researchers, service providers, database hold-
supports. The projects conducted within this             ers, and service users. They expressed a desire
program of research (and reported elsewhere)             to focus on data collection efforts that could
provide some baseline information to this end.           inform service improvements needed to pro-
                                                         mote social inclusion, and would represent more
Fourth, most data collection activities do not           than a “checklist” approach to measurement.
focus on issues persons with IDD and their               Stakeholders also discussed the importance of
families consider most relevant. Data collec-            focussing data collection efforts on elements that
tion focuses mostly on compliance to processes.          are meaningful and actionable by the agencies
Information on many key personal outcomes –              collecting them, and that go beyond measuring
i.e., the impact of services and supports on the         issues related to compliance to process stan-
lives of persons with IDD and their families, as         dards. To support stakeholders in their efforts to
well as on critical service needs – is missing or        improve service quality monitoring, we propose
cannot be aggregated. For example, the sense of          the following five guiding principles.
burden and perceived distress of caregivers are
critical indicators of the system’s capacity to care     Guiding Principles to Quality
for persons with IDD that are currently only             Improvement Monitoring
assessed by some agencies (Ouellette-Kuntz,
Lunsky, Blinkhorn, Robinson, & Tint, 2013b).
                                                         Research and consultation with Ontarian
                                                         stakeholders have permitted the identification
Last, and arguably most importantly, current
                                                         of five principles that should guide the assess-
data collection efforts often miss the perspective
                                                         ment and monitoring of supports and services
of persons with IDD, since many of the ques-
                                                         to inform quality improvement efforts in IDD
tionnaires and surveys used rely on parents and
caregivers as proxy respondents. The perspective
of family members and significant others in the
lives of persons with IDD is important to inform         Principle 1: Activities Are Embedded Within
quality improvement efforts, but their opinions          a Continuous Quality Improvement Cycle
should not substitute those of persons with IDD,
since discrepancies between their opinions are           In the context of IDD services, continuous qual-
not unusual (Foxx, Faw, Taylor, Davis, & Fulia,          ity improvement requires that stakeholders
1993; Golubovic & Skrbic, 2013; Reid, Everson, &         engage in a cycle of ongoing data collection and
Green, 1999; Stancliffe, 1995). Techniques exist to      analysis to develop knowledge about actions
support the participation of persons with IDD in         needed to improve the quality of services with
interviews and questionnaires (Finlay & Lyons,           regards to desired outcomes for service users
2001; Finlay & Lyons, 2002; Lloyd, Gatherer, &           (van Loon et al., 2013). The cycle leads to the
Kalsy, 2006) and should be considered when               translation of knowledge into actions, and
planning data collection. Among others, the              monitoring of their impact on outcomes for ser-
following tips apply: (1) Avoid using complex            vice users. From the lessons learned through
multiple-choice questions; (2) Minimize acquies-         the research program, a graphic representation
cence and be sensitive to the tendency to please         of this cycle has been developed and is repre-
interviewers or carers; (3) Abstract concepts            sented in Figure 1.
should be avoided, as well as negative questions;
and (4) The level of literacy should be consistent       A continuous quality improvement approach
with the cognitive abilities of interviewees.            is distinct from Quality Assurance Measures

v.20 n.2
94      Cobigo et al.

                                                                                  DATA COLLECTION
                                                             Meaningful, Actionable, Multidimensional & Multiperspective

                                                                  SYSTEM          ORGANIZATIONAL         PERSONAL
                               Willingness to act promptly

                                                                STRATEGIES           OUTPUTS             OUTCOMES

                                                               Legislation and    Resources and            Improved
                                                                  policies,       processes used        social inclusion
                                                             community services to provide services    for service users
                                                                and supports,      and supports
                                                                 and system

                                                                  REPORTING                            ANALYSIS AND
                                                             Translation into actions
                                                                                                       into knowledge

 Figure 1. Continuous quality improvement in the developmental services system

which focus on compliance to process stan-                                               Principle 2: Activities Focus on Personal
dards (Schalock & Verdugo, 2012). The lat-                                               Outcomes That Are Important to Persons
ter, while important to ensure safeguards
                                                                                         with IDD
against harm and abuse, lacks a collaborative
approach that could lead to quality improve-
ment, and does not focus on outcomes for the                                             Decision-making should ideally be done based
service users. Stakeholders engaged in con-                                              on empirical evidence of the effectiveness and
tinuous quality improvement take owner-                                                  efficacy of services and programs. However,
ship of the process, identify improvements in                                            empirical knowledge of best practices for sup-
their own field of action, and are committed                                             porting adults with IDD and improving their
to respond (Bryson & Patton, 2010; Cousins &                                             social inclusion remains scarce. In the absence
Whitmore, 1998; O’Sullivan, 2012). A collabo-                                            of such supporting evidence, there is a need for
rative approach is recommended to enhance                                                quality improvement activities to focus on the
multidirectional communication between ser-                                              outcomes that are important to service users,
vice providers, service users, policy makers,                                            rather than solely on those relevant to the needs
data analysts and researchers, and to ensure                                             of policy makers and service providers (Schalock
their meaningful contribution (O’Sullivan,                                               & Verdugo, 2012; van Loon et al., 2013).
2012). Contrasting stakeholders’ perspectives
and identifying common priorities for action                                             Without knowledge of the outcomes deemed
should be central to quality improvement prac-                                           important by service users, it is difficult to
tices. In a continuous quality improvement                                               determine what services and policies are need-
cycle, all stakeholders should be responsive to                                          ed, and how to prioritize what actions are
the knowledge developed and put it into action.                                          required to improve social inclusion (van Loon
Actions include service and support improve-                                             et al., 2013). Social inclusion outcomes include
ments, as well as improvements to data collec-                                           experiencing valued and meaningful social
tion and analysis processes.                                                             roles, as well as feelings of trust, reciprocity
                                                                                         and belonging shared among members of a

Guiding Principles for Quality Improvement
community (Cobigo, 2012; Cobigo et al., 2012;          Principle 4: Knowledge is Developed from
Lemay, 2006; Sherwin, 2010). Monitoring the            Multiple Perspectives
quality of services for Ontarians with IDD,
therefore, should appraise the extent to which         Because of the diverse systems in which they
services improve these outcomes (see Table 1           belong, stakeholders’ perspectives on the qual-
for a list of relevant personal outcomes).             ity of services and the priorities for action are
                                                       likely to differ. It is important that all points of
Principle 3: Activities Capture the                    view are considered to support decision-mak-
Complexity and Multidimensional Nature                 ing when planning service quality improve-
of the Outcomes Being Measured                         ment (Buntinx & Benjamins, 2010; Buntinx,
                                                       2008). The perspective of service users and their
                                                       families is crucial, and they are the best respon-
A conceptual framework is important in
                                                       dents when measuring service users’ outcomes.
transforming data into meaningful knowl-
                                                       The perspective of service agencies and their
edge (Buntinx & Schalock, 2010; Schalock &             staff, as well as the perspective of the funding
Verdugo, 2012; van Loon et al., 2013). It provides     entity (i.e., the entity funding services, which is
an understanding of the complex interactions           the MCSS in Ontario), must also be considered.
between input, processes and outcomes, and
therefore goes beyond the “checklist” approach         Social inclusion is the outcome of complex
of quality assurance measures and accredita-           interactions between personal and environ-
tion bodies. From the MAPS research program,           mental factors (Cobigo, 2012; Cobigo et al.,
a theoretical framework of social inclusion,           2012; Cummins & Lau, 2003). A linear rela-
developed in consultation with persons with            tionship between services or policies and
IDD and their families, providers of develop-          improve social inclusion is difficult to draw;
mental services, policy makers and research-           factors independent of the service system will
ers, has emerged (Cobigo, 2012; Cobigo et al.,         influence observed outcomes. An ecological
2012). Therefore, the information to be collect-       approach is thus more appropriate, where the
ed should relate to the various dimensions of          individual is at the centre of a microsystem (i.e.,
social inclusion, if quality improvement efforts       family and friends), a mesosystem (i.e., connec-
are to be successful in promoting social inclu-        tions between elements of the microsystem),
sion.                                                  an exosystem (i.e., services and benefits), and
                                                       a macrosystem (i.e., values, attitudes and laws)
In addition, gathered information should focus         (Buntinx & Schalock, 2010).
on areas where the developmental services sys-
tem can act (i.e., organizational outputs and          Principle 5: Knowledge Is Meaningful,
system-level strategies). Organizational outputs       Actionable, and Informs Policy and Service
include efficiency measures, staff-related mea-        Improvements
sures, and program and service options avail-
able to service users. A focus on the quality of       Continuous quality improvement means that
outputs from the perspective of users is sug-          the information collected should lead to knowl-
gested (Schalock & Verdugo, 2012; Zeithaml,            edge and actions on how to improve outcomes
Parasuraman, & Berry, 1990) (see Table 1 for a         for the individuals supported (Schalock &
list of the characteristics of outputs appraised       Verdugo, 2012). To do so, stakeholders must
positively by users). System-level strategies to       develop a management information system
improve the social inclusion of adults with IDD        that allows for the collection, analysis and
include: legislation and policy enactment; com-        reporting of meaningful, timely and useful
munity-based and person-centred services and           data from various perspectives (Poister, 2010).
supports; anti-stigma and anti-discrimination          Stakeholders must also be responsive to the
initiatives, and system monitoring (Cobigo &           knowledge developed so that it leads to action.
Stuart, 2010).                                         Ongoing exchange among policy makers, ser-
                                                       vice providers, service users and researchers is
                                                       essential if one is to collect relevant data, devel-

v.20 n.2
96    Cobigo et al.

Table 1. Quality Improvement in IDD Services: Domains Proposed by the Multidimensional
          Assessment of Providers and System.

FOCUS                  DOMAIN             DEFINITION

PERSONAL               Valued             Being socially included means experiencing a range
OUTCOMES               social roles       of social roles within a variety of contexts. Social roles
Benefits for                              are valued when they respect the individual and the
service users                             community’s expectations, choices and needs.
                       Competency         To perform a given social role, the person has to have
                                          acquired the required skills to be competent in that role.
                       Reciprocity        Complementary social roles; social roles only exist
                                          when reciprocated by others.
                       Belonging          A sense of belonging is a subjective feeling of value
                                          and respect that is built from reciprocal relationships
                                          and shared experiences.
                       Choice             Social roles are the result of an individual’s choices.
                       Satisfaction       Social inclusion relies on the satisfaction of all
                                          members of the community that their social roles are
                                          valued, meaningful and respectful of choices.
ORGANIZATIONAL Tangibles               Quality of physical facilities, equipment and people.
OUTPUTS          Reliability           Ability to perform the promised service dependably
Resources and                          and effectively.
processes used   Responsiveness        Willingness to help consumers and provide prompt
by agencies to                         services.
provide services
                 Assurance             Knowledge and courtesy of employees and their ability
                                       to convey trust and confidence.
                       Empathy         Caring and individualized attention provided to
                       Availability    Presence of needed services and supports.
                       Affordability   Ability to procure the services and supports.
                       Awareness       Degree to which the service user is aware of potential
                                       services or supports.
                       Accessibility   Ability to access the services and supports (e.g.,
                                       barrier-free environment).
                       Extensiveness   Variety of services and supports.
                       Appropriateness Degree of correspondence between needs of the
                                       consumer and availability of relevant services or
SYSTEM                 Legislation and    Human rights discourse and legislation that protect
STRATEGIES             policy             the rights and freedoms of persons with IDD.
Ways in which          Supports and       Community-based and person-centered supports and
agencies and MCSS      services           services.
enhance personal       Anti-stigma        Efforts to eliminate social barriers and promote
outcomes and           Initiatives        adequate and equitable access to public goods and
organizational                            services.
                       System             Developing evidence of the effectiveness of actions.

Guiding Principles for Quality Improvement
op knowledge of where improvement is need-             Building Capacity for a Province‑Wide
ed, and transform this knowledge into action.          Quality Improvement System
Social inclusion outcomes are complex, and
                                                       An important factor that emerged from our
causal links between actions from the devel-
                                                       review of evidence-based practices is the criti-
opmental services system and improved social
                                                       cal role of organizational readiness (Hodges &
inclusion cannot be ascertained. Activities are
                                                       Hernandez, 1999) to manage data collection,
intended to gather evidence that the devel-
                                                       analyse results, and translate findings into
opmental services system is moving towards
                                                       action. Key to building capacity is the develop-
targeted actions, and the social inclusion of
                                                       ment of partnerships between all stakeholders
persons with IDD is improving over time.
                                                       who play an important role in quality improve-
However, it is unlikely that a specific action
                                                       ment including persons with IDD and their fam-
will be the unique cause of observed changes.
                                                       ilies, services agencies, funding entities, respon-
In addition, factors external to the develop-
                                                       sible ministries, and researchers (see Table 1).
mental services system have an impact on the
                                                       These partnerships will open discussions rela-
social inclusion of adults with IDD (Hartley,
                                                       tive to obtaining legal agreements for data shar-
Finkenflugel, Kuipers, & Thomas, 2009). For
                                                       ing and linkages; understanding technical issues
example, well executed supported employment
                                                       for data sharing and linkages; and developing
programs are likely to improve access to pro-
                                                       the infrastructure for an efficient information
ductivity options for adults with IDD, although
                                                       management system, including data collection,
these also increase as a result of support from
                                                       analysis, interpretation, and reporting of mean-
family and friends, as well as changes in the
                                                       ingful and actionable information.
attitudes of employers. It is thus important to
observe trends in both service users’ outcomes
                                                       In addition to having common goals and objec-
and organisational outputs when monitoring
                                                       tives, these partnerships must be based on
the quality of the services. Trend analysis and
                                                       mutual trust and a commitment to improving
interpretation require multiple data points (i.e.,
                                                       the quality of services. Differences in perspec-
repeated measures at the population/system
                                                       tives, interests and power must be acknowl-
level), an understanding of random variability
                                                       edged. Persons with IDD and their families
in population/system level measures, and con-
                                                       must believe that their views and contributions
sideration of external factors contributing to
                                                       are valued, and trust that the information they
observed changes (Poister, 2010). Most impor-
                                                       provide will be used to improve the quality of
tantly, trend data are informative if analyzed
                                                       services. Further, there needs to be a clear com-
and understood in light of program and policy
                                                       mitment among all stakeholders to act based
changes (Rosenberg, 1997).
                                                       on the knowledge gained through quality
                                                       improvement activities, which means making
                                                       relevant changes to services and policies.
Movement toward a continuous quality                   Over the three years of this program of
improvement approach to the assessment and             research, active and effective collaborations
monitoring of developmental services and sup-          have been developed and fostered with persons
ports demands a culture that is supportive of          with IDD (through Consumer Consultations),
knowledge-informed policy and practice. To             families and developmental services agencies
create that culture in Ontario, it is suggest-         (through Local Advisory Committees and
ed that three goals be pursued concurrently:           individual research projects), as well as with
(1) building capacity; (2) improving data col-         representatives from various levels of MCSS
lection and interpretation; and (3) exchanging         (through Local Advisory Committees, consul-
knowledge.                                             tations, and individual research projects). This
                                                       type of partnership model and stakeholder
                                                       involvement is strongly suggested for advanc-
                                                       ing quality improvement activities.

v.20 n.2
98      Cobigo et al.

Improving Data Collection                            Exchanging Knowledge
and Interpretation
                                                     While key stakeholder groups have been
The quality of the data on which decisions to        engaged throughout the program of research,
create or change policy and practice is key.         there is a need for further collaboration and
Moving forward, several steps are recommend-         exchange to build a consensus on how to
ed to improve data collection and interpreta-        improve the quality of supports provided to
tion. The first step is to better understand how     adults with IDD in Ontario and, more widely,
data currently collected could be interpreted        to promote a culture of knowledge-informed
and used for quality improvement. As previ-          policy and practice. In particular, electronic
ously mentioned, the system collects a range         forms of dissemination (e.g., websites, news-
of data that could be integrated into a mean-        letters); presentations, webinars, videoconfer-
ingful framework, and would provide addi-            ences; as well as workshops with stakeholders
tional information if linked and aggregated.         interested in more detailed information about
Modifications to current data collection and         how to assess and monitor the quality of their
management is also required. Adopting new            services are needed.
data collection processes would be an import-
ant step to allow agencies and their funders         To some extent, this has already started with
to capture indicators of outcomes for service        the MAPS team leading a variety of activities to
users. Finally, developing efficient systems for     engage with multiple stakeholder groups. For
generating reports meaningful to stakeholder         example, over 60 participants attended a webi-
groups is a crucial next step for data to be used    nar from about 30 sites across the province in
in a timely and relevant manner.                     February 2014, a panel presentation was held
                                                     at the Ontario Association for Developmental
Consultations with stakeholders, exploration         Disabilities (OADD) annual conference in April
of current data collection, as well as scientif-     2014, and persons with IDD, service providers
ic knowledge has led to the development of a         and representatives of MCSS participated in
set of 150 indicators (available upon request;       workshops in June 2014. These activities help
Table 1 presents the list of domains and sub         support the sharing of knowledge, ideas, and
domains within which indicators are orga-            experiences related to improving the quality
nized) related to personal outcomes, organiza-       of services and supports, as well as to (1) build
tional outputs, and system-level strategies used     capacity within the developmental services
to improve the social inclusion of Ontarians         system through connection and exchange;
with IDD. Indicators appear to have face validi-     (2) build on existing relationships with per-
ty (i.e., deemed relevant by stakeholder groups,     sons with IDD, their families, agencies, and the
including persons with IDD) and to be con-           representatives from the entity funding servi-
ceptually sound (i.e., supported by a review         ces; (3) promote the involvement of new stake-
of the scientific knowledge). A necessary next       holders; and (4) foster interest in promoting the
step is to appraise their validity and reliability   social inclusion of persons with IDD.
through pilot testing among service agencies,
and relevant data holders. The proposed list of
indicators is not meant to be a rigid set of indi-                   Conclusion
cators that must be used in its entirety. Rather,
                                                     Over the past five years, Ontario has demon-
it is intended to serve as a flexible framework
                                                     strated leadership in the field of IDD through
guiding a phased approach to quality improve-
                                                     the adoption of the Services and Supports to
ment activities. Implementation should reflect
                                                     Promote the Social Inclusion of Persons with
a partnership including service users, service
                                                     Developmental Disabilities Act, 2008 and its relat-
providers, policy makers, data analysts, and
                                                     ed transformation agenda. The commitment to
                                                     social inclusion is evident by a desire to under-
                                                     stand fully what social inclusion means and
                                                     how it should be measured. As services and
                                                     supports are transformed, the system’s pro-
                                                     cesses for data collection, use, and interpreta-

Guiding Principles for Quality Improvement
tion must be adapted. Having already invested          Rouette, Suzanne Robinson, Robyn Saaltink,
in the MAPS program of research, MCSS has              Ami Tint, Sharon Walker, Colleen Webber, and
taken an important first step towards know-            Andrea Winkelman-Gleed.
ledge-informed policy and practice. Next steps
require a collaborative approach whereby all           MAPS is thankful for the contribution of: Valerie
stakeholder groups – i.e., policy-makers, ser-         Bradley, President of Human Services Research
vice providers, service users and their families,      Institute in Cambridge, Massachusetts, USA;
as well as researchers – engage in developing          Wil Buntinx, Honorary Associate Research
a quality improvement monitoring system, and           Professor in the Department of Health Services
become responsive to the developed knowledge.          Research, Maastricht University and Research
                                                       Associate at the Governor Kremers Center
                                                       for Intellectual Disability, The Netherlands;
Key Messages From This Article                         Heather Stuart, Professor, Department of
                                                       Public Health Sciences, Queen’s University at
Persons with disabilities: It is important to          Kingston and Bell Canada Mental Health and
know how your services and supports help you           Anti-Stigma Research Chair; Yves Lachapelle,
to have the life you want.                             Professor, Département de psychoéducation,
                                                       Université du Québec à Trois-Rivières & Chaire
Professionals: Agencies collect a lot of infor-        de recherche sur les technologies de soutien à
mation on the supports they provide, but a             l’autodétermination des personnes présentant
standardized way of collecting this data across        une déficience intellectuelle ou un trouble
all agencies is needed to maximize its use and         envahissant du développement; Roy I. Brown,
inform quality improvement. It is also impor-          Emeritus Professor, University of Calgary,
tant to involve persons with IDD and their fam-        Alberta, Canada.
ilies in order to obtain meaningful information
on how to improve services and supports.
Policymakers: To improve the quality of the
services and supports provided to adults with          Bryson, J. M., & Patton, M. Q. (2010). Analyzing
IDD, it is crucial to collect data on their impact         and engaging stakeholders. In J. S. Wholey,
in a consistent and useable manner.                        H. P. Hatry, & K. E. Newcomer (ed.),
                                                           Handbook of Practical Program Evaluation
                                                           (3rd ed., pp. 30–54). San Francisco, CA:
           Acknowledgements                                Jossey-Bass.
                                                       Buntinx Training & Consultancy. (2012).
The study was undertaken as part of the                    International consulting in disability
Multidimensional Assessment of Providers                   support services, creating value from
and Systems (MAPS). MAPS was supported                     knowledge. Retrieved from http://buntinx.
by a research grant from the Government of                 org/english
Ontario’s Ministry of Community and Social             Buntinx, W. H. E. (2008). Quality of life based
Services (2010–2013). The views expressed in               evaluation of support service performance.
this report are not necessarily the views of all           Journal of Intellectual Disability Research, 52,
MAPS partners, researchers, collaborators or               794.
those of the Ministry.                                 Buntinx, W. H. E., & Schalock, R. L. (2010).
                                                           Models of disability, quality of life, and
The MAPS team would like to thank mem-                     individualized supports: Implications
bers of its Consumer Consultation Groups                   for professional practice in intellectual
and Local Advisory Committees in Kingston,                 disability. Journal of Policy and Practice
Toronto, and Thunder Bay for their invaluable              in Intellectual Disabilities, 7, 283–294.
contributions, as well as research assistants and          doi:10.1111/j.1741-1130.2010.00278.x
                                                       Buntinx, W. H. E., & Benjamins, C. (2010).
associates, fellows and students. The following
                                                           Content analysis of client, family and staff
individuals have particularly contributed to
                                                           perceptions of service quality. Journal of
the present paper: Ashleigh Blinkhorn, Philip
                                                           Applied Research in Intellectual Disabilities,
Burge, Jamie Hagen, Maureen McDonald,
                                                           23, 486.
Alyson Mahar, Jami-Lynn Petner-Arrey, Julie

v.20 n.2
100              Cobigo et al.

Cobigo, V. (2012). To be or not to be included?        Foxx, R. M., Faw, G. D., Taylor, S., Davis, P. K.,
    that is no longer the question. reaching a             & Fulia, R. (1993). Would I be able to…?
    consensus on the definition of social inclusion.       Teaching clients to assess the availability
    Kingston, Ontario, Canada: Department of               of their community lifestyle preferences.
    Public Health Sciences, Queen’s University.            American Journal on Mental Retardation, 98,
    Multidimensional Assessment of Providers               235–248.
    and Systems (see www.mapsresearch.ca).             Golubovic, S., & Skrbic, R. (2013). Agreement
Cobigo, V., Lysaght, R., & Ouellette-Kuntz, H.             in quality of life assessment between
    (2010). Understanding and defining social              adolescents with intellectual disability and
    inclusion. Journal of Applied Research in              their parents. Research in Developmental
                                                           Disabilities, 34, 1863–1869.
    Intellectual Disabilities, 23, 508.
                                                       Hartley, S., Finkenflugel, H., Kuipers, P., &
Cobigo, V., Mahar, A., & Stuart, H. (2012).
                                                           Thomas, M. (2009). Community-based
    Belonging: What does it mean to belong.
                                                           rehabilitation: Opportunity and challenge.
    Journal of Intellectual Disability Research, 56,       The Lancet, 374, 1803–1804.
    806.                                               Hickey, R. (2011). The role and practices of
Cobigo, V., Ouellette-Kuntz, H., Lysaght,                  direct support workers in promoting greater
    R., & Martin, L. (2012). Shifting our                  social inclusion, choice and independence: An
    conceptualization of social inclusion.                 exploratory review of the literature. Kingston,
    Stigma Research and Action, 2, 75–84.                  Ontario, Canada: Department of Public
Cobigo, V., Ouellette-Kuntz, H., Hickey, R.,               Health Sciences, Queen’s University.
    Lunsky, Y., Lysaght, R., & Martin, L. (2013).          Multidimensional Assessment of Providers
    Multidimensional assessment of providers               and Systems (see www.mapsresearch.ca).
    and systems: Informing quality improvement         Hodges, S. P., & Hernandez, M. (1999). How
    practices in Ontario’s developmental services          organizational culture influences outcome
    system. Kingston, Ontario, Canada:                     information utilization. Evaluation and
    Department of Public Health Sciences,                  Program Planning, 22, 183–197.
    Queen’s University. Multidimensional               Human Services Research Institute, & The
    Assessment of Providers and Systems (see               National Association of State Directors
    www.mapsresearch.ca)                                   of Developmental Disabilities Services.
Cobigo, V., & Stuart, H. (2010). Social inclusion
                                                           (2013). NCI. Retrieved from http://www.
    and mental health. Current Opinion in
    Psychiatry, 23, 453–457.
                                                       Legislative Assembly of Ontario. (2014). Select
Cobigo, V., & Webber, C. (2012). Measuring our
    success in promoting choices. Journal of               Committee On Developmental Services Interim
    Intellectual Disability Research, 56, 723.             Report. Toronto, ON: Legislative Assembly
Cousins, J. B., & Whitmore, E. (1998). Framing             of Ontario.
    participatory evaluation. In Elizabeth             Lemay, R. (2006). Social role valorization
    Whitmore (ed.), New directions for program             insights into the social integration
    evaluation (80th ed., pp. 5–23) doi:10.1002/           conundrum. Mental Retardation, 44, 1–12.
    ev.1114                                            Lloyd, V., Gatherer, A., & Kalsy, S. (2006).
Cummins, R. A., & Lau, A. L. D. (2003).                    Conducting qualitative interview research
    Community integration or community                     with people with expressive language
    exposure? A review and discussion in                   difficulties. Qualitative Health Research, 16,
    relation to people with an intellectual                1386–1404.
    disability. Journal of Applied Research in         Lunsky, Y., Klein-Geltink, J. E., & Yates, E. A.
    Intellectual Disabilities, 16, 145–157.                (2013). Atlas on the primary care of adults with
Finlay, W. M. L., & Lyons, E. (2001).                      developmental disabilities in Ontario. Toronto,
    Methodological issues when interviewing                ON: Institute for Clinical Evaluative
    and using self-report questionnaires                   Sciences (ICES) and Centre for Addiction
    with people with mental retardation.                   and Mental Health.
    Psychological Assessment, 13, 319–335.             Lunsky, Y., Robinson, S., Tint, A., Vickar, M.,
Finlay, W. M. L., & Lyons, E. (2002).                      & Ouellette-Kuntz, H. (2013). System-wide
    Acquiescence in interviews with people                 information about family caregivers of
    who have mental retardation. Mental                    adults with intellectual/developmental
    Retardation, 40, 14–29.                                disabilities – a scoping review of the
                                                           literature. Journal of Policy and Practice in
                                                           Intellectual Disabilities, 11, 8–14.

Guiding Principles for Quality Improvement
Lunsky, Y., Tint, A., Robinson, S., Vickar, M.,          Martin, L., Ouellette-Kuntz, H., & Cobigo,
    & Ouellette-Kuntz, H. (2012). System-wide               V. (2013). Supporting person-directed
    information about family caregivers of adults           living through planning. Kingston,
    with intellectual/developmental disabilities – A        Ontario, Canada: Department of Public
    scoping review of the literature. Kingston,             Health Sciences, Queen’s University.
    Ontario, Canada: Department of Public                   Multidimensional Assessment of Providers
    Health Sciences, Queen’s University.                    and Systems (see www.mapsresearch.ca).
    Multidimensional Assessment of Providers             Martin, L., Ouellette-Kuntz, H., Cobigo,
    and Systems (see www.mapsresearch.ca).                  V., & Ashworth, M. (2012b). Survey of
    ww.mapsresearch.ca)                                     planning practices in Ontario. Kingston,
Lysaght, R., Cobigo, V., & Hamilton, K. (2012).             Ontario, Canada: Department of Public
    Inclusion as a focus of employment related              Health Sciences, Queen’s University.
    research in intellectual disability: A                  Multidimensional Assessment of Providers
    scoping review. Disability and Rehabilitation,          and Systems (see www.mapsresearch.ca).
    34, 1339–1350.                                       Martin, L., Ouellette-Kuntz, H., Petner-Arrey,
Lysaght, R., Cobigo, V., Petner-Arrey, J.,                  J., & Walker, S. (2013). Understanding
    & Ouellette-Kuntz, H. (2014). Work                      planning experiences in Ontario through
                                                            case studies of planning teams. Kingston,
    preparation and participation in
                                                            Ontario, Canada: Department of Public
    Ontario for people with intellectual and
                                                            Health Sciences, Queen’s University.
    developmental disabilities – A cross-
                                                            Multidimensional Assessment of Providers
    region analysis. Journal on Developmental
                                                            and Systems (see www.mapsresearch.ca).
    Disabilities, 20(2), 34–43.
                                                         MCSS. (2013). News release: Enhancing support
Lysaght, R., Ouellette-Kuntz, H., Cobigo, V.,
                                                            for adults with a developmental disability
    & Petner-Arrey, J. (2013). Maximizing social
                                                            – Ontario government reducing waitlist
    inclusion, choice and independence through
    productivity options – A multi-phase study.             pressures, improving services for those at high
    Kingston, Ontario, Canada: Department of                risk. Retrieved from http://news.ontario.
    Public Health Sciences, Queen’s University.             ca/mcss/en/2013/07/enhancing-support-
    Multidimensional Assessment of Providers                for-adults-with-a-developmental-disability.
    and Systems (see www.mapsresearch.ca).                  html
Mahar, A., Cobigo, V., & Stuart, H. (2013).              O’Sullivan, R. G. (2012). Collaborative
    Conceptualizing belonging. Disability and               evaluation within a framework of
    Rehabilitation, 35, 1026–32. doi:10.3109/0963           stakeholder-oriented evaluation
    8288.2012.717584                                        approaches. Evaluation and Program
Mahar, A., Cobigo, V., & Stuart, H. (2014).                 Planning, 35, 518–522.
    Comments on measuring belonging as a                 Ouellette-Kuntz, H., Blinkhorn, A., Rouette,
    service outcome. Journal on Developmental               J., Blinkhorn, M., Lunsky, Y., & Weiss, J.
    Disabilities, 20(2), 20–33.                             (2014). Parent resilience – an important
Martin, L., Ashworth, M., & Ouellette-Kuntz,                indicator when planning services for
    H. (2012). Planning practices in Ontario’s              adults with intellectual and developmental
    developmental services agencies. Journal                disabilities. Journal on Developmental
    on Developmental Disabilities, 18, 68–72.               Disabilities, 20(2), 55–66.
Martin, L., & Cobigo, V. (2011). Definitions             Ouellette-Kuntz, H., Cobigo, V., Hickey, R.,
    matter in understanding social inclusion.               Lysaght, R., Lunsky, Y., & Martin, L. (2011).
    Journal of Policy and Practice in Intellectual          Multidimensional Assessment of Providers and
    Disabilities, 8, 276–282.                               Systems – Progress Report Year 1. Kingston,
Martin, L., & Ouellette-Kuntz, H. (2014).                   Ontario, Canada: Department of Public
    Quality improvement in Ontario’s                        Health Sciences, Queen’s University.
    developmental services system. Journal on               Multidimensional Assessment of Providers
    Developmental Disabilities, 20(2), iv–vii.              and Systems (see www.mapsresearch.ca).
Martin, L., Ouellette-Kuntz, H., Cobigo, C.,
    & Ashworth, M. (2012a). Measuring the
    planning process: Review of existing measures.
    Kingston, Ontario, Canada: Department of
    Public Health Sciences, Queen’s University.
    Multidimensional Assessment of Providers
    and Systems (see www.mapsresearch.ca).

v.20 n.2
102              Cobigo et al.

Ouellette-Kuntz, H., Cobigo, V., Hickey, R.,         Schalock, R. L., & Verdugo, M. A. (2012).
    Lysaght, R., Lunsky, Y., & Martin, L. (2012).        A leadership guide for today’s disabilities
    Multidimensional Assessment of Providers and         organizations: Overcoming challenges and
    Systems – Progress Report Year 2. Kingston,          making change happen. Baltimore, MD: Paul
    Ontario, Canada: Department of Public                H. Brookes Publishing Co.
    Health Sciences, Queen’s University.             Sherwin, J. (2010). Leadership for social
    Multidimensional Assessment of Providers             inclusion in the lives of people with
    and Systems (see www.mapsresearch.ca).               disabilities. International Journal of
Ouellette-Kuntz, H., Cobigo, V., Hickey, R.,             Leadership in Public Services, 6 supp., 84–93.
    Lysaght, R., Lunsky, Y., & Martin, L.            Stancliffe, R. J. (1995). Assessing opportunities
    (2013a). Multidimensional Assessment of              for choice-making: A comparison of self
    Providers and Systems. MCSS-Queen’s                  and staff reports. American Journal on
    Research Agreement 2010–2013. Year 3                 Mental Retardation, 99, 418–429.
    (2012–2013) and Final Report. Kingston,          Thompson, J. R., Bryant, B. R., Campbell, E. M.,
    Ontario, Canada: Department of Public                Craig, E. M., Hughes, C. M., Rotholz, D. A.,
    Health Sciences, Queen’s University.                 … Wehmeyer, M. (2004). Supports Intensity
    Multidimensional Assessment of Providers             Scale (SIS). Washington, DC: American
    and Systems (see www.mapsresearch.ca).               Association on Mental Retardation.
Ouellette-Kuntz, H., Lunsky, Y., Blinkhorn,          United Nations Ad Hoc Committee on a
    A., Robinson, S., & Tint, A. (2013b). Parents        Comprehensive and Integral International
    seeking adult developmental services for             Convention on the Protection and
    their children: A provincial study [Phase 1].        Promotion of the Rights and Dignity of
    Kingston, Ontario, Canada: Department of             Persons with Disabilities (2006). Convention
    Public Health Sciences, Queen’s University.          on the rights of persons with disabilities.
    Multidimensional Assessment of Providers             Retrieved from http://www.un.org/
    and Systems (see www.mapsresearch.ca).               disabilities/convention/conventionfull.
Poister, T. H. (2010). Performance                       shtml
    measurement: Monitoring program                  van Loon, J. H. M., Bonham, G. S., Peterson, D.
    outcomes. In J. S. Wholey, H. P. Hatry, &            D., Schalock, R. L., Claes, C., & Decramer,
    K. E. Newcomer (ed.), Handbook of practical          A. E. M. (2013). The use of evidence-based
    program evaluation (3rd ed., pp. 100–124). San       outcomes in systems and organizations
    Francisco, CA: Jossey-Bass.                          providing services and supports to
Reid, D. H., Everson, J. M., & Green, C.                 persons with intellectual disability.
    W. (1999). A systematic evaluation of                Evaluation and Program Planning, 36, 80–87.
    preferences identified through person-           Webber, C., & Cobigo, V. (2014). What should
    centered planning for people with                    service providers know when measuring
    profound multiple disabilities. Journal of           how they impact consumers’ freedom
    Applied Behavior Analysis, 32, 467–477.              to make choice? Journal on Developmental
Rosenberg, D. (1997). Trend analysis and                 Disabilities, 20(2), 8–19.
    interpretation. Rockville, MD: Division of       World Health Organization. (2010). Community-
    Science, Education and Analysis, Maternal            based rehabilitation guidelines. Geneva,
    and Child Health Bureau, US Department               Switzerland: WHO Press.
    of Health and Human Services.                    World Health Organization. (2011). World report
Saaltink, R., & Ouellette-Kuntz, H. (2014).              on disability. New York, NY: World Health
    What it takes to obtain services for                 Organization, World Bank.
    adults with IDD in Ontario: An analysis          Zeithaml, V. A., Parasuraman, A., & Berry, L.
    of parents’ narratives. Journal on                   L. (1990). Delivering quality service: Balancing
    Developmental Disabilities, 20(2), 44–54.            Customer Perceptions and Expectations. New
Services and Supports to Promote the Social              York, NY: Simon and Schuster.
    Inclusion of Persons with Developmental
    Disabilities Act, S.O. c.14. (2008). Retrieved
    from http://www.e-laws.gov.on.ca/html/

You can also read
Next slide ... Cancel