TRIBAL VOCATIONAL REHABILITATION INSTITUTE PROGRAM EVALUATION 2018-2019 ACADEMIC YEAR - JOAN LAFRANCE MEKINAK CONSULTING SEATTLE, WASHINGTON ...
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Tribal Vocational Rehabilitation Institute
Program Evaluation
2018-2019 Academic year
Joan LaFrance
Mekinak Consulting
Seattle, WashingtonThis report is an evaluation of a Northwest Indian College program funded by the Rehabilitation Services Administration (RSA) through award H315C150002 in the U.S. Department of Education. The report represents the findings of an independent evaluation conducted by Joan LaFrance of Mekinak Consulting.
Tribal Vocational Rehabilitation Institute
Program Evaluation 2018-2020 Academic Year
Table of Contents
Executive Summary ......................................................................................................................... i
Introduction ..................................................................................................................................... 1
Evaluation Plan ........................................................................................................................... 1
Evaluation Design and Process ............................................................................................... 2
Organization of the Report...................................................................................................... 2
Section 1: Tribal Vocational Rehabilitation Enrollment and Graduation....................................... 3
Pre-post Assessments .................................................................................................................. 4
Gains in Knowledge ................................................................................................................ 5
Gains in Application of Skills ................................................................................................. 8
Gains in Confidence ................................................................................................................ 8
Section II Assessment of the TVRI Courses and Technical Assistance ....................................... 10
Course Development and Improvement ................................................................................... 10
Course Enrollment and Evaluation Surveys ......................................................................... 11
Technical Assistance ............................................................................................................. 14
Section III Interviews with Graduates and Supervisors ................................................................ 14
Graduates’ Experiences ............................................................................................................ 14
Reasons for Enrolling in the TVRI ....................................................................................... 15
Course of Study and Instructional Quality............................................................................ 15
Engaging Colleagues ............................................................................................................ 16
Application of Course Content ............................................................................................. 17
Contributions to their Participants ........................................................................................ 18
Benefits to Graduates ............................................................................................................ 19
Barriers to Services and Further Educational Needs ............................................................ 20
Final Reflections ................................................................................................................... 20Directors’ and Supervisors’ Experiences .................................................................................. 20
Supervisors’ Expectations ..................................................................................................... 20
Quality of Instruction and Communications with the TVRI ................................................ 21
Influence of the Institute on TVR Programs ......................................................................... 22
Recommendations for Additional Training .......................................................................... 23
Final Reflections ................................................................................................................... 24
Section IV: Evidence of the Benefits of the TVRI ....................................................................... 24
Final Reflections and Recommendations.............................................................................. 25
Appendix ....................................................................................................................................... 27
Table 1. Means and t-test for Significance for Knowledge of Skills for the Initial Pre- and
Post-Assessments ...................................................................................................................... 27
Table 2. Means and t-test for Significance for Application of Skills for the Initial Pre- and
Post-Assessments ...................................................................................................................... 28
Table 3. Means, Standard Deviation (SD) and Wilcoxon Test for Significance for Knowledge
of Skill Areas for the Revised Pre- and Post-Assessments (Number=9) .................................. 29
Table 4. Means, Standard Deviation (SD) and Wilcoxon Test for Significance for Confidence
in Skill Areas for the Revised Pre- and Post-Assessments (Number=9) .................................. 31
Table 5. Number of Students Completing Courses and Survey Response Rate....................... 33
Table 6. Students’ Satisfaction with their Courses ................................................................... 34
Table 7. Met Expectations for Course Organization ................................................................ 34
Table 8. Met Expectations for Course Materials ...................................................................... 35
Table 9. Met Expectations for Course Instructor ...................................................................... 35
Table 10. Content of Documents Related to My Work ............................................................ 36
Table 11. Documents Related to My Educational Interests ...................................................... 36
Table 12. Ease of Understanding Documents ........................................................................... 36
Table 13. Clarity of Course Requirements ............................................................................... 37
Table 14. Clarity of Course Materials....................................................................................... 37
Table 15. Clarity of GoToTraining Sessions ............................................................................ 37
Table 16. Clarity of Discussion Sessions.................................................................................. 37
Table 17. Number of Students Choosing Suggestions for Improving Courses ........................ 38
Table 18. Number of Students who Noted Following as Challenges ....................................... 38Table 19. Before and After Means and Standard Deviations for 301 and Wilcoxon Test for Statistical Significance .............................................................................................................. 39 Table 20. Before and After Means and Standard Deviations for 302 and Wilcoxon Test for Statistical Significance .............................................................................................................. 39 Table 21. Before and After Means and Standard Deviations for 303 and Wilcoxon Test for Statistical Significance .............................................................................................................. 40 Table 22. Before and After Means and Standard Deviations for 304 and Wilcoxon Test for Statistical Significance .............................................................................................................. 41 Table 23. Before and After Means and Standard Deviations for 305 and Wilcoxon Test for Statistical Significance .............................................................................................................. 41 Table 24. Before and After Means and Standard Deviations for 306 and Wilcoxon Test for Statistical Significance .............................................................................................................. 42 Table 25. Before and After Means and Standard Deviations for 307and Wilcoxon Test for Statistical Significance .............................................................................................................. 42 Table 26. Before and After Means and Standard Deviations for 308 and Wilcoxon Test for Statistical Significance .............................................................................................................. 43 Table 27. Before and After Means and Standard Deviations for 309 and Wilcoxon Test for Statistical Significance .............................................................................................................. 43 Table 28. Before and After Means and Standard Deviations for 310 and Wilcoxon Test for Statistical Significance .............................................................................................................. 44 Table 29. Before and After Means and Standard Deviations for 311 and Wilcoxon Test for Statistical Significance .............................................................................................................. 44
NWIC TVR Institute 2018-2019 Evaluation Report
Executive Summary
The Tribal Vocational Rehabilitation Institute (TVRI) is a partnership between Northwest Indian College
(NWIC) and Western Washington University (WWU). Its purpose is to provide comprehensive training
on service provision for tribal citizens with disabilities. Through NWIC, the Institute offers twelve
courses on various aspects of tribal vocational rehabilitation. The first seven sequential courses are
required to receive an Award of Completion from NWIC. The courses are designed to transfer to four-
year institutions. They are delivered online using Canvas software and are taught synchronously. The
Institute serves current and potential employees of Tribal Vocational Rehabilitation (TVR) programs.
The second evaluation conducted by Mekinak Consulting in Seattle, Washington focused on the academic
year of 2018-19, the fourth year of the grant from the U.S. Department of Education’s Rehabilitation
Services Administration. Major findings from the evaluation are summarized below.
• Over its first four years the institute assumed its outreach efforts would reach a minimum of 300
potential students, however they were able to advertise the program to over 2,093 individuals. Over
half of the applicants to the Institute were counselors (64%). Program Directors were 14% of the
enrollees, and assistants, aides and technical staff were 15%. Other staff such as job developers,
coaches, transition liaisons were 5% of the applicants.
• In its first four years the Institute has enrolled a total of 167 students in one or more courses which is
84% of the target enrollment over five years. With many of the students currently enrolled planning to
complete the seven-course Award of Completion program, the Institute should meet or exceed its
course enrollment goals by year five.
• By the end of the 2018-19 academic year, the TVRI had enrolled students from 61 TVR programs
which is 71% of the tribal programs in the United States. Thirty-nine or two-thirds (64%) of the 61
programs have two or more staff enrolled in Institute courses.
• The pre-post assessment of 40 TVR skills completed by 32 graduates indicated strong gains in
knowledge. The gains were statistically significant. The assessment was revised to measure students
sense of confidence. The 9 graduates who took the revised instrument described large gains in
confidence and all, but one of the changes, were statistically significant.
• Data from the Institute indicates that there were 243 student enrollments for the eleven courses taught
in the academic year. Of these 216 students completed the course for a completion rate of 89%.
• Analysis of students’ evaluations of their courses found that large majorities were very to extremely
satisfied with them, and they indicated they were able to apply their learning.
• Technical Assistance greatly expanded during the 2018-19 academic year with services offered
to 178 participants through different events.
• Interviews with 80% of the 2018-19 graduates found that they were very complimentary in their
description of the course of study and its value to their own practice. They described ways in which
the course of study influenced improvements in their own practices and to those of their TVR
program.
Mekinak Consulting
March 2020 iNWIC TVR Institute 2018-2019 Evaluation Report
• Interviews with 79% of the graduates’ supervisors found that their expectations for their workers
professional development were not only met, they were exceeded. They verified the graduates’
reports of improvements in policies and procedures at the program level.
• The Institute used ongoing evaluation and student feedback to revise and develop new courses.
• The Institute can continue to grow in enrollment, course offerings, and technical supports. Some of
the recommendations for future work based on the data gathered during the evaluation include:
o Courses 305-Case Management and 306-Job Search and Employment had the lowest
percentage of objectives meeting the statistical test for significance. Revisions and
improvements of these courses should be reviewed for better alignment with the course
learning objectives.
o Develop courses or modules, or technical assistance events that cover areas noted as needed
by some of the graduates and directors. These include transition services for youth, improving
motivational interviewing skills, discussions on how to maintain healthy self-care, ways to
manage participants with challenging behaviors.
o Consider mechanisms to help workers manage the challenges faced by working in
environments with large extended families when some members are related to workers in the
TVR program.
o Offer refresher courses to program directors or to graduates who move into director or
supervisory roles.
o Support a forum to help Institute faculty in both cultural content and pedagogical methods
appropriate for Native American learners.
o Continue to move towards the offering of a four-year online degree program that builds on
the curriculum of the Institute and easily transitions to continuing education.
Overall, the enrollment data, student course evaluations, and interviews with graduates, TVR directors,
Institute faculty and staff indicate that the TVRI is doing an excellent job in training tribal vocational
rehabilitation workers. Its work is contributing not only to individual professional development but also to
overall improvements in many TVR offices. In addition, the Institute is building a strong network and
communities of practice for TVR across the nation.
Mekinak Consulting
March 2020 iiNWIC TVR Institute 2018-2019 Evaluation Report
Introduction
The Tribal Vocational Rehabilitation Institute (TVRI) is a partnership between Northwest Indian
College (NWIC) and Western Washington University (WWU). Its purpose is to provide
comprehensive training on service provision for tribal citizens with disabilities. The training is
offered to current and potential future American Indian Vocational Rehabilitation (AIVRS)
counselors and staff. The TVRI is funded by the Rehabilitation Services Administration (RSA)
through award H315C150002 in the U.S. Department of Education. The Institute has five goals:
1. Conducting outreach and recruitment to American Indian and Alaskan Native TVR
programs.
2. Develop a curriculum for a structured, culturally relevant academic program on the
foundational vocational rehabilitation knowledge and skills based on RSA training topics.
3. Provide accessible courses that lead to an Award of Completion in Tribal Vocational
Rehabilitation Studies.
4. Provide technical assistance to graduates of the Institute.
5. Conduct ongoing internal and external evaluations to assess how the Institute are meeting
its goals.
The TVRI offers an Award of Completion in Tribal Vocational Rehabilitation Studies (hereafter
referred to as the Award) to enrollees who complete seven courses designed to strengthen the
capacity of TVR staff to provide services to American Indian and Alaska Natives. The courses
are taught in a synchronous online format by experts in the field of tribal vocational
rehabilitation. Although more courses have been added to the Institute’s curriculum over the past
two years, the original seven-course sequence is the requirement for the Award, with additional
courses offered as electives. The courses are designed to transfer to four-year institutions so
students who want to can continue their education.
The Institute is funded for five years. This is the second evaluation report prepared by Mekinak
Consulting. Although it focuses on the fourth year, the 2018-2019 academic year, data from all
four years as well as some from the specific academic year were used to evaluate the Institute’s
program. The enrollment, graduation, assessment data, and student evaluation survey results
were provided by the Institute. This information was augmented by qualitative information
collected by Mekinak Consulting of Seattle, Washington. Mekinak Consulting conducted the
analysis of all the evaluation data and developed the final report.
Evaluation Plan
The evaluation addresses five basic questions using data collected internally by TVRI staff and
by the external evaluator. The questions are:
1. Who was served by the program and were the goals for recruitment, training, and technical
assistance met?
Mekinak Consulting
March 2020 1NWIC TVR Institute 2018-2019 Evaluation Report
2. What evidence is there that the curriculum developed met the needs and expectations of
AIVRS personnel, and how did the curriculum and technical assistance work in practice
towards achieving the intended outcomes?
3. How do AIVRS directors and participants describe their experiences with the Institute?
4. What evidence is there that feedback from American Indians with disabilities was included in
the evaluation and used to improve program services and what adjustments and
improvements were made to the project and training of AIVRS personnel?
5. What evidence is there that the TVR Institute is having an impact on improving the delivery
of VR services to American Indians with disabilities and increasing employment outcomes
for American Indians with disabilities,
Evaluation Design and Process
The evaluation uses a quantitative/qualitative mixed-method design with data collected through a
combination of enrollment records, student surveys, pre-post assessments, interviews and focus
groups.
• Upon entry into the Institute students rate their knowledge of and confidence in doing 40
different vocational rehabilitation competencies. Those completing the seven-course
program and receiving the Award retake the assessment. By the fourth year, a large
enough pool of graduates allowed for a statistical comparison of their pre-/post-
assessments of their knowledge and confidence.
• Each participant enrolled in a class is invited to complete an online course evaluation.
The evaluation is done midway through the term to provide formative information to
course instructors, and again at the end of the course for a summative assessment of the
course. The end of the course evaluations were used for the evaluation analyses.
• Interviews were conducted with 80% of the 2018-19 academic year graduates, 79% of
their supervisors, and 57% of the course instructors. A focus group was conducted with
TVRI staff.
Quantitative data was analyzed using the Statistical Package for the Social Sciences (SPSS).
Dedoose, an online qualitative analysis software was used to examine qualitative information.
Organization of the Report
Findings from the evaluation are reported in four sections that closely align with the evaluation
questions. Additional information regarding the assessment and course evaluation data are
included in tables in the appendix. Several direct quotes from interviewees are included in this
report. Their statements are italicized to emphasize the voices of TVR workers, their directors
and supervisors, and other participants in the Institute.
Mekinak Consulting
March 2020 2NWIC TVR Institute 2018-2019 Evaluation Report
Section 1: Tribal Vocational Rehabilitation Enrollment and
Graduation
The evaluation question -- who was served by the program and were the goals for
recruitment, training, and technical assistance met – is addressed in this section except for the
information on Technical Assistance which is included in the next section. Analysis of
recruitment outreach, enrollment in courses, completion of courses for the Award indicate that
the TVRI is on track to meet its five-year projections.
The Institute’s outreach and recruitment goals were successful in reaching the target audiences.
Over its first four years it assumed its outreach efforts would reach a minimum of 300 potential
students, however, they were able to advertise the program to over 2,093 individuals. Over half
of the applicants to the Institute were counselors (64%). program directors were 14% of the
enrollees, assistants, aides and technical staff were 15%. Other staff such as job developers,
coaches, transition liaisons were 5% of the applicants. There were a few applicants who were not
employed in a TVR program (2%).
The enrollment in one or more courses was projected to be 200 by the end of year five. In the
first four years the Institute has enrolled a total of 167 students in one or more courses which is
84% of the target enrollment over five years. Enrollment in courses increased dramatically
between the first and second years and fluctuated in the third and fourth years (Figure 1). With
many of the students currently enrolled planning to complete the seven-course Award program,
the Institute should meet or exceed its course enrollment goals by year five.
Figure 1. Enrollment Numbers for One or More Courses Each Academic Year
2018-2019 58
2017-2018 39
2016-2017 62
2015-2016 8
0 10 20 30 40 50 60 70
By the end of the 2018-19 academic year, the TVRI had enrolled students from 61 TVR
programs which is 71% of the tribal programs in the United States (based on a list of 86
programs from the list provided by the Rehabilitation Services Administration dated February 2,
Mekinak Consulting
March 2020 3NWIC TVR Institute 2018-2019 Evaluation Report
2020). Thirty-nine or two-thirds (64%) of the 61 programs have two or more staff enrolled in
Institute courses. Table 1 illustrates the distribution of enrollment across the programs.
Table 1. Number and Percent of Enrollees by TVR Program
Enrollees Number Percentage
1 to 2 enrollees 10 26%
3 to 4 enrollees 22 56%
5 to 6 enrollees 4 10%
7 or more enrollees 3 8%
Total 39 100%
The percentage of staff per office enrolled is not reflected in the Table 1 as the size of the TVR
programs differ in relationship to their service populations and geographic spread, and the
number of TVR offices serving the tribe or tribal groups. A small tribe may have only three to
four staff and almost 100% enrollment in TVRI courses. The largest number enrolled from one
program is 10.
The Institute set a goal of 45 graduates over 5 years. At the end of the 2018-19 academic year, 33
students had received an Award for a completion rate of 74%. With 15 students on track to
graduate in 2019-2020, the goal of 45 will be met. Table 3 describes the target for the year and
the actual number who graduated.
Figure 2. Actual Award of Completion Compared to Target for the Year
20 18
18
16 15
14 12 12
12 11
10
8
6 4
4
2
0
2016-2017 2017-2018 2018-2019
Target Actual
Pre-post Assessments
All students who register for TVRI courses take a pre-assessment survey. The survey lists 40
tribal vocational rehabilitation skills and asks respondents to list their level of knowledge on a
five-point scale (1=none, 2=little, 3=moderate, 4=high, 5=very high). The students were asked to
Mekinak Consulting
March 2020 4NWIC TVR Institute 2018-2019 Evaluation Report rate their application of this set of skills using the same scale. Those students who graduate with an Award take the survey as a post-assessment of their knowledge of and application of the skills after completing the Institute’s course of study. An initial analysis of the pre-assessments completed from September 22, 2017 to April 11, 2019 noted that “application” was conditioned by the student’s position in TVR and the opportunity to provide the service described in the skill. Since the intent of the assessment was to measure knowledge gained and the student’s confidence in implementing the skill area, the assessment was changed from “application” of a skill to a self-reported “sense of confidence” in doing the skill. New students enrolling after April 2019 were given the revised assessment. Gains in Knowledge The initial assessment that compared a pre-/post-assessment of knowledge and application of the 40 skills was completed by twenty-three graduates. The mean ratings for knowledge of all skill areas improved on the post-assessment. All skills showed a statistically significant gain in knowledge using a t-test based on the differences in the mean scores and setting a one-tailed standard for statistical significance (p ≤ .05). Figure 3 describes the top 10 skills where the difference in the pre-/post-mean was one or nearly one full point. The revised assessment was completed by nine graduates. Given the smaller number, The Wilcoxon test for statistical significance was used to analyze the pre/post scores. A non- parametric test, the Wilcoxon statistic is based on the differences in the rankings given for the pre-assessment compared to the post-assessment and is a more effective test for a small sample. The changes in the rankings in changes in knowledge were positive for all 40 skills and all but one had gains that were statistically significant using the Wilcoxon one-tailed test at p ≤ .05. The skill, “collaboration with other counselors and programs” did not meet the test for significance. The 10 skill areas showing the highest gain in the mean scores for knowledge is described in Figure 4. There are six skill areas that were among the top ten on both assessments. These included: 1) skills related to working with clients to prepare the Individual Plan for Employment (IPE), 2) providing culturally appropriate services under the IPE, 3) using disability documentation to prepare an eligibility statement, 4) caseload data gathering, 5) learning more about the history of tribal vocational rehabilitation, and 6) provision of post-employment. Mekinak Consulting March 2020 5
NWIC TVR Institute 2018-2019 Evaluation Report
Figure 3. Largest Gains in Mean Pre- and Post Level of Knowledge
on the Initial Assessement Instrument (N=23)
Using disability documentation to prepare an 1.00
4.00
eligibility statement 3.00
Providing VR services under the IPE using 0.92
3.83
appropriate cultural protocols 2.91
0.91
Organization and time management 4.04
3.13
0.91
Managing multiple cases 3.87
2.96
0.91
Conducting a vocational assessment 3.43
2.52
0.91
Caseload data gathering 3.78
2.87
0.87
Partnering with the consumer to create the IPE 3.96
3.09
0.87
Determining consumer job readiness 3.87
3.00
0.87
Post-employment services 3.74
2.87
0.87
History of VR and tribal VR 3.48
2.61
0 1 2 3 4 5
Difference Post Mean Pre Mean
Mekinak Consulting
March 2020 6NWIC TVR Institute 2018-2019 Evaluation Report
Figure 4. Largest Gains in Post Mean Scores for Knowledge on the
Revised Assessment Instrument (N=9)
1.91
History of VR and tribal VR 4.13
2.22
Providing VR services under the IPE using 1.83
4.5
appropriate cultural protocols 2.67
1.61
Post-employment services 4.50
2.89
1.6
Conducting a vocational assessment 4.38
2.78
1.57
Job placement strategies 4.13
Axis Title
2.56
Accessing and using labor market 1.57
4.13
information 2.56
1.53
Caseload data gathering 4.75
3.22
Partnering with the consumer to create the 1.52
4.63
IPE 3.11
Using disability documentation to prepare an 1.5
4.5
eligibility statement 3.00
Helping the consumer to identify a vocational 1.49
4.38
goal 2.89
0.00 1.00 2.00 3.00 4.00 5.00
Difference Post Mean Pre Mean
Mekinak Consulting
March 2020 7NWIC TVR Institute 2018-2019 Evaluation Report
Gains in Application of Skills
Given that many skill areas on the assessment are closely related to a TVR position, the
application of skill areas which was measured by the initial assessment differed from gaining
knowledge of the skill. Eleven of the 40 sills had statistically significant positive changes (p ≤
.05) using a t-test. Figure 5 describes these eleven skills.
Table 5. Largest Gain in Post-Mean Scores for Application of Skills on
the Initial Assessment Instrument (N=23)
0.82
Conducting a vocational assessment 2.83
3.65
0.82
Psychosocial impacts of disability 3.57
4.39
0.75
Job placement strategies 3.21
3.96
0.65
History of VR and tribal VR 2.96
3.61
0.65
Providing traditional healing services 2.61
3.26
0.61
How "functional limitations" affect employment 3.87
4.48
0.57
Communicating with employers 3.34
3.91
0.57
Vocational implications of various disabilities 3.91
4.48
0.57
Concept of "barriers to employment" 4.00
4.57
0.48
Organization and time management 4.26
4.74
0.44
Caseload data gathering 3.69
4.13
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Difference Post Mean Pre Mean
The students noted being able to apply their increased knowledge of characteristics of disabilities
when working with their participants. They indicated improved application of job placement
strategies and communication with employers as well as application of better organization and
time management to their work and collect caseload data.
Gains in Confidence
The revised assessment measured students’ confidence in doing the skills rather than application
of them. Given the smaller number of students, the Wilcoxon test for statistical significance was
used to analyze the pre/post scores. All but the one of the skill areas, collaboration with other
Mekinak Consulting
March 2020 8NWIC TVR Institute 2018-2019 Evaluation Report
counselors and programs, demonstrated statistically significant higher rankings on the post-
assessment (p ≤ .05). Figure 6 describes the 10 skill areas with largest confidence gains in the
post-mean scores.
Figure 6. Largest Gains in Post Mean Scores for Confidence on
Revised Instrument (N=9)
Providing VR services under the IPE using 1.83
4.5
appropriate cultural protocols 2.67
Partnering with the consumer to create the 1.75
4.75
IPE 3
Accessing and using labor market 1.69
4.25
information 2.56
Using disability documentation to prepare 1.63
4.63
an eligibility statement 3
1.52
Types of case closure and documentation 4.63
3.11
1.49
Exploring comparable services and benefits 4.38
2.89
How "functional limitations" affect 1.49
4.38
employment 2.89
1.47
Assistive technology 4.25
2.78
1.46
Job placement strategies 4.13
2.67
1.42
Required elements for TVR eligibility 4.75
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
Difference Post Mean Pre Mean
Both assessments indicate that graduates believe they made considerable gains in their
knowledge of the skill areas needed to provide effective tribal vocational rehabilitation services.
The revised assessment is demonstrating that graduates are confident they can apply the skills. In
the appendix Tables 1 to 4 describe the pre-/post means and standard deviations and the results
of statistically significant testing for each of the 40 skills.
Mekinak Consulting
March 2020 9NWIC TVR Institute 2018-2019 Evaluation Report
Section II Assessment of the TVRI Courses and Technical
Assistance
This section addresses the evaluation questions: What evidence is there that the curriculum
developed met the needs and expectations of AIVRS personnel, and how did the
curriculum and technical assistance work in practice towards achieving the intended
outcomes? Data used to address these questions include interviews the external evaluator
conducted with the TVRI staff, graduates of the program, their supervisors, and the faculty who
taught courses. Student evaluations of their classes, and the types of and participation in
technical assistance opportunities are described.
Course Development and Improvement
The Institute developed seven courses that comprised an Award of Completion in TVR. The
courses cover the history of the program, regulatory information, and explanation of the practices
and procedures for TVR. They are 300 level courses designed to transfer to four-year degree
programs at NWIC, WWU, or other institutions.
• 301: TVR Foundations: History
• 302: TVR Foundations: Communications
• 303: TVR Foundations: Assessment and Eligibility
• 304: TVR Foundations: IPE Development
• 305: TVR Foundations: Case Record and Management
• 306: TVR Foundations: Job Search and Employment
• 307: TVR Foundations: Resource Management
The Institute added elective courses during the year. Both internal and external evaluations noted
the need for more information on management of TVR and more in-depth understanding of the
nature of addiction, and the psychiatric and medical aspects of disabilities. As a result of this
information, the TVRI added five new courses of which four were taught for the first time in the
2018-19 academic year. The new courses are:
• 308: TVR Foundations: Addictions and TVR Services
• 309: TVR Foundations: Medical Aspects
• 310: TVR Foundations: Psychiatric Disabilities
• 311: TVR Foundations: Financial Management
• 312: TVR Foundations: Advanced Techniques
To develop the courses, NWIC with support from WWU created a template to submit to the
Curriculum Committee at NWIC. The template is given to the instructors, who have
supplemented courses with their own materials. One faculty member who taught two of the
newer courses as well as one in the seven required for the Award noted that the core course had a
very rich set of materials that had been added by different instructors over the years. However,
Mekinak Consulting
March 2020 10NWIC TVR Institute 2018-2019 Evaluation Report
working in collaboration with NWIC and WWU, she was much more involved in developing the
two new courses.
In addition to developing new elective courses, the TVRI staff have continued to expand the
materials for use in classes. Videos of different TVR worker participant interactions have been
produced to illustrate problematic as well as exemplary engagement. Featuring Native American
actors, the videos are used in different classes to provoke discussions among the students.
The Institute is responding to student evaluations and feedback to make improvements to the
courses. In one case they revised a course to eliminate some of the redundancy students were
experiencing. In another course where students had concerns, the Institute is working with an
instructor with more experience in the subject area to improve the course.
The faculty recruited to teach courses are satisfied with the support they receive from the
Institute. They reported little difficulty connecting with their students. They appreciated the
synchronicity of the courses that enable students to actively engage with each other. A couple of
faculty members commented on students’ willingness to be open about their own experiences
and issues that they struggle with. They have openly discussed feeling burned out at times and
are helping others understand how to cope and care for themselves.
One faculty explained her experience teaching for the Institute:
I've taught as a faculty member in other community colleges both online and in person,
and this is the easiest place to work for. I've had the most support. The platform that they
use, Canvas, is the easiest online platform I've used, and I've used a few different ones. I
also think that they're most supportive of their students.
Course Enrollment and Evaluation Surveys
Data from the Institute indicates that there were 243 student enrollments for the eleven courses
taught in the academic year. Of these 216 students completed the course for a completion rate of
89%.
Evaluation surveys are administered to students midway through the course to provide formative
feedback to the instructors and at the end of course for a summative assessment of the course.
The surveys are comprehensive and solicit students’ views on satisfaction with and expectations
for the course, the value of the course to their work, ways in which they shared their learning at
work, the clarity of various course elements, challenges and ways to improve the course. They
also note in a reflective set of questions how well they understood the different course learning
objectives at the beginning of the course and at the completion of the course.
Most of the seven core courses were taught more than one quarter. Evaluations for these courses
were combined to create an overall evaluation for the course. The response rate for responding to
the course evaluation varies from a low of 45% to a high of 100%. The average response rate
Mekinak Consulting
March 2020 11NWIC TVR Institute 2018-2019 Evaluation Report
across all eleven courses was 64%. Table 5 in the appendix describes completion for each of the
courses and the student evaluation response rate.
Students rated their satisfaction on a seven-point scale. They could indicate whether they were
slightly, moderately or extremely dissatisfied, neither satisfied or dissatisfied, or slightly,
moderately or extremely satisfied. Students were very satisfied with their courses. As illustrated
in Table 2, most of the students rated the courses in the satisfied range with a large majority
noting that they were extremely satisfied. The full range of their responses is in Table 6 in the
appendix.
Table 2. Students’ Satisfaction with their Courses
Slightly to Neither Slightly to
Number of
Course Extremely satisfied/ Extremely
respondents
Dissatisfied dissatisfied Satisfied
301 24 4% 4% 92%
302 11 9% 91%
303 12 8% 92%
304 18 6% 6% 88%
305 12 8% 92%
306 14 14% 86%
307 8 100%
308 10 100%
309 4 100%
310 8 100%
311 3 100%
The students’ rated how well the course’s information improved their effectiveness as an
employee. The five-point scale ranged from extremely well to not at all well. Table 3 indicates
that most students rated the course as very to extremely well.
Table 3. Improved Effectiveness as a TVR Employee
Number of
Not well Not Moderately Very Extremely
Course respondents to
at all well well well well
the item
301 14 14% 36% 50%
302 9 22% 11% 67%
303 10 30% 10% 60%
304 18 11% 56% 33%
305 11 9% 55% 36%
306 9 11% 22% 56% 11%
307 7 14% 72% 14%
308 9 22% 56% 22%
Mekinak Consulting
March 2020 12NWIC TVR Institute 2018-2019 Evaluation Report
Table 3. Improved Effectiveness as a TVR Employee
Number of
Not well Not Moderately Very Extremely
Course respondents to
at all well well well well
the item
309 2 50% 50%
310 5 60% 40%
311 2 100%
The ratings for the qualities of instruction and materials, their recommendations for
improvements and challenges they faced are displayed in Tables 7 through 18 in the appendix.
The evaluation included a reflective set of items that asked students to rate their understanding of
each course learning objective before the course and after they completed the course. Table 4
describes the number of respondents to this item, the number of learning objectives for each
course, and the percent of those objectives that had a positive change from the before and after
ratings that were statistically significant using a one-tailed Wilcoxon test (p ≤ .05). Although
there were increases in the mean ratings for each learning objective after the courses were
completed, in four courses less than 50% of the gains were statistically significant (Table 4).
Table 4. Percentage of Students with Statistically Significant Gains in Course Learning
Objectives by Course
Number Percent
Number
TVRI Course of Significant
Respondents
Objectives Change*
301: TVR History 21 11 100%
302: Communications 10 11 100%
303: Assessment and Eligibility 11 8 100%
304: Developing the IPE 18 10 100%
305: Case Records and Management 11 9 33%
306: Job Search and Employment 10 9 44%
307: Resource Management 7 8 75%
308: Addictions and TVR Services 9 6 100%
309: Medical Aspects 2 7 0%
310: Psychiatric Disabilities 5 6 100%
311: Financial Management 2 8 0%
The means for the new courses, 309 and 311, went up; however, the changes were not significant
which likely is owing to small number of students in the courses and the fact that only two
completed the evaluation survey. Courses 305-Case Records and Management and 306-Job
Search and Employment had the lowest percentage of objectives meeting the statistical test for
Mekinak Consulting
March 2020 13NWIC TVR Institute 2018-2019 Evaluation Report
significance. These courses should be reviewed for better alignment of the curriculum with the
learning objectives. Tables 19 through 29 in the appendix describe the learning objectives for
each course, the change in the mean ratings, and the results of the Wilcoxon test for significance.
Technical Assistance
The goals of the Institute’s Technical Assistance (TA) are to support their participants with
the application of knowledge and skills learned in the courses and to create a framework
for a Community of Practice for TVR students and graduates. The TA greatly expanded
during the 2018-19 academic year with services offered to 178 participants through
different events. The staff at the Institute provided one-on-one assistance to 23
participants by responding to direct questions. They also responded to 13 questions posted
on a TA forum on Canvas. The inquiries fell into three major categories including those
related to regulatory or administrative issues, employment and post-employment services,
and IPE, eligibility and case management issues. They also responded to questions on
transition and cultural services and staff training.
A Coffee Break program was initiated which is accessed through the TA Forum on Canvas.
Guest speakers discuss TVR topics and entertain questions. Time is allotted for discussions
among Coffee Break participants. The program is publicized through the Institute newsletter and
has attracted participation from TVR staff who have not attended Institute courses. The Coffee
Breaks are recorded and archived along with the materials presented and discussed. There were
142 participants in Coffee Breaks during the year. In addition to providing real time assistance,
the TA questions and concerns allow the Institute to collect information that can be integrated
into courses. There are no formal evaluations of the TA as they are intended to provide a more
casual outreach and educational opportunity to TVR staff.
Section III Interviews with Graduates and Supervisors
To address the evaluation question: How do AIVRS directors and participants describe their
experiences with the Institute, telephone interviews were conducted with graduates and their
supervisors. Sixteen of the twenty students who graduated between the end of the 2018 summer
quarter and the Spring quarter of 2019 were interviewed in December 2019. The graduates
included TVR staff who were new to the program when they started to veterans with over 10
years of experience. These graduates worked for fourteen different directors or supervisors.
Eleven of these were interviewed via telephone in December 2019 and January 2020.
Graduates’ Experiences
Graduates were asked to describe their reasons for enrolling, the quality of the curriculum and
instruction, their application of the course content, and their experience working with the
Institute staff. Overall, the graduates were complimentary in their description of the course of
Mekinak Consulting
March 2020 14NWIC TVR Institute 2018-2019 Evaluation Report
study and its value to their own practice and to its contribution to improving policies and
procedures in their TVR programs.
Reasons for Enrolling in the TVRI
Graduates had different reasons for enrolling in the Institute. Although all expressed interest in
developing more knowledge and skills in TVR, there were specific motivations for enrolling in
the Award program. Some were encouraged to enroll by their colleagues who were taking the
courses. Others were urged by their supervisors. A few noted that the financial support provided
an incentive. One graduate summed up the different reasons by noting her three: “One, they were
paid for, two, it encouraged me to be a better counselor in my field, and three, the director
encouraged participation.”
Another student explained her motivation as follows:
Well, the director kept urging, urging, urging us. Since I've been in this position for a
long time, I thought I knew what I needed to know, but I'm glad I signed up. Because I'm
going to tell you, when I first started in '93, I started out as a social worker and then I
just got into this vocational [position]. But I really didn't understand vocational
rehabilitation. So, to me, it was just a job. Then taking these courses, I realized it's a lot
more than what I thought it was.
Being new to TVR was also a motivator. As one graduate explained:
I was actually the first rehabilitation technician for our program, so I was unfamiliar
with the program or overall. So, I felt like that was a good way for me to learn more
about vocational rehabilitation as well as disabilities. When our program manager
offered it, I went ahead and took the opportunity right away. [The TVRI] really helped
me learn not only my rehab tech position, but I ended up being hired as a Vocational
Rehabilitation Counselor. So, I think it really helped me prepare for the counseling
position.
Course of Study and Instructional Quality
All those interviewed praised the curriculum, noting that it was comprehensive and presented in
a well-designed sequence of classes. When asked which if any of the classes were the most
helpful, most of the graduates noted that they were all useful. For those who chose to pick out a
course, the most frequently mentioned was the course on preparing the IPE, followed by the
second course on communications and the first course on the history of TVR. The following is
an example of responses regarding the usefulness of the courses.
Well, I'll tell you what that they all stood out. I mean every course that I took I learned
something, and a lot in each one of those courses I was able to apply, to my work with my
participants. But I guess if there was one of them, probably the history of Tribal Vocational
Mekinak Consulting
March 2020 15NWIC TVR Institute 2018-2019 Evaluation Report
Rehabilitation. I think that was the first one that I had taken. But just to get that footing, the
foundation, was very interesting to me.
The main complaint shared by some of the graduates related to a course that had an instructor
that relied on lecture and did not allow for much interaction. As one graduate described, “[name
of course] was difficult for me, but I believe most of that was the teacher's style and less about
what the class was about. It was a hard class for me to take because I wasn't very compatible
with the teaching style. She did most of the talking and then there was very little time allowed for
us to interact with each other.” However, this instructor was the exception as all the graduates
praised the instructional quality, including the opportunities to share information and learn from
other classmates.
One graduate explained the value of having a mixed group of TVR staff in the courses: “Overall,
I loved everything about my classes because [we were] able to interact with other people at all
levels. We had supervisors or program managers all the way down to intake specialists.” So,
being able to interact with all those people and learn and teach with each other that was really
good for me. One student was pleased to be asked to share documents she had worked on with
others in her class. She noted that these then became exemplars that the instructor shared in other
classes.
The graduates found the Canvas software easy to learn and were able to get assistance from the
Institute staff if they encountered any problems with connections or the software. One student
described the format as follows: “The format was great. The learning platform was great. It
almost reminded me of being in a talking circle.” They complimented their instructors’
knowledge and experience. An example is the statement from one of the graduates: “We had
high quality instructors that knew their stuff, that knew where to get the resources and the
information we received were high quality.”
A few of the graduates noted the challenge faced taking courses while working full time.
However, they were supported by their supervisors and most were able to do their homework
during working hours. They explained that the instructors and the TVRI staff supported them
when they faced personal or family issues and accommodated their need for extra time in
completing assignments.
There were a few suggestions for improving the courses. Two recommendations included
conducting classes outside of work hours, perhaps on a Saturday, and creating a textbook for the
course of study.
Engaging Colleagues
All the graduates shared information they were learning with their colleagues. Most often they
described reporting at staff meetings. In some cases, a graduate would share material from a
class with another staff person whose position aligned with the resource. She explained that
Mekinak Consulting
March 2020 16NWIC TVR Institute 2018-2019 Evaluation Report
when she and others in her office who were also enrolled in the TVRI shared information, it
encouraged office wide discussions: “There were different things that would pop up that was in
class that we would bring up in our staff meeting and I would question as to how things were
going to be done. Do we do it this way, do it that way. You know? So it kind of, it kind of put a
little bubble in some people's heads, because I needed to understand more as to what way are we
going, how is it going to go, and what is the procedure and stuff like that.”
Another graduate explained that her supervisor audited the courses with her. Together they
discussed ideas for changes in their own procedures and policies. She described the support she
had to talk with other members of the staff: “[the supervisor’s interest] also helped me to go to
other people in my office and say, ‘Hey, what about this one, can we change this up a little bit,
will that help us to work better in our program?’ And I think it was very helpful.” Other sharing
was more formal with graduates describing opportunities to make presentations during office
training sessions.
In offices where several staff were enrolled in the Institute, the students would work together on
projects. In some instances, students would share information within their networks and promote
the TVRI to colleagues in other programs within their state or region.
The Institute was able to arrange several TVRI face-to-face class sessions at various sites around
the country. In a TVRS 302 face-to-face session in March 2018, most of the students enrolled
were from Wisconsin and Oklahoma. The Wisconsin students hosted those from Oklahoma. The
joint visit enabled the Wisconsin TVR staff to share their cultural practices such as the process
and ceremony involved in tapping maple syrup.
Application of Course Content
All the graduates had applied aspects of the courses to their own work and in many cases, they
worked with their colleagues to modify their office practices and procedures. The most
frequently mentioned changes were made to the IPE based on suggestions from the course and
examples of ways the IPE was organized in other TVR programs. The course on
communications and eligibility influenced changes in the orientation information. One graduate
described the following orientation for participants as follows:
So, one of [the changes] we did was a flow chart of how our services work. And we have
it shaped like a tree growing up to the sky. We use that to show our clients how the
process is, the TVR process, because there are a bunch of steps to it. Having that upfront
so that they know that this is a process and from start to finish -- what to expect was
really good.
Another graduate developed an “action tool” to tailor the process for the participant.
It provided me with ideas on how I could work with my consumers and like the action
tool I developed with my consumers.…When I started here, again when it came to
Mekinak Consulting
March 2020 17NWIC TVR Institute 2018-2019 Evaluation Report
eligibility, I wasn't sure how to develop the plan for certain individuals. ... I had some
individuals that were medically disabled that didn't have any cognitive issues. Whereas
some came in with cognitive [issues]. So, by being able to come up with the action tool
along with the IPE plan, it gave them a better understanding of following step by step,
what they're needing to do for the program.
A few graduates noted changes made in their office’s application forms and intake interviews to
improve their ability to get more information from the participant regarding their disabilities and
needs. One graduate described improvements in their presentations about TVR to outside
organizations.
Contributions to their Participants
When asked how services to their clients had improved, the most frequent reply related to the
quality of the graduates’ interactions with their clients. They described being able to
communicate more effectively and to better understand their clients’ choices and be more
empathetic during counseling sessions. In the words of one graduate: “I'm a better case manager
for taking the classes. So being able to work with clients better to offer them the choices that they
need to have and to give better service is probably the best way that I could describe that.”
Another described the overall improvement as follows: “I think that it has made me a better
counselor overall. The services that are delivered and the time I spend with them face-to-face,
I'm better at what I'm doing and more confident to better help them.”
One graduate explained using a reflection tool that was given out by one of the course
instructors. It helped the graduate and participant to reflect on each of the steps in the process
with an emphasis on the next steps to be taken: “That tool I use all the time with my consumers
and it's really helpful because not only does it help me remember what I need to write down from
my notes, it's right there. The consumer themselves were able to tell me what they learned that
day or what they needed to get done before our next appointment.”
One graduate was moved by a story she heard from a fellow student who had taken a participant
who had little knowledge of his culture to a sweat lodge. She decided to connect one of her more
problematic clients with his culture by arranging for a visit to an Alaskan village to witness
traditional lifestyles. She described his reaction to the visit.
It had a profound effect on the consumer who had a history of addiction and
incarceration. So, it's really empowering for them. He came to our office. We have a
morning meeting every day. He came to our morning meeting and he talked for about 30
minutes about how that trip had really changed his view of himself and really felt
empowered. His parents were both alcoholics and so he was left in the care of the State
early on in his life and left the village. And he never had that experience, that
empowering experience like this place where I'm from, these people that I'm from, they're
important and they're great at doing things. And he's a part of that too.
Mekinak Consulting
March 2020 18You can also read
