Creating an Educational Research Culture of Assessment for Programmatic Excellence
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Creating an Educational
Research Culture of
Assessment for
Programmatic Excellence
APTA EDUCATIONAL LEADERSHIP CONFERENCE 2019
Benchmarks Taskforce
u Peter Altenburger, Assoc Prof, Indiana Univ
u Jacklyn Brechter, Assoc Prof, Chapman Univ
u Gary Chleboun, Prof, Ohio Univ
u Amy Heath, Assist Prof, Western Michigan Univ
u Diane Jette, Prof Emeritus, MGH Institute of Health Professions
u Denise Schilling, Prof, Western Univ of Health Sciences
u Kimberly Topp, Prof Emeritus, Univ of California San Francisco
u Barbara Tschoepe, Visiting Prof, Franklin Pierce Univ
1Presentation Goals
u Illustrate the value of creating a culture of programmatic assessment
u Integrate data from internal and external stakeholders
u Differentiate between the value of programmatic and professional shared
data assessment
u Create a commitment to develop a professionally supported national
program assessment instrument
u Ensure that such an instrument is able to share data across multiple
stakeholders to strengthen informed decisions and enhance ongoing
improvement
Defining Assessment
u Establishing clear, measureable outcomes for student learning
u Ensuring that students have the necessary resources to achieve those
outcomes
u Systematically gathering, analyzing, interpreting, evidence to determine
whether student outcomes are consistent with expected goals and
objectives
u Creating an ongoing process that uses assessment outcomes to inform
decision-making about where and how to improve student learning
Angelo, 1995
2Defining Assessment
u Students learn best when courses and activities are purposefully designed as coherent,
integrated learning experiences, in which courses and out-of-class experiences build on
and reinforce one another
“When students can see connections between learning experiences, learning is deeper and more
lasting”
Graff, 2008
"Successful learners build pathways, or make connections between what they are trying to learn
and what they already know. .."
Niemczyk, 2012
u Collaborative curricular designs require collaborative assessment
Assessment
Culture
3A Culture of Assessment
u What is your visceral response to this comment from a peer-faculty
member?
u The accreditation self-study starts next week
u What data can you gather quickly for the Dean's queries:
u How do our numbers of PhD/DSc faculty compare to those of other
public/private or Masters/R1 institutions?
u What is our student indebtedness as compared to other peer institutions?
Assessment Culture Defined
u Culture: "The set of shared attitudes, values, goals, and practices that
characterizes an institution or organization" (Merriam-Webster Dictionary)
u Assessment Culture: An environment that values reflection in action and
reflection on action using a systematic method of collecting, analyzing
and acting on data to inform decisions, implement improvements and
aspire toward excellence. (Diane Jette)
u Setting high expectations and holding each other accountable.
(Jensen et al., 2017)
u Fundamental processes for benchmarking or quality assurance are 1)
identifying indicators of excellence, and 2) mechanism(s) for capturing
performance (Heath et al., 2018)
4...any field that deserves
to be described as a
profession must take on
the highest standard of
self-evaluation, both for
individual practitioners
and the entire
professional community.
Lee S. Shulman, PhD
President, Carnegie
Foundation for the
Advancement of Teaching
Levels of PROGRAMMATIC
INSTITUTIONAL
Assessment PROFESSIONAL
5Programmatic Assessment
u Student learning is measured based on achievement of the established
program outcomes --- faculty performance and resources are also
included in the assessment
u "Programmatic assessment is an approach in which routine information
about the learner’s competence and progress is continually collected,
analyzed and, where needed, complemented with purposively collected
additional assessment information..." (Schuwirth et al., 2017)
Institutional Assessment
u Institutions assess student outcomes as linked to the institutional mission,
values and goals
u As student learning is an institutional outcome, programmatic and
institutional assessments are linked
u Accreditation bodies are looking for longitudinal outcome data to
demonstrate continuous quality improvement in programs
u Challenges are found in identifying data elements that fit multiple
professions and programs, and efficient methods to collect data
longitudinally after graduation
6Professional
Assessment
Professional Assessment
u Define professional assessment – PROFESSIONAL EDUCATION
u Is only as good as the programmatic and institutional culture
u Collective commitment to each other
u The value
u Organization literature
u Understanding learning processes and value of assessment to improve
u Make the cultural shift for data collection and assessment
7Professional Assessment
u Shared Programmatic Assessment
u Definition
u Value
u Importance
u Culture creates
u Expectation and need
Shared
Programmatic
Assessment
8Shared Programmatic Assessment –
Value for the Profession
u Comparator professions overviews and examples
u AAMC Site www.aamc.org/data-reports
u AACP – Assessment and Accreditation Management System (AAMS)
u www.aacp.org/research/assessment-accreditation-management-system-aams
u What is possible for physical therapy given what other professions have
accomplished?
Where is Shared Programmatic
Assessment in Physical Therapy?
u The challenge is what to measure
§ Breakdown the idea that we need to have a perfect measure
§ Not everyone is the same
9Where is Shared Programmatic
Assessment in Physical Therapy?
§ Challenges/Successes
§ Limitations
§ Pair-share on how to shift the culture
§ uses
§ benefits
§ barriers
Solutions for Implementing Shared
Programmatic Assessment
u How does the profession create a measure of quality education?
u Provide a discussion on the possible educational metrics
u Provide a discussion on the amount and types of data needed for critical
assessment
10Data
u Presenters will use data to review outcomes at the programmatic,
comparative school, or collectively at the level of the profession.
u They will share educational trends across all institutions including
student debt, program cost, faculty trends and other key
administrative metrics.
Data Integration and Summary
u Examples of data integration
u Program and Institution
u Program and Profession
u Program, Institution, and Profession
11Creating a AUDIENCE ACTIVITY
Culture
Example: From a Developing Program
Blueprints Foundation Building
• Planning • Faculty Champion • Putting the plan to
action…not just for CAPTE,
• Master Assessment Plan (MAP) • Buy in from Leadership
but for continuous quality
• Details: who is responsible for Program Director improvement
collecting data, when, where to
get the data, when to report the Dean • Continuously reviewing the
data and to whom, the pre- plan and the timeline to
established thresholds to trigger refine the plan
change
• Plan and Timeline to Refine Plan
12Example: One University
Various Strategies
Resources like apartments are the same – but décor/utilization varies
u Culture of assessment is influenced by
u People
u Profession – expectations of the profession
The Building – Office of Institutional
Research & University Assessment Committee
u Mandated assessment services
u Alumni Survey
u Financial Aid Assessment
u Admissions data
u Research activity - grants & publication
u Student Outcomes
u Demographics – student & faculty
u Program completion data - retention
u Accreditation report/self-study
13The Apartments – individual programs
u Varied Utilization
u PT – representative attends CAPTE workshop and assists with data
collection & analysis
u PA – representative attends all faculty meetings, identifies assessment
opportunity & performs analysis
u Podiatry & Veterinary Medicine – representative administers, analyze &
report on administrator effectiveness/evaluation
u Osteopathic Medicine – representative collects overall curriculum &
individual course effectiveness
u Dental - Employment surveys
Questions for discussion/reflection
Many strategies and many resources available
u Can we agree to a culture of assessment and a similar/standardized
process?
u Can we create a data bank of assessment options that allow programs to
select the best option for them?
u What is “your” culture of assessment?
u How are you utilizing available resources?
u How are you fostering a culture of assessment?
14Cultural Summary
u The movement toward comprehensive programmatic assessment has
been around for over 25 years.
u Must Establish
u Value
u Importance
u Commitment from leadership is Key
u Programmatic leadership
u Campus Leadership
u Professional Leadership
Sustained Professional Assessment
u Striving for Academic Excellence
u Advancing the profession – put aside competition for the good of the profession
u Student Centered Education
u Educational advancement and sustained excellence
u can only be achieved through collaboration,
u collective data sharing,
u and commitment to wide-spread assessment practices.
15Sustained Professional Assessment
u A crucial factor to success
u Development of a culture of assessment that is based
u A sound theoretical framework
u Integration of multiple sources of data
u Current Professional Models
u Barriers to national Physical Therapy educational assessment
Future Directions
16The Future
u Use existing empirical data and theoretical frameworks as a basis for
defining excellence
u Reconfirm a commitment by all stakeholders to participate in
standardized professional assessment
u Collaborate with CAPTE to interface data elements and collection.
CLUSTER ATTRIBUTE Descriptions
Cluster 1 Diverse and Engaged Participants Learning and
Diverse and Engaged Faculty/Students leadership environments that
foster student/faculty growth
Engaged Leaders and development
Cluster 2 Participatory Cultures
Shared Program Direction
Program innovation through
Community of Learners
collaborative interaction
Risk Taking/Interactive Learning
Cluster 3 Interactive Teaching and learning
Mentoring/Cooperative Learning High impact practices that foster
Experiential Learning student learning
Cluster 4 Connected Program Requirements
Depth/Breadth Coursework Curricular design that promotes
Professional Residency professional development
Tangible Products
Support for Faculty
Cluster 5 Adequate Resources
Infrastructure
17The Future
u Require assessment participation as a professional expectation for being “a member in good
standing” in ACAPT.
u Designate appropriate resources in assessment activities for data collection, storage, analysis
technology, personnel, and communication.
u Educate constituents on VALUE and ways data can be useful for program benchmarking,
quality improvement, strategic planning and accreditation self-study.
18The Future
u The BenEx Taskforce envisions the development of a Research Center
u designed to support ACAPT initiatives,
u provide assistance with data collection and interpretation
u to guide ongoing growth and development of our educational enterprises.
u Profession needs to support assessment
u Elevate programs beyond level set by CAPTE
u Support programmatic need to substantiate improvement
Assessment is Necessary for Excellence
u Call to action from Jensen et al., 2018:
u ”Create a national data set that includes essential metrics of performance
outcomes, structures, and processes that can be used for meaningful research
to guide future evidence-based change”
u An excellent academic program demonstrates a culture of excellence by
continually and intentionally striving to transform learners, advance
knowledge, and improve societal health.
19ACAPT Criteria
for Excellence
20Questions?
References
u Busche G. The Appreciative Inquiry Model. In: Kessler EH, ed. Encyclopedia of Management Theory. Thousand Oaks, CA: Sage Publications; 2013.
u Grignon TP, Henley E, Lee KM, Abentroth MJ, Jette DU. Expected graduate outcomes in US physical therapist education programs: a qualitative study. J
Phys Ther Educ. 2014;28(1):48.
u Haworth JG, Conrad C. Emblems of quality in higher education: developing and sustaining high-quality programs. Needham, MA: Allyn and Bacon; 1997.
u Heath, A., Altenburger, P., Brechter, J.H., Chleboun, G., Jette, D.U., Pike, G., Schilling, D., Topp, K., Tschoepe, B., 2018. J Phys Ther Educ. 32:109-117.
u Higgs J, McMeeken J. Achieving Quality in Physiotherapy Programmes through Benchmarking. NZ J Physiother. 1997;25(3):19-22.
u Hobson R, Rolland S, Rotgans J, et al. Quality assurance, benchmarking, assessment and mutual international recognition of qualifications. Eur J Dent Educ 2008;12(s1):92-100.
u Jensen, G.M., Hack, L.M., Nordstrom, T., Gwyer, J., Mostrom, E. National study of excellence and innovation in physical therapist education: part 2 – a call to reform. Phys Ther. 2017.
97:875-888.
u Newswander LK, Borrego M. Engagement in two interdisciplinary graduate programs. Higher Education. 2009;58(4):551-562.
u Niemczyk, M., (2012) Using your Brain to Learn: Strategies for Success, Performance Optimization Solutions LLC. Arizona State University
u Schuwirth, L., van der Vleuten, C., Durning, S.J. What programmatic assessment in medical education can learn from healthcare. 2017. Perspect Med Educ. 6:211-215.
u Standards and Required Elements for Accreditation of Physical Therapist Education Programs, (2016).
u Suskie L. Assessing Student Learning: A common sense guide. San Francisco, CA: Jossey Bass; 2009.
u Wang Hm. What matters in graduate school? Exploring patterns of student engagement, academic and personal development. Association for Institutional Research; 2003; Tampa,
FL.
u Warden S. Testing the Engagement Theory of Program Quality in CACREP-Accredited Counselor Education Programs. Counselor Education and Supervision. 2012;51(2):127-140.
21You can also read
