Urgent CD-CAT Research During The COVID-19 Pandemic - Hua-Hua Chang Purdue University
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Urgent CD-CAT Research During The COVID-19 Pandemic Hua-Hua Chang Purdue University 12 November 2020
Purdue Univ. 2
Indiana COVID-19 Data Report (11-10-2020) At PURDUE: 3
Challenges and Solutions • With the COVID-19 pandemic, many schools have moved from in-class instructions to online instruction, which has created new challenges for both instructors and students. • It is likely that an increased number of courses will remain online or hybrid even after the pandemic ends. • Receiving help in large gateway courses in which students are mostly anonymous is already difficult and becomes more problematic when switching to online or hybrid learning environments. • CD-CAT-powered online assessments are expected to provide instructors with precise information about student achievement levels and problem areas so they can intervene in students’ learning with clarifying information at the optimal time. • Time-constrained CAT can be used to administer all kinds of classroom assessment as an alternative to paper-and-pencil based tests including final exams. 4
Challenges in Teaching and Learning Before & After the Pandemic • Example: STEM DFW rates (proportion of students earning a D, F, or Withdrawal) – Each year, most large universities in the US enroll thousands of students in gateway courses that are taught in gigantic lecture halls. Due to this “one-size-fits-all” approach, often without adequate teaching staff or resources (e.g., enough TAs or graders), the DFW rates are as high as 30%-50%. 5
• DFW rates are higher still for URM. For example, a math course, Algebra and Trigonometry, taught recently at a Midwestern university reached DFW of 66% for URM. • Many majors require these “gateway” courses so failing them could force a student to change majors or add time and expense to the degree. To teach a course with hundreds or even thousands of students, instructors face the daunting task of generating, administering, and evaluating endless assessments to meet their teaching goals. 6
What is CD? What is CAT? • CAT: Computerized Adaptive Testing, also called tailor-made test • CD: Cognitive Diagnosis -- a new trend in psychometrics – Provide examinees with more information than just a single score. – How? By considering the different attributes measured by the test. – An attribute is a “task, subtask, cognitive process, or skill” assessed by the test, such as addition or reading comprehension. 7
The Item-Attribute Relationship Which items measure which attributes is represented by the Q-matrix: i1 i2 i3 i4 A1 0 1 0 1 A2 1 0 0 1 A3 1 0 1 0 8
CAT can be utilized to get Cognitive Diagnostic Information: CD-CAT • A-matrix Attributes 1 2 3 4 5 01100 1 Examinee 00111 2 ik 10001 3 00001 4 • Q-matrix Attributes 1 2 3 4 5 01100 q jk 1 Item 2 00111 3 10001 4 00001 9
10
◼ The student will determine the meaning of words in a variety of written texts. Texas 3rd ◼ The student will identify supporting ideas in a variety of written texts. grade reading ◼ The student will summarize a variety of written texts. assessment ◼ The student will perceive relationships and recognize outcomes in a variety of written texts. 6 attributes ◼ The student will analyze information in a variety of written texts in order to make inferences and generalizations. ◼ The student will recognize points of view, propaganda, and/or statements of fact and opinion in a variety of written texts.
New development --- CD-CAT ---AI-powered tool for both teachers and students • Cognitive diagnostic modeling • Diagnostic report -- how many specific skills the student has mastered? • D-CAT selects the best items that provides the information • Teachers will be able to assign CAT quizzes and use feedback from the resulting cognitive diagnostic reports, which will provide remedial recommendations. In this way, the technology provides feedback both by alerting teachers to areas that require additional supports, thereby making big-class teaching more individualized. 12
Challenges Bring Opportunities --- Proposals for the 2021 NCME Conference 1. From CAT to Smart Learning – Urgent Research During the Pandemic 2. A Machine Learning Method for Classifying Student’s Learning Status 3. A Time Constrained CAT Design to Support Online Testing 4. Using Cognitive Diagnostic Analysis to Construct Learning Path 5. …. 13
Example: Learning Trajectory of China and USA Wu et al. (In press). A comparative study on cognitive diagnostic assessment of mathematical key competencies and learning trajectories—PISA data analysis based on 19454 students from 8 countries. 14
Zhu & Chang (2020) Person FIT by Machine Learning (1,1,1) normal response patterns (1,1,1) normal response pattern In Out (1,1,0) normal response patterns (1,1,0) normal response pattern …… …… In Out • (0,0,0) normal response patterns (0,0,0) normal response pattern • • • (1,1,1) aberrant response patterns • (1,1,1) aberrant response pattern • • (1,1,0) aberrant response patterns • • (1,1,0) aberrant response pattern …… In Out …… (0,0,0)aberrant response patterns (0,0,0)aberrant response pattern Input Layer Hidden Layer Output Layer
Testing companies in the US are laying off people. What is the future of assessment? FROM ADAPTIVE TESTING TO PERSONALIZED LEARNING 16
• The idea of personalized learning is not new. But before technology is ready it is impossible to provide 1-to-1 teaching on a large scale. • CAT can help! – Selecting items sequentially helps students CD-CAT helps better understand the concepts being taught – CAT provides more flexibility learning • Examples 1. Some schools in China are using CAT to help classroom teaching 2. At UIUC, CAT has been utilized to help low- performing students in an undergraduate physics course.
A Large-Scale CD-CAT with 2000 PC’s in Dalian, China In December 2011, 30,000 Grade 5 Students in Dalian China were taking a cognitive diagnostic CAT for their English proficiency assessment. 18
Utilizing CAT in Classroom Teaching, Students are learning “Area of a Circle” “圆的面积” 课例展示（北京市海淀区西颐小学六年级二班） 图片说明：1．集体学习系统中“圆的面积”的视频内容； 19
CAT Is Revolutionarily Changing the Way We Address Challenges in Learning █ Students really enjoy the new mode of testing, which makes learning more enjoyable comparing with regular teaching and P&P testing 20
Help Teachers Know Their Students Better. According to the diagnostic report, remedial planning is on the way █ The in-class CAT provides more information to teachers, which facilitates research and career development 图片说明： 三位实验教师在讨论学 习内容 【郑州市金水区纬一路小学】的老师在实验中，借助易学通系统对习题进行 钻研，通过对学生学习情况的不断分析总结，促进教师在反思中提高自身的 教学技能，在提高教学质量的同时，也使自身的专业素养得到提升。 21
Another Example in China (Dalian, China): Adaptive Testing Administered By Paper/pencil How to make P&P Test Adaptive ? z zzv
• Assessment in most on-line learning systems CD-CAT are not well developed ▪ “adaptive” or “tailor-made” ? supports ▪ Termination rule? all-the-time ▪ Diagnostic report? ▪ How to detect learning? and • Learners should be further encouraged and everywhere inspired • Also, how to help teachers in classrooms? learning
Adaptive Learning System ---Measurement model, learning model, recommendation strategy • Skill profile: = ( 1 , … , ( )). • Learning intervention (learning material): ( ). • Reward (learning outcome): . e.g. = σ =1 + 1 − ( ) • Total reward (overall learning outcome): σ −1 =0 ( )
Method Breakthrough of item selection method • Maximum Priority Index (MPI) Cheng and Chang, 2009 – can be considered as a variant of the maximum information method – non-statistical constraints were also accommodated • Various constraints such as content balancing, item exposure rate and etc – Choose the item that maximizes the priority index (PI): # of constraints Weight of kth constraint = ෑ cjk: 0 or 1. indicates if item j is relevant to the kth constraint =1 Information at the current ability scaled quota left of kth constraint estimate 25
Incorporating information from both θ & α • Dual Information (Cheng and Chang, Wang and Chang) proposed to maximize KL j (ˆm , ˆm ) = wKL j (ˆm ) + (1 − w) KL j (ˆm ) KL alpha Information KL theta information weight 26
COMBINING CD-CAT AND SMART LEARNING --TEACHER-BASED CLASSROOM (Zhang & Chang, 2015) 27
• Personalized learning routes – Knewton system • Blended learning Personalized – Khan Academy – Intelligent Tutoring Systems Education – – e-Schoolbags/student portfolios • Open Educational Resources and Massive Innovations Online Open Courseware and Attempts – Coursera – EdX • International expansions and personal device use
Knewton System --- Personalized Learning Paths 29
Other Commercial Platforms
Adaptive Learning Platform in Taiwan http://adaptive-learning.ntcu.edu.tw 31
32
Contents (2017 data) Domain Math Chinese Science Grades 1~9 1~9 1~6 Knowledge Nodes >1,100 >1,200 >800 Diagnostic Items > 8,000 >8,880 >4000 Interactive Tutoring >100 >20 >20 Dynamic Assessment >1000 >40 >40 33
Knowledge Structure for Curriculum Standard 34
Knowledge Structure for Subskills (subconcepts) Demo Video 35
Videos and Quizzes Learning Recommendation For the next steps Learning Recommendation For the prerequisite nodes 36
Learning Paths for Individuals Learning Path for Learning Path for Student B Student A Learning Path for Student C 37
Users in Taiwan (2017 data) >1000 > 200,000 Schools Students > 9,000 Teachers 38
Teacher Instruction and Group Discussion using 因材網
Thanks to Zhiliang Ying and his Colleagues at Columbia University for Organizing 7 international workshops on CD 40
Chang606@purdue.edu THANK YOU! 41
You can also read
