The only research-proven online game that assesses and teaches social and emotional skills
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
The only research-proven online game that assesses
and teaches social and emotional skills
Introduction
The Problem
Research demonstrates that positive social and emotional
skills and relationships in childhood are associated with
children’s positive behavioral, emotional, and academic well-
being. However, many children struggle socially throughout
their school years. And without early intervention, these
children can face a wide range of social, emotional, and 1 in 4 children experience
academic problems, including school dropout, drug abuse, some type of social problem.
depression, anxiety, physical aggression, and antisocial
behavior—and these problems can persist into adulthood.
• 10% of children report having
no friends in their class and no
Social and emotional learning helps children develop the one to play with at school.
skills they need to thrive socially, but conducting in-person
training presents a variety of challenges for clinicians, • 30% of children are involved
in bullying on a regular basis—
schools, and parents. Logistical barriers, particularly the
as a bully, victim, or both.
financial costs to families and schools, demanding resource
needs, and time and travel requirements for children to
participate, can deny many children the benefits of participating in a social and emotional learning
program.
Determining which children have social and emotional skills deficits can also be problematic.
Traditional assessment methods such as behavioral observation and behavior rating scales can be
costly and require extensive time and training to implement. In addition, they also pose challenges
such as unreliable or biased observers, lack of social comparison data, inability to capture a
student’s behavior in specific situations, potential for students to modify their behavior because
they’re being observed, and inappropriate recording techniques.
The Solution
With funding from the U.S. Department of Education, 3C Institute developed
Zoo U, an online game-based social and emotional skills assessment and
skill-building program for children aged 7–12 (www.zoougame.com). In Zoo
U, the only online game that assesses and teaches social and emotional skills,
children become students at a virtual school for future zookeepers and build
their social and emotional skills by working through a series of common
social scenarios.
Zoo U leverages powerful technology to eliminate the barriers of traditional social and emotional
skills assessment and training methods (Figure 1). For example:
• Administration of Zoo U requires minimal training.
• Subjective bias and recording errors are eliminated because the assessment system—
rather than observers—scores the child’s behaviors.
• Social comparison data can be collected efficiently from a large group of children.
1
• Situations that are important for assessment but unlikely to be observed because they
occur infrequently can be incorporated into the assessment.
• “Stealth assessment” techniques—in which assessments are embedded into a game and
students aren’t even aware they’re being assessed—greatly reduce the likelihood that
children will alter their behavior to please an observer.
Figure 1. Advantages of Zoo U over traditional social and emotional skills assessment methods
Behavior Behavioral
Zoo U
Rating Scale Observation
Minimal training required ü ü
Time-efficient ü ü
Produces data about less frequent behaviors ü ü
Objected report of behavior ü ü
Situational specificity ü ü
Reporter/recorder objectivity ü ü
Stealth observation ü ü
Assigns appropriate weight to behaviors ü
ü= applicable; ü= somewhat applicable
Zoo U offers an engaging educational experience for students, an affordable method for broadscale
everyday use by schools, and a seamless means of integrating data-driven decision making into
school-based social interventions.
And in addition to strong endorsements by both children and school professionals, research shows
that Zoo U is effective in improving children’s social and emotional skills.
about the product
The Child Experience: Assessment
In Zoo U, players are students in a school for future zookeepers, where Principal Wild and a host
of friendly animals help them learn essential social and emotional skills as they navigate common
school-based social scenarios. Social and emotional skills addressed in Zoo U include:
The child’s Zoo U experience begins by creating a character, or avatar (Figure 2), which promotes
engagement by helping students visualize themselves in the social scenarios the game presents.
2
Next, the child enters the assessment component and chooses one of the assessment scenes
(Figure 3). Each scene focuses on a specific social skill area.
Throughout the assessment, the child encounters six scenarios similar to those they might
experience at school. To navigate these scenes, the child makes dialogue, action, and behavior
choices (Figure 4).
Figure 2. Character Creator Figure 3. Scene Selection
Figure 4. Sample scene from Zoo U Assessment
The Student Experience: Skill Builder
Once the assessment is complete, the child enters the skill-building component of the game (Figure
5). Throughout the skill builder, the child completes 30 social scenes—five in each social skill area.
As with the assessment, the child makes dialogue, action, and behavior choices to navigate these
scenes, and the child can earn virtual coins for making good choices.
Following each scene, Principal Wild reviews the child’s choices, giving praise and reinforcement
for skills the child demonstrated well and constructive feedback on areas that the child hasn’t quite
mastered (Figure 6).
Zoo U also helps children practice and strengthen their newly acquired skills through mini-games
and other activities, such as deciphering a code with a classmate to improve cooperation skills
(Figure 7).
3
Figure 5. Samle skill-builder scene Figure 6. Principal Wild feedback sample
Figure 7. Interactive activity sample
The Educator Experience
Educators have access to an online dashboard (Figure 8) and downloadable graphic assessment and
skill builder reports (Figures 9 and 10) full of useful information about their students’ progress and
recommendations for specific resources based on the student’s strengths and weaknesses. Reports
can also be shared and saved.
Zoo U also includes a Resource Center with supplemental social and emotional learning resources
and activities.
Educators can create classes or small groups and upload student information for easy, organized
program administration. Reports can be generated for individual students or entire classes.
4
Figure 8. Educator Dashboard
Alex Herman
5
Figure 9. Graphic assessment report sample with interpretation guidelines
6Figure 10. Graphic skill builder report sample with interpretation guidelines
7RESEARCH BEHIND ZOO U
Zoo U is Effective in Assessing and Building Social and Emotional Skills
3C Institute researchers investigated the effectiveness of Zoo U with a group of children aged 7–11.
Parents completed questionnaires about their child’s social and emotional skills and behaviors both
prior to and after their child played Zoo U, and children also completed questionnaires before and
after playing the game.
Analyses revealed that children who played Zoo U …
• Showed significant improvements in controlling impulses, initiating
conversation, and managing emotions (see Table 1 in "Summary
Statistics Section)
• Showed less aggression in social interations (see Table 1)
• Reported feeling more confident about social interactions and more
accepted by peers (see Table 2)
• Made significant gains in social and emotional skills knowledge,
especially in the areas of communication, cooperation, and empathy
(see Table 3)
What children learned from Zoo U:
• “I learned how to use better communication and how to treat others better.” A
H
• “I learned to control my feelings and that I should be nice to people.”
• “I learned more about considering other people and how they would react to a situation.”
• “I learned that sometimes I need to take a moment to relax and think about my actions
S
before I jump right to the next thing.”
• “I learned how to deal with tough situations in real life.”
The Zoo U Assessment Produces Reliable Data
Researchers at 3C Institute conducted three field tests—including one study that spanned 27
classrooms and 12 states—to determine the validity of the scoring algorithms in the Zoo U
assessment component. These studies enabled the game developers to refine the algorithms
throughout the course of development and ultimately demonstrate that the assessment data
produced through Zoo U gameplay accurately reflect the player’s social skill level in each of Zoo U’s
six social skill areas.
To test Zoo U’s assessment validity, researchers compared Zoo U scores with teachers’ independent
external assessment of their students’ social and emotional skills using established rating scale
measures such as the Social Skills Improvement System.
8These field tests revealed strong correlations between the students’ Zoo U assessment scores and
the scores of the measures completed by the students’ teachers (see Tables 4 and 5 in “Summary
Statistics” section). In addition, a national field test supported how differential performance on Zoo
U (high, moderate, low) predicted teachers’ ratings of children’s social and emotional skills at school
(see Table 6) as well as teacher-rated real-world adjustment at school, including discipline problems
and academic performance (see Table 7).
Zoo U Also Helps Identify Potential Bullies
Study data also revealed that Zoo U is effective in
identifying potential bullies.
Students who consistently selected the most negative
dialogue option were significantly more likely than other
children to be identified as a bully: 53% of these children
were considered bullies by their teacher.
In addition, students who scored poorly in the
cooperation, empathy, and impulse control
assessment scenes were significantly more likely than
others to be identified as a bully by a teacher. In fact, 67%
percent of children with low scores in all three areas were
identified by their teacher as bullies.
And 78% of the students who fell into both of these
categories—consistently negative responses and low
scores in cooperation, empathy, and impulse control—
were identified as a bully by their teacher (Figure 11).
The ability to identify bullies or a predilection for bullying behavior is especially powerful when it
enables school staff to address the social and emotional skills deficits that lead to bullying before the
child actually exhibits this behavior. Social-emotiona2 skills training such as Zoo U’s skill-building
component helps children learn how to interact appropriately with others and develop the capacity
to resolve conflicts peacefully.
Zoo U Engages Students
To ensure usability with the target audience, 3C Institute researchers tested Zoo U’s usability with a
group of children aged 7–12. In this study, children liked the game and thought their friends would
also like it. Many of them were interested in playing Zoo U again or playing other games like Zoo U.
Feedback from this study (Figure 12) reinforced the positive findings from an earlier field test
conducted with the Zoo U prototype. (Also see Table 8 in "Summary Statistics" section.)
9Figure 12. Child ratings of Zoo U
87% thought the game was fun
80% wanted to play more games like Zoo U
83% thought their friends would like it
Children also shared feedback in their own words.
Comments included:
• “I liked the way you made learning social and emotional
skills fun. Observations during a
I also liked the way you made the animals play an active Zoo U field test revealed
part of Zoo U.” high levels of student
• “The thing I liked the most about Zoo U was that it engagement with 96%
was a fun game to play, but it helped me learn about on-task behavior.
my feelings and how to control them.” Students easily understood
• “I liked that it was like a play just for fun online game how to navigate the game
and that it also teaches kids important lessons.” with almost no instruction.
• “It was fun to play, and I could not wait to see what
was next.”
• “I loved the way you have an option of what you want to
say.”
School Professionals Found Zoo U Innovative and Effective
During the game development process, 3C Institute researchers tested Zoo U with elementary
school providers (teachers, counselors, and school administrators). Feedback about the game
was extremely positive; school staff thought the software would be easy for teachers to use in the
classroom, and the game was seen as a highly valuable tool for helping teachers assess students’
social skill levels and monitor student progress over time. They viewed Zoo U as highly innovative
and effective compared to current social and emotional skills programs (Figure 13). (Also see Table 9
in “Summary Statistics” section.)
Provider praise for Zoo U:
• “This product is particularly useful because, unlike most assessments, it collects data
directly from the student.”
A
H
• “Teachers need products like this that are easy to implement and will impact student
learning.”
• “I love this approach when working with students. It's very engaging and appealing to
S
students.”
• “This program would be an innovative way to collect baseline data and progress
monitoring data on a child’s social and emotional skills.”
10Figure 13. Provider ratings for Zoo U
100%
75%
% AGREED
50%
25%
NOTE: See Summary Statistics section for detailed study data.
summary
Zoo U not only helps children build critical social and emotional skills, it gives educators reliable,
affordable, easy-to-use tools for discovering children’s specific social strengths and weaknesses and
monitoring their progress throughout the intervention. Highly rated by school staff and children
alike and proven effective through multiple research studies, Zoo U offers a compelling, interactive
educational experience that enables children to safely explore different responses to social
scenarios and the consequences of their choices.
Future Directions
Research:
3C Institute plans to conduct another randomized control trial with a larger sample of children
in early 2015 to further explore the impact of Zoo U on children’s social and emotional skills and
behavior at home and at school.
Product Development:
Ongoing enhancements will be made to continually improve the gameplay and user interface
experience as well as expand the supplemental resources. Additionally, data collected in the
2015 study will be used to refine Zoo U’s scoring algorithms. The development team also plans
to integrate emotional responses into the intelligent engine to further personalize the user
experience. Additional enhancements will also be made based on user feedback once the product is
commercialized.
Commercialization:
Zoo U is currently available for purchase by schools at www.zoougame.com.
11ABOUT THE GAME DEVELOPERS
Zoo U was developed by a diverse team of researchers, content
experts, game developers, web programmers, and artists at 3C
Institute, an award-winning research and development company
(www.3cisd.com). 3C Institute’s mission is to bring research to practice
to improve health and well-being in children, adolescents, and adults.
3C Institute's areas of expertise include evidence-based programs for
social and emotional development, bullying prevention, and mental
health; autism spectrum disorder; adaptive online assessment and
game technology; and implementation science. The company is also developing several other social
and emotional learning games.
3C Institute has partners with Personalized Learning Games (www.personalizedlearninggames.com)
to market and distribute Zoo U and other innovative technologies.
The development of Zoo U was funded by the U.S. Department of Education through the following
grants:
• ED-IES-10-P-0114 ($99,992)
• ED-IES-11-C-0039 ($849,989)
PUBLICATIONS & PRESENTATIONS
DeRosier, M. E. (Ed.). (2014). Social skills assessment through games: The new best practice. Cary,
NC: Interchange Press.
DeRosier, M. E. (2012, September). Making effective social skill assessment more accessible, informative,
and effective for schools on a broad-scale through an interactive game-based platform
for students. Presentation at the Society for Research on Educational Effectiveness (SREE) Fall
Conference, Washington, DC.
DeRosier, M. E., Craig, A. B., & Sanchez, R. P. (2012). Zoo U: A stealth approach to social skills
assessment in schools. Advances in Human-Computer Interaction. doi:10.1155/2012/654791
Hehman, C. & Craig, A. (2012, June). Stealth assessment of social skills through interactive games.
Presentation at the 8th Annual Games for Health Conference, Boston, MA.
NOTE: 3C Institute researchers are in the process of publishing study results and are conducting
additional testing on Zoo U to further support its effectiveness.
12SUMMARY STATISTICS
13Data supporting the effectiveness of Zoo U (continued):
Table 1 (continued). Summary of Relevant Descriptive Information, ANCOVA Statistics, and
Effective Sizes for Children’s Social and Emotional Skills and Behaviors via Parents’ Reports on
the Social and Emotional Skills Behavior Inventory
Mean (SE) ANCOVA
Pre- Post- Change 2
F ɳ
Intervention Intervention
Social Behaviors
Internalizing
Treatment 3.24 (.07) 3.93 (.05) .69
5.851* .120
Wait-list Control 3.21 (.10) 3.27 (.05) .06
Externalizing
Treatment 2.28 (.11) 2.12 (.12) -.16
5.682* .117
Wait-list Control 2.58 (.15) 2.67 (.14) .09
Assertion
Treatment 3.52 (.24) 3.78 (.21) .26
4.260* .090
Wait-list Control 3.33 (.22) 3.17 (.22) -.16
ᵻ
Note. p < .10, *p < .05, **p < .001
Table 2. Summary of Relevant Descriptive Information, ANCOVA statistics, and Effect Sizes for
Child-Reported Self-Efficacy and Peer Acceptance.
NOTE: Two measures were used: (1) The Self-Efficacy and Outcome Expectancy Scale (SEOES; Ollendick &
Schmidt, 1987), was used to assess children’s social self-efficacy. This 20-item measure asks children to
rate on a 5-point Likert-type scale how sure they felt about engaging in social behaviors and how sure
they are that other children would respond in a specific way. (2) The Loneliness and Social Dissatisfaction
Scale (LSDS; Cassidy & Asher, 1992) is a 19-item measure of children’s self-reported feelings of loneliness
and dissatisfaction with peer relationships.
Mean (SE) ANCOVA
Pre- Post- Change
2
F ɳ
Intervention Intervention
Self-Efficacy
(based on a 5-point scale)
Treatment 3.922 (.11) 4.296 (.09) .374
4.176* .089
Wait-list Control 3.852 (.13) 3.989 (.13) .137
Peer Acceptance
(based on a 4-point scale)
Treatment 2.498 (.04) 3.415 (.13) .917
8.214** .160
Wait-list Control 3.296 (.11) 3.222 (.10) -.074
Note. *p < .05, **p < .001
Data supporting the effectiveness of Zoo U (continued):
Table 3. Summary of Relevant Descriptive Information, MANOVA
Statistics, and Effective Sizes for Child Performance on the Achieved
Learning Questionnaire (ALQ)
NOTE: Each unit of Zoo U is accompanied by a unit quiz, which, when
compiled, comprises the Achieved Learning Questionnaire (ALQ), a 36-item
measure of children’s social literacy across six social-emotional domains.
Children were asked to demonstrate their knowledge in each of these
domains by answering multiple choice and true/false questions.
Mean (SE) MANOVA
(% correct on ALQ) F ɳ
2
Impulse Control
Treatment .841 (.05)
2.003 .044
Wait-list Control .746 (.05)
Communication
Treatment .978 (.03)
17.513** .285
Wait-list Control .819 (.03)
Cooperation
Treatment .964 (.02)
5.569* .112
Wait-list Control .884 (.02)
Social Initiation
Treatment .949 (.03)
1.925 .042
Wait-list Control .899 (.03)
Empathy
Treatment .920 (.03)
35.232** .445
Wait-list Control .681 (.03)
Emotion Regulation
Treatment .725 (.03)
4.445* .092
Wait-list Control .826 (.03)
Overall
Treatment .896 (.02)
9.131** .172
Wait-list Control .809 (.02)
Note. *p < .05, **p < .001
Data supporting the reliability of the Zoo U assessment:
To establish the scoring algorithms for Zoo U’s six social emotional skills, 3C Institute researchers
rd th
conducted a field test with 50 3 and 4 grade students in one central North Carolina elementary
school. Students completed the six scenes of the Zoo U assessment while research staff supervised
and observed. The students’ teachers independently completed online ratings of the social and
behavioral adjustment of the students in their classroom using the Social Skills Improvement System
(SSIS), a widely used behavioral rating scale with considerable evidence supporting its reliability and
validity. 3C researchers computed correlations between the Zoo U and SSIS subscale scores. Results
are displayed in Table 4:
Table 4. Correlations between SSIS and Zoo U Scores.
Zoo U Social Skill Areas
SSIS Subscales Impulse Emotion Communi- Social
Empathy Cooperation
Control Regulation cation Initiation
Attentive .50 .29 .08 .25 .41 .34
Self-Control .42 .45 .27 .32 .36 .45
Communication .21 .22 .39 .36 .35 .36
Empathy .19 .16 .23 .45 .39 .32
Engagement .12 .23 .49 .37 .31 .39
Cooperation .33 .28 .23 .40 .47 .40
Note. All correlations ≥ .20 were significant at the p3C Institute researchers conducted a third field test to evaluate Zoo U with a broader selection of
rd th
schools and teachers. The participating sample included 289 3 and 4 graders in 27 classrooms
across 12 states. Teachers independently completed the SSBI for each student in their class, rating
children for each of the SSBI social skill subscales as well as subscales assessing children’s adjustment
for school-related outcomes. Researchers investigated whether differential performance on Zoo U
predicted teachers’ ratings of children’s social skills at school (Table 6) and teacher-rated real-world
adjustment at school (Table 7). Results are displayed in Tables 6 and 7:
Table 6. Means and Mean Comparisons for Teacher-rated SSBI Social and Emotional Skills by Zoo U Categories.
Zoo U Category
Social Skill Area High Moderate Low
A B C
Impulse Control .38 -.04 -.21
A A B
Emotion Regulation .24 -.01 -.38
A AB B
Communication .17 .00 -.19
A B B
Empathy .34 -.12 -.17
A A B
Social Initiation .20 -.04 -.42
A AB B
Cooperation .20 .05 -.22
Note. Means within a row with different superscript letters are
significantly different from one another.
Table 7. Significant Prediction of School Outcomes by Grade Level.
SSBI School Impulse Emotion Communi- Social
Empathy Cooperation
Outcome Subscale Control Regulation cation Initiation
Internalizing th th th
4 4 4
behavior problems
Externalizing rd th th rd th rd th
3 &4 4 3 &4 3 &4
behavior problems
Disruptive behavior th th th
4 4 4
problems
th rd th th rd
Discipline actions 4 3 &4 4 3
Academic rd th th rd th rd th
3 &4 4 3 &4 3 &4
performance
Table 8 (Data for Figure 10). 3C Institute researchers tested Zoo U’s usability with 29 children aged 7–
12, who provided feedback through a brief evaluation survey. Table 8 shows child ratings of Zoo U
using a 5-point scale from 1=Strongly disagree to 5=Strongly agree.
% Agreed or
Area rated Mean Strongly
Agreed
Liked the game 4.55 87%
Thought the game was fun 4.52 87%
Wanted to play more games like Zoo U 4.34 80%
Thought their friends would like it 4.17 83%
Wanted to play Zoo U again 4.21 76%
Table 9 (Data for Figure 11). 3C Institute researchers tested Zoo U with 60 elementary school
providers (teachers, counselors, and school administrators), who provided feedback through a brief
evaluation survey. Table 9 shows school staff ratings of Zoo U using a 5-point scale from 1=Strongly
disagree to 5=Strongly agree.
Mean % Agreed or
Area Rated
(SD) Strongly Agreed
Engaging for students 4.60 (.69) 95%
Easy to use in the classroom 4.43 (.67) 90%
Easy to use with individual students 4.63 (.52) 98%
Feasible for use in schools 4.55 (.62) 94%
A valuable tool for teachers 4.52 (.70) 92%
A valuable assessment tool 4.55 (.59) 95%
Useful for monitoring progress 4.50 (.62) 94%
High overall quality 4.45 (.75) 95%
More effective than current methods 4.48 (.70) 88%
You can also read
