Omnibus Learning Gain Study: Towards Inclusive Learning Gain Metrics - Inside Government
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Omnibus Learning Gain Study: Towards Inclusive Learning Gain Metrics ‘…an attempt to measure the improvement in knowledge, skills, work-readiness and personal development made by students during their time spent in higher education.’ (HEFCE, 2015) Dr Fuad Ali Continuum Centre for Widening Participation Policy Studies @FuadContinuum #learninggain
Inclusive Learning Gain
• Connecting WP & Learning Analytics
• Data as Assemblage (DeLanda, Dixon-Román)
• Value-laden & affecting
• Relationality, socio-political forces of difference &
inequality
• Power, performance data & performativity
• Survey Scales
• Academic Behaviour Confidence*
• Need for Cognition*
• United Kingdom Engagement Survey (partial)*
• Predict your Grade
• Multiple Motivation* & Creatical Employability
• Institutional Engagement and Study Outputs
• Stakeholder-centric analytics
• Institutional momentum & fluency
• Efficient, flexible survey instrument
• Evidence-base for policy and practice
Emerging Learning Gain Ecology of Interest
To restructure
To market
To rank
To support To inform
students teaching
Student perspectives
“There are quite a lot of similar ones here and I think it's the same thing, saying it four or five
times, basically.”
“I found [question 1] a bit crazy, like we'd say it this way because nobody likes complexity. So,
if somebody prefers complex to simple problems, it's a bit weird. Not necessarily negative,
Need for but it's weird to say that … in the educational system as .. a student, we're trying to find
shortcuts rather than complexities. So, I would not agree with that.”
Cognition “It's quite a struggle…because it is focusing on the negative, it's actually making it more
difficult because you're really having to question like, oh, is that something that I don't like to
do?”
(tries to) measures an
individual’s tendency to “...the term, thinking, is very vague. Not to like bludgeon the vagueness of it all but thinking,
what does that mean? Thinking, like imagining, like meditating, like critically thinking,
analysing, what does that mean? It's very vague. They use it a lot”
engage in and enjoy
effortful cognitive “The notion of thinking abstractly is appealing to me. Yeah, I said minus because I like to have
clear ideas. I don't like this word abstract. Anything that is confusing is not - it doesn't make
endeavours. me comfortable and it can be very annoying if I don't understand something or if I don't have
the support, help, to be able to find a solution or to be able to move forward, I can be very
frustrated.”
Cacioppo, J. T., Petty, R. E. & Kao, C. F. (1984) The efficient assessment of need for cognition. Journal of Personality Assessment 48, pp. 306–307
Student perspectives
“…in the light of my situation I feel like it missed, I don't
know, personally growth I guess. But that doesn't
really fall under the umbrella of academic behaviour.
Academic Or at least strictly it doesn't.”
Behaviour “How can you honestly pick 24 questions that are going
to suit everyone and actually be something that
Confidence everyone is able to answer?”
Psychometric means of assessing “These ones I relate to like I think what makes a good
the confidence that graduate is to attend everything, to go to lectures, to
undergraduates have in their own ask the lecturers to be on time because you need to
anticipated study behaviours in succeed. If you don't follow these then you'll have - you
relation to their degree will struggle.”
programme.
Sander, P. & Sanders, L. (2009) Measuring academic behaviour confidence: the ABC scale revisited, Studies in Higher Education 34:1, pp. 19-35.
• Ready reckoner • 4 or 6 sub- scales • Poll of Student anxieties in the room • Programme level approach • Pre-entry potential • ‘Some’ correlation with attendance monitoring
Student reflections on Motivation
“I went to university because it was just the next step really. I don't know. “
“So, for me, I just thought it was a natural thing for me to do, based on my family background.”
“I was never going to come. I never had it on - I applied, purely because my school - we're the type of school that
were like, you've got to apply to uni. It was that sort of thing. We never really got told about apprenticeships and
stuff like that. It was always uni, uni, uni - get your grades and go.”
“We just have the role to make children, cook and clean basically. There's no education… So yeah, I want to do
something, get a job and better myself.”
“My first experience of university was horrible. So, I dropped out and then I didn't study for a good few years. But
then my mum began - was diagnosed with …, so that made me experience counselling and …made me realise I was
interested in psychology…. Also, I want to graduate for my mum, so it's something she can look forward to in the
future and remember and look back and boast to her friends.”
“I was at that point in my life where I felt that I needed more than the job that I loved. I used to work in a school as a
teaching assistant but I was very often given more responsibility than my job description. I loved it but I felt that it
was not right and I wanted to say more, but because I did not have the qualification, my voice was not heard. …you
can't have your voice heard unless you are well equipped academically. You can't share what you know unless
people know your background, know on which ground you're standing, your knowledge, the information you
provide. You can't convince the people unless you have that information inside you.”
Discussion prompted by: Vallerand et al (1992) The Academic Motivation Scale: A measure of Intrinsic, Extrinsic and Amotivation in Education. Education and Psychological Measurement 52(4)
Multiple Motivation Scale 1. To experience pleasure and satisfaction whilst learning new things 2. There is no alternative 3. To improve my career and employment prospects 4. Successfully completing a degree is important for my self-esteem 5. To make a greater contribution to society 6. To support my family 7. To realise family's ambition 8. To get to know people and to expand social horizons
HEI nWave 1 nWave 2 nBoth
Participation Problems UEL 939 531 361
Roehampton 381 123 72
Brunel 549 255 186
• ~ 3000 students surveyed overall
Total 1869 909 619
• Wave 2 drop off across institutions
• Implication for data
• Only 1/3 of data ‘complete’
• High student churn across sector
• Physically unengaged excluded
• Attendance problem compounded
• Nboth data overestimates scoresThe Student Calibration Conundrum
HEI nWave 1 nWave 2 nOverlap ABCWave 1 ABCWave 2 ABCOverlap NfCWave 1 NfCWave 2 NfCOverlap
(mean) (mean) (mean) (mean) (mean) (mean)
UEL 939 531 361 5.31 5.06 5.36 -> 5.13 9.90 8.39 10.29 -> 8.73
Roehampton 381 123 72 5.13 4.89 5.37 -> 5.05 10.09 7.67 13.44 -> 10.22
Brunel 549 255 186 5.20 4.98 5.30 -> 5.03 N/A 10.82 NA -> 10.11“There are times when I would feel like I'm
struggling when I feel like I work full time and I've
got other stuff going on as well. So, I feel like I want
to just drop it. Sometimes, my mum doesn't make it
easy because she's *** so she's very stubborn and
hard to deal with sometimes. It makes me feel why
am I doing this for her? I don't want to do it
anymore. Then I have to calm my emotions down
and then keep going and not give up sometimes.”
“Are you working more than 20 or 30 hours?”
“I work 37 hours on nights.”
“While studying?”
“Part-time. I had to change to part-time. I couldn't
do full-time.”
Acknowledging
structural inequalityViscosity of Engagement • UKES time spent scale
Paid Work and Student Age : Multilevel breakdown
Data Dialogues • Shiny app development • Learning Gain in histograms • Deepening gatekeeper engagement • Micro-lectures • College level T&L Strategies • Interest & Vulnerability
Temporal Considerations
• Academic Cycle
• Time to Measure =/= Time to Act
• Survey Administration
• Student completion
• Distribution & Collection
• Study Explanation
• Learning Gain Duration
• Pre-Entry Exploitation
• PG and FE
• State of perpetual institutional becoming
• Ontological HE transformation
i.e. National Education ServiceRecommendations 1) Data is not neutral, but vibrant and interactive, be alive to this. 2) LG as an opportunity for complementary contextualisation 3) Connect with varying tastes of partners in teaching and learning 4) Establish core and optional metrics to support creative programme level agency in survey-item authorship 5) Integrate with Moodle to save teaching time. 6) Enable student & cohort level analytics, but consider fragility 7) Develop a cooperative, accountable and non-proprietary approach
Further Work • Updates: On Continuum blog www.wideningparticipation.wordpress.com • January: Student focus groups to fine tune Wave 4 instrument • February: Wave 4 data collection begins • Attainment data, Ethnicity & Confidence • Cross institutional engagement with partners • Selection of LG items for enrolment task • Publications: Targeting Widening Participation & Critical Learning Analytics Space • Dissemination: Special Session on Learning Gain @ Forum for Access & Continuing Education 2018
You can also read
