IMPACT AND REF 2021 New challenges for universities - Dr Molly Morgan Jones and Dr Catriona Manville - My.Anglia Homepage
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
IMPACT AND REF 2021… New challenges for universities Dr Molly Morgan Jones and Dr Catriona Manville
Session outline
• Introductions
• Why measuring impact is difficult but not
impossible
• REF and Impact – what did we learn last time?
• The path to REF 2020: Strengths, weaknesses
and opportunities for ARU’s case studies
• Coffee
• Institutional case studies – breakout discussions
• Reflections and Next steps
2Objectives
• Review the purpose of the case studies in the wider
context and learning from REF 2014
• Identify and understand the strengths, weaknesses
and opportunities for ARU case studies
• Establish a forward plan for strengthening the case
studies and gathering evidence with a particular
focus on institutional case studies
• Your objectives?
3INTRODUCING RAND EUROPE
4RAND Europe
• Independent not-for-profit public policy research institute
– “help improve policy and decision making through research and
analysis”
• Not a university and not a management consultancy —but with the
capabilities of both
• Part of the global RAND Corporation
• Work across the breadth and depth of government
• Strongly held values of quality and objectivity
• Provider of evidence
5Our experience of research impact
6CHALLENGES IN MEASURING
IMPACT
7There are many reasons to evaluate impact...
• Advocacy
– ‘Make the case’ for research funding
• Accountability
– To taxpayer, employer, donors etc
• Analysis
– What works in designing or disseminating research?
• Allocation
– What to fund (institution, field, people …)
8Impact is an important component of REF
• One 4* case study is (roughly) equivalent to thirteen 4*
publications/outputs
• The back of the envelope…
– 4 outputs/researcher vs. 1 case study/10 researchers
– 10 FTEs = 40 outputs
• The ratio of weightings is…
– 65%/20% = 1/3
• Therefore…
– 1/3 weighting of impact x 40 outputs = 13 outputs/case study
9Measuring and identifying impact is not easy
10The challenge of attribution
11Dealing with serendipity
12Addressing time lags
13Will users engage?
14Examining the margin
15Measuring impact is difficult, but not impossible
16REF AND IMPACT – WHAT DID
WE LEARN LAST TIME?
17Plurality of impacts
• 'For the purposes of the REF, impact is defined as an effect on,
change or benefit to the economy, society, culture, public policy or
services, health, the environment or quality of life, beyond academia'
(HEFCE et al 2012a).
18The case study’s ‘job’ is to establish the impact
pathway
19Why the contribution of the research is important
20The critical role of evidence
• Significance must be demonstrated through nature of benefits
• Reach must be demonstrated by the scale of impact
• Evidence will vary for different types of impact
– Quantitate and qualitative
– Value of fact vs evidence of opinion
• Breadth versus depth of case study
• Importance of the chain of evidence
21What counts as evidence?
• All panels give examples of types of evidence
• The case study database is a great source of relevant examples
• International examples:
– Canadian Academy of Health Sciences (CAHS) Payback Framework
• Five categories of impact with indicators
– Australian Research Quality Framework provides list of indicators
22RQF indicators
• Reduced pollution • Creative works commissioned
• Regeneration or arrested degradation of natural • Change in procedures, behaviours, outlook etc
resources • New policies, guidelines, legislation etc
• Lives saved • Citations of research in legal judgements which
• Reduced infection rates become case law.
• Reduced treatment time and cost • Contracts and industry funding
• Increased literacy and numeracy rates • Repeat business
• Positive reviews of creative publications and • Number of presentations involving contact with
performances end-users
• Increased cultural awareness • Community awareness of research
• Royalties • Non-academic publications & performances
• Increased employment • Collaborative projects with end-users
• Reduced costs and resource usage • Citations in Government reports, Hansard, etc
• Increased competitiveness of Australian • Provision of expert advice and submissions to
industry enquiries etc
• Spin off companies • Invitations to be a visiting researcher or
• New products & inventions researcher in residence at an end-user
• Licences institution (if based on research merit)
• Invitations to be on reference, advisory and/or
steering committees (if based on research
merit)
23REF guidance on indicators/evidence
– Measures of improved clinical – Citation in a public discussion or
outcomes consultation
– Documented changes to public health – Measures of improved inclusion, welfare or
guidelines equality
– Evidence of adoption of best practice – Visitor or audience numbers, or number of
– Evidence of use of process or participants
technology – Quantitative data relating, for example, to
– Documented evidence of influence on effectiveness or organisational
policy process performance
– Economic impacts cost-effectiveness – Satisfaction measures
measures – Incorporation in training or CPD material
– Priority shifts in expenditure profiles or – Publication and sales figures, tourism data
quantifiable reallocation of budgets – Critiques or citations in users’ documents,
– Documented evidence of policy debate e.g. in the media, in teaching materials, for
– Measures of improved international public or commercial bodies
welfare or inclusion – Number and profile of people engaged,
– Evidence of sustained engagement types of audience
with a group – Formal partnership agreements with major
– Evidence from clinical trials institutions, NGOs and public bodies
– Traceable references to inclusion of – Independent testimony
research in industry standards or 24
authoritative guidanceCONSIDERATIONS WHEN
DEVELOPING CASE STUDIES
25Overall observations from your case studies
• There is a need for better understanding of what impact stands for
• Understanding eligibility and key parameters for the REF is crucial for
building the impact case study
• There is no one-size-fits-all approach – either the case studies or
assessment
• Collecting the evidence needs to be seen as a planned, systematic,
and recurring activity
• Contributing to / influencing policy, public debate needs some
forethought (and luck!)
• Avenues for dissemination and resources within ARU
• Mapping within and between units across ARU
• Going forwards the writing of the case study will be important
26Understanding of what impact stands for
• Impacts listed must be outside academia
– Note this is different from establishing the impact pathway
– Impact needs to have been realised, not potential
– Impact demonstrated needs to be within the impact window
• Dissemination at academic conferences can be an aspect of the
case study but not impact itself
• Reach and significance are two sides of the ‘impact' coin
– Both are equally important
– Don’t forget about contribution!
27Understanding eligibility and key parameters for
the next REF
• Uncertainty around revision of existing impact: timelines for REF
2021and new contribution since 2014 for resubmission case studies.
• Impact stays with the institution at which the research was
conducted and does not travel
– ARU’s unique contribution in case of multi-institutional effort
• Stern review and its implications
28Stern review recommendations about impact
• Recommendation 5: Institutions should be given more flexibility to
showcase their interdisciplinary and collaborative impacts by
submitting ‘institutional’ level impact case studies, part of a new
institutional level assessment.
• Recommendation 6: Impact should be based on research of
demonstrable quality. However, case studies could be linked to a
research activity and a body of work as well as to a broad range of
research outputs.
• Recommendation 7: Guidance on the REF should make it clear
that impact case studies should not be narrowly interpreted, need
not solely focus on socio-economic impacts but should also include
impact on government policy, on public engagement and
understanding, on cultural life, on academic impacts outside the
field, and impacts on teaching.
29No one-size-fits-all approach
• This applies to putting the case study together and also to its
assessment
• Important to recognise that the panel assessing the case study
would take a subject/field-specific approach
• The panel's approach to assessment would be based on the
strengths and weaknesses of the subject/field
30Collecting the evidence
• Evidence needed to demonstrate reach and significance
• Reach achievable through (amongst other methods) public talks,
writing in the trade press, the conversation, twitter/social media
communication, download data, number of users, etc.
• Demonstrating significance requires evidence of change over a
length of time. This change can be at an individual / collective level
• Significance can also be about preserving cultural heritage / or a
unique contribution to the field or discourse
• Planning and persistence needed to collect evidence (e.g. feedback
forms, twitter hashtags, emails, feedback surveys, testimonials,
written responses, book reviews, Amazon reviews, online
comments/feedback)
31Contribution / influence on policy, public debate
• Has the research fed directly / indirectly into policy debate or related
outcomes?
• Testimonials from the stakeholders (written, email, audio, or video) -
government, NGOs, consumer bodies, or industry consortia
• Has the research been referenced in consultations, select
committee hearings, and/or parliamentary debate?
• Look out for any citations in guidelines, reference documents, or
policy papers published by the government
32Avenues for dissemination and resources within
ARU
• Consider using MSc/PhD students for communicating through social
media and collecting evidence such as participant/audience
feedback
• Engage with stakeholders
• Create forums for people to engage with research
– Using 'The Conversation' for reaching a non-academic audience,
tracking feedback and debate
– Dissemination through relevant charities and public professional bodies
• Consider availability of centralised resources
– University press officer to publicise talks, book and article publications
– Technology transfer team to support commericalisation and discussions
with industry
– Shared resources through Impact officers who provide practical and
individual support across multiple UOAs 33Mapping within and between units across ARU
• Different ways of clustering research and impact to frame a case
study
– Maturity of research
– Chronological
– Type of impact
– Cross-cutting
– Type of stakeholder group
– Grand challenges
• Use of less developed or smaller scale impact alongside other
elements of a case study
34Going forwards the writing of the case study will be
important
• Title
• Summary of impact
• Underpinning research
• References to research
• Impact section
• Corroborating sources
35Title of the case study
• The first thing that reviewers will read
• A simple way to clarify and strengthen case studies
• Should include a clear statement of the impact
• Should express the focus of the case study
• Titles should be impact rather than research-led
• Also attention grabbing, where possible
• Do not make them too long
36Examples of message driven titles from REF 2014
submissions
• Improving health access and equity in India through health financing
reform
• Changing the way history of the Cold War is taught in schools across
the United States and Europe
• A framework for establishing how to increase global food production at
least cost to biodiversity
• Innovative chemistry reduces the environmental impact of mining and
pharmaceutical manufacture
• The use of citizen science in recording wildlife: ladybirds and invasive
alien species
• Attraction explained: The science of how we form relationships
37Examples from your case studies
• Too long
– “The prevalence of vision loss due to various causes had been occasionally
analysed by the WHO in the past, however the outputs were limited in there was
minimal attention to age-specificity and no gender-specificity to the results.”
– Despite occasionally analysing the prevalence of vision loss, WHO's past
studies have paid minimal attention to age and gender-specific causes.
• Too short
– Labour History Research Unit
– A new understanding of the civil wars in the Labour party and their
influence on the livelihoods of the British working class
38Summary of the impact
• From the template: “This section should briefly state what specific impact is
being described in the case study”
• Summary of key points, like an abstract
• Communicate three things:
– What the problem is/was (1 sentence)
– What the research was (1-2 sentences)
– What impact you had (2-3 sentences)
• Bullet points or numbered lists could aid clarity and help keep within word
limit
• Use figures where possible
39Underpinning research
• Needs to state what was unique about the particular research which
contributed to the impacts claimed
– Distinct contribution c.f. research elsewhere
• Research should be clearly defined and bounded
• Focus on the impact, not career of particular researchers
40References to the research
• From template: “This section should provide references to key outputs from
the research described in the previous section, and evidence about the
quality of the research”
• Include: Author(s); Title; Year of publication; reference details (e.g. Journal
title and issue); DOI or URL if required
• Indicative maximum of 6 references
• Need to provide evidence of quality, e.g.:
– rigorous peer-review process (e.g. Journal impact factors)
– end of grant reports referencing a high quality grading
– favourable reviews from authoritative sources
– prizes or awards made to individual research outputs cited
– evidence that an output is a reference point for further research beyond
the original institution
– rating in output submission
41Details of the impact
• Core of the case study
• Needs to present a clear narrative outlining how the impact came about,
and how it is linked to the research
• From the template: “This section should provide a narrative, with supporting
evidence, to explain:
– how the research underpinned (made a distinct and material
contribution to) the impact
– the nature and extent of the impact”
42General approach to impact section
• Overview of the impact at the start of the section
• The narrative should outline the key steps which took place from the
research being conducted to impact being realised
– NB: This may not be linear!
• Focus in this section on the impact
– Specific references and information on reach and significance
– Quantitative information can be useful here
• Always think one step further, beyond the initial impact
• Do not restate academic research, impacts
• Do not use the words ‘will’, ‘could’, ‘may’, etc…
43Sources to corroborate the impact
• From the template:
– “This section should list sources external to the submitting HEI that
could, if audited, provide corroboration of specific claims made in the
case study... the information in this section will be used for audit
purposes only”
• Should provide evidence for each specific impact claimed
– What are the crucial steps in your narrative?
• Link evidence to impacts claimed in Section 4
– Similar structure for each section
– Use a numbering system
4445
Breakout session: Introduction to institutional case
studies
• Stern Recommendation 5: Institutions should be given more
flexibility to showcase their interdisciplinary and collaborative
impacts by submitting ‘institutional’ level impact case studies, part of
a new institutional level assessment.
46REF 2014 case studies were interdisciplinary in
nature
47Breakout sessions
• The purpose of the breakout sessions will be to:
– Identify areas of overlap between UOAs which could be developed into
institutional case studies
– Start developing plans to create and track impact
• A discussion in three parts:
– Part 1: Reflect on learning and identify areas of synergy across UOAs
(20 mins)
• (By doing this exercise in groups we hope to encourage and enable areas of
overlap and synergy to come out)
– Part 2: Developing and mapping your impact strategy (20 mins)
• (Intention is to encourage people to think about cross-cutting case studies,
but generally this mapping tool could be used at an individual case study
level and overall we’d like to get people comfortable with using it so you can
go away from the day with a planning tool to hand).
– Part 3: Reporting back to the wider group (20 mins)
48How to think about case studies
• Different ways to frame the case study
– Chronological
– Type of impact
– Cross-cutting
– Type of stakeholder group
– Grand challenges
• Ideas for institutional case studies
– Medical
– Teaching/schools/social work
– Veterans/military
49Part 1: Reflect on learning and identify areas of
synergy across UOAs (20 mins)
50UOA W UOA X UOA Y UOA Z
Grand Challenges: what are the ‘big’ social, economic, political, health,
What are the
research strengths
cultural, or other issues which cut across these UOAs?
of my UOA?
What are the impact
strengths of my
UOA?
What are the
research and impact
opportunities
looking forward?
Who are our main
external
stakeholders?
What other UOAs
could our UOA work
with?
Summary of overlapping areas of impacts across UOAs
Realised impacts:
Potential impacts:
51Part 2: Developing and mapping your impact
strategy (20 mins)
52Area of exploration Notes
Grand challenge This will allow researchers to identify their ‘hook’ for the case
- What’s the ‘vision’ for the impact case study? What broader study early on.
social challenges is it addressing?
What impacts will be highlighted? This will encourage researchers to think about how they will
- What are the 3-4 main areas of impacts which will be group various impacts under different headings. There can be
demonstrated? (e.g. improved social well being, changes in multiple kinds of impacts, but it is important to carve out a clear
policy, changes in practice, improved health outcomes, economic narrative.
growth, etc.)
What outputs will lead to the impacts? This will enable people to plan what research outputs might be
- 2*+ research publications needed to support the case study, but also what additional, more
- Research networks and collaborations impact-focussed outputs might be required.
- Publications targeted at stakeholders
What dissemination efforts will be needed and what audiences By separating out dissemination as a unique step, this allows for
will you target? the distinction between dissemination and impact to be
- Dissemination outputs (media, blogs, etc.) reinforced, but will also encourage people to think about what
- Audiences (local councillors, professional bodies, general dissemination activities are needed to create impact.
public, policymakers)
What activities do we need to proceed with in order to produce This step will allow people to plan different activities.
our outputs/outcomes?
- Proposed new research
- Proposed new stakeholder engagements
- Proposed new dissemination activities
- Proposed new evaluations
What resources are already available and what will be needed? This step will allow people to think about what resources are
- Existing research( mark with a *) required for each activity.
- Investment required (time, money, people, space, project)
- Existing networks/stakeholder relationships
53Logic model worksheet
54Reflections and next steps
55UOA Specific sheets to be printed for
task 1
56Grand Challenges: what are the ‘big’ social, economic, political, health, cultural, or other
UOA 1 UOA 3 UOA 5 UOA 6
What are the
research strengths of
my UOA?
What are the impact
strengths of my
issues which cut across these UOAs?
UOA?
What are the
research and impact
opportunities looking
forward?
Who are our main
external
stakeholders?
What other UOAs
could our UOA work
with?
Summary of overlapping areas of impacts across UOAs
Realised impacts:
Potential impacts:
57Grand Challenges: what are the ‘big’ social, economic, political, health, cultural, or other
UOA 11 UOA 15 UOA 16 UOA 17 UOA 26
What are the
research
strengths of my
UOA?
What are the
impact strengths
issues which cut across these UOAs?
of my UOA?
What are the
research and
impact
opportunities
looking forward?
Who are our main
external
stakeholders?
What other UOAs
could our UOA
work with?
Summary of overlapping areas of impacts across UOAs
Realised impacts:
Potential impacts:
58Grand Challenges: what are the ‘big’ social, economic, political, health, cultural, or other
UOA 19 UOA 20 UOA 22 UOA 23 UOA 25
What are the
research
strengths of my
UOA?
What are the
impact strengths
issues which cut across these UOAs?
of my UOA?
What are the
research and
impact
opportunities
looking forward?
Who are our main
external
stakeholders?
What other UOAs
could our UOA
work with?
Summary of overlapping areas of impacts across UOAs
Realised impacts:
Potential impacts:
59Grand Challenges: what are the ‘big’ social, economic, political, health, cultural, or other
UOA 28 UOA 29 UOA 30 UOA 34 UOA 35 UOA 36
What are the
research
strengths of
my UOA?
What are the
impact
issues which cut across these UOAs?
strengths of
my UOA?
What are the
research and
impact
opportunities
looking
forward?
Who are our
main external
stakeholders?
What other
UOAs could
our UOA work
with?
Summary of overlapping areas of impacts across UOAs
Realised impacts:
Potential impacts:
60You can also read
