IMPACT AND REF 2021 New challenges for universities - Dr Molly Morgan Jones and Dr Catriona Manville - My.Anglia Homepage

 
IMPACT AND REF 2021…
New challenges for universities

Dr Molly Morgan Jones and Dr Catriona Manville
Session outline
  • Introductions
  • Why measuring impact is difficult but not
    impossible
  • REF and Impact – what did we learn last time?
  • The path to REF 2020: Strengths, weaknesses
    and opportunities for ARU’s case studies
  • Coffee
  • Institutional case studies – breakout discussions
  • Reflections and Next steps

                                                        2
Objectives

• Review the purpose of the case studies in the wider
  context and learning from REF 2014
• Identify and understand the strengths, weaknesses
  and opportunities for ARU case studies
• Establish a forward plan for strengthening the case
  studies and gathering evidence with a particular
  focus on institutional case studies
• Your objectives?

                                                        3
INTRODUCING RAND EUROPE

                          4
RAND Europe

 • Independent not-for-profit public policy research institute
     – “help improve policy and decision making through research and
       analysis”
 • Not a university and not a management consultancy —but with the
   capabilities of both
 • Part of the global RAND Corporation
 • Work across the breadth and depth of government
 • Strongly held values of quality and objectivity
 • Provider of evidence

                                                                       5
Our experience of research impact

                                    6
CHALLENGES IN MEASURING
IMPACT

                          7
There are many reasons to evaluate impact...

 • Advocacy
    – ‘Make the case’ for research funding
 • Accountability
    – To taxpayer, employer, donors etc
 • Analysis
    – What works in designing or disseminating research?
 • Allocation
    – What to fund (institution, field, people …)

                                                           8
Impact is an important component of REF

• One 4* case study is (roughly) equivalent to thirteen 4*
  publications/outputs
• The back of the envelope…
    – 4 outputs/researcher vs. 1 case study/10 researchers
    – 10 FTEs = 40 outputs
• The ratio of weightings is…
    – 65%/20% = 1/3
• Therefore…
    – 1/3 weighting of impact x 40 outputs = 13 outputs/case study

                                                                     9
Measuring and identifying impact is not easy

                                               10
The challenge of attribution

                               11
Dealing with serendipity

                           12
Addressing time lags

                       13
Will users engage?

                     14
Examining the margin

                       15
Measuring impact is difficult, but not impossible

                                                    16
REF AND IMPACT – WHAT DID
  WE LEARN LAST TIME?

                            17
Plurality of impacts

• 'For the purposes of the REF, impact is defined as an effect on,
  change or benefit to the economy, society, culture, public policy or
  services, health, the environment or quality of life, beyond academia'
  (HEFCE et al 2012a).

                                                                           18
The case study’s ‘job’ is to establish the impact
pathway

                                                    19
Why the contribution of the research is important

                                                    20
The critical role of evidence

 • Significance must be demonstrated through nature of benefits
 • Reach must be demonstrated by the scale of impact
 • Evidence will vary for different types of impact
     – Quantitate and qualitative
     – Value of fact vs evidence of opinion
 • Breadth versus depth of case study
 • Importance of the chain of evidence

                                                                  21
What counts as evidence?

• All panels give examples of types of evidence
• The case study database is a great source of relevant examples
• International examples:
   – Canadian Academy of Health Sciences (CAHS) Payback Framework
       • Five categories of impact with indicators
   – Australian Research Quality Framework provides list of indicators

                                                                         22
RQF indicators

•   Reduced pollution                                 •   Creative works commissioned
•   Regeneration or arrested degradation of natural   •   Change in procedures, behaviours, outlook etc
    resources                                         •   New policies, guidelines, legislation etc
•   Lives saved                                       •   Citations of research in legal judgements which
•   Reduced infection rates                               become case law.
•   Reduced treatment time and cost                   •   Contracts and industry funding
•   Increased literacy and numeracy rates             •   Repeat business
•   Positive reviews of creative publications and     •   Number of presentations involving contact with
    performances                                          end-users
•   Increased cultural awareness                      •   Community awareness of research
•   Royalties                                         •   Non-academic publications & performances
•   Increased employment                              •   Collaborative projects with end-users
•   Reduced costs and resource usage                  •   Citations in Government reports, Hansard, etc
•   Increased competitiveness of Australian           •   Provision of expert advice and submissions to
    industry                                              enquiries etc
•   Spin off companies                                •   Invitations to be a visiting researcher or
•   New products & inventions                             researcher in residence at an end-user
•   Licences                                              institution (if based on research merit)
                                                      •   Invitations to be on reference, advisory and/or
                                                          steering committees (if based on research
                                                          merit)

                                                                                                            23
REF guidance on indicators/evidence

–   Measures of improved clinical                –   Citation in a public discussion or
    outcomes                                         consultation
–   Documented changes to public health          –   Measures of improved inclusion, welfare or
    guidelines                                       equality
–   Evidence of adoption of best practice        –   Visitor or audience numbers, or number of
–   Evidence of use of process or                    participants
    technology                                   –   Quantitative data relating, for example, to
–   Documented evidence of influence on              effectiveness or organisational
    policy process                                   performance
–   Economic impacts cost-effectiveness          –   Satisfaction measures
    measures                                     –   Incorporation in training or CPD material
–   Priority shifts in expenditure profiles or   –   Publication and sales figures, tourism data
    quantifiable reallocation of budgets         –   Critiques or citations in users’ documents,
–   Documented evidence of policy debate             e.g. in the media, in teaching materials, for
–   Measures of improved international               public or commercial bodies
    welfare or inclusion                         –   Number and profile of people engaged,
–   Evidence of sustained engagement                 types of audience
    with a group                                 –   Formal partnership agreements with major
–   Evidence from clinical trials                    institutions, NGOs and public bodies
–   Traceable references to inclusion of         –   Independent testimony
    research in industry standards or                                                                24
    authoritative guidance
CONSIDERATIONS WHEN
DEVELOPING CASE STUDIES

                          25
Overall observations from your case studies

 •   There is a need for better understanding of what impact stands for
 •   Understanding eligibility and key parameters for the REF is crucial for
     building the impact case study
 •   There is no one-size-fits-all approach – either the case studies or
     assessment
 •   Collecting the evidence needs to be seen as a planned, systematic,
     and recurring activity
 •   Contributing to / influencing policy, public debate needs some
     forethought (and luck!)
 •   Avenues for dissemination and resources within ARU
 •   Mapping within and between units across ARU
 •   Going forwards the writing of the case study will be important

                                                                               26
Understanding of what impact stands for

• Impacts listed must be outside academia
    – Note this is different from establishing the impact pathway
    – Impact needs to have been realised, not potential
    – Impact demonstrated needs to be within the impact window
• Dissemination at academic conferences can be an aspect of the
  case study but not impact itself
• Reach and significance are two sides of the ‘impact' coin
    – Both are equally important
    – Don’t forget about contribution!

                                                                    27
Understanding eligibility and key parameters for
the next REF
• Uncertainty around revision of existing impact: timelines for REF
  2021and new contribution since 2014 for resubmission case studies.
• Impact stays with the institution at which the research was
  conducted and does not travel
    – ARU’s unique contribution in case of multi-institutional effort
• Stern review and its implications

                                                                        28
Stern review recommendations about impact

• Recommendation 5: Institutions should be given more flexibility to
  showcase their interdisciplinary and collaborative impacts by
  submitting ‘institutional’ level impact case studies, part of a new
  institutional level assessment.
• Recommendation 6: Impact should be based on research of
  demonstrable quality. However, case studies could be linked to a
  research activity and a body of work as well as to a broad range of
  research outputs.
• Recommendation 7: Guidance on the REF should make it clear
  that impact case studies should not be narrowly interpreted, need
  not solely focus on socio-economic impacts but should also include
  impact on government policy, on public engagement and
  understanding, on cultural life, on academic impacts outside the
  field, and impacts on teaching.

                                                                        29
No one-size-fits-all approach

• This applies to putting the case study together and also to its
  assessment
• Important to recognise that the panel assessing the case study
  would take a subject/field-specific approach
• The panel's approach to assessment would be based on the
  strengths and weaknesses of the subject/field

                                                                    30
Collecting the evidence

• Evidence needed to demonstrate reach and significance
• Reach achievable through (amongst other methods) public talks,
  writing in the trade press, the conversation, twitter/social media
  communication, download data, number of users, etc.
• Demonstrating significance requires evidence of change over a
  length of time. This change can be at an individual / collective level
• Significance can also be about preserving cultural heritage / or a
  unique contribution to the field or discourse
• Planning and persistence needed to collect evidence (e.g. feedback
  forms, twitter hashtags, emails, feedback surveys, testimonials,
  written responses, book reviews, Amazon reviews, online
  comments/feedback)

                                                                           31
Contribution / influence on policy, public debate

• Has the research fed directly / indirectly into policy debate or related
  outcomes?
• Testimonials from the stakeholders (written, email, audio, or video) -
  government, NGOs, consumer bodies, or industry consortia
• Has the research been referenced in consultations, select
  committee hearings, and/or parliamentary debate?
• Look out for any citations in guidelines, reference documents, or
  policy papers published by the government

                                                                             32
Avenues for dissemination and resources within
ARU
• Consider using MSc/PhD students for communicating through social
  media and collecting evidence such as participant/audience
  feedback
• Engage with stakeholders
• Create forums for people to engage with research
    – Using 'The Conversation' for reaching a non-academic audience,
      tracking feedback and debate
    – Dissemination through relevant charities and public professional bodies
• Consider availability of centralised resources
    – University press officer to publicise talks, book and article publications
    – Technology transfer team to support commericalisation and discussions
      with industry
    – Shared resources through Impact officers who provide practical and
      individual support across multiple UOAs                                      33
Mapping within and between units across ARU

• Different ways of clustering research and impact to frame a case
  study
   – Maturity of research
   – Chronological
   – Type of impact
   – Cross-cutting
   – Type of stakeholder group
   – Grand challenges
• Use of less developed or smaller scale impact alongside other
  elements of a case study

                                                                     34
Going forwards the writing of the case study will be
important
• Title
• Summary of impact
• Underpinning research
• References to research
• Impact section
• Corroborating sources

                                                       35
Title of the case study

• The first thing that reviewers will read
• A simple way to clarify and strengthen case studies
• Should include a clear statement of the impact
• Should express the focus of the case study
• Titles should be impact rather than research-led
• Also attention grabbing, where possible
• Do not make them too long

                                                        36
Examples of message driven titles from REF 2014
submissions

 •   Improving health access and equity in India through health financing
     reform
 •   Changing the way history of the Cold War is taught in schools across
     the United States and Europe
 •   A framework for establishing how to increase global food production at
     least cost to biodiversity
 •   Innovative chemistry reduces the environmental impact of mining and
     pharmaceutical manufacture
 •   The use of citizen science in recording wildlife: ladybirds and invasive
     alien species
 •   Attraction explained: The science of how we form relationships

                                                                                37
Examples from your case studies

•   Too long
     – “The prevalence of vision loss due to various causes had been occasionally
       analysed by the WHO in the past, however the outputs were limited in there was
       minimal attention to age-specificity and no gender-specificity to the results.”
     – Despite occasionally analysing the prevalence of vision loss, WHO's past
       studies have paid minimal attention to age and gender-specific causes.

•   Too short
     – Labour History Research Unit
     – A new understanding of the civil wars in the Labour party and their
       influence on the livelihoods of the British working class

                                                                                         38
Summary of the impact
•   From the template: “This section should briefly state what specific impact is
    being described in the case study”
•   Summary of key points, like an abstract
•   Communicate three things:
     – What the problem is/was (1 sentence)
     – What the research was (1-2 sentences)
     – What impact you had (2-3 sentences)
•   Bullet points or numbered lists could aid clarity and help keep within word
    limit
•   Use figures where possible

                                                                                    39
Underpinning research
•   Needs to state what was unique about the particular research which
    contributed to the impacts claimed
     – Distinct contribution c.f. research elsewhere
•   Research should be clearly defined and bounded
•   Focus on the impact, not career of particular researchers

                                                                         40
References to the research
•   From template: “This section should provide references to key outputs from
    the research described in the previous section, and evidence about the
    quality of the research”
•   Include: Author(s); Title; Year of publication; reference details (e.g. Journal
    title and issue); DOI or URL if required
•   Indicative maximum of 6 references
•   Need to provide evidence of quality, e.g.:
     – rigorous peer-review process (e.g. Journal impact factors)
     – end of grant reports referencing a high quality grading
     – favourable reviews from authoritative sources
     – prizes or awards made to individual research outputs cited
     – evidence that an output is a reference point for further research beyond
       the original institution
     – rating in output submission
                                                                                      41
Details of the impact
•   Core of the case study
•   Needs to present a clear narrative outlining how the impact came about,
    and how it is linked to the research
•   From the template: “This section should provide a narrative, with supporting
    evidence, to explain:
     – how the research underpinned (made a distinct and material
       contribution to) the impact
     – the nature and extent of the impact”

                                                                                   42
General approach to impact section
•   Overview of the impact at the start of the section
•   The narrative should outline the key steps which took place from the
    research being conducted to impact being realised
     – NB: This may not be linear!
•   Focus in this section on the impact
     – Specific references and information on reach and significance
     – Quantitative information can be useful here
•   Always think one step further, beyond the initial impact
•   Do not restate academic research, impacts
•   Do not use the words ‘will’, ‘could’, ‘may’, etc…

                                                                           43
Sources to corroborate the impact
•   From the template:
     – “This section should list sources external to the submitting HEI that
       could, if audited, provide corroboration of specific claims made in the
       case study... the information in this section will be used for audit
       purposes only”
•   Should provide evidence for each specific impact claimed
     – What are the crucial steps in your narrative?
•   Link evidence to impacts claimed in Section 4
     – Similar structure for each section
     – Use a numbering system

                                                                                 44
45
Breakout session: Introduction to institutional case
studies
• Stern Recommendation 5: Institutions should be given more
  flexibility to showcase their interdisciplinary and collaborative
  impacts by submitting ‘institutional’ level impact case studies, part of
  a new institutional level assessment.

                                                                             46
REF 2014 case studies were interdisciplinary in
nature

                                                  47
Breakout sessions
• The purpose of the breakout sessions will be to:
    – Identify areas of overlap between UOAs which could be developed into
      institutional case studies
    – Start developing plans to create and track impact
• A discussion in three parts:
    – Part 1: Reflect on learning and identify areas of synergy across UOAs
      (20 mins)
        • (By doing this exercise in groups we hope to encourage and enable areas of
          overlap and synergy to come out)
    – Part 2: Developing and mapping your impact strategy (20 mins)
        • (Intention is to encourage people to think about cross-cutting case studies,
          but generally this mapping tool could be used at an individual case study
          level and overall we’d like to get people comfortable with using it so you can
          go away from the day with a planning tool to hand).
    – Part 3: Reporting back to the wider group (20 mins)
                                                                                           48
How to think about case studies

• Different ways to frame the case study
    – Chronological
    – Type of impact
    – Cross-cutting
    – Type of stakeholder group
    – Grand challenges
• Ideas for institutional case studies
    – Medical
    – Teaching/schools/social work
    – Veterans/military

                                           49
Part 1: Reflect on learning and identify areas of
synergy across UOAs (20 mins)

                                                    50
UOA W                UOA X      UOA Y   UOA Z

                                                                      Grand Challenges: what are the ‘big’ social, economic, political, health,
What are the
research strengths

                                                                             cultural, or other issues which cut across these UOAs?
of my UOA?

What are the impact
strengths of my
UOA?

What are the
research and impact
opportunities
looking forward?

Who are our main
external
stakeholders?

What other UOAs
could our UOA work
with?

Summary of overlapping areas of impacts across UOAs
Realised impacts:

Potential impacts:

                                                                                                                                                  51
Part 2: Developing and mapping your impact
strategy (20 mins)

                                             52
Area of exploration                                                Notes

Grand challenge                                                    This will allow researchers to identify their ‘hook’ for the case
- What’s the ‘vision’ for the impact case study? What broader      study early on.
social challenges is it addressing?
What impacts will be highlighted?                                  This will encourage researchers to think about how they will
- What are the 3-4 main areas of impacts which will be             group various impacts under different headings. There can be
demonstrated? (e.g. improved social well being, changes in         multiple kinds of impacts, but it is important to carve out a clear
policy, changes in practice, improved health outcomes, economic    narrative.
growth, etc.)
What outputs will lead to the impacts?                             This will enable people to plan what research outputs might be
-  2*+ research publications                                       needed to support the case study, but also what additional, more
-  Research networks and collaborations                            impact-focussed outputs might be required.
-  Publications targeted at stakeholders
What dissemination efforts will be needed and what audiences       By separating out dissemination as a unique step, this allows for
will you target?                                                   the distinction between dissemination and impact to be
-     Dissemination outputs (media, blogs, etc.)                   reinforced, but will also encourage people to think about what
-     Audiences (local councillors, professional bodies, general   dissemination activities are needed to create impact.
      public, policymakers)
What activities do we need to proceed with in order to produce     This step will allow people to plan different activities.
our outputs/outcomes?
-   Proposed new research
-   Proposed new stakeholder engagements
-   Proposed new dissemination activities
-   Proposed new evaluations

What resources are already available and what will be needed?      This step will allow people to think about what resources are
-  Existing research( mark with a *)                               required for each activity.
-  Investment required (time, money, people, space, project)
-  Existing networks/stakeholder relationships

                                                                                                                                         53
Logic model worksheet

                        54
Reflections and next steps

                             55
UOA Specific sheets to be printed for
task 1

                                        56
Grand Challenges: what are the ‘big’ social, economic, political, health, cultural, or other
                        UOA 1                UOA 3    UOA 5   UOA 6

What are the
research strengths of
my UOA?

What are the impact
strengths of my

                                                                                               issues which cut across these UOAs?
UOA?

What are the
research and impact
opportunities looking
forward?

Who are our main
external
stakeholders?

What other UOAs
could our UOA work
with?

Summary of overlapping areas of impacts across UOAs
Realised impacts:

Potential impacts:
                                                                                                                                                                     57
Grand Challenges: what are the ‘big’ social, economic, political, health, cultural, or other
                     UOA 11           UOA 15          UOA 16   UOA 17   UOA 26

What are the
research
strengths of my
UOA?

What are the
impact strengths

                                                                                                          issues which cut across these UOAs?
of my UOA?

What are the
research and
impact
opportunities
looking forward?

Who are our main
external
stakeholders?

What other UOAs
could our UOA
work with?

Summary of overlapping areas of impacts across UOAs
Realised impacts:

Potential impacts:
                                                                                                                                                                                58
Grand Challenges: what are the ‘big’ social, economic, political, health, cultural, or other
                     UOA 19           UOA 20          UOA 22   UOA 23   UOA 25

What are the
research
strengths of my
UOA?

What are the
impact strengths

                                                                                                          issues which cut across these UOAs?
of my UOA?

What are the
research and
impact
opportunities
looking forward?

Who are our main
external
stakeholders?

What other UOAs
could our UOA
work with?

Summary of overlapping areas of impacts across UOAs
Realised impacts:

Potential impacts:
                                                                                                                                                                                59
Grand Challenges: what are the ‘big’ social, economic, political, health, cultural, or other
                 UOA 28          UOA 29           UOA 30   UOA 34   UOA 35   UOA 36

What are the
research
strengths of
my UOA?

What are the
impact

                                                                                                               issues which cut across these UOAs?
strengths of
my UOA?

What are the
research and
impact
opportunities
looking
forward?

Who are our
main external
stakeholders?

What other
UOAs could
our UOA work
with?

Summary of overlapping areas of impacts across UOAs
Realised impacts:

Potential impacts:
                                                                                                                                                                                     60
You can also read
NEXT SLIDES ... Cancel