Using Responsible Research Metrics to prepare for REF 2021 - Dr Evangelia Lipitakis
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Using Responsible Research Metrics to prepare for REF 2021 Dr Evangelia Lipitakis Research Evaluation Specialist 9th October 2019
Agenda
1 Provision of citation data to REF 2O21
2 Citation data and background to contextualization
3 Use cases
2
What is Clarivate
1. Matching API service
providing to the
REF? • Link submitted outputs to our citation data
• Incorporated into the REF submission system
2. Contextual data
• Support panels in their evaluation
• https://www.ref.ac.uk/guidance/citation-and-contextual-data-
guidance/
3. Matching support services
• Assist HEIs to match outputs to Web of Science
4. Interpretative support
•Support and training about the responsible use and potential pitfalls
of citation data
3
Citation data • Panels only using citation data to inform their assessment of the
and preparing significance of outputs as part of their peer review of the
for the REF ‘originality, significance and rigour’.
• Cautious use of citations for evaluating recent publications.
• We will not be providing citation data for the most recent year.
• Bibliometric methods do not make a value judgement on why a
given paper is cited which could make reliance on them at the
paper-level somewhat misleading.
• Andrew Wakefield’s original paper on a link between MMR and autism
has been cited 1,248 times, putting it among the most highly-cited
papers.
• Clarivate Analytics and the REF team are committed to
responsible use of metrics.
• Working with the UK Forum for Responsible Research Metrics, who are
also represented on the REF Panel User Group for metrics.
4
Database
coverage Data we provide to the REF will be derived from
the Web of Science Core Collection:
• Science Citation Index – Expanded.
• Social Sciences Citation Index.
• Arts and Humanities Citation Index.
• Emerging Sources Citation Index.
• Conference Proceedings Citation Index.
Citation counts will be consistent with those
provided in out Web of Science product based on
the Core Collection.
5
Why do citation Because citations credit the intellectual contribution of earlier
counts require research, the number of citations a paper receives reflects the
contextualisation? impact it has had on the field to which it relates.
But citation counts are also affected by:
• The field of research.
6
Background to
contextualising Citation rates and behaviours vary by field:
citation data
Field
60 Note: Chart shows citations recieved
by global papers to 31 Jan 2019
Biology & biochemistry
50
Citations per paper
40 Clincial medicine
30 Economics & business
20 Materials science
10
0
2002
2003
2004
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
1999
2000
2001
7Why do citation Because citations credit the intellectual contribution of earlier
counts require research, the number of citations a paper receives reflects the
contextualisation? impact it has had on the field to which it relates.
But citation counts are also affected by:
• The field of research.
• The age of the paper (i.e. the publication year).
8Background to
contextualising Older papers are more likely to be cited and have had more time
citation data to accumulate citations:
Publication year
40 100
Percentage documents cited
30 75
Citations per paper 20 50
10 25
0 0
2009 2010 2011 2012 2013 2014 2015 2016 2017 2018
Note: Chart shows citations recieved by UK papers to 31 Jan 2019
9Why do citation Because citations credit the intellectual contribution of earlier
counts require research, the number of citations a paper receives reflects the
contextualisation? impact it has had on the field to which it relates.
But citation counts are also affected by:
• The field of research.
• The age of the paper (i.e. the publication year).
• The type of document (e.g. article, review, proceedings
paper, book).
10Background to
contextualising Reviews and articles tend to receive more citations
citation data than other document types:
Document type
12 Note: Chart shows citations recieved by global papers published in 2017
10
Citations per paper
8
6
4
2
0
Books Articles Reviews Proceedings
papers
Chemistry Physics Economics and business
11What is Clarivate Contextual data has been agreed and published on the
providing to the REF website:
REF? https://www.ref.ac.uk/guidance/citation-and-contextual-data-guidance/
Contextual data
• Global mean number of citations per publication.
• Number of citations equating to different thresholds of
excellence:
• The world’s top 1%, 5%, 10%, 25% and 50% of most
highly cited papers.
• We will provide the data to the REF team as a spreadsheet.
• Additional look-up function.
Three snapshots during the project:
• Start of the pilot phase (autumn 2019)
• Start of the submission phase (January 2020)
• For the evaluation phase (November 2020)
12Monitor data in
Web of Science
https://www.ref.ac.uk/guidance/citation-and-contextual-data-guidance/Monitor article level indicators in InCites B&A
https://www.ref.ac.uk/guidance/citation-and-contextual-data-guidance/How citation counts can be contextualized?
Normalization of citation counts per year, document type and research area
Category expected citations Category Normalized Citation Impact Percentile in Subject Area position
average number of citations for the ratio of actual times cited to category out of 100% in the citation distribution
same document type published in expected citations for the same document type,
the same year in the category publication year, and subject category
ü 19 citations mean an impact 4.9
times higher than the category’s ü Out of 100%, this paper is
expected impact positioned well in the top 3%
15Output classification by
The REF 2021 UoA Research Schema is mapped on the
The context of REF 2021 research schema
Journal Subject Categories and is available in InCites B&A.
This is a proxy mapping based on the scope notes associated with
our categories (not HESA cost centres mapping).
The Journal Subject Categories are the 254 Web of
Science Subject Categories
161 How to create and upload a dataset for papers which
have been identified as potential submissions
Use cases
2 How to inform submission selection using normalised
indicators and contextual data
3 How to download article level metrics and
calculate percentile thresholds
4 How to collect REF compliant outputs in InCites using filters
(Timespan, Document Type, OA, Research Schemas)
5 How to Benchmark against other UK/international
universities in the context of REF 2021 (or other schemas)
17Creating a dataset - Organizational Level
Organizational level and refining with filters
18Creating a dataset – Custom dataset
Upload a File of Identifiers (WoS Accession Number, DOI, PubMedID)
A custom dataset can be created from a list of document identifiers that you compile. There are three valid
identifiers:
•Web of Science Core Collection accession number. Example: WOS:000388713500004
•DOI. Example: 10.1088/1478-3975/13/6/066006
•MEDLINE ID (PubMed ID). Example: MEDLINE:24843332
The format of the file you upload must be *.csv or *.txt. Upload up to 50 k records per file, in minutes.
19Creating a dataset – My Organization’s Faculties, Departments, Schools
Researcher, team & departmental research analytics reporting using bibliometric data – based on your own data
20Drill down to contextual metrics to verify identified outputs from any dataset. For every level of aggregation of your datasets, you can get the contextualized article level metrics
Access the article level contextual indicators in the research schema of
your choice
Get the full list of article level metrics
with contextualized indicators for the
research area of your choice
22Create customized percentiles thresholds: 1%, 5%, 10%, 25%, 50%
Identify the publications per percentile threshold
Download the csv file with
article level indicators. Sort the
dataset for Percentile in
Biochemistry &
Subject area (smaller to larger).
Molecular Biology
Identify the % of articles within
articles in the top 1%
1%, 5%, 10%, 25% and 50%
thresholds.
23Benchmark with other UK/International institutions
SWOT Analysis of Russell Group’s 2013-2018 articles in Biochemistry & Molecular Biology
24Understand the contribution of multi-authored papers to impact
Use threshold filters which permit the inclusion or exclusion of multi-authored
papers and limit output to comparable sizes
Web of Science Documents: Threshold
filter which you can limit output to a
minimum or maximum number of
publications. (i.e. use a minimum # of
papers and then rank your results by
Category Normalized Citation Impact
indicators to exclude smaller datasets
that might skew results
Authors per Documents: Threshold filter
which permits the inclusion or exclusion
of multi-authored papers (i.e. see the
papers which only have between 1 and 9
authors)
25Verify open access output with contextualized metrics
Institutional evaluation of OA policy
Open Access status is available across the Web of Science platform already as a result of a partnership with Impact Story, a
not-for-profit organization that recently launched a knowledge base of Open Access (OA) content.
This knowledge base makes it possible to discover and link to legal Gold or Bronze (free content at a publisher's website)
and Green (e.g., author self-archived in a repository) OA versions. Learn more about Open Access versioning in Web of
Science
26Identify Open Access trends and impact of output
Verify Gold Directory of Open Access Journals (DOAJ) output with contextualized metrics
27Thank you Questions? Dr Evangelia Lipitakis Research Evaluation Specialist Evangelia.lipitakis@clarivate.com https://clarivate.com/webofsciencegroup/solutions/ref2021/ Web of Science Group retains all intellectual property rights in, and asserts rights of confidentiality over, all parts of its response submitted within this presentation. By submitting this response we authorise you to make and distribute such copies of our proposal within your organisation and to any party contracted directly to solely assist in the evaluation process of our presentation on a confidential basis. Any further use will be strictly subject to agreeing appropriate terms.
You can also read
