Revisiting the journal impact factor
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Revisiting the journal impact
factor
Vera Morgan and Aleksandar Janca
INTRODUCTION
I
n 1955, Eugene Garfield proposed “a bibliographic system for science
literature that can eliminate the uncritical citation of fraudulent,
incomplete, or obsolete data by making it possible for the conscien-
tious scholar to be aware of criticisms of earlier papers”.1 This led to the
publication of the Science Citation Index (SCI) by the Institute for Scien-
tific Information (ISI) and the development, by Garfield and Sher of the
ISI, of the journal impact factor to facilitate the selection of journals for
the SCI and Current Contents2 (Table 1). Some 44 years later, and 36
years after the publication of the first Science Citation Index, Garfield
wrote that “[l]ike nuclear energy, the impact factor has become a mixed
blessing”.2 Hecht et al are more unequivocal in their criticism, describ-
ing the impact factor as “a misnamed, misleading, misused measure”.3
This paper revisits the concept of the impact factor. It identifies two
main uses and several limitations in its use. It argues that distinguish-
ing the two uses of the impact factor helps one to decide whether the
journal impact factor is valuable or whether it is an ill-conceived meas-
ure that should be abandoned.
WHAT IS A JOURNAL’S IMPACT FACTOR
The impact factor of a journal is an ISI measure found in the Journal
Citation Reports (JCR), an index of published journals in the sciences and
social sciences produced annually by the ISI. It is one of several values
in the JCR based on the frequency with which articles published by a
journal are cited. The impact factor, the most common of these values,
is used to rank journals both overall and within a discipline area
(Table 1).
The impact factor measures the average number of times articles pub-
lished within the previous two years by a journal are cited in the current
Australasian Psychiatry • Vol 8, No 3 • September 2000
Table 1: Terms and abbreviations
Vera Morgan, BA, DipEd, MSocSc ISI Institute for Scientific Information which publishes:
Research Fellow
University of Western Australia SCI Science Citation Index
Department of Psychiatry and Behavioural Science
Perth, Western Australia
Aleksandar Janca, MD, MSc, FRCPsych, FRANZCP
JCR Journal Citation Report
Associate Professor of Psychiatry
University of Western Australia Impact factor The impact factor measures the average number of times articles
Department of Psychiatry and Behavioural Science published within the previous two years by a journal are cited in
Perth, Western Australia
Correspondence: Vera Morgan, University of Western the current year.
Australia, Department of Psychiatry and Behavioural Science,
Medical Research Foundation Building, Rear 50 Murray Street, Cited half-life The cited half-life measures the number of publication years
Perth, WA 6000.
Tel: 08 9224 0235 preceding the current year which account for 50% of total citations
Fax: 08 9224 0285 within a journal in the current year.
Email: vmorgan@cyllene.uwa.edu.au
230year covered by the JCR. For example, the 1998 However, the JCR also adds that its citation scores
impact factor for an individual journal is the number “should not be relied on as the sole source of infor-
of citations in 1998 of articles published by the jour- mation when comparing and evaluating publica-
nal in 1996–97 divided by the number of articles pub- tions. The quantitative citation data it reports are
lished by the journal in the same two-year period. intended to complement, not replace, traditional
Only original research articles, review articles and qualitative and subjective inputs, such as peer sur-
technical notes are included in JCR counts of articles veys and specialist opinions”.5 Nonetheless, there is
in the denominator. However, counts of citations in growing evidence that the impact factor is used in
the numerator also include editorials, letters, academic appointments and promotion,6,7 and in
abstracts and the like. resource allocation.7 It would appear that some
scientific committees, while recognising that a sim-
A second JCR measure is the cited half-life. The cited ple count of articles published is poor evidence of
half-life measures the number of publication years the quality of research output, are coming to rely on
preceding the current year covered by the JCR which journal impact factors as a shortcut to assessing the
account for 50% of total citations within a journal in quality of individual papers and their authors. Hecht
the current year. Only journals with at least 100 cita- argues that it has had a particularly strong impact in
tions are listed. For example, in 1998 a journal may Western Europe, possibly in response to a need for
have included citations going as far back as 1964 but an objective system of review following a history of
50% of the citations in the journal were for items “favouritism, nepotism, and social-rank privileges”.3
published between 1993 and 1998. The JCR formula Benítez-Bribiesca observes that its popularity is
calculates a cited half-life of some 7 years, with the growing in developing countries who are endeavour-
exact figure based on the cumulative percentage of ing to ensure that their own research committees
citations per year. apply the same rigorous standards found in highly
developed countries.6
The JCR complements the SCI. The SCI compiles a
searchable list of all substantive items published in
journals during the period covered by the index
LIMITATIONS IN THE USE OF IMPACT
(Source Index), and further compiles all citations to FACTORS
that item (Citation Index). These indexes are organ- Limitations due the ISI method of calculation
ised in two parts: by items cited and by articles cit-
ing those items, both sorted on first author. Because There are a number of inherent limitations in the ISI
calculation of the impact factor which call into ques-
there is also a subject index (Permuterm Subject
tion its usefulness. First, the source database, the
Index), it is possible to trace published articles
SCI, is not comprehensive. It does not include books
through the subject matter. Thus the SCI allows
and, in 1997, it only covered some 3200 journals out
researchers to determine whether their publication
of an estimated world total of 126,000.7 There are
has been cited by others, and to trace those refer-
strong biases as to which journals are covered. There
ences. It is also a means of following backwards or
is a preponderance of journals written in English
forwards work published by a particular author or in
and, if in English, published in North America.7
a particular field of research. Indexes, similar to the
Opthof cites de Jong and Schaper’s 1996 data for
science index, have been published for the arts and
clinical cardiovascular papers.8 They analysed cita-
humanities, and for the social sciences. The JCR is tions for 137,019 papers published between 1981 and
compiled using data from these three indices. 1992 from the G7, seven major industrial democra-
cies, Canada, France, Germany, Italy, Japan, United
THE USE OF THE IMPACT FACTOR
Australasian Psychiatry • Vol 8, No 3 • September 2000
Kingdom and USA, as well as 7 smaller European
Garfield argues that the primary aim of the impact countries. Some 46% of these papers were never
factor is to help libraries decide what to purchase cited, ranging from 31% for Norway to 69% for
and authors decide where to submit their manu- Japan. The average number of citations per paper
scripts.2 The JCR also recommends its general use for was 7.5 for USA down to 2.0 for Japan. Moreover, it
tracking bibliographic trends, as well as a source of appears that discipline area also affects inclusion on
evaluative data for the editorial and publication the ISI database. One institution found that, in 1987,
boards of journals. Editors of journals such as the the database included 90% of publications from its
British Journal of Psychiatry endorse the use of the faculty of chemistry but only 30% of those from the
journal impact factor as a measure of journal status: faculty of biology.7
“Readers and future authors may like to know that Second, articles that are counted in the denominator
work published in the Journal is cited relatively of the impact factor equation — substantive citable
frequently — an indication that articles in the Jour- articles — may be hard to classify as such. This has
nal are read and have an impact on the psychiatric led to incorrect classifications which, in turn, affect
community”.4 the impact factor which decreases in value as the
231denominator increases, assuming the numerator Other aspects of journals will affect the impact fac-
remains static. For example, Joseph and Hoey calcu- tor. Elite generalist journals can be more selective
lated there were between 176 and 208 citable articles about what they publish as they are not limited to a
in the Canadian Medical Association Journal (CMAJ) specific field, nor do they have obligations to pub-
for the period 1995–96 compared to the JCR’s count lish the work of small and/or foreign research
of 350. The erroneous counting, by the ISI, of certain groups.3 Journals published in English and, if in Eng-
items as citable was a mistake identified by the CMAJ lish, in North America are not only more likely to be
which will be corrected in the JCR from 2000.9 included on the ISI database for the calculation of
Third, citations which are the basis of the numerator the impact factor (see above), they are also more
may be positive or negative, a consideration not likely to be read and cited for linguistic and cultural
included in the calculation of the impact factor reasons.4,7,8
which merely counts the number of citations. Finally, editorial policy may be influential. For
Finally, a journal’s impact factor will be adversely example, the time taken to review manuscripts is
affected either if it has a change of name over the important given a two-year limit on the calculation
two-year calculation period as the two journal titles of citations.12 Self-citations will also boost a journal
are not linked for the purposes of calculating the impact factor;7 it has been demonstrated that two
impact factor,10 or if names in a citation are mis- spurious self-citations per source article will increase
spelled, a not insignificant problem which affected a journal impact factor of 0.1 to 2.1.13
10% of citations in cardiovascular research in 1994.8
Biases arising out of journal characteristics QUALITY AND THE IMPACT FACTOR
Other problems in the use of the impact factor arise Given these limitations, what can be said of the use
out of biases affecting the algorithm for calculating of the impact factor as a measure of quality. Seglen
the impact factor introduced through journal char- argues, with good evidence, that the impact factor is
acteristics. Many of these biases arise because the useful in assessing the quality of a journal but not the
source for citations in the impact factor numerator is quality of any individual paper or author. For one
more broadly defined than the source for substantive thing, he contends, the impact factor cannot
items in the denominator; journal characteristics represent author impact where only 15% of journal
that increase the numerator at a greater rate than the articles account for 50% of citations.7 However, the
denominator will tend to result in higher impact analysis of simultaneous publication in multiple
factors. journals suggests that, while the level of citation is
more influenced by the quality of an article than by
The nature of the research published in a journal will
the journal impact factor, journal visibility may
exert a strong influence on its impact factor.
increase the citation rate by up to 80%.8 This, in
Research in the basic sciences is more likely to be
cited compared to articles with clinical and indus- turn, makes these prestigious journals highly attrac-
trial applications, or relating to health care pro- tive to good researchers, be it for the prestige or to
grams, which may be widely read and applied but ensure wider dissemination of their findings or
not cited much.6,7,10 Likewise, published research in because publication in high impact journals will
disciplines in which knowledge is expanding rapidly overtly or covertly increase their chances of gaining
and publication is a high priority such as genetics funding or promotion. “However, if the papers we
will be cited with greater frequency within the two- submit are not of high quality, they will not be
year time limit for the calculation of the impact accepted by prestigious journals and we have to sub-
Australasian Psychiatry • Vol 8, No 3 • September 2000
factor than publications in other disciplines such as mit articles of moderate quality to journals of inferior
mathematics.6,7,10 The cited half-life of journals gives impact”, writes Hoeffel.14
a good indication of the heterogeneity that exists These trends lie at the heart of warnings that high
between disciplines in the timeframe within which impact factor journals will attract the highest quality
articles will be cited; in some cases, that heterogene- contributions to the detriment of other journals, and
ity can even be seen within a discipline, for example, their use as a measure of quality in scientific evalua-
oral science.10 tions will have important repercussions for new
The type of items published in a journal can also fields, young researchers and research groups in
lead to biases. Review articles generate more cita- developing countries.6 They also underlie a concern
tions than research papers, because it is easier to ref- that, although high impact journals are not always
erence a review article than many separate original the most appropriate vehicle for some publica-
articles.3,7 Similarly published abstracts are often tions,6,15 an excellent article in an average impact
cited but do not affect the denominator in the cal- journal may be poorly received compared to an aver-
culation of the journal impact factor.11 age article in a high impact journal.3
232EXPLOITING THE IMPACT FACTOR: CONCLUSION
EFFECT ON EDITORIAL POLICY Two primary uses of a journal impact factor have
A further danger is that a journal may introduce been identified. The first use is in assessing the
changes merely to boost its impact factor. Hecht impact of a journal, either overall or within a field.
et al. warn that “[e]ditorial policies once determined This use seems inevitable and even warranted in a
by scientists-editors may increasingly be dictated by period where the number of journals is expanding
executives and accountants”. It is inevitable that rapidly and time for evaluation is limited. However,
some provisos are essential. Given the limitations of
journals should begin to use impact factors as mar-
the database and the effect of journal characteristics
keting tools, and evidence of this use is growing.3,4,16
on the impact factor, it is important for users to
The number of journals has increased exponentially
become familiar with the mechanisms underlying
over the last half-century. At the same time,
the calculation of the impact factor, in order to bet-
researchers, faced with fewer resources and substan-
ter understand its virtues and deficits. In particular,
tially larger administrative roles, are increasingly
it is essential to recognise that, while it appears to
judicious in their use of time and money. Similar hold that journals with high impact factors are of
constraints apply to departmental and institutional high quality, the converse is not necessarily true and
libraries. In these circumstances, journals will do that at least some high quality journals will have
what they can to win their share of readers. There is moderate, low or missing impact factors for any of
some evidence that this includes, not just marketing the reasons already outlined.
by a journal of its impact factor, but proactive
changes to a journal format to increase its impact At the same time, changes to journal policy need to
factor. The Canadian Medical Association Journal be monitored to ensure that the quest for a higher
notes the importance of increasing its impact factor impact factor is not at the expense (directly or indi-
rectly) of substantive changes to the quality of scien-
to attract authors.9 One measure planned is a shorter
tific publication.
time between receipt of a manuscript and publica-
tion.15 The Lancet now uses explicit citations to orig- Whether one should go so far as to separate review
inal articles in its letters of comment rather than journals from non-review journals and elite journals
including them in the text as it did formerly, thus from discipline-specific journals, and rename the
increasing the number of self-citations.13 Other jour- impact factor a “citation rate index” in keeping with
nals such as the FASEB Journal (the journal of the its actual role3 is open to debate. However, one sim-
Federation of American Societies for Experimental ple procedure available to all users is to consider the
Biology) and Nature have reclassified some items so discipline area when comparing impact factors
that they are not counted as source items for the cal- across journals. Standardising the impact factor
culation of the denominator but may be used in the across discipline areas takes into account at least
count of citations. Thus, for example, when meeting some of the influences affecting the citation rate
abstracts changed from being counted as source to within a particular area. The simplest way of doing
being counted as non-source in the FASEB Journal, this is to divide the impact factor of any journal by
its impact factor rose from 0.24 in 1988 to 18.3 in the impact factor of the journal ranked top within
the same discipline area. A multiplier can be used to
1989. 13 Numerous writers have pointed to an
set the maximum possible range.
increase in the publication of review articles in order
to boost the number of citations; Hecht et al. observe The second use of the impact factor is to evaluate
that 15 of the top ranked journals do not publish individuals and their work on the basis of the impact
Australasian Psychiatry • Vol 8, No 3 • September 2000
fresh results, only compilations of reviews and sum- factor of journals in which their publications appear.
maries of past research.3 This extension of what amounts to an ecological fal-
lacy into the area of citation analysis is highly con-
Advances in computer technology may lead to new troversial. It accepts that the quality of published
ways of enhancing one’s impact factor. For example, research can be evaluated from its “wrapping” rather
a journal that alerts its readership to its current con- than from an examination of its “contents”.7 The
tents and provides easy and timely access to full text use of the impact factor in these circumstances may
articles may be indirectly increasing the number of be unwarranted where other options are available.
citations made to it.
Unfortunately, while the published literature has
The trend to editorial changes to boost one’s impact debated extensively the use of the impact factor as a
factor is relatively new. Many changes documented means of predicting author quality, relatively little
have been cosmetic rather than substantive. discussion has focussed on the development of alter-
Whether such changes ultimately improve or detract native means of researcher evaluation. One such
from the substantive quality of a journal remains to alternative is to further develop the use of actual
be seen. author citation data in the SCI. The use of author
233Table 2: JCR classification of journals within psychiatry and their rankings in 1998.
1. Arch Gen Psychiatry 9.398 40. Int J Eat Disorder 1.139
2. Am J Psychiatry 5.939 41. J Psychosom Res 1.129
3. J Clin Psychopharmacol 5.338 42. Prog Neuro-Psychoph 1.114
4. Mol Psychiatry 4.756 43. Can J Psychiatry 1.058
5. Schizophr Bull 4.455 44. J Psychiatry Neurosci 0.966
6. Neuropsychopharmacology 4.318 45. Eur Arch Psychiatry Clin Neurosci 0.959
7. Dementia 4.148 46. Aust NZ J Psychiatry 0.847
8. J Clin Psychiatry 4.073 47. Neuropsychobiology 0.846
9. J Am Acad Child Adolesc Psychiatry 3.732 48. J Intellect Disabil Res 0.754
10. Br J Psychiatry 3.503 49. Dement Geriatr Cogn 0.710
11. Psychol Med 3.124 50. Br J Med Psychol 0.702
12. Psychosom Med 3.046 51. Nervenarzt 0.696
13. Psychopharmacology 3.032 52. Behav Med 0.649
14. J Neurol Neurosurg Psychiatry 2.938 53. Stress Medicine 0.648
15. Psychopharmacol Bull 2.590 54. Int J Psychiatry Med 0.630
16. Schizophr Res 2.496 55. J Psychosom Obst Gyn 0.606
17. Biol Psychiatry 2.405 56. Hum Psychopharm Clin 0.584
18. Pharmacopsychiatry 2.304 57. Convulsive Ther 0.500
19. Am J Orthopsychiatry 2.267 58. Neuropsy Neuropsy Be 0.500
20. Psychother Psychosom 2.103 59. Verhaltenstherapie 0.486
21. J Neuropsychiatry Clin Neurosci 1.697 60. Psychopathology 0.474
22. Addiction 1.620 61. Subst Use Misuse 0.457
23. J Child Adol Psychop 1.609 62. Child Psychiatry Hum De 0.452
24. J Affect Disord 1.586 63. Fortschr Neurol Psychiatr 0.441
25. Psychiatr Serv 1.570 64. Eur Psychiat 0.394
26. Int Clin Psychopharmacol 1.557 65. Psychiat Clin Neuros 0.382
27. Acta Psychiatr Scand 1.554 66. J Geriatr Psychiatry 0.375
28. Psychosomatics 1.541 67. Encephale 0.290
29. Exp Clin Psychopharm 1.510 68. Neurol Psychiat Br 0.216
Australasian Psychiatry • Vol 8, No 3 • September 2000
30. Drug Alcohol Depen 1.485 69. Nervenheilkunde 0.211
31. Psychiatry Res 1.424 70. Ann Med Psychol 0.208
32. J Psychiat Res 1.362 71. Arq Neuropsiquiatr 0.141
33. Psychiat Res-Neuroim 1.329 72. Neuropsychiatrie 0.125
34. Psychiatry 1.255 73. Acta Neuropsychiatr 0.089
35. J Nerv Ment Dis 1.250 74. Zh Nevropatol Psikhiatr 0.080
36. Compr Psychiatry 1.234 75. Giorn Neuropsi Evol 0.031
37. J Geriatr Psychiatry Neurol 1.222 76. Actas Luso Esp Neurol Psiquiatr Cienc Afines 0.013
38. Z Psychosom Med Psycoanal 1.209 77. J Ect —
39. Gen Hosp Psychiatry 1.144
234citation data is not without its problems, many of 3. Hecht F, Hecht B, Sandberg A. The journal ‘impact factor’: a misnamed, mislead-
which relate to limitations inherent in the SCI data- ing misused measure. Cancer Genetics and Cytogenetics 1998; 104: 77–81.
base. One author may appear under multiple names, 4. Howard L, Wilkinson G. Impact factors of psychiatric journals. British Journal of
particularly authors with particle or compound Psychiatry 1997; 170: 109–112.
names, those with names that are not English, or 5. Institute/for/Scientific/Information. 1998 JCR Journal Citation Reports. Science
those who have changed their name. A further prob- Edition. Philadelphia: Institute for Scientific Information; 1999.
lem occurs where an author is cited incorrectly — or 6. Benítez-Bribiesca L. The impact factor of medical journals: its use and misuse.
even with variant initials. Other shortcomings aris- Archives of Medical Research 1999; 30: 161–162.
ing out of the incomplete coverage of the ISI data-
7. Seglen P. Why the impact factors of journals should not be used for evaluating
base and raised in relation to the impact factor also research. British Medical Journal 1997; 314: 497.
apply here. However, there is some evidence that the
8. Opthof T. Sense and nonsense about the impact factor. Cardiovascular Research
number of citations correlates reasonably well, if not
1997; 33: 1–7.
perfectly, with peer esteem.8 Furthermore, it may be
possible to shift from mere numeration to the devel- 9. Joseph K, Hoey J. CMAJ’s impact factor: room for recalculation. Canadian Med-
ical Association Journal 1999; 161: 977–978.
opment of more detailed longitudinal profiles of
researcher performance. Researchers subject to scien- 10. Linde A. On the pitfalls of journal ranking by impact factor. European Journal of
tific scrutiny have an obligation to develop and Oral Sciences 1998; 106: 525–526.
refine the use of suitable performance indicators for 11. van Zanten S. Impact of abstracts and short reports. Letter. Canadian Medical
their own industry. And, once established, they have Association Journal 2000; 162: 489.
a right to demand that these indicators be used as 12. Hansson S. Impact factor as a misleading tool in the evaluation of medical jour-
the basis for a more equitable evaluation of scientific nals. Letter. Lancet 1995; 346: 906.
merit than currently available through the use of the 13. Gowrishankar J, Divakar P. Sprucing up one’s impact factor. Letter. Nature 1999;
impact factor. 401: 321–322.
REFERENCES 14. Hoeffel C. Journal impact factors. Allergy 1998; 53: 1225.
1. Garfield E. Citation indexes to science: a new dimension in documentation through 15. Weir E, Joseph K. How quickly does CMAJ evaluate submissions? Canadian Med-
association of ideas. Science 1955; 122: 108–111. ical Association Journal 1999; 161: 985.
2. Garfield E. Journal impact factor: a brief overview. Canadian Medical Association 16. Howard L, Wilkinson G. Impact factors of psychiatric journals. British Journal of
Journal 1999; 161: 979–980. Psychiatry 1998; 172: 457.
Australasian Psychiatry • Vol 8, No 3 • September 2000
235Copyright of Australasian Psychiatry is the property of Sage Publications, Ltd. and its content may not be copied or emailed to multiple sites or posted to a listserv without the copyright holder's express written permission. However, users may print, download, or email articles for individual use.
You can also read
