Academic Career Confidential: The Truth about Journal Impact Factors
The formula is straightforward:
Number of citations in the current year to items published in the journal’s previous 2 years
-------------------------------------------------------------------------------------------------------------------------------
Total number of substantive articles and reviews published by the journal in the same 2 years
Yet the journal impact factor—Thomson Corporation’s measure of the frequency with which a journal’s hypothetical “average article” was cited in a particular year1—continues to stimulate controversy more than 40 years after its debut. This is probably not surprising, given the ever-higher stakes for publishers and editors competing for authors, readers and, in some cases, advertisers; researchers competing for tenured positions; and research institutions competing for external funding.
Created in the early 1960s by Eugene Garfield and Irving H. Sher as a tool for selecting additional journals for Science Citation Index, the impact factor can be helpful in evaluating a journal’s relative importance, especially when compared with other publications in the same field. Journal Citation Reports, the source of impact factor information, is one component of ISI Web of Knowledge, available from the Quick Links on the HSLS home page <www.hsls.pitt.edu>. Like any surrogate measure, however, it has its limitations. Critics over the years have pointed to several weaknesses in the assumptions on which it is based:
Room for editorial maneuvering
While the numerator includes citations to all types of documents--including editorials, letters, and abstracts--the denominator includes only substantive items such as research and review articles. In a recent editorial, the editors of PLoS Medicine charge that editorial manipulation is commonly used to “massage” a publication’s impact factor.2 For example, an editor may publish extra review articles, since these tend to be cited more frequently than other publication types (hence the higher-than-average impact factors enjoyed by Annual Reviews publications). Controversial editorials or a popular correspondence section can also inflate the numerator.3
Time factors
Since the impact factor calculation window is set at two years, the pace of the editorial cycle and of discovery in different research fields also make a difference. Articles in a given journal tend to cite articles from that same journal, so journals with a short publication turnaround time will accumulate a relatively larger number of citations. Rapidly moving research fields, such as the biosciences, are also well served by the two-year window, while more stable fields, such as mathematics, are relatively penalized.
Variations among the disciplines
Varying patterns and customs also exaggerate the discrepancies among journal rankings in different fields. In one study, biochemistry and molecular biology articles were cited five times as often as pharmacy articles.4 And articles in some disciplines are simply referenced more heavily than in others.
Human error
Impact factors are only as accurate as the citations they derive from. In one study, erroneous citation elements (year, volume or page number, or article title) occurred in 10 percent of publications.5
Scientific isolationism
Impact factors are based on data from two of Thomson’s citation databases: Science Citation Index and Social Sciences Citation Index. Like all literature indexes (including, for example, MEDLINE), these are selective, covering only a fraction of the world’s journals. Even in the age of the international scientific community, some critics say that Science Citation Index is dominated by English language publications and American science in particular.
--Pat Weiss
References:
1. Impact factor trend graph. Journal Citation Reports 2006 [cited 2006 6/7/2006]; 2004 JCR Science Edition: Available from: <http://portal.isiknowledge.com/portal.cgi?DestApp=JCR&Func=Frame>.
2. The impact factor game. PLoS Med 2006;3(6):e291.
3. Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314(7079):498-502.
4. Narin F, Hamilton K. Bibliometric performance measures. Scientometrics 1996;6:293-310. Cited in: Seglen PO. Why the impact factor of journals should not be used for evaluating research. BMJ 1997;314(7079):498-502.
5. Opthof T. Sense and nonsense about the impact factor. Cardiovasc Res 1997;33(1):1-7. Cited in: Cheek J, Garnham B, Quan, J. What's in a number? Issues in providing evidence of impact and quality of research(ers). Qual Health Res 2006;16(3):423-35.
