July 2022, Volume 72, Issue 7

Editorial

The Journal Citation Indicator: Have we found the Holy Grail for optimal assessment of research impact?

Muhammed Mubarak  ( Department of Histopathology, Sindh Institute of Urology and Transplantation, Karachi, Pakistan. )
Shabana Seemee  ( College of Physicians and Surgeons, Pakistan, Karachi, Pakistan. )

DOI: https://doi.org/10.47391/JPMA.22-77

 

 

A transparent, objective, fair and comprehensive assessment of the impact of research is very important in the modern academic and scientific world. It serves many purposes, and many stakeholders use it for varying objectives. Unfortunately, till date, there is no single universally agreed upon parameter by which all scholarly and academic activities can be judged without some degree of bias or prejudice. Till recent past, the most valuable and widely used marker for assessment of research impact has been the number of citations. A number of metrics have been developed over years based on citation counts. These serve as an immediate, valid marker of research influence and impact, as they reflect the decisions that researchers themselves make while recognizing important work. However, citations must be considered carefully and in context. For validity and fairness in assessing the impact of published research, analysis of citation counts must control for many variables such as subject category, journal publication frequency and size, document type and year of publication.

The oldest and still widely used citation-based metric is the Journal Impact Factor (JIF), promulgated annually in the form of Journal Citation Reports (JCR).1 The JCR is an annual compendium of citation data first published by The Institute for Scientific Information (ISI) in 1975. It was later on owned by Thomson Reuters and more recently by Clarivate Analytics. The JCR provides quantitative tools for evaluating, ranking, categorizing and comparing journals. Originally, developed by Drs. Eugene Garfield and Irving H. Sher for selecting the journals for inclusion in Science Citation Index (SCI) list of journals for libraries, the use and misuse of JIF have increased tremendously over the succeeding decades.2 The popularity and supremacy of JIF were un-paralleled till recent past, when some alternative citation metrics have made their entry in the field, such as Citescore by Scopus, another international indexing database.3 As alluded to earlier, JIF was originally intended to select the journals for libraries for purchasing and cataloging in ISI list of journals, which were then few in number and in print form. With great rise in the number and scope of journals in ever expanding fields of science, its overuse, and unwarranted expansion of its use in many other areas in the academic and scientific field, many misconceptions were generated and its limitations, overuses and misuses were exposed. Some of the aforementioned uses of JIF include hiring and promotion of academicians and faculty members, and grant approvals. A fundamental understanding of how the JIF is actually calculated highlights how inappropriate and unwarranted these applications are. Those who use it in such indiscriminate manner clearly fail to either understand its true derivation or appreciate the original intended use.4-6

There are several caveats in using JIF indiscriminately. The JIF can be influenced and biased by many factors. First of all, it is an average figure and not every article in that journal is cited for as many times as the JIF. In fact, some of the articles may not be cited at all in a given journal with medium to high JIF. One frequently cited article may artificially raise the IF of all articles published in a journal. Thus, JIF calculation is skewed. Secondly, its calculation is not as straightforward as it seems. Many factors are taken into consideration during its calculation, including variable criteria for citable documents and the denominator of the equation used for calculating IF. The suitability and exact nature of the methods with which citations and citable documents are defined for calculation have been seriously debated. Moreover, JIF counts only those citations which are in journals indexed and included in the SCI Expanded (SCIE) database or Web of Science (WoS) core collection. Any citations in journals not included in the above repositories are not counted. A vast majority of prestigious and fairly reputable national journals in many developing countries are not included in this list for one reason or the other. Their articles are frequently cited in other local or regional journals. But, many such journals from developing countries are either out of this IF game or have very low IFs. Their citation performances are not at all represented in the above platform, but that does not mean that they have zero citations. Only journals indexed and covered by the SCIE and social sciences citation index (SSCI) databases are currently scanned for their bibliographic lists, which is only a very small proportion (about 12,000 titles) of the world total of over 125,000 English language journals and chiefly those from the USA are favoured, and tend to enjoy higher IF than non-English language journals. Books and book chapters are not recorded for their bibliographies, nor are they included in IF calculation. The citation counting window spans only one year. The citable documents comprise of articles published during previous two years in the most widely used 2-year JIF. Such limited time frames in the calculation of JIF have also been criticized. It is also well known that journals in different disciplines have different publication and citation patterns. This difficulty is easily overcome by comparing like with like – comparing journals with others in the same discipline, rather than with journals from another discipline. Many more metrics were developed to overcome some of the above shortcomings, e.g. Immediacy index, 5-year IF, source-normalized impact per paper (SNIP) etc., but none has been able to completely replace 2-year JIF.7-9

In view of the above shortcomings, starting from 2021, Clarivate has introduced another category-normalized single number citation-based metric to serve as complementary to JIF. JCI normalizes for different fields and categories of research with JCI (Journal Citation Indicator) their widely varying rates of publication and citation patterns.10  As JIF, it is a journal-level metric but one, which is, easy to interpret and compare across disciplines. It is calculated for all editions included in WoS including SCIE, SSCI, arts and humanities citation index (AHCI) and emerging sciences citation index (ESCI), including those journals that do not have a JIF. Thus, its scope is broader than that of JIF. The citation counting window spans four years including JIF year and preceding three years. This will also facilitate the citation performance of journals, as most research takes time to be cited in other articles. The citable documents included are those published during previous three years as compared to two-year span of JIF. It measures the citation performance of a journal’s articles or reviews against the expected citation rate of the category of journals in which it is placed. A JCI value of one represents performance at par with world average, values above one are considered above average, and values below one are considered below average. A JCI value of two is considered twice the world average.10  

The actual calculation of JCI is complex and requires considerable computing power. During the calculation of JCI for a given journal, another Clarivate measure is used: Category Normalized Citation Impact (CNCI), a metric found in the analytic and benchmarking tool, the InCites. CNCI is a fair indicator of impact standardized for the category, year of publication, subject focus, and document type. The value of JCI is the mean CNCI for all articles and reviews published in a journal in the preceding three years. For example, for the 2020 JCI value, the years under analysis are 2017, 2018 and 2019 and the counting window includes 2017, 2018, 2019, and 2020 (Figure).

 

 

The most notable advantage of JCI is that its nomenclature accurately reflects the task it is meant to denote, i.e., citation performance, as compared to JIF terminology, where “impact” has more broad connotations. According to Clarivate, JCI is not meant to supplant or replace JIF, but is complementary to JIF and its use is likely to increase in future. The spread of JCI values among all journals is also narrow as compared to broad spread of JIF. As an example, JCI value is 26.14 as compared to JIF of 91.245. Compared to a journal of New England Journal of Medicine with JIF of 1, the above JIF value seems very high as compared to JCI value.

In conclusion, the quest for Holy Grail for fair, transparent and field-normalized assessment of research impact is ongoing. JCI represents an important new metric launched by Clarivate to showcase the performance of journals in a standardized way. Its use will likely increase in future with increasing awareness.

 

 

References

 

1.       Mubarak M, Seemee S. Journal prestige index: Expanding the horizons of assessment of research impact. J Coll Phys Surg Pak 2021; 31:1261-2.

2.       Jawad F. The Journal Impact Factor- Does it reflect the quality of a journal? J Pak Med Assoc 2020;70:1-2.

3.       Garfield E. The history and meaning of the journal impact factor. JAMA. 2006 ;295:90-3.

4.       Baker DW. Introducing CiteScore, Our Journal's Preferred Citation Index: Moving Beyond the Impact Factor. Jt Comm J Qual Patient Saf 2020;46:309-10.

5.       Kumar V, Upadhyay S, Medhi B. Impact of the impact factor in biomedical research: its use and misuse. Singapore Med  2009;50:752-5.

6.       Plebani M. Journal impact factor: the debate continues. Clin Che Lab Med 2013;51:2247-8.

7.       Zupanc GK. Impact beyond the impact factor. J Comp Physiol  Neuroethol Sens Neural Behav Physiol 2014;200:113-6.

8.       Crotty D. Other Metrics: beyond the Impact Factor. Eur Heart  2017;38:2646-2647 9. Hong ST. Over the journal impact factor. J Korean Med Sc 2013;28:969.

10.    Introducing the Journal Citation Indicator. Accessed from https://clarivate.com/wp-content/uploads/dlm_uploads/2021/05 Journal-Citation-Indicator-discussion-paper.pdf.

Journal of the Pakistan Medical Association has agreed to receive and publish manuscripts in accordance with the principles of the following committees: