By Author
  By Title
  By Keywords

April 2013, Volume 63, Issue 4

Original Article

Accuracy of references: comparison between two premier Pakistani medical journals

Tariq Wahab Khanzada  ( Isra University, Hyderabad, Pakistan. )
Basant Kumar  ( Isra University, Hyderabad, Pakistan. )
Kamran Ahmed Malik  ( Sultan Qaboos University Hospital, Muscat, Sultanate of Oman. )
Abdul Samad  ( Isra University, Hyderabad, Pakistan. )


Objective: To compare two premier Pakistani medical journals for errors in references of original articles published in the year 2008.
Methods: All original articles of the Journal of Pakistan Medical Association and the Journal of the College of Physicians and Surgeons Pakistan published in 2008 were included in the study. References pertaining to journals were included in the study, whereas references pertaining to other sources such as books, internet articles, websites and newspapers were excluded. Errors were categorised into author error, article title error, journal title error, year of publication error, volume error and page number error. The data was analysed through SPSS 16.0.
Results: References from 200 original articles (100 each from both journals) were evaluated. Only 19 (9.5%) articles were found to be free of error with no significant difference between the two journals. On evaluation of 3783 references, the overall reference error was found to be 1015 (26.8%): 531 (31%) in JPMA and 484 (23.4%) in JCPSP. The author error was the commonest error among these references (n=490; 13%), followed by page error (n=297; 7.9%), article title error (n=222; 5.9%), and journal title error (n=189; 5%). JCPSP had statistically significant more article errors, whereas JPMA had statistically significant more journal title and page errors.
Conclusion: Reference errors constitute an avoidable but substantial lapse of medical literature. The magnitude of reference errors is much higher than expected.
Keywords: References, Citation, Errors, Accuracy. (JPMA 63: 445; 2013)


Bibliography of references section is an integral but often neglected component of most of the biomedical literature. One of the major parametres for determining the standard of any medical journal is the accuracy of its references. The major objectives of writing references include the acknowledgement of research work done by other researchers, easy accessibility of relevant literature for the readers, and corroboration of authors’ statements.1 Accordingly, the authenticity and accuracy of references is a touchstone to decide the standard of a medical journal. The accuracy of references is an essential requirement for proper transmission of scientific information, and a careful documentation of references facilitates the search of an individual reference in the massive ocean of knowledge. The Vancouver style for formatting references is considered the ‘gold standard’, and is followed by most medical journals.2
A reference is said to be correct if its each element of citation is identical to its source.3 In a research paper, the references serve to provide background information and allow the researcher to compare and contrast the work of others in relation to his own study. This correct listing and validation of the reference not only helps in citation-tracking, but also helps in calculating the journal’s impact factor by Scopus and the Web of Science.4 Some of the errors in writing references may not jeopardise their identification, but the errors in vital components of a reference may result in substantial delay or even failure to locate a reference. Accuracy of the information derived from an article comes under substantial doubt and debate if there are too many errors in references.
This study was planned to compare the two premier Pakistani medical journals — the Journal of Pakistan Medical Association (JPMA), and the Journal of College of Physicians and Surgeons Pakistan (JCPSP) — for the errors in references of original articles.

Materials and Methods

The selection criteria for inclusion of Pakistani medical journals in this study included Pub Med indexation and publication of regular monthly issues during 2008. Only two journals, the JPMA and the JCPSP, fulfilled the above mentioned selection criteria. All issues of JPMA and JCPSP published during 2008 — 12 issues each — were thoroughly evaluated for the references of original articles. References cited in all the original articles published by the two journals during 2008 were included. References pertaining to sources other than journals, like books, newspapers, websites, internet articles etc., were excluded from the study. The selected references were evaluated for all types of errors. These errors were categorised into ‘author errors,’ ‘article title errors,’ ‘journal title errors,’ ‘year of publication errors,’ ‘volume errors’ and ‘page number errors’. The data was analysed through SPSS 16.0. Chi-square test was used to determine the statistical significance and the differences having p-value of 0.05 or less were considered statistically significant for the purpose of comparison.


It was purely coincidental that the number of articles fulfilling the selection criteria for both journals was exactly same i.e. 100. Among these 200 articles, only 19 (9.5%) were found to be free of any error in references and there was no statistically significant difference in this regard between these two premier journals (p <0.469) (Table).

Overall, 3783 references fulfilling the selection criteria were included in the study; 1715 (45.33%) for JPMA, and 2068 (54.66%) for JCPSP. The incorrect references were found to be 531 (31%) and 484 (23.4%) for the two journals respectively, but the difference between two was not statistically significant (p <0.744). The error related to the author component was observed in 490 (13%) references and this was found to be the commonest error in the study. This was followed by errors related to page number (n=297; 7.9%), article title (n=222; 5.9%), journal title (n=189; 5%), volume (n=28; 0.7%) and year (n=22; 0.6%). On comparison of the two journals for errors in various components of references, JCPSP had statistically significant more errors in article title component (p <0.001), whereas JPMA had statistically significant more errors in journal title (p <0.001) and page number (p <0.001) components of references. No statistically significant differences were observed between the two journals regarding errors related to the author (p= 0.514), year (p= 0.408) and volume (p= 0.378) components of references.


In a manuscript, the reference section may be an area where one can encounter common errors in different components. Nothing is more frustrating than to cover unidentified references, as they reflect poorly on the field of bio-medical informatics. Reference citation errors can make the job of a reader extremely difficult to retrieve the references and verify the information to which the manuscript refers. Contributors to any journal have the responsibility to cite references accurately, but many fail to do so.
Literature search has demonstrated a very broad range of errors for citation of references in different biomedical journals ranging from 3% to 60%.1,3,5-9 This is consistent with our study in which the overall reference error is 26.8%. The study reveals the ‘author error’ being the highest (13%) among all reference errors. This observation is consistent with some other international studies10-13 demonstrating 8% to 18.1% ‘author errors,’ while other studies showed much higher ‘author errors’ ranging from 23.3% to 48%.14-18 Errors in author name make retrieving the article not only difficult, but also critical when author citations are used to measure research productivity.19
Incidentally, the commonest category of error in the study was ‘author error’, and it was 13% in both journals. Higher proportion of author errors has been quoted in other local20 and international studies.9,14 The next most common error was related to page numbers in this study. Overall, this was about 7.9%; 13.4% for JPMA, and 3.2% for JCPSP. The similar pattern of page error was seen in some international studies,9,14-17 whereas a higher rate of 17.3% was observed in a local study.20 The ‘article title error’ in this study was 5.9% which was much lower compared to other studies9,14-17,20 in which it ranged between 17.7%17 and 49.2%.15 The ‘journal title error’ in this study was 5% which is similar to the error observed in other studies (2.5% to 8.9%),14-16 but low compared to the error (17.3%) observed by another study.20 The ‘year error’ and ‘volume error’ in this study were 22 (0.6%) and 28 (0.7%) respectively. One study did not consider such errors important in the era of internet and various easy methods to access the electronic data.1
The uniform requirement for manuscripts submitted to biomedical journals, as stated in their guidelines, is that the ultimate responsibility of ensuring accuracy of all references in published article lies with the authors.2 Authors must verify the references and these should be in the same format as required by the journal concerned. All components of references should be verified. The main reasons of errors in citing the references is the usual practice of copying reference from the reference list of main article rather than gaining the original article. Besides, the other important reason for these errors is inaccurate transcription by the editorial staff, but this error is related to those journals where actual proofs are not seen by the authors and incorrect transcription of hand-written notes is done.16 Sometimes the responsibility of verifying or cross checking the cited references is delegated to junior staff (other than the author), student, or secretary, thus resulting in errors.20,21
Different studies have proposed various strategies to minimise reference errors. These include submitting the copy of first page of each quoted reference and limitation of the number of references.9,15-17,20 Limiting the number of references in a manuscript may result in omission of important citation and can jeopardise the importance of subject discussed in the manuscript. Electronic editorial management tools and softwares can be used to minimise this error rate. Proofs may be provided to the authors to cross-check prior to publication and, hence, authors should be made responsible for any mistake after that.
The study was confined to two Pakistani medical journals and did not include any international medical journal. The findings, as such, may be reflective of Pakistani journals at best, and may not be considered for other international medical journals. This analysis of two premier Pakistani journals may give a false impression that Pakistani medical journals are generally at par with international standards because most of the other journals do not have strong organisational background and their standards are too low to be compared with these two premier journals.


Reference errors constitute an avoidable but substantial lapse of medical literature. The magnitude of reference errors is much higher than expected. This shortcoming may be rectified by simple and inexpensive steps of carrying out a more careful initial formatting of manuscript and providing the final manuscript to authors for proofreading.


1. Reddy MS, Srinivas S, Sabanayagam N, Balasubramanian SP. Accuracy of references in general surgical journals – an old problem revisited. Surgeon 2008; 6: 71-5.
2. International Committee of Medical Journal Editors. Uniform requirements for manuscripts submitted to biomedical journals: writing and editing for publication. (Online) Updated April 2010. (Cited 2011 October). Available from URL:
3. Mc Lellan MF, Case LD, Barnett MC. Trust, but verify. The accuracy of references in four anesthesia journals. Anesthesiology 1992; 77: 185-8.
4. Gasparyan AY, Ayvazyan L, Kitas GD. Biomedical journal editing: elements of success. Croat Med J 2011; 52: 423-8.
5. Celayir AC, Sander S, Celayir S. Accuracy of references in the pediatric surgery journals. J Pediatr Surg 2003; 38: 653-4.
6. Fenton JE, Brazier H, De Souza A, Hughes JP, McShane DP. The accuracy of citation and quotation in otolaryngology / head and neck surgery journals. Clin Otolaryngol 2000; 25: 40-4.
7. Jackson K, Porrino JA Jr, Tan V, Daluiski A. Reference accuracy in the Journal of Hand Surgery. J Hand Surg [Am] 2003; 28: 377-80.
8. Siebers R. The accuracy of references of three allergy journals. J Allergy Clin Immunol 2000; 105: 837-8.
9. Vargas-Origel A, Gomez-Martinez G, Vargas-Nieto MA. The accuracy of references in paediatric journals. Arch Dis Child 2001; 85: 497-8.
10. Holt S, Siebers R, Suder A, Loan R, Jeffery O. The accuracy of references in Australian and New Zealand Medical Journals. NZ Med J 2000; 113: 416-7.
11. Raja UY, Cooper JG. How accurate are the references in Emergency Medical Journal? Emerg Med J 2006; 23: 625-6.
12. Doms CA. A survey of reference accuracy in five national dental journals. J Dent Res 1989; 68: 442-4.
13. Awrey J, Inaba K, Barmparas G, Recinos G, Teixeira PG, Chan LS et al. Reference accuracy in general surgery literature. World J Surg 2011; 35: 475-9.
14. Nagan Kee WD, Roach VJ, Lau TK. The accuracy of references in the Hong Kong Medical Journal. Hong Kong Med J 1997; 3: 377-80.
15. Asano M, Mikawa K, Nishina K, Maekawa N, Obara H. Improvement of the accuracy of references in the Canadian Journal of Anaesthesia. Can J Anaesth 1995; 42: 370-2.
16. Unver B, Senduran M, Unver Kocak F, Gunal I, Karatosun V. Reference accuracy in four rehabilitation journals. Clin Rehabil 2009; 23: 741-5.
17. Aronsky D, Ransom J, Robinson K. Accuracy of references in five biomedical informatics journals. J Am Med Inform Assoc 2005; 12: 225-8.
18. Adhikari P. Accuracy of references in indexed journals of Nepal. Nepal Med Coll J 2009; 11: 130-2.
19. Hecht F, Hecht BK, Sandberg AA. The journal “impact factor”: a misnamed, misleading, misused measure. Cancer Genet Cytogenet 1998; 104: 77-81.
20. Midarullah, Butt IF, Mubarik A. The accuracy of references in manuscripts selected for publication in Pakistan Armed Forces Medical Journal (PAFMJ). Pak Armed Forces Med J 2008; 58: 299-303.
21. Roland CG. Thoughts about medical writing. XXXVII: verify your references. Anesth Analg 1976; 55: 717-8.

Journal of the Pakistan Medical Association has agreed to receive and publish manuscripts in accordance with the principles of the following committees: