By Author
  By Title
  By Keywords

April 1996, Volume 46, Issue 4

Original Article

Critical Appraisal by Reading for Medical Students - A Case Study from Pakistan

F. F. Fikree  ( Department of Community Health Sciences, The Aga Khan University, Karachi. )
D. R. Marsh  ( Department of Community Health Sciences, The Aga Khan University, Karachi. )

Abstract

Doctors read literature to keep abreast of medical advances. A recommendation from the 1993 World Summit for Medical Education is that medical schools should teach medical students to critically appraise scientific reports. The Department of Community Health Sciences of Aga Khan University Medical College teaches basic research methods to medical students. This is now supplemented with “Critical Reading”.
Critical reading was first taught to 67 third yearstudents between October, 1993 and May, 1994. A “validity check-list for critical readers” was introduced in a two week orientation consisting of three one-hour classroom sessions and four one-hour small group sessions. Thereafter, small groups met monthly to critique clinical epidemiological reports relevant to current organ system teaching. The students reading attitudes and critical appraisal skills were assessed through continuous assessment and a written final examination with questionnaire. All but three students passed the final examination (mean score (74%, standard deviation 12%). Sixty-four of 67 (96%) completed questionnaires. All (73% strongly) agreed that critical reading skills were essential, but only 30% strongly agreed that they had, indeed, mastered the skills. Ninety-seven percent (56% strongly) disagreed that year three was too early to start critical reading. Clinical teaching staff expressed interest in learning these skills. Students benefited from and enjoyed this first critical reading course. It strengthened ties between clinical and community health sciences teaching staff. The critical reading skills of the clinical teaching staff is being addressed in seminars to strengthen institutional research capacity (JPMA 96: 80, 1996).

Introduction

Medical schools traditionally impart to students a body of information from basic sciences and clinical sciences. Ironically, this central role of science is principally that of a “fact fount”. But these facts themselves are open to interpreta­tion, revision and even rejection. Thus, medical educators world-wide increasingly insist that medical graduates have sufficient scientific background to critically appraise medical literature, the practical medium of new information. The recommendations froni the 1993 World Summit for Medical Education stated that: Science lies at the heart of an effective curriculum. The methods of science are essential to defining problems and measuring impact of intervention in the care of both patients and communities. The curriculum as a whole requires scien­tific and critical thinking, the basis of evidence-based medi­cine. [Curricula should] emphasise economic, statistical, managertial and informational sciences as relevant to clinical work [so as to produce] a multi-science-based practitioner who can remain abreast of the scientific advances and understand their relevance to the field of practice1.
The Aga Khan University (AKU), the first private University in Pakistan, teaches community-based, problem-solving, community health science to its medical students2. The University devotes twenty percent of the first two years\\\' pre-clinical curruculum to community medicine including courses in practical biostatistics and epidemiology with field application. We introduce the beginning their clinical education in year three. Despite a University movement to wards a horizontally and vertically intergrated curriculum, joint educational activites between the Department of Community Health Sciences and other basic or clinical science disciplines have remained elusive. In part, this is becuse basic science, clinical and community health teaching staff are unsure where their respective activities interest and how they can be mutually strengthened.
To develop its junior teaching staff, the Department of Community Health Sciences has conducted a monthly critical reading journal club for three years. We incorporated many of the principles reported by others’ descriptions of post-gradu­ate journal clubs: use of check-list, participants presenting in rotation and emphasising methods before contene3-5. Despite the experience of several western medical colleges in teaching critical reading to undergraduates6-8, we hesitated until: 1. the World Summit for Medical Education espoused the universal teaching of critical appraisal of scientific literature at the undergraduate level; and 2. local curriculum revision of year three provided an opportunity to introduce critical reading and describe its impact.
By third year, the students had already mastered basic quantitative methods and terminology. Under faculty guid­ance, they had formulated researchquestions, designed survey protocols and collected and analysed data. Our mission, then was to facilitate theirbringing this spirit of scientific inquiry to reading clinical reports and ultimately to clinical care. Our learning objectives were for students to develop: (1) a method of choosing pertinent articles to read; and (2) a systematic approach to evaluating published scientific literature. The purpose of this paper is to report our preliminary experience.

Method

The classes at Aga Khan University are small and the selection pmcess is rigorous. Twenty-six hundred of the 3000 applicants have the academic qualifications to take an admission examination. Two hundred and fifty top applicants are then interviewed, of whom, only 70-80 are chosen. Consideration is given to those from remote areas.
All 67 entering third year students (42 boys and 25 girls, average age 20 years), graduating class of 1996, participated inthe critical reading course conducted between October1993 and May, 1994. This first course offering of critical reading consisted of two components: an intense introduction during the first two weeks and a continuation phase over the remaining eight months. The introduction had seven one-hour components: three plenaiy sessions and four small group tutorials. The plenary sessions: (1) justified proficiency in evaluating medical literature as alife-long skill; (2) introduced a critical reader’s validity check-list (Table I)

(3) explained the organisation of scientific reports and (4) reinforced epidemiological and biostatistical points which had been introduced in the preceding two years.
Small group tutorials gave the students experience in critical reading guided by the check-list. This guide helps less experienced readers evaluate a paper’s conclusions (internal validity) and the generalisability of these conclusions (exter­nal validity). The check-list has 35 points, such as clarity of conceptual framework, subject sampling method and study limitations. Students write comments for each in the space provided to prepare for the tutorial.
Two papers were methodically reviewed simultane­ously over the first three tutorials, one each on abstracts and introductions, methods, and results and discussion. The two papers were a community-based risk factor assessment for childhood malnutrition and a clinical epidemiology report assessing the validity ofjunior doctors’ diagnoses of myocar­dial infarction. They were chosen to underscore the applicabil­ity of the check-list’s points to both community and clinical reports. During the fourth introductory tutorial, the entire paper a report addressing the clinical presentationof ill infants was reviewed. Clinical, epidemiological papers were chosen for the remainder of the course.
The University recently integrated the third year cur­riculum into organ system blocks9. Thus, a continuation phase of the critical reading course was planned for the six largest organ system blocks: respiratory, gastroenterology, infectious diseases, urology, cardiology and neurology. A different faculty tutor took responsibility in turn for preparing each tutorial. In collaboration with clinical counterparts, he or she chose a paper relevant to block objectives, to the interests of involved clinical faculty and to the students’ methodological ability. The tutor then organised a preview with the other course faculty using the Delphi method4. Each independently reviewed the article with the same check-list used by the students. Then all met to reach a consensus regarding the best responses to each point and to identify main teaching issues. Meanwhile, students read the papers and completed their check-lists. Each tutorial consisted of five small groups of 10-13 students each. These sessions took different formats within the framework of the check-list. Each student might publicly “grade” the paper prior to its detailed critique. The tutor might facilitate an orderly discussion “down the check­list”, or he might pre-select individual students to take responsibility for major sections of the review.
Students evaluated the course at the end of the first month and again at the end. The students’ critical reading ability was evaluated at the end of the course using the paper assigned for the neurology block. Students received 10 marks for a take-home completed check-list. The remaining 90 marks were awarded on the basis of an open-book. multiple-choice examination based on their assessment of the likely validity of the paper’s conclusions and their applicability to other settings.

Results

At the end of the introductory sessions, 23 students (4-5 each of the 5 small groups) were asked to complete an anonymous questionnaire. Overall, 63 percent of the students graded the module as good; no student reported that the sessions were poor. They wrote that the sessions taught them a “new perspective” whichcouldbe “important forthe informed doctor” and would “assist them in their future careers”. However, the students also wrote that the “technical aspects (especially of biostatistics) were at times complicated” and requested “more time for small group discussions”.
At the end of the course, each student was asked to complete a more detailed anonymous questionnaire about the course. Sixty- four of 67 (96%) returned completed forms (Table II).

All (73% strongly agreed that acquiring critical reading skills was essential. Far fewer (30% strongly) agreed that they had, indeed, learned how to “assess the truth of a scientific paper”. Almost all (56% strongly) disagreed that year three was too early to begin to learn these skills. The vast majority (9 1%) found the small group sessions useful. Most (75%) planned to use the check- lists for future reading. Almost half (48%) reported that the papers were uninteresting. Reported student comments included: “avoid afternoon group sessions”, “choose better papers” and are there no good papers?!”
The students’ critical reading abilities as measured on the final examination showed that all completed take-home check-lists and all but three (96%) passed the in-class exercises with a mean score of 74%±l2% (1 standard deviation).

Discussion

This preliminary report describes The Aga Khan Uni­versity ‘s first attempt to teach the critical appraisal of scientific literature to medical students. It will be interesting to learn about similar experiments in other developing country medi­cal colleges.
Audet et al10 have recently critically reviewed 10 reports of teaching critical reading in western settings. They found methodological weaknesses so prevalent that the concluded: “the effectiveness of teaching critical appraisal of the literature remains uncertain. More rigorous methods are needed in the research in this area”. A “critical reading” of this report is no exception. Given its non-experimental, post-course only design, AKU students could observe that assess­ing the course’s impact is hampered by the lack of pre-course measurements and a comparison group. On the other hand, the initial student feedback noted that critical reading skills were difficult despite their prior experience with quantitative methods. Although not a baseline measurement, the notion that skills improved by the final examination is somewhat supported.
The purpose of this report is not primarily to demon­strate impact, but to (1.) describe the course; (2) detail the facilitating institutional context; and (3) show that it is popular and suggest that it is probably valuable.
Assessing its value should also include measuring other unanticipated outcomes. The principal boon has been in­creased communication between the Department of Commu­nity Health Sciences and clinical departments. Several clinical teachers remarked during the planning and implementation of the critical reading sessions that they wished they knew what the students were being taught! Indeed, the Department of Community Health Sciences has responded to this widely observed need by consulting to clinical departmental journal clubs and offering a year’s fortnightly course on quantitative methods, including critical reading, for AKU teaching staff. The average attendance to these fortnightly, 2-hour weekend sessions is 49. including: clinical (44%), communit health (23%), basic science (7%) and nursing (6%) faculty; admini­stration personnel (5%); arid student and visitors (15%).
Students rightly objected to “over-critiquing” papers. It was felt that one of the aims of the critical reading course should be to teach students to distinguish carefully between scientists’ practical constraints and imperfect thinking. The students should be taught to differentiate between minor, major and “fatal” flaws in reasoning. Occasionally metho­dologically unsound papers were chosen to demonstrate teaching points. Perhaps the same points could be as well, if not better, taught from excellent papers. Indeed, almost all papers have redeeming points and the students’ are correct in so reminding us.
A more complete assessment of the course is in progress. Reports will follow on changes in third year medical students’ reading habits and critical appraisal after the critical reading course. Additionally, reading practices and attitudes will be compared among departing fifth year medical students (who have not had the course) and the current fourth year medical students who did.

Acknowledgements

We are indebted to Professors Henry J. Walton, Joseph B. McCormick and N. Jafarey foruseful suggestions onearlier manuscripts. We also acknowledge our fellow tutors, past and present: Drs. P. Nayani, S.N.B. Inam, M. Lobo, M. Kadir, S. Sadruddin, R.W. Zuberi and G. Perveen,

References

1. Walton, H. B. World Federalion for Medical Education. Proceedings of the World Summit on Medical Education. Medical Education, 1993;28f,Supplement l):146.
2. Bryant, J.H., Marsh, DR., Khan, KS. et al. A developing country’s university oriented toward strengthening health systems: challenges and results. Am. J. Public Health, 1993;83:1537-43
3. Gehlbach, S.H.,Bobula,J.A. andDickinson, J.C. leaching reaidenta to read the medical literature. 3. Med. Educ., 1980;55:362-5.
4. Linzer, M, Browen, J.T., Frazier, L.M. etal. lmpact of a medical journal club on house-staff reading habits, knowledge and critical appraisal skills: A randomized control trial. JAMA., 1988;260:2537-41.
5. Thumau, G.R. and Fiahburne, J.1. Format of an obstetrics and gynecology journal club and fouryeara experiencc. Am. 3. Obstet. Gynecol., 1989;160:313-6.
6. Riegelman. R.K., Povar, G.H. and Ott, J.E. Medical students skills, attitudes and behavior needed for literature reading. 3. Med. Educ., 1 83;58 :411-7.
7. Reigelman, R.K. Effects of teaching first-year medical students skills to read medical literature. J. Med. Educ., 1986;61 ,454-60.
8. Bennett, K.J., Sackett, D.L., Haynca, RB. et al. A controlled trial of teaching critical appraisal of the clinical literature to medical students. JAMA., 1987;257:2451-454.
9. Ham, T.H. Medical education at Westem Reacrve University. N. EngI. J. Med., 1962;267:868-74.
10. Audet, N., Gagnon, R., L adouceur, R. et al. Enseignement del analyse critique dea publication acicntifique mcdicalea est-il cificace? Revision dea etudes et deleur qualite methodologique. Can. Med. Aaaoc. J., 1993:148:945-52.

Journal of the Pakistan Medical Association has agreed to receive and publish manuscripts in accordance with the principles of the following committees: