Junaid Sarfraz Khan ( Department of Examinations, University of Health Sciences, Lahore, Pakistan. )
Saima Tabasum ( Department of Examinations, University of Health Sciences, Lahore, Pakistan. )
Osama Mukhtar ( Department of Examinations, University of Health Sciences, Lahore, Pakistan. )
Objective: To compare the awards of various pre-medical academic achievements with aptitude test scores obtained by candidates taking the Entrance Test 2011, and to identify demographic differences in the trend, if any.
Methods: The cross-sectional study involved a 22-item aptitude test administered to all the students taking the Medical College Admission Test conducted by the University of Health Sciences, Lahore, on September 13, 2011. The Matriculation, Intermediate, Entrance Test and Aptitude Test scores of all the students along with their demographic variables were entered into SPSS 16 and statistically analysed by using parametric tests. A p value of <0.05 was considered statistically significant.
Results: Out of the 32,746 students, 22,935 (70%) were females. Students who scored more than 75% marks (n=4,723; 14%) also scored higher marks in the Aptitude Test (p <0.05). Female candidates in general scored higher marks in the Entrance Test (p<0.05) and in the Aptitude Test (p<0.05). Overall, students from the economically and academically underdeveloped districts of Punjab scored less marks in the Entrance Test (p<0.05) and the Aptitude Test (p<0.05). The difference in Entrance Test and Aptitude Test marks of students from underdeveloped districts was greater than that of students for the developed districts (p<0.05). However, in Matriculation and Intermediate examinations, students from the underdeveloped districts scored higher marks than the developed districts (p<0.05). Students scoring higher marks in Matric and Intermediate, scored low marks in the Aptitude Test (p <0.05).
Conclusion: The Entrance Test and Aptitude Test scores correlated poorly with Intermediate and Matriculation scores. There is a need to strengthen the underdeveloped areas academically. Besides, the predictive value of the Aptitude Test scores related to future performance of candidates selected needs to be ascertained with further studies.
Keywords: Aptitude Test, Entrance Test, Medical and Dental graduate programmes, Pre-medical academic achievements, Underdeveloped districts. (JPMA 63: 552; 2013).
Selection of candidates for enrollment into the graduate medical and dental programme has to be made through reliable, just, credible, transparent and fair methods, especially when the majority of the applicants are suitably academically qualified to enter into the realm of professional medical education.1 In both the developed and the developing worlds, the competition for selection amongst the candidates is intense. As selectors, individuals, organisations, institutions and governments need to justify their choices not only to the public and the students, but to themselves as well. The selection choices made at any given point affect the medical education system and environment and the healthcare delivery system nationally and internationally for decades to come. This is because, for any given merit-based seat/vacancy in health professional education, there are up to 10-15 prospective candidates, again all academically suitable to fill in that vacancy. In the Pakistani province of Punjab, each year up to 35,000 candidates scoring 60% and above marks in their Higher Secondary School Certificate (HSSC) examination are eligible to be accommodated against the 3,500 public and private medical and dental education seats in the 32 institutions across the province, which has a total population of 80 million.2
Worldwide, a number of methods and processes are applied for selection for undergraduate admission into professional medical education. They range from the widely used pre-medical academic achievements to aptitude testing and interviews of applicants, to a combination of these in one way or the other.3 In one country in South America, candidates are selected through a system of national lottery. Given the stakes involved, it is no wonder that whatever selection criteria and methodology is applied, controversies abound. The paradigm shift away from the use of pre-medial achievement scores alone for selection has been slow; slower still in the developing world, where aptitude testing has not really taken root and interviewing has been fraught with problems of subjectivity, lack of standardisation, limited physical resources and time constraints.
In Punjab, Medical Colleges Admission Test (MCAT) has been used as a compulsory weighted component of the selection process since 1998. Currently its scores weigh in at 50% of the selection criteria, with Matriculation and HSSC scores representing 10% and 40% of the weighted total. MCAT in Punjab is not without controversy. Public, students and various organisations criticise the weight MCAT pulls into the total score considering it as a one-off 150-minute test using 220 MCQs (30 English; 70 Biology; 60 Chemistry; and 60 Physics) from the HSSC syllabus, when compared to the more rigorous and extensive evaluation processes lasting over many hours every day for months in the Matriculation and HSSC examinations which pull in only up to 50% weight in the total.
The paradigm shift away from the exclusive use of pre-medical achievement scores as the admission criteria resulted in the introduction of Entrance Tests. In some countries, such tests alone were used for selecting candidates without taking any other variables, like pre-medical achievement, into consideration. With such a large number of candidates, all potentially suitable, competing for limited seats, this shift alone was found to be lacking in justifying the selection. Aptitude tests were then introduced as a separate weighted factor in the selection process in many developed countries like the US, UK and Australia, designed to assess general attributes and abilities gained through pre-medical experiences and studies, including critical thinking, critical analysis, problem-solving, professionalism, empathy, understanding, context and relevance and abstract non-verbal reasoning. Whether these abilities, skills and knowledge have any relevance to professional health education and especially, even if it is relevant, what relevance the timing (pre-medical, medical, post-medical) in their acquisition has in differentiating the good medical student/practitioner form an average one, still needs to be ascertained. There is also at present limited published evidence available on how, if at all, pen-and-paper aptitude testing can be related to aptitude testing through interviews, or whether aptitude tests or interviews are mutually complementary or can one replace the other. This though is clear that the additional tests, interviews and protocol of admission place a huge psychological burden on the community, especially the students, which is not without its share of economic burden.4
Nevertheless, for the sake of an argument, it sounds reasonable to select candidates for professional health education not only on the basis of their pre-medical academic achievements, but also skills and attitudes that are critical for this profession. The skills to communicate with peers, faculty, public and patients, including both verbal and non-verbal cues; the ability to present oneself and one\\\'s thoughts appropriately and to make a positive and empathic impression vis-à-vis \\\'a caring attitude\\\' can perhaps only be \\\'measured\\\' using interviews/ observations. Structured interviews, including the semi-structured mini-interviews, gained considerable popularity in countries which had the resources, both economic and human, to conduct them, but it was evident form the very beginning that no matter how structured the interviews got, subjectivity to some extent remained its Achilles heel. Some institutions and countries, therefore, having started it, are now completely abandoning the interview process.5 And yet the skills, attitudes and knowledge that were being tested through these interviews cannot for the present be tested by any other means.
Medical education is subsidised by the government in all public-sector profession health education institutions in Punjab (3488 seats).6 The private sector charges upto 15000 US Dollars per annum for medical education from candidates selected on merit and using the guidelines laid down by the Pakistan Medical and Dental Council (PMDC).7
In order to strengthen the admission process, aptitude testing was piloted in the MCAT 2011. All the 33,694 candidates sitting in the MCAT 2011, were also required to respond to a 22-item, Aptitude Test that looked for ethical, moral, professional and personality attributes, quantitative and abstract reasoning and analytical reasoning and problem-solving skills.8
The purpose of the study was to compare the awards of various pre-medical academic achievement scores with Aptitude Test scores obtained by the candidates sitting in the MCAT 2011 and to identify gender and demographic difference in the trends if any.
Subjects and Methods
The cross sectional study comprised a 22-item Aptitude Test which was administered to all the students taking the MCAT 2011 which was held on September 13, 2011, by the university of Health Sciences, Lahore, for the government of Punjab.
The Aptitude Test was designed by a team of psychologists, behavioural scientists, psychiatrists, health professionals, educationists and senior and junior basic sciences and clinical faculty members at the University of Health Sciences, Lahore. The test was administered to another group similar to the one above and there followed between the groups, three, one-hour discussion sessions to validate individual items based on the responses. The candidates knew that the Test would not, in 2011, have any effect on their selection or otherwise.
The response form were read through Optical Mark Reader (OMR) machines and the scores of students together with their Matriculation and HSSC scores and their demographic and gender variables were entered into SPSS 16 and analysed by using Student\\\'s t-test and Analysis of Variance (ANOVA) parametric tests for the comparison of different variables. All these scores were adjusted for the purpose of analysis. A p-value of <0.05 was considered statistically significant.
Overall, 32,746 students took the Entrance Test 2011, of which 22,935 (70.04%) were females and 9,811 (29.96%) were males. Only 4,723 (14%) candidates scored more than 75% marks.
Candidates who scored more than 75% marks in the Entrance Test also scored higher marks in the Adjusted Aptitude Test (t=29.99; p <0.0001).
Gender comparison of the Entrance and Adjusted Aptitude Test marks showed that female candidates in general scored higher marks in the Entrance test (p <0.05) and in the Aptitude Test (p <0.05) (Table).
A significant difference was found in the mean Entrance Test marks (F= 26.373; p <0.0001) and Aptitude Test marks (F=44.33; p <0.0001) of candidates of different localities. Overall, students from economically and academically underdeveloped districts of Punjab scored less marks in the Entrance Test and the Aptitude Test. The difference in marks of students from the underdeveloped districts was greater than that of students from the developed districts (p<0.05) (Figure-1).
Significant difference was also found in Adjusted Matriculation (F= 8.57; p <0.0001) and HSSC (F= 21.43; p <0.0001) marks of the candidates from different localities. However, in Matriculation and HSSC examinations, students from the underdeveloped districts scored higher marks than the developed districts. Students overall scoring higher marks in Matric and HSSC scored low marks in the Aptitude Test (Figure-2).
By law,9 all public and private medical, dental and allied health sciences institutions within the geographical boundaries of Punjab, which has an estimated population of 80 million, have to be affiliated with the University of Health Sciences, Lahore (UHS). At present, there are 80 institutions affiliated with the UHS,10 which also conducts the MCAT on behalf of the provincial government every year in summer/autumn. The candidature consists of HSSC students with at least 60% marks and are \\\'residents\\\' of the province by birth. The total population of students every year varies between 30,000 to 35,000 candidates. Currently, according to PMDC rules,10 students are admitted through merit in the public institutions based on the following formula:6
50% MCAT score +10% Matriculation score + 40% HSSC examination score.
There is no gender quota and the admissions are on an equal opportunity basis. Nearly 70% of the admitted candidates each year are females.
The need for the MCAT arose due to the non-standardisation of the evaluation and assessment amongst the nine Boards of Intermediate & Secondary Education (BISE)11 across the various districts of Punjab, and the differences in the academic and socio-economic facilities and conditions amongst the districts. MCAT was introduced by the government of Punjab in 1998.12 MCAT has been criticised in the past for testing factual knowledge again, since it is made up of 220 Multiple Choice Questions (MCQs) from the HSSC course and syllabus. The UHS initiative to introduce aptitude testing and weigh it with pre-medical achievement scores and MCAT scores in the selection process was postponed by the government. However, permission was granted to run the test without weighing it into the selection process in any way.
In the study, female candidates scored higher marks in the MCAT as well as in the aptitude test than the males. Our results for the academic achievement scores for the females compared to the males were similar to findings reported in other studies.13,14 This, given that the Male:Female population ratio of Punjab is 1.056 is even more striking.15 Not much literature is available on the reasons as to why females achieve academically better results, but findings of an earlier study16 support the argument that gender specifications should always be taken into account when analysing academic-related data or constructing academic policies. Can the healthcare delivery system support up to 70% female medical and dental graduates every year in Pakistan or any other developing country with a huge healthcare burden, especially in rural setting, is a question future research needs to answer. The admission criteria and polices then need to be refined keeping them in line with the strategic planning for improving the healthcare delivery in the province.
One study reported that females in the UK in Biomedical Admission Test (BMAT) where women scored lower than males in both the aptitude and skills component (Section 1) and the scientific knowledge and applications component (Section 2) of the BMAT.17 This contrasts with the findings in our study. The study argued that the finding could be due to the widening participation schemes in UK, with the male applicants probably belonging to a higher portion of their wider population\\\'s ability distribution than females. We can present a similar argument to support our contrasting result. Females belonging to higher academic and socio-economic status enter into HSSC education, whereas far more males belonging to the lower academic and socio-economic backgrounds enter into secondary and post-secondary education. This could be coupled with the socio-economic conditions in the province with the males having to study and work at the same time to support themselves while the females can concentrate only on their academics, leading them to secure higher marks and achieve better than the males.18
Our study showed that there was a significant difference in both academic and aptitude scores between candidates from socioeconomically developed and underdeveloped districts of the Punjab, with candidates from the developed districts securing higher marks. This was a reversal in trend seen in 2011 compared to the MCAT results previously reported for Punjab. This finding was also in contrast to the fact higher Matriculation and HSSC scores were reported by students of underdeveloped districts in the study. MCAT, as mentioned already, was set up in Punjab due to the sub-standardisation of education and evaluation amongst the BISEs11 in various districts. It is possible that BISEs in the underdeveloped districts do not follow the same stringent evaluation criteria as those followed by Boards in the developed districts. Then again, in the underdeveloped districts, the more academically and socio-economically robust candidates seek higher secondary education leading to a higher mean score for the underdeveloped district cohort i.e. fewer more academically sound students compared with a larger group of students in the developed districts belonging to all academic and socio-economic backgrounds. Further studies are required to establish this argument. The finding that students from higher social and economic groups were better in Aptitude Test, came as no surprise and has been reported in other studies as well.17,19 The 22-item MCQ test was administered in English and the candidates did not have any prior training, teaching or guidance in attempting the test. It is likely that the language-barrier, especially for candidates in the underdeveloped districts, could have resulted in candidates from there securing lower Aptitude Test scores. Further studies requiring validity of the Aptitude Test in the mother tongue is required. Otherwise, the test shall probably always be biased towards candidates with better command over the language from the developed districts\\\' English Medium educational institutions and socio-economic backgrounds.
Nevertheless, the finding that on average, the difference between the academic and aptitude scores was wider amongst candidates of the under-developed districts, is important to consider here and cannot be explained just on the basis of the language divide.20 Better education facilities are available in socio-economically affluent districts, which allow development of skills related to critical thinking, analysing, problem-solving and abstract thinking in students. Once selected in medical education, all candidates will be guided in achieving these skills in the course of their medical education. Can aptitude testing, when it can be related to socio-economic factors, further deprive students from socio-economically poorer backgrounds from higher education. It is a question that should concern us in deciding on further changes, if any, in the admission criteria.
With limited resources, merit-based selection of candidates for the Medical and Dental programmes is crucial to improve the healthcare delivery system. Entrance Test and Aptitude Test scores correlated poorly with HSSC and Matriculation scores which is a worrying sign. Besides, underdeveloped areas need to be strengthened.
1. Powis D. Selecting medical students. Med J Aust 2008; 188: 323-4.
2. "Punjab - World Gazetteer". (Online) 2010 (Cited 2012 March 8). Available from URL: http://www.world-gazetteer.com/ wg.php?x=&men=gpro&lng= en&des=wg&geo=437641435&srt= pnan&col=abcdefghinoq&msz=1500&geo=-2943.
3. Mercer A, Puddey IB. Admission selection criteria as predictors of outcomes in an undergraduate medical course: a prospective study. Med Teach 2011; 33: 997-1004.
4. Ladd MAB, Thomas KF. What\\\'s at stake in High-Stakes Testing Teachers and Parents Speak Out. J Teach Edu 2000; 51: 384-97.
5. Wilkinson D, Zhang J, Byrne G, Luke H, Ozolins I, Parker MH, et al. Medical school selection criteria and the prediction of academic performance. Med J Aust 2008; 188: 349-54.
6. Government of the Punjab. Prospectus for admission to Government Medical and Dental Colleges of the Punjab Session 2010-2011. University of the Health Sciences Lahore.
7. Pakistan Medical & Dental Council: Conditions for admission in MBBS/BDS courses and conditions for house job regulations 2010. Islamabad: 2010; 1-12.
8. University of Health Sciences. Syllabus for entrance test 2011, University of Health Sciences. (Online). Pakistan: 2011 (Cited 2011 Jan 16). Available from URL: http://www.uhs.edu.pk/mcat/ ETsyllabus2011.pdf.
9. The University of Health Sciences, Lahore Ordinance 2002 (Ordinance LVIII of 2002). (Online) (Cited 2011 March 5). Available from URL: http://punjablaws.gov.pk/laws/446.html.
10. University of health sciences. (Online) 2011 (Cited 2011 Jan 16). Available from URL: http://www.uhs.edu.pk/.
11. Pakmed: Boards of Intermediate & Secondary Education in Pakistan. (Online) Pakmed Board Home: (Cited 2011 August 16). Available from URL: http://www.pakmed.net/board/.
12. Students inn: serving for better future. (Online) 2011 (Cited 2011 August 16). Available from URL: http://www.studentsinn.edu.pk/ Aboutus.html.
13. Sharpe R. As leaders, women rule. Businessweek online [serial on the internet]. 2000 November 20; (Cited 2011 March 6); [about 12 screens]. Available from URL: http://www.businessweek.com/ 2000/00-47/b3708145.html.
14. Fernardo N, Prescott G, Cleland J, Greaves K, Mckenzie H. A comparison of the United Kingdom Clinical Aptitude Test (UK-CAT) with a traditional admission selection process. Med Teach 2009; 31: 1018-23.
15. World Census Organization. Census of population: 2011 (Online): Islamabad-Pakistan (Cited 2011 August 16). Available from URL: www.census.gov.pk.
16. Ferguson E, james D, Madeley L. Factors associated with success in medical school: systematic review of the literature. B Med J 2002; 324: 952-7.
17. Emery J L, Bell H F, Vidal Rodeiro CL. The biomedical admission test for medical student selection: Issues of fairness and bias. Med Teach 2011; 33: 62-70.
18. Khan J S. Facts regarding Medical Colleges Admission Test (MCAT) in Punjab, Pakistan: The good, the bad and the ugly. Presented in "Trends in Medical Education - 6th Congress of Asian Medical Education Association (AMEA 2011)", Kuala Lumpur, Malaysia; 2011.
19. McManus IC, Powis DA, Wakeford R, Ferguson E, Jmaed D, Richards P. Intellectual Aptitude tests and a levels for selecting UK school leaver entrants for medical school. BMJ 2005; 331: 555-9.
20. Geo L. The link between teacher quality and student outcomes: a research synthesis [PhD Thesis]. Washington, National comprehensive Center for Teacher Quality; 2007.