Junaid Sarfraz Khan ( University of Health Sciences, Lahore, Pakistan. )
Osama Mukhtar ( University of Health Sciences, Lahore, Pakistan. )
Saima Tabasum ( University of Health Sciences, Lahore, Pakistan. )
Objective: To identify the strengths and weaknesses of the current selection process for admissions to public medical colleges being followed in the province of Punjab and to suggest changes.
Methods: The prospective study comprised candidates sitting the Medical and Dental Colleges Entrance Tests administered by the University of Health Sciences, Lahore, from 2008 to 2011. The marks of pre- and post-admission examinations were entered in SPSS 17 and performance of candidates in different examinations up to 2012 was compared using parametric statistical tests. P<0.05 was considered statistically significant. Reliability of the entrance test question paper was calculated through Cronbach\'s Alpha.
Results: Overall 1,01,273 candidates sat in the entrance test during the study period, and of them 14, 995 (14.8%) students were admitted to public and private medical colleges in Punjab affiliated with the University of Health Sciences, Lahore. Cronbach\'s Alpha showed excellent internal consistency reliability of the question paper of 220 items for all these four years. Overall, students scored significantly lower marks (p<0.05) in all their MBBS Professional Examinations than their pre-admission tests. A similar pattern of difference was seen in entrance test marks in different geographical localities as in the case with MBBS Professional Examination. Students from Islamabad and Rawalpindi regions scored significantly higher marks in all MBBS examinations as well as in the entrance test and aptitude test, but lower in the Higher Secondary School Certificate examination. Internal consistency reliability of the aptitude test was found to be 0.83. Linear regression analysis showed that Estimated Regression Coefficient of the Higher Secondary School Certificate marks was negative, but positive for entrance test and indicated that the latter results were a good positive predictor for the post-selection performance. In the sub-components, only the Biology and Chemistry regression coefficient section was found to be positive.
Conclusions: Entrance and aptitude tests in Punjab were found to be valid and reliable with incremental predictive validity at least for performance in the pre-clinical years. It is suggested that Higher Secondary School Certificate -marks should have little, if any, weightage in the selection process.
Keywords: Medical & Dental Colleges Admission Test, Aptitude test, Health, Professional education, Selection process, Predictive validity. (JPMA 64: 1132; 2014).
Professional health educational institutions in Pakistan are faced with the dilemma of having a limited number of seats against a disproportionately high number of potential candidates, all seemingly suitable for admission into medical and dental colleges. In Pakistan, medical and dental colleges can be divided into two categories — private and public. Public medical colleges are under the direct control of provincial governments and professional health education in these institutions is subsidised and students pay a very nominal tuition fee. Both public and private medical colleges require accreditation, licensure and certification by the provincial governments, and the Pakistan Medical and Dental Council (PMDC), which is the regulatory authority for medical education in the country as well as by respective degree-awarding universities. In the province of Punjab, currently there are 14 public and 18 private medical colleges with a total number of 3,305 seats of which 2,980 are in the public sector.1 For admissions in both private and public medical colleges, candidates who have scored over 60% marks in the Higher Secondary School Certificate (HSSC) examination are required to sit in the Entrance Test (ET) organised by the respective provincial government in the province of Punjab which has a population of over 80 million.2 The ET is conducted for the provincial government by the University of Health Sciences (UHS), Lahore in September each year. The ET consists of 220 multiple choice questions (MCQs) of single best type with four options and the paper is divided into four sections with 88 questions in Biology, 44 in Physics, 58 in Chemistry and 30 in English.3 Each correct answer by a candidate carries 5 marks and each incorrect answer a -1 mark. The duration of the exam is two-and-a-half hours. There is also a non-weighted additional half-an-hour test termed the \'Aptitude and Feedback\' test (AT) comprising 60 questions of which 8 are related to feedback regarding the conduct of the ET, 30 are related to personality assessment and the rest to aptitude assessment with a mix of questions on ethics, critical thinking and abstract non-verbal reasoning.
The problem of mismatch between the available numbers of suitable candidates is not unique to Pakistan. In fact the situation has been the same all over the world for many decades.4,5 To justify selection, various countries have adopted different strategies.6-12 For example, in the Netherlands, the selection is based partly on institutional selection procedures (up to 50%) and partly on a national lottery weighted for school performance.4 In Australia, they use three components in the selection process; the pre-medical academic score, the Undergraduate Medicine and Health Sciences Admission Test (UMAT) and a selection interview.5 In the United Kingdom, clinical aptitude test assessing cognitive ability in verbal reasoning, quantitative reasoning, abstract reasoning and decision analysis is part of the selection process which may include interviews, pre-medical achievement scores and letters of reference.13,14
The purpose of the selection procedures adopted in the developed countries is to assess cognitive abilities which reflect intellectual capabilities, factual recall and performances of test on declarative knowledge, while another component commonly known as the non-cognitive component is gaining popularity in the selection process and includes personal qualities, attributes, attitudes, interests and values.14 Of these, good communication skills have been found to be an important attribute for both medical students and doctors, whereas other cognitive attributes and pre-medical achievement scores are good predictors for performance in the early non-clinical years of medical education.5 Similarly, personality traits like openness, agreeableness, emotional stability and extroversion have been found to be good predictors of better performance in the clinical years of medical education and in professional practice.14
The province of Punjab is vast and potential candidates come from varying socio-economic and educational backgrounds. Pre-medical academic achievement amongst the candidates in the Punjab has been found to be unreliable in predicting future performance of medical students.15 Pakistan is a developing country which requires rationalisation of resource allocation in all areas of governance, healthcare, education etc. For Pakistan and its population, it is therefore important to choose the best possible candidates for admission in professional health education in order to maximise the impact of professionals on the healthcare delivery system.15
The aim of this study was to identify the strengths and shortcomings of the current selection process for admissions in public medical colleges being followed in Punjab and to suggest changes in order to improve the quality of the selection based on the Best Evidence Medical Education (BEME) practices worldwide.
Material and Methods
The prospective study comprised candidates sitting the Medical and Dental Colleges Entrance Tests administered by UHS, Lahore, from 2008 to 2011. Reliability of ET question paper was calculated using Cronbach\'s Alpha. Students who qualified for admission in medical colleges affiliated with UHS all over Punjab following the ET were the focus of the study. The marks of pre-admission examinations Secondary School Certificate (SSC), HSSC, ET and AT and post-admission examinations (MBBS Professional Examinations) were entered into SPSS 17. The pre- and post-admission performances of these candidates in different examinations up to 2012 were compared using paired sample t-test. Marks of all MBBS Professional Examinations and SSC marks were adjusted to make the total equivalent to 1100 as per the ET and HSSC marks. Analysis of Variance (ANOVA) was used to compare the performances of candidates from different geographic localities. The whole cohort was distributed in 11 different geographical localities; Nine (09) regions (Divisions) of Punjab which consisted of 36 districts and two others were Islamabad and Azad Jammu and Kashmir. There were 6 districts that were considered underdeveloped i.e. D.G. Khan, Muzaffargargh, Rajanpur, (included in D.G. Khan division), Layyah (included in Multan) and Bhakkar (included in Sargodha). Regression analysis was used to determine the relationship between the performance of the students in MBBS Professional Examinations and their pre-admission examinations. P<0.05) was considered statistically significant.
Selection processes, procedures, protocols, methodologies etc, adopted over the years in various countries and regions have not been without their controversies.3,16-18 These controversies arise as a result of the high political, societal and psychological stakes involved in conducting admission or selection tests. In Pakistan, in the province of Punjab, around 36000 candidates secure over 60% marks in the HSSC examination. The seats available for admissions in public medical colleges are limited to only 2980, so all these 36000 candidates cannot be admitted. A simple lottery weighted for academic achievement as is the case in the Netherlands is difficult to justify when faced with this huge volume of applicants. The same holds for other auxiliary methods of selection like letters of references or intent and semi-structured mini-interviews.2,3 The authors believe that interviews alone for 36000 candidates will not be cost-effective and will be beset with the problems of lack of standardization, inter-rater bias, gender and demographic bias and political influences. With reported evidence of poor-reliability of pre-medical academic achievement for predicting future performance in health professions education, this variable alone also cannot be the sole determining factor for selection. ET for selection of candidates for admission in medical colleges in Punjab has been in place since 1998. The Government of Punjab handed over the conduct of ET to the University of Health Sciences, Lahore (UHS) in 2008. And to date, the UHS has conducted five regular ETs for the province of Punjab for admission of candidates in health professions education. The syllabus for 2010 was revised to reflect the distribution presented in the introduction section together with introduction of a non-weighted \'Aptitude and Feedback\' test (AT). Since the population for the test comes from varied educational backgrounds including both the Punjab HSSC Board education and other educational systems, national as well as foreign, like for example Advanced Level of UK (A-level), a syllabus was developed by the UHS with learning objectives derived from commonalities in the various syllabi. The syllabus and the table of specifications derived from the syllabus reflect the crystallized intelligence derived from the premedical education streams. This, the authors believe establishes the construct validity of entrance test, since, all questions asked in ET can be cross-referenced from the textbooks and syllabi of both the Punjab Higher Secondary School Board and the non-board based educational streams. The reliability co-efficient presented in the result section of the paper for the ET and its sub-components indicate a high internal consistency for this test over the years.
With the construct validity and reliability established, the authors would now like to focus their attention on predictive validity. The results clearly indicate that Entrance test has high incremental predictive validity, with better performance in Entrance test predicting higher scores in health professions education. This is not the case when comparing pre-ET academic achievement with post-medical- selection academic performance. This incremental predictive validity is at least established for the pre-clinical years of health profession education. Only time will prove the worth of ET in predicting educational performance in the clinical years and later on in professional life.
The authors believe that by combining the data across the years, they have presented a more realistic picture of predictive validity. However, as is evident from Table-2 and Figures 1 & 2 and the data from different years presented separately, there is high positive correlation between Entrance Test scores and post-selection educational performance when compared with the correlations between HSSC and SC performance and the same post-selection medical college performance.
From the result, it is also evident that all four components of Entrance test do not have similar Predictive validity for post-selection performance. However, it is not the intent of the authors to suggest that one or more components may be excluded from the Entrance test. Nevertheless, the evidence in support of excluding English and Physics sub-components from the ET in the future is substantial and these portions in the opinion of the authors can be replaced with the aptitude test discussed next in this section.
In discussing the \'Aptitude and Feedback\' test, the authors would like to add that in this study, the data presented is only for the first 22 out of 60 questions. These 22 questions evaluate ethical and behavioral concepts, abstract non-verbal reasoning and aptitude and hence are a mix of both crystallized intelligence and fluid intelligence.3 0.5 as a value for Crohnbach\'s alpha is acceptable given the short version of this test (22 items).18 Of greater importance is the fact that the data from 2011 suggests incremental predictive validity comparable to that of the Entrance Test for post-selection academic achievement. This is particularly exciting, for the authors believe that over the years, further data shall be available to match predictive validity of Entrance Test and the aptitude test against each other in predicting not only pre-clinical academic performance but also clinical academic performance and clinical practice in professional life.
The data suggests bias related to social class and educational background. Surprisingly, students from socio-economically and educationally under-developed districts (DG Khan, Lodhran, Muzaffargarh, Rajanpur, Layyah) are seen consistently to score higher marks in SC and HSSC examinations than in post-medical selection professional examinations. Since it is the performance after selection that is the subject of interest here, the authors believe that Entrance and Aptitude tests being better predictors of post selection academic performance should be given greater or sole weightage in the selection process. Whereas, it is not the purpose of this study to comment on the reasons for this poor predictive validity of School Certificate (SC) and HSSC results, the authors would like to add only that better performance by candidates from underdeveloped districts in SC and HSSC examinations in their own districts may be a result of leniency on the part of the individual Districts Boards of underdeveloped districts trying to follow a policy of widening participation unofficially.
Based on the results of the study, the authors conclude that Entrance test conducted by UHS, Lahore for Government of Punjab for admissions in medical colleges in the province is a valid and reliable test with incremental predictive validity at least for performance in the pre-clinical years of medical education, so far. However, the authors would like to suggest that in order to reduce the economic and psychosocial burden of the test, the eligibility for applicants to sit in the test might be raised from 60% HSSC marks to 70%, thus reducing the test burden. Under the current circumstances in which HSSC and SC scores have poor predictive validity whereas Entrance and Aptitude tests have high predictive validity at least for performance in the pre-clinical years, it is suggested that HSSC and SC marks should have little if any weightage in the selection process.
1. Government of the Punjab. Prospectus for admissions to Government Medical and Dental Institutions of the Punjab session 2011-2012. University of health Sciences. (Online) (Cited 2013, May 24); Available from URL: http://www.uhs.edu.pk/mcat/full_prospectus_final.pdf
2. Pakistan Medical & Dental Council: Conditions for admission in MBBS/BDS courses and conditions for house job/internship/foundation. (Online) 2013 (Cited 2013 May 25); Available from URL: http://pmdc.org.pk/portals/0/admission% 20housejob%20regulations.pdf
3. University of Health Sciences. Syllabus for Entrance Test 2013, University of Health Sciences. (Online) 2013 (Cited 2013 May 24); Available from URL: http://www.uhs.edu.pk/mcat/etsyllabus2013.pdf
4. Urlings-Strop LC, Stegers-Jager KM, Stijnen T, Themmen AP. Academic and non-academic selection criteria in predicting medical school performance. Med Teach 2013; 35: 497-502.
5. Mercer A, Puddey IB. Admission selection criteria as predictors of outcomes in an undergraduate medical course: A prospective study. Med Teach 2011; 33: 997-1004.
6. Mcmanus IC, Ferguson E, Wakeford R, Powis D, James D. Predictive validity of the Biomedical Admissions Test: An evaluation and case study. Med Teach 2011; 33: 53-7.
7. Ferguson E, James D, Madelely L. Factors associated with success in medical school: Systematic review of the literature. BMJ 2002; 324: 952-7.
8. Parry JM, Mathers JM, Stevens AJ, Parsons A, Lilford R, Spurgeon P et al. Admissions processes for five year medical courses at English schools: review. BMJ 2006; 332: 1005-9.
9. Kulatunga-Moruzi C, Norman GR. Validity of admissions measures in predicting performance outcomes: The contribution of cognitive and non-cognitive dimensions. Teach Learn Med 2002; 14: 32-42.
10. Siu E, Reiter HI. Overview: What\'s worked and what hasn\'t as a guide towards predictive admissions tool development. Adv Health Sci Educ 2009; 14: 759-75.
11. Patterson F, Ferguson E. Selection for medical education and training. In: Swanwick T, editor. Understanding medical education. London UK: Wiley-Blackwell, 2010.
12. Prideaux D, Roberts C, Eva K, Centeno A, Mc Corie P, Mc Manus C et al.. Assessment for selection for the health care professions and specialty training: conference. Med Teach 2011; 33: 215-23.
13. Turner R, Nicholson S. Can the UK Clinical Aptitude Test (UKCAT) select suitable candidates for interview? Med Educ 2011; 45: 1041-7.
14. Hojat M, Erdmann JB, Gonnella JS. Personality assessments and outcomes in medical education and the practice of medicine: AMEE Guide no. 79. Med Teach 2013; 1-35.
15. Khan JS, Tabasum S, Mukhtar O. Comparison of pre-medical academic achievement, entrance test and aptitude test scores in admission selection process. J Pak Med Assoc 2013; 63: 552-7.
16. Harden RM. If it matters, it produces controversy. Med Teach 2011; 33: 4-5.
17. Mcmanus IC, Ferguson E, Wakeford R, Powis D, James D. Response to comments by Emery and Bell, Medical Teacher 33(1): (this issue). Med Teach 2011; 33: 60-1.
18. Emery JL, Bell JF. Comment on I.C. McManus, Eamonn Ferguson, Richard Wakeford, David Powis and David James (2011). Predictive Validity of the BioMedical Admissions Test (BMAT): An evaluation and case study. Medical Teacher 33(1): (this issue). Med Teach 2011; 33: 58-9.