Welcome to Journal Of Pakistan Medical Association
February, 2017 >>

Medical studentsí perception of the progress test as a quality-controlled assessment tool for improving learning and teaching, at a public sector medical college in Saudi Arabia

Kamran Sattar, Tauseef Ahmad, Mahmoud Salah Khalil, Mona Mohamed Soliman, Hamza Mohammad Abdulghani  ( Department of Medical Education, College of Medicine, King Saud University, Saudi Arabia. )

Gominda Giatry Punnamperuma  ( University of Colombo, Sri Lanka. )

Abstract

Progress test's distinguishing characteristics make it pertinent worldwide. We explored medical students' perceptions and opinions about Progress Test (PT) with a view to identifying areas concomitant with it's execution. This cross-sectional study took place at College of Medicine, Saudi Arabia, during the academic year 2015-16. A questionnaire (14 items) was administered. Reason for majority n=96 (89.7%) of the total participants to take the PT was their keenness to compare their academic standing with their peers from other participating medical colleges. The majority of students were highly satisfied with PT implementation; i.e. its orientation (58.9%) and allocated time (90.7%). Students (76.6%) considered PT to offer academic support as future physicians. Students (75.7%) also agreed to participate in the future PT. Students being highly satisfied with the organization of PT. They found it to be a tool helping them to focus on improving the knowledge domain.
Keywords: Medical Students, Student Experience, Progress Test, Lessons Learned, Future Directions, Saudi Arabia.


Introduction

As an assessment tool in medical education, the progress test (PT) offers some distinctive characteristics that sets it apart from other types of assessment. Since the 1970s, it has been considered a unique type of examination developed independently by two medical schools, one in the United States1 the other in the Netherlands.2 This exam concept has since been embraced by many other medical schools, especially McMaster in Canada, the initiator of the full problem-based learning medical curriculum.3 Usually the PT is designed to be applied to all students enrolled in the medical programme at the same time. The results of the PT help consistently measure the objectives of the entire medical programme, due to its repetitive and longitudinal nature. This lets the organizations using the PT to target and identify several important functions in the medical programmes. It is widely believed that what is asked in examinations drives what students learn.4 In this way the test is successful in assessing functional knowledge and breaking the relationship between the taught programme and assessment.5 It was found that various medical schools used questions which were usually either multiple choice (University of Missouri-Kansas City,1 McMaster3) or true-false (University of Limburg2).
Since PT has newly been introduced in Saudi Arabia, a need was identified to determine medical students' perception and opinions about PT. Therefore, this study explored the essential factors in terms of students' feelings about their participation in the PT. We also identified areas concomitant with the execution of PT which require further strengthening to achieve more positive sustained outcomes in terms of students' participation in future PT.


Subjects and Method

In the academic year 2015 to 2016, a total of 113 medical students participated in the PT and all of them were requested to complete a prestructured questionnaire. With our PT (to be organised twice yearly), the student population (from 2nd to 5th year) of our medical college sits the same test. This PT consisted of 200 items of the single best answer format added with an option as "don't know". The question items represented basic medical science and clinical subjects. Students' test scores are indicated as the percentage of correct minus incorrect answers (i. e. 0.25 marks to be deducted for each wrong answer) to reduce the students' attempts at guessing.
The details about the first ever PT at College of Medicine were discussed with the students. This helped motivate students to take the PT, as they could compare their performance with those from 30 participating medical colleges all over the Kingdom of Saudi Arabia.

Study Instrument; Preparation & Validation

A team of experts, comprising four senior faculty members did an extensive literature review. Initially 20 items were included in the questionnaire, later the experts discussed each item for its face validity and agreed upon selecting 14 items to be kept in the final questionnaire.
The participants were asked to answer this anonymous self-administered, English questionnaire comprising 14 items. Of the 14 items, 11 used a five-point Likert scale, ranging from strongly agree to strongly disagree. Additionally, there were three 'non-mandatory' open-ended questions, asking students what 3 things they liked the most and three things that they did not like. There was a last item asking students to write any suggestion.
Students were from the 2nd, 3rd, 4th and 5th academic years. The students from the 1st year were not subjected to the PT as there was an exam to be held at the same time. An item analysis of the 11 items revealed a Cronbach's alpha of 0.61 (Figure-1).

The corrected item-total correlation of the 11 item scale ranged from 0.03 to 0.50. This caused the correlation indices to remain satisfactory and the removed item 8 would cause further improvement to the alpha value (0.68) (Table 1).



Data collection and analysis

The collected data were saved in a secured computer and analyzed using SPSS version 21.0 statistical software. Chi-Square test was used to measure the associations between the different categorical variables such as participating students' year of study. A p-value of <0.05 was considered statistically significant. The correlation between each item and the whole test was also reviewed. The Cronbach's alpha was used to estimate the reliability of the test.

Ethics Approval

The Institutional Review Board approved the study.


Results

The College of Medicine has a total of 1300 students. 113 (8.69%) students participated in the PT. Of them, 107 students (94.69%) completed the questionnaire. 25 (23.3%) were second year students, 26 (24.2%) were third year students, 5 (4.6%) were fourth year students, and 51 (47.6%) were fifth year students. Of the total 107 participating students 47 (43.9%) were male, and 60 (56%) were female students.


Table-2 summarizes the students' perception of the progress test. Most students (57.4% of the male, and 60% of female) reported that progress test orientation session helped them clear their confusions. Furthermore, most of the students (79.4% of the male, and 76.6% of the female) reported that they were satisfied with the way the progress test is organized, showing a result which is not significant (p=0.18), this is due to an absence of group difference. The time allowed to answer was sufficient, as 90.7% students agreed, of which 91.5%were male and 90% were female. As such, the p value for investigating a gender difference on claiming that testing time was sufficient was statistically insignificant (p=0.52). Most of students (76.6%) reported that the progress test was important for them as a future physician
Majority (81.3%) of the students agreed that the participation in the PT stimulated them to learn more. This was also found to be consistent with the students' written comments (against the open ended question):
"PT helps me for my other MCQs exams".
"Yes... I would again participate in next PT exam …but I guess I would need to study more from lectures but it's not enough so I would get help from other learning resources"
Most (75.7%) students (second= 88%, third= 73.1%, fourth= 100%, fifth = 68.6%) agreed to sit the PT again next year but there was no group differences between male and female students and the p value was 0.63.
A weak agreed response rate (37.4%) was found for the item "I already knew about the progress test before it was arranged this year". Students' open comments in this regard were:
"I did know that there is an exam … fortunately my friend called me one night before exam and told me about it".
"I think my result will not be good because I knew about this exam very late and could not prepare for it".
"Why weren't we being informed many days before the actual exam day?"
Moreover, 86.9% (including 91.5% of the male, and 83.3% of the female) students were excited to know about their academic standing, as compared to other students of King Saud University Medical College. There were also some students written comments recorded as follows:
"I took the PT exam for only one reason and that is my chance to know my knowledge about the subjects and my standing as compared to other students in my own university".
And almost the same proportion (89.7%) of students were excited to know the academic standing as compared to students of other medical colleges. Students' comments were found to be as follows:
"It's good that I will be able to judge myself and see where do I stand in the field of medicine when compared with students from other medical universities"
Of the total 107 students, 47(43.92%) responded to the three open ended questions.
For the item, "three things that you liked the most", the majority of the students were found to be very much satisfied with PT because it allowed them to have a clear judgement about their academic standing, when compared to their peers.
Furthermore, the responses recorded by the majority of the students regarding the item,
"three things that you did not like about the PT", revealed that the students were dissatisfied with the number of exam questions, the time allocated, and for not being informed about the commencement of PT well in advance. Few students also highlighted the need for a short break during the exam and also suggested that the future PT exam should not be held in mid-semester.


Discussion

The majority of the students were satisfied with the PT test. However, the students indicated that the administration of the test could be improved. This PT was conducted concurrently at various medical colleges. A test which is commenced at different schools is a powerful instrument for comparison of academic standing among students from various colleges.6
Around 77% students agreed that the topics covered in the progress test were important for them. PT is a useful tool for students' achievements.7 The majority (81.3%) of the students agreed that the participation in the PT stimulated them to learn more. This is compatible with the reason why progress testing was originally developed.8
The organizing committee discussed the situation in a post PT meeting and agreed to plan and work for a robust and appropriate organization for introductory sessions utilizing all available resources of communication e.g. students' email system and social networking sites commonly used by the majority of the students.9
Students' comments for reducing the number of questions is deemed impractical as the PT encompasses all subjects of the 5 year medical curriculum. The majority of the students were highly satisfied with the way the PT was organized (its orientation session, allocated time). Students agreed that progress tests assisted to monitor the progress of their educational performance had an optimistic predictive impact on their study efforts during the entire academic year.10
The limitation of our study include the fact that the1styear medical students, n=300(23%) were not able to take part in the PT; therefore, their perception and opinions could not be included.
In conclusion, this study found that the majority of the students were satisfied with the PT test, and they intended to sit the test again next year. The fact that there were no significant differences in the perceptions of male and female students and among the students of different years indicated that the test is acceptable to all categories of students. We therefore, recommend that the identified issues as mentioned and discussed above should be sincerely and seriously addressed to ensure more positive and sustained outcomes in terms of students' extensive participation in future PTs.


Acknowledgements

Authors thank all the students who took part in answering the questionnaire and helped us identify the pertinent issues for quality of organization and adaption of PT to the very specific needs of the college of medicine, KSU. The study was funded by the College Research Center, Deanship of Scientific Research, King Saud University, Riyadh, Saudi Arabia.
Disclosure: Authors have no conflict of interest, and the work was not supported or funded by any drug company.
Disclaimer: Authors declare that this Manuscript has not been previously presented or published in a conference.
Conflict of Interest: Authors declare that we do not have any financial, personal, or professional interests that could be construed to have influenced the work.


References

1. Wood DF. Formative Assessment, one of 29 booklets in the ASME series Understanding Medical Education. Edinburgh: Association for the Study of Medical Education; 2007
2. Arnold L, Willoughby TL. The quarterly profile examination, Acad Med 1990; 65: 515-6.
3. Van der Vleuten CPM, Verwijnen GM, Wijnen WHFW. Fifteen years of experience with progress testing in a problem-based learning curriculum. Med Teach 1996; 18: 103-9.
4. Newble DI, Jaeger K. The effect of assessments andexaminations on the learning of medical students. Med Educ 1983; 17: 165-71.
5. Abdulghani HM, Ponnamperuma GG, Ahmad F, Amin. A comprehensive, multi-modal evaluation of the assessment system of an undergraduate research methodology course: Translating theory into practice. Pak J Med Sci 2014; 30: 227-32.
6. Muijtjens AMM, Schuwirth LWT, Cohen-Schotanus J, Vleuten CPM van der. Differences in knowledge development exposed by multi-curricular progress test data. Adv Health Sci Educ Theory Pract 2008; 13: 593-605
7. Schauber SK, Hecht M, Nouns ZM, Kuhlmey A, Dettmer S. The role of environmental and individual characteristics in the development of student achievement: a comparison between a traditional and a problem-based-learning curriculum. Adv Health Sci Educ Theory Pract 2015; 20: 1033-52.
8. Arnold L, Willoughby TL. The quarterly profile examination. Acad Med 1990;65: 515-6.
9. Sattar K, Ahmad T, Abdulghani HM, Khan S, John J, Meo S. Social networking in medical schools: medical student's viewpoint.Biomed Res 2016; 27: 1378-84
10. Wijnia L, Loyens SMM, Derous E. Investigating effects of problem-based versus lecture-based learning environments on student motivation. Contemp Educ Psychol 2011; 36: 101-3.


NEWS AND EVENTS

Eurasian Orthopedic Forum will be held from 29-30 June 2017 in Moscow, Russia. Detail information can be obtained from:
http://eoforum.ru/en



FOR REVIEWERS

ANNOUNCEMENT

SUPPLEMENT

INDEX

INSTRUCTIONS TO AUTHORS

COMMITTEE ON PUBLICATION ETHICS


This journal is a member of and subscribes to the principles of the Committee on Publication Ethics.

Copyrights © 2015 JPMA- All rights reserved
Powered by: PakCyber