July 2014, Volume 64, Issue 7

Short Communication

Introducing an innovative viva format for assessment of integrated knowledge

Arshia Samin Naqvi  ( Quality Enhancement Cell (QEC), Ziauddin University, Clifton Campus, Karachi, Pakistan. )
Bilal Aheed  ( Department of Biochemistry, Ziauddin Medical College, Ziauddin University, Clifton Campus, Karachi, Pakistan. )

Abstract

Level of cognitive learning in undergraduate medical students can be enhanced by a commensurate change in teaching/learning and assessment strategies. Medical institutions with integrated curriculum find it difficult to assess the integration of Basic Health Sciences knowledge. Generally, viva being one of the traditional tools of assessment, is not appropriately utilized for achieving the desired objective of assessing an amalgamation of basic sciences knowledge in pre-clinical year\'s examinations. In order to harness the most important determinant of learning i.e. assessment, changes in the viva format are carried out at institutional level and three scenarios are used around which the viva questions are designed for each semester examination. Evaluation of the new format has been done. Encouraging results are obtained which add credence to the scenario based viva format to be used in situations where integration of different disciplines is practiced.
Keywords: Assessment, Assessment tool, Integrated curriculum, Evaluation.

Introduction

Today\'s ever evolving healthcare system has increased the need for change in the methodology of learning and assessment in medical education.1 Assessment and assessment instruments have a great impact on learning hence the adage \'Assessment drives learning".2,3 It is important to use accurate assessment tool to improve the desired type of learning.2
If the assessment tool requires descriptive knowledge the students will study in detail while for simple recall type multiple true false items students will simply memorize the facts.
Older assessment techniques may not be able to meet the demands of the improved/innovative curriculum. A number of emerging innovative assessment tools are now being used internationally. George Sweeney4 was convinced that medical professionals require the development of ability to solve problems by using their theoretical basic sciences knowledge Choice of an effective tool for assessment of student\'s integrated learning of basic health sciences knowledge is a challenging task.5 What is deemed important is the alignment of assessment tools to the teaching/learning strategies as well as to the objectives of the curriculum.6
Studies have shown that most of the medical schools around the world have adapted new teaching/learning strategies to attain their objectives;7 but without accurate tool of assessment these strategies prove useless.2

Innovative Assessment Tool
MBBS curriculum at Ziauddin Medical College has an integrated modular approach to the Basic Health Sciences. Achievement of the specified learning objectives is ensured by means of multimodal assessment keeping in mind the strengths and weaknesses of each tool.5
For assessing integrated multidisciplinary knowledge of basic sciences a structured clinical scenario based viva was introduced in 2012 in which the students integrate and apply their basic sciences knowledge to answer the viva questions.1.8 Since then, a Viva examination comprises of a set of three scenarios, each scenario having viva questions derived from all the integrated basic sciences disciplines. The number of learning objectives addressed by each scenario may vary. The scenarios reflect the professional framework that the students will encounter in future.8 This format ensures good sampling of core knowledge, objectivity of assessment and improves validity of the examination.3
Scenario based viva exam is organized in the form of a circuit similar to the OSCE circuits. Arrangement is such that six students are allowed into the examination hall together. Students are presented with the three scenarios and allowed five minutes to go through them. Every student faces both an external and an internal examiner for five minutes each, for every basic science discipline. Presence of external examiners improves the credibility of assessment and is a requirement of Pakistan Medical and Dental Council.



Evaluation is essential to assess the quality of any innovative teaching or assessment methodology.
The first three scenarios based viva examination was evaluated by students, external examiners and internal and external observers by means of structured questionnaires.
Responsibility of evaluating the new viva format was given to the Quality Enhancement Cell (QEC) of Ziauddin University.
Different feedback forms were designed for students, examiners and observers. Every student had to fill the form and submit it to QEC before leaving the examination hall. Examiners filled the forms at the end of each viva. Two neutral observers, one being the member of QEC and another invited by QEC from a sister medical institution, gave detailed reports of their observations.
Feedback data, received from students, examiners and observers was compiled. It was followed by qualitative analysis and conclusions drawn on its basis.
Students being the major stakeholders were informed of the new viva format much in advance of the semester exam and their feedback was taken by means of a questionnaire given to them after the end of viva. Each statement was scored on a Likert scale with 3 being the maximum and 1 minimum score. The students rated clarity of instructions given during the examination process as the highest with a mean score of 2.53 and difficulty level of questions got a minimum mean score of 1.94.
The questionnaire provided to the external examiners for evaluation of scenario based viva, rated various attributes on a Likert scale where 1 was minimum and 5 the maximum score.
The external examiners involved in the viva found the new format to be relevant to the curriculum with a mean score of 4.66. Efficiency of the format to assess what was to be assessed and the quality of questions scored a mean of 4.58 each. Role of examiners in the new format had a mean score of 4.33 while mean score of reliability was 4.25. Responding to open ended questions regarding the strengths of the format the external examiners commented that biasness was least, examiner friendly format with a broader view, felt more comfortable as it provided more options for questions within short time. It motivated students and was interesting for teacher and student both.
External observer reported that it\'s a novel method of testing application and integration of basic and clinical sciences through viva voice examination. He further stated that the Content coverage was adequate, having both Face and Content validity as it was designed appropriately to test application of basic sciences in clinical context through a series of questions with reference to a case. Examiner training is essential and helped to improve the inter-examiner reliability.9
The internal observer commented that increasing the number of cases allowed wider coverage. Objectivity of the exam was fairly good. It is resource intensive as a number of examiners and proper arrangements are required. The assessment was fair and based on student\'s performance.
Evaluation of the assessment tool shows that it is a useful format, well appreciated by students, examiners and observers as it serves the purpose of integrating knowledge and giving a holistic approach to basic sciences viva exam.
In spite of its limited drawbacks, structured scenario based viva voice is an appropriate tool provided the curricular objectives are well addressed.10
The innovative assessment method serves the purpose of integrating knowledge of various basic science disciplines; assess higher cognitive skills that enhance the effectiveness of the medical curriculum. Its repeated use followed by evaluation will help to establish its reliability and validity on a larger scale.

Conclusion

In spite of its limited drawbacks, structured scenario based viva voice is an appropriate tool provided the curricular objectives that are not assessed by other tools are well addressed. The innovative assessment method serves the purpose of integrating knowledge of various basic science disciplines; assess higher cognitive skills that enhance the effectiveness of the medical curriculum. Its repeated use followed by evaluation will help to establish its reliability and validity on a larger scale.
Further improvements can be made with the help of effective feedback from faculty, examiners and students.

References

1. Mckimm J. Current trends in undergraduate medical education: teaching learning and assessment. Samoa Med J 2010; 2: 38-44.
2. Macdonald R. Assessment strategies for enquiry and problem based learning. In: Barrett T, Mac Labhrainn I, Fallon H (Eds). Handbook of Enquiry & Problem Based Learning. Galway: CELT; 2005, pp 85-93.
3. McCrorie P. Basic and clinical sciences. In: A Practical Guide for Medical Teachers. Edinburgh, London, New York: Elsevier Churchill Livingstone; 2009; pp 255-9.
4. Sweeney G. The challenge for basic science education in problem-based medical curricula. Clin Invest Med 1999; 22: 15-22.
5. Shumway JM1, Harden RM; Association for Medical Education in Europe. AMEE Guide No. 25: The assessment of learning outcomes for the competent and reflective physician. Med Teach 2003; 25:569-84.
6. Prideaux D. Integrated learning. In: A Practical Guide for Medical Teachers. Edinburgh, London , NewYork: Elsevier Churchill Livingstone, 2009; pp  181-6. 
7. Blumberg P. Assessing Students During the Problem-Based Learning (PBL) process. Med Sci Educat 2012; 15: 92-8.
8. Epstein RM, Hundert EM. Defining and assessing professional competency. JAMA 2002; 287: 226-35.
9. Palmer JE, Devitt GP. Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions? Research paper. BMC Educat 2007; 7: 1-7.
10. Asimakis GA, Ainsworth MA, Aronson JF, Frye AW, Lieberman SA , Rabek JP. Evolution of Student Assessment Following Implementation of an Integrated Medical Curriculum: Contribution to Improved Educational Outcomes. Med Sci Educat 2011; 21: 181-9.

Journal of the Pakistan Medical Association has agreed to receive and publish manuscripts in accordance with the principles of the following committees: