Rizwana Shahid  ( Rawalpindi Medical University, Rawalpindi )
Rehan Ahmed Khan  ( Riphah Academy of Research and Education, Riphah International University, Islamabad,Pakistan )
Rahila Yasmeen  ( Riphah Academy of Research and Education, Riphah International University, Islamabad,Pakistan )

Objective: To establish the construct validity of Assessment of Medical Educational Environment by the Teachers inventory.
Methods: The cross-sectional analytical study was conducted from January to May 2017 and comprised doctors working as faculty in Rawalpindi Medical College, Rawalpindi, Pakistan, and its 3 teaching hospitals. Non-probability (purposive) sampling was used to meet the criteria of 5 participants per item of the Assessment of Medical Educational Environment by the Teachers inventory. Exploratory factor analysis was done using SPSS 20 and confirmatory factor analysis was done with version 16 of the Analysis of Moment Structures software.
Results: Of the 250 subjects, 126(50.4%) were males and 124(49.6%) were females. Exploratory factor analysis ended with extraction of 11 components. It showed sufficiency of sample size and no multi-collinearity. Three (50%) of the six domains were finalised on the whole and 10(20%) of the 50 items were debarred from the inventory. All three domains had high reliability. Root mean square residual and chi square / degree of freedom were within acceptable limit. However, comparative fit index, goodness of fit index, normed fit index and root mean square error of approximation portrayed not only poor model fit after re-running confirmatory factor analysis, but also led to omission of further 16(32%) items with poor loadings from the inventory. Thus, there was exclusion of total
26(52%) items from the tool and the finalised Assessment of Medical Educational Environment by the Teachers inventory comprised 24(48%) items.
Conclusion: Construct validity of Assessment of Medical Educational Environment by the Teachers inventory could not be established, but the tool was found to be reliable.
Keywords: AMEET inventory, Exploratory factor analysis, Confirmatory factor analysis, Construct validity. (JPMA 69: 34; 2019)

Introduction

Diverse teaching and learning strategies are employed in medical institutes all over the world to enhance learning of the students.¹ Communication gap between teachers and students was also found to be adversely affecting the learning climate of medical students. Therefore the need was felt to arouse interest among medical teachers to work for the improvement of the educational climate.² Numerous instruments were devised to assess educational climate by getting viewpoints of the students like Postgraduate Hospital Educational Environment Measure (PHEEM), Dundee Ready Educational Environment Measure (DREEM) etc., but the Assessment of Medical Educational Environment by the Teachers (AMEET) inventory is a tool that is actually designed to get judgments of the tutors pertinent to educational environment of medical students.³ AMEET is an instrument that is currently used to gauge educational climate of medical students by gathering opinion of the teachers. This tool is designed by medical teachers of the United Arab Emirates. It is based on 50 items and six domains. This inventory is grounded on a five-point Likert scale from 'strongly agree' to 'strongly disagree'. This tool has excellent reliability (Cronbach alpha 0.94) but unluckily the inventors could not establish its construct validity due to insufficient sample.⁴ Construct validity can be established by factor analysis keeping in view five respondents per item.⁵ Construct validity is the extent to which any tool or trial measures what it is supposed to gauge.⁶ Construct validity is broad-ranged. It covers all the verifications supporting specific interpretation of a score.⁷ The validation process embraces accretion of proofs in order to have methodical explanation. Establishing construct validity of any tool is a strenuous practice and this could be done through factor analysis both exploratory factor analysis (EFA) and confirmatory factor analysis (CFA).⁸ We are confronted with diverse terminologies during EFA, while CFA results in the creation of a model and calculation of certain indices.⁹ The current study was planned to ascertain the construct validity of AMEET inventory in relevance to Pakistani culture and context following its factor analysis. The study was likely to confirm applicability of AMEET inventory in our setup for appraising the learning climate of our medical institutions.

Subjects and Methods

The cross-sectional analytical study was conducted from January to May 2017 and comprised doctors working as faculty in Rawalpindi Medical College (RMC), Rawalpindi, Pakistan, and its 3 teaching hospitals: Holy Family Hospital (HFH), Benazir Bhutto Hospital (BBH) and District Head Quarters (DHQ) Hospital. For factor analysis, data has to be collected from 5 participants per item.5 Non-probability (purposive) sampling was used to meet the criteria for the 50-item inventory. The subjects filled AMEET inventory (Annexure-1)



after first giving informed consent. Permission for use of AMEET inventory for the establishment of its construct validity was taken from one of the principal inventors of this instrument.⁴ Approval was obtained from the ethics review board of Riphah International University, Rawalpindi and RMC. All doctors working in RMC and the three teaching hospitals were enrolled. House officers and postgraduate trainees were excluded. EFA was conducted using SPSS 20. Items of the AMEET inventory found with Eigen value less than 1 after EFA were eliminated from the inventory. It is mandatory for each domain to have at least 3-5 items for accuracy of the results from EFA.¹⁰ CFA was carried out by using version 16 of the Analysis of Moment Structures (AMOS) software. Parameters of CFA were also considered before eliminating irrelevant items. In AMOS 16, visual paths were drawn on graphic windows.

Results

Of the 250 subjects, 126(50.4%) were males and 124(49.6%) were females. Institutions and designations of the subjects were noted (Table-1).



Kaiser-Meyer Olkin (KMO) measure of sample adequacy was computed to be 0.900 which indicated sufficient sample size. In addition, highly significant Bartlett's test of sphericity value (0.000) suggested rejection of null hypothesis and efficient application of EFA on the data set with varimax rotation (Table-2).



The computed determinant equivalent was 9.6, revealing no multi-collinearity. Percentage of variance and eigen value constituted by each domain was separately (Table-3).



Also, 11 components / domains were extracted from AMEET inventory following EFA and varimax data rotation (Table-4).



Ten (20%) items were totally eliminated from the inventory after EFA, out of which 8(16%) were excluded because of their negative scores and 2(4%) others because they did not load in any component. Out of six domains, 3(50%) were finalised and the items of other domains (with less than 3 items) were also shifted to the 3 major domains keeping in view their theoretical confirmation to the respective domain. Ultimately CFA also led to the elimination of further 16(32%) items which were computed to have poor loading (<0.4) in their respective domains (Annexure-2).



The 3 major domains were established from results of EFA keeping in view rotated component matrix and reliability (internal consistency) of all the items within each domain (Table-5).



EFA was followed by CFA and the initial model was designed pertinent to all the factors and their relevant items (Figure-1).



Items with negative loadings and poor loading in each factor were further removed and modification indices were computed after re-running CFA in order to get values closer to an acceptable model fit (Figure-2).



The reliability index (Cronbach α) of domains 1,2 and 3 established after elimination of items with poor loadings in relevant domains and re-running CFA was computed to be 0.94, 0.88 and 0.71 respectively. However, reliability of modified tool was also found to be acceptable (Cronbach α= 0.77). Fit indices were computed for both models and their comparison with benchmarks was done (Table-6).



Discussion

Establishing construct validity is actually evaluation of the degree to which statistical and theoretical confirmation support the aptness of inferences. ¹¹ Moreover, researchers should try to establish construct validity of an already developing instrument instead of developing and validating a new tool.¹² Both EFA and CFA were carried out in the present study for the establishment of construct validity of AMEET inventory by getting this tool filled by the faculty of RMC and its allied hospitals. A similar study was conducted by doctors of a Saudi university medical school that was aimed at establishing the validity of PHEEM inventory by doing EFA.¹³ Like the present study, the Saudi doctors applied principal component analysis with varimax rotation, and also retained those items in the inventory that had Eigen value greater than 1.13 but they did not do CFA. Psychometric analysis of Jefferson scale of physician empathy¹⁴ was done by doing both EFA and CFA. EFA was done by performing principal component analysis to appraise the relationship between variables and factors. Like the present study, factor co-efficient of 0.40 or greater was selected for retention of variables in this international research. ¹⁴ But contrary to the current study, factors with Eigen value greater than 1.25 were retained and apart from using AMOS software, structural equation modelling was also done for CFA. Jefferson scale revealed excellent goodness of fit compared to our study. This difference might be due to huge sample of 853 respondents who filled the questionnaire compared to the current sample of 250 doctors. In 2002 a study opined that we should not rely on favourable approximate indices calculated from CFA because the appropriateness of goodness of fit indices might be due to huge variance and low correlation between the variables.¹⁵ Moreover, it has been suggested that paradoxical effect of reliability seems to greatly affect confirmatory fit index (CFI).¹⁶ A study among 656 Malaysian medical students for the establishment of construct validity of DREEM was done using CFA. This 50-item tool ultimately had 17 items after CFA.¹⁷ The current study eliminated 26 items from the AMEET inventory to make it valid for Pakistani culture, while 33 items were removed from the DREEM questionnaire following CFA to make it fit the Malaysian culture and context.¹⁷ Contrary to the current study, the Malaysian study was done on a huge sample size of 656. This could be one of the reasons of our inability to establish the construct validity of AMEET inventory. Similar to our study, the faculty of the School of Dentistry in Indonesia worked for the establishment of construct validity of DREEM by getting it filled by 352 medical students. Apart from CFA, Pearson Product Moment Correlation test was applied to test the validity of this tool.¹⁸ Ultimately 17 items out of total 50 were sorted out as bad due to weak correlation (r<0.3). Again, this tool was found to have good reliability (Cronbach α = 0.883) but its construct validity could not be proven in Indonesian culture and context. The limitation of that study ¹⁸ was found to be insufficient sample size. Although study participants in the Indonesian research were more than those enrolled in the current study, it failed to determine the construct validity of the tool. This is a matter of great concern and should be scrutinised by further research. Psychometric assessment of DREEM inventory has also been carried out by researchers of Ireland by getting responses of 239 final year medical students.¹⁹ Like the present study, it revealed acceptable reliability (Cronbachα = 0.89) of the inventory, but CFA showed a weak model fit. Contrary to our study on AMEET inventory within which all domains in the final model showed good reliability, the reliability of one subscale (students' social
self-perception) of DREEM had poor reliability (Cronbach α = 0.55) which also arouses suspicion regarding vagueness of construct validity of DREEM inventory.¹⁹ The reason might be the inadequacy of the sample size. Ultimately it was suggested that factor analysis of DREEM inventory should be followed by huge sample full Structural Equation Modelling (SEM) analysis for psychometric evaluation of this instrument.¹⁹ IN terms of limitations, the current study had doctors alone as respondents and were likely to have studied in English-medium schools. Therefore, the questionnaire was not translated in Urdu. Besides, data was not gathered by probability sampling technique in order to have a large sample size. In future studies, methods other than factor analysis should also be employed to determine the construct validity of a tool. In addition, construct validity of this tool should be established in other cultural contexts by getting responses from still more medical teachers so as to get a better model fit. The factor analysis carried out on still large sample size followed by SEM technique is most likely to prove the construct validity of a tool. Moreover, criteria of 5 participants per item need to be revised for authenticity of factor analysis.

Conclusion

AMEET inventory was found to have acceptable reliability. Two indices of CFA met standards, but 4 indices could not, so construct validity could not be established.
Disclaimer: This manuscript is part of the MHPE (Masters in Health Professions Education) research project of one of the authors.
Conflict of Interest: None.
Source of Funding: None.

References

1. Reem A, Ramnarayanan K, Vinod P. Students' Perceptions of Learning Environment in Indian Medical School. BMC Med Educ 2008; 8: 20.
2. Rajan M, Chacko T. Improving educational environment in medical colleges through transactional analysis practice of teachers. F1000Res 2012; 1: 24.
3. Soemantri D, Herrera C, Riquelme A. Measuring the educational environment in health professions studies. Medical Teacher 2010; 32: 947-952.
4. Shehnaz SI, Premadasa G, Arifulla M, Sreedharan J, Gomathi KG. Development and validation of the AMEET inventory: An instrument measuring medical faculty members' perceptions of their educational environment. Medical Teacher 2015; 37: 660-669.
5. Gorsuch RL. Factor Analysis. 2nd ed. Hillsdale, NJ: Erlbaum. 1983.
6. Polit DF, Beck CT. Nursing Research: Generating and Assessing Evidence for Nursing Practice. 9th ed. Philadelphia, USA: Wolters Klower Health Lippincott Williams & Wilkins.2012.
7. Messick S. Standards of validity and the validity of standards in performance assessment. Educational Measurement 1995; 14: 5-8.
8. Cronbach LJ, Meehl PE. Construct Validity in Psychological Tests. Psychological Bulletin 1955; 52: 281-302.
9. Ugulu I. Confirmatory factor analysis for testing validity and reliability of traditional knowledge scale to measure university students' attitudes. Educ Res Rev 2013; 8: 1399-1408.
10. MacCallum RC, Widaman KF, Zhang S, Hong S. Sample size in factor analysis. Psychological Methods 1999; 4: 84-89.
11. Messick S. Test validity: A matter of consequences. Social Indicators Research 1998; 45: 35-44.
12. Kimberlin CL, Winterstein AG. Validity and reliability of measurement instruments used in research. Am J Health-Syst Pharm 2008; 65: 2276-2284.
13. Al-Sheikh MH, Ismail MH, Al-Khater SA. Validation of the postgraduate hospital educational environment measure at a Saudi university medical school. Saudi Med J 2014; 35: 734-773.
14. Tavakol S, Dennick R, Tavakol M. Psychometric properties and confirmatory factor analysis of the Jefferson scale of physician empathy. BMC Medical Education 2011; 11: 54.
15. Browne MW, MacCallum RC, Kim, C-T, Anderson BL, Glaser R. When fit indices and residuals are incompatible. Psychological Methods 2002; 7: 403-421.
16. Miles J, Shevlin M. A time and a place for incremental fit indices. Personality and Individual differences 2007; 42: 869-874.
17. Yusoff MSB. The Dundee Ready Educational Environment Measure: A confirmatory factor analysis in a sample of Malaysian medical students. Int J Humanities and Social Sci 2012; 2: 313-321.
18. Leman MA. Construct Validity Assessment of Dundee Ready Educational Environment Measurement (DREEM) in a school of Dentistry. The Indonesian J Med Edu 2017; 6: 11-19.
19. Hammond SM, O'Rourke M, Kelly M, Bennett D, O'Flynn S. A Psychometric Appraisal of the DREEM. BMC Med Educ 2012; 12: 2-5.