Abstract

Background

Medical education has witnessed a paradigm shift towards competency-based medical education (CBME) over the past two decades. CBME is a concept where teaching, learning and assessment are driven by the needs of the population which in turn direct the kind of competencies learners should attain in order to address those needs [1]. Therefore, the goal of CBME is to graduate health professionals with defined abilities, who can meet local health care demands and optimize their care of the communities they serve [2, 3].

CBME curricula use assessments to verify that students have acquired each of the established competencies and to provide opportunity for progressive monitoring, mastery and maintenance of skills throughout the educational process [4, 5]. These assessments can utilize multiple methodologies, including written and oral exams, defined problem sets, presentations of research, as well as observations of student practice and behavior in the classroom, clinic or field through the use of the Objective Structured Clinical Examinations (OSCEs) and student portfolios [6]. The specific assessment methods chosen should have the psychometric rigor to demonstrate that the students have successfully attained the targeted competencies and have reached an acceptable level of proficiency [6]. For instance, assessment methods may differ between the pre-clinical and clinical years or when transitioning from knowledge-based to performance-based assessments [7]. Since all assessment methods have limitations, ideally competencies should also be addressed by multiple assessment methods [8]. The ability to design robust, valid and reliable assessment methods is a cornerstone of CBME. In addition, faculty must be supportive of utilizing optimal assessment methods and should have the knowledge and skills to develop valid assessments [3], if implementation of CBME curricula is to be effective and sustainable [9].

Therefore, any educational institution utilizing CBME should perform iterative systematic evaluations of their competency assessments and assess the willingness and ability of faculty to optimize their course assessments [10–13]. While implementation of competency-based reforms has been written about in other contexts, little is published on the implementation and evaluation of CBME in sub-Saharan Africa medical schools [2, 12].

The goal of the study was to evaluate the CBME curriculum in a sub-Saharan Africa medical school, Makerere University College of Health Sciences (MaKCHS), to evaluate whether the assessment tools within the current curriculum address the stated competencies to better inform future curriculum changes. In this paper, we report the method and findings from this evaluation.

Methods

Results

Determination of competencies assessed

A total of 188 assessments were documented on the ECEFs for 57 courses. Due to insufficient detail provided, competencies were not able to be assigned for 11 assessments documented on the ECEF of three courses: (a) Systemic Pathology I (Gastrointestinal tract, Cardiovascular system, Respiratory system & Central Nervous System); (b) Systemic Pathology II (Endocrine, Central Nervous System, Genitourinary tract); & (c) Chemotherapy. The remaining 177 assessments from fifty-four courses documented that 58 (98%) of the 59 competencies were assessed at least once in the curriculum. One competency in Domain 8-Continuous Improvement of Care Through Reflective Practice (8.ii Demonstrate lifelong learning in the areas of science, health care and public health) was not addressed in any of the 177 assessments reviewed. Nine (15%) of the 59 competencies were addressed by less than five assessments. These competencies fall within domains 3-Critical Inquiry and Scientific Method (1 of 4 competencies), 6-Leadership and Management (4 of 7 competencies), 7-Population Health Skills (2 of 6 competencies), 8-Continuous Improvement of Care Through Reflective Practice (2 of 7 competencies). (Supplemental Table 2)

A single course could have multiple assessments that address the same competency. Therefore, we also looked at the number of courses across the curriculum that provided an assessment for each competency. Fourteen (24%) of the fifty-nine curriculum competencies were addressed by fewer than five courses across the curriculum. Figure 1 shows the number of times each competency domain is addressed and which assessment method was used for each. This graph highlights that Domain 1-Medical Knowledge is assessed using all assessment method categories and is the most heavily assessed of all competency domains, addressed in 165 (88%) of the 188 total assessments documented. Figure 1 also shows that all domains are addressed by a variety of assessment methods and each is addressed by a minimum of 30 assessments.

Open in a separate window

Fig. 1

Number of Times Each Method is Used for Each Competency

Determination of assessment methods

To determine the frequency that each assessment method is used across the MBChB curriculum and the alignment between assessment method and competency, we looked at the method of assessment frequency and the number of times each method was used throughout the MBChB curriculum. As described in Table 1, a method category was assigned to each of the 188 assessments documented for 57 courses. While both written exams mainly differ in the amount of course content addressed, progressive (given mid semester) and summative (given end of semester) exams were used with high frequency and as such, we chose to categorize them separately. Also shown in Table 1 is the usage of each assessment method, both as an absolute number of courses the method was used and as a percentage of the total number of courses in the curriculum. Each method is used in a minimum of 20% of the courses within the MBChB curriculum. Progressive and summative written exams are the most heavily used assessment method (65% & 77% respectively) across courses in the curriculum, while practical problems/lab work and reports/write-up/presentation are used in 21% and 23% of courses, respectively.

Table 1

Assessment Method Category and Description and Number of Courses Using Each

Category	Description	# Courses	% of Total Courses
Progressive Written Exam	a written exam given sometime during a course	37	65
Summative Written Exam	a written exam given at the end of a course/semester	44	77
Practical/Objective Structured Clinical Examination/Steeple Chase Exam	includes any form of practical performance or hands-on application/demonstration of skill	23	40
Oral/Vive Voce Exam	includes all forms of oral examination	15	26
Clinicals/Clerkship/Field Work	includes all clinical work on the wards, community, and field work	18	32
Practical Problems/Lab Work	any assignment requiring students to respond/complete in specific problems, questions, or laboratory work	12	21
Report/Write-Up/Presentation	any assignment requiring students to generate a report or present on a topic they’ve explored, e.g., journal club, or project they’ve done.	13	23
Tutorial/Problem-based Learning/Seminars	any activity requiring students to participate in a topical discussion or individual/group project they’ve been assigned and/or asked to research. Students could be required to share with larger group or participate in discussion with faculty.	26	46

Open in a separate window

As shown in Fig. 2, all of the 26 pre-clinical courses (100%) used both a progressive and summative exam, compared to their use in the 31 clinical courses, 35% and 58% respectively.

Open in a separate window

Fig. 2

Percent of Courses in the Pre-Clinical and Clinical Years Using Each Assessment Category

Discussion

Makerere University College of Health Science’s transition to CBME was designed to optimize skills development amongst health professionals that would ultimately be needed to address Uganda’s priority health needs [1]. As reported in previously published work describing this change process, nine key competency domains were defined along with their respective sub-competencies [1]. The medical curriculum was subsequently revised to incorporate these new competencies, but also to align the teaching, learning and assessment methods with the defined competencies. However, since the transition to a CBME curriculum, no formal evaluation had been conducted to determine whether assessments being performed did in fact address the defined competencies. With the goal of CBME to provide students the skills to optimize health care of local communities served, it would be expected that all established competencies are assessed and that assessment takes place multiple times during the curriculum using multiple methods. This study was done to provide a monitoring and evaluation framework to map course assessments to established competencies for the MBChB curriculum. The methodology and outcome of such evaluations can inform curriculum improvement and competency adjustments and provide an iterative process to ensure the MBChB curriculum at Makerere University College of Health Sciences continues to be aligned with the health care demands of the Ugandan population.

Results from the evaluation indicated that all nine competency domains and 98% of the 59 competencies established for the MBChB program were being addressed by assessments utilized throughout the curriculum. However, competencies across several domains were assessed less than five times across the curriculum. Although five times was arbitrarily chosen, this raises the questions whether these competencies are being adequately assessed and whether they remain relevant within the curriculum. If these competencies are still relevant, efforts to further incorporate these into the current curriculum teaching and assessment framework are needed. However, it should be noted that from the evaluation, all but one of the fifty-nine competencies were targeted by at least one assessment method which is a positive finding. With shifting local health care demands, faculty changes, and assessment modifications, this highlights the need to continually evaluate established competencies for a curriculum and the assessments that address them. In doing so, competencies and assessments can be revised to ensure the curriculum continues to optimize the training students receive to care for the communities they serve.

In addition to knowing if competencies are being addressed in course assessments, it is helpful to understand how they are being addressed. All assessment methods have limitations and as such, the adoption of multiple assessment methods across a curriculum provide opportunity to align method to the competency. Results of this evaluation demonstrate that the courses across the MBChB curriculum use a range of assessment methods to assess program competencies, ranging from written exam to problem-based learning, use of OSCEs, practicals etc. Each of the assessment methods is used in over 20% of courses evaluated within the curriculum as well as in assessing all nine competency domains.

Differences in the application of the various assessment methods in the pre-clinical versus clinical years were also found. Although these differences might be expected, especially in the use of written and oral exams, both methods are used significantly less frequently in the clinical years compared to pre-clinical years, where competencies are focused more on psychomotor skills necessary for patient care.

As with any change in curriculum, it is the faculty that ultimately carry out these changes. Results of this evaluation demonstrate that within their courses, faculty have been using assessments that do, overall, address the competencies identified for the MBChB program. However, this evaluation also provided important feedback both to Makerere leadership and its faculty. Despite the overall success in addressing the competencies for the MBChB program, instructional training and insufficient faculty/student ratios are two key areas faculty identify as impacting their ability to successfully implement a CBME curriculum at Makerere. The first area underscores the significance of continued faculty development to not only equip the teaching faculty with knowledge of the curriculum competencies but also to provide ongoing training of the necessary skills to successfully teach and assess these competencies. Indeed, in a context where CBME is a new concept to teaching faculty, faculty expressed a need for greater understanding of the concept itself, as well as training in varied assessment methods. This requires continued efforts to train faculty in the most feasible and valid assessment methods for the various competencies amidst increasing student numbers. This observation is likely generalizable to other institutions struggling with the implementation of CBME especially from low- and middle-income countries (LMIC) and where student numbers are increasing throughout sub-Saharan Africa, as has been previously reported in [14].

Addressing the second area, insufficient faculty/student ratios, highlights a potential burden on faculty in carrying out assessment methods that require higher levels of individual feedback. This also has the potential to lead to inconsistencies across faculty in terms of the quality and quantity of feedback provided. Addressing this area of faculty concern is beyond the scope of this evaluation, however, it does highlight a critical issue in many LMIC medical schools and most especially the medical schools in Sub-Saharan Africa that were mandated during the implementation of the Medical Education Partnership Initiative (MEPI) programme to increase the number doctors graduating [14].

The complexity of implementing and evaluating a CBME curriculum has been reported by groups outside SSA [15–19]. These primarily have focused on post-graduate training [15–18], using national assessment data and promotional timelines [15], survey of competence committees [17] and lecture and syllabus review with student feedback [19]. However, the goal of this study was to specifically assess alignment of student assessments throughout the curriculum to established programme competencies. It is not possible to compare these findings with other schools in sub-Saharan Africa since there is a dearth of published research on the mapping of assessments to established competencies in CBME curriculum.

This evaluation not only provides information to MBChB teaching faculty and leadership on alignment of curriculum competencies and assessment, but also provides a framework for conducting such evaluations for other Sub-Saharan Africa medical schools implementing CBME. Globally, many medical schools are implementing CBME and many, especially from LMICs, are faced with the same questions addressed in this evaluation. This study thus provides some useful insights into implementing a CBME curriculum, essentially through an ongoing, iterative review of defined competencies and corresponding assessment as well continued faculty training in CBME strategies.

Limitations and challenges with the review

We interfaced with faculty who lead the courses for the interviews, but not all teaching faculty in the institution which may have led to the omission of some information. In the data capture, we collected only formal assessments from documents and from faculty interviewed. We did not capture informal assessments with no standardized performance measure, all of which could certainly play a role in addressing the stated competencies. Data was also collected as a snapshot and such cross-sectional data may not capture variations over time that could influence outcomes. Finally, the eight method categories defined were subjective and assignment of all assessments to these categories for analysis were at the discretion of the HEPI study team doing the analysis.

Conclusion and recommendations

CBME was successfully implemented at MaKCHS as evidenced by all but one of the fifty-nine competencies within the nine domains established for the MBChB program being assessed within the curriculum and by the use of multiple methods. The faculty interviewed were largely aware of it. We recommend more faculty training in CBME to align the competencies with the assessment methods within a formal standardized framework across the pre-clinical and clinical years and across departments. We also recommend future longitudinal studies to further interrogate the curriculum over some time across the years, to involve more faculty and other key stakeholders like the community and graduates of the programme to assess whether the curriculum is addressing the community’s health needs.

Electronic supplementary material

Acknowledgements

Abbreviations

Authors’ contributions

JMW and SW collected, analyzed and interpreted the data for this evaluation. JMW and AGM were major contributors in writing the manuscript. All authors read and approved the final manuscript.

Funding

Research reported in this publication was supported by the Fogarty International Centre of the National Institutes of Health (NIH), U.S. Department of State’s Office of the U.S. Global AIDS Coordinator and Health Diplomacy (S/GAC), and President’s Emergency Plan for AIDS Relief (PEPFAR) under Award Number 1R25TW011213. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health (NIH). This funder supported the ethics approval expenses and data collection process including designing tools and the actual data collection in the field.

Data availability

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Declarations

Footnotes

References