Abstract

INTRODUCTION

Key performance indicators (KPIs) are quantifiable measures of performance developed based on an organization’s strategic plan that serve as the basis for tracking progress and measuring success in achieving the organization’s goals.1 Continuous performance evaluation is key to understand past events and plan future quality improvement strategies.

In higher education, KPIs focus on key elements of the higher education enterprise including students, research, outreach, effectiveness and efficiency of administrative and educational processes, communications, and educational outcomes.2 Previous authors compiled comprehensive lists of performance measures for higher education in the United States.3-5 However, the use of KPIs is not mandated for colleges and universities in the United States unlike other countries such as the United Kingdom, New Zealand, and Canada.6-9 KPIs allow higher education institutions to not only track their own performance but also to measure their performance against peer institutions.2 KPIs can also provide insight into resource allocation and determine funding decisions by governments.2,10,11

Little research focusing on performance measurement in schools and colleges of pharmacy has been conducted to date.12 It is likely that most of these data are kept for internal use only and not published. Information pertaining to, for example, the North American Pharmacist Licensure Examination (NAPLEX) pass rate, residency matching rates or National Institutes of Health (NIH) funding rank is commonly disseminated on the schools and colleges’ websites. With the rise in the number of pharmacy schools in recent decades, and a decline in the number of applications and enrollment over the past years, it is crucial that pharmacy schools develop robust strategic plans. KPIs emanating from those strategic plans will allow assessment of the organization’s performance and inform necessary changes for continuous quality improvement.13-15

A previous report described the implementation of a continuous quality improvement program focusing on programmatic advancement at a college of pharmacy.12 The authors used the Studer Group’s Hardwiring Excellence approach to guide implementation of an excellence program that would enhance achievement of education, research and service goals, which included development of a strategic plan and performance measurement by means of KPIs.16 Accountability and alignment between the college’s goals and individual faculty, staff and administrators’ performance were ensured by including KPIs specific to each person’s job responsibilities in their annual evaluations.12 A similar management approach was adopted at the Virginia Commonwealth University (VCU) School of Pharmacy (SOP). KPIs were developed for key areas by teams of faculty, presented to the VCU-SOP Executive Committee and remaining faculty for discussion and, ultimately, approved by faculty in May 2015. Feedback from faculty was obtained over the years resulting in an undesirably high number of KPIs. In 2019, the Outcomes and Assessment Committee at the VCU-SOP was tasked with refining the KPIs for the school in conjunction with the Dean. The aim of this study was to identify essential KPIs to be adopted school-wide using a modified Delphi consensus process. Rather than aiming at the generalization of results to other schools and colleges, our goal was to illustrate how a Delphi technique can be used in committee service to promote faculty and staff involvement in KPI selection instead of following a top-down approach wherein school administrators make all decisions. Delphi techniques have been applied in the pharmacy education literature to achieve consensus regarding multiple topics, including: general practice pharmacists’ educational requirements, faculty members’ leadership characteristics, learning objectives of model courses within the pharmacy curriculum, attributes of an organizational culture measure in academic pharmacy, competencies for individuals designated as assessment leads, among others.17-24 Despite having been previously utilized to develop KPIs in clinical areas, the application of the Delphi technique to the refinement of KPIs in higher education is unique in the pharmacy literature.25-30

METHODS

A three-round modified Delphi technique was used to gather consensus about essential KPIs defined as the minimum indispensable KPIs needed to track the VCU-SOP performance. Despite not having provided a definition for what constitutes essential and non-essential KPIs to the panel members, the goal of the process and what we expected to accomplish was explained at multiple school-wide meetings prior to survey launch. The Delphi technique is a consensus method based on the agreement of a panel of experts about a given subject. In the first phase of a traditional Delphi technique, experts contribute information pertaining to the subject area being studied in response to open-ended questions.31,32 In this modified Delphi technique, the initial list of items (KPIs) was provided a priori because those were the existing KPIs adopted to assess the school’s performance. Additionally, participants targeted included all faculty and staff at the VCU-SOP because the goal was to promote all personnel engagement in the KPIs redefinition initiative, rather than to generalize the results beyond our institution. Thus, despite not being subject matter experts (i.e., performance management) as required by best practices in conducting a Delphi consensus technique, our approach ensured that all individuals were given an opportunity to participate.31,32 The final result of the Delphi process served as the basis for internal discussions at the Outcomes and Assessment and Executive Committees levels regarding adoption of KPIs achieving consensus as school wide KPIs. Ultimately, the Executive Committee, with input from the Outcomes and Assessment Committee, adjudicated the final list of KPIs.

The VCU Institutional Review Board determined that this study was not considered human subjects research, thus no review was necessary.

RESULTS

Of the 109 faculty and staff invited, 62 unique individuals participated in the Delphi process (56.9% response rate). Of these, 49 completed the first round, 51 the second, 42 the third, and 32 completed all three rounds. Of the 49 individuals who completed round 1, 38 also completed round 2. Participants had been working at the VCU-SOP for an average of 12 years, half were female, and over 70% were faculty members (Table 1).

Delphi Round 1: Consensus (≥90% agreement) for KPIs considered essential was achieved for 7 of the initial 50 (14%) KPIs assessed (Table 2 and

). Consensus items in this first round pertained to faculty, staff and student satisfaction, scholarly output (i.e., number of peer reviewed journal publications), and funding (i.e., total amount of private funds received as cash gifts annually and NIH funding ranking). No KPIs achieved consensus as being non-essential. By the end of the first round, and after reviewing input from panelists, one KPI was reworded and 26 new KPIs were included. Ultimately, 69 (=50-7+26) items were included in the second consensus round, including the new proposed KPIs and excluding KPIs that achieved consensus during round 1.

Delphi Round 2: Consensus (≥80% agreement) was achieved for 16 of the 69 (23.2%) KPIs, 15 of which were considered essential and one non-essential (Table 2 and Online appendix), representing an accumulated consensus of 23 (=7+16) out of 76 (=50+26) KPIs, i.e., 30.3%. None of the KPIs in the community outreach and research and scholarship modules achieved consensus in this round. Twelve new KPIs were suggested at the end of round 2 to be included in round 3. After excluding items that achieved consensus and adding proposed KPIs during round 2, 65 (=69-16+12) KPIs were included in the third Delphi round.

Delphi Round 3: Consensus (≥75% agreement) in the last round was achieved for 15 (23.1%) of the 65 KPIs, 13 considered essential and 2 non-essential (Table 2 and Online appendix), representing an accumulated consensus of 38 (=7+16+15) out of 88 (=50+26+12) KPIs, i.e., 43.2%. Ultimately, accumulated consensus was achieved for 35 out of 88 (39.8%) KPIs that were considered essential and 3 out of 88 (3.4%) that were considered non-essential.

DISCUSSION

The purpose of this report is to describe the process used to reassess the KPIs tracked by the school in order to reduce their number and improve programmatic assessment by focusing on the most important indicators. Burke and Minassians recommend that colleges or universities adopt no fewer than 10 and no more than 25 KPIs.33 Their research showed that a greater number of KPIs made it more cumbersome for institutions to track performance.33 The Delphi process proved effective, allowing a reduction in the number of KPIs from 56 to 44 (including 9 mandatory KPIs), while engaging faculty and staff in the process, and efficient given that no additional meetings or discussions were required to attain consensus. The Outcomes and Assessment and Executive Committees reviewed the KPIs that achieved consensus as essential and further reduced the number from 44 to 36 (including 9 mandatory KPIs). Despite being above the recommended upper limit, our initiative was successful in reducing the number of KPIs.33 KPIs that did not achieve consensus as either essential or not essential may continue to be monitored by departments or units per their needs.

As expected, a majority of participants (slightly over 70%) were faculty members. We did not have an a priori staff involvement goal but consider that having almost 30% participants being a mix of staff, university and academic professionals, and others is indicative of success. Providing all personnel with an equitable opportunity to participate in the process was a priority to the VCU-SOP administration. The panel size in our study was larger than the recommended number of experts in a Delphi study, between 15 and 30.31 Research indicated that Delphi panels of more than 30 experts do not improve results.34,35

Three rounds, as opposed to the traditional four, were used given that previous evidence showed that three rounds were sufficient to achieve consensus, with most changes to items occurring between the first and second rounds.34,35 The three-round Delphi technique is a common approach in the pharmacy literature and elsewhere, but other approaches including two rounds have also been adopted.18,22,26,28,36-38 We deliberately used high consensus criteria (as high as ≥90% in the first round and ≥80% in the second) to identify essential KPIs. Previous studies adopted a consensus threshold of ≥80%, or ≥75% throughout all rounds.18,26,38 Our study used varying consensus thresholds throughout the three rounds similar to previous research, with the goal to differentiate between KPIs for which consensus was obvious, and those that would generate more internal discussion regarding their inclusion for performance measurement purposes.17,37,39 This approach allowed us to clearly identify student satisfaction, scholarly output, and funding as the most critical KPIs to evaluate the school’s performance in round 1. These areas, in addition to the graduate program, were those for which more KPIs achieved consensus as essential at the end of round 3. These areas are consistent with KPIs described by Arif and Smiley.40 No specific trends were noted in KPIs achieving consensus in the second and third rounds (Online appendix).

Different rating scales have been utilized to assess consensus in Delphi studies. Some used Likert scales with varying formats (4-, 5-, 6- or 9-point), while others used dichotomous scales.17-19,26,36,38,39 When using Likert-type scales, consensus is often assessed by collapsing responses to the two most extreme options of agreement, technically rendering the scale dichotomous (i.e., agree or strongly agree versus everything else). In our study, we opted for a dichotomous option to obtain unequivocal decisions from participants regarding the essentiality of each KPI. We included an ‘unable to respond’ option to allow participants who were not familiar with certain aspects of the organization to avoid weighing in on the decision to keep or eliminate a KPI. This decision was left to the discretion of each participant rather than imposed a priori based on their role within the school. Other schools following a similar approach may choose to target individuals who they consider experts in a certain area to rate specific KPIs, which would result in different sets of KPIs being sent out to different groups of individuals. In our study, the number of individuals selecting the ‘unable to respond’ option ranged between 2-19 in round 1, 2-18 in round 2, and 1-12 in round 3 and did not negatively affect the recommended number of panelists in a Delphi process (between 15 and 30) in either round.31

Once the Delphi process was completed, the Outcomes and Assessment Committee recommended removal of 4 of the 35 KPIs that achieved consensus (excluding 9 mandatory KPIs) because they were either unable to be accurately measured, were already captured by other KPIs, or were not considered essential as a school wide KPI. Of note, all the KPIs removed had achieved consensus in the third round. The Executive Committee accepted these recommendations and proposed further removing two KPIs that also achieved consensus in the third round, combining two KPIs pertaining to fundraising into one, and rewording another to improve clarity (Table 3). KPIs approved by the Executive Committee were presented to and reviewed by the school’s National Advisory Committee, which includes alumni and other stakeholders, as well as University administration.

Regarding lessons learned, the Delphi technique allowed transparency in the process of adding, refining and removing school wide KPIs. Without the use of this technique, we would likely have remained with an undesirably high number of KPIs, a lack of focus, and low buy-in from faculty and staff. Prior to this initiative, most faculty and staff were not aware of the school’s KPIs. Their inclusion in this systematic process ensured that new voices, namely staff, junior faculty, or faculty without administrative duties, were heard.

KPIs are the basis for both internal and external accountability at the VCU-SOP. One of the recommendations by Burke and Minassians is for institutions to implement internal performance reporting.41 At the VCU-SOP, members of the Executive Committee are charged with setting target goals and responsible units for each KPI at the beginning of the fiscal year. Both the Executive and Outcomes Assessment Committees are responsible for designing and implementing actionable quality improvement strategies. This continuous review of KPIs ensures that we are using valid measures to assess the school’s performance and that we are measuring critical outcomes for accreditation purposes and state priorities, per Burke and Minassians’s recommendations.41 Furthermore, select KPIs applicable to each employee’s role have been integrated into annual performance evaluations. As a result of this process, the VCU-SOP added new items to already existing surveys and created new surveys to collect newly proposed KPIs (e.g., KPIs related to staff experience) or enhance collection of existing ones. As for external accountability, achievement of the nine KPIs mandated by the university is reported to the Vice Provost of Health Sciences who, in turn, communicates how the VCU-SOP contributes to University goals to higher levels of administration, as well as to alumni and donors.

One limitation is that the panel was not comprised of a group of experts in performance management, per best practices in conducting a Delphi technique.31 Additionally, the results are not generalizable to other pharmacy schools. However, our goal was to illustrate a way to engage faculty and staff in the process of refining the KPIs that other institutions may find useful.

CONCLUSIONS

This study illustrates how a Delphi consensus technique can be used in committee service as a means to promote faculty and staff involvement in school’s decisions, in this case KPIs selection, instead of following a top-down approach. The process resulted in the successful refinement of the number of KPIs to be tracked by the VCU-SOP while engaging faculty and staff.

Pharmacy schools’ administrators may opt to evolve to a more integrated perspective of organizational success by using widely known strategic management tools such as the balanced scorecard. The Delphi technique could be a viable approach to elicit specific measures of performance for the balanced scorecard dimensions, i.e., financial, customer, learning and innovation, and internal process perspectives. The Delphi technique can also be utilized for multiple other purposes in committee or college work when seeking to obtain consensus about specific skills, competencies, programmatic content, organizational aspects of the school, among others.

Footnotes

Contributor Information

Teresa M. Salgado, MPharm, PhD. Assistant Professor. Department of Pharmacotherapy & Outcomes Science, School of Pharmacy, Virginia Commonwealth University . Richmond, VA ( United States ). ude.ucv@odaglasmt .

Taylor N. Reynolds, BS. School of Pharmacy, Virginia Commonwealth University . Richmond, VA ( United States ). ude.ucv.liamym@ntsdlonyer .

Laura M. Frankart, PharmD, MEd, BCPS. Associate Professor and Director of Education and Assessment. Department of Pharmacotherapy & Outcomes Science, School of Pharmacy, Virginia Commonwealth University . Richmond, VA ( United States ). ude.ucv@nagromal .

David A. Holdford, BSPharm, MS, PhD. Professor. Department of Pharmacotherapy & Outcomes Science, School of Pharmacy, Virginia Commonwealth University . Richmond, VA ( United States ). ude.ucv@ofdlohad .

Joseph T. Dipiro, PharmD. Dean. Dean’s Office, School of Pharmacy, Virginia Commonwealth University . Richmond, VA ( United States ). ude.ucv@oripidtj .

VCU School of Pharmacy KPIs Redefinition Taskforce:

References