Study Setting

The study was conducted at HealthPartners Medical Group (HPMG), a multispecialty medical group in Minnesota that provided care to 175,000 adults in 18 clinics in 1996. The study was designed to take advantage of EMR implementation in 1 clinic of a multiclinic medical group by comparing results with those achieved in a clinic that was not using the EMR. On May 7, 1997, an EMR developed by Epic Systems was introduced at the intervention clinic, with extensive formal and ongoing one-on-one support provided to physicians. This EMR system included visit notes typed by physicians, automated ordering of pharmaceuticals, current displays of all laboratory and test results on request, a problem list, and other features. Physicians typically consulted the EMR on a computer monitor while in the examination room with each patient. All clinical data were loaded from several previous years, and after EMR implementation, paper charts were no longer available at the time of patient visits. An EMR enhancement in March 1998 included prompts to physicians if a patient with diabetes had no HbA_1c test within 6 months or no urine microalbuminuria test within 1 year. In mid-1999, further enhancements included prompts to physicians when diabetic patients had blood pressures of ≥130/85 mm Hg, LDL levels of ≥130 mg/dL, HbA_1c levels ≥8%, or no aspirin use if aged 40 years or older. The prompts and reminders were visible on the screen, but a response to them was not obligatory. At no time during this study was the EMR widely used to provide information or printouts directly to patients.

The EMR clinic was relatively small, with a stable staff of 4 to 5 physicians, and was a system leader in quality of care. The comparison clinic was selected because it was similar to the EMR clinic in size, physician stability, and suburban location, and the diabetic patients had similar baseline HbA_1c test values. No residency training was done at either study clinic.

The EMR evaluated in this report was developed by Epic Systems (Madison, Wis) and updated regularly as new systems and technology were developed. Support was provided through Information Services at Health-Partners, with expert consultation from Epic as needed. The cost of the EMR at this clinic is not available for public disclosure, but more resources were invested in EMR implementation and maintenance than were invested for other care improvement interventions at the comparison clinic or at other clinics in the medical group during the study period. The EMR clinic participated in other diabetes-related care improvement activities within the medical group during the study period, as previously described.¹³ All clinics in the medical group had access to physician-specific diabetes registries that were distributed quarterly in printed form, in-clinic diabetes teaching nurses for patient education, and a common diabetes clinical guideline developed regionally by the Institute for Clinical Systems Improvement (http://www.icsi.org).

The EMR used a Windows interface and a Virtual Basic (Microsoft Corp, Redmond, Wash) operating system linked to laboratory test results and pharmacy databases. Each examination room had a computer screen and keyboard, and physicians typically constructed office notes and orders while in the examination room with the patient. Although we did not formally evaluate acceptability of the EMR to physicians or patients in this study, it is widely believed at the EMR clinic and within HPMG that both physicians and patients are quite satisfied with the system.

Study Patients

To evaluate the impact of the EMR on process and intermediate outcomes of diabetes care, we focused on all adults with an established diagnosis of diabetes at study baseline in both study clinics. Those older than 18 years were classified as having diabetes mellitus if in calendar year 1994 they had either (1) 1 or more inpatient or 2 or more outpatient ICD-9 codes for diabetes (ICD-9 codes 250.xx), or (2) a filled prescription for a diabetes-specific drug (insulins, sulfonylureas, metformin, thiazolidinediones, alpha-glucosidase inhibitors, meglitinides). This diabetes identification method was previously evaluated with estimated sensitivity of 0.91, specificity of 0.99, and positive predictive value of 0.94.²¹

The clinic that each diabetes patient attended was identified in 1996, 1998, and 2000 based on number of visits and administrative data. Patients were included in the analysis only if they attended their original study clinic in all 3 of the study years and were still alive and enrolled in HPMG on December 31, 2000. We have previously shown that disenrollment rates for diabetes patients averaged 3.9% a year from 1993 to 1996, and that the HbA_1c level was not a significant predictor of either death or disenrollment in adults with diabetes.²² There were 122 study patients who met these inclusion criteria and whose diabetes care data are analyzed in this report; of these patients, 57 received care at the EMR clinic and 65 received care at the comparison clinic.

Dependent Variables

Dependent variables included measures of diabetes process of care and measures of intermediate outcomes of diabetes care. Process of care measures included the number of HbA_1c tests and LDL cholesterol tests done for patients with diagnosed diabetes in each study clinic in each of the calendar years 1996, 1998, and 2000. Additional process measures assessed whether patients met minimum thresholds for HbA_1c and LDL testing. Specifically, 3 threshold measures assessed whether the patient had at least 2 HbA_1c tests in a calendar year; at least 1 LDL test in a calendar year; or at least 2 HbA_1c tests and 1 LDL test in a calendar year.

Intermediate diabetes outcome measures included glycemic control and lipid control, as measured by HbA_1c and LDL test values in calendar years 1996, 1998, and 2000. All HbA_1c tests during the study period at these clinics were performed at a single accredited clinical chemistry laboratory using a standard liquid chromatography assay²³ with a coefficient of variation of 0.58% at an HbA_1c value of 8.8% and a normal range of 4.5% to 6.1%. In October 1998 a minor modification of the assay method was made, which did not change median HbA_1c values, but which led to statistically significant lower values for HbA_1c concentrations of more than 10%. This change in assay applied equally to both clinics in the study in both timing and degree.

LDL cholesterol (LDL) was calculated from measures of total cholesterol, high-density lipoprotein cholesterol, and fasting (12 hours or more) triglycerides using a standard equation.²⁴ All lipid assays were done in the same accredited clinical chemistry laboratory using standard assay techniques that did not change during the study period.

All test results from both clinics were entered directly into an electronic database at the central laboratory. In the EMR clinic, results were then downloaded into Epic. When collecting data for analysis, we drew all data from the electronic database at the central laboratory, assuring equivalent and complete capture of relevant laboratory test data for the EMR and comparison clinics. Neither clinic used desktop analyzers for any tests during the study period.

Independent Variables

There were 2 independent variables of interest. The first was an indicator of EMR use. The second was an indicator of the year (1996, 1998, or 2000) in which the outcome measures were taken. The interaction between these 2 independent variables constituted the effect of most interest for the study. A significant interaction between time and EMR status would indicate that the relationship between time and testing (either test frequency or test values) was different as a function of whether the patient was in an EMR or comparison clinic. Examining predicted cell means in the context of a significant interaction could provide evidence of whether the rate of testing or values of tests performed improved more in the EMR clinic than in the comparison clinic.

Covariates

Covariates included patient age, sex, and Charlson comorbidity score. Age and sex were obtained from administrative records at HPMG. Charlson comorbidity score was based on the method of Charlson et al²⁵ as modified by Deyo et al,²⁶ and Rush et al.²⁷ The modifications of Rush et al included (1) assigning points only when a diagnostic code appears 2 or more times during a 12-month period, and (2) excluding all points related to diabetes, because every patient in the analysis was identified as having diabetes as a condition of being in the study. This modified Charlson score has been validated and found to predict both mortality and costs of care.²⁷ Because of the nonnormal distribution of the Charlson score, patients were grouped and analyzed according to whether their Charlson score was less than 2, 2, or more than 2.

Plan of Analysis

The analysis was designed to test 3 hypotheses: (1) EMR use predicts the number of HbA_1c and LDL tests performed, (2) EMR use predicts the proportion of patients with recommended number of HbA_1c and LDL tests in a given year,^28,29 and (3) EMR use predicts change in values of HbA_1c and LDL tests with time.

Generalized linear modeling techniques (SAS version 8.2, SAS Institute, Cary, NC) were used to predict whether dependent variables changed with time in the EMR and comparison settings. Normally distributed continuous outcome variables (test values) were predicted using PROC MIXED, specifying a normal distribution and identity link function. PROC GEN-MOD was used for binary outcomes (testing thresholds) specifying a logit link function and a binomial distribution. Number of tests was predicted using both the normal distribution with an identity link function and the Poisson distribution with a log link function in PROC GENMOD with identical results. In all models, the unit of analysis was the patient, and the covariance structure among the repeated measures was unspecified. We evaluated relevant interaction terms and included those that were significant.

Implications of Findings for EMR Design

The data suggest that despite the substantial cost and increasing technical sophistication of EMRs, EMR use failed to achieve desirable levels of clinical improvement. Although it is commendable to perform regular HbA_1c testing, it is the HbA_1c level, not the number of HbA_1c tests done, that predicts future risk of complications and increased health care costs.^30–32 Moreover, recommendations for HbA_1c test frequency are not based on clinical trial data, but on expert opinion, whereas the HbA_1c level has been shown in clinical trials to be a strong predictor of outcomes and a moderate predictor of costs. The tenuous link between process and outcome of diabetes care is well described in the literature; frequency of tests and procedures is not strongly linked to HbA_1c levels or clinical events.^33,34

HbA_1c levels at both study clinics had been improving steadily for 4 years before implementation of the EMR. After EMR implementation, HbA_1c levels in the EMR clinic worsened for a period of about 2 years. This finding suggests an urgent need to develop more effective and less disruptive methods of introducing EMRs into clinics.

Figure 1 shows that the slope of improvement for HbA_1c levels in the EMR clinic was similar that for the non-EMR clinic after the EMR implementation period. This observation underscores that there are many ways to improve HbA_1c levels in the absence of EMRs—in fact, most recent reports of successful improvement in HbA_1c levels in the published literature were accomplished without the aid of EMRs.^13–16

That the EMR clinic did not outperform the non-EMR clinic indicates the need to further enhance the capabilities of EMRs. Based on our work and that of others, there are 2 particular domains of care where EMR performance needs major improvement.^35,36 The first domain is more sophisticated clinical decision support. For example, EMRs can integrate data on patient age, current HbA_1c, levels, renal function, liver function, comorbid conditions, and current medications to recommend specific changes in treatment that may be beneficial. Algorithm-driven decision support is now becoming more widely available in commercial EMRs, but it is utilized only in a minority of practices.^10,37

The second unrealized frontier is effective use of the EMR as a patient education and patient activation tool.^38,39 Although computerized information support to patients has shown promise in recent studies,⁴⁰ additional strategies, such as patient access to the medical record, prompts and reminders to patients, and generation of customized information and educational materials through EMR databases, should be more completely developed and evaluated.