Abstract

Background and Significance

Electronic health record (EHR) systems hold great promise for pediatric clinical research. 1 2 3 4 Primarily designed to improve clinical care, EHRs store vast amounts of routinely generated patient and provider data. 5 Through dissemination and development of sophisticated clinical research informatics tools, it is increasingly feasible to reuse EHR data for research. 6 7 This strategy is particularly attractive for the study of rare diseases including those affecting children, the study of uncommonly used drugs, or the characterization of infrequent adverse drug events in adults and children. 8 9 10 11 12 13

Despite the value of retrospective studies, prospective trials remain essential to drug development. 14 In response to the gap in pediatric drug labeling, the Eunice Kennedy Shriver National Institute for Child Health and Human Development (NICHD) created the Pediatric Trials Network (PTN), tasked under the Best Pharmaceuticals for Children Act with the design and conduct of prospective off-patent drug and device trials in children. 15 16 The PTN has enrolled >7,000 children in 38 phase 1 to phase 4 studies in 18 therapeutic areas.

From its inception, the PTN recognized the value of EHR data to support clinical trials. The network leveraged the Mednax Clinical Data Warehouse, the American Academy of Pediatrics-supported CER 2 (Comparative Effectiveness Research through Collaborative Electronic Reporting) database, and individual site EHRs to identify target drugs and study populations, inform protocol development, and support findings from prospective trials. 17 18 19 20 21 22

Objective

To take full advantage of data generated at sites, and because of our initial positive experience with EHR-generated data, the PTN sought to develop a multicenter pediatric inpatient data repository: The Pediatric Trials Network Data Repository. Data from this repository will be used in support of PTN trials and the labeling of drugs and devices for children. Herein, we describe the design and creation of this EHR-derived repository and share lessons learned in the process.

Methods

Repository Design

Each participating site was provided with an implementation guide, including instructions on how to perform EHR extractions, transformation, and testing, and how to package the data for submission. The transmissions were performed via secured file transfer protocol (FTP) delivery method. Data were downloaded from the secured FTP drop site and processed using 15 Informatica (Informatica, Redwood City, California, United States) ETL workflow processes, and stored in an Oracle repository (Oracle Corp., Redwood Shores, California, United States). All data were secured within a medium-level FISMA-compliant environment ( Fig. 1 ). The data intake process was designed to be agnostic to the specific type of EHR run at each site. Rather, sites were required to map their data according to the PTN specifications.

Open in a separate window

Fig. 1

Pediatric Trials Network (PTN) contextual data model.

Via a priori method, we defined a set of 520 rules designed to detect errors or warnings in the submitted data. We created an automated script that was built into the ETL process to check submitted data against all 520 rules. Errors were generated and logged if a data point violated a critical rule (e.g., invalid formats, required value not present, or nonunique primary key). Warnings were generated and logged if a data point violated a noncritical rule (e.g., certain reasonable logical inconsistencies, nonrequired value missing, or dates outside of expected sequences). Each new data submission was tested against all 520 rules, and a warning and error report was generated and shared with sites. For a data submission to be considered acceptable, 0% errors and <2% warnings were required across all records.

The database and ETL logic were first created and tested within development and validation environments. After successful loads in both, the same logic was migrated to the production environment. All three environments were constructed in an identical manner.

Results

We included all children <17 years of age at the time of admission between June 1, 2013, and June 30, 2017, whose admission source records were available in the EHR. The type of EHR used varied by site and represented two different platforms: Epic (Epic Systems Corporation, Verona, Wisconsin, United States) and Cerner (Cerner Corporation, North Kansas City, Missouri, United States). We excluded outpatient encounters and inpatient admissions without an EHR record. Our final data model included 147 mandatory and 99 optional data elements across 14 domains ( Fig. 2 ). Patient and site identity and admission date were used as primary keys to each file related to an admission, while patient and site identity were used as primary keys for all files not related to an admission. In addition to 10 PCORnet domains, we created four custom data domains because of their perceived importance to PTN trial design and execution: vital signs including weight, enteral and parenteral fluids, flowsheet assessments, and respiratory support. We included any flowsheet assessment provided with a tool name, result, and date and time of assessment.

Open in a separate window

Fig. 2

Repository design and infrastructure.

All data mapping was conducted by the sites using the implementation guide. The mapping was iterative, with each site first completing a trial submission on a small subset of their records across all 14 domains. Following each data submission and testing against rules, a report of warnings and errors was generated and shared with sites. Individual conference calls between each site and the coordinating center technical team were then scheduled to review results of each submission and discuss strategies for resolution if needed. Over a 14-month period, 177 data submissions were received. Final transfer of a site file to the repository required approval of the entire coordinating center team.

An average of 20 submissions per site (range 13–29) was required before data were moved into the repository. Site participation from kickoff to final data submission ranged from 8 to 14 months. The first data submissions were received within 2 months of sending out the implementation guide, and the time from first to final submission ranged from 6 to 10 months. Reasons for repeat submissions varied by site and submission, and were systematically recorded in site-specific project trackers. Most common reasons for repeat submissions included: failure of site to supply a required data element, failure of site to submit data in correct format, and failure of site to submit a required domain. Time to process the final complete site files ranged from 30 to 36 hours. All initial data submissions contained at least one error. However, all sites achieved 0% errors and <2% warnings.

The complete repository includes 386,159 inpatient encounters from 264,709 children across 9 participating sites. The encounters consist of 563,886,147 data points. Encounters were evenly distributed across the study period, but varied by site as expected based on hospital admission volume. Among the 14 data domains, 2 domains account for the majority of data points: “assessments” (33%) and “vital signs” (31%). Clinical and demographic data are shown in Table 1 .

Table 1

Summary of clinical data from participants in the repository

Variables	Counts, median (range), or %
Patients	264,709
Encounters	386,159
Patient age	2.3 (0–17)
Patient sex
Female	53%
Male	47%
Patient race
American Indian or Alaska Native	<1%
Asian	4%
Black or African American	19%
Native Hawaiian or Other Pacific Islander	<1%
White	58%
Multiple races	2%
Other	9%
Unknown	6%
Non-Hispanic	82%
Patient ethnicity
Hispanic	18%
Encounters by discharge year
2013	11%
2014	25%
2015	26%
2016	26%
2017	12%
Encounters by site
01	12%
02	21%
03	15%
04	7%
05	6%
06	9%
07	11%
08	7%
09	12%
Common drug administration records
Albuterol sulfate	974,551
Acetaminophen	656,159
Lorazepam	494,086
Furosemide	482,635
Morphine	385,013
Common diagnosis records
Feeding difficulties/intolerance	40,767
Gastroesophageal reflux disease	38,178
Fever, unspecified	25,388
Dehydration	18,143
Other dyspnea and respiratory anomaly	18,039

Open in a separate window

Discussion

We created a FISMA-compliant data repository extracted from inpatient pediatric EHR encounters at nine U.S. hospitals participating in the NICHD PTN. Our approach utilized a customized data model heavily influenced by the PCORnet format, site-based data mapping, a comprehensive set of data testing rules, and an iterative process of data submission. Crucial to our success were the close-working relationships between dedicated teams with expertise in IT, informatics, and clinical medicine at both the coordinating center and participating sites.

Multiple subspecialties have successfully leveraged clinical data from the EHR for research applications, including cardiovascular medicine, emergency care, epidemiology, and pediatrics. 8 9 22 34 35 36 37 38 39 PEDSnet is a PCORnet clinical data research network developed to provide the digital infrastructure for a national pediatric learning health system. 40 41 This objective requires PEDSnet to have extensive capabilities including dual methods for data submission and sharing (i2b2 and OMOP), terminology harmonization across sites and existing networks, a link to NICHD terminology efforts, the ability to integrate patient-reported outcomes and biospecimen data, and partnerships with national data partners including pharmacy benefits−managing companies and multipayer administrative databases. 42 Even though the scope of the PTN repository is much narrower, we were able to adapt features of the PEDSnet model including the use of a centralized data coordinating center, a predefined set of ETL conventions, and an iterative building process with prespecified data quality checks. 43 Similar to the PEDSnet approach, we initially focused on completeness of data and conformity with predefined conventions. As the repository evolves, we will gradually shift our focus toward assessing plausibility. Despite the considerable breadth and reach of PCORnet data, we chose to develop our own, modified data model. This approach was motivated by small but important differences in the anticipated use of the data, including the desire to capture clinical data commonly recorded on inpatient flow sheets, such as pain and sedation scores and vital signs, and data on daily fluid intake and output. These data are of particular importance for early phase pharmacokinetic and pharmacodynamic trials commonly conducted by the PTN. Further, not all PTN sites are members of the PCORnet distributed research network, and while we acknowledge that joint participation of all pediatric research sites in one data-sharing network would be preferable, this goal is beyond the current scope of the PTN.

The applications of EHR data to clinical research are diverse, and include protocol design, participant selection, translational research efforts, safety analyses, and trial execution. 44 45 46 47 In its effort to secure pediatric labeling of off-patent drugs and devices, the PTN has successfully reused EHR data collected by collaborating networks and individual participating sites. Analyzing EHR data, we described drug utilization practices and target molecules for study, reported on real-world drug safety, and performed comparative effectiveness studies. 18 48 49 50 51 The PTN has also leveraged EHR data as an innovative tool in pharmacometric analyses, a critically important step in pediatric drug development in concert with the Food and Drug Administration. 17 52 53 54 55 56

Despite these successes, more widespread application of EHR data within the PTN is contingent upon access to detailed clinical information including drug administration, laboratory results, vital signs, diagnoses and procedures, and clinical assessments. Data sources from previous PTN collaborating networks including the Pediatrix Medical Group Clinical Data Warehouse do not routinely capture this level of detail, collaborations with CER 2 were focused on outpatient data, and site-based collection of data from the EHR for a specific study is resource-consuming and inefficient. This multicenter repository provides access to detailed data, including new domains not routinely collected in registries such as vital signs, fluid intake and outputs, flowsheet assessments, and respiratory support variables.

This repository creates an opportunity to advance the PTN mission to design and conduct off-patent drug and device trials in children. Specific applications made possible by the data include clinical trial simulation, cohort enrichment strategies, synthetic control groups of patients whose data are captured in the EHR and used for comparison against prospective trial populations, retrospective safety assessments, and comparative effectiveness. To meet these goals, a team of clinicians, IT and informatics experts, and biostatisticians has been assembled to design and implement appropriate analytics, quality control, and data curation tools. The team will also leverage the repository to explore solutions to critical real-world data aresues including record linkage, missing data elements, and quality. 57 58 59 To maximize the pediatric public health benefit of the repository while maintaining data privacy and confidentiality, PTN investigators will be offered the opportunity to propose studies and research questions through a Web-based interface ( www.pediatrictrials.org ). Further, we will explore opening the repository to participation by other PTN sites.

We anticipate that through ongoing use of the repository, errors, discrepancies, and limitations of the data will become apparent. Whenever possible, we will remediate these with the assistance of our sites.

At this stage, several limitations and important lessons have been identified. By design, we did not include textual data, which is challenging to collect, deidentify, and analyze. As new natural language process methods become available, however, textual data reuse for clinical research will grow. 60 The repository also focused on inpatient data only. Collection of ambulatory data may require additional considerations of its formats, linking to inpatient encounters, and availability of qualified personnel to perform data collection. A successful example of outpatient EHR data use for research is the American Academy of Pediatrics CER 2 network. 22 To ensure success, CER 2 also relies on a common data model (derived from the OMOP model), and was built by allying existing outpatient practice networks, rather than recruiting individual clinics and sites. Because our data model relied heavily on the PCORnet model, which is itself derived from OMOP, our model can be viewed as yet another extension of OMOP focused on the inpatient setting. A similar effort within the PTN was beyond the scope of this work but could be implemented in future iterations of this repository. Site-based mapping, iterative submission and error reporting, and intense bidirectional communication was ultimately successful in creating a repository, but may be challenging to scale up to a larger number of sites. At this time, prospective new sites would have to undergo the same data submission process. While lessons learned to date may help identify errors and warnings prior to submission, we would anticipate a high workload for site research teams to process multiple rounds of submission prior to the elimination of all errors. In our experience, the majority of site-based efforts were concentrated on the mapping of data elements, a process that requires considerable informatics expertise. This is in turn associated with significant costs at the site level, which would need to be carefully considered and potentially shared between sites and coordinating institutions to ensure success. Finally, once a data repository is created, comprehensive high-quality analyses of these data will be performed by trained experts not traditionally involved in clinical research networks including biostatistics, IT, informatics, machine learning and advanced analytics, and systems medicine. Integration of this workforce into clinical trial teams will help maximize the value of EHR data.

Clinical Relevance Statement

We created a multicenter, pediatric inpatient, EHR-derived data repository to support PTN trials and the labeling of drugs and devices for children. In our multicenter model, clinical, informatics-trained, and information technology personnel performed site-based data mapping with central research support and intense bidirectional communication for the successful creation of our data repository.

Multiple Choice Questions

1. The data model of the newly developed PTN repository was based on which of the following national data standardization efforts:

   1. Health Level Seven International (HL7).

   2. Fast Healthcare Interoperability Resources (FHIR).

   3. Biomedical Research Integrated Domain Group (BRIDG).

   4. Patient-Centered Clinical Research Network (PCORnet).

   Correct Answer: The correct answer is option d, patient-Centered Clinical Research Network (PCORnet). We chose to pattern the first version of the PTN repository data points using the PCORnet data model. This choice was made after careful review of all existing data models, and alignment of their content with the requirement of our repository to help support drug and device trials in children. Despite PCORnet's strengths and significant advantages, the data model did not encompass all domains felt to be of relevance to pediatric drug and device development. Additional fields were therefore added, including vital signs, fluids, flowsheet assessments, and respiratory support.

2. Which of the following strategies were implemented in support of site-based teams responsible for data mapping?

   1. Financial incentives.

   2. Threat of study termination.

   3. An implementation guide.

   4. Written instructions only.

   Correct Answer: The correct answer is option c, an implementation guide was one of the strategies employed to support site-based research teams. Other strategies included individual conference calls, and customized error and warning reports. An early decision during the development process was to employ site-based rather than centralized data mapping. This decision was made based on several perceived advantages given the overall constraints of the project, but immediately raised the question of how best to support sites during the mapping efforts. The PTN team ultimately decided on a combination of written instructions in the form of an implementation guide, combined with individual conference calls with technical support teams, customized error and warning reports, and iterative submission processes designed to incrementally submit data and correct errors. Overall, the individual communication with sites was essential to the success of this project, although it raises some concerns regarding scalability of these efforts.

Acknowledgments

Funding Statement

Funding This work was funded under National Institute for Child Health and Human Development (NICHD) contract HHSN27520100–003I for the Pediatric Trials Network (PI: Danny Benjamin). C.P.H. receives support for research from NICHD grant K23HD090239. M.L. receives funding from FDA grant RO1 5R01FD005101–03.

Protection of Human and Animal Subjects

References