Affordability

One major barrier to corpus annotation is affordability. Minimizing the manual effort required and time spent to generate annotations would be a considerable contribution to the development of semantic resources.

Pre-annotation, providing machine-generated annotations based on e.g. dictionary lookup from knowledge bases such as the Unified Medical Language System (UMLS) Metathesaurus [11], can assist the manual efforts required from annotators. A study by Lingren et al. [12] combined dictionaries with regular expressions to pre-annotate clinical named entities from clinical texts and trial announcements for annotator review. They observed improved reference standard quality, and time saving, ranging from 14% to 21% per entity while maintaining high annotator agreement (93-95%). In another machine-assisted annotation study, a machine learning system, RapTAT, provided interactive pre-annotations for quality of heart failure treatment [13]. This approach minimized manual workload with significant improvements in inter-annotator agreement and F1 (89% F1 for assisted annotation compared to 85%). In contrast, a study by South et al. [14] applied cue-based dictionaries coupled with predictions from a de-identification system, BoB (Best-of-Breed), to pre-annotate protected health information (PHI) from synthetic clinical texts for annotator review. They found that annotators produce higher recall in less time when annotating without pre-annotation (from 66-92%).

Another strategy to mitigate time and cost is to use different annotation methods for creating corpora meant to train a statistical system versus creating a gold standard. A recent study found that double annotation and consensus annotation are not always necessary when creating corpora to train a statistical system [15]. They showed that there was no statistically significant difference in results when training a model on single annotated data compared to double annotated plus consensus. A three step process is suggested: 1) use double annotation for only a small targeted sample to ensure guideline adherence, 2) allow annotators to work independently on different sections of the corpus, 3) train a machine learning model on the human annotations and apply this to a new dataset.

Utility

Utility of clinical texts can be affected when clinical eponyms such as disease names, treatments, and tests are spuriously redacted, thus reducing the sensitivity of semantic queries for a given use case. For example, if mentions of Huntington’s disease are spuriously redacted from a corpus to understand treatment efficacy in Huntington’s patients, knowledge may not be gained because disease/treatment concepts and their causal relationships are not extracted accurately. One de-identification application that integrates both machine learning (Support Vector Machines (SVM), and Conditional Random Fields (CRF)) and lexical pattern matching (lexical variant generation and regular expressions) is BoB (Best-of-Breed) [25-26]. BoB applies the highest performing approaches from known de-identification systems for each PHI type, resulting in balanced recall and precision results (89%) for a configuration of individual classifiers, and best precision (95%) was obtained with a multi-class configuration. This system was also evaluated to understand the utility of texts by quantifying clinical information loss following PHI tagging i.e., medical concepts from the 2010 i2b2 Challenge corpus, in which less than 2% of the corpus concepts partially overlapped with the system [27].

Generalizability

Generalizability is a challenge when creating systems based on machine learning. In particular, systems trained and tested on the same document type often yield better performance, but document type information is not always readily available. By creating training and testing sets on clinical documents that were partitioned into similar types – categorized by measures of writing complexity and clinical vocabulary usage – de-identification results were improved (avg. F1 92%), compared to using randomly selected clusters (avg. F1 88%), on a collection of over 4500 various document types from Vanderbilt University Medical Center [28].

Another challenge related to generalizability is to apply and evaluate de-identification methods not only on various document types, but also on corpora in non-English languages. Toward this goal, Grouin and Névéol [29] developed a reference corpus in French for de-identification. Two pre-annotation tools were used, one rule-based and one CRF-based, evaluated by two annotators. The rule-based system produced better pre-annotations, but the manual revision of the CRF-based system was faster. In both cases, human agreement was high (> 90% F1), and only 20 documents were needed to build a statistical system that outperformed pre-annotation tools. Two gold standard sets of French clinical notes were created. For Swedish, CRF models were also used to refine a set of de-identification annotations, along with manual revision, resulting in a new gold standard and system performance of 80% F1 [30]. PHI annotations on a subset of this corpus (100 notes) have been pseudonymized [31] and approved for release to the research community. In Korea, clinical notes are written both in English and Korean. To handle Korean names in a de-identification system, a heuristic approach using regular expressions was adopted and verified on 6,502 clinical notes from the Asian Medical Center, resulting in 89% precision and 97% recall [23].

Scalability

Scalability of de-identification for larger corpora is also a critical challenge to address as the scientific community shifts its focus toward “big data”. Deleger et al. [32] showed that automated de-identification models perform at least as well as human annotators, and also scales well on millions of texts. This study was based on a large and diverse set of clinical notes, where CRF models together with post-processing rules performed best (93% recall, 96% precision). Moreover, they showed that the task of extracting medication names on de-identified data did not decrease performance compared with non-anonymized data.

Adaptability to the Clinical Domain

Several standards and corpora that exist in the general domain, e.g. the Brown Corpus and Penn Treebank tag sets for POS-tagging, have been adapted for the clinical domain. Fan et al. [34] adapted the Penn Treebank II guidelines [35] for annotating clinical sentences from the 2010 i2B2/VA challenge notes with high inter-annotator agreement (93% F1). This adaptation resulted in the discovery of clinical-specific linguistic features. This new knowledge was used to train the general-purpose Stanford statistical parser, resulting in higher accuracy than models trained solely on general or clinical sentences (81%).

New morphological and syntactic processing applications have been developed for clinical texts. cTAKES [36] is a UIMA-based NLP software providing modules for several clinical NLP processing steps, such as tokenization, POS-tagging, dependency parsing, and semantic processing, and continues to be widely-adopted and extended by the clinical NLP community. The variety of clinical note types requires domain adaptation approaches even within the clinical domain. One approach called ClinAdapt uses a transformation-based learner to change tag errors along with a lexicon generator, increasing performance by 6-11% on clinical texts [37].

Semantic Analysis Method Development – Information Models and Resources

One notable effort for a rich information model with several annotation layers is the MiPACQ (Multi-source Integrated Platform for Answering Clinical Questions) dataset consisting of annotated Treebank POS tagged tokens, PropBank predicate-argument frames, and UMLS encoded entities on a corpus of randomly-selected Mayo Clinic clinical and pathology notes related to colon cancer with high agreement (93%, 89-93% and 70-75%, respectively) [40]. This dataset has promoted the dissemination of adapted guidelines and the development of several open-source modules.

Other development efforts are more dependent on the integration of several information layers that correspond with existing standards. The latter approach was explored in great detail in Wu et al. [41] and resulted in the implementation of the secondary use Clinical Element Model (CEM) [42] with UIMA, and fully integrated in cTAKES [36] v2.0.

The organization of shared tasks, or community challenges, has also been an influential part of the recent advancements in clinical NLP not only in corpus creation and release, annotation guideline development and schema modeling, but also in defining semantically-related tasks. Furthermore, NLP method development has been enabled by the release of these corpora, producing state-of-the-art results [17].

Many of these corpora address the following important subtasks of semantic analysis on clinical text.

Named Entity Recognition and Contextual Analysis

Correctly identifying the entities or concepts to which semantic modifiers or relations belong is crucial for information extraction. Often, concepts are defined as noun phrases (e.g. diabetes mellitus), requiring algorithm solutions that deal with sequences of words, either rule- or machine learning-based, as shown in the solutions for previous challenges – the 2010 i2b2 challenge [43], the 2014 ShARe/CLEF eHealth challenge [44] and the SemEval 2015 Task 14: Analysis of Clinical Text [45] - on concept classification, with system performance as high as 85% F1. In clinical settings, semantic type information is also essential, for instance, knowing that a concept is a problem, a test or a treatment, as in the definition of the 2010 i2b2 challenge [43]. Clinical entity recognition has also been studied for non-English languages. For instance, one study employed CRF models on Swedish clinical data for the types disorders, findings, pharmaceuticals and body structures, resulting in F1s ranging between 69-81% [46], in line with or slightly lower than results reported for English.

For accurate information extraction, contextual analysis is also crucial, particularly for including or excluding patient cases from semantic queries, e.g., including only patients with a family history of breast cancer for further study. Contextual modifiers include distinguishing asserted concepts (patient suffered a heart attack) from negated (not a heart attack) or speculative (possibly a heart attack). Other contextual aspects are equally important, such as severity (mild vs severe heart attack) or subject (patient or relative).

The ShARe (Shared Annotated Resources) corpus - a subset of discharge summaries, radiology, echocardiogram, and electrocardiogram reports from the MIMIC II database [47] - consists of templates with disease/disorder events encoded with SNOMED CT concept unique identifiers (CUI), with rich contextual attributes from the Clinical Element Model (CEM) [42] and with temporal expression mentions [44]. This dataset is unique in its integration of existing semantic models from both the general and clinical NLP communities.

In the 2014 ShARe/CLEF eHealth task 2, in an effort to leverage this annotated dataset, several approaches were taken to normalize semantic modifiers such as body site and severity which included optimizing cTAKES modules, developing rules based on resources from UMLS, and employing grammatical relations [48]. For example, Dligach et al. [49] treat these two problems as a relation extraction task, building SVM models and evaluating on two clinical corpora, resulting in F1 scores of 74-91% for body site and 91-93% for severity. The created models have been released as cTAKES modules. In addition to normalization of specific modifiers, the SemEval 2015 Task 14: Analysis of Clinical Texts also included an end-to-end system evaluation [45] that assessed NLP system performance for identifying and normalizing disease/disorders and their modifiers from the ShARe corpus.

An ensemble machine learning approach leveraging MetaMap and word embeddings from unlabeled data for disorder identification, a vector space model for disorder normalization, and SVM approaches for modifier classification achieved the highest performance (combined F1 and weighted accuracy of 81%) [50].

When encoding semantic concepts, lexicon- and rule-based NLP systems have the advantage of being almost language independent since the underlying algorithms do not necessarily depend on the source language. However, they require language-specific rules and lexicons. pyConTextNLP [51], a rule-based system for classifying assertions (negation and/or uncertainty modifiers) of disease mentions, was ported from English to Swedish [52]. The system relies on a cue lexicon with scoping rules. To create a Swedish lexicon, the authors translated and added cues from several sources, and final overall results were reported as 81% F1. When comparing negation and uncertainty cues across languages, the most frequent cues are often similar, but rarer cues are more prone to individual language particularities [53-54]. Further, the negation detection algorithm NegEx was evaluated on additional languages, where negation cues were translated also to French and German [53]. The system has also been adapted to Dutch, where adaptations for the contextual attributes negation, experiencer, and temporality were developed through translations from English along with enhanced rules and regular expressions [55]. Final results for negation and experiencer were high (> 87% and > 99% F1, respectively), but lower for historical and hypothetical temporality properties (26-54% and 13-44% F1, respectively).

Experiencer and temporality attributes were also studied as a classification task on a corpus of History and Physical Examination reports, where the ConText algorithm was compared to three machine learning (ML) algorithms (Naive Bayes, k-Nearest Neighbours and Random Forest). There were no statistically significant differences in results for classifying experiencer between these approaches, but the ML approach (specifically, Random Forest) outperformed ConText on classifying temporality (historical or recent), resulting in 87% F1 compared to 69% [56].

Coreference Resolution

A challenging issue related to concept detection and classification is coreference resolution, e.g. correctly identifying that it refers to heart attack in the example “She suffered from a heart attack two years ago. It was severe.” NLP approaches applied on the 2011 i2b2 challenge corpus included using external knowledge sources and document structure features to augment machine learning or rule-based approaches [57]. For instance, the MCORES system employs a rich feature set with a decision tree algorithm, outperforming unweighted average F1 results compared to existing open-domain systems on the semantic types Test (84%), Persons (84%), Problems (85%) and Treatments (89%) [58]. Another approach deals with the problem of unbalanced data and defines a number of linguistically and semantically motivated constraints, along with techniques to filter co-reference pairs, resulting in an unweighted average F1 of 89% [59]. Domain knowledge and domain-inspired discourse models were employed by Jindal & Roth on the same task and corpus with comparable results (unweighted average F1 between 84-88%), where the authors concluded that most recall errors could be handled by addition of further domain knowledge [60].

Temporal Reasoning

Temporal modeling has been the focus of many recent studies. In order to understand disease progression, adverse drug reactions, and other clinically relevant events over time, semantic models of temporality are needed. For instance, knowing when particular symptoms were present for a specific disease can be used to build predictive models to ensure timely treatment. The TimeML model [61], a rich model to represent temporal information in text through events (high blood pressure, heart attack), time expressions (two years ago), and their temporal relation (high blood pressure BEFORE heart attack), has been adapted in at least two ways to the clinical domain. Styler et al. [62] adapted the model to pathology and clinical texts from Mayo clinic, creating the THYME (Temporal History of Your Medical Events) corpus, while another adaptation was used in the 2012 i2b2 challenge [63-64]. For example, in contrast to previous corpora, the THYME events are encoded to the linguistic head of a phrase, and a new temporal expression type was introduced to capture time expressions such as preoperative. The THYME corpus was released as part of the SemEval-2015 Task 6: Clinical TempEval challenge [65], where the system approach results met or were close to human agreement on all subtasks except temporal relationships using machine learning approaches [66].

The first step in a temporal reasoning system is to detect expressions that denote specific times of different types, such as dates and durations. A lexicon- and regular-expression based system (TTK/GUTIME [67]) developed for general NLP was adapted for the clinical domain. The adapted system, MedTTK, outperformed TTK on clinical notes (86% vs 15% recall, 85% vs 27% precision), and is released to the research community [68]. In the 2012 i2b2 challenge on temporal relations, successful system approaches varied depending on the subtask. For instance, hybrid approaches combining rule-based systems such as HeidelTime [69], SUTIME [70] and GUTIME [67] with CRF or SVM machine learning models proved useful for time expression classification (up to 90% span F1), CRF models for event span detection (up to 92% F1), SVM models for event attribute detection (86% accuracy), while temporal relationships were classified with a variety of approaches, resulting in up to 68% F1 [63].

Other studies define coarser time representations. For instance, Raghavan et al. [71] created a model to distinguish time-bins based on the relative temporal distance of a medical event from an admission date (way before admission, before admission, on admission, after admission, after discharge). The model was evaluated on a corpus of a variety of note types from Methicillin-Resistant S. Aureus (MRSA) cases, resulting in 89% precision and 79% recall using CRF and gold standard features. In a study to classify patient history episodes in Bulgarian discharge notes, the authors defined rules to identify temporal markers (absolute or relative moments of time), resulting in 87% precision and 68% recall, and the direction of time for the episode starting point (backwards or forward) resulting in 74% precision [72].

Most studies on temporal relation classification focus on relations within one document. Cross-narrative temporal event ordering was addressed in a recent study with promising results by employing a finite state transducer approach [73].

Identifying Disease and Associated Symptomology

A method for identifying progress notes pertaining to diabetes was developed using a supervised machine learning framework (SVM), using a bag-of-words (BoW) representation, on notes from different institutions, resulting in F1 scores > 93% [84]. Interestingly, richer NLP features (named entities, synonym resolution, negation) were not found to be useful for this task. Similarly, Yetisgen-Yildiz et al. [85] obtained best results (79% F1) using n-gram features (uni-, bi- and trigrams) in a study to identify patients confirmed to have Acute Lung Injury. They found that assertion values were not useful for this task. Their study was evaluated on a corpus of 1,748 free-text chest x-ray reports related to patients at an intensive care unit at the Harborview Medical Center.

Another example in psychiatry showed that models incorporating NLP (using the HiTeX system [86]) improved determining mood states for diagnosing major depressive disorders compared to using diagnostic codes alone (area under receiver operating characteristic curve of 85-88% vs 54-55%) [87]. The underlying NLP methods were mostly based on term mapping, but also included negation handling and context to filter out incorrect matches.

NLP has also been used for mining clinical documentation for cancer-related studies. Spacíc et al. [88] present a review of current state-of-the art in this area, where they conclude that named entity recognition methods perform well (F1 between 80-90%) but that there is room for improvement for handling non-standard wordings and that the main bottleneck for progress in this area is the lack of available corpora.

Assigning Diagnostic Billing Codes

ICD-9 and ICD-10 (version 9 and 10 respectively) denote the international classification of diseases [89]. ICD codes are usually assigned manually either by the physician herself or by trained manual coders. They are used primarily for billing purposes for hospital administrations. Manually assigned codes are, however, often erroneous. In an investigation carried out by the National Board of Health and Welfare (Socialstyrelsen) in Sweden, 4,200 patient records and their ICD-10 coding were reviewed, and they found a 20 percent error rate in the assignment of main diagnoses [90]. NLP approaches have been developed to support this task, also called automatic coding, see Stanfill et al. [91], for a thorough overview. The best performing systems obtain F1-scores of around 90%. Perotte et al. [92], elaborate on different metrics used to evaluate automatic coding systems. Other recent approaches for automatic coding support are described in e.g. Martinez et al. [93].

Detecting Adverse Drug Events

An important aspect in improving patient care and healthcare processes is to better handle cases of adverse events (AE) and medication errors (ME). A study where NLP was used to automate detection of IV infiltrations, narcotic medication over sedation and dosing errors in a neonatal intensive care setting provided higher sensitivity and positive predicted value compared with manual trigger tools (as high as 100% precision and recall for some AE/ME types) [94]. A study on Danish psychiatric hospital patient records [95] describes a rule- and dictionary-based approach to detect adverse drug effects (ADEs), resulting in 89% precision, and 75% recall. Another notable work reports an SVM and pattern matching study for detecting ADEs in Japanese discharge summaries [96].

Bridging the Healthcare Policy and Practice Gap

There are new governmental policies and initiatives, e.g., Meaningful Use, advocating for next generation EHR technologies to enhance the efficiency and accuracy of healthcare delivery to the patient through clinical decision support, patient engagement, self-reported/self-monitored data integration, and quality measure reporting [104]. For example, the Precision Medicine Initiative advocates for the development of tools to integrate patient genetic, environmental, and lifestyle data (e.g., data from medical/personal devices or social media) into the electronic medical record to support precision medicine e.g., patient-centered prevention, diagnostic, and treatment models for disease [105]. Similarly, the European Commission emphasizes the importance of eHealth innovations for improved healthcare in its Action Plan [106]. Such initiatives are of great relevance to the clinical NLP community and could be a catalyst for bridging health care policy and practice.

Integrating New Media for Accessing Population Health Status

Furthermore, with growing internet and social media use, social networking sites such as Facebook and Twitter have become a new medium for individuals to report their health status among family and friends. These sites provide an unprecedented opportunity to monitor population-level health and well-being, e.g., detecting infectious disease outbreaks, monitoring depressive mood and suicide in high-risk populations, etc. Additionally, blog data is becoming an important tool for helping patients and their families cope and understand life-changing illness.

We wish to thank Pierre Zweigenbaum, Aurélie Névéol, Wendy Chapman, and Stephane Meystre for valuable comments on this survey. This work was partially funded by Swedish Research Council (350-2012-6658), the Department of Veteran Affairs CREATE (CRE 12-312), National Library of Medicine (NLM R01LM010964) and the project High-Performance Data Mining for Drug Effect Detection at Stockholm University, funded by the Swedish Foundation for Strategic Research under grant IIS11-0053.