Automatic Classification of Ultrasound Screening Examinations of the Abdominal Aorta

Study Setting

This retrospective observational study was reviewed and approved by the VA Greater Los Angeles Research Institutional Review Board with a waiver of informed consent. The study population included ultrasound imaged patients in the Imaging Department from January 1, 2009 through Dec 31, 2013. We are a 740 bed urban teaching hospital with over 1,300,000 outpatient visits per year. The gender mix of our patient population is primarily adult male 91.6 % and adult female 9.4 %. The reports were dictated by five different fellowship trained body imagers. The average number of years’ experience as an attending abdominal imager is 18.2 ± 10.4 years. Dictation was performed with speech recognition software integrated with our PACS system. Spell checking is done electronically but final approval on all changes must be approved by the radiologist.

The main objective of our research is to appropriately determine the correct VA national diagnostic code for AAA from radiology reports. The three steps required to complete the objective involve detection of the pertinent negative and positive AAA findings within the impression or conclusion section of an ultrasound report, (2) extract the measurement quantifying the maximal diameter from the report narrative, and (3) find the trigger words that define sub-optimal imaging studies. These three steps were implemented using the ConText algorithm. Table 1 shows the three AAA codes used to classify patients.

Evaluation (Gold Standard)

Three board certified radiologists, with 10+ years of post-graduate experience, manually tagged each report from 2012 to 2013 for pertinent positive, pertinent negative, and unacceptable image quality. The AC1 statistic, which is similar to a generalized kappa, allows one to access inter-rater reliability of multiple readers. For readers 1 and 2, AC1 = 0.9969, 95 % CI (0.9933–1.000), and 2-sided p < 0.001. Readers 1 and 3, AC1 = 0.9682, 95 % CI (0.9568–0.9796), and 2-sided p < 0.001. Finally, readers 2 and 3, AC1 = 0.9854, 95 % CI (0.9789–0.9919), and 2-sided p < 0.001. Our results indicate excellent agreement between independent manual tagging of the AAA reports by our three radiologists. The significant p value indicates that this strong agreement did not happen by chance. The final gold standard was decided by two additional radiologists who analyzed all the manually tagged results from the three radiologists and decided ground truth for those reports where disagreement occurred. The gold standard committee also reviewed other imaging exams (CT and/or MR) of the abdomen to determine ground truth when such concurrent studies were available.

Processing VA Patient Records

To obtain a set of AAA patient records, we queried the VA DHCP/VISTA/CPRS electronic medical record (EMR) system for all ultrasound (US) AAA screening examinations from 2009 to 2013 performed at VA Greater Los Angeles Healthcare System. The records in free-text format were then processed using the GATE (General Architecture for Text Engineering natural language processing system [12–14]. GATE is an open source text analytics engine for natural language processing developed at the University of Sheffield and includes an information extraction module called ANNIE (A Nearly-New Information Extraction System) which is composed of a tokenizer, gazetteer, sentence splitter, part of speech tagger, named entities transducer, and a co-reference tagger (Fig. 1) [12]. The text of each report was decomposed into sentences and each sentence in turn was parsed into tokens. Tokens were obtained by simply splitting the text of each sentence by using spaces as a delimiter. Terms of interest were then identified within each sentence and were mapped to their appropriate semantic representations. The semantic representation provides the meaning of the term of interest. The term of interest can be a negation, i.e., “no abdominal aortic aneurysm found,” positive affirmation, i.e., “3 cm abdominal aortic aneurysm found,” or ambiguous finding, i.e., “abdominal aortic aneurysm not visualized by poor quality of ultrasound exam.” In our particular case, the terms of interest consisted of phrase variations of AAA. The following is a sample list: abdominal aortic aneurysm, aortic aneurysm, and aneurysm of the abdominal aorta, ectasia of abdominal aorta, dilation of the abdominal aorta, etc. We were also interested in the size associated with each mention of AAA, and thus numeric terms were also identified as terms of interest. Numeric terms were extracted using a regular expression pattern. The advantage of a rule-based information extraction methodology includes the following: declarative statements, easy to incorporate domain knowledge, easy to comprehend and maintain, and easy to trace and fix errors. The disadvantages of rule-based text processing systems are the heuristic methodology, and tedious manual creation of new rules [15].

The general ConText algorithm in pseudo-code format:

1. Find trigger term in sentence,

2. If term is a pseudo-trigger term,

   • Find next trigger term,

3. Determine scope of trigger term,

4. If termination term within scope,

   • Terminate scope before termination term,

5. Assign appropriate contextual feature to corresponding clinical concept within scope. The contextual feature can be negation term, size measurement, or non-visual term applied to clinical concept.

ConText, which is an expansion of NegEx, relies on three types of terms: trigger terms, pseudo-trigger terms, and termination terms. In the case of negation, the trigger terms could be “no” or “not” along with a clinical concept (i.e., AAA). A clinical concept that falls within the scope of the trigger term will result in a negated clinical concept. Pseudo-trigger terms contain negation trigger terms but are false positive for negation of a clinical concept. For example, “no increase in AAA diameter measurement,” “no increase” is a pseudo-trigger term that does not negate the clinical concept AAA. A termination term such as “but” can terminate the scope of the negation before the end of the window, as in “The AAA exists, but is not measureable due to the for-shortened view.” In our experiment, the window between the trigger term and clinical concept was up to five terms.

These are the steps performed in the training phase:

1. Input AAA reports into GATE as GATE document.

2. Add GATE documents into GATE corpus

3. Process corpus with ANNIE utilizing custom Java Annotation Patterns Engine (JAPE) transducers [16, 17]. JAPE transducers allow finite state processing of annotations with regular expressions.

4. Generate feature vector based on rules that were captured during step 3.

5. Respectively code reports based on feature vector utilizing decision table classifier.

A feature vector consisting of JAPE transducer rules was generated for each finding that mentioned AAA. The vector consisted of five rules: negative assertion of AAA, positive assertion of AAA, measurement of abdominal aorta, mild ectasia of AAA, and measure value of abdominal aorta ≥3.0 cm. The training phase was an iterative process to improve the precision and accuracy of detecting the non-diagnostic phrases, and pertinent positive and negative findings. This phase involved processing 468 ultrasound reports from the year 2009 to 2011 using GATE’s built-in JAPE. Using JAPE, regular expressions rules were written to discover pertinent negative and positive findings within the reports. The training set also was used to identify measurement patterns found in the free-text reports. AAA diameter sizes are usually expressed as one-, two-, or three-dimension (anteroposterior (AP), transverse, and length) and described as (e.g., 2.9 cm, measuring 3.5 × 4.1 cm, 3.9 × 4.8 × 5.5 cm). Finally, we discovered trigger words for sub-standard image exam. The training phase helped identify NLP findings under which concept matching (proper identification of AAA pertinent positive, pertinent negative, and non-visual study concepts) of desired term failed. In addition, we were able to determine contextual phrases that identified non-diagnostic image studies. All 468 reports were ultrasound screening procedures for AAA, and the analysis of this particular type of radiology report focused on the impression section. The second phase involves testing the final versions of the custom JAPE transducers on the test dataset. A separate set of 1402 ultrasound AAA screening reports from 2012 to 2013 were analyzed using GATE. Two of the authors identified all of the pertinent positive and negative AAA findings in the reports and the other two authors were involved in the failure analysis of the GATE output.

The reports were processed using a Java-based driver for executing the GATE system in batch mode. The output of GATE is a set of annotations that provide syntactic or semantic labels to segments of text. More specifically, each annotation consists of a unique identifier, a document identifier, start and end indices, an annotation type, and a set of features. The document identifier and start and end indices uniquely specify the lexical location of each annotation. Annotation types are names given to the labels assigned to text segments, and features are other metadata associated with each annotation type. For instance, the “Token” annotation represents a token (e.g., a word) and token features may consist of the part-of-speech of the token, the token’s word root, or whether the token was capitalized. All annotations were stored in a backend relational database to facilitate retrieval and provide organizational structure. To maximize flexibility and reduce processing overhead, features were stored as Javascript Object Notation (JSON) strings because different annotation types will be associated with different sets of features.

Diameter Extraction

AAA diameter sizes are usually expressed as one-, two-, or three-dimension (anteroposterior (AP), transverse, and length) and described as (e.g., 2.9 cm, measuring 3.5 × 4.1 cm, 3.9 × 4.8 × 5.5 cm). Although there maybe one or more dimensions describing the aneurysm size. Only the maximum size of either AP or transverse dimension is selected for the final diameter measurement.

Outcome Measures

For each contextual feature (i.e., proper meaning of AAA depending on the other terms surrounding the concept AAA), we compared ContText’s value to a value assigned by the gold standard. We calculated recall, precision, and F-measure or harmonic mean of precision and recall using the following formulas:

We also calculated the 95 % confidence intervals (CI) for the outcome measures.

Limitations and Future Work

Some of ConText’s errors can be resolved by adding additional trigger and termination terms to the dictionary. There was a misspelled trigger term, i.e., “In the visible portion of the aorta there is no aneurysm but non- visualization of the complete abdominal aorta” (extra space after -) that was not detected by our algorithm which lead to a false negative for sub-standard image quality. This type of error was seen in Table 3, the test case was 3 but the classifier response was 1 since there was no mention of an abdominal aneurysm and the non-visualization contextual term was not detected. Another example of the same type of errors: “Ectasia of the abdominal aorta, mid portion of the aorta not seen.” This report indicates possible portion of the aorta containing aneurysm, but after the comma the sentence acknowledges poor visualization of the middle portion of the abdominal aorta. Since a measurement is not given, the “ectasia” was not significant but the non-visualization of a portion of the aorta led the classifier to select response 3, but the majority of the abdominal aorta had no aneurysm greater than 3 cm. In this case, the report is insufficient in the description of the visual evidence. We also encountered a word pair form of aneurysm not seen in our training set “aneurysmal/dilation.” For example: “Extensive calcified atherosclerotic plaque throughout the abdominal aorta, without evidence of aneurysmal dilatation.” Another typical example: “Mild aneurysmal dilation of the infernal abdominal aorta.” Table 3 shows test 2 but the response is 1 in this case since our algorithm failed to identify an abdominal aortic aneurysm. The most common error for the non-visualization of the abdominal aorta was an additional finding that indicated negation of the AAA for the portion of the abdominal aorta that was visible. Another difficult case was the compound sentence, “mild ectasia of the abdominal aorta and the iliac artery was not visualized.” Our algorithm mistakenly identified the “not visualized” portion of the sentence with the abdominal aorta. This again is depicted as test 1 but response 3 in Table 3. Currently, our study represents only one VA institution, but generalization of this work to additional VA hospitals is possible by discovering each individual institutions particularly unique phrasing of positive, negative, and visually impaired AAA exams.