Anatomy-guided deep learning for object localization in medical images

Outline of paper and approach

A schematic representation of the proposed approach, which we will refer to as AAR-DL, is shown in Figure 1. The AAR-DL approach consists of 4 stages which are shown by 4 modules in the figure – Automatic Anatomy Recognition-Recognition (AAR-R), Deep Learning-Recognition (DL-R), refined Automatic Anatomy Recognition-Recognition (rAAR-R), and refined Deep Learning-Recognition (rDL-R).

The first module AAR-R performs recognition of objects in a given body region B via the AAR approach [1] by making use of the already created anatomy model of the objects in B. It outputs a fuzzy model mask FM^t(O), a transformed version of the original fuzzy model FM(O), for each object O included in the AAR anatomy model, which indicates where object O is likely to be in input image I. FM^t(O) provides the needed region-of-interest (ROI) information in the form of a container, shaped like O, albeit fuzzy, for the second module DL-R. DL-R outputs a stack BB(O) of 2D bounding boxes (BBs) for each O by detecting the BBs in each slice within the ROI for O. In the third module rAAR-R, the AAR fuzzy model is refined by making use of the more precise localization information in BB(O) provided by DL-R. rAAR-R deforms the fuzzy model and outputs a refined recognition result (fuzzy model) rFM^t(O) for each object O. Finally, the fourth module rDL-R, performs refined DL-based recognition by utilizing the refined fuzzy model rFM^t(O) as an additional input channel and outputs a refined stack of 2D BBs, rBB(O), for each object O. This module has its own pre-trained network which is denoted rDL-R model.

Automatic Anatomy Recognition-Recognition (AAR-R module)

AAR is a general approach [1,9], developed before the advent of DL techniques, based on fuzzy anatomy modeling for recognizing and delineating all objects in a body region. It consists of three stages – model building, object recognition, and object delineation. Since the goal of this paper is object recognition, here we are concerned with only model building and recognition processes of AAR.

Deep Learning-Recognition (DL-R module)

DL-R uses a 3-channel image and the recognition mask FM^t(O) output by AAR-R to determine the region for search for 2D BBs, and outputs a stack of 2D BBs, denoted BB(O), for each object O as described in this section. DL-R takes a slice-by-slice approach, and as such, the three channels represent the slice in question and the two neighboring slices in the superior and inferior directions.

Considering accuracy, efficiency, and flexibility, our recognition network design is inspired by one-stage dense detection networks including RetinaNet [2] and YOLO [3, 4]. In order to capture richer contextual dependencies, an optimized attention mechanism is also incorporated into our recognition network. Target objects are separated into two groups: sparse and non-sparse. Spatially less compact and less filling, thin, tubular, and small objects, such as the salivary glands, thoracic esophagus, and thoracic spinal cord, form the sparse group. Large, blob-like, and spatially compact objects, such as the lungs and heart, form the non-sparse group. For non-sparse objects, DL-R cares more about semantic and abstract information from higher/deeper levels of the neural network. Conversely, the network cares more about detailed structural information from lower/shallow levels of the network for sparse objects. Thus, anatomic knowledge (NI) is infused into the network architecture at the design stage itself. The overall network architecture is depicted in Figure 2.

Although feature maps directly generated from the backbone network contain information from different levels, they are not sufficient for effective object recognition which involves classification (“what it is”) and regression (“where it is”) tasks. The neck network further integrates and refines these feature maps. A neck network is usually composed of several bottom-up paths and several top-down paths. Networks that utilize this mechanism include Feature Pyramid Network (FPN) [5], Path Aggregation Network (PAN) [6], weighted Bi-directional Feature Pyramid Network (BiFPN) [7], and Neural Architecture Search Feature Pyramid Network (NAS-FPN) [8]. To improve the performance of feature maps, FPN extracts in-network feature hierarchy, with a top-down path as well as lateral connections to propagate semantically strong features. It is known that neurons in higher layers strongly respond to entire objects while those in lower layers are more likely to be activated by local texture and patterns. To recognize a sparse object like trachea or proximal bronchial tree, not only local information but also information from deeper network layers is needed, since the locations of these structures have a close relationship to the lungs around them.

In order to further enhance the recognition capability of the entire neck network by propagating strong responses of low-level patterns, instead of the FPN and inspired by PAN, we build a path with clean lateral connections from the low level to top level. Figure 3 shows the complete structure of our neck network. This design is based on our desire that accurate recognition of non-sparse objects also requires detailed local information, especially at the object edges and corners. The lungs in 2D slices usually have sharp corners anteriorly in the superior slices. To localize lungs including such sharp angles accurately, local information from lower layers is also required. The complementary information gathered in different layers from the backbone network is fully exploited by the neck network. Each building block takes a higher resolution feature map and a coarser map through lateral connections and generates the new feature map Fi.

Object recognition in medical images is often more challenging than in natural images. However, in medical images there is abundant prior information and there are reasonable assumptions that can be made to further improve recognition performance. For example, parotid glands and submandibular glands almost always occur in pairs, but are separated by a certain distance. Attention mechanisms are designed to take advantage of such information. The superiority of attention mechanisms has been demonstrated especially in complicated segmentation applications.

In this paper, we apply a Dual Attention Network to adaptively integrate local features with their global dependencies further for each Fi. We call this attention network Self Attention (SA) module given that it involves attention inside each feature map Fi. In the SA module, an attention mechanism is implemented in spatial and channel dimensions of each feature map. The spatial attention component selectively aggregates the feature at each location by a weighted sum of the features at all locations. Similar features have connections to each other regardless of their distances. Channel attention selectively emphasizes channel interdependence by integrating associated features between channels. At the design core of convolutional neural network (CNN), the receptive field in a convolutional operation is a local neighborhood operator. This imposes a limitation on modeling global/expansive and richer contextual representation. Owing to the complexity of human anatomy, organs are not related to just the adjacent tissues alone. The SA module enables finding any tissues that have potential connection with the target organ despite their distance. Each feature map contains spatial and channel dimension information. In order to fully process the feature maps, the SA module consists of Position Attention Module (PAM) and Channel Attention Module (CAM) as depicted in Figure 4.

For PAM, F is an input feature map of size (W, H, C) from the previous step, denoting the width, height, and number of channels, respectively. To make Figure 4 clearer, we replace Fi by F. F is passed through a convolution layer to generate two new feature maps F’ and F’’ of size (W, H, C/8). They are reshaped to size ((W × H), C/8) and (C/8, (W × H)). Then a matrix multiplication is performed between the two feature maps and a softmax layer is attached to generate the spatial attention map S^p with size ((W × H), (W × H)). Meanwhile, F is passed through another convolution layer and reshaped into F’’’ of size ((W × H), C). Then, a matrix multiplication is performed between F’’’ and S^p to obtain a new ((W × H), C) feature map. Subsequently, a map S^pam is generated with the shape as the input feature map F by a reshape operation. Finally, a trainable parameter α is attached to S^pam and an element-wise sum operation with feature map F is performed. The attention feature map P^AM of PAM is generated as:

where α is initialized to 0. Gradually, the network learns to increase α to enhance the importance of spatial attention. α allows the network to first rely on the cues in the local neighborhood which are easier to train, and then to gradually learn to assign more weight to the non-local cues.

For CAM, the network is similar to PAM but more simplified. Different from PAM, without the convolution operation, input feature map F is reshaped into F’ and F’’ of size (C, (W × H)) and ((W × H), C), respectively. Then a matrix multiplication is performed between them and a softmax operation is followed to generate S^C which has size (C, C). Meanwhile, F is passed through another convolution layer and reshaped into F’’’ of size ((W × H), C). Then, a matrix multiplication is performed between S^C and F’’’ to obtain a new feature map of size ((W × H), C). Subsequently, a map S^cam is generated with the shape as the input feature map F by a reshape operation. Finally, the attention feature map C^AM of CAM is generated in the same way as PAM:

Similar to α in PAM, β is a trainable parameter and initialized to 0, and gradually, the network learns to increase β. Finally, the output of the SA module is the sum of the two attention feature maps: