COVID‐19 lung infection segmentation from chest CT images based on CAPA‐ResUNet

2.1. Network structure

Figure 1 shows the structure of the proposed CAPA‐ResUNet. Inspired by the variant of UNet ⁹ and the encoder‐decoder structure ³¹ , ³² we use a similar frame, mainly including a down‐sampling path and an up‐sampling path. While the down‐sampling path expands the receptive field, it reduces the computational cost, and the up‐sampling path restores the missing feature information in the down‐sampling path. All encoder modules are composed of 3 × 3 convolution operations with stride 1, batch normalization (BN) layer, ³³ rectified linear unit (ReLU) activation function ³⁴ and residual block. ³⁵ The down‐sampling block is consisted of a 3 × 3 convolution operation with stride 2, a BN layer, a ReLU activation function and a pre‐activated residual block, ³⁶ and makes channel numbers remain unchanged. The structure of the residual block and the pre‐activated residual block is explained in Figure 2. In E1 (Figure 1), the first convolution operation changes channel numbers from 1 to 64, and the latter two convolution operations only extract feature information without changing channel numbers. Similarly, in other layers from E2 to E5, the number of channels turns 128, 256, 512, and 1024. When the number of channels is 1024, the feature information is up‐sampling through the content‐aware sampling operation, ³⁷ , ³⁸ , ³⁹ and then the feature in the down‐sampling path is connected with those in the up‐sampling path by jumping connection for supplementing feature information. From D4 to D1, the number of channels decreases to 512, 256, 128, and 64. When the number of channels is 64, 1 × 1 convolution operation and Sigmoid activation function are utilized to convert feature maps of final segmentation results into probability maps. Next, we will introduce the details of the network and loss function.

2.2. Residual block

In our model, the residual block is applied to the encoder module to extract feature information, as shown in the residual block in Figures 1 and 2A, and explicated in Figure 2A. In the residual block, the first is two 3 × 3 convolution layers with equal output channel numbers, and each convolution operation is followed by a BN layer and a ReLU activation function. To alleviate the gradient disappearance problem, we introduce the residual connection ⁴⁰ and append inputs to the final ReLU activation function.

Assuming that the input is $x$, the ideal mapping that the model needs to fit is $F\left(x\right)$, and this mapping is an identity mapping. However, due to the effect of multiple nonlinear layers, it is difficult to directly fit the identity mapping. After adding the residual connection, the model output is $F\left(x\right)+x$. To make $F\left(x\right)+x$ an identity mapping, we only need to set the weights and bias of multiple nonlinear layers to 0, that is, $F\left(x\right)=0$, and the mapping obtained at this time is an identity mapping. In the model training, the residual connection is easier to be optimized, ⁴⁰ and the gradient disappearance problem is also alleviated.

2.3. Pre‐activated residual block

Since the foreground distribution of COVID‐19 lesion segmentation data is complex, for instance, shapes, sizes, and locations of lesions are different, which has a certain impact on the training results of the network. In addition, during the training, the changes of network parameters at each layer will result in changes in data distributions at the latter layer.

For that matter, inspired by Lu et al., ³⁶ we designed the pre‐activated residual block by using BN layer, and applied the pre‐activated residual block to the down‐sampling block, as shown in Figure 2B. In a word, adding pre‐activated ResBlock before inputting at each layer, and then enter for learning. Inputs of each layer are preceded by a pre‐activated residual block, which is then input for learning.

2.4. Content‐aware up‐sampling module

Feature sampling is an important operation in image processing. At present, there are three main up‐sampling methods. The first one is a bilinear interpolation, but it only considers the adjacent sub‐voxel space, it cannot obtain sufficient semantic information. The second is deconvolution, which extends dimensions by utilizing convolution layers. However, deconvolution uses the same convolution for the whole image, limiting the capture of local information changes. It not only increases the parameters but also reduces the computational efficiency. The third is the feature up‐sampling operator content‐aware reassembly of features (CARAFE), ³⁷ which is mainly divided into two parts, namely, kernel prediction module and content‐aware reassembly module. First, the kernel prediction module is used to predict the sampling convolution kernel, and then take advantage of the content‐aware reassembly module completing the sampling and obtaining the feature maps. In this article, we adopt the third method above for up sampling (Algorithm 1). The specific steps are as follows:

2.5. Loss function

In the early stage of COVID‐19 infection, the proportion of lesion area in the whole image is small, which may lead to the prediction results of the network being more biased towards the background. Inspired by Taghanaki et al., ³⁰ we have effectively overcome this problem by using similar combo loss forms to balance the proportion of foreground and background.

where $N$ is the total number of voxels in the images. $\mathit{S}=\left\{s_i,i=1,2,\cdots ,N\right\}$ is the probability set of prediction voxels, $\mathit{G}=\left\{g_i,i=1,2,\cdots ,N\right\}$ is the probability set of corresponding label voxels.

Another important problem is that the boundaries of COVID‐19 lesion regions are fuzzy and locations are scattered and irregular, which makes the prediction results of the network more prone to misclassification. To solve this problem, we construct a loss function based on the area of misclassified regions. As shown in Figure 3, ${\Omega }_1$ and ${\Omega }_3$ are misclassified regions, and ${\Omega }_2$ is the correct segmented region. Taking ${\Omega }_1$ as an example, the purpose is to continuously reduce the area of ${\Omega }_1$ enclosed by ${\phi }_1\left(y\right)$ and ${\phi }_2\left(y\right)$. We use the idea of converting curve integral into double integral to construct the loss function of this part, as shown in Equation (2).

Since the integral form is difficult to realize and optimize in network training, we intend to discrete Equation (2). First, the region ${\Omega }_1$ is divided into $n$ rectangles, and then the area of $n$ rectangles is summed to obtain the approximate value of the area of the region ${\Omega }_1$. To simplify the calculation, we further zoom it into the product of the maximum distance between ${\phi }_1\left(y\right)$ and ${\phi }_2\left(y\right)$ and the distance from point A to point B, as shown in Equation (3).

where $y_A$ and $y_B$ are vertical coordinates of point A and point B, respectively, and ${\sigma }_i$ is the area of the red rectangle in Figure 3. Finally, the loss function based on the misclassified area is shown in Equation (4).

Finally, the total loss function $L_{\mathrm{seg}}$ in this article is the linear combination of $L_{\text{Combo}}$ and $L_{\text{Area}}$, as shown in Equation (5). Experimental results demonstrate that the linear combination of $L_{\text{Combo}}$ and $L_{\text{Area}}$ is effective to train model.

where $\lambda$ denotes the weight of $L_{\text{Combo}}$ and $L_{\text{Area}}$.

3.1. Datasets

The public dataset ⁴¹ was used in our experiment, which is the COVID‐19 Lung CT Lesion Segmentation Challenge—2020 ⁴² dataset, in which the CT data were provided by The Multi‐national NIH Consortium through the NCI TCIA public website. ⁴³ All of the CT data were positive for SARS‐CoV‐2 RT‐PCR, and the COVID‐19 lesions were labeled by the gold standard.

Since the challenge has ended, we only got part of the dataset, namely, we got a total of 13 595 CT slices. In this article, the dataset was grouped into the train set and the test set according to the ratio of 3:1. Specifically, the train set includes 10 876 CT slices and the test set contains 2719 CT slices. Before the training, we preprocessed the dataset, such as automatically cutting the image according to the boundary of the lung area and reshaping the image size into 256 × 256. In addition, we performed data augmentation by using the Transforms method in Pytorch, mainly including horizontal or vertical flip, random rotation 90°.

3.2. Training details

The experiments in this article are implemented using Pytorch ⁴⁴ and a server with two NVIDIA PH402 GPUs 32 GB card. Ranger optimizer ²⁹ was used to train the model. Ranger optimizer was the combination of RAdam ⁴⁵ and Lookahead. ⁴⁶ On the one hand, we used RAdam to reduce the variance at the beginning of training and alleviate the problem of easily converging to the local optimal solution. On the other hand, we enhanced training stability and decreased the training variance by using Lookahead.

Hyper‐parameters were tuned by performing five‐fold validation on the train set. All the control experiments used the same parameters. In our experiments, the number of training epochs was 60. Our proposed loss function $L_{\mathrm{seg}}$ was combined with the Ranger optimizer and weight decay 0.0001. The learning rate was initialized to $5\times {10}^{-4}$ and decayed 0.1 times at the 40th epoch, once every 5 epochs. Also, we used a batch size of 4. In our loss function, we set $\lambda$ to 0.5.

3.3. Evaluation metrics

To assess the performance of all models, five different metrics were used, namely, dice coefficient (Dice), sensitivity (Sen), specificity (Spe), precision (PC), and intersection over union (IoU).

1. Dice: Dice measures the similarity between the model segmentation results and the ground truth. Dice is defined as,

• 2

  Sen: Sen represents the proportion of the infected areas that are accurately predicted. Its definition is as follows,

• 3

  Spe: Spe represents the proportion of non‐infected areas which are accurately predicted and is defined as,

• 4

  PC: PC represents the probability of pulmonary infections which are accurately predicted, which is formulated as,

• 5

  IoU: IoU is the overlapping areas divided by the joint areas between segmentation results and the ground truth. Its definition is as follows,

where $\mathit{TP},\mathit{FP},\mathit{TN},\text{and}\mathit{FN}$ denote the true positive, false positive, true negative, and false negative between segmentation results and ground truth, respectively.

3.4. Comparative experiments

To evaluate the effect of our model, some comparative experiments were carried out. Concretely, CAPA‐ResUNet was compared with various advanced deep learning models, including UNet, ⁹ SegNet, ¹¹ CopleNet, ²⁴ DDANet, ²⁵ and CARes‐UNet. ²⁷ All models were trained from the open‐source codes in the article. In our experiments, all networks shared the same dataset to ensure fairness.

Compared with the other models mentioned above, the influence of label data complexity on experiments was reduced by adding pre‐activated residual blocks in our network. And considering the expansion of the wrong regions due to blurring lesion boundary, the area loss function was proposed to reduce the area of the false positive regions. Experiment results of all models on the COVID‐19 Lung CT Lesion Segmentation Challenge—2020 dataset are shown in Table 2.

Our model obtains the best results in all models, especially in Dice, Sen, Spe, PC, and IoU. Specifically, in terms of Dice, it is 2.51% higher than the second‐best model CARes‐UNet and 4.31% higher than the classic UNet model, while Dice was the significant evaluation index in image segmentation. Furthermore, while maintaining a high Spe, Sen also significantly increased by 11.6%. Compared with CARes‐UNet, the PC value of the model also increased from 0.527 to 0.533, which increased by 1.14%, and the IoU value of the model increased from 0.628 to 0.646, an increase of 2.87%. The effectiveness of the proposed model is fully proved.

Analysis of variance (ANOVA) based on Dice was calculated to test statistical differences between methods. When the p‐value is less than 0.05, it indicates a significant difference. The experimental results are shown in Table 3.

We selected CT images with different degrees of infection as presentations. The results of CT images segmentation are shown in Figure 4. It can be observed from Figure 4 that CAPA‐ResUNet was more accurate than any other model in the segmentation of COVID‐19 lesions.

First, for CT images with lighter COVID‐19 infection, the lesion area is relatively small. As shown in the first row of Figure 4, the segmentation results of CAPA‐ResUNet model are closer to ground truth, while the lesion areas segmented by UNet, SegNet, and CopleNet are smaller.

Second, for CT images with deep infection, the range of lesion area is expanded, the distribution is dispersed, and the shapes are complex and different, as shown in the CT images from the second to fourth rows in Figure 4. For the segmentation of such CT images, the results of CAPA‐ResUNet model show that the false‐positive and false‐negative regions are small, which are more accurate than those of other comparative models. In particular, the comparison of the image results of the third and fourth lines is the most obvious. The lesion regions segmented by SegNet, CopleNet, DDANet, and CARes‐Unet are much larger than those segmented by ground truth, and the lesion regions segmented by UNet are smaller than those segmented by ground truth.

Finally, for CT images with the deepest infection, as shown in the fifth row of CT images in Figure 4, most areas in the lungs have been infected. It can be seen from the segmentation results that the CAPA‐ResUNet model has better segmentation effect. In summary, it can be concluded from the segmented CT images that CAPA‐ResUNet model can better deal with CT images with different infection levels.

3.5. Ablation study

In this section, we performed a few experiments to verify the effectiveness of different components in our model, including pre‐activated residual block and area loss function based on the error segmentation regions. We showed the results in Figure 5. For the basic network, our network adopted the CARes‐UNet framework, using combo loss for training. As can be seen from Figure 5, all the proposed modules had great contributions to the performance of CARes‐UNet.

3.6. Discussion on hyper‐parameters

Furthermore, we also studied the effect of hyper‐parameter $\lambda$ on the predicted results, as shown in Table 4. The hyper‐parameter $\lambda$ was set to 0.4, 0.45, 0.5, 0.55, 0.6, and 0.7 in turn, and six groups of experiments were carried out. As can be seen from Table 4, when λ was set to 0.5, the best result was achieved with Dice up to 0.767. This demonstrated that the combo loss and area loss based on error segmentation regions were equally important in the model training. That is to say, for the segmentation of the pulmonary infection area, we should not only consider the imbalance of foreground and background categories but also consider the fuzzy boundaries of lesion regions to obtain better segmentation results.