Deep Learning-Based Approaches for Nucleus Segmentation

Ⅰ. Introduction

The segmentation of cell nuclei is a crucial step in the analysis of biomedical microscopy images^[1]. It serves as the foundation for various medical analyses like cell counting^[2] and cell type classification^[3]. However, this task is quite challenging due to variations in staining, tissue types, and the diverse visual characteristics of different cell types, all of which affect how nuclei appear^[1]. Manual segmentation of nuclei is not only time-consuming but also becomes impractical when dealing with large datasets. Furthermore, the accuracy of segmentation results relies heavily on the expertise of individuals and is often not reproducible. Consequently, there is a significant demand for automated methods for instance-based nuclei segmentation in microscopy images.

In the literature, various techniques have been proposed for automating the segmentation of nuclei, spanning from basic background subtraction to more advanced methods. These include approaches like the Otsu-based method^[4], the watershed method^[5], Grab Cut^[6], and active contour^[7]. However, these conventional methods have their limitations. They are often sensitive to the choice of parameters, and their effectiveness is typically limited to specifc categories of structured nuclei.

In contrast, deep learning-based techniques have gained significant traction in the field of medical imaging, being applied to various applications such as medical image super-resolution^[8], classification^[9], and notably, medical imaging segmentation^[10,11]. These approaches have also been extensively utilized for cell and nuclei segmentation, with numerous deep learning-based methods dedicated to this task^[12]. For instance, Pan et al. introduced a deep semantic network designed for nuclei segmentation in pathological images^[13]. Vuola et al. employed MaskRCNN to segment nuclei^[14]. Van et al.presented DeepCell, a method tailored for analyzing cells in live-cell imaging^[15]. Ronneberger et al. proposed the U-Net model, which has become a popular choice for nuclei segmentation^[16]. Zeng et al. developed a modified version of U-Net for nuclei segmentation, incorporating features like residual blocks and channel attention mechanisms^[17]. Zhou et al. introduced CIA-Net, which utilizes two separate decoders for distinct tasks and incorporates a multilevel information aggregation module to capture dependencies between nuclei and their contours^[18].

Fig. 1.

FCN architecture

In the current study, we conduct a feasibility study on deep learning-based approaches for nuclei images convolution Network (FCN)^[19], SegNet^[20], UNet^[16], DoubleUNet^[21]. We demonstrate the utility of such models by evaluating them for cell segmentation and recommend which one is most suitable for nuclei segmentation tasks.

The paper is organized as follows: in Section II, we review several deep learning models for image segmentation. Section III presents the comparative results when apply deep learning reviewed in Section II. Conclusions of this paper are given in Section IV.

Ⅱ. Materials and Methods

2.1 Fully Convolutional Neural Network

Long et al. introduced the Fully Convolutional Neural Network (FCN) for addressing semantic image segmentation challenges^[22]. This architecture has since been extended to tackle various other segmentation tasks, including ventricle segmentation^[19]. As presented in Fig.1, the network includes 15 convolution layers, 3 max pooling layers, followed by upsampling layers and a classifier layer.

The network is divided into two main parts, contracting path (also called encoder) and expanding path (called decoder). The encoder consists of convolutional layers used to preserve the spatial structure of the feature map and max pooling layers to reduce the resolution. In the improved version of FCN by Tran^[19] each convolution layer is followed by a rectified linear unit (ReLU) and a mean-variance normalization (MVN). The purpose of MVN is to normalize the intensity distribution of the feature map so that the feature map pixel values have zero mean and unit variance. The decoder, symmetric to the encoder, consists of transposed convolution layers, and upsampling layers. The feature map in the decoder is combined with the corresponding feature map in the encoder to preserve spatial information that might be lost during pooling operations. Finally, a classifier such as softmax is used to produce class probabilities for each pixel in the image to be segmented.

2.2 SegNet

The SegNet architecture was proposed by Badrinarayanan et al.^[20] for semantic pixel-wise segmentation. Similar to the FCN architecture, the SegNet consists of an encoder path account for downsampling with convolution and max-pooling layers, and a decoder path used for upsampling the feature maps to the same size of the input image. To retain information lost due to pooling, Ninh et al.^[23], proposed the skip connection mechanism to fuse between the encoder to decoder of the SegNet. In addition, compared to the original SegNet model, their model has also fewer learned parameters since the downsampling and upsampling layers are reduced. The improved version of the SegNet architecture is presented in Fig. 2.

Fig. 2.

SegNet architecture

Fig. 3.

U-Net architecture

2.3 U-Net

The U-Net neural network architecture, introduced by Ronneberger and colleagues in reference^[16], has become widely recognized as the standard approach for medical image segmentation tasks. It draws inspiration from the Fully Convolutional Network (FCN)^[19] and can be conceptually divided into two main sections: the contracting part (encoder) and the expanding part (decoder). The contracting part employs a combination of convolutional layers and max-pooling operations to achieve downsampling, while the expanding part involves upsampling and convolutional layers. To preserve crucial spatial information that might be lost during downsampling, U-Net employs skip connections, where feature maps from the encoder are concatenated with their counterparts in the decoder at the same spatial resolution. This architectural design facilitates accurate segmentation of medical images.

2.4 DoubleU-Net

The DoubleU-Net has recently been proposed by Jha et al.^[21] shown in Fig 4. As can be seen from this figure, DoubleU-Net starts with an VGG-19 pre-trained network as an encoder in the first sub-network, and then follows by an astrous spatial pyramid pooling (ASPP) block and finally is the decoder. The input image is passed through the first sub-network, then the first output is created. This output then multiplies with the input, the result becomes the input of the second sub-network and the second output is created. Finally, these two outputs are concatenated and become the output of DoubleU-Net. What makes DoubleU-Net differ from U-Net is that there are two subnetworks. Squeeze-and-Excitation block is used inside to enhance the power of convolution layer. Moreover, the skip connection is performed from the first encoder to the first decoder and from either the first encoder or the second encoder to the second decoder, which maintains the spatial resolution and enhances the quality of the output feature maps.

Fig. 4.

DoubleU-Net architecture

2.5 Loss function

For deep learning-based nuclei image segmentation, the binary cross-entropy function and Dice loss are widely used to train the networks. Let y be the predicted map by the network, [TeX:] $$\hat{y}$$ be the corresponding label map, and N be the number of pixels of the maps. The Binary Cross Entropy loss is express as:

The Dice loss is computed as

In our paper, we evaluate a loss function which is conjunction between the Binary Cross Entropy loss and Dice loss. While Dice loss captures the result between pixel overlap, BCE loss is associated with models that output probabilities. The combination of these two loss functions will help the optimization process become more general and converge better. Our loss function is express as

Ⅲ. Experimental Results

3.1 Datasets

The Data Science Bowl 2018 (DSB2018) dataset presented a global challenge to scientists, tasking them with the automatic identification and segmentation of cells within a collection of microscopic images. The primary objective was to develop image segmentation techniques that could be universally applied across multiple experiments without the need for additional human intervention. This approach aimed to decrease the time required for image quantification, enabling future researchers to readily apply and evaluate various experiments for both research and clinical purposes.

The Data Science Bowl 2018 (DSB2018) dataset comprises a total of 670 training pairs and 65 testing pairs. Each pair consists of an image and its corresponding masks. Notably, the DSB2018 dataset encompasses five distinct types of cell images, including Small Fluorescent, Purple Tissue, Pink and Purple Tissue, Large Fluorescent, and Grayscale Tissue. These images exhibit variations in their content and characteristics, making the dataset diverse and challenging to work with, as depicted in Fig 5.

Fig. 5.

Five types of nuclei images in Data Science Bowl 2018 dataset

Fig. 6.

Distribution of the dataset for training and testing.

The distribution of data of nucleus types of the train and test sets of the DSB2018 dataset are given in Fig. 6.

3.2 Training

We implemented our model and utilized the Adam algorithm for optimizing the trainable parameters of the model with original learning rate [TeX:] $$s \times 10^{-4}$$. The training process is looped on the dataset for 200 epochs with batch size 16 and the data is augmented after every epoch. Early stopping and Reducelronplateau have already been used.

3.3 Evaluation metrics

We use Dice Similarity Coefficient (DSC) to evaluate the segmentation performance by the neural network. The DSC metric is defined as

where TP, FN, FP represent for True Positive, False Negative and False Positive, respectively.

In addition to DSC, we also use the Intersection over Union (IoU) index as an alternative evaluation measure, defined as

where TP, FN, FP represent for True Positive, False Negative and False Positive, respectively.

3.4 Results and validation

In this research paper, we undertake an investigation into the practicality of employing deep learning-based methods for the segmentation of nucleus images. Our focus is on evaluating the performance of various cutting-edge deep learning models on the Data Science Bowl 2018 (DSB2018) dataset. The models we assess encompass the FCN, SegNet, U-Net, and DoubleU-Net architectures. To quantitatively assess their segmentation results, we present a comprehensive evaluation in Tab 1, which includes metrics like the Dice Similarity Coefficient (DSC) and Intersection over Union (IoU).

As depicted in Table 1, it becomes evident that the DoubleU-Net model outshines its counterparts, achieving the highest DSC and IoU scores. In comparison to its predecessor model, U-Net, DoubleU-Net demonstrates remarkable improvements, surpassing it by 1.8% in DSC and 2.5% in IoU. Moreover, DoubleU-Net also exhibits substantial advancements in Recall and Precision scores, surpassing U-Net by 7.9% and 2.3%, respectively.

Furthermore, when pitted against the FCN and SegNet models, DoubleU-Net exhibits superior performance in terms of DSC, IoU, and Precision. These findings position DoubleU-Net as the standout model, delivering the best overall performance among all the state-of-the-art methods we evaluated for nucleus image segmentation.

However, the pursuit of higher performance inevitably introduces greater complexity into the model. In the case of DoubleU-Net, it comprises two pairs of encoder-decoder structures, and each of these encoder-decoder pairs is essentially a U-Net model. While this complexity is instrumental inachieving the impressive segmentation results mentioned earlier, it does come with the trade-off of reduced computational efficiency. In contrast to U-Net and other models like FCN and SegNet, DoubleU-Net's computational speed is notably slower. This means that while DoubleU-Net excels in accuracy and segmentation quality, it may not be the ideal choice in scenarios where real-time or near-real-time processing is a critical factor, as its computational demands are more substantial due to its intricate architecture.

Fig. 7.

Quantitative segmentation results by state of the arts neural networks on the DSB2018 dataset

In Table 2, we delve into a detailed experiment to compare the output of the two subnetworks in DoubleU-Net, when every network can be considered as an U-Net model. We can observe that the output from Network 2 is more accurate than the output from Network 1, which confirms the preeminence of DoubleU-Net in the Tab 1 when developed from a single U-Net network. This all-encompassing approach allows us to draw robust and well-founded conclusions regarding the DoubleU-Net's capability to handle nuclei image segmentation tasks, helping to improve model performance and provide more accurate predictions.

In Table 3, we showcase the results achieved through the fusion of the Binary Cross-Entropy Loss and Dice Loss functions, resulting in improved model performance. Through a thorough analysis of the enhancements observed in each model, we can make informed conclusions regarding the appropriateness and effectiveness of the selected loss function. It's worth noting that the BCE loss function utilized in the study by Jha et al.^[21] may encounter challenges related to sigmoid saturation, wherein the model's output reaches extreme values close to 0 or 1. Consequently, the incorporation of the Dice loss serves as a countermeasure to alleviate this effect. The results are shown in Table 3.

To provide better evidence of higher performance, we create visual representations of segmented results produced by the aforementioned models in Fig7. Upon careful scrutiny of this figure, it becomes apparent that the segmentation masks generated by DoubleU-Net display the closest resemblance and alignment with the ground truth annotations. In other words, when comparing the model's output to the actual reference data, DoubleU-Net consistently demonstrates a superior ability to accurately identify and delineate the nuclei in the images. This visual evidence further reinforces the notion that DoubleU-Net outperforms other models in terms of segmentation quality and precision.

Ⅳ. Conclusion

In this study, we have explored the feasibility of utilizing deep learning-based methods for the segmentation of nuclei images. We conducted experiments using several state-of-the-art models on the Data Science Bowl 2018 dataset. Our results and evaluations revealed that the DoubleU-Net model achieved superior segmentation performance when compared to other state-of-the-art alternatives. Looking ahead, our future research will be directed towards reducing the training time or enhancing the accuracy of the DoubleU-Net for nuclei image segmentation, as well as extending its application to other segmentation tasks.

Additional strategies to enhance the approach's performance may involve considering alternatives like switching to a different pre-trained encoder, adopting a more contemporary neural network architecture, or seeking out an improved loss function