3-D Fully Convolutional Networks for Multimodal Isointense Infant Brain Image Segmentation

Dong Nie, Li Wang, Ehsan Adeli, Cuijin Lao, Weili Lin, Dinggang Shen

Research output: Contribution to journalArticle

  • 1 Citations

Abstract

Accurate segmentation of infant brain images into different regions of interest is one of the most important fundamental steps in studying early brain development. In the isointense phase (approximately 6-8 months of age), white matter and gray matter exhibit similar levels of intensities in magnetic resonance (MR) images, due to the ongoing myelination and maturation. This results in extremely low tissue contrast and thus makes tissue segmentation very challenging. Existing methods for tissue segmentation in this isointense phase usually employ patch-based sparse labeling on single modality. To address the challenge, we propose a novel 3-D multimodal fully convolutional network (FCN) architecture for segmentation of isointense phase brain MR images. Specifically, we extend the conventional FCN architectures from 2-D to 3-D, and, rather than directly using FCN, we intuitively integrate coarse (naturally high-resolution) and dense (highly semantic) feature maps to better model tiny tissue regions, in addition, we further propose a transformation module to better connect the aggregating layers; we also propose a fusion module to better serve the fusion of feature maps. We compare the performance of our approach with several baseline and state-of-the-art methods on two sets of isointense phase brain images. The comparison results show that our proposed 3-D multimodal FCN model outperforms all previous methods by a large margin in terms of segmentation accuracy. In addition, the proposed framework also achieves faster segmentation results compared to all other methods. Our experiments further demonstrate that: 1) carefully integrating coarse and dense feature maps can considerably improve the segmentation performance; 2) batch normalization can speed up the convergence of the networks, especially when hierarchical feature aggregations occur; and 3) integrating multimodal information can further boost the segmentation performance.

LanguageEnglish (US)
JournalIEEE Transactions on Cybernetics
DOIs
StateAccepted/In press - Feb 8 2018

Fingerprint

Image segmentation
Brain
Tissue
Magnetic resonance
Network architecture
Fusion reactions
Labeling
Agglomeration
Semantics
Experiments

Keywords

  • 3-D fully convolutional network (3D-FCN)
  • Biomedical imaging
  • Brain
  • brain MR image
  • Convolution
  • Image segmentation
  • isointense phase
  • Magnetic resonance imaging
  • multimodality MR images
  • Solid modeling
  • tissue segmentation

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Information Systems
  • Human-Computer Interaction
  • Computer Science Applications
  • Electrical and Electronic Engineering

Cite this

3-D Fully Convolutional Networks for Multimodal Isointense Infant Brain Image Segmentation. / Nie, Dong; Wang, Li; Adeli, Ehsan; Lao, Cuijin; Lin, Weili; Shen, Dinggang.

In: IEEE Transactions on Cybernetics, 08.02.2018.

Research output: Contribution to journalArticle

@article{05497e314fcd4563bd1d6ca52ef7e2fe,
title = "3-D Fully Convolutional Networks for Multimodal Isointense Infant Brain Image Segmentation",
abstract = "Accurate segmentation of infant brain images into different regions of interest is one of the most important fundamental steps in studying early brain development. In the isointense phase (approximately 6-8 months of age), white matter and gray matter exhibit similar levels of intensities in magnetic resonance (MR) images, due to the ongoing myelination and maturation. This results in extremely low tissue contrast and thus makes tissue segmentation very challenging. Existing methods for tissue segmentation in this isointense phase usually employ patch-based sparse labeling on single modality. To address the challenge, we propose a novel 3-D multimodal fully convolutional network (FCN) architecture for segmentation of isointense phase brain MR images. Specifically, we extend the conventional FCN architectures from 2-D to 3-D, and, rather than directly using FCN, we intuitively integrate coarse (naturally high-resolution) and dense (highly semantic) feature maps to better model tiny tissue regions, in addition, we further propose a transformation module to better connect the aggregating layers; we also propose a fusion module to better serve the fusion of feature maps. We compare the performance of our approach with several baseline and state-of-the-art methods on two sets of isointense phase brain images. The comparison results show that our proposed 3-D multimodal FCN model outperforms all previous methods by a large margin in terms of segmentation accuracy. In addition, the proposed framework also achieves faster segmentation results compared to all other methods. Our experiments further demonstrate that: 1) carefully integrating coarse and dense feature maps can considerably improve the segmentation performance; 2) batch normalization can speed up the convergence of the networks, especially when hierarchical feature aggregations occur; and 3) integrating multimodal information can further boost the segmentation performance.",
keywords = "3-D fully convolutional network (3D-FCN), Biomedical imaging, Brain, brain MR image, Convolution, Image segmentation, isointense phase, Magnetic resonance imaging, multimodality MR images, Solid modeling, tissue segmentation",
author = "Dong Nie and Li Wang and Ehsan Adeli and Cuijin Lao and Weili Lin and Dinggang Shen",
year = "2018",
month = "2",
day = "8",
doi = "10.1109/TCYB.2018.2797905",
language = "English (US)",
journal = "IEEE Transactions on Cybernetics",
issn = "2168-2267",
publisher = "IEEE Advancing Technology for Humanity",

}

TY - JOUR

T1 - 3-D Fully Convolutional Networks for Multimodal Isointense Infant Brain Image Segmentation

AU - Nie, Dong

AU - Wang, Li

AU - Adeli, Ehsan

AU - Lao, Cuijin

AU - Lin, Weili

AU - Shen, Dinggang

PY - 2018/2/8

Y1 - 2018/2/8

N2 - Accurate segmentation of infant brain images into different regions of interest is one of the most important fundamental steps in studying early brain development. In the isointense phase (approximately 6-8 months of age), white matter and gray matter exhibit similar levels of intensities in magnetic resonance (MR) images, due to the ongoing myelination and maturation. This results in extremely low tissue contrast and thus makes tissue segmentation very challenging. Existing methods for tissue segmentation in this isointense phase usually employ patch-based sparse labeling on single modality. To address the challenge, we propose a novel 3-D multimodal fully convolutional network (FCN) architecture for segmentation of isointense phase brain MR images. Specifically, we extend the conventional FCN architectures from 2-D to 3-D, and, rather than directly using FCN, we intuitively integrate coarse (naturally high-resolution) and dense (highly semantic) feature maps to better model tiny tissue regions, in addition, we further propose a transformation module to better connect the aggregating layers; we also propose a fusion module to better serve the fusion of feature maps. We compare the performance of our approach with several baseline and state-of-the-art methods on two sets of isointense phase brain images. The comparison results show that our proposed 3-D multimodal FCN model outperforms all previous methods by a large margin in terms of segmentation accuracy. In addition, the proposed framework also achieves faster segmentation results compared to all other methods. Our experiments further demonstrate that: 1) carefully integrating coarse and dense feature maps can considerably improve the segmentation performance; 2) batch normalization can speed up the convergence of the networks, especially when hierarchical feature aggregations occur; and 3) integrating multimodal information can further boost the segmentation performance.

AB - Accurate segmentation of infant brain images into different regions of interest is one of the most important fundamental steps in studying early brain development. In the isointense phase (approximately 6-8 months of age), white matter and gray matter exhibit similar levels of intensities in magnetic resonance (MR) images, due to the ongoing myelination and maturation. This results in extremely low tissue contrast and thus makes tissue segmentation very challenging. Existing methods for tissue segmentation in this isointense phase usually employ patch-based sparse labeling on single modality. To address the challenge, we propose a novel 3-D multimodal fully convolutional network (FCN) architecture for segmentation of isointense phase brain MR images. Specifically, we extend the conventional FCN architectures from 2-D to 3-D, and, rather than directly using FCN, we intuitively integrate coarse (naturally high-resolution) and dense (highly semantic) feature maps to better model tiny tissue regions, in addition, we further propose a transformation module to better connect the aggregating layers; we also propose a fusion module to better serve the fusion of feature maps. We compare the performance of our approach with several baseline and state-of-the-art methods on two sets of isointense phase brain images. The comparison results show that our proposed 3-D multimodal FCN model outperforms all previous methods by a large margin in terms of segmentation accuracy. In addition, the proposed framework also achieves faster segmentation results compared to all other methods. Our experiments further demonstrate that: 1) carefully integrating coarse and dense feature maps can considerably improve the segmentation performance; 2) batch normalization can speed up the convergence of the networks, especially when hierarchical feature aggregations occur; and 3) integrating multimodal information can further boost the segmentation performance.

KW - 3-D fully convolutional network (3D-FCN)

KW - Biomedical imaging

KW - Brain

KW - brain MR image

KW - Convolution

KW - Image segmentation

KW - isointense phase

KW - Magnetic resonance imaging

KW - multimodality MR images

KW - Solid modeling

KW - tissue segmentation

UR - http://www.scopus.com/inward/record.url?scp=85041812887&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85041812887&partnerID=8YFLogxK

U2 - 10.1109/TCYB.2018.2797905

DO - 10.1109/TCYB.2018.2797905

M3 - Article

JO - IEEE Transactions on Cybernetics

T2 - IEEE Transactions on Cybernetics

JF - IEEE Transactions on Cybernetics

SN - 2168-2267

ER -