Loading...

Abstract:

In this paper, we propose a novel deep convolutional neural network to solve the general multi-modal image restoration (MIR) and multi-modal image fusion (MIF) problems. Different from other methods based on deep learning, our network architecture is designed by drawing inspirations from a new proposed multi-modal convolutional sparse coding (MCSC) model. The key feature of the proposed network is that it can automatically split the common information shared among different modalities, from the unique information that belongs to each single modality, and is therefore denoted with CU-Net, i.e., Common and Unique information splitting network. Specifically, the CU-Net is composed of three modules, i.e., the unique feature extraction module (UFEM), common feature preservation module (CFPM), and image reconstruction module (IRM). The architecture of each module is derived from the corresponding part in the MCSC model, which consists of several learned convolutional sparse coding (LCSC) blocks. Extensive numerical results verify the effectiveness of our method on a variety of MIR and MIF tasks, including RGB guided depth image super-resolution, flash guided non-flash image denoising, multi-focus and multi-exposure image fusion.


Framework

In this paper, we aim to solve the general multi-modal image restoration and fusion problems, by proposing a deep convolutional neural network named the Common and Unique information splitting network (CU-Net). To the best of our knowledge, this is the first time a universal framework is proposed to solve both the MIR and MIF problems. Compared with other empirically designed networks, the proposed CU-Net is derived from a new multimodal convolutional sparse coding (MCSC) model,and thus each part of the network has good interpretability.

Our method can be applied to various multi-modal image restoration and fusion tasks, as the following figure shows.

The CU-Net architecture is as follows.

Our results

Some numerical results compared with other SOTA methods on three MIR tasks.

Guided modality (RGB image)                                  Target modality (Depth image) 4X upscaling

Guided modality (RGB image)                                    Target modality (Multi-spectral image) 4X upscaling

Guided modality (Flash image)                                  Target modality (Non-flash image) sigma=75

Under-exposed image                     Our fused image                 Over-exposed image

Far-focused image                  Our fused image              Near-focused image

Database and code

Download the paper, databse and code.

Our results

The testing datasets and our results in paper are available to be downloaded.













Software code


Software code is released on GitHub.