D2Net: Deep Denoising Network in Frequency Domain for Hyperspectral Image

Erting Pan, Yong Ma, Xiaoguang Mei, Jun Huang,Fan Fan, and Jiayi Ma

Dear Editor,

Since the existing hyperspectral image denoising methods suffer from excessive or incomplete denoising, leading to information distortion and loss, this letter proposes a deep denoising network in the frequency domain, termed D2Net.Our motivation stems from the observation that images from different hyperspectral image (HSI)bands share the same structural and contextual features while the reflectance variations in the spectra are mainly fallen on the details and textures.We design the D2Net in three steps: 1) spatial decomposition, 2) spatial-spectral denoising, and 3) refined reconstruction.It achieves multi-scale feature learning without information loss by adopting the rigorous symmetric discrete wavelet transform (DWT)and inverse discrete wavelet transform (IDWT).In particular, the specific design for different frequency components ensures complete noise removal and preservation of fine details.Experiment results demonstrate that our D2Net can attain a promising denoising performance.

Introduction: Due to various unstable factors in the complex imaging chain, HSIs are always contaminated by noises, which will severely degrade the visual qualtiy and affect their further analysis and subsequent interpretations [1], [2].Therefore, removing HSI noise is of utmost significance for HSI exploitation.

Traditional HSI denoising methods commonly employ a definite model such as low-rank matrix recovery [3], domain transform [4],sparse representation [5], [6], tensor decomposition [7], etc.Unfortunately, they suffer from manual parameterization for specific noise cases and befall complex optimization problems.With the benefit of solid nonlinear representative abilities and fewer priors, convolutional neural networks (CNN)-based techniques have achieved impressive success in image restoration [8].However, every coin has two sides.Few priors imply a large number of parameters, resulting in huge redundant calculations, especially with high-dimensional HSIs.On the other hand, mainstream CNN-based denoising approaches typically follow either an encoder-decoder [9] or a high-resolution (single-scale) [10] feature processing architecture.The former achieves extensive contextual feature learning by upsampling and downsampling operations, but it loses fine spatial details, making it challenging to reconstruct clean HSIs accurately.The latter do not vary the spatial resolution and the limitation of the receptive field makes such networks incapable of encoding contextual information.Furthermore, due to a neglect of the intrinsic HSI spatial-spectral properties, these methods represent quite finite denoising performance.Therefore, investigating a tailor-made denoising approach is an urgent and challenging task, which requires efficiently removing HSI noises while carefully preserving the high-frequency details.

We have observed that capturing from the same scene, images in different bands show strong correlations.As evident in Fig.1, they share the same structural and contextual features, which are also called low-frequency features.On the other hand, the reflectance difference between bands may induce by details, textures, and noises,which mainly belong to high-frequency features.Motivated by this,we argue that denoising HSIs in the frequency domain can achieve better results and propose D2Net (as shown in Fig.2) to efficiently and precisely restore a clean HSI.First, we replace common up-sampling and down-sampling operations with DWT/IDWT, which is mathematically strict and symmetric, to provide rich frequency features for domain transformation and clean HSI reconstruction.Second, DWT/IDWT produces multi-scale features without any information loss, enabling us to design sub-branches for denoising in different frequency components.In particular, we propose a progressive spatial-spectral mixed convolution block (PMCB) to protect the effective transfer of high-frequency information.Third, we deploy a spatial-spectral consistency regularization block (SCRB) to explore its coherence further and finely reconstruct the clean HSI.The experimental results prove that the proposed method has a good trade-off between efficiency and denoising performance.

Fig.1.Examples of an HSI [1] taken from the PolyU hyperspectral face database.Sample bands covering the visible range from 420 nm to 690 nm in 30 nm intervals.

Fig.2.The proposed D2Net framework for HSI denoising.

Methodology: The goal of the HSI denoising task is to recover a clean HSIXfrom a noise-contaminated HSIY, whereX,Y∈R(C×H×W), andCpresents the number of spectral bands,HandWdescribe their spatial scale.

1) Spatial decomposition: As aforementioned, mainstream CNNbased HSI denoising methods easily fail in recovering spatial and spectral details.On the contrary, denoising in the frequency domain utilizing strictly symmetric DWT/IDWT enables multi-scale feature learning without information loss, alleviating this problem.In this paper, we utilize DWT in the second level with theHaarwavelet kernel to spatially decompose the noisy HSIYbefore denoising so that the network can perform customized and accurate denoising for different frequency components.It can be formulated as

whereYL(2)is in size of (B,1,C,H/2,W/2),YH(2)andYH(1)are in size of (B,3,C,H/4,W/4),Bis the batch size, 1 and 3 indicate the corresponding number of wavelet components.Intuitively, the employed DWT can derive wavelet subbands in multi-resolution,allowing multi-scale feature learning and benefiting the non-local similarities exploration in HSIs.Besides, DWT/IDWT would not affect the end-to-end training of our network, making the proposed D2net simple and effective.

Fig.3.Illustration of two main blocks in detail: (a) progressive spatial-spectral mixed convolution block; (b) spatial-spectral consistency regularization block.

Experiments: We organize training and evaluation of the proposed D2Net via mimicking synthetic Gaussian noise in different intensities and typically mixed noise cases on the ICVL hyperspectral dataset (http://icvl.cs.bgu.ac.il/hyperspectral/).To fairly assessment, we chose a series of advanced denoising methods for comparison, including BM4D [4], TDL [7], GLF [3], HSID-CNN [10],QRNN3D [9] and DSWN [12].We employ five metrics for HSI denoising performance evaluation, inclusive of peak signal-to-noise ratio (PSNR), structure similarity (SSIM) and feature similarity(FSIM) for spatial-based image quality measurement, spectral angle mapper (SAM) for spectral fidelity evaluation, and time-consuming during testing per HSI.

1) Quantitative comparisons: As listed in Table 1, where Cases 1,2, 3 are: Gaussian noise intensity of σ =70, blind σ, mixed noise of Gaussian noise and deadline noise, D2Net represents superior in the majority of quantitative metrics, especially the spatial quality metrics.It indicates the superior flexibility of our proposed D2Net.However, it falls slightly (0.007) on the SAM metric behind the QRNN3D in the complex noise case.It might be caused by the fact that the DWT/IDWT are conducted only in the spatial domain while QRNN3D emphasis spectral features by recurrent.On the other hand,specific designs in D2Net like PMCB and SCRB have greatly compensated for the gap in spatial-spectral fidelity, bringing considerable gains involving nearly 1dB gains in PSNR and average 0.005 incomes in SSIM and FSIM, but a bit of sacrifice in speed (average 0.04 s slower than QRNN3D).

Table 1.Quantitative Results on the ICVL Dataset.The best and the second results are shown in red and blue, respectively.

2) Spatial quality comparisons: Fig.4 illustrates denoising results of some representative scenes.From the specifics aspect, taking the second-row results in Fig.4 as an example, traditional method like TDL has removed some noise but still obtain poor results; QRNN3D shows a cleaner result but loses some details due to excessive denoising; DSWN retains more high-frequency detail information, but fails to eliminate high-frequency noise.In contrast, our proposed D2Net obtains the best denoising results with its model superior, achieving high fidelity recovery and showing results much clearer with fewer artifacts and sharp edges.These results suggest that the proposed D2Net has a more robust capability to remove HSI noises.

Fig.4.Illustration of comparison experiments on synthetic noises and real HSIs.Bounded by the black dotted line, the left side shows denoising results (involving zoom-in region and residual noise map) on the ICVL dataset and recovered spectral curves of the sampled pixel, where three rows from top to bottom are in three noise cases; the right side presents denoising results on the Indian Pines dataset.

3) Spectral fidelity comparisons: To verify the superiority of our method in spectral fidelity, we select one typical pixel in each noise case and draw the recovered spectral curves of some representative comparing methods in Fig.4.Apparently, compared to the other results, the recovered spectra curves of our D2Net are much closer to the reference.It indicates that our method accurately eliminates the negative effect of noise in the spectral domain and further confirms the advantage of our D2Net in maintaining high spectral fidelity.

4) Denoising on real HSIs: We also conduct denoising experiments on the Indian Pines dataset to verify the effectiveness and flexibility of our model.As Fig.4 shown, our method still shows stable performance in dealing with varying noisy levels, confirming its generalization ability in a real-world scenario.We also draw the spectral curves of three typical pixels recovered by some leading methods in Fig.4.It is quite clear that the spectral curves recovered by our proposed method are smoother after denoising, and their overall shapes are more consistent with the original data.

5) Ablation studies: To verify the effectiveness of three blocks in the D2Net, we conduct corresponding ablation studies and list the results in Table 2.Compared the first row with the second, replaced down-sampling/up-sampling convolution (D/U) with DWT/IDWT,denoising performance of the backbone model has improved without extra model parameters burden, except for some decreases in SAM due to neglecting the spectral properties.Similarly, the effectiveness of PMCB and SCRB also have been verified.

Conclusion:This letter insightfully combines a multi-branch network with DWT/IDWT and proposes D2Net to achieve fine recovery of noisy HSIs.On the one hand, DWT/IDWT employed in this work, which is mathematically strict symmetric, can support multiscale feature decomposition and recomposition without information loss.On the other hand, HSI noise distribution varies in different frequency components, which inspired us to design tailor-made subbranches and develop PMCBs to achieve accurate detachment of noise and texture details for high-frequency sub-branches.In addition, we deploy an SCRB in refined reconstruction to further explore its coherence in original resolution and enhance the spatial and spec-tral fidelity of the recovered HSIs.Experiments demonstrate the superiority of our D2Net.Such an idea also can be flexibly transferred or promoted to other vision tasks for future insightful research,like HSI reconstruction, super-resolution, and abnormal detection.

Table 2.Ablations on Remove Synthetic Gaussian Noise With σ =50 on ICVl Dataset.Our D2Net is indicated by boldface.

Acknowledgments:This work was supported by the National Natural Science Foundation of China (61903279).