Dark channelprior based blurred image restoration method using totalvariation and morphology

,*

1.College of Information and Communication Engineering,Harbin Engineering University,Harbin 150001,China; 2.Departmentof Information and Communication Engineering,University of Electro-Communications,Tokyo 182-8585,Japan

Dark channelprior based blurred image restoration method using totalvariation and morphology

Yibing Li1,Qiang Fu1,Fang Ye1,*,and Hayaru Shouno2

1.College of Information and Communication Engineering,Harbin Engineering University,Harbin 150001,China; 2.Departmentof Information and Communication Engineering,University of Electro-Communications,Tokyo 182-8585,Japan

The blurred image restoration method can dramatically highlightthe image details and enhance the globalcontrast,which is ofbene fi tto improvementofthe visualeffectduring practicalapplications.This paper is based on the dark channelprior principle and aims atthe priorinformation absentblurred image degradation situation.A lot of improvements have been made to estimate the transmission map ofblurred images.Since the dark channelprior principle can effectively restore the blurred image at the cost of a large amount of computation,the total variation(TV)and image morphology transform(speci fi cally top-hat transform and bottomhat transform)have been introduced into the improved method. Compared with originaltransmission map estimation methods,the proposed method features both simplicity and accuracy.The estimated transmission map together with the element can restore the image.Simulation results show that this method could inhibit the ill-posed problem during image restoration,meanwhile it can greatly improve the image quality and de fi nition.

image restoration,dark channel prior,total variation (TV),morphology transform.

1.Introduction

Owing to the complicated environment med ium and the uncertain ambient noise interference,outdoor images frequently suffer from the visualization problems such as low resolution and blurred edges,which also bring inconvenience to further processing.Aiming atthe image restoration,various methods have been researched and proposed. In 2009,He fi rst proposed the dark channel prior based image defogging method[1-3].The method focuses on the blurred images interfered by complex noise together with the uneven atmosphere.Furthermore,the dark channel prior is a model based on a great quantity of clear images,and the transmission depth map can be acquired thus the restoration can be made.

While the dark channel prior is being widely acknowledged and accepted,many developments of blurred image restoration have been made these years[4-6].Integrated with the traditionalimage restoration and enhancement methods,it has been applied more widely[7-10]. In general,histogram equalization,Retinex and wavelet methods are commonly selected and combined with dark channel prior,while they are not perfect[11-14].Histogram equalization might cause the loss of grayscale and perform not well on low contrast blurred images. Retinex employs the weighted average of results from multi-scale processing and contains much convolution operation.Thresholding wavelet depends mainly on the subjective judgment,which makes the manipulation uncertain and inaccurate.Furthermore,these common estimation methods would bring the block effect through the acquisition of the transmission map,and the estimation among different color zones might also produce errors, which mightlead to excessive enhancementor detailloss.

Considering the de fi ciency of these methods and aiming at a better solution to the image restoration problem, a transmission map estimation method combining total variation(TV)and image morphology is proposed in this paper.Based on the estimated transmission map,image restoration can be ful fi lled.

The paper is organized as follows.Section 2 reviews the background and the principle of dark channel prior.Section 3 analyzes the TV based transmission map estimation. In Section 4 experimentalresults and comparison are provided and Section 5 is the discussion and conclusion.

2.Scattering and dark channelprior model

2.1 Atmospheric scattering based degradation model description

Generally,the vague degradation model[15-17]can be ex-pressed as

where I is the observed intensity,J represents the scene radiance,A is the global atmospheric light,and t stands for the transmission map which can be speci fi cally described as t(x)=e?βd(x)by transmission factorβand scene depth d.The fi rst term J(x)t(x)on the right-hand side of the equation is called the direct attenuation term,and the second term A(1-t(x))is called the air lightcomponent. The goalof haze removalis to recover J,A and t from I.

2.2 Dark channelprior restoration model

After He’s observation of numerous haze-free outdoor images,it has been fi gured out that in most of the nonsky patches atleastone channelhas a very low intensity at some pixels,which means the minimum intensity in such a patch is close to zero.Let Jcrepresenta single color channel of image J andΩ(x)is a local patch centering on x. The dark channelof image J can be de fi ned as

Assume the atmospheric light A is given and the transmission map~t in a local patch is fi xed.Taking the minimum operation in the local patch on the haze imaging equation,we can get

Then taking division on both sides by A,

Since the minimum operation is performed on three colorchannels independently,then the minimum operation among three color channels is used again to obtain

According to the dark channel prior,the dark channel tends to be zero,namely

Also since Acis positive,we have

The transmission map can be estimated by

Optionally,a very smallamountof haze can be reserved to preventthe unnaturaldepth feeling in mostcases.Therefore a constantparameterω(0＜ω＜1)can be introduced to the equation.

3.TV and morphology based transmission map estimation

According to the above-mentioned analysis of dark channel based image restoration,the acquirement of the transmission map t is the key procedure to the following restoration.Since the transmission among local patches is not constant,the traditional estimation method might cause blocking artifact,and the fi nal restoration result obtained will not be satisfying.To solve the blocking artifact,soft mapping has been introduced to settle the white balance problem of spatialvariables to some extent.The transmission map after softmatting reveals the approximate edges of the objects,and the discontinuity of sharp edges can be reserved.Nevertheless,the sparse matrix based soft mapping is computationally complicated and related research shows thatsoft mapping based re fi ning operation takes up the majority of the entire calculated amount.

Moreover,the massive convolution computation and large size Laplacian matrices occupy too much memory thus the method has many restrictions to the image.In order to compromise the effectiveness and the algorithm complexity,a novel estimation method of transmission map combining TV and image morphology is proposed in this paper.

3.1 TV restoration

In order to reserve the image edges,image enhancement should follow the principle that:the mutational edge area has low smoothness while the fl at site has high smoothness,the direction along the edge has the maximum evenness and the minimum perpendiculardirection.The largest smoothness willappear along the edge direction while the least is vertical to the edge.The basic idea of TV is to build the image model and minimize the energy function to achieve a balance state.The anisotropy partialdifferentialequations are introduced to coordinate the edge-reserve problem and the noise restrain.

AssumeΩ?Rn,where(n∈N+)is the closed domain of dimension,andΩstands for the de fi nitional domain of the image,as to two-dimensionalimage n=2.

In two-dimensionalgrayscale images,the gradientis thederivative of the variables x and y,which is de fi ned by?u as

In the TV restoration model,the Euler-Lagrange equa-

Uniformly-spaced sampling has been conducted to the image at the interval of 1.As shown in Fig.1,the objective pixelis u(i,j),the neighboring position coordinate set isΛ={(i,j+1),(i-1,j),(i,j-1),(i+1,j)},labeled as e,n,w,s respectively,and a parameterβsmall enough is also introduced to avoid the denominator|?u|β=

Fig.1 Object pixel u(i,j)and its neighborhood

Similarly,other three directions can be acquired.Thus we have

After simplication,the Euler-Lagrange equation can be rewritten as

Fig.2 depicts the TV restoration comparison of the noisy image and the restored output.

3.2 Image morphology transform

Image morphology belongs to the non-linear fi lter which could modify the geometrical characteristic of signals by corresponding transformation.Morphology fi ltering has been widely used for its bene fi cial property,namely reserving target information and removing interference simultaneously.In the case of haze image restoration,the features of objects are dif fi cult to extract for the sake of background interference.Other traditional fi lters,e.g.median fi lter and high-pass fi lter,mighterroneously judge the small targets as noise and powerless to remove the highlight spot.In our proposed method top-hat transform and bottom-hattransform are introduced.Top-hatcan work as the high-pass fi lter while bottom-hatcan detect the valley value of the image.Above all,top/bottom-hattransform is of especialbene fi twhen the lightis asymmetry.

The fundamental operation contains dilation,corrosion,and opening and closing.The prede fi ned structural element is employed to collect the image information. Through the movement of the structural element,the interrelation ofdifferentimage sections can be acquired,and the structuralcharacteristic can be comprehended.

f is assumed to be the original image in a continuous space,and b is the structuring element.And the dilation of f can be de fi ned as

The geometricalsigni fi cance of dilation is thatthe center of b keeps moving on the surface of f,the envelop will be made by b and Dbis the range of b.The empty holes and the sunk parts willbe fi lled.

The de fi nition of erosion is

which means the track made by the movementof the center of structuring element b when b keeps tightly to f.The erosion can be used to weaken oreven eliminate the bright area smaller than b.

Opening and closing are the other two operations made by the combination of dilation and erosion.Opening is defi ned as(f?b)=(fΘb)⊕b,which means erosion followed by dilation,using the same structuring element for both operations.It can be used to reduce the edge blur and remove the smallobjects.

On the contrary,the closing operation is dilation followed by erosion,which can be written as(f?b)= (f⊕b)Θb.It is useful to join the circles in the image together by fi lling in the gaps between them and by smoothing their outeredges.

The top-hattransform is the process to cutthe openingprocessed image from the originalimage while the bottomhattransform is to cutthe originalimage from the closingprocessed image.If the objecthas the same localcontrast, the top-hattransform can be used to adjustthe non-uniform light and the large-scale isotropy structuring element can be used as a high-pass fi lter.On the otherhand,bottom-hat transform is more suitable for the brightobjects in the dark background.Thus top-hatand bottom-hatcan be combined to stretch the image grayscale:the originalimage plus the top-hatprocessed image and then minus the bottomed-hat processed image,the fi nal result will be the enhanced image.

Fig.3 shows the comparison between top-hat and bottom-hattransform.Fig.3(a)is the original image,Fig. 3(b)and Fig.3(c)are the images after bottom-hat transform and top-hat transform respectively,and Fig.3(d)is the enhanced image.

Fig.3 Comparison of top-hat and bottom-hat transform

3.3 Acquisition of transmission map

Three images“Urban”,“Hilly”,“Rural”shown in Fig.4 have been chosen to be experimented while the architectures and mountain rocks detail should be particularly noticed.Transmission maps have been estimated respectively according to the fl owchartshown in Fig.5.Firstthe RGB image is transferred into grayscale images,then the bottom-hat transform is conducted.After that the TV enhancement will be made to the image.Finally the inverse color t of the whole image is taken,and the top-hat and bottom-hatenhancementis used;the eventualtransmission map tx can be acquired.

Fig.4 Original blurred outdoor images

Fig.5 Flowchart of the transmission map acquisition

Step 1Change the RGB image form into grayscale.

Step 2Make the bottom-hattransform.

Step 3Make the TV enhancement,and the algorithm can be realized by the following three-layer periodic nesting.

Initialize u(0)=u0, for n=1,2,...

for i=1,2,...,Mfor j=1,2,...,N

Calculateωp,hp,hoas wellas

End

End

End

Step 4Take the inverse complementof t.

Step 5Make the top-hatand bottom-hatenhancement to achieve the eventualtransmission map tx.

As to the comparison displayed from Figs.6-8,the left columns are the transmission maps obtained via the original dark channelmethod;the middle columns are through the TVbased estimation;and the rightcolumns are through the TV-morphology based estimation.

Fig.6 Comparison of transmission maps of urban

Fig.7 Comparison of transmission maps of hilly

Fig.8 Comparison of transmission maps ofrural

From the transmission map contrast,it can be indicated that the original maps have obvious blocking artifact;without morphology transform,the TV-based maps are still fuzzy comparing to the morphology transformadded maps.

The computation time of the original method and the proposed one has been measured for these three images. Each image estimation has been made fi ve times and the average has been calculated.These experiments have been executed on the Intel Core 2 E8400 Duo CPU,with Windows XP and Matlab R2010a.

Comparison results are shown in Table 1.Although the fl ow path ofthe proposed method may seem a bitcomplex, considering the better quality,the outcome still indicates the competitiveness of the TV-morphology based method.

Table 1 Computation time ofestimation methods

3.4 Integrated restoration method

The fl owchart of the improved restoration is shown in Fig.9 and the detailprocedures are as follows.

Fig.9 Flowchart of image restoration method

Step 1Decompose the originalhaze image into R,G, B channels respectively.

Step 2Estimate the corresponding atmosphere light component A and transmission map tx of each channel. The highestpixelintensity is used as the atmosphere light A,while tx can be acquired through the fl owchart shown in Fig.5.

Step 3Calculate the new components NR,NG,NB through the combination of original R,G,B,atmosphere lightcomponent A and transmission map tx.Step 4Synthesize these three new components into the restored image.

4.Experimentalresult and analysis

In this section our experiments willbe explained.Outdoor haze images have been tested to verify the effectiveness and the performance will be compared with He’s method and the soft-matting method.

Our experiments are conducted on three sets of color images as shown in Fig.4.The haze removal results are shown from Figs.10-12,which compare the restoration images of urban,hilly and rural,respectively.

Fig.10 Comparison of restoration image urban

Fig.11 Comparison of restoration image hilly

Fig.12 Comparison of restoration image rural

Itcan be fi gured outthatthe raw transmission map based restoration cannot recover the edge,also the estimate of color is less accurate,and arti fi cial color blocks often appear.The proposed method can preserve both regions of deep colorand lightcolor,and excessive-enhancementcan be avoided.As to the signal-to-noise ratio,the proposed method can improve about4 dB.

Among the three test images,“urban”and“rural”have more edges and details,therefore the edge detector can be used to inspectthe restoration results.The Canny operator has been used to detect the detail edges of objects.From the results shown in Fig.13,it can be seen that compared with He’s method,the proposed method can recover more speci fi c details and the complete edge can be extracted.

Fig.13 Edge detection comparison by Canny operator

5.Conclusions

In this paper,the dark channel prior based TV and the morphologicalhaze image restoration method has been explained for single image haze removal.Utilizing the new transmission map estimation method,it has become more effective to recoverthe image.

Since the limitation ofthe haze removalmodel,the more complicated phenomena could be further researched.Relatively speaking,ourwork focuseson a single kind ofblur, the mixed blurnoise willbe the nextfocus.Meanwhile,we will also investigate ways to simplify the processing procedures and speed up execution.

Acknowledgment

We are particularly grateful to the University of Electro Communications and the Short-term Exchange Program of JUSST(Japanese University Studies in Science and Technology)for providing the precious research assistance.We also express gratitude to the kind guidance from Professor Hayaru Shouno.

[1]K.He,J.Sun,X.Tang.Single image haze removalusing dark channel prior.IEEE Trans.on Pattern Analysis and Machine Intelligence,2011,33(12):2341-2353.

[2]Y.W.Wen,Ng.K.Michael,Y.M.Huang.Ef fi cienttotalvariation minimization methods for color image restoration.IEEE Trans.on Image Processing,2008,17(11):2081-2088.

[3]F.Chen,X.Huang,W.Chen.Texture-preserving image deblurring.IEEE Signal Processing Letters,2010,17(12): 1018-1021.

[4]A.G.Hanbury,J.Serra.Morphological operators on the unit circle.IEEE Trans.on Image Processing,2001,10(12):1842-1850.

[5]S.Parthasarathy,P.Sankaran.A Retinex based haze removal method.Proc.ofthe 7th IEEE InternationalConference on Industrial and Information Systems,2012:1-6.

[6]S.C.Pei,T.Y.Lee.Effective image haze removal using dark channelprior and post-processing.Proc.of the IEEE International Symposium on Circuits and Systems,2012:2777-2780.

[7]H.Xu,J.Guo,Q.Liu,et al.Fast image dehazing using improved dark channel prior.Proc.of the International Conference on Information Science and Technology,2012:663-667.

[8]B.Xie,F.Guo,Z.Cai.Improved single image dehazing using dark channelprior and multi-scale Retinex.Proc.ofthe International Conference on Intelligent System Design and Engineering Application,2010:848-851.

[9]H.Y.Yang,P.Y.Chen,C.C.Huang,et al.Low complexity underwater image enhancement based on dark channel prior. Proc.of the 2nd International Conference on Innovations in Bio-inspired Computing and Applications,2011:17-20.

[10]K.B.Gibson,T.Q.Nguyen.On the effectiveness of the dark channelpriorforsingle image dehazing by approximating with minimum volume ellipsoids.Proc.of the IEEE International Conference on Acoustics,Speech and Signal Processing,2011: 1253-1256.

[11]Y.Wang,Y.Li,T.Zhang.The method of image restoration in the environments of dust.Proc.of the International Conference on Mechatronics and Automation,2010:294-298.

[12]J.Guo,X.Wang,C.Hu.Single image dehazing based on scene depth and physical model.Journal of Image and Graphics, 2012,17(1):27-32.(in Chinese)

[13]B.Wu,Y.Wu,H.Zhang.Totalvariation based image restoration technology.Beijing:Peking University Press,2008.

[14]R.C.Gonzales,R.E.Woods.Digital image processing.Beijing:Publishing House of Electronics Industry,2011.

[15]H.Zhang,P.Zhou,M.Xue.Foggy weather image enhancement algorithm based on dark channel prior and histogram matching.Computer Engineering,2012,38(1):215-216, 219.(in Chinese)

[16]W.Jin,Z.Mi,X.Wu,etal.Single image de-haze based on a new dark channel estimation method.Proc.of the IEEE International Conference on Computer Science and Automation Engineering,2012:791-795.

[17]Q.Fu,Y.Li,Y.Liu.Improving image quality in poor visibility with Retinex-based adaptive image enhancementmethod. International Journal of Digital Content Technology and its Applications,2011,5(10):296-301.

Biographies

Yibing Li was born in 1967.He is a doctor and a professor in Harbin Engineering University.Also he is a member of IEEE.He has published more than 80 papers and 3 monograph teaching materials.In 2004 he received Huo Yingdong Award for young teachers and in 2006 he was selected to join the academic support plan of Heilongjiang province.His research interests are digital image processing and information fusion.

E-mail:liyibing0920@sina.cn

Qiang Fu was born in 1988.He was admitted to graduate school direct from B.S.to Ph.D.,by passing the M.S.degree without quali fi cation examination in Harbin Engineering University in 2010.He has been to the University of Electro Communications in Tokyo in the second year of Ph.D.as an exchange student.His research interests include digital image processing and probabilistic-based image inference.

E-mail:funq2012@hotmail.com

Fang Ye was born in 1980.She is a doctorand an associate professor in Harbin Engineering University. She has been to University of Southampton as a visiting scholar from 2007 to 2008.As a technical director,she has taken partin many scienti fi c projects and has published more than 30 papers.Herresearch interests are UWBwireless communication and cognitive radio.

E-mail:yefang0815@sina.cn

Hayaru Shouno is a Ph.D.graduating from Osaka University.He has been the researcher in Osaka University and Nara Women’s University and Yamaguchi University.Now he is an associate professor in the University of Electro Communications,also a member of IEEE and Information Processing Society of Japan and the Institute of Electronics,Information and Communication Engineers(IEICE). His research interests include digital image processing,data mining and probabilistic-based image inference.

E-mail:shuono@uec.ac.jp

10.1109/JSEE.2015.00042

Manuscriptreceived July 03,2013.

*Corresponding author.

This work was supported by the National Natural Science Foundation of China(61301095),the Chinese University Scienti fi c Fund (HEUCF130807),and the Chinese Defense Advanced Research Program of Science and Technology(10J3.1.6).