Video-based construction vehicles detection and its application in intelligent monitoring system

Wenyng Ji,Lingjun Tng,Dedi Li,Wenming Yng,*,Qingmin Lio

aShenzhen Key Lab.of Information Sci&Tech/Shenzhen Engineering Lab.of IS&DCP Department of Electronic Engineering/Graduate School at Shenzhen, Tsinghua University,China

bShenzhen Municipal Planning&Land Realestate Information Center,China

Available online 13 October 2016

Video-based construction vehicles detection and its application in intelligent monitoring system

Wenyang Jia,Lingjun Tangb,Dedi Lia,Wenming Yanga,*,Qingmin Liaoa

aShenzhen Key Lab.of Information Sci&Tech/Shenzhen Engineering Lab.of IS&DCP Department of Electronic Engineering/Graduate School at Shenzhen, Tsinghua University,China

bShenzhen Municipal Planning&Land Realestate Information Center,China

Available online 13 October 2016

While vehicle detection on highways has been reported before,to the best of our knowledge,intelligent monitoring system that aims at detecting hydraulic excavators and dump trucks on state-owned land has not been explored thoroughly yet.In this paper,we present an automatic,video-based algorithm for detecting hydraulic excavators and dump trucks.Derived from lessons learned from video processing,we proposed methods for foreground detection based on an improved frame difference algorithm,and then detected hydraulic excavators and dump trucks in the respective region of interest.From our analysis,we proposed methods based on inverse valley feature of mechanical arm and spatial-temporal reasoning for hydraulic excavator detection.In addition,we explored dump truck detection strategies that combine structured component projection with the spatial relationship.Experiments on real-monitoring sites demonstrated the promising performance of our system.

State-owned land protection;Construction vehicle detection;Hydraulic excavator detection;Dump truck detection;Inverse-V feature;Intelligent monitoring system

1.Introduction

1.1.Background

The increasing population and rapid urbanization in China have led to the frequent occurrence of encroachments in state-owned land,such as illegal construction.Although concerned departments have already taken some regulatory methods including remote sensing[1],human inspection,and vehicle video monitoring,some intractable problems still happen.Poor real-time performance and insufficient details exist in remote sensing.Human inspection has the bestfl exibility,but it has the problems of low efficiency and high cost of labor resource.Vehicle video monitoring saves labor resources,but it is unavailable in the bumpy areas.To overcome these problems,an intelligent monitoring system is presented.

Protection of state-owned land is an important duty for every local land and resources bureau as a result of the rapid urbanization in China.Even with such strict supervision,stateowned land is encroached upon regularly by developers and individuals.In 2010,372 typical illegal occupation cases occurred in Shenzhen,the first special economic zone in China.The majority of these cases are illegal construction on state-owned land.To forewarn illegal construction in a timely manner,an intelligent monitoring system aimed at detecting construction vehicles on sites is introduced.A variety of engineering vehicles are used in heavy civil construction.Many of these vehicles are manufactured to carry out specific operations,while others,such as hydraulic excavators and dumptrucks,can handle multiple activities and can be used in different project stages.Thus,detecting engineering vehicles, especially hydraulic excavators and dump trucks on stateowned land can increase the state's confidence in preventing illegal construction activities.

1.2.General introduction

Using an intelligent monitoring system to forewarn of illegal construction can greatly save on cost.Once the monitoring system is built,all processing procedures are fully automatic.The system can not only provide complete information of every video site,but also detailed information of every pre placed position.Furthermore,the system we propose has good real-time performance and high efficiency;it can process videos in real time and handle multiple videos simultaneously.Our system is also fully automatic,and thus no specialized supervision is required.

According to our research,studies or algorithms related to the detection of hydraulic excavators and dump trucks on state-owned land do not exist.Thus,resources that can be learned and used for reference is limited.Through our research and on-the-spot investigation,we gathered that a forewarning before illegal construction for more than two layers is necessary.On the basis of this analysis,forewarning information should be given before construction,that is, during the stage when foundations are being laid or construction activities have just started.This analysis coincides with our detection target because hydraulic excavators can be used in the foundation-laying stage,which in this case includes excavation,loading,trimming,and moving materials, as well as in the construction stage.Furthermore,dump trucks can be used in the foundation-laying stage for muck transportation and in the construction stage for building material transportation.

In this paper,an improved three-frame difference algorithm that consider processing efficiency is put forward for foreground detection.Detecting construction vehicles in the region of interest(ROI)can not only improve the accuracy rate,but also greatly reduce the calculation amount.We therefore propose the inverse-V feature of mechanical arm and spatialtemporal reasoning to recognize hydraulic excavators in construction videos.In dump truck detection,we explore dump truck recognition strategy combined with structured component projection with spatial relationship (SCPSR).The detailed processing flow is shown in Fig.1.

1.3.Related work

To effectively protect state-owned land from encroaching, relevant functional departments adopt various methods for supervision.All the different kinds of method can be roughly divided into satellite remote sensing,vehicle video monitoring,artificial patrolling,and video monitoring.Satellite remote-sensing method uses satellite remote sensing images to inspect the illegal use of state-owned land.Although this method is inexpensive,it has the disadvantages of lackingdetails,long-circuit-cycle and no real-time scenes.Vehicle video monitoring is proposed to overcome the disadvantage of lacking details and poor real-time performance.However,this method is not practical for monitoring vehicles in bumpy areas.Artificial patrolling and video monitoring have the bestflexibilities,but they have problems of low efficiency and high cost of labor resource.Therefore,the best choice is to use an intelligent monitoring system based on videos.

1.4.Site situation

The applied location of our system is in Shenzhen,the first special economic zone of China.Eighty video sites distributed in each Shenzhen district.The detailed distribution is shown in Table 1.We divide each video point into multiple presets according to the site conditions.Dividing multiple presets can acquire detailed informations and avoid blind areas.The monitored scene is extremely complex,involving grass, stones,water,and other natural scenes.To reduce false alarm and increase detection efficiency,foreground detection is performed first,that is recognizing the ROI only.Our system is built on a multi thread,so it can thus handle 80 videos at the same time.

1.5.Outline

In the following section,we first elaborate a foregrounddetection algorithm based on the frame difference.We describe in detail the improved three frame difference algorithm for the ROI of dump truck detection and the normal frame difference algorithm for the ROI of hydraulic excavator detection.In Section 3,we describe hydraulic excavator detection based on inverse-V feature and spatia-temporal reasoning.We use structural information in Section 4 to develop a method for dump truck detection based on SCPSR. In Section 5,we present the experimental results in practical engineering projects.A conclusion finishes this contribution.

2.Method

For a given video sequence,the foreground detection algorithm,which is the improved three-frame differential method,is performed first to judge whether a candidate dump truck is present.Then,we find the"V feature"in the candidate wheel area and determine the ROI of the cab and hopper according to the structural relationship of the dump truck.For cab and hopper detection,we use a method that makes a decision based on projection.The detailed handling process is shown in Fig.1.

2.1.Foreground detection

Fig.1.Overall processing flow for video.

Owning to theneed forintelligentmonitoring developments,foreground detection has caught the attention of numerous scholars and engineering researchers,which has resulted in many new methods and ideas.All those methods can be roughly divided into frame difference,background subtraction,optical flow,sports competition,motion template, and time entropy,among others.These varieties of methods have different advantages and applications.According to the application of our project,the frame-difference method is the best choice because it has good real-time performance and does not accumulate background.Foreground detection is different between hydraulic excavators and dump trucks because their moving speeds are different.

2.2.Foreground detection for hydraulic excavators

Hydraulic excavator is a type of hydraulic equipment,that is mainly used for excavation.It is widely used in loading, trimming,and moving materials,and is,therefore,widely used in construction activities.Through observation,we found that the movement of a hydraulic excavator is relatively slow compared with other vehicles.Thus,using continuous two frames to determine the difference is inadequate.Through analysis and experiments,the distinguished performance of using the two frames at a distance of 4 is achieved.Detection process for hydraulic excavators is specified in Algorithm 1.

Fig.2.The processing procedure and result of foreground detection for hydralic excavator.

The first step is frame difference,that is,the absolute value difference of Frameiand Framei+5,which are illustrated in Fig.2a and Fig.2b.The next step is to select an appropriate method for binarization.Inspired by image segmentation,we can obtain the threshold value automatically based on the segmentation idea.The origin of the data is introduced.After the frames difference operation,a gray image,which can be viewed as an"a difference"spectrum,is obtained.As illustrated in Fig.2c,the difference spectrum is composed of numerous values which can be expressed as numbers from 0 to 255."A large difference"in spectrum value means a large difference between the two images.

A large number in the"difference"spectrum can be regarded as a large difference between two images. Conversely,a smaller value of"difference"spectrum means that more similarities of the two images can be seen.Thisfinding indicates that differences between the two images are small[3],as shown in Fig.2d.Our purpose is to extract regions that change dramatically and suppress imperceptible changes.Through the analysis above,we can obtain the difference distribution map,which can be used to acquire the segmentation threshold.

The threshold for binary operation is easy to use according to the Eq.(1)after the segmentation threshold is obtained. Plenty of noises and pseudo foreground points exist,which morphological operations should eliminate. After the morphological operations,a contour filter is performed according to the size of the hydraulic excavator that appears in the video.The processing result is shown in Fig.2f.The process makes it easy to determine whether qualified foreground exists after the above operation.

2.3.Foreground detection for dump trucks

The process flow and idea of foreground detection for dump truck is similar to the flow for hydraulic excavator.Their biggest difference of them is that the speed of dump truck is faster than that of the hydraulic excavator.Thus,using the same method to judge whether a candidate foreground exists is unreasonable.Through our analysis,we put forward the improved three-frame difference algorithm for judgment.The detection process for dump trucks is specified in Algorithm 2.

The difference between normal and improved three frame difference algorithm is that the former is conducted and operated by using Difi-1and Difi+1.The advantage of this operation is to eliminate a ghost image.However,our purpose is to enhance changing areas.In that sense,we add two absolute values of Difi-1and Difi+1to enhance the change areas. However,it depressed the less changing areas.To obtain the enhanced changing image,subsequent processing processes, which are discussed in Section 2.2,are performed.The frame difference algorithms have a slight difference in contour filter. As the shape of dump trucks is rectangle,contour filter should satisfy this restriction.

3.Hydraulic excavators detection

Hydraulic excavators are highly deformable machines composed of mechanical arms and their hinged supports. Typical deformations of the machine are illustrated in Fig.3. Hydraulic excavators can slew 360°and rotate their mechanical arms,which consist of boom,dipper and attachment around their hinged supports.Considering the countless forms, it is impossible to detect hydraulic excavators with a limited number of training samples as used in the case of rigid body equipment.Studies show that the mechanical arm of the working hydraulic excavator presents an inverse-V feature. Thus,capturing this feature for further analysis,and spatialtemporal reasoning based on this judgment are wise choices.

3.1.Recognition of mechanical arm based on inverse-V feature

Hydraulic excavators have countless forms in operation process,except that the boom is aligned with camera view.In this case,it is impossible to distinguish the mechanical arm. Our objective is to detect the hydraulic excavator by its mechanical arm.As illustrated in Fig.3,the positions of boom and dipper can be divided into horizontal,vertical,leftinclined,and right-inclined.Lines of boom or dipper can be regarded as horizontal,which can be defined as-10°to 10°and 170°to 190°respectively.The bins mentioned above can be defined according to practical application.The spatialtemporal reasoning can be conducted and combined with the inverse-V feature for further decision(Fig.4).

1)Edge detection:Edge detection is performed.To detect

inverse-V feature of the mechanical arm of hydraulic excavator,diagonal Sobel mask for edge detection is employed, which considers the intensity response of the diagonal edge. 2)Binary operation using the idea of segmentation:After the gray-scale image which indicates edge response is obtained,the next step is to perform threshold operation naturally.Global and partial thresholds are two main methods used for threshold operation.However,a robust method that can not only suppress local noise but also keep strong edges is needed to obtain the binary image. Inspired by image segmentation,gray-scale image can be divided into two classes,namely,strong edges and background and noise.Therefore,the main problem is to segment the two classes.We can use the method mentioned in Section 2 to segment the image.

3)Contour filter:Through the above process,a binary image that represents edges is obtained.However,some annoying noises and pseudo edges may still exist.Morphological operations are typically used for further processing,but such operations are unlikely to fulfill this task because of background complexity.Considering that the mechanical arm contour is larger than the rest and that other scattered contours are relatively small,we use contour filter to purify the binary image.In our project,we discard the contours whose binding rectangles are greater than 256*256 or less than 64*64,because it can not only remove small, annoying outlines but also wipe out large contours,which are caused by missing data.

4)Line detection:As stated in Chapter 2,the bins,in which the lines of boom and dipper belong to should be identifi ed.It is convenient to confirm line angles that represent the boom and dipper.Hough transformation is one of the classical methods to determine angles for specific shape detection.Any valid points in the image space can bemapped to transform domain as a line or sinusoid as illustrated in Fig.5.Thus,a line can be determined by if nding the extreme local value in the transform domain. Detailed algorithm is specified in Algorithm 2.

Fig.3.The typical deformation of the hydraulic excavator and division of bins.

Fig.4.Edges detection procedure.

5)Bins and positions filtering:This section states how to extract inverted-V characteristics in detail.The characteristics can be realized by two filters.The first one is called bin filter which is used to divide lines into horizontal,left-inclined,vertical or right-inclined bins;and the second filter is position filter,which is used to restrict the space position relationships of lines.

It is convenient to map the lines into corresponding bins in the Hough space.As stated in Section 2,the abscissa of the Hough map is an angle to know whether lines in a corresponding angle scope exist.In detail,a line is horizontal if its angle lies at-10°to+10°or+170°to+190°.After the above operation,our algorithm projects the lines into correspondingbins.Location filter means that lines in corresponding bins should satisfy the corresponding position relationship.In detail,the pair of lines that satisfy the inverse-V feature[2] should not be far away from each other,and their vertices should be close.

Fig.5.Lines detection in Hough space.

3.2.Spatial-temporal reasoning

Existing algorithms are far from ideal,let alone achieve the ideal target of matching human vision.False alarms are unavoidable because monitoring scenes are done manually.In particular,grasslands and woods are swaying in the wind, which may result in an unpredictable false alarm.To avoid false alarms,spatial-temporal reasoning is introduced(Fig.6).

Spatial-temporal reasoning is an artful concept in computer visionsciencethatfusesaprioriandbackgroundknowledgeinto detection.Ittakesfulladvantageofimagesequencestoaboutthe targetlocationinspaceandtimetoreasoning.Therefore,logical reasoning is employed for further judgment to improve the detection rate in videos.In this framework,we assert that a working hydraulic excavator is present in videos via different combinations of inverse-V feature of the mechanical arm.

3.3.Judgment

The detecting process can be described as the followingflow chart.For a given video,foreground detection is performed first.If a foreground is detected,the next step is to detect the inverse-V feature of the mechanical arm.A specific combination of inverse-V feature is recorded when the mechanical arm is detected.

4.Dump trucks detection

In investigations and surveys,numerous scholars are devoted to vehicle detection,especially,vehicles on highway that present the front or rear-view.Literature which about dump trucks that present the side-view is limited.Given the practical application of our project,cameras are hung on buildings or in the corner.It is unlikely that the dump truck will drive toward the camera,so it generally shows its sideview.Owing to the complexity of natural scenes and the calculation efficiency of state-of-the-art methods considered,a new method that can detect dump truck rapidly is needed. Through our observation,side-view structure characteristics ofthe dump truck,which can be divided into wheel,cab and hopper zones,are obvious.As illustrated in Fig.7,different parts of a typical dump truck satisfy certain proportion relationships.Therefore,once one of the parts is detected,it will be more convenient to locate the rest of the parts.

Fig.6.The combination of different Bins.

4.1.Wheels detection

Wheels zone should be detected first because it has an obvious valley feature,which is the distribution of gray-value present in obvious peaks and valleys.To increase wear resistance and non-skid of tires,manufacturers add black carbon into the rubber,so tires will look blacker than the background.

Through the above analysis,if a scanning line is used tofind gray value statistics of the wheel zone,the statistics curve will present the V-feature.The specific handling method is using a scanning line to scan the ROI region frome bottom. Once the V-feature of the wheel zone is located,the position and size of wheels are also determined.Structural relationships of wheels,cab,and hopper of dump trucks are shown in Fig.8.It is easy to acquire the ROI of cab and hopper according to the structural relationships of dump trucks(Fig.9).

4.2.Cab detection

It is easy to determine the ROI of the cab according to the wheel position and size and structural relationship of dump trucks.An efficient method is needed to extract cab surface contour,which is a continuous entirety,except for the cab window presented as black in the image because component projection is used for judgment.A specific approach is scattering f i ve seed points in ROI according to the proportion and location relationships.Then,the consistency of seed point scattering is judged.If points are inconsistent,special points are removed, and points are re-scattered until consistency is satisfied.Data points are clustered according to the scattered seed points to generate cab surface contour[4].Some image denoising and morphology operations are used to purify the image.The final stepistoprojectthesurfacecontourtothehorizontalandvertical directions,respectively.Then,whether the surface contour satisfies the constraint condition according to projection curve is judged.The detection process is specified in Algorithm 3.

Fig.7.The structure of dump truck and it's proportional relation.

4.3.Hopper detection

The hopper detection method is similar to cab detection,but theorientationofthehopperneedsinformationonboththefront and rear wheels.Once the ROI of the hopper is found,the detection algorithm is the same as Algorithm 3.Post processing for hopper contour is different from the cab.As the hopper is strictly rectangular,we use a rectangular filter to handle the binaryimage.Specifically,we usea rectangularsliding windowto purify the binary image.If the space occupation ratio is larger than the threshold,the rectangular sliding window is filled with 255,the maximum gray-scale value.Otherwise,the rectangular slidingwindowisnotfilled.Todescribethehopperrectangle,we locate the rectangle and project it to horizontal and vertical directions respectively.Specially,we locate the ROI of the hopper on the basis of the wheel projection curve,which is the position and size of the front and rear wheels.The methods of clustering and projection are similar with that of cab detection.The detection results are shown in Fig.10.

5.Results

5.1.Detection rate

To test the effectiveness of our system,we use the videos captured by the Land and Resources Committee of Shenzhen, China.The continuous four-day recorded videos are used for testing.The processing results as statistically recorded in Table 2.

Fig.8.Detect"Valley feature"in the ROI of dump truck.

Fig.9.Determine the ROI of cab and make a decision.

Fig.10.Determine the ROI of hopper and make a decision.

According to the table,it is natural to obtain rates of detection,false alarm,average detection and average false alarm.The detailed statistical results are shown in Fig.11.

5.2.Time consumption

Achieving only the preferred detection result is not enough. As the goal of our system is to conduct actual application,a good real-time performance is thus necessary for the system. To test the real-time efficiency of our system,we randomly select 10 sections of the video for testing.The testing is performed on a PC with double Intel Core 2.5 G processors and 4 G RAM.The detailed processing results are shown in Fig.11.

6.Conclusion

A system for solving practical problems with Chinese characteristics is introduced in this paper.The whole system is composed of two parts:foreground detection and construction vehicle detection.In construction vehicle detection,two of the most commonly used engineering vehicles,which are the hydraulic excavators and dump trucks,are studied.Inverse-V feature modelofmechanicalarm and spatial-temporal reasoning are introduced to detect hydraulic excavator.In addition,SCPSR is introduced for dump truck detection in this journal.This system not only shows promising results in recognizing hydraulic excavator and dump truck in onlinevideos from stationary cameras,but also creates an intelligent monitoring application pioneer on the state-owned land. Future works should focus on exploring more effective detection algorithms and features.

Table 2Results of the method.

Fig.11.The statistic of detection.

Acknowledgement

This work was supported in part by the Natural Science Foundation of China under Grant No.61471216 and in part by the Special Foundation for the Development of Strategic Emerging Industries ofShenzhen underGrantNo.JCYJ20150831192224146 and No.JCYJ20150601165744635.

[1]C.El Amrani,G.L.Rochon,T.El-Ghazawi,G.Altay,T.-e.Rachidi, Development of a real-time urban remote sensing initiative in the mediterranean region for early warning and mitigation of disasters,in:GeoscienceandRemoteSensingSymposium (IGARSS),2012IEEE International,IEEE,2012,pp.2782-2785.

[2]W.Yang,D.Li,D.Sun,Q.Liao,Hydraulic excavators recognition based on inverse"v"feature of mechanical arm,in:Pattern Recognition,6th Chinese Conference on Pattern Recognition,CCPR,2014, pp.536-544.

[3]N.Dalal,B.Triggs,Histograms of oriented gradients for human detection,in:Computer Vision and Pattern Recognition,2005.CVPR 2005.IEEE Computer Society Conference on,vol.1,IEEE,2005,pp. 886-893.

[4]B.Catanzaro,B.-Y.Su,N.Sundaram,Y.Lee,M.Murphy,K.Keutzer, Efficient,high-quality image contour detection,in:Computer Vision,2009 IEEE 12th International Conference on,IEEE,2009,pp.2381-2388.

Wenyang Jireceived the B.E.degree in electronics engineering in 2014,Tsinghua university,and is now working toward the Master degree in electronics engineering in visualinformation processing lab, Tsinghua university.His research interests include image processing and pattern recognition.He has published articles in IEICE transactions on information and systems,Chinese conference on biometrics recognition.

Lingjun Tang,senior engineer,received his master degree of urban planning and design in Wuhan Technical University of Surveying and Mapping in 1999, and serve as the minister of development research department in the Information Center of Planning& Land&Real-Estate of Shenzhen,China.His research interests include the construction of digital city,and the informatization of the management of planning& land.He has won many related awards such as Surveying and Mapping Science and Technology Progress Award,Geographic Information Science and Technology Progress Award and Excellent engineering of Geographic Information Industry.

Wenming Yangreceived the B.S.and M.S.in material science from Harbin University of Science and Technology,Harbin,China in 2000 and 2003,respectively. In 2006,he received the Ph.D in electronic engineering from Zhejiang University,Hangzhou,China.From 2007 to 2009,He was a Postdoctoral Fellow in DepartmentofElectronicEngineering,Tsinghua University.Since 2013,He has been associate professor in Department of Electronic Engineering at Graduate School at Shenzhen,Tsinghua University. His research interests include image processing and pattern recognition,computer vision,biometrics,video surveillance,image super-resolution and non-destructive testing based on artificial vision.

QingminLiaobornin1963.HereceivedtheB.S.degree in radio technology from the University of Electronic Science and Technology of China,Chengdu,China,in 1984,and the M.S.and Ph.D.degrees in signal processing and telecommunications from the University of Rennes 1,Rennes,France,in 1990 and 1994,respectively.Since 1995,he has been joining with Tsinghua university,Bei-jing,China.He became Professor in the Department of Electronic Engineering of Tsinghua University,in 2002.From2001 to2003,he servedasthe InvitedProfessorwithatri-yearcontractattheUniversity of Caen,France.Since 2010,he has been Director of the Division of Information Science and Technology in the Graduate School at Shenzhen,Tsinghua University,Shenzhen,China.His research interests include image/video processing, transmission and analysis;biometrics;and their applications to teledetection, medicine,industry,and sports.He has published over 90 papers internationally.

*Corresponding author.Shenzhen Key Lab.of Information Sci&Tech, DepartmentofElectronic Engineering/Graduate SchoolatShenzhen, Tsinghua University,China.

E-mail address:yangelwm@163.com(W.Yang).

Peer review under responsibility of Chongqing University of Technology.

http://dx.doi.org/10.1016/j.trit.2016.09.001

2468-2322/Copyright?2016,Chongqing University of Technology.Productionand hostingby Elsevier B.V.Thisis an openaccess article under the CCBY-NC-ND license(http://creativecommons.org/licenses/by-nc-nd/4.0/).

Copyright?2016,Chongqing University of Technology.Production and hosting by Elsevier B.V.This is an open access article under the CC BY-NC-ND license(http://creativecommons.org/licenses/by-nc-nd/4.0/).