A Study on Hovering Control of Small Aerial Robot by Sensing Existing Floor Features

Chinthaka Premachandra,,Dang Ngoc Hoang Thanh,Tomotaka Kimura,,and Hiroharu Kawanaka,

Abstract—Since precise self-position estimation is required for autonomous flight of aerial robots, there has been some studies on self-position estimation of indoor aerial robots. In this study, we tackle the self-position estimation problem by mounting a small downward-facing camera on the chassis of an aerial robot. We obtain robot position by sensing the features on the indoor floor.In this work, we used the vertex points(tile corners) where four tiles on a typical tiled floor connected,as an existing feature of the floor. Furthermore,a small lightweight microcontroller is mounted on the robot to perform image processing for the on board camera. A lightweight image processing algorithm is developed.So, the real-time image processing could be performed by the microcontroller alone which leads to conduct on-board real time tile corner detection. Furthermore,same microcontroller performs control value calculation for flight commanding.The flight commands are implemented based on the detected tile corner information. The above mentioned all devices are mounted on an actual machine,and the effectiveness of the system was investigated.

I.Introduction

SMALL aerial robots are capable of motion such as vertical takeoff and landing in narrow spaces, horizontal movements like forward, backward,and hovering.Studies can be performed into autonomous flight considering applications such as room surveillance and transporting cargo by using these motions.Examples of research into autonomous flight include hovering[1],[2], movement to a target location[3],obstacle avoidance[4],[5],and object detection[6]–[8].Furthermore,when performing autonomous flight indoors,since the flight area is limited,only a small robot can be used.Therefore,it is necessary to use the hardware that is mounted on the aerial robot small and lightweight.

To implement the autonomous flight of an aerial robot,it is essential for the robot to perform self-position estimation.The GPS cannot be used in indoor environments.Indoor selfposition estimation of general robots has been studied using distance sensors such as ultrasonic sensors and infrared sensors[9]–[13],laser range finders[14],and cameras[15]–[21].Although ultrasonic sensors and infrared sensors need only light processing and many lightweight devices are available,but they suffer from low precision.Laser range finders are usually large,and are not suitable for mounting on a small aerial robot.In contrast,cameras can visually capture the surrounding environment,and can implement precision measurement through image processing.

In this study,we study autonomous flight control of an aerial robot by estimating its self-position using an on-board camera.There are some studies in the literature, regarding self-position estimation and autonomous flight control of aerial robots by using on-board cameras.Those studies include the method of obtaining position information by capturing images of markers embedded in the ground surface,using downward-facing camera mounted on the robot[15]–[18].In those methods,the camera mounted on the robot captures images of the markers.Here,the size and shape of the markers are already known.The robot orientation and position information are obtained from the geometrical relationship between the markers.However, those methods require advance installation of markers inside the flight area.In this case,robot can conduct operation only within the marker existing area.

Some studies in the literature also perform self-position estimation and robot control by capturing natural features from images[19],[20].Those methods capture natural features those exist on the floor and calculate the optical flow to obtain the robot position information.Since markers are not used,there are no limitations on the flight region,and measurement can also be performed in unknown locations.The common methods of calculating the optical flow include the block matching method and gradient method,and these allow high-precision measurements as long as the distances are short.However,the cumulative error becomes larger as the measurement distance becomes longer.In addition,it is difficult to implement them using light weight hardware like microcontrollers due to complexity of the algorithms.

Therefore,in order to address these problems,in this study we investigate recognizing tile corners using a camera mounted on an aerial robot to obtain position information and perform autonomous flight.Tiled floors are widely used in indoors.This removes the installing markers in advance because the tile corners can be used as an existing feature.Furthermore, by memorizing the initial position of the tile corner,the error can be reset if the robot returns to the initial position during measurement.Furthermore,since the tile corners are arranged with equal spacing,errors of moving distance can be corrected by using the constant distance between tile corners.

In this study,the robot position information is obtained based on these tile corners,and flight commands are sent to the robot based on the obtained position information.The final goal of this study is to perform autonomous flight indoors.Since the flight area is limited due to the size of the room and so when performing autonomous flight indoors,it is necessary to use a small robot.Due to the limitations of robot body surface and the weight that robot can carry,a small lightweight camera is mounted on its body surface.Furthermore,small lightweight microcontroller(Raspberry pi3)is mounted on the robot. A lightweight image processing algorithm is developed to perform real-time image processing by this on-board microcontroller alone.In addition,same microcontroller is used to perform flight control values decisions,following detected self-position information by image processing.The entire system is mounted on an actual machine,and the effectiveness of the system was investigated.

The results showed that it is possible to obtain robot position information by sensing tile corners.Furthermore,we confirmed the possibility of real-time autonomous flight control based on the detected robot positions.

II.System of Sensing Aerial Robot Position by Using Existing Floor Features

Fig.1 shows the system configuration of this study.A downward-facing monocular camera is mounted on the robot.In the position information calculation unit,floor images are captured by the camera,and they are sent to the microcontroller mounted on the robot.The microcontroller performs image processing to obtain the robot position information.The control signals are generated by the same microcontroller based on the obtained position information.Ultimately,the robot is controlled by those control signals which is sending to on-board flight controller.All the parts of the system will be detailed in next sub-sections.

A. Position Information Calculation Unit

The robot position information is obtained by recognizing the tile corners in images of the floor,captured by the camera.When the position capturing is started,the position of the tile corner first recognized is used as the initial position.Furthermore,the position information is sent to the control value calculation unit.

Fig.1.System configuration.

B.Control Value Calculation Unit

The control values are determined based on the position information from the position information calculation unit.Then determined control values are sent to the flight controller of the aerial robot.In this paper,we target flight control to achieve hovering of the robot.The detailed information on control value calculation is presented in section five.

III.Estimation of Aerial Robot Position Information

A. Position Estimation Stream

Fig.2 shows the procedure of the robot position estimation.Fig.3 shows the floor to be captured,whereas Fig.4 shows the points on the floor to be recognized.In this study,tile corners are extracted by analyzing the directional component of the floor pattern those appear on the images.To achieve this tile corner detection,the general corner detection methods such Harris corner detection and its related methods[22],[23]cannot be applied since they detect all the corner points in the image and the extraction of this specific point(tile corner)from them is very complicated.In this paper,the tile corners are recognized by image processing and then the robot position information is obtained.The methods of each process is present in next sections.In addition, the horizontal translation distance of the robot is measured by comparing the coordinates of the tile corners between consecutive frames.

Fig.2.Position information estimation procedure.

Fig.3.Captured image.

Fig.4.Tile corner.

Since we select a floor that is more complicated than a regular tiled floor,the system is expected to be able to handle floors with various tile shapes.Furthermore,we develop an algorithm with a small amount of calculations in order to perform the real time image processing on a microcontroller.

IV.Tile Corner Recognition

Fig.5 show s the sequence of the image processing during tile corner recognition.In Fig.5,the 80×80 pixel original floor image(a)is converted into a brightness layer image(b),and pattern enhancement and binarization are performed (c).The floor pattern direction components are extracted from the binarized image(d),and image angle correction is performed if needed (e).The direction components are extracted after the rotation correction (f),and noise removal is then performed on the image to obtain a low noise image(g).Finally,processing to recognize the tile corners is performed (h).

Fig.5.Sequence of image processing.

A.Conversion to Brightness Layer Image

In order to perform the calculations more efficiently in the image processing, the color image obtained from the camera is converted into a brightness layer image.A brightness layer image resembles a regular grayscale image,and is converted into an image that represents only the luminosity information from the color RGB image.The brightness layer image colored with 32 levels of luminance as shown in Fig.6.The luminosity valuesYin the brightness layer image are determined following(1).Here,R,G,and B denote red,green,and blue component value of each pixel in the color image,respectively.The color image is converted into a brightness layer image by storing the found luminosity valueYin each of the corresponding pixels.The larger luminosity valueYis denoted by the brighter pixels.

Fig.6.Brightness layer image.

B. Pattern Enhancement and Binarization

In this study,tile corners are recognized based on the regularity of the floor pattern.Because of this,it is important to extract the direction component of the floor pattern.However,the floor pattern is not clear in the brightness layer image,so a pattern enhancement filter is used to make it clearer.In this study,the patterns in the horizontal and vertical directions are enhanced.We design a simple filter to remove unnecessary information and extract only the target information by aligning then×nfilter shown in Fig.7.This can be expressed in the form of an equation as defined by(2),by takingI(x,y)as the image,G(i,j)as then×n(n=3)filter(where 0≤i≤n–1,0 ≤j≤n–1),andH(x,y)as the value after filter processing.TheH(x,y)value for each pixel in the image is calculated.Finally,each pixel is replaced by correspondingH(x,y).We experimentally designed this filter to carry on this work.This filtering can be done with some filters in literature[24]–[28].But, they are comparatively time consuming with compared to above mentioned filter.

Fig.7.Pattern enhancement filter.

After the filter processing,an image with the floor pattern enhanced as shown in Fig.5(c).Here binarized image is obtained by setting a threshold.C. Extraction of Direction Components After Pattern Enhancement Filtering and Binarization

To recognize whether the angle of the binarized image Fig.5(c)needs to be corrected,direction component extraction (DCE)is performed.

First,a pattern matching process is performed on binarized image Fig.5(c)to achieve directional components.In this process,original pixel pattern is compared with the defined three patterns.Fig.8 illustrates the defined patterns for direction component extraction.They can extractX-direction,Y-direction,and no direction component in the image.This process is performed to entire image to achieve DCE image(Fig.5(d)or 5(f)).We describe about image shown in Fig.5(f)in next sub-section.In Figs.5(d)and 5(f),above three directional components are plotted using three different colors.Fig.9 shows an example of aY-directional component.Based on this processing,whether the image needs to be rotated is judged as detailed in next sub-section.

Fig.8.Pattern matching.

Fig.9.Example of direction component pattern.

D. Angle Correction by Image Rotation

We take the camera angle shown in Fig.10 as 0°angled image.In reality,the robot will rotate while aerial,and the camera image will be rotated from 0°in the plus or minus directions as shown in Fig.11.Since the recognizedXandYdirection components become smaller due to the rotation,it becomes harder to obtain the floor patterns.As a result, the accuracy of recognizing the tile corners decreases.In order to solve this problem, rotation of the image is calculated,and then angle correction is performed before the tile corner determination.

Fig.10.At angle0°.

Fig.11.Rotated image.

An image,after the pattern enhancement and binarization is considered(e.g.,Fig.5(c)).We determine the total number ofX-direction components in the image asXtotaland the total number ofY-direction components asYtotal.The total of them(Dsum)is calculated following(3).IfDsumexceeds a defined threshold value,we judge that image rotation correction is not needed.IfDsumis less than the threshold value,the following process is conducted to achieve angle correction by rotating the image.

As shown in Fig.12, the value ofDsumgradually decreases until the camera is rotated by ±45°.Correction is performed to have a 0°angled image following this variation, rotating the image by angler°.The images obtained by rotating in to plus and minus directions are different each other.Furthermore,the direction in which the correction needs to be performed,cannot be judged based on the initially obtainedDsumalone.Therefore,above mentioned pattern matching for directional component extraction is applied to the images,those obtained by rotating the binarized image in to both directions as shown in Fig.13.The value ofDsum+obtained by rotating the image in to plus direction.In the other words,Dsum+is sum of calculatedDsumvalues for each image during the plus direction rotation.The value ofDsum–is obtained by rotating the image in to minus direction in same manner.They are compared based on the rule in(4)to determine the rotation direction.After determining the direction,The image which shows maximumDsumvalue in that direction is selected as angle corrected image.Fig.5(e)is an example image for angle corrected image.In addition,its corresponding DCE image is also generated during this process.following the Fig.5,the image shown in Fig.5(f)is an example for this DCE image.

After this process,noise removal as described in Section III.Eis performed on DCE image.Furthermore,if it is judged that angle correction is not needed,noise removal is performed on the DCE image without conducting the angle correction.following the Fig.5,step(e)and(f)are not conducted in this case.

Fig.12.Relationship between camera angle and D sum.

Fig.13.Deciding the rotation direction.

E. Noise Removal

In order to detect the tile corners, noise removal is performed on DCE image since the image that has been processed up to this contains a lot of noise (Figs.5(d) or 5(f)). Fig.14 illustrates the noise removal process.The DCE image is divided into 5×5 pixel segments.The number ofXdirection components(Xi,j)and the number ofYdirection components(Yi,j)in each segment is compared.The direction component that accounts for the highest proportion of each segment is used as the direction componentDk,lfor the entire segment following(5).After this, the segment is filled with defined color for each direction as illustrated in Fig.14.Once it is confirmed that this processing is finished throughout the entire area,the noise removal finishes.The Fig.15 illustrates an image after this noise removal,and this allows the tile corners to be recognized with relatively high precision.

Fig.14. Noise removal.

F. Deciding the Tile Corner Coordinates

The coordinates of the tile corner are decided based on the noise-removed direction component image.Fig.15 shows an example of noise-removed direction component image.Here,the Brown denotesXdirection component while Green denotesYdirection component.Fig.16 shows the tile corner coordinate searching method.The designated row in the image is scanned from the left to search for the coordinates of the boundary between theXdirection component andYdirection component.If a boundary with three consecutiveYdirection components are found they are labeled as①,②,and③.The same process repeat within all designated rows and labeled as illustrated in Fig.16.The line which connects maximum boundaries is selected as border on vertical direction.The border on the horizontal direction is also determined in same manner.Finally,the cross point of these two lines is selected as tile corner.

Fig.15.Direction component image.

Fig.16.Searching for the coordinates of the tile corner.

G. Horizontal Translation Distance Measurement

The horizontal translation distance of aerial robot is measured by tracking the tile corners.We take the coordinate of the tile corner recognized in then-th frameFnas(xn,yn).The translation distanceLn,n+1betweenFnandFn+1is then given by (6).

Therefore,the translation distanceL1,nfrom the first frame to then-th frame is given by (7).

V.Hovering Control

A. Hovering Control

There are many studies can be found in the literature regarding the controlling different type of robots[29]–[31].In this paper,we decided to use pulse position modulation (PPM)[32]to transfer the shift between tile corner and aerial robot to achieve hovering control of aerial robot,since it is less time consuming.Furthermore,PPM can easily be implemented on a small microcontroller.The PPM method is a modulation method that converts a signal into time-based pulse position(phase difference)relative to a period.The information is determined by the position of the pulse.The PPM signal used in this study is illustrated in Fig.17.It is composed of 8 channels of pulses and dummy pulse.The dummy pulse is for adjusting the period of the overall signal.This time,hovering control is performed with PID control and PD control following the PPM value, to hover aerial robot over the detected tile corner.Here,PID control is used to achieve the altitude(zdirectional control)and PD control is used to achieve horizontal directional control(xandydirectional control)[33].

Fig.17.PPM signal.

We use the PID for thez-axis instead of a PD controller is that the battery voltage/performance decreases with flight time.A full battery needs upwards lift equal to the gravitational force of the aerial robot.A battery at 50%however needs a throttle command in the same motor speeds respectively same upwards force.This loss of “power”,upwards lift over time or acceleration in thez-axis can be compensated by an integrator in the controller.Here the I-part of the PID can constantly increase the throttle command to hold the same height/set point(hover).For thex-andy-axis,we do not need to compensate the loss of battery voltage,since it does not affect thex-yspace positions.The aerial robot would theoretically just stay at the same horizontal position,as long as the roll and pitch angles are 0°.However,we also chose PD-controllers, because the aerial robot only is operated inside a certain range inx-yspace near to a tile corner.

As mentioned in Section II-B, the control signal is generated by on-board microcontroller based on the position information.Generally,a detected tile corner does not move more than several pixels within a two consecutive frames,during hovering control.Therefore,in this paper,if the detected tile corner in current frame moves irregularly,more than a pre-defined threshold, we ignore the detected position.In this case,hovering control is performed following the detected tile corner in previous frame.

In this study,we keep both manual control and autonomous control of the aerial robot.Webring the aerial robot to defined altitude by manual control. After that,when the robot detects a tile corner,it starts the autonomous hovering control keeping the current altitude by PID control(zdirectional control).As mentioned above,xandydirectional robot shift from the detected tile corner is modified by PD control.To achieve an effective hovering control, the camera pose has to be horizontal.We have solved this matter applying a pose keeping hardware.

B. Deciding Flight Commands

In this study,the center of the 80×80 pixel image at(40,40)is taken as a reference coordinate(Fig.18).When the recognized tile corner is shifted from the reference coordinate,in other words,when the robot is moved from directly above the tile corner,control is performed to remain above the reference coordinate to achieve hovering.Since thex-axis direction in the image corresponds to the forward and backward motion of the robot,if thexcoordinate changes from the reference coordinate,the CH1 pulse width in Fig.17 is changed according toxcoordinate shift value and the aileron is controlled following PD control to return the robot to reference coordinate.Furthermore,since they-axis direction in the image corresponds to the left and right motion of the robot,if theycoordinate changes from the reference coordinate,the CH2 pulse width in Fig.17 is changed according to theycoordinate shift and the elevator is controlled following PD control to return the robot to reference coordinate[33].In this paper,the parameters of PD control is determined by sub-experiments.

Fig.18.Reference coordinate(The center of the80×80 pixel image).

VI.Experiments

A. Investigation of Tile Corner Recognition and Horizontal Translation Distance Measurement

To investigate the accuracy of the system,we performed accuracy verification experiments on the tile corner recognition and horizontal translation distance measurement by using the on-board downward-facing monocular camera and Raspberry pi3 microcontroller.During the flight,the rotation around thezaxis is assumed to be zero.The camera pixel configuration is set to 80×80 pixels.

B. Experimental Method of Tile Corner Recognition

The images of the floor were captured with on-board camera and the accuracy of the tile corner recognition and horizontal translation distance measurement were investigated.Here,the recognition rate was investigated with the approximate camera elevation at 1000 mm.At this camera elevation,there is only one tile corner appear in an image.Furthermore,the machine with camera was approximately moved 200mm horizontally,and the actual horizontal translation distance and measurement result values were compared.The experiment environment used in buildings of the Katsushika Campus,Tokyo University of Science.Fig.19 shows an example image of a floor.In this work,in order to support different kinds of tiled floors,a more complicated tile shape than typical tiled floors were chosen.

Fig.19.Example of floor image used in experiments.

C. Results of Tile Corner Detection

Fig.20 shows the experimental results of tile corner detection.When the tile corner is recognized,a blue point is displayed in the image.Through the experimental results,we confirm that recognition was possible even when the camera was rotated.Furthermore,the recognition success rate as a result of capturing 30 000 frames while flight was 97.6%.This result confirms that the tile corner could be recognized with high accuracy.Furthermore,Fig.21 shows the main stages of the image processing.According to this,it confirms that the subsequent processing was performed after judging whether angle correction was needed from the image.

Fig.20.Experimental results of tile corner detection.

Fig.21.Experimental results of tile corner detection with all processing stages.

Furthermore, the average error of horizontal translation measurement when the machine elevation was approximately at 1000 mm is 2.7%.This shows that high accuracy measurement of horizontal movement was possible.This tile detection could be conducted in real-time since the average computational time was18 ms per frame.

D. Hovering Control Procedure Following Tile Corner

To confirm whether the system built in this work can perform robot control,control verification experiments were performed on Raspberry pi3 microcontroller.

The(x,y)coordinate in the center of the image at (40,40)is treated as a reference coordinate.As shown in Fig.22,it is assumed that the robot will be controlled to return the camera coordinates to the position of the reference coordinate if the robot is moved,and the PPM signal is output to perform this kind of control.In this work, we performed an investigation assuming that the camera coordinates are returned to the position of the reference coordinate when the robot is moved forward,backward,left,and right.For example,a PPM signal is output that controls the elevators to return to(40,40)when(x,y)changes to(20,40)or(60,40)and that controls the ailerons to return to(40,40) when (x,y)changes to(40,20)or(40,60).Fig.23 shows the coordinate relationship between the robot and the camera image.

Fig.22.Investigation of control command output.

Fig.23.Coordinate relationship between the robot and camera image.

E. Results of PPM Generation

We analyze the PPM variation during the flight movement.Figs.24 and 25 show a comparison of the PPM signal when the camera is at the reference coordinate with the PPM signal when the camera is at(x,y)=(20,40)and(60,40),and Figs.26 and 27 show a comparison of the PPM signal when the camera is at the reference coordinate and the PPM signal when the camera is at (x,y)= (40,20)and (40,60).Fig.24.Comparison of signals between the reference coordinate and (x,y) =(20,40).

Fig.25.Comparison of signals between the reference coordinate and (x, y) =(60,40).

Looking at the brown waveform in Fig.24,we can see that when the camera moves from the reference coordinate to(20,40),a PPM signal output becomes shorter following the required minus shift from the original.Similarly,looking at the green waveform in Fig.25,we can see that when the camera moves from the reference coordinate to(60,40),a PPM signal output becomes wider,following the required plus shift.

Furthermore,looking at the green waveform in Fig.26, we can see that when the camera moves from the reference coordinate to(40,20),a PPM signal output becomes wider following the required plus shift from the original.Similarly,looking at the green waveform in Fig.27,we can see that when the camera moves from the reference coordinate to(40,60),a PPM signal output become smaller according to the required minus shift.

Fig.26.Comparison of signals between the reference coordinate and (x, y)= (40,20).

Fig.27.Comparison of signals between the reference coordinate and (x, y)= (40,60).

This experiment confirms that PPM signals are output based on the obtained position information.

F. Hovering With a Real Machine

Fig.28 illustrates the prototype aerial machine prepared for flight experiments.It mainly consists of general parts of a quadcopter.However,to carry on this work,we installed Pixhawk4 flight controller, microcontroller(Raspberry pi3),and small type camera which can be connected to the microcontroller.Tile corner detection is conducted by the microcontroller by processing image from the camera.Pixhawk4 flight controller is connected with microcontroller to communicate each other to achieve data transform.Through the different experiments,we could confirm that hovering control is possible by sensing tile corners on the ground.

Fig.28.Prepared aerial robot for experiments.

VII.Conclusion

In this research,we developed a lightweight image processing algorithm such that image processing can be performed using only a microcontroller with the aim of obtaining the position information of a small aerial robot based on tile corners as an existing floor feature.We then investigated autonomous flight of the robot by sensing the tile corner positions.As a result of the investigation,it was found that the position information for a robot can be obtained by sensing the tile corner using a downward-facing camera mounted on the robot.We also implemented flight commands based on the obtained position information,and confirmed the possibility of autonomous hovering control.