Deep Learning Control for Autonomous Robot

Rihem Farkh,Saad Alhuwaimel,Sultan Alzahrani,Khaled Al Jaloud and Mohammad Tabrez Quasim

1College of Engineering,Muzahimiyah Branch,King Saud University,Riyadh,11451,Saudi Arabia

2Laboratory for Analysis,Conception and Control of Systems,LR-11-ES20,Department of Electrical Engineering National Engineering School of Tunis,Tunis El Manar University,1002,Tunisia

3King Abdulaziz City for Science and Technology,Saudi Arabia

4College of Computing and Information Technology,University of Bisha,Bisha,67714,Saudi Arabia

Abstract: Several applications of machine learning and artificial intelligence,have acquired importance and come to the fore as a result of recent advances and improvements in these approaches.Autonomous cars are one such application.This is expected to have a significant and revolutionary influence on society.Integration with smart cities,new infrastructure and urban planning with sophisticated cyber-security are some of the current ramifications of self-driving automobiles.The autonomous automobile,often known as selfdriving systems or driverless vehicles,is a vehicle that can perceive its surroundings and navigate predetermined routes without human involvement.Cars are on the verge of evolving into autonomous robots,thanks to significant breakthroughs in artificial intelligence and related technologies,and this will have a wide range of socio-economic implications.However,in order for these automobiles to become a reality,they must be endowed with the perception and cognition necessary to deal with high-pressure real-life events and make proper judgments and take appropriate action.The majority of self-driving car technologies are based on computer systems that automate vehicle control parts.From forward-collision warning and antilock brakes to lane-keeping and adaptive drive control,to fully automated driving,these technological components have a wide range of capabilities.A self-driving car combines a wide range of sensors,actuators,and cameras.Recent researches on computer vision and deep learning are used to control autonomous driving systems.For self-driving automobiles,lane-keeping is crucial.This study presents a deep learning approach to obtain the proper steering angle to maintain the robot in the lane.We propose an advanced control for a selfdriving robot by using two controllers simultaneously.Convolutional neural networks (CNNs) are employed,to predict the car’and a proportionalintegral-derivative(PID)controller is designed for speed and steering control.This study uses a Raspberry PI based camera to control the robot car.

Keywords: Autonomous car;cascade PID control;deep learning;convolutional neural network;differential drive system;raspberry PI;road lane detector

1 Introduction

Since the middle of the 1980s,many universities,research centers,car companies,and companies in other industries have studied and developed self-driving cars(Fig.1)(also known as autonomous cars and driverless cars).

Figure 1:self-driving car

The Navlab mobile platform[1],the University of Pavia and University of Parma’s car,ARGO[2],and UBM’s vehicles,VaMoRs and VaMP,are all important examples of self-driving car research platforms from the last two decades.DARPA(Defense Advanced Research Projects Agency)conducted a test in 2004 to see if self-driving cars could complete the course,but no one did.Sebastian Thrun led his team to victory in the second challenge,which took place in 2005.Google developed the efficient self-driving car technology in the Toyota Prius,which was also licensed by the Department of Motor Vehicles in 2012[3].

Deep learning and artificial intelligence have been key technologies behind many breakthroughs in computer vision and robotics over the last decade [4,5].They also have a huge influence on the emerging autonomous driving revolution in academia and industry.A self-driving or autonomous car can sense its environment and travel to a pre-determined destination without the assistance of a human driver on a route that has not been specifically built for it[6].

With the advent of technologies and transportation,more cars now have self-driving modes to help passengers in protecting their health when driving long distances and reducing the risk of traffic accidents.The advantages of self-driving vehicles include the ability to predict and manage traffic problems and provide built-up accessibility for all users.Academic and industry research groups have developed various low-cost systems for studying research problems in autonomous driving and control[7-12].

Self-driving vehicles must maintain their lane.Because of numerous sensors on autonomous vehicles,such as radar,LiDAR,ultrasonic sensors,and infrared cameras,ordinary color cameras remain critical because of their low cost and capacity to collect rich data [13,14].Lane detection is a critical environmental perception tool for creating a self-driving vehicle.It is critical to monitor a vehicle’s movement in compliance with the road lane in which it is driving[10,11,15,16].

After detecting two lines,we must quantify and draw a virtual line in the center,then approximate the offset angle between the vehicle’s body and the virtual line to change the vehicle’s steering such that the vehicle is still in the middle of two lines under all circumstances[17].

The above-mentioned steering angle calculations are complicated and can result in several errors.Several roads have no lane or blurry lane markings.Furthermore,as cars are speeding along a sloping driveway,the camera installed previously will point to the sky and fail to keep up with the lane ahead,which can also lead to erroneous detection[18].

Recent developments in low-cost sensors and small-scale robot cars have provided new incentives and prospects for solving and overcoming the limitations,such as TurtleBot[19],Donkey car[20],and DFROBOT[21].Students and researchers interested in self-driving can use these robot cars.They are also permitted to design and build their robot cars using low-cost sensors and units,such as remotecontrol cars,Raspberry Pi(RPi),GPS,Wi-Fi,Bluetooth,and single/stereo cameras[22].

Convolutional neural networks(CNNs)have been the most prevalent deep learning architecture due to its efficacy on image classification problems since their introduction in the 1990.For selfdriving vehicles,deep learning has been widely used.Obstacle detection,scene recognition,and lane recognition are all major issues in self-driving cars that must be solved[23].

However,using CNNs,[24]showed an end-to-end learning method for self-driving vehicles.Reference[25]introduced using YOLO and CNN algorithms,which allow the system to detect road lanes,objects,and make suggestions to the driver.However,the system was recommended only for use on highways,not on city streets.

This study proposes a self-driving vehicle prototype,in which experimental hardware components and methodologies are used in a self-driving car.The device consists of an RPi as the key part operating the algorithms,cameras linked to the RPi,and various sensors.The Arduino is also a critical component because it controls the vehicle motors and their movements.

2 Autonomous Robot Car Architecture

Training the car in a real-time environment,such as a motorway or road with heavy traffic,is a risky and cost-driven task.To manage such a situation,a small RPi-based car and a custom environment were established so that training can be conducted feasibly.

This section describes the architecture and system block.A suitable configuration was chosen for developing a line-tacking robot using a Pi camera connected to the motor driver IC through an Raspberry PI3 B+.Fig.2 illustrates this configuration.

Figure 2:self-driving robot structure

The system’s implementation on the Raspberry Pi ensured the following functions:

? Data and image collection using the Pi camera

? PID controllers and the expected steering angle are used to determine the speed of the left and right motors

? directing the robot in the desired direction.

3 Deep Learning Based PID Control for Self-Driving Vehicle

To make the vehicle autonomous,image processing,AI,and deep learning approaches are used.To control the robot,a cascaded PID controller is designed.Fig.3 shows the overall proposed control diagram.

Figure 3:Main controller for autonomous lane-keeping robot

To control the robot’s maneuver,steering control is needed.Therefore,two PID controllers are employed to control the robot’s direction.The first PID controls the throttling speed (basic speed)based on the steering angle calculation.The steering angle is determined using an image processing controller.The input of the system is images;the output is steering angle in degrees(θ).Algorithm is applied on one single image(one frame)and will be repeated on all frames.The second PID determines the car’s deviation speed using a CNNθprediction.The trained network receives an image frame from a single forward-looking camera as input and generates a predictedθvalue as the output at each control period.

Ifθ=90°,the car will drive forward without needing to be steered.A constant throttling speed of 50%PWM is used to ensure that the robot car moves forward at 50%off the total speed.Ifθ＞90°,the car should steer to the right;otherwise,it should steer left.

The driving signals for the left and right motors are calculated using the following equations.

Right Speed=throttling speed(PID1)-speed correction(PID2)

Left Speed=throttling speed(PID1)+speed correction(PID2)

These driving signals will be used to generate the motor speed set-point for the robot.The speed control is tuned to make the robot move according to the speed set-point input using PID3 and PID4.For precise speed control of the two DC motors,encoders and PID controllers are required to obtain the actual positions and shaft rotations speeds.

3.1 Steering Angle Calculator

Lane detection is a critical component of intelligent transportation systems For more than a decade,lane identification has been an important topic research,with numerous techniques suggested,such as the Hough transform,Kalman filtering and probabilistic fitting.Fig.4 represents the entire operation.The system’s input is images and its output isθ.

Figure 4:Steering angle(θ)calculation process

Camera

The camera will start recording a video with a resolution of 100×100(Fig.5).Since frames per second(fps)will drop after applying the processing techniques to each frame,we propose lowering the resolution to obtain a better fps.

Figure 5:road line

hue saturation,value(HSV)color space conversion

After taking the video recording as frames from the camera,the following step is to convert each frame into an HSV color space(Fig.6).The main benefit is that it helps you distinguish between colors based on their luminance levels[26].

Figure 6:convert the image to HSV color

Detect white color and Edges

After converting the image to an HSV color space,we can now detect only the color we’re interested in (i.e.,white,since it is the color of the lane lines).Edges are only detected using canny edge detectors to minimize the overall distortion in each frame(Fig.7)[26].

Figure 7:Canny edge detection

Select region of interest(ROI)

To concentrate only on one area of the picture,a ROI must be chosen.We do not want the robot car to see many things in the environment in this situation.We only focus on the lane lines in the bottom half of the frame and disregard the rest(Fig.8)[26].

Figure 8:ROI selection

Detect line segments

Figure 9:Identification of lane lines on the road

Average slope and intercept(m,b)

Here,the average of line segment slopes and intercepts identified using the Hough transform will be determined.Left lane points are defined as all lines with a negative slope;the right lane is the opposite.To improve detection accuracy,two boundary lines divide each picture into two sections(right and left).Right lane estimation is correlated with all width points greater than the right boundary line.If all width points are less than the left boundary line,they are linked to measuring the left lane(Fig.10)[26].

Figure 10:superimpose the left and right lines onto the original image

Calculate and display heading line

The heading line determines the direction in which the steering motor should rotate and the speed at which the throttling motors should run.Ifθ=90°,the car can drive forward without having to steer.Ifθ＞90°,the car should turn right;otherwise,it should turn left.

3.2 Steering Angle Estimation Using CNN

3.2.1 Data Description

When working with deep learning networks,data preparation is needed.The RPi captures images and data as the user drives the vehicle around the track at 4-5 km/h.Approximately 30,000 images are paired with steering angles in the data collected(Fig.11).The image’s initial resolution is 100×100 pixels.

To avoid the blur caused by vibration as the vehicle drives on the track,the Pi Camera is set to capture at 10 fps with an exposure time of 5000 us.Fig.12 provides several examples of this dataset’s images.

3.2.2 Data Augmentations

21.Offered meat and drink:If the king is fasting as a form of penance to take the offered food would be a break of that fast. Because he refused, he shows his piety like his wife does when she prays.Return to place in story.

Because there are too few examples to train on,deep learning models overfit the limited dataset,resulting in a model with poor generalization performance.Data augmentation manipulates incoming training data to produce more instances of training data by randomly converting existing examples into new ones.This approach expands the training size,reducing overfitting.Horizontal flip,brightness changes,random shadow,height,and width change are all common transformations.Furthermore,data augmentation is done only on the training data,not the validation or test data[27].

3.2.3 Training Process

The dataset was divided into three subsets:training(60%),validation(20%)and test(20%).The training dataset contains 36,000 augmented frames,and the validation set contains 6000 frames.The model was copied back to the RPi after it was trained on the desktop computer.The vehicle’s main controller then uses the model to feed a picture frame from the Pi camera as input.Each control cycle can process 10 images per second,and the vehicle’s top speed around curves is 5-6 km/h.

Figure 11:Collected data set

Figure 12:Samples from the driving dataset

3.2.4 Convolutional Neural Networks

CNNs is a powerful artificial neural network technique CNNs,are multilayer neural networks designed specifically for 2D input,such as video and pictures,and have become widely used in computer vision.These networks preserve the problem’s spatial structure,make effective use of structural information and patterns in a picture and were designed for object identification and classification tasks[28].

CNNs get their name from the convolutional layers that make up its structure.Convolutional layers’primary task is to detect specific local characteristics in each area of the input picture.A convolution layer is made up of a number of separate filters.The filter is slid over the entire image,and the dot product of the filter and chunks of the input image is calculated.Each filter is independently convolved with the image to produce feature maps.We can utilize a feature map for a variety of purposes,including reducing the size of an image while keeping its semantic content.

The network architecture comprises 18 layers,including 6 convolutional layers,and 4 fully connected ones.The input image is 100×100×3 (height×width×depth format).Convolutional layers were designed to perform feature extraction and were chosen empirically through a series of experiments with varied layer configurations.Convolutional layers use a 3×3 kernel and a stride of 2×2.The respective depth of each layer is 32,32,64,64,128,and 128 to push the network going deeper.

The max-pooling layer is a powerful tool used by CNN.This is a method that resizes large images but keeps the most important information about them.All max-pooling layers were chosen with a kernel of 2×2 and non-stride.The output is flattened after the convolutional layers,followed by a succession of fully connected layers of decreasing size:512,256,64,and 1.

Batch normalization is a layer that allows the network’s layers to learn more independently.It is used to normalize the output of the preceding layers and can be used as regularization to prevent the model from overfitting.By placing the BN after the rectified linear unit(ReLU)yields a slightly higher accuracy and lower loss.

All hidden layers are equipped with the ReLU to improve the convergence.From feature vectors,we apply Softmax to calculate steering wheel angle probability.The entire network will have～2,813,921 parameters (Tab.1) and will offer excellent training performance on modest hardware.Additionally,to make the architecture more robust and avoid overfitting,we use batch normalization and dropout[28]at a rate of 0.5 for the last two fully connected layer gatherings of most parameters.

Table 1:The convolutional neural network structure

Table 1:Continued

3.3 Proportional-Integral-Derivative Control

The PID constants were tuned separately in this cascaded PID method.A PID controller adjusts the output to keep the input variable close to the reference set-point.Three parameters can be tweaked to fine-tune this process(Kp,Ki,and Kd).Here is a continuous representation of the well-known PID controller equation.

Kp,Ki,and Kd represent the proportional,integral,and derivative gain constants,respectively.

To implement the controller equation in digital form,the reverse Euler method for numerical integration is employed.

u(kT)and e(kT)present the discrete time control and error signals at T sampling time.

The PID regulates both the left and right motor speeds based on the expected error measurement.The PID controller generates a control signal to establish the left and right speeds of the robot wheels(PID-value).It is a differential drive system.

The right and left speed used the duty cycle of the PWM and applied at the input pins of the motor driver IC.

The Ziegler-Nichols tuning method[29]was used to obtain the PID parameters(Kp,Ki,and Kd).To begin,the system’s Ki and Kd were set to 0.The Kp was raised from 0 to the maximum gain Ku,after which the robot oscillated constantly(Tab.2).

Table 2:Proportional-integral-derivative control tuning parameters

Because of the variations in line curvature,a traditional PID controller was found to be inadequate for the line-follower robot after extensive testing.To fix this problem,only the last three error numbers were summarized rather than aggregating all preceding values.The updated controller equation is as follows:

This technique provides satisfactory performance for lane detection.

4 Conclusions

We presented a vision navigation autonomous robot in which a camera takes an image input and a CNN model is employed to take a decision accordingly.To provide a smooth tracking line,we designed a deep learning based cascade PID controllers.The results indicate that the suggested approach is well suited to autonomous robots,as they can operate with imprecise information.More advanced controllers based deep learning could be developed in the future.

Acknowledgement:The authors acknowledge the support of King Abdulaziz City of Science and Technology.

Funding Statement:The authors received no specific funding for this study.

Conflicts of Interest:The authors declare that they have no conflicts of interest to report regarding this study.