Defect Detection in Printed Circuit Boards with Pre-Trained Feature Extraction Methodology with Convolution Neural Networks

Mohammed A.Alghassab

Electrical Engineering Department,College of Engineering,Shaqra University,Riyadh,11911,Saudi Arabia

Abstract:Printed Circuit Boards(PCBs)are very important for proper functioning of any electronic device.PCBs are installed in almost all the electronic device and their functionality is dependent on the perfection of PCBs.If PCBs do not function properly then the whole electric machine might fail.So,keeping this in mind researchers are working in this field to develop error free PCBs.Initially these PCBs were examined by the human beings manually,but the human error did not give good results as sometime defected PCBs were categorized as non-defective.So,researchers and experts transformed this manual traditional examination to automated systems.Further to this research image processing and computer vision came into actions where the computer vision experts applied image processing techniques to extract the defects.But,this also did not yield good results.So,to further explore this area Machine Learning and Artificial Intelligence Techniques were applied.In this study we haveapplied Deep Neural Networks to detect the defects in the PCBS.Pretrained VGG16and Inception networks were applied to extract the relevant features.DeepPCB dataset was used in this study,it has 1500 pairs of both defected and non-defected images.Image pre-processing and data augmentation techniques were applied to increase the training set.Convolution neural networks were applied to classify the test data.The results were compared with state-of-the art technique and it proved that the proposed methodology outperformed it.Performance evaluation metrics were applied to evaluate the proposed methodology.Precision 94.11%,Recall 89.23%,F-Measure 91.91%,and Accuracy 92.67%.

Keywords: Printed circuit board;convolution neural network;inception;vgg16;data augmentation

1 Introduction

Printed circuit boards (PCB) are included in almost all the electronic devices and it plays an important role in proper functioning of these devices.These components are being utilized in industrial usage for decades and for domestic purposes.There are many fields where PCBs are used such as logistics,defense,aeronautics and for the applications utilized for automobiles and industries related to medical,because PCBs are considered as the basic need for the design of electronic items.PCBs are made up of solid thin plates which are equipped with plastic coted materials like fiberglass,sometimes with composite epoxy which actually form a base to support the chips and electronic components [1,2].

These board are designed in such a way that would make pathways to form a circuit for the power electronic devices which are fixed to PCBs.PCBs need inspection for the proper functionality of the devices as the manufacturing is continuously increasing so also the inspection is getting on demand especially for the modern manufacturing.Industries like production and manufacturing are making utmost efforts in getting 100% quality assurance for the PCBs [3,4].PCBs support various electronic components such as integrated circuits,transistors and capacitors as shown in Fig.1.These components are mounted on the board by drilling holes and soldering them to form circuit pattern [5].While undergoing these steps the PCBs could get spoiled and these are not in a position to be used in real and needs to be detected before exploring into the market.Inspection is very important step in the manufacturing and distribution of PCBs and needs to have 100% assured of all the parts.There are variety of approaches for automated inspections for PCBs during the past two decades.PCBs are in much demand as they are utilized in many consumer electronic gadgets such as laptops,tablets and smartphones.Based on this demand the manufacturing of PCBs is done in large quantities.So,the manufacturing of efficient and error free PCBs are essential and crucial.As they are manufactured in large quantities the accuracy and error free production is a big challenge and maintaining the quality is a big concern [6].So,to overcome this challenge,inspection systems are deployed which could help in maintaining the quality of PCBs,but these teams are manual which are prone to error.So,automated inspection teams are needed which can prove to be efficient and less prone to errors.These automated inspection on PCBs also prove to be fast and detect the defects in the PCBs more accurately.Detecting the defected PCBs is very crucial because if it is not done with accuracy then it might lead to big destructions as the whole device might fail in proper functioning where these defected PCBs were installed.In the traditional methodology these defects in PCBs are detected by automatic inspection (AOI) machines [7].After that a skilled quality inspection engineer would check all the PCBs.Many boards are classified as defected by AOI but,they are not.Since the machine has erroneously classified a PCB as defective based on some scratches or in the presence of tiny holes,dust particles like nanoparticles,paper fragments or even a small air bubble.Sometimes small difference about sample might also lead AOI machines to declare the PCB as defected as shown in Fig.2.They are a need for efficient automated machines to handle these kinds of issues.Therefore,it is very essential to handle detection of defected PCBs more efficiently to prevent further obstacles and to fix the errors occurred by the visual inspection.Inspection of the PCBs consists of two major processes: Detection of defect and classification of defects.To overcome the errors and increase the defect detection accuracy image processing techniques are widely applied in PCBs manufacturing companies [8].These image processing techniques can locate the parts in PCBs where the defects have occurred and further classify them.But,still they are some limitations in these techniques and to overcome those machine learning techniques are applied in the very recent times [7].Detecting the defection in PCBs has been a source for many techniques being introduced.In the very recent times machine vision-based algorithm were presented,these techniques focused on automated detection of the defect.Machine vision-based techniques also helped in recognizing the different components in the PCB image.Depending on the situation and requirements,these techniques were implemented.These algorithms were categorized into certain number of classes and divided based on the requirement.Basically the algorithms are of four different types: Machine learning based [9,10],Projection types [11,12],algorithm based on filters [13,14],and finally hybrid methodologies [15,16].In the current time machine learning is applied extensively,especially in solve image processing problems because of its ability to automatically produce discriminative features.These features are developed with little training samples with the help of introducing learning and pattern recognition algorithms.These algorithms include artificial neural network (ANN) and support vector machines (SVM).Currently with the development and introduction of deep learning which has attracted lot of researchers globally in various fields [17-19].With the implementation of deep learning many real-time issues were handled efficiently,especially for long-standing machine-vision tasks that improved the classification accuracy significantly.These implementations include image segmentation [20] and also many image classification problems [21,22].Because of the extra ordinary capability of deep learning it can be applied for PCBs defect detection.In PCB defect detection the uneven distribution of color which exists ion the chip location could be detected.This study is based on brightness improvement distribution on the PCB images.As discussed above manual detection is done by trained engineers which are failing at sometimes because of the limitations,to overcome this issue an automated system should be introduces which can do this job more efficiently.So,machine learning based techniques which is both accurate and fast in detecting the defects in PCB images are introduced with the help of deep learning techniques.The objective of this research is to introduce deep learning techniques to detect the defection in the PCB images which can reduce the false detection rate and further increase the production rate.According to few studies like this [23],it has revealed that deep learning is outperforming traditional machine learning approach for classification and algorithms used for extraction of features.In another work which is related to PCBs defect detection also have applied deep learning approach [24-26].Other approaches are also implemented for PCBs like [27] which is based on template-matching.Similarly,another approach for PCB defect detection is applied [28],in this method the authors have used OpenCV for image subtraction.There are so many methodologies applied but convolution neural network (CNN) is applied in many applications like [29,30],where the core idea is image recognition for the detection of objects.

Figure 1:Basic printed circuit board with all the components

The rest of the paper is as follows: Section 1 give the full introduction about the research field,Section 2 discusses the literature review,Section 3 describes the dataset used in this study and also image pre-processing and data augmentation details,Section 4 discussed the actual proposed methodology,followed by Section 5 discuss the implementation,and Section 6 with results and analysis and Section 7 has the conclusion and Section 8 with discussion followed by references.

Figure 2:Printed broad circuit without the extra components to detect the defect on the board

2 Literature Review

Detecting the defects in PCB could be classified into two types: image processing-based techniques,machine learning based techniques,and object detection method.In the image processing technique,the errors are detected on the PCB images and then classify them if found any error.Some of the techniques are listed as shown in the Tab.1.

Image processing-based techniques:Further to this research,the authors of this work [31],have introduced a PCB inspection system to detect the defect in PCB bars by applying image processing techniques.The outcome of this method is recognition of common defects such as missing of holes and open circuits.In another work [6] the authors have introduced an automated visual inception system for detecting the defects in PCBs.A template image is used as reference and then mapped with the real PCB image and if any difference is found after subtraction then the inspected image is classified as defected.The authors in this work [5],have collected all the possible defects that could occur on PCBs and introduce a MATLAB based image processing technique to classify them.They have discovered 14 different types of defects which were further classified into 5 different groups.In another approach,the authors in this work [32] have introduced a unique methodology based on morphological image segmentation for detecting the defects in PCBs and classify them as defected,the approach is image segmentation algorithm together with image processing theories.Another image processing technique is applied to detect the defect in PCBs.In this work [33],the authors have applied wavelet-based image difference algorithm.This is an automatic system which checks the difference in wavelet form and classifies the image as defected if it finds some difference with the template image.A different approach is adapted in this work [34],the authors have tried hybrid methodology by combining genetic algorithm and neural network classifier to detect the defect in PCB.

Machine learning based techniques:Feature extraction was done by genetic algorithms.PCBs had another type of defect caused by solder-balls,this defect was studied by [35],they detected this defect with the help of data mining approach.The authors in this work [36] have adapted latest machine learning technique called deep learning as it best suited for dataset without unstructured and unlabelled data.In this work they have compared the result with different machine learning classifiers like support vector machine to detect the defects in PCBs.In another work [8]Convolution neural networks (CNN) was applied as a classification algorithm to detect defects in PCBs.Similarly in another approach [37] CNN is applied to recognize the defects in PCBs.Deep discriminative features were applied to detect the defect in PCBs [38].The defects were located by applying sliding window approach.In another approach,the authors in [39] applied Auto-VRS technique.It uses deep neural networks to check if the defects detected with correct or not.A multi-label learning problem was handled by [40],in this work multi-task convolution neural network was proposed.The proposed methodology developed a model which has the capability of defining label learning like a binary classification task.Deep learning technique was applied to detect the defects in PCBs by [41],in this work DeepPCB dataset was used which has 1500 images.These images are pairs of defective and intact PCBs.In another work [42] related deep learning.The authors have proposed a new approach by utilizing Autoencoder for the extraction of discriminative features.

Object detection method:Object detection methodology can be efficiently applied in PCBs to detect any defects.Based on this assumption many researchers have proposed this technique,some of them are discussed here.In recent time numerous generic object detection techniques are proposed as discussed in [43].Some of the as example are two stage methodology (e.g.,RCNN [44],and FastRCNN [45]) and some are based on single stage detection like (e.g.,YOLO [46] and SSD [47]),all these techniques have demonstrated excellent accuracy by applied the dataset presented in [43].Recently object recognition technique is extensively applied for PCBs defect detection.Some of the works are discussed here.In this work [48],the authors have proposed machine learning and Deep learning both for the detection of PCB components by applying AdaBoost classifier to detect the capacitors ion the PCBs.Similarly in [49,50],they have applied DL to investigate the PCB components.In [49],they have made an intensive analysis by applying YOLO architecture to find the capacitors and resistors on the PCBs.In [50],object detection technique is applied which was based on neural networks to identify numerous PCB components like capacitors,resistors,and other smaller components.Some of the major electronic components where PCBs could be implemented are [51] and [52].

Table 1:List of techniques applied for the detection of defects in the PCBs

3 Dataset

The dataset used in this work for the training of proposed methodology and development of the model is DeepPCB.This dataset is used for the detection of defects in PCBs.It consists of 1500 pairs of images,where one image in the pair is a template image and the other is the defected image.There are six types of defects annotated to the test images.The defected images are names as open,short,mousebite,spur,pin hole and spurious copper.This dataset is freely available to the research community.The following are the advantages of this dataset:a) The images of each corresponding pairs (template and test image with defects) are aligned by template matching methodology.This technique of aligning the images already reduces lots of efforts especially for the image pre-processing.b) This dataset is available for free as shown in Fig.3.It can be used by the researchers to detect the defect in PCB as the dataset is available publicly.This dataset is collected by following the regulations of common industrial settings.The images collected are obtained with the help of linear scan CCD with the resolution of around 48 pixels per 1 millimetre.The clean image which defect-free acting as template image is collected manually and further it is monitored and dressed in respect to the sample image.The raw size of the image for both template and defected are 16k × 16k pixels.These raw size images are then clipped and reshaped to sub-images with new size as 640 × 640 pixles and then they are aligned with template matching method.This methodology reduces the translation and rotation offset among the pairs of the images.Further to the data collection process,a threshold is then carefully designed to apply binarization for avoiding the disturbances caused by illumination.Image preprocessing is also applied depending on the PCB defects.The techniques are image registration and thresholding,these techniques are applied to get high-accuracy PCB defect localization [26].The images are annotated by applying axis-aligned bounding box technique.The images were annotated into six types and each one with a defect type.The following were the annotation name of PCB defects: open,short,mouse bite,spur,pin hole and spurious copper.

3.1 Pre-Processing

The PCB images were applied for image processing.There are various image pre-processing techniques available like smoothing,background subtraction,resizing etc.These techniques are applied to enhance the quality of the image and make it able for any kind of Machine Learning technology.

3.2 Data Augmentation

Data augmentation technique is applied on the images of the dataset if the size of the dataset is not enough to be implemented on deep learning neural network.Since deep learning technique needs lot of data to process and train the model.To be able to train and test the proposed model in deep learning,the size of the dataset should be increased.To increase the size few techniques are needed to be applied such as cropping,rotating,scaling,translating and flipping.There are few other techniques like Generative Adversarial Networks (GANs) which are applied to generate new images for training which have proved in the past to be more significant especially for data generation task.The purpose of applying data augmentation is not only increasing the data size but also this approach improves the current state of the art classification task.In another study feature-space and data-space can also be used to perform data augmentation.

Figure 3:DeepPCB sample pair images of both template and defected image

4 Proposed Methodology

The proposed model in this work takes care of only one class for training and the goal is to extract features which are not similar concerning the normal samples.The following is the architecture designed for neural network.The desired outcomes would be: a) the proposed model should extract the comprehensive features for the normal samples.The purpose is to distinguish between normal and defective samples based on the features,b) this designed architecture should be flexible and can be adopted to other domains.

Convolution Neural Network (CNN) in the current times has proved to be an efficient technique for image classification and detection of objects,as shown in Fig.4.But,the training process of CNN needs a big amount of data to train the proposed model and it needs extensive computation power.Keeping this in mind the researchers have proposed an alternative to overcome this shortcoming by introducing transfer learning technique.This technique is proved to be efficient especially for the experimentation where the training data is in limited size [53].The basic idea behind transfer learning is based on the learning done on one specific problem could be re-used for solving different set of problems.The layers in CNN which is trained be: Starting layers are learned to produce generic low-level image representations like blobs and textures.The later layers are responsible for producing high level of semantic representations for the output.ImageNet [54] dataset is used for training the model which is not similar to PCBs dataset,but he advantage of using ImageNet is its number of images trained which proves to be robust for the next level of learning the different datasets for the extraction of features.

Figure 4:Basic building block of convolution neural networks

The problem in this study is portraited as detection of anomaly wherein the defects are calculated without much collection of big data related to defects,but rather discover the important features which would help in detecting the defects in the image.In this work transfer learning approach is implemented which is considered as the common approach to extract features and further gives a customized data.Here the transfer learning is done based on VGG16 network [55].It is a deep neural network which has 16 layers (13 of them are convolution and the rest are fully connected as shown in Fig.5.This network was trained to classify images with 1000 categories.The total images used for training are around 1 million.ImageNet dataset is used to train this network with the available pertained weights for transfer learning.Features are extracted based on different approaches for custom image datasets.These approaches cold be removing the final fully connected layer or sometimes beginning layers are frozen with retraining the other layers based on the custom dataset.As a regular practice it is used for supervised transfer learning,the last layer has neurons equal to the number of classes corresponding to the training model for classification.The original features vector for fc2 is 4096 dimensions and further the final output of the convolution is around 25,099 elements.

The proposed approach has two main parts: Initially we apply VGG16 pretrained convolution layers which are frozen to extract the features.Then,in the second phase we follow the approach presented in [56],this technique is based on unsupervised representation of learning on simple geometric transformations.By following this step,it is expected to give a rich data representation for transfer learning.This data is then fine-tuned for representing as training data.After transfer learning approach,several experiments were conducted for the optimum VGG16.As shown in the figure is the most efficient VGG16 configuration which proved to be successful.The description of our configured VGG16 is as follows: all the layers of VGG16 were frozen starting from the input layer till the flatten layer,after freezing these layers then we detached the final layers fc1 and fc2 layers and upon that new two dense layers were added.These two layers had 1000 and 66 neurons respectively.After all these changes softmax layer was added.The parameters which were output from the frozen layers are served as an input to the FC1.

Figure 5:A building block of VGG16 pretrained network

Another pretrained model is utilized to extract the features for classification.In this process Inception-V3 network [57] is applied.This network is also trained on ImageNet dataset for the purpose of feasibility study in the mid-level for the extraction of features for further classification of PCB images.The mid-level representations [58] of the PCB images are extracted from the output of the third Inception module i.e.,‘Mixed-2’layer of the Inception-V3 network as shown in Fig.6.Inception module consists of multiple-sized convolutional filters which provide features at different scales.We train an adaptation network,consisting of three convolutional layers,one max pool layer and two fully connected layers,using the extracted mid-level representations of the PCB images.The combination of the pre-trained Inception-V3 network up to its ‘Mixed-2’layer and the adaptation network trained on the midlevel representations provides the complete CNN to classify the PCB images.

Figure 6:Inception pretrained network architecture

5 Implementation

As can be seen in the below Fig.7 which gives the clear picture about the implementation steps being carried on in this research.Initially DeepPCB dataset is downloaded which consists of 1500 paired images of both defected and non-defected images as discussed in the dataset section of this paper.Then pre-processing steps were applied wherein image processing techniques were applied on the PCB images.Then pretrained networks are applied to extract relevant features and after that image classification technique is implemented to detect the defects in the images.

Figure 7:Overall workflow of proposed method

6 Results and Analysis

The process of defect detection includes threshold methodology.The difference between the template image and the defected image is calculated based on the threshold.The amount of difference would determine the possibility of the image being defected.The defect is calculated if the difference in the two images crosses the threshold,then the image is considered as a defective image.If the difference is below the threshold then is considered as non-defective image.In this proposed method this threshold is determined which categorizes the image.The best threshold is calculated based on true positive rate recall,precision,selectivity,accuracy and,F-score.The evaluation of the proposed methodology could be evaluated based on these four equations.These equations would estimate the accuracy of prediction for the input images if they belong to defected image or non-defected image.The following are the parameters in these equations:True Positive (TP),False Positive (FP),True Negative (TN),and False Negative (FN).Here TP represents the correctly classified positive cases,TN represents the correctly classified negative cases,FP represents as the positive cases being classified incorrectly,FN represents the negative cases being incorrectly classified.After all these parameters,the exact measurement is further evaluated based on the accuracy which ultimately interprets the performance of the proposed model.The accuracy is expressed by using TP,TN,FP,FN as represented by Eq.(1).The other significant performance metric for multi-class classification are precision recall,and F1-score are expressed using Eqs.(2)-(4) respectively.The results of evaluation metrics are as shown in the Tab.2.

As it is shown in Tab.2 when threshold equal to 100 the accuracy is maximum,and it is equal to 0.986.It should be noted that this threshold works fine with all kinds of defects.The proposed methodology was implemented using Python,TensorFlow and Keras framework.We have taken 100 images in random and the accuracy of detecting the defects were very good.Testing was done on 80 PCB images.The following are the defect detection categories discovered in this study.“Missing hole”is a kind of defect which was detected lowest because all the incorrect holes were incorrectly identified and located.There is another reason for the detection of low defects,like threshold estimation.If the results are measured by evaluation metrics,then sensitivity was around 94% and specificity of 92%.Another reason for better results is the technical system which has pre-trained models for the extraction of discriminative features for detecting the defects.The Unsupervised learning had good information which used to extract useful knowledge from the unlabeled data.Transfer Learning also proved handy for the excellent results which helped in extracting the relevant information.This approach also helped in escaping the overfitting issue,this overfitting occurs when the machine cannot adequately understand the structure of the data since the results in the model does not fit because it is insufficient and also because if the smaller number of relevant features.Apart from Transfer learning,data augmentation also played an important role which forced the model to extract the relevant features which were considered as useful and essential for the detection of defects.The information extracted by rotating the images helped in enhancing the defect detection accuracy.This approach helps in getting more training data which would make the model more robust and efficient to tackle other serious issues.The results as shown in Tab.3.describes the test set classification accuracy achieved based on the proposed pretraining models.The comparison is done with the results of other methodologies [24]and the proposed methodology of detecting the defects in PCBs.As can be seen in the Tab.3,the proposed methodology is giving good results for classifying the defected PCBs.The accuracy achieved by [24] is 70.8% and the proposed technique achieved better results.Now it is confirmed that the proposed methodology with Transfer Learning outperformed the high-level representation method [24].The comparison is done between features extracted traditionally which was based on engineering and further classifying them to detect the defects more often based on predefined PCB conductor shape and illumination patterns.Then the comparison is done with the Transfer Learning based approach which helps is improving and increasing the training data which reflects in enhancing the classification accuracy.

Table 2:Performance matric comparison for the threshold

Table 3:Comparative analysis

7 Conclusion

Printed Circuit Board are used in almost every electronic device.The functionality of these devices strongly depends on the PCBs being implemented in them.If the PCB is defected,then no matter how good the devices is designed it would give a disastrous performance.So,proper implementation and proper development of PCBs are the most important thing.Initially these PCBs were checked manually by human beings with their naked eyes which had many issues as defected PCBs were considered as good and the prefect PCBs were categorized as defected because of the human error.When this thing happened very commonly the researchers took this issue and tried to solve it automatically by developing automated systems.These automated systems were developed based on the latest technologies.Computer vision and image processing came into picture with its ability to detect the objects.There are many image processing techniques available to detect the object.In this work DeepPCB dataset was used to detect the defects in the PCB images.Convolution neural networks were applied.Feature extraction techniques were applied especially pretrained VGG16 and Inception networks were applied to extract the relevant features.Transfer Learning were applied to the training dataset.Before that data pre-processing and data augmentation techniques were applied to increase the training data to make the model more robust.Ultimately by applying pretrained models the accuracy was measured with performance metrics and finally compared our proposed method with some other methods.These comparative analyses clearly stated that the proposed method outperformed other methods in detecting the defects in the PCBs.

8 Discussion

In this work Printed Circuit Board images are studied for any defects in them.DeepPCB dataset is used for this study.The dataset has pairs for defect and non-defect images.The nondefected images are considered as template images.Data pre-processing steps are applied to the images with data augmentation techniques like rotation,flipping and transformation.This data augmentation helped in increasing the training data size which helps in making a goof training model.Transfer learning technique is applied using ImageNet dataset to increase the feature counts.After pre-processing and data augmentation,a pretrained models were applied to extract most relevant features.VGG16 and Inception networks were applied to the training data to extract most relevant and discriminative features.These pretrained models helped in classifying the defected PCBs from that of the non-defective’s PCB images.Finally,we have compared our results with others work who used only traditional feature extraction techniques.The results show that the proposed methodology outperformed other methods.

9 Limitations and Future Scope

In this work the defects in the PCB were detected.But more accuracy could be achieved by applying more recent feature extraction and pretrained models.Also,this technique could be applied on other applications like detecting the defects in plastic injection molding products.

Funding Statement: The author would like to thank Deanship of Scientific Research at Shaqra University for their support to carry this work.

Conflicts of Interest: The author declare that they have no conflicts of interest to report regarding the present study.