Method for Aircraft Sheet Metal Part Recognition Based on Cascading Virtual-Real Fusion

，，

Chengdu Aircraft Industrial（ Group） Co.Ltd.， Chengdu 610073， P.R.China

Abstract: A cascading virtual-real fusion approach is proposed to recognize various aircraft sheet metal parts （SMPs）with remarkable similarities.The SMPs are identified and numbered by cascading “Rough” “Fine”， and “Preferred”Through the virtual-real fusion approach of virtual workbench modelling and physical workbench actual recognition.The“Rough” SMP set is identified by gathering the main direction image of the sheet metal item on the real workbench and obtaining an eight-dimensional （8D） shape-description vector from the image.This leads to the discovery of a candidate SMP set.Then， template matching is conducted on the candidate SMP set based on the image’s grey information， and “Fine” matching is obtained.A quantitative index of recognition reliability is proposed to subsequently initiate the “Preferred” recognition process， which is accomplished with an augmented reality 3D projection.The effectiveness and superiority of the propsed method are verified by real experiments， and the best accuracy rate of 96.9% is achieved in testing parts.With the help of 3D projection， the accuracy of man-machine combination is 100%.

Key words：image recognition； aircraft sheet metal part； simulation imaging； 3D projection；virtual-real fusion

0 Introduction

Sheet metal parts （SMPs）［1］are the main parts of the body of an aircraft， accounting for a large proportion of the aircraft parts， with several types，small batches， and various shapes［2］.Aircraft sheet metal surfaces often require heat treatment and painting after batch forming.To improve the efficiency， various SMPs are generally mixed for batch processing［3］.After treatment， the SMP is identified for subsequent use.Currently， SMP recognition after heat treatment and painting is primarily performed through manual comparisons using 2D drawings or 3D tooling.Owing to the wide variety of SMPs， there are various parts that must be recognized， and some of these high-similarity parts are difficult to be distinguished； these are vulnerable to misidentification， resulting in low efficiency and frequent errors and causing subsequent production problems.

With developing computing and sensor technology， machine vision is widely used in industrial production.Wang et al.［4］extracted speckle and scaleinvariant feature transform （SIFT） features from images of aerospace electrical connectors［5］and input them into a support vector machine （SVM） to obtain the training model of parts.Subsequently，through the Hough transform and training-model prediction， the online recognition of aerospace connectors was realised， with an average recognition accuracy of more than 90%.However， the shape characteristics of aerospace connectors are quite different from those of SMPs； therefore， this method cannot be used to solve the problem stated in this study.Wu et al.［6］combined three geometric invariants to construct feature vectors of parts and subsequently associated feature vectors of part images with class labels to construct training and test sets.Thereafter， the optimised SVM［7］was used to recognize part images， achieving an average recognition accuracy of 95.31%.This method of combining labels and feature vectors inspired this study.Joshi et al.［8］extracted fourteen shape-invariant factors，such as roundness and the average grey level from the part image， and combined them with the SVM to complete the classification and recognition task of regular mechanical parts， e.g.gears.The recognition accuracy was as high as 98%， indicating that the shape factor is an important feature in image recognition.Yin et al.［9］developed a convolutional neural network （CNN） based on Fast R-CNN［10］in the field of machine vision to recognize assembly parts.This approach utilizes the latest advancements in artificial intelligence technology.The recognition accuracy of ten types of electrical connectors reaches 94.25%， but the interpretability and adaptability of the neural network to SMPs are still unknown； the training time of the data set was quite long.Lv et al.［11］proposed a cross-granularity recognition method for SMPs， which achieved better recognition results than conventional visual methods.However， in this method， the SMP must be placed manually multiple times according to the image superimposition on the computer screen to perform the finegrained identification method of contour matching，which involves multiple manual adjustments and is inconvenient to operate.

Based on the aforementioned research， the process characteristics of SMPs are analysed， and the key points for recognition are summarised as follows： （1）Due to the process requirements of SMPs for matching the shape of the body of aircraft， there are often pairs or series of indiscernible SMPs with highly similar structural shapes， and even if the feature vectors such as the shape-invariant factor or geometric invariant moment are used in the image，it is still difficult to distinguish them.（2） Due to the process characteristics of SMPs， the surface lacks texture information， and the number of local features， such as SIFT， extracted from the part image is small and unstable.（3） Because of the time-variant changes of aircraft shape， the placement in the recognition stage is different from that in the design stage.Additionally， the positioning reference from the design cannot be used for recognition as the profiling tool， clamping plate， and fixture for auxiliary recognition are not applicable.（4） Periodical upgrades or modifications to the production system lead to continuous expansion and improvement of the batch plan and part database.New types of SMPs may be introduced at any time， making it difficult for methods that rely heavily on sample training datasets， such as classification recognition trainer or neural networks， to meet actual production requirements.Periodic retraining and testing may be unacceptable in such cases.

Due to the actual production requirements of SMPs， a cascading SMP-recognition method based on virtual-real fusion and an aircraft sheet metal parts identification system （APIS） are proposed.The virtual-real fusion recognition is conducted through a cascading procedure of “Rough”“Fine”，and “Preferred” recognition using virtual workbench modelling and physical workbench recognition.Only one industrial camera and a projector are required for conducting SMP recognition.

1 Basic Workflow

Fig.1 SMP pairs and series

Fig.2 SMP similarity in different views

Fig.3 Basic workflow of the proposed method

SMPs usually appear in pairs or series as a result of the production process， as shown in Fig.1（a）.In the series of SMPs， similar parts exist with the same shape and structure of sheet metal parts and only small size differences， which are easy to confuse， as shown in Fig.1（b）.However， significant differences exist among the major categories of SMPs； therefore， they are relatively easier to distinguish.The analysis shows that most SMPs have obvious differences from the top view， but the differences in similar parts are small， as shown in Fig.2.Therefore， a cascading method for recognising SMPs with virtual-real fusion is proposed.The main workflow of the SMP-recognition method proposed in this study is shown in Fig.3.It is primarily composed of two parts.The first part is the physical processing part， which mainly collects the information of the physical SMP and extracts the feature factors， including the processing of the sample physical SMP and the SMP to be recognized.The second part is the virtual processing part， which primarily includes the use of a virtual 3D digital model of a SMP to generate simulated images for projecting and the corresponding bench calibration.The information of both parts is stored in the SMP information database.In the recognition process， the feature information of the SMP to be recognized is extracted and matched with the information in the database to determine the candidate SMP information set.This is the “Rough” recognition， in which the paired or SMPs to be matched can be obtained.Then， the template-matching method is used to perform“Fine” matching from the candidate SMPs，and the results are evaluated according to the quantitative indicator.If the reliability meets the requirements， the default selection is appropriate.If the reliability does not meet the requirements， the proposed 3D projection-aided recognition method is used to project the pre-generated simulation image of the candidate matching SMP.The “Preferred”recognition is determined when the image is completely consistent with the physical SMP through human-machine cooperation.

2 Virtual-Real Fusion

Fig.4 APIS hardware

Based on the structural characteristics analysis of SMPs， the hardware of the APIS system developed in this study primarily comprises an industrial camera， a small projector， a computer， a workbench， and structural supports， as shown in Fig.4.The industrial camera is installed on the support directly above the workbench， and it is used to collect the top view image of the physical SMP.The projector is installed beside the camera and projects the simulated image to clearly show the image differences.Database establishment， workbench calibration，and projection simulation image generation are important components of virtual-real fusion.

2.1 Database establishment

Before the SMP recognition， the information database should be established for storing， updating， and management of the information of various SMPs.Each record is composed of the part number（PN）， top view image， an 8D feature， and a 3D projection simulation image.The basic process of establishing a sheet metal information database is as follows：

（1） A sample of each SMP is placed on the workbench under the top-view camera.The camera is turned on to snap the image of the SMP sample，and image feature information is extracted and stored in the database with the PN.

（2） The SMP profile in the image is obtained through the method［12］.The area， perimeter， convex hull［13］area， minimum bounding box length［14］，minimum bounding box width， minimum bounding box aspect ratio， and first-order and second-order［15］with good rotation and translation invariance are extracted and stored as the shape feature indicator.

（3） According to the virtual 3D digital model and calibration information of the SMP sample， a projection simulation image is generated and stored in the database as the attribute information of the SMP.The workbench calibration and projection simulation image generation are described in detail in Sections 2.2 and 2.3.

2.2 Workbench calibration

To perform the SMP recognition based on virtual-real fusion， the positional relationship between the camera， projector， and top plane of the real workbench， as well as the virtual workbench must be determined.In this study， the internal and external parameters of the projector are calibrated using a 2D plane calibration plate and three frequency fourphase shift-structured light［16-17］.During the calibration of the internal and external parameters of the projector， the planeXwOwYwof the world coordinate systemOw-XwYwZwis selected on the plane calibration plate.The position and attitude of the plane calibration plate are altered multiple times.Under each position and attitude， the projector projects structural light to the calibration plate and the top view camera captures the corresponding calibration plate image.To accurately calibrate the positional relationship between the projector and the workbench top plane， one position of the calibration plate is set as the plane calibration plate， and it is completely attached to the top plane of the workbench.The planeXwOwYwof the world coordinate system is set parallel to the top plane of the workbench.The projection is like the inverse process of camera imaging， and its internal parameters include the equivalent focal length of two directionsfx，fy， as well as the coordinatesu0，v0of the principal point.The posture transformation matrix [Rp tp] between the calculated projector and world coordinate systems is determined by the calibration process，whereRpdenotes the rotation matrix andtpthe translation vector.

After the projection calibration is completed， a virtual recognition platform consistent with the actual recognition platform is established.The workbench of the virtual recognition platform is shifted downward by a plane calibration plate thickness in theXwOwYwcoordinate plane to ensure consistency of the position， posture of the virtual recognition platform， and the actual recognition platform in the world coordinate system.Subsequently， the 3D digital model of the SMP is imported， and the stable placement state of the SMP on the actual workbench is simulated.Further， the virtual model is triangulated to determine the coordinatesPiandNinormal information of each vertex of the triangular mesh， wherei=1，2，…，n，ndenotes the total number of vertices of the triangular patch after triangulation， as shown in Fig.5.In this manner， the coordinate information and normal information of each point on the virtual SMP in the world coordinate system are obtained.

Fig.5 Virtual SMP coordinates and normal information

2.3 Projection simulation image generation

To recognize a SMP based on virtual-real fusion， appropriate projection simulation images must be generated.After the workbench has been fully calibrated， the image coordinates of the virtual pointPion the projection image plane are calculated based on the coordinate informationPiand normal informationNiof the virtual SMP in the world coordinate system combined with the projection parameters obtained through the calibration.In this study，the projection model shown in Eq.（1） is established based on OpenGL.

where [u v1 ]Tis the homogeneous coordinate of the image point on the projector image plane corresponding to each point to be calculated on the sheet metal； [xwywzw1 ]Tthe homogeneous coordinate corresponding to a point of the sheet metal part in the world coordinate system；Mthe transformation matrix from the world coordinate system，determined when the calibration plate is completely fitted on the identification workbench， to the projector coordinate system [Rptp]；Pthe projection matrix， which can be defined by the OpenGL perspective projection cone［18］， as shown in Fig.6.Here，Op-XpYpZpis the projector coordinate system，Znear，Zfarare the distances from the viewpoint to the near and far cut-off surfaces of the viewing cone， respectively， and (xmin，ymin，Znear)， (xmax，ymax，Zfar) the lower-left and upper-right corner coordinates of the viewing cone.The projection matrix is expressed as

In Eq.（1），F(xiàn)is used to perform viewport transformation on the normalised coordinates after perspective projection.The image obtained by perspective projection is mapped onto an image plane with a resolutionW×Hof pixels through affine transformation.Affine matrixFis expressed as

Fig.6 View frustum in OpenGL

wherewandhare the width and height of the image plane in pixels，respectively.

To make the simulation image more realistic，Phong’s illumination model is used to simulate the illumination.The ambient light component of the illumination is denoted asal，dldenotes the diffuse light component，slthe specular light component，amthe ambient light reflection component of the material，dmthe diffuse reflection component， andsmthe specular highlight reflection component.al，dl，sl，am，dm， andsmare 3D vectors.The lighting and material are specified in advance， combined with coordinate informationPiand normal informationNi，the colour outputCican be accurately calculated.

where diff denotes the diffuse reflection illumination influence factor， spec the specular illumination influence factor，Plthe position vector of the light source，Pvthe position vector of the projector，nml(Ni) the normalisation operation of vectorNi，rfl(N1，N2) reflection vectorN1along the normal axis of vectorN2，a×bthe Hadamard product of vectoramultiplied byb， andmthe highlight index（32 in this experiment）.

The image coordinates and pixel colours of each point on the SMP obtained above are input into the OpenGL， rendering a pipeline for rasterisation and segment interpolation to generate a projection simulation image of the SMP under the perspective of the projector with the same resolution（W×H） as that of the projector， which is stored in the database as the auxiliary information.

3 SMP Cascading Recognition

3.1 “Rough” feature recognition

The SMPxitem to be recognized is horizontally placed on the workbench under the top-view camera， as it is positioned when establishing the information database.A top-view image of the SMPxitem is taken with the top camera， and then 8D feature vector is extracted from the image， denoted asFx.SMPi，i=1，2，…，n，wherenis the information stored in the database.The corresponding 8D feature vectors (A，C，Ac，L，W，R，Hu1，Hu2) are denoted asFi=(x1，x2，x3，x4，x5，x6，x7，x8)T.

Using the global nearest neighbour-searching（NNs） algorithm from Ref.［13］， relative Euclidean distanceDibetween feature vectorFxof the part to be identified and feature vectorFiof theith SMP in the database is calculated as

wherexjandyjrefer to thejth components of the vectorFiandFx， respectively.

The relative Euclidean distance threshold is set asDT.IfDi≤DT， SMPiwithDiis sufficiently similar to the part SMPxbeing recognized； therefore， it is a candidate SMP with the same PN as the part SMPx.Using this preliminarily “Rough” screening method， the candidate SMP set is recorded asS.Because aircraft SMPs usually appear in pairs or series， there are usually two or more candidates inS.

3.2 “Fine” template matching

To further determine the exact PN of the SMP being recognized， “Fine” template matching is adopted based on the main direction alignment.The item is denoted as SMPj， wherej=1，2，…，J（Jis the total number of SMPs in the candidate setS）.Due to the different shapes of SMPs， accurate repeated positioning cannot be realised， resulting in different placement angles of the part SMPxon the workbench and SMPj， which will affect the results of direct template matching.Therefore， the method described in Ref.［16］ is adopted to solve the minimum bounding rectangle for the image SMPjmatching SMPx.Anticlockwise angles between the long side of the minimum bounding rectangle and the positive direction of the horizontal axis of the image are denoted asθj，θx.After the top-view image is rotatedθj-θxanticlockwise， the rectangular image area surrounding the object is recognized， and the candidate image is cropped； then the normalised square difference in Eq.（8） is used to calculate the matching degree between SMPxand SMPj.

whereT(m，n) andI(m，n) represent the pixel values of SMPxbeing recognized and candidate SMPj，respectively， at the pixel points (m，n)；M×Nis the rectangular image area.

The highest matching degreeCmin1=MIN(Cx1，Cx2，…，CxJ) is calculated within candidate setMwith SMPx， and the second highest matching degreeCmin2=MIN( {Cx1，Cx2，…，CxJ}Cmin1).The reliability index of template matching is taken as

IfΔ>CT，CTis the given threshold， theCmin1corresponding candidate is the reliable final recognition result.Otherwise， smallΔindicates that it is difficult to reliably distinguish the two candidates of optimal and sub-optimal matching.If the matching method cannot obtain reliable matching results in cascading recognition， the subsequent projection-assisted “Preferred” recognition is conducted.

3.3 3D projection auxiliary“ Preferred” recognition

In aircraft production， there are strict requirements for the reliability of identification.If the recognition result obtained in Section 3.2 is unreliable，further steps are required.Therefore， the projection auxiliary “Preferred” human-machine cooperation method， which projects a simulation image of the virtual model from the projector for an SMP candidate with a given PN， is proposed.During humanmachine interaction recognition， if the real SMP placed on the workbench is correctly corresponding to the virtual one， the virtual image should completely coincide with the physical SMP.

For SMPxbeing recognized， if multiple candidate SMPs are selected according to the method in Section 3.2 but theΔis not reliable， the projection auxiliary “Preferred” recognition process will be triggered.On the actual workbench， SMP is placed for recognition， and its location is adjusted to verify if the projection simulation image can perfectly match the 3D shape of the SMP item.If it does completely coincide， the SMP being recognized and the current candidate SMP should be identical with the same PN.If it is impossible to completely overlap，simulation images of the remaining candidates should be projected separately until the PN of the SMP being recognized is determined and the recognition task is completed.

4 Results and Analysis

To verify the recognition efficiency and accuracy of the cascading virtual-fusion SMP recognition method proposed in this study， thirty-nine categories of actual SMPs are selected for recognition experiments.Each category has at least one highly similar counterpart which is vulnerable to error.In the experiment， the threshold is set asDT=0.1，CT=0.3.

4.1 Candidate set generation experiment based on the feature index

Each SMP is subjected to ten recognition tests， and 3 309 number recognitions are performed.First， the candidate sets， of which the relative Euclidean distance to the shape feature of object SMPxto be recognized is greater than the threshold valueDTfound in the information database， are investigated.For the 39 types of SMP items involved in the experiment， the correct recognition results of each kind being recognized are included in the candidate set obtained by the algorithm used in this study， and no missing recognition occurs.Because the number of elements in the obtained candidate set does not exceed two， and they all contain correct recognition results， the Top 2（Top 2 refers to the proportion of the first two places of the similarity index containing correct recognition results in all experiments， and Top 1 has a similar meaning） recognition accuracy rates of all SMPs participating in the experiment based on the similarity discrimination of the 8D feature vector relative to the Euclidean distance in this study reach 100%.If the result with the smallest relative Euclidean distance is directly selected as the recognition result， the Top 1 recognition accuracy rate of the 390 experiments is only 58.5%，and the main reason for the recognition error is that the relative Euclidean distance difference between easily mixed parts is small.The relative Euclidean distance of the feature vector can “Roughly” distinguish the SMP items from other categories， but it cannot achieve further subdivision between the pairs or series of items.

Table 1 shows the feature information of SMPs in the information database and the relative Euclidean distanceDibetween them for recognising SMPxitems.Because the feature data and values of SMP in the image are small， they are processed by taking common logarithms and absolute values to ensure stability when calculating the Euclidean distance.The second column of Table 1 presents the characteristic information value； Columns 3—8 present the feature information values and corresponding SMPxrelative Euclidean distancesDi（Diin descending order from left to right） of the Top 6 SMPi（i=1，2，…，6） with the highest similarity obtained from the information database.In the last line， the difference between the relative Euclidean distance between SMP1， SMP2and SMPxbeing recognized is small， and both are less than the threshold valueDT.Therefore， results revealed that these two SMPs are highly similar to the parts being recognized and serve as the constituent elements of the candidate setM.

Table 1 Feature information and relative Euclidean distance of SMPx and the Top 6 candidates

The above experiments show that the candidate set generation method based on the 8D shapefeature information “Rough” recognition proposed in this paper can effectively extract candidate setS，which highly similar to the recognized parts.

4.2 Cascading template-matching and recognition reliability experiment

Fig.7 Easily misrecognized SMP pair items

In the 390 recognition tests performed by cascading “Fine” template matching of candidate easily misrecognized SMP items， the corresponding candidate parts are incorrect 12 times.Moreover， after“Fine” cascading template matching， the Top 1 recognition accuracy reached 96.9%.Fig.7 shows a pair of easily mixed SMPs.As shown in Fig.7（a），the detail difference between the two parts is that the positions of the convex structures on the bending edges are different.However， there is no obvious difference in the top-view image shown in Fig.7（b）；therefore， the difference between the shape feature vector and the template matching output is small.Table 2 shows the average values of the templatematching degree and reliability measurement index between the candidate sets of the pair of SMPs in 10 experiments，Cˉmin1，Cˉmin2andΔˉ.For the cases of the misrecognized Top 1 SMP item， the reliability index of recognition is less than the threshold value of 0.3； therefore， projection auxiliary “Preferred” process is automatically triggered.For comparison with the state-of-the-art deep learning algorithm， ViT with excellent performance in image classification task has also been tested on the same dataset of SMP.Table 3 shows the recognition results of the corresponding network model.In the table， Accuracy 1 represents the result of directly recognizing all the aforementioned experimental pictures， and Accuracy 2 the result of recognition after applying the screening method proposed in this paper.The proposed cascading method outperforms ViT method with more than 3% in this SMP recognition task.

Table 2 Values of easily misrecognized SMP pair items

Table 3 Comparison of deep learning algorithms %

4.3 3D-projection auxiliary recognition experiment

The pair of easily misrecognized SMP items in Fig.7 is taken as an example to illustrate the 3D-projection auxiliary recognition process.As shown in Figs.8（a， b）， the projection simulation images of the two candidates of the SMP being recognized are denoted asI1andI2， respectively.Because the value of the reliability index is below the given threshold，the projection of the candidate is automatically triggered.The coincidence between virtual projectionI1and the physical item being recognized is shown in Fig.9.The projection cannot completely coincide with the 3D shape of the real object； therefore， the SMP being recognized and the candidate SMP1item being projected have not the same PN.The coincidence situation of the truly identical candidate SMP2item projectionI2for physical item is shown in Fig.10.The projection images are completely overlapping with the SMP item being recognized.Therefore， it is practical and intuitive to use our 3D-projection auxiliary method for “Preferred” recognition.The coincidence of virtual imageI2with a real item determined that they should be identical with the same PN.

Fig.8 Simulation images of a pair of candidate items

Fig.9 Partial coincidence of projection I1 on a real item

Fig.10 Complete coincidence of projection I2 on a real item

Through the proposed 3D-projection auxiliary human-machine cooperation “Preferred” recognition method， the small difference between highly similar SMPs can be practically and intuitively recognized，enabling the operator to classify the type of the SMP.

5 Conclusions

An aircraft SMP-recognition method based on virtual-real fusion is proposed， and a cascading“Rough” “Fine”， and “Preferred” process is designed and implemented to robustly recognize highly similar SMP items.Different from the object-recognition method using a classification-recognition trainer or neural network， the method proposed in this study does not require various image samples for each type of SMP in advance， and only a top-view image is collected for each piece of SMP.The results of 390 recognition experiments on performed 39 SMPs indicate that the cascading recognition method based on template matching with the relative Euclidean distance of a feature vector index can automatically achieve a Top 1 accuracy of 96.9%，which adheres to most recognition requirements.For the remaining 3.1%， the Top 2 candidates are all correctly selected， and the human-machine cooperation process is automatically triggered with the reliability index.It is concluded that the proposed method is highly practical and robust for recognising and distinguishing the truly identical SMP items according to the coincidence level of the 3D virtual projection and the corresponding real item.