Statistical analysis of helium bubbles in transmission electron microscopy images based on machine learning method

Zhong-Hang Wu · Ju-Ju Bai · Di-Da Zhang · Gang Huang · Tian-Bao Zhu ·Xi-Jiang Chang · Ren-Duo Liu · Jun Lin · Jiu-Ai Sun

Abstract Helium bubbles, which are typical radiation microstructures observed in metals or alloys, are usually investigated using transmission electron microscopy(TEM). However, the investigation requires human inputs to locate and mark the bubbles in the acquired TEM images,rendering this task laborious and prone to error.In this paper,a machine learning method capable of automatically identifying and analyzing TEM images of helium bubbles is proposed, thereby improving the efficiency and reliability of the investigation. In the proposed technique,helium bubble clusters are first determined via the densitybased spatial clustering of applications with noise algorithm after removing the background and noise pixels. For each helium bubble cluster, the number of helium bubbles is determined based on the cluster size depending on the specific image resolution. Finally, the helium bubble clusters are analyzed using a Gaussian mixture model,yielding the location and size information on the helium bubbles. In contrast to other approaches that require training using numerous annotated images to establish an accurate classifier, the parameters used in the established model are determined using a small number of TEM images. The results of the model formulated according to the proposed approach achieved a higher F1 score validated through some helium bubble images manually marked. Furthermore, the established model can identify bubble-like objects that humans cannot facilely identify.This computationally efficient method achieves object recognition for material structure identification that may be advantageous to scientific work.

Keywords TEM · GMM · DBSCAN · Helium bubble ·Machine learning

1 Introduction

A number of researchers have reported that the analysis of the evolution behavior of metals or alloys exposed to the complex and extreme environment in a reactor is a considerably exigent task [1–3]. The irradiation-induced damage to nuclear reactor components is one of the major problems that extremely detrimental to nuclear power generation because of the high cost of component replacement and the introduction of uncertainty in predicting the lifecycle of components. Neutron interactions with the material lead to atomic collisions that create point defects and defect clusters. A problem of particular interest is the accumulation of nanoscale helium bubbles in nickel-based superalloys used in nuclear reactors. Helium is mainly produced through the interaction of thermal neutrons emitted by the reactor core with nickel atoms when collisions occur. The nuclear reaction channel indicates that the nickel nucleus can crack into two nuclei of iron and helium after the absorption of one neutron. The presence of transmutation-generated helium has a critical role in the microstructural evolution of metals exposed to neutron irradiation [4–6]. When the concentration of helium atoms exceeds a certain threshold,helium coalesces into bubbles in the microstructure. Moreover, nanosized helium bubbles tend to appear and develop in the matrix,dislocations, and grain boundaries of the material, leading to swelling, hardening, or embrittlement [7–9]. Helium bubbles in metals have a complex evolution mechanism during service life of the material that is crucial to reactor safety.To prevent the decline in the performance of metals or alloys caused by helium bubbles,gaining insight into the dynamic evolution of helium bubbles, especially their movement and aggregation, is critical. To accomplish the foregoing, the use of a suitable technique to identify and analyze the microstructure of helium bubbles is essential.

Several experimental tools including small-angle neutron scattering [10–12], positron annihilation spectrometry[13–15], grazing incidence small-angle X-ray scattering,and transmission electron microscopy (TEM) are available to study the behavior of helium bubbles in metals or alloys[16–18].Among these tools,TEM is typically employed to capture helium bubble images for investigating their morphological evolution at the nanometer scale spatial resolution [19, 20]. Technically, bubbles images appear as white and black dots at the same location under underfocused and over-focused imaging conditions,respectively;hence,they are clearly distinguishable.However,owing to the powerful magnification in microscopy, the helium bubble images acquired by TEM are extremely sensitive to the quality of sample preparation and exposure setting. A typical TEM image in which the artifact caused by the mixed foreground and background and the nonuniform distribution of illumination intensity are easily differentiated is shown in Fig. 1. However, in the current practice,because the location, size, and density of helium bubbles are marked and analyzed manually by researchers,the task of characterizing the helium bubbles remains labor-intensive and prone to error. Presently, sample preparation and helium bubble marking in the image require days or even weeks. The analysis results are susceptible to inaccuracies due to several factors, such as visual fatigue during marking, subjective human judgment, uncontrollable image quality, and bubbles with irregular shapes.

Fig. 1 Typical TEM image of helium bubbles

To facilitate the characterization of helium bubble morphology, an image processing algorithm or a machine vision system may be introduced as an alternative to automatically count and identify the bubbles in TEM images. By exploiting electronic engineering and machine intelligence, the tedious task performed by humans can be easily accomplished. The necessity of detecting and characterizing objects, such as bubbles and cells from images,has been observed in various industrial fields [21]; generally, shape-based and template-based approaches are employed. In the former, object shapes are normally presumed to follow certain shapes, such as circles or ellipses.This assumption allows the use of shape detectors, such as the Hough transform,to project the candidate points inside circles in the image into the parameter space of center coordinates and the radius of circles[22].As the number of objects to be identified increases, these shape detectors generally have high computational consumption and require large memory storage. Moreover, small targets are typically undetected because the number of points is insufficient to cluster them in the parametric space. Both problems are associated with helium bubble images captured by TEM. In contrast, the template-based method relies on the sliding window technique rather than voting for each point on the circle. This means that a patch window containing the pattern of the object of interest is convoluted across all pixels in the image. The result with the highest similarity to the pattern is selected as the target object. This approach utilizes all the pixel information in the template to identify the location and shape of all objects of interest without the necessity of using high-frequency filters to extract the edges of objects. As a result, the robustness of detection is improved [23]. Similarly, the kurtosis wavelet energy[24],kurtosis curvelet energy[25],and hidden Markov random field-expectation maximization[26] are three efficient methods that may be implemented to analyze the statistical characteristics of helium bubbles.However, if the electronic lens magnification or the illumination conditions are altered during image acquisition,or if the objects are deformed or partially occluded, then the template-based approach may experience difficulty in correctly matching candidates.

As the shape and size of helium bubbles continue to evolve in the course of the investigation, sample preparation and image acquisition become difficult to control to achieve satisfactory reproducibility. Moreover, the image features of helium bubbles actually deviate from the hypothesis of traditional image processing approaches.This limits the applicability of these traditional techniques for automatic helium bubble detection. To resolve this, a more delicate machine learning approach has been proposed to augment the helium bubble identification process.The method is capable of achieving accuracy and reproducibility comparable to that accomplished by humans but requires a considerably less amount of time to generate the TEM data[27].By combining traditional image processing and the latest convolutional neural network techniques,objects can be detected with improved exactness [28, 29].To further enhance the detection precision, large datasets with qualified annotations are normally required for training and testing deep neural networks. However, although training using synthetic data has been employed in biological experiments, the complexity of the procedure involving TEM images may preclude the possibility of utilizing simulated images [30–34].

In contrast to the deep learning approach, where learning numerous labeled real or synthetic images is generally required to establish an accurate classifier, this paper proposes a new machine learning method. This proposed technique is implemented to automatically identify and analyze helium bubbles in the TEM images of nickel-based alloys implanted with helium ions at high temperatures.The images were processed and analyzed using the densitybased spatial clustering of applications with noise(DBSCAN) algorithm and Gaussian Mixture Model(GMM). After determining the key parameters of the model, the optimized model could be used to identify and count the helium bubbles in the TEM images of the nickelbased alloy. In case only a few marked TEM images were available for training, the performance achieved by the proposed approach is comparable to manual identification and analysis of helium bubbles.

This paper is mainly composed of three parts.Section 1 describes the research background of the identification and analysis of helium bubbles.Section 2 presents the proposed analysis methods as well as the details of the datasets used for modeling and training. Section 3 elaborates on the experimental results obtained using the proposed model;further discussions are also presented. Finally, the paper concludes with a brief summary.

2 Method

2.1 Dataset preparation

The helium bubble images analyzed in this work were captured from nickel-based alloy samples irradiated with helium ions at 650 °C on a 4-MV electrostatic accelerator at the Shanghai Institute of Applied Physics. The samples containing helium bubbles were prepared using a focused ion beam,and the microstructure was characterized using a Tecnai-G2-F20 transmission electron microscope under bright-field imaging conditions.

Helium bubbles can be identified under either out-offocus or under-focused imaging conditions. As demonstrated in Fig.1,the helium bubble images analyzed in this work are mainly captured under bright-field under-focused TEM imaging conditions. All bright-field TEM images contained three channels with the same size(1024 × 1024 pixels).For convenient processing,each image was equally cropped into 16 small images (256 × 256 pixels).

To evaluate the detection performance of the proposed algorithm, two experts in TEM analysis were requested to manually mark and double check the bubbles found in the TEM images.The annotated images were used for training and evaluating the capability of the new method in identifying TEM bubble images.

2.2 Statistical analysis of helium bubbles

In this study, the helium bubble TEM image was processed and analyzed in three steps: First, the image was preprocessed, eliminating the background and noise pixels from the image. Second, the remaining pixels were clustered by DBSCAN. Finally, the GMM was applied to analyze the helium clusters. A schematic of the entire model and a detailed description of the implementation steps are shown in Fig. 2.

2.2.1 Background and noise pixel elimination

The pixel matrix of the helium bubble image has three channels, each with different intensity values. To afford utmost convenience in data processing and maintaining the three-dimensional signal information, merging the three channel datasets into a single channel dataset is more advantageous.The intensity of each pixel is converted into a single value using the following:

Fig. 2 Schematic of designed model

where gray represents the combined intensity value of the components of the three channels; and R, G, and B are the red, green, and blue channel intensities, respectively.

Before attempting to remove the background intensity from the image matrix,the background threshold should be defined. The helium bubbles are assumed to be evenly distributed in the TEM image and have different intensity depths from the background. The matrix data of helium bubble intensities are considerably distinguishable from the background, and the most appropriate threshold value is derived by the training process.

After determining the background threshold value, the image matrix can be processed using the following:

where vthcorresponds to the threshold of background.

The background pixel value of the helium bubble image has a nonuniform distribution owing to the nonuniform illumination intensity; hence, determining an appropriate background threshold is difficult. To resolve this, the histogram equalization technology is introduced to adjust the helium bubble brightness against the background. This approach enhances the imaging contrast and facilitates the identification of helium bubbles.

Histogram equalization is a simple and effective image enhancement technology for improving the contrast of an image with small dynamic range intensity. It alters the grayscale value of each pixel to distribute the intensity histogram across the entire intensity range; consequently,the new image tends to be clearer. For example, the grayscale level of an overexposed image is concentrated in the high brightness range,whereas that of an underexposed image is concentrated in the low brightness range.Through histogram equalization, the histogram of an original image can be transformed into a uniformly distributed(equalized)form. This transformation increases the dynamic range of the grayscale value difference among pixels to enhance the overall image contrast. In other words, histogram equalization enhancement is achieved by compressing the grayscale value with numerous pixels (i.e., the grayscale value that performs a major function in the image) and expanding the grayscale value with a small number of pixels (i.e., the grayscale value that does not have a primary function in the image). The original bright-field under-focused TEM image and the results of histogram equalization operation are shown in Fig. 3. The separation of the helium bubble from the background is evidently more difficult in some regions, especially in the upper-left area. Nevertheless, the image contrast is significantly improved after performing histogram equalization on the original TEM micrograph. Thus, to a certain extent, preprocessing through histogram equalization can aid in setting a proper threshold value for distinguishing helium bubbles from the background.

2.2.2 Clustering

Clustering analysis and detection, which are known to have led to the rapid development of data mining techniques, are widely applied to fields, such as pattern recognition, data analysis, market research, and image processing. As an unsupervised machine learningalgorithm,its basic principle is to group similar objects into the same cluster. The more similar the objects in the cluster, the better the clustering effect. Highly similar helium bubbles evidently belong to a particular bubble type. Accordingly, a clustering detection algorithm is adopted to analyze the irradiated-alloy TEM micrograph for helium bubble detection and statistics. Specifically, the DBSCAN algorithm was applied to this study. This algorithm assigns cluster labels based on dense regions of points.In DBSCAN,the notion of density is defined as the number of points within a specified radius.

Fig. 3 a Original TEM image and b TEM image after histogram equalization processing

Numerous helium bubble clusters can be obtained after the background and noise are eliminated; thus, clustering the remaining pixels in the preprocessed TEM image data is necessary. This section presents the application of DBSCAN to cluster the calculated helium bubble pixels.

A special label is assigned to each sample point using the following criteria.

(1) A point is considered as a core point if a specified number of neighboring points falls within the specified radius.

(2) A border point is one that has fewer neighbors than the specified number within a specified threshold;however, it lies within the threshold radius of the core point.

(3) All points that are neither core nor border points are considered as noise points.

After labeling the points as core, border, or noise, the DBSCAN algorithm processes them following two simple steps.

Step 1: A separate cluster for each core point or connected group of core points is formed (core points are connected if they are not more distant than the threshold).

Step 2:Each border point is assigned to the cluster of its corresponding core point.

The DBSCAN algorithm can be used to identify the degree of pixel density and classify the pixels according to their distribution based on the pixel distribution density.At the same time, the relatively isolated pixels are marked as noise points, rather than helium bubbles.

Two parameters are extremely important in the DBSACN algorithm: eps and neibth. The first, eps, is the scanning radius whose value is set to 1 in this model; that is, only eight pixels around the current pixel were considered. The second, neibth, is a threshold value for identifying kernel points; its value is obtained by the training process. The main steps of DBSCAN are as follows:

1. Randomly choose an unvisited pixel as a starting point,and then find all of its neighbors within the scanning radius (eps) range.

2. If the number of neighboring pixels is greater than or equal to neibth, mark the current point and its neighboring pixels as kernel points, and put the neighbors in a queue. Otherwise, the current pixel is temporarily marked as a noise point.

3. Consider another unvisited point from the queue, find its neighboring pixels, mark the unvisited pixels as kernel points, and put them in the queue.

4. Repeat step 3 until the queue is empty;at this instance,the marked kernel points form a new cluster.

5. Repeat steps 1–4 until all pixels are marked as kernel points or noise points, and terminate the algorithm.

The shape of the cluster able to be detected from the above steps can be any of the forms; hence, DBSCAN remains considerably suitable for clustering helium bubbles with irregular shapes.

2.2.3 GMM

The GMM is a well-known basic statistical machine learning model widely applied to visual media fields.It is a type of unsupervised clustering model and density model consisting of a number of Gaussian function components.Within its framework, the feature density function is regarded as belonging to different component variables whose combination provides multimodal density. The GMM can afford greater flexibility and precision in modeling as well as deriving the underlying statistics of sample data;accordingly,it is popularly used in object recognition and classification.

Virtually all helium bubbles in metals or alloys irradiated by helium ions are spherical or ellipsoidal; thus, their two-dimensional images are represented by circles or ellipses. Based on this observation and GMM characteristics, the assumption that helium bubble images follow a normal distribution is reasonable. In addition, considering that some helium bubbles in TEM images may intersect or coincide, the multidimensional GMM, which is a probabilistic model for representing normally distributed subpopulations within the overall population, is applicable to helium cluster analysis. In the multidimensional GMM,knowing the subpopulation to which a data point belongs to is unnecessary;hence,the subpopulations are automatically learned.

where D is the dimension of the Gaussian distribution (in this work, D = 2 because the helium bubble cluster pixels have two dimensions).

If the component weights are not learned, then they can be considered as the a priori distribution over components such that p(x) = φk(where x is generated by component Ck). If they are learned, then they are the a posteriori estimates of the component probabilities given by the data.

Parameters, such as φk, μk→, and Σk, may be evaluated using the expectation maximization (EM) algorithm. The EM algorithm is a numerical technique for maximum likelihood estimation and typically used when closed-form expressions for updating model parameters can be calculated. The EM is an iterative algorithm with the property that the maximum likelihood of data strictly increases with subsequent iterations. This guarantees that the local maximum or saddle point is approached.

The EM for GMM consists of two steps:the expectation step(E step)and the maximization step(M step).The main objective of E step is to compute the expectation of the loglikelihood estimation. It uses the current estimated distributions of latent variables based on the parameters inferred from the previous step.For the E step,the probability, ^ωik,that xiis generated by component Ckis evaluated using the following equation:

The E and M steps are alternately run until the parameters converge. The EM algorithm is employed to iteratively optimize the parameter estimate. Note that the EM requires the a priori selection of the model order. Usually,the user selects a suitable number approximately corresponding to the length of the training expression.

In this work,the pixel value,pixel（i）,is used as the pixel mesh weight required in the GMM.Therefore,Eqs.(7)–(9)can be transformed into Eqs.(10)–(12),respectively,in this model.

Based on the foregoing statement and formulae, an automatic bubble detection model is developed, and the clustering classification algorithm is formulated. The results yielded by the GMM-based model provide the helium bubble information;that is,μk→describes the central points, and the covariance matrix contains the size and shape information of helium bubbles, respectively.

2.3 Assessment criteria

As described above, several parameters have to be determined to formulate the proposed model. The various parameters evidently lead to considerably different detection results and statistics. Therefore, it is necessary to first determine the assessment criteria for identifying the optimal parameters to achieve excellent detection results.

In machine learning,the three important indices that are frequently used for clustering evaluation are recall, precision, and F1 values. The precision index indicates the portion of positive identifications in a classification set that are actually correct.In contrast,the recall index represents the proportion of actual positives that are correctly identified. In the statistical analysis of binary classification,the F1 value (also called F1-score or F1-measure) considers the precision and recall of the test. The F1 value can be interpreted as a weighted average of precision and recall;the best F1 value is 1, whereas the worst score is 0.

The precision and recall values were calculated and monitored during the training and formulation of the proposed model. The deduced F1 value was calculated using these two parameters and utilized as the comprehensive performance evaluation criterion to determine the optimal parameters of the model. The F1, precision, and recall indices are expressed in the following forms:

where P and R represent the precision and recall of the model,respectively,and F1 is the harmonic mean value of precision and recall; Nmodelis the number of helium bubbles recognized by the desired model; Nlabelis the number of helium bubbles manually marked by the TEM analysis experts; and Nmlindicates the number of helium bubbles detected by the clustering analysis algorithm(these bubbles are concurrently marked manually by the experts). Precision P and recall R cannot be promoted at the same time;hence, F1 is defined to balance these two assessment criteria.

3 Results and discussion

As described in the previous section, first, the determination of four parameters is necessary: bgth(the threshold value of the pixel background), neibth(the minimum number of neighboring points of a pixel), clusterth(the minimum number of a cluster), and ndot(the number of pixels of a helium bubble). These parameters vary with a limited number of discrete integer values. The rational values can be conveniently determined by testing the combinations of all values.In this study,eight TEM images and corresponding annotated images were employed for model training to determine the best combination of the four parameters. Based on the generated results, eight candidate models were established and denoted as Models 1–8. For each TEM image, a combination of parameters is given at the highest F1 value,as well as the P and R results,as listed in Table 1.

Second, eight candidate models were formed using a combination of the four parameters after the training process.As listed in Table 2,four TEM images are selected to determine the optimal parameters for the proposed model with the F1 value as the evaluation criterion. The parameters of Model 5 are easily identified as having the best values; hence, they are chosen as candidates for the final model for bubble clustering selection.

Finally,four different TEM images were used to test the performance of the formulated model for bubble clustering.The corresponding performance evaluation results including the four datasets and their mean values are listed in Table 3.

The position of the helium bubble was evaluated under the Gaussian mixture analysis framework, and the coordinate deviation was compared with the manually marked images in this model. The mean bias for describing the deviation is computed using the following:

Table 1 Parameters of each sample image and F1, P, and R values

Table 2 F1 values of each test image evaluated on eight models

Table 3 Performance results of formulated model

where Pland PMare the central points of helium bubbles manually marked and estimated using the model, respectively.Here,the Manhattan distance is adopted to compute the distance between Pland PM.

The list in Table 3 indicates that the mean coordinate bias is 3 pixels and the results yielded by the model approximate those marked manually.

For example, Figs. 4–7 show the helium bubble detection results yielded by the formulated model and manual marking. In Fig. 4, the red points represent the helium bubbles detected by the described model, and the blue points represent those marked by the TEM analysis experts.The numbers of red and blue points are expressed as nredand nblue, respectively.

As shown in Figs.4–7,the number of red points exceeds that of the blue points. Virtually all helium bubbles have been recognized by the formulated model.Moreover,some objects similar to the bubbles, but with no manual markings, have also been identified by the established model.The superior recall (i.e., 94%) achieved by the proposed model means that it is capable of automatically locating and marking most of the manually marked helium bubbles.

Fig. 6 (Color online) a Original TEM and detected results with nred3 = 38 and nblue3 = 24

In contrast to the excellent recall performance, the precision level achieved by the model was relatively low.This may have been caused by two factors. First, some objects with a similar morphology as the bubbles are extremely dark or minuscule to be observed by human sight, but they can be identified by the machine learning method.Second,helium bubbles with irregular shapes may be manually marked as a single bubble; however, they are distinguished as multiple bubbles by the model. Using the proposed model, these two aspects may yield higher counting results compared with those manually marked.The foregoing may also explain the relatively low precision associated with the manual approach. Accordingly, the model established in this work is observed to be extremely sensitive. It can detect bubble-like objects and identify more candidates of helium bubble clusters than the manual marking approach. By considering Fig. 4 as an example,Fig. 8 indicates that the model developed in this work is considerably sensitive and capable of dealing with situations in which helium bubbles may have irregular shapes or small sizes.

4 Summary

This study is the first to employ the DBSCAN clustering method and the GMM model for efficient identification and analysis of helium bubbles in the TEM images of alloys in a high-temperature irradiation environment. In the current practice,considering that the manual counting of bubbles is time-consuming, the ability of the formulated model to identify and count these bubbles automatically considerably saves time.The new method is found to have a higher recall but low precision performance because it canidentify objects with irregular shapes and those unnoticed by the naked eye. The nature of automatic identification and the substantial time saved afforded by the designed method highlights new opportunities in the use of machine learning to expedite helium bubble identification and counting.

Fig.7 (Color online)a Original TEM and b detected results with nred4 = 46 and nblue4 = 45

Fig. 8 (Color online) Comparison of results obtained by model and manual marking

The results obtained in this study are based exclusively on TEM datasets under the under-focused condition of bright-field TEM micrographs. Nevertheless, the model is also expected to be capable of adapting to the circumstances of the over-focused condition. A more accurate helium bubble counting performance can be derived by considering the results of these two conditions together.Furthermore,the variation in the shape and size of bubbles must also be considered because helium bubbles evolve dynamically and diversely throughout the entire service.Hence, considerable work awaits completion and verification through follow-up research.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing,adaptation,distribution and reproduction in any medium or format,as long as you give appropriate credit to the original author(s) and the source,provide a link to the Creative Commons licence,and indicate if changes were made.The images or other third party material in this article are included in the article’s Creative Commons licence,unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.

Author contributionsAll authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by Zhong-Hang Wu, Ju-Ju Bai, Di-Da Zhang, Tian-Bao Zhu and Ren-Duo Liu. The first draft of the manuscript was written by Zhong-Hang Wu and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.