Transfer Learning-Based Image Recognition of Nitrogen and Potassium Nutrient Stress in Rice

Nitrogen (N) and potassium (K) are essential elements for rice growth.When N and K are in deficient rice,specific symptoms appear on the leaves that are similar and difficult to distinguish with the naked eye.To identify the category and degree of N and K nutrient stress in rice leaves as early as possible,we investigated two convolutional neural networks (CNNs) by scanned rice leaf images under nine nutrient stress degrees,including four different degrees of N stress [the concentration of N1 to N4 as 0,28.6,57.2 and 85.7 mg/L,respectively,with the concentration of K (K2SO4) as 89.3 mg/L],and four different degrees of K stress [the concentration of K1 to K4 as 0,22.3,44.7 and 67.0 mg/L,respectively,with the concentration of N (NH4NO3) as 114.3 mg/L],as well as a control group [CK,with the concentration of N (NH4NO3) as 114.3 mg/L and K (K2SO4) as 89.3 mg/L] under a normal condition.We chose the advanced EfficientNet-B0 model and the widely used Inception-V3 model,which are first trained separately by transfer learning,and then fused to form EVI(EfficientNet-V3 Integration) model to obtain more accurate results.In this study,the classification effects of the models were compared before and after transfer learning,and the effects of the two models after using transfer learning were also compared with the effects of EVI fusion model separately.The results proved that the use of transfer learning was effective,and the EVI model obtained a good accuracy of 97.1% on the category of rice leaf nutrient stress,which was 3.7% and 1.5%better than the Inception-V3 and EfficientNet-B0 models,and a good accuracy of 94.0% on the degree of rice leaf nutrient stress,which was 3.3% and 1.1% better than the Inception-V3 and EfficientNet-B0 models,respectively,effectively identifying the category and degree of nutrient stress in rice leaves.

Identifying and diagnosing rice nutrient deficiencies is an integral part of scientific fertilizer application,and plays an important role in the management and decision-making of agricultural production.If nutrient stress is present,identifying the category of nutrient stress and determining the degree of nutrient stress before irreversible damage developed would effectively improve planting efficiency.Both N and K are major nutrients for rice growth and are involved in the physiological and biochemical metabolic activities of rice in different ways (Ma et al,2022).N deficiency is characterized by slow plant growth,short plant height,long and thin stem,and small and yellowing leaf.Leaf yellowing starts at the leaf tip and extends along the midrib to the base of the leaf.K is an activator of many enzymes in plants.Rice under low K stress starts with slow growth and later the leaves begin to yellow with burnt leaf tips and margins,sometimes with irregularly distributed necrotic spots on the leaves.As N deficiency,symptoms of K stress appear firstly on older leaves.In order to meet the needs of modern agricultural production,it is imperative to study an automatic observation method for the early determination of N and K nutrient deficiencies in crops.Most traditional machine learning methods focus on pre-computing key points and features of domain-specific image,and then training classifiers in the corresponding representation space (Sun et al,2018;Chen et al,2019;Saleem et al,2019;Jeyaraj et al,2022).However,these methods have some limitations,such as hand-crafted features for specific categories of data,which may not work for different tasks.In recent years,deep learning-based detection methods have entered the agricultural sector (Porikli et al,2018;Jeyaraj et al,2022),with CNN being one of the most common methods,such as plant disease damage prediction (Liu et al,2020;Krishnamoorthy et al,2021),plant disease image recognition(Ghosal et al,2018;Chen et al,2020;Sethy et al,2020;Gui et al,2021),fruit counting (Bhattarai and Karkee,2022),plant tassel stage observation (Bai et al,2018),plant species identification (W?ldchen and M?der,2018;Jeyaraj et al,2022),and nutritional defect (Ferentinos et al,2019).In the production of field,sometimes rice would appear similar symptoms under N and K stress,which affects agricultural workers to judge the growth status of rice with the experience.Therefore,this study first used deep learning to identify the types of rice stress,and then on this basis,identified the stress levels under each type.

Fig.1.Analysis of nitrogen (N) and potassium (K) nutrient stress in rice leaves based on different models.

When growing rice,the supply of N and K fertilizer is usually essential.However,when rice is slightly stressed by N or K,it shows similar symptoms.It is difficult to distinguish the nutritional status of rice at this time and give correct nutritional supplies only through expert experience.Therefore,to enhance the accuracy of the model for the identification of rice nutritional status,rice samples were cultivated by hydroponics under nine nutrient stress degrees,with four different degrees of N stress (N1 to N4) and K stress (K1 to K4)from extreme to slight deficiency,and a control group (CK)under normal conditions (Figs.1-A and S1;Table S1).The top-four leaves under nine nutrition levels,totaling 4 495 samples,were collected on July to September of 2013,2015 and 2016.A total of 1 930 rice leaf samples under four different N levels,2 031 samples under four different K levels and 534 samples with normal nutrition levels were collected to build the diagnosis rules and the identification models.

All the samples were analyzed in the laboratory.First,the leaves were placed on a scanner (EPSON GT20000,Seiko Epson Corporation,Suwa,Nagano-ken,Japan) with a maximum scanning area of 11.7 × 17.0 inches and an RGB/BK color CCD line sensor.The resolution was set to 300 dpi (dots per inch). Since the number of acquired leaf images was relatively small compared to other deep learning datasets and the number of healthy leaf images was much lower than the number subjected to N and K nutrient stress,image enhancement(random scaling,flipping,rotating and mirroring) was performed on this dataset to increase the overall number with the data enhancement model in Python (Fig.S2) The Inception-V3 model was slightly weaker and slower than the EfficientNet-B0 model (Fig.1-B),probably because the EfficientNet-B0 model adjusted the network depth,width and resolution of the input network,whereas the Inception-V3 model only adjusted the depth of the network,which was less effective than the EfficientNet-B0 model.The EVI fusion model was better than the two models trained separately at identifying the category of nutrient stress and the degree of stress in rice (Fig.S3),indicating that the EVI fusion model is a best-of-breed model that can well find the optimal weight parameter matrix of the two CNN models,which was selected to fuse and obtain the final prediction results.The EVI fusion model can achieve high recognition and classification accuracy with good discriminative ability for the enhanced dataset (Fig.S2).As shown in Fig.1-B,CK leaves and leaves stressed by K deficiency were easily confused and interfered with the results,but the EVI model integrated the best results of the other two models,so it still could maintain a high discrimination ability,and the EVI model had lower misclassification results compared with the other two models.In terms of specificity,the EVI model always had a high accuracy rate,which indicated that the EVI model can accurately identify and classify most leaf nutrient stress categories,which is the most important factor to diagnosis the growth status of rice in the field.From the above three evaluation criteria (accuracy,sensitivity and specificity),it can be seen that the EVI model had good recognition and classification abilities,and its performance was superior in all aspects.

In this enhanced rice leaf dataset,the models were trained in two ways,one directly and the other based on transfer learning,and the recognition of each category gradually improved with increasing time.Taking the rice leaf nutrient stress category as an example (Fig.1-C),the accuracies of the directly trained models were both lower than those of the models using transfer learning,and the losses of the directly trained models were both higher than those of the models using transfer learning,with the directly trained Inception-V3 model having the lowest accuracy and the highest loss score for rice leaf nutrient stress categories.To find out the reason for this situation,we trained the test Inception-V3 network model several times on the enhanced dataset.Here,the number of misjudgment samples can be calculated by using the confusion matrix of Inception-V3 model.The results showed that K deficient category was always mixed with K healthy category,and the number of recognition error images reached 10,with much higher than those of other categories,which proved our previous conjecture that the healthy category was not easily identified with images that were slightly deficient in K nutrient elements.When rice is slightly deficient in element K,the color characteristics of the leaves do not differ much from those of healthy normal leaves,and it is not surprising that this phenomenon arises because the amount of data for normal healthy leaves is small compared with the other two deficient nutrients in the original dataset,and a large number of images of healthy normal rice leaves will be needed to improve accuracy in the future.The EVI fusion model based on transfer learning fuses the information in the Inception-V3 model and EfficientNet-B0 model,compensates some defects of the Inception-V3 model,and improves the classification accuracy of EfficientNet-B0,reaching 97.1% (Fig.1-B).Taking the degrees of nutrient stress in rice leaves as an example (Fig.1-D),the overall situation was the same as the recognition classification of leaf nutrient stress categories,which further justified the use of transfer learning in this small dataset of experiments.Transfer learning not only simplifies the training of the model,but also reduces the computing space and time.However,the loss of the Efficient-B0 model and the Inception-V3 model based on transfer learning gradually approached and stabilized.As with the identification of leaf nutrient stress degree,the EVI fusion model selected the best results from the optimized Inception-V3 model and EfficientNet-B0 for fusion,so that the highest identification accuracy of 94.0% was achieved (Fig.1-B).For individual nutrient stress levels,the EVI model was also the best in terms of accuracy,with 84.7%,88.7%,86.0%,85.2%,84.6%,83.5%,88.2%,85.8% and 85.1% under different N (N1-N4) stresses,K (K1-K4) nutrient stresses and CK,respectively,while the Inception-V3 model had 74.0%,79.1%,75.3%,74.7%,73.8%,77.9%,78.3%,78.1% and 78.5%,respectively,and the EfficientNet-B0 model had 80.4%,84.7%,84.9%,83.9%,80.2%,81.6%,85.1%,84.8% and 79.9%,respectively.These data also proved the application significance of EVI model construction.The identification and classification of rice leaf nutrient stress categories outperformed those of rice leaf nutrient stress degrees (Fig.1-C and -D).The reason for this phenomenon may be due to the small amount of data in each category in the rice leaf nutrient stress degree recognition dataset,with only one-third of the categories in the dataset being rice leaf nutrient stress categories,and the model’s inadequate knowledge of their characteristics,leading to misclassification of some images (Bai et al,2018;Barbedo,2019;Saleem et al,2019).To address this issue,it can be hoped that subsequent experiments will collect a sufficient number of data images to increase the dataset as a way to improve the accuracy of model recognition and classification.

Compared with the single model,the fusion model EVI proposed in this study made full use of the difference and complementarity of different CNN models in terms of network structure and parameters,and achieved the fusion and semantic complementarity of network models by fusing the advantages of different deep CNN models,which improved the recognition accuracy of the category and degree of N and K nutrient stresses in rice leaves,and helped to make more accurate in different environments.In addition,the model can be used for diagnosis of nutrient stress in other plant leaves.By deploying these improved models in mobile environments (e.g.portable scanners),plant pathologists and farmers will be able to quickly and easily diagnose plant diseases and take the necessary preventive measures to promote fine agriculture.

ACKNOWLEDGEMENTS

This study was funded by the National Natural Science Foundation of China (Grant Nos.31801255 and 52071200).

SUPPLEMENTAL DATA

The following materials are available in the online version of this article at http://www.sciencedirect.com/journal/rice-science;http://www.ricescience.org.

File S1.Methods.

Fig.S1.Different degrees of nitrogen and potassium nutrient stresses.

Fig.S2.Dataset of nutrient stress type and degree of rice leaves.

Fig.S3.Framework of EfficientNet-V3 Integration model.

Table S1.Nutrient contents of nitrogen and potassium at different nutrient levels.