Evaluation of UAV-derived multimodal remote sensing data for biomass prediction and drought tolerance assessment in bioenergy sorghum

Jiting Li ,Dniel P.Schchtmn ,Coy F.Creech ,Lin Wng ,Yufeng Ge,c ,Yeyin Shi,*

a Department of Biological Systems Engineering,University of Nebraska-Lincoln,Lincoln,NE 68583,USA

b Department of Agronomy and Horticulture,Center for Plant Science Innovation,University of Nebraska-Lincoln,Lincoln,NE 68588,USA

c Center for Plant Science Innovation,University of Nebraska-Lincoln,Lincoln,NE 68588,USA

d Department of Agronomy and Horticulture,Panhandle Research and Extension Center,University of Nebraska-Lincoln,Scottsbluff,NE 69361,USA

Keywords:Crop phenotyping Unmanned Aerial System (UAS) Thermal Machine learning Drought stress

ABSTRACT Screening for drought tolerance is critical to ensure high biomass production of bioenergy sorghum in arid or semi-arid environments.The bottleneck in drought tolerance selection is the challenge of accurately predicting biomass for a large number of genotypes.Although biomass prediction by lowaltitude remote sensing has been widely investigated on various crops,the performance of the predictions are not consistent,especially when applied in a breeding context with hundreds of genotypes.In some cases,biomass prediction of a large group of genotypes benefited from multimodal remote sensing data;while in other cases,the benefits were not obvious.In this study,we evaluated the performance of single and multimodal data (thermal,RGB,and multispectral) derived from an unmanned aerial vehicle(UAV) for biomass prediction for drought tolerance assessments within a context of bioenergy sorghum breeding.The biomass of 360 sorghum genotypes grown under well-watered and water-stressed regimes was predicted with a series of UAV-derived canopy features,including canopy structure,spectral reflectance,and thermal radiation features.Biomass predictions using canopy features derived from the multimodal data showed comparable performance with the best results obtained with the single modal data with coefficients of determination(R2)ranging from 0.40 to 0.53 under water-stressed environment and 0.11 to 0.35 under well-watered environment.The significance in biomass prediction was highest with multispectral followed by RGB and lowest with the thermal sensor.Finally,two well-recognized yieldbased drought tolerance indices were calculated from ground truth biomass data and UAV predicted biomass,respectively.Results showed that the geometric mean productivity index outperformed the yield stability index in terms of the potential for reliable predictions by the remotely sensed data.Collectively,this study demonstrated a promising strategy for the use of different UAV-based imaging sensors to quantify yield-based drought tolerance.

1.Introduction

Bioenergy sorghum is a photoperiod-sensitive sorghum that either flowers late or not at all in the northern hemisphere.Owing to its enhanced photoperiod sensitivity and prolonged vegetative growth duration,bioenergy sorghum has been found to produce more than twice the biomass of grain sorghum [1,2] and is therefore being developed as an important feedstock for bioenergy production.It grows in marginal areas,such as arid regions;hence,drought tolerance is one of the most important traits for the production of bioenergy sorghum.Although the prolonged vegetative growth stage is one factor that enhances its drought-tolerance ability [3],selecting the most drought tolerant sorghum genotypes is an important breeding goal to maximize biomass yields in marginal environments.

Over many decades,researchers have used direct or indirect selection criteria for drought tolerance.As a direct selection criterion,primary traits such as grain yield or biomass are of great interest to researchers.Based on primary traits,numerous drought tolerance indices have also been developed to evaluate the drought-adaptive performance of different plant genotypes.Some commonly used indices include geometric mean productivity(GMP),mean productivity(MP),harmonic mean(HM),stress tolerance index (STI),stress susceptibility index (SSI),tolerance index(TOL),and yield stability index(YSI)[4,5].These indices can be largely grouped into two categories.The first group includes GMP,MP,HM,and STI,which have been found to be more reliable for identifying genotypes with both high yield potential and stress tolerance potential[6,7].YSI and SSI fall into the second group that is useful for discriminating genotypes with higher stability across multiple environments[8,9].The indices within each group generally have strong correlations with each other[5,9].In this study,we were interested in the genotypes with both high biomass and yield stability.Therefore,we identified two drought tolerance indices,i.e.,GMP and YSI,as the drought tolerance indicators for selecting the best drought tolerant bioenergy sorghum genotypes.

In addition to the direct selection criterion,numerous secondary traits have also been evaluated and used as indirect selection criteria for drought tolerance[10].Secondary traits are related to primary traits but provide additional information on crop growth.Compared to primary traits that are usually measured at the end of the season,the secondary traits are easier and faster to measure during the season,are highly heritable,and may have good correlations with the primary traits [11,12].For sorghum,a suite of secondary traits has been evaluated in drought tolerance breeding,such as leaf rolling,leaf water potential,stay-green,stomatal conductance,plant height,and canopy temperature[13,14].Although these traits have shown to be useful in drought tolerance selection,phenotyping them in a high-throughput manner in the field remains very labor-intensive and slow.A promising solution is to replace the traditionally labor-intensive manual measurements with new advanced remote sensing techniques.

Remote sensing is capable of collecting data in a faster,nondestructive,and more cost-effective manner [15].Remote sensing data can be further used to estimate the above primary and secondary traits.For example,one of the most well-characterized drought adaptative traits in sorghum,stay-green [16],has traditionally been evaluated using visual scores based on the fraction of senesced leaf area[17]or the fraction of green leaves[18].Using remote sensing tools,it is possible to estimate stay-green nondestructively from spectral indices,such as the normalized difference red edge (NDRE) [19] and the normalized difference vegetation index (NDVI) [20].Additionally,another important drought adaptative trait,canopy temperature,has been well-documented in sorghum [21],wheat [22],potato [23],and other crops [24-26].It is usually measured by remote sensors,such as thermal infrared imagery and infrared thermometry.Early in 1989,researchers used handheld infrared thermometers to measure canopy temperature and reported that the drought-resistant sorghum genotypes had lower canopy temperatures than the drought susceptible ones[27].These successful applications of remote sensors in measuring drought responsive secondary traits indicate the possibility of applying such techniques to a drought tolerance breeding program.

Among various remote sensing platforms,the unmanned aerial vehicle (UAV) has gained increasing attention since it provides higher spatial resolution than satellite platforms and allows faster data collection than ground platforms.Moreover,with the miniaturization of optical cameras,a large number of successful applications of UAV systems have been reported in agricultural fields[28-31].Many canopy features can be derived from the sensors attached to UAVs that may be directly or indirectly related to drought responsive traits,such as canopy temperature,vegetation indices,and plant height.This has led to the use of the UAV system for evaluating the drought tolerance in a bioenergy sorghum association panel in this study.

Only a limited number of studies have applied UAV systems to drought tolerance screening [5,32-34].UAV thermal imaging was used to evaluate the drought responses of 503 poplar tree genotypes [5].One-fourth of the population was identified as candidates for drought-tolerant genotypes using a stress susceptibility index calculated with canopy temperature.In addition to UAVacquired thermal imaging,RGB images were also added [32,33]to assess the drought tolerance of 390 genotypes of forage grasses.In that study,the drought tolerance was rated by visual scoring of the amount of green biomass using the human eye.Multiple RGBbased indices and thermal-based indices correlated with the breeder scores.The results showed that the thermal indices had weaker correlations than the RGB indices with the visual scores.Given the intrinsic complexity of plant drought adaptation mechanisms[10],different genotypes may have a unique set of physiological and biochemical responses.Therefore,many studies have emphasized the importance of combining multiple selection criteria for selecting drought-tolerant genotypes[11,22].Using UAV technology,the utilization of data sets from multiple sensors such as thermal,RGB,and multispectral cameras makes this approach more feasible.Image features extracted from multiple UAV-based cameras were used together with a support vector machine classifier to distinguish between two groups of genotypes with differential wilting times with an average classification accuracy of 80% [34].While earlier results suggested that UAVs provide a rapid,objective,and labor-saving method for distinguishing drought tolerance within large populations of different plant genotypes,further studies are needed to quantitatively evaluate the potential of UAVbased multimodal images (thermal,RGB,and multispectral) for yield-based drought tolerance selection.

In our study,the quantitative evaluation was focused on the estimation of biomass since this was the primary trait for the selection of drought tolerant bioenergy sorghum genotypes.Although UAV-based remote sensing data have been used to estimate biomass for various crops,such as maize [35],wheat [36],pea [37],and sorghum [38,39],the estimation performance varies amongst studies on the same crop species.For example,Masjedi et al.[38]conducted experiments for sorghum biomass prediction using UAV-based RGB,hyperspectral and LiDAR sensors.The number of genotypes evaluated ranged from 4 to 840.Results showed that the R2between estimated and measured end-of-season biomass ranged from 0.64 to 0.89.In another study that targeted biomass estimation of ten sorghum varieties using vegetation indices derived from UAV-based remote sensing data,the maximum R2achieved was 0.91[39].Differences in the estimation performance among these studies may be attributed to variation in experimental design,types of sensors used,modeling approach,number of genotypes,sample size,and other factors.In contrast to other studies highlighted above,our study sought to assess the use of UAV-based single and multimodal data (thermal,RGB,and multispectral) to quantify biomass production and to evaluate drought tolerance in a large bioenergy sorghum association panel [40].A linear support vector regression model was first established to predict biomass using the canopy features extracted from UAV multimodal images.Thereafter,we assessed the reliability of using the UAV predicted biomass to derive the yield-based drought tolerance indices (i.e.,GMP and YSI).

2.Materials and methods

2.1.Field experiment and biomass sampling

The experiment was conducted at the Panhandle Research and Extension Center of the University of Nebraska-Lincoln located in Scottsbluff,NE,USA (Fig.1).The soil was relatively uniform throughout the field.Four randomized blocks were arranged in the field,including two well-watered (WW) replicates and two water-stressed (WS) replicates in 30 plots × 12 ranges grid.The size of each block was 75 m long and 52 m wide.A total of 360 sorghum genotypes were planted in each block,consisting of 357 accessions of the Bioenergy Association Panel (BAP) population[40],one Grassl [40],one BTx623 [41],and one PI_92264.The entries in each block were fully randomized.Individual plots were 4.5 m long and 1.5 m wide,consisting of two rows spaced 76 cm apart.Each plot was planted with 100 seeds,with 50 seeds per row.

On April 16,2020,112 kg ha-1of nitrogen urea was applied to the field and incorporated using tillage.Planting occurred on June 1,2020 for all 1440 plots using a precision vacuum planter.For the WW replicates,29.35 cm of supplemental irrigation was applied on a weekly basis.For the WS replicates,3.05 cm of supplemental irrigation was applied initially to establish the plants in the field.Fresh and dry biomass was harvested on September 2-4,using a tractor pulled a forage harvester with a weigh hopper.The entire plot was harvested and chopped.Whole plot weights were recorded by load cells below the weigh hopper to the nearest 0.23 kg.A subsample of the wet chopped-up plant material was taken from each plot,and its fresh biomass was obtained.This subsample was then put into a small mesh drawstring bag and placed in drying ovens until sample weights stopped decreasing.Dried plant material was taken out of the bag and weighed.Both fresh and dry biomass units were finally converted to kg m-2for each plot.

2.2.Statistical analysis for fresh and dry biomass

Two-way ANOVA(analysis of variance)was carried out to evaluate the effects of water treatment,genotype,and the interaction between genotype and treatment (G × T) on the observed fresh and dry biomass.The difference was considered to be statistically significant at α=0.001.From the ANOVA output,the expected mean of squares (EMS) can be estimated by ‘mean of squares’,which were further used to calculate genotypic varianceG×T variance,and residual variancefollowing Equation (1-3) [11].Thereafter,the broad-sense heritabilitywas estimated over combined treatments according to Equation(4)[5].

Where EMSG,EMSG×T,and EMSεare the mean of squares from ANOVA output for genotype,G×T,and residual error components,respectively.The number of replicates for a genotype within each treatment is r,and t is the total number of treatments.The total phenotypic variance is

Additionally,the impact of water deficit on biomass was evaluated by computing mean,percent reduction (PR) in WS as compared to WW (Equation (5)) [42],and coefficient of variation (CV,Equation(6)).Pearson correlation coefficient(r)was used to examine the correlations between biomass and UAV-derived individual canopy feature.

Where μwwand μwsare the mean value under well-water and water-stressed conditions.SD and μ are the standard deviation and mean of the target variable under well-water or waterstressed conditions.

2.3.UAV systems and flight missions

Two unmanned aerial systems were used for data collection,as depicted in Fig.1.The six-rotor UAV is the DJI Matrice 600 Pro(DJI,Shenzhen,Guangdong,China),equipped with the Zenmuse X5R RGB camera (DJI,Shenzhen,Guangdong,China) and the five-band RedEdge multispectral camera (MicaSense,Inc.Seattle,WA,USA).RGB and multispectral images were obtained using this system.Detailed specifications of the RGB and the multispectral cameras were provided in Table S1.The four-rotor aerial vehicle,DJI Matrice 210 RTK (DJI,Shenzhen,Guangdong,China),was used to collect thermal images by the carried Zenmuse XT2 camera (DJI,Shenzhen,Guangdong,China).This camera is a dual-sensor system: a radiometric thermal sensor made up of uncooled Vox microbolometers and a visible sensor.In this study,only the images from the thermal sensor were used.Each thermal pixel records temperature value in Celsius degrees.More specifications of this thermal sensor are listed in Table S1.

Two UAV data collections were conducted in 2020.The corresponding drone flight settings and weather conditions were listed in Table S2.The weather during the flights was mostly sunny with mild wind.All the images were collected within the period from 11:30 to 13:30 local time,except on August 5 when the thermal images were collected at 15:50 local time due to unexpected system malfunction.As suggested by Kelly et al.[43],the thermal camera was turned on and stabilized for about 30 mins before the flight.The purpose was to alleviate any temperature dependency effect,which is the effect of the thermal sensor’s temperature on the thermal responsive signal [44].

2.4.Image processing and canopy feature extraction

Pix4Dmapper (Pix4D,Lausanne,Switzerland) software was used to generate orthomosaic from raw aerial images.During the map generation,geometric correction was performed in Pix4Dmapper to correct geometric distortion and to ensure that maps from different sensors or different times were well-aligned.The correction utilized five ground control points,i.e.,checkered wooden boards,that were distributed evenly in the field.The geo-locations of the center of those boards were surveyed in the field with a Real-Time Kinematic GPS receiver(Topcon Positioning Systems,Inc.,Tokyo,Japan) with centimeter-level accuracy.The radiometric calibration for the multispectral images involved two sources of calibration references: (1) the MicaSense’s Calibrated Reflectance Panel provided by the manufacturer;(2) three standard reflectance tarps laid out in the field.Details on the multispectral radiometric calibration are described in Li et al.[45].The thermal calibration description can be found in Fig.S1.Specifically,three reference targets were laid out in the middle of the field,including a cold reference (a cooler containing water and ice mixture),a plant sample representing medium temperature,and a hot reference (the darkest radiometric correction tarp).The surface temperature of these targets was measured concurrently with the flights,using a handheld infrared radiometer (Apogee Instruments,Inc.,Logan,UT,USA).For each data collection,a linear calibration function was established between the measured reference temperatures of the targets and their corresponding pixel values averaged from the thermal map (Fig.S1).The linear calibration function was then used to calibrate the original thermal map.From the corrected orthomosaic(i.e.,thermal,RGB,and multispectral maps),six canopy features were extracted: normalized relative canopy temperature (NRCT),canopy cover,plant height,and three vegetation indices.

2.4.1.Normalized relative canopy temperature (NRCT) and canopy cover

The NRCT and canopy cover were extracted together,using the RGB co-registration method to remove soil pixels from the RGB and thermal map [46].The co-registration method makes use of the higher spatial resolution of the RGB map and could partially alleviate the effect of the mixed pixels in the thermal map with lower spatial resolution.The mixed pixel is the pixel that covers both vegetation and soil [47].The RGB co-registration workflow is shown in Fig.S2.First,the RGB map was converted to the HSV(i.e.,Hue,Saturation,Value) color space.Based on the literature[48] and manual adjustment,a pixel was considered to be plant pixel when its value in the Hue channel ranged from 20 to 120,in the Saturation channel ranged from 50 to 255,and in the Value channel ranged from 80 to 255.Using this approach,a binary plant mask was created,where one represents plant pixel and zero represents non-plant pixel.Second,the original thermal map was resampled to the same size as the plant mask using the nearest neighbor interpolation method.The resampling allows us to overlay the plant mask onto the thermal map to generate the segmented thermal map,where the non-plant pixels were set to Null value.

From the segmented thermal map,mean canopy temperature(°C)was derived from each plot.To normalize across different data collection events,the canopy temperature was converted to NRCT,according to Equation(7)[49,50].In other studies canopy-air temperature difference was used,but because of the speed with which the UAV can cover the entire field(approximately 10 mins),the air temperature did not change significantly so that it was not necessary to factor the air temperature into our data analysis.The canopy cover was also derived from the segmented thermal map,as the ratio between the number of plant pixels and the number of all pixels falling into each plot.

Where Tcanopyis the mean canopy temperature (°C) from the soil-removed thermal map;Tminand Tmaxrepresent the minimum and maximum temperature (°C) within all the pixels in the soilremoved thermal map.

2.4.2.Plant height

To obtain the plant height map,the digital surfaced model(DSM)was generated from the RGB images using the Pix4Dmapper software.The DSM represents the elevation above sea level of the surface,including the surface of natural or artificial features.Moreover,a digital elevation model (DEM) indicating the elevation above sea level of the bare soil was also required.In this study,the DEM was created by interpolating soil points sampled from the DSM map,using the Kriging interpolation tool in ArcMap 10.5.1(Esri Inc.,Redlands,CA,USA).The plant height map was then derived as the difference between the DSM and the DEM maps.Then,for each plot,the trimmed mean value was extracted to represent plot-level average plant height: the mean value of all the pixels falling into the range of 25th to 75th percentiles of plant height pixel values within one plot.

2.4.3.Vegetation indices

Finally,three vegetation indices were computed from the multispectral bands: NDVI,NDRE,and RDVI (i.e.,renormalized difference vegetation index).These indices have already been widely used and were reported to be efficient in estimating crop water stress [51].The formulae for these indices were given in Equation(8-10).

Where RNIR,RRED,RRedEdgeare reflectance value of the nearinfrared,red,and red edge spectral band,respectively.

2.5.Linear support vector regressor for biomass prediction

To estimate the biomass using the extracted canopy features,the support vector regressor with the linear kernel (Linear-SVR)was utilized.The support vector regressor is the regression equivalent of the support vector machine(SVM),a classical and popular supervised machine learning algorithm [52].Different nonlinear kernel functions have been integrated into SVM to solve nonlinear problems,such as Gaussian radial basis function,polynomial function,and sigmoid function[53].While in this study,the linear kernel was found to be the most efficient and was thus selected.The objective of the SVR model is to find an optimal hyperplane that fits the data.Besides,a flexible boundary is formed symmetrically around the hyperplane.Within this boundary,the estimation errors are tolerant during model training [54].The width of this boundary can be predefined or optimized by the analyst,referring to the hyperparameter ε.Conversely,the estimations outside this boundary are penalized by a regularization term C.In this study,both ε and C were optimized for each model.

Regression models were established separately for WS and WW conditions.Under each water regime,70% of the 360 genotypes were randomly selected as the training set (i.e.,252 genotypes),and 30% of the genotypes were used as the testing set (i.e.,108 genotypes).Given that each treatment was replicated twice,the resulting training sample size was 504,and the testing sample size was 216.The genotypes assigned to the testing set were consistent between two water regions.Model hyperparameters were optimized through a ten-fold cross-validation nested in a grid search process,using the training samples only.Then,the model with its optimized hyperparameters was further evaluated on the stand-alone testing set.Three evaluation criteria,i.e.,the coefficient of determination (R2) (Equation (11)) and the root mean squared error (RMSE) (Equation (12)),and relative RMSE (RRMSE,Equation (13)) were computed between the model predicted and the measured biomass.

Where yiandare the actual and predicted values for ith sample;is the mean value of n total number of samples.

2.6.Yield-based drought tolerance indices

Selection criteria for drought-tolerant genotypes vary from case to case.In this study,we selected GMP and YSI as drought tolerance indices,computed by equations(14)[6]and(15)[55],respectively.These formulae were originally established based on grain yield production.In this study,however,as biomass production is the primary trait of bioenergy sorghum,grain yield terms were replaced with biomass.Accordingly,we renamed these two indices as ‘biomass-GMP’ and ‘biomass-YSI’,respectively.We understand that the final decision on whether a genotype is drought tolerant or not depends on individual breeders and breeding purposes in each breeding program.In the present study,we considered the top-ten genotypes as drought tolerant ones by ranking the genotypes based on biomass-GMP and biomass-YSI indices.

In addition,to evaluate the biomass predicted from UAVderived canopy features,biomass-GMP and biomass-YSI computed using the predicted biomass(i.e.,all the canopy features were used as predictors) were compared with those computed using ground truth biomass.

Where BiomassWWis the biomass under well-watered conditions;BiomassWSis the biomass under water-stressed condition.

3.Results

3.1.Statistical analysis for fresh and dry biomass

The biomass was statistically different (α=0.001) due to treatment,genotype,but not the interaction between treatment and genotype (Table 1).The effects of water deficit on fresh and dry biomass were further compared using mean values and CV within each water regime and the percent reduction (PR) under WS condition relative to WW condition(Table 1).In general,biomass production was reduced under the water-stressed environment.The reduction in the fresh biomass (32.99% PR) was larger than that of the dry biomass (10.81% PR),which probably indicates that the water deficit had a significant impact on plant water content.Using the output of ANOVA,we computed the variance components and the broad-sense heritability of fresh and dry biomass(Table 1).Results showed that the heritability of fresh biomass was higher than that of dry biomass.

Table 1 Mean value,coefficient of variation (CV),percent reduction (PR),effect of source of variable in ANOVA,and variance component analysis for fresh and dry biomass.

3.2.Correlation between individual UAV derived canopy feature and biomass

The correlations (i.e.,correlation coefficient r) between UAVderived canopy features and biomass under each water regime were explored (Fig.2).In general,the correlations under WS conditions were stronger than those under WW conditions,regardless of fresh or dry biomass.For the three vegetation indices,the correlations on August 5 were all stronger than those on August 28,regardless of the water regimes or biomass components.Among all canopy features derived from two dates of UAV data collections,RDVI under WS conditions on August 5 had the strongest correlation with fresh biomass(r=0.70)and with dry biomass(r=0.69).On the other hand,the plant height under WW conditions on August 5 and NRCT under WW conditions on August 28 showed nonsignificant correlations with fresh biomass (α=0.001).For dry biomass,NRCT under WW conditions on both August 5 and 28,as well as the canopy cover from the WS block on August 5,showed nonsignificant correlations.

3.3.Biomass predicted by combined canopy features

To predict fresh and dry biomass from UAV-based canopy features,the linear-SVR model was trained with 70%of the 360 genotypes and was tested on the remaining 30% genotypes under two water regimes.Prediction results were given in Fig.3 for fresh biomass and in Fig.4 for dry biomass.According to the dates of data collection,three sets of input variables were compared: (1) combined six canopy features(NRCT,plant height,canopy cover,NDVI,RDVI,and NDRE)from August 5(Fig.3A,D,and 4A,D);(2)combined six canopy features from August 28 (Fig.3B,E,and 4B,E);(3) integrating (1) and (2) (Fig.3C,F,and 4C,F).Overall,the model performed better under WS (R2=0.47-0.53 for fresh biomass in Fig.3D-F;R2=0.40-0.47 for dry biomass in Fig.4D-F) than WW environment (R2=0.28-0.35 for fresh biomass Fig.3A-C;R2=0.11-0.29 for dry biomass in Fig.4A-C).Among the three sets of inputs,the one that combined the canopy features from two UAV data collections improved the estimation of the fresh biomass but had no significant improvement for dry biomass prediction.

To further compare the contribution of different sources of aerial images,we grouped these canopy features based on their source image.The source images were: (1) thermal;(2) RGB;(3) multispectral images.The corresponding canopy features were: (1)NRCT from 5 and 28 August;(2) plant height and canopy cover from two dates;(3) NDVI,RDVI,and NDRE from two dates.The model was trained on each group of features and compared with the model that used the canopy features from all three image sources.As shown in Fig.5,except for the dry biomass prediction under WW condition,the multispectral images had the lowest RRMSE and highest R2values among the three images sources.On the other hand,the thermal images mostly performed the worst among the three image sources.Nevertheless,the model trained with all canopy features performed the best for estimating the fresh and dry biomass,which suggests that utilizing multimodal data for yield-based drought tolerance evaluation is optimal.

Fig.2.Correlations between UAV-derived canopy features and biomass.(A)Fresh biomass.(B)Dry biomass.Canopy features were derived from August 5,2020 and August 28,2020.Blue circles represent well-watered conditions;orange circles represent water-stressed conditions.*indicates correlation is statistically significant given α=0.001.NRCT,normalized relative canopy temperature;PH,plant height;CC,canopy cover;NDVI,NDRE,and RDVI are three vegetation indices.

Fig.3.Predicted versus measured fresh biomass from the testing dataset(n=216)under(A-C)well-watered condition and(D-F)water-stressed condition.Canopy features input into each model were derived from (A,D) August 5,2020,(B,E) August 28,2020,and (C,F) August 5 and 28,2020.

Fig.4.Predicted versus measured dry biomass from the testing dataset (n=216) under(A-C) well-watered condition and (D-F) water-stressed condition.Canopy features input into each model were derived from (A,D) August 5,2020,(B,E) August 28,2020,and (C,F) August 5 and 28,2020.

Fig.5.Performance of different image sources(i.e.,thermal,RGB,and multispectral)for(A,C)fresh biomass and(B,D)dry biomass prediction.R2 and RRMSE values are based on the testing dataset (n=216).Multimodal indicates the combination of all three types of images.

3.4.Yield-based drought tolerance indices

In this section,we extended the biomass prediction to the drought tolerance assessment using two well-recognized yieldbased drought tolerance indices,i.e.,the biomass-GMP and the biomass-YSI indices.As shown in Fig.6,biomass-GMP and biomass-YSI were computed using predicted biomass and compared with those computed using manually measured biomass.A significantly better correlation was found for the biomass-GMP index than that for the biomass-YSI index.

In Fig.7,we highlighted the top ten genotypes by ranking the biomass-GMP and biomass-YSI calculated from both measured and predicted biomass.An assumption we made here was that the higher biomass-GMP or biomass-YSI value corresponds to the better drought tolerance.Results showed that using the biomass-GMP index,four out of the top ten genotypes were selected by the index derived by predicted biomass and the one derived by manually measured biomass were in common (Fig.7A,B);while two or three out of ten genotypes were in common between biomass-YSI indices calculated from predicted and manually measured biomass(Fig.7C,D).These findings(Figs.6,7)indicated that the biomass-GMP drought tolerance index might be the one that can be estimated more reliably from remotely sense data.

4.Discussion

4.1.Capability and limitation of UAV-based remote sensing data for biomass prediction and drought tolerance assessment of bioenergy sorghum

In this study,we investigated the potential of UAV-based multimodal remote sensing data (including thermal,RGB,and multispectral data) for fresh and dry biomass predictions and drought tolerance evaluation over a large group of bioenergy sorghum genotypes and systematically compared the results with those obtained from single data sources.Multimodal data has been used previously for yield predictions for other crops such as soybean[56] and maize [57],but to the best of our knowledge,this study is the first for a large number of sorghum genotypes.In our study,multimodal data did not result in noticeably better predictions than the single-source data for both fresh and dry biomass.Results showed that the multimodal data had comparable or slightly better performance than the multispectral data in this study (Fig.5).Yet,in the study on soybean yield prediction [56],the multimodal data had consistently superior prediction performance than single modal data.The better performance obtained using multimodal data might be due to the fewer soybean varieties (i.e.,three varieties),or different algorithms employed,or the addition of texture features to spectral,structure,and thermal features.We noticed that the correlation between plant height and sorghum fresh biomass was 0.41 in the current study,which is not as high as in a similar study with only 24 bioenergy sorghum genotypes(r=0.79) [58].A major challenge in our study was the large amount of genotypic diversity(360 genotypes).On the other hand,we published a similar finding regarding the performance ranking of three UAV-based sensors in the soybean study[56].Namely,the multispectral sensor contributed the most to the estimation of sorghum biomass.RGB was also important,but its performance in modeling was inferior to multispectral images in most cases.The thermal sensor had the worst performance.Nevertheless,with relatively better performance for fresh biomass prediction identified using multiple image sources (R2=0.53 and RRMSE=23.57%),and given the inconsistency among different studies,collecting multimodal data to start with the investigation may still be a more reliable way to ensure better biomass prediction accuracy for yield-based drought tolerance assessment in breeding studies.

Fig.6.Drought tolerance indices calculated from predicted versus measured FB (fresh biomass) and DB (dry biomass).(A,B) Biomass-GMP.(C,D) Biomass-YSI.

Fig.7.Top ten genotypes selected by (A,B) biomass-GMP and (C,D) biomass-YSI calculated from measured versus predicted biomass.The x-axis label is the pseudo-ID number for each genotype.The length of the vertical color bar represents the magnitude of biomass-GMP or biomass-YSI value.FB is fresh biomass;DB is dry biomass.

In the applications that a drought tolerance index is preferred,this study demonstrated the possibility of predicting the biomass-GMP index from the remotely sensed data (Fig.6).The UAV predicted fresh and dry biomass was used to compute two drought tolerance indices,namely,biomass-GMP and biomass-YSI.The results (Figs.6,7) suggested that the UAV predicted biomass was more reliable when using biomass-GMP as an index of drought tolerance.The capability to obtain a rapid and reliable estimation of biomass-GMP from the UAV multimodal data is encouraging because biomass-GMP is one of the most widely used indexes to identify elite genotypes in drought tolerance screening programs[7-9,59-61].On the other hand,the biomass predicted in the current study appeared to be less reliable for derivation of biomass-YSI.Future UAV-based remote sensing studies will be needed to improve the biomass prediction for better estimation of biomass-YSI,but the multimodal approach used in this study shows great promise for the estimation of biomass-GMP.

Admittedly,the UAV-based remote sensing data applied in this study also had certain limitations.As the results shown in Figs.3,4,the overall performance under WS condition was better than that under WW condition.This coincides with the generally stronger correlations between canopy features and biomass under the WS conditions than those under the WW conditions(Fig.2).One factor that possibly impacts the UAV data’s performance under WW environment is the saturation of some canopy features,such as the NDVI.The canopy under WW becomes dense quicker than that under WS.However,the broadband reflectance that was sensed in this study mainly comes from the top canopy layer.As the biomass kept accumulating under water sufficient environment,some of the remotely sensed canopy features would reach a plateau.By looking at the distribution of each canopy feature (Fig.S3),values of NDVI under WW condition had less variation and almost reached one.The genotypic variation in NDVI was not enough to explain the genotypic variation in the biomass.Previous studies have suggested that using vegetation indices based on the red edge band,such as NDRE,is better than NDVI in terms of saturation issue [62].In this study,however,the correlations between NDRE and fresh/dry biomass were not as good as those between NDVI and fresh/dry biomass,even though NDRE showed less saturation(Fig.2).This may be due to the complication of unknown factors related to genotypic differences.

4.2.Perspectives on the use of UAV-equipped thermal camera in yieldbased drought tolerance screening

The utility of UAV-equipped miniaturized thermal cameras in agriculture has been demonstrated in a number of studies that have estimated plant water status to detect water deficit [28-31,46,50,63,64].Most of these studies used thermal data to quantify instantaneous plant water status,usually with a single or a few genotypes.Based on these studies,the use of thermal indices appears to be useful for measuring short-term responses,such as stomatal conductance and transpiration.For example,the crown temperature within a citrus orchard was found to be well correlated with manually measured stomatal conductance (r=0.88)and plant water potential (r=0.58) [30].In a study of cotton[63],the crop water stress index (CWSI) was derived from a UAV thermal map and correlated well with stomatal conductance(r=0.81).However,the good correlations between canopy temperature and instantaneous traits may not be replicable in a drought tolerance screening program that targets final yield or biomass.In a study assessing genetic variation of 120 recombinant inbred lines of Setaria grass under drought stress,correlations ranging from-0.32 to-0.49 were observed between canopy temperature and total biomass [65].In our study,negative but relatively weak correlations were found between canopy temperature and biomass,with absolute r value ranged from 0.09 to 0.44(Fig.2).Moreover,the comparative results among the three image sources from this study show that the thermal images performed the worst for modeling biomass.

Under drought,the most common early response of plants is stomatal closure [66].Stomatal closure also may vary according to a diurnal cycle with a midday reduction in transpiration occurring due to elevated temperatures[67].This may partly explain the lower correlation between canopy temperature and biomass for the drought stressed genotypes as compared to the stronger correlations with NDVI,NDRE,and RDVI.However,canopy temperature is not solely regulated by stomatal conductance and may not be perfectly linked to transpiration rates in sorghum under drought[67].Canopy temperature is also affected by air temperature,solar radiation,wind speed,etc.[68].In addition,drought responses are not only crop species specific but also vary amongst genotypes of a single species [69].Although midday reductions in stomatal conductance occur and variation in genotype responses were expected,even with a large number of sorghum genotypes there was a significantly negative correlation between canopy temperature and biomass under drought conditions(Fig.2).An earlier study evaluated 300 sorghum genotypes under irrigated conditions[21] and showed no strong correlation between the leaf temperature and grain yield,which is similar to our results under well-watered conditions.The earlier study [21] also validated the simulation models and the hypothesis [67] that higher canopy temperatures in sorghum may not be tightly linked to lower yields due to stomatal closure that leads to conservation of water resulting in higher biomass.Therefore,the significant negative correlations observed in our dataset between canopy temperature and biomass may have been less than that observed for the spectral indices because some high biomass genotypes conserved water by stomatal closure as hypothesized [67] and shown previously[21].In this regard,we suggest that the use of thermal sensors alone may not be sufficient to identify the most drought tolerant sorghum lines in a screening program.Additional sources of phenotypic information are needed to fully elucidate the complexities in genotypic responses for drought tolerance in sorghum.

From the sensing perspective,another possible factor that limits the performance of the thermal index(i.e.,NRCT)in estimating sorghum biomass is the sensitivity and accuracy of the thermal camera.In our study,the thermal camera we used was an uncooled camera,which is suitable for UAV applications due to its compact size and lightweight.However,compared to the cooled thermal cameras that use cooling systems to reduce measurement noise,the uncooled type is more likely to be affected by the temperature of the camera body,as well as the environmental conditions such as air temperature,relative air humidity,wind speed,cloud cover,etc.[70,71].Furthermore,according to the manufacturer,the estimated temperature accuracy of the thermal camera used in this study was around ± 5 °C under ideal environmental conditions.This accuracy may not be sufficient in a screening or breeding program that uses small plot designs,where the required accuracy may be greater than 1 °C [72].The measurement sensitivity to weather conditions,along with the relatively low sensing accuracy,may have reduced the performance of the thermal sensor used in this study.Future studies will be required to investigate whether the major limitation is because of different genotypic response rates to canopy temperature and biomass production or because of the measurement accuracy of this thermal camera.For the latter question,a thermal camera with higher measurement accuracy,such as ICI 8640P thermal camera that has an accuracy of ± 1 °C[72],maybe an alternative for the UAV system.

4.3.Future improvements

In our study,images including six spectral bands were used for sorghum biomass prediction and drought tolerance assessment:blue,green,red,red edge,near-infrared,and thermal infrared.The combination of this information provided acceptable performance for evaluating drought tolerance which was benchmarked with biomass accumulation.However,given the complexity in genotypic responses under drought stress,future work is required to facilitate the UAV-based remote sensing in drought tolerance selection.First,additional improvement will likely be gained if more sources of data could be included,such as the information from the hyperspectral imagery that has hundreds of continuous spectral bands.With the development of sensing technologies,UAV systems can now carry miniaturized spectrometers [73] that have been used for yield or biomass estimation [74,75],crop disease detection [76],and water stress detection [30].In the future,UAV-based point spectrometers or hyperspectral cameras will likely improve our ability to more precisely evaluate the drought tolerance of a large population of plant genotypes.Another future improvement is the collection of time-series data.In the present study,the canopy temperature index was measured at only two time points and thus provided only a snapshot in time.Therefore,it does not provide an integrated canopy assessment that reveals a long-term response to water deficit (i.e.,biomass) [33].Instead of single or a few snapshots of crop traits,multitemporal crop traits may be more beneficial for predicting biomass which is an accumulated product over the growth season.In addition,intensive data collection over the course of crop growth allows the researcher to pinpoint the critical growth stages for final biomass estimation,which will be helpful when frequent data collection is not practical.

5.Conclusions

This study examined the use of UAV-based multimodal images(i.e.,thermal,RGB,and multispectral) for estimating biomass and assessing drought tolerance of bioenergy sorghum.Biomass predictions using canopy features derived from the multimodal data showed comparable performance with the best results obtained from the single modal data with R2up to 0.57 under the WS condition where the canopy was less dense.Vegetation index RDVI derived from the multispectral data had the highest correlations with biomass and had a major contribution in the predictions when multimodal data were used.RGB contributed less than the multispectral images but was also important.Thermal images alone had the lowest prediction power,indicating the limitation of using this thermal camera alone for this particular drought tolerance screening project.These findings were similar to some previous studies but different than some others;hence,it indicated the advantage of starting with multimodal data under unknown conditions.Finally,for drought tolerance assessments using yield-based indices,it turned out that the biomass-GMP index outperformed the biomass-YSI index in terms of the potential to be directly and more reliably predicted by the remotely sensed data.Follow-up improvement in biomass prediction within a breeding context based on low-altitude remote sensing is promising,by enriching the available data sources through the use of timeseries data,hyperspectral imagery,or by incorporating a higher accuracy thermal camera.Ultimately,UAV systems will be an essential tool for drought tolerance screening programs.In this case,instead of destructively sampling biomass or yield production from all experimental plots,subsampling can be done together with UAV mapping to greatly reduce cost and decrease time commitment.

Data availability

The UAV image data has been published on Dryad and is available at https://datadryad.org/stash/share/fxAffx2rXRHVdIyGQR 9RUV2GMq0epPMuW6wqHtxsPqc.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

CRediT authorship contribution statement

Jiating Li:Conceptualization,Data curation,Formal analysis,Investigation,Methodology,Software,Visualization,Writing -original draft,Writing -review &editing.Daniel P.Schachtman:Conceptualization,Funding acquisition,Data curation,Investigation,Methodology,Project administration,Resources,Writing -review &editing.Cody F.Creech:Resources,Data curation,Writing -review &editing.Lin Wang:Investigation,Writing -review&editing.Yufeng Ge:Methodology,Writing -review &editing.Yeyin Shi:Conceptualization,Funding acquisition,Methodology,Project administration,Resources,Supervision,Writing -review&editing.

Acknowledgments

This research was funded by US Department of Energy,BER(DE-SC0014395 to DPS),a USDA-NIFA Grant (2021-67021-34417),and the Nebraska Agricultural Experiment Station through the Hatch Act Capacity Funding Program(1011130)from the USDA National Institute of Food and Agriculture.We would like to thank Allyn Pella for her help in coordinating the data,thank Stephanie Futrell for help in ground truth data collection and lab analysis,and thank Dr.Xin Qiao and his team for their help during UAV data collections.

Appendix A.Supplementary data

Supplementary data for this article can be found online at https://doi.org/10.1016/j.cj.2022.04.005.