Materials and methods

Estimations of crop parameters

Fruit number

Fruit detections obtained from the mobile scans of all the measurement plots in the stone fruit and Sundial experimental orchards were reprocessed using a calibration factor derived from predictions against ground truth values. This calibration factor was equivalent to the slope of a linear regression with intercept = 0 obtained by the mobile platform counts (predicted) in experimental plots against total fruit counts per plot obtained with a commercial fruit grader equipped with optical sensors (InVision 9000; Compac Sorting Equipment Ltd, Colliver, Victoria, Australia) at harvest (ground truth). Fruit number was obtained on different crop × training system combinations: ‘Golden May’ apricot and ‘Angeleno’ plum under Tatura trellis and vase training systems; ‘August Flame’ peach under Tatura trellis and vertical leader training systems; and ‘Majestic Pearl’ nectarine under Tatura trellis, vertical leader, cantilever trellis 1, and cantilever trellis 2 training systems. Calibrated spatial maps were used to portray fruit densities (e.g., fruit per tree, fruit per linear meter) throughout the orchard.

FD and color

Ground truth FD and color for validations of stationary mobile platform predictions were obtained on fruit from different crops (apricot, peach, nectarine, plum), training systems (Tatura trellis, vertical leader, vase, cantilever trellis 1, and cantilever trellis 2), row spacing (4.5 and 3.5 m), and canopy height [position of the fruit within the canopy: low (< 1.2 m), medium (1.2 to 2.0 m), and high (> 2.0 m)]. The stone fruit cultivars scanned at the Tatura SmartFarm were Majestic Pearl and Autumn Bright nectarine, Golden May apricot, Snow Flame 25 and August Flame peach, and Angeleno plum. Both FD and color were measured at different times (1 month before harvest and at harvest) during the season to increase the variability of the measures in the collected data. Data from different collection times were then pooled together and ground truth FD and color data were compared with stationary predictions. The mobile platform predictions were generated based on the information within the fruit bounding boxes generated by the mobile platform fruit detector (Fig. 1A).

View Figure

• Download as Powerpoint
• Download Figure

Ground truth FDs were measured perpendicular to the main axis of the fruit with a digital Bluetooth caliper (OriginCal; iGAGING, San Clemente, CA, USA) (Fig. 1B). Estimates of FD were obtained from stationary and mobile scans at different stages (1 month before harvest and at harvest) throughout the season. Two independent reliability tests were carried out to determine the reliability of FD estimates obtained with 1) stationary and 2) mobile scans, due to the potential effect of the mobile platform motion on shape distortion of detected objects in images. Estimating size involved two main steps. First, fruit was detected by the same neural network deployed for fruit counts. Second, LiDAR and vision data were combined to estimate the metric width of the bounding box in meters. These two steps were intentionally separated to verify the accuracy of the approach in each of them. In the first phase, the ability to measure the metric width of fruit in the images was verified based on the assumption that fruit had perfect bounding boxes placed around them (Fig. 1A). Every image obtained from stationary scans was hand labeled for fruit with pixel precision. The labeling and the choice of which fruit to label were carried out manually rather than using software. A first reliability test was then carried out to determine the accuracy of FD predictions on stationary scans. Subsequently, a proprietary Green Atlas algorithm was used to automatically select a subset containing the most accurate FD estimates to use in mobile scans. This algorithm reduced the number of fruit on which FD was measured compared with the total detected fruit. However, in common orchard scans, thousands of fruit are measured, and it is not necessary to measure the size of every single fruit to obtain an accurate estimate of FD for the orchard- or row-subset. Mobile scan data were used to validate FD estimation. Special purpose computer vision technique was used to combine camera image and LiDAR data to improve the accuracy of FD estimation.

Validation of FD predictions in peach, nectarine, and plum commercial orchards obtained from mobile scans was carried out by comparing 1) the average FD of the detected fruit in three row sections per crop (nectarine row section length = 25 m; peach row section length = 10 m; plum row section length = 12 m) against 2) the average FD in sub-samples (n = 60 to 200) measured in the same row sections.

Ground truth fruit color was measured using a portable colorimeter (Instrument & Data Tools Pty Ltd, Rowville, VIC, Australia) (Fig. 1C). The colorimeter generated different color attributes in the CIELab/LCh color wheel (i.e., L*, a*, b*, C*, and h°), as previously shown by Scalisi et al. (2021a). Color was predicted with the mobile platform within the same bounding boxes used for fruit number and diameter predictions. Fruit color data collected by the mobile platform was extracted to produce RGB values for each data point, representing the average RGB values of fruit in the image. Afterward, RGB values were converted to CIELab/LCh using a Python script (Islam 2021). CIELab/LCh predictions were first generated on images logged with stationary mobile platform for training purposes. In addition to CIELab/LCh color attributes, to represent fruit skin color for stone fruit and capture the transition from greener to redder fruit, a color index developed in a previous study (Scalisi et al. 2022) that accounted for the distance from pure green (0) to pure red (1) was used. This parameter, called color development index (CDI), reflects the departure from skin greenness whether using the bottom half or the top half of the wheel. The CDI (a diagram of relationship with hue as shown in Fig. 2) is easier to interpret for users without prior knowledge of the CIELab/LCh color space. It provides greater flexibility for use in a broad range of crops or cultivars, as it can be calculated from color attributes that best describe the quality and maturity of that specific crop. In this study, CDI was derived exclusively from values of hue angle, as shown in Eq. [1]. $\text{CDI = |180 - h°/180|}$[1]

View Figure

• Download as Powerpoint
• Download Figure

Spatial maps of CDI were generated from the mobile platform scans in the commercial orchards planted with ‘Glacier’ peach (7 Jan 2021), ‘Summer Flare 34’ nectarine (28 Jan 2021), and ‘October Sun’ plum [26 Jan and 25 Feb 2021 (i.e., 1 month before and at harvest, respectively)] at different times of the day.

Results

Reliability of fruit number, FD, and fruit color predictions

Fruit number

The relationships of detected against ground truth fruit number in apricot, plum, peach, and nectarine are shown in Fig. 3. Distinct clusters were observed for different training systems with clearly separate linear relationships within each cluster. The only exception occurred in plum, where the linear fits for Tatura trellis and vase had similar slopes (Fig. 3B), although Tatura trellis trees bore more fruit than vase trees. Linear model statistics and calibration factors for each crop and training system are reported in Table 1. Tatura trellis and vase systems both had a good linear association in apricot (Fig. 3A) and peach (Fig. 3C), but the slopes of the relationships were visibly different between training systems. Different calibration factors were used for each training system, a standard practice advised by Green Atlas to improve the performance of the prediction. Prediction of fruit number in plum was similar between Tatura trellis and vase trees (i.e., similar calibration factor), although percent standard error (SE) was slightly higher in the latter. Overall, the percent SE was always ≤ 10%. In apricot, plum, and peach, predictions in Tatura trellis systems had lower errors compared with vase (apricot and plum) and vertical leader (peach) (Table 1). Highest errors were observed in nectarine systems, due to a smaller sample size (number of plots, n) compared with apricot, plum, and peach (Table 1). The highest RMSE was 14 fruit/tree and was obtained in experimental plots of Tatura trellis plum that had a high average fruit number per tree (313) and a percent SE of only ∼2.5%.

View Figure

• Download as Powerpoint
• Download Figure

Table 1.Calibration factor, root mean square error (RMSE), and percent standard errors (SE) of the estimates of fruit number detected with the mobile platform in apricot, plum, peach, and nectarine cultivars.

View Fullscreen

Fruit diameter

Estimations of FD with the stationary mobile platform were tightly associated with ground truth measurements, regardless of row spacing (Fig. 4A), training system (Fig. 4B), canopy height (Fig. 4C), and crop (Fig. 4D). These factors, although thought to influence the prediction models, did not form distinct clusters in the linear association. Therefore, data of fruit from all the crops, training systems, row spacings, and canopy heights were pooled together (Fig. 4E). The best linear fit was very close to the y = x line and the prediction model had an r_c > 0.90 and an RMSE < 5.0 mm.

View Figure

• Download as Powerpoint
• Download Figure

The validation of the mobile platform predictions of FD from mobile scans in commercial orchards generated a similar confidence to the reliability of predictions from stationary scans (Fig. 5) The best fit line was very close to the line of perfect fit (y = x), accuracy (r_c) was also more than 90%, and the RMSE was <2.0 mm.

View Figure

• Download as Powerpoint
• Download Figure

Fruit color

Predictions of CIELab/LCh color parameters measured by the mobile platform had different associations with ground truth measurements obtained with the portable colorimeter (Fig. 6). Linear models and statistics of predictions vs. observed CIELab/LCh color parameters are reported in Table 2. In peach and nectarine fruit, that is, fruit in which h° is a well-documented quality parameter in literature (Ferrer et al. 2005; Robertson et al. 1990; Scalisi et al. 2020), the r_c of the h° prediction models was 0.858 and 0.848, respectively, and the estimation error was below 10% (Table 2). In apricot and plum, despite h° being the better predicted color attribute, the reliability of the measure was much poorer (r_c < 0.40) than in peach and nectarine, with apricot predictions showing large noise and plum’s highlighting distinct clusters of data in the linear association (Fig. 6). Relationships between other CIELab color attributes (i.e., L*, a*, b*, and C*) measured by the mobile platform and by the portable colorimeter were poor (r_c < 0.40) in all the stone fruit crops (Table 2).

View Figure

• Download as Powerpoint
• Download Figure

Table 2.Linear regression models of predicted fruit color attributes for peach, nectarine, apricot, and plum shown in Fig. 6. Linear models, probability (P), coefficient of determination (R²), Lin’s concordance correlation coefficient (r_c), root mean square error (RMSE), and percent standard error (SE) of color attributes in stone fruit cultivars. Sample number reported in brackets under cultivar names.

View Fullscreen

Relationships between predicted and ground truth fruit CDI for ‘Snow Flame’ peach, ‘Majestic Pearl’ nectarine, ‘Golden May’ apricot, and ‘Angeleno’ plum are presented in Fig. 7A–D, respectively. The CDI prediction accuracies in peach, nectarine, and apricot were similar to the ones observed for h°. The use of CDI corrected the discontinuity that created distinct point clusters in predictions of h° in plum (Fig. 6D). Nevertheless, the relationship between predicted and observed CDI did not improve the relationship obtained with h° (Fig. 6D).

View Figure

• Download as Powerpoint
• Download Figure

Spatial mapping of validated fruit number, FD, and fruit color

  Density maps representing the spatial distribution of calibrated fruit number per tree and FD within the commercial orchard of ‘October Sun’ plum are shown in Fig. 8A and B, respectively. Summary statistics obtained at harvest in the ‘Glacier’ peach, ‘Summer Flare 34’ nectarine, and ‘October Sun’ plum commercial orchards are reported in Table 3. The prediction of plum fruit number had the smallest error, but overall, errors below 8% were obtained. Within-orchard variability of fruit number per tree was higher in plum (CV = 47%) than in other crops (Table 3), and was affected by the considerably higher fruit number in the south end of the orchard block (Fig. 8A). Both FD and CDI were also slightly more variable in plum than in nectarine and peach, although the overall spatial variability was low (CV ≤ 12% and ≤ 7% for FD and CDI, respectively). The mobile platform predictions indicated that ‘Summer Flare 34’ nectarine had similar FD metrics to ‘Glacier’ peach (Table 3).

View Figure

• Download as Powerpoint
• Download Figure

Table 3.Summary block statistics obtained in Varapodio S T & R (Ardmona, Victoria, Australia) commercial orchards at harvest. Standard errors (SE) of calibration factors are shown in brackets and block coefficients of variation (CV) are reported for fruit number, fruit diameter (FD), and color development index (CDI).

View Fullscreen

Spatial maps of fruit color were generated using the average CDI for each spatial point measured by the mobile platform. Figure 9A and B show CDI spatial color maps at harvest in the peach and nectarine commercial orchards, respectively. Visually, a small variation of color across the orchard was noticeable in ‘Glacier’ peach and ‘Summer Flare 34’ nectarine, in line with CVs reported in Table 3. Figure 9C and D show CDI in ‘October Sun’ plum 1 month before and at harvest, respectively. Figure 9C unveiled the position of pollenizers—a different cultivar that yielded fruit maturing earlier than ‘October Sun’ plum—with dark-purple points, whereas ‘October Sun’ fruit was still greener than pollenizers. At harvest, fruit reached higher CDIs across the block and differences in fruit color between main cultivar and pollenizers were almost canceled. Also, the temporal distinctive coloration can be clearly differentiated from Fig. 9C and D for ‘October Sun’ plum, where plum scanned 1 month before harvest appeared much greener (Fig. 9C) than they did at harvest (Fig. 9D).

View Figure

• Download as Powerpoint
• Download Figure

Discussion

The mobile platform fruit detector was tested for its capability to predict fruit number, FD, and fruit color in peach, nectarine, plum, and apricot. Predictions of fruit number per tree were possible by calibrating detections of fruit number per image. To obtain absolute values of fruit number per tree, it is important to apply customized calibration factors for each tree training system and overall orchard conditions (i.e., the standard commercial practice for Green Atlas). Without calibration, the mobile platform currently generates relative fruit number densities that can be spatially mapped to support orchard management decisions at a block level.

The percentage of fruit number prediction error in the models was steadily between 2% and 6% in apricot and plum trained to Tatura trellis or vase and on peach trained to vertical leader or Tatura trellis (Table 1). The higher errors observed in nectarine were due to the smaller sample size but remained <10.2%. Fruit number calibration factors varied under different crops and training systems (Table 1). Overall, there was a tendency to lower prediction errors in tree systems with canopies trained on trellises, where fruit is typically found in configurations more comparable to 2D walls. This observation is in line with Bortolotti et al. (2021), who found that fruit detections are improved in 2D systems. In our study, predictions of fruit number in Tatura trellis systems had lower errors than in other training systems in all the crops (Table 1). Some of the reasons behind the occurrence of false negatives (i.e., fruit not detected) and false positives (i.e., single fruit detected as doubles, or non-fruit objects identified as fruit) were as follows: 1) higher leaf area density that caused high fruit occlusion, 2) smaller fruit size relative to leaf area, 3) the three-dimensionality of canopy architecture, 4) dark fruit skin color that makes fruit appearance similar to shadows and dark areas in the images, and 5) fruit detections from adjacent rows. To the best of our knowledge, literature on orchard fruit detections in peach, nectarine, plum, or apricot is limited (Kurtulmus et al. 2014; Saedi and Khosravi 2020).

FD estimations were very accurate (r_c > 0.90) in images collected with stationary (Fig. 4) and mobile state of the platform (Fig. 5). Crop, row spacing, training system, or canopy height had no significant effects on the estimation of fruit size (Fig. 4). These results suggest that no calibration is necessary for accurate estimations of fruit size. Data of fruit size at harvest from the commercial orchard blocks were not available and orchard average FD (and total fruit number) predicted by the mobile platform could not be compared with packhouse results to further validate the prediction method. However, the validations carried out in this study unveiled fruit size estimation errors <5 mm, in line with previous errors reported in mango by Wang et al. (2017).

After fruit number calibration, it is possible to estimate yield (e.g., tonnes per hectare, kilograms per tree) by multiplying the total orchard block fruit number by the average FW derived from FD relationships, because final fruit fresh weight is strongly related to FD. Scalisi et al. (2019) observed a robust power relationship between FW and FD in ‘September Bright’ nectarine fruit at different fruit growth stages (FW = 0.0018 × FD^2.709; R² = 0.997, RMSE = 2.2 g, n = 150). Crop- or cultivar-specific FW to FD relationships need to be established for best yield predictions. Data obtained in a packing shed from a fruit grader where FD and FW are measured on individual pieces of fruit can be used to derive this relationship. A correct estimation of FD is important to calculate FW and in turn generate predictions of yield (Wang et al. 2017).

Estimation of fruit color is generally more complex than fruit number and FD because of the sensitivity of sensors to external environments and the typically subjective perception of color. McGuire (1992) highlighted how the use of CIELab and/or LCh color attributes guarantees an objective color measure. Literature showed that h° is associated with redness and maturity in peach and nectarine (Ferrer et al. 2005; Robertson et al. 1990; Scalisi et al. 2020, 2021a). Our results showed good accuracy of h° predictions in peach and nectarine (Fig. 6; Table 2). The clustering of points observed in the prediction of h° in ‘Angeleno’ plum was likely due to a combination of the range of the measured variable (0–360) and the color properties of this cultivar. In fact, near harvest, most fruit have a dark purple coloration and high levels of wax coating, meaning that they will have h° between 0° and 90° and between 200° and 360°, leading to a gap in coloration between 90° and 200°. Little errors in the predictions of fruit skin color with h° ∼0° or 360° will project the linear regression points near the 0° or the 360° ticks in the graph (Fig. 6D), generating two additional distant clusters in the bottom right and top left corners of the graph. Poor relationships (r_c < 0.40) between other CIELab color attributes (i.e., L*, a*, b*, and C*) measured by the mobile platform and by the portable colorimeter in all the stone fruit crops (Table 2) were most likely due to the fundamental difference between the contact sensing technology used for ground truthing and the prediction from proximal sensing. Proximal sensing might have been influenced by external light and skin reflectivity. However, our results suggest that predictions of h° were relatively sensor- and light-independent, as good accuracy was achieved in peach and nectarine without calibrations. Our findings are in line with what has been observed in previous work carried out in apricot, where two contact sensors with different illuminant settings measured similar h° values in ‘Golden May’ apricot collected at different maturity stages (Scalisi et al. unpublished data). Therefore, among the other color parameters, h° appears to be the most promising color attribute for proximal machine vision.

The new color parameter CDI, which is derived from h°, was shown to be a useful index to predict the development of fruit color in peach (Fig. 7A) and nectarine (Fig. 7B) with relatively high accuracy (r_c > 0.84) and can be successfully used for spatial mapping in these crops (Fig. 9A and B). In addition, spatial mapping of CDI over time, portraying the temporal and spatial variability in a plum orchard (Fig. 9C and D), showed the position of pollenizers within the block. The relationship between predicted and observed CDI in plum (Fig. 7D) did not improve the relationship obtained with h° ground truth. The lack of a significant relationship between ground truth CDI and CDI predicted from the mobile platform in plum was likely driven by 1) the disagreement of h° measured by the two devices, and by 2) an interference with the waxy cuticle layer present on plum skin when measuring with the contact sensor. Therefore, there is no evidence that CDI predictions from the mobile platform do not reflect the actual color development of plum. In fact, spatial maps in Fig. 9C and D suggest that CDI in plum truly followed a temporal evolution that reflected the change in fruit skin pigmentation. The use of CDI to monitor fruit color development and potentially maturity in fruit crops allows the estimation of fruit color (i.e., one of the most important fruit quality parameters). In terms of orchard management, spatial mapping of CDI can support decision making on 1) utilization of reflective mulching, 2) spraying of bio-stimulants that enhance anthocyanins and red pigmentation, 3) mechanical defoliation, and 4) thinning. Harvest timing could possibly be determined from fruit achieving a pre-established CDI threshold that meets fruit quality standards for specific destination markets.

In this study, fruit color measurements were collected at different times of the day under different light environments. An influence of external light on color detection cannot be ruled out, as this study does not present enough evidence for a definitive recommendation. Uniform light conditions are generally only achieved at night, assuming the absence of fog. In addition, nocturnal scans have the potential to improve the prediction of CIELab/LCh parameters that can be more useful in other crops. Hue angle is the single best predictor of redness development in fruit, as it is less affected by external light because it measures the angular relationship between yellow-blueness vs. greenness-redness, regardless of lightness and color intensity. This was empirically demonstrated in a previous study that compared measurements of CDI (i.e., a transformation of hue angle) under different light environments (Scalisi et al. 2022). Future evaluations of fruit skin color predictions in plum cultivars with yellow, orange, or red fruit skin will be needed to validate the utilization of CDI as the best color predictor for machine vision applications in stone fruit.

Green Atlas currently supports an experimental feature that produces spatial maps of h°. Our results suggest that spatial maps of CDI over time are the preferred method of presenting variability in stone fruit skin color. The mobile platform can additionally deliver estimates of tree size, such as canopy area and density, that can be modeled in relation to fruit number, fruit size, and fruit color. This provides a powerful tool for researchers and growers to establish best practices to obtain uniform and high-quality fruit. Furthermore, this study highlighted the utility of the experimental orchards at the Tatura SmartFarm to test and evaluate cutting-edge technology in different crops and very diverse orchard conditions.

Conclusions

This study presented pioneering research for the application of reliable, fast, and mobile predictions of fruit number, FD, and fruit color in stone fruit crops (peach, nectarine, plum, apricot). In summary, fruit number estimations were accurate after an initial calibration process. The data processing time is on average between 1.5- and 2.0-fold the time needed for scanning the orchard, based on Green Atlas’ experience in commercial orchards. Uncalibrated spatial maps can be used as a visual tool to assess spatial variability in several crop parameters (e.g., fruit number, fruit size, and fruit color). Potential future research would assess the technology to accurately predict fruit number for different crops, cultivars, row spacings, and training systems without the need for initial calibration. We found FD predictions were accurate under different measurement conditions (e.g., row spacing, crop, training system, fruit position in the canopy). When used in combination, spatial maps of fruit number, FD, yield, and fruit color represent a practical tool for growers to better understand the variability across an orchard block to manage labor and orchard operations like fruit thinning, pruning, spraying, and harvest logistics. Early fruit size, fruit quality, and yield predictions can inform and support orchard decision management in the logistics and post-harvest handling of crops and can help forecast production revenues.

Future development of the mobile platform for stone fruit crops should focus on the following: 1) reducing the need for orchard-specific calibrations by using stone fruit datasets from different regions, crops, cultivars, tree and row spacings, and training systems; 2) evaluating the estimation of yield by applying established conversions from FD to FW for forecasting purposes; 3) comparing the accuracy of fruit color estimations in nocturnal and diurnal scans; and 4) streamlining the application of the spatial data to orchard operations and precision spatial management like flower and fruitlet thinning, pest and disease spraying, and fruit harvesting.

Units

View Fullscreen