Flesh firmness is a characteristic used to indicate fruit quality. Experimental design and data analysis are important when comparing devices that measure fruit firmness. We compared the Effegi penetrometer operated by hand, mounted in a drill press and then operated by hand, and mounted on a motorized drive and operated remotely; the hand-operated EPT pressure tester; the Instron with an Effegi probe; and a hand-operated prototype of the twist tester. Devices varied in operator differences and precision. Comparisons between devices were at the within-fruit level of variability and, therefore, more precise than comparisons where different device-operators used different fruit. We demonstrate statistical methods that are appropriate for making the comparisons of interest and discuss the possible cause of differences between operators and between devices. We also discuss how the mechanical properties of the devices may affect results and consider implications for their practical use. In this study, we found the precision of discrimination between soft and hard apples was best using the Instron in 1992, while the Instron and hand-held Effegi penetrometer were comparable in 1991. For kiwifruit, the hand-held Effegi penetrometer consistently gave the most precise measurements of softening in 1991, while the twist test was the most precise in 1992.