Chinese chestnut (Castanea mollissima), which is native to China, has been cultivated as a nontimber forest tree species for 4000 years. This species has been found to display self-sterility, which results in a significantly lower seed set following self-pollination (SP) compared with that following cross-pollination (CP). Self-sterility can be induced by prezygotic or postzygotic late-acting self-incompatibility (LSI) or early-acting inbreeding depression (EID). To elucidate the causes of self-sterility in chestnut, we investigated pollen–pistil interactions, fertilization, and early ovule development following SP and CP by using a paraffin section technique and fluorescence microscopy. The fruit set percentage and seed characteristics also were evaluated among different pollination treatments. The results show that there were no significant differences in pollen tube behavior following SP vs. CP, regardless of the stigmatic or stylar level. Double fertilization was significantly greater following CP (18.09%) than SP (2.58%). The significantly lower percentages of ovule penetration and double fertilization in the selfed vs. crossed ovules support a prezygotic LSI mechanism in C. mollissima. The fruit set resulting from chase-pollination (CHP; 53.85% to 63.64%) was greater than that resulting from SP (12.12% to 14.00%). In addition, the distribution of aborted seed sizes after SP showed a widely clumped pattern. Abortion occurred at different stages during seed development rather than at a uniform stage, which supported the idea that EID was operating in C. mollissima. Levels of self-sterility in the chinese chestnut trees ranged from 88.2% to 90.5%. Thus, partial prezygotic LSI and EID contributed to self-sterility in the C. mollissima ‘Yanshanzaofeng’, with prezygotic LSI rejecting part of the self-pollen in the ovary and EID aborting part of the self-fertilized seeds.