Camellia oleifera, a major woody oil plant, has a low oil yield because of self-incompatibility. For commercial oil production, compatible pollen and optimal cross-pollination combinations are required. To evaluate the effects of pollination compatibility and pollen source on oil yield and quality, four C. oleifera cultivars—Huashuo (HS), Huajin (HJ), Huaxin (HX), and Xianglin XLC15 (XL)—were subjected to self-, cross-, and natural pollination. Pollen compatibility, oil yield, and quality indices were analyzed. There were no significant differences in pollen germination and tube growth between self- and cross-pollination. Following self-pollination, fertilization was unsuccessful, resulting in severe ovule dysplasia; cross-pollination decreased the ovule abortion rate. Pollen source significantly affected the fruit set, fruit traits, seed traits, and fatty acid content, implying xenia in C. oleifera. In cross-pollinated plants, HX pollen produced more seeds, and HJ pollen increased linoleic acid content relative to naturally pollinated plants. For the XL and HS combinations, linolenic acid contents were significantly higher than other pollination combinations. However, oleic acid content was not significantly affected by pollen source, in any of the cultivars. Cultivar HX was, therefore, the most effective pollen donor, and HS × HX was the optimal cross-pollination combination for improving oil yield and sustainability.