The determination of nutrient removal from olive orchards could be of interest to estimate tree consumption and to provide information about the amount of nutrients to be applied when leaf analysis indicates the need for fertilization. In this work, nutrient removal from yield and pruning was determined from the control plots of two olive orchards located in different locations, in which two long-term experiments dealing with nitrogen fertilization were conducted. The trees from these plots received only potassium fertilizers during the 7 years of the experiments, because the previous season’s leaf analysis showed that the other nutrients were always above the threshold of sufficiency. Potassium was the most abundant element in the harvested fruits with an average of 4.42 g·kg−1 fresh fruit, which represents more than 50% of the mineral composition of the olive fruit, whereas calcium was the more abundant element in the pruning material with an average of 12.0 g·kg−1 and 6.87 g·kg−1, depending on the location, which represents more than 50% of the mineral composition of the pruning material. Nitrogen was the second more abundant element in both fruits (2.87 g·kg−1) and pruning material (6.87 and 5.40 g·kg−1, depending on the location), representing ≈35% of the mineral composition of both fruit and pruning material. The other nutrients were removed only in very small amounts. Expressed per hectare, the amounts of nutrients removed annually were: 57.9 kg·ha−1 per year calcium (Ca), 54.4 kg·ha−1 per year nitrogen (N), 45.5 kg·ha−1 per year potassium (K), 6.87 kg·ha−1 per year phosphorus (P), 3.79 kg·ha−1 per year magnesium (Mg), 0.12 kg·ha−1 per year copper (Cu), 0.11 kg·ha−1 per year boron (B), 0.08 kg·ha−1 per year manganese, and 0.05 kg·ha−1 per year zinc (Zn). These data show that olive trees remove small amounts of nutrients and, therefore, the need for fertilization is relatively low.