The adoption of a sustainable soil management system is essential for the steep slopes and low fertility soils still supporting rainfed olive orchards in the Mediterranean basin. The effect of the soil management on olive yield, tree nutritional status and soil fertility was studied in a rainfed olive orchard located in NE Portugal that had been managed since its earliest days as a sheep-walk. In 2001, three different soil management systems were established: Sheep-walk, in which the vegetation was managed with a flock of sheep; Tillage, where the vegetation was controlled by conventional tillage; and Glyphosate, where a glyphosate-based herbicide was applied. The soil management systems had a pronounced effect on olive yield. The accumulated olive yields between 2002 and 2011 were 187.2, 142.9 and 89.5 kg tree-1, respectively in the Glyphosate, Tillage and Sheep-walk treatments. However, the effect of soil management on tree nutritional status was not so clear. On the other hand, the pools of organic carbon and N in the soil, and also the soil available N and phosphorus (P), were found to be less in the Glyphosate and Tillage treatments in comparison with the Sheep-walk. In these soils, N appeared as a much more limiting factor for crop growth than P. In rainfed orchards, the tolerance to herbaceous vegetation appears to be a determining factor in sustainability, which regulates annual crop yields and soil fertility. The higher the tolerance to herbaceous species, the lower the olive yields, but the better are the soil fertility parameters.