Optical surfaces with high quality have been widely applied in high-tech industries for their excellent performances. To precision manufacture those surfaces efficiently and effectively, various machining technologies involved become extremely crucial. As one of the promising ultra-precision machining technologies, inflated or solid elastic tool polishing has attracted more attention for its own superiority. However, there is still lack of understanding on material removal mechanisms especially with the consideration of curvature effect, and it is of great importance to determine the surface quality and form control in ultra-precision polishing process. In this paper, originating from the famous macro-scale Preston equation, the curvature effect-based material removal model in polishing using a flexible ball-end tool has been developed successfully on the basis of two key sub-models, one is the generic model of effective relative velocity and the other refers to the semi-experimental contact pressure model. A series of spot polishing experiments subsequently are conducted on concave surfaces with a curvature radius range from 75 mm to 225 mm. The experimentally measured section profiles of polishing spots do match well with the predicted data, which verifies the effectiveness of the proposed material removal model. On the measured polishing spots, it is also observed that there have two nonuniform material removal phenomena, one is analyzed along the central axis and the other is discussed by two regions symmetrical about the central axis. Compared with the effective relative velocity, it is found that, the contact pressure is more sensitive to curvature effect by investigating the variation of maximum removal depth within a broader curvature radius range from 75 mm to 1000 mm. This study can provide a valuable foundation for polishing optical surfaces with deterministic removal.