Semiconductor quantum dots and wires are important building blocks for future electronic and optoelectronic devices. The common way of producing semiconductor nanostructures is by molecular beam epitaxy (MBE). In this additive growth process atoms are deposited onto crystalline surfaces and self-assemble into 3D structures. Here we present a subtractive process, in which surface vacancies are created by ion impacts. On terraces of crystalline surfaces their nucleation forms depressions which coarsen and finally lead to a self-organized 3D morphology. It is shown that this kind of spontaneous pattern formation is inherent to the ion induced erosion process on crystalline surfaces and is analogous to 3D growth by MBE. However, novel facets are found due to slightly different energetics and kinetics of ad-atoms and surface vacancies, especially at Ehrlich-Schwoebel step-edge barriers. Depending on the crystal orientation, three-fold, four-fold, six-fold symmetry, as well as extremely regular periodic nanogrooves can be produced on different orientations of group IV (Si, Ge) and III-V (GaAs, InAs) semiconductors.