Investigation of the mechanical properties of materials at the nanoscale is often performed by atomic force microscopy nanoindentation. However, substrates with large surface roughness and heterogeneity demand careful data analysis. This requirement is even more stringent when surface indentations with a typical depth of a few nanometers are produced to test material hardness. Accordingly, we developed a geometrical model of the nanoindenter, which was first validated by measurements on a reference gold sample. Then we used this technique to investigate the mechanical properties of a coating layer made of Balinit C, a commercially available alloy with superior anti-wear features deposited on steel. The reported results support the feasibility of reliable hardness measurements with truly nanosized indents.