Atomic fingerprints are commonly used for the characterization of local environments of atoms in machine learning and other contexts. In this work, we study the behavior of two widely used fingerprints, namely, the smooth overlap of atomic positions (SOAP) and the atom-centered symmetry functions (ACSFs), under finite changes of atomic positions and demonstrate the existence of manifolds of quasi-constant fingerprints. These manifolds are found numerically by following eigenvectors of the sensitivity matrix with quasi-zero eigenvalues. The existence of such manifolds in ACSF and SOAP causes a failure to machine learn four-body interactions, such as torsional energies that are part of standard force fields. No such manifolds can be found for the overlap matrix (OM) fingerprint due to its intrinsic many-body character.