The present paper aims at introducing the concepts and mathematical details of unsupervised neural learning with orthonormality constrains. The neural structures considered are single non-linear layers and the learnable parameters are organized in matrices, as usual, which gives the parameters spaces the geometrical structure of the Euclidean manifold. The constraint of orthonormality for the connection-matrices further restricts the parameters spaces to differential manifolds such as the orthogonal group, the compact Stiefel manifold and its extensions. For these reasons, the instruments for characterizing and studying the behavior of learning equations for these particular networks are provided by the differential geometry of Lie groups. In particular, two sub-classes of the general Lie-group learning theories are studied in detail, dealing with first-order (gradient-based) and second-order (non-gradient-based) learning. Although the considered class of learning theories is very general, in the present paper special attention is paid to unsupervised learning paradigms.