The present manuscript treats the problem of adapting a neural signal processing system whose parameters belong to a curved manifold, which is assumed to possess the structure of a Lie group. Neural system parameter adapting is effected by optimizing a system performance criterion. Riemannian-gradient-based optimization is suggested, which cannot be performed by standard additive stepping because of the curved nature of the parameter space. Retraction-based stepping is discussed, instead, along with a companion stepsize-schedule selection procedure. A case-study of learning by optimization of a non-quadratic criterion is discussed in detail.