We investigate the modeling of inverse dynamics without prior kinematic information for holonomic rigid-body robots. Despite success in compensating robot dynamics and friction, general inverse dynamics models are nontrivial. Rigid-body models are restrictive or inefficient; learning-based models are generalizable yet require large training data. The structured kernels address the dilemma by embedding the robot dynamics in reproducing kernel Hilbert space. The proposed kernels autonomously converge to rigid-body models but require fewer samples; with a semi-parametric framework that incorporates additional parametric basis for friction, the structured kernels can efficiently model general rigid-body robots. We tested the proposed scheme in simulations and experiments; the models that consider the structure of function space are more accurate.