The dynamics of complex systems is characterized by oscillatory components on many time scales. To study the interactions between these components we analyze the cross modulation of their instantaneous amplitudes and frequencies, separating synchronous and antisynchronous modulation. We apply our novel technique to brain-wave oscillations in the human electroencephalogram and show that interactions between the alpha wave and the delta or beta wave oscillators as well as spatial interactions can be quantified and related with physiological conditions (e.g., sleep stages). Our approach overcomes the limitation to oscillations with similar frequencies and enables us to quantify directly nonlinear effects such as positive or negative frequency modulation.