We present a semiclassical calculation of the generalized form factor Kab(tau) which characterizes the fluctuations of matrix elements of the operators a and b in the eigenbasis of the Hamiltonian of a chaotic system. Our approach is based on some recently developed techniques for the spectral form factor of systems with hyperbolic and ergodic underlying classical dynamics and f = 2 degrees of freedom, that allow us to go beyond the diagonal approximation. First we extend these techniques to systems with f > 2. Then we use these results to calculate Kab(tau). We show that the dependence on the rescaled time tau (time in units of the Heisenberg time) is universal for both the spectral and the generalized form factor. Furthermore, we derive a relation between Kab(tau) and the classical time-correlation function of the Weyl symbols of a and b.