We argue semiclassically, on the basis of Gutzwiller's periodic-orbit theory, that full classical chaos is paralleled by quantum energy spectra with universal spectral statistics, in agreement with random-matrix theory. For dynamics from all three Wigner-Dyson symmetry classes, we calculate the small-time spectral form factor K(tau) as power series in the time tau. Each term tau(n) of that series is provided by specific families of pairs of periodic orbits. The contributing pairs are classified in terms of close self-encounters in phase space. The frequency of occurrence of self-encounters is calculated by invoking ergodicity. Combinatorial rules for building pairs involve nontrivial properties of permutations. We show our series to be equivalent to perturbative implementations of the nonlinear sigma models for the Wigner-Dyson ensembles of random matrices and for disordered systems; our families of orbit pairs have a one-to-one relationship with Feynman diagrams known from the sigma model.