Fermi's golden rule applies in the limit where an initial quantum state is weakly coupled to a continuum of other final states overlapping its energy. Here we investigate what happens away from this limit, where the set of final states is discrete, with a nonzero mean level spacing; this question arises in a number of recently investigated many-body systems. For different symmetry classes, we analytically and/or numerically calculate the universal crossovers in the average decay of the initial state as the level spacing is varied, with the golden rule emerging in the limit of a continuum. Among the corrections to the exponential decay of the initial state given by Fermi's golden rule is the appearance of the spectral form factor in the longtime regime for small but nonzero level spacing.