Controllability--the possibility of performing any target dynamics by applying a set of available operations--is a fundamental requirement for the practical use of any physical system. For finite-dimensional systems, such as spin systems, precise criteria to establish controllability, such as the so-called rank criterion, are well known. However, most physical systems require a description in terms of an infinite-dimensional Hilbert space whose controllability properties are poorly understood. Here, we investigate infinite-dimensional bosonic quantum systems--encompassing quantum light, ensembles of bosonic atoms, motional degrees of freedom of ions, and nanomechanical oscillators--governed by quadratic Hamiltonians (such that their evolution is analogous to coupled harmonic oscillators). After having highlighted the intimate connection between controllability and recurrence in the Hilbert space, we prove that, for coupled oscillators, a simple extra condition has to be fulfilled to extend the rank criterion to infinite-dimensional quadratic systems. Further, we present a useful application of our finding, by proving indirect controllability of a chain of harmonic oscillators.