Brain rhythms recorded in vivo, such as gamma oscillations, are notoriously variable both in amplitude and frequency. They are characterized by transient epochs of higher amplitude known as bursts. It has been suggested that, despite their short-life and random occurrence, bursts in gamma and other rhythms can efficiently contribute to working memory or communication tasks. Abnormalities in bursts have also been associated with e.g. motor and psychiatric disorders. It is thus crucial to understand how single cell and connectivity parameters influence burst statistics and the corresponding brain states. To address this problem, we consider a generic stochastic recurrent network of Pyramidal Interneuron Network Gamma (PING) type. Using the stochastic averaging method, we derive dynamics for the phase and envelope of the amplitude process, and find that they depend on only two meta-parameters that combine all the model parameters. This allows us to identify an optimal parameter regime of healthy variability with similar statistics to those seen in vivo; in this regime, oscillations and bursts are supported by synaptic noise. The probability density for the rhythm's envelope as well as the mean burst duration are then derived using first passage time analysis. Our analysis enables us to link burst attributes, such as duration and frequency content, to system parameters. Our general approach can be extended to different frequency bands, network topologies and extra populations. It provides the much needed insight into the biophysical determinants of rhythm burst statistics, and into what needs to be changed to correct rhythms with pathological statistics.