Water can exist in a metastable liquid state under tension for long times before the system relaxes into the vapor via cavitation, i.e., bubble nucleation. Microscopic information on the cavitation process can be extracted from experimental data by the use of the nucleation theorem, which relates measured cavitation rates to the size of the critical bubble. To apply the nucleation theorem to experiments performed along an isochoric path, for instance, in cavitation experiments in mineral inclusions, knowledge of the bubble entropy is required. Using computer simulations, we compute the entropy of bubbles in water as a function of their volume over a wide range of tensions from free energy calculations. We find that the bubble entropy is an important contribution to the free energy that significantly lowers the barrier to bubble nucleation, thereby facilitating cavitation. Furthermore, the bubble entropy per surface area depends on the curvature of the liquid-vapor interface, decreasing approximately linearly with its mean curvature over the studied range of bubble volumes. At room temperature, the entropy of a flat liquid-vapor interface at ambient pressure is very similar to that of critical bubbles over a wide range of tensions, which justifies the use of the former as an approximation when interpreting data from experiments. Based on our simulation results, we obtain an estimate for the volume of the critical bubble from experimentally measured cavitation rates.