We investigate system-size effects on the rotational diffusion of membrane proteins and other membrane-embedded molecules in molecular dynamics simulations. We find that the rotational diffusion coefficient slows down relative to the infinite-system value by a factor of one minus the ratio of protein and box areas. This correction factor follows from the hydrodynamics of rotational flows under periodic boundary conditions and is rationalized in terms of Taylor-Couette flow. For membrane proteins like transporters, channels, or receptors in typical simulation setups, the protein-covered area tends to be relatively large, requiring a significant finite-size correction. Molecular dynamics simulations of the protein adenine nucleotide translocase (ANT1) and of a carbon nanotube porin in lipid membranes show that the hydrodynamic finite-size correction for rotational diffusion is accurate in standard-use cases. The dependence of the rotational diffusion on box size can be used to determine the membrane viscosity.