We describe the design, construction, and characterization of a medium-finesse Fabry-Perot cavity for simultaneous frequency stabilization of two lasers operating at 960 and 780 nm wavelengths, respectively. The lasers are applied in experiments with ultracold rubidium Rydberg atoms, for which a combined laser linewidth similar to the natural Rydberg linewidth (≈10 kHz) is desired. The cavity, with a finesse of ≈1500, is used to reduce the linewidth of the lasers to below this level. By using a spacer made of ultra low expansion (ULE) glass with active temperature stabilization, the residual frequency drift is limited to ≲1 MHz/day. The design optimizes for ease of construction, robustness, and affordability.