We report on an experimental investigation of the transition of a quantum system with integrable classical dynamics to one with violated time-reversal (T) invariance and chaotic classical counterpart. High-precision experiments are performed with a flat superconducting microwave resonator with circular shape in which T-invariance violation and chaoticity are induced by magnetizing a ferrite disk placed at its center, which above the cutoff frequency of the first transverse-electric mode acts as a random potential. We determine a complete sequence of ≃1000 eigenfrequencies and find good agreement with analytical predictions for the spectral properties of the Rosenzweig-Porter (RP) model, which interpolates between Poisson statistics expected for typical integrable systems and Gaussian unitary ensemble statistics predicted for chaotic systems with violated Tinvariance. Furthermore, we combine the RP model and the Heidelberg approach for quantum-chaotic scattering to construct a random-matrix model for the scattering (S) matrix of the corresponding open quantum system and show that it perfectly reproduces the fluctuation properties of the measured S matrix of the microwave resonator.