Background The Bosniak classification system for cystic renal masses was updated in 2019 in part to improve agreement compared with the 2005 version. Purpose To compare and investigate interrater agreement of Bosniak version 2019 and Bosniak version 2005 at CT and MRI. Materials and Methods In this retrospective single-center study, a blinded eight-reader assessment was performed in which 195 renal masses prospectively considered Bosniak IIF-IV (95 at CT, 100 at MRI, from 2006 to 2019 with version 2005) were re-evaluated with Bosniak versions 2019 and 2005. Radiologists (four faculty members, four residents) who were blinded to the initial clinical reading and histopathologic findings assessed all feature components and reported the overall Bosniak class for each system independently. Agreement was assessed with Gwet agreement coefficients. Uni- and multivariable linear regression models were developed to identify predictors of dispersion in the final Bosniak class assignment that could inform system refinement. Results A total of 185 patients were included (mean age, 63 years ± 13 [standard deviation]; 118 men). Overall interrater agreement was similar between Bosniak version 2019 and version 2005 (Gwet agreement coefficient: 0.51 [95% CI: 0.45, 0.57] vs 0.46 [95% CI: 0.42, 0.51]). This was true for experts (0.54 vs 0.49) and novices (0.50 vs 0.47) and at CT (0.56 vs 0.51) and MRI (0.52 vs 0.43). Nine percent of masses prospectively considered cystic using Bosniak version 2005 criteria were considered solid using version 2019 criteria. In general, masses were more commonly classified in lower categories when radiologists used Bosniak version 2019 criteria compared with version 2005 criteria. The sole predictor of dispersion in Bosniak version 2019 class assignment was dispersion in septa or wall quality (ie, smooth vs irregular thickening vs nodule; 72% [MRI] and 60% [CT] overall model variance explained; multivariable P < .001). Conclusion Overall interrater agreement was similar between Bosniak version 2019 and version 2005; disagreements in septa or wall quality were common and strongly predictive of variation in Bosniak class assignment. © RSNA, 2021 Online supplemental material is available for this article. See also the editorial by Eberhardt in this issue.