Coarse-graining is commonly used to decrease the computational cost of simulations. However, coarse-grained models are also considered to have lower transferability, with lower accuracy for systems outside the original scope of parametrization. Here, we benchmark a bead-necklace model and a modified Martini 2 model, both coarse-grained models, for a set of intrinsically disordered proteins, with the different models having different degrees of coarse-graining. The SOP-IDP model has earlier been used for this set of proteins; thus, those results are included in this study to compare how models with different levels of coarse-graining compare. The sometimes naive expectation of the least coarse-grained model performing best does not hold true for the experimental pool of proteins used here. Instead, it showed the least good agreement, indicating that one should not necessarily trust the otherwise intuitive notion of a more advanced model inherently being better in model choice.