RNA is a key participant in many biological processes, but studies of RNA using computer simulations lag behind those of proteins, largely due to less-developed force fields and the slow dynamics of RNA. Generating converged RNA ensembles for force field development and other studies remains a challenge. In this study, we explore the ability of replica exchange molecular dynamics to obtain well-converged conformational ensembles for two RNA hairpin systems in an implicit solvent. Even for these small model systems, standard REMD remains computationally costly, but coupling to a pre-generated structure library using the reservoir REMD approach provides a dramatic acceleration of ensemble convergence for both model systems. Such precise ensembles could facilitate RNA force field development and validation and applications of simulation to more complex RNA systems. The advantages and remaining challenges of applying R-REMD to RNA are investigated in detail.