Disorder is an intrinsic attribute of any realistic molecular system. It is known to lead to localization, which hampers efficient transport. It was recently proposed that in molecular ensembles strongly coupled to photonic cavities, moderate disorder leads to delocalization and increases of the transport and chemical reaction rates. Vibrational polaritons involve molecular vibrations hybridized with an infrared cavity. When the coupling strength largely exceeds the molecular inhomogeneity, polaritons are unaffected by disorder. However, in many experiments, such a homogeneous limit does not apply. We investigated vibrational polaritons involving molecular ensembles with systematically modified disorder. Counterintuitively, moderate disorder leads to an increase in Rabi splitting and the modification of the polariton bandwidths. Experimental spectroscopic data agree with a Tavis-Cummings-like model that suggests enhanced delocalization of the reservoir states occurs via the admixture of the cavity mode. Our results provide new insights into the paradigm of disorder-induced cavity-assisted delocalization in molecular polaritons.