We report on the polarization-dependent optical response of elongated nanoholes in optically thin gold films. We measured elastic scattering spectra of spatially isolated ellipsoidal nanoholes with varying aspect ratio and compared the results to electrodynamic simulations. Both experiments and theory show that the plasmon mode that is polarized parallel to the short axis of the ellipsoidal hole red-shifts with increasing aspect ratio. This behavior is completely opposite to the case of elongated metal particles. We present a simple analytical model that qualitatively explains the observations in terms of the different orientations of the induced dipole moments in holes and particles.