Vortex matter in mesoscopic superconductors is known to be strongly affected by the geometry of the sample. Here we show that in nanoscale superconductors with coherence length comparable to the Fermi wavelength the shape resonances of the order parameter results in an additional contribution to the quantum topological confinement-leading to unconventional vortex configurations. Our Bogoliubov-de Gennes calculations in a square geometry reveal a plethora of asymmetric, giant multivortex, and vortex-antivortex structures, stable over a wide range of parameters and which are very different from those predicted by the Ginzburg-Landau theory. These unconventional states are relevant for high-T(c) nanograins, confined Bose-Einstein condensates, and graphene flakes with proximity-induced superconductivity.