Marangoni instabilities can emerge when a liquid interface is subjected to a concentration or temperature gradient. It is generally believed that for these instabilities bulk effects like buoyancy are negligible compared to interfacial forces, especially on small scales. Consequently, the effect of a stable stratification on the Marangoni instability has hitherto been ignored. Here, however, we show that they can matter. We report, for an immiscible drop immersed in a stably stratified ethanol-water mixture, a new type of oscillatory solutal Marangoni instability that is triggered once the stratification has reached a critical value. We experimentally explore the parameter space spanned by the stratification strength and the drop size and theoretically explain the observed crossover from levitating to bouncing by balancing the advection and diffusion around the drop. Finally, the effect of the stable stratification on the Marangoni instability is surprisingly strongly amplified in confined geometries, leading to an earlier onset.