We report an electric-field-induced giant modulation of the hole g factor in SiGe nanocrystals. The observed effect is ascribed to a so-far overlooked contribution to the g factor that stems from the mixing between heavy- and light-hole wave functions. We show that the relative displacement between the confined heavy- and light-hole states, occurring upon application of the electric field, alters their mixing strength leading to a strong nonmonotonic modulation of the g factor.