The magnetic-field dependence of the storage modulus for bimodal magnetic elastomers consisting of carbonyl iron with a diameter of 2.8 μm (magnetic) and aluminum hydroxide with a diameter of 1.4 μm (nonmagnetic) was measured, and the effect of nonmagnetic particles on the magnetic-field sensitivity of the storage modulus was investigated. The coefficient of the magnetic-field sensitivity for the monomodal magnetic elastomer increased from 0.018 to 0.026 mT-1 for the bimodal one by embedding nonmagnetic particles of 6.6 vol %. At volume fractions above 5.4 vol %, the bimodal magnetic elastomer exhibited significant nonlinear viscoelasticity at 0 mT and a high storage modulus at 500 mT, simultaneously, the coefficient of the magnetic-field sensitivity demonstrated high values. This strongly indicates that both the particles form a particle network at the off-field, and they make a well-developed chain structure under magnetic fields. The time profiles of the storage modulus for bimodal magnetic elastomers can be fitted by a linear combination of two exponential functions with two characteristic times showing the alignment of magnetic particles. The alignment time for the fast and slow processes was distributed around 3.3 ± 0.3 and 176 ± 12 s, respectively. The alignment time was independent of the volume fraction of the nonmagnetic particles; however, the increment in the storage modulus for the fast process significantly increased at volume fractions above 5.4 vol %. It was also revealed that the coefficient of the magnetic-field sensitivity can be scaled by a power function of the increment in the storage modulus divided by the off-field modulus, ΔG'/G'0, not only for the bimodal magnetic elastomers but also for the monomodal ones.