The majority of optical lenses have spherical surface profiles because they are convenient to fabricate. Replacing spherical optics with aspheric optics leads to smaller size, lighter weight, and less complicated optical systems with a superior imaging quality. However, fabrication of aspheric lenses is expensive and time-consuming. Here, we introduce a straightforward and low-cost casting method to fabricate polymeric aspheric lenses. An elastomeric ferrogel was formed into an aspherical profile by using a designed magnetic field and then was used as a mold. Different types of aspherical profiles from parabola to hyperbola can be formed with this method by tuning the magnetic field. A home-built Shack-Hartmann sensor was employed to characterize the cast polymeric lenses. The effects of magnetic field intensity, gradient of the magnetic field, and magnetic susceptibility of the ferrogel on the lens profiles were investigated. This technique can be used for rapid-forming polymeric aspherical lenses with different sizes and shapes.