We report the rapid formation of photonic crystal structures by assembly of uniform nonmagnetic colloidal particles in ferrofluids using external magnetic fields. Magnetic manipulation of nonmagnetic particles with size down to a few hundred nanometers, suitable building blocks for producing photonic crystals with band gaps located in the visible regime, has been difficult due to their weak magnetic dipole moment. Increasing the dipole moment of magnetic holes has been limited by the instability of ferrofluids toward aggregation at high concentration or under strong magnetic field. By taking advantage of the superior stability of highly surface-charged magnetite nanocrystal-based ferrofluids, in this paper we have been able to successfully assemble 185 nm nonmagnetic polymer beads into photonic crystal structures, from 1D chains to 3D assemblies as determined by the interplay of magnetic dipole force and packing force. In a strong magnetic field with large field gradient, 3D photonic crystals with high reflectance (83%) in the visible range can be rapidly produced within several minutes, making this general strategy promising for fast creation of large-area photonic crystals using nonmagnetic particles as building blocks.