A new optimization method based on the topological derivative concept is developed for the electromagnetic design problem. Essentially, the purpose of the topological derivative method is to measure the sensitivity of a given shape functional with respect to a singular domain perturbation, so that it has applications in many relevant fields such as shape and topology optimization for imaging processing, inverse problems, and design of metamaterials. The topological derivative is rigorously derived for the electromagnetic scattering problem and used as gradient descent direction to find local optima for the design of electromagnetic devices. We demonstrate that the resulting topology design algorithm is remarkably simple and efficient and naturally leads to binary designs, while depending only on the solution of the conventional finite element formulation for electrodynamics. Finally, several numerical experiments in two and three spatial dimensions are presented to illustrate the performance of the proposed formulation.