We introduce a 3-dimensional electromagnetic eigenmodal algorithm for the theoretical analysis of resonating nano-optical structures. The method, a variant of the Jacobi-Davidson algorithm, solves the electric field vector wave, or curl-curl, equation for the electromagnetic eigenmodes of resonant optical structures with a finite element method. In particular, the method includes transparent boundary conditions that enable the analysis of resonating structures in unbounded space. We demonstrate the performance of the method. First, we calculate the modes of several dielectric resonator antennas and compare them to theoretically determined results. Second, we calculate the modes of a nano-cuboid and compare them to theoretically determined results. Third, we numerically analyze spherical nanoparticles and compare the result to the theoretical Mie solution. Fourth, we analyze optical dipole antenna configurations in order to assess the method's capability for solving technologically relevant problems.