In order to identify the extent to which results from topological graph models are useful for modeling vulnerability in electricity infrastructure, we measure the susceptibility of power networks to random failures and directed attacks using three measures of vulnerability: characteristic path lengths, connectivity loss, and blackout sizes. The first two are purely topological metrics. The blackout size calculation results from a model of cascading failure in power networks. Testing the response of 40 areas within the Eastern U.S. power grid and a standard IEEE test case to a variety of attack/failure vectors indicates that directed attacks result in larger failures using all three vulnerability measures, but the attack-vectors that appear to cause the most damage depend on the measure chosen. While the topological metrics and the power grid model show some similar trends, the vulnerability metrics for individual simulations show only a mild correlation. We conclude that evaluating vulnerability in power networks using purely topological metrics can be misleading.