Reproducing characteristics of observed sea ice extent remains an important climate modeling challenge. In this study we describe several approaches to improve how model biases in total sea ice distribution are quantified, and apply them to historically forced simulations contributed to the Coupled Model Intercomparison Project phase 5 (CMIP5). The quantity of hemispheric total sea ice area, or some measure of its equatorward extent is often used to evaluate model performance. We introduce a new approach which investigates additional details about the structure of model errors, with an aim to reduce the potential impact of compensating errors when gauging differences between simulated and observed sea ice. Using several observational data sets, we apply several new methods to evaluate the climatological spatial distribution and the annual cycle of sea ice cover in 41 CMIP5 models. We show that in some models, error compensation can be substantial, for example resulting from too much sea ice in one region and too little in another. Error compensation tends to be larger in models that agree more closely with the observed total sea ice area, which may result from model tuning. Our results suggest that consideration of only the total hemispheric sea ice area or extent can be misleading when quantitatively comparing how well models agree with observations. Further work is needed to fully develop robust methods to holistically evaluate the ability of models to capture the fine scale structure of sea ice characteristics, however, our "sector scale" metric aids to reduce the impact of compensating errors in hemispheric integrals.