Iodinated contrast-enhanced X-ray imaging of the breast has been studied with various modalities, including full-field digital mammography (FFDM), digital breast tomosynthesis (DBT), and dedicated breast CT. Contrast imaging with breast CT has a number of advantages over FFDM and DBT, including the lack of breast compression, and generation of fully isotropic 3-D reconstructions. Nonetheless, for breast CT to be considered as a viable tool for routine clinical use, it would be desirable to reduce radiation dose. One approach for dose reduction in breast CT is spectral shaping using X-ray filters. In this paper, two high atomic number filter materials are studied, namely, gadolinium (Gd) and erbium (Er), and compared with Al and Cu filters currently used in breast CT systems. Task-based performance is assessed by imaging a cylindrical poly(methyl methacrylate) phantom with iodine inserts on a benchtop breast CT system that emulates clinical breast CT. To evaluate detectability, a channelized hoteling observer (CHO) is used with sums of Laguerre-Gauss channels. It was observed that spectral shaping using Er and Gd filters substantially increased the dose efficiency (defined as signal-to-noise ratio of the CHO divided by mean glandular dose) as compared with kilovolt peak and filter settings used in commercial and prototype breast CT systems. These experimental phantom study results are encouraging for reducing dose of breast CT, however, further evaluation involving patients is needed.