Selecting the single best blastocyst based on morphological appearance for implantation is a crucial part of in vitro fertilization (IVF). Various deep learning and computer vision-based methods have recently been applied for assessing blastocyst quality. However, to the best of our knowledge, most previous works utilize classification networks to give a qualitative evaluation. It would be challenging to rank blastocyst quality with the same qualitative result. Thus, this paper proposes a regression network combined with a soft attention mechanism for quantitatively evaluating blastocyst quality. The network outputs a continuous score to represent blastocyst quality precisely rather than some categories. As to the soft attention mechanism, the attention module in the network outputs an activation map (attention map) localizing the regions of interest (ROI, i.e., inner cell mass (ICM)) of microscopic blastocyst images. The generated activation map guides the entire network to predict ICM quality more accurately. The experimental results demonstrate that the proposed method is superior to traditional classification-based networks. Moreover, the visualized activation map makes the proposed network decision more reliable.