Only one-third of embryo transfer cycles via invitro fertilization, the most common fertility treatment, leads to a clinical pregnancy. Identifying embryos with the highest potentials for transfer is an essential step to optimize in-vitro fertilization outcome. However, human embryos are complicated by nature and some of their developmental aspects has still remained a mystery to expert biologists. In this paper, the first-ever attempt is made to estimate probability of implantation using a single blastocyst image. First, a semantic segmentation system is proposed for human blastocyst components in microscopic images. Second, a multi-stream classification model is proposed for the prediction of embryos' implantation outcome. The proposed classification model features an architectural component, Compact-Contextualize-Calibrate (C3) to guide the feature extraction process and a slow-fusion strategy to learn cross-modality features. Experimental results confirm that the proposed method delivers the first-reported implantation outcome prediction via a single blastocyst image to date with a mean accuracy of 70.9%.