The refinement of gamete micromanipulation techniques has made conception possible for couples with male factor infertility who otherwise would remain childless. Moreover, intracytoplasmic sperm injection (ICSI) has ensured that such refractory cases can now generate offspring as successfully as in couples that merely require in vitro insemination. However, despite the now sterling record of ICSI it does not assure a successful outcome for every patient. This can be due, for instance, to the inability of the spermatozoon to activate the oocyte, and applies obviously in cases where spermatozoa are absent from the ejaculate or testicular biopsy. In the present paper we describe in detail the reasons for such failure and review the options that may help overcome it. In particular, we outline the treatment protocol for the situation in which spermatozoa are unable to induce oocyte activation. Further, we report on the clinical outcome achieved with spermatozoa retrieved from the testis, and in cases of extreme oligozoospermia we also explore the option of replicating a single spermatozoon while gaining information on its genomic content. For the most extreme situation in which men have no identifiable germ cells, we will discuss the current status of efforts to accomplish neo-gametogenesis through embryonic stem cell differentiation.