Retrieval Question Answering (ReQA) is an essential mechanism of information sharing which aims to find the answer to a posed question from large-scale candidates. Currently, the most efficient solution is Dual-Encoder which has shown great potential in the general domain, while it still lacks research on biomedical ReQA. Obtaining a robust Dual-Encoder from biomedical datasets is challenging, as scarce annotated data are not enough to sufficiently train the model which results in over-fitting problems. In this work, we first build ReQA BioASQ datasets for retrieving answers to biomedical questions, which can facilitate the corresponding research. On that basis, we propose a framework to solve the over-fitting issue for robust biomedical answer retrieval. Under the proposed framework, we first pre-train Dual-Encoder on natural language inference (NLI) task before the training on biomedical ReQA, where we appropriately change the pre-training objective of NLI to improve the consistency between NLI and biomedical ReQA, which significantly improve the transferability. Moreover, to eliminate the feature redundancies of Dual-Encoder, consistent post-whitening is proposed to conduct decorrelation on the training and trained sentence embeddings. With extensive experiments, the proposed framework achieves promising results and exhibits significant improvement compared with various competitive methods.