Automated retinal vessel segmentation is among the most significant application and research topics in ophthalmologic image analysis. Deep learning based retinal vessel segmentation models have attracted much attention in the recent years. However, current deep network designs tend to predominantly focus on vessels which are easy to segment, while overlooking vessels which are more difficult to segment, such as thin vessels or those with uncertain boundaries. To address this critical gap, we propose a new end-to-end deep learning architecture for retinal vessel segmentation: hard attention net (HAnet). Our design is composed of three decoder networks: the first of which dynamically locates which image regions are "hard" or "easy" to analyze, while the other two aim to segment retinal vessels in these "hard" and "easy" regions independently. We introduce attention mechanisms in the network to reinforce focus on image features in the "hard" regions. Finally, a final vessel segmentation map is generated by fusing all decoder outputs. To quantify the network's performance, we evaluate our model on four public fundus photography datasets (DRIVE, STARE, CHASE_DB1, HRF), two recent published color scanning laser ophthalmoscopy image datasets (IOSTAR, RC-SLO), and a self-collected indocyanine green angiography dataset. Compared to existing state-of-the-art models, the proposed architecture achieves better/comparable performances in segmentation accuracy, area under the receiver operating characteristic curve (AUC), and f1-score. To further gauge the ability to generalize our model, cross-dataset and cross-modality evaluations are conducted, and demonstrate promising extendibility of our proposed network architecture.