Whole slide imaging (WSI), is an essential technology for digital pathology, the performance of which is primarily affected by the autofocusing process. Conventional autofocusing methods either are time-consuming or require additional hardware and thus are not compatible with the current WSI systems. In this paper, we propose an effective learning-based method for autofocusing in WSI, which can realize accurate autofocusing at high speed as well as without any optical hardware modifications. Our method is inspired by an observation that sample images captured by WSI have distinctive characteristics with respect to positive / negative defocus offsets, due to the asymmetry effect of optical aberrations. Based on this physical knowledge, we develop novel deep cascade networks to enhance autofocusing quality. Specifically, to handle the effect of optical aberrations, a binary classification network is tailored to distinguish sample images with positive / negative defocus. As such, samples within the same category share similar characteristics. It facilitates the followed refocusing network, which is designed to learn the mapping between the defocus image and defocus distance. Experimental results demonstrate that our method achieves superior autofocusing performance to other related methods.