Ghost imaging (GI) is an unconventional optical imaging method making use of the correlation measurement between a test beam and a reference beam. GI using deep learning (GIDL) has earned increasing attention, as it can reconstruct images of high quality more effectively than traditional GI methods. It has been demonstrated that GIDL can be trained completely with simulation data, which makes it even more practical. However, most GIDLs proposed so far appear to have limited performance for random noise distributed patterns. This is because traditional GIDLs are sensitive to the under-estimation error but robust to the over-estimation error. An asymmetric learning framework is proposed here to tackle the unbalanced sensitivity to estimation errors of GIDL. The experimental results show that it can achieve much better reconstructed images than GIDL with a symmetric loss function, and the structural similarity index of GI is quadrupled for randomly selected objects.