Because coal is quite weak compared with conventional sandstone, shear failure downhole will produce a large amount of nanoscale coal fines during the drainage process. Since the size of pores in coal is on the nanoscale range, these fines will sometimes cause serious damage problems downhole. The origin of coal fines cannot be explained by conventional sand prediction theory, which was previously designed for conventional sandstone. During the drainage process, the in situ stress change in coal was caused by the combination of the poroelastic effect, methane desorption and compression around the borehole. To prevent nanoscale coal fines, the critical pressure drawdown can be predicted by the comprehensive stress model. A special test was also designed to determine the key model parameters, making the model easy to use. It was proven that the induced stress due to methane desorption can exaggerate the shear failure, which is different from conventional sand prediction theory. Based on the stress model, the safe window of bottom hole pressure was applied for open-hole horizontal wells to prevent the origin of nanoscale coal fines.