As a member of the bromodomain and extra terminal domain (BET) protein family, bromodomain-containing protein 4 (BRD4) is an epigenetic reader and can recognize acetylated lysine residues in histones. BRD4 has been regarded as an essential drug target for cancers, inflammatory diseases and acute heart failure, and therefore the discovery of potent BRD4 inhibitors with novel scaffolds is highly desirable. In this study, the crystalline water molecules in BRD4 involved in ligand binding were analyzed first, and the simulation results suggest that several conserved crystalline water molecules are quite essential to keep the stability of the crystalline water network and therefore they need to be reserved in structure-based drug design. Then, a docking-based virtual screening workflow with the consideration of the conserved crystalline water network in the binding pocket was utilized to identify the potential inhibitors of BRD4. The in vitro fluorescence resonance energy transfer (HTRF) binding assay illustrates that 4 hits have good inhibitory activity against BRD4 in the micromolar regime, including three compounds with IC50 values below 5 μM and one below 1 μM (0.37 μM). The structural analysis demonstrates that three active compounds possess novel scaffolds. Moreover, the interaction patterns between the hits and BRD4 were characterized by molecular dynamics simulations and binding free energy calculations, and then several suggestions for the further optimization of these hits were proposed.