Although several cell-based reporter assays have been developed for screening of viral protease inhibitors, most of these assays have a significant limitation in that numerous false positives can be generated for the compounds that are interfering with reporter gene detection due to the cellular viability. To improve, we developed a mammalian cell-based assay based on the reverse two-hybrid system to monitor the proteolytic activity of human enterovirus 71 (EV71) 3C protease and to validate the cytotoxicity of compounds at the same time. In this system, the GAL4 DNA binding domain (M3) and transactivation domain (VP16) were fused, in-frame, with 3C or 3C(mut). The 3C(mut) was an inactivated protease with mutations at the predicted catalytic triad. The reporter plasmid contains a secreted alkaline phosphatase (SEAP) gene under the control of GAL4 activating sequences. We demonstrated that M3-3C-VP16 failed to turn on the expression of SEAP due to the separation of M3 and the VP16 domains by self-cleavage of 3C. In contrast, SEAP expression was induced by the M3-3C(mut)-VP16 fusion protein or the M3-3C-VP16 in cells treated with AG7088, a potent inhibitor of human rhinoviruses (HRVs) 3C protease. Potentially, this protease detection system should greatly facilitate anti-EV71 drug discovery through a high-throughput screening.