The dynamics of a membrane coupled to an active fluid on top of a substrate is considered theoretically. It is assumed that the director field of the active fluid has rotational symmetry in the membrane plane. This situation is likely to be relevant for in vitro reconstructed actomyosin-membrane system. Different from a membrane coupled to a polar active fluid, this model predicts that only when the viscosity of the fluid above the membrane is sufficiently large, a contractile active fluid is able to slow down the relaxation of the membrane for perturbations with wavelength comparable to the thickness of the active fluid. Hence, our model predicts a finite-wavelength instability in the limit of strong contractility, which is different from a membrane coupled to a polar active fluid. However, a membrane coupled to an extensile active fluid is always unstable against long-wavelength perturbations due to active extensile stress enhanced membrane undulation.