In the design of a compliant admittance controller for physical human-robot interaction, it is necessary to ensure stable and effective cooperation. The stability of the admittance controller is mainly threatened by a stiff environment. Many methods that guarantee stability in arbitrary environments, impose conservative control gains that limit the effectiveness of the cooperation. Inspired by previous work in frequency domain stability observers, a method is proposed in this paper to detect unstable behavior and stabilize the robot with online adaptation of the admittance control gains. The introduced instability index is based on frequency domain analysis, which very quickly detects unstable behavior by monitoring high frequency oscillation in the force signal. To treat the instability, an adaptation scheme of the admittance parameters is proposed, that relaxes conservative gains and improves the cooperation by considering the effect of variable admittance on the operators' effort. We investigate two human-robot co-manipulation tasks; cooperation within a zero stiffness environment and cooperation in contact with a stiff double-wall virtual environment. The proposed methods are validated experimentally with a number of subjects in cooperation with an LWR manipulator.