We theoretically investigate the optical response of a cavity optomechanical system via the nonlinear coupling between optical and mechanical resonators, which is expected to be strong. Our results show that the nonlinear coupling will significantly influence the optical bistability. We compare the transmission spectrum of linear coupling with that of nonlinear coupling up to the second order and observe a shift between the transmission peak, which indicates the energy-level modification induced by the nonlinear coupling. Based on this nonlinear optomechanical phenomenon, we propose a theoretical method to determine the mechanical frequency and the second-order coupling constant. This means will eliminate the influence caused by the first-order coupling, which enables much easier detection of second-order coupling constant.