This paper studies the nonlinear behavior of a nano-optomechanical actuator, consisting of a free-standing arc in a ring resonator that is coupled to a bus waveguide through evanescent waves. The arc deflects when a control light of a fixed wavelength and optical power is pumped into the bus waveguide, while the amount of deflection is monitored by measuring the transmission spectrum of a broadband probe light. This nanoactuator achieves a maximal deflection of 43.1 nm, with a resolution of 0.28 nm. The optical force is a nonlinear function of the deflection of the arc, leading to pull-back instability when the control light is red-tuned. This instability is studied by a combination of experiment and modeling. Potential applications of the nanoactuator include bio-nanomotor, optical switches, and optomechanical memories.