Boron carbonitride (BCN) films were deposited by d.c. unbalanced magnetron sputter deposition where a substrate bias ranging from -50 V to -300 V was applied to the sample. X-ray photoelectron spectroscopy and Fourier transform infrared spectroscopy were used to confirm the composition and bonding structure of the BCN films. Surface morphology and roughness were analyzed by atomic force microscopy. The nano-mechanical properties and nano-scratch behavior of the prepared films were determined by a nano-indentation system equipped with continuous stiffness measurement and lateral-force measurement attachments. The results indicate that there is little change in the content of B, C, and N. The films deposited are compounds with hybridized B-C-N bonds and the disordered degree of the structure increases with increasing the substrate bias. The substrate biasing can enhance the nanohardness, elastic modulus, nano-scratch resistance, and cohesion strength of the deposited films. During the nano-scratch test, plastic deformation and ploughing wear appears for the BCN films deposited at lower bias. Elastic deformation becomes the dominant deformation mechanism for the films deposited at higher bias. The coefficient of friction between the deposited BCN films and the diamond tip depends on the loading critical load. The increasing of the substrate bias leads to the improvement of the critical load and the elastic deformation proportion.