Light has shown an incredible capability in precision measurement based on optomechanic interaction in high vacuum by isolating environment noises. However, there are still obstructions, such as displacement and mass estimation error, highly hampering the improvement of absolute accuracy at the nanoscale. Here, we present a nonlinearity based metrology to precisely measure the position and mass of a nanoparticle with optical levitation under 10^{-5} mbar. By precisely controlling the oscillation amplitude of the levitated nanoparticle at the nonlinear regime for high accuracy calibration, we realized a feasible sub-picometer-level position measurement with an uncertainty of 1.0% without the prior information of mass, which can be further applied to weigh the femtogram-level mass with an uncertainty of 2.2%. It will also pave the way to construct a fine-calibrated optomechanic platform in high vacuum for high sensitivity and accuracy measurement in force and acceleration at the nanoscale and the study in quantum superposition at the mesoscopic scale.