The effect of rotation on small-scale characteristics and scaling law in the mixing zone of the three-dimensional turbulent Rayleigh-Taylor instability (RTI) is investigated by the lattice Boltzmann method at small Atwood number. The mixing zone width h(t), the root mean square of small scale fluctuation, the spectra, and the structure functions are obtained to analyze the rotating effect. We mainly focus on the process of the development of plumes and discuss the physical mechanism in the mixing zone in rotating and nonrotating systems. The variation of kinetic energy spectra E_{u} and temperature energy spectra E_{θ} with the dimensionless rotation Ωτ demonstrate the suppression effect of rotation. Two scaling laws between the mixing layer width h(t) and dimensionless time t/τ are obtained at various Coriolis forces(sqrt[h(t)]≃t^{0.9} and sqrt[h(t)]≃t^{0.35}). The rotation increasingly suppresses the growth of the mixing layer width h(t). The velocity and temperature fluctuations are also suppressed by the rotation effect. The relation between the Nusselt number (Nu) and the Rayleigh number (Ra) indicates that the heat transfer is suppressed by the rotation effect in the rotating RT system. The width of the inertial subrange increasingly narrows with increasing Ωτ.