Soil nitrogen (N) deficiencies can affect the photosynthetic N-use efficiency (PNUE), mesophyll conductance (gm), and leaf N allocation. However, lack of information about how these physiological characteristics in N-fixing trees could be affected by soil N deficiency and the difference between N-fixing and non-N-fixing trees. In this study, we chose seedlings of two N-fixing (Dalbergia odorifera and Erythrophleum fordii) and two non-N-fixing trees (Castanopsis hystrix and Betula alnoides) as study objects, and we conducted a pot experiment with three levels of soil N treatments (high nitrogen, set as Control; medium nitrogen, MN; and low nitrogen, LN). Our results showed that soil N deficiency significantly decreased the leaf N concentration and photosynthesis ability of the two non-N-fixing trees, but it had less influence on two N-fixing trees. The LN treatment had lower gm in D. odorifera and lower leaf N allocated to Rubisco (PR), leaf N allocated to bioenergetics (PB), and gm in B. alnoides, eventually resulting in low PNUE values. Our findings suggested that the D. odorifera and E. fordii seedlings could grow well in N-deficient soil, and adding N may increase the growth rates of B. alnoides and C. hystrix seedlings and promote the growth of artificial forests.