The strength of light-matter interaction is of central importance in photonics and optoelectronics. For many widely studied two-dimensional semiconductors, such as MoS2, the optical absorption due to exciton resonances increases with thickness. However, here we will show, few-layer black phosphorus exhibits an opposite trend. We determine the optical conductivity of few-layer black phosphorus with thickness down to bilayer by infrared spectroscopy. On the contrary to our expectations, the frequency-integrated exciton absorption is found to be enhanced in thinner samples. Moreover, the continuum absorption near the band edge is almost a constant, independent of the thickness. We will show such scenario is related to the quanta of the universal optical conductivity of graphene (σ0 = e2/4ħ), with a prefactor originating from the band anisotropy.