The temperature dependence of the band gap is crucial to a semiconductor. Bulk black phosphorus is known to exhibit an anomalous behavior. Through optical spectroscopy, here we show that the temperature effect on black phosphorus band gap gradually evolves with decreasing layer number, eventually turns into a normal one in the monolayer limit, rendering a crossover from the anomalous to the normal. Meanwhile, the temperature-induced shift in optical resonance also differs with different transition indices for the same thickness sample. A comprehensive analysis reveals that the temperature-tunable interlayer coupling is responsible for the observed diverse scenario. Our study provides a key to the apprehension of the anomalous temperature behavior in certain layered semiconductors.