Black phosphorus (BP) has recently been rediscovered as a new and interesting two-dimensional material due to its unique electronic and optical properties. Here, we study the linear and nonlinear optical properties of BP flakes. We observe that both the linear and nonlinear optical properties are anisotropic and can be tuned by the film thickness in BP, completely different from other typical two-dimensional layered materials (e.g., graphene and the most studied transition metal dichalcogenides). We then use the nonlinear optical properties of BP for ultrafast (pulse duration down to ~786 fs in mode-locking) and large-energy (pulse energy up to >18 nJ in Q-switching) pulse generation in fiber lasers at the near-infrared telecommunication band ~1.5 μm. We observe that the output of our BP based pulsed lasers is linearly polarized (with a degree-of-polarization ~98% in mode-locking, >99% in Q-switching, respectively) due to the anisotropic optical property of BP. Our results underscore the relatively large optical nonlinearity of BP with unique polarization and thickness dependence, and its potential for polarized optical pulse generation, paving the way to BP based nonlinear and ultrafast photonic applications (e.g., ultrafast all-optical polarization switches/modulators, frequency converters etc.).