Measured spectral absorption coefficients were inverted to infer phytoplankton concentration in three size classes (picoplankton, nanoplankton, and microplankton), chlorophyll concentration [Chl], and both magnitude and spectral shape of absorption by colored detrital matter (CDM). Our algorithm allowed us to solve for the nonlinear factor of CDM absorption slope separately from the other linear factors, thus fully utilizing the additive characteristic inherent in absorption coefficients. We validated the inversion with three datasets: two spatially distributed global datasets, the Laboratoire d'Océanographie de Villefranche dataset and the NASA bio-Optical Marine Algorithm Dataset, and a time series coastal dataset, the Martha's Vineyard Coastal Observatory dataset. Comparison with high performance liquid chromatography analyses showed that the phytoplankton size classes can be retrieved with correlation coefficients (r)>0.7, root mean square errors of 0.2, and median relative errors of 20% in oceanic waters and with similar performance in coastal waters. Much improved agreement was found for the entire phytoplankton population, with r>0.90 for [Chl] and absorption coefficients (aph) for all three datasets. The inferred aCDM(400) and CDM spectral slope agree within ±4% of measurements in both oceanic and coastal waters. The results indicate that the chlorophyll-a specific absorption spectra used as an inversion kernel represent well the global mean states for each of the three phytoplankton size classes. The method can be applied to either bulk or particulate absorption data and is spectrally flexible.