Paper-based immunoassays are becoming powerful and low-cost diagnostic tools, especially in resource-limited settings. Inexpensive methods for quantifying these assays have been shown using desktop scanners, which lack portability, and cameras, which suffer from the ever changing ambient light conditions. In this work, we introduce a novel approach of quantifying colors of colorimetric diagnostic assays with a smartphone that allows high accuracy measurements in a wide range of ambient conditions, making it a truly portable system. Instead of directly using the red, green, and blue (RGB) intensities of the color images taken by a smartphone camera, we use chromaticity values to construct calibration curves of analyte concentrations. We demonstrate the high accuracy of this approach in pH measurements with linear response ranges of 1-12. These results are comparable to those reported using a desktop scanner or silicon photodetectors. To make the approach adoptable under different lighting conditions, we developed a calibration technique to compensate for measurement errors due to variability in ambient light. This technique is applicable to a number of common light sources, such as sun light, fluorescent light, or smartphone LED light. Ultimately, the entire approach can be integrated in an "app" to enable one-click reading, making our smartphone based approach operable without any professional training or complex instrumentation.