Global agriculture is under pressure to meet increasing demand for food and agricultural products. There are several global assessments of crop yields, but we know little about the uncertainties of their key findings, as the assessments are driven by the single best yield dataset available when each assessment was conducted. Recently, two different spatially explicit, global, historical yield datasets, one based on agricultural census and the other largely based on satellite remote sensing, became available. Using these datasets, we compare the similarities and differences in global yield gaps, trend patterns, growth rates and changes in year-to-year variability. We analyzed maize, rice, wheat and soybean for the period of 1981 to 2008 at four resolutions (0.083°, 0.5°, 1.0° and 2.0°). Although estimates varied by dataset and resolution, the global mean annual growth rates of 1.7-1.8%, 1.5-1.7%, 1.1-1.3% and 1.4-1.6% for maize, rice, wheat and soybean, respectively, are not on track to double crop production by 2050. Potential production increases that can be attributed to closing yield gaps estimated from the satellite-based dataset are almost twice those estimated from the census-based dataset. Detected yield variability changes in rice and wheat are sensitive to the choice of dataset and resolution, but they are relatively robust for maize and soybean. Estimates of yield gaps and variability changes are more uncertain than those of yield trend patterns and growth rates. These tendencies are consistent across crops. Efforts to reduce uncertainties are required to gain a better understanding of historical change and crop production potential to better inform agricultural policies and investments.