Telemedicine has potential to improve quality and delivery of medical care, particularly in image-oriented specialties where decisions are based on appearance of morphological features during examination. In the ophthalmology domain, nearly all published telemedicine studies have measured accuracy against a gold standard of ophthalmoscopic examination. The purposes of this study are to examine difficulties in defining an absolute gold standard and to compare diagnostic speed in a representative disease, retinopathy of prematurity. We compare results from ophthalmoscopic and telemedicine examinations by the same physicians. In 180 (86.5%) of 208 eyes, the two examinations produced the same diagnosis. In some discrepancies, there was rationale suggesting that telemedicine may have provided a more accurate diagnosis than ophthalmoscopic examination. The quantity and nature of these disagreements has important implications for evaluation of telemedicine systems in image-based specialties, and for the definition of gold standards in future studies.