Amyloid plaques are one of the central manifestations of Alzheimer's disease pathology. Aggregation of the amyloid beta (Aβ) protein from amorphous oligomeric species to mature fibrils has been extensively studied. However, structural heterogeneities in prefibrillar species, and how that affects the structure of later-stage aggregates are not yet well understood. The integration of infrared spectroscopy with atomic force microscopy (AFM-IR) allows for identifying the signatures of individual nanoscale aggregates by spatially resolving spectra. We use AFM-IR to demonstrate that amyloid oligomers exhibit significant structural variations as evidenced in their infrared spectra. This heterogeneity is transmitted to and retained in protofibrils and fibrils. We show that amyloid fibrils do not always conform to their putative ordered structure and structurally different domains exist in the same fibril. We further demonstrate that these structural heterogeneities manifest themselves as a lack of β sheet structure in amyloid plaques in Alzheimer's tissue using infrared imaging.