Aflatoxin M1 (AFM1) is a carcinogenic compound commonly found in milk in excess of the WHO permissible limit, especially in developing countries. Currently, state-of-the-art tests for detecting AFM1 in milk include chromatographic systems and enzyme-linked-immunosorbent assays. Although these tests provide fair accuracy and sensitivity, they require trained laboratory personnel, expensive infrastructure, and many hours to produce final results. Optical sensors leveraging spectroscopy have a tremendous potential of providing an accurate, real-time, and specialist-free AFM1 detector. Despite this, AFM1 sensing demonstrations using optical spectroscopy are still immature. Here, we demonstrate an optical sensor that employs the principle of cavity attenuated phase shift spectroscopy in optical fiber cavities for rapid AFM1 detection in aqueous solutions at 1550 nm. The sensor constitutes a cavity built by two fiber Bragg gratings. We splice a tapered fiber of < 10 μm waist inside the cavity as a sensing head. For ensuring specific binding of AFM1 in a solution, the tapered fiber is functionalized with DNA aptamers followed by validation of the conjugation via FTIR, TGA, and EDX analyses. We then detect AFM1 in a solution by measuring the phase shift between a sinusoidally modulated laser input and the sensor output at resonant frequencies of the cavity. Our results show that the sensor has the detection limit of 20 ng/L (20 ppt), which is well below both the U.S. and the European safety regulations. We anticipate that the present work will lead towards a rapid and accurate AFM1 sensor, especially for low-resource settings.