Detection of Brucella, causing brucellosis, is very challenging, since the applied techniques are mostly time-demanding and not standardized. While the common detection system relies on the cultivation of the bacteria, further classical typing up to the biotype level is mostly based on phenotypic or genotypic characteristics. The results of genotyping do not always fit the existing taxonomy, and misidentifications between genetically closely related genera cannot be avoided. This situation gets even worse, when detection from complex matrices, such as milk, is necessary. For these reasons, the availability of a method that allows early and reliable identification of possible Brucella isolates for both clinical and epidemiological reasons would be extremely useful. We evaluated micro-Raman spectroscopy in combination with chemometric analysis to identify Brucella from agar plates and directly from milk: prior to these studies, the samples were inactivated via formaldehyde treatment to ensure a higher working safety. The single-cell Raman spectra of different Brucella, Escherichia, Ochrobactrum, Pseudomonas, and Yersinia spp. were measured to create two independent databases for detection in media and milk. Identification accuracies of 92% for Brucella from medium and 94% for Brucella from milk were obtained while analyzing the single-cell Raman spectra via support vector machine. Even the identification of the other genera yielded sufficient results, with accuracies of >90%. In summary, micro-Raman spectroscopy is a promising alternative for detecting Brucella. The measurements we performed at the single-cell level thus allow fast identification within a few hours without a demanding process for sample preparation.