Metastasis is the leading cause of mortalities in cancer patients due to the spreading of cancer cells to various organs. Detecting cancer and identifying its metastatic potential at the early stage is important. This may be achieved based on the quantification of the key biomolecular components within tissues and cells using recent optical spectroscopic techniques. The aim of this study was to develop a noninvasive label-free optical biopsy technique to retrieve the characteristic molecular information for detecting different metastatic potentials of prostate cancer cells. Herein we report using native fluorescence (NFL) spectroscopy along with machine learning (ML) to differentiate prostate cancer cells with different metastatic abilities. The ML algorithms including principal component analysis (PCA) and nonnegative matrix factorization (NMF) were used for dimension reduction and feature detection. The characteristic component spectra were used to identify the key biomolecules that are correlated with metastatic potentials. The relative concentrations of the molecular spectral components were retrieved and used to classify the cancer cells with different metastatic potentials. A multi-class classification was performed using support vector machines (SVMs). The NFL spectral data were collected from three prostate cancer cell lines with different levels of metastatic potentials. The key biomolecules in the prostate cancer cells were identified to be tryptophan, reduced nicotinamide adenine dinucleotide (NADH) and hypothetically lactate as well. The cancer cells with different metastatic potentials were classified with high accuracy using the relative concentrations of the key molecular components. The results suggest that the changes in the relative concentrations of these key fluorophores retrieved from NFL spectra may present potential criteria for detecting prostate cancer cells of different metastatic abilities.