Exosomes are nanosized extracellular vesicles (ranging from 30 to 120 nm) released from many cells that provide promising biomarkers for the noninvasive diagnosis of cancer. However, traditional exosome-isolation methods are tedious, nonstandardized, and require bulky instrumentation, thus limiting its clinical applications. In this paper, an anion-exchange (AE)-based isolation method was first proposed to isolate exosomes directly from plasma and cell-culture medium with AE magnetic beads within 30 min. Exosomes isolated with AE magnetic beads had higher recovery efficiency (>90%) and less protein impurities than those isolated by ultracentrifugation (UC). Prostate-cancer (PCa) exosomes in plasma were detected in a visual, label-free, and quantitative manner with aptamer-capped Fe3O4 nanoparticles for the first time. The linear range of PCa exosomes was estimated from 0.4 × 108 to 6.0 × 108 particles/mL with a detection limit of 3.58 × 106 particles/mL. The present study provides an efficient and practical approach for the rapid isolation and visible detection of exosomes, which is promising for the early diagnosis of PCa.