Direct asymmetric three-component Mannich reaction involving simple ketones (such as cyclohexanone, acetone, and acetophenone) as donors and catalyzing by silver tartaric acid-derived phosphate was realized to afford a series of optically active β-amino-ketone derivatives in high yields (up to 96%) and good-to-high enantioselectivities (up to 97%) with moderate-to-good diastereoselectivities. This is the first example of direct catalytic asymmetric three-component Mannich reaction via a chiral counteranion-directed strategy.