Conventional micro-particle manipulation technologies have been used for various biomedical applications using dynamics on a plane without vertical movement. In this case, irregular topographic structures on surfaces could be a factor that causes the failure of the intended control. Here, we demonstrated a novel colloidal particle manipulation mediated by the topographic effect generated by the "micro hill" and "surface gradient" around a micro-magnet. The magnetic landscape, matter orbital, created by periodically arranged circular micro-magnets, induces a symmetric orbit of magnetic particle flow under a rotating magnetic field. The topographic effect can break this symmetry of the energy distribution by controlling the distance between the source of the driving force and target particles by several nanometers on the surface morphology. The origin symmetric orbit of colloidal flow can be distorted by modifying the symmetry in the energy landscape at the switching point without changing the driving force. The enhancement of the magnetic effect of the micro-magnet array can lead to the recovery of the symmetry of the orbit. Also, this effect on the surfaces of on-chip-based devices configured by symmetry control was demonstrated for selective manipulation, trapping, recovery, and altering the direction using a time-dependent magnetic field. Hence, the developed technique could be used in various precise lab-on-a-chip applications, including where the topographic effect is required as an additional variable without affecting the existing control method.