We demonstrate a unidirectional motion of a quasiparticle without explicit symmetry breaking along the space-time coordinate of the particle motion. This counterintuitive behavior originates from a combined action of two intrinsic asymmetries in the other two directions. We realize this idea with the magnon-driven motion of a magnetic domain wall in thin films with interfacial asymmetry. Contrary to previous studies, the domain wall moves along the same direction regardless of the magnon-flow direction. Our general symmetry analysis and numerical simulation reveal that the odd order contributions from the interfacial asymmetry is unidirectional, which is dominant over bidirectional contributions in the realistic regime. We develop a simple analytic theory on the unidirectional motion, which provides an insightful description of this counterintuitive phenomenon.