We deal with a system of prisoner's dilemma players undergoing continuous motion in a two-dimensional plane. In contrast to previous work, we introduce altruistic punishment after the game. We find punishing only a few of the cooperator-defector interactions is enough to lead the system to a cooperative state in environments where otherwise defection would take over the population. This happens even with soft nonsocial punishment (where both cooperators and defectors punish other players, a behavior observed in many human populations). For high enough mobilities or temptations to defect, low rates of social punishment can no longer avoid the breakdown of cooperation.