It is puzzling how altruistic punishment of defectors can evolve in large groups of nonrelatives, since punishers should voluntarily bear individual costs of punishing to benefit those who do not pay the costs. Although two distinct mechanisms have been proposed to explain the puzzle, namely voluntary participation and group-level competition and selection, insights into their joint effects have been less clear. Here we investigated what could be combined effects of these two mechanisms on the evolution of altruistic punishment and how these effects can vary with nonparticipants' individual payoff and group size. We modelled altruistic punishers as those who contribute to a public good and impose a fine on each defector, i.e., they are neither pure punishers nor excluders. Our simulation results show that voluntary participation has negative effects on the evolution of cooperation in small groups regardless of nonparticipants' payoffs, while in large groups it has positive effects within only a limited range of nonparticipants' payoff. We discuss that such asymmetric effects could be explained by evolutionary forces emerging from voluntary participation. Lastly, we suggest that insights from social science disciplines studying the exit option could enrich voluntary participation models.