Urban biodiversity conservation often aims to promote the quality of life for urban residents by providing ecosystem services as well as habitats for diverse wildlife. However, biodiversity inevitably brings some disadvantages, including problems and nuisances caused by wildlife. Although some studies have reported that enhancement of nature interaction among urban children promotes their affective attitude toward of favorable animals, its effect on tolerance toward problem-causing wildlife is unknown. In this study, we assessed the tolerance of 1,030 urban residents in Japan toward hornets and wild boar, and analyzed the effects of childhood experience with nature on tolerance using a structural equation model. The model used sociodemographic factors and childhood nature experience as explanatory variables, affective attitude toward these animals as a mediator, and tolerance as a response variable. The public tolerance toward hornets and boars was low; over 60% of the respondents would request the removal of hornets and wild boar from nearby green spaces by government services, even when the animals had not caused any damage. Tolerance was lower in females and elderly respondents. Childhood experience with nature had a greater influence on tolerance than did sociodemographic factors in the scenario where animals have not caused any problems; however, its effect was only indirect via promoting positive affective attitude toward wildlife when the animals have caused problems. Our results suggest that increasing people's direct experience with nature is important to raise public tolerance, but its effect is limited to cases where wildlife does not cause any problems. To obtain wider support for conservation in urban areas, conservationists, working together with municipal officials, educators and the media, should provide relevant information on the ecological functions performed by problem-causing wildlife and strategies for avoiding the problems that wildlife can cause.