Considering beneficial effects of leisure activities in later life on well-being and health, we investigated which type of social network among older adults is associated with starting their participation in leisure activities. We used data from a longitudinal Japan Gerontological Evaluation Study (JAGES) conducted in Japan every three years from 2010 to 2016. We extracted types of social networks of older adults who did not participate in leisure activities in 2013 and responded to items related to social networks (n = 3436) relying on latent class analysis to examine changes in leisure activity participation over a three-year period within each latent class while controlling for participants' activity in 2010. As a result, we identified five latent classes of social networks: the Neighborhood network, the Restricted network, which is characterized by limited social contacts, the Colleagues network, the Same-Interest network, and the Diverse network, from the most to the least prevalent. We found that members of the Neighborhood (Cohen's d = 0.161) and Same-Interest networks (d = 0.660) were significantly more likely to, and members of the Diverse (d = 0.124) and Colleague networks (d = 0.060) were not significantly more likely to start leisure activities than those in the Restricted network. Furthermore, we found that lower age, better mental health, and higher education level were positively associated with starting participation in leisure activities in some latent classes. Horticulture or gardening was most likely to be chosen across all latent classes. Supporting the formation of social networks facilitating leisure activities, and recommending activities that were likely to be selected could be one solution for getting and keeping older adults active.