Sexual activity in older people has become a topic of growing interest. The aim of this study is to investigate the effect of physical health and socioeconomic factors on the sexual activity of middle-aged and elderly Taiwanese men. From August 2007 to April 2008, 744 men older than 40 years were enrolled from a free health screening in Kaohsiung, Taiwan. All participants received detailed physical examination and answered questionnaires that collected demographic and lifestyle information, and medical history as well as answered items from the International Prostate Symptoms Score and five-item version of the International Index of Erectile Function (IIEF-5). Overall, 100 (13.4%) participants reported to be sexually inactive in previous 6 months. Older age, lower education levels, loss of a partner, erectile dysfunction, and increased number of comorbidities were found to be independent predictors for sexual inactivity. In conclusion, most middle-aged and elderly Taiwanese men remain sexually active. In addition to erectile dysfunction and loss of a partner, lower education levels and increased number of comorbidities were found to be predictors for sexual inactivity. Further research would need to elucidate whether improvement of those factors could help to preserve sexual activity.