Most studies on the health effects of leisure time physical activity have focused on mortality. There has been limited research regarding physical activity undertaken at work or around the home and mortality. This study assessed the associations between leisure, work, and household physical activity and subsequent all-cause mortality among older adults aged 65 years and older (n = 2133) in Taiwan, over 8 years. Physical activity was evaluated with the Physical Activity Scale for the Elderly. Cox proportional hazard models were used to estimate the association of physical activity with the risk of mortality. This study demonstrated that a low level of total physical activity is predictive of increased all-cause mortality in both men and women in an East Asian population. It also indicates that leisure- and household-related but not work-related activity are significant contributors to this relationship.