The association of physical activity and fitness with sleep still remains unclear since there is a lack of studies in this field of research using objective measurements of these variables. This study aimed to investigate the association of objectively-measured sedentariness, physical activity levels, and physical fitness with sleep quantity and quality in sedentary middle-aged adults. A total of 74 volunteers (52.7% women; aged 53.7 ± 5.1) were recruited for the present study. Cardiorespiratory fitness was measured through a maximal treadmill test, and muscular strength by extension and flexion peak torque, and by the hand grip test. Physical activity and objective sleep parameters were determined through accelerometry, and subjective sleep by the Pittsburgh Sleep Quality Index (PSQI). Reduced levels of sedentariness, greater VO2max, and greater muscular strength were positively related to improved objective sleep quantity and quality (all P ≤ 0.05). Furthermore, higher levels of overall physical activity, VO2max, and muscular strength were related to better subjective sleep quantity and quality (all P ≤ 0.05). Reduced sedentariness and increased physical activity and fitness may be a potential prevention and/or treatment pathway to reduce sleep disturbances and, in general, to improve patients physical and psychological health for a successful aging process.