In light of the differences in hip fracture rates between men and women of different ages, age-related changes in bone structure that lead to bone fragility might differ depending on both age and gender. To investigate age-related bone loss and geometric deterioration of the femur, hip scans of 1,504 men and 2,076 women aged 19-92 years acquired during the Fourth Korean National Health and Nutrition Examination Survey (KNHANES IV) using dual-energy X-ray absorptiometry (DXA) were analyzed with a structural analysis program. Cross-sectional area and cortical thickness with bone mineral density in men started to decline from the third decade and continued to decline at a constant rate. However, in women, these parameters remained nearly constant until the fifth decade and then declined at a more rapid rate than that seen in men. Consequently, changes in the buckling ratio, earlier onset, and continuation of increase over the lifetime were observed in men. A relatively later onset with a greater acceleration with aging was observed in women. Taken together, there were obvious gender and age differences in structural trends with age. Bone aging, i.e., bone loss and geometric deterioration, actually begins at a young age, especially in men, and osteoporosis prevention strategies should target not only the elderly but also younger individuals.