Our objective was to assess factors associated with the physical (Short-Form 12-physical component summary [SF-12-PCS]) and mental (Short-Form 12-mental component summary [SF-12-MCS]) quality of life (QOL) in the Lebanese population. A cross-sectional study, conducted between July and September 2017, enrolled 57 patients with substance use disorder from the Psychiatric Hospital of the Cross. The SF-12 scale was used to assess the QOL among participants. A first stepwise linear regression, using the SF-12-PCS score as the dependent variable, showed that females compared with males (β = -4.86) and having a family history of substance use disorder (β = -4.70) were associated with lower physical QOL (lower SF-12-PCS scores). A second stepwise linear regression, using the SF-12-MCS score as the dependent variable, showed that having a generalized anxiety disorder (β = -10.52) and having suicidal ideation (β = -8.05) were associated with lower mental QOL (lower SF-12-MCS scores). Strategic interventions are needed to manage the health-related QOL in those with substance abuse.