Early life family relationships affect the sexuality in adulthood, and these influences might be reflected in sexual dreams. The present study was designed to investigate the exact associations between family relationships and sexual dream experience. We therefore invited 62 frequent sexual dreamers (dreamers) and 104 healthy volunteers (controls) to answer the Sexual Dream Experience Questionnaire (SDEQ) and the Family Relationship Questionnaire (FRQ). Compared to controls, dreamers scored higher on all SDEQ factors and sexual dream frequency, higher on FRQ Paternal Abuse, and lower on FRQ General Attachment and Maternal Freedom Release. In controls, Paternal Abuse was associated with Joyfulness, Maternal Dominance with Aversion, and Maternal Abuse with dream frequency (-). In dreamers, Paternal Abuse was associated with Aversion, Bizarreness and dream frequency, and Maternal Freedom Release with Aversion (-). In conclusion, there were pronounced associations between sexual dreams and family relationships in frequent sexual dreamers. Paternal Abuse in particular was associated with sexual dream experience. Adverse family relationships might induce frequent sexual dream occurrence, and family therapy or early intervention of Paternal Abuse might alleviate the negative sexual dream experience.