Sexual relations between drug injectors (IDUs) and crack smokers (CS), and non-drug users are a major means of HIV spread to the broader population. However there is little literature describing community processes that regulate sexual and social partnerships among these groups. We describe these relationships in Bushwick, a low-income, mainly Latino neighbourhood in Brooklyn, NY. In this community, IDU and CS are heavily stigmatized, both by non-users and by some users. Known IDU/CS may find it harder to start and maintain social and sexual relationships, and to get jobs or support. Partially as a result of this stigma, IDU/CS attempt to 'keep it together' and hide either their drug use or its extent from other residents. Nevertheless, other residents believe, sometimes falsely, that they can distinguish users from nonusers. We describe some potential negative consequences of these beliefs and interactions, including their effects on risk for HIV and other sexually transmitted diseases.