There is growing evidence the microbiota of the large bowel may influence the risk of developing colorectal cancer as well as other diseases including type-1 diabetes, inflammatory bowel diseases and irritable bowel syndrome. Current sampling methods to obtain microbial specimens, such as feces and mucosal biopsies, are inconvenient and unappealing to patients. Obtaining samples through rectal swabs could prove to be a quicker and relatively easier method, but it is unclear if swabs are an adequate substitute. We compared bacterial diversity and composition from rectal swabs and rectal mucosal biopsies in order to examine the viability of rectal swabs as an alternative to biopsies. Paired rectal swabs and mucosal biopsy samples were collected in un-prepped participants (n = 11) and microbial diversity was characterized by Terminal Restriction Fragment Length polymorphism (T-RFLP) analysis and quantitative polymerase chain reaction (qPCR) of the 16S rRNA gene. Microbial community composition from swab samples was different from rectal mucosal biopsies (p = 0.001). Overall the bacterial diversity was higher in swab samples than in biopsies as assessed by diversity indexes such as: richness (p = 0.01), evenness (p = 0.06) and Shannon's diversity (p = 0.04). Analysis of specific bacterial groups by qPCR showed higher copy number of Lactobacillus (p < 0.0001) and Eubacteria (p = 0.0003) in swab samples compared with biopsies. Our findings suggest that rectal swabs and rectal mucosal samples provide different views of the microbiota in the large intestine.