Inflammatory bowel disease has been associated with differential abundance of numerous organisms when compared to healthy controls (HCs); however, few studies have investigated variability in the microbiome across intestinal locations and how this variability might be related to disease location and phenotype. In this study, we have analyzed the microbiome of a large cohort of individuals recruited at Mount Sinai Hospital in Toronto, Canada. Biopsies were taken from subjects with Crohn's disease, ulcerative colitis, and HC, and also individuals having undergone ileal pouch-anal anastomosis for treatment of ulcerative colitis or familial adenomatous polyposis. Microbial 16S rRNA was sequenced using the Illumina MiSeq platform. We observed a great deal of variability in the microbiome characterizing different sampling locations. Samples from pouch and afferent limb were comparable in microbial composition. When comparing sigmoid and terminal ileum samples, more differences were observed. The greatest number of differentially abundant microbes was observed when comparing either pouch or afferent limb samples to sigmoid or terminal ileum. Despite these differences, we were able to observe modest microbial variability between inflammatory bowel disease phenotypes and HCs, even when controlling for sampling location and additional experimental factors. Most detected associations were observed between HCs and Crohn's disease, with decreases in specific genera in the families Ruminococcaceae and Lachnospiraceae characterizing tissue samples from individuals with Crohn's disease. This study highlights important considerations when analyzing the composition of the microbiome and also provides useful insight into differences in the microbiome characterizing these seemingly related phenotypes.