Size bidisperse granular materials in a spherical tumbler segregate into two different patterns of three bands with either small particles at the equator and large particles at the poles or vice versa, depending upon the fill level in the tumbler. Here we use discrete element method simulations with supporting qualitative experiments to explore the effect of the tumbler wall roughness on the segregation pattern, modeling the tumbler walls as either a closely packed monolayer of fixed particles resulting in a rough wall or a frictional geometrically smooth wall. Even though the tumbler wall is in contact with the flowing layer only at its periphery, the impact of wall roughness is profound. Smooth walls tend toward a small-large-small (SLS) band pattern at the pole-equator-pole at all but the highest fill fractions; rough walls tend toward a large-small-large (LSL) band pattern at all but the lowest fill fractions. This comes about because smooth walls induce poleward axial drift of small particles and an equator-directed drift for large particles, resulting in an SLS band pattern. On the other hand, rough walls result in both sizes of particles moving poleward at the surface of the flow. Due to radial segregation, small particles percolate lower in the flowing layer and when arriving near the pole are caught in the return current drift that carries them back toward the equator incrementally with each passage through the flowing layer, while large particles remain at the surface near the pole, resulting in an LSL band pattern. The tendency toward either of the two segregation patterns depends on the fill level in the tumbler and the roughness of the tumbler's bounding wall.