A layered oxychloride Bi4NbO8Cl is a visible-light responsive catalyst for water splitting, with its remarkable stability ascribed to the highly dispersive O-2p orbitals in the valence band, the origin of which, however, remains unclear. Here, we systematically investigate four series of layered bismuth oxyhalides, BiOX (X = Cl, Br, I), Bi4NbO8X (X = Cl, Br), Bi2GdO4X (X = Cl, Br), and SrBiO2X (X = Cl, Br, I), and found that Madelung site potentials of anions capture essential features of the valence band structures of these materials. The oxide anion in fluorite-like blocks (e.g., [Bi2O2] slab in Bi4NbO8Cl) is responsible for the upward shift of the valence band, and the degree of electrostatic destabilization changes depending on building layers and their stacking sequence. This study suggests that the Madelung analysis enables a prediction and design of the valence band structures of bismuth and other layered oxyhalides and is applicable even to a compound where DFT calculation is difficult to perform.