The optical response of two-dimensional (2D) perovskites, often referred to as natural quantum wells, is primarily governed by excitons, whose properties can be readily tuned by adjusting the perovskite layer thickness. We have investigated the exciton fine structure splitting in the archetypal 2D perovskite (PEA)2(MA)n-1PbnI3n+1 with varying numbers of inorganic octahedral layers n = 1, 2, 3, and 4. We demonstrate that the in-plane excitonic states exhibit splitting and orthogonally oriented dipoles for all confinement regimes. The evolution of the exciton states in an external magnetic field provides further insights into the g-factors and diamagnetic coefficients. With increasing n, we observe a gradual evolution of the excitonic parameters characteristic of a 2D to three-dimensional transition. Our results provide valuable information concerning the evolution of the optoelectronic properties of 2D perovskites with the changing confinement strength.