We report pulse-field magnetization, ac susceptibility, and 100 GHz electron spin resonance (ESR) measurements on the S = 5/2 two-dimensional triangular compound Ba3NbFe3Si2O14 with the Néel temperature T(N) = 26 K. The magnetization curve shows an almost linear increase up to 60 T with no indication of a one-third magnetization plateau. An unusually large frequency dependence of the ac susceptibility in the temperature range of T = 20-100 K reveals a spin-glass behavior or superparamagnetism, signaling the presence of frustration-related slow magnetic fluctuations. The temperature dependence of the ESR linewidth exhibits two distinct critical regimes; (i) ΔH(pp)(T) is proportional to (T-T(N))(-p) with the exponent p = 0.2(1)-0.2(3) for temperatures above 27 K, and (ii) ΔH(pp)(T) is proportional to (T-T*)(-p) with T* = 12 K and p = 0.8(1)-0.8(4) for temperatures between 12 and 27 K. This is interpreted as indicating a dimensional crossover of magnetic interactions and the persistence of short-range correlations with a helically ordered state.