A growing body of empirical evidence suggests that the dynamics of wealth within a population tends to be nonergodic, even after rescaling the individual wealth with the population average. Despite these discoveries, the way in which nonergodicity manifests itself in models of economic interactions remains an open issue. Here we shed valuable insight on these properties by studying the nonergodicity of the population average wealth in a simple model for wealth dynamics in a growing and reallocating economy called reallocating geometric Brownian motion (RGBM). When the effective wealth reallocation in the economy is from the poor to the rich, the model allows for the existence of negative wealth within the population. In this work, we show that in the negative reallocation regime of RGBM, ergodicity breaks as the difference between the time-average and the ensemble growth rate of the average wealth in the population. In particular, the ensemble average wealth grows exponentially, whereas the time-average growth rate is nonexistent. Moreover, we find that the system is characterized with a critical self-averaging time period. Before this time period, the ensemble average is a fair approximation for the population average wealth. Afterwards, the nonergodicity forces the population average to oscillate between positive and negative values since the magnitude of this observable is determined by the most extreme wealth values in the population. This implies that the dynamics of the population average is an unstable phenomenon in a nonergodic economy. We use this result to argue that one should be cautious when interpreting economic well-being measures that are based on the population average wealth in nonergodic economies.