We report on theoretical investigations of the exotic magnetism in rubidium sesquioxide Rb4O6, a model correlated system with an open 2p shell. Experimental investigations indicated that Rb4O6 is a magnetically frustrated insulator. The frustration is explained here by electronic structure calculations that incorporate the correlation between the oxygen 2p electrons and deal with the mixed-valent oxygen. This leads to a physical picture where the symmetry is reduced because one third of the oxygen in Rb4O6 is nonmagnetic while the remaining two thirds assemble in antiferromagnetic arrangements. A degenerate, insulating ground state with a large number of frustrated noncollinear magnetic configurations is confidently deduced from the theoretical point of view. These findings demonstrate in general the importance of electron-electron correlations in open-shell p-electron systems.