The t_{2g} orbitals of an edge-shared transition-metal oxide with a honeycomb lattice structure form dispersionless electronic bands when only hopping mediated by the edge-sharing oxygens is accessible. This is due to the formation of isolated quasimolecular orbitals (QMOs) in each hexagon, introduced recently by Mazin et al. [Phys. Rev. Lett. 109, 197201 (2012)], which stabilizes a band insulating phase for t_{2g}^{5} systems. However, with the help of the exact diagonalization method to treat the electron kinetics and correlations on an equal footing, we find that the QMOs are fragile against not only the spin-orbit coupling (SOC) but also the Coulomb repulsion. We show that the electronic phase of t_{2g}^{5} systems can vary from a quasimolecular band insulator to a relativistic J_{eff}=1/2 Mott insulator with increasing the SOC as well as the Coulomb repulsion. The different electronic phases manifest themselves in electronic excitations observed in optical conductivity and resonant inelastic x-ray scattering. Based on our calculations, we assert that the currently known Ru^{3+} and Ir^{4+} based honeycomb systems are far from the quasimolecular band insulator but rather the relativistic Mott insulator.