Many biological systems can switch between two distinct states. Once switched, the system remains stable for a period of time and may switch back to its original state. A gene network with bistability is usually required for the switching and stochastic effect in the gene expression may induce such switching. A typical bistable system allows one-directional switching, in which the switch from the low state to the high state or from the high state to the low state occurs under different conditions. It is usually difficult to enable bi-directional switching such that the two switches can occur under the same condition. Here, we present a model consisting of standard positive feedback loops and an extra negative feedback loop with a time delay to study its capability to produce bi-directional switching induced by noise. We find that the time delay in the negative feedback is critical for robust bi-directional switching and the length of delay affects its switching frequency.