The interplay of small, noncoding microRNAs (miRNAs), mRNAs and proteins plays crucial roles in almost all cellular processes. MiR-124, widely known as a memory-related miRNA, can regulate LTM by binding to the mRNA of CREB1 stimulated with 5-HT. In this paper, we establish a regulatory network model of CREB1 and miR-124 stimulated by 5-HT, in which miR-124 inhibits CREB1, which in turn enhances miR-124. Our model validates three protocols based on 5-HT in experiments on the induction of LTM in Aplysia. A steady-state analysis and numerical bifurcations of the abstracted system beyond memory formation, when the fast reaction has been in the equilibrium, can facilitate more abundant dynamical behaviors such as bistability and oscillation. The original system also exhibits bistability under appropriate feedback strengths, which is relevant to the mechanism of LTM formation. Furthermore, we specifically show a change in the transition from a reversible switch to an irreversible switch via bifurcations of the negative regulation of miR-124 on CREB1, which eventually maintains a high phosphorylated CREB1 level after initially elevated by 5-HT. These findings indicate that miR-124 provides an inhibitory constraint on long-term synaptic plasticity through the regulation of CREB1.