Biological membranes can exhibit various morphology due to the fluidity of the lipid molecules within the monolayers. The shape transformation of membranes has been well described by the classical Helfrich theory, which consists only a few phenomenological parameters, including the mean and the Gaussian curvature modulus. Though various methods have been proposed to measure the mean curvature modulus, determining the Gaussian curvature modulus remains difficult both in experiments and in simulations. In this paper we study the buckling process of a rectangular membrane and a circular membrane subject to compressive stresses and under different boundary conditions. We find that the buckling of a rectangular membrane takes place continuously, while the buckling of a circular membrane can be discontinuous depending on the boundary conditions. Furthermore, our results show that the stress-strain relationship of a buckled circular membrane can be used to determine the Gaussian curvature modulus effectively.