In the cognitive radio system, spectrum sensing for detecting the presence of primary users in a licensed spectrum is a fundamental problem. Energy detection is the most popular spectrum sensing scheme used to differentiate the case where the primary user’s signal is present from the case where there is only noise. In fact, the nature of spectrum sensing can be taken as a binary classification problem, and energy detection is a linear classifier. If the signal-to-noise ratio (SNR) of the received signal is low, and the number of received signal samples for sensing is small, the binary classification problem is linearly inseparable. In this situation the performance of energy detection will decrease seriously. In this paper, a novel approach for obtaining a nonlinear threshold based on support vector machine with particle swarm optimization (PSO-SVM) to replace the linear threshold used in traditional energy detection is proposed. Simulations demonstrate that the performance of the proposed algorithm is much better than that of traditional energy detection.