The support vector machine (SVM) has been receiving increasing interest in an area of QSAR study for its ability in function approximation and remarkable generalization performance. However, selection of support vectors and intensive optimization of kernel width of a nonlinear SVM are inclined to get trapped into local optima, leading to an increased risk of underfitting or overfitting. To overcome these problems, a new nonlinear SVM algorithm is proposed using adaptive kernel transform based on a radial basis function network (RBFN) as optimized by particle swarm optimization (PSO). The new algorithm incorporates a nonlinear transform of the original variables to feature space via a RBFN with one input and one hidden layer. Such a transform intrinsically yields a kernel transform of the original variables. A synergetic optimization of all parameters including kernel centers and kernel widths as well as SVM model coefficients using PSO enables the determination of a flexible kernel transform according to the performance of the total model. The implementation of PSO demonstrates a relatively high efficiency in convergence to a desired optimum. Applications of the proposed algorithm to QSAR studies of binding affinity of HIV-1 reverse transcriptase inhibitors and activity of 1-phenylbenzimidazoles reveal that the new algorithm provides superior performance to the backpropagation neural network and a conventional nonlinear SVM, indicating that this algorithm holds great promise in nonlinear SVM learning.