Planetary gearboxes exhibit complicated dynamic responses which are more difficult to detect in vibration signals than fixed-axis gear trains because of the special gear transmission structures. Diverse advanced methods have been developed for this challenging task to reduce or avoid unscheduled breakdown and catastrophic accidents. It is feasible to make fault features distinct by using multiwavelet denoising which depends on the feature separation and the threshold denoising. However, standard and fixed multiwavelets are not suitable for accurate fault feature detections because they are usually independent of the measured signals. To overcome this drawback, a method to construct customized multiwavelets based on the redundant symmetric lifting scheme is proposed in this paper. A novel indicator which combines kurtosis and entropy is applied to select the optimal multiwavelets, because kurtosis is sensitive to sharp impulses and entropy is effective for periodic impulses. The improved neighboring coefficients method is introduced into multiwavelet denoising. The vibration signals of a planetary gearbox from a satellite communication antenna on a measurement ship are captured under various motor speeds. The results show the proposed method could accurately detect the incipient pitting faults on two neighboring teeth in the planetary gearbox.