Multiple modality clustering seeks to partition objects via leveraging cross-modality relations to provide comprehensive descriptions of the same objects. Current clustering methods rely heavily on accurate affinity measurements among samples. The samplewise affinity is costive to be constructed yet easy to corrupt by the heterogeneous gap. In the era of big data, fast and accurate clustering of multiple modality data remains challenging. To fill the gap, we propose a novel approach to achieve the clustering by focusing on feature matching across different modalities instead of samplewise affinity. First, a feature matching matrix is calculated by measuring the potential featurewise correlations. The obtained matching matrix is decomposed into two bases corresponding to the column and row spaces of feature matching, acting as coded bases within feature spaces of the different modalities. Then, the sample assignment is obtained by jointly reconstructing the samples by the two bases. The feature matching potential and sample assignment are collaboratively learned by an alternating optimization scheme. The proposed method dramatically reduces the computational cost by avoiding the costive samplewise affinity estimation, without sacrificing accuracy. Extensive experiments on the synthetic and real-world datasets demonstrate its superior speed and high accuracy.