Visual object tracking is a challenging computer vision task with numerous real-world applications. In this paper, we propose a simple but efficient Spectral Filter Tracking (SFT) method from the view of graph, where each candidate image region is modeled as a pixelwise grid graph. Instead of the conventional graph matching, we formulate the tracking as a plain least square regression problem of learning spectral filters on graphs to predict an optimal vertex, which indicates the center of the target. To bypass computationally expensive eigenvalue decomposition on graph Laplacian L, we parameterize spectral graph filters as a polynomial of L to aggregate local graph features according to spectral graph theory, in which Lk exactly encodes a k-hop local neighborhood of each vertex. Thus, different from the holistic regression in those correlation filter based methods, SFT can operate on localized regions around a pixel (i.e., a vertex), which can effectively reduce the influence of local variations and cluttered backgrounds. Furthermore, we observe that the correlation filter tracking may be viewed as a specific case of our proposed spectral filtering method. The implementation of SFT can simply boil down to only a few line codes, but surprisingly it beats the correlation filter based model with the same feature input, and achieves the state-of-the-art performance on OTB-2015 and VOT2016 under the same feature extraction strategy.