The massively and rapidly spreading disinformation on social network platforms poses a serious threat to public safety and social governance. Therefore, early and accurate detection of rumors in social networks is of vital importance before they spread on a large scale. Considering the small-world property of social networks, the source tweet-word graph is decomposed from the global graph of rumors, and a rumor detection method based on graph attention network of source tweet-word graph is proposed to fully learn the structure of rumor propagation and the deep representation of text contents. Specifically, the proposed model can adequately capture the contextual semantic association representation of source tweets during the propagation and extract semantic features. For the data sparseness of the early stage of information dissemination, text attention mechanism based on opinion similarity can aggregate and capture more tweet propagation structure features to help improve the efficiency of early detection of rumors. Through the analysis of the experimental results on real public datasets, the rumor detection performance of the proposed method is better than that of other baseline methods. Especially in the early rumor detection tasks, the proposed method can detect rumors with an accuracy of nearly 90% in the early stage of information dissemination. And it still has good robustness with noise interference.