Graph-based semisupervised learning is of great importance in many effective learning systems, particularly in agnostic settings where no parametric information or other prior knowledge about the data distribution is available. It leverages the graph structure to propagate labels from labeled nodes to unlabeled ones. Two separate stages are usually involved: constructing an affinity graph and propagating labels on the graph for transductive inference. It is suboptimal to manage them independently, as the correlation between the affinity graph and the labels would not be fully exploited. In this article, we integrate these two stages into one unified framework by formulating the graph construction as a regularized function estimation problem, similar to label propagation. We then propose an alternating diffusion process to solve them alternately, which allows us to learn the graph and unknown labels in an iterative fashion. With the proposed framework, we can construct a dynamic graph adapted to the given and predicted labels iteratively, resulting in more accurate and robust label propagation performance. Extensive experiments on synthetic data and various real-world data have demonstrated the superiority of the proposed method compared with other state-of-the-art methods.