Fusing a low-resolution hyperspectral image (HSI) with a high-resolution (HR) conventional image into an HR HSI has become a prevalent HSIs super-resolution scheme. However, in most previous works, little attention has been paid on exploiting the underlying manifold structure in the spatial domain of the latent HR HSI. In this paper, we advance a provable prior knowledge that the clustering manifold structure of the latent HSI can be well preserved in the spatial domain of the input conventional image. Inspired by this, we first conduct clustering in the spatial domain of the input conventional image and adopt the intra-cluster self-expressiveness model to implicitly depict the clustering manifold structure, which enables learning the complicated manifold structure via solving a constrained ridge regression model without knowing the exact form of the manifold. Then, we incorporate the learned structure into a variational super-resolution framework to regularize the latent HSI. The resulted framework can be effectively optimized by a standard alternating direction method of multipliers. Since the learned structure can well depict the underlying spatial manifold of the latent HSI, the proposed method shows the state-of-the-art super-resolution performance on two benchmark data sets.