Based on the study of typical selenium-rich soil in the middle area of Zhejiang province, the influencing factors of selenium contents were summarized. We found that the selenium contents were controlled by the geological background. It was much higher in Tangxi, Zhijiang, and Jinhua than in other geological background regions. In addition, the contents of selenium in soil under different land use types showed a great difference. The content in the dry land was much more enriched than that in the paddy field. Further research showed that selenium was closely related to soil organic matter because of its adsorption and fixation. Moreover, we found that the soil texture was another important influencing factor. Selenium contents increased with increasing content of clay in the soil.