Broad learning system (BLS), which emerges as a lightweight network paradigm, has recently attracted great attention for recognition problems due to its good balance between efficiency and accuracy. However, the supervision mechanism in BLS and its variants generally relies on the strict binary label matrix, which imposes limitations on approximation and fails to adequately align with the data distribution. To address this issue, in this article, two novel flexible label-induced BLS models with the manifold manner are proposed, whose notable characteristics are as follows. First, two proposed label relaxation strategies can both enlarge the margins between different categories and simultaneously enhance the diversity within labels. Second, the integration of manifold geometrical criterion enables the models to capture local feature structures, ensuring the obtained flexible labels align better with the similarity between samples. Third, the proposed models can be optimized efficiently with the alternating direction method of multipliers. Each iteration benefits from a closed-form solution, facilitating the optimization process. Extensive experiments and thorough theoretical analysis are intended to show the advantages of our proposed models compared to other state-of-the-art recognition algorithms.