Semantic segmentation of surgery scenarios is a fundamental task for computer-aided surgery systems. Precise segmentation of surgical instruments and anatomies contributes to capturing accurate spatial information for tracking. However, uneven reflection and class imbalance lead the segmentation in cataract surgery to a challenging task. To desirably conduct segmentation, a network with multi-view decoders (MVD-Net) is proposed to present a generalizable segmentation for cataract surgery. Two discrepant decoders are implemented to achieve multi-view learning with the backbone of U-Net. The experiment is carried out on the Cataract Dataset for Image Segmentation (CaDIS). The ablation study verifies the effectiveness of the proposed modules in MVD-Net, and superior performance is provided by MVD-Net in the comparison with the state-of-the-art methods. The source code will be publicly released.