In this paper, we propose a multiple-instance discriminant analysis algorithm for weakly supervised segment annotation. We introduce a selection parameter for each image/video with weak labels and expect that it can sift out object regions from the background clutter to train a better transformation vector. The selection parameter and the transformation parameter are incorporated into a single objective function and optimized in an alternate way. The optimization is an iteration between the eigenvalue decomposition and a set of quadratic programming. We also integrate a regularization term into the objective function to formulate the spatial constraint of segments, which is ignored in ordinary multiple-instance learning methods. The algorithm is able to overcome the limitations that arise when applying ordinary multiple-instance methods to the task. The experimental results validate the effectiveness of our method.