To train a deep neural network relies on a large amount of annotated data. In special scenarios like industry defect detection and medical imaging, it is hard to collect sufficient labeled data all at once. Newly annotated data may arrive incrementally. In practice, we also prefer our target model to improve its capability gradually as new data comes in by quick re-training. This work tackles this problem from a data selection prospective by constraining ourselves to always retrain the target model with a fix amount of data after new data comes in. A variational autoencoder (VAE) and an adversarial network are combined for data selection, achieving fast model retraining. This enables the target model to continually learn from a small training set while not losing the information learned from previous iterations, thus incrementally adapting itself to new-coming data. We validate our framework on the LGG Segmentation dataset for the semantic segmentation task.Clinical relevance- The proposed VAE-based data selection model combined with adversarial training can choose a representative and reliable subset of data for time-efficient medical incremental learning. Users can immediately see the improvement of the medical segmentation model whenever new annotated images are contributed (after a few minutes of model retraining).