With the presence of complex background noise, parasitic light, and dust attachment, it is still a challenging issue to perform high-precision laser-induced damage change detection of optical elements in the captured optical images. For resolving this problem, this paper presents an end-to-end damage change detection model based on siamese network and multi-layer perceptrons (SiamMLP). Firstly, representative features of bi-temporal damage images are efficiently extracted by the cascaded multi-layer perceptron modules in the siamese network. After that, the extracted features are concatenated and then classified into changed and unchanged classes. Due to its concise architecture and strong feature representation ability, the proposed method obtains excellent damage change detection results efficiently and effectively. To address the unbalanced distribution of hard and easy samples, a novel metric called hard metric is introduced in this paper for quantitatively evaluating the classification difficulty degree of the samples. The hard metric assigns a classification difficulty for each individual sample to precisely adjust the loss assigned to the sample. In the training stage, a novel hard loss is presented to train the proposed model. Cooperating with the hard metric, the hard loss can up-weight the loss of hard samples and down-weight the loss of easy samples, which results in a more powerful online hard sample mining ability of the proposed model. The experimental results on two real datasets validate the effectiveness and superiority of the proposed method.