Classification of molecular structures is a crucial step in molecular dynamics (MD) simulations to detect various structures and phases within systems. Molecular structures, which are commonly identified using order parameters, were recently identified using machine learning (ML), that is, the ML models acquire structural features using labeled crystals or phases via supervised learning. However, these approaches may not identify unlabeled or unknown structures, such as the imperfect crystal structures observed in nonequilibrium systems and interfaces. In this study, we proposed the use of a novel unsupervised learning framework, denoted temporal self-supervised learning (TSSL), to learn structural features and design their parameters. In TSSL, the ML models learn that the structural similarity is learned via contrastive learning based on minor short-term variations caused by perturbations in MD simulations. This learning framework is applied to a sophisticated architecture of graph neural network models that use bond angle and length data of the neighboring atoms. TSSL successfully classifies water and ice crystals based on high local ordering, and furthermore, it detects imperfect structures typical of interfaces such as the water-ice and ice-vapor interfaces.