Hardware artificial neural network (ANN) systems with high density synapse array devices can perform massive parallel computing for pattern recognition with low power consumption. To implement a neuromorphic system with on-chip training capability, we need to develop an ideal synapse device with various device requirements, such as scalability, MLC characteristics, low power operation, data retention, and symmetric/linear conductance changes under potentiation/depression modes. Although various devices have been proposed for synapse applications, they have limitations for application in neuromorphic systems. In this paper, we will cover various RRAM synapse devices, such as filamentary switching RRAM (HfOx, TaOx, Cu-CBRAM) and analog RRAM devices, based on interface resistive switching (Pr0.7Ca0.3MnOx and TiOx) and ferroelectric polarization (HfZrOx). By optimizing potentiation/depression conditions, we could improve the conductance linearity and MLC characteristics of filamentary synapse devices. Interface RRAM has better MLC characteristics with limited retention and conductance linearity. By controlling the reactivity of metal electrodes and the oxygen concentration in oxides, we can modulate the synapse characteristics. Metal-Ferroelectric-Insulator-Semiconductor (MFIS) FET devices exhibit good retention characteristics and analog memory characteristics due to polarization. Based on various synapse device characteristics, we have estimated the pattern recognition accuracy of MNIST handwritten digits and CIFAR-10 datasets. We have confirmed that synapse device characteristics directly affect the pattern recognition accuracy of ANNs. In order to simultaneously satisfy all the requirements of synapse devices, it is necessary to develop new technology capable of controlling the movement of oxygen vacancies and metal ions at the atomic scale. Considering the limited synapse characteristics of current 2-terminal RRAM devices, hardware ANNs capable of only off-chip training can be constructed by optimizing the current RRAM devices by limiting the bit number. A 3-terminal synapse device or a device based on a new operation principle should be developed as an alternative for on-chip training applications.