Reference-frame-independent quantum key distribution (RFI-QKD) provides a practical way to generate secret keys between two remote parties without sharing common reference frames. On the other hand, measurement-device-independent QKD (MDI-QKD) offers a high level of security, as it is immune to all quantum hacking attempts to measurement devices. The combination of these two QKD protocols, i.e., RFI-MDI-QKD, is one of the most fascinating QKD protocols, since it holds advantages of both practicality and security. For further practicality of RFI-MDI-QKD, it is beneficial to reduce the implementation complexity. Here, we show that RFI-MDI-QKD can be implemented using fewer quantum states than those of its original proposal. We find that, in principle, the number of quantum states for one of the parties can be reduced from six to three without compromising security. Compared to conventional RFI-MDI-QKD where both parties transmit six quantum states, it significantly simplifies the implementation of the QKD protocol. We also verify the feasibility of the scheme with a proof-of-principle experiment.