Efficient and cost-effective electrocatalysts for the hydrogen evolution reaction (HER) and oxygen evolution reaction (OER), especially bifunctional catalysts for overall water splitting, are highly desired. In this work, with rationally designed sandwich-type metal-organic framework/graphene oxide as a template and precursor, a layered CoP/reduced graphene oxide (rGO) composite has been successfully prepared via pyrolysis and a subsequent phosphating process. The resultant CoP/rGO-400 exhibits excellent HER activity in acid solution. More importantly, the catalyst manifests excellent catalytic performances for both the HER and OER in basic solution. Therefore, it can be utilized as a bifunctional catalyst on both the anode and cathode for overall water splitting in basic media, even displaying superior activity to that of the integrated Pt/C and IrO2 catalyst couple.