We study the ground state (GS) and the phase transition in a hexagonal-close-packed lattice with both XY and Ising models by using extensive Monte Carlo simulation. We suppose the in-plane interaction J1 and interplane interaction J2, both antiferromagnetic. The system is frustrated with two kinds of GS configuration below and above a critical value of η=J1/J2 (ηc). For the Ising case, one has ηc=0.5 which separates in-plane ferromagnetic and antiferromagnetic states, while for the XY case ηc=1/3 separates the collinear and noncollinear spin configurations. The phase transition is shown to be of first (second) order for η>(<)ηc. The phase diagram in the space (η,T) is shown for both cases.