The missing member of the rosiaite family, CoGeTeO6, was synthesized by mild ion-exchange reactions and characterized by magnetization M and specific heat Cp measurements. It exhibits a successive short- and long-range magnetic ordering at Tshort-range ≈ 45 K and TN = 15 K, respectively. Based on these measurements, the magnetic H-T phase diagram was established, showing two antiferromagnetic phases separated by a spin-flop transition. The reason why the pronounced short-range correlation occurs at a temperature nearly three times higher than TN was found by evaluating the Co-O⋯O-Co exchange interactions using energy-mapping analysis. Although CoGeTeO6 has a layered structure, its magnetic structure consists of three-dimensional antiferromagnetic lattices made up of rhombic boxes of Co2+ ions. The experimental data obtained at high temperatures agree well with the computational results by treating the Co2+ ions of CoGeTeO6 as S = 3/2 ions, but the heat capacity and magnetization data were obtained at low temperatures by treating the Co2+ ion as a Jeff = 1/2 ion.