Surface functionalized ultrafine CoFe2O4 nanoparticles (NPs), with mean diameter ∼5 nm, were investigated by means of DC magnetization and AC susceptibility over the temperature range of 4-400 K. All NPs present the same CoFe2O4 core, with different molecular surface coatings, increasing gradually the number of carbon atoms in the coating layer: glycine (C2H5NO2), alanine (C3H7NO2), aminobutanoic acid (C4H9NO2), aminohexanoic acid (C6H13NO2), and aminododecanoic acid (C12H25NO2). Samples were intentionally fabricated in order to modulate the core-core magnetic dipolar interaction, as the thickness of the coating layer increases with the number of carbon atoms in the coating molecule. The magnetic data of the uncoated CoFe2O4 NPs were also collected for comparison. All investigated CoFe2O4 NPs (coated and uncoated) are in a magnetically blocked state at room temperature as evidenced by ZFC/FC measurements and the presence of hysteresis with ∼700 Oe coercivity. Low temperature magnetization scans show slightly constricted hysteresis loops with coercivity decreasing systematically with a decreasing number of carbon atoms in the coating molecule, possibly resulting from differences in magnetic dipole coupling between NPs. Large thermomagnetic irreversibility, slow monotonic increase in the FC magnetization and non-saturation of the magnetization give evidence for the cluster glass (CG) nature in the CoFe2O4 NPs. The out of phase part (χ'') of AC susceptibility for all samples shows a clear frequency dependent hump which was analyzed to distinguish superparamagnetic (SPM), cluster glass (CG) and spin glass (SG) behavior by using Néel-Arrhenius, Vogel-Fulcher, and power law fittings. These analyses rule out the SPM state and suggest the presence of significant inter-cluster dipolar interaction, giving rise to CG cooperative freezing in the high-temperature region. In the low-temperature range, however, the disordered spins on the nanoparticle's surface play an important role in the formation of the SG-like state, as evidenced by Arrott plots and temperature dependency of dM/dH in the initial magnetization curves. In summary, the magnetic measurements showed that undercooling the system evolves from a SPM state of weakly interacting spin clusters, through the CG state induced by strong dipolar interaction, to the SG state resulting from the frustration of the disordered surface spins.