A detailed x-ray scattering and high-resolution ac calorimetric study has been carried out near the smectic-A to chiral smectic-C phase transition of liquid-crystal compounds 4-(2-methyl butyl) phenyl 4-n-octylbiphenyl-4-carboxylate (CE8) and p-( n-decyloxy) benzylidene-p-amino-(2-methylbutyl) cinnamate (DOBAMBC) confined in hydrophilic and hydrophobic aerosil nanoparticle networks. The character of the transition, which is mean field near a tricritical point in bulk, is changed dramatically with an increase of aerosil-induced disorder. X-ray measurements revealed pretransitional behavior and compression of the smectic layers, phenomena that are strongly pronounced in high aerosil concentrations. A theoretical model that takes into account the interplay of relevant mechanisms is proposed to explain the observed phenomena. The effect of chirality on the interaction of liquid crystals with aerosils is discussed.