BCS-Bose-Einstein condensation (BEC) crossover is effected by increasing pairing strength between fermions from weak to strong in the particle-particle channel, and has attracted a lot of attention since the experimental realization of quantum degenerate atomic Fermi gases. Here we study the effect of the (often dropped) particle-hole channel on the zero T gap Δ(0), superfluid transition temperature Tc, the pseudogap at Tc, and the mean-field ratio 2Δ(0)/, from BCS through BEC regimes, using a pairing fluctuation theory which includes self-consistently the contributions of finite-momentum pairs and features a pseudogap in single particle excitation spectrum. Summing over the infinite particle-hole ladder diagrams, we find a complex dynamical structure for the particle-hole susceptibility χph, and conclude that neglecting the self-energy feedback causes a serious over-estimate of χph. While our result in the BCS limit agrees with Gor'kov et al., the particle-hole channel effect becomes more complex and pronounced in the crossover regime, where χph is reduced by both a smaller Fermi surface and a big (pseudo)gap. Deep in the BEC regime, the particle-hole channel contributions drop to zero. We predict a density dependence of the magnetic field at the Feshbach resonance, which can be used to quantify χph and test different theories.