The formation of primordial black holes from inflationary fluctuations is accompanied by a scalar induced gravitational wave background. We perform a Bayesian search of such background in the data from Advanced LIGO and Virgo's first, second, and third observing runs, parametrizing the peak in the curvature power spectrum by a log-normal distribution. The search shows no evidence for such a background. We place 95% confidence level upper limits on the integrated power of the curvature power spectrum peak which, for a narrow width, reaches down to 0.02 at 10^{17} Mpc^{-1}. The resulting constraints are stronger than those arising from big bang nucleosynthesis or cosmic microwave background observations. In addition, we find that the constraints from LIGO and Virgo, at their design sensitivity, and from the Einstein Telescope can compete with those related to the abundance of the formed primordial black holes.