Flow-induced aggregation of colloidal particles leads to aggregates with fairly high fractal dimension (df approximately 2.4-3.0) which are directly responsible for the observed rheological properties of sheared dispersions. We address the problem of the decrease in aggregate size with increasing hydrodynamic stress, as a consequence of breakup, by means of a fracture-mechanics model complemented by experiments in a multipass extensional (laminar) flow device. Evidence is shown that as long as the inner density decay with linear size within the aggregate (due to fractality) is not negligible (as for df approximately 2.4-2.8), this imposes a substantial limitation to the hydrodynamic fragmentation process as compared with nonfractal aggregates (where the critical stress is practically size independent). This is due to the fact that breaking up a fractal object leads to denser fractals which better withstand stress. In turbulent flows, accounting for intermittency introduces just a small deviation with respect to the laminar case, while the model predictions are equally in good agreement with experiments from the literature. Our findings are summarized in a diagram for the breakup exponent (governing the size versus stress scaling) as a function of fractal dimension.