The dynamics of rumor spreading is investigated using a model with three kinds of agents who are, respectively, the Seeds, the Agnostics, and the Others. While Seeds are the ones who start spreading the rumor being adamantly convinced of its truth, Agnostics reject any kind of rumor and do not believe in conspiracy theories. In between, the Others constitute the main part of the community. While Seeds are always Believers and Agnostics are always Indifferents, Others can switch between being Believer and Indifferent depending on who they are discussing with. The underlying driving dynamics is implemented via local updates of randomly formed groups of agents. In each group, an Other turns into a Believer as soon as m or more Believers are present in the group. However, since some Believers may lose interest in the rumor as time passes by, we add a flipping fixed rate 0<d<1 from Believers into Indifferents. Rigorous analysis of the associated dynamics reveals that switching from m=1 to m≥2 triggers a drastic qualitative change in the spreading process. When m=1, even a small group of Believers may manage to convince a large part of the community very quickly. In contrast, for m≥2, even a substantial fraction of Believers does not prevent the rumor dying out after a few update rounds. Our results provide an explanation on why a given rumor spreads within a social group and not in another and also why some rumors will not spread in neither groups.