The weakening of the geomagnetic field causes an increase in galactic cosmic ray (GCR) flux. Some researchers argue that enhanced GCR flux might lead to a climatic cooling by increasing low cloud formation, which enhances albedo (umbrella effect). Recent studies have reported geological evidence for a link between weakened geomagnetic field and climatic cooling. However, more work is needed on the mechanism of this link, including whether the umbrella effect is playing a central role. In this research, we present new geological evidence that GCR flux change had a greater impact on continental climate than on oceanic climate. According to pollen data from Osaka Bay, Japan, the decrease in temperature of the Siberian air mass was greater than that of the Pacific air mass during geomagnetic reversals in marine isotope stages (MIS) 19 and 31. Consequently, the summer land-ocean temperature gradient was smaller, and the summer monsoon was weaker. Greater terrestrial cooling indicates that a reduction of insolation is playing a key role in the link between the weakening of the geomagnetic field and climatic cooling. The most likely candidate for the mechanism seems to be the increased albedo of the umbrella effect.