X-ray grating interferometry, which has been spotlighted in the last decade as a multi-modal X-ray imaging technique, can provide three independent images, i.e., absorption, differential-phase, and visibility-contrast images. We report on a cause of the visibility contrast, an effect of insufficient temporal coherence, that can be observed when continuous-spectrum X-rays are used. This effect occurs even for a sample without unresolvable random structures, which are known as the main causes of visibility contrast. We performed an experiment using an acrylic cylinder and quantitatively explained the visibility contrast due to this effect.