We investigated the switching dynamics of optical modulators consisting of a Si waveguide with a VO2 cladding layer by utilizing the photothermal effect, which induces a metal-insulator transition in VO2. The devices exhibited stable optical switching with a high extinction ratio exceeding 16 dB. The switching time of the insulator-to-metal transition (heating process) ranged from tens of nanoseconds to microseconds depending on the incident light power, and that of the metal-to-insulator transition (cooling process) was several microseconds regardless of the incident light power. The heat transfer in the devices was numerically simulated to reproduce the switching characteristics and revealed that the temperature change in the first few micrometers of the VO2/Si waveguide governed the switching time. The thermal structural design of the device is thus of key importance to improve the switching speed of the device.