Methods for regulating peptide conformation by non-harmful light stimuli can be useful for remotely controlling cellular functions in vitro. Here, we synthesized a series of p-heteroatom-substituted azobenzenes and studied their photoisomerization properties. The trans-isomer of p-sulfur-substituted azobenzene was effectively isomerized by visible light irradiation and the cis-isomer was thermally stable at physiological temperature. We developed a novel visible light sensitive amino acid (AZO), via p-sulfur-substituted azobenzene, and utilized it as a photosensitive modulator of the SV40 nuclear localization signal (NLS). The cellular uptake of the AZO-NLS conjugate was controlled by visible light irradiation. Our technology can be utilized for regulating not only the cellular uptake, but also the function of peptides within cells by non-harmful visible light irradiation.