We present an experimental realization of a robust quantum communication scheme [Phys. Rev. Lett. 93, 220501 (2004)] using pairs of photons entangled in polarization and time. Our method overcomes errors due to collective rotation of the polarization modes (e.g., birefringence in optical fiber or misalignment), is insensitive to the phase's fluctuation of the interferometer, and does not require any shared reference frame including time reference, except the need to label different photons. The practical robustness of the scheme is further shown by implementing a variation of the Bennett-Brassard 1984 quantum key distribution protocol over 1 km optical fiber.