Measurements of fragmentation functions for jets associated with an isolated photon are presented for the first time in pp and Pb-Pb collisions. The analysis uses data collected with the CMS detector at the CERN LHC at a nucleon-nucleon center-of-mass energy of 5.02 TeV. Fragmentation functions are obtained for jets with p_{T}^{jet}>30 GeV/c in events containing an isolated photon with p_{T}^{γ}>60 GeV/c, using charged tracks with transverse momentum p_{T}^{trk}>1 GeV/c in a cone around the jet axis. The association with an isolated photon constrains the initial p_{T} and azimuthal angle of the parton whose shower produced the jet. For central Pb-Pb collisions, modifications of the jet fragmentation functions are observed when compared to those measured in pp collisions, while no significant differences are found in the 50% most peripheral collisions. Jets in central Pb-Pb events show an excess (depletion) of low (high) p_{T} particles, with a transition around 3 GeV/c. This measurement shows for the first time the in-medium shower modifications of partons (quark dominated) with well-defined initial kinematics. It constitutes a new well-controlled reference for testing theoretical models of the parton passage through the quark-gluon plasma.