There are intraindividual differences in empathic tendencies across situations. Empathy research, however, has traditionally focused on differences between people. This between-person approach does not necessarily describe the structure of empathy for individual people. Additionally, it is unclear whether the valence of affect perceived in the target person moderates the empathic process. Through a microlongitudinal experience sampling study (N = 117 adults, Nobservations = 6,307) we investigated within-person variation in empathy. Applying a variance component analysis, we found substantial individual differences in affect-specific empathy. Then, using multilevel structural equation modeling, we developed a within-person and between-person model of affect-specific empathy, identifying different factor correlation patterns at the within-person and between-person levels. For the within-person level, participants mostly empathized cognitively and affectively either for positive or negative affect, but rarely at the same time. For the between-person level, we found separate affect-specific dimensions for affective empathy, indicating that in general, individuals report affective empathy for one affect (e.g., positive) more than the other (e.g., negative). However, there was no affect-specific differentiation for cognitive empathy. Beyond the average within-person model for all participants combined, we found large differences between persons in the extent to which they simultaneously experienced positive and/or negative affective and/or cognitive empathy. Implications and future directions are discussed. (PsycInfo Database Record (c) 2023 APA, all rights reserved).