Interpersonal physiological synchrony has been shown to play important roles in social activities. While most studies have shed light on the effects of physiological synchrony on recognition of the group state, such as cohesion or togetherness, the effect of physiological synchrony on the recognition of emotional experience has not been adequately researched. In this study, we examined how physiological synchrony is associated with first- and third-person emotion recognition during a joint task. Two participants played a cooperative block-stacking game (Jenga), alternating their roles as player and adviser, while their heart rates were recorded. The participants evaluated their own emotional experience for each turn. Bystanders watched the game to evaluate the players' emotions. Results showed that the players' subjective excitement increased not only with their own heart rate, but also with increased heart rate synchrony with their adviser. Heart rate synchrony between player and adviser also related to increased intensity in perceived excitement from the bystanders. Given that both first- and third-person emotion recognition can have cumulative impacts on a group, the relationship between physiological synchrony and emotion recognition observed in the present study will help deepen understanding of the psychophysiological mechanisms underlying larger group phenomena such as crowd excitement.