Previous research indicates that different exercise modes might create different effects on cognition and peripheral protein signals. This study aimed to compare the effects of long-term participation in an open and closed-skill exercise on cognitive functions and Brain-derived neurotrophic factor and Cathepsin B levels. 18 fencers, 18 swimmers, 18 sedentary controls between 18-25 years old participated in the study. Participants performed visuospatial working memory, verbal fluency and selective attention tasks. Blood samples were tested for Brain-derived neurotrophic factor and Cathepsin B using ELISA. The results showed that fencers performed superiorly on some part of visuospatial working memory, verbal fluency, and selective attention tasks than swimmers and sedentary controls. Athlete groups showed higher scores on some subtests of visuospatial working memory and selective attention tasks than sedentary controls. The basal serum Brain-derived neurotrophic factor level was not significant between the groups, but Cathepsin B was higher in fencers than swimmers and sedentary controls. The peripheric protein signal response to acute exercise was significantly higher in athletes, particularly in the open-skill group for Cathepsin B. Our research provided noteworthy results that more cognitively challenging exercise may provide more benefits for some aspects of cognition. Since our findings suggest that open-skill exercise improves specific types of executive-control functioning, this exercise mode might be included in training programs to support cognition and prevent cognitive impairment.