Dual-task gait systems can be utilized to assess elderly patients for cognitive decline. Although numerous research studies have been conducted to estimate cognitive scores, this field still faces two significant challenges. Firstly, it is crucial to fully utilize dual-task cost representations for diagnosis. Secondly, the design of optimal strategies for effectively extracting dual-task cost representations remains a challenge. To address these issues, in this paper, we propose a deep learning-based framework that implements a spatio-temporal graph convolutional neural network (ST-GCN) with single-task and dual-task pathways for cognitive impairment detection in gait. We also introduce a novel loss, termed task-specific loss, to ensure that single-task and dual-task representations are distinguishable from each other. Furthermore, dual-task cost representations are calculated as the difference between dual-task and single-task representations, which are resilient to individual differences and contribute to the robustness of the framework. These representations provide a comprehensive view of single-task and dual-task gait information to generate task predictions. The proposed framework outperforms existing approaches with a sensitivity of 0.969 and a specificity of 0.940 for cognitive impairment detection.