Driving assistance systems that support drivers by adapting to driver characteristics can provide appropriate feedback and prevent traffic accidents. Cognitive function is helpful information for such systems to assist older drivers, and automatic estimation of drivers' cognitive function enables systems to utilize this information without being burdensome to these drivers. Therefore, this study aims to estimate drivers' cognitive function from daily driving data. We focus on modeling the scores of Trail Making Test (A) and (B) as measures of cognitive function, which indicate general cognitive ability. The main challenge is learning the generalized mapping function to the cognitive status from driving behavioral features extracted from the different driving routes of each driver. To address this problem, the proposed method focuses on particular driving scenarios in which differences in cognitive function can be observed. We evaluate the performance of the proposed model and the effectiveness of driving scenario information. Experimental results show that the results of Trail Making Tests (A) and (B) can be estimated with Spearman rank correlation coefficients of r = 0.34 and 0.48, respectively. In addition, the proposed method makes it easier to analyze the relationships between driving behaviors and cognitive function by comparing driving behaviors (e.g., steering angle velocity) in specific driving scenarios (e.g., intersections).