Previous studies have proved that the vulnerable plaque is a major factor leading to the onset of acute coronary syndrome (ACS). Recognizing vulnerable plaques is essential for cardiologists to treat illnesses, early. However, this task often comes with the challenge of insufficient annotated data sets and subtle differences between lesion regions and normal regions. In this paper, we apply the visual attention model with deep neural network to improve the performance of recognizing vulnerable plaques. There are two key ideas about our method: 1) using a top-down attention model to extract salient regions (blood vessels) according to the doctor's prior knowledge, and 2) employing a multi-task neural network to complete the recognition task. The first branch, a typical classification task, is to distinguish whether the image contains vulnerable plaques. The other branch uses a column-wise segmentation to locate vulnerable plaques. We have verified the effectiveness of our proposed method on the data set provided by 2017 CCCV-IVOCT Challenge. The proposed method obtains good performance.