Kidney cancer is one of the common cancers in the world. Automatic segmentation of the kidney and kidney tumor from CT images is of great significance for the therapy treatment of kidney cancer. Due to the diversity of the kidney tumor in terms of location, size, and shape, current methods have limited performance on the tumor segmentation, especially on the boundary. This paper proposes an effective deep neural network with multi-task learning paradigm to improve the boundary segmentation accuracy. The network is an improved U-Net model enhanced by a novel boundary attention mechanism, named boundary attention U-Net (BAU-Net). It consists of a main branch to segment the target regions and an auxiliary branch to generate boundary attention maps to boost the segmentation. Our method is an extension of our original competition method, in which we improve the classical coarse-to-fine framework by using kidney information to segment tumors. Our method is quantitatively evaluated on a public dataset from MICCAI 2021 Kidney and Kidney Tumor Segmentation Challenge (KiTS2021), with mean Dice similar-ity coefficient (DSC) as 98.04% and 84.09% for the kidney and kidney tumor respectively, outperforming all competitive methods of KiTS2021. Clinical relevance---The proposed method based on deep learning can automatically segment kidneys and kidney tumors with high accuracy, which can be used in the computer-aided diagnosis (CAD) system to assist clinicians in the nephrectomy.