Pedestrian trajectory prediction is an important technique of autonomous driving. In order to accurately predict the reasonable future trajectory of pedestrians, it is inevitable to consider social interactions among pedestrians and the influence of surrounding scene simultaneously, which can fully represent the complex behavior information and ensure the rationality of predicted trajectories obeyed realistic rules. In this article, we propose one new prediction model named social soft attention graph convolution network (SSAGCN), which aims to simultaneously handle social interactions among pedestrians and scene interactions between pedestrians and environments. In detail, when modeling social interaction, we propose a new social soft attention function, which fully considers various interaction factors among pedestrians. Also, it can distinguish the influence of pedestrians around the agent based on different factors under various situations. For the scene interaction, we propose one new sequential scene sharing mechanism. The influence of the scene on one agent at each moment can be shared with other neighbors through social soft attention; therefore, the influence of the scene is expanded both in spatial and temporal dimensions. With the help of these improvements, we successfully obtain socially and physically acceptable predicted trajectories. The experiments on public available datasets prove the effectiveness of SSAGCN and have achieved state-of-the-art results. The project code is available at.