Pancharatnam-Berry geometric phase has attracted enormous interest in subwavelength optics and electromagnetics during the past several decades. Traditional theory predicts that the geometric phase is equal to twice the rotation angle of anisotropic elements. Here, we show that high-order geometric phases equal to multiple times the rotation angle could be achieved by meta-atoms with highfold rotational symmetries. As a proof of concept, the broadband angular spin Hall effect of light and optical vortices is experimentally demonstrated by using plasmonic metasurfaces consisting of space-variant nanoapertures with C2, C3, and C5 rotational symmetries. The results provide a fundamentally new understanding of the geometric phase as well as light-matter interaction in nanophotonics.