Linearly polarized vectorial vortices are analyzed according to their Pancharatnam phase and experimentally demonstrated using a geometric phase element consisting of space-variant subwavelength gratings. It is shown that in the absence of a Pancharatnam phase, stable vectorial vortices that have no angular momentum arise. In contrast, if a Pancharatnam phase is present the vectorial vortices have orbital angular momentum and collapse upon propagation.