By using the strong nonlinear effect and ultrafast electronic response of cholesteric liquid crystals (CLC), ultrafast all optical switching between polarization vortex and phase vortex is realized in a system combining CLC and q-plate. The experimental result shows that switching with high modulation depth can be accomplished in less than 1 picosecond. Furthermore, CLC and q-plates will enable compact integrated devices with sub-mm thicknesses.