We analytically study the vortex formation associated with the classical periodic orbits in a nonseparable quantum billiard. The influence of vortex formation on the quantum flux in the mesoscopic region is clearly demonstrated. Remarkably, the high-order vortex structures in an equilateral triangle billiard display the triangular, swirl, and kagomè vortex lattices. The relationship between various vortex lattices is found to rely on the relative phase factor in the coherent state.