The panoramic annular lens (PAL) system can capture plentiful scene information in real time. The locally described freeform surface can provide more degrees of freedom for the optical system design than the globally described one. In this paper, we propose a locally described annular Gaussian radial basis function freeform surface that enables a high-performance yet compact PAL system. The designed PAL system achieves an ultra-wide field of view (FOV) of (30∘∼125∘)×360∘. The F-theta distortion of the system is less than 3%, and the modulation transfer function across the whole FOV is greater than 0.5 at 100 lp/mm. The proposed system can be implemented for various applications such as video conferencing or robotic navigation.