Simultaneous localization and mapping (SLAM) is widely used in autonomous driving and intelligent robot positioning and navigation. In order to overcome the defects of traditional visual SLAM in rapid motion and bidirectional loop detection, we present a feature-based PAL-SLAM system for a panoramic-annular-lens (PAL) camera in this paper. We use a mask to extract and match features in the annular effective area of the images. A PAL-camera model, based on precise calibration, is used to transform the matched features onto a unit vector for subsequent processing, and a prominent inlier-checking metric is designed as an epipolar constraint in the initialization. After testing on large-scale indoor and outdoor PAL image dataset sequences, comprising of more than 12,000 images, the accuracy of PAL-SLAM is measured as typically below 1 cm. This result holds consistent in conditions when the camera rotates rapidly, or the Global Navigation Satellite System (GNSS) signals are blocked. PAL-SLAM can also detect unidirectional and bidirectional loop closures. Hence it can be used as a supplement or alternative to expensive commercial navigation systems, especially in urban environments where there are many signal obstructions such as buildings and bridges.