Panoramic annular lens (PAL) is a type of special optical system that can transform the cylindrical side view onto a planar annular image. We propose a compact polarization-based dual-view panoramic lens to solve the two major limitations of the conventional PAL structure: large front annular lens and lack of forward view. Dual-view imaging capability, which is highly needed for some applications, is achieved by using a polarization technique. The diameter ratio of the proposed front PAL lens and the sensor is reduced to less than 1.5, while it is larger than 2.5 for traditional PAL systems, expanding its applications in space-confined environments.