In the last two decades, several new concepts for improving the performance of infrared detectors have been proposed. These new concepts particularly address the drive towards the so-called high operating temperature focal plane arrays (FPAs), aiming to increase detector operating temperatures, and as a consequence reduce the cost of infrared systems. In imaging systems with the above megapixel formats, pixel dimension plays a crucial role in determining critical system attributes such as system size, weight and power consumption (SWaP). The advent of smaller pixels has also resulted in the superior spatial and temperature resolution of these systems. Optimum pixel dimensions are limited by diffraction effects from the aperture, and are in turn wavelength-dependent. In this paper, the key challenges in realizing optimum pixel dimensions in FPA design including dark current, pixel hybridization, pixel delineation, and unit cell readout capacity are outlined to achieve a sufficiently adequate modulation transfer function for the ultra-small pitches involved. Both photon and thermal detectors have been considered. Concerning infrared photon detectors, the trade-offs between two types of competing technology-HgCdTe material systems and III-V materials (mainly barrier detectors)-have been investigated.