Hot-carrier photodetectors are drawing significant attention; nevertheless, current researches focus mostly on the hot-electron devices, which normally show low quantum efficiencies. In contrast, hot-hole photodetectors usually have lower barriers and can provide a wide spectral range of photodetection and an improved photoconversion efficiency. Here, we report a comparable study of the hot-electron and hot-hole photodetectors from both underlying physics and optoelectronic performance perspectives. Taking the typical Au/Si Schottky contact as an example, we find obvious differences in the energy band diagram and the sequent hot-carrier generation/transport/emission processes, leading to very distinguished photodetection performances. Compared with hot electrons, hot holes show higher density below the Fermi level, the longer mean free path arising under the lower electron-electron and electron-phonon scatterings, a lower barrier height, and a lighter effective mass in Si, all of which lead to larger number of high-energy hot holes, larger transport probability, higher emission efficiency, and higher photoresponsivity. However, the low barrier height can cause poor performances of hot-hole device in dark current density and detectivity. The study elucidates the intrinsic physical differences and compares the key performance parameters of the hot-hole and hot-electron photodetections, with the objective of providing complete information for designing hot-carrier devices.