Liver-infiltrating T cells have been implicated in the pathogenesis of primary sclerosing cholangitis (PSC), however little information is available about changes in other cellular compartments in the liver during PSC. This study aimed to characterize non-parenchymal intrahepatic cells in PSC livers and to find associations between phenotypes and disease severity. Using immunohistochemistry, followed by automated image analysis and quantification and a principal component analysis, we have studied non-parenchymal intrahepatic cells in PSC-patient livers (n = 17) and controls (n = 17). We observed a significant increase of T cells in the PSC patients, localized to the fibrotic areas. MAIT cells, normally present at high numbers in the liver, were not increased to the same extent. PSC patients had lower expression of MHC class I than controls. However, the levels of NKp46+ NK cells were similar between patients and controls, nevertheless, NKp46 was identified as a phenotypic marker that distinguished PSC patients with mild from those with severe fibrosis. Beyond that, a group of PSC patients had lost expression of Caldesmon and this was associated with more extensive bile duct proliferation and higher numbers of T cells. Our data reveals phenotypic patterns in PSC patients associated with disease severity.