Understanding the self-assembly of conjugated organic materials at the molecular level is crucial in their potential applications as active components in electronic and optoelectronic devices. The type of aggregation significantly influences the intriguing electronic and optical characteristics differing from their constituent molecules. Perylenediimides (PDIs), electron-deficient molecules exhibiting remarkable n-type semiconducting properties, are among the most explored organic fluorescent materials due to their high fluorescence efficiency, photostability, and optoelectronic properties. PDI derivatives are reported to form well-tailored supramolecular architectures: cofacial with minor slip (H-aggregates), staggered with major slip (J-aggregates), magic angle stacking (M-aggregates), rotated (X-aggregates), rotated orthogonal ((+)-aggregates), etc. H*-aggregates are defined here as an ideal case of H-aggregate with an eclipsed configuration. Although numerous reports regarding the formation and optical properties of various PDI aggregates are known, the key driving force within the PDI units guiding the self-assembly to form distinct aggregate systems remains elusive. To unravel the molecular-level mechanisms behind the self-assembly of PDI units by probing the intermolecular interactions, symmetry-adapted perturbation theory-based energy decomposition, potential energy surface scans, and non-covalent interaction index analyses were employed on PDI dimer models. Quantum theory of atoms in molecules and frontier molecular orbital analyses were implemented on the dimer models to comprehend the effect of heteroatoms and orbital interactions in stabilising the X-aggregates over the other PDI aggregate systems. Competition between the attractive and repulsive non-covalent interactions dictates a stability order of X > H > J > M > (+) > H* for the PDI aggregate system, while in the parent perylene system, the stability order was found to be X > (+) > H > M > J > H*.