Colloidal particles suspended in a binary liquid mixture can interact via solvent mediated interactions, known as critical Casimir forces. For anisotropic colloids this interaction becomes directional, which leads to rich phase behavior. While experimental imaging and particle tracking techniques allow determination of isotropic effective potentials via Boltzmann inversion, the modeling of effective interaction in anisotropic systems is non-trivial precisely because of this directionality. Here we extract effective interaction potentials for non-spherical dumbbell particles from observed radial and angular distributions, by employing reference interaction site model (RISM) theory and direct Monte Carlo simulations. For colloidal dumbbell particles dispersed in a binary liquid mixture and interacting via induced critical Casimir forces, we determine the effective site-site potentials for a range of experimental temperatures. Using these potentials to simulate the system for strong Casimir forces, we reproduce the experimentally observed collapse, and provide a qualitative explanation for this behavior.