When dealing with system-reservoir interactions in an open quantum system, such as a photosynthetic light-harvesting complex, approximations are usually made to obtain the dynamics of the system. One question immediately arises: how good are these approximations, and in what ways can we evaluate them? Here, we propose to use entanglement and a measure of non-Markovianity as benchmarks for the deviation of approximate methods from exact results. We apply two frequently-used perturbative but non-Markovian approximations to a photosynthetic dimer model and compare their results with that of the numerically-exact hierarchy equation of motion (HEOM). This enables us to explore both entanglement and non-Markovianity measures as means to reveal how the approximations either overestimate or underestimate memory effects and quantum coherence. In addition, we show that both the approximate and exact results suggest that non-Markonivity can, counter-intuitively, increase with temperature, and with the coupling to the environment.