Dynamical mean field theory (DMFT), combined with finite-temperature exact diagonalization, is one of the methods used to describe electronic properties of strongly correlated materials. Because of the rapid growth of the Hilbert space, the size of the finite bath used to represent the infinite lattice is severely limited. In view of the increasing interest in the effect of multi-orbital and multi-site Coulomb correlations in transition metal oxides, high-T(c) cuprates, iron-based pnictides, organic crystals, etc, it is appropriate to explore the range of temperatures and bath sizes in which exact diagonalization provides accurate results for various system properties. On the one hand, the bath must be large enough to achieve a sufficiently dense level spacing, so that useful spectral information can be derived, especially close to the Fermi level. On the other hand, for an adequate projection of the lattice Green's function onto a finite bath, the choice of the temperature is crucial. The role of these two key ingredients in exact diagonalization DMFT is discussed for a wide variety of systems in order to establish the domain of applicability of this approach. Three criteria are used to illustrate the accuracy of the results: (i) the convergence of the self-energy with the bath size, (ii) the quality of the discretization of the bath Green's function, and (iii) comparisons with complementary results obtained via continuous-time quantum Monte Carlo DMFT. The materials comprise a variety of three-orbital and five-orbital systems, as well as single-band Hubbard models for two-dimensional triangular, square and honeycomb lattices, where non-local Coulomb correlations are important. The main conclusion from these examples is that a larger number of correlated orbitals or sites requires a smaller number of bath levels. Down to temperatures of 5-10 meV (for typical bandwidths W ≈ 2 eV) two bath levels per correlated impurity orbital or site are usually adequate.