Polyelectrolytes may be classified into two primary categories (strong and weak) depending on how their charge state responds to the local environment. Both of these find use in many applications, including drug delivery, gene therapy, layer-by-layer films, and fabrication of ion filtration membranes. The mechanism of polyelectrolyte complexation is, however, still not completely understood, though experimental investigations suggest that entropy gain due to release of counterions is the key driving force for strong polyelectrolyte complexation. Here we perform a comprehensive thermodynamic investigation through coarse-grained molecular simulations permitting us to calculate the free energy of complex formation. Importantly, our expanded-ensemble methods permit the explicit separation of energetic and entropic contributions to the free energy. Our investigations indicate that entropic contributions indeed dominate the free energy of complex formation for strong polyelectrolytes, but are less important than energetic contributions when weak electrostatic coupling or weak polyelectrolytes are present. Our results provide a new view of the free energy of polyelectrolyte complex formation driven by polymer association, which should also arise in systems with large charge spacings or bulky counterions, both of which act to weaken ion-polymer binding.