We consider the Dicke model in the ultrastrong-coupling limit to investigate thermal phase transitions and their precursors at finite particle numbers N for bosonic and fermionic systems. We derive partition functions with degeneracy factors that account for the number of configurations and derive explicit expressions for the Landau free energy. This allows us to discuss the difference between the original Dicke (fermionic) and the bosonic case. We find a crossover between these two cases that shows up, for example, in the specific heat.