We have used two models based on the valence force field and the regular solution model to study the immiscibility of InAlN ternary alloy, and have got the spinodal and binodal curves of InAlN. Analyzing the spinodal decomposition curves, we obtain the appropriate concentration region for the epitaxial growth of the InN-AlN pseudobinary alloy. At a temperature most common for the epitaxial growth of InAlN (1000 K), the solubility of InN is about 10%. Then we introduce the mismatch strain item into the Gibbs free energy, and the effect of different substrates is taken into consideration. Considering Si, Al2O3, InN, GaN, AlN as a substrate respectively, it is found that all the five systems are stabilized with the upper critical solution temperature largely reduced. Finally, InN and GaN are potential substrates for In-rich InAlN, while AlN and GaN substrates are recommended in the Al-rich region. Si and Al2O3 may be ideal substrates for thin InAlN film.