Existing tumor growth models based on fluid analogy for the cells do not generally include the extracellular matrix (ECM), or if present, take it as rigid. The three-fluid model originally proposed by the authors and comprising tumor cells (TC), host cells (HC), interstitial fluid (IF) and an ECM, considered up to now only a rigid ECM in the applications. This limitation is here relaxed and the deformability of the ECM is investigated in detail. The ECM is modeled as a porous solid matrix with Green-elastic and elasto-visco-plastic material behavior within a large strain approach. Jauman and Truesdell objective stress measures are adopted together with the deformation rate tensor. Numerical results are first compared with those of a reference experiment of a multicellular tumor spheroid (MTS) growing in vitro, then three different tumor cases are studied: growth of an MTS in a decellularized ECM, growth of a spheroid in the presence of host cells and growth of a melanoma. The influence of the stiffness of the ECM is evidenced and comparison with the case of a rigid ECM is made. The processes in a deformable ECM are more rapid than in a rigid ECM and the obtained growth pattern differs. The reasons for this are due to the changes in porosity induced by the tumor growth. These changes are inhibited in a rigid ECM. This enhanced computational model emphasizes the importance of properly characterizing the biomechanical behavior of the malignant mass in all its components to correctly predict its temporal and spatial pattern evolution.