The human placenta is a highly invasive tumor-like structure in which a subpopulation of placental trophoblast cells known as the "extravillous trophoblast" (EVT) invades the uterine decidua and its vasculature to establish adequate fetal-maternal exchange of molecules. By utilizing in vitro-propagated short-lived EVT cell lines we found that molecular mechanisms responsible for their invasiveness are identical to those of cancer cells; however, unlike cancer cells, their proliferation, migration, and invasiveness in situ are stringently controlled by decidua-derived transforming growth factor (TGF)-beta. By SV40T antigen transfection of normal EVT cells followed by a forced crisis regimen in culture we produced an immortalized premalignant derivative that is hyperproliferative, hyperinvasive, and deficient in gap-junctional intercellular communication. Both premalignant and malignant EVT (JAR and JEG-3 choriocarcinoma) cell lines were found to be TGF-beta-resistant. Using these cell lines, we investigated genetic changes responsible for transition of the normal EVT cells to premalignant and malignant phenotype. Hyperinvasiveness in both cases resulted from a downregulation of tissue inhibitor of metalloprotease (TIMP)-1 and plasminogen activator inhibitor (PAI)-1 genes. In contrast to normal EVT cells, both cell types failed to upregulate these genes in response to TGF-beta. Loss of TGF-beta response in malignant EVT cells was explained by the loss of expression of Smad3 gene. Differential mRNA display of normal and premalignant EVT cells identified up- and down-regulation of numerous known or novel genes in premalignant EVT cells, with potential oncogenic and (or) tumor-suppressor functions, e.g., loss of fibronectin and insulin-like growth factor binding protein (IGFBP-5). Premalignant EVT cells also lost IGF receptor type 2 (IGFR-II). IGFBP-5 was shown to be a negative regulator of IGF-1-induced proliferation of premalignant EVT cells, so that loss of IGFBP-5 as well as IGFR-II permitted their unrestricted proliferation in an IGF-I-rich microenvironment of the fetal-maternal interface. The present model may be a good prototype for identifying genetic changes underlying epithelial tumor progression.