Although treatment options for localized prostate cancer (CaP) are initially effective, the five-year survival for metastatic CaP is below 30%. Mutation or deletion of the PTEN tumor suppressor is a frequent event in metastatic CaP, and inactivation of the transforming growth factor (TGF) ß signaling pathway is associated with more advanced disease. We previously demonstrated that mouse models of CaP based on inactivation of Pten and the TGFß type II receptor (Tgfbr2) rapidly become invasive and metastatic. Here we show that mouse prostate tumors lacking Pten and Tgfbr2 have higher expression of stem cell markers and genes indicative of basal epithelial cells, and that basal cell proliferation is increased compared to Pten mutants. To better model the primarily luminal phenotype of human CaP we mutated Pten and Tgfbr2 specifically in luminal cells, and found that these tumors also progress to invasive and metastatic cancer. Accompanying the transition to invasive cancer we observed de-differentiation of luminal tumor cells to an intermediate cell type with both basal and luminal markers, as well as differentiation to basal cells. Proliferation rates in these de-differentiated cells were lower than in either basal or luminal cells. However, de-differentiated cells account for the majority of cells in micro-metastases consistent with a preferential contribution to metastasis. We suggest that active TGFß signaling limits lineage plasticity in prostate luminal cells, and that de-differentiation of luminal tumor cells can drive progression to metastatic disease.