Most patients with rectal cancer receive neoadjuvant radiochemotherapy (RCT), causing a variable decrease in tumor mass. We evaluated the prognostic impact of pathologic parameters reflecting tumor response to RCT, either directly or indirectly. Seventy-six rectal cancer patients receiving neoadjuvant RCT between 2006 and 2009 were included. We studied the association between disease-free survival (DFS) and the "classical" clinicopathologic features as well as tumor deposits, circumferential resection margin (CRM), Dworak regression grade, and tumor and nodal downstaging. Patients with tumor downstaging had a longer DFS (p = 0.05), indicating a more favorable prognosis when regression was accompanied by a decrease in tumor infiltrative depth, referred to as tumor shrinkage. Moreover, tumor downstaging was significantly associated with larger CRM and nodal downstaging (p = 0.02), suggesting that shrinkage of the primary tumor was associated with a decreased nodal tumor load. Higher Dworak grade did not correlate with tumor downstaging, nor with higher CRM or prolonged DFS. This implies that tumor mass decrease was sometimes due to fragmentation rather than shrinkage of the primary tumor. Lastly, the presence of tumor deposits was clearly associated with reduced DFS (p = 0.01). Assessment of tumor shrinkage after RCT via tumor downstaging and CRM is a good way of predicting DFS in rectal cancer, and shrinkage of the primary tumor is associated with a decreased nodal tumor load. Assessing regression based on the amount of tumor in relation to stromal fibrosis does not accurately discern tumor fragmentation from tumor shrinkage, which is most likely the reason why Dworak grade had less prognostic relevance.