Plant dwarfism is a desirable agronomic trait in non-timber trees, but little is known about the physiological and molecular mechanism underlying dwarfism in woody plants. Castor bean (Ricinus communis) is a typical woody oilseed crop. We performed cytological observations within xylem, phloem and cambia tissues, revealing that divergent cell growth in all tissues might play a role in the dwarf phenotype in cultivated castor bean. Based on bulked segregant analyses for a F2 population generated from the crossing of a tall and a dwarf accession, we identified two QTLs associated with plant height, covering 325 candidate genes. One of these, Rc5NG4-1 encoding a putative IAA transport protein localized in the tonoplast was functionally characterized. A non-synonymous SNP (altering the amino acid sequence from Y to C at position 218) differentiated the tall and dwarf plants and we confirmed, through heterologous yeast transformation, that the IAA uptake capacities of Rc5NG4-1Y and Rc5NG4-1C were significantly different. This study provides insights into the physiological and molecular mechanisms of dwarfing in woody non-timber economically important plants, with potential to aid in the genetic breeding of castor bean and other related crops.