Memory devices with high speed and high density are highly desired to address the 'memory wall' issue. Here we demonstrated a highly scalable, three-dimensional stackable ferroelectric diode, with its rectifying polarity modulated by the polarization reversal of Hf0.5Zr0.5O2 films. By visualizing the hafnium/zirconium lattice order and oxygen lattice order with atomic-resolution spherical aberration-corrected STEM, we revealed the correlation between the spontaneous polarization of Hf0.5Zr0.5O2 film and the displacement of oxygen atom, thus unambiguously identified the non-centrosymmetric Pca21 orthorhombic phase in Hf0.5Zr0.5O2 film. We further implemented this ferroelectric diode in an 8 layers 3D array. Operation speed as high as 20 ns and robust endurance of more than 109 were demonstrated. The built-in nonlinearity of more than 100 guarantees its self-selective property that eliminates the need for external selectors to suppress the leakage current in large array. This work opens up new opportunities for future memory hierarchy evolution.