The El Niño/Southern Oscillation has been traditionally linked to the extremes in the Indian summer monsoon rainfall (ISMR) affecting more than a billion people in the region. This trans-oceanic influence is seen to be moderated by the Indian Ocean Dipole (IOD) phenomenon in recent decades. In the presence of a positive IOD (pIOD), the otherwise subdued ISMR in an El Niño year remains close to normal even in the face of record breaking El Niños. While this general influence of pIOD on ISMR is understood, the influence of negative IOD (nIOD) on ISMR is not yet recognized. In this study, it is revealed that those opposite phases of IOD are associated with distinct regional asymmetries in rainfall anomalies. The pIOD is associated with a tripolar pattern in rainfall anomalies with above normal rainfall in central parts of India and below normal rainfall to north and south of it. Conversely, the nIOD is associated with a zonal dipole having above (below) normal rainfall on the western (eastern) half of the country. This spatial quasi-asymmetry arises from the differences in the atmospheric responses and the associated differences in moisture transports to the region during contrasting phases of the IOD.