Previous studies showed similar mappings between sounds and colours for synaesthetes and non-synaesthetes alike, and proposed that common mechanisms underlie such cross-modal association. The findings between vowels and colours, and between pitch and lightness, were investigated separately, and it was also unknown how language background would influence such association. The present study investigated the cross-modal association between sounds (vowels and pitch) and colours in a tone language using three groups of non-synaesthetes: Cantonese (native), Mandarin (foreign, tonal), and English (foreign, non-tonal). Strong associations were found between /a/ and red, /i/ with light colours, and /u/ with dark colours, and a robust pitch effect with a high tone eliciting lighter colours than a low tone in general. The pitch effect is stronger than the vowel associations. Significant differences among the three language groups in colour choices of other vowels and the strength of association were found, which demonstrate the language-specificity of these associations. The findings support the notion that synaesthesia is a general phenomenon, which can be influenced by linguistic factors.