Linear and nonlinear counter-traveling waves in a fluid-filled annular cylinder with realistic no-slip boundary conditions uniformly heated from below and rotating about a vertical axis are investigated. When the gap of the annular cylinder is moderate, there exist two three-dimensional traveling waves driven by convective instabilities: a retrograde mode localized near the outer sidewall and a prograde mode adjacent to the inner sidewall with a different wave number, frequency and critical Rayleigh number. It is found that the retrogradely propagating mode is always more unstable and is marked by a larger azimuthal wave number. When the Rayleigh number is sufficiently large, both the counter-traveling modes can be excited and nonlinearly interacting, leading to an unusual nonlinear phenomenon in rotating Rayleigh-Bénard convection.