We observe asymmetric color filtering under unpolarized incidence in a structured metallic (Ag) film, where the center of an optically thick circular Ag disk surrounded by a step gap appears dark when observed from one side, and bright from the other. The latter situation corresponds to abnormally high optical transmission through the optically thick film. We explain this by a three-step process: coupling of photons to surface plasmon polaritons (SPPs), wave interference of SPPs forming resonant cavity modes, and out-coupling from SPPs to photons. Full wave electromagnetic simulations based on the finite element method support our findings. These results may have potential applications in areas such as optical color filtering and biosensing via dielectric detection within the step gap plasmonic cavity.