Plasmonic color filters, exhibiting great promise as an alternative for existing colorant-based filters, often only output one fixed color. Developing active color filters with controllable color output will lead to more compact color filter-based devices. In this paper, we present an approach to achieve active color filtering with a polarization-dependent plasmonic structural color filter, which comprises arrays of asymmetric cross-shaped nanoapertures in an ultrathin film of silver. A systematical study for aperture size, array period, and the thickness of silver film dependences of color filter properties is carried out, and strategies for polarization-dependent color filter designing are generated. A polarization-dependent and high tunability of color can be achieved by selecting the appropriate nanostructure parameters, which imply many potential applications.