Vortex beam has the potential to significantly improve the performance of lidar (light detection and ranging) and optical communication applications in which low signal-to-noise ratio (SNR) limits the detection/transmission range. The vortex beam method allows for spatially separating the coherent light (laser signal) from the incoherent light (the background radiation and multiple-scattered light) of the received signal. This paper presents results of a simulation model in which the optical vortex acts as an optical filter. We present instrument parameters that describe the filtering effect, e.g., the form of the vortex phase modulation function, the topological charge of the vortex and the focal length of a virtual Fresnel lens that is used for optical filtering. Preliminary experimental results show that the background radiation within the spectral filter bandwidth can be suppressed by as much as 95%. At the same time, we retain 97% of the coherent laser signal. Our simulation model will be used in future design of lidar instruments and optical communication systems in which the optical vortex method is used for optical filtering of the detected signals.