A self-propelled artificial microswimmer is often modeled as a ballistic Brownian particle moving with constant speed aligned along one of its axis, but changing direction due to random collisions with the environment. Similarly to thermal noise, its angular randomization is described as a memoryless stochastic process. Here, we speculate that finite-time correlations in the orientational dynamics can affect the swimmer's diffusivity. To this purpose, we propose and solve two alternative models. In the first one, we simply assume that the environmental fluctuations governing the swimmer's propulsion are exponentially correlated in time, whereas in the second one, we account for possible damped fluctuations of the propulsion velocity around the swimmer's axis. The corresponding swimmer's diffusion constants are predicted to get, respectively, enhanced or suppressed upon increasing the model memory time. Possible consequences of this effect on the interpretation of the experimental data are discussed.