Resistance switching (RS) devices with ultra-thin Ta2O5 switching layer (0.5-2.0 nm) with a cell diameter of 28 nm were fabricated. The performance of the devices was tested by voltage-driven current-voltage (I-V) sweep and closed-loop pulse switching (CLPS) tests. A Ta layer was placed beneath the Ta2O5 switching layer to act as an oxygen vacancy reservoir. The device with the smallest Ta2O5 thickness (0.5 nm) showed normal switching properties with gradual change in resistance in I-V sweep or CLPS and high reliability. By contrast, other devices with higher Ta2O5 thickness (1.0-2.0 nm) showed abrupt switching with several abnormal behaviours, degraded resistance distribution, especially in high resistance state, and much lower reliability performance. A single conical or hour-glass shaped double conical conducting filament shape was conceived to explain these behavioural differences that depended on the Ta2O5 switching layer thickness. Loss of oxygen via lateral diffusion to the encapsulating Si3N4/SiO2 layer was suggested as the main degradation mechanism for reliability, and a method to improve reliability was also proposed.