Angular momentum is an important physical property that plays a key role in light-matter interactions, such as spin-orbit interaction. Here, we investigate theoretically and experimentally the spin-orbit interaction between a circularly polarized optical (spin) and a transverse vortex acoustic wave (orbital) using Brillouin backscattering in a silica optical nanofiber. We specifically explore the state of polarization of Brillouin backscattering induced by the TR21 torso-radial vortex acoustic mode that carries an orbital angular momentum. Using a full-vectorial theoretical model, we predict and observe two operating regimes for which the backscattered Brillouin signal is either depolarized or circularly polarized, depending on the input pump polarization. We demonstrate that when the pump is circularly polarized and thus carries a spin angular momentum, the backscattered signal undergoes a handedness reversal of circular polarization due to opto-acoustic spin-orbit interaction and the conservation of overall angular momentum.