We study the effect of the core ellipticity and core-induced thermal stress on the conversion of LP11 modes to vortex modes in gradually twisted highly birefringent PANDA fibers using an improved perturbation-based modeling method. We show that these two technologically unavoidable factors have a significant impact on the conversion process, which manifests itself in shortening the conversion length, altering the assignment between the input LP11 modes and output vortex modes, and modifying the vortex mode structure. In particular, we demonstrate that for certain fiber geometries, it is possible to obtain output vortex modes with parallel and antiparallel spins and orbital angular momenta. The simulation results obtained using the modified method are in good agreement with recently published experimental data. Furthermore, the proposed method provides reliable guidelines for choosing fiber parameters that ensure a short conversion length and the desired polarization structure of the output vortex modes.