In layered superconductors the order parameter may be modulated within the unit cell, leading to nontrivial modifications of the vortex core if the interlayer coherence length ξ(c)(T) is comparable to the interlayer spacing. In the iron pnictide SmFeAs(O,F) (T(c)≈50 K) this occurs below a crossover temperature T(⋆)≈41 K, which separates two regimes of vortices: anisotropic Abrikosov-like at high and Josephson-like at low temperatures. Yet in the transition region around T(⋆), hybrid vortices between these two characteristics appear. Only in this region around T(⋆) and for magnetic fields well aligned with the FeAs layers, we observe oscillations of the c-axis critical current j(c)(H) periodic in 1/sqrt[H] due to a delicate balance of intervortex forces and interaction with the layered potential. j(c)(H) shows pronounced maxima when a hexagonal vortex lattice is commensurate with the underlying crystal structure. The narrow temperature window in which oscillations are observed suggests a significant suppression of the order parameter between the superconducting layers in SmFeAs(O,F), despite its low coherence length anisotropy (γ(ξ)≈3-5).