In this work, we analyze different types of recurrent neural networks (RNNs) working under several different parameters to best model the nonlinear optical dynamics of pulse propagation. Here we studied the propagation of picosecond and femtosecond pulses under distinct initial conditions going through 13 m of a highly nonlinear fiber and demonstrated the application of two RNNs returning error metrics such as normalized root mean squared error (NRMSE) as low as 9%. Those results were further extended for a dataset outside the initial pulse conditions used on the RNN training, and the best-proposed network was still able to achieve a NRMSE below 14%. We believe that this study can contribute to a better understanding of building RNNs employed for modeling nonlinear optical pulse propagation and of how the peak power and nonlinearity affect the prediction error.