Somehow, our brain and other organisms manage to predict their environment. Behind this must be an input-dependent dynamical system, or recurrent neural network, whose present state reflects the history of environmental input. The design principles for prediction-in particular, what kinds of attractors allow for greater predictive capability-are still unknown. We offer some clues to design principles using an attractor picture when the environment perturbs the system's state weakly, motivating and developing some theory for continuous-time time-varying linear reservoirs along the way. Reservoirs that inherently support only stable fixed points are generically good predictors, while reservoirs with limit cycles are good predictors for noisy periodic input.