The non-ergodicity problem of temporal averaging in the laser speckle contrast imaging (LSCI) is discussed both theoretically and numerically. We demonstrate that temporal speckle statistics assessed within a finite time window might differ from the statistics of the speckle ensemble. The dependence of temporal speckle contrast on sample dynamics is non-monotonic and demonstrates regimes of negative (ergodic), as well as positive correlations with dynamics (non-ergodic). The ergodic regime is similar to an ensemble (spatial) averaging case and is typically assumed for interpretation of LSCI measurements. The positive relation in the non-ergodic regime is an artifact of temporal statistics which we further quantify and describe the transition between two regimes.