A theory of systems with long-range correlations based on the consideration of binary N-step Markov chains is developed. In our model, the conditional probability that the ith symbol in the chain equals zero (or unity) is a linear function of the number of unities among the preceding N symbols. The correlation and distribution functions as well as the variance of number of symbols in the words of arbitrary length L are obtained analytically and numerically. If the persistent correlations are not extremely strong, the variance is shown to be nonlinearly dependent on L. A self-similarity of the studied stochastic process is revealed. The applicability of the developed theory to the coarse-grained written and DNA texts is discussed.