The discrete and random occurrence of chemical reactions far from thermodynamic equilibrium, and low copy numbers of chemical species, in systems biology necessitate stochastic approaches. This review is an effort to give the reader a flavor of the most important stochastic approaches relevant to systems biology. Notions of biochemical reaction systems and the relevant concepts of probability theory are introduced side by side. This leads to an intuitive and easy-to-follow presentation of a stochastic framework for modeling subcellular biochemical systems. In particular, we make an effort to show how the notion of propensity, the chemical master equation (CME), and the stochastic simulation algorithm arise as consequences of the Markov property. Most stochastic modeling reviews focus on stochastic simulation approaches--the exact stochastic simulation algorithm and its various improvements and approximations. We complement this with an outline of an analytical approximation. The most common formulation of stochastic models for biochemical networks is the CME. Although stochastic simulations are a practical way to realize the CME, analytical approximations offer more insight into the influence of randomness on system's behavior. Toward that end, we cover the chemical Langevin equation and the related Fokker-Planck equation and the two-moment approximation (2MA). Throughout the text, two pedagogical examples are used to key illustrate ideas. With extensive references to the literature, our goal is to clarify key concepts and thereby prepare the reader for more advanced texts.