Tests of local realism and macrorealism have historically been discussed in very similar terms: Leggett-Garg inequalities follow Bell inequalities as necessary conditions for classical behavior. Here, we compare the probability polytopes spanned by all measurable probability distributions for both scenarios and show that their structure differs strongly between spatially and temporally separated measurements. We arrive at the conclusion that, in contrast to tests of local realism where Bell inequalities form a necessary and sufficient set of conditions, no set of inequalities can ever be necessary and sufficient for a macrorealistic description. Fine's famous proof that Bell inequalities are necessary and sufficient for the existence of a local realistic model, therefore, cannot be transferred to macrorealism. A recently proposed condition, no-signaling in time, fulfills this criterion, and we show why it is better suited for future experimental tests and theoretical studies of macrorealism. Our work thereby identifies a major difference between the mathematical structures of local realism and macrorealism.