We develop the framework of classical observational entropy, which is a mathematically rigorous and precise framework for nonequilibrium thermodynamics, explicitly defined in terms of a set of observables. Observational entropy can be seen as a generalization of Boltzmann entropy to systems with indeterminate initial conditions, and it describes the knowledge achievable about the system by a macroscopic observer with limited measurement capabilities; it becomes Gibbs entropy in the limit of perfectly fine-grained measurements. This quantity, while previously mentioned in the literature, has been investigated in detail only in the quantum case. We describe this framework reasonably pedagogically, then show that in this framework, certain choices of coarse-graining lead to an entropy that is well-defined out of equilibrium, additive on independent systems, and that grows toward thermodynamic entropy as the system reaches equilibrium, even for systems that are genuinely isolated. Choosing certain macroscopic regions, this dynamical thermodynamic entropy measures how close these regions are to thermal equilibrium. We also show that in the given formalism, the correspondence between classical entropy (defined on classical phase space) and quantum entropy (defined on Hilbert space) becomes surprisingly direct and transparent, while manifesting differences stemming from noncommutativity of coarse-grainings and from nonexistence of a direct classical analog of quantum energy eigenstates.