Given image data of a fluid flow, the flow field, <u,v>, governing the evolution of the system can be estimated using a variational approach to optical flow. Assuming that the flow field governing the advection is the symplectic gradient of a stream function or the gradient of a potential function-both falling under the category of a potential flow-it is natural to re-frame the optical flow problem to reconstruct the stream or potential function directly rather than the components of the flow individually. There are several advantages to this framework. Minimizing a functional based on the stream or potential function rather than based on the components of the flow will ensure that the computed flow is a potential flow. Next, this approach allows a more natural method for imposing scientific priors on the computed flow, via regularization of the optical flow functional. Also, this paradigm shift gives a framework--rather than an algorithm--and can be applied to nearly any existing variational optical flow technique. In this work, we develop the mathematical formulation of the potential optical flow framework and demonstrate the technique on synthetic flows that represent important dynamics for mass transport in fluid flows, as well as a flow generated by a satellite data-verified ocean model of temperature transport.