This article studies the bearing-based formation tracking control problem of multiple double-integrator agents. A leader-following structure, where the leader moves with the reference dynamics, is adopted. Different from the existing methods, which require complete information of the time-varying reference velocity, in this article, only the time-varying reference orientation information is known by part of the followers and the amplitude of the reference velocity is unknown. To solve the problem, this article proposes a velocity-estimation-based control scheme, which consists of an estimator for estimating the varying rate of the reference orientation, an adaptation law for estimating the amplitude of the reference velocity, and bearing-based control inputs for tracking the leader and achieving the bearing-based formation based on the estimations. Moreover, the scaling formation maneuver can be achieved by using an auxiliary distance measurement. It shows that both the estimation errors and control errors converge to zero under the connectivity of the topology and properties of bearing rigidity. The closed-loop system is analyzed to be semiglobally uniformly asymptotically stable based on the cascaded system theory. Numerical simulations are presented to demonstrate the effectiveness of our method.