Aerial manipulators have the potential to perform various tasks with high agility and mobility, but the requirement of system parameters and the complicated dynamic model impede the implementation in practice. To deal with uncertain parameters and complexity of the coupled dynamic model, a decoupling approach is presented in this article by utilizing the adaptive/robust techniques and reinforcement learning approach for the tracking control of quadrotors with position control on the robotic arm. A reinforcement learning approach is proposed to control the robotic arm ensuring minimal effect on the quadrotor dynamics while following the desired trajectory. With the design of nominal inputs, the dynamic uncertainties from the quadrotor, robotic arm, and payload are coped with by utilizing the proposed adaptive algorithms. In addition, the residue of interactive force/torque after the use of DDPG is compensated by robust controllers so that the stability and tracking performance are guaranteed. Numerical examples and experiments are illustrated to demonstrate the efficacy of the presented aerial manipulator control structure and algorithms.