Heterogeneous ultra-dense networking (HUDN) with energy harvesting technology is a promising approach to deal with the ever-growing traffic that can severely impact the power consumption of small-cell networks. Unfortunately, the amount of harvested energy, which depends on the transmission environment, is highly random and difficult to predict. Since there may be multiple sources of energy in the HUDN, e.g., macro base stations or TV towers, the challenging issue is when and where to harvest energy. Optimally controlling the HUDN can profoundly influence the performance of both data transmission and energy harvesting. However, the working pattern of individual small cell base stations needs to be determined in every time slot. To find an optimal solution in a highly random environment we propose reinforcement learning methods, such as deep deterministic policy gradient (DDPG) and wolpertinger DDPG (W-DDPG). Since the action space is large and discrete for the controlling tasks, a W-DDPG algorithm has been found to be the best approach. The simulation results verify that, compared with the original DDPG algorithm and deep Q-learning, the proposed W-DDPG method can achieve a superior performance in terms of both energy efficiency and throughput.
Keywords: DDPG; Reinforcement learning; energy harvesting; heterogeneous network; mmWave.