Deep Quantization (below eight bits) can significantly reduce the DNN computation and storage by decreasing the bitwidth of network encodings. However, without arduous manual effort, this deep quantization can lead to significant accuracy loss, leaving it in a position of questionable utility. We propose a systematic approach to tackle this problem, by automating the process of discovering the bitwidths through an end-to-end deep reinforcement learning framework (ReLeQ). This framework utilizes the sample efficiency of proximal policy optimization to explore the exponentially large space of possible assignment of the bitwidths to the layers. We show how ReLeQ can balance speed and quality, and provide a heterogeneous bitwidth assignment for quantization of a large variety of deep networks with minimal accuracy loss (≤ 0.3% loss) while minimizing the computation and storage costs. With these DNNs, ReLeQ enables conventional hardware and custom DNN accelerator to achieve 2.2× speedup over 8-bit execution.