Histone deacetylases (HDACs) are a family of enzymes that have been of interest in drug discovery for more than 30 years. Inhibitors of HDACs are potential therapeutics for various diseases, such as neurodegenerative diseases, inflammation, viral infection, and especially cancer. Most HDAC inhibitors (HDACi) are designed for cancer therapy. In 2006, suberoylanilide hydroxamic acid was approved by the US Food and Drug Administration for once-daily oral treatment of advanced cutaneous T-cell lymphoma. In the meantime, there have been aggressive efforts to bring HDACi to the market for every major tumor type, either as a single therapy or in combination, and a number of compounds are currently undergoing clinical trials. Multiple strategies have been applied to the rational design of drugs targeting HDACs by taking advantage of the new developments in proteomics, chemogenomics, cheminformatics, and computational chemistry/biology. Herein, we review the current methods successfully used in developing novel HDACi.