HDAC6 plays a central role in the recruitment of protein aggregates for lysosomal degradation and is a promising target for combination therapy with proteasome inhibitors in multiple myeloma. Pharmacologically displacing ubiquitin from the zinc-finger ubiquitin-binding domain (ZnF-UBD) of HDAC6 is an underexplored alternative to catalytic inhibition. Here, we present the discovery of an HDAC6 ZnF-UBD-focused chemical series and its progression from virtual screening hits to low micromolar inhibitors. A carboxylate mimicking the C-terminal extremity of ubiquitin, and an extended aromatic system stacking with W1182 and R1155, are necessary for activity. One of the compounds induced a conformational remodeling of the binding site where the primary binding pocket opens up onto a ligand-able secondary pocket that may be exploited to increase potency. The preliminary structure-activity relationship accompanied by nine crystal structures should enable further optimization into a chemical probe to investigate the merit of targeting the ZnF-UBD of HDAC6 in multiple myeloma and other diseases.