Mono-ADP-ribosylation is a major post-translational modification performed by bacterial toxins, which transfer an ADP-ribose moiety to a substrate acceptor residue. Actin- and Rho-specific ADP-ribosylating toxins (ARTs) are typical ARTs known to have very similar tertiary structures but totally different targets. Actin-specific ARTs are the A components of binary toxins, ADP-ribosylate actin at Arg177, leading to the depolymerization of the actin cytoskeleton. On the other hand, C3-like exoenzymes are Rho-specific ARTs, ADP-ribosylate Rho GTPases at Asn41, exerting an indirect effect on the actin cytoskeleton. This review focuses on the differences and similarities of actin- and Rho-specific ARTs, especially with respect to their substrate recognition and cell entry mechanisms, based on structural studies.