Berenil is an antitrypanosomal agent that binds to nucleic acid duplexes. Recently, we reported that this drug can bind to both DNA and RNA duplexes, while exhibiting properties characteristic of both intercalation and groove binding [Pilch, D. S., Kirolos, M. A., Liu, X., Plum, G. E., & Breslauer, K. J. (1995) Biochemistry 34, 9962-9976]. In this work, we use spectroscopic, calorimetric, and hydrodynamic techniques to demonstrate that berenil also can bind to DNA and RNA triplexes. Our results reveal the following significant features: (i) Berenil binds to the poly(dA).2poly(dT) DNA triplex and to the poly(rA).2poly(rU) RNA triplex without displacing the major groove-bound third strands. (ii) Both berenil-bound triplexes melt via two distinct transitions: initial conversion of the triplex to the duplex state, with the berenil remaining bound, followed by denaturation of the duplex to its component single strands. (iii) The magnitude and even the direction of the impact of berenil binding on the thermal stability of the DNA triplex depends on both the Na+ concentration and the drug binding density (the [base triplet]/[total berenil] ratio). Specifically, at Na+ concentrations < or = 0.08 M, the DNA triplex to duplex transition is thermally stabilized, while at Na+ concentrations > or = 0.125 M it is thermally destabilized. Between these two salt concentrations, berenil binding either enhances or diminishes the thermal stability of the DNA triplex in a manner that depends on the [base triplet]/[total berenil] ratio. (iv) The effect of berenil binding on the thermal stability of the RNA triplex to duplex equilibrium also depends on the [base triplet]/[total berenil] ratio, having a weakly destabilizing effect on this equilibrium at [base triplet]/[total berenil] ratios > 5, while thermally stabilizing this equilibrium at [base triplet]/[total berenil] ratios < 5. (v) The apparent "site sizes" associated with berenil binding to the triplexes range from approximately 1 to 12 base triplets per bound berenil and depend, in part, on the host triplex. One of the site sizes common to both triplexes is consistent with berenil binding to the minor groove. (vi) Berenil exhibits a higher apparent binding affinity for the DNA triplex relative to the RNA triplex. (vii) Viscometric data reveal nonintercalative binding properties when berenil complexes with both triplexes, consistent with a minor groove binding mode. (viii) Berenil binding to either the DNA or the RNA triplex is enthalpically more favorable than berenil binding to the corresponding duplex. (ix) Berenil binding to both triplexes decreases the cooperativity of the triplex to duplex melting event.(ABSTRACT TRUNCATED AT 400 WORDS)