The association and dissociation of particles via specific anisotropic interactions is a fundamental process, both in biology (proteins) and in soft matter (colloidal patchy particles). The presence of alternative binding sites can lead to multiple productive states and also to non-productive "decoy" or intermediate states. Besides anisotropic interactions, particles can experience non-specific isotropic interactions. We employ single replica transition interface sampling to investigate how adding a non-productive binding site or a nonspecific isotropic interaction alters the dimerization kinetics of a generic patchy particle model. The addition of a decoy binding site reduces the association rate constant, independent of the site's position, while adding an isotropic interaction increases it due to an increased rebinding probability. Surprisingly, the association kinetics becomes non-monotonic for a tetramer complex formed by multivalent patchy particles. While seemingly identical to two-particle binding with a decoy state, the cooperativity of binding multiple particles leads to a kinetic optimum. Our results are relevant for the understanding and modeling of biochemical networks and self-assembly processes.