We search for the production of a heavy W' gauge boson that decays to third generation quarks in 0.9 fb-1 of pp collisions at square root(s)=1.96 TeV, collected with the D0 detector at the Fermilab Tevatron collider. We find no significant excess in the final-state invariant mass distribution and set upper limits on the production cross section times branching fraction. For a left-handed W' boson with SM couplings, we set a lower mass limit of 731 GeV. For right-handed W' bosons, we set lower mass limits of 739 GeV if the W' boson decays to both leptons and quarks and 768 GeV if the W' boson decays only to quarks. We also set limits on the coupling of the W' boson to fermions as a function of its mass.