The Q-state Potts model can be extended to noninteger and even complex Q by expressing the partition function in the Fortuin-Kasteleyn (F-K) representation. In the F-K representation the partition function Z(Q,a) is a polynomial in Q and v=a-1 (a=e(betaJ)) and the coefficients of this polynomial, Phi(b,c), are the number of graphs on the lattice consisting of b bonds and c connected clusters. We introduce the random-cluster transfer matrix to compute Phi(b,c) exactly on finite square lattices with several types of boundary conditions. Given the F-K representation of the partition function we begin by studying the critical Potts model Z(CP)=Z(Q,a(c)(Q)), where a(c)(Q)=1+square root[Q]. We find a set of zeros in the complex w=square root[Q] plane that map to (or close to) the Beraha numbers for real positive Q. We also identify Q(c)(L), the value of Q for a lattice of width L above which the locus of zeros in the complex p=v/square root[Q] plane lies on the unit circle. By finite-size scaling we find that 1/Q(c)(L)-->0 as L-->infinity. We then study zeros of the antiferromagnetic (AF) Potts model in the complex Q plane and determine Q(c)(a), the largest value of Q for a fixed value of a below which there is AF order. We find excellent agreement with Baxter's conjecture Q(AF)(c)(a)=(1-a)(a+3). We also investigate the locus of zeros of the ferromagnetic Potts model in the complex Q plane and confirm that Q(FM)(c)(a)=(a-1)(2). We show that the edge singularity in the complex Q plane approaches Q(c) as Q(c)(L) approximately Q(c)+AL(-y(q)), and determine the scaling exponent y(q) for several values of Q. Finally, by finite-size scaling of the Fisher zeros near the antiferromagnetic critical point we determine the thermal exponent y(t) as a function of Q in the range 2</=Q</=3. Using data for lattices of size 3</=L</=8 we find that y(t) is a smooth function of Q and is well fitted by y(t)=(1+Au+Bu2)/(C+Du) where u=-(2/pi)cos(-1)(squareroot[Q]/2). For Q=3 we find y(t) approximately 0.6; however if we include lattices up to L=12 we find y(t) approximately 0.50(8) in rough agreement with a recent result of Ferreira and Sokal [J. Stat. Phys. 96, 461 (1999)].