Well controlled gas phase experiments of the size and dopant dependent reactivity of gold clusters can shed light on the surprising discovery that nanometer sized gold particles are catalytically active. Most studies that investigate the reactivity of gold clusters in the gas phase focused on charged, small sized clusters. Here, reactivity measurements in a low-pressure reaction cell were performed to investigate carbon monoxide adsorption on neutral bare and silver doped gold clusters (Au(n)Ag(m); n = 10-45; m = 0, 1, 2) at 140 K. The size dependence of the reaction probabilities reflects the role of the electronic shells for the carbon monoxide adsorption, with closed electronic shell systems being the most reactive. In addition, the cluster's reaction probability is reduced upon substitution of gold atoms for silver. Inclusion of a single silver atom causes significant changes in the reactivity only for a few cluster sizes, whereas there is a more general reduction in the reactivity with two silver atoms in the cluster. The experimental observations are qualitatively explained on the basis of a Blyholder model, which includes dopant induced features such as electron transfer from silver to gold, reduced s-d hybrization, and changes in the cluster geometry.