The flow instabilities of Rayleigh-Bénard convection in a cylinder with effect of uniform internal heat source are investigated numerically. The instabilities of the static state and of axisymmetric flows are investigated by linear stability analysis. The convection threshold depends on the strength of internal heat source q and the aspect ratio of the cylinder Γ. The stability of axisymmetric flows is strongly affected by these two parameters, as well as the Prandtl number Pr. Depending on the value of q, three regimes are identified: weak internal heating, moderate internal heating, and strong internal heating regime. In a weak internal heating regime, the instability characteristics are similar to Rayleigh-Bénard convection. In a moderate internal heating regime, intense interaction of buoyancy instability and hydrodynamic instability result in complex instability curves. When q is large enough, the internal heating effect overwhelms the boundary heating effect. Specifically, the influence of Pr on instability is studied at a moderate internal heat strength q=6.4. An extremely multivalued stability curve is observed. At most five critical Rayleigh numbers can be determined for the axisymmetry-breaking instability at a certain Prandtl number. An axisymmetric unsteady instability mode is observed as well. By nonlinear simulation, the oscillatory flow patterns are obtained, and the axisymmetry-breaking bifurcation of the unsteady toroidal flow is studied.