We investigate the dynamics of an ensemble of smooth inelastic hard spheres confined between two horizontal plates separated by a distance smaller than twice the diameter of the particles, in such a way that the system is quasi-two-dimensional. The bottom wall is vibrating and, therefore, it injects energy into the system in the vertical direction and a stationary state is reached. It is found that if the size of the plates is small enough, the stationary state is homogeneous. Otherwise, a cluster of particles is developed. The instability is understood by using some effective hydrodynamic equations in the horizontal plane. Moreover, the theoretical prediction for the size of the system above which it becomes unstable agrees very well with molecular dynamics simulation results without any fitting parameter.