The chemical vapor deposition method is widely used in preparation of graphene, hexagonal boron nitride (h-BN) and other 2D materials. To improve the quality of h-BN, it is essential to learn details of the growth mechanism including dissociation of precursors, the initial nucleation, the stabilities of various clusters and the origins behind them. Here, the dissociation process of ammonia borane (AB) and its later nucleation on the Ru(0001) surface were simulated with density functional theory. The results show that the key step before dissociation is the chemical adsorption of AB molecule on the Ru(0001) surface. In the approaching process of AB molecule to Ru(0001), all bonds connected with H, irrespective of B-H or N-H, are stretched by the underlying metal surface. The H atoms connected with boron prefer to be dissociated followed by those connected to nitrogen. After full dehydration, BN dimer accumulates by forming chain-like, ring-shaped and honeycomb clusters. The chain-like geometries are energy-favoured in BN clusters with size lower than 6. Beyond this, the honeycomb configuration becomes the energy-favoured one, resulting in a geometry evolution from chain-like to honeycomb. The nucleation barrier and the critical nuclei size of BN clusters on Ru(0001) at different chemical potentials are discussed, which can be realized by tuning ratios of B to N in the feedstock. The growth mode observed for h-BN on Ru(0001) can be easily extended to growth on other transition metals and other 2D materials, thus should provide important information in optimizing experimental schemes.