The geometric and electronic structures of Si(n), Si(n)-, and PSi(n-1) clusters (2 < or = n < or = 13) have been investigated using the ab initio molecular orbital theory formalism. The hybrid exchange-correlation energy functional (B3LYP) and a standard split-valence basis set with polarization functions (6-31+G(d)) were employed to optimize geometrical configurations. The total energies of the lowest energy isomers thus obtained were recalculated at the MP2/aug-cc-pVTZ level of theory. Unlike positively charged clusters, which showed similar structural behavior as that of neutral clusters [Nigam et al., J. Chem. Phys. 121, 7756 (2004)], significant geometrical changes were observed between Si(n) and Si(n)- clusters for n = 6, 8, 11, and 13. However, the geometries of P substituted silicon clusters show similar growth as that of negatively charged Si(n) clusters with small local distortions. The relative stability as a function of cluster size has been verified based on their binding energies, second difference in energy (Delta2 E), and fragmentation behavior. In general, the average binding energy of Si(n)- clusters is found to be higher than that of Si(n) clusters. For isoelectronic PSi(n-1) clusters, it is found that although for small clusters (n < 4) substitution of P atom improves the binding energy of Si(n) clusters, for larger clusters (n > or = 4) the effect is opposite. The fragmentation behavior of these clusters reveals that while small clusters prefer to evaporate monomer, the larger ones dissociate into two stable clusters of smaller size. The adiabatic electron affinities of Si(n) clusters and vertical detachment energies of Si(n)- clusters were calculated and compared with available experimental results. Finally, a good agreement between experimental and our theoretical results suggests good prediction of the lowest energy isomeric structures for all clusters calculated in the present study.