In many Western societies there are rising concerns about increasing polarization in public debate. However, statistics on private attitudes paint a different picture: the average attitudes in societies are more moderate and remain rather stable over time. The present paper presents an agent-based model of how such discrepancies between public opinion and private attitudes develop at the scale of micro-societies. Based on social psychological theorizing, the model distinguishes between two types of agents: a) those seeking to gain or maintain a good reputation and status, and b) those seeking to promote group harmony by reaching consensus. We characterized these different types of agents by different decision rules for either voicing their opinion or remaining silent, based on the behavior of their proximal network. Results of the model simulations show that even when the private attitudes of the agents are held constant, publicly expressed opinions can oscillate and (depending on the reputational concerns of individual actors) situations can occur in which minorities as well as majorities are silenced. We conclude that the macro-level consequences of micro-level decisions to either voice an opinion or remain silent provide a foundation for better understanding how public opinions are shaped. Moreover, we discuss the conditions under which public opinion could be considered a good representation of private attitudes in a society.