Individuals often refer to opinions of others when they make decisions in the real world. Our question is how the people's reference structure self-organizes when people try to provide correct answers by referring to more accurate agents. We constructed an adaptive network model, in which each node represents an agent and each directed link represents a reference. In every iteration round within our model, each agent makes a decision sequentially by following the majority of the reference partners' opinions and rewires a reference link to a partner if the partner's performance falls below a given threshold. The value of this threshold is common for all agents and represents the performance assessment severity of the population. We found that the reference network self-organizes into a heterogeneous one with a nearly exponential in-degree (the number of followers) distribution, where reference links concentrate around agents with high intrinsic ability. In this heterogeneous network, the decision-making accuracy of agents improved on average. However, the proportion of agents who provided correct answers showed strong temporal fluctuation compared to that observed in the case in which each agent refers to randomly selected agents. We also found a counterintuitive phenomenon in which reference links concentrate more around high-ability agents and the population became smarter on average when the rewiring threshold was set lower than when it was set higher.