With the rapid growth of the Internet and overwhelming amount of information that people are confronted with, recommender systems have been developed to effectively support users' decision-making process in online systems. So far, much attention has been paid to designing new recommendation algorithms and improving existent ones. However, few works considered the different contributions from different users to the performance of a recommender system. Such studies can help us improve the recommendation efficiency by excluding irrelevant users. In this paper, we argue that in each online system there exists a group of core users who carry most of the information for recommendation. With them, the recommender systems can already generate satisfactory recommendation. Our core user extraction method enables the recommender systems to achieve 90% of the accuracy of the top-L recommendation by taking only 20% of the users into account. A detailed investigation reveals that these core users are not necessarily the large-degree users. Moreover, they tend to select high quality objects and their selections are well diversified.