Due to the extraordinary abilities in extracting complex patterns, graph neural networks (GNNs) have demonstrated strong performances and received increasing attention in recent years. Despite their prominent achievements, recent GNNs do not pay enough attention to discriminate nodes when determining the information sources. Some of them select information sources from all or part of neighbors without distinction, and others merely distinguish nodes according to either graph structures or node features. To solve this problem, we propose the concept of the Influence Set and design a novel general GNN framework called the graph influence network (GINN), which discriminates neighbors by evaluating their influences on targets. In GINN, both topological structures and node features of the graph are utilized to find the most influential nodes. More specifically, given a target node, we first construct its influence set from the corresponding neighbors based on the local graph structure. To this aim, the pairwise influence comparison relations are extracted from the paths and a HodgeRank-based algorithm with analytical expression is devised to estimate the neighbors' structure influences. Then, after determining the influence set, the feature influences of nodes in the set are measured by the attention mechanism, and some task-irrelevant ones are further dislodged. Finally, only neighbor nodes that have high accessibility in structure and strong task relevance in features are chosen as the information sources. Extensive experiments on several datasets demonstrate that our model achieves state-of-the-art performances over several baselines and prove the effectiveness of discriminating neighbors in graph representation learning.