This paper presents a kernel-based nonlinear mixing model for hyperspectral data, where the nonlinear function belongs to a Hilbert space of vector valued functions. The proposed model extends the existing ones by accounting for band-dependent and neighboring nonlinear contributions. The key idea is to work under the assumption that nonlinear contributions are dominant in some parts of the spectrum, while they are less pronounced in other parts. In addition to this, we motivate the need for taking into account nonlinear contributions originating from the ground covers of neighboring pixels by practical considerations, precisely the adjacency effect. The relevance of the proposed model is that the nonlinear function is associated with a matrix valued kernel that allows to jointly model a wide range of nonlinearities and includes prior information regarding band dependences. Furthermore, the choice of the nonlinear function input allows to incorporate neighboring effects. The optimization problem is strictly convex and the corresponding iterative algorithm is based on the alternating direction method of multipliers. Finally, experiments conducted using synthetic and real data demonstrate the effectiveness of the proposed approach.