The wakefield, or ion focus, formed by ions streaming past dust particles trapped in the plasma sheath leads to two types of instabilities: the Schweigert instability in multilayer systems and the mode-coupling instability that already appears in single-layer dust systems. Here, a model is presented that treats both types of instability in a common description. The parameter space for the onset of the instabilities is determined. A new variant of the mode-coupling instability is found to arise from the interaction among the layers. For weak confinement, all instabilities continuously merge into each other. For stronger confinement of the dust mainly the Schweigert type of instability is observed.