Stimuli-responsive self-assembly of nanoparticles is a versatile approach for the bottom-up fabrication of adaptive and functional nanomaterials. For this purpose, anisotropic building blocks are of particular importance due to the unique shapes and structures that can be obtained upon self-assembly. Here, we demonstrate the photo-responsive self-assembly of plasmonic magnetic "dumbbell" Janus nanoparticles (Au-Fe3O4) via the host-guest interaction of the supramolecular host cyclodextrin and the molecular photoswitch arylazopyrazole. We developed efficient ligand exchange procedures that enable the introduction of functional ligands, respectively, to the surface of the gold or magnetite core of the dumbbell. Our results indicate that distinct nanoparticle superstructures arise in aqueous solutions if nanoparticle aggregation is crosslinker-induced or self-induced and that the reversible formation and fragmentation of the superstructures can be modulated with light.