We demonstrate controlled manipulation of semiconductor and metallic nanoparticles (NPs) with 5-15 nm diameters and assemble these NPs into hybrid structures. The manipulation is accomplished under ambient environment using a commercial atomic force microscope (AFM). There are particular difficulties associated with manipulating NPs this small. In addition to spatial drift, the shape of an asymmetric AFM tip has to be taken into account in order to understand the intended and actual manipulation results. Furthermore, small NPs often attach to the tip via electrostatic interaction and modify the effective tip shape. We suggest a method for detaching the NPs by performing a pseudo-manipulation step. Finally, we show by example the ability to assemble these small NPs into prototypical hybrid nanostructures with well-defined composition and geometry.