Focused ion beam (FIB) milling is a common fabrication technique to make nanostencil masks which has the unintended consequence of gallium ion implantation surrounding milled features in silicon nitride membranes. We observe major changes in film structure, chemical composition, and magnetic behaviour of permalloy nanostructures deposited by electron beam evaporation using silicon nitride stencil masks made by a FIB as compared to stencil masks made by regular lithography techniques. We characterize the stenciled structures and both types of masks using transmission electron microscopy, electron energy loss spectroscopy, energy dispersive x-ray spectroscopy, magnetic force microscopy and kelvin probe force microscopy. All these techniques demonstrate distinct differences at a length scale of a 1-100 nm for the structures made using stencil mask fabricated using a FIB. The origin of these differences seems to be related to the presence of implanted ions, a detailed understanding of the mechanism however remains to be developed.