In this paper, we report a successful realization and integration of a gold two-dimensional hole array (2DHA) structure with semiconductor InAs quantum dot (QD). We show experimentally that a properly designed 2DHA-QD photodetector can facilitate a strong plasmonic-QD interaction, leading to a 130% absolute enhancement of infrared photoresponse at the plasmonic resonance. Our study indicates two key mechanisms for the performance improvement. One is an optimized 2DHA design that permits an efficient coupling of light from the far-field to a localized plasmonic mode. The other is the close spatial matching of the QD layers to the wave function extent of the plasmonic mode. Furthermore, the processing of our 2DHA is amenable to large scale fabrication and, more importantly, does not degrade the noise current characteristics of the photodetector. We believe that this demonstration would bring the performance of QD-based infrared detectors to a level suitable for emerging surveillance and medical diagnostic applications.