Organic-inorganic hybrid perovskite has become one of the most important photoactive materials owing to its intense light-harvesting property as well as its facile solution processability. Besides its photovoltaic applications, a novel photo-programmed transistor memory was recently developed based on the device architecture of a floating-gate transistor memory using a polymer/perovskite blend as the gate dielectric with the non-volatile memory characteristics of decent light response, applicable On/Off current ratio, and long retention time. In this study, we further clarify the influence of polymer matrix selection on the photo-response and memory properties of derived hybrid perovskite-based photo-memory devices. Four different host polymers, polystyrene (PS), poly(4-vinylphenol) (PVPh), poly(methyl methacrylate) (PMMA), and poly(methacrylic acid) (PMAA), were systematically investigated for comparison herein. This revealed that dissimilar chemical interactions existed between the host polymers and perovskite, resulting in the distinct memory behavior of the derived photo-memory devices, attributable to the different morphologies of the hybrid dielectric layers and the different sizes of the distributed perovskite nanoparticles (NPs). The photo-response behavior and the resultant On/Off current ratio increased as the size of the embedded perovskite NPs decreased, due to the enhanced photo-induced charge transfer across the dielectric/pentacene interface, benefiting from the better confinement effect of perovskite NPs in the polymer matrix. These results demonstrate the influence of perovskite NP aggregation at the dielectric/pentacene interface on the resultant memory behavior of the newly developed photo-memory device.