In this paper, an approach for optimizing sub-Nyquist lenses using an end-to-end physics-informed deep neural network is presented. The simulation and optimization of these sub-Nyquist lenses is investigated for image quality, classification performance, or both. This approach integrates a diffractive optical model with a deep learning classifier, forming a unified optimization framework that facilitates simultaneous simulation and optimization. Lenses in this work span numerical apertures from approximately 0.1 to 1.0, and a total of 707 models are trained using the PyTorch-Lightning deep learning framework. Results demonstrate that the optimized lenses produce better image quality in terms of mean squared error (MSE) compared to analytical lenses by reducing the impact of diffraction order aliasing. When combined with the classifier, the optimized lenses show improved classification performance and reduced variability across the focal range. Additionally, the absence of correlation between the MSE measurement of image quality and classification performance suggests that images that appear good according to the MSE metric may not necessarily be beneficial for the classifier.