Optical systems have been crucial for versatile applications such as consumer electronics, remote sensing and biomedical imaging. Designing optical systems has been a highly professional work due to complicated aberration theories and intangible rules-of-thumb, hence neural networks are only coming into this realm until recent years. In this work, we propose and implement a generic, differentiable freeform raytracing module, suitable for off-axis, multiple-surface freeform/aspheric optical systems, paving the way toward a deep learning-based optical design method. The network is trained with minimal prior knowledge, and it can infer numerous optical systems after a one-time training. The presented work unlocks great potential for deep learning in various freeform/aspheric optical systems, and the trained network could serve as an effective, unified platform for generating, recording, and replicating good initial optical designs.