Lensless imaging has emerged as a robust means for the observation of microscopic scenes, enabling vast applications like whole-slide imaging, wave-front detection and microfluidic on-chip imaging. Such system captures diffractive measurements in a compact optical setup without the use of optical lens, and then typically applies phase retrieval algorithms to recover the complex field of target object. However existing techniques still suffer from unsatisfactory performance with noticeable reconstruction artifacts especially when the imaging parameter is not well calibrated. Here we propose a novel unsupervised Diffractive Neural Field (DNF) method to accurately characterize the imaging physical process to best reconstruct desired complex field of the target object through very limited measurement snapshots by jointly optimizing the imaging parameter and implicit mapping between spatial coordinates and complex field. Both simulations and experiments reveal the superior performance of proposed method, having > 6 dB PSNR (Peak Signal-to-Noise Ratio) gains on synthetic data quantitatively, and clear qualitative improvement on real-world samples. The proposed DNF also promises attractive prospects in practical applications because of its ultra lightweight complexity (e.g., 50× model size reduction) and plug-to-play advantage (e.g., random measurements with a coarse parameter estimation).