We propose a scheme for modulating phase computationally in light field imaging systems. In a camera system based on the scheme, light field (LF) data is obtained by array-based optics, and the data is computationally projected into a single image with arbitrary phase modulation. In a projector system based on the scheme, LF data with arbitrary phase modulation is computationally generated before optical projection, and the phase-modulated image is projected by array-based optics. We describe the system design and required conditions based on the sampling theorem. We experimentally verified the proposed scheme based on camera and projector systems. In the experiment, we demonstrated a super-resolution camera and projector with extended depth-of-field without estimating the object's shape.