The dispersive characteristics and wavelength-dependent behaviors of passive photonic integrated circuits (PICs) can be well described by S-parameters. However, circuit-level simulations of PICs that commonly consist of both passive and active components have to be conducted in the time domain. Thus, S-parameters need to be converted into a time-domain representation without losing accuracy and violating physical properties (e.g., causality). To address this issue, this paper proposes an approach for extracting causal impulse responses of passive PICs by extrapolating their baseband band-limited S-parameters. The method is efficient and robust for a wide range of passive PICs.