Owing to the limited spectral response of the fiber directional coupler used in a balanced optical coherence tomography configuration, the spectra are different in the two outputs. This affects unfavorably operation of the balanced photodetector unit. Excess photon noise makes a larger contribution than a directional coupler with a flat spectral response. A theoretical model is developed that shows that an optimum set of parameters may be defined to maximize the achievable signal-to-noise ratio. The model leads to a redefinition of the effective noise bandwidth, which takes into account the nonflat response of the directional coupler used. The model also predicts a limitation on the signal-to-noise ratio even when the stray reflectances in the interferometer are brought to zero.