We present an optimization of spot-size converter (SSC) of waveguide photodetector (PD) for small polarization dependent loss (PDL). Beam-propagation method simulation gives responsivity for each polarization and SSC structure. From the calculated responsivity data, optimum structure of SSC is determined that can be implemented with a sufficient process tolerance. We confirm the optimization by measuring PDL of waveguide PD designed according to the structure obtained through the simulation.