Phase estimation, at the heart of many quantum metrology and communication schemes, can be strongly affected by noise, whose amplitude may not be known, or might be subject to drift. Here we investigate the joint estimation of a phase shift and the amplitude of phase diffusion at the quantum limit. For several relevant instances, this multiparameter estimation problem can be effectively reshaped as a two-dimensional Hilbert space model, encompassing the description of an interferometer phase probed with relevant quantum states--split single-photons, coherent states or N00N states. For these cases, we obtain a trade-off bound on the statistical variances for the joint estimation of phase and phase diffusion, as well as optimum measurement schemes. We use this bound to quantify the effectiveness of an actual experimental set-up for joint parameter estimation for polarimetry. We conclude by discussing the form of the trade-off relations for more general states and measurements.