We studied the effect of impurity on the first order superconducting (SC) transition and the high field-low temperature (HFLT) SC state of CeCoIn5 by measuring the specific heat of CeCo(In1-xCdx)_{5} with x=0.0011, 0.0022, and 0.0033 and CeCo(In1-xHgx)_{5} with x=0.000 16, 0.000 32, and 0.000 48 at temperatures down to 0.1 K and fields up to 14 T. Cd substitution rapidly suppresses the crossover temperature T0, where the SC transition changes from second to first order, to T=0 K with x=0.0022 for H parallel[100], while it remains roughly constant up to x=0.0033 for H parallel[001]. The associated anomaly of the proposed FFLO state in Hg-doped samples is washed out by x=0.000 48, while remaining at the same temperature, indicating high sensitivity of that state to impurities. We interpret these results as supporting the nonmagnetic, possibly FFLO, origin of the HFLT state in CeCoIn5.