Centered kernel alignment (CKA), also known as centered kernel-target alignment, is useful as a similarity measure between kernels and as a kernel-based similarity measure between feature representations. We prove that CKA based on a Gaussian RBF kernel converges to linear CKA in the large-bandwidth limit. The result relies on mean-centering of the feature maps and on a Hilbert-Schmidt Independence Criterion (HSIC) identity. We show that convergence onset is sensitive to the geometry of the feature representations, and that a notion of representation eccentricity, ρ, constrains the bandwidth range for which Gaussian CKA can differ noticeably from linear CKA. Our experimental results suggest that Gaussian bandwidths less than ρ should be selected in order to enable nonlinear modeling.