In a leakage channel fiber, the desired fundamental mode (FM) has negligible waveguide loss. Higher-order modes (HOM) are designed to have much higher waveguide losses so that they are practically eliminated during propagation. Coherent reflection at the fiber outer boundary can lead to additional confinement especially for highly leaky HOM, leading to lower HOM losses than what are predicted by conventional FEM mode solver considering infinite cladding. In this work, we conducted, for the first time, careful measurements of HOM losses in two leakage channel fibers (LCF) with circular and rounded hexagonal boundary shapes respectively. Impact on HOM losses from coiling, fiber boundary shapes and coating indexes were studied in comparison to simulations. This work, for the first time, demonstrates the limit of the simulation method commonly used in the large-mode-area fiber designs and the need for an improved approach. More importantly, this work also demonstrates that a deviation from circular fiber outer shape may be an effective method to mitigate HOM loss reduction from coherent reflection from fiber outer boundary, even in double-clad fibers, with HOM losses in excess of 20 dB/m measured in the hexagonal LCF with ~50 µm core diameter while keeping FM loss negligible.