Overcoming the IOTLB wall for multi-100-Gbps Linux-based networking

Alireza Farshin; Luigi Rizzo; Khaled Elmeleegy; Dejan Kostić

doi:10.7717/peerj-cs.1385

Overcoming the IOTLB wall for multi-100-Gbps Linux-based networking

PeerJ Comput Sci. 2023 May 16:9:e1385. doi: 10.7717/peerj-cs.1385. eCollection 2023.

Authors

Alireza Farshin¹, Luigi Rizzo², Khaled Elmeleegy², Dejan Kostić¹

Affiliations

¹ School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Kista, Sweden.
² Google, Mountain View, California, United States of America.

Abstract

This article explores opportunities to mitigate the performance impact of IOMMU on high-speed network traffic, as used in the Linux kernel. We first characterize IOTLB behavior and its effects on recent Intel Xeon Scalable & AMD EPYC processors at 200 Gbps, by analyzing the impact of different factors contributing to IOTLB misses and causing throughput drop (up to 20% compared to the no-IOMMU case in our experiments). Secondly, we discuss and analyze possible mitigations, including proposals and evaluation of a practical hugepage-aware memory allocator for the network device drivers to employ hugepage IOTLB entries in the Linux kernel. Our evaluation shows that using hugepage-backed buffers can completely recover the throughput drop introduced by IOMMU. Moreover, we formulate a set of guidelines that enable network developers to tune their systems to avoid the "IOTLB wall", i.e., the point where excessive IOTLB misses cause throughput drop. Our takeaways signify the importance of having a call to arms to rethink Linux-based I/O management at higher data rates.

Keywords: 200 Gbps; Hugepages; IOMMU; IOTLB; Linux kernel; Packet processing; iPerf.

Grants and funding

This work has been supported by an unrestricted gift received for Google Ph.D. Fellowship 2021 in Systems and Networking. The work was also funded by the Swedish Foundation for Strategic Research (SSF). This project has received funding from the European Research Council (ERC) under the European Union’s Horizon 2020 research and innovation programme (grant agreement No 770889). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.