Efficient sequencing data compression and FPGA acceleration based on a two-step framework

Shifu Chen; Yaru Chen; Zhouyang Wang; Wenjian Qin; Jing Zhang; Heera Nand; Jishuai Zhang; Jun Li; Xiaoni Zhang; Xiaoming Liang; Mingyan Xu

doi:10.3389/fgene.2023.1260531

Efficient sequencing data compression and FPGA acceleration based on a two-step framework

Front Genet. 2023 Sep 21:14:1260531. doi: 10.3389/fgene.2023.1260531. eCollection 2023.

Authors

Shifu Chen^#^{1

2}, Yaru Chen^#¹, Zhouyang Wang¹, Wenjian Qin², Jing Zhang¹, Heera Nand³, Jishuai Zhang³, Jun Li¹, Xiaoni Zhang¹, Xiaoming Liang³, Mingyan Xu¹

Affiliations

¹ HaploX Biotechnology, Shenzhen, Guangdong, China.
² Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, Guangdong, China.
³ Xilinx Inc., San Jose, CA, United States.

^# Contributed equally.

Abstract

With the increasing throughput of modern sequencing instruments, the cost of storing and transmitting sequencing data has also increased dramatically. Although many tools have been developed to compress sequencing data, there is still a need to develop a compressor with a higher compression ratio. We present a two-step framework for compressing sequencing data in this paper. The first step is to repack original data into a binary stream, while the second step is to compress the stream with a LZMA encoder. We develop a new strategy to encode the original file into a LZMA highly compressed stream. In addition an FPGA-accelerated of LZMA was implemented to speedup the second step. As a demonstration, we present repaq as a lossless non-reference compressor of FASTQ format files. We introduced a multifile redundancy elimination method, which is very useful for compressing paired-end sequencing data. According to our test results, the compression ratio of repaq is much higher than other FASTQ compressors. For some deep sequencing data, the compression ratio of repaq can be higher than 25, almost four times of Gzip. The framework presented in this paper can also be applied to develop new tools for compressing other sequencing data. The open-source code of repaq is available at: https://github.com/OpenGene/repaq.

Keywords: FPGA acceleration; LZMA; compression; fastq; repaq; sequencing.

Grants and funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This work was partially supported by Shenzhen Science and Technology Innovation Committee Technical Research Project (Grant No. JSGG20201103153801005 and No. CJGJZD20200617102403009) and Development of real-time monitoring molecular diagnostic products for tumor based on liquid biopsy (Grant No. 2023B1111040002).