META-pipe cloud setup and execution

Aleksandr Agafonov; Kimmo Mattila; Cuong Duong Tuan; Lars Tiede; Inge Alexander Raknes; Lars Ailo Bongo

doi:10.12688/f1000research.13204.3

META-pipe cloud setup and execution

F1000Res. 2017 Nov 29:6:ELIXIR-2060. doi: 10.12688/f1000research.13204.3. eCollection 2017.

Authors

Aleksandr Agafonov¹, Kimmo Mattila², Cuong Duong Tuan³, Lars Tiede⁴, Inge Alexander Raknes⁵, Lars Ailo Bongo¹

Affiliations

¹ Department of Computer Science, UiT The Arctic University of Norway, Tromsø, Norway.
² CSC - IT Center for Science, Espoo, 02150, Finland.
³ CESNET, Prague 6, 160 00 , Czech Republic.
⁴ Department of Information Technology, UiT The Arctic University of Norway, Tromsø, Norway.
⁵ Department of Chemistry, UiT The Arctic University of Norway, Tromsø, Norway.

Abstract

META-pipe is a complete service for the analysis of marine metagenomic data. It provides assembly of high-throughput sequence data, functional annotation of predicted genes, and taxonomic profiling. The functional annotation is computationally demanding and is therefore currently run on a high-performance computing cluster in Norway. However, additional compute resources are necessary to open the service to all ELIXIR users. We describe our approach for setting up and executing the functional analysis of META-pipe on additional academic and commercial clouds. Our goal is to provide a powerful analysis service that is easy to use and to maintain. Our design therefore uses a distributed architecture where we combine central servers with multiple distributed backends that execute the computationally intensive jobs. We believe our experiences developing and operating META-pipe provides a useful model for others that plan to provide a portal based data analysis service in ELIXIR and other organizations with geographically distributed compute and storage resources.

Keywords: AAI federation; Amazon Web Services; Apache Spark; EGI Federated Cloud; ELIXIR; META-pipe; OpenStack; Portability.

Grants and funding

This work was funded by ELIXIR, The Research Council of Norway (project number 270675), EGI-Engage, and UiT The Arctic University of Norway. ELIXIR received funding from the European Union’s Horizon 2020 research and innovation program (ELIXIR- EXCELERATE, grant agreement no 676559). The EGI-Engage project is co-funded by the European Union (EU) Horizon 2020 program under Grant number 654142.