ilifu – Research cloud infrastructure

A primary goal of IDIA is to build capacity and expertise in data intensive research at South African universities to support MeerKAT science projects, specifically servicing five of the eight MeerKAT large survey projects. IDIA manages the Ilifu research cloud infrastructure which supports both astronomy and bioinformatics South African research communities. The Ilifu facility provides both compute and storage resources and offers infrastructure, platform, software and support as a service. Data transfer nodes and services provide fast data transfers from the SARAO archive. A new client-server astronomy visualisation tool, the Cube Analysis and Rendering Tool for Astronomy (CARTA), allows for web-based, efficient interactive visual analytics of large astronomy data sets hosted on Ilifu.

The Ilifu infrastructure is managed through the open source cloud computing OpenStack platform, allowing for both dedicated services and infrastructure as a service solutions. The Ilifu cluster is the primary service for data processing and includes a pool of compute resources and access to storage. Using the Slurm job scheduling software, the cluster supports batch job submission, for running data processing pipelines, and workflows. A development and analytics environment is implemented through a Jupyter notebook and JupyterLab service (jupyter.ilifu.ac.za) with resources selected by the user to fit the scale of the task. Several Jupyter kernels are available with common astronomy software environments for processing and analytics, e.g. CASA, source finding, machine learning, plotting, etc.

In addition to the above infrastructure, Ilifu also provides a well developed software ecosystem. Software environments are encapsulated using Singularity containers. The containers allow for software stacks to be made available across distributed infrastructure. This allows for easy access to and sharing of custom software stacks, provides a flexible software environment and supports reproducible science.

Access

Access and eligibility to the Ilifu cloud computing research facility is governed by the Ilifu research facility policies. User documentation for the Ilifu facility is available here, including the process of registering an account on Ilifu.

Resources

Training videos have been recorded and are available here. These training videos cover both basic and advanced topics surrounding the use of different ilifu services.