On the Road to the SKA: Big Data Transfer
Big-data transfer work at IDIA demonstrates the engineering tools needed for future SKA regional science data centres.
The File Transfer Service (FTS), originally developed at CERN, provides a flexible way to handle large data transfers through graphical and programmatic interfaces and multiple data protocols.
In the MeerKAT context, FTS was used to ship data from the MeerKAT archive at the CHPC data centre in Cape Town to IDIA, and from IDIA to ASTRON in the Netherlands.
Data integrity
Transfer checks help ensure data arrives complete and uncorrupted.
Parallel transfers
Multiple simultaneous connections improve throughput for large datasets.
Optimisation
FTS monitors errors and manages connection counts to maximise throughput without overloading links.
Engineering for the SKA
More than 90% of the 10G link capacity between CHPC and IDIA was demonstrated for suitable file sizes. MeerKAT is therefore also a precursor for the engineering tools needed for the global SKA, including reliable very-large-data transfer.