To Swiftly Go: An Adventure in Terabytes

We’ve been working with the SETI Institute to analyze massive quantities of radio data generated by their telescope array. Each telescope observation generates multiple 2.5 Terabyte files in just a few hours, and they want to use cloud-based analytics tools like Apache Spark to gain insight from it. A prerequisite for using cloud analytics on a dataset is, naturally, that the data needs […]

Read more

E pluribus unum – OpenStack Swift Manifest Objects

By default, the content of an OpenStack Swift object cannot be greater than 5 GB. However, you can use a number of smaller objects to construct a large object via the concept of segmentation. From OpenStack Large Object Support, “Segments of the larger object are uploaded and a special manifest file is created that, when downloaded, sends all the segments concatenated as […]

Read more