CF Large Objects

Share: Share on FacebookTweet about this on TwitterShare on Google+Share on LinkedInShare on RedditEmail this to someonePrint this page

UPDATE: This utility has been updated to a general purpose Object Storage plugin. You can read more here.


A Cloud Foundry CLI Plugin for Working With Large Objects

Object Storage has two very powerful types of aggregate files, Static Large Objects (SLOs) and Dynamic Large Objects (DLOs). You can find a ton of great information on SLOs and DLOs in this post. Both types of large objects use a manifest file to allow users to create large files out of small chunks. This is especially useful when trying to upload files greater than OpenStack’s maximum file size, 5GB. However, there is not much tooling around the use of large objects. Previously, users needed to upload each individual chunk via an HTTP request which can be quite tedious.

We have developed a new plugin, CF Large Objects (CFLO), to make working with these large objects a much simpler task. This new Cloud Foundry CLI plugin works on IBM Bluemix as well as on other Cloud Foundry based platforms.  CFLO provides Cloud Foundry CLI users with four new commands for interacting with OpenStack Object Storage –  get-auth-info, make-slo, make-dlo, and put-object.  Now it is possible to fetch authentication credentials as well as upload DLOs, SLOs and standard objects from the CLI. This gets rid of the hassle of HTTP requests as well as the need to go through the long OpenStack authentication process. CFLO takes care of all of that for you.

Why Use Large Objects?

Great question, I’m glad you asked! In addition to uploading files that exceed the size limit, large objects have many other uses and benefits. But let’s start there. The main reason users will find themselves creating SLOs and DLOs is because their files are too large to upload directly. Both have the ability to break files into manageable chunks, the main difference being that DLOs provide the ability to add and remove chunks as desired. SLOs, on the other hand, must be provided with a user-generated manifest that defines exactly what chunks to include.

This strict manifest gives SLOs the advantage of being able to store their chunks across multiple containers. This allows us to exploit parallelism to upload and download large files faster. Because DLO manifests are actually zero-byte files that only define a container and filename prefix, all of their chunks must be located in the same container. This means users can upload chunks after the manifest is generated, but lose the ability to upload to or download from multiple containers at the same time.

How about a concrete example? We could use manifests to create sample subsets of large files without the need for redundant data uploads. This is quite useful when your files are large data sets that you may want to run some preliminary or test analysis on. Multiple manifests can take different sets of chunks (perhaps representing rows of a data set) and make different sample sets of varying sizes. Instead of having a bunch of large subsets of the same data, we just have a couple small manifest files.

Installation and Usage

CF Large Objects’ GitHub page contains detailed instructions for installation and usage of this plugin. The plugin is not currently supported in one of Cloud Foundry’s official repos, so it must be installed from binary. After downloading the binary from GitHub, running the following command from the binary’s directory will install CFLO.

Now you can use  cf help with any of the four included commands ( get-auth-info, make-slo, make-dlo, and put-object) to view the usage details.

A sample run of  make-slo  may look something like this.

Again, you can find CF Large Objects here. Feedback is much appreciated!

Share: Share on FacebookTweet about this on TwitterShare on Google+Share on LinkedInShare on RedditEmail this to someonePrint this page
Ryan Soley
Ryan began working with the jStart team as a co-op in May 2016. He is currently a senior at Penn State University pursuing his Bachelor's degree in Computer Science.
Ryan Soley

Latest posts by Ryan Soley (see all)

Ryan Soley

Leave a Reply