Creating a Cloud Foundry plugin to automate Cloudant replication

Share: Share on FacebookTweet about this on TwitterShare on Google+Share on LinkedInShare on RedditEmail this to someonePrint this page

cloudant configuration blog

In February, Lee Surprenant published Configure and run a multiregion Bluemix application with IBM Cloudant and Dyn. He explained how, when building robust cloud applications on Bluemix, one should consider a multiregion architecture. This allows developers to incorporate active failover and geographical load balancing into their applications. However, when working with multiregion applications, keeping dynamic content in sync across regions can be a little tricky. Fortunately, Cloudant has replication capabilities that enable synchronization of Cloudant databases on a global scale.

Unfortunately, configuring that replication (step 3 of the article) can be cumbersome and often painful. To set up replication between the databases you must juggle credentials for 3 different Cloudant accounts, modify the database permissions under each account, and post replication documents for each database in each region to each account. Not only was it time consuming, but the likelihood of making a mistake was relatively high.

Lee approached me about automating this task which resulted in the creation of bluemix-cloudant-replicator. bluemix-cloudant-replicator is a Cloud Foundry cli plugin written in Go that automates the process of setting up continuous replication between Cloudant databases.

bluemix-cloudant-replicator

Guiding Thoughts

When creating this utility, we really wanted to make it user friendly and as frictionless as possible. We gravitated towards a cf cli plugin because it took advantage of the user’s pre-existing Bluemix connection and allowed for easy setup/invocation.

After deciding on a cf cli plugin, Go seemed to be a natural choice to develop it in. The cf cli itself is written in Go and it includes excellent samples for writing plugins in Go as well. Additionally, Go has some qualities that makes it nice for tasks like this:

  1. Strong concurrency features that allow us to easily send concurrent http requests to each Bluemix region
  2. A simple cross-compilation story that allows us to easily build self-contained executables for multiple platforms

Cloud Foundry allows users to create cli plugins in a very friendly way. The two things that I found most useful while developing my plugin were the basic_plugin example and the Plugin API. With these two resources alone and a little knowledge of Go, you can quickly create custom plugins to enhance your cf cli experience.

The Mechanics

In order for replication to be set up for our Cloudant databases, we must first have a way of interacting with Cloudant. Fortunately for us, there exists a Cloudant API and Go has a nice package for handling http requests/responses. With these resources I automated the following process:

  1. Retrieve a cookie for each account to authenticate all subsequent requests
  2. Create a “_replicator” database for every account
  3. Retrieve and modify permissions for each selected database in each account. This is done to give the other accounts permission to read and replicate to one another.
  4. Create pair-wise replications between the databases in each region.
  5. Delete the cookies retrieved in step 1

Specifically, the replication is accomplished by posting a replication document with the following JSON structure to each account’s “_replicator” database:

This document tells Cloudant to continuously check for changes applied to the REMOTE source and apply the same changes to the LOCAL target.

The image below shows how your databases will be linked after cloudant-replicate has executed.

bluemix-cloudant-replicator_diagram_2

For more information about the processes and the calls being sent, follow the links below to the Cloudant API docs:

Authentication, ReplicationCreating DatabasesCreating Documents

Assumptions and Limitations

For our implementation we made several assumptions to fit our specific use-case. We did so to minimize the amount of required configuration (at the expense of flexibility). As a result, bluemix-cloudant-replicator only needs 3 things from the user:

  1. a multiregion application containing a Cloudant service in each region
  2. the list of databases to replicate between regions (often a list of 1)
  3. the user’s Bluemix password (used to access the various Bluemix regions)

After these arguments are passed in, or interactively gathered, everything else is inferred.

  • We assume that the org, space, and application name are the same in each region
  • We use the current cf target to set the initial values for the org and space name
  • We use the application’s first Cloudant service binding to retrieve the Cloudant credentials
  • We assume the Cloudant databases exist (again with the same name) in each region

We would love to hear what you think about our design. Do you have any use-cases where our plugin is too restricting?


If you want to know more about the jStart team please visit our site, and talk to us more from our contact page.

Share: Share on FacebookTweet about this on TwitterShare on Google+Share on LinkedInShare on RedditEmail this to someonePrint this page
Luke Stack
Luke joined IBM's jStart team as a co-op in January 2016 . Luke is a junior, pursuing Bachelor’s of Science degrees in both Computer Science and Mathematics at Appalachian State University.
Luke Stack
Luke Stack

Latest posts by Luke Stack (see all)

One comment

Leave a Reply

Your email address will not be published. Required fields are marked *