Creating a Cloud Foundry plugin to automate Cloudant replication
In February, Lee Surprenant published Configure and run a multiregion Bluemix application with IBM Cloudant and Dyn. He explained how, when building robust cloud applications on Bluemix, one should consider a multiregion architecture. This allows developers to incorporate active failover and geographical load balancing into their applications. However, when working with multiregion applications, keeping dynamic content in sync across regions can be a little tricky. Fortunately, Cloudant has replication capabilities that enable synchronization of Cloudant databases on a global scale.
Unfortunately, configuring that replication (step 3 of the article) can be cumbersome and often painful. To set up replication between the databases you must juggle credentials for 3 different Cloudant accounts, modify the database permissions under each account, and post replication documents for each database in each region to each account. Not only was it time consuming, but the likelihood of making a mistake was relatively high.
Lee approached me about automating this task which resulted in the creation of bluemix-cloudant-replicator. bluemix-cloudant-replicator is a Cloud Foundry cli plugin written in Go that automates the process of setting up continuous replication between Cloudant databases.
When creating this utility, we really wanted to make it user friendly and as frictionless as possible. We gravitated towards a cf cli plugin because it took advantage of the user’s pre-existing Bluemix connection and allowed for easy setup/invocation.
After deciding on a cf cli plugin, Go seemed to be a natural choice to develop it in. The cf cli itself is written in Go and it includes excellent samples for writing plugins in Go as well. Additionally, Go has some qualities that makes it nice for tasks like this:
- Strong concurrency features that allow us to easily send concurrent http requests to each Bluemix region
- A simple cross-compilation story that allows us to easily build self-contained executables for multiple platforms
Cloud Foundry allows users to create cli plugins in a very friendly way. The two things that I found most useful while developing my plugin were the basic_plugin example and the Plugin API. With these two resources alone and a little knowledge of Go, you can quickly create custom plugins to enhance your cf cli experience.
In order for replication to be set up for our Cloudant databases, we must first have a way of interacting with Cloudant. Fortunately for us, there exists a Cloudant API and Go has a nice package for handling http requests/responses. With these resources I automated the following process:
- Retrieve a cookie for each account to authenticate all subsequent requests
- Create a “_replicator” database for every account
- Retrieve and modify permissions for each selected database in each account. This is done to give the other accounts permission to read and replicate to one another.
- Create pair-wise replications between the databases in each region.
- Delete the cookies retrieved in step 1
Specifically, the replication is accomplished by posting a replication document with the following JSON structure to each account’s “_replicator” database:
This document tells Cloudant to continuously check for changes applied to the REMOTE source and apply the same changes to the LOCAL target.
The image below shows how your databases will be linked after cloudant-replicate has executed.
For more information about the processes and the calls being sent, follow the links below to the Cloudant API docs:
Assumptions and Limitations
For our implementation we made several assumptions to fit our specific use-case. We did so to minimize the amount of required configuration (at the expense of flexibility). As a result, bluemix-cloudant-replicator only needs 3 things from the user:
- a multiregion application containing a Cloudant service in each region
- the list of databases to replicate between regions (often a list of 1)
- the user’s Bluemix password (used to access the various Bluemix regions)
After these arguments are passed in, or interactively gathered, everything else is inferred.
- We assume that the org, space, and application name are the same in each region
- We use the current cf target to set the initial values for the org and space name
- We use the application’s first Cloudant service binding to retrieve the Cloudant credentials
- We assume the Cloudant databases exist (again with the same name) in each region
We would love to hear what you think about our design. Do you have any use-cases where our plugin is too restricting?