The Cloud is a way of thinkingfeeling and implementing  platforms and software that enable us to move past the concerns of unitary machines and into those of autonomic services.

There are many blogs, books, talks and presentations which define ways to build a ‘cloud’ architecture for your services. Sadly most concentrate on how to work within one framework or another built by corporations trying to tie you to their way of thinking.

The cloud has always been much more than this. I speak of course of RackSpace, Amazon and any number of other ‘cloud’ providers. Put simply:

Neither virtualization nor tooling defines the Cloud

The plethora of tools and tooling frameworks which surround these group of companies is immense. All of them designed to fix the woeful inadequacies of the underlying platform when dealing with the Cloud idea.  These companies and their products have their place in the world and many a business owes them their existence and prosperity. However:

The general acceptance of a method of execution does not define the success of understanding of an idea.

These companies do not offer you access to a real cloud. They offer large-scale virtualization technologies which mimic what a cloud should do – it’s a bit like putting an after market exhaust and a dump valve on an on old Nissan. Sure it sounds good but you really won’t pull away from anything very fast at all.

The speed, ease, reliability and flexibility of deployment, scaling, monitoring and operation within these platforms is inadequate.

The idea isn’t to scale in 30 minutes. It’s to allow the machines to scale themselves within context and constraint within minutes of understanding the need to do so.

The idea isn’t for a humans or machines to watch simple statistics to aid decision-making. It’s to allow your services to gain a contextual understanding of each constituent part’s operation and allow the convergent intelligence inherent to make real-time decisions.

The idea isn’t to re-invent the wheel when it comes to deployment but to leverage decades of experience to make sure we move past this triviality and tackle the hard problems.

That’s impossible in most current ‘cloud’ platforms unless dealing with the most trivial of services.  The last decade of my professional life has been in one ‘cloud’ environment or another. Uniformly, all have failed to show any glimmer of real understanding of the Cloud idea. Till last year, but more on that later.

This is the small, gentle introduction to what will be a series showing you how to build a real cloud services architecture.

You’ll need a GitHub account, a Joyent public cloud account, a local CFEngine installation and some understanding of Python and Bash.

It will be fun, I hope you’ll join me .

Follow me (@khushil) or bookmark this blog to get each installment as it arrives. Click on the ‘About Me‘ page to find out how to get in touch with me.


The term ‘cloud‘ is bandied about quite readily these days. To my mind there is one operating system that can actually be called a ‘cloud operating system‘ – SmartOS. I’ll dedicate another post to the differences that distinguish SmartOS from other OS in this ‘cloud’ space, but for now, take my word for it 🙂

I’m going to be setting up a medium sized infrastructure based on Joyentsmartmachines‘ – which you can think of as non-Global zones (for those from the Solaris world).

So, let’s get started.

First, we create our smartmachine – I’ve opted for a 2vCPU and 2GB machine for now, I can easily resize as required in the future thanks to SmartOS. Once I’ve logged in I make sure my base OS is up to date:

pkgin -fy update && pkgin -fy upgrade

This should pull down anything needs upgrading and leave you with a nice new smartmachine ready to roll.

Now, handily the chaps at Joyent have provided a quick guide to install CFEngine which I’ll be basing this tutorial around at this stage. It’s pretty basic but it get’s you to a good place to move forward from. Bear in mind that it seems to be geared toward installing a Global Zone within the Joyent SDC product.

First we need to install some pre-req’s using the pkgin installation system on smartmachines:

pkgin in gcc47 gmake tokyocabinet openssl pcre
export PATH=/opt/local/bin:/opt/local/sbin:/opt/local/gcc47/bin:$PATH

With those out of the way, let’s get the latest source and get it built:

cd /tmp
curl -k -o cfengine-3.5.1.tar.gz 'https://cfengine.com/source-code/download?file=cfengine-3.5.1.tar.gz'
gtar xfvz cfengine-3.5.1.tar.gz

This next step strays from the Joyent tutorial as we specify the working directory for CFEngine3 – a good idea IMHO.

cd cfengine-3.5.1

Whilst compiling this for ouselves, one of my engineers found a small bug as documented in https://cfengine.com/dev/issues/3097 – follow the advice therein for the fix.

Now issue the following command – all on one line:

CPPFLAGS="-I/opt/local/include" LDFLAGS="-L/opt/local/lib -R/opt/local/lib" ./configure --prefix=/opt/sfw/cfengine3 --enable-static --with-workdir=/var/cfengine

Now make and install CFEngine3

gmake
gmake install

You need to give it’s default policy files. These would normally live in /var/cfengine/masterfiles but with the source install, they get dumped in /opt/cfengine3/share/CoreBase – so go ahead and

cp -R /opt/sfw/cfengine3/share/CoreBase/* /opt/sfw/cfengine3/var/cfengine/masterfiles/

Now take a look in /opt/sfw/cfengine3/var/cfengine/masterfiles/def.cf and in particular at the ‘acl => slist‘ clause.

Make sure that includes the network or IP of the install machine itself. Once you’re happy with that clause we can go ahead and bootstrap :

/opt/sfw/cfengine3/bin/cf-agent --bootstrap XXX.XXX.XXX.XXX

If no errors occur you will get a ‘Bootstrap completed’ message and you’re away!

That’s it – you’ve installed and got the hub to bootstrap.

Now comes the hard bits – the configuration!