Monday August 17, 2009

A framework for running anything on EC2: Terracotta tests on the Cloud - Part 1

One of the most fun things about working as the CTO's SWAT team is he's always thinking of new ideas and things he wants to do. If you've ever met Ari, you know the guy is an idea fountain.

One recent project he asked me to do was run a large test of Terracotta on a cloud. Any cloud. So I set out to find an appropriate cloud environment and a way to run one of our session clustering tests. What I wound up with was a framework that can be used to scale anything on EC2 with relative ease.

I was able to leverage a few sysadmin tools that I already knew about and some I was just learning, to create an environment where running an arbitrary size test is as simple as saying "runTest < number >" and the appropriate number of AMIs would be launched, configured, and the test would be run. In the end, collecting the results was just as simple.

The nice thing about running Terracotta on the cloud is that it's dead simple. There are a few reasons for this. One is that all clustering is done over standard TCP/IP, no multicast. One of the initial parts of the task was to run a comparison of Tomcat session clustering with Terracotta but I couldn't get it to work. There is no multicast traffic on EC2 so that method was eliminated. I also tried Tomcat's support for unicast clustering but it is so poorly documented that I gave up pretty quickly. I don't know if it even works. I expect this is a problem for any clustering/datagrid/distributed cache product that relies heavily on multicast support. Another great thing, is that the recommended way for Terracotta clients to get their configurations is to pull it directly from the server. This is a huge advantage in the cloud and anywhere you are trying to configure massive numbers of machines. As you'll see below, Puppet uses the same model.

This first blog post will primarily cover what it took to come up with a machine image for running the simple tests. For the actual results of the tests themselves, you'll have to stay tuned for Part 2.

The Building Blocks

CentOS

The first task was to pick an operating system. I'd run Ubuntu on the cloud with the smart guys at Tubemogul before. But I've been really intrigued with a lot of the stuff coming out of the Fedora labs, specifically, Cobbler and Func (even though I ultimately did not wind up using either). CentOS meant those were a possibility, and RedHat is also a supported Terracotta platform. I chose to take one of the excellent CentOS images made public by the guys at Rightscale (who also have a REALLY impressive platform) and modify it so that it had all the software we needed, as well as any configurations necessary for management and security.

EC2

As you already know by now, we ran the tests on EC2. The reason I mention it here is that the framework behind the tests is probably applicable to any cloud computing environment as long as it has an API. There are a few code changes that would be required but it shouldn't be too hard to abstract the underlying cloud environment away so that requesting more machines is as simple as that.

Puppet

I must admit when I began the project I was a complete Puppet newbie. After the project, I'd moved up to intermediate. This is probably the main reason that Puppet wasn't used even more. The great thing is now that all the machines in the cloud actually talk to the puppetmaster, it can be embraced and extended to take advantage of all the great things Puppet can do. (Puppet is my new sysadmin's best friend, seriously)

PSSH

Pssh is a great piece of Python written by Brent Chun over at The Ether.Org. It is actually a suite of tools like pssh, pslurp, prsync, and pcsp that will allow you to ssh onto dozens of machines in parallel (by default 32) and perform the above operations. We could have done some of these things with Puppet and puppetrun (though before 0.25 Puppet is terrible at delivering directory trees), but instead pssh was used to run the tests, whereas Puppet was used to configure the OS.

Perl and Shell

The last parts were plain old Perl and Shell taking advantage of the Net-Amazon-EC2 Perl module. Using some scripting in both languages, I was able to have the AMIs auto-configure themselves on launch to be ready for the cluster.

Making it work

The first task was to be able to have the machines join our "cluster" when they are launched. This process works like this:

1. The AMI is launched on EC2 (we used a c1.medium instance for this) to play the role of the Terracotta server/puppetmaster.
2. A credentials file is edited to contain the information needed to be able to spawn more copies of the AMI
3. A script is run that fixes puppet and creates a TC user, then a tarball is extracted as root that sets up the puppet config and some cron jobs necessary to make everything work. The reason this is done manually is that we don't want each system to try and become the puppetmaster or the munin-host. This one time operation allows us to use the same AMI for both master and workers.
4. The keys and software for the Terracotta user to run are copied onto the launched AMI. For security reasons we didn't want to store private ssh keys on the AMI. Additionally, we didn't want to have to build a new AMI every time a new version of Terracotta or Tomcat were released. With this method, we can stay current and the only limitation is the speed of the file copy. We only have to do this once however, as all the remaining file distribution is done from the master and occurs at gigabit (see my previous post on this topic) speeds within the cloud. There could definitely be a method developed where the host pulls files/configuration from S3 and gets the latest version, but for the purposes of our test, we went the simple route.
5. Now that all the pieces are in place, we launch the workers from the master using a Perl script and they all call in to the puppetmaster and are configured in such a way that they can be managed from the master itself. This step could also be performed instead of the previous one as there is no dependency on the Terracotta user for the machines to be configured. We used the m1.small image for these tests.
6. Additionally, all the workers are configured as Munin nodes so that we can track the various metrics of system activity during the lifetime of the launched AMIs.
7. Finally the tests can be run from the Terracotta user on the master.

After the AMI is launched, we logon as root and run a script (called newmaster.sh, listed below) that will setup the machine as the master, install some cronjobs, the puppet config, and launch the appropriate daemons. We also need to put the proper credentials in /mnt/creds.cfg (AWS keys, AMI type, keypair name). This file is used by the launching script. Here is the script we run to create the master out of our AMI.

#!/bin/bash

# setup this machine as a new L2 master/puppetmaster/etc.

/sbin/service puppet stop
/sbin/service puppetmaster start
/usr/sbin/groupadd -g 481 tc
/usr/sbin/useradd -g 481 -u 481 -c "Terracotta user" -d /mnt/tc -m  tc

/bin/cp /etc/ec2/creds.cfg.template /mnt/creds.cfg
echo "** You must edit /mnt/creds.cfg to be able to launch workers **"

cd /
tar -xf ~root/l2master.tar

Here is the script used to launch the hosts:

#!/usr/bin/perl -w

use strict;
use Net::Amazon::EC2;
use MIME::Base64;
use Config::Tiny;
use LWP::Simple;

die "launch.EC2.workers.pl numinstances" unless (scalar @ARGV == 1);


my $Config = Config::Tiny->new();
$Config = Config::Tiny->read('/mnt/creds.cfg') || die "Can't read credentials file";

# get these from creds.cfg
my $ec2 = Net::Amazon::EC2->new(
    AWSAccessKeyId => $Config->{_}->{accesskey},
    SecretAccessKey =>$Config->{_}->{secretkey}  
);


my $workernum=$ARGV[0];

my $url = "http://169.254.169.254/latest/meta-data/local-hostname";
my $hostname = get $url;
die "Couldn't get $url" unless defined $hostname;


print "Launching $workernum instance(s) with master node $hostname\n";

my $userdata = encode_base64("L2ip=$hostname");

# launch the workers
my $instance = $ec2->run_instances(ImageId => $Config->{_}->{workerami},
   MinCount => $workernum, MaxCount => $workernum, KeyName => $Config->{_}->{keypair},
  UserData => $userdata);
# should put some error checking here

The most important thing to note here is that we identify the master server with the user-data parameter. This is the key to the entire cloud deployment. When the machine boots, it executes the following code which populates the puppetmaster and munin host information on the worker. The munin configuration could be done by Puppet itself, and probably should be. This might require the creation of an Augeas lens for the munin config file.

#!/bin/bash
# tcec2init        Init script for setting up customizations on EC2
#
# Author:       Dave Mangot 
#
# chkconfig: - 89 02
#
# description: gets system ready for TC/EC2 management

PATH=/usr/bin:/sbin:/bin:/usr/sbin
export PATH

RETVAL=0

# Source function library.
. /etc/rc.d/init.d/functions


start() {
        echo -n $"Getting my configs: "
        RETVAL=$?
        /usr/local/bin/get.EC2.userdata.pl
        [ $RETVAL = 0 ] 
        return $RETVAL
}

stop() {
        echo -n $"Nothing"
        RETVAL=$?
        echo
        [ $RETVAL = 0 ] 
}

reload() {
        echo -n $"Nothing"
        RETVAL=$?
        echo
        return $RETVAL
}

restart() {
    stop
    start
}


case "$1" in
  start)
        start
        ;;
  stop) 
        stop
        ;;
  restart)
        restart
        ;;
  reload|force-reload)
        reload
        ;;
  status)
        echo ""
        RETVAL=$?
        ;;
  *)
        echo $"Usage: $0 {start|stop|status|restart|reload|force-reload"
        exit 1
esac

exit $RETVAL

Very simply this is an init.d script that starts early in the boot cycle. This way the values for the Puppet startup and Munin startup are already configured before those daemons try and start.

#!/usr/bin/perl -w 

use strict;
use LWP::Simple;
use Tie::File;

sub setupPuppet {

    my $puppetMaster = shift;

    my @lines;
    tie @lines, 'Tie::File', "/etc/sysconfig/puppet" or die "Can't read file: $!\n";
    foreach ( @lines ) {
        s/\#PUPPET\_SERVER\=puppet/PUPPET_SERVER=$puppetMaster/;
    }
    untie @lines;
}


sub setupMunin {

    # could have made this a function that took munin/puppet arg but, why bother, so trivial
    my $notNode = shift;
    
    # we want notNode as specially formatted IP, thankfully, Perl will allow us to construct this easily
    my (undef,undef,undef,undef,@addrinfo) = gethostbyname($notNode);
        
    my ($a, $b, $c, $d) = unpack('C4', $addrinfo[0]);
    my $modNode= "^" . $a . "\\." . $b . "\\." . $c . "\\." . $d . "\$";
                 
    my @lines;
    tie @lines, 'Tie::File', "/etc/munin/munin-node.conf" or die "Can't read file: $!\n";
    foreach ( @lines ) {
         s/\\.1\$/\\.1\$\nallow $modNode\n/;
    }
    untie @lines;
 }

sub setupDaemons {

     my $masterIP = shift;
     
     setupMunin($masterIP);
     setupPuppet($masterIP);     
     }
      
      
      
my $req = get 'http://169.254.169.254/latest/user-data';
my @returned = split /\|/, $req;

# here we store each keypair in /var/local/key
foreach my $item (@returned) {
    my @datakey = split /=/, $item;
    chomp($datakey[1]);
    open EC2DATA , ">/var/local/$datakey[0]" or die "can't open datafile: $!";
    print EC2DATA "$datakey[1]\n";
    close EC2DATA;

    

    setupDaemons($datakey[1]) if ($datakey[0] eq "L2ip");
}

This script is one of the keys to the entire infrastructure. We are passing in the IP address of the L2 (the master) but we could also pass in other data as key-value pairs. If an 'L2ip' key is detected then we will setup the Munin and Puppet daemons. If any other key-value pairs are passed in (by modifying the launch script) then they will also be placed as a file in /var/local/. This way if we have other software on our system that needs access to any of those values, they will always be available. In reality, we can simulate much of this functionality in Puppet if desired. We will use the L2ip file to identify the Terracotta server when we run our sessions test.

Puppet

autosign

While it's great that the workers know how to find the master, the master still has to recognize them as valid clients and accept their certificates as valid workers. For this we use the autosign.conf ability of Puppet. We populate /etc/puppet/autosign.conf using the following cron entry and script.

* * * * * /usr/local/bin/get.EC2.workers.pl > /etc/puppet/autosign.conf

#!/usr/bin/perl -w

use strict;
use Net::Amazon::EC2;
use MIME::Base64;
use Config::Tiny;
use Sys::Hostname;


exit 1 unless (-e "/mnt/creds.cfg");

my $Config = Config::Tiny->new();
$Config = Config::Tiny->read('/mnt/creds.cfg') || die "Can't read credentials file";

my $ec2 = Net::Amazon::EC2->new(
    AWSAccessKeyId => $Config->{_}->{accesskey},
    SecretAccessKey =>$Config->{_}->{secretkey} 
);

open MUNIN, ">/etc/munin/munin.conf" or die "cant't open munin.conf: $!";
print MUNIN <describe_instances;      

foreach my $reservation (@$running_instances) {
    foreach my $instance ($reservation->instances_set) {
        if ($instance->private_dns_name) {
          # don't want to use the master as a worker node
           next if ($instance->private_dns_name =~ /$hostname/); #. "compute-1.internal");
           print $instance->private_dns_name . "\n";
           
           my (undef,undef,undef,undef,@addrinfo) = gethostbyname($instance->private_dns_name);
           my ($a, $b, $c, $d) = unpack('C4', $addrinfo[0]);
           my $nodeaddress= $a . "\." . $b . "\." . $c . "\." . $d;
           my $nodename = $instance->private_dns_name;
           print MUNIN "[tcscale;$nodename]\n\taddress $nodeaddress\n\tuse_node_name yes\n\n";
        }
    }
 }

close MUNIN;

This script updates the munin config on the master and prints out all the workers hostnames to STDOUT. The output when run from cron is redirected to /etc/puppet/autosign.conf so that when the clients call in after booting, they will automatically be accepted by Puppet. From a security perspective, there is an obvious race condition if an attacker can guess the hostnames of the machines that will be assigned by EC2, or is able to determine those hostnames before those hosts can boot initially. One possibility would be to also write some IP tables rules to wall off anyone but the proper IPs. This way, an attacker would also have to spoof an IP address which could potentially be more difficult to execute successfully in the EC2 environment if they have proper security mechanisms in place. For the purposes of our test, an attacker could disrupt the test but they would not gain access to any of our virtual infrastructure as a result.

puppet config

import "classes/*.pp"


node default {
        include tcuser
        }

Very simple, all our test nodes are configured with a default configuration. Obviously we can add whatever classes we wish into this configuration to do things like install app servers, db server, load balancers, etc. or any of the configurations necessary to get those services up and running. This is what is great about puppet, and what makes it so powerful for these arbitrarily sized cloud deployments.

class tcuser {
    group { "tc":
        ensure => "present",
        provider => 'groupadd',
        gid => 481,
        name => "tc"
    }
    user { "tc":
        comment => "TC user",
        gid => 481,
        home => "/mnt/tc",
        name => "tc",
        shell => "/bin/bash",
        managehome => true,
        uid => 481
    }
    file { "mnttcssh":
        name => "/mnt/tc/.ssh",
        owner => tc,
        group => tc,
        ensure => directory,
        mode => 700,
        require => File["mnttc"];
    }
    file { "mnttc":
        name => "/mnt/tc",
        owner => tc,
        group => tc,
        ensure => directory,
        mode => 700,
        require => User["tc"];
    }
    file { "/mnt/tc/.ssh/authorized_keys":
        content => "ssh-dss AAAA..truncated..meI8A==",
        owner => tc,
        group => tc,
        mode => 700,
        require => File["mnttc"]
    }

}

Though we actually store the private key off the cloud for security reasons, this creates the user under which we will be running all our tests (not root!). Once this is created, we do all our management of the remote workers as the tc user.

Munin

One of the best things about Munin is that you get so many great graphs and data collected for very, very, little effort. On the master machine, our /etc/munin/munin.conf is updated with the same script as the autosign.conf to look something like this:

dbdir   /var/lib/munin
htmldir /var/www/html/munin
logdir  /var/log/munin
rundir  /var/run/munin
tmpldir /etc/munin/templates
[localhost]
    address 127.0.0.1
    use_node_name yes
        
[tcscale;domU-12-31-39-03-C1-06.compute-1.internal]
        address 10.249.194.244
        use_node_name yes

The clients are just updated on boot to point at the munin master. I like to look at my munin graphs through SSH port forwarding (you will need to start Apache which is not started by default). You could also open up the EC2 firewall to allow traffic in from your location.

Additional software

There are a few additional packages that are installed that are convenient for testing or that could be used by workers for testing or deployments. Some examples:

• nttcp
• haproxy
• iozone
• Sun JDK 1.6 (in /usr/java)

Setting up the Sessions Test

#!/bin/sh

# sudoers has a NOPASSWD field for this, want a clean environment for each test
sudo /usr/sbin/puppetca --clean --all > /dev/null

/usr/local/bin/launch.EC2.workers.pl $1
workerhosts=`cat /etc/puppet/autosign.conf | wc -l `
while [ $workerhosts -lt $1 ] 
do
        sleep 60
        workerhosts=`cat /etc/puppet/autosign.conf | wc -l `
done
echo "determining L1s and  load generators"
~/bin/partworkers.sh
if [ $? != 0]; then
    exit $?
fi
~/bin/tcgo.sh
~/bin/tomcatgo.sh
~/bin/lggo.sh

We'll cover the actual content of these scripts in the next post, which will actually be running Terracotta on the cloud, as opposed to setting up the infrastructure. Briefly, we clear out our puppet clients between runs as we use fresh workers for each run of the test (though, this is by no means a requirement). We launch any number of workers and wait until they are all up and running. We partition our workers into load generators and Tomcat servers. We launch, Terracotta, then Tomcat, then the load generators. Simple!

Conclusion

As you can see, we have a very flexible infrastructure available to us now on EC2. We can launch any number of clients from our "master" server and control them as well. They can be partitioned any way we like to serve any number of roles and we keep track of them in a central location which allows us to make decisions about our "cluster" and take actions if we like. For the Terracotta test, we'll be able to show how Terracotta scales linearly in the cloud and just how simple it is to configure for this purpose. Obviously, this is not a complete solution for anything other than running our current test. Here are some things we could do to extend our functionality:

• The newest puppet (0.25) release candidate has REST so could deploy directories that way instead of using prsync (which we'll see in part 2).
• EC2 gives you the ability to add a "tag" description to your instances. We could add that capability to the Net::Amazon:EC2 module and classify servers in puppet based on tag (e.g. load generator, load balancer, tomcat instance). This would allow us to deploy only the software we need to each node.
• We could create an Augeas lens for munin.conf. This way we wouldn't have to edit the file with Perl but simply define the configuration in Puppet. (yes, I am currently in love with Puppet)
• The timescales for munin by default are really long. They also cannot be chosen for some reason. For our short tests, it's not really that effective. For a longer running task, it would probably be fine. We could modify the Munin source code for a shorter timescale or just choose something else. I chose Munin to start with because it give you so much for so little effort. Caveat emptor.
• We could improve our error checking, of course.
• Everything we need for autoscaling is already there. We can do node classification in puppet to add webservers, app servers, etc. We could use metrics from Munin to determine when we need to add nodes. Currently there is no way to remove nodes but that would be pretty quick to script. We could also monitor a load balancer like haproxy either locally or remotely and add nodes based on traffic.
• We could also rebuild the AMI as an x86_64 architecture. When running the Terracotta tests, we just kept adding nodes until the L2 eventually fell over. We could scale much higher with a bigger EC2 instance. Currently, we are limited to very small virtual machines. We could also run the Terracotta FX product and scale the Terracotta server across nodes. Combining bigger machines and FX would give us an ridiculous amount of capacity.

There are probably a number of different directions this could take and I'm sure people have really good ideas. If you'd like to take the AMI for a spin it's available on EC2 as ami-e14dac88 (search for tcscale). Enjoy and stay tuned for part 2!