Michael,

Thanks for your comment. I will definitely be posting more about our config. We are using rrdtool with Ganglia but sending the data to graphite is certainly an option for us. We would need to write something that consumed the data stream from sflowtool and sent it to Graphite.

Our Ganglia instance is patched to handle the mod_sflow packet natively. I think the patches are available by request until things settle down with the new sFlow standards. If you're interested, I'd ask on the host-sflow-discuss mailing list available on Sourceforge.

Cheers,

-Dave

Hi,

Thanks for the post, this looks very interesting. Are you able to post a bit more about your config, especially how you have configured ganglia to handle the output from mod-sflow? Are you using rrdtool at all, or is your backend for this entirely graphite? Keenly awaiting any more info you wish to provide.

Fixed, thanks!

Broken link:

http://http//about-tagged.com/news/monitor should be http://about-tagged.com/news/monitor

This is a great post. Thank you so much for sharing this with everyone!

Shlomo,

In fact, I was under the impression that the tag I was able to give in ElasticFox was something in the API but not exposed through the Perl module. Your blog post cleared that all up and your solution is great! It would also be trivial to implement in the scripts above. I like it.

Thanks for the note.

Cheers,

-Dave

Thanks for this article. It helps show what is possible with puppet on a high level, as well as providing enough technical nitty-gritty to see how to hook things up in practice.

You say:

EC2 gives you the ability to add a "tag" description to your instances.

What do you mean - to which API are you referring? Or are you referring to the technique of using security groups to tag instances, as described here:
http://clouddevelopertips.blogspot.com/2009/06/tagging-ec2-instances-using-security_30.html

That's pretty funny. The puppet setup is really pretty simple. I just use a default node and it creates a user for the demos and puts in an ssh key so we can use the pssh tools from http://www.theether.org/pssh/ to push out any last minute directories (puppet had an 'out of file descriptors bug' when distributing big directories in the past). Most of the other main configuration comes from DHCP, though on second thought, I should have done our packages installation through puppet too.

I will try and get the configs up soon.

Cheers,

-Dave

Great writeup. Would love to hear more about your puppet setup as well.

Thanks for the great writeup. Is there any chance to get more pr0n on your puppet setup?!

Dirk,

Thanks for the information about CloudClimate. I know Hyperic has something similar. In the graphs above I was measuring connectivity between instances on the same cloud because I was trying to determine the suitability of the cloud for a compute grid type (or session clustering, or distributed cache, etc..) scenario that we can use with Terracotta.

In the link you posted about your EC2 graphs, you do monitoring of "Inter-Cloud HTTP Requests: ... the performance of the Internet network connectivity for the cloud servers by performing HTTP requests between all cloud systems every 30 seconds...For an ideal server and network environment the measured times should be constant over time. Spikes in the graph are caused by network latencies or connectivity problems."

Maybe I don't understand the idea of talking about ideal network environments when your traffic is traveling over the Internet. The wild Internet is far from an ideal networking environment. You aren't measuring anything about the cloud in this case. You're measuring the performance of the different ISPs the traffic traverses between the clouds. I guess I just don't understand the point of taking such a measurement. (other than, because it's there).

Thanks for the info. I will definitely check out NewServers.

Cheers,

-Dave

Then we took the concept of cloud monitoring to the next level with http://www.cloudclimate.com: CloudClimate displays the current performance of selected cloud hosting providers. Using our network monitoring software PRTG Network Monitor we monitor the performance of cloud hosting services Amazon EC2 (US), Amazon EC2 (EU), GoGrid CloudServers, and Newservers.

When you look at the graphs you will notice the same level of variations for EC2 instances as you have measured:

Detailed graphs for EC2 US: http://www.cloudclimate.com/ec2-us/

You will also notice that especially our test system at NewServers has almost no variations. NewServers does not use virtualization - if you purchase a server you actually get your own blade of a blade center server. This fact is the reason for the constancy of our cpu and disk measurements.

We have performed performance tests between EC2 instance (for a week) and m1.small instances have the strongest variations for our cpu, disk and memory load tests. It is no surprise since Amazon can put more m1.small instances onto one hardware server than larger instances. We think that you get much better performance and more constancy for the buck with c1.medium.

Our instance comparison:
http://www.paessler.com/blog/2009/04/03

We have performed performance tests between EC2 instance (for a week) and m1.small instances have the strongest variations for our cpu, disk and memory load tests. It is no surprise since Amazon can put more m1.small instances onto one hardware server than larger instances. We think that you get much better performance and more constancy for the buck with c1.medium.

Our instance comparison:
http://www.paessler.com/blog/2009/04/03

Then we took the concept of cloud monitoring to the next level with http://www.cloudclimate.com: CloudClimate displays the current performance of selected cloud hosting providers. Using our network monitoring software PRTG Network Monitor we monitor the performance of cloud hosting services Amazon EC2 (US), Amazon EC2 (EU), GoGrid CloudServers, and Newservers.

When you look at the graphs you will notice that especially our test system at NewServers has almost no variations. NewServers does not use virtualization - if you purchase a server you actually get your own blade of a blade center server. This fact is the reason for the constancy of our cpu and disk measurements.

It seems to me that one way to think about the variability you experienced is to decompose it in to:

1) variability due to virtualization (machine, network, disk sharing), present in any virtualized infrastructure under load.

2) additional variability due to being in the "cloud" of someone else's hardware, i.e. beyond what you would see in an in-house virtualization deployment.

I wonder how much of what you experienced was due to #1 vs #2.

Setting up a wifi connection on Windows XP isn't really all that hard, but in my experience it's easier on Mac OS X. In fact, I've had exactly the opposite experience Mark explains above: I had to dumb down wireless encryption from WPA2 (any kind) to WPA so that XP could connect.

Interesting post. I look forward to playing with Leopard fairly soon. Your comment about Wireless Networking has me at a loss. I've found Wifi to be much easier to work with on Windows than OSX. I had to dumb down my Wifi encryption WEP because the Mac had a problem with WPA.

Honestly if it weren't for Textmate I think I would dump OSX & Windows and setup Ubuntu. For now, thank God for VMWare.

You can set the system property java.rmi.server.hostname to the IP address of the server if InetAddress.getLocalHost().getHostAddress() returns 127.0.0.1. You might get some other ideas from my blog entry about RMI on multihomed machines. On the whole this is a pretty tough problem.