There’s times when we need to enforce adhoc control over our machines. This blog post by Alex Honor does a way better job than I could to explain why we still need to worry about adhoc command execution, but a quick summary would probably be the following:

1. Sometimes, we just need to execute commands ASAP - unplanned downtime and adhoc information gathering both spur this activity.
2. We need to orchestrate any set of commands across many machines - things like getting your DB servers up and prepped before your web servers come online is one example of this.

RunDeck can handle these two scenarios perfectly. It’s a central job server that can execute any given set of commands or scripts across many machines. You can define machines via XML, or use web servers that respond with a set of XML machine definitions (see chef-rundeck for a good example of this.) You can code in whatever language you want for your job scripts - RunDeck will run them on the local machine if the execute bit is set (or, if you’re running jobs across many machines, it will SCP them onto the machine and execute them via SSH.)

You can define multiple job steps within a job (including using other jobs as a job step) - thus, you can, for example, have a job step that pulls down your web application code from git across all web application servers, another job step that pulls down dependencies, then one more to start the web application across all of those machines, all of which is coordinated by the RunDeck server. As another example, you can also coordinate scheduled jobs that provision EC2 servers at a certain time of the day, run some kind of processing job across all of those machines, then shut all of them down as soon as the job is finished.

There’s another way these jobs can be started, not only just by manual button pushing and cron schedules: they can also be started by hitting a URI via the provided API with an auth token. To expand on the previous example, if say you don’t have a reliable way to check to see if the processing is finished from RunDeck’s point of view, you can probably have some kind of job that’s fired from the workers, which hits the API and tells RunDeck to run a ‘reap workers’ job.

mcollective vs. RunDeck?

A few people have asked me how this compares with mcollective.

I would probably say that they’re actually complimentary with each other - mcollective is incredibly nice, in that it has a bit of a better paradigm for executing commands across a fleet of hosts (pub/sub vs. SSH in a for loop.) You can actually use mcollective as an execution provider in RunDeck, by simply specifying ‘mco’ as the executor in the config file (here’s a great example of this.) RunDeck is nice, in that you can use arbitrary scripts with RunDeck and it’ll happily use them for execution - plus, it has a GUI, which makes it nice if you need to provide a ‘one button actiony thing’ for anyone else to use. RunDeck can also run things ‘out of band’ (relative to mcollective) - for example, provisioning EC2 machines is an ‘out of band’ activity (though you can certainly implement this with mcollective as well, mcollective’s execution distribution model doesn’t seem to ‘fit’ well with actions that are meant to be run on a central machine.)

But again, you can tie mcollective with RunDeck, and they’ll both be happy together. I can see right away that the default SSH executor will likely be a pain in the ass once you get to about 100+ machines or so.

Conclusion

RunDeck is pretty damn nice. We’ve been putting it through its paces in our staging environment, and it’s held up very nicely with a wide variety of tasks that we’ve thrown at it. The only worries I have are the following:

1. Default SSH command executor will likely start sucking after getting to about 100+ machines.
2. Since it can execute whatever you throw at it, it’s possible that you might end up with yet another garbled mess of shell and Perl scripts. This is probably better solved with team discipline, however.
3. There doesn’t seem to be a clear way to automate configuration changes for RunDeck - thus, be sure to keep good backups for now (though, this is probably because of my own ignorance - the API is pretty full featured, and I believe you can add new jobs via the API, but it would’ve been awesome to be able to just edit job definitions in a predictable location on the disk.)

Despite these worries, I highly recommend RunDeck - it’s helped us quite a bit so far, it’s quick to get setup, and quick to start coding against for fleet wide adhoc control.

So, I’ve transitioned from Jekyll to Octopress. It comes with a really nice default HTML5 template that I’ve edited a bit (changed some fonts, colors, etc.), and has a decent fallback for mobile devices, all of which was lacking in my last site design with Jekyll (if anyone’s interested, the old site is still on my GitHub account.)

What’s awesome about Octopress, anyway?

• It comes with a nice set of Rake tasks for handling common tasks, such as generating a new site and rsync’ing the pages over to a server. Also comes with Rake tasks for creating new posts and pages.
• The repository is very well laid out, with nice separation between themes and content.
• The default theme comes with scss files, which is way better to deal with than regular old css.
• Comes with a sane set of plugins that make things look awesome, such as code snippets, image embedding, and HTML5 video embedding (with Flash video fallback.)

Anyway, the site looks way better now that it’s on Octopress, and it’s a bit easier to maintain as well.

Over a couple of caffeine induced nights, I’ve converted a handful of our Puppet modules over to Chef cookbooks (by hand) to see how it’d all turn out. We’ve not yet decided whether we will actually use Chef in production, but I figure that if I’ve lowered the bar of entry, it would make the decision much easier (we’d still have to worry about whether the technical merits justify putting in time to transition everything over to Puppet, but at least the initial cost of converting things over isn’t as painful as it would’ve been.)

Here’s a sample snippet of Puppet code that I’ll be referring to for the rest of the article:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

class foobar {
  $version = "1.0.0"
  $tarball = "foobar-${version}.tar.gz"
  $url     = "http://example.com/${tarball}"

  file { "/opt/foobar":
    ensure => directory,
    mode   => 0755,
  }

  exec { "download tarball":
    path    => "/bin:/usr/bin:/sbin:/usr/sbin",
    cwd     => "/opt/foobar",
    command => "wget ${url}",
    creates => "/opt/foobar/${tarball}",
  }

  exec { "extract tarball":
    path    => "/bin:/usr/bin:/sbin:/usr/sbin",
    cwd     => "/opt/foobar",
    command => "tar zxvf ${tarball}",
    creates => "/opt/foobar/foobar-${version}",
    before  => File["/opt/foobar/foobar-${version}/config.cfg"],
  }

  file { "/opt/foobar/foobar-${version}/config.cfg":
    content => template("foobar/config.cfg.erb"),
  }

  file { "/opt/foobar/foobar-${version}/staticfile":
    source => [ "puppet:///modules/foobar/staticfile.${hostname}",
                "puppet:///modules/foobar/staticfile" ],
  }
}

Puppet resources port pretty well over to Chef resources - you simply have to make sure that you’re using the right Chef resource when rewriting it (for example, you use “file” in Puppet for templates, regular files transferred from the server, regular files managed only locally, and directories - these are all separate things in Chef, with resources such as “cookbook_file”, “template”, “file”, “directory”, and “remote_directory.”) As for variables, this might be a bit of a judgment call - I’ve mostly thrown tunables into node attributes, and leave temporary recipe-specific variables where they were in Puppet. In this case, I’d likely throw $version into “node[:foobar][:version]” as an attribute - the tarball name, if the naming convention doesn’t change between versions, could probably stay inside the recipe itself. The URL might be a tunable too, and so could probably be stuffed into “node[:foobar][:url]” - what if you had mirrors of these files in each geographical area?

Here’s what our code will look like so far, given the above:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

# foobar/attributes/default.rb:
default[:foobar][:version] = "1.0.0"
default[:foobar][:tarball] = "foobar-#{node[:foobar][:version]}.tar.gz"
default[:foobar][:url]     = "http://example.com/#{node[:foobar][:tarball]}"
# Replaces the following Puppet code:
# $version = "1.0.0"
# $tarball = "foobar-${version}.tar.gz"
# $url     = "http://example.com/${tarball}"

# foobar/recipes/default.rb:

directory "/opt/foobar" do
  mode "0755"
end
# Replaces the following Puppet code:
# file { "/opt/foobar":
#   ensure => directory,
#   mode   => 0755,
# }

Now, what do we do with the exec resources? There’s still a few things that make sense to port directly to Chef’s “execute” resources - things like updating database users, running a program to alter configuration (think “make” for creating sendmail database files), or extracting tarballs. However, we do have one exec resource that can be replaced by a regular Chef resource - the exec resource that downloads a tarball onto local disk. This is supplanted by the “remote_file” resource - keep in mind that we likely do not want to keep downloading the file on each Chef run (we guarantee this in Puppet with some kind of command in the “onlyif” or “unless” statement - test -f? Current downloaded file md5sum hash matches? etc.) Probably the simplest way to deal with this is to use a SHA256 hash in the remote_file resource - that’s what we’ll do in this next code example:

1
2
3
4
5
6
7
8
9
10
11
12
13

# foobar/recipes/default.rb:
remote_file "/opt/foobar/#{node[:foobar][:tarball]}" do
  source node[:foobar][:url]
  mode "0644"
  checksum "deadbeef"
end
# Replaces the following Puppet code:
# exec { "download tarball":
#   path    => "/bin:/usr/bin:/sbin:/usr/sbin",
#   cwd     => "/opt/foobar",
#   command => "wget ${url}",
#   creates => "/opt/foobar/${tarball}",
# }

There isn’t a way to do a simple existence check for the file before attempting to download it (like what we’ve done in our Puppet code above), but this is a bit more robust for dealing with remote files.

The tarball extraction exec resource in Puppet is easy to port over, and we’ll just go ahead and just do a straight translation over to Chef:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

# foobar/recipes/default.rb:

execute "extract tarball" do
  cwd "/opt/foobar"
  command "tar zxvf #{node[:foobar][:tarball]}"
  creates "/opt/foobar/foobar-#{node[:foobar][:version]}"
end

# We'll go ahead and throw in the template, too:

template "/opt/foobar/foobar-#{node[:foobar][:version]}/config.cfg" do
  source "config.cfg.erb"
end

# Replaces the following Puppet code:
# exec { "extract tarball":
#   path    => "/bin:/usr/bin:/sbin:/usr/sbin",
#   cwd     => "/opt/foobar",
#   command => "tar zxvf ${tarball}",
#   creates => "/opt/foobar/foobar-${version}",
#   before  => File["/opt/foobar/foobar-${version}/config.cfg"],
# }
#
# file { "/opt/foobar/foobar-${version}/config.cfg":
#   content => template("foobar/config.cfg.erb"),
# }

Since Chef executes things in order, we don’t have to specify that the tarball extraction step must come before writing the template file - we simply place it before the template creation step.

Also, Chef doesn’t require you to specify a global “Path” variable in the top level of your class hierarchy - it uses the PATH variable that’s sourced in the shell that’s running Chef. If you need to override this, you can by setting the “path” attribute in the execute resource.

Lastly, dealing with the regular static file at the end of the Puppet module is a bit simpler in Chef than in Puppet - sometimes, we need to have specific files for specific hosts or operating systems, and the way we do this is by throwing the file into different directories, instead of coming up with a naming convention and trying to stick with it throughout Puppet module development. In this case, we have file specificity broken up by hosts, and so we can create the following folders and files in our Chef cookbook:

1
2
3

foobar/files/host-host1.example.com/staticfile
foobar/files/host-host2.example.com/staticfile
foobar/files/default/staticfile

We then write the following in our recipe:

1
2
3
4
5
6
7
8
9
10

# foobar/recipes/default.rb:

cookbook_file "/opt/foobar/foobar-#{node[:foobar][:version]}/staticfile" do
  source "staticfile"
end
# Replaces the following Puppet code:
# file { "/opt/foobar/foobar-${version}/staticfile":
#   source => [ "puppet:///modules/foobar/staticfile.${hostname}",
#               "puppet:///modules/foobar/staticfile" ],
# }

All in all, it’s not too tough to port things over to Chef - there’s nothing like chef2puppet for converting things automatically to Chef, but Chef resources map pretty well to Puppet resources, and so the task isn’t nearly as hard as it could have been. Chef has a bit more structure in place for stashing tunables separate from what actually executes, but it could possibly be a bit more constraining than what others could decide on with structuring their Puppet modules.

If anyone has any questions about this, don’t hesitate to let me know via email or through the comments section below.

We’ve come a long way with monitoring in general - we have groups such as ##monitoringsucks on Freenode who are discussing how to break down monitoring into pieces so that we can come up with better tools to handle each component. We have tools such as logstash that can structure our logs, graylog2 for presenting them in real time, and backends such as elasticsearch for storing them in a way that makes it easy to search for the logs again.

We’re looking at problems differently than how it was seen before. Currently, at Seattle Biomed, we have an installation of logwatch that’s on a centralized rsyslog host that cuts down noise and sends us an email every hour, showing us the log entries that might seem useful to us. Unfortunately, this can sometimes be very useless - it’s easy to add in a filter that gets rid of a bunch of noise, yet those same logs can end up being incredibly useful down the road for catching a recurring bug.

We see this with other related tools as well: RRD’s are usually used to store metrics on events that we collect, yet it has become the de facto standard for storing time series data, while assuming too much about how we want to store our data (sometimes, we don’t want to rotate our data, or we want to rotate our data in some other way. There’s other issues too, such as assuming that all metrics are gathered regularly - what if I wanted to graph irregular Puppet runs?)

The tools we now have on the table look at the problem differently: collect as much data as possible, and make them useful. Split away functionality as much as possible, so we have a bit more of a loosely coupled system.

What we’re implementing right now in Seattle Biomed is the following:

1. logstash, for structuring different log formats into JSON, and shipping them off to other services

2. graylog2, for presenting real time logs and divvying up the view of these logs into different “streams”

3. elasticsearch + logstash web, for long term archival of logs and search

4. rabbitmq, for storing application log events from logstash-forwarders and for forwarding logs between networks

To be honest, I was a bit leery about setting all of this up - I was afraid it’d add a lot more complexity to our network. And perhaps it did, but now that we have something that structures any crappy logs I throw at it, something to visualize all of our logs (not just syslog events), something to store these events for long-term archival, and an AMQP broker that can also be used for other applications, I think this little bit of added complexity has paid off.

To show how all of this hooks up together, I hope this ASCII art drawing tides everyone over:

1
2
3
4
5

syslog events
-----------------> LOGSTASH -----> GRAYLOG2
                   /       \_____> ELASTICSEARCH
app events    AMQP
--------------/

It’d be nice to throw everything into AMQP before having it processed by logstash, but it looked like it required either another logstash agent (to grab all of the syslog events and throw them into AMQP), or agents running on all machines that watches a bunch of files. For now, I’d rather not implement either solution - if anyone else has other ideas, I’m all ears. The way we have it setup seems to work just fine though, but YMMV.

Logstash comes with support for ‘grok’, which is a library that allows you to group regex’s into macros, so you can write a pattern like this:

1

%{TIMESTAMP} %{USER} %{SYSLOGPROG} %{MESSAGE}

instead of writing a huge, ugly regex for everything. You can also assign names to a macro, which is used to structure the logs in logstash:

1

%{TIMESTAMP:timestamp} %{USER:user} %{SYSLOGPROG:program} %{MESSAGE:message}

The subsequent structure for the event will now look like this, assuming we wrote our macros as we did above:

1
2
3
4
5
6
7
8

{
  @message => <whatever was matched in MESSAGE and assigned to @message>
  @fields => {
               "timestamp" => <TIMESTAMP>
               "user" => <USER>
               "program" => <SYSLOGPROG>
             }
}

(the patterns I’ve used above aren’t exactly what’s included in logstash, but hopefully it gets the point across as to how powerful grok + logstash can be.)

Logstash can prevent events from going through the rest of the pipeline, but I’ve decided to simply write grok patterns that structure incoming events, and do any filtering later on in the pipeline (right now, we don’t filter away anything - to me, it seems better to have every message available and be able to drill down to what entries might be relevant, instead of getting rid of a bunch of INFO/DEBUG messages and find out later that I really needed to see that information.)

After structuring the data with logstash, we ship the structured events off to both elasticsearch and graylog2 using logstash’s elasticsearch and GELF output plugins.

graylog2 can match against any arbitrary key that’s in the event, so if we wanted to look at only Puppet runs, we can create a stream called “Puppet Runs” that matches against “program=puppet-agent”. We can setup alarms and stream subscriptions for that specific stream, thereby giving us the same sort of functionality that we expect from logwatch and swatch. Right now, I have three streams for Puppet runs: “Puppet Runs” (aggregate of all Puppet Run logs), “Puppet Runs - Changes” (if anything looks like a Puppet action, we stuff it into this stream), and “Puppet Runs - Error” (anything that has severity level ERROR and is a Puppet Run.)

Elasticsearch stores every log entry. We capture as much as possible, so we can look back at an entire history of logs and correlate events together a bit better. Again, I’d rather worry about disk space than drop log entries that might’ve been a bit noisy at one point but end up being useful down the road. Elasticsearch has a fairly rich language for querying the database, so I’d imagine it’s possible to come up with an expression strong enough to get exactly what you need.

Down the road, my wish is to have logstash pull certain metrics out of events, and throw them over to graphite for visualization (this is already implemented in logstash 1.0.16, but there’s a few showstopping bugs that prevented me from keeping 1.0.16 installed.)

Anyway, log monitoring has advanced quite a bit over the past few years in OSS software - this solution, while more complex to setup than something like Splunk, also costs absolutely nothing in upfront costs, and covers much of the same ground as Splunk. This is also a way better way to view logs than using tools such as swatch or logwatch, where you do your filtering right at the source, rather than gather as much as possible and filter at the tail end of your pipeline (where you’re more likely to have a bit better judgment as to whether those INFO logs were spewing a bunch of crap or not.)

Once we’re hitting the magic spot of roughly 50 - 100 machines or more, we usually need to start looking at ways on how to manage all of these machines a bit more easily.

Configuration management is for keeping servers in spec with what you’ve written down. You deploy packages, configuration, and start services with configuration management. For the most part, we’ve gotten to the point where we can describe enough in our configuration management language so that we don’t have to login directly to our machines to change anything.

But if say we want to grab information, execute an action, or test something out on several machines at once, we likely have to look at solutions that can log into several machines at once and execute an action. We have applications such as ClusterSSH and dsh for controlling several SSH sessions at the same time, general helper applications such as GNU Parallel, and good ol’ SSH in a for loop. However, we’ll also likely have to maintain lists of different classes of machines depending on what OS it is (what if we wanted to run “apt-get install foo” on all of our Linux machines, yet we have a few CentOS machines?), as well as lists of machines belonging to different roles (we likely don’t want to install “apache2” on our DB servers, for instance.)

Further, we might not be interested in stdout output of the commands we run, nor even the stderr output - we just want to know whether the command succeeded or failed, and maybe some useful information to go along with it (aside from attempting to grep out a few things from stdout/stderr.)

mcollective helps with all of these issues. It’s much more than “SSH in a for loop” - it uses a STOMP broker such as ActiveMQ or RabbitMQ (with the STOMP plugin) as a pubsub broker, and allows you to use metadata straight from the server as filters (so you can classify servers in different groups as needed, and since the server itself is reporting its metadata, information is always up to date.) One example of using mcollective is to collect inventory information and compile a report:

1
2
3
4
5

inventory do
    format "%s:\t\t\t%s"

    fields { [ identity, facts["lsbdistdescription"] ] }
end

1

mco inventory --script=inventory.mc

1
2

foobar1.example.com: CentOS release 5.5 (Final)
foobar2.example.com:  Ubuntu 11.04

Another example is to kick off a Puppet run on all machines with the “country=us” fact, with no more than three concurrent runs at a time:

1

mc-puppetd runonce -W country=us 3

Three Drunken SysAds also has a nice article on using capistrano and mcollective for updating code on the Puppet master, and kicking off a subsequent Puppet run on all of the machines in the fleet with mcollective.

All we had to do in order to query all of these nodes intelligently is not by maintaining a list of machines on the client, but by directly querying all of the servers in the fleet, with any server satisfying the filter replying back to the client. The servers themselves are the source of truth, not anything else.

–

There’s a few caveats that I’ve run into that should be taken into consideration:

Security

By default, security is through a preshared key that’s distributed between the clients and all of the servers. This is bad, especially if you plan on having mcollective do anything useful, such as run privileged commands or kicking off Puppet runs. We’ve decided to use the AES + RSA security plugin for encrypting traffic between clients and servers, which is much more secure than using a PSK.

We’ve enabled RPC Auditing as well, just to make sure that nothing’s amiss. It’d be nice to centralize all of the logs in one place: we haven’t found a good way to do this yet (mcollective does have a logstash plugin, though it stuffs all of the audit logs in the STOMP server, which, at this point in time, can’t be pulled from logstash due to a bug in jruby.) For now, it might be worth it to install the centralized log audit plugin, pull an aggregate log file from a single machine, and run logstash against that particular file.

Further, you’ll likely want to implement an authorization plugin. We use the ActionPolicy plugin for authorizing specific users to be able to do certain actions. For ActionPolicy to work, you have to make sure your client’s computer has UID’s synced with the servers you are querying, since the RPC client grabs the UID from your client machine, and ships it over as “request.caller” to the server (UPDATE: this is not true for the SSL and AES + RSA security plugins, only for the PSK security plugin.) Since mcollective basically gives you root level access to each system it runs on (even if it only exposes a few hooks to the client), you’ll want to make sure that you only give just enough permissions to mcollective that a person (or system user) might need.

Keep in mind that if you’re using AES + RSA security or SSL security with ActionPolicy, you’ll want to use “cert=” instead of “uid=” - the AES + RSA security plugin sends the cert name in the “caller” section of the RPC request, which you can confirm if you enable RPC auditing and drill into the logs.

STOMP

This was a little annoying, since we already had a rabbitmq server up and running for logstash and I didn’t want to spin up another machine just for mcollective. Thankfully, rabbitmq has a STOMP connector - we installed it on our RabbitMQ server, and have it churning through both mcollective and logstash requests just fine.

You can also code a different connector plugin for mcollective agents - I haven’t yet seen code out in the wild for connecting with an AMQP broker (or anything else for that matter), but the option is out there, if you really don’t want to run a STOMP broker.

mcollective from tarball

There’s no mcollective gem to be found anywhere, so in order to get a consistent installation across all of our machines, we resorted to using the tarball. A couple of things to keep in mind if you choose to go this route:

• Be sure to set RUBYLIB to point to the lib directory in the tarball.
• Install the STOMP gem.
• If you’re using the applications directly (as an mcollective client for example), add the unpacked mcollective directory to your PATH (or copy over the Ruby binaries over to a bin folder of your choice.)

Configuration files

It’s not immediately obvious that you can split your configuration files into multiple files, but only for plugin configuration. If your server.cfg file has entries such as “plugin.foo.bar = blah” and “plugin.bar.baz = 20”, you can write the following cfg files in /etc/mcollective/plugin.d instead of stuffing those previous entries in server.cfg:

foo.cfg:

bar = blah

bar.cfg:

baz = 20

Again, it’s not totally obvious from the documentation, but at least there’s a little bit of flexibility there for your configuration management system of choice.

–

I’m quite happy with mcollective - it’s easily extensible and it gives me much better control over our UNIX machines. We have it running in production and it makes it a hell of a lot easier to inventory and control several machines at once.