redbluemagenta

RunDeck

2012-01-08T21:50:00-08:00

There’s times when we need to enforce adhoc control over our machines. This blog post by Alex Honor does a way better job than I could to explain why we still need to worry about adhoc command execution, but a quick summary would probably be the following:

Sometimes, we just need to execute commands ASAP - unplanned downtime and adhoc information gathering both spur this activity.
We need to orchestrate any set of commands across many machines - things like getting your DB servers up and prepped before your web servers come online is one example of this.

RunDeck can handle these two scenarios perfectly. It’s a central job server that can execute any given set of commands or scripts across many machines. You can define machines via XML, or use web servers that respond with a set of XML machine definitions (see chef-rundeck for a good example of this.) You can code in whatever language you want for your job scripts - RunDeck will run them on the local machine if the execute bit is set (or, if you’re running jobs across many machines, it will SCP them onto the machine and execute them via SSH.)

You can define multiple job steps within a job (including using other jobs as a job step) - thus, you can, for example, have a job step that pulls down your web application code from git across all web application servers, another job step that pulls down dependencies, then one more to start the web application across all of those machines, all of which is coordinated by the RunDeck server. As another example, you can also coordinate scheduled jobs that provision EC2 servers at a certain time of the day, run some kind of processing job across all of those machines, then shut all of them down as soon as the job is finished.

There’s another way these jobs can be started, not only just by manual button pushing and cron schedules: they can also be started by hitting a URI via the provided API with an auth token. To expand on the previous example, if say you don’t have a reliable way to check to see if the processing is finished from RunDeck’s point of view, you can probably have some kind of job that’s fired from the workers, which hits the API and tells RunDeck to run a ‘reap workers’ job.

mcollective vs. RunDeck?

A few people have asked me how this compares with mcollective.

I would probably say that they’re actually complimentary with each other - mcollective is incredibly nice, in that it has a bit of a better paradigm for executing commands across a fleet of hosts (pub/sub vs. SSH in a for loop.) You can actually use mcollective as an execution provider in RunDeck, by simply specifying ‘mco’ as the executor in the config file (here’s a great example of this.) RunDeck is nice, in that you can use arbitrary scripts with RunDeck and it’ll happily use them for execution - plus, it has a GUI, which makes it nice if you need to provide a ‘one button actiony thing’ for anyone else to use. RunDeck can also run things ‘out of band’ (relative to mcollective) - for example, provisioning EC2 machines is an ‘out of band’ activity (though you can certainly implement this with mcollective as well, mcollective’s execution distribution model doesn’t seem to ‘fit’ well with actions that are meant to be run on a central machine.)

But again, you can tie mcollective with RunDeck, and they’ll both be happy together. I can see right away that the default SSH executor will likely be a pain in the ass once you get to about 100+ machines or so.

Conclusion

RunDeck is pretty damn nice. We’ve been putting it through its paces in our staging environment, and it’s held up very nicely with a wide variety of tasks that we’ve thrown at it. The only worries I have are the following:

Default SSH command executor will likely start sucking after getting to about 100+ machines.
Since it can execute whatever you throw at it, it’s possible that you might end up with yet another garbled mess of shell and Perl scripts. This is probably better solved with team discipline, however.
There doesn’t seem to be a clear way to automate configuration changes for RunDeck - thus, be sure to keep good backups for now (though, this is probably because of my own ignorance - the API is pretty full featured, and I believe you can add new jobs via the API, but it would’ve been awesome to be able to just edit job definitions in a predictable location on the disk.)

Despite these worries, I highly recommend RunDeck - it’s helped us quite a bit so far, it’s quick to get setup, and quick to start coding against for fleet wide adhoc control.

Transitioned to Octopress

2011-10-03T16:20:00-07:00

So, I’ve transitioned from Jekyll to Octopress. It comes with a really nice default HTML5 template that I’ve edited a bit (changed some fonts, colors, etc.), and has a decent fallback for mobile devices, all of which was lacking in my last site design with Jekyll (if anyone’s interested, the old site is still on my GitHub account.)

What’s awesome about Octopress, anyway?

It comes with a nice set of Rake tasks for handling common tasks, such as generating a new site and rsync’ing the pages over to a server. Also comes with Rake tasks for creating new posts and pages.
The repository is very well laid out, with nice separation between themes and content.
The default theme comes with scss files, which is way better to deal with than regular old css.
Comes with a sane set of plugins that make things look awesome, such as code snippets, image embedding, and HTML5 video embedding (with Flash video fallback.)

Anyway, the site looks way better now that it’s on Octopress, and it’s a bit easier to maintain as well.

Converting Puppet Modules to Chef Cookbooks

2011-09-20T00:10:00-07:00

Over a couple of caffeine induced nights, I’ve converted a handful of our Puppet modules over to Chef cookbooks (by hand) to see how it’d all turn out. We’ve not yet decided whether we will actually use Chef in production, but I figure that if I’ve lowered the bar of entry, it would make the decision much easier (we’d still have to worry about whether the technical merits justify putting in time to transition everything over to Puppet, but at least the initial cost of converting things over isn’t as painful as it would’ve been.)

Here’s a sample snippet of Puppet code that I’ll be referring to for the rest of the article:

class foobar {
  $version = "1.0.0"
  $tarball = "foobar-${version}.tar.gz"
  $url     = "http://example.com/${tarball}"

  file { "/opt/foobar":
    ensure => directory,
    mode   => 0755,
  }

  exec { "download tarball":
    path    => "/bin:/usr/bin:/sbin:/usr/sbin",
    cwd     => "/opt/foobar",
    command => "wget ${url}",
    creates => "/opt/foobar/${tarball}",
  }

  exec { "extract tarball":
    path    => "/bin:/usr/bin:/sbin:/usr/sbin",
    cwd     => "/opt/foobar",
    command => "tar zxvf ${tarball}",
    creates => "/opt/foobar/foobar-${version}",
    before  => File["/opt/foobar/foobar-${version}/config.cfg"],
  }

  file { "/opt/foobar/foobar-${version}/config.cfg":
    content => template("foobar/config.cfg.erb"),
  }

  file { "/opt/foobar/foobar-${version}/staticfile":
    source => [ "puppet:///modules/foobar/staticfile.${hostname}",
                "puppet:///modules/foobar/staticfile" ],
  }
}

Puppet resources port pretty well over to Chef resources - you simply have to make sure that you’re using the right Chef resource when rewriting it (for example, you use “file” in Puppet for templates, regular files transferred from the server, regular files managed only locally, and directories - these are all separate things in Chef, with resources such as “cookbook_file”, “template”, “file”, “directory”, and “remote_directory.”) As for variables, this might be a bit of a judgment call - I’ve mostly thrown tunables into node attributes, and leave temporary recipe-specific variables where they were in Puppet. In this case, I’d likely throw $version into “node[:foobar][:version]” as an attribute - the tarball name, if the naming convention doesn’t change between versions, could probably stay inside the recipe itself. The URL might be a tunable too, and so could probably be stuffed into “node[:foobar][:url]” - what if you had mirrors of these files in each geographical area?

Here’s what our code will look like so far, given the above:

# foobar/attributes/default.rb:
default[:foobar][:version] = "1.0.0"
default[:foobar][:tarball] = "foobar-#{node[:foobar][:version]}.tar.gz"
default[:foobar][:url]     = "http://example.com/#{node[:foobar][:tarball]}"
# Replaces the following Puppet code:
# $version = "1.0.0"
# $tarball = "foobar-${version}.tar.gz"
# $url     = "http://example.com/${tarball}"

# foobar/recipes/default.rb:

directory "/opt/foobar" do
  mode "0755"
end
# Replaces the following Puppet code:
# file { "/opt/foobar":
#   ensure => directory,
#   mode   => 0755,
# }

Now, what do we do with the exec resources? There’s still a few things that make sense to port directly to Chef’s “execute” resources - things like updating database users, running a program to alter configuration (think “make” for creating sendmail database files), or extracting tarballs. However, we do have one exec resource that can be replaced by a regular Chef resource - the exec resource that downloads a tarball onto local disk. This is supplanted by the “remote_file” resource - keep in mind that we likely do not want to keep downloading the file on each Chef run (we guarantee this in Puppet with some kind of command in the “onlyif” or “unless” statement - test -f? Current downloaded file md5sum hash matches? etc.) Probably the simplest way to deal with this is to use a SHA256 hash in the remote_file resource - that’s what we’ll do in this next code example:

# foobar/recipes/default.rb:
remote_file "/opt/foobar/#{node[:foobar][:tarball]}" do
  source node[:foobar][:url]
  mode "0644"
  checksum "deadbeef"
end
# Replaces the following Puppet code:
# exec { "download tarball":
#   path    => "/bin:/usr/bin:/sbin:/usr/sbin",
#   cwd     => "/opt/foobar",
#   command => "wget ${url}",
#   creates => "/opt/foobar/${tarball}",
# }

There isn’t a way to do a simple existence check for the file before attempting to download it (like what we’ve done in our Puppet code above), but this is a bit more robust for dealing with remote files.

The tarball extraction exec resource in Puppet is easy to port over, and we’ll just go ahead and just do a straight translation over to Chef:

# foobar/recipes/default.rb:

execute "extract tarball" do
  cwd "/opt/foobar"
  command "tar zxvf #{node[:foobar][:tarball]}"
  creates "/opt/foobar/foobar-#{node[:foobar][:version]}"
end

# We'll go ahead and throw in the template, too:

template "/opt/foobar/foobar-#{node[:foobar][:version]}/config.cfg" do
  source "config.cfg.erb"
end

# Replaces the following Puppet code:
# exec { "extract tarball":
#   path    => "/bin:/usr/bin:/sbin:/usr/sbin",
#   cwd     => "/opt/foobar",
#   command => "tar zxvf ${tarball}",
#   creates => "/opt/foobar/foobar-${version}",
#   before  => File["/opt/foobar/foobar-${version}/config.cfg"],
# }
#
# file { "/opt/foobar/foobar-${version}/config.cfg":
#   content => template("foobar/config.cfg.erb"),
# }

Since Chef executes things in order, we don’t have to specify that the tarball extraction step must come before writing the template file - we simply place it before the template creation step.

Also, Chef doesn’t require you to specify a global “Path” variable in the top level of your class hierarchy - it uses the PATH variable that’s sourced in the shell that’s running Chef. If you need to override this, you can by setting the “path” attribute in the execute resource.

Lastly, dealing with the regular static file at the end of the Puppet module is a bit simpler in Chef than in Puppet - sometimes, we need to have specific files for specific hosts or operating systems, and the way we do this is by throwing the file into different directories, instead of coming up with a naming convention and trying to stick with it throughout Puppet module development. In this case, we have file specificity broken up by hosts, and so we can create the following folders and files in our Chef cookbook:

foobar/files/host-host1.example.com/staticfile
foobar/files/host-host2.example.com/staticfile
foobar/files/default/staticfile

We then write the following in our recipe:

# foobar/recipes/default.rb:

cookbook_file "/opt/foobar/foobar-#{node[:foobar][:version]}/staticfile" do
  source "staticfile"
end
# Replaces the following Puppet code:
# file { "/opt/foobar/foobar-${version}/staticfile":
#   source => [ "puppet:///modules/foobar/staticfile.${hostname}",
#               "puppet:///modules/foobar/staticfile" ],
# }

All in all, it’s not too tough to port things over to Chef - there’s nothing like chef2puppet for converting things automatically to Chef, but Chef resources map pretty well to Puppet resources, and so the task isn’t nearly as hard as it could have been. Chef has a bit more structure in place for stashing tunables separate from what actually executes, but it could possibly be a bit more constraining than what others could decide on with structuring their Puppet modules.

If anyone has any questions about this, don’t hesitate to let me know via email or through the comments section below.

Modern Log Management and Monitoring

2011-08-19T05:29:00-07:00

We’ve come a long way with monitoring in general - we have groups such as ##monitoringsucks on Freenode who are discussing how to break down monitoring into pieces so that we can come up with better tools to handle each component. We have tools such as logstash that can structure our logs, graylog2 for presenting them in real time, and backends such as elasticsearch for storing them in a way that makes it easy to search for the logs again.

We’re looking at problems differently than how it was seen before. Currently, at Seattle Biomed, we have an installation of logwatch that’s on a centralized rsyslog host that cuts down noise and sends us an email every hour, showing us the log entries that might seem useful to us. Unfortunately, this can sometimes be very useless - it’s easy to add in a filter that gets rid of a bunch of noise, yet those same logs can end up being incredibly useful down the road for catching a recurring bug.

We see this with other related tools as well: RRD’s are usually used to store metrics on events that we collect, yet it has become the de facto standard for storing time series data, while assuming too much about how we want to store our data (sometimes, we don’t want to rotate our data, or we want to rotate our data in some other way. There’s other issues too, such as assuming that all metrics are gathered regularly - what if I wanted to graph irregular Puppet runs?)

The tools we now have on the table look at the problem differently: collect as much data as possible, and make them useful. Split away functionality as much as possible, so we have a bit more of a loosely coupled system.

What we’re implementing right now in Seattle Biomed is the following:

logstash, for structuring different log formats into JSON, and shipping them off to other services
graylog2, for presenting real time logs and divvying up the view of these logs into different “streams”
elasticsearch + logstash web, for long term archival of logs and search
rabbitmq, for storing application log events from logstash-forwarders and for forwarding logs between networks

To be honest, I was a bit leery about setting all of this up - I was afraid it’d add a lot more complexity to our network. And perhaps it did, but now that we have something that structures any crappy logs I throw at it, something to visualize all of our logs (not just syslog events), something to store these events for long-term archival, and an AMQP broker that can also be used for other applications, I think this little bit of added complexity has paid off.

To show how all of this hooks up together, I hope this ASCII art drawing tides everyone over:

syslog events
—————–> LOGSTASH —–> GRAYLOG2
                   /       \_____> ELASTICSEARCH
app events    AMQP
————–/

It’d be nice to throw everything into AMQP before having it processed by logstash, but it looked like it required either another logstash agent (to grab all of the syslog events and throw them into AMQP), or agents running on all machines that watches a bunch of files. For now, I’d rather not implement either solution - if anyone else has other ideas, I’m all ears. The way we have it setup seems to work just fine though, but YMMV.

Logstash comes with support for ‘grok’, which is a library that allows you to group regex’s into macros, so you can write a pattern like this:

%{TIMESTAMP} %{USER} %{SYSLOGPROG} %{MESSAGE}

instead of writing a huge, ugly regex for everything. You can also assign names to a macro, which is used to structure the logs in logstash:

%{TIMESTAMP:timestamp} %{USER:user} %{SYSLOGPROG:program} %{MESSAGE:message}

The subsequent structure for the event will now look like this, assuming we wrote our macros as we did above:

{
  @message => <whatever was matched in MESSAGE and assigned to @message>
  @fields => {
               "timestamp" => <TIMESTAMP>
               "user" => <USER>
               "program" => <SYSLOGPROG>
             }
}

(the patterns I’ve used above aren’t exactly what’s included in logstash, but hopefully it gets the point across as to how powerful grok + logstash can be.)

Logstash can prevent events from going through the rest of the pipeline, but I’ve decided to simply write grok patterns that structure incoming events, and do any filtering later on in the pipeline (right now, we don’t filter away anything - to me, it seems better to have every message available and be able to drill down to what entries might be relevant, instead of getting rid of a bunch of INFO/DEBUG messages and find out later that I really needed to see that information.)

After structuring the data with logstash, we ship the structured events off to both elasticsearch and graylog2 using logstash’s elasticsearch and GELF output plugins.

graylog2 can match against any arbitrary key that’s in the event, so if we wanted to look at only Puppet runs, we can create a stream called “Puppet Runs” that matches against “program=puppet-agent”. We can setup alarms and stream subscriptions for that specific stream, thereby giving us the same sort of functionality that we expect from logwatch and swatch. Right now, I have three streams for Puppet runs: “Puppet Runs” (aggregate of all Puppet Run logs), “Puppet Runs - Changes” (if anything looks like a Puppet action, we stuff it into this stream), and “Puppet Runs - Error” (anything that has severity level ERROR and is a Puppet Run.)

Elasticsearch stores every log entry. We capture as much as possible, so we can look back at an entire history of logs and correlate events together a bit better. Again, I’d rather worry about disk space than drop log entries that might’ve been a bit noisy at one point but end up being useful down the road. Elasticsearch has a fairly rich language for querying the database, so I’d imagine it’s possible to come up with an expression strong enough to get exactly what you need.

Down the road, my wish is to have logstash pull certain metrics out of events, and throw them over to graphite for visualization (this is already implemented in logstash 1.0.16, but there’s a few showstopping bugs that prevented me from keeping 1.0.16 installed.)

Anyway, log monitoring has advanced quite a bit over the past few years in OSS software - this solution, while more complex to setup than something like Splunk, also costs absolutely nothing in upfront costs, and covers much of the same ground as Splunk. This is also a way better way to view logs than using tools such as swatch or logwatch, where you do your filtering right at the source, rather than gather as much as possible and filter at the tail end of your pipeline (where you’re more likely to have a bit better judgment as to whether those INFO logs were spewing a bunch of crap or not.)

mcollective

2011-08-19T04:41:00-07:00

Once we’re hitting the magic spot of roughly 50 - 100 machines or more, we usually need to start looking at ways on how to manage all of these machines a bit more easily.

Configuration management is for keeping servers in spec with what you’ve written down. You deploy packages, configuration, and start services with configuration management. For the most part, we’ve gotten to the point where we can describe enough in our configuration management language so that we don’t have to login directly to our machines to change anything.

But if say we want to grab information, execute an action, or test something out on several machines at once, we likely have to look at solutions that can log into several machines at once and execute an action. We have applications such as ClusterSSH and dsh for controlling several SSH sessions at the same time, general helper applications such as GNU Parallel, and good ol’ SSH in a for loop. However, we’ll also likely have to maintain lists of different classes of machines depending on what OS it is (what if we wanted to run “apt-get install foo” on all of our Linux machines, yet we have a few CentOS machines?), as well as lists of machines belonging to different roles (we likely don’t want to install “apache2” on our DB servers, for instance.)

Further, we might not be interested in stdout output of the commands we run, nor even the stderr output - we just want to know whether the command succeeded or failed, and maybe some useful information to go along with it (aside from attempting to grep out a few things from stdout/stderr.)

mcollective helps with all of these issues. It’s much more than “SSH in a for loop” - it uses a STOMP broker such as ActiveMQ or RabbitMQ (with the STOMP plugin) as a pubsub broker, and allows you to use metadata straight from the server as filters (so you can classify servers in different groups as needed, and since the server itself is reporting its metadata, information is always up to date.) One example of using mcollective is to collect inventory information and compile a report:

inventory.mc

inventory do
    format "%s:\t\t\t%s"

    fields { [ identity, facts["lsbdistdescription"] ] }
end

mco command

mco inventory –script=inventory.mc

Sample Output

foobar1.example.com: CentOS release 5.5 (Final)
foobar2.example.com:  Ubuntu 11.04

Another example is to kick off a Puppet run on all machines with the “country=us” fact, with no more than three concurrent runs at a time:

mc-puppetd runonce -W country=us 3

Three Drunken SysAds also has a nice article on using capistrano and mcollective for updating code on the Puppet master, and kicking off a subsequent Puppet run on all of the machines in the fleet with mcollective.

All we had to do in order to query all of these nodes intelligently is not by maintaining a list of machines on the client, but by directly querying all of the servers in the fleet, with any server satisfying the filter replying back to the client. The servers themselves are the source of truth, not anything else.

–

There’s a few caveats that I’ve run into that should be taken into consideration:

Security

By default, security is through a preshared key that’s distributed between the clients and all of the servers. This is bad, especially if you plan on having mcollective do anything useful, such as run privileged commands or kicking off Puppet runs. We’ve decided to use the AES + RSA security plugin for encrypting traffic between clients and servers, which is much more secure than using a PSK.

We’ve enabled RPC Auditing as well, just to make sure that nothing’s amiss. It’d be nice to centralize all of the logs in one place: we haven’t found a good way to do this yet (mcollective does have a logstash plugin, though it stuffs all of the audit logs in the STOMP server, which, at this point in time, can’t be pulled from logstash due to a bug in jruby.) For now, it might be worth it to install the centralized log audit plugin, pull an aggregate log file from a single machine, and run logstash against that particular file.

Further, you’ll likely want to implement an authorization plugin. We use the ActionPolicy plugin for authorizing specific users to be able to do certain actions. For ActionPolicy to work, you have to make sure your client’s computer has UID’s synced with the servers you are querying, since the RPC client grabs the UID from your client machine, and ships it over as “request.caller” to the server (UPDATE: this is not true for the SSL and AES + RSA security plugins, only for the PSK security plugin.) Since mcollective basically gives you root level access to each system it runs on (even if it only exposes a few hooks to the client), you’ll want to make sure that you only give just enough permissions to mcollective that a person (or system user) might need.

Keep in mind that if you’re using AES + RSA security or SSL security with ActionPolicy, you’ll want to use “cert=” instead of “uid=” - the AES + RSA security plugin sends the cert name in the “caller” section of the RPC request, which you can confirm if you enable RPC auditing and drill into the logs.

STOMP

This was a little annoying, since we already had a rabbitmq server up and running for logstash and I didn’t want to spin up another machine just for mcollective. Thankfully, rabbitmq has a STOMP connector - we installed it on our RabbitMQ server, and have it churning through both mcollective and logstash requests just fine.

You can also code a different connector plugin for mcollective agents - I haven’t yet seen code out in the wild for connecting with an AMQP broker (or anything else for that matter), but the option is out there, if you really don’t want to run a STOMP broker.

mcollective from tarball

There’s no mcollective gem to be found anywhere, so in order to get a consistent installation across all of our machines, we resorted to using the tarball. A couple of things to keep in mind if you choose to go this route:

Be sure to set RUBYLIB to point to the lib directory in the tarball.
Install the STOMP gem.
If you’re using the applications directly (as an mcollective client for example), add the unpacked mcollective directory to your PATH (or copy over the Ruby binaries over to a bin folder of your choice.)

Configuration files

It’s not immediately obvious that you can split your configuration files into multiple files, but only for plugin configuration. If your server.cfg file has entries such as “plugin.foo.bar = blah” and “plugin.bar.baz = 20”, you can write the following cfg files in /etc/mcollective/plugin.d instead of stuffing those previous entries in server.cfg:

foo.cfg:

bar = blah

bar.cfg:

baz = 20

Again, it’s not totally obvious from the documentation, but at least there’s a little bit of flexibility there for your configuration management system of choice.

–

I’m quite happy with mcollective - it’s easily extensible and it gives me much better control over our UNIX machines. We have it running in production and it makes it a hell of a lot easier to inventory and control several machines at once.

Why XDA Developers Forums Suck

2011-06-12T14:36:00-07:00

I’ve been a member of the XDA Developers Forum forums for a few months, and after following development forums for various phones (mostly the Nexus S and the HTC Thunderbolt forums), I’ve concluded that the XDA Developers forums suck. Perhaps an even stronger claim is that the Android developer community sucks, but I won’t be defending that claim in this post.

So, why does the XDA Developers forums suck?

For one, the purpose of the forums is to centralize discussion and cooperation of development projects for Android. Based on what I’ve seen so far on the Nexus S and Thunderbolt forums, I believe that the forums have largely failed at this task. ROM’s are largely developed independently with about over half of the ROMs maybe sharing source code to the community, with the biggest offender being CM7 for Thunderbolt (the source code for most of the OS is shared to the public, but the most important part that adds support for the Thunderbolt’s radio will not be open sourced “until it’s done.”) Worse, there’s a lot of prima donnas in the community - in most other dev communities, most of the work is done in teams, though there may be benevolent dictators or celebrities (but almost none of the things that are present in the XDA community - witholding of source code and “heroism.”) As much as slayher has contributed to the community, it’s telling when you see CM7 on Thunderbolt completely contingent on him finishing his radio interface layer code, and having to go to a channel called #slayher for CM7 Thunderbolt support. Who the fuck creates a channel based on their handle for a software project?

Not only that, but the discussion that does take place on the forums around development is almost always centered on end user support. Many other communities solve this with mailing lists in order to help focus branching topics in a thread (most forums are notoriously bad at this, given that the default view in most forums and the way that a forum focuses your conversation often defaults to a flat hierarchy of posts.) Any relevant developer discussion is drowned in a sea of user support questions, and I would not blame anyone who wishes to take their conversation elsewhere.

What I propose as an alternative is the following:

A site that is an aggregate of mailing lists for various phones, software projects associated with each phone, etc.
User support may be provided on this same site on forums, like how XDA is setup right now. The only difference is that dev discussion is separated into mailing lists (and make it crystal clear that any developer related discussion should be posted on the mailing lists.)
This site should not post anything that doesn’t have any source code freely available under the GPL/BSD/Apache/etc. licenses.

Puppet vs. Chef

2011-05-21T12:25:00-07:00

I’m approaching this from the view of a systems administrator - it’s possible that my world view is quite different from a developer, so my preferences might seem a bit weird to a couple of people. I’m not going to outright say which one is “better” than the other - in fact, both CM systems are quite awesome, it’s just that they both have different philosophies on how to approach configuration management.

Puppet, for example, explicitly exposes the dependency graphs - if you want to impose order, you are required to declare how you’re imposing it. Is it because of a dependency? Is it because you’ve grouped resources in different stages and you want your resource to execute in one of those stages? The DSL forces the sysadmin to think about exactly how their system is built - ideally, you’d setup dependencies between various disparate resources, and each group of resources should be able to execute independently of each other (for example, I could setup a group of dependencies that describe a web server running Rails, and another group of dependencies describing NFS services - unless the web server depends on NFS, I should be able to setup the web service and the NFS service independently from each other. If one set of resources fails to execute, the other set should still be able to execute regardless of the state of the first set of resources.)

Puppet has a Ruby DSL - however, there’s a lot of institutional momentum to continue writing in the old Puppet DSL. Writing manifests in the old Puppet DSL enables us to write things more clearly, but at the cost of flexibility - for example, the way I’ve been applying an action on each element of an array is by using “definitions”:

define foo {
  # do something to ${name}
}

...

$bar = [ 'one', 'two', 'three' ]

foo { $bar: }

In Chef, I’d instead write it like this:

%w{one two three}.each do |foo|
  # do something to #{foo}
end

For slightly harder things that rely on data structures such as arrays and hashes, Puppet makes it harder to clearly express what I want. But there’s a danger with Chef: you can write arbitrary Ruby code, which can sometimes obfuscate exactly what you want to do on the system, or at the very least, can make it confusing to read compared to equivalent Puppet code.

Chef is much more conducive for programmers. You can declare dependencies like in Puppet, but by default, the resources execute in order as they’re listed in the Chef recipe. You can write arbitrary Ruby code alongside Chef DSL code - however, aside from the dangers outlined above, you’ll also have to be very careful about the execution order of Ruby code and Chef DSL code. Ruby code always executes during the compilation stage (unless you wrap the Ruby code within a “ruby_block” resource declaration.) The compilation stage executes before the execution stage - this can lead to things like this:

# foobar.conf, before cookbook_file resource:
hello

# foobar.conf, as defined in cookbook:
hi

# Recipe:
cookbook_file "/etc/foobar.conf" do
  source "foobar.conf"
  mode 0644
end

grab_state = %x[cat /etc/foobar.conf]

grab_state will only be “hello” on the first execution of this recipe - otherwise, it will have the value “hi” in every other execution. This might be confusing, even if grab_state is located below “cookbook_file” in the recipe.

This will do the right thing, if you expect “grab_state” to be given a value after “cookbook_file” does its thing:

cookbook_file ...

ruby_block "grab state" do
  block do
    grab_state = %x[cat /etc/foobar.conf]
  end
end

This is because we’ve now placed our arbitrary Ruby code within a Chef DSL resource, which is executed after the compilation phase.

In short, the largest differences between Puppet and Chef is not only the language, but also the way they approach configuration management. Chef breaks the mold by enforcing execution order - most other CM systems execute things in random order, which is just simply a different way of looking at things (the description is more important than how we get there.) Puppet, in my opinion, is neither better nor worse than Chef, but simply different - there are pains in programming in both. Other issues, such as administering the server, are also just “different” - Puppet has quite a bit fewer dependencies, but also doesn’t come with as many things that are taken for granted in Chef (such as an external node classifier, and other things that make Chef a bit better in larger deployments, such as a separate data store, queues, etc.) Chef Server is a pain in the ass to setup, and is totally overkill for small deployments, but once you do get it setup, you do get a lot more language features out of the box than you do in Puppet.

VERDICT: DIFFERENT

Presentation from CasITConf

2011-03-12T12:17:00-08:00

For those interested, I’ve posted my presentation on my website. I’ve written it in PowerPoint 2011, so you might have to convert it down if you can’t open pptx files.

I talk about Chef in the presentation - was hoping there was an audio or video recording, but alas, all you can have is my slide deck. :)

EDIT: It’s also now on scribd.

More thoughts on DevOps

2011-03-03T11:04:00-08:00

After working at a few places, watching interviews, and thinking about system administration culture, I’ve ended up slightly changing my views about the DevOps movement.

Automation, repeatability, consistent root cause analysis: these are all things that any decent sysadmin strives for. We are all problem solvers, and we all either want to have good documentation, automation, or both! This new culture that’s developing is a great thing - it’s of my opinion that automation is key for system administration, and that requires a bit of coding, as well as knowing systems in and out.

I was talking to a friend of mine who works at Amazon, and he mentioned that everyone in the company is, indeed, a systems engineer in some capacity. This is what DevOps strives for - software engineers who have systems clue, or at the very least, two groups who collaborate very closely with each other.

HOWEVER: Amazon, Google, etc. have had to adopt this strategy because they were forced to do so at the scale that they were operating at. Benjamin Black had a great interview on NoSQL Tapes which covers roughly the same ground, though mostly on the technology rather than the culture (to summarize what he said, you want to look at your application and infrastructure, figure out what the problem might be, and then fix it; similarly, before adopting new technology, see whether the semantics of the new technology actually fits in with your infrastructure and application.)

Similarly, DevOps tackles an issue that ought to be tackled, yet it’s operating at the “wrong scale”: we do need to automate stuff, but some sites might be fine with just writing shell scripts, some sites might be fine with a few Puppet/Chef recipes but might not to manage their entire site that way. Some companies end up assigning SA’s to program stuff in applications, or vice versa - sometimes the result is disastrous! (Why? I know I can’t write clean code worth crap [sorry folks who have to maintain my code], and I know of a few instances of devs attempting to manage systems with Chef and neglecting certain operational issues, such as performance tuning, packaging, or whatever else.)

If we want to execute DevOps well, we should look at Amazon and Google for guidance, but we should also look at how they’re organized, what sort of scale they’re operating at, and see if it actually makes sense to assign roles in the same way.

I’d love to hear your guys’ thoughts on this, too. Comment below. :)

Why I switched to Android

2011-01-15T00:00:00-08:00

I’ve had an iPhone 3GS for roughly six months; I bought it second hand from my brother after owning a Nokia N900 for a while. I was pretty disenchanted with the Maemo platform, especially with Nokia’s run-ins with the Osborne Effect (for those who don’t know, roughly one or two months after the N900 was released, Nokia announced that they were developing Maemo 6, which was then merged with the Moblin project in order to give birth to the MeeGo project. They then announced that MeeGo was not to be supported officially on the N900.)

I love the iOS platform, don’t get me wrong; the UI is quite polished and every behavior is consistent between every app, thanks to Apple’s libraries and policies regarding UI design. However, I was tired of not having a level of customization in an un-jailbroken phone as one would have on the Android platform, plus I was not happy with how notifications were handled, nor was I happy about the restrictions on iOS’s multitasking abilities. I had to cut costs somehow on my monthly bills as well, and after evaluating AT&T’s exuberant costs for a 2 GB data plan along with unlimited texting and a decent amount of minutes, I decided that I need to hop on another provider who had much lower costs for data and texting (I don’t care much for talking on the phone, so I don’t really care how minutes I get in a mobile plan.)

I’ve heard about the Nexus S being released in December, so I decided to dive right in and buy the phone from Best Buy, along with switching over to T-Mobile. T-Mobile, thankfully, offers a contract-free “Even More Plus” plan that costs only $60 per month for unlimited data (likely capped at 5 GB, which is then throttled for any usage beyond 5 GB), unlimited texting, and 500 minutes of voice.

After buying the Nexus S, the very first thing I’ve noticed was how graceful the notifications were handled on Android. Every notification is stuffed in the notifications drawer on the top, which you can read at your own leisure. The second thing I’ve noticed was that the UI wasn’t really that bad at all - sure, there’s a bit of inconsistency between each application, but you can be assured that the four buttons on the bottom of the phone will each consistently do what they suggest they do in every app. In short, there’s a lot of FUD about how terrible the UX/UI is in Android, and really, it’s still all very easy to use.

The level of customization on the Android platform is amazing. I’ve switched out the software keyboard with SwiftKey, which I think is an awesome software keyboard for the Android. Some people like Swype. Others like the plain old Android keyboard that comes stock on many phones. The fact of the matter is, you’re not stuck with one particular input method - you have the choice of switching out the keyboard, the home screen style, etc. You have no such choice on iOS - you have to stick with everything Apple provides by default.

Android’s integration with the whole suite of Google web applications is awesome, too. Being able to use the default phonebook and be able to call out to any number with Google Voice by default is extremely helpful. On iOS, I would’ve had to use yet another application which doesn’t work so well as a default dialer. Being able to check in with Google Latitude, integrating your list of contacts with GMail, etc. is handled very well on Android.

I haven’t noticed any issues with the Android App Market at all. In fact, as a consumer, it’s pretty awesome being able to get games/applications at a much lower price than on Apple’s App Store. It integrates with Google Checkout, which isn’t any harder to use than using your Apple ID.

My only complaints about Android have to do with device fragmentation and carrier restrictions, though I’ve mostly avoided the whole question by getting a Nexus S. When I bought my Nexus S, there was buzz about dual core processor phones coming out for the Android platform, and sure enough, at CES, there were a whole slew of them announced. Am I jealous? Sure I am, even though my Nexus S is plenty powerful. With the Android platform, there’s always new devices being announced that are way better than the previous set of phones, which hinders device satisfaction quite a bit. iPhone users don’t have this issue, as there’s only one line of phones that are released fairly predictably every year.

Even with the above in mind, I’m satisfied with the Nexus S and Android platform, even more so than the iPhone I’ve had for a little while. It’s shaping up to be a very strong competitor to Apple’s iOS platform.

Chef as a system integration tool, with a few examples.

2010-12-02T08:40:00-08:00

I haven’t been using configuration management tools for too long, but for my particular use case (if I was directly managing a fleet of machines), I am usually fine with just having an easier to use interface for managing my machines.

However, I was tasked with building a Chef deployment that was to be a “lights out”, fully automated deploy that would not only setup the machines automatically with their particular roles, but to also have said machines gather system state from other machines and compute exactly what to do based on other system states. This required a lot of introspection: what sort of things do I do when configuring machines to talk to each other? Most of this is all second nature to me, and I tend to err on the side of caution when configuring multiple services to talk to each other. For this particular project of which I have almost no experience in doing, I figure that if I can at least write down my steps, I can automate those steps in code.

Setting up MySQL machines is a good example of the differences between the two methods of managing machines with a tool like Chef. You can setup my.cnf on each machine with Chef and have my.cnf be configured so that it properly sets up MySQL to be a slave or a master depending on any attributes that the user provides, along with any system specific tuning depending on the output of ohai. The additional steps required for the second method is to actually setup replication automatically - this requires the system to know exactly which systems are configured as masters, where the masters are at in its binlogs, and what sort of credentials it needs in order to communicate with the server.

In general, you must know your application before proceeding - as an example, if you haven’t worked with MySQL replication at all, then it’s easy to screw everything up, doubly so if you automate your ignorance into the code.

What helps for me is to first document exactly what steps I’ve taken to setup said application, then document how the application ties with any other service on the network. Then redo your documentation in your favorite configuration management system. Figure out if you need each machine to know every other machine’s state; if so, then you might want to look into chef-server, the Opscode platform, or Puppet + mcollective.

Once you’ve determined that you do need some sort of way to grab machine states for each machine, you can now sit down and start writing your recipes with the assumption that you’ll be using chef-server or Puppet + mcollective.

Since I’ve been working with chef-server the most lately, I’ll talk about how I automated MySQL replication with chef-server.

In the recipes I’ve written, I’ve used the “search” feature of chef-server quite extensively in order to gather collections of nodes that match certain criteria. For instance, I have two roles, “db-master” and “db-slave” which set attributes on the nodes (that can be later pulled with ohai, knife, etc.) that says exactly what database role that was applied to the machine. For instance, role[db-master] sets node[:mysql][:server][:role] = master, and similarly, role[db-slave] sets node[:mysql][:server][:role] = slave.

Figure 1: A slice of the role file for configuring the node’s DB role.

default_attributes "mysql" => { "server" => { "role" => "slave", ... } }

When the role is thrown into the run_list of a particular node, these attributes will be set under mysql.server and will be stored by chef-server.

Now, let’s say I want to collect all of the slaves that are known by chef-server, so I can update the binlog positions/log file on each slave, or at the very least, setup slave replication if it isn’t already setup. I use the search facility in chef-server in order to poll chef-server, grab the results and stuff them into a variable.

Figure 2: An example on using search in order to collect all of the DB slaves known to chef-server, then doing something to each of these nodes.

db_slaves = search(:node, 'mysql_server_role:slave')
db_slaves.each do |slave|
  # execute CHANGE MASTER TO query…
  # or maybe, puts "ohai #{slave[:fqdn]}!"
end

Now, of course, we need a way to know which binlog position and log file the slaves need to use from the master server. Further, we need to allow the replication user to connect to the master server for each slave machine that’s known by chef-server (you really don’t want to blow open your permissions with ‘replication’@’%’, someone can guess the password and stream your data from your server. Not a tractable solution.) It ends up that we can assign values to attributes in the recipes as well: we simply grab the master status from the master, put the values into new attributes that’s pushed onto chef-server (and can be pulled with knife/ohai/etc.), and then pull those attributes out again on the slave machines.

Figure 3: Grab the master status from the master MySQL machine and throw them into attributes, then on the slave machines, pull those attributes out again and apply the changes to the slaves.

# Recipe snippet applied to master DB node
ruby_block "master-log" do
  block do
    logs = %x[mysql -u root -e "show master status;" | grep mysql].strip.split
    node[:mysql][:server][:log_file] = logs[0]
    node[:mysql][:server][:log_pos] = logs[1]
  end
  action :create
end

# Make sure that we create replication users for each slave machine known
# by chef-server.

slaves = search(:node, 'mysql_server_role:slave')

ruby_block "create-replication-users" do
  block do
    slaves.each do |slave|
      # slave_ip = <favorite mechanism to get IP address from hostname>
      %x[mysql -u root -e "GRANT REPLICATION SLAVE ON *.* TO 'replication'@'#{slave_ip}' IDENTIFIED BY 'password';]
    end
  end
  action :create
end

# Recipe snippet applied to slave DB nodes
masters = search(:node, 'mysql_server_role:master')

# Let's assume we only have one master for now
ruby_block "import-master-log" do
  block do
    # master_ip = <favorite mechanism to figure out an IP address \
                   from a hostname>
    master_log_file = masters[0][:mysql][:server][:log_file]
    master_log_pos = masters[0][:mysql][:server][:log_pos]
    %x[mysql -u root -e "CHANGE MASTER TO master_host='#{master_ip}', master_port=3306, master_user='replication', master_password='password', master_log_file='#{master_log_file}', master_log_pos=#{master_log_pos};"]
    %x[mysql -u root -e "START SLAVE;"]
  end
  action :create
end

Of course, you should REALLY add in checks to make sure you don’t accidentally clobber your data - you really shouldn’t be applying these things if the machine already has the required info in place. You’ll definitely want to add in checks to make sure that you don’t apply master binlog positions if the slave machine is already streaming updates from a master.

Another example is configuring a service that needs to have information of multiple machines on the network; mysql-mmm is a great example of this. As far as I know, with mysql-mmm conf files, you cannot have a conf.d folder with individual configurations for each machine you want to provision, plus you have defined sections in each mysql-mmm configuration file that cannot easily be split apart. You’ll definitely need to use something that can gather all of that info into a single place, then generate the conf file based on some kind of central store’s knowledge of the state of every machine.

If you guys have any questions on any of the above, don’t hesitate to let me know.

Hope this article helps!

Further Reading:

Chef Search Syntax - http://wiki.opscode.com/display/chef/Search
Solr - http://lucene.apache.org/solr/ (the search engine powering chef-server’s search capabilities)
Slides from Orchestration Panel at Cloud Connect 2010 - http://www.slideshare.net/dev2ops/orchestration-panel-at-cloud-connect-2010

Yet another site redesign

2010-10-24T00:00:00-07:00

So, I’ve decided to cut out all of the distracting fancy stuff and just go with a very, very minimal site design this time around. My intention is that the content is at the forefront, and that all of the space should only be used to house content, and nothing else.

I can easily revert back to the previous site design if people don’t like this one. Also, as always, I’ve made the site available on GitHub if you wish to use the site design. The site design is licensed under the 3-clause BSD licensed, for the exception of anything in the plugins directory (for now) and anything in the posts and _drafts directories.

UNIX and Literacy

2010-10-22T12:56:00-07:00

I was reading through this article on UNIX as literature a while ago; it resonated with me very strongly, if only because I studied philosophy (and math, which required a lot of prose writing as well) and decided to hop into system administration as my career.

I started thinking about how UNIX has affected my thoughts on computing and how I express my thoughts on paper (or, as is the case these days, on the internet.) Perhaps it was the “UNIX way” of using pipes between programs and transmitting clear, parsable text that helped me understand computing on a different level than, say, doing things the “Windows way”; with Windows, as a regular user or even as a sysadmin, your main method of expression is through the mouse: you point and grunt at things, and that’s about the extent of your grammar when communicating with your computer. On the other hand, with UNIX, your grammar is expanded quite a bit; you operate on entire sentences, phrases, and words, and the way you tell your machine to do anything is by writing things in the command line.

Yes, you might have the same experience in DOS as you might in UNIX; after all, you do have pipes in DOS and that gets you most of the way there. UNIX applications, however, often do one thing really really well, and it takes a bit of knowledge and tinkering in order to figure out how to really use each program to its full potential. sed and awk are probably two of the most powerful programs in the whole UNIX suite of programs, but most people don’t realize it until they start digging deeper. Hell, I still use sed as a simple string replacement program, without considering any of the other features that’s available in sed.

But the biggest thing that distinguishes UNIX from all other operating systems is that words and sentences are of the primary focus in the OS; it’s only later in the history of UNIX that you start seeing windowing systems, but even then, text is at the forefront of all interactions in the system.

Powershell, though a valiant effort (and a better shell than bash, from what I hear), is ultimately disappointing. Why? The things you operate on in Powershell are objects, not words, and definitely not sentences. This may be a bit better if we’re talking about our perception of the world, for we at least observe objects before we can ever talk about them with language (this is a separate topic for a different field, this can be debated for eons), but we wield words as easily as we move our arms around, and so it is much more natural to operate on words on a computer than it is to operate on objects that are constructed artificially.

As the article asserts, it’s freeing to be able to use language to express your thoughts to a computer; I feel the same exact way, too. It’s hard as hell to get used to it, especially since a lot of us have grown up using Windows, but once you’ve mastered how to express your thoughts on a UNIX system, it’s extremely hard to go back to any other paradigm.

All I have to say is:

$ echo "woot" | cowsay
    _____
   < woot >
    —–
         \   ^__^
          \  (oo)\_______
             (__)\       )\/\
                 ||—-w |
                 ||     ||

Web Security 101

2010-10-22T10:12:00-07:00

A friend of mine recently asked me what sort of security issues she should keep in mind when she’s designing web pages and uploading them to a web server.

Note that I’m differentiating between a web designer and a web developer, the latter of which would mostly work with writing code in some sort of language/framework that’s meant to be used in a web application, such as Javascript, Ruby, Python, Java, etc. This article is meant for web designers, who usually mock up UI prototypes and write the presentational stuff for a given website.

The short answer is, “not too much.” If you’re only designing static pages in HTML and CSS, there’s really not too much you’d have to be worried about - the language, for instance, dictates only structure, content, and presentation, and doesn’t touch upon anything that would interact with the system in any significant way. The only scenario that I could think of that could really screw with a web designer is if you’re writing up a web form that talks with a database, or if you’re writing up a web form that transmits passwords over the internet, to which you have to worry about sanitizing your inputs for any strings that may try to execute an arbitrary SQL statement, or making sure that the web form is strictly communicating over SSL.

The long answer is, “well, sure, there’s a few. You can’t be too cautious.” The biggest issue that could arise is improper file permissions on your files. I’ll go ahead and enumerate each issue under a different header and explain a few of the basic things that one needs to keep in mind when uploading stuff to a server, and writing up static web pages.

File Permissions

Most web servers these days are hosted on a UNIX platform; there’s a small number that are hosted off of Windows, but more likely than not, when you’re signing up for a web server at a random company, it’ll be on a UNIX system.

Therefore, you’ll need to learn a few basics of UNIX permissions, at least enough to know what to do.

There’s three groups of people that UNIX recognizes for permissions: “owner,” “group,” and “the world.” As you might expect, the owner of the file is a single user that has complete ownership over the file. Groups are simply groups of users on a UNIX system that are named explicitly on the UNIX system. Finally, “the world” is simply everyone else that is both not the owner of the file nor a part of the group that owns the file.

On UNIX, this is expressed in a three digit number attached to each file and directory.

7    7    7
|    |    |
owner group world

These numbers specify exactly what sort of permissions are applied to the file for each user, group, and everyone else. (In this particular case, 777 = read/write/execute for the user, group, and world.)

How do these numbers specify the individual permissions for users, groups, and the world?

UNIX adds up three different numbers (4, 2, and 1), with each specifying a different permission type.

4 signifies read access for the file/directory. 2 signifies write access for the file/directory. Finally, 1 signifies execute access for the file/directory.

You can also deny all permissions by simply using “0” for the specific permission bit - say we only want read/write/execute for the owner, but not for anyone else. Then we apply the permission “700” to the file, where “7” is for the owner, the second digit is for the group and the third digit is for the world.

If I wanted read/write access for all users, then I add 4 + 2 together to get 6 (since I want both read (4) and write (2) access), and then I use that number for every single group of people. Therefore, I get 666. :)

If I wanted read/write/execute for the owner, but only read only access for everyone else, then I use 744. For the first bit, 4 + 2 + 1 = 7, and then I use 4’s for the other people for read-only access.

In most FTP clients, you can right click the file and set the permissions within the FTP interface. If you have a UNIX shell, you can execute the following command to change permissions on a specific file:

$ chmod 744 [filename]

How does this affect security when you’re uploading HTML/CSS files to a web server?

If you have 777 permissions on every file, then that means you’re also granting full access to everyone on the system, including anyone who decides to hack the web server. The latter scenario isn’t totally implausible, but the former scenario is even more likely - since most shared web servers (usually the ones costing roughly $5-$10 per month) don’t have much separation between each user on the system, that means any other customer can easily muck up your website without as much having explicit access to your account.

When in doubt, use “644” permissions on your files. That sets the file to be, as you might guess by now, read/write for the owner, and read only for everyone else. For directories, use “755” permissions. If you set directories to not have the execute permission set for a specific user/group, then you won’t be able to change over to that directory in the future (this is easily fixed by setting the permissions back to 755.)

Firewalls

Typically, if you’re using another web provider like A Small Orange or whoever else, you won’t have to worry too much about this - they usually take care of those issues for you.

However, if you’re using a “virtual private server” (VPS) provider like Linode, prgmr, Slicehost, or Blue Box Group (yeah, sorry, shameless plug :)), or even your own server in your house, then you’ll have to worry about setting up a firewall.

Usually, you can setup the firewall to block all incoming traffic BUT the specific set of traffic associated with someone browsing a website. The best way to determine, uniquely, whether someone’s browsing a website or not, is by looking at what port they’re trying to use. For web traffic, the port number is port 80. If you’re serving secure content through HTTPS, then you’ll have to open up port 443 as well.

I can’t possibly cover all of the different firewall products out there, so all I can say is that you should take a look at your firewall/router manual and see how you can do port forwarding for a particular machine.

HTTPS

This wikipedia article does a way better job of explaining the issue than I ever could.

In short, you want to make sure that the server is vouched by some sort third party as a “legitimate website”, and you want to make sure that any sensitive information (like passwords and the like) are transmitted in HTTPS. HTTPS guarantees that any data being transferred between two parties (in this case, you and the server) is not transmitted in clear text.

Conclusion

This article ended up being much longer than I thought. :) If anyone needs any clarification on any of the concepts, let me know.

MySQL Replication - master slave replication with multiple R/O masters

2010-10-12T13:41:00-07:00

(NOTE: I know, the blog post says that I’m working with multiple R/O masters. The guide doesn’t tell you how to do such a thing. However, the principles are quite the same, and you can easily extend this guide to make one of your other R/O slaves to be an R/O master.)

So, after battling MySQL for a couple of days, I’ve come out with a bit more experience with replication and a few pitfalls that I’ve encountered that you may encounter as well.

The MySQL installation I was wrestling with consisted of a single R/W master, a couple of R/O slaves, a R/O slave that also acted as an R/O master, and several R/O slave machines slaving off of the R/O master.

Apologies for the ASCII art:

* - R/W Master
                 |
                /|\
               / | \
 R/O slave  - *  *  * - R/O slave
                 |\_ R/O slave/master
                 |
                /|\
               * * * - R/O slaves

We assume that the R/W master already has a bunch of data, and that we need to copy all of the data over to each machine.

The biggest thing to keep in mind is that it’s REALLY REALLY easy to mess this up - if your replication settings are off by just a little bit on the slave machines, then you’re going to corrupt your data on the slave machines.

Also, when we do the data copy to each machine, we will want to wipe out any relay logs, any master binary logs, the master.info file, and the slave-relay-bin.info file on the slave machines. We’ll recreate the master.info file through the MySQL CLI.

So, without further ado, here’s how you set everything up:

Step 1: Create a backup and ship it off to all of the slave machines.

You’ll have to make sure that either an existing slave server, or your master server, either has a lock on all tables (for mysqldump), or the daemon is switched off completely (for raw data backup with tar.) In either case, you’ll need to make sure you can read all of the data without it being written to at the same time.

In my particular case, I’ve shut off the daemon and grabbed a snapshot using tar, since I had a spare slave server with an up to date copy of the data. I streamed the compressed tar data over SSH to the slave servers:

cd /var/lib ; tar -czf - mysql | \
ssh <host> "tar -C /destination/folder -zxvf -"

Depending on how large your database is, it may or may not be popcorn time. :)

Step 2: Remove existing relay logs, master.info, etc.

By default, these files are stored in /var/lib/mysql.

Remove all of them:

cd /var/lib/mysql ; rm mysql-bin* slave-relay-bin* \
relay-log* mysqld-relay-bin* master.info

Step 3: Setup my.cnf on the master and all slave machines.

Okay, so I know this is pedantic, but we know for sure how to identify different machines from each other in some way or another, either by an IP address, a hostname, or some other metadata. MySQL uses an internal identification scheme for identifying different machines in an MySQL replication environment. You can define this ID under the [mysqld] section in my.cnf (usually located in /etc/mysql/my.cnf), using the configuration option “server-id”. These must be unique for every machine in the entire set of machines that’s participating as either a master or a slave in the replication setup.

I usually start my numbering from the master, and work my way down. For instance, I’ll number my R/W master as server-id = 1, my R/O master as server-id = 2, one of the slaves as server-id = 3, etc.

We’ll split this step into two parts: first part will be for master machines (R/W and R/O), and R/O slave machines.

R/W and R/O masters:

You’ll want to make sure that the following are defined in my.cnf:

log-bin = /var/log/mysql/mysql-bin.log
log-bin-index = /var/log/mysql/mysql-bin.index
log-slaves-updates

For the R/O master, you’ll want to also set read_only = 1.

R/O slave:

relay-log = /var/log/mysql/slave-relay.log
relay-log-index = /var/log/mysql/slave-relay.index
relay-log-info = /var/log/mysql/slave-relay.info
replicate-ignore-db = mysql
read_only = 1

One more note: make sure skip-slave-start is set on all slave machines in my.cnf for now; we’ll delete the line later, but this is necessary so we can have as much time as possible in setting up the slave machines.

Step 4: Setup MySQL replication on all slaves.

We’re now ready to set up MySQL replication on all of the slave machines.

We’re going to set up grants on the R/W master to the two slave machines and the R/O master, and then we’re going to set up grants on the R/O master for each of its slaves.

Let’s assume that the IP address scheme is as such:

R/W Master: 10.0.0.1
|_ R/O Slave: 10.0.0.2
|_ R/O Slave: 10.0.0.3
|_ R/O Master: 10.0.0.4
 \_ R/O Slave: 10.0.0.11
 |_ R/O Slave: 10.0.0.12
 |_ R/O Slave: 10.0.0.13

Alright, let’s login to the R/W master and grant replication rights to its three slaves:

rwmaster# mysql -u root
mysql> GRANT REPLICATION SLAVE ON *.* TO \
’replicate’@’10.0.0.2’ IDENTIFIED BY ‘foobar’;
mysql> GRANT REPLICATION SLAVE ON *.* TO \
’replicate’@’10.0.0.3’ IDENTIFIED BY ‘foobar’;
mysql> GRANT REPLICATION SLAVE ON *.* TO \
’replicate’@’10.0.0.4’ IDENTIFIED BY ‘foobar’;
mysql> FLUSH PRIVILEGES;
mysql> \q

Stop the master’s daemon:

rwmaster# /etc/init.d/mysql stop

Now, let’s login to the R/O master and grant replication rights to its three slaves:

romaster# mysql -u root
mysql> GRANT REPLICATION SLAVE ON *.* TO \
’replicate’@’10.0.0.11’ IDENTIFIED BY ‘barfoo’;
mysql> GRANT REPLICATION SLAVE ON *.* TO \
’replicate’@’10.0.0.12’ IDENTIFIED BY ‘barfoo’;
mysql> GRANT REPLICATION SLAVE ON *.* TO \
’replicate’@’10.0.0.13’ IDENTIFIED BY ‘barfoo’;
mysql> FLUSH PRIVILEGES;
mysql> \q

We grab the master status from the R/W master and the R/O master, and note it down for later usage:

rwmaster# mysql -u root
mysql> SHOW MASTER STATUS\G
…
File: mysql-bin.log.000500
Position: 5150150
…

romaster# mysql -u root
mysql> SHOW MASTER STATUS\G
…
File: mysql-bin.log.000001
Position: 4
…

Again, note these down. The R/W master binlog file and position will be needed for its immediate slaves and the R/O master, and the R/O master file and position will be relevant for its immediate slaves. Also note down the passwords and username for each grant: we’re going to use this information to connect to the respective masters in order to start replication.

YOU CANNOT USE THE SAME BINLOG FILE AND INFORMATION FOR ANY SLAVES NOT DIRECTLY CONNECTED TO THE MASTER. If you do, expect replication to break. I’ve learned this the hard way. Always assume that any replication you’re setting up is relevant only for any direct connection between a master/slave.

With these binlog files and positions in mind, you’ll now set up each slave to connect to its direct master.

We’re going to start with the R/W master’s direct slaves.

Connect to each slave and enter in the following information:

mysql> CHANGE MASTER TO \
MASTER_HOST=’10.0.0.1’, \
MASTER_PORT=’3306’, \
MASTER_LOG_FILE=’mysql-bin.log.000500’, \
MASTER_LOG_POS=’5150150’, \
MASTER_USER=’replicate’, \
MASTER_PASSWORD=’foobar’;

where MASTER_LOG_FILE and MASTER_LOG_POS is grabbed from the output of ‘SHOW MASTER STATUS\G’ on the R/W master.

Now, connect to the R/O slaves that are connected directly to the R/O slave, connect to the MySQL daemon, then type in the following:

mysql> CHANGE MASTER TO \
MASTER_HOST=’10.0.0.4’, \
MASTER_PORT=’3306’, \
MASTER_LOG_FILE=’mysql-bin.log.000001’, \
MASTER_LOG_POS=’4’, \
MASTER_USER=’replicate’, \
MASTER_PASSWORD=’barfoo’;

In order to be extra cautious about the data on each slave, we want to start from the least significant slaves (the ones attached to the R/O master, since they won’t receive their updates until the R/O master receives its updates from the R/W master), and then work our way up until we get to the R/W master. (Reason why we’re being paranoid about this is because it’s very possible, if we worked in the opposite direction, for the R/W master to get too ahead of the slaves for us to actually set up the slaves for replication. In this case, we’d have to start over from scratch and create a new snapshot of the data. Not good.)

Go to the R/O master’s slaves and type the following in:

mysql> START SLAVE;

Then go to the R/O master and the R/W master’s slaves and type the following in:

mysql> START SLAVE;

Now start the R/W master’s daemon back up:

rwmaster# /etc/init.d/mysql start

You can run ‘SHOW SLAVE STATUS;’ on each slave machine to see if it’s replicating normally.

If not, you can check /var/log/mysql/mysqld.log (or mysqld.err) and see what’s breaking.

That’s it!

Let me know if this guide has been helpful for you guys. :)

Quick update on everything as of late

2010-10-09T16:52:00-07:00

Hey guys.

I haven’t been posting all too much lately - I’ve recently started over at Blue Box Group, LLC and have hit the ground running.

Now that I have my breath from working here for two weeks, I can go over a few things that’s been happening as of late (not necessarily work related.)

Again, I’ve been hired by Blue Box Group as a senior system administrator. This job is really fucking awesome - I’ve had several projects thrown at me since I’ve started, and I’m learning quite a bit about everything. Things are excellent on this front. This is to no detriment to my previous employer, 3TIER. They’re an excellent group to work for, and they’re awesome people in general. I just think, at this point in time, I need to get my feet very much wet - this means going in head first into crazy problems right away.
I’ve been reading a lot of things… and I really should stop doing that. Why? Because my mind is way too scatterbrained to keep focus on many things. I think the one thing I need to sit down with, until I’m at a really good point to stop, is probably “Structure and Interpretation of Computer Programs.” I’m not even through the first chapter; it’s a challenging book, but a good one to read, and I hope to learn a bit of much needed computer science training that I’ve skipped over when I was in college. While I was reading this book, I’ve also found a bunch of tutorials and a few other books that I should read; the two that stick out for me at the moment are “Web Operations,” and “Let’s Build a Compiler.” “Web Operations” will likely be the next thing I’ll concentrate on, though I’ve read through half of the book so far.
I’ve finally ordered my tickets for LISA ‘10! I’ve my hotel and flight booked already, and so I’m all set for this year’s LISA! There’s a chance I’ll speak about the LOPSA Mentorship Program at LISA at one of the BoF’s, but we’ll see. Can’t wait to meet all of you guys in the flesh. :)
Finally, I’ve whittled down a few outstanding holes in my knowledge that I really need to work on. First, I’m extremely weak with databases. I’ve only a cursory understanding of MySQL replication, I haven’t crafted many statements longer than a “SELECT * FROM TABLE”, and I also haven’t seen many of the problems that can go wrong in a database. Second, I’m extremely weak with essential computer science concepts. I don’t know too many data structures nor algorithms, which hampers a lot of things when it comes to performance tuning and optimization of code. The way I plan to tackle these two deficiencies is to both tackle a ton of projects that involve exercising both skills, and also attend conferences and read books on the subject. I’ve signed up for the basic databases tutorial at LISA ‘10, and while this will obviously not be exhaustive, it’ll at least get the ball rolling. I imagine a lot more reading and a lot more programming will help me with my comp-sci-fu.

Yep. :)

Experiences with leading teams

2010-09-27T08:11:00-07:00

I’ve been chairing the LOPSA Mentorship Program for the past few months; I was approached by Lois Bennett about leading the program, and so I took the helm, without much of any experience whatsoever. I didn’t really use any methodologies that are already established as “best practices”; the Agile framework definitely was an influence, but my style is to not be heavy handed with much of anything.

I think the biggest thing that I learned to do is to always, always let the team decide the best course of action. My role is to only guide the discussion, and make sure that any meetings don’t last too long. If I had an opinion, I would throw it into the ring, but more often than not, it would be torn apart by the rest of the team members; this is good, the best ideas win when things are done this way, in my opinion. If a discussion went on for too long, I’d guide people to talk about it either on IRC or on the mailing list, so we can get the meeting done faster.

I never try to assert my position of power. I’ve never once considered myself to be “above” anyone else - everyone else in the team are considerably more experienced than me, and even if they weren’t, it would do no one good if I asserted my power at every discussion. People need to be autonomous and be able to say, without fear, what their opinion is.

We communicated over the phone once a week with a free conference call provider, sussed out any minute details on our mailing list, and if anyone wished, they could also talk to us through IRC. The way we communicated was already established before I started leading the program, and so it was important to keep it the same.

I think, what’s also important, is to always delegate the work. This does several things for you: one, it frees up your time so you can prepare for more discussion; two, the people on the team are now accountable for getting the things done, which, I suppose in theory, should empower the people on the team. It’s not exactly good to be the leader and hog all the work.

I think, what I personally need to work on, is better organization, and just simple “thank you’s” to people. Everyone’s done a really great job, and I’m not sure if I did such a great job in letting everyone know that.

Any sort of suggestions are welcome - I see my career sticking on the technical track, but I figure that I’ll be leading teams of people at work sometime down the road.

Why work in system administration?

2010-09-22T20:10:05-07:00

I will admit right away: system administration is really not a sexy field to be in, at least compared to the other technical fields out there. However, I personally feel quite at home in system administration, and based on my limited exposure so far, it can also be one of the most fulfilling fields to work in.

The range of responsibilities may vary between teams and different companies, but at its core, we manage multiple machines, whether they be hooked into a network or not. Some departments employ sysadmins in order to not only look over the network and systems of computers, but also to administer any other device that hooks into a computing device, like a network printer, fax machines, etc. Other roles are far removed from the actual physical hardware, to which the sysadmin takes more on a role of ensuring that such systems of computers are actually doing what they’re supposed to do (which is to serve that website you always go to, or to let you login with your username and password without compromising security, etc.)

But why work as a system administrator rather than as a software developer, or as a lawyer, doctor, etc.?

The biggest reason that the field appeals to me is that I get to solve interesting problems, day after day; it’s awesome to solve issues involving hundreds, or even thousands, of computers. It’s a natural extension of the knack that one gains when they’re fixing a single computer - now that you’re working with hundreds/thousands of machines, you now have to think about picking a solution that is tractable to deploy to a whole system of computers.

The system administrator often keeps his nose as close to the particular inner workings of a system, and thus, usually is the domain expert on weird issues that happen with machines. As of late, this knowledge is not necessarily valued as much as it was before, but it’s still ever more important - for instance, when deploying things to EC2 or even bare metal, you need to know how to measure IOPs, and you even need to know why you’re measuring that in the first place (maybe you’re deploying an I/O intensive service?) When deploying services in the cloud, more often than not, you’re working with machines that are significantly underpowered compared to what you can get as a bare metal machine; therefore, these weird things that you’ve been learning about file system performance and whatever else suddenly become important.

No matter what anyone says, the system administrator is, more often than not, one of the most important roles in a company, especially when your company is providing large amounts of computing resources for customers. They are the guys who both make sure your desktop is running smoothly, and make sure that your high traffic website doesn’t fall under the pressure of traffic, nor under the pressure of system crackers. They’re responsible for all computing resources, all the way up (but not necessarily including) the application level, where the software developers thrive (I’m not saying that software devs aren’t concerned about system-level issues, it’s just that it’s not usually their central focus point, and it’s something that most devs aren’t responsible for in production environments.)

It’s not sexy, but it’s extremely fulfilling. I encourage everyone to take a look at the field and see if it’s for you.

LOPSA Mentorship Program

2010-09-22T19:01:30-07:00

We’ve launched the LOPSA mentorship program about a week ago, and we’re now soliciting for proteges and mentors to sign up for the program.

Why sign up for the LOPSA mentorship program even if you have great mentors at work already? We think that it only helps you, as a system administrator, to have several mentors (if you already have one or two guiding you through your career); if you don’t already have a mentor, then you’ll benefit greatly by signing up for the program, no matter your skill level.

And for those wondering whether you should be a mentor or not, you don’t have to worry about where your skill level is at, either - you don’t have to have 10-15 years of experience in the field to help someone else. It’s also one of the better ways to learn more about your field - helping someone with a particular problem can also solidify your own skills in that particular problem domain.

So, if you’re wanting to be a protege, you don’t have to be a LOPSA member - it’s totally free. If you’re wanting to be a mentor, you do have to be a LOPSA member, though for a short period of time, you can register for only $35.

Thanks everyone! And be sure to sign up if you’re interested!

Official press release:

http://lopsa.org/mentor_program_PR

My foray into Lisp, SICP, and emacs

2010-09-12T07:40:45-07:00

So, after many jokes between me, my coworkers, and various friends, I’ve finally decided to break down and learn emacs. This is a bit after I decided to learn a bit of material from Structure and Interpretation of Computer Programs, which uses Lisp as its primary programming language.

I’ll first go over the history of what editors I’ve used over the years.

When I first started using Linux back in high school, it was quite hard for me to wean myself away the Windows Way of doing things. “You mean I have to learn how to use the terminal?!?” I did get away with just using gedit, which was a comfortable text editor to use when I was in front of a Linux machine with GNOME installed.

However, I eventually had to work on remote machines with no GUI at all, which means I had to learn some sort of terminal text editor for me to be somewhat productive. Thus, the next easiest thing for me to use at the time was nano. It was very simple to adapt to, in the sense that I didn’t have to memorize a ton of different key combinations, nor did I have to figure out how to insert text into a document. I just typed stuff and it worked as I expected it to.

I forgot exactly when I started using vi(m), but I realized that I had to learn it, since it’s usually included by default on all UNIX systems. It took me a while to get used to the way vi(m) worked (especially alternating between command mode and insert mode), and it took me a while to also figure out that it was The Better Way compared to nano. I became a bit more fluent in it, and was able to edit files with a lot more ease than with nano.

So anyway, I switched between nvi and vim for a while because I thought BSD vi was a lot better, it didn’t impose any stupid syntax highlighting rules which didn’t work half the time in vim, etc.

I’ve now just learned that emacs handles syntax highlighting way better than vi(m). Plus, it’s easy to extend! It’s all in Lisp; I figure this is a good way to sink my teeth into Lisp along with learning Lisp through SICP.

SICP is a great text by the way - I’m not through the first chapter quite yet, but the problem sets seem to be both challenging and deep in its concepts that it’s trying to convey. It’s taking me a while to really understand how Lisp works; there’s a lot of “unlearning” I have to do in order to really grok Lisp.

Anyway, that’s really about it. I heavily recommend SICP to anyone else who’s wanting a great introduction to programming and computer science.

I’m not quite sure yet if I would recommend emacs over vi(m) quite yet - I don’t want to incite a religious war. :)