A few weeks ago, we at Ticketmaster, silently open-sourced our internal system configuration software which consists of 3 pieces of software. I led the effort to get them open-sourced, so I’m particularly happy about and proud of this. In addition, I develop on all 3 pieces of software, and am the primary author of one of them.<\/p>\n
Now, there are already many pieces of open-source system configuration software out there: cfengine and puppet are the two best known. So why release our own? Well, the software out there didn’t meet our needs – not just on a feature level, but on a design level. Several years ago, we wrote out own, and have been developing it ever since. Having matured, we felt this was something the community could benefit from.<\/p>\n
The software 3 pieces of software we released were:<\/p>\n
With these three pieces of software, Ticketmaster manages a global infrastructure of more than 3000 machines. In fact, the majority of those machines (almost 3000) are run by a team of 8 people. The team was recently grown to help account for the many new projects we’re taking on, but when I joined, 4 people (including myself) were managing ~2000 Linux systems running a variety of configurations and software on a variety of hardware, in a variety of locations. What this means is automation<\/strong> and high server-to-admin ratios<\/strong>.<\/p>\n “Sure,” you say, “but it assumes you’re running an infrastructure like Ticketmaster’s!” Well, that’s not true. The last several months of development time were purely dedicated to abstraction and configurability of the way in which these tools work. That isn’t to say there aren’t assumptions – there are. However the assumptions are not specific to Ticketmaster, and every assumption made is documented and either on the list to remove or justified.<\/p>\n So, lets have a look at what each of these tools does and why you want to use it.<\/p>\n I mentioned that Spine is a configuration management framework<\/em>. Unlike other configuration management systems, it’s not designed to, for example, know how a DNS server should be configured. Instead, it’s designed for you to show it how you want<\/em> a DNS server to be configured, and to be able to configure that server, 100% reproducibly. Further, it’s designed to be able to account for, say, slight variations among your DNS servers, but still allow you to have a single definition of what the configuration files should look like. This is done through a combination of a hierarchical data model<\/strong> and templates<\/strong>.<\/p>\n Spine, like most extensible tools, is broken into a set of plugins. You have PackageManager plugin which makes calls to your package-manager (yum, apt, etc. – it currently only supports apt-rpm, but that’s in the process of changing). PackageManager make sure the software you need installed is installed and further that only<\/strong> the software you need installed is there. It does this by walking the dependency tree for the allowed list of software and removing anything else installed. This prevents bit rot<\/strong> and enforces consistency<\/strong> – two common problems in large environments.<\/p>\n Spine also has a SystemHarden plugin to perform various hardening tasks like strip setuid\/setgid bits from anything that’s not supposed to have them. There’s a Startup plugin to ensure the services that are supposed to be running are running and configured to start at boot. It also provides service restarting for services whose configuration is changing due to spine.<\/p>\n These are all the basics that are required for configuration system, but they’re not unique. What is unique is that every configuration file you put into the spine configuration is template-ized and evaluated on that box using the configuration data that applies to that box. Before one can see how powerful this is, we need to take a look at the data model.<\/p>\n The data-model for spine is hierarchical. This hierarchy is configurable, but let’s make up an example hierarchy. We might have a network level for every network – for example, lets say spine is running on a machine with the IP address of 192.168.1.1\/22, it’ll descend the configuration for the \/network\/192.168.1.0-22 part of the tree. Next up, we might want to descend the businessunit for that machine (which at Ticketmaster is determinable by the hostname, but spine can get this information from a variety of places). I work in a group called ‘websys,’ so spine would descend \/websys. Here things like which sysadmins have access will get added to the configuration. Next up, we may want configurations based on the type of environment: dev, qa, staging, prod, etc. and descended under the business unit<\/strong>. So, if this is a machine in our dev environment, spine would add configuration data for \/websys\/dev. Of course, we could also configure spine to descend some global \/dev area too. Next, the configuration for the product in dev would make sense to descend. For example, \/websys\/dev\/front_web, and so on, until there’s nothing left to descend.<\/p>\n Of course, hierarchy is great, but sometimes you want to bypass it. For example, all machines of product “<product>” might need a common config regardless of being in dev, or qa, or prod, or anywhere else. So, each level can include other parts of the tree. These other parts of the tree that are not inherently in your hierarchy are called “config groups”… they can be stored anywhere, but at Ticketmaster, we store them at the business-unit level since that makes the most sense for us.<\/p>\n So, lets take an example. Lets say, at your global level, you have a key called “packages” which is what the PackageManager plugin uses to determine what should be installed. At the global level you have the base packages for your whole company. Then, at your business unit, you add a few packages your team uses. Keys are cumulative, so at your business unit, you just define the extras. Of course, dev will need compilers, debuggers and profilers, so you just add that at the dev level. And so on. Now you have a complete package list that’s dynamic per class of machine.<\/p>\n Using this hierarchical data, you can define templates written in TemplateToolkit for configuration files such as \/etc\/hosts<\/em>, \/etc\/fstab<\/em>, httpd.conf<\/em>, odbc.ini<\/em>, etc. While we currently have a special plugin for handling authentication (\/etc\/passwd, authorized_keys, \/etc\/shadow, etc.), you could even write templates to handle these (and we did for years).<\/p>\n What this means is having one<\/strong> \/etc\/hosts for your entire businessunit (or for your entire company if that makes sense). Just update the data so that the template can be rendered correctly for each machine. TemplateToolkit is a fully-featured (if somewhat odd) programming language. If you write your templates correctly, you rarely ever have to change them – you just check in new config data and your templates adjust themselves. This makes managing thousands of hosts a lot like managing 10 hosts.<\/p>\n Plugins are easy to write. Spine is written in perl and has a standard API for new plugins.<\/p>\n Provision was written by yours truly to handle a gap. Lets say spine knows how to configure webservers in your webfarm, and lets say you need to add 10 to one of your farms. Well, once it’s up and running, spine will do the rest. But from nothing to ‘running’ isn’t handled. This is were provision comes in. Provision will find an IP in the right network, allocate DNS for it (forward and reverse), allocate NFS storage (well, assuming you have NetApp filers, but a plugin could easily be written for anything else), and optionally create a VMWare virtual machine for it, and start that machine booting and kickstarting. If this machine needs to be built on real hardware, the only step it won’t cover is starting the kickstart process.<\/p>\n Provision assumes that your reverse DNS zones are the authoritative location for IP allocations, and that RCS is a valid locking mechanism. Like spine, most of provision’s work happens in plugins: the DNS plugin, the filer plugin, and the VM plugin. As such you can chose what sorts of allocations will happen. And since spine is so good at writing configuration files, you can have it write out the config file for provision.<\/p>\n Provisions configuration comes from an easy and flexible configuration file format. Using this file, it’s easy to represent a very wide variety of configurations so that provision can properly provision things in your environment. However, if the configuration has a limitation that prevents you from expressing your environment, you can write a plugin hooking into the necessary callbacks to change how provision will act.<\/p>\n Plugins for provision are also very easy to write. In fact, there are two kinds: provisioning plugins, and local decision plugins. A provisioning plugin is something you write to provision some resource. The DNS plugin is an example of this. Local decision plugins are what I mentioned in the previous paragraph. Provision provides a series of callbacks for local decision plugins so you can hook into provision to make complicated decisions. While provision is highly configurable, there’s no way to predict the kind of configurations every shop will have – so provision was designed with callbacks throughout all of its logic for maximum flexibility.<\/p>\n Onall is a simple perl utility to ssh to multiple machines in parallel and execute some commands – or alternatively, to copy a script to the remote machines, run it, and remove it. It’s configurable – you can choose how many machines to do in parallel, what timeouts to set, how long to wait between parallel sets, and more.<\/p>\n Unlike clusterssh, onall doesn’t open a window to each machine. Onall is for non-interactive work, so don’t go throwing away clusterssh – it’s an invaluable tool for interactive parallel work. However, it doesn’t scale for highly parallel work – especially when interactivity isn’t needed.<\/p>\n Onall can dump the output of your work to your terminal window, or to a directory – one file per system. If you let it go to your terminall, you can choose to let onall buffer it and try to group it into readable segments per host, or to just dump it as it comes. Onall takes a list of hostnames on standard in, and the commands to run as command-line arguments. While we have an internal tool to generate a list of hosts for us (called nhs), it’s highly Ticketmaster specific and releasing it would have done no one any good. We assume you have or can write a similar tool.<\/p>\n I covered a few of the strengths of these tools in their descriptions, but lets go ahead and list the big strengths and weaknesses of these tools here.<\/p>\n Our strengths are…<\/p>\n Our weaknesses are…<\/p>\n We are constantly working on improvements. Coming very soon are:<\/p>\n Those are the big ones, but there are other smaller features and a variety of bugs always been worked on and fixed respectively.<\/p>\n If you decide to use these tools, here are a few things to keep in mind:<\/p>\n A few weeks ago, we at Ticketmaster, silently open-sourced our internal system configuration software which consists of 3 pieces of software. I led the effort to get them open-sourced, so I’m particularly happy about and proud of this. In addition, I develop on all 3 pieces of software, and am the primary author of one […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[5],"tags":[],"_links":{"self":[{"href":"https:\/\/www.phildev.net\/phil\/blog\/index.php?rest_route=\/wp\/v2\/posts\/109"}],"collection":[{"href":"https:\/\/www.phildev.net\/phil\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.phildev.net\/phil\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.phildev.net\/phil\/blog\/index.php?rest_route=\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.phildev.net\/phil\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=109"}],"version-history":[{"count":0,"href":"https:\/\/www.phildev.net\/phil\/blog\/index.php?rest_route=\/wp\/v2\/posts\/109\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.phildev.net\/phil\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=109"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.phildev.net\/phil\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=109"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.phildev.net\/phil\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=109"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Spine<\/h2>\n
Provision<\/h2>\n
Onall<\/h2>\n
Strengths and Weaknesses<\/h2>\n
\n
\n
Things coming<\/h2>\n
\n
Getting the most out of these tools<\/h2>\n
\n
\nSpine can’t keep your systems in-sync if it’s not run regularly. Depending on your environment, your ability to perform regular maintenance may differ, but I recommend trying to never let a system go more than a month without a spine run. If you have a highly redundant environment where clusters can be taken out of service easily, frog-leaping spine-runs between clusters may be a good choice for you as well. Running spine regularly also helps enforce the next rule.<\/li>\n
\nMake sure all<\/em> your configurations are in spine. That doesn’t mean every config file has to be in spine. But any config file you ever need to change must go into spine. If, for example, you’ve never changed \/etc\/inittab, don’t put it in spine. But the first time you need to modify, it’s time to write a template for it. Even if that template is just a copy of the file with one little IF statement. If you don’t do this, then there’s no guarantee of your changes sticking around – and your system fails to be reproducible. Also, regular spine runs can enforce this because the spine run will remove (some) configurations it doesn’t know about.<\/li>\n
\nIf you use provision with spine (I say if, because people are more likely to have this piece automated already), make sure you *always* use provision. Provision is designed have everything provisioned through it. If you know what you’re doing, you can do things manually, but provision does not attempt to sort it’s zonefiles or do other things to make manual editing particularly easy – and this is very much on purpose. If you don’t use provision, human errors like not provisioning reverse DNS are liable to happen. If this happens, provision now has inaccurate data. While provision is very good at detecting this, such a state is probably not one you want to be in.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"