Puppet-ing my home network
For just over 12 months, I’ve been using Puppet as a way of configuring the various machines I have dotted around my home network. The initial desire to move to an automated configuration management tool was to keep tabs on the various requirements across pieces of software I run, however I’ve now moved to the point where all deployments of new hardware and software (both my own, and third party) are all managed through Puppet, and life is much simpler (most of the time!). I’ve been meaning to write this post for a while, mainly as a “getting started” point for anyone who is thinking about embarking on a similar journey, hopefully this will help people avoid making some of the same mistakes I made along the way!
Firstly, a little about my network, to give some of the points context. Over the last 5 years, I’ve evolved from having a single desktop machine, to today having the same desktop, 2 microservers (one for mass storage, the other for backup, services and acting as the Puppetmaster), a Home Theatre PC, and 7 Raspberry Pis in various guises (an alarm clock, appliance monitors and TV encoder between them). All of these machines were originally configured manually, and re-imaged from scratch every time something went wrong, leading to endless pages of scribbled notes about various requirements and processes to get things working. Now, all of them pull a catalogue from a single Puppetmaster, which I update as requirements change. Adding a new host is a matter of installing a fresh operating system, installing Puppet, and adding a few lines to the Puppetmaster detailing what I want installed – the rest is taken care of!
I’ve ended up using Puppet for much more than I originally imagined, including:
- SSH Keys
- MySQL hosts & databases
- Apache site configuration
- Tomcat configuration (auto-deploying apps is on my to-do list)
- Sendmail
- Automated backups
- Media sorting
- ZNC configuration
Plus even more besides that! Most bits of my own software run through supervisord, which is also configured through Puppet.
The first, and arguably most important lesson I learned through my Puppet-isation process is to keep the configuration in some sort of versioning system. Initially I did not keep track of the changes I was making to Puppet manifests, and requirements that were added at 3am seemed completely illogical in the cold light of day – but by self-documenting the manifests as I went along, and tracking changes through my own Subversion repository, I can look back at the changes I’ve made over time, and save myself the hassle of accidentally removing something essential. I’ve lost track of the number of times I’ve had to revert back to a previous configuration, and having a system in place to do this automatically will definitely help you in the long run!
While writing your configuration, I’ve learnt to be explicit about your requirements. A lot of the modules and class definitions I wrote early on did not use any of the relationships you can define between attributes, and as a result I’d often be bitten by the Puppet parser attempting to configure things in an illogical order. For the sake of adding a few arrows into your manifests, it’s worth using these just to avoid the head-bashing when a configuration run fails intermittently! I still run into problems when configuring a new host, when a package I require isn’t installed in time to start up a service – using relationships from the ground up will hopefully avoid this ever becoming an issue.
Arguably the second most useful tool I installed (after Puppet itself) is the Puppet Dashboard. This fantastic tool pulls in the reports that Puppet generates, and spits them out into a very readable format which allows you to get straight to the heart of what’s causing failed runs, rather than having to resort to diving through the depths of the raw logs. It took me a while getting around to installing this, and I honestly regret not installing it straight away – it has saved me an incredible amount of time since. A word of warning though – the logs can take up an awful lot of space (my dashboard database is 400MB+ with only the last 3 months data stored) – make sure your SQL server is up to the task before starting this!
While installing things like the Puppet Dashboard, heira and the like, I’ve taken to Puppet-ing your Puppet, by which I mean writing the configurations for these tools into your Puppet manifests. Initially this seemed very strange to me, I didn’t want some almost-sentient piece of software configuring itself on my machine! However, the client will quite happily run on the same machine as the server to configure itself (in a strange cyclical dogfooding loop). There are plenty of third-party modules available that will do a lot of this for you, but I ended up writing my own for several things – as long as the installation is managed somehow, you’re going to have a much better time!
Linked to that, and my last tip that I’ll post here, is to write your own modules wherever you can. For the first few months after starting using Puppet, I tried to cram everything imaginable into the main manifest directory, and barely used templates and source files at all. Subsequently I’ve learned to separate the concerns wherever possible, which leads to much cleaner code in the module definitions, and much nicer looking node definitions as well. Third-party modules have been a mixed blessing for me, some of them have been coded extremely well to allow multi-platform support, but some have been hard-coded to alien platforms making them useless to me. I’d absolutely recommend going looking for someone else who’s done the hard work first, but don’t be afraid to roll up your sleeves and write a module yourself. Maybe even post it online for other people to enjoy! (I’m very hypocritical making that last point, none of my modules have made it online as they’re all terrible!)
I hope that some of the above might help steer someone in the right direction when embarking on a journey with Puppet. Obviously I’d recommend some more official reading before converting your professional network of 10,000+ machines across, but if you’re like me and need a comparatively small network taming, then take the plunge and get started!