Category Archives: servers

Puppet-ing my home network

Posted on by

For just over 12 months, I’ve been using Puppet as a way of configuring the various machines I have dotted around my home network. The initial desire to move to an automated configuration management tool was to keep tabs on the various requirements across pieces of software I run, however I’ve now moved to the point where all deployments of new hardware and software (both my own, and third party) are all managed through Puppet, and life is much simpler (most of the time!). I’ve been meaning to write this post for a while, mainly as a “getting started” point for anyone who is thinking about embarking on a similar journey, hopefully this will help people avoid making some of the same mistakes I made along the way!

Firstly, a little about my network, to give some of the points context. Over the last 5 years, I’ve evolved from having a single desktop machine, to today having the same desktop, 2 microservers (one for mass storage, the other for backup, services and acting as the Puppetmaster), a Home Theatre PC, and 7 Raspberry Pis in various guises (an alarm clock, appliance monitors and TV encoder between them). All of these machines were originally configured manually, and re-imaged from scratch every time something went wrong, leading to endless pages of scribbled notes about various requirements and processes to get things working. Now, all of them pull a catalogue from a single Puppetmaster, which I update as requirements change. Adding a new host is a matter of installing a fresh operating system, installing Puppet, and adding a few lines to the Puppetmaster detailing what I want installed – the rest is taken care of!

I’ve ended up using Puppet for much more than I originally imagined, including:

  • SSH Keys
  • MySQL hosts & databases
  • Apache site configuration
  • Tomcat configuration (auto-deploying apps is on my to-do list)
  • Sendmail
  • Automated backups
  • Media sorting
  • ZNC configuration

Plus even more besides that! Most bits of my own software run through supervisord, which is also configured through Puppet.

The first, and arguably most important lesson I learned through my Puppet-isation process is to keep the configuration in some sort of versioning system. Initially I did not keep track of the changes I was making to Puppet manifests, and requirements that were added at 3am seemed completely illogical in the cold light of day – but by self-documenting the manifests as I went along, and tracking changes through my own Subversion repository, I can look back at the changes I’ve made over time, and save myself the hassle of accidentally removing something essential. I’ve lost track of the number of times I’ve had to revert back to a previous configuration, and having a system in place to do this automatically will definitely help you in the long run!

While writing your configuration, I’ve learnt to be explicit about your requirements. A lot of the modules and class definitions I wrote early on did not use any of the relationships you can define between attributes, and as a result I’d often be bitten by the Puppet parser attempting to configure things in an illogical order. For the sake of adding a few arrows into your manifests, it’s worth using these just to avoid the head-bashing when a configuration run fails intermittently! I still run into problems when configuring a new host, when a package I require isn’t installed in time to start up a service – using relationships from the ground up will hopefully avoid this ever becoming an issue.

Arguably the second most useful tool I installed (after Puppet itself) is the Puppet Dashboard. This fantastic tool pulls in the reports that Puppet generates, and spits them out into a very readable format which allows you to get straight to the heart of what’s causing failed runs, rather than having to resort to diving through the depths of the raw logs. It took me a while getting around to installing this, and I honestly regret not installing it straight away – it has saved me an incredible amount of time since. A word of warning though – the logs can take up an awful lot of space (my dashboard database is 400MB+ with only the last 3 months data stored) – make sure your SQL server is up to the task before starting this!

While installing things like the Puppet Dashboard, heira and the like, I’ve taken to Puppet-ing your Puppet, by which I mean writing the configurations for these tools into your Puppet manifests. Initially this seemed very strange to me, I didn’t want some almost-sentient piece of software configuring itself on my machine! However, the client will quite happily run on the same machine as the server to configure itself (in a strange cyclical dogfooding loop). There are plenty of third-party modules available that will do a lot of this for you, but I ended up writing my own for several things – as long as the installation is managed somehow, you’re going to have a much better time!

Linked to that, and my last tip that I’ll post here, is to write your own modules wherever you can. For the first few months after starting using Puppet, I tried to cram everything imaginable into the main manifest directory, and barely used templates and source files at all. Subsequently I’ve learned to separate the concerns wherever possible, which leads to much cleaner code in the module definitions, and much nicer looking node definitions as well. Third-party modules have been a mixed blessing for me, some of them have been coded extremely well to allow multi-platform support, but some have been hard-coded to alien platforms making them useless to me. I’d absolutely recommend going looking for someone else who’s done the hard work first, but don’t be afraid to roll up your sleeves and write a module yourself. Maybe even post it online for other people to enjoy! (I’m very hypocritical making that last point, none of my modules have made it online as they’re all terrible!)

I hope that some of the above might help steer someone in the right direction when embarking on a journey with Puppet. Obviously I’d recommend some more official reading before converting your professional network of 10,000+ machines across, but if you’re like me and need a comparatively small network taming, then take the plunge and get started!

Media storage & sorting

Posted on by

It’s been a long time (over a year in fact) since I last blogged about my home AV setup, and I left the last post on a real cliffhanger – how do I sort and store all these videos? Well, wait in suspense no longer, because here’s the answer!

Firstly, a bit about the storage – for a while, I have crammed more and more hard-drives into a single old desktop machine that I re-purposed to be my media centre/SVN repository/Tomcat/Apache/seedbox/etc, but it was getting to the point where I had files spread across 3 different drives, and no redundancy in the event of drive failure. Given I’ve had more than a few drives die on me recently, some sort of recovery strategy was becoming very necessary. We’re not talking small size requirements here either – somehow I’ve managed to accrue over 4TB of videos, music, pictures and other files. With these requirements in mind, I went shopping for some sort of Network Attached Storage style of device, but with more of a DIY edge to it. Having looked closely at Drobo and the like, I decided I’d rather have full control of the OS and RAID setup; the idea of not knowing what’s going on under-the-hood, and not being able to recover the data apart from on another Drobo, was not appealing at all. I ended up getting a HP ProLiant N40L Microserver, sadly well after the £100 rebate ended, but still at a very good price for a decent specification of machine and a nicely engineered case. I moved the 250GB system drive to the optical drive bay as documented here, in went three 3TB hard drives, on went a fresh copy of Ubuntu 12.04.2 server, and I was away. The fourth drive port on the ProLiant is left empty for now, with a view to putting in another 3TB drive when I eventually run out of space again.

I’ve heard interesting things about ZFS recently, but decided to stick with what I know, and set up a RAID5 array across the three disks, giving me 6TB usable space, and allowing for the failure of any single disk. This arrayis then shared over NFS so that my other server can access to add files, and my Mac Mini (running XBMC) can access the content to display on the TV.

So, travelling back to the end of my last post on my home media setup – how do we sort the files we’ve downloaded to keep them in an OCD-friendly manner? The answer – SortTV. This program is one that I wish I had found years ago, it does a fantastic job of looking up program metadata on various sites (IMDB and TheTVDB I believe) and then moving files into sub-folders per-programme and -series, or per-movie, and also downloading artwork for XBMC and the like to show. I’m not going to go into much more detail on how I set this up, there are great instructions available on configuring it for your system. One recommendation I will make is to use something like trickle to limit the bandwidth available for moving when you’re using NFS, otherwise it can grind things to a halt if moving several large files. I ended up with the following in my crontab:

30 * * * * trickle -s -d 100 -u 500 perl /path/to/sorttv/sorttv.pl

So that’s it! I think I’ve finally reached the end of this mini-series on how my home AV setup works, more than 2 years after I first set out writing it. I’m sure I’ll be back on the subject at some point soon though, something is bound to break…

Dishing up videos

Posted on by

Continuing my line of posts about my home media setup, I come to the main reason for putting myself through all this pain – being able to watch my selection of movies and TV shows with (next to) zero effort required.

I’ve posted before about the merits of XBMC, an open-source project to create a digital media hub that is accessible to all. I’ve not had anything better recommended since, so I’m still using this as my main 10-foot UI on my new TV, running off the Mac Mini I bought 18 months ago to start doing some iPhone development work. I didn’t really appreciate the iOS SDK, but the Mac Mini makes a fantastic media centre PC! The box is pretty much silent, the Apple Remote is pretty intuitive for getting around XBMC and similar interfaces, and they’re available on eBay for pretty reasonable prices. I may get fed up of Apple one day and revert to a custom built media box, but for now, the Mini will do!

I’ll post at some point in the future about how I actually store my files (it’s all about to change anyway, I’m running out of space!) but the stuff I’m using for downloading and sorting downloads is pretty neat if I do say so myself! A caveat at this point – I’m not publicly condoning the below, I obey the law when it comes to content – so should you!

First up – iPlayer. This part is only applicable in the UK (currently), so feel free to skip this paragraph if you’re viewing from overseas. There is a fantastic package available called get_iplayer that allows you to set a list of desired programmes, and then run a cron script to download them. For instance, the following will install the package, and add Have I Got News For You to your PVR.

$ sudo apt-get install get-iplayer
$ get-iplayer --pvradd "HIGNFY" "Have I Got News For You"

You’ll then need to add get-iplayer to your crontab (using crontab -e or similar) and fire it off every so often:

0 * * * * /usr/bin/get_iplayer --pvr --modes flashhd,flashvhigh,flashhigh --output /home/matt/DownloadComplete/

Will schedule get-iplayer to run every hour, on the hour, and attempt to download your content in HD, then descend through quality if necessary, and output the files to /home/matt/DownloadComplete/ (change as appropriate).

Update 11/02/12 – Since writing this, I’ve updated my get-iplayer cron job to be the following, which is a bit more useful when it comes to sorting – more on that in a future post!

15,45 * * * * /usr/bin/get_iplayer --pvr --vmode=flashhd,flashvhigh,flashhigh --output /home/matt/Download/ --command='mv "<filename>" "/home/matt/DownloadComplete/<fileprefix>.<ext>"' --file-prefix="<nameshort> <senum>" --whitespace

Next up, fetching non-UK TV shows. There is a fantastic website called ShowRSS that lets you build custom RSS feeds of popular (and unpopular, they cover 267 shows at the time of writing) TV show torrents, which can then be used by most good torrent software to automagically add torrents as they become available. One very good way of achieving this downloading of torrents is Flexget (again, available in Ubuntu and other systems with a simple apt-get install flexget), which is a package for parsing the data returned from feeds such as ShowRSS, and performing the necessary logic to download the actual .torrent file. Once again, we need to add Flexget to our crontab, set to run hourly (in a slightly fancier way!) 

@hourly /usr/local/bin/flexget --cron

Flexget uses a configuration file to determine what to do each time it’s run, which can be found in ~/.flexget/config.yml. As you may notice, it uses YAML (pronounced like camel) to store the preferences. YAML confused the hell out of me initially, but it’s all in the spacing! An example of my (truncated) config.yml is below.

feeds:
  tv-shows:
    rss: http://showrss.karmorra.info/rss.php?user_id=[REMOVED]&hd=null&proper=null
    series:
      - ncis
      - ncis:la
      - family guy:
          min_quality: hdtv
      - futurama
    download: ~/watch/

As you can see, this uses the tv-shows rss plugin to get my ShowRSS feed, then scans for the series listed (I believe you can do a catch-all rather than repeating the list), and then downloads the files into ~/watch/ – this will be useful in a minute. You can view Flexgets status on individual series using the following:

$ flexget --series
 Name                          Latest              Status               
-------------------------------------------------------------------------------
 Ncis                          S09E11 - 14d 20h    *hdtv                
 Ncis:La                       S03E11 - 14d 20h    *hdtv                
 Futurama                      S06E26 - 109d 12h   *hdtv                        
 Family Guy                    S10E09 - 16d 21h    *hdtv                             
-------------------------------------------------------------------------------

Amazing! Now we’ve got our files downloaded and going into ~/watch/ automatically. Now, to actually download them…

For my torrenting needs, I use rTorrent (again, packages available for Ubuntu in the repositories), which runs quite nicely in a detached screen downloading my files for me! rTorrent uses a file called ~/.rtorrent.rc for its settings, an example exert from mine:

schedule = watch_directory,5,5,load_start=./watch/*.torrent
schedule = untied_directory,5,5,stop_untied=
directory = ./Download/
system.method.set_key = event.download.finished,move_complete,"execute=mv,-u,$d.get_base_path=,~/DownloadComplete/;d.set_directory=~/DownloadComplete/"

dht=on
dht_port=6881
peer_exchange=yes
port_range = 7000-7500

This makes rTorrent keep an eye on the ~/watch directory for any files with the .torrent extension, and will add them to the active downloads if a new one is discovered, and remove any that are deleted. All torrents are downloaded into ~/Download/, but moved to ~/DownloadComplete/ when finished, for reasons that will be evident soon. DHT is enabled, and rTorrent will choose a random port between 7000 and 7500 on load.

So that’s all of our media downloading (you can add music/movies manually, or do some other clever scripts – I have an IRC bot that I’ll blog about in future do it for me!), but how do we sort that off into the right places for XMBC to find? It would be terribly inelegant just to make XBMC look in ~/DownloadComplete for it’s files… Well, that’s a question for a future post!!

Dishing up music

Posted on by

I’ve been meaning for a long time to continue my line of posts about my home media set up. Anyone who vaguely knows me (or stalks this blog) will know that I recently moved down to London, into a flat I can (sort of) call my own, so for the first time I’ve been able to play about with AV equipment in my way!

In this first post, I’m going to talk a bit about how I’m currently using the set up to serve my music collection around the flat, to work, and to my phone for on-the-go listening. It’s worth noting at this point that everything here relies on an old machine running Ubuntu server that’s currently sat in my cupboard, a solution that won’t work for everyone…

I’ve mentioned previously that I was looking into using DAAP to get at my music. This ended pretty quickly when I realised the clients available for listening are fairly crap. The actual configuration is fiddly, and I was after a nice functional GUI to access my music through. DAAP failed at pretty much all of this, so I went in search of another solution. After trying many different software packages (commercial and otherwise), I struck upon Subsonic, which seemed at first glance to do everything I was after. Essentially, it requires a server (available for all platforms) with access to your music collection, and will then spit the content back out to its own web UI (accessible through a personal URL if you’re not like me and running some kind of dynamic DNS service) plus it has a number of native players available for different platforms (including mobile). Best of all – it’s free! They withhold some features until you’ve given a small donation, but it’s entirely usable as a free package!

The web UI is basic, but skinnable (I’m still working with the basic skin – works well), and gives you access at a glance to all your artists, albums and playlists (although I’ve not done much with playlists yet, that part does seem a bit tricky). This UI also lets you play with the ID3 tags on your music, including the grabbing of album artwork off the internet, so if you’re as OCD as me about music tagging, your needs are well catered for! Within the web application your play queue appears at the bottom, with the current playing song in a lovely flash player. From the main window you can add things onto the end of the play queue, or change the playing song entirely. It’s basic, but functional, and if you want anything more advanced there are a number of native players available for all operating systems.

What interested me most was the quality of the iOS native applications. I’ve tried all of the ones recommended on Subsonics App page, and a couple that are in the app store but not endorsed officially, but I have now struck upon iSub as a favourite. It allows the caching of songs on your phone for playback when no data signal is available (although you have to have played the song for it to be cached – I’d like a feature where you could pre-cache songs to the phone), plus it integrates nicely with the iOS player API (play/pause whilst locked etc).

One other massive bonus was my discovery of a XMBC plugin for Subsonic (XMBC is what I use to browse media on my TV, as I’ve mentioned previously, more in a future post). This lets me listen to songs through my TV in the flat without having to bother turning a computer on – a massive bonus, and much more sociable if you have company!

The only downside of Subsonic is the reliance on a stable internet connection to an always-on music server. There are commercially hosted solutions available, but since I already had a machine running 24/7, and have recently splashed out on Virgin Media’s 30Mb fibre connection, it made sense to run the package from home.

So there it is – a whole-hearted recommendation for Subsonic (they’re honestly not paying me for this!) and it’s associated other applications. Tune in soon for a peek into the world of my TV/Movie system!

Dishing up media

Posted on by

As my latest project for ‘what to do when incredibly bored in the evening’ is to find a way of creating a central server for all of my media (starting with music, the rest is pretty easy) and then being able to play it off any device, preferably with some kind of specialised app for iPhone that would let me download tracks and playlists for local playback. If anyone has undertaken a similar project, I’d love to hear about it!

So – the task list so far:

  • Stable running media server
  • Automatic feed-in of new content (from download server – will blog about that one some other time!)
  • Automatic tagging (ID3 etc) of new content (this may be a step too far!)
  • Playlist viewing/editing
  • Access over the network
  • Access over the internet
  • Access from the iPhone (including local playback)
  • Front end media player (using XBMC or similar) for video/picture

Sounds like another one of those pipe-dreams, but I’m off to a pretty good start it seems! I’ve struck upon using some sort of DAAP server (currently running forked-daapd, will probably end up with the original daapd as it seems a bit more stable) off a dedicated test machine (so I don’t bork up my other running server, I’ve got plenty of old machines lying around the house at the moment!) which can then be connected to anywhere off the internal network, and some simple port forwarding should sort that for the wider world.

The only gripe at the moment is the lack of playlist support – the DAAP protocol isn’t designed to have anything to do with playlist editing, and that’s a pretty major feature I want to have! I may end up with some kind of web interface to the server that’ll let me edit playlists directly – not ideal, but I guess it’ll do the trick.

I’m currently struggling with the iPhone access part – I’ve found a single DAAP client (Simple DAAP Client), but that doesn’t support any caching, and is very basic (but free). I may have to resort back to some kind of manual syncing to get the iPod functionality, which would be a shame, but not an insurmountable problem.

The automatic tagging is definitely a big problem – what I’m envisaging at the moment is a daemon running on the server that uses Musicbrainz or something similar to update ID3 tags on MP3s as they are added on to the server, however such a thing may be slightly beyond the realms of possibility at the moment unfortunately. One day maybe…

I’ll try and keep this post up to date with my progress, but it’s likely to be slow work as my entire ‘server farm’ is being uprooted and moved to London in a few weeks. Flat hunting later this week, better make sure it’s got plenty of cupboard space for all these machines!