Category Archives: linux

Unable to load libcec.so on Kodi

Posted on by

As part of a complete re-vamp of my home media setup (blog post coming soon on that!), I found myself trying to connect a Sony TV to an Intel NUC running Kodi v16 via one of PulseEight‘s brilliant USB-CEC Adapters. However, no matter what I tried (combinations of cables, all the settings on the TV, different ports on the NUC etc) I could not get Kodi to recognise the adapter. Despite the Kodi Wiki page on CEC suggesting that this should plug-and-play, I did not get any suggestion from either the TV or Kodi that the CEC dongle was being recognised.

After quite a lot of searching in the wrong place (at first I was convinced that my fresh install of Kodibuntu wouldn’t be the root cause), I found a couple of log lines that finally pointed me in the right direction:

DEBUG: Loading: libcec.so.3.0
ERROR: Unable to load libcec.so.3.0, reason: libcec.so.3.0: cannot open shared object file: No such file or directory

Turns out that Kodi is built against libcec.so.3.0, however the version in the repository (I believe that was bundled or at least suggested by Kodibuntu) is 3.1. Kodi should (theoretically) be built against libcec.so.3 (note the missing .0), which the appropriate symlinks would have taken care of, however it seems there are some changes between 3.0 and 3.1 that Kodi doesn’t like, so some simple switching of the links (I tried creating /usr/local/lib/libcec.so.3.0 -> /usr/local/lib/libcec.so.3.1, which in hindsight isn’t a great idea) caused various crashes that I wasn’t prepared to start debugging – so instead I used the suggestion in this forum thread:

$ apt-get install --reinstall libcec3=3.0.1-1~trusty

Which will put the old version of libcec on your machine. After a quick reboot, Kodi instantly recognised the CEC dongle as promised, and it has been working beautifully ever since!

Puppet-ing my home network

Posted on by

For just over 12 months, I’ve been using Puppet as a way of configuring the various machines I have dotted around my home network. The initial desire to move to an automated configuration management tool was to keep tabs on the various requirements across pieces of software I run, however I’ve now moved to the point where all deployments of new hardware and software (both my own, and third party) are all managed through Puppet, and life is much simpler (most of the time!). I’ve been meaning to write this post for a while, mainly as a “getting started” point for anyone who is thinking about embarking on a similar journey, hopefully this will help people avoid making some of the same mistakes I made along the way!

Firstly, a little about my network, to give some of the points context. Over the last 5 years, I’ve evolved from having a single desktop machine, to today having the same desktop, 2 microservers (one for mass storage, the other for backup, services and acting as the Puppetmaster), a Home Theatre PC, and 7 Raspberry Pis in various guises (an alarm clock, appliance monitors and TV encoder between them). All of these machines were originally configured manually, and re-imaged from scratch every time something went wrong, leading to endless pages of scribbled notes about various requirements and processes to get things working. Now, all of them pull a catalogue from a single Puppetmaster, which I update as requirements change. Adding a new host is a matter of installing a fresh operating system, installing Puppet, and adding a few lines to the Puppetmaster detailing what I want installed – the rest is taken care of!

I’ve ended up using Puppet for much more than I originally imagined, including:

  • SSH Keys
  • MySQL hosts & databases
  • Apache site configuration
  • Tomcat configuration (auto-deploying apps is on my to-do list)
  • Sendmail
  • Automated backups
  • Media sorting
  • ZNC configuration

Plus even more besides that! Most bits of my own software run through supervisord, which is also configured through Puppet.

The first, and arguably most important lesson I learned through my Puppet-isation process is to keep the configuration in some sort of versioning system. Initially I did not keep track of the changes I was making to Puppet manifests, and requirements that were added at 3am seemed completely illogical in the cold light of day – but by self-documenting the manifests as I went along, and tracking changes through my own Subversion repository, I can look back at the changes I’ve made over time, and save myself the hassle of accidentally removing something essential. I’ve lost track of the number of times I’ve had to revert back to a previous configuration, and having a system in place to do this automatically will definitely help you in the long run!

While writing your configuration, I’ve learnt to be explicit about your requirements. A lot of the modules and class definitions I wrote early on did not use any of the relationships you can define between attributes, and as a result I’d often be bitten by the Puppet parser attempting to configure things in an illogical order. For the sake of adding a few arrows into your manifests, it’s worth using these just to avoid the head-bashing when a configuration run fails intermittently! I still run into problems when configuring a new host, when a package I require isn’t installed in time to start up a service – using relationships from the ground up will hopefully avoid this ever becoming an issue.

Arguably the second most useful tool I installed (after Puppet itself) is the Puppet Dashboard. This fantastic tool pulls in the reports that Puppet generates, and spits them out into a very readable format which allows you to get straight to the heart of what’s causing failed runs, rather than having to resort to diving through the depths of the raw logs. It took me a while getting around to installing this, and I honestly regret not installing it straight away – it has saved me an incredible amount of time since. A word of warning though – the logs can take up an awful lot of space (my dashboard database is 400MB+ with only the last 3 months data stored) – make sure your SQL server is up to the task before starting this!

While installing things like the Puppet Dashboard, heira and the like, I’ve taken to Puppet-ing your Puppet, by which I mean writing the configurations for these tools into your Puppet manifests. Initially this seemed very strange to me, I didn’t want some almost-sentient piece of software configuring itself on my machine! However, the client will quite happily run on the same machine as the server to configure itself (in a strange cyclical dogfooding loop). There are plenty of third-party modules available that will do a lot of this for you, but I ended up writing my own for several things – as long as the installation is managed somehow, you’re going to have a much better time!

Linked to that, and my last tip that I’ll post here, is to write your own modules wherever you can. For the first few months after starting using Puppet, I tried to cram everything imaginable into the main manifest directory, and barely used templates and source files at all. Subsequently I’ve learned to separate the concerns wherever possible, which leads to much cleaner code in the module definitions, and much nicer looking node definitions as well. Third-party modules have been a mixed blessing for me, some of them have been coded extremely well to allow multi-platform support, but some have been hard-coded to alien platforms making them useless to me. I’d absolutely recommend going looking for someone else who’s done the hard work first, but don’t be afraid to roll up your sleeves and write a module yourself. Maybe even post it online for other people to enjoy! (I’m very hypocritical making that last point, none of my modules have made it online as they’re all terrible!)

I hope that some of the above might help steer someone in the right direction when embarking on a journey with Puppet. Obviously I’d recommend some more official reading before converting your professional network of 10,000+ machines across, but if you’re like me and need a comparatively small network taming, then take the plunge and get started!

PCTV tripleStick 292e with TVHeadend

Posted on by

I’ve blogged before about my home AV set up, but something I’ve not talked about is the recent addition of a couple of TV tuners so that I can watch and record live Freeview channels. Until recently I’d been using TVHeadend version 3.2 on a Raspberry Pi, with a PCTV nanoStick T2 that worked out of the box on Raspbian for me. However, the time came when I wanted to be able to record and/or view multiple channels at once, so I set about getting a second tuner to be able to do this. Through a lack of attention paid while ordering, I ended up with a PCTV tripleStick rather than a second nanoStick, and this one sadly was not as easy to set up. I bounced around a lot of forums and blog posts in getting mine working, so I thought I’d consolidate my learnings here, in the hope that someone else may find this useful!

First off, the chipset on the tripleStick (Silicon Labs Si2168) is different to the nanoStick (Sony CXD2820R), hence the incompatibility with the old drivers. There’s a very detailed teardown and comments at Antti’s LinuxTV Blog which does a great job of explaining what’s under the hood, and the comments do offer some useful guidance (but also some misdirection!). I was previously running an older version of Raspbian (kernel 3.12 if I recall correctly), which failed to recognise the tripleStick as a DVB tuner at all, but several sources suggested that firmware was included in 3.16 and higher. I updated my Raspberry Pi with the usual apt-get update; apt-get upgrade; apt-get dist-upgrade to move up to a newer kernel version (3.18) which did get the dongle recognised in TVHeadend, however it appeared to not get any signal, despite being plugged in to the same aerial as the working nanoStick.

At this point I attempted upgrading to TVHeadend 4.0, something I should have done a considerable time ago anyway, however this had no effect and the dongle continued to show no signal through TVHeadend. Checking my logs, I found that my /var/log/syslog had repeated entries referring to “found a 'Silicon Labs Si2168' in cold state“, and claiming that firmware files had not been found. Many different message boards carried many different links to firmware, and suggesting different combinations that needed to be installed, several of which I found to be corrupt, however the one that worked for me was installed using the following:

$ wget http://palosaari.fi/linux/v4l-dvb/firmware/Si2168/dvb-demod-si2168-02.fw -O /lib/firmware/dvb-demod-si2168-02.fw

There are many suggestions that the file dvb-demod-si2168-b40-01.fw is also needed from that same source, however it seems to be working fine for me without this present. I’ve seen some reports that the tuner should appear as two separate entries in TVHeadend (one as a DVB-T tuner, and another as a a DVB-S), however since I’m only using DVB-T I’ve not seen any problems – your mileage may vary!

Blinkytape

Posted on by

Yet another one of my Kickstarter jaunts turned up just before Christmas – the Blinkytape by BlinkinLabs. Essentially, this product is a strip of 60 LEDs connected to a USB interface, which allows you to address each “pixel” individually through a little bit of coding so you can build up your own programmable lighting show! So far I’ve only had chance to use this as a very nerdy alternative to Christmas lighting, and more generally expanding my knowledge of Python, but I’ve got big plans for it in future!

First up – getting started. I decided to use this in conjunction with a Raspberry Pi I had going spare from another project, as it gives me network connectivity and a platform to write and run Python scripts on. Conveniently, no powered external USB hub is required to run the Blinkytape off a Pi (as I had no other peripherals plugged in, your mileage may vary!), so it was just a case of plugging it in and installing the necessary Python libraries:

$ sudo apt-get install python-pip
$ sudo pip install pyserial

There is an official Blinkytape python library available from their GitHub repository (along with some other languages), however at the time when I was playing with this (before Christmas) their base class was lacking a lot of features – so I wrote my own! To get my integration script, run the following:

$ svn co http://projects.mattdyson.org/projects/blinkytape blinkytape

This will give you the main class (BlinkyTapeV2.py) and a couple of example files, all of which are commented in a (hopefully!) helpful manner to show what’s going on. The following video shows an example of the BouncingBlocks.py class in action (by running sudo python BouncingBlocks.py) followed by a more ‘festive’ example, something I knocked together very quickly to cycle through a series of effects in very Christmas-y red and green colours!

Overall, I’m very impressed by the quality of this product. I was expecting something very rough-and-ready, being a rather specialist product marketed through Kickstarter – however the LEDs themselves are very bright, and nicely packaged up in a plastic flexible strip in order to protect the circuitry. The ease with which I managed to write my own integration library is also a testament to how simple the electronic design of this product is.

So what am I planning on using this for? First up, I’m looking at building my own alarm clock that reads from Google Calendar to only wake me up when I need to be up – normal alarms don’t seem to have been built with shift work in mind! I’m hoping to integrate the Blinkytape into this project by creating an ambient light that gradually fades up after the alarm has gone off, hopefully easing the transition into daylight hours! There are also plenty of projects I was hoping to do with a Moore’sCloud Light, another Kickstarter project that sadly failed to meet their funding goals, but hopefully Blinkytape will fill the void! I’ll make sure to post back here with further updates when my Blinkytape gets put to use!

Managing music with beets

Posted on by

In a previous post, I talked about how I use Subsonic in order to make my entire music collection available over the internet to either my phone, or any computer via its web-interface. I’m still using Subsonic to achieve this, but had one fairly major gripe left with the set-up – I was managing the library on disk manually. Subsonic is great for editing ID3 tags on individual songs, but it relies on files being in sensible per-album folders in order to populate its library, something which I very quickly got fed up of doing manually.

I’ve known for some time about the Musicbrainz project, which maintains a database of all music releases, and has a number of applications built on top of it which will scan, tag and move your music collection as desired. I recall using Musicbrainz Picard back in the day to sort my library before I moved to hosting music on my own server, but never found anything similar that I could run on my Ubuntu server, until now. Enter beets.

Beets is a program that will manage your entire music library, allowing command line access, and interfaces directly with the Musicbrainz API to tag tracks appropriately. Plugins for beets also allow you to update genres according to Last.fm, download cover art (which Subsonic will quite nicely pick up!), and even acoustically fingerprint unknown files to figure out what they are!

Installing Beets was as simple as following the instructions on their getting started guide, however importing my existing music proved a little more tricky, and I had a few false-starts at doing this. I eventually found that the easiest way to do this was to move my existing collection into a separate folder, and set up beets to sort my collection back into the original place. The ~/.config/beets/config.yaml file I ended up using looks like this:

directory: /media/music
library: /media/backup/beets/musiclibrary.blb
import:
    write: yes
    move: yes
    resume: yes
replace:
    '[\\/]': _
    '^\.': _
    '[\x00-\x1f]': _
    '[<>:"\?\*\|]': _
    '\.$': _
    '\s+$': ''
art_filename: cover
plugins: fetchart embedart lastgenre

Put simply, my music lives in /media/music (an NFS share), with my library file on a separate backup share. When importing files, I want them writing (moving) to their new location, so all I need to do is run beet import New\ Album/ and the files will be tagged and moved into place (with most dodgy characters removed – unfortunately some special characters still seem to slip through). Album art is also downloaded into the new folder as cover.jpg, embedded into the files themselves, and also the genre field is populated using Last.fm. A nightly scan configured in Subsonic picks up the new files and adds them into the library, making them available to listen!

The next step for me is to integrate importing into the same process as my automatic sorting of TV shows, as I currently still need to manually import newly downloaded tracks. However, even this is a massive improvement on the tedious processed previously needed for getting OCD-quality tags on new music!!

Using 20×4 RGB LCD over i2c with a Raspberry Pi

Posted on by

Now there’s a specialist blog post title if ever there were one…

Recently, I’ve been dabbling with electronics to fill the void of spare time I’ve found myself with while I’m between jobs. I’m currently working on a half-baked idea to create some sort of digital assistant who will take instructions in some form, and then read stuff back to me in a Siri-esque manner. Nothing sounds more awesome than having twitter @replies read out to you, right?! To kick off this project, and get me motivated to actually do something, I ordered a boatload of parts from Adafruit, and set about learning how to use them. First challenge – connecting up their 20×4 RGB backlight negative LCD screen to my Raspberry Pi.

In order to assist with this, I also bought the i2c / SPI character LCD backpack in order to save some GPIO pins for other uses. Due to my lack of attention while ordering, I failed to notice that the LCD backpack only has 16 pins, whereas the LCD screen I ordered has 18 (2 more for the extra background LEDs). Rather than giving up and being limited to only a single channel of control for the backlights, I decided to connect pins 14 to 18 direct into the Pi, and mash two separate libraries together to give myself full control. This is what I ended up with (click for big):

2013-05-02 21.36.11

Now, that looks like an absolute mess. That’s because it is. In an attempt to make that a bit more readable, here’s a Fritzing diagram of how it’s wired (again, click for big).

lcd_test_bb

Now, that’s even more confusing as I couldn’t find a Fritzing library with the right parts – so I’ve fudged a few things. Imagine there are ports 17 and 18 on the LCD, and that the LCD itself is 20×4 rather than 16×2. Secondly, imagine the chip in the middle is actually the i2c backpack mentioned above, so everything on the bottom is connected straight to ports 1 to 16 on the LCD, and the VCC/GND/CLK/DAT are connected to the Pi. So, in terms of wiring we get:

  • LCD #1 to #14 -> i2c backpack #1 to #14
  • LCD #15 -> 5V0
  • LCD #16 -> Raspberry Pi GPIO 17
  • LCD #17 -> Raspberry Pi GPIO 27
  • LCD #18 -> Raspberry Pi GPIO 22
  • i2c backpack GND -> GND
  • i2c backpack VCC -> 5V0
  • i2c backpack CLK -> Raspberry Pi SCL
  • i2c backpack DAT -> Raspberry Pi SDA

Now that’s all set up, you can use the standard AdafruitLcd Python library (nice adaptation that I used can be found here) to control the text shown on screen, but we need something bespoke for our background lighting. For future projects, I wanted the ability to control each colour individually, so I can set arbitrary RGB values on the screen, and also brighten/dim appropriately. The latest version of RPi.GPIO will let you do software Pulse Width Modulation, which will achieve this quite nicely for us. To install the latest version (0.5.2a at the time of writing), you’ll need to run the following on your Pi (as root):

$ wget https://pypi.python.org/packages/source/R/RPi.GPIO/RPi.GPIO-0.5.2a.tar.gz
$ tar xf RPi.GPIO-0.5.2a.tar.gz
$ cd RPi.GPIO-0.5.2a
$ python setup.py install

So, combining some standard example code for PWM on the Pi with the AdafruitLcd library, I developed my own little library for controlling a LCD wired up in this manner. To get up and running with the code I wrote, you will need (again, as root):

$ mkdir lcdtest
$ cd lcdtest
$ svn co http://projects.mattdyson.org/projects/LCDControl@889 .
$ git clone https://github.com/PDKK/RpiLcdBackpack.git
$ touch RpiLcdBackpack/__init__.py
$ python testLCD.py

Note: If you see IOError: [Errno 5] Input/output error when running testLCD.py, you may need to edit RpiLcdBackpack/RpiLcdBackpack.py and change the line self.__bus=smbus.SMBus(0) to self.__bus=smbus.SMBus(1). This should only happen on newer versions of the Pi, where the i2c bus number changed to 1 from 0.

Note 2 (added 15/10/14): The version of my LCDControl library that you’re checking out with the above command is now out-dated, I’ve updated the library to use pigpio instead of RPi.GPIO, as the latter was causing me flickering problems when the Pi was under load. To get the latest version, remove the @889 from the svn co command, you will need to have pigpio installed and running for this to work.

Once you run testLCD.py, you should see the screen flash a series of colours, followed by some messages appearing on the screen. Yaaay – it works!

The LCDControl class I’ve written is pretty basic (I’m still learning Python… slowly!) but allows you to set RGB or individual colour values for the backlights, and also pass in any message without worrying about formatting. Currently (version 1.0 at the time of writing), the LCDControl.setMessage method will split by the newline character (\n) and do the logic regarding line numbers for you (as the third display line on the LCD is actually carried over from the first line passed to the controller, and the fourth with the second) – future iterations of this code will allow you to do things such as full text wrapping, and scrolling text.

So there we have it – a 20×4 RGB LCD screen talking to a Raspberry Pi over i2c, retaining individual control over the background LEDs. As always, please leave a comment if you spot anything wrong with what I’ve written here, or have any feedback/suggestions/requests!

Monitoring MDADM RAID arrays with Cacti & SNMP

Posted on by

As I’ve mentioned in a previous post, I use MDADM RAID arrays on Ubuntu for storing all of my media, plus a separate array elsewhere for backups. I also use Cacti to keep an eye on my different machines, combined with the Threshold plugin to allow monitoring of anything going wrong within the network. While MDADM does allow you to use the monitoring feature to email when a disk gets dropped, I was looking for something that would hook into my current infrastructure, and was surprised to find that no templates existed to allow graphing MDADM details in Cacti – so I set about making some! This post explains how to set this up on your own machine.

Rather than using a local script, I have used SNMP to allow for this being added on multiple machines and monitored remotely. So, our first step is to get the information into our SNMP configuration. This is accomplished using two scripts (as MDADM requires root to run; we’ll see how this is achieved shortly) – in order to get them, you need to run (as root):

$ cd /etc/snmp/
$ svn co http://projects.mattdyson.org/projects/cacti/mdadm/scripts/ mdadm
$ cd mdadm
$ chmod -R 0777 *

This will download 2 files – mdadm-monitor and mdadm-read. The first one is the “business end” of the operation, this will take various arguments (as you’ll see in the snmpd.conf shortly) and output an indexed list representing the value for each array in our system. The mdadm-read script simply calls mdadm --detail /dev/$1, so we can add this to /etc/sudoers to allow us to access this information without requiring a root password each time – add the following line into /etc/sudoers in order to make this happen. You may need to change the user (“snmp” in this example) to whatever snmpd runs under on your system

snmp    ALL=(ALL) NOPASSWD: /etc/snmp/mdadm/mdadm-read

At this point, you should be able to run /etc/snmp/mdadm/mdadm-monitor Name as your SNMP user and see a list of all MDADM arrays on your system!

In order to access these through SNMP, we need to edit /etc/snmp/snmpd.conf and add the following lines:

extend .1.3.6.1.4.1.2021.80 Num /etc/snmp/mdadm/mdadm-monitor Num
extend .1.3.6.1.4.1.2021.81 Index /etc/snmp/mdadm/mdadm-monitor Index
extend .1.3.6.1.4.1.2021.82 Name /etc/snmp/mdadm/mdadm-monitor Name
extend .1.3.6.1.4.1.2021.83 Active /etc/snmp/mdadm/mdadm-monitor Active
extend .1.3.6.1.4.1.2021.84 Working /etc/snmp/mdadm/mdadm-monitor Working
extend .1.3.6.1.4.1.2021.85 Failed /etc/snmp/mdadm/mdadm-monitor Failed
extend .1.3.6.1.4.1.2021.86 Spare /etc/snmp/mdadm/mdadm-monitor Spare
extend .1.3.6.1.4.1.2021.87 Degraded /etc/snmp/mdadm/mdadm-monitor Degraded

Once this is done, you’ll need to restart SNMP with /etc/init.d/snmpd restart, and then you should be able to run snmpwalk -v1 -cpublic localhost .1.3.6.1.4.1.2021.82 (if you have snmpwalk installed) and see some output similar to this:

iso.3.6.1.4.1.2021.82.1.0 = INTEGER: 1
iso.3.6.1.4.1.2021.82.2.1.2.4.78.97.109.101 = STRING: "/etc/snmp/mdadm/mdadm-monitor"
iso.3.6.1.4.1.2021.82.2.1.3.4.78.97.109.101 = STRING: "Name"
iso.3.6.1.4.1.2021.82.2.1.4.4.78.97.109.101 = ""
iso.3.6.1.4.1.2021.82.2.1.5.4.78.97.109.101 = INTEGER: 5
iso.3.6.1.4.1.2021.82.2.1.6.4.78.97.109.101 = INTEGER: 1
iso.3.6.1.4.1.2021.82.2.1.7.4.78.97.109.101 = INTEGER: 1
iso.3.6.1.4.1.2021.82.2.1.20.4.78.97.109.101 = INTEGER: 4
iso.3.6.1.4.1.2021.82.2.1.21.4.78.97.109.101 = INTEGER: 1
iso.3.6.1.4.1.2021.82.3.1.1.4.78.97.109.101 = STRING: "md0"
iso.3.6.1.4.1.2021.82.3.1.2.4.78.97.109.101 = STRING: "md0"
iso.3.6.1.4.1.2021.82.3.1.3.4.78.97.109.101 = INTEGER: 1
iso.3.6.1.4.1.2021.82.3.1.4.4.78.97.109.101 = INTEGER: 0
iso.3.6.1.4.1.2021.82.4.1.2.4.78.97.109.101.1 = STRING: "md0"

Excellent! We’re now up and running with details of all MDADM arrays being output to SNMP, including numbers of active, working, failed and spare disks for each array! Next up, lets add the SNMP query XML into our Cacti resource directory (substitute the relevant Cacti path for your system). The code I’ve written works fine on net-snmp 5.4.3, but I believe lower versions may not play nicely with using the long OIDs in net-snmp_mdadm.xml – you may need to remove everything in the OID after .1.3.6.1.4.1.2021.8<X>.4

$ cd /usr/share/cacti/site/resource/snmp_queries/
$ svn co http://projects.mattdyson.org/projects/cacti/mdadm/queries/ mdadm

This is referenced by the Cacti Data Template that I have put together, which can be downloaded here. Once you’ve used the “Import Templates” feature to import this file, add the SNMP – Get MDADM Arrays template to a host, and create the relevant graphs for each of your arrays. Here’s one of mine that’s a little boring, but at least we know everything is okay!

saturn_md0_1hr

There we have it – MDADM data available in Cacti! I’ve also created a Threshold template that monitors the Failed data source for any value other than 0, and then kicks off the usual alerting to let me know to replace a disk, but Cacti won’t let me export that one easily, and it was simple enough.

I haven’t fully tested this method with multiple arrays on the same host yet, please let me know in the comments if you find any problems, and I’ll do my best to help out.

Update 08/05/13: I’ve updated the graph template to make a bit more sense when an array is rebuilding.

Update 05/11/13: The graph template should now parse properly – thanks to Evandro Nabor for spotting the error!

Update 28/12/13: Doh! My ‘fix’ for the graph template above introduced another problem! I’ve updated the file again, and hopefully things should now be working – thanks to C. Alexander for spotting this!

Media storage & sorting

Posted on by

It’s been a long time (over a year in fact) since I last blogged about my home AV setup, and I left the last post on a real cliffhanger – how do I sort and store all these videos? Well, wait in suspense no longer, because here’s the answer!

Firstly, a bit about the storage – for a while, I have crammed more and more hard-drives into a single old desktop machine that I re-purposed to be my media centre/SVN repository/Tomcat/Apache/seedbox/etc, but it was getting to the point where I had files spread across 3 different drives, and no redundancy in the event of drive failure. Given I’ve had more than a few drives die on me recently, some sort of recovery strategy was becoming very necessary. We’re not talking small size requirements here either – somehow I’ve managed to accrue over 4TB of videos, music, pictures and other files. With these requirements in mind, I went shopping for some sort of Network Attached Storage style of device, but with more of a DIY edge to it. Having looked closely at Drobo and the like, I decided I’d rather have full control of the OS and RAID setup; the idea of not knowing what’s going on under-the-hood, and not being able to recover the data apart from on another Drobo, was not appealing at all. I ended up getting a HP ProLiant N40L Microserver, sadly well after the £100 rebate ended, but still at a very good price for a decent specification of machine and a nicely engineered case. I moved the 250GB system drive to the optical drive bay as documented here, in went three 3TB hard drives, on went a fresh copy of Ubuntu 12.04.2 server, and I was away. The fourth drive port on the ProLiant is left empty for now, with a view to putting in another 3TB drive when I eventually run out of space again.

I’ve heard interesting things about ZFS recently, but decided to stick with what I know, and set up a RAID5 array across the three disks, giving me 6TB usable space, and allowing for the failure of any single disk. This arrayis then shared over NFS so that my other server can access to add files, and my Mac Mini (running XBMC) can access the content to display on the TV.

So, travelling back to the end of my last post on my home media setup – how do we sort the files we’ve downloaded to keep them in an OCD-friendly manner? The answer – SortTV. This program is one that I wish I had found years ago, it does a fantastic job of looking up program metadata on various sites (IMDB and TheTVDB I believe) and then moving files into sub-folders per-programme and -series, or per-movie, and also downloading artwork for XBMC and the like to show. I’m not going to go into much more detail on how I set this up, there are great instructions available on configuring it for your system. One recommendation I will make is to use something like trickle to limit the bandwidth available for moving when you’re using NFS, otherwise it can grind things to a halt if moving several large files. I ended up with the following in my crontab:

30 * * * * trickle -s -d 100 -u 500 perl /path/to/sorttv/sorttv.pl

So that’s it! I think I’ve finally reached the end of this mini-series on how my home AV setup works, more than 2 years after I first set out writing it. I’m sure I’ll be back on the subject at some point soon though, something is bound to break…

Using an Xbox 360 Wireless Controller with Raspberry Pi

Posted on by

As part of a project I’m working on at the moment (more information to come soon… more information here) I’ve been attempting to get my Xbox 360 Wireless Controller for Windows talking to my Raspberry Pi. Having spent a fair amount of time chasing various options around the internet, I thought I would share my eventual (and rather simple) solution here.

The first thing I found was PyGame – a python library that offers support for joysticks and gamepads, but primarily designed for game development. This post suggested that PyGame may solve the connectivity problems, and gives some example code for echoing out events, however I could not get this to work.

The Ubuntu wiki suggested a module called xpad, which is included by default on Ubuntu, but not on the Rasbian image I am using (Rasbian “wheezy”), although it is available through apt-get in the default repository. Unfortunately, this didn’t work for me either.

The eventual solution that worked for me came up in a blog post by Zephod about using an Xbox controller to run a remote control car, which suggested using Xboxdrv – a userspace driver for the Xbox controllers in Linux. There were suggestions on the Raspberry Pi forums that this would require building, but a simple `apt-get install xboxdrv` on the Pi worked for me. Once installed, execute the program (as root), and then re-sync the Xbox controller – this had me stumped for quite a while, and seemingly only needs to be done the first time you attempt to use the controller since the module was loaded. A re-sync for the wireless controller means holding the button on the reciever for ~3 seconds until it starts to flash, and then holding the button on top of the controller (to the right of where ‘Microsoft’ is written) until the lights flash. Once this has completed, you should see a new line in stdout for every event that happens on the controller – so press a button and see what happens!

This is the output I saw when starting up xboxdrv (having already done the sync) and pressing the “A” button on my controller – notice the A:1 changing to A:0 as I release the button (about 3/4 of the way across the terminal). Success! (Click for a larger image)

Zephod has written a small Python class to read from the output of xboxdrv and allow it to be read in a more usable format – I’ve not yet had chance to fully digest what it does and how it works (this is an exercise in me learning Python as well!), but it looks very promising, and I’m looking forward to continuing with my project!

Update 05/01/2013
I’ve had chance to play around with the xbox_read.py file I linked above – and it works a treat! I’ve set up a quick test python script that you can use to print out events. In order to use it, do:

$ git clone https://github.com/zephod/lego-pi.git legopi
$ touch legopi/__init__.py
$ wget http://projects.mattdyson.org/projects/robotarm/readEventTest.py
$ sudo python readEventTest.py

You may have to re-sync your controller as described above, and then once you start to move sticks/press buttons you should see a single line for each event! Magic!

Dishing up videos

Posted on by

Continuing my line of posts about my home media setup, I come to the main reason for putting myself through all this pain – being able to watch my selection of movies and TV shows with (next to) zero effort required.

I’ve posted before about the merits of XBMC, an open-source project to create a digital media hub that is accessible to all. I’ve not had anything better recommended since, so I’m still using this as my main 10-foot UI on my new TV, running off the Mac Mini I bought 18 months ago to start doing some iPhone development work. I didn’t really appreciate the iOS SDK, but the Mac Mini makes a fantastic media centre PC! The box is pretty much silent, the Apple Remote is pretty intuitive for getting around XBMC and similar interfaces, and they’re available on eBay for pretty reasonable prices. I may get fed up of Apple one day and revert to a custom built media box, but for now, the Mini will do!

I’ll post at some point in the future about how I actually store my files (it’s all about to change anyway, I’m running out of space!) but the stuff I’m using for downloading and sorting downloads is pretty neat if I do say so myself! A caveat at this point – I’m not publicly condoning the below, I obey the law when it comes to content – so should you!

First up – iPlayer. This part is only applicable in the UK (currently), so feel free to skip this paragraph if you’re viewing from overseas. There is a fantastic package available called get_iplayer that allows you to set a list of desired programmes, and then run a cron script to download them. For instance, the following will install the package, and add Have I Got News For You to your PVR.

$ sudo apt-get install get-iplayer
$ get-iplayer --pvradd "HIGNFY" "Have I Got News For You"

You’ll then need to add get-iplayer to your crontab (using crontab -e or similar) and fire it off every so often:

0 * * * * /usr/bin/get_iplayer --pvr --modes flashhd,flashvhigh,flashhigh --output /home/matt/DownloadComplete/

Will schedule get-iplayer to run every hour, on the hour, and attempt to download your content in HD, then descend through quality if necessary, and output the files to /home/matt/DownloadComplete/ (change as appropriate).

Update 11/02/12 – Since writing this, I’ve updated my get-iplayer cron job to be the following, which is a bit more useful when it comes to sorting – more on that in a future post!

15,45 * * * * /usr/bin/get_iplayer --pvr --vmode=flashhd,flashvhigh,flashhigh --output /home/matt/Download/ --command='mv "<filename>" "/home/matt/DownloadComplete/<fileprefix>.<ext>"' --file-prefix="<nameshort> <senum>" --whitespace

Next up, fetching non-UK TV shows. There is a fantastic website called ShowRSS that lets you build custom RSS feeds of popular (and unpopular, they cover 267 shows at the time of writing) TV show torrents, which can then be used by most good torrent software to automagically add torrents as they become available. One very good way of achieving this downloading of torrents is Flexget (again, available in Ubuntu and other systems with a simple apt-get install flexget), which is a package for parsing the data returned from feeds such as ShowRSS, and performing the necessary logic to download the actual .torrent file. Once again, we need to add Flexget to our crontab, set to run hourly (in a slightly fancier way!) 

@hourly /usr/local/bin/flexget --cron

Flexget uses a configuration file to determine what to do each time it’s run, which can be found in ~/.flexget/config.yml. As you may notice, it uses YAML (pronounced like camel) to store the preferences. YAML confused the hell out of me initially, but it’s all in the spacing! An example of my (truncated) config.yml is below.

feeds:
  tv-shows:
    rss: http://showrss.karmorra.info/rss.php?user_id=[REMOVED]&hd=null&proper=null
    series:
      - ncis
      - ncis:la
      - family guy:
          min_quality: hdtv
      - futurama
    download: ~/watch/

As you can see, this uses the tv-shows rss plugin to get my ShowRSS feed, then scans for the series listed (I believe you can do a catch-all rather than repeating the list), and then downloads the files into ~/watch/ – this will be useful in a minute. You can view Flexgets status on individual series using the following:

$ flexget --series
 Name                          Latest              Status               
-------------------------------------------------------------------------------
 Ncis                          S09E11 - 14d 20h    *hdtv                
 Ncis:La                       S03E11 - 14d 20h    *hdtv                
 Futurama                      S06E26 - 109d 12h   *hdtv                        
 Family Guy                    S10E09 - 16d 21h    *hdtv                             
-------------------------------------------------------------------------------

Amazing! Now we’ve got our files downloaded and going into ~/watch/ automatically. Now, to actually download them…

For my torrenting needs, I use rTorrent (again, packages available for Ubuntu in the repositories), which runs quite nicely in a detached screen downloading my files for me! rTorrent uses a file called ~/.rtorrent.rc for its settings, an example exert from mine:

schedule = watch_directory,5,5,load_start=./watch/*.torrent
schedule = untied_directory,5,5,stop_untied=
directory = ./Download/
system.method.set_key = event.download.finished,move_complete,"execute=mv,-u,$d.get_base_path=,~/DownloadComplete/;d.set_directory=~/DownloadComplete/"

dht=on
dht_port=6881
peer_exchange=yes
port_range = 7000-7500

This makes rTorrent keep an eye on the ~/watch directory for any files with the .torrent extension, and will add them to the active downloads if a new one is discovered, and remove any that are deleted. All torrents are downloaded into ~/Download/, but moved to ~/DownloadComplete/ when finished, for reasons that will be evident soon. DHT is enabled, and rTorrent will choose a random port between 7000 and 7500 on load.

So that’s all of our media downloading (you can add music/movies manually, or do some other clever scripts – I have an IRC bot that I’ll blog about in future do it for me!), but how do we sort that off into the right places for XMBC to find? It would be terribly inelegant just to make XBMC look in ~/DownloadComplete for it’s files… Well, that’s a question for a future post!!