My latest task at work was to setup a server running virtual box to allow all of the developers access to all test machines and to allow us to keep the virtual machines up to date (by updating one set of appliance rather than multiple sets).

I found an excellent tutorial on how to do that here.

After you follow the tutorial to the letter and have your Headless VirtualBox and phpVirtualBox running, you’ll need to add your virtual machines. We already had several that we were actively using so I set out to import them into the new environment.

This was met with resounding failure.

The import feature was not working for me (Could not create the clone medium ‘/path/to/VirtualBox VMs/Disk1.vmdk’ (VERR_VD_VMDK_INVALID_FORMAT)). So I scraped the .ova files that I had exported and copied over the vmdk files – the hard drives of the virtual machines. Once these were added to the virtual box users home directory and chown’d accordingly, I created new machines in the phpVirtualBox interface. For each new Virtual Box, I selected already existing Hard Drives.

Some minor tweaks were required (like removing the CDROM’s) so they would boot, but after that, we were able to connect no problem.

There was an issue with the overall phpVirtualBox throwing errors after clicking on anything. It turned out to be we added (or came with the clean install) a vboxnet0 ethernet connection under File > Preferences > Network. Removing that solved the error (The object functionality is limited (0x80070005)).

So, now we have the following virtual environments and all are working extremely well and fast:

Windows XP – Internet Explorer 6
Windows 7 – Chrome (current)
Windows XP – Internet Explorer 7
Windows 7 – Internet Explorer 8
Windows 7 – Internet Explorer 9
Windows 8 – Internet Explorer 10 (To get this appliance check out this site)
Windows 7 – Firefox (latest)
Windows 7 – Opera (latest)

Sadly, MAC OS is not supported by Virtual Box, so we have to continue to use our MAC test machine over this new virtual test environment.

There is a Safari 5.1 for Windows so I may create another appliance for that.

Hope this helped.

After searching for a template for Munin that was a little easier to use than the standard, I came across the Raven template by Simian Uprising. This worked great for Munin 1.2.x but does not work for version 2.0.x. So, rather than use the standard template or fix Raven, I created a new template called Modern.

This theme has been testing in the Chrome browser only but should work in all the majors.

Update 2014-10-18: The Theme can now be found on Git Hub: https://github.com/sjoudry/modern-munin

Alright, the inaugural post!

Background

One of the most recent challenges at work was to implement a system to backup our internal servers to a local backup device and to a remote backup device. This covers my ass in the event of a fire at the office or a hurricane or whatever.

The existing backup was a 7 day rolling snapshot backup which was better than nothing. However the solution was not as elegant as what I am proposing here.

The inspiration for this solution comes from here where Mr. Rubel outlines how to create incremental backups using rsync/cp -al and also using rsync/–link-dest. For my purposes rsync/–link-dest was the answer.

The Solution (in Theory)

The solution is hard links. If I create a file called “unique” I have one physical file and a link to that file. If I then create another link to the same file called “not_unique” I still only have one physical file but there are now two links to it. If I remove one link, the file remains. If I remove all links, the file is destroyed. As usual, Wikipedia explains it well.

So, based on that theory, the idea is to create a complete snapshot backup once and only once. After that, incremental backups would be created using hard links.

• Any files that have been added since the last incremental backup would be created in the current increment.
• Any files that have been changed since the last increment would be re-created in the latest increment (which would preserve the state in previous backups).
• Any files that have been removed since the last increment would be removed from the current increment (which would again preserve the state in previous backups).

The only hit to the disk is for files that have changed. The links take space, but it’s minimal.

Demonstration

First, I create the initial backup. I’m using -i on the ls command to show the inode id of the file:

$ ls -lRi
.:
total 12
41451556 drwxr-xr-x 2 scott scott 4096 2012-11-06 22:28 2012.11.06/

./2012.11.06:
total 12
41423324 -rw-r--r-- 1 scott scott 93 2012-11-06 22:28 existing_file

Next, I create the first increment with a new file added. Make note of the inode id number that I have coloured to show the hard links:

$ ls -lRi
.:
total 16
41451556 drwxr-xr-x 2 scott scott 4096 2012-11-06 22:28 2012.11.06/
41451557 drwxr-xr-x 2 scott scott 4096 2012-11-06 22:32 2012.11.07/

./2012.11.06:
total 12
41423324 -rw-r--r-- 2 scott scott 93 2012-11-06 22:28 existing_file

./2012.11.07:
total 16
41423324 -rw-r--r-- 2 scott scott 93 2012-11-06 22:28 existing_file
41423328 -rw-r--r-- 1 scott scott 143 2012-11-06 22:32 new_file

Next, I create the second increment and update the existing file. The first two versions of the file have the same inode id and the third is different:

$ ls -lRi
.:
total 20
41451556 drwxr-xr-x 2 scott scott 4096 2012-11-06 22:28 2012.11.06/
41451557 drwxr-xr-x 2 scott scott 4096 2012-11-06 22:32 2012.11.07/
41451558 drwxr-xr-x 2 scott scott 4096 2012-11-06 22:38 2012.11.08/

./2012.11.06:
total 12
41423324 -rw-r--r-- 2 scott scott 93 2012-11-06 22:28 existing_file

./2012.11.07:
total 16
41423324 -rw-r--r-- 2 scott scott 93 2012-11-06 22:28 existing_file
41423328 -rw-r--r-- 2 scott scott 143 2012-11-06 22:32 new_file

./2012.11.08:
total 12
41423329 -rw-r--r-- 1 scott scott 0 2012-11-06 22:38 existing_file
41423328 -rw-r--r-- 2 scott scott 143 2012-11-06 22:32 new_file

Finally, I create the third increment and remove the original file. The first two versions of the file have the same inode id as before, the third is different as before but the fourth version is missing as expected:

$ ls -lRi
.:
total 16
41451556 drwxr-xr-x 2 scott scott 4096 2012-11-06 22:28 2012.11.06
41451557 drwxr-xr-x 2 scott scott 4096 2012-11-06 22:32 2012.11.07
41451558 drwxr-xr-x 2 scott scott 4096 2012-11-06 22:38 2012.11.08
41451560 drwxr-xr-x 2 scott scott 4096 2012-11-06 22:50 2012.11.09

./2012.11.06:
total 4
41423324 -rw-r--r-- 2 scott scott 93 2012-11-06 22:28 existing_file

./2012.11.07:
total 8
41423324 -rw-r--r-- 2 scott scott 93 2012-11-06 22:28 existing_file
41423328 -rw-r--r-- 3 scott scott 143 2012-11-06 22:32 new_file

./2012.11.08:
total 4
41423329 -rw-r--r-- 1 scott scott 0 2012-11-06 22:38 existing_file
41423328 -rw-r--r-- 3 scott scott 143 2012-11-06 22:32 new_file

./2012.11.09:
total 4
41423328 -rw-r--r-- 3 scott scott 143 2012-11-06 22:32 new_file

The Solution (in Practice)

I created a starting point using the following command:

rsync -avzh --delete SOURCE -e ssh root@SERVER:PATH

• SOURCE is the absolute path on the source device.
• SERVER is your destination device (IP Address or domain name)
• PATH is the absolute path on the destination device where the backup will be created

Next, I created the first increment:

rsync -avzh --delete --link-dest=PATH1 SOURCE -e ssh root@SERVER:PATH2

• SOURCE is the absolute path on the source device.
• SERVER is your destination device (IP Address or domain name)
• PATH1 is the absolute path on the destination device that rsync will copy to create the next increment.
• PATH2 is the absolute path on the destination device where the backup will be created

Finally, I created a script to run via cron nightly:

#!/bin/bash

CURRENT=PATH/$(date +'%Y.%m.%d')
LINKDEST=PATH/$(date +'%Y.%m.%d' -d "yesterday")

ssh root@SERVER mkdir -p $CURRENT
rsync -avzh --delete --link-dest=$LINKDEST SOURCE -e ssh root@SERVER:$CURRENT

• SOURCE is the absolute path on the source device.
• SERVER is your destination device (IP Address or domain name)
• PATH is the absolute path on the destination device where the backup will be created

This script requires that you have set up authorized keys. This is explained here.