as days pass by

The server that this site runs on is a Bytemark Hosting virtual machine. Those of you who have been reading for a while will remember that we had a serious fuckup a couple of months ago and the site went away. And we didn’t have any backups. Fortunately, we were able to retrieve an old version of the sites hosted here, and reconstructed the data. But that’s not happening again, so I’ve set up some backups. Bytemark give you some free backup space to back up your VM. The following steps should give you full nightly backups of everything you want to back up.

First, mount your backup space over NFS. You need to get access to the backup space as a filesystem, although we’re not using NFS for the backups; we’ll be using rsync. Mount the backup space at /backup. All this stuff needs to be done as root.

mkdir /backup/.config
mkdir /backup/rsync
mkdir /backup/rsync/files
mkdir /backup/rsync/db
nano /backup/.config/rsync

Put the following line in that file:

VMNAME-backup:PASSWORD:::1:rsync

This sets up rsync. VMNAME must be the name of your Bytemark VM (NB: not the domain name you’ve got pointed to it; it’ll also be called something like foobar.vm.bytemark.co.uk, and foobar is your VM’s name.) PASSWORD is your choice of rsync password. Pick something random. Wait for ages. Rsync stuff only gets activated a few times a day, so go away and come back the following day or eight hours later or something.

mkdir /usr/local/backups
echo "RSYNC_PASSWORD_FROM_ABOVE" > /usr/local/backups/backup-password
chmod og-rwx /usr/local/backups/backup-password

Now we’ll test that rsync is working.

rsync -a 
--password-file=/usr/local/backups/backup-password 
/etc/motd 
VMNAME-backup@VMNAME.backup.vm.bytemark.co.uk::VMNAME-backup/files/test/

If that worked, there should now be a /backup/rsync/files/test directory with an “motd” file inside. That means that rsync is working! If it didn’t work then, er, rtfm, etc, etc.

Now to create two backup scripts, one to do your MySQL databases and one to do your files. If you don’t use MySQL, skip the DB one (or replace it with one that does your own choice of DB.)

/usr/local/backups/backup-databases.sh:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

#!/bin/bash

# Dump all databases to a file, and rsync that file into the backup space,
# inside a directory named for data and time.

OUTPUT_FILE=/usr/local/backups/dbdump
NEW_BACKUP_DIR=`date +%Y%m%d@%H%M%S`

mysqldump --all-databases --user=root --password=_yourMySQLRootPassword 
   --opt | bzip2 > $OUTPUT_FILE

rsync -a 
--password-file=/usr/local/backups/backup-password 
 $OUTPUT_FILE 
 <em>vmname</em>-backup@_vmname_.backup.vm.bytemark.co.uk::_vmname_-backup/db/$NEW_BACKUP_DIR/

logger -t Backups Backup of all databases done to $NEW_BACKUP_DIR

Run that file to test it. It should create a directory in /backup/rsync/db named after the date and time, and put a DB backup in it of all your databases. /usr/local/backup/backup-files.sh:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25

#!/bin/bash
# Backs up files with rsync

NEW_BACKUP_DIR=`date +%Y%m%d@%H%M%S`
FILES_TO_BACKUP=/usr/local/backups/files-to-include
EXCLUDE_FILES=/usr/local/backups/files-to-exclude
LAST_BACKUP_FILE=/usr/local/backups/last-file-backup

# Work out what the <strong>last</strong> backup was, to use as a base for this one
if [ -f $LAST_BACKUP_FILE ]; then
  read LAST_BACKUP < $LAST_BACKUP_FILE
else
  LAST_BACKUP=nonexistent
fi

echo $NEW_BACKUP_DIR > $LAST_BACKUP_FILE

rsync -a --delete 
  --password-file=/usr/local/backups/backup-password 
  --include-from=$FILES_TO_BACKUP 
  --link-dest=../$LAST_BACKUP 
  / 
 VMNAME-backup@VMNAME.backup.vm.bytemark.co.uk::VMNAME-backup/files/$NEW_BACKUP_DIR/

logger -t Backups Backup of files done to $NEW_BACKUP_DIR

You’ll also need to create /usr/local/backups/files-to-include. The syntax of this file is a great big pain in the arse. Mine is below:

# List of directories to back up
# If you want to back up directory /a/b, then you need the following lines:
# /a         (since otherwise we never get as far as b, because a is excluded)
# /a/b       (the directory we want to back up)
# - /a/*/    (don't back up any other directories in /a)
# - /a/*     (don't back up any files in /a either)
# Yes, this is completely shit. Rsync's fault.

/var/
/var/www/
- /var/*/
- /var/*
/home/
/etc/
/usr/
/usr/local/
- /usr/*/
- /usr/*
/usr/local/backups/
- /usr/local/*/
- /usr/local/*

# Directories to explicitly exclude. Note that you don't need all
# that parent directory shit here; just put in dirs that you don't
# want to back up for whatever reason.

- /var/www/jonobacon.org/mysqlbackups/
- /var/www/jonobacon.org/mysqlbackups.old/

# Don't fuck about with the stuff here; it ensures that logs directories
# aren't backed up, and neither is anything else in / (otherwise we'll
# back up the whole machine or something equally dreadful).

- **/*.mp3
- **/logs/
- /*/
- /*

Pay attention to the comments. I mean it.

This is actually a super clever backup method. All the magic is done by rsync’s—link-dest option. What that does is, every time you tell it to back up, it creates a tree of hardlinks in the backup space from the last backup. Then it rsyncs any changed files over the top. This means that every single backup is a full backup, but it only takes up the space of an incremental backup. This only works because hardlinks exist. I love Linux, I really do.

Anyway, those two files should do the job of backing up everything. Stick two lines in your crontab to kick them off.

/etc/crontab:

# ########### Backups ###############
# mysql
30 5 * * * root /usr/local/backups/backup-databases.sh

# files (4.40am)
45 5 * * * root /usr/local/backups/backup-files.sh

and it should all work.

Note that these instructions probably won’t work for you right off, but they will at least give you a starting point.