Distributed backups to friends

It ought to be possible to have a backup system with the following characteristics:

  1. You download a backup client and run it. It asks for a backup group name and a password. It is cross-platform, or at least ported to Linux, Mac OS X, and Windows. It also asks how much space you’re prepared to devote to backups.
  2. You choose directories and files to back up by finding them in your file manager and tagging them as “For backup”.
  3. That’s then all the user interaction that there is.

The way the backup actually works is that:

  • It takes the stuff you want to back up, and creates a big backup file out of it, every night.
  • It breaks the file up into bits, using the PAR stuff from parchive.sourceforge.net. This means that to recover your backup, you need some but not all of the bits, so if some bits get lost it doesn’t matter.
  • It then ships the bits out to other people in your backup group and stores it on their systems, not on yours, giving you off-site backups.

That would make it very easy for a group of people to do mutual backups without having to think very hard about it.

Implementation thoughts

You’d need a server somewhere, to store password details for backup groups and to co-ordinate shipping the data around (since everyone’s likely to be behind a firewall). No-one should ever see or know about this server, though. There is no “sign-up procedure”; to create a new backup group, you just run the client and provide a backup group name and password. That’s all. You don’t have to sign up on the web or explicitly invite anyone into the group; anyone with the username and password can join. There’s nothing in the above about how to restore from a backup, I know. That needs some kind of UI, but I’m not sure what that should be. It needs to warn you if there’s not enough space out there on the group to back up all the stuff you’re trying to back up. Some kind of algorithm which demands that if you want to back up N megabytes you have to offer 3N megabytes of space to the group or something. There should be some rsyncness in it. If not much has changed, it shouldn’t need to send much out to the group. However, this might be complex, because the previous backup is in scattered bits, and you don’t want to do incremental backups because then you need the full backup as well. Backups must be encrypted, because they’re stored on someone else’s machine. There will probably need to be some kind of UI to provide a passphrase or similar. This also makes rsyncness difficult. I think this would be a really useful project. The key point, the absolutely critical point, is that the client must be as described above: it just asks you which backup group you’re in and that’s all. No ten pages of options, no need for you to tell it who else is in the group or to maintain a list of who that is or where you want backups to go. If it’s in any way difficult, it won’t get used, and then no-one has backups. Wish I had time to write this. The big problem that needs solving is how to have the rsyncness in it, so that it only ships changes around rather than a full backup every night. Other than that, it’s all doable, and not all that difficult.

I'm currently available for hire, to help you plan, architect, and build new systems, and for technical writing and articles. You can take a look at some projects I've worked on and some of my writing. If you'd like to talk about your upcoming project, do get in touch.

More in the discussion (powered by webmentions)

  • (no mentions, yet.)