ZFS and Cyrus

ZFS allows you to take snapshots of live filesystems, which is a great way to solve the 'oops, I deleted the wrong file' backup problem. 'zfs send' for snapshots allows you to effectively deal with the disaster recovery problem. Snapshots are smart in that they only make copies of new or modified files.

Where this breaks down, unfortunately, is things like databases, where relatively large files wind up being treated as modified and copied into each snapshot. It's much better to use the database's own tools to generate backups, which tend to be much smaller. They also are safer, as a snapshot of a running database may wind up not being cleanly recoverable (obviously they're supposed to be, if the database is fully ACID compliant, but there is always a difference between theory and practice).

Fortunately, zfs makes it cheap and easy to create separate filesystems for data that has different needs.

Cyrus IMAP, unfortunately, by default mixes two different flows of data together - each mailbox directory has one message per file, plus a handfull of opaque database files that are always changing as the content of the mailbox changes. What's more, these database files are entirely reconstructible, so backing them up is unnecessary.

The problem is that you want to snapshot the filesystem where the partition is, but the snapshot will also backup these index files, which is a pointless waste of space.

Fortunately, Cyrus 2.4 has added the ability to separate the meta-data from the partition. This way you can create snapshots of just the mail itself. The snapshots will wind up being much smaller, as a single message will only be present on the disk once (because with cyrus, once a message is written to disk, it's not touched after that).

Before I began, the imapd.conf file had in it:

partition-default: /home/imap-spool

I added to that

metapartition-default: /home/imap-meta

metapartition_files: header index cache expunge squat lock

I created a new zfs filesystem for /home/imap-meta, and chown'd and chmod'd it to match imap-spool. I then shut down the cyrus system.

At this point, there are two choices to migrate. I chose the safer path, which was to simply run 'reconstruct' and then 'find imap-spool -name cyrus\* -delete'. Unfortunately, this resulted in all of the messages being marked as unseen.

The other possible choice would have been to replicate the directory structure under imap-spool to imap-meta, and then move all of the files that don't match the pattern [0-9]*\. from imap-spool to imap-meta.

With either of these paths taken, you should be able to restart Cyrus and see that everything is basically unchanged and still works.

But having done this, you don't have to set up imap-meta for snapshots or backup.