ZFS and Cyrus

ZFS allows you to take snapshots of live filesystems, which is a great way to solve the 'oops, I deleted the wrong file' backup problem. 'zfs send' for snapshots allows you to effectively deal with the disaster recovery problem. Snapshots are smart in that they only make copies of new or modified files.

Where this breaks down, unfortunately, is things like databases, where relatively large files wind up being treated as modified and copied into each snapshot. It's much better to use the database's own tools to generate backups, which tend to be much smaller. They also are safer, as a snapshot of a running database may wind up not being cleanly recoverable (obviously they're supposed to be, if the database is fully ACID compliant, but there is always a difference between theory and practice).

Fortunately, zfs makes it cheap and easy to create separate filesystems for data that has different needs.

Cyrus IMAP, unfortunately, by default mixes two different flows of data together - each mailbox directory has one message per file, plus a handfull of opaque database files that are always changing as the content of the mailbox changes. What's more, these database files are entirely reconstructible, so backing them up is unnecessary.

The problem is that you want to snapshot the filesystem where the partition is, but the snapshot will also backup these index files, which is a pointless waste of space.

Fortunately, Cyrus 2.4 has added the ability to separate the meta-data from the partition. This way you can create snapshots of just the mail itself. The snapshots will wind up being much smaller, as a single message will only be present on the disk once (because with cyrus, once a message is written to disk, it's not touched after that).

Before I began, the imapd.conf file had in it:

partition-default: /home/imap-spool

I added to that

metapartition-default: /home/imap-meta

metapartition_files: header index cache expunge squat lock

I created a new zfs filesystem for /home/imap-meta, and chown'd and chmod'd it to match imap-spool. I then shut down the cyrus system.

At this point, there are two choices to migrate. I chose the safer path, which was to simply run 'reconstruct' and then 'find imap-spool -name cyrus\* -delete'. Unfortunately, this resulted in all of the messages being marked as unseen.

The other possible choice would have been to replicate the directory structure under imap-spool to imap-meta, and then move all of the files that don't match the pattern [0-9]*\. from imap-spool to imap-meta.

With either of these paths taken, you should be able to restart Cyrus and see that everything is basically unchanged and still works.

But having done this, you don't have to set up imap-meta for snapshots or backup.

BSD in the cloud

For almost 20 years now, I've had static IP addressing at home. It makes me a bit of a throwback to when the Internet was young and September only had 30 days.

I had a static address because I had a server, and servers generally need to have their addresses be well known and stable. Yes, there are dynamic DNS tricks you can use to make do, but they demand that you at least have a stable address for your DNS server, and I've never really been comfortable with the proposition.

So for a while I've had the very best DSL that PacBell/SBC/AT&T would offer, because they also offered static IP addresses for residential service. Comcast, alas, does not. And it sort of makes some sense that they wouldn't. Cable modems are optimized for relatively light upstream demands. And their terms of service discourage the sorts of uses that imply it. Comcast does offer business class services, and they do include static addresses, but that means having a separate business account for service, and they don't actually offer the fastest speeds like they do for their residential customers, and you can't take advantage of bundle pricing and on and on.

I considered getting Comcast and keeping the DSL just for the server, but that winds up being expensive, and the tiny uplink channel for the DSL complicates things like backups.

With all of the advancement in virtualization technologies, however, I decided it was finally time to set up a VPS node and retire the machine in the garage.

I decided to go with RootBSD. They support FreeBSD, and had a reasonably good setup to let you perform your own installation. This let me perform a rather advanced ZFS based configuration that is not actually supported by the current FreeBSD installer. They went the extra mile and connected up the FreeBSD installation/Live DVD image on the virtual optical drive, though normally they install via PXE booting and installing the packages via FTP.

I made a couple of missteps in my attempt at installing, but worked around it by NFS exporting my own live CD across the Internet to fix my errors and everything was fine.

If you're wondering at this point how you get to the console of the virtual machine, they provide you with a VNC based console that you can connect to. The result is no different than if you were seated in front of a physical machine. You also get a web based power switch and reset button. They set your machine up with the amount of RAM and hard disk space you're paying for and they give you graphs so you can see how much network and disk I/O and CPU your VM is using over time. If you decide to upgrade, they can attach more RAM to your VM with just a reboot, and can add extra disk space as additional disk devices.

If you have multiple machines, another feature they offer is a 2nd (virtual, of course) Ethernet interface that connects to a private network connecting all of the machines to each other, and to a SAN where they host backup storage space. Bandwidth used on this private network between your machines (and your machines and the SAN) don't count towards your monthly allowed bandwidth.

All in all, 24 hours in, it's working well. I am using ZFS snapshots for most of the backup needs, and downloading a weekly snapshot as a disaster recovery mechanism. I'll probably retain the physical hardware for a while, just in case, and should be able to recover from a disaster by using the downloaded snapshots. With the Comcast 50/10 service, the weekly backup only takes a couple hours (in the wee hours of the morning).

So far, so good.

Ancient firmware updates

It's not every day that you set out to upgrade the firmware of a 10+ year old device.

One of the oldest pieces of gear still operational at our house is the fax modem out in the garage. It waits patiently for the phone to ring and receives faxes. Most of them are for Scarlet. The few that aren't are junk. But every once in a while sending an image of a signature or form or some such is necessary, and e-mailing a PDF won't work, so it's nice to have the option to use the old methods.

I was poking around the Internet looking for something else, when I managed to wander down the path towards US Robotics, which was bought some time ago by 3Com. In fact, they still have a USR website, including a support section for a lot of their obsolete products. I was able to root around and find my model of Courier V.Everything. I bought it sometime in, oh, 1997 or 1998 or so. I don't remember.

I opened up the serial port and did a ATI7 and, in fact, it informed me that the firmware was out-of-date, and that they had a DOS based updater for it. Ew. Well, I read a little further, and believe it or not, they had a mechanism for updating the firmware for folks running without DOS/Windows. If you send AT~X! to the modem, it starts doing an XMODEM receive for a firmware update. How convenient! Well, I was using kermit as a modem terminal program, but shelling out from kermit's command mode allowed me to use the lrzsz package from the ports tree to send the firmware. And it was done.

I'm not sure if the update did a lot. The same set of AT commands appears to be present. Hopefully it might have a positive impact on the fax reliability. I don't really use the modem for anything else.

FreeBSD 8.0

I upgraded quack to FreeBSD 8.0 RC3 this evening. I also re-enabled the journaling on /home. The last time I tried the journaling for UFS, it was very unstable. Hopefully in the interim, things have gotten better. If I don't post bad news, then you'll all know it went well.

The only other minor upgrade gotcha was that the APC daemon that monitors the UPS failed. All I had to do was change the device name and rebuild the port and it worked once again.

uuencode -> base64

If you have a modern version of Unix, you'll have a more recent version of the 'uuencode' command that can generate base64 output with the -m flag.

If you don't, you can convert uuencode output to base64 easily enough. You just have to remove the 'begin' and 'end' lines, strip off the 'count' character off the front of each line, and then convert the characters from the uuencode character set to the base64 one.

This is all relatively easy to do as a shell script pipeline. The hardest part is putting the special characters in the uuencode character set into a variable. But careful escaping can make it happen:

#! /bin/sh

# The uuencode character set
UU=' !"#$%&'"'"'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`'
# The base64 character set - note the extra ` -> A conversion
B6='ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/A'

tail +2 | egrep -v ^end - | tr "$UU" "$B6" | sed -e "s/^.//g"

You will have to remove the line breaks stuck in there by blogger. Sorry.

This script also leaves out one step that makes the output potentially invalid base64 - if the number of characters in the output stream is not an even multiple of 4, then '=' characters are added to represent 0 padding. You can do that experimentally by hand if you absolutely need it, but having to count the number of characters (not including the newlines) as you output them would be rather painful. At that point, it makes more sense to just write base64 yourself in C.

VoIP traffic shaping with FreeBSD IPFW + Dummynet

We have an Airport Extreme, and while it is a good router, it isn't perfect. Among the deficiencies in it, it lacks support for traffic shaping (sometimes called QoS).

Using the FreeBSD machine as the firewall instead of the AirPort is one option, but the problem with that is that FreeBSD lacks support for uPNP or NAT-PMP, which would seriously complicate things like Back-to-my-Mac.

The best compromise is to keep the AirPort Extreme in the routing role for the inside machines, and use the FreeBSD machine as a traffic shaping bridge.

It's fairly simple:

ipfw pipe 1 config bw 640kbps
ipfw queue 1 pipe 1 weight 100
ipfw queue 2 pipe 1 weight 1

queue 1 is for Vonage traffic, and queue 2 is for everything else.

ipfw queue 1 ip from voip to any xmit re0
ipfw queue 2 ip from not voip to any xmit re0

in this case, re0 is the "outside" LAN interface (connected to DSL).

The issue is in figuring out what the number for the bandwidth is. It's not actually the same as the DSL connection's uplink line rate. The problem is that there are several layers of encapsulation. The best way to figure out what to put in there is to start with a number far higher than the uplink bandwidth and run a speed test. Reduce the number until the speed test shows a reduction in the uplink bandwidth. In our case, the line is a 6M/768K one and the right number to use is 640kbps. The speed tests show an actual data throughput of closer to 600 kbps. The Ethernet and ATM framing for the packets takes up the rest of the bandwidth.

FreeBSD and Journaled UFS

Since I have been in an experimental mood lately, I convinced myself to redo the /home partition on my FreeBSD machine to enable journaling. It went pretty smoothly, though since the disk was fully partitioned, it did mean doing a dump and a restore. Fortunately, though /home is relatively large, it doesn't have much stuff in it, so a dump of it fit in the free space on /usr (if this hadn't been the case, I would have just used an external disk over USB). So here's what I did:

1. Boot into single user mode.

2. fsck -p && mount -a

3. umount /home

4. dump 0af /usr/home.dump /home

Now, at this point we're actually going to destroy the existing /home partition, so make sure the dump worked correctly first. In this case, my home partition was on /dev/ad0p5.

5. gjournal load

6. gjournal label -f /dev/ad0p5

I didn't bother specifying a -s argument, so the default journal size of 1 GB will be in effect.

7. newfs -J -L home /dev/ad0p5.journal

8. At this point, /dev/ufs/home should reappear. Edit /etc/fstab and find the line that mounts /home and change 'rw' to 'rw,async' since with journaling enabled we'll be using an asynchronous mount.

9. mount /home . Now type 'mount' and make sure that both 'asynchronous' and 'journal' show up in the list of mount options for /home.

10. cd /home && restore -rf /usr/home/dump

11. Edit /boot/loader.conf and add a line that says geom_journal_load="YES"

12. reboot.

So why bother? Because now /home will not need to have an fsck performed on it ever again.

GPT for FreeBSD!

A lot has changed since the PC was first introduced in the early 1980s. Unfortunately, a lot of legacy nonsense has remained behind that has to be hacked-around because of that history. In essence, a modern Intel based computer starts the boot process as an 8088 and, effectively, climbs up out of the primordial ooze stepwise, getting a little more advanced with each tweak.

One of those bits of legacy nonsense still with us is the old FDISK based partitioning scheme, which still has nonsense like cylinders, heads and sectors in it (which mean nothing at all to modern disks, which in essence are just single-dimensioned arrays of 512 byte blocks).

FreeBSD has existed within this framework by using the old BSD "slice" system to chop up a single FDISK partition. Linux, instead, has tended to use FDISK extended partitioning, but that has led, to some extent, to Linux systems living in a single filesystem. One of the benefits of the multi-partitioning that FreeBSD uses instead are that two of the typical partitions (/ and /usr) tend to be "read-mostly," while the other two (/var and /home) tend to have the majority of write activity. This is somewhat safer, since the volatility is kept away from important configuration data (in /etc) and system code (in /boot and /usr).

Macs have discarded a lot of the Intel legacy, since they don't have to boot 20 year old operating systems (instead, they boot Unix, which is a 30 year old operating system, but I digress). So the Intel macs have adopted GPT partitioning. GPT is great, since it can operate on disks with 2^64 blocks, which is ridiculously huge, and the number of partitions is unlimited (it's actually limited to the number of blocks allocated to the GPT itself, and by default it's 128 entries).

Well, with the advent of FreeBSD 7.1, support has been added to boot from a GPT partitioned disk. The way this works is that the legacy FDISK partition (which basically marks the entire disk as allocated to protect against FDISK based tools trying to do something to the disk) contains boot code that looks for a special GPT partition that contains the second stage boot loader. That partition and the FDISK boot sector is written with the 'gpt boot' command. That boot loader looks for the first UFS partition on the disk and loads the main FORTH-based loader, which in turn loads the kernel and so on.

Just to be on the safe side, I migrated my setup by buying a new disk (a 320GB disk for $55 - can't go wrong!), partitioning it and then copying all the stuff over. As a bonus, the new disk is faster than the old one, and we get to start the warranty over.

There is, however, a 'gpt migrate' command that purports to convert a disk from FDISK and BSD slice format to GPT, but that concept sounds a bit scary to me, at least for filesystems I really care about. :)

One interesting thing is that the gpt command opens the disk for writing, even if all you want to do is read information. But if there are mounted partitions, then this will fail. There is a -r option that will make the open read-only, which is useful for the 'show' command.

Now that the disk is GPT partitioned, converting to an EFI based machine would simply require replacing the special FreeBSD boot partition with an EFI system partition (it would need to be slightly larger, but space from the adjacent swap partition could be stolen), and EFI boot code for FreeBSD would need to be added to it (if it existed).

FreeBSD 7.1 hasn't yet has been released - it's at RC2 at the moment - but - on new year's day, in fact. I'm quite happy with it.

sid-milter and SMTP AUTH

I use sid-milter. It works well enough for what it does, but there is a stumbling block. I have my laptop set up for SMTP AUTH so that if I'm not at home, I can still send e-mail as long as I can reach my home mail server. My home domain has an SPF record that includes a "-all" clause in it so that if someone tries to joe-job me, at least those who look at SPF records won't get misled. The problem is that sid-milter doesn't seem to know to check to see if a message was sent with SMTP authentication. I get the Authentication-Result: header with all failures in it. That tends to look bad, even if a spam filter somewhere doesn't trigger on it and junk my e-mail.

Well, since I am out of town and it's happening to me, I decided to fix it. I got into the source of the sid-milter and added a check nearby the "ignore domain" list code. It checks to see if the message has the {auth_authen} macro set, and if it does, it does the same thing as if the "ignore domain" list check matched - it just accepts the message without doing anything.

Here's the patch:

--- sid-filter.c.orig 2006-07-20 15:28:09.000000000 -0700
+++ sid-filter.c 2008-03-24 15:55:00.000000000 -0700
@@ -1892,6 +1892,13 @@
   return (testmode ? SMFIS_ACCEPT : SMFIS_REJECT);
  }
 
+ char *auth_authen;
+ auth_authen = smfi_getsymval(ctx, "{auth_authen}");
+ if (auth_authen != NULL) {
+  /* User used SMTP auth. No problems */
+  return SMFIS_ACCEPT;
+ }
+
  /* if the responsible domain is one we trust, just accept */
  if (domains != NULL)
  {

Oh, note that you must add define(`confMILTER_MACROS_ENVFROM', `i, {auth_authen}') to your sendmail.mc file.