ZFS and Cyrus

ZFS allows you to take snapshots of live filesystems, which is a great way to solve the 'oops, I deleted the wrong file' backup problem. 'zfs send' for snapshots allows you to effectively deal with the disaster recovery problem. Snapshots are smart in that they only make copies of new or modified files.

Where this breaks down, unfortunately, is things like databases, where relatively large files wind up being treated as modified and copied into each snapshot. It's much better to use the database's own tools to generate backups, which tend to be much smaller. They also are safer, as a snapshot of a running database may wind up not being cleanly recoverable (obviously they're supposed to be, if the database is fully ACID compliant, but there is always a difference between theory and practice).

Fortunately, zfs makes it cheap and easy to create separate filesystems for data that has different needs.

Cyrus IMAP, unfortunately, by default mixes two different flows of data together - each mailbox directory has one message per file, plus a handfull of opaque database files that are always changing as the content of the mailbox changes. What's more, these database files are entirely reconstructible, so backing them up is unnecessary.

The problem is that you want to snapshot the filesystem where the partition is, but the snapshot will also backup these index files, which is a pointless waste of space.

Fortunately, Cyrus 2.4 has added the ability to separate the meta-data from the partition. This way you can create snapshots of just the mail itself. The snapshots will wind up being much smaller, as a single message will only be present on the disk once (because with cyrus, once a message is written to disk, it's not touched after that).

Before I began, the imapd.conf file had in it:

partition-default: /home/imap-spool

I added to that

metapartition-default: /home/imap-meta

metapartition_files: header index cache expunge squat lock

I created a new zfs filesystem for /home/imap-meta, and chown'd and chmod'd it to match imap-spool. I then shut down the cyrus system.

At this point, there are two choices to migrate. I chose the safer path, which was to simply run 'reconstruct' and then 'find imap-spool -name cyrus\* -delete'. Unfortunately, this resulted in all of the messages being marked as unseen.

The other possible choice would have been to replicate the directory structure under imap-spool to imap-meta, and then move all of the files that don't match the pattern [0-9]*\. from imap-spool to imap-meta.

With either of these paths taken, you should be able to restart Cyrus and see that everything is basically unchanged and still works.

But having done this, you don't have to set up imap-meta for snapshots or backup.

HD, FTW

A few years ago (maybe it was in 2001. I don't remember), 2001: A Space Odyssey had a brief nationwide theatrical run. This was my first opportunity to see a movie that was made to be shown in a theater before I was born on a big screen, as it was intended. I had, of course, seen the film before, but always on Television, which until only recently meant NTSC video. Seeing it in the theater was a tremendous revelation. There was just an amazing amount of detail in the original film that I had never seen in all of the times I had seen the film on TV. The most striking example of this is in the scene when the lunar lander was being lowered into the moonbase. On either side of the elevator there are numerous windows, none of which were truly distinctly discernible on TV. There were actors walking around doing stuff and video screens displaying changing information while the lander was descending, all of which lent an extra air of reality to the scene. No doubt Stanley Kubrick went to a lot of extra trouble to add that in, knowing that people were going to be able to see it and that it would make the scene look just that much more realistic.

All that was nice and all, but I had my doubts that even HD would be able to present as much detail as that.

Turns out, I was wrong.

Just on a whim I decided to watch 2001 via Netflix Watch Now on the TiVo. Our Internet connection is good enough that we pretty routinely now get the highest quality streams available.

The stream they're showing now is every bit as nice as what I saw in the theater that day. All those details are clearly visible on our 50" TV from 8 feet away.

In hindsight, it should have been obvious to me. All you need to do is watch a little bit of an NFL game in SD and then HD to see the difference. It's not subtle. Not even a little.

Cisco Ūmi - say what?

Let's just call it what it is. The Cisco Ūmi is FaceTIme for your living room. As such, it's a pretty cool idea.

Cisco wants you to pony up $600 and $25/mo for the privilege.

Are they nucking futs?

For the same amount of money you literally could buy a mac mini and a webcam and plug that in your TV. You could then download Apple's FaceTime software for free and chat with your similarly equipped friends with no monthly cost at all (besides your Internet connection, which you'd have to pay for with Ūmi anyway).

I may have been born on a Saturday, but it wasn't last Saturday.

Verizon iPhone

Welcome, Verizon.

It'll be interesting to see how many people jump ship from AT&T and how many people trade in various other Verizon phones for iPhones.

At the moment, AT&T and Verizon offer a Morton's Fork to the phone connoisseur: AT&T's network is fast, when it works properly, but has lots of holes, even around here (AT&T is notoriously bad in Palo Alto, for instance). Verizon's network may have somewhat better coverage, but it's as slow as EDGE for data.

Maybe it'll be better with LTE, but of course, the Verizon iPhone isn't going to do LTE (which is only fair - the first iPhone was EDGE). We'll have to wait for that for probably at least another year.

The big news is the word that the Verizon phone will include a WiFi hotspot feature. It'll be interesting to see whether AT&T will add this feature, and how they'll price it (it really should be added for free to their existing tethering feature).

What has been left unsaid so far is whether or not Facetime will be supported without a WiFi connection (that is, over CDMA data). There's a much better case to be made for restricting it on CDMA given the constricted bandwidth compared to HSUPA.

My prediction is that Verizon will offer Facetime without restriction, which will force/shame AT&T into dropping the restriction on Facetime, and AT&T will follow suit with the wifi hotspot for folks who have the tethering plan.

No more POTS lines

I am about to cross a generational rubicon.

I am, sometime in the next week or two, going to call up AT&T and cancel our last remaining POTS line.

This line was "under" our DSL connection, and I had it on the cheapest measured-rate service I could get, because at the time you couldn't get Naked DSL. And, at the time, we had DirecTV receivers that required a connection to a land-based phone line, and we had our alarm system and we used the number to receive faxes.

Well, within a space of about 2 months, I've managed to make all of those justifications vanish. We tossed DirecTV and bought a TiVo Premiere and Comcast cablecards; we replaced our DSL connection by moving the server off into the cloud and buying a cable modem; I've equipped the alarm with a GSM modem and an AT&T prepaid SIM card; and the fax receive capability broke when I shut the server down, and wasn't really being used for anything anyway.

So now the only RJ-11 wiring in the entire house runs between the Vonage box, the cordless phone base, and our printer (for fax sending). Everything else we do is either via cell phones or IP.

How different it was 15 years ago. In 1995 I had a dialup ISP in my home with dialup PPP connectivity to the Internet that cost 4 times what I pay now for 50/10 megabit service. I subsidized the cost of the Internet connection with the ISP business. I had, at the height of it, 5 dialup modems and a couple dozen users. From 6 copper phone lines, down to 0.

Mac App store first impressions

I've perused the Mac App Store. There are some nice apps in there, but I have to say that just based on the value to me being offered, almost every single paid app I've seen so far has been a minimum of double the price I'd be willing to pay for that app. I'd be happy to pay $10 for Daisy Disk, but not $20. Maybe $15 (more like $10) for Earthdesk, but not $25. The one exception is the Contact Cleaner and Calendar Cleaner apps. I don't know if they're any good or not, because my calendar and contacts are clean enough for me at the moment, but if, for example, I had my Mother's calendar and contacts, it'd for sure be worth $5 to fix them.

We'll see if the market as a whole shares my view. Early indications are that at least some developers who got in on the ground floor are ecstatic about first day sales, but I do suspect that, as with the iPod/Pad/Phone app store that preceded it, prices will plummet as we see a race for the bottom.

iPhone and IPv6

I've been a devotee of IPv6 for years now. I bought, and continue to use, an AirPort Extreme because of it's support for 6to4 and tunneling.

Long story short, I've discovered that when an iPhone with IOS 4.x is connected to a WiFi network that's got an IPv6 router serving a globally reachable prefix, it will, in fact, use it!

Here's proof: The Dancing Kame on an iPhone.



This page has logic on it to detect whether or not you are fetching it with IPv6 or not, and though you can't see the animation in the screenshot above, the text surrounding it is different than what you get over IPv4.

So, Apple.... what's stopping you from at least optionally supporting 6to4 over 3G?

BSD in the cloud

For almost 20 years now, I've had static IP addressing at home. It makes me a bit of a throwback to when the Internet was young and September only had 30 days.

I had a static address because I had a server, and servers generally need to have their addresses be well known and stable. Yes, there are dynamic DNS tricks you can use to make do, but they demand that you at least have a stable address for your DNS server, and I've never really been comfortable with the proposition.

So for a while I've had the very best DSL that PacBell/SBC/AT&T would offer, because they also offered static IP addresses for residential service. Comcast, alas, does not. And it sort of makes some sense that they wouldn't. Cable modems are optimized for relatively light upstream demands. And their terms of service discourage the sorts of uses that imply it. Comcast does offer business class services, and they do include static addresses, but that means having a separate business account for service, and they don't actually offer the fastest speeds like they do for their residential customers, and you can't take advantage of bundle pricing and on and on.

I considered getting Comcast and keeping the DSL just for the server, but that winds up being expensive, and the tiny uplink channel for the DSL complicates things like backups.

With all of the advancement in virtualization technologies, however, I decided it was finally time to set up a VPS node and retire the machine in the garage.

I decided to go with RootBSD. They support FreeBSD, and had a reasonably good setup to let you perform your own installation. This let me perform a rather advanced ZFS based configuration that is not actually supported by the current FreeBSD installer. They went the extra mile and connected up the FreeBSD installation/Live DVD image on the virtual optical drive, though normally they install via PXE booting and installing the packages via FTP.

I made a couple of missteps in my attempt at installing, but worked around it by NFS exporting my own live CD across the Internet to fix my errors and everything was fine.

If you're wondering at this point how you get to the console of the virtual machine, they provide you with a VNC based console that you can connect to. The result is no different than if you were seated in front of a physical machine. You also get a web based power switch and reset button. They set your machine up with the amount of RAM and hard disk space you're paying for and they give you graphs so you can see how much network and disk I/O and CPU your VM is using over time. If you decide to upgrade, they can attach more RAM to your VM with just a reboot, and can add extra disk space as additional disk devices.

If you have multiple machines, another feature they offer is a 2nd (virtual, of course) Ethernet interface that connects to a private network connecting all of the machines to each other, and to a SAN where they host backup storage space. Bandwidth used on this private network between your machines (and your machines and the SAN) don't count towards your monthly allowed bandwidth.

All in all, 24 hours in, it's working well. I am using ZFS snapshots for most of the backup needs, and downloading a weekly snapshot as a disaster recovery mechanism. I'll probably retain the physical hardware for a while, just in case, and should be able to recover from a disaster by using the downloaded snapshots. With the Comcast 50/10 service, the weekly backup only takes a couple hours (in the wee hours of the morning).

So far, so good.

How the TSA kills americans

There's a big to-do this week with a big passenger backlash to the perceived excesses of airport passenger screenings. There are those with luddite attitudes towards the new backscatter X-ray scanners, but it's fairly easy to see how the machines themselves are safe. But facing the Hobson's choice of either a virtual strip-search or a pat-down not dissimilar to what happens when suspected criminals are arrested just because you choose to use an airplane to exercise your constitutional right to free (as in speech) travel is unreasonable.

But we can go a step further. There is a web campaign calling on travelers to "opt-out" of the X-ray, forcing the TSA to give all of those passengers pat-downs instead, in an act of civil disobedience. And the TSA has vowed not to give in. And so, the prospect looms of air travel becoming, at least for a day, even more unpleasant.

The more air travel becomes more expensive, unpleasant or otherwise untenable, the more people forego it for their cars. And while individually they don't make headlines, people die on a daily basis on America's roads, to the tune of more than 30,000 per year (as of 2009). That compares to a fatality rate for domestic commercial aviation of approximately 100 per year. So the TSA is doing everything they can to funnel people to a transportation system that has a fatality rate 300 times higher. They should be proud of themselves.

The attacks of September 11, 2001 were only possible because of the "rules of engagement" that were in place at the time. Those rules said that passengers and flight crew should cooperate with hijackers and let them go where they wanted and let the police handle matters when the plane lands (as it inevitably must). When Al Queda demonstrated that aircraft could be turned into weapons, those rules changed. In actual fact, Al Queda's plan stopped working even before it was complete - the plane that crashed in Pennsylvania did so because the passengers revolted when they figured out what the plan was. No terrorist since then has been able to take control of or destroy an aircraft while on board because of the vigilance of the passengers.

Really, the only thing the TSA needs to do, given this state of affairs, to make air travel sufficiently safe, is to insure that the cockpits remain secure during flight, that each piece of luggage in the cargo hold belongs to a passenger, and perform the level of passenger screening that was commonplace for the 3 decades between DB Cooper and 9/11.

Who does Nancy Grace remind you of?

Every time I see her on The Soup, I can't help but think of Roland Freisler.

So what does health care cost?

Scarlet was in the hospital a few weeks ago. Fortunately, she's all better, but we got some paperwork from our health insurance today. The long and short of it: Her one week hospital stay had a retail price of $87,000. But the "network" price was $18,000, and the patient responsibility (that is, what we have to pay) is $0.

So the last of those numbers - the $0 - is fine with me. I can't complain. It is, after all, what health coverage is for. But I'm reminded of my last visit to Safeway where I bought a box of crackers for $1.99 at the "club" price, instead of the "regular" price of $3.99.

Taiwanese animated news: Snooki

I've been loving the NMA animated news "reenactments" that have been featured on The Soup and elsewhere, but googling around for it, I found perhaps the best one of all, centered around Jersey Shore inmate, Snooki:



Best of all is the last few seconds of the video boldly predicting the coming end of her 15 minutes.

There really is an "I" in team

You just have to use the correct font.

The Mehserle mess

In honor of the sentencing this week, I thought it long overdue to pontificate on the shooting of Oscar Grant.

I think we can take at face value the supposition that Grant did nothing that evening that warranted the use of deadly force. But at the same time I think we also can take at face value Mehserle's assertions that he meant to use his Taser rather than his gun. After all, to believe otherwise would be to ascribe some motive that Mehserle had to kill Grant - a man whom we have no evidence Mehserle had ever met before (and, no, we have no evidence that Mehserle harbored any racist leanings that might explain his actions).

If anyone thinks Mehserle got off lightly, let's just put some things into perspective: he is going to spend another year of his life in prison, having already spent the better part of a year there. He will have a felony conviction on his record, which will make him utterly unemployable in any career even remotely related to what he is trained to do (since he is a felon, he will no longer have legal access to firearms, so he can't work as a security guard, and I don't believe felons can get a private investigator's license). And then, there's all the baggage that goes along with being an ex-con - having to check the "yes" box on all of his future employment applications that ask about felony convictions. All of this because he committed an error.

That said, we can, and I believe should hold police officers to a higher standard. But at the same time, I believe at least some of the blame should be attributed squarely to where it belongs. This was, fundamentally, a failure of ergonomic engineering.

Policemen and soldiers train themselves so that in times of stress they can rely on their instincts to carry out actions quickly and without thinking about them. They need to do so in order to stay alive, given that they are in situations where they are in contact with other people bent on doing them bodily harm. They need to act in the amount of time that the rest of us would mentally say, "Oh shit!" and soil ourselves.

I'm sure Mehserle trained for hours to pull a gun, raise it up to a target and pull the trigger.

Now go back to the picture of the actual model of Taser that Mehserle carried.

I'm sure the Taser folks were thinking that if they shaped their weapon that way that they could leverage the training that the users already would have in operating similar, but more deadly weapons.

Mehserle had time for exactly one thought that night, and his brain said, "Quick! Tase him!" Everything that followed from that decision was instinct and training. He reached for and grabbed his weapon. At that instant, had his Taser training been with a weapon that had a different shaped grip, or that you operated with a thumb trigger rather than an index finger trigger, then the reptilian portion of the brain that was following along the script would have said, "wait, this doesn't feel right." And that probably would have been enough to change the outcome. But the fact that the gripping action for the Taser X-26 was the same as his service revolver didn't give him a chance to recognize his error before it was too late.

It's too bad the book has already been written about technology failures like this. The story of Johann Mehserle and Oscar Grant absolutely deserves to be written alongside those of the Bhopal disaster and the Therac 25.

What's Hollywood-speak for Deja vú?

Remember when SkyLine used to be Independence Day?

Or perhaps you remember when it was called War of the Worlds?

Come on, Hollywood. Please.

New MacBook Air

I've decided to trade in my old MacBook Pro for an Air. I used to think that a big screen was what I wanted, but I've had a change of heart and now light weight is key, since I'm taking it with me on the train most mornings.

For a little while I had appropriated Scarlet's (original model) Air, and it was nice, though that machine only had 2 GB of RAM and was a little sluggish. It also was a little awkward only having a single USB port that was recessed inside that little door gizmo. Basically, it didn't work with anything except a cable, for the most part.

The new Air is a worthy successor. Not only can you get it with 4 GB of RAM, but they addressed just about all of the issues the original Air had (2 USB ports, and both them and the magsafe connector are mounted on a vertical, rather than a canted surface).

As we speak, I'm writing this post on CalTrain with the new Air. With the MiFi for connectivity, it's an excellent experience. Time will tell how the battery stacks up, but I may be able to leave the power adapter at home from now on if Apple's claims are justified. And leaving that behind makes my load about a third lighter, it turns out (the power adapter weighs almost as much as the Air. The rest is my tote).

Most remarkable is that the restore media for the Air is in the form of a USB ROM drive. It's slightly longer and narrower than a postage stamp, yet it contains all of the pre-loaded software and is bootable. The first announcements of third-party replacement flash drives for the Air have come out today, so this would be how you'd get your machine back up and running after swapping the drive out. Of course, that presumes you can find one of those pesky pentacle security Torx screwdrivers to get the bottom case off (really, Apple? What's the purpose of keeping me out of my own machine?).

I haven't tried the USB SuperDrive we bought for the old Air, though I assume it works just the same way with the new one. Of course, with USB ports on both sides of the machine, you no longer are forced to keep the drive on the right, if that's not what you like.

Once again, I was struck with how easy the transition to a new mac is with the migration tool. But at the same time, I was also struck with how bloody long the process takes. And I had to do it twice, since I was migrating two machines down into one. The second time, however, I used a USB Ethernet interface and connected the Air directly to the target machine, which improved the speed (instead of using WiFi) markedly.

Why don't they stock the good stuff?

Scarlet wants a speakerphone for the van. I did some research and I found the exact right one. Which is unusual. It's the Kensington LiquidAUX Bluetooth car kit. It's the most promising product pitch I've had: It's a speakerphone that does A2DP and has an auxiliary output (instead of an FM transmitter - we went to great trouble to get an aux port for the stereo to avoid FM transmitters). It plugs into the power port (formerly known as cigarette lighter) and has a USB power outlet.

So it's wonderful. I'm sold. I want one.

So why isn't there any in any stores nearby? I swear, it's supernatural. The one thing you want to buy is the one thing nobody ever stocks.

We'll actually have to see whether it lives up to the hype or not, but we won't know for a week because I had to buy one through the internet.

Grumble.

No more POTS wiring

Well, this evening I decided we were done with POTS wiring in the house. Everywhere CAT-5 wire goes now, it goes to a T568B jack. The whole house is thusly set up for gigabit Ethernet.

We still have a couple of uses for analog phone lines, but they don't need to go over the house wiring.

I went to Fry's and bought a wall mount set of punchdown CAT5 8P8C jacks and just punched down the whole house onto it. We have an 8 port gigabit switch in the front of the garage, and it's all lit up for gigabit now.

So... w00t.

Are you have problems with the 3G Microcell?

So, I know I'm not the only one having a terrible experience with AT&T's 3G Microcell. If you're one of the unfortunate folks who have one of these boat anchors and it's not working, I'd suggest you join me in my campaign to spread the word.

Here's a YouTube video posted by a guy in the marketing department of Cisco, the folks who built the Microcell for AT&T.

Here's a YouTube video posted by AT&T demonstrating the setup procedure for the Microcell.

Go leave a comment on these videos. Tell them your Microcell story of pain. Maybe if enough people do this we can shame them into some kind of public response.

It's official, the 3G Microcell sucks

It's been 6 months now. It's time to take it to the streets.