A few years ago, I decided to build myself a fileserver. Based on my criteria (free software, extensible, parity-based redundancy), a custom-built system was the best option; and for an operating system, I settled on FreeNAS. It let me get features I wanted (basically the entire ZFS featureset) without jumping headfirst into administering a BSD system, something I'd never done at the time. And it worked very well; installation is as simple as flashing an image onto a USB thumb drive, booting the system off that thumb drive, and setting things up through the surprisingly usable web interface. (I have a general antipathy to web anything as bad technology used by bad people for bad things. But the FreeNAS folks, as of version 8.3 anyway, kept their pants on about it and so, at the client end, it's solidly simple and usable.)
One of the options, which I ticked because it seemed like a reasonable idea, is to run full-disk encryption over the disks used for storage. In retrospect, this was not a great idea in my case, as my workflow makes entering a passphrase on boot excessively inconvenient (I leave the system off most of the time, turning it on only when it's needed) so the keyfile is just sitting there unenecrypted anyway, and it adds another layer of things that can go wrong if you ever need to recover data. And unfortunately, another questionable decision on the part of the FreeNAS developers means that you will almost certainly have to recover your system at some point.
See, USB thumb drives aren't a great choice to host the root of an OS. Unlike SSDs, they don't do wear leveling as a rule (at least, the 8GB drive I had on hand three years ago didn't) and OSs like to log things and generally do lots of bookkeeping on their root as a rule, and the intersection of those behaviors is that whatever thumb drive you have FreeNAS installed on will likely fail early. Mine finally bit the dust a week ago. (It had been failing before that; the first sign I noticed, in retrospect, was that the web interface was failing to load. That started months ago but didn't interfere with my use of the device as a fileserver, so I put off digging out a keyboard and monitor to plug into the thing to diagnose the issue, something I probably should have been more proactive about.) It's throwing hard read errors every few blocks and is, unfortunately, almost certainly completely shot.
So I decided to buy a small, cheap SSD, throw FreeBSD on it, and rebuild the thing. The data volumes are all fine and I'd backed up the encryption key, so the only thing left to do was figure out FreeNAS's encryption scheme and reconstruct it in FreeBSD. (By the way, if you have an encrypted fileserver and haven't backed the key up somewhere, do it now. I'll wait. Without that key, your data is gone forever, that being rather the point of encryption, after all. Actually, FreeNAS has an option to export the entire configuration to a file, a feature I wholeheartedly recommend making use of. Importing your configuration into a new FreeNAS install is even easier than the process I describe here.) Unfortunately, it seems nobody's documented exactly how FreeNAS builds its encrypted volumes, so I had to piece that together from bits and pieces.
(If you're looking for the TL;DR, fortunately, it's very simple:
the drives are encrypted with geli and a zpool is built over the crypto
/dev/*.eli. Easy as that.)
The first step to setting everything up was to install FreeBSD to the SSD I'd gotten for the OS root. I unplugged the data drives while installing to prevent mishaps. Installing FreeBSD is simple, so I won't go over it it any detail; anyone who's installed a Linux distro can handle it, and their documentation is pretty good.
After plugging the data drives back in, booting the system and getting
sshd working so I could fiddle with the thing from the comfort of my
living room with a web browser to hand, the next step was to import
the encrypted volumes. Fortunately, this is actually really easy (you
get to skip my obsessive worry that I would do something wrong and
nuke my data). The drives are encrypted with geli, so they're attached
root@databox:~# geli attach -k keyfile /dev/encrypted slice
for each device you have. The devices have already been set up by
FreeNAS, so no importing or configuration is required. Once you've
imported all of them, you should have a bunch of crypto devices in
At this point, for me at least, ZFS automatically picked up on the
pool stored on the crypto devices. I only needed to mount it:
root@databox:~# zfs mount pool name
and everything worked. Setting up NFS to export the share is
well-documented, so I won't describe it here.
Hope this helps anyone else with FreeNAS encrypted volumes on hand. Happy hacking!