🏠 Go home.

Catastrophic Data Loss and Me

Published on

So, my server decided to not mount its root partition after moving a mile up the road. This seems like it shouldn't be bad, because we have backups and use them, right?

Well…

My /srv/data/ is backed up, but my / was not. And it turns out all of my VMs and containers' persistent data were on that / partition. I lost a fair amount of data by leaning in to an architecture I didn't fully understand, trying to use Btrfs for my server's root partition to (ab)use systemd-nspawn's ability to generate containers as subvolumes of a root volume. Create a base install, run containers as subvolumes of that, get all the niceties of COW and dedup without having to deal with Docker or any of that crap. It turns out that at some point, I added a file as a non-loopback device to the volume, in an attempt to fix a full btrfs metadata subvolume. That file, upon inspection on reboot, was all 0's.

Woops.

Oh well.

Anyways.

I lost a fair amount of stuff, it turns out.

Most importantly, I lost the private keys for the whatthefuck.computer Matrix.org homeserver. I'm now @rrix:kickass.systems. This is incredibly unfortunate, and it's why you should always have backups of your keys. I need to figure out what to do with Kickass Systems Common because the only admin in that room was on whatthefuck.computer. Oh well. If you need me, message me on @rrix:kickass.systems using the matrix.to link above. You cannot contact me at @rrix:whatthefuck.computer any more.

I also lost all of the stuff on my git host, anything I didn't have cloned on to my laptop. That's the pits. I replaced my cgit + gitolite setup with Gogs which is publicly accessible on a nice URL now. I'm going to start updating all of the README and, I guess, make Gogs support README.org. I'll also need a Gogs Issues<->Org Mode syncer, in general. Luckily I keep most things I care about cloned in to /srv/data/ so I didn't lose that much stuff from my Git repositories, mostly just old things. If by chance someone had actually cloned those things and wants to see them, email me and I might have them in an old backup somewhere.

I lost a bunch of other stuff, too, but nothing irreplaceable. A bunch of old ZNC logs that needed to get purged, along with a Bitlbee setup that needed to get updated. An outdated Gollum wiki that I don't use.

I got lucky, but I also learned my lesson. Revision 3 of my home infrastructure is built on top of Docker, for better or worse. It's quite alright, so far, I was able to bootstrap most of the stuff that I care about fairly quickly, since there are some decent minimal Dockerfile setups for a lot of the things that I am using right now. I built a little script to generate systemd service files for them to keep them running, I just need to make sure /etc/systemd/system gets backed up in my new regiment. Or maybe when I get a new server, I'll just make it CoreOS and be able to express these as a bunch of yaml. The nice thing about Docker, is that I explicitly have to mount state in to the container, which means that I can make sure they are in a directory on my NAS spinning disks that gets backed up by script offsite.

Respond to this note:

Ryan Rix is a computer infrastructure fanboy who dabbles in decentralized systems. Reach him on twitter as @rrrrrrrix, via email to ryan@whatthefuck.computer or on Facebook or on Matrix as @rrix:kickass.systems.