Tag - backups

Linux, Backups, and Trust

Every 2 years, Ubuntu releases a new LTS. These Long Term Support versions are supported for 5 years with bug fixes, security updates, and generally are a little more conservative with new features or overhauls. I generally wipe my computer and start fresh every 2 years as these LTS releases out, then I go back to work and keep my machines stable and clean for 2 years until the next one. The couple of weeks before and after the LTS release however, I experiment with other distributions, window managers, workflows, applications, etc. It's my window of seeing what else is out there before I settle in for stability.

This time around, I changed how I do backups. For years, I have relied on Crashplan along with their cloud storage for backups, management, and storage. I have nothing negative to say about them. They have saved me plenty of times, the application has native support for Linux, and works really well. On Ubuntu 20.04 however, I ran into a small problem with restores, which I'm sure was fixable, but it did lead me down a path where I no longer use Crashplan.

I've switched to Cloudberry's Backup software and send my data to Backblaze's B2 object storage. This combination works really well so far. Configuration is powerful, backups are fast, and restores work great from the Cloudberry app. Everything from the encryption standpoint seems to be above board as it was with Crashplan also.

Flameshot Configuration Screenshot
What my backup plan looks like at a high level.

In my backup plan, I start by selecting my entire home directory /home/sajan, then excluding a bunch of directories that are noisy and are not needed for backup. I also select /var/lib/zerotier-one as this directory holds all of my Zerotier keys and configuration.

If you aren't familiar with Zerotier, it's an overlay network protocol that is used for VPN. Our team at IP ArchiTechs has started to use this consistently both internally, and for clients as a quick, secure method of getting access to devices without the need for a traditional VPN with advanced capabilities due to how it transmits packets and performance with larger MTUs.

This means being able to backup and restore your keys is an easy way to distribution hop or recover from disaster without having to ask all the administrators of the networks you were connected to re authenticate your new Zerotier ID. Which can get annoying.

With Crashplan, backing up this directory wasn't an issue (or maybe it was as we'll find out). The Crashplan service ran as root and was able to gobble up anything on my system at any time. Cloudberry on the other hand, runs the backup job as the user that created it. Which was actually quite refreshing and I enjoyed that piece, even though Cloudberry had management processes running as root. In this case, me. As Cloudberry started sending me warning emails, I realized that my user doesn't have permissions to read all of the files in /var/lib/zerotier-one. So let's fix this.

My first glance at this, I didn't look too closely. My instinct told me to just add my user to the zerotier-one group and call it a day. Simple enough, usermod -a -G zerotier-one sajan. Restarted all the processes, and I still had permission issues. Looking at the above files and directories closer, you see that authtoken.secret, controller.d, and identity.secret can only be read by the owner, the group isn't allowed. Which makes sense.

The next option would be to manually set the group read bit on those three items and be done with it. Philosophically, that didn't sit well with me and the other two options went through my brain. However, even as I type this, I guess I'm not entirely sure which option makes the most sense.

Option #1: Allow Group Read

These files are sensitive. Anyone that gains access to them would essentially be able to blindly connect to our corporate network. From Zerotier's perspective, no user on the system should have access to these files. I agree with this, so skip for now.

Option #2: Run Cloudberry As Root

This seems like the best options as I've already put Cloudberry in my circle of trust and it already has processes running as root, even if the individual backup workers are spun up as the user that created them. I could create the backup plan as root, and it will be able to process all the files.

Option #3: Set ACLs On The Files

Here you can see the standard perms, setting the ACL, and what it looks like afterward.

This was a viable option as well, and I gave it a shot. It does work. However, now I'm worried about any random third party application that runs as my user being able to access the sensitive information. So I've removed it for now.

Furthermore, this option doesn't make a whole lot of sense over option #2. At some level, Cloudberry is going to have to read these files. So achieving that by also letting all of the sajan user processes read these files also makes no sense.

In Conclusion

Either I get away from proprietary backup software that needs to run as root and switch to an open source utility and manage it all myself, trust Cloudberry and its local encryption, or not back these configurations up at all. However, it did make me think the bucket I more or less blindly put Crashplan in. Since Crashplan ran as root by default, I was never faced with this problem or a prompt to think about it.

Read more...