Table of Contents

Recently I setup Authelia to use as an SSO (OIDC) for lots of my compatible applications. With Authelia running, it was time to start implementing it into an application. I’m a huge fan of Bookstack, and decided to start there. But, before making a big change like this, I wanted to make sure I had a good, recent backup.

Background

If you’ve read through some of my older posts, I created a backup script that utilizes restic and runs nightly on all of my servers. This backups up databases and important files. But just to be sure, I decided to double check that the backups were actually working.

Let’s look at our snapshots (backups)

restic snapshots

Hmmm, anything in the error log?

tail restic.err

That is not good, I’m probably about 2 months behind on backups!

OK, let’s make sure the databases have at least been backing up correctly:

restic ls -l --human-readable latest /tmp/dbbackup

OK, let’s add the Firefly-III database back into my backup script, and run a backup to see if it works now:

Well shit.

TLDR;

Looks like not only do I not have recent backups of my Bookstack DB, but I also don’t have recent backups of another application I forgot to add to my script. Upon digging into the restic error log, I also noticed I wasn’t backing up my main personal data directory!! This last one was due to a failed hard drive and subsequently changing file paths to a different ZFS pool.

Remediation

Turns out, that when I switched my Podman containers over to the new Quadlet system (awesome btw!), I changed how I was passing in the database password. Thus, the script was unable to create a backup file.

I added back my Podman secrets, rewrote the script to allow for different password methods, and re-ran the backup.

Postmortem

OK so what went wrong, and how can I do better in the future?

First, I have a reminder that pops up once a month to check my backups. I have been dutifully ignoring it the past few months as other things have come up. But that begs the question: How should I be checking my backups? A few ideas come to mind:

• Check for existence of certain files (like the database backups)
• Verify regularly I’ve included new applications and changing directories
• Use notifications for bad backups

The first point could easily be taken care of with a script, and perhaps I will write something small just to notify me if a file is missing or less than an expected size. Not foolproof, but this could help to identify issues.

The second point is on me. I intentionally don’t backup EVERYTHING on the servers, just the most critical files. But I need to be reminded that part of my regular “check your backups!” routine, is to double check everything is still the same.

Finally, I have in place a system for backup notifications… Which is apparently not working. So that’s my next order of business. Why are my ntfy notifications not working with this script?

Conclusion

This was just a short reminder to properly check your backups! It should also be noted that just because a file exists, doesn’t mean it’s a “good” backup. Files can get corrupted, database backups may not be complete, etc. The best way would be to download a db backup file, import it into a fresh db, and verify that everything works as expected.

But that’s for another day. Comments and suggestions are welcome in the section below!