Discussion
Loading...

Discussion

  • About
  • Code of conduct
  • Privacy
  • About Bonfire
DocYeet :verified:
@docyeet@mastodon.halis.io  ·  activity timestamp 9 hours ago

Outage of the year award goes to 🥁🥁🥁

Synapse !

The reverse proxy in front of my s3 bucket for backups went out because the host node ran out of space, so no backups could be made for the last 6 days…
On top of that, it seems like the PVC holding synapse’s media got reset for some odd reason ?

We couldn’t share pictures so I thought something else was off, the only real problem came up when I tried updating Synapse to the latest version, which through out various weird errors

Now trying to bring it back up…

#selfhosted #homelab #synapse #elementhq #s3 #servers #longhorn #k3s #kubernetes #backups

  • Copy link
  • Flag this post
  • Block
DocYeet :verified:
@docyeet@mastodon.halis.io replied  ·  activity timestamp 9 hours ago

Good thing is, the last, 6 days old, backup is also the most up to date backup I have, ‘cause we couldn’t send media either way 🤷‍♂️

The Longhorn volume is getting rebuilt as we speak, let’s see if that works
At least it looks bigger than the initially displayed 111Mio…

  • Copy link
  • Flag this comment
  • Block
DocYeet :verified:
@docyeet@mastodon.halis.io replied  ·  activity timestamp 9 hours ago

I had been thinking about changing my approach to S3 backups, getting rid of the out of cluster reverse proxy and second Wireguard network, and only use the main k3s wg network to distribute backups to the NAS… That might be the triggering opportunity

  • Copy link
  • Flag this comment
  • Block
DocYeet :verified:
@docyeet@mastodon.halis.io replied  ·  activity timestamp 2 hours ago

And now it's back up and running...

Recreating ~50Gb worth of media took longer than expected because of the reverse proxy bottleneck...

And resetting the db at the same time while forgetting to specify the recovery point didn't help either..

But that's just yet another example for why BACKUPS TO MATTER

  • Copy link
  • Flag this comment
  • Block
Log in

BT Free Social

BT Free is a non-profit organization founded by @ozoned@btfree.social . It's goal is for digital privacy rights, advocacy and consulting. This goal will be attained by hosting open platforms to allow others to seamlessly join the Fediverse on moderated instances or by helping others join the Fediverse.

BT Free Social: About · Code of conduct · Privacy ·
Bonfire community · 1.0.0 no JS en
Automatic federation enabled
  • Explore
  • About
  • Public Groups
  • Code of Conduct
Home
Login