
Incident report:

* Previously in the week it was noticed that Cilium had an update to 1.19.0
* Upon further inspection, this looked to be a pre-release, so it was left alone, cilium is a load-bearing component and should only be touched with care
* At some point in the last week I forgot about that and hit merge in forgejo, not an issue, since argocd won't auto-sync any load-bearing components (cilium, forgejo, argo, cert-manger, and a handful of others)
* Over the last few days I have repeatedly restarted various components while troubleshooting some OIDC issues. Including ArgoCD and Forgejo
* This caused a few sync errors or argo state refresh errors as pods were unceremoniously exploded
* At some point during this time Cilium ended up out of sync/errored
* By this point I had forgotten about the cilium major`*` update
* I hit sync to clear that out and see what is wrong. Everything is green, and nothing breaks.
* BGP sessions continue
* I go about my afternoon
* BGP sessions expire, causing immediate issues. since my old config was depricated
* I start the investigation with DNS, since the TV stopped playback and sites stopped loading on my laptop
* Yup, DNS is down.
* But not from my dev console, that means ad-guard DNS is down.
* Ad-guard DNS is throwing errors connecting to quad9 via DoT, I am not sure the cause of this, maybe the UI has a clue
* ad-guard DNS ui isn't opening, oh. No cluster-based site is opening actually.
* Confirmed, all LB services are down, must be BGP related
* Looking in the Mikrotik router I see two BGP sessions, so I restart the BGP service on the router, they drop and don't re-appear, must have been stale on that side.
* Restart cilium to see logs
* BGP config error? Wait, did cilium update???
> ⚠️ You may need to take action during upgrade to Cilium v1.19 if you use Network Policies, Cluster Mesh, LoadBalancer IPAM or BGP.
* Oh right, I've been seeing depreciation warnings about the BGP config for ages now
* After converting the docs format I am rewarded with the TV abruptly starting the playlist, time to commit this to git and move on, phew

#Homelab #Networking #Kubernetes #BGP