Incident report: * Previously in the week it was noticed that Cilium had an update to 1.19.0 * Upon further inspection, this looked to be a pre-release, so it was left alone, cilium is a load-bearing component and should only be touched with care * At some point in the last week I forgot about that and hit merge in forgejo, not an issue, since argocd won't auto-sync any load-bearing components (cilium, forgejo, argo, cert-manger, and a handful of others) * Over the last few days I have repeatedly restarted various components while troubleshooting some OIDC issues. Including ArgoCD and Forgejo * This caused a few sync errors or argo state refresh errors as pods were unceremoniously exploded * At some point during this time Cilium ended up out of sync/errored * By this point I had forgotten about the cilium major`*` update * I hit sync to clear that out and see what is wrong. Everything is green, and nothing breaks. * BGP sessions continue * I go about my afternoon * BGP sessions expire, causing immediate issues. since my old config was depricated * I start the investigation with DNS, since the TV stopped playback and sites stopped loading on my laptop * Yup, DNS is down. * But not from my dev console, that means ad-guard DNS is down. * Ad-guard DNS is throwing errors connecting to quad9 via DoT, I am not sure the cause of this, maybe the UI has a clue * ad-guard DNS ui isn't opening, oh. No cluster-based site is opening actually. * Confirmed, all LB services are down, must be BGP related * Looking in the Mikrotik router I see two BGP sessions, so I restart the BGP service on the router, they drop and don't re-appear, must have been stale on that side. * Restart cilium to see logs * BGP config error? Wait, did cilium update??? > ⚠️ You may need to take action during upgrade to Cilium v1.19 if you use Network Policies, Cluster Mesh, LoadBalancer IPAM or BGP. * Oh right, I've been seeing depreciation warnings about the BGP config for ages now * After converting the docs format I am rewarded with the TV abruptly starting the playlist, time to commit this to git and move on, phew #Homelab #Networking #Kubernetes #BGP