Post

@stefano
Wow! Cool story

@stefano that's impressive. meanwhile I accidentally stumbled on your website:
You have shared many useful items in a thoughtful way. I appreciate it, and am glad to let you know. 😀

@stefano This immediately brought to mind coming into the office after a holiday weekend in 2005 and finding “my” computer room dark. I found our infrastructure manager, who told me that they had an unexpected power outage over the weekend. Confused, I said “But how is that possible? We have multiple feeds and a huge uninterruptible power supply!”

I will never forget his response, delivered in his thick Scottish brogue: “Yes, we do. But it doesn’t do much good when the UPS catches fire.” 😳

@stefano This is a pretty important knowledge to have!

@stefano thanks for sharing this.

@stefano thank you for this knowledge, I have boosted it for reference for others. 🤗

@stefano Cool story bro, but it's too fictional, I'd say.
First off, as a Ukrainian, I know that powerlines can survive "the spikes" by just cutting the power at the very input. No damage to equipment behind the input electric circuit breaker, nope. You just get damaged input.
Next, I used to work in a bank. And here we had a clear requirement for data storage center: more than one power input -- is a must.

@stefano There was an attack a few years back near here where they dropped burning rubbish into manholes around a a data centre; the theory at the time was it was to try and cut off some CCTV or alarm monitoring for something. Well caught!

@stefano I wonder how they generate a big enough power surge.

@stefano 10+ years ago i started volunteering at a festival. Everything was new that year including the small outdoor racks for the area field routers (Juniper MX80). They barely fit but we managed. The racks were left in the sun in the summer. It was only when we enabled Observium (LibreNMS predecessor) that graphs almost everything it gets from SNMP that we discovered the inlet temperature was getting close to 80 degrees C. #monitorallthethings

@stefano About 15 years ago, the place I worked had a supercomputer. One night, the aircon in the machine room failed. The machine kept computing, and the temperature rose. It rose *quite a lot*.

Sadly, the first thing to fail from the heat was the core switch for the room. You know, the one that handles all of the network for everything in the room. Including the temperature alerts.

It was finally spotted about 8am when the security patrol wondered why the door shutters were so hot.

@stefano Great job!

This is why is always run up time on different servers in other places!

Perfect!

@stefano so refreshing to read a quality tech tale on Mastodon. Thanks for sharing!

@stefano Uptime Kuma instance from waaaaay downtown!!!

@stefano

You are the hero I aspire to be!

@stefano That’s a rather cool war story. Great for a lecture.

@stefano Sounds like a case of either good design or *very* good luck too that the UPS took the brunt of it.

We can't protect against everything, but we *can* have an idea for what to do when the unimagined happens.

@stefano that advice also applies to monitoring scheduled backup jobs (or any other automated process). I use a service that emails me if I don't hit a specific URL roughly every 24 hours, and I hit that at the end of my backup job if it was successful.

Better than finding out the hard way at some point in the future that something happened with my backup job, preventing it from running for the last month.

@stefano Only in BSDcafé can you read actual techno thrillers like this.

@stefano nice story! and, yeah, internal monitoring is a must, but you also need an external one, operated by someone else than yourself.