@EnigmaRotor reading this at lunch in a cafe near my house and I keep chuckling and smiling from ear to ear. @stefano is such a treasure 🙌🏆
Discussion
Loading...
Damn you Stefano.
You just spoiled a future Netflix movie.
Instead of watching in 2027 : `The Power Surge Heist`... we will have `The Uptime` with Stefano as sysadmin.
Following you so i can keep up with all the movies i will be missing.
@stefano Good for you. If next time, you could solve your problems without involving people who are sick at home with a serious family issue on top, that would be great.
@fennek@cyberplace.social pretty harsh. As an external provider, @stefano@bsd.cafe likely would have no idea their primary point of call was sick and/or had a family issue.
Really, if the primary point of call was out of action, it would be up to the business itself to arrange alternatives, allowing the sick person to stay out of action.
Really, if the primary point of call was out of action, it would be up to the business itself to arrange alternatives, allowing the sick person to stay out of action.
@paul @fennek I am familiar with that organization - and I know that the person (the one who was home sick, even if I didn't know he was home sick) has a deep sense of loyalty, but he is not reckless. If he hadn't been well enough, he wouldn't have gone. I even offered to go in his place myself. It is a healthy environment, not "that typical company" that exploits its employees. For obvious reasons, I cannot disclose details (and I work with several similar companies in different areas), but I can guarantee that everyone acted with the utmost respect for human decency. Fortunately, not all businesses operate like malicious entities that only think about harming their employees and collaborators.
I always strive to distance myself from such organizations, as they do not align with my outlook on life and the world.
@fennek Calling these 'my' problems is inaccurate; I am simply providing services to this company and I have no formal contract or obligation regarding this specific issue. I could have easily ignored the alert, especially since I wasn't aware the person in charge was out sick. Despite this, I offered to step in and handle it myself - even though it’s hours away - to help out and allow them to stay home.
@stefano Have to integrate this story into the pitch for our monitoring service 😁
@stefano wow. Great example. 🙏🏻
@stefano knowledge to take out a security system: aquired
@stefano I wasn't aware of this kind of problems with internal monitoring and the importance of external monitoring. However, I think is more important to monitor the monitoring server or to have one heartbeat of the monitoring system (external or internal). Because the external monitoring system could also fail without being aware of it.
@zako sure. Monitoring the monitor is more important than monitoring the services.
@stefano
I just want to say, this is one of those long, esoteric, fascinating, entertaining threads like you used to see on Reddit, and it's great to see here on the Fedi, minus all the Reddit bullshit. Good job everyone!
@stefano
even my new home alarm is coupled with a external monitoring alarm center that recognize tampering/sabotage jn addition to the "normal" alarms based on sensors etc. it costs a yearly subscription, but having a break in in the past, we considered it worthwile when we renovated our home.
@stefano wow, cool story and well done! 👍
And yes sometimes the truth is really better than fiction (thinking about about something a while back I was part in in my job that could have been easily from a badly scripted reality TV show. Can't go into details because of nda 🙈 )
@Tionisla Thank you. Yes, this is true. Sometimes things IRL are stranger than in fiction. And, if I look back, I've lived some incredible experiences. If I told it to my 20-year-old self, I would never have believed it
@stefano heh, yeah and even now you have to sit down rub your eyes and go "wtf". :-D
In the first sentence you mention a "data center", but such an attack would not work with a data center, to be one you need to have two buildings with independent power supply, at a safe distance, etc etc. I think this was at best a hosting room, not a data center.
@lorenzo @stefano
I think Stefano, the mild mannered barista of the BSD Cafe who posts pictures of sunsets and from his walks in nature is just a cover, and in reality he is a tough-as-nails secret military agent who's chasing cybercriminals around the globe.
See also his comment to my blog post about "just telling people to call the Barista" to make them crap their pants... this Barista has a secret! 🕵️
Internal monitoring can go dark.
External monitoring tells the truth.
Great example of why both matter.
@stefano AFAIK, professional alarm systems should function based on the principle that "if it doesn't send periodic alerts saying that everything is ok, and there's no scheduled downtime, then something clearly isn't ok, and somebody needs to be send to investigate it asap."
@miki I agree. In fact, their first idea is to check why they didn't call/intervene
@stefano The true horror part of this story:
> The office was closed for the holidays, but I contacted the IT manager anyway. He was home sick with a serious family issue, but he got moving.
Home for the holidays, sick, serious family issue?? Who cares! You know what's more important?? Keeping that data center up and running!
Glory to sacrificing yourself for the system!!
Or maybe get someone else next time.
@danvolchek to be honest, I offered him to rush there. But he refused and decided to go (he wasn't far from there)
@stefano zapping the power lines, eh? Looks like the perfect solution to my nuisance neighbors with the big loudspeakers.
@stefano And while not relying on internal monitoring make sure your external monitoring doesn't share anything with the monitored systems:
Different ISP, different cloud provider if in the cloud, no shared infra at any level
@stefano Thanks for all the info about the company's internal setup.
@OndrejZizka I never named the company 😉
@stefano
Hey! Thanks for the inside story! I love happy endings.
@stefano Great story and appropriate setup!
@stefano
Wow! Cool story
@stefano that's impressive. meanwhile I accidentally stumbled on your website:
You have shared many useful items in a thoughtful way. I appreciate it, and am glad to let you know. 😀
@stefano This immediately brought to mind coming into the office after a holiday weekend in 2005 and finding “my” computer room dark. I found our infrastructure manager, who told me that they had an unexpected power outage over the weekend. Confused, I said “But how is that possible? We have multiple feeds and a huge uninterruptible power supply!”
I will never forget his response, delivered in his thick Scottish brogue: “Yes, we do. But it doesn’t do much good when the UPS catches fire.” 😳
@thegaffer @stefano That reminds me of an incident that happened at work. We have multiple sources of electricity and generators, but none of that matters if the room with the UPS and power controller where all the power sources meet floods from an overflowing toilet a floor above 🙃😅
Whoopsie daisy!
I just finished bypassing all the network switches in the closets from that circuit when they managed to bypass it and catastrophe averted.
That was a fun night! /s
@stefano This is a pretty important knowledge to have!
@stefano why do you assume that via 4G there would be connectivity? I don't get this part, what am I missing?
@stefano thanks for sharing this.
@stefano thank you for this knowledge, I have boosted it for reference for others. 🤗
@stefano Cool story bro, but it's too fictional, I'd say.
First off, as a Ukrainian, I know that powerlines can survive "the spikes" by just cutting the power at the very input. No damage to equipment behind the input electric circuit breaker, nope. You just get damaged input.
Next, I used to work in a bank. And here we had a clear requirement for data storage center: more than one power input -- is a must.
@stefano
Third, given it's a data center, power consumption is probably tens of KW. The "gang" could probably be killed in action playing with it.
Fourth, if there is a power spike and cut off, it won't go unnoticed by those who control power lines. They will be the first on site to see what happened.
@stefano There was an attack a few years back near here where they dropped burning rubbish into manholes around a a data centre; the theory at the time was it was to try and cut off some CCTV or alarm monitoring for something. Well caught!
@stefano I wonder how they generate a big enough power surge.
@stefano 10+ years ago i started volunteering at a festival. Everything was new that year including the small outdoor racks for the area field routers (Juniper MX80). They barely fit but we managed. The racks were left in the sun in the summer. It was only when we enabled Observium (LibreNMS predecessor) that graphs almost everything it gets from SNMP that we discovered the inlet temperature was getting close to 80 degrees C. #monitorallthethings
@lasseleegaard true. I'm using my switch's fan speed to understand if my home office room is too warm
@stefano About 15 years ago, the place I worked had a supercomputer. One night, the aircon in the machine room failed. The machine kept computing, and the temperature rose. It rose *quite a lot*.
Sadly, the first thing to fail from the heat was the core switch for the room. You know, the one that handles all of the network for everything in the room. Including the temperature alerts.
It was finally spotted about 8am when the security patrol wondered why the door shutters were so hot.
@darkling nice story! Unfortunately I had to manage something like that, too (A/C broken - switch dead, etc)
@stefano Fortunately, the only thing that did fail after the aircon was the switch. (And a pair of ear muffs which had been hanging on a metal rail -- they'd melted).
The fire brigade turned up, checked everything, and ran some big positive pressure fans to get airflow through the room from one door to the other to cool everything down.
@stux thank you! Yes, that's a very wise approach. I have some internal and external monitoring tools. And the monitoring tools monitoring the monitoring tools, with different technologies (so a bug won't hit all the tools at the same time). Yet, I always feel I need moooarrr monitoring 🙂
@stefano you’re a hero Stefano! As your Fedi friend and documentary filmmaker I hope I get preferential treatment when one of your amazing stories gets optioned for a film 🤗
@_elena Thank you! Sure, I will 👍
But, to be honest, I don't think any of those stories will ever be a film.
The big, most scary one is yet to come, anyway...
@stefano I don't know, you told this short story like a pro. Starts out, ya, data center suddenly goes dark over the holidays. UPS fails, kinda of ya, ya , still interesting then you introduce the gold, two-meter thick walls, professional thieves, wow, that's some drama! Although, I wonder how they were able to send such a massive power surge down the lines and why the bus mains didn't blow before the equipment was damaged? Looking forward to your next tale!
@stefano This is such a good, if niche, example of "paying attention to the fundamentals and the alerts covers all sorts of things you'd never imagine happening."
Thanks for sharing.
@neurovagrant thank you! My rule is: we need moooarr alerts, as you never know how and when (not if - we know it will happen) your alertil system will break.
@stefano so refreshing to read a quality tech tale on Mastodon. Thanks for sharing!
@bojanlandekic thank you! I'm just trying to spread some real life experiences
@stefano it is the criminals among us who make life difficult for all. Not even the greatest sci-fi authors have been able to imagine how beautiful and fun a future we all would have without them!
@stefano Uptime Kuma instance from waaaaay downtown!!!
You are the hero I aspire to be!
@n_dimension ahah thank you, but I'm not a hero. I'm just doing my job anche checking the alerts.
@stefano feeling of :xkcd:`705` intensifies 😁
@stefano Sounds like a case of either good design or *very* good luck too that the UPS took the brunt of it.
We can't protect against everything, but we *can* have an idea for what to do when the unimagined happens.
@stefano that advice also applies to monitoring scheduled backup jobs (or any other automated process). I use a service that emails me if I don't hit a specific URL roughly every 24 hours, and I hit that at the end of my backup job if it was successful.
Better than finding out the hard way at some point in the future that something happened with my backup job, preventing it from running for the last month.
@rhoot exactly, that's the right approach. I'm using something similar.
@stefano Only in BSDcafé can you read actual techno thrillers like this.
@EnigmaRotor Sometimes the lights are low and the atmosphere is dark...
@stefano Stefano Jones P.A. a very noir series.
@EnigmaRotor reading this at lunch in a cafe near my house and I keep chuckling and smiling from ear to ear. @stefano is such a treasure 🙌🏆
@_elena@mastodon.social When you direct the movie, can I star as the legendary @stefano@mastodon.bsd.cafe ?
1 more replies (not shown)
@EnigmaRotor /me making coffee in the dark, while whispering some IT horror stories
@stefano @EnigmaRotor Competing with @mwl are we?
@Dianora @EnigmaRotor @mwl I could never compete with the Mentor. He's more of a spiritual guide
2+ more replies (not shown)
@stefano Oh, if genre is horror, then don’t forget to tell the tale of the guy who pronounced “Microsoft” 3 times before his mirror. What happened next, the blue mirror of death, is frightening to the bones.
I am quite keen to look into Uptime Kuma. Our current monitor is antiquated.
On a side note, you guys are hilarious! I genuinely had a good laugh at your comments.
1 more replies (not shown)
@stefano nice story! and, yeah, internal monitoring is a must, but you also need an external one, operated by someone else than yourself.