Kia ora.
The big piece of news over the weekend was the CrowdStrike outage, as an update pushed to an offering from the cyber security company crippled businesses around the world.
I work in IT as part of my day job, and there’s a couple of golden rules around deploying any sort of change or patch:
Never deploy on a Friday; you’ll be left with limited support if something goes wrong
Make sure your change is well-tested, with robust testing plans and scripts
Make sure your change has been reviewed by others, including somebody who knows first-hand exactly what the change might impact
Now, it seems pretty likely that at least one (if not all) of these rules were broken, and there’s probably a software developer feeling pretty sheepish somewhere. These are what you call “Resume Generating Events” - someone, somewhere is to blame, and they’re probably getting fired.
I know the feeling.
Large companies often use what is called an ERP (Enterprise Resource Planning) system to run their business - these can do all manner of things from managing finances to generating sales to procurement of inventory to everything in between. The company I work for uses one such system; and in my first six months with them, I crashed it. For a day.
Long story short, I was seeing some unusual behavior in our data, so went to test a possible cause to see if I could replicate the issue - and accidentally did my test in our Production environment, basically taking the business offline for the better part of a day. Goody.
Thankfully, the issue was resolved without too much gnashing of teeth, and I immediately admitted to my mistake - I’m not the type to try and just cover my butt, and there was no point anyway, as telemetry data would identify me as the user who set the whole thing in motion. Thankfully, outside of a stern talking-to there were no further repercussions, but I did learn a very valuable lesson and was MUCH more careful from then on.
So, what’s the dumbest thing you’ve done at work? Or what’s your biggest screw-up? Funnily enough, the two don’t always overlap...
Matty’s track of the day: Buggin’ Out, by A Tribe Called Quest
And now for your links:
The Athletic ($$$) on Djed Spence’s chances of getting back into Ange Postecoglou’s good books
Alasdair Gold shares his talking points from Tottenham’s win over QPR on the weekend