i said this before to my friends but i guess it’s worth posting here: i truly did not expect the scope of how bad the twitter acquisition would go. i genuinely thought musk was going to talk big and whine before the deal, then slowly over time wreck the site with a series of poor decisions.
the pace and magnitude of the fuckups he managed to pull in such a short period of time is unprecedented. it’s SO fucking bad. since i at least have some technical understanding behind running a social network, i feel like i’m staring into the sun every time a new headline comes out. there is no hyperbole when it comes to describing how badly elon musk is fumbling the bag.
if you’re absorbing this news and think that maybe it’s not as bad as it sounds, i assure you it is probably WORSE than what we’re hearing. this isn’t just haters amplifying small mistakes, musk is weaving the goddamn rope.
it’s a goddamn tragedy that billionaires never really suffer for their choices, but i think this is as close as we’re gonna get before we start beheading them in the streets
The really funny thing? For as bad as it gotten, as quickly as it has… None of the current problems are what’s going to kill Twitter.
Like yes, users are leaving because they have no trust in the site. Executives are resigning. Advertisers are pulling their ads. Companies are getting impersonated and are going to sue them. The FTC is going to fine them a bazillion dollars.
But all those things take time. They’re all bad, bad things, but they won’t kill Twitter overnight. They’re a slow bleed, an eventual demise when a court case resolves, that kind of thing.
But one of the first things Musk did was fire much of the SRE teams, the on-call staff. These are the people who work to keep the site up and running, and they’re the kind of people you pay to do nothing most of the time because what you’re really paying them for is to be there at 3am when some server goes down and you need someone to log in and quickly fix it.
They’re the server administration equivalent of firefighters. Musk fired THOUSANDS of them. And just like getting rid of all the firefighters, this might seem to be going fine for a little while… Because there’s not a fire yet. But once there is, oh boy. Things are going to get Real Exciting fast.
Twitter is one bad day away from just ceasing to LOAD at all. Someone hacks a server, someone famous dies and a system gets a little too overloaded due to the traffic spike, or hey, maybe a server just fills up with junk files because the person who had been manually cleaning them up lost their job a week ago… And Twitter dot Com is just gonna be a big message saying “sorry we couldn’t load that information, try Twitter/help or try again later”.
For hours. Or days. Because the other thing about firing all your SREs… They’re also the people who know the weak points of the system, and know where things go wrong when they do go wrong. You can’t just replace them with any warm bodies, elm sku just had a ton of institutional knowledge walk out the front door and it’s not coming back.
Yes, Twitter is the titanic, but we haven’t even hit the iceberg yet. The CEO has demanded full speed on a dark night and no one has the binoculars. It’s only a matter of time.
This is it, there’s a lack of operational knowledge and twitter is careening to the point where it’s going to be necessary.
This is all made up and I don’t have any direct knowledge but i’ve been in operations for almost two decades now and this is how things work in the general case:
[ID: discord text from me, Tevruden stating:
from a guy who’s very computer: sometimes things just work because dave goes in and runs a script that has some third order effect that if it didn’t happen everything would break
said script cleans a cache system that the ‘can you see this tweet api checks and since the cache isn’t working properly sometimes you CAN see the tweet
or the script was running as a scheduled job in an account that’s no longer allowed to login so cant run]
Dave never had a chance to fix this because it was working fine as-is and if he changed the process that might fuck something up so that was relegated to his copious amounts of free time.
Dave doesn’t think about this at ALL. He’d point it out if he was leaving because he found a new job, but if he’s getting fired, he’s busy processing this and it’s not gonna come up, if it was as sudden as it it appears to be for twitter employees getting laid off there is no place for it to come up.
At my current job, my direct predecessor was laid off in 2017. I think we found the last gotcha in 2020. Our shit is not as complex as twitter.
They’re going to be in for a bumpy ride for YEARS.
Oh also? Their first test is December 12. (Two guesses, It’s Not DNS)