Builds and operates billion-QPS distributed systems. Senior Director @MongoDB. Core infra, databases, reliability, and engineering management. Views are my own.
Dublin, Ireland
Joined May 2009
- Tweets3.8K
- Following445
- Followers3.3K
- Likes11.5K
A lesson on the value of redundancy for improved uptime: Either your bank should have a multi-zone deployment, or you should have two banks.
🚨 Google lost power in London this morning and Revolut, a bank with 80 million customers, went down with it.
I've spent the morning reading what Revolut users went through. It's wild.
People stranded at petrol stations, unable to pay for fuel or a taxi to the airport.
One traveller couldn't get out of a train station in Japan.
A 71-year-old woman stood at a supermarket till for an hour. Her kid had to drive 30km to bring her cash.
Card payments, transfers, ATM withdrawals, currency exchange, share trading. All down.
Revolut's statement said "all our customers' money and data are safe."
Safe. You just can't touch it.
Google's explanation so far is a "power event" in one zone of its London cloud region. Its advice to affected customers was to move their workloads to another zone or region "if feasible."
Must be nice. Revolut customers couldn't even move £10.
In July the UK government put Google Cloud, AWS, Microsoft and Oracle under direct Bank of England oversight, because one outage at any of them could shake the entire financial system.
12 days ago the Bank of England ran a drill with the UK's 38 biggest banks and market operators. The scenario they practised was a global cloud outage.
And exactly one year ago today, AWS went down and took Lloyds and Halifax with it.
Same date. Different cloud. Lloyds was having problems again this morning.
Revolut spent five years earning a UK banking licence.
Bitcoin has been running for almost 18 years. It doesn't have a London region.
A special-purpose database optimized for a specific workload can indeed do better than a general multi-purpose one. The hard part is to operate this database with a similar reliability posture.
Should review the cost of operating this and the uptime figures in a year's time.
Highlights from the Aug 25 Buildkite incident:
- Deployment caused an EKS pod to scale up.
- Infra couldn't scale up in time. More scale up requests piled on. New pods put extra pressure on internal DNS.
- Discovery slowed down by 780X and went down with a huge blast radius.
Full incident write up: buildkitestatus.com/incident…
Today I started as a Senior Director of Engineering at @MongoDB. I'll own Observability Engineering. Ping me if you have pain points and/or ideas!
Today was my last day at Amazon after 6+ years. I worked on some of the world's largest scale distributed systems and learnt things I wouldn't be able to imagine before joining Amazon. I'll miss the daily thrill. Next week I'm taking on a new challenge and looking forward to it!
Highlights from the Aug 17 Github incident:
* Load balancers got overloaded due to autoscaling config being insufficient.
* Cascaded to more LBs, causing auth latency and failures.
* Retries made it worse and recovery harder.
So, yet another one for the Retry Storm Hall of Fame.
The RCA for the outage we had at GitHub yesterday is live.
We're sorry it happened and hope this answers your questions about it. The team is already working on the actions listed here to fix it.
githubstatus.com/incidents/z…
The deeper problem is operational risks are probabilistic and the same risk may not materialise repeatedly, which makes it difficult to defend a big strategic investment to fix it. This turns the ops into an endless whack-a-mole via small tactical improvements that won’t hold up.
Seniority is less about the code you write but more about the size of the outcome you're accountable for and how much ambiguity you can navigate while delivering it.
I wrote about how expectations change as engineers become more senior within an organization.
Link below:
eonem.substack.com/p/how-do-…
Evren Önem retweeted
I follow the AI & math discourse a bit - without understanding the actual math. I thought I might offer some thoughts as a scientist in structural biology - a field transformed by various tech over the years, and recently by AlphaFold. It might perhaps interest @nasqret et al.🧵
I'm currently returning to Toronto from a summit on the future of mathematics, at OpenAI. @SebastienBubeck asked me to talk a bit about the future we'd all like to avoid, where humans are mathematically disempowered. @Jacob_Tsimerman advised us to try to prioritize detail over correctness, and I have no doubt that I succeeded in deprioritizing correctness.
I tried to find a title that wasn't too bombastic:
* How should it decide what scaling action will help the situation at hand, e.g. a hot key read scenario so create more read replicas, or a hot key write scenario so re-partitioning is required?
Evren Önem retweeted
The elimination of engineers was one of the wilder hypotheses out there. Just like absurdly wrong. We just gave engineers a power tool that can accelerate the development of whatever we want.
Of course their value not only remains critical in that world, but actually goes up in many domains because we can apply engineering to far more work than before.
If you’re trying to automate drug discovery, you need engineers. If you’re trying to automate manufacturing, you need engineers. If you’re taking on larger and larger software projects, you need engineers.
This will happen in domains beyond engineering too. AI causes companies to take on way more work than before, which leads to needing more experts to oversee the work.
Even as models get better and better, they can be better utilized by the experts in those fields than the novices. Great time to be an expert.
One take-away seems to be that if your deployment requires taking hosts down, the process should first check if the remaining capacity suffices to serve the traffic.
githubstatus.com/incidents/q…
“My services are well and healthy despite not reviewing a single line of LLM-generated code”. Meanwhile, the production system:
Code != value. You can ship 1M lines a day and still ship absolutely nothing anyone needs, or an unreliable product that breaks often. My favorite species is the “I don’t waste time reviewing anything, I use that time to ship more” bros. These guys ship tomorrow’s archaeology.
we're in a strange transitionary period where people are confusing what was historically scarce with what is actually valuable.
for the last few decades, knowing how to code was highly correlated with making a lot of money and building valuable things.
so now that AI has given everyone the ability to code, the obvious reaction has been to start building software.
but the value of coding was partially downstream of the fact that most people couldn't do it.
which makes vibe coding kind of paradoxical - millions of people are rushing to acquire the output of a skill at the exact moment that output is becoming abundant.
i think the trend dies down once people realize that not knowing how to build software was never really their problem.