Software Engineer. Speaker and writer on DevOps, InfoSec, SRE & Continuous Delivery. OSCP Pen Tester. He/Him
Ireland
Joined March 2009
- Tweets1.2K
- Following244
- Followers835
- Likes1.6K
Pierre Vincent retweeted
Not so long ago (okay, 23 years ago… 👴🏻), my dear friend @PierreVincent and I would build websites entirely offline, transfer the files to floppy disks (several of them, for redundancy), and… ride our bloody bikes 🚴🏻♂️ to the library to upload updates to free FTP servers!
Back in the day, the biggest bottleneck of the deployment pipeline was the library opening hours.
Not so long ago (okay, 23 years ago… 👴🏻), my dear friend @PierreVincent and I would build websites entirely offline, transfer the files to floppy disks (several of them, for redundancy), and… ride our bloody bikes 🚴🏻♂️ to the library to upload updates to free FTP servers!
And now today I'm the SRE on the Pagerduty rota. One would think my father should have warned me about this 😂
This is throwing me back 30+ years, when my father used to bring his Minitel home some week-ends, while being the on-call system admin!
As kids, we used to use it offline to draw line by line just using white squares and spaces 😅
For the last attempt, the traffic team at slack went back to strengthen run books (especially for very risky rollbacks), increasing observability on DNS (route53 logs for full visibility of dns requests, breakdown by resolvers). @rdelvira #QConLondon
You know you're in a really bad spot when you need to ask all DNS resolvers operator to clear their cache for your main domain 😱. "This was a very big spreadsheet..." @rdelvira #QConLondon
When encountering issues with the rollout, the traffic team decided to rollbck DNSSEC, with confidence of having done it many times in testing. That wasn't taking into account the 24h cache DNS resolvers worldwide. @rdelvira #QConLondon
Using netlog capture to debug issues in the chromium engine used behind the Slack client. @rdelvira #QConLondon
The challenge on DNSSEC is that it needs to be applied per domain, cannot be split per subdomain or be progressively rolled out.
The rollout was done per domain, with slack.com being the last one...causing 3 different outages @rdelvira #QConLondon
"We planned DNSSEC carefully, with the necessary changes and replicated most of our DNS use cases... And you'll see later why I said 'most'..." @rdelvira #QConLondon
"Who here tried to rollout DNSSEC?, Ok one person... Now how failed when trying to rollout DNSSEC? Welcome to the club!" 😂 @rdelvira #QConLondon
Here we go, last talk of the day on the production track at #QConLondon, with @rdelvira and "an entertaining outage story" (his own words) when slack rolled out DNSSEC
"If we have proper visualisation and better metaphors, we set much better conditions for our operators to be comfortable in understanding and responding to variations in our systems." @yurynino #QConLondon
Visual metaphors as alternative solutions to hard to read bar charts. For traffic, using nodes and edges of variable size - and for latency using color coded gauges. @yurynino #QConLondon
When surveyed, only 55% of engineers answered correctly when looking at a standard line graph - but they all got it right when viewing it in a city metaphor @yurynino #QConLondon
Introducing visual metaphors: displaying data and metrics in real world shapes and analogies. @yurynino #QConLondon
Chaos engineering isn't just about injecting the chaos, it's about increasing knowledge about our system (through observability) and overcome its weaknesses. @yurynino #QConLondon
Collecting metrics and signals are only one part of the solution - observability has to come with good visualisation, and engineering a solution for humans. @yurynino #QConLondon
"In our field, observability is about humans and about how humans interact with technology" @yurynino #QConLondon
