For the last attempt, the traffic team at slack went back to strengthen run books (especially for very risky rollbacks), increasing observability on DNS (route53 logs for full visibility of dns requests, breakdown by resolvers). @rdelvira #QConLondon

When encountering issues with the rollout, the traffic team decided to rollbck DNSSEC, with confidence of having done it many times in testing. That wasn't taking into account the 24h cache DNS resolvers worldwide. @rdelvira #QConLondon

Using netlog capture to debug issues in the chromium engine used behind the Slack client. @rdelvira #QConLondon

The challenge on DNSSEC is that it needs to be applied per domain, cannot be split per subdomain or be progressively rolled out. The rollout was done per domain, with slack.com being the last one...causing 3 different outages @rdelvira #QConLondon

"We planned DNSSEC carefully, with the necessary changes and replicated most of our DNS use cases... And you'll see later why I said 'most'..." @rdelvira #QConLondon

Here we go, last talk of the day on the production track at #QConLondon, with @rdelvira and "an entertaining outage story" (his own words) when slack rolled out DNSSEC

"If we have proper visualisation and better metaphors, we set much better conditions for our operators to be comfortable in understanding and responding to variations in our systems." @yurynino #QConLondon

Visual metaphors as alternative solutions to hard to read bar charts. For traffic, using nodes and edges of variable size - and for latency using color coded gauges. @yurynino #QConLondon

When surveyed, only 55% of engineers answered correctly when looking at a standard line graph - but they all got it right when viewing it in a city metaphor @yurynino #QConLondon

Introducing visual metaphors: displaying data and metrics in real world shapes and analogies. @yurynino #QConLondon

Chaos engineering isn't just about injecting the chaos, it's about increasing knowledge about our system (through observability) and overcome its weaknesses. @yurynino #QConLondon

Collecting metrics and signals are only one part of the solution - observability has to come with good visualisation, and engineering a solution for humans. @yurynino #QConLondon

"In our field, observability is about humans and about how humans interact with technology" @yurynino #QConLondon

Continuing the #QConLondon production track with @yurynino, and using visual metaphors to understand our production data in a different way.

Is 2022 the year of Continuous Profiling? @mhausenblas #QConLondon

Correlation remains a challenge with profiles, to link together all the different signal types, rather than dealing with multiple independent tools. @mhausenblas #QConLondon

Looking ahead, eBPF is becoming the unified collection method for profiles. Profiling standards seem to be converging towards pprof with its increasing support @mhausenblas #QConLondon

Continuous Profiling with Parca, open source project, with concepts and query language very close to Prometheus. @mhausenblas #QConLondon