It’s never been easier to ship code, but testing has still been a massive bottleneck for us… so I decided to change that 🫡 With @Cap, we’re currently making a full transition to GPUI, and a screen recorder is a strange thing to test reliably, because you can export a perfectly playable video that has the wrong frames in it. Everything appears to work until you look closer! I don’t want to ship slop, and any regressions are my worst nightmare. The faster we’re shipping code (and generously from contributors!) the more time I find myself spending trying to 100% prove that we haven’t broken something So I’ve spent the last couple of days building an automated testing harness for Cap… 🥁 It runs our actual media code through repeatable scenarios whenever a relevant PR changes it and runs simultaneously across macOS, Windows and Linux. Also made it so that a draft gets a smaller suite so feedback comes back quicker (this is great for agents!) The interesting part though is how we know whether the video is correct… To do this, we generate test videos with an ID encoded into every frame, alongside audio markers placed at exact moments. These are our video fixtures. When we put one through Cap’s exporter, we can decode the result and check which frames actually made it through So a cut would change which frame should appear next, or maybe a speed change alters when it should appear. We calculate that independently of Cap’s implementation, so the test has something to check the output against For example, if frame 125 appears where frame 126 should be, we will know. We don’t need someone to sit there watching for it! Performance gets measured too. The PR and its merge base are built on the same machine, then benchmarked in alternating runs after warming up. That gives us a comparison of CPU time before and after the change, while reducing the noise you’d get from comparing unrelated CI runs We’ve moved most of this onto @useblacksmith . One Windows test batch went from 9m 52s on GitHub-hosted runners to 2m 59s, about 3.3x faster (!!) Another went from 5m 08s to 45s. Nearly 7x faster for that test step (!!!) What I hadn’t quite expected though was how much we’d fix while building the actual harness itself In one crash recovery test, the footage lost went from 11.03 seconds to 1.03 seconds. We also fixed a decoder bug that could show a frame from 89 frames earlier after seeking, without meaningfully increasing CPU usage There’s something almost self healing about having this in a loop. We get a reproducible failure, fix the cause, and the test stays behind to catch it if it ever comes back. Each fix leaves us less dependent on remembering everything that’s gone wrong before I think now that we have this in place, the logical next step is to scale it up and run performance/regression testing autonomously from things like Sentry logs and user feedback. Definitely going to commit a decent amount of $ to it. I probably need at least another 4x Claude subs! Really excited to ship the improvements we’ve already made just from building this. With the harness mostly in place, I can move much faster on my own changes and contributor PRs. Being able to make that progress without the constant worry of breaking someone’s recording feels pretty fucking good I can’t lie 🤩