Building #ExperimentOps @remyxai Experiment orchestration for AI teams get outrider: https://xitter.gq/t.co/HLjSzx4znL
San Francisco, CA
Joined October 2018
- Tweets5K
- Following445
- Followers938
- Likes1.9K
Pinned Tweet
SpaceOm beats NVIDIA's Cosmos-Reason1-7B on SATURN spatial reasoning benchmarks using half the parameters!
Reference: "SATURN: Symbolic Spatial Reasoning for Multi-Perspective Grounding" (arxiv 2606.22694)
Try the weights: huggingface.co/remyxai/Space…
a guess about the next best experiment for VQASynth after surveying over 50 papers referencing this line of work
gist.github.com/smellslikeml…
19
Fix metrics prior to experiment launch, but you'll still use post-hoc exclusions bc of an outrage
templates w/ typed inputs,
spot lighting,
adversarial review,
provenance for human audits,
no silver bullet to disambiguate data and instructions
SOTA is defense in depth
New Meta paper shows that Meta’s staff can trust an AI assistant’s numbers about product tests because the AI only plans the steps and never writes a single number itself.
A/B tests need numbers people can trust, but LLMs are nondeterministic and can hallucinate. Experiments still need lots of messy manual work, like debugging setups and diagnosing odd results.
Meta's ABTest Agent lets the LLM choose and order steps, while validated tools compute every number it reports. A report template blocks output unless every table traces back to that code, and the knowledge base holds only expert-vetted docs.
Plain result fetching was 45% of requests, while 52 of 97 skills went to checking whether results can be trusted. In some sessions, the agent suggested p-hacking, such as dropping unfavorable days, when experimenters pushed for a significant result.
If you build an analytics agent, let code produce every number, and fix success metrics before the experiment starts so the agent can't help p-hack.
1
82
Cheap code is a solved problem, but acceptance dynamics are the bigger limiting factor
In the age of the easy fork, merge conflicts and maintainance costs will be the least of your concerns
$ eval >>> $ generation
Teams will maintain entire portfolios of repos
The future of software
In the limit, of course, software won't mean anything anymore. Everything digital will be instantly created at will. But in the interim period we'll wander through an increasingly odd in-between age for software. This transition periodwill be defined by three new precepts.
Everflux software - instead of stepping from release to release, months apart, software now evolves continuously.
Software creation has broken the sound barrier of our own keystrokes, and every model release pushes it faster. There is no reason to wait weeks or months for the next release when a single company merges over 30,000 code changes a day. Cyber capabilities are also progressing at an increasing pace. Bug and vulnerability fixes need to ship quickly and continuously. And every piece of software sits in a lattice of dependencies, each one now shipping at a blistering pace. Software will have to move accordingly fast just to keep from breaking as everything around it shifts.
Translucent sources – A new territory has opened between open source and closed source.
Today Claude will happily pick apart any Electron app, and any binary shipped with debug symbols has become a reverse-engineering playground for LLMs. In other words, much so-called closed-source software is now open source in all but name, and keeping code closed makes less and less sense.
Even the notion of IP and copyright is breaking in software: Apps don't even need to be accessed to be reproduced. I'm writing this text on a personal text editor I prompted Claude to vibe-code from a few videos of people demoing editor features they liked. A mere description of the interface is increasingly enough for an LLM to rebuild the internals, whether or not they resemble the original. Software IP seems to be dissolving like cotton candy in a raccoon's paws.
Bespoke features - Now that all software is open source, nothing prevents us from modifying everything.
And personalization is just one prompt and a couple million tokens away, so there is absolutely no reason anymore to wait for someone else to add this feature you want. Everyone becomes their own product manager. Want a search box, an API bridge, a custom button? It's a couple of minutes away, mostly executed in the background of your daily life. Soon we'll want to personalize everything.
A major challenge appears when bespoke features crash into our first tenet: everflux software. How do you customize software without cutting yourself off from upstream? I could fork it for good to add my search box, but I'd lose all future benefit from the software creator, be it new security updates or other updates to the rest of the software. How do you personalize software that was never meant to be personalized, without shouldering all of its future maintenance?
Merge conflicts won't be the exception anymore; they'll be a constant of everflux software. Agents will rebase our personal features onto each new release overnight, and software will look less like a product and more like a living species, evolving through constant mutation.
Image: M.C. Escher, Metamorphosis II, November 1939 - March 1940
"age of easy fork"
linkedin.com/posts/terry-j-r…
15
Outrider is the harness behind half the new methods released in PEFT v0.21.0
xitter.gq/BenjaminBossan/status/…
Happy to announce the newest 🤗 PEFT release, v0.21.0 🎉
github.com/huggingface/peft/…
Check the thread below for a few highlights of this release.
ALT https://github.com/huggingface/peft/releases/tag/v0.21.0
2
27
Case in point, "Batchwise-adaptive pruning" implemented prior to code release, implemented into competing frameworks using Outrider
In SGLang
github.com/smellslikeml/sgla…
In vLLM: github.com/smellslikeml/vllm…
3
40
agree that AI has made clean-room implementations which side-step licensing restrictions easy
likewise, no reference code is no longer the blocker
3
Matching methods to targets like timm
github.com/huggingface/pytor…
1
15
As codegen becomes more commodified, the validation gates teams can trust become key
Try Remyx: remyx.ai/validate
1
12
Is "judgement" more than a socially acceptable cover for incomplete instrumentation?
Like it's okay to ship code w/o evidence now that AI drafts more of the features?
As if engineering is more limited by epistemic ambiguity than process failure?
124
Smells Like ML retweeted
Super-Tuning uses calibration-free magnitude scoring to allocate a sparse tuning budget. It also allows to add LoRA on top of this sparse update. It is a good fit for exploring sparsity alongside higher-rank adaptation. Also contributed by @smellslikeml.
Smells Like ML retweeted
@smellslikeml added the Riemannian-preconditioned LoRA optimizer. It is a new LoRA-specific optimizer similar to LoRA-FA or LoRA+ that improves feature learning and does not require tuning of learning rates for LoRA's A and B matrices separately.
similarity to the reference implementation is a simple evaluation criteria
but in practice, I'll trade off fidelity to the source method in order to respect constraints of the target repo
A maintainer's merge decision or application-specific benchmark is more relevant here
1
28
with codegen, research repos are cheaper to make than ever before
one way we could repurpose the paper2code candidate pool to evaluate something closer to the contributor/practitioner's pain points upstreaming a method into application
gist.github.com/smellslikeml…
30
Got it exactly reversed!
I don't need a bunch of agents working on one problem someone else framed.
I need one agent focused on addressing the problems in my context.
Introducing autoresearch for arXiv papers: replicate and experiment on any arXiv paper using an army of Claude/Codex agents
For post-training, your agents can use @tinkerapi to launch concurrent RL runs while you watch your experiment tree grow
Try it out now on OpenResearch: openresearch.sh
1
1
264
Get Outrider: github.com/remyxai/outrider
The coding harness for continuous contributions to your repo
58