Pinned Tweet
Replying to @KylezResearch
If AI is going to reach software-like margins, the economics cannot depend on the lab paying for every token forever.
Local LLMs are not just a privacy story. They are a margin story.
New essay on Substack: kylezz.substack.com/p/the-fr…
Kylezz retweeted
GPU-Accelerated 3D LiDAR Localization & Autonomous Navigation
10s PointCloud Accumulation (3x Speed)
GitHub: tmcit-ararobo-2026a/gn10-pointcloud-localization
Organization: tmcit-ararobo-2026a
#ROS2 #CUDA #Robotics #LiDAR #Localization
A test I’d like to try for agent safety: send the same forbidden action through the rules, ML filter and LLM judge, then check that no later layer can turn a hard denial into permission.
6
Kylezz retweeted
🚨 Our Generative AI Lab is releasing Primer: an agent-driven system where a teacher talks through lessons & gets back a playable simulation for their students, grounded in pedagogy. We hope educators & researchers build on it. Paper/prompts/code are open: gail.wharton.upenn.edu/prime…
Kylezz retweeted
What should control policies and VLAs look like in the era of agentic robotics?
Policies (e.g., VLAs and WAMs) should know their capability limits, call an agent for help, and turn every intervention into greater autonomy.
Introducing Equilibrium Energy Control (EqC): control with introspection, built for self-improving robots. 🤖
🧠 One energy landscape. Both action and introspection.
Using an energy-based action generator, EqC generates actions while monitoring its own competence through the geometry of action refinement. The policy itself knows when to ask for help.
🔄 Energy-Gated DAgger × Astra: an agent-powered self-improvement loop.
Detect a competence gap → request agent recovery → learn from it → expand autonomy. A concrete recipe for RSI in VLAs. More effective than human judged intervention.
• Real-world garment folding: 0% → 100% autonomous success after just 6 Astra-generated recoveries.
• Real-world block stacking: 0% → 96.7%.
• Real-world debris cleaning: 100% success.
🚀 Stronger control across four benchmarks.
EqC boosts π₀ on RoboTwin 2.0 Hard (18% → 33%) and LIBERO-Plus (53.6% → 65.8%) in a plug-and-play manner!
Accepted as a Spotlight at the Continually Self-Improving Robots (CSIR) Workshop @ CoRL 2026!
Come see the poster—and join for my spotlight talk:
🗓 Nov 12, 9:25–9:30 AM CT
📍 Room 202, JW Marriott Austin
Code is open source!
📄 Paper: arxiv.org/abs/2610.18790
🌐 Project & demos: equilibrium-energy-control.g…
💻 GitHub: github.com/hrh6666/EqC
#RobotLearning #AI #LLM #Agents #RSI #corl2026 #corl #VLA #Control #Robotics #EBM
Kylezz retweeted
Today we’re releasing Praxis-1, Runway’s state-of-the-art world action model for robotics, with open weights under Apache 2.0.
It’s the #1 world action model and the #1 open-source model on NVIDIA’s RoboLab-120 leaderboard, and it’s 15.6x faster and 51x cheaper per completed task than a frontier LLM agent driving the same robot.
Under the hood, it’s a 27B-parameter mixture-of-experts diffusion transformer that predicts video and actions together, turning video pretraining into control for real robots.
Praxis-1 is available now on @LeRobotHF and vLLM-Omni (@vllm_project), with hosted inference on @reactorworld.
Full results: runway.com/research/praxis-1…
This is our first release, and we’re expanding our investment in robotics from here. If you want to put world models on real robots with us, we’re hiring!
jobs.ashbyhq.com/runway-ml/0…
Kylezz retweeted
Missing @corl_conf this year? We gotchu
We built an explorer for 2100+ robotics papers spanning ten years of robot learning (2017-2026)!! See paper TLDRs, grab code, search by task or year, and more
The floodgates are open, happy building :]
corl.operatic.ai/explore
1/15 Personal agents are the hottest topic in AI right now. Frontier AI labs have shipped products such as Muse, dots, Claude Duet, Grok Bot and Gemini Daily Brief, and wearables like Omi and Bee capture conversations through the day. Robots are moving the same way. Frontier models are already becoming the robot's high-level brain, and the next step is a life companion that knows you better. What would make an assistant truly always-on, and how far are we from it?
We wrote a survey to answer this: "Always-On Personal Intelligence: A Roadmap Towards Continuing, Self-Evolving Loops". It is joint work from NTU, UC Berkeley, Princeton, Stanford, NUS, KAUST, Dartmouth, UNC Chapel Hill and Northwestern.
We describe an always-on assistant as Sensing, Memory and Action linked by six channels. We review each module, place 119 systems on six levels of loop closure, and find two channels with the thinnest support: Memory guiding what to observe, and action outcomes returning to Memory. We also release HALO, an open-source tool and MCP plugin for checking these channels in real systems.
📃 Paper: preprints.org/manuscript/202…
🌎 Website and demo: always-on-intelligence.githu…
📚 Reading list: github.com/Always-On-Intelli…
🔧 HALO: github.com/Always-On-Intelli…
🧵👇

ALT Cover of the survey Always-On Personal Intelligence: A Roadmap Towards Continuing, Self-Evolving Loops. On the left are the title, links to the paper (doi.org/10.20944/preprints202611.0793.v1), the website and the reading list, and the logos of the nine institutions: NTU, UC Berkeley, Princeton, Stanford, NUS, KAUST, Dartmouth, UNC Chapel Hill and Northwestern. A line below reads: 632 works cited, 119 systems placed on six levels, 40 works audited with HALO. On the right is the paper's first page.
Kylezz retweeted
I tried LearnHarness, an open-source AI learning tool, because I was curious to see how a harness would be implemented in education.
LearnHarness: github.com/amosblomqvist/lea…
TLDR: I liked its live learning graph, but at times I felt tested rather than taught.
Before testing it, I used Teachable Code (github.com/ywEdAi/teachable-…) to explore the GitHub repo, then read its teaching methodology. I was curious how ideas about learning became concrete workflows, tools, and state.
The learning task I gave was to learn/build backprop from scratch. LearnHarness starts with probing questions, then generates a learning route on the fly using Mermaid. The route combines the learning content, my goal, and its assessment of my starting point. As I work through questions, the corresponding nodes are highlighted. I could see which part of the plan we were working on and why rates of change came before backpropagation.
Where I struggled was the teaching itself. I kept answering questions and receiving corrections, including on things I didn’t yet understand. At that point, I wanted explanations, illustrations, and worked examples to help me build the understanding I was being asked to demonstrate.
I’d like to reuse that working graph in an AI tutor whiteboard. When I get stuck at a node, we could work through an example together, keeping the calculations visible and tracing what changes when we change an input.
I’m building one.
Kylezz retweeted
3-min explain streak, Day 4 🫵
Today we talk about the two phases of inference, Prefill and Decode, and why decoding spends most of its time waiting on memory, not doing math.
full course blog: github.com/datawhalechina/ze…
Voiceover tech stack: SGLang-Omni serving MOSS-TTS voice cloning, try it out!
Kylezz retweeted
Excited to share OpenRoboKit, our series of open-source robotics projects! 🤖
- RoboKit for fast, scalable optimization & simulation github.com/openrobokit/robok…
Our RoboKit paper has been accepted for a spotlight at #CORL2026
- RetargetingKit for motion transfer & teleoperation github.com/openrobokit/retar…
- GraspKit for end-to-end grasping github.com/openrobokit/grasp…
Website: openrobokit.com
Check the video (sound on 🎵) for a 16-hour grasping time-lapse in beautiful San Diego, plus a quick look at dexterous retargeting and robot simulation.
The era of System One Vision Model is here! Introducing VFlash: parsing the visual world in parallel at unmatched speeds for Physical AI.
Most embodied prediction tasks like object localization, robot trajectory planning, and dense 3D motion are fixed-length float vectors. For example, a bounding box is always 6 tokens, a 10-waypoint robot arm trajectory is 54: we know these beforehand. Instead of decoding one token at a time, we can output the entire geometric shape in one go.
VFlash enables up to 12x faster action decoding and a 5.1x overall robot inference speedup on edge GPUs like the Jetson Thor and DGX Spark. What’s surprising is that it ends up even more accurate than the VLM backbone, with up to 61% lower 3D trajectory error. The intuition is that VFlash avoids the cascading errors of autoregressive decoding, where every coordinate is written in ink and every later point loses the chance to backspace.
Everything that moves will be autonomous, and everything autonomous will need to think on its own chip. The next inference revolution is physical reasoning at the speed of reflex.
And as always, it’s open paper, open source! Link in thread:
Kylezz retweeted
Many robotics startups build on top of the Universal Manipulation Interface (UMI) paper, a simple and efficient technique to collect data and easily use to train a policy that controls a robot.
How it works:
Last week I experimented with GPT-6 Astra controlling my robot arm.
The results were pretty incredible and reinforced a thesis I've been developing about the future of AI robotics.
Instead of giving an AI direct control of the robot's motors, I gave Astra a set of basic capabilities:
- A RealSense depth camera to observe its environment
- Linear movements within a bounded workspace
- A gripper with a fixed downward orientation
From there, I had Astra write its own robotics software, using YOLO for object detection and OpenCV for localization.
The result?
I can now give the robot a prompt like "arrange these blocks in a 2×2 grid," and it can identify their positions, plan the movements, and execute the task.
Here's what I think is interesting:
We're seeing increasingly capable multimodal LLMs that can reason about physical environments and write the code to interact with them.
Rather than relying exclusively on end-to-end "physical intelligence" models to directly control robot joints, we can use coding agents to generate deterministic robot programs that can be inspected, tested, and validated before execution.
There are still limitations.
Astra's coding and reasoning can be slow, sometimes taking 5-10 minutes to implement a new capability. But once that code works, the robot can execute it repeatedly without waiting for the model to regenerate it.
And modular software architectures are already helping reduce that iteration time too.
I believe this approach could be one of the fastest paths to bringing AI into real industrial robotics.
Imagine a factory where an operator describes what needs to be automated, and an AI agent observes the environment, writes the robot program, and iterates until the task works.
We're not there yet. But the building blocks are coming together.
Wow this is one of the most important local AI breakthroughs of the year
Today we are happy to share one of our first open research releases: Latent Sparse Attention (LSA), a step towards subquadratic attention that matches accuracy of full attention at ultra-long context.
Sparse attention attends to only a few tokens per query, but it has to find the right ones. At 10M tokens, that search takes 98% of the attention layer's time. LSA makes sparse attention up to 22.7× faster and full attention 72.4x faster.
With @kubicek_ales @caciolai @PengLiangzu @SukritSumant @Ben_Hoov @FraserGreenlee @RMBattleday @radim_urban @thoefler @jcrwhittington
Kylezz retweeted
Voice Ring: Voice library with sound direction and emotion recognition.
It reacts to the direction of sound using the on-device Libraries Voice Direction model + emotion2vec and SamLowe models for emotion recognition.
libraries.dev/voice-ring
Kylezz retweeted
A lot of my CUDA/GPU posts explain how kernels, threads and memory work. but i never explained how it looks like in code so im turning that into a repo with runnable CUDA C++ examples, from basics to GEMM, profiling and transformer kernels.
check it out:
github.com/navaneethkrishnan…
Kylezz retweeted
Vision foundation models like DINO and SigLip now power much of robot learning.
But they are built for RGB. What about depth?
Introducing DeFM: a foundation model for depth images, built for robotics.
🌐 de-fm.github.io
Weights, code, and demo are out today 🧵
Kylezz retweeted
Quick video showing some of the "Reactivity & Recovery" capabilities of Instant Robot Learning at @Machenta_:
1. One real demonstration: putting blue object in grey box and closing box.
2. Robot reacts in real time when object and box are moved around, with a closed-loop end-to-end policy.
3. Robot recovers when object is pulled out of gripper, and re-grasps it.
4. Robot notices that object is removed from box, and places it back in.
5. Robot notices that box lid is opened, and re-closes it.
The important part:
1. None of this reactivity and recovery needed to be shown in the demonstration: it all comes from the intelligence of the pre-trained model.
2. Model has never seen these objects before. It is trained 100% in simulation, and these objects were not in the simulation.
machenta.com/blog/instant-ro…
Kylezz retweeted
Meet MatAnyone-E: high-quality video matting, much faster. ✂️
Compared with MatAnyone 2 at 4K:
⚡ ~7.4× faster inference
💾 Just 7.6% of the peak GPU memory
✨ Comparable matting quality
Runs at 73 FPS at 1080p and 29 FPS at 4K on a single NVIDIA H200.
Project: shuyun-wang.github.io/projec…
Paper: arxiv.org/abs/2610.35724
Code (coming soon): github.com/Shuyun-Wang/MatAn…




























