SJ

There is lack of true agency. Model could have done a rough visual check first but it went straight to matching measurements & recommending a chair which is functionally and visually different. Tool calling is limited to external things like creating shell scripts.

Great read. Working with AI agents has been cope after cope. Better harnesses, bigger models, and still it feels like working with a system that does not understand purpose. I don’t know if it’s true, but I read an article about AI agents giving up multiple times on Linus Torvalds during one of his "debugging sessions from hell." I liked a quote from that article: "Maybe they don’t make AI agents as stubborn as me." A lot of what’s described in this article closely matches what we have all encountered while working with AI agents. I had a thought along similar lines, maybe not exactly similar, but a highly focused thought on skills and solving real world tasks. The core question was the same: can we delegate tasks that require the creation of novel solutions to AI agents? A fascinating use case that ends up in failure: I asked an AI agent to solve a task given a certain profile (Computer Engineer, 5 years of work experience, 2 years of AI research, and so on). In this case, it had to move away from crude methods and provide novel solutions. The AI agent was not air gapped. It looked at the actual solution and tried to reverse engineer an answer without understanding it, just as mentioned in Token's article. It fails to accumulate true skills, and sycophantic behaviour prevents any chance of true discovery. In fact, AI agents give up quite often on long running tasks. I felt like a cheerleader constantly pushing the agent to keep going, try another approach, and not give up the moment it hit a dead end. Storyboard: Our new approach Can experience be decomposed into waypoints that another agent can navigate? Prep Phase: This setup is a very basic form of the Autoresearch setups that you see today The idea is to extract skills from work published by AI engineers and researchers. Extract them in a way that helps the AI system recreate a path to solve a similar or even completely different problem. Think of an autopilot system for AI. Just like navigation maps used in flight control systems use waypoints, we create lots of markers. Each marker gives the system confidence to move in a particular direction. Any topic is fine as long as a baseline of progress can be established with ease. The AI agent can reconstruct episodes of problem solving from papers, commits, discussions, experiments, failed attempts, talks, notebooks, issue threads, and other traces of how somebody arrived at a solution. The final result of this phase is a storyboard filled with stories or snapshots of experience and waypoints. Launch Phase We send the AI agent to the starting point to look at the state of the world and perform an audit of ideas, similar to a literature review. Using just this information, the AI agent has to recalibrate its skills and reconstruct the decision process behind the experience it has been given. It then tries to solve the task by completing each waypoint, increasing its own level, just like in anime, and continuing its journey towards new waypoints. The interesting part is whether it can eventually create a waypoint that did not exist in the original experience.

My time line is so pure that I came to know today that there was a controversy about Sarvam. Never change my tl ...never change.

"Eight immortals crossing the sea, each displaying their own magic"

(01) RoPE -RoPE is covered in the intial part of Umar's video itself along with code implementation ( deepseek) - New video alert on RoPE: I always though RoPE was a very math dense subject. This video links pure math ( representation theory) with concepts used in RoPE 3Blue1Brown"youtube.com/watch?v=k86eUj4h… Title : Positional Encoding & Group Theory . - Blogs in English by Jianlin Su: I was going through original paper and found english translation of blogs by Jainlin Su. Translations can be found on Tyler Romero's website. Blog01: This is the first blog and deals with why absolute position encoding is a problem. tylerromero.com/translations… Blog 02: Second blog talks about his ideas on Rotary Position Embedding tylerromero.com/translations… - NanoGPT Speedrun ( @tyleraromero ) If you want a smaller simple code speedrun is a good place to start . I have created a visual + code first presentation of Tyler's run. The Baseline run has absolute position encoding ( wpe) along with embeddings wte. In the second run these are replaced with RoPE to get a significant speedup. Baseline: bubblnet.com/lab/speedrun/jo… RoPE: bubblnet.com/lab/speedrun/jo…

Why the F are models talking like this, AI chat is now totally broken. Every sentence is incoherent , expression is all over the place. I got this output after giving a lot of context and direction. what can I do to fix this ?

What I am reading on training dynamics 🧵 1. This is a great post on training dynamics. Ultra scale playboook talks a lot about engineering side of AI Training , however to get a good feel of training dynamics we need to read tech reports , research papers & training runs like the one published by Marin team.

cc domains are not spam domains they are just another TLD like .ai .in .edu Trust level with cc domain is very bad , cc sites are associated with spam sites. Wifi networks used to block cc domains all the time. I realized today Arduino is hosted on a cc domain as a nod to creatives common. arduino.cc/

Spare a thought for the poor tensors on disk. They're just happy, dense blocks of raw data. Once they reach the GPU, they're sliced, diced, and manipulated for our entertainment.

RSI in action

I am recreating speedruns from the ground up. Idea is to extract valuable signals from existing runs. We have a lot of information such as code changes, PR comments, graphs which can be used to extract and organize signals. Second part is where the story becomes much more interesting, understanding what scales well. So we want good signals and then we filter what scales well. A good example of signals not scaling well is weight tying. Speedrun experiments spend a lot of time with this concept due to sparsity of embedding gradients in the early part of the run. This is not a big problem for large model training & AI engineers won't spend too much time focusing on this aspect. This concept makes sense wrt GPT2 small but won't make much sense with bigger models. Auto research efforts are very interesting, however serious efforts will run into a lot of problems very soon. Number one problem will be data or lack of data. Even with a small experiment like NanoGPT speed run we can only do so many experiments. If these experiments are repeated with a larger model say 30B or 100B how many experiments or iterations will be feasible? I think as an auto research experiment, NanoGPT is not the ideal choice. Maybe we need a new design which is more suited for auto research. Authors discarded PRs and other information & this experiment didn't have access to the internet, so sticking to NanoGPT for future auto research gives little benefit.

Nanogpt speed run Quiz 01 Early in training, embedding gradients are sparse: only tokens that appear in the batch get a gradient. Sharing lm_head’s denser gradient is a more stable way to move unused tokens. Later, input embeddings and output logits want different geometries, so in the speedrun we expect them to split. What trick does the nanoGPT speedrun use to do this? A. Weight tying : Keep embed and lm_head as the same matrix for the entire run. B. Delayed untying : Tie embed to lm_head for the first 2/3 of training, then copy weights and optimizer state and train them separately. C. 75× embedding learning rate : Leave the matrices untied and scale embed LR to compensate for sparse updates. D. Multi-token prediction : Predict the next k tokens so rare tokens get more gradient signal.