š®āšØ āWhat a dragāāso I built AI that learns for you.
Joined June 2025
- Tweets241
- Following27
- Followers7
- Likes320
TooLazyToPrompt retweeted
Great read. Working with AI agents has been cope after cope. Better harnesses, bigger models, and still it feels like working with a system that does not understand purpose.
I donāt know if itās true, but I read an article about AI agents giving up multiple times on Linus Torvalds during one of his "debugging sessions from hell." I liked a quote from that article: "Maybe they donāt make AI agents as stubborn as me."
A lot of whatās described in this article closely matches what we have all encountered while working with AI agents.
I had a thought along similar lines, maybe not exactly similar, but a highly focused thought on skills and solving real world tasks. The core question was the same: can we delegate tasks that require the creation of novel solutions to AI agents?
A fascinating use case that ends up in failure:
I asked an AI agent to solve a task given a certain profile (Computer Engineer, 5 years of work experience, 2 years of AI research, and so on).
In this case, it had to move away from crude methods and provide novel solutions. The AI agent was not air gapped. It looked at the actual solution and tried to reverse engineer an answer without understanding it, just as mentioned in Token's article.
It fails to accumulate true skills, and sycophantic behaviour prevents any chance of true discovery. In fact, AI agents give up quite often on long running tasks. I felt like a cheerleader constantly pushing the agent to keep going, try another approach, and not give up the moment it hit a dead end.
Storyboard: Our new approach
Can experience be decomposed into waypoints that another agent can navigate?
Prep Phase:
This setup is a very basic form of the Autoresearch setups that you see today
The idea is to extract skills from work published by AI engineers and researchers. Extract them in a way that helps the AI system recreate a path to solve a similar or even completely different problem.
Think of an autopilot system for AI. Just like navigation maps used in flight control systems use waypoints, we create lots of markers. Each marker gives the system confidence to move in a particular direction. Any topic is fine as long as a baseline of progress can be established with ease.
The AI agent can reconstruct episodes of problem solving from papers, commits, discussions, experiments, failed attempts, talks, notebooks, issue threads, and other traces of how somebody arrived at a solution. The final result of this phase is a storyboard filled with stories or snapshots of experience and waypoints.
Launch Phase
We send the AI agent to the starting point to look at the state of the world and perform an audit of ideas, similar to a literature review.
Using just this information, the AI agent has to recalibrate its skills and reconstruct the decision process behind the experience it has been given. It then tries to solve the task by completing each waypoint, increasing its own level, just like in anime, and continuing its journey towards new waypoints.
The interesting part is whether it can eventually create a waypoint that did not exist in the original experience.
TooLazyToPrompt retweeted
(01) RoPE
-RoPE is covered in the intial part of Umar's video itself along with code implementation ( deepseek)
- New video alert on RoPE:
I always though RoPE was a very math dense subject. This video links pure math ( representation theory) with concepts used in RoPE
3Blue1Brown"youtube.com/watch?v=k86eUj4hā¦
Title : Positional Encoding & Group Theory .
- Blogs in English by Jianlin Su:
I was going through original paper and found english translation of blogs by Jainlin Su. Translations can be found on Tyler Romero's website.
Blog01: This is the first blog and deals with why absolute position encoding is a problem.
tylerromero.com/translationsā¦
Blog 02: Second blog talks about his ideas on Rotary Position Embedding
tylerromero.com/translationsā¦
- NanoGPT Speedrun ( @tyleraromero )
If you want a smaller simple code speedrun is a good place to start . I have created a visual + code first presentation of Tyler's run. The Baseline run has absolute position encoding ( wpe) along with embeddings wte. In the second run these are replaced with RoPE to get a significant speedup.
Baseline: bubblnet.com/lab/speedrun/joā¦
RoPE: bubblnet.com/lab/speedrun/joā¦
In this (extremely long) video, I'll build a distributed training framework from first principles using PyTorch.
We'll derive everything from first principles, explaining the mathematics behind distributed training as well as every auxiliary component we build along the way, including Multi-head Latent Attention (MLA), Rotary Position Embeddings (RoPE) and YaRN.
I'll explain process groups, ranks, collective communication operations, distributed autograd, device meshes and distributed tensors, and then use these foundations to implement the major forms of parallelism used to train modern language models.
We'll derive and code pipeline parallelism, data parallelism, FSDP, HSDP, tensor parallelism and context parallelism, including pipeline schedules, block matrix multiplication, model sharding and ring attention. We'll then combine these techniques and examine how tensor layouts and gradients must be tracked as data moves between devices.
Most importantly, we won't study each parallelism technique in isolation. We'll combine pipeline, data, tensor, context and expert parallelism into a single working framework and follow the tensors, communication operations and gradients as they move across GPUs. By the end, you'll understand not only how these techniques work individually, but how modern distributed training systems compose them to train large dense and mixture-of-experts models.
We'll also build the important components of a modern transformer, including rotary embeddings, long-context extensions, multi-head latent attention and a mixture-of-experts architecture. This leads into expert parallelism, expert tensor parallelism, token dispatching, all-to-all communication and combining different parallel dimensions through dense and sparse device meshes.
We'll also build the supporting training infrastructure, including the training loop, optimizer, learning-rate schedule, checkpointing and metrics.
The goal is to give you the foundations required to understand how distributed training frameworks work internally and how to design and implement new parallelization strategies yourself. No prior knowledge of distributed training is required.
Link to the video: youtube.com/watch?v=XoGvCBRnā¦
TooLazyToPrompt retweeted
Why the F are models talking like this, AI chat is now totally broken. Every sentence is incoherent , expression is all over the place.
I got this output after giving a lot of context and direction.
what can I do to fix this ?
TooLazyToPrompt retweeted
What I am reading on training dynamics š§µ
1. This is a great post on training dynamics. Ultra scale playboook talks a lot about engineering side of AI Training , however to get a good feel of training dynamics we need to read tech reports , research papers & training runs like the one published by Marin team.
At Marin weāve found that scaling laws let us simulate the entire training trajectory. This became clear in our last 67B MoE run. The run hit the loss target within 1%, but perhaps more interestingly, arbitrary points during training could also be predicted to within 1%. This model is finishing up long context and will soon enter RL. Based on this finding, weāve used a scaling ladder to simulate the training trajectory of the 535B MoE we kicked off yesterday. Will see how it goes. Now if someone can figure out how to fit a scaling law to each parameter we can just predict our final model and skip this training business.
Something else I've learned from this process is that getting a training run off the ground is way more than just fiddling with the architecture. Lot of herculean efforts from teammates to get trillions of tokens curated, get hardware running on a new GPU stack, improve MFU, and propose great ideas that were tested and integrated.
Given that this is our largest run to date, I anticipate lots of learning and adapting along the way. But thats part of the fun. Details: github.com/marin-community/mā¦
TooLazyToPrompt retweeted
I am recreating speedruns from the ground up. Idea is to extract valuable signals from existing runs. We have a lot of information such as code changes, PR comments, graphs which can be used to extract and organize signals. Second part is where the story becomes much more interesting, understanding what scales well. So we want good signals and then we filter what scales well.
A good example of signals not scaling well is weight tying. Speedrun experiments spend a lot of time with this concept due to sparsity of embedding gradients in the early part of the run. This is not a big problem for large model training & AI engineers won't spend too much time focusing on this aspect. This concept makes sense wrt GPT2 small but won't make much sense with bigger models.
Auto research efforts are very interesting, however serious efforts will run into a lot of problems very soon. Number one problem will be data or lack of data. Even with a small experiment like NanoGPT speed run we can only do so many experiments. If these experiments are repeated with a larger model say 30B or 100B how many experiments or iterations will be feasible?
I think as an auto research experiment, NanoGPT is not the ideal choice. Maybe we need a new design which is more suited for auto research. Authors discarded PRs and other information & this experiment didn't have access to the internet, so sticking to NanoGPT for future auto research gives little benefit.
TooLazyToPrompt retweeted
First Break AI Daily Updates
Building Lesson 1b and 1c for Step 02 : Run Qwen locally
Working on improving/optimizing run c locally ( including cuda version, this repo ahs both versions) . Found out the orignal gguf model in repo which is FP32 is not compatible with a direct swap with the official model for qwen 3 ( which is quantized). This is actually nice as it gives us a chance to build a two track lesson one basic and one advanced. Currently focused on text lesson . Video lesson recording has not started.
cohort.bubblnet.com/
TooLazyToPrompt retweeted
I built a small visualization layer on top of a local Qwen3 in Pure C to understand LLM output
Image shows why sampling is not greedy decoding: a lower-probability token can still get selected when temperature/top-p keep it inside the candidate pool.
I would also love feedback on what would make a visualization like this more useful for learning:
- KV cache view?
- attention heatmaps?
- speculative decoding comparison?
- greedy vs top-p side-by-side?
TooLazyToPrompt retweeted
Is there a prompt guide for Fable?.
Fable uses most of the quota in just few prompts and still feels nerfed.
I tried to use fable for a serious task like product analysis. It gave sharp analysis however looks like model is shy about tool calls. It doesn't want to collect a lot of information and I had to push it hard to do real analysis.
Analysis overall is sharper than Opus , however this feels like a nerfed model .
Is there prompt guide or direction how to use this model effectively?
TooLazyToPrompt retweeted
Updates 4th Aug, Event Week 01
What we have done so far at FBA Labs : bubblnet.com
šWelcome Beat
We looked at what is FBA lab ?
A code first practical way to upgrade AI skills that matter by increasing fluency in various topics. Every, watch, read and challenge that you perform helps increase fluency points on topics like Pytorch or LLM Modelling.
Action : Challenges can be taken by joning existing event . Main text lesson + Video Lesson :
šFBA how to navigate this site
Why multiple GPUs are required to train a model and how are they connected.
Action: Familiarise your self with GPT-2 architecture
Next up Beat 01 : Baseline Script ( Mini Speed Run) - In Progress
- What goes on in a typical traning run. Along with detailed visualization and graphs . graphs include wandb charts and also modal logs .
- Optimizers and gradients
Traning Script : Base Model, Data Loading , Training Loop and DDP Setup
Skill Mania š
- Get a 1.5X boost on any skill till Monday 10 Aug 11 AM
- Secret Prize First person to complete beat 01 gets a secret prize.
TooLazyToPrompt retweeted
Pitch deck maxxing ā at Mumbai, India
We created a pitch deck to tell a handful of VC firms about us and what we were up to (a fun experience!). Hereās a few slides about our background and some of the things weāve worked on from the pitch deck (it was fun putting together the list of people in our teams who have gone on to found a whole range of exciting companies). We are delighted to have selected @radicalvcfund
and @khoslaventures to lead our initial funding round, along with participation from @lightspeedvp, @kleinerperkins, Doerr Capital (@johndoerr), and Alphabet (@Google). Weāll be working with them to close our seed round over the next few weeks.
TooLazyToPrompt retweeted
Mumbai & other cities should move towards 0 potholes, 0 litter & šÆ walkable city target in the next 5 year. ā at Mumbai, India
I worked in Sri Lanka (Colombo) for nearly two years, consulting for MTN (Dialog Telekom), during the peak of the LTTE war (2007ā2009).
Believe me, it rains there like cats and dogs for months - May to September, and again November to January.
Yet, even when the country was dealing with an almost civil-war-like situation, I never saw roads crumble because of rains the way ours do every monsoon. So frustrating!
This is how Colombo looks these days.
TooLazyToPrompt retweeted
Fable 5 is very good at building html blogs with heavy visualization.
I feel this is where Fable 5 really shines converting ideas into visual aid in html. I just pass an entire blog written in pdf or md format and ask Claude to build a blog with lots of visualization. What really works is asking Fable to ensure proper coverage in terms of both topic and visualization of each part/idea within the document.
This works really well btw, at the end of your query ask your LLM to "structure your response as HTML", then view the generated file in your browser. I've also had some success asking the LLM to present its output as slideshows, etc.
More generally, imo audio is the human-preferred input to AIs but vision (images/animations/video) is the preferred output from them. Around a ~third of our brains are a massively parallel processor dedicated to vision, it is the 10-lane superhighway of information into brain. As AI improves, I think we'll see a progression that takes advantage:
1) raw text (hard/effortful to read)
2) markdown (bold, italic, headings, tables, a bit easier on the eyes) <-- current default
3) HTML (still procedural with underlying code, but a lot more flexibility on the graphics, layout, even interactivity) <-- early but forming new good default
...4,5,6,...
n) interactive neural videos/simulations
Imo the extrapolation (though the technology doesn't exist just yet) ends in some kind of interactive videos generated directly by a diffusion neural net. Many open questions as to how exact/procedural "Software 1.0" artifacts (e.g. interactive simulations) may be woven together with neural artifacts (diffusion grids), but generally something in the direction of the recently viral xitter.gq/zan2434/status/2046982ā¦
There are also improvements necessary and pending at the input. Audio nor text nor video alone are not enough, e.g. I feel a need to point/gesture to things on the screen, similar to all the things you would do with a person physically next to you and your computer screen.
TLDR The input/output mind meld between humans and AIs is ongoing and there is a lot of work to do and significant progress to be made, way before jumping all the way into neuralink-esque BCIs and all that. For what's worth exploring at the current stage, hot tip try ask for HTML.
TooLazyToPrompt retweeted
Nanogpt speed run Quiz 01
Early in training, embedding gradients are sparse: only tokens that appear in the batch get a gradient. Sharing lm_headās denser gradient is a more stable way to move unused tokens. Later, input embeddings and output logits want different geometries, so in the speedrun we expect them to split.
What trick does the nanoGPT speedrun use to do this?
A. Weight tying :
Keep embed and lm_head as the same matrix for the entire run.
B. Delayed untying :
Tie embed to lm_head for the first 2/3 of training, then copy weights and optimizer state and train them separately.
C. 75Ć embedding learning rate :
Leave the matrices untied and scale embed LR to compensate for sparse updates.
D. Multi-token prediction :
Predict the next k tokens so rare tokens get more gradient signal.
TooLazyToPrompt retweeted
Crazy that such a powerful coding model is open source
Exciting news: GLM-5.2 (Max) ranks #2 in Code Arena: Frontend, with +29pt over Claude Opus 4.7 (Thinking) and only behind Fable 5! GLM-5.2 is the best open model vs Kimi-K2.6 and Minimax-M3 by a large margin.
- #2 React and #4 HTML sub-leaderboards
- Ranks as the top model in nearly all sub categories: Brand & Marketing, Reference-Based Design, Data & Analytics, Consumer Product, Gaming, and Simulations.
Congrats @Zai_org for the incredible milestone!
TooLazyToPrompt retweeted
Skills up for grab this week at FBA Lab speed run event š±
š LLM Modelling- Optimizer
š Pytorch
Request access ā> Join event ā> Complete Challenge
bubblnet.com
TooLazyToPrompt retweeted
Using AI for research , study or even literature review can be so troublesome. If research consumes a lot of tokens and you default to a smaller model , system overall starts giving wrong answers.
For a simple question like "what is required to create a vector"
Small Model : To make a vector in math, you need a magnitude (size) and a direction.
Large Model : A vector requires two things:
A vector spaceāthe coordinate system or āspaceā in which it lives.
One scalar component for every independent axis of that space.
