TensorTonic

Why does dropout divide by (1 − p)? (a popular ML interview question) Take 4 activations: [2, 4, 1, 3]. Sum = 10. Apply dropout with p = 0.5. On average, half the neurons get zeroed, so the next layer learns to expect a sum of around 5. Then at inference, dropout turns off. Every neuron fires, and the next layer suddenly gets 10. Double what it was trained on. The fix: multiply every surviving neuron by 1/(1 − p). Each neuron is kept with probability (1 − p) and boosted by exactly the inverse, so the two cancel. Its expected value stays the same. Training and inference now see the same scale, and at inference dropout becomes a plain identity. No mask, no rescaling. The visual below shows all three steps.

A 6 month plan for becoming a research engineer. > Math and core ML Linear algebra, probability, calculus and optimization, followed by loss functions, classification, metrics and evaluation. Evaluation needs more hours because a research engineer is usually the person who has to say whether a result is real. > Deep learning from scratch Neural networks, activations, optimizers, PyTorch and an autograd engine that you write yourself. > Transformers and language models Attention, embeddings and the full encoder block, then BERT and GPT pretraining. > Post training Fine tuning with LoRA and QLoRA, then SFT, reinforcement learning with verifiable rewards and how RL environments are built. > GPUs and inference GPU programming in CUDA and Triton, inference and KV caching, Quantization and speculative decoding. This stage lets you make your own experiments faster instead of waiting on someone else’s infrastructure. Reproduce papers alongside all of it. Pick a recent one, implement it end to end, run the ablations and write up what matched and what did not. It is the closest thing to the actual job and it is what you can show in an interview. Most of these stages are available on TensorTonic as problems where you implement the idea yourself and get graded against test cases, with GPUs provided for the heavy ones. Start at tensortonic.com

pretraining any LLM is a lot of small pieces that all have to be right: > the tokenizer > the shifted targets > the causal mask > the loss > the optimizer > the learning rate schedule and any one of them can be wrong without throwing an error. if you want to read more in detail, we wrote a full lesson on TensorTonic - tensortonic.com/llm-internal…

Gradient checking is the most underrated debugging skill in deep learning. 1. Use a tiny input, a few parameters and float64. Not your full model. 2. Compute the gradient with your backward pass. 3. Compute it numerically too 4. Compare them with relative error, not the plain difference. 5. Check one parameter at a time to find which layer is wrong. Build Micrograd from Scratch has 25 exercises, including gradient checks. tensortonic.com/study-plans/…

Implement FlashAttention from scratch on TensorTonic in PyTorch and Triton. This project makes you write each piece yourself: safe softmax, online softmax over chunks, merging two partial results, the tiled forward pass, skipping causal tiles, and counting reads and writes. link - tensortonic.com/projects

yesterday we released RL projects on TensorTonic. you build the RL environment, write the agent, and implement Q-learning yourself until the agent learns to reach the goal. tensortonic.com

We just released a new Multimodal AI project. Train your own tiny CLIP from scratch- > build the scaled similarity matrix > write the symmetric contrastive loss > measure retrieval with recall at K > classify zero-shot with prompts > add hard negatives to fix the shortcut > write the pairwise sigmoid loss tensortonic.com

bringing major updates to tensortonic