Calc Consulting

𝗥𝗲𝗻𝗼𝗿𝗺𝗮𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻, 𝗘𝗺𝗲𝗿𝗴𝗲𝗻𝗰𝗲, 𝗮𝗻𝗱 𝗪𝗵𝘆 𝗗𝗲𝗲𝗽 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 𝗪𝗼𝗿𝗸𝘀. Turns out, BackProp looks remarkably like defining a Wilsonian RG beta function for each layer. One of the deepest ideas in theoretical physics is that complex, emergent behavior can arise from simple microscopic rules. Kenneth Wilson's Nobel Prize-winning renormalization group (RG) theory gives us a mathematical framework for understanding how this happens across scales. We've been developing a spectral RG theory of deep learning, motivated by empirical observations of heavy-tailed weight matrices, scale invariance, and the remarkable tendency of well-trained neural networks to approach marginal spectral behavior near α ≈ 2. But here's something surprising. You don't even need to look at the weight spectra to see an intriguing connection to RG. Just look at the basic equations of backpropagation. In Wilsonian RG, a beta function describes how a coupling changes with logarithmic scale: β(g) = dg/d(log b) Now consider a neuron with preactivation z = wx. Backpropagation gives us the exact identity: w ∂L/∂w = z ∂L/∂z = ∂L/∂log|z| That's precisely the mathematical form of a logarithmic scaling derivative. And in a multilayer neural network, something even more interesting happens: → Each layer has its own beta-like logarithmic derivative. → These quantities are coupled across layers through the chain rule. → At stationary points of training, the corresponding dataset-averaged quantities vanish. → The resulting mathematical structure resembles a system of coupled RG beta functions. This is a striking connection hiding in plain sight. Of course, identifying these derivatives does not by itself establish a Wilsonian coarse-graining transformation. But it gives us a concrete mathematical starting point for investigating how coupled layerwise RG flows might emerge during learning. And that brings us back to the bigger question: Can we understand deep learning as a theory of emergence, in which different neural-network layers develop coupled, scale-dependent structures, potentially approaching their own fixed points? Our spectral RG theory approaches this question empirically, through the eigenvalue distributions of trained weight matrices. The backpropagation equations suggest another route toward understanding the underlying dynamics. We're working to bring these two perspectives together. I've put together a thread walking through the mathematics, starting with Wilsonian RG and deriving the backpropagation identities step by step Stay tuned for more

alpha per layer for Lambda3

𝗦𝗽𝗲𝗰𝘁𝗿𝗮𝗹 𝗨𝗻𝗶𝘃𝗲𝗿𝘀𝗮𝗹𝗶𝘁𝘆 𝗼𝗳 𝗟𝗲𝗮𝗿𝗻𝗶𝗻𝗴 11/11 The spectral RG view may extend from neural networks to XGBoost and other learners approaching fixed points. Spectral universality asks whether different learning rules converge toward similar effective correlation structures. Public arXiv link to follow.

𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝗿𝗲𝗰𝗼𝘃𝗲𝗿𝘆 𝗳𝗿𝗼𝗺 𝗼𝘃𝗲𝗿𝘁𝗿𝗮𝗶𝗻𝗶𝗻𝗴 10/11 We also fix an overfit checkpoint. In this preliminary experiment, continuing with WW-PGD recovers more test accuracy than the AdamW and Muon continuations shown.

𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝘀𝗽𝗲𝗰𝘁𝗿𝗮𝗹 𝗰𝗼𝗻𝘁𝗿𝗼𝗹 9/11 On MNIST, adding WW-PGD to AdamW moves layerwise exponents toward 2 while keeping test accuracy close to the baseline across five runs. This tests whether we can steer the spectrum using WW-PGD while preserving predictive performance.

𝗧𝘂𝗿𝗻𝗶𝗻𝗴 𝘁𝗵𝗲 𝘁𝗵𝗲𝗼𝗿𝘆 𝗶𝗻𝘁𝗼 𝗮𝗻 𝗶𝗻𝘁𝗲𝗿𝘃𝗲𝗻𝘁𝗶𝗼𝗻 8/11 This suggests an optimizer intervention: WW-PGD. WW-Projected Gradient Descent After a base optimizer step, select a spectral window, nudge its shape toward α = 2, preserve its trace-log before blending, and mix the correction back into the weights. You can test this now using the Python package: ww-_pgd

𝗧𝗲𝘀𝘁𝗶𝗻𝗴 𝘁𝗵𝗲 𝗽𝗿𝗼𝗽𝗼𝘀𝗲𝗱 𝗰𝗼𝗻𝗻𝗲𝗰𝘁𝗶𝗼𝗻 7/11 Trained checkpoints let us test the RG picture directly. Near α = 2, we compare energy across logarithmic bands, the ECS with the fitted power-law tail, and their normalization scales. Together, these measurements turn the theory into a concrete experimental program.