Calculation Consulting is a boutique consultancy that specializes in machine learning, AI, and data science
San Francisco
Joined January 2013
- Tweets15K
- Following4K
- Followers4.9K
- Likes88.3K
๐ฅ๐ฒ๐ป๐ผ๐ฟ๐บ๐ฎ๐น๐ถ๐๐ฎ๐๐ถ๐ผ๐ป, ๐๐บ๐ฒ๐ฟ๐ด๐ฒ๐ป๐ฐ๐ฒ, ๐ฎ๐ป๐ฑ ๐ช๐ต๐ ๐๐ฒ๐ฒ๐ฝ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด ๐ช๐ผ๐ฟ๐ธ๐. Turns out, BackProp looks remarkably like defining a Wilsonian RG beta function for each layer. One of the deepest ideas in theoretical physics is that complex, emergent behavior can arise from simple microscopic rules.
Kenneth Wilson's Nobel Prize-winning renormalization group (RG) theory gives us a mathematical framework for understanding how this happens across scales.
We've been developing a spectral RG theory of deep learning, motivated by empirical observations of heavy-tailed weight matrices, scale invariance, and the remarkable tendency of well-trained neural networks to approach marginal spectral behavior near ฮฑ โ 2.
But here's something surprising.
You don't even need to look at the weight spectra to see an intriguing connection to RG. Just look at the basic equations of backpropagation.
In Wilsonian RG, a beta function describes how a coupling changes with logarithmic scale:
ฮฒ(g) = dg/d(log b)
Now consider a neuron with preactivation z = wx. Backpropagation gives us the exact identity:
w โL/โw = z โL/โz = โL/โlog|z|
That's precisely the mathematical form of a logarithmic scaling derivative.
And in a multilayer neural network, something even more interesting happens:
โ Each layer has its own beta-like logarithmic derivative.
โ These quantities are coupled across layers through the chain rule.
โ At stationary points of training, the corresponding dataset-averaged quantities vanish.
โ The resulting mathematical structure resembles a system of coupled RG beta functions.
This is a striking connection hiding in plain sight.
Of course, identifying these derivatives does not by itself establish a Wilsonian coarse-graining transformation. But it gives us a concrete mathematical starting point for investigating how coupled layerwise RG flows might emerge during learning.
And that brings us back to the bigger question:
Can we understand deep learning as a theory of emergence, in which different neural-network layers develop coupled, scale-dependent structures, potentially approaching their own fixed points?
Our spectral RG theory approaches this question empirically, through the eigenvalue distributions of trained weight matrices. The backpropagation equations suggest another route toward understanding the underlying dynamics.
We're working to bring these two perspectives together.
I've put together a thread walking through the mathematics, starting with Wilsonian RG and deriving the backpropagation identities step by step
Stay tuned for more
๐ ๐ฆ๐ฝ๐ฒ๐ฐ๐๐ฟ๐ฎ๐น ๐ฅ๐ฒ๐ป๐ผ๐ฟ๐บ๐ฎ๐น๐ถ๐๐ฎ๐๐ถ๐ผ๐ป-๐๐ฟ๐ผ๐๐ฝ ๐ฉ๐ถ๐ฒ๐ ๐ผ๐ณ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด
1/11 Neural networks leave a spectral record of learning in their weights. My new paper asks why strong trained layers often develop power-law tails near ฮฑ โ 2.
And how this can help us design better optimizers and prevent overfitting
๐ฆ๐ฝ๐ฒ๐ฐ๐๐ฟ๐ฎ๐น ๐จ๐ป๐ถ๐๐ฒ๐ฟ๐๐ฎ๐น๐ถ๐๐ ๐ผ๐ณ ๐๐ฒ๐ฎ๐ฟ๐ป๐ถ๐ป๐ด
11/11 The spectral RG view may extend from neural networks to XGBoost and other learners approaching fixed points. Spectral universality asks whether different learning rules converge toward similar effective correlation structures.
Public arXiv link to follow.
๐ง๐ฒ๐๐๐ถ๐ป๐ด ๐ฟ๐ฒ๐ฐ๐ผ๐๐ฒ๐ฟ๐ ๐ณ๐ฟ๐ผ๐บ ๐ผ๐๐ฒ๐ฟ๐๐ฟ๐ฎ๐ถ๐ป๐ถ๐ป๐ด
10/11 We also fix an overfit checkpoint.
In this preliminary experiment, continuing with WW-PGD recovers more test accuracy than the AdamW and Muon continuations shown.
๐ง๐ฒ๐๐๐ถ๐ป๐ด ๐๐ฝ๐ฒ๐ฐ๐๐ฟ๐ฎ๐น ๐ฐ๐ผ๐ป๐๐ฟ๐ผ๐น
9/11 On MNIST, adding WW-PGD to AdamW moves layerwise exponents toward 2 while keeping test accuracy close to the baseline across five runs.
This tests whether we can steer the spectrum using WW-PGD while preserving predictive performance.
๐ง๐๐ฟ๐ป๐ถ๐ป๐ด ๐๐ต๐ฒ ๐๐ต๐ฒ๐ผ๐ฟ๐ ๐ถ๐ป๐๐ผ ๐ฎ๐ป ๐ถ๐ป๐๐ฒ๐ฟ๐๐ฒ๐ป๐๐ถ๐ผ๐ป
8/11 This suggests an optimizer intervention: WW-PGD.
WW-Projected Gradient Descent
After a base optimizer step, select a spectral window, nudge its shape toward ฮฑ = 2, preserve its trace-log before blending, and mix the correction back into the weights.
You can test this now using the Python package: ww-_pgd
๐ง๐ฒ๐๐๐ถ๐ป๐ด ๐๐ต๐ฒ ๐ฝ๐ฟ๐ผ๐ฝ๐ผ๐๐ฒ๐ฑ ๐ฐ๐ผ๐ป๐ป๐ฒ๐ฐ๐๐ถ๐ผ๐ป
7/11 Trained checkpoints let us test the RG picture directly. Near ฮฑ = 2, we compare energy across logarithmic bands, the ECS with the fitted power-law tail, and their normalization scales.
Together, these measurements turn the theory into a concrete experimental program.
๐ช๐ต๐ฎ๐ ๐๐ต๐ฒ ๐ง๐ฟ๐ฎ๐ฐ๐ฒ-๐๐ผ๐ด ๐บ๐ฒ๐ฎ๐๐๐ฟ๐ฒ๐
6/11 The ERG Trace-Log condition has a geometric meaning. For the retained Jaconbian map,
J = โdet(X_R),
so
log J = ยฝ Tr log(X_R).
Evaluated on the normalized operator, zero trace-log means unit retained volume.
This connects volume-preserving training dynamics to the scale-invariant trace-log SETOL construction.
๐ง๐ต๐ฒ ๐ฅ๐ ๐ฉ๐ถ๐ฒ๐
5/11 Renormalization-group (RG) ideas let us follow the correlations that survive training. At each checkpoint, normalize the full spectrum, then select the Effective Correlation Space (ECS).
Optimization supplies the flow;
Selecting the ECS gives us an RG view of it.
๐ช๐ต๐ ๐ฎ๐น๐ฝ๐ต๐ฎ = ๐ฎ ๐บ๐ฎ๐๐๐ฒ๐ฟ๐
4/11 The physical argument begins with spectral energy. For an ideal tail ฯ(ฮป) โ ฮปโปยฒ, equal logarithmic bands contribute equal energy.
In Renormalization Grouo (RG) language, we say that :
ฮฑ = 2 is marginal.
๐ง๐ต๐ฒ ๐๐๐ฎ๐๐ถ๐๐๐ถ๐ฐ๐ฎ๐น ๐ฒ๐
๐ฝ๐น๐ฎ๐ป๐ฎ๐๐ถ๐ผ๐ป
3/11 The statistical idea is self-averaging: as a system grows, relative fluctuations shrink and averages become reliable.
When fluctuations persist, adding more components may not stabilize the result.
We propose this as one way to understand overfitting.
๐ช๐ต๐ ๐๐ฟ๐ฎ๐ถ๐ป๐ถ๐ป๐ด ๐ฝ๐ฒ๐ฟ๐ณ๐ผ๐ฟ๐บ๐ฎ๐ป๐ฐ๐ฒ ๐ถ๐ ๐ป๐ผ๐ ๐ฒ๐ป๐ผ๐๐ด๐ต
2/11 Training can keep improving its own score while the model gets worse on unseen data. In this MNIST run, test accuracy eventually falls, and Correlation Traps (blue) accumulate.
As the model overfits, ฮฑ drops below 2.
The weights carry a record of the breakdown.
Let me share an update to the RG theory paper,
In the new subsection 5.6, I provide an argument that explains how the normalization for the SETOL ERG Trace-Log condition can be equivalent to normalization in power-law fits defined by the MLE estimator in weightwatcher
Hopefully this is useful to you
Ross Perot predicted exactly what happened. We used to have a vibrant manufacturing economy. A strong middle class. A great country.
Then we decided to outsource the entire economy to the CCP, just a few years after the Tiananmen Square massacre
And those of us were old enough to remember her the giant sucking sound
Ross Perot ran for President and told the world what President Trump has also been saying all this time
Our prior Presidents made horrible economic deals that basically gave us managed decline for the last 3 decades
Perot was the first person to tell the nation things that were in plain sight but ignored or worse spun to make all of us think that the mistakes were not mistakes but inevitable
Proud to be a Morgan (from my motherโs side)
The British army once sentenced Daniel Morgan to 500 lashes, a punishment that usually killed a man, for the crime of knocking out an officer who struck him. He survived every stroke. And he spent the rest of his life joking that the drummer had miscounted and only given him 499, so the British still owed him one. He collected that debt at a place called Cowpens.
Morgan was not born to anything. He was a rough, barely educated frontiersman who made his living hauling freight by wagon on the Virginia frontier, a big, powerful, hot-tempered young man. During the French and Indian War, serving as a civilian teamster with the British army, a British lieutenant struck him with the flat of a sword. Morgan answered with his fist and dropped the officer cold. For a colonial to strike a British officer was close to a capital offense, and the court-martial sentenced him to five hundred lashes with the cat-o'-nine-tails, a flogging meant to flay a man and very often to kill him.
They tied him up and laid it on, and Daniel Morgan did not die. He carried the ridged scars on his back for the rest of his life. But instead of breaking him, that brutal punishment forged something. It left him with a cold, permanent hatred of the British army and its contempt for colonial men like him, a hatred he would one day repay with interest. And it gave him his lifelong joke: the drummer lost count and stopped at 499, so the British, he liked to say, were still one lash short of paying him in full.
When the Revolution came, that uneducated wagoner turned out to be a natural soldier of rare gifts. He raised a company of frontier riflemen, crack shots who could kill at distances that terrified British regulars, and he led them north to Saratoga, where his marksmen helped shatter Burgoyne's army and win the turning-point victory of the war. Morgan had a feel for battle that trained European officers lacked, an instinct for ground, for timing, and above all for the men he led. He understood that his militia were not professional soldiers and could not be asked to do what professionals did, so instead of cursing them for it, he built his tactics around what they could do.
That genius reached its peak at the Battle of Cowpens in January 1781, one of the most perfectly fought battles in American history. Facing the dreaded Banastre Tarleton, Morgan set a trap built entirely on human nature. He put his unreliable militia in the front line and asked them only for two volleys, then permission to fall back, which they gladly did. Tarleton's British, seeing the Americans retreat, charged forward in a confident rush straight into Morgan's real line of hardened Continentals, and then Morgan sprang the jaws, hitting the British on both flanks at once in a double envelopment while his cavalry rode down their rear. In minutes Tarleton's command was destroyed, hundreds killed and captured, one of the most crushing and complete victories of the war, and Tarleton himself barely escaped. It was the frontier wagoner out-generaling the darlings of the British army.
So the debt was paid. The boy they tried to flog to death grew up to be the man who methodically dismantled the British in the field, who understood soldiers because he had been treated like dirt as one, and who turned the hard lessons of his own back into some of the finest tactics the Revolution ever saw. Remember Daniel Morgan. The British gave him 499 lashes and made an enemy who never forgot, and he collected the last one at Cowpens, with interest.
