Bigpeachpie

Continued training on junk social posts makes LLMs think worse, and causes irreversible reasoning decline and negative behaviors. Feeding LLMs lots of short, popular, or low-quality Twitter/X text causes lasting drops in reasoning, long-context use, and safety. This big concern here is that because many teams keep topping up models with web crawls. Feeding highly popular or shallow posts can cause lasting, hard to undo damage. They ran controlled tests that kept token counts and training steps the same. Junk came in 2 forms, M1 based on popularity and short length, and M2 based on clickbait style and shallow topics. Popularity of posts is a stronger predictor of harm than length. Training on very popular posts hurts step-by-step reasoning more, while training on very short posts mainly weakens long-context use. Across 4 instruction tuned models, junk exposure reduced reasoning, long context retrieval, and safety. With more junk, performance dropped in a dose like way. For example, ARC with chain of thought fell from 74.9 to 57.2, and RULER CWE fell from 84.4 to 52.3. They also measured personality style and saw higher narcissism and psychopathy with lower agreeableness. The main failure was thought skipping, the model answered without planning or it dropped steps. Instruction tuning and more clean pretraining helped a little, but they did not restore the baseline. Popularity predicted harm better than length, so curation during continual pretraining matters for safety.