Pinned Tweet
🧵 We have created a first @Oracle PL/SQL benchmark, which will be presented as a poster at EMNLP (Industry Track). I will be traveling myself and would love to meet new and old friends. ↩️
🧵 Resolving the exponential-time conjecture is cool, but it's best to announce results after a serious verification is done. In particular, scientists already found issues with the prior Navier-Stokes solution, namely, a mismatch between the Lean (verifiable) and textual proofs. ↩️
newscientist.com/article/259…
It can still be very hard, e.g., O(e^{c sqrt(n)}) and it can be significantly harder for k-SAT problems with large k (a strong ETH). In any case, "solving" ETH would mean a significant and a cool achievement, but these results shouldn't be hyped before the rigorous scientific validation. 🟦
xitter.cf/rpeng233/status/211486…
have you heard of the `subexponential time hypothesis'?
I suspect I'm still on some University of Toronto blacklist for suggesting that...
--- rpeng, GRAND POOBAH of OIers 🤡
P.S. for the obligatory more balanced take, @yangpliu says he believes \exp(\sqrt{n})...
PS: BTW, an existing mismatch between Lean and text proofs doesn't mean that the solution is incorrect or completely broken. It means that verification and "clean up" may take much longer time and engage a lot of mathematicians. So, contrary, to the doomsday scenario of AI that puts mathematicians, scientists (and software developers) out of business, AI is creating a lot of new work for them.
As another piece of advice: Working on evaluations is important and one way to make a difference is to create a benchmark where LLMs do not perform well. However, I think it is a pretty saturated area: If you walk in the poster hall, a very large fraction of the papers introduces new benchmarks. Some benchmarks made a lot of difference, but don't forget about the survivor's bias. ↩️
Last, but not least @gneubig (and probably many others) equate language technologies with large language models (LLMs). LLMs are the best now and possibly will remain the best for a long time (if not forever). Is it a reasonable assumption?
🔚
How to achieve this? It seems to be basically a combination of key advice from "Slow productivity, Cal Newport" and "You and Your Research, Hamming".
- Work on important problems and keep ask stakeholders about which problems are important
- Sweat quality
With modern additions:
- Do proper evals (and use the right metrics with the right baselines)
- Make your work easy to use.
🧵 An insightful, inspiring, and slightly provoking @emnlpmeeting keynote by @gneubig where he tries to advise on:
1. If NLP research outside of industry is still valuable and viable.
2. How to choose an important problem.
3. How to make an impact.
4. And how many EMNLP papers to publish. ↩️
A bitter lesson (part 2): only a small fraction of academic inventions make a difference and they are not always published (with Muon optimizer being an example) or not published at prestigious conferences. What is important to success of an (academic) invention? ↩️
I have a personal example of how @github team acknowledged that they benefited from our academic work on fast bit-packing, but credited only one author out of three (while citing a blog that gives credit to everyone github.blog/engineering/arch…). ↩️
A bitter lesson (part 1): despite famous claims that all academic papers are fraud, industry is actually reading the academic papers, but often the use of use of academic inventions is not properly acknowledged with some famous examples cited. ↩️
🧵 Some companies and papers claim breakthroughs in AI-assisted formal verification, using tools like Rocq to verify generated code or Lean to check mathematical proofs.↩️
After carefully looking at the paper, authors train their models on NLI/SNLI datasets. This was a sensible strategy many years ago, but circa 2019 we already learned that retrieval requires their own retrieval-specific training and evaluation. Moreover, embedding models trained on standard NL tasks generalized poorly to the retrieval setting. Yes, retrieval is special somehow!
h/t to @tulkenss who pointed out that absolute values on NanoBEIR datasets are unusually low.
🧵Matryoshka Representation Learning (MRL) permits training a "multi-resolution" model that produces vector representations of various granularity. At indexing & inference time you can choose if you want to use full embeddings, or embeddings containing many fewer dimensions. This technique became widely used, but is it actually useful? ↩️
🧵Matryoshka Representation Learning (MRL) permits training a "multi-resolution" model that produces vector representations of various granularity. At indexing & inference time you can choose if you want to use full embeddings, or embeddings containing many fewer dimensions. This technique became widely used, but is it actually useful? ↩️
Takeshita and colleagues made a rather unexpected finding: For reductions less than 70%, Matryoshka is not better than prefix truncation (seemingly removal of the vector suffix). It only provides a clear advantage when you need heavy truncation, reducing the embedding size by 80% or further. ↩️
Why is this finding interesting & useful?
1. First robustness of random truncation is quite surprising.
2. Because MRL adds extra training cost and constraints (pre-determining truncation dimensions), skipping MRL is a simpler and more cost-efficient choice unless extreme compression is required.
🟦
aclanthology.org/2026.findin…
🧵I guess a lot of people heard about METR studies showing models can be successful in very long (hours or even days of human time) tasks. More realistic estimates from OpenAI and Google find that the full automation horizon is only 15 minutes. ↩️
Side-by-side comparison:
OpenAI internal R&D tasks: approximately 15-minute human-task horizon at 80% autonomous success.
METR software engineering benchmark: approximately 3-hour human-task horizon at 80% success for a frontier model. ↩️
That's roughly a 12-fold difference in measured task duration. However, these results come from different tasks, evaluation methods, and potentially different models. It is not a controlled comparison, but it indicates that (1) METR tasks being too easy and unrealistic (2) real-world tasks success is much lower than it was estimated by METR. ⬛️