Shawn Morel retweeted
Is this guy talking about DSPy?
We're missing (at least one) major paradigm for LLM learning. Not sure what to call it, possibly it has a name - system prompt learning?
Pretraining is for knowledge.
Finetuning (SL/RL) is for habitual behavior.
Both of these involve a change in parameters but a lot of human learning feels more like a change in system prompt. You encounter a problem, figure something out, then "remember" something in fairly explicit terms for the next time. E.g. "It seems when I encounter this and that kind of a problem, I should try this and that kind of an approach/solution". It feels more like taking notes for yourself, i.e. something like the "Memory" feature but not to store per-user random facts, but general/global problem solving knowledge and strategies. LLMs are quite literally like the guy in Memento, except we haven't given them their scratchpad yet. Note that this paradigm is also significantly more powerful and data efficient because a knowledge-guided "review" stage is a significantly higher dimensional feedback channel than a reward scaler.
I was prompted to jot down this shower of thoughts after reading through Claude's system prompt, which currently seems to be around 17,000 words, specifying not just basic behavior style/preferences (e.g. refuse various requests related to song lyrics) but also a large amount of general problem solving strategies, e.g.:
"If Claude is asked to count words, letters, and characters, it thinks step by step before answering the person. It explicitly counts the words, letters, or characters by assigning a number to each. It only answers the person once it has performed this explicit counting step."
This is to help Claude solve 'r' in strawberry etc. Imo this is not the kind of problem solving knowledge that should be baked into weights via Reinforcement Learning, or least not immediately/exclusively. And it certainly shouldn't come from human engineers writing system prompts by hand. It should come from System Prompt learning, which resembles RL in the setup, with the exception of the learning algorithm (edits vs gradient descent). A large section of the LLM system prompt could be written via system prompt learning, it would look a bit like the LLM writing a book for itself on how to solve problems. If this works it would be a new/powerful learning paradigm. With a lot of details left to figure out (how do the edits work? can/should you learn the edit system? how do you gradually move knowledge from the explicit system text to habitual weights, as humans seem to do? etc.).
Shawn Morel retweeted
Biology’s lack of data is holding back its AI boom.
Epoch’s latest report shows explosive growth in biological model training data size from 2017–2021 (9.7×/year), but a (2.1x/year) plateau since.
AI models for biology are ready to transform science if the data can keep up.
Shawn Morel retweeted
The first of (hopefully) many more tools and tutorials for the comp bio community on how to use Evo 🧬
What’s also incredible, you can prompt with text to control dna design
We aim for impact and to accelerate science where we can 🧪 🧬 ⚡️
@MichaelPoli6 @ai_with_brains
In case you missed it: a new 7B StripedHyena model is out (the longest context one, yet), Evo-1 7B 🧬.
And it now runs in a single notebook (powered by @togethercompute), from DNA generation to protein fold.
Shawn Morel retweeted
every time someone says they are building a product for data scientists i have to remind them that there are only like 100,000 data scientists total
Shawn Morel retweeted
Paper copies of The Engineering Executive’s Primer now shipping from Amazon as of last night! Just have to figure out audiobook details.
Excited to see copies start showing up in the world!
Shawn Morel retweeted
Announcing our preprint understanding transfer learning for protein language models (PLMs), led by former MSRNE intern @francescazfl, with @KevinKaichuang @avapamini @yisongyue
Key takeaway:
PLMs do not scale for anything except structure!
🧵👇
biorxiv.org/content/10.1101/…
Shawn Morel retweeted
I’m super excited to share that I’m joining @controlplaneio where I’ll be helping the company expand their security services portfolio to @fluxcd.
Shawn Morel retweeted
Vision transformers need registers!
Or at least, it seems they 𝘸𝘢𝘯𝘵 some…
ViTs have artifacts in attention maps. It’s due to the model using these patches as “registers”.
Just add new tokens (“[reg]”):
- no artifacts
- interpretable attention maps 🦖
- improved performances!
Shawn Morel retweeted
Our North Star is a Cargo-like experience for Python: a streamlined workflow whereby your tools “just work” and compose into a unified toolchain.
The Ruff formatter is the next step in that direction. If you like the linter, I think you'll like the formatter too.
Shawn Morel retweeted
pair programming is very high leverage — you don’t just get the immediate task done, but also teach someone to be able to do it themselves
Shawn Morel retweeted
How well do🧬 DNA LMs perform on prediction tasks that are biologically relevant, are naturally very imbalanced and require reasoning over long contexts? We present BEND, a benchmark for DNA LMs on the human genome
arxiv.org/abs/2311.12570
Shawn Morel retweeted
The Mojo internals technical talk is now available👇. Dig in to learn more about how Mojo's compiler architecture is a step forward from other languages, about our quest to lift performance engineering out of compilers and into libraries, and Mojo for heterogenous compute. 🔥🔥
Our 2nd keynote, "Mojo 🔥: A system programming language for heterogenous computing", from the 2023 LLVM Developers' Meeting is now available: youtu.be/SEwTjZvy8vw #Mojo #LLVMDevMtg
Shawn Morel retweeted
Interested in LLMs for genomic research but don't know where to start? looking for a review/survey to get started in this field? 👇👇😀
I am very excited to share that our review paper titled "To Transformers and Beyond: Large Language Models for the Genome" is now available as a preprint (arxiv.org/abs/2311.07621)! Our review unveils a revolution in genomics analysis with Genome LLMs. 🧬
🔍 What's Inside:
✅ The power & challenges of transformers in genomics.
✅ Cutting-edge models like HeynaDNA and scGPT & their impact.
✅ Deep dives into Enformer, DNABERT, and other Genome LLMs.
🌍 Why it Matters:
1. GPT-4's influence reshapes AI in genomics.
Unmatched insights into transformers' role in genomics.
2. Critical analysis of new models, addressing interpretability, privacy, & computational needs.
3. Essential for computational biologists & computer scientists to navigate the future of genomic data analysis.
This is a work led by the amazing PhD student, Mica Consens, in the lab! Also, a huge collaborative work with lots of field leaders @fabian_theis @genophoria @MKarimzade @michaelwainberg and Alan Moses!
@UofT @VectorInst @UofTCompSci @UofT_LMP @UHN @pmcc_ai @UHNAIHUB
Shawn Morel retweeted
Replying to @LondonBreed @SF_DPH
Could you maybe start classing fentanyl as food, so that you'll also be able to crack down on unlicensed street vendors of that?
Shawn Morel retweeted
Check out this incredible blog post benchmarking gradient descent, showing Mojo 🔥 outperforming C++ and JAX 😱 Another incredible community project 🚀💯
control.limited/mojo-does-gi…
#mojo #modular #programming #ai #ml
Shawn Morel retweeted
Mojo🔥 0.4 just dropped today with a MASSIVE number of updates across language, library and tools built in the last two weeks. 🥰 Mac support didn't quite make the train but is looking promising for release soon! 🧑💻
docs.modular.com/mojo/change…




































