David Andrzejewski (@davidandrzej) header
David Andrzejewski (@davidandrzej) profile picture

Software! Systems! Data! @SFMachineLearn Elsewhere: @davidandrzej@sigmoid.social @davidandrzej.bsky.social

San Francisco
Joined September 2008
Photo by David Andrzejewski (@davidandrzej)Photo by David Andrzejewski (@davidandrzej)Photo by David Andrzejewski (@davidandrzej)Photo by David Andrzejewski (@davidandrzej)Photo by David Andrzejewski (@davidandrzej)Photo by David Andrzejewski (@davidandrzej)Photo by David Andrzejewski (@davidandrzej)Photo by David Andrzejewski (@davidandrzej)Photo by David Andrzejewski (@davidandrzej)Photo by David Andrzejewski (@davidandrzej)
Proposal to re-brand "deep learning" as "stochastic regularized estimation of compositionally structured nonlinear functions".
5
24
75
Fun stuff - "In-Context Learning Creates Task Vectors" (via @_akhaliq) isolates something (more or less) resembling the "program key" described below as an internal attention state, demonstrating with state substitution/patching experiments huggingface.co/papers/2310.1…
My interpretation of prompt engineering is this: 1. A LLM is a repository of many (millions) of vector programs mined from human-generated data, learned implicitly as a by-product of language compression. A "vector program" is just a very non-linear function that maps part of the latent space unto itself. 2. When you're prompting, you're fetching one of these programs and running it on an input -- part of your prompt serves as a kind of "program key" (as in database key) and part serves as program argument(s). Like, in "write this paragraph in the style of Shakespeare: {my paragraph}", the part "write this paragraph in the stye of X: Y" is a program key, with arguments X=Shakespeare and Y={my paragraph}. 3. The program fetched by your key may or may not work well for the task at hand. There's no reason why it should be optimal. There are lots of related programs to choose from. 4. Prompt engineering represents a search over many keys in order a find a program that is empirically more accurate for what you're trying to do. It's no different than trying different keywords when searching for a Python library. 5. Everything else is unnecessary anthropomorphism on the part of the prompter. You're not talking to a human who understands language the way you do. Stop pretending you are.
1
1
651
Frequent SF (Nob Hill) sighting recently: astonished & delighted tourists taking photos/videos of the self-driving cars.
232
Paper itself is an absolute treat: great diagrams, abundant code samples, and as a nice bonus some historical context around the origins and development of the research ideas. Credit: saw it in @deliprao's "AI Research & Strategy" newsletter deliprao.substack.com/
6
Easy to get "breakthrough fatigue" in ML recently, but in this work the NN learns *how to learn* linear regression, decision trees, 2-layer ReLU nets 😲 "...we train Transformer models to discover algorithms for different learning problems." arxiv.org/abs/2208.01066
8
68
5
446
...furthermore: outperforms XGBoost, does Lasso in one-pass, seems not to rely on nearest-neighbor. Future work: "...an intriguing possibility where we might be able to reverse engineer the Transformer to obtain better learning algorithms." (!)
2
1
25
Day 1 of #Illuminate22 starts tomorrow! Check out some of the talks that will kick off at 1:00 am PT/4:00 am ET including @SumoLogic's general session on #reliability management and #security in one platform as well as speakers from @eSentire and @Zyston. CVSoci.al/1hILg6Wx
lmao coinbase: “I understand”
Pondering the TBs of data crunched in an ultramodern cloud ML platform in order to power cutting-edge causal models targeting tightly business-aligned KPIs in one of the world’s most esteemed tech firms, all leading up to this email.
2
2
I’m hiring for an ML Platform Engineer role in the SF Bay Area - we’ve got some interesting problems, please let me know if you’d want to learn more! boards.greenhouse.io/sumolog…
2
2
Is there any writing attempting to project current SF hot-button issues like schools or crime into a hypothetical near-future where local tax revenues and property values are annihilated by Tech-exodus/WFAnywhere, the collapse of tourism/conference industry, etc?
1
2
2
Guess the subject area: "An obvious extension of the basic arithmetic theories discussed so far is to add multiplication. Unfortunately, this dramatically increases the complexity of the problem, and so is often avoided in practice."
Striking example of Simpson's Paradox in Covid-19 Case Fatality Rates (CFR): Italy has higher total/overall CFR while China has higher CFR within each individual age group - from @bschoelkopf Causality lecture at MLSS 2020: youtu.be/btmJtThWmhA
2
“A Blueprint of Standardized and Composable Machine Learning” - how to unify the “zoo” of disjointed ML approaches? Thought-provoking talk by Eric Xing (@mldcmu) as part of the consistently excellent @IASMLSeminars series video.ias.edu/machinelearnin…
2
4
“The Idea Factory: Bell Labs and the Great Age of American Innovation” by Jon Gertner vs The Onion theonion.com/new-technologic…
1
I listened to a story about extreme spelunking on NPR 10 years (!?) ago and the description of some phenomenon known as "the rapture" has vividly stuck with me to this day npr.org/templates/story/stor…
Highly recommend the excellent “Projects To Know” newsletter from @sarahcat21 & @AmplifyPartners: short, well-curated digests of interesting recent ML research papers, code repos, & blogs mailchi.mp/amplifypartners/p…
7
6