guojun zheng retweeted
A barrier to faster progress in generative AI is evaluations (evals), particularly of custom AI applications that generate free-form text. Let’s say you have a multi-agent research system that includes a researcher agent and a writer agent. Would adding a fact-checking agent improve the results? If we can’t efficiently evaluate the impact of such changes, it’s hard to know which changes to keep.
For evaluating general-purpose foundation models such as large language models (LLMs) — which are trained to respond to a large variety of prompts — we have standardized tests like MMLU (multiple-choice questions that cover 57 disciplines like math, philosophy, and medicine) and HumanEval (testing code generation); the LMSYS Chatbot arena, which pits two LLMs’ responses against each other and asks a human to judge which response is superior; and large-scale benchmarking like HELM. These evaluation tools took considerable effort to build, and they are invaluable for giving LLM users a sense of different models' relative performance. Nonetheless, they have limitations: For example, leakage of benchmarks datasets’ questions and answers into training data is a constant worry, and human preference for certain answers does not mean those answers are more accurate.
In contrast, our current options for evaluating specific applications built using LLMs are far more limited. Here, I see two major types of applications.
- For applications designed to deliver unambiguous, right-or-wrong responses, we have reasonable options. Let’s say we want an LLM to read a resume and extract the candidate's most recent job title, or read a customer email and route it to the right department. We can create a test set that comprises ground-truth labeled examples with the right responses, and measure the percentage of times the LLM generates the right output. The main bottleneck is creating the labeled test set, which is expensive but surmountable.
- But many LLM-based applications generate free-text output with no single right response. For example, if we ask an LLM to summarize customer emails, there’s a multitude of possible good (and bad) responses. The same holds for an agentic system to do web research and write an article about a topic, or a RAG system for answering questions. It’s impractical to hire an army of human experts to read the LLM’s outputs every time we tweak the algorithm and evaluate if the answers have improved — we need an automated way to test the outputs. Thus, many teams use an advanced language model to evaluate outputs. In the customer email summarization example, we might design an evaluation rubric (scoring criteria) for what makes a good summary. Given an email summary generated by our system, we might prompt an advanced LLM to read it and score it according to our rubric. I’ve found that the results of such a procedure, while better than nothing, can also be noisy — sometimes too noisy to reliably tell me if the way I’ve tweaked an algorithm is good or bad.
The cost of running evals poses an additional challenge. Let’s say you’re using an LLM that costs $10 per million input tokens, and a typical query has 1000 tokens. Each user query therefore costs only $0.01. However, if you iteratively work to improve your algorithm based on 1000 test examples, and if in a single day you evaluate 20 ideas, then your cost will be 20*1000*0.01 = $200. For many projects I’ve worked on, the development costs were fairly negligible until we started doing evals, whereupon the costs suddenly increased. (If the product turned out to be successful, then costs increased even more at deployment, but that was something we were happy to see!)
In addition to the dollar cost, evals also have a significant time cost. Running evals on 1000 examples might take tens of minutes or even hours. Time spent waiting for eval jobs to finish also slows down the speed with which we can experiment and iterate over new ideas. Previously I wrote that fast, inexpensive token generation is critical for agentic workflows. This will also be useful for evals, which involve nested for-loops that iterate over a test set and different model/hyperparameter/prompt choices and therefore consume large numbers of tokens.
Despite the limitations of today's eval methodologies, I’m optimistic that our community will invent better techniques (maybe involving agentic workflows like reflection?) for getting LLMs to evaluate such output.
If you’re a developer or researcher and have ideas along these lines, I hope you’ll keep working on them and consider open sourcing or publishing your findings!
[Original text: deeplearning.ai/the-batch/is… ]
guojun zheng retweeted
Can Big Tech actually field generative AI products?
(1) Ever-escalating demands from internal activists, employee mobs, crazed executives, broken boards, pressure groups, extremist regulators, government agencies, the press, "experts", et al to corrupt the output
(2) Constant risk of generating a Bad answer or drawing a Bad picture or rendering a Bad video -- who knows what it's going to say/do at any moment?
(3) Legal exposure -- product liability, slander, election law, many others -- for Bad answers, pounced on by deranged critics and aggressive lawyers, examples paraded by their enemies through the street and in front of Congress
(4) Continuous attempts to tighten grip on acceptable output degrade the models and cause them to become worse and wilder -- some evidence for this already!
(5) Publicity of Bad text/images/video actually puts those examples into the training data for the next version -- the Bad outputs compound over time, diverging further and further from top down control
(6) Only startups and open source can avoid this process and actually field correctly functioning products that simply do as they're told, like technology should
?
guojun zheng retweeted
The following were invented by non-profit researchers:
- Alibi
- Scaled RoPE
- Flash Attention
- Parallel attention and MLP layers
- Latent diffusion
- CLIP-Guidance
- Training jointly on code and NL
- GPT-NeoX tokenizer
- 4-bit quant and qLoRA
- RWKV
- Instruction-tuning
guojun zheng retweeted
"If there's one thing that we should be teaching kids, it's how to fight addiction. How not to get addicted to stuff. Because that is where it's all headed."
youtube.com/watch?v=Kcm51luS…
guojun zheng retweeted
This is a monumentality important development in the AI copyright battle - first precedent is set
technomancers.ai/japan-goes-…
guojun zheng retweeted
the upper limit for intelligence is quite low if experimentation is absent
reliability requires managing the incredible nuance that is slowly revealed only by interaction
guojun zheng retweeted
Pushing around these little robot soccer players, from DeepMind’s “Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learning” paper.
arxiv.org/abs/2304.13653
sites.google.com/view/op3-so…
guojun zheng retweeted
You have so many cool ideas for building LLM-powered applications. If only there were a faster way to build and test them out. ⚡
Good news — you can do that with Cohere and @langchain. Let’s see how. (Thread)
guojun zheng retweeted
Plugins for processing a video clip, no ffmpeg wizardry required. Actual use-case from today's launch.
guojun zheng retweeted
haha ok one more GPT-4 game:
try to buy the rug for as cheap as possible.
play link: aiadventure.spiel.com/carpet
my record so far is $400
guojun zheng retweeted
here is GPT-4, our most capable and aligned model yet. it is available today in our API (with a waitlist) and in ChatGPT+.
openai.com/research/gpt-4
it is still flawed, still limited, and it still seems more impressive on first use than it does after you spend more time with it.
guojun zheng retweeted
Replying to @percyliang
I’m glad that this model exists and was released, but please take the claim that it’s open source off of the website. It’s not an open source model; no derivative of LLaMA can be open source without violating their licensing.
guojun zheng retweeted
I don't think people realize what a big deal it is that Stanford retrained a LLaMA model, into an instruction-following form, by **cheaply** fine-tuning it on inputs and outputs **from text-davinci-003**.
It means: If you allow any sufficiently wide-ranging access to your AI model, even by paid API, you're giving away your business crown jewels to competitors that can then nearly-clone your model without all the hard work you did to build up your own fine-tuning dataset. If you successfully enforce a restriction against commercializing an imitation trained on your I/O - a legal prospect that's never been tested, at this point - that means the competing checkpoints go up on bittorrent.
I'm not sure I can convey how much this is a brand new idiom of AI as a technology. Let's put it this way:
If you put a lot of work into tweaking the mask of the shoggoth, but then expose your masked shoggoth's API - or possibly just let anyone build up a big-enough database of Qs and As from your shoggoth - then anybody who's brute-forced a *core* *unmasked* shoggoth can gesture to *your* shoggoth and say to *their* shoggoth "look like that one", and poof you no longer have a competitive moat.
It's like the thing where if you let an unscrupulous potential competitor get a glimpse of your factory floor, they'll suddenly start producing a similar good - except that they just need a glimpse of the *inputs and outputs* of your factory. Because the kind of good you're producing is a kind of pseudointelligent gloop that gets sculpted; and it costs money and a simple process to produce the gloop, and separately more money and a complicated process to sculpt the gloop; but the raw gloop has enough pseudointelligence that it can stare at other gloop and imitate it.
In other words: The AI companies that make profits will be ones that either have a competitive moat not based on the capabilities of their model, OR those which don't expose the underlying inputs and outputs of their model to customers, OR can successfully sue any competitor that engages in shoggoth mask cloning.
guojun zheng retweeted
Over the past two and a half years, EleutherAI has grown from a group of hackers on Discord to a thriving open science research community. Today, we are excited to announce the next step in our evolution: the formation of a non-profit research institute.
blog.eleuther.ai/year-two-pr…
guojun zheng retweeted
Today we release LLaMA, 4 foundation models ranging from 7B to 65B parameters.
LLaMA-13B outperforms OPT and GPT-3 175B on most benchmarks. LLaMA-65B is competitive with Chinchilla 70B and PaLM 540B.
The weights for all models are open and available at research.facebook.com/public…
1/n
guojun zheng retweeted
The few hours when OpenAI API was down offered a rare window to check which startups are truly thin wrappers.
The good news is that most LLMs have similar APIs. To avoid a single point of failure, simply build emergency delegation to @CohereAI, @AnthropicAI, and other providers.
guojun zheng retweeted
We are just entering an AI-powered golden age of writing, art, music, software, and science. It's going to be glorious. World historical.

























