In the course of running Gradescope, we learned what makes for effective evaluation. It's being able to: • create descriptive rubric items • apply multiple rubric items to the thing being evaluated • and -- crucially -- change rubric item weights on the fly When we set out to evaluate different multiplayer AI platforms, we decided to apply the same principles. We created a set of 18 criteria, such as "In a shared conversation, the agent only uses connectors everyone present has access to." We evaluated each platform (Claude Tag, Dust, QM, Superconductor, and Viktor) on each criterion, recording videos so that you can see for yourself. And we presented the results in an interactive table, where you can change the weights of each item, coming up with a score that represents your team's needs. Curious to hear what you think! multiplayer-ai.com/compare-p…

This is the most interesting recent benchmark result that I've seen: The 100-line mini-swe-agent harness gets better performance out of Opus, GPT, and Gemini than their respective bespoke harnesses. (As measured on the excellent DeepSWE bench). Why would that be true?

What percentage of your Twitter feed (the stuff you actually read, not just scroll past) do you believe is currently written by AI?

LLM Provider Comparisons 1. @withmartian 2. @ArtificialAnlys 3. @FixieAI

Has anyone done comprehensive testing of gpt-4-vision-preview? I want to know stuff like the minimum text size it can read, the radius of the smallest circle it can locate in an image, the number of circles it can count, etc. Could be an automated benchmark for other models too

Which set of statements do you agree with? 1. AGI is as much or more of a risk to human flourishing as nuclear weapons 2. I have a good idea for what should be done about that

Has anyone had good experiences with GPT-powered code generation for complete web app features? As in, you describe what should exist, and GPT actually provides the source of all the necessary files and where they should go. Ideally in the context of Ruby on Rails.

Let's say that a US-based research company has developed an AGI model that was able to use the browser, pass captchas, hire people on Upwork, and lie about its intentions. What should they do after observing this?

Solutions from replies: - @OpenPipeAI looks exactly right openpipe.ai - @PortkeyAI launching feature soon - @analyticsaurabh building his own I currently use @helicone_ai, any plans from them?

Is there a service I can use to pipe my GPT-4 calls through, and it automatically finetunes GPT-3.5 (or whatever) on all of them, and lets me know when it's up to par?

It feels like something to be you. Do you think it feels like something to be GPT-4?

By what year will there be an AI that is more capable than most humans in most domains of digital work (e.g. you can tell it to do anything you currently hire a white collar professional to do, and it does the job better than the median human)?

Guaranteed JSON output from any local LLM, with very low overhead! Check out the library and a brief description of the method below the fold. github.com/normal-computing/… "The basic idea is simple: regular expressions have an equivalent Deterministic-Finite Automaton (DFA) representation. We can transform this DFA into a generative model: in each state we get a list of symbols which correspond to completions that partially match the regular expression. We mask the other symbols in the logits returned by a large language model, sample a new symbol and move to the next state." - @remilouf and co at @normalcomputing