E𝛁a🏳️‍⚧️ princess of edge cases

GPT-4 is getting worse over time, not better. Many people have reported noticing a significant degradation in the quality of the model responses, but so far, it was all anecdotal. But now we know. At least one study shows how the June version of GPT-4 is objectively worse than the version released in March on a few tasks. The team evaluated the models using a dataset of 500 problems where the models had to figure out whether a given integer was prime. In March, GPT-4 answered correctly 488 of these questions. In June, it only got 12 correct answers. From 97.6% success rate down to 2.4%! But it gets worse! The team used Chain-of-Thought to help the model reason: "Is 17077 a prime number? Think step by step." Chain-of-Thought is a popular technique that significantly improves answers. Unfortunately, the latest version of GPT-4 did not generate intermediate steps and instead answered incorrectly with a simple "No." Code generation has also gotten worse. The team built a dataset with 50 easy problems from LeetCode and measured how many GPT-4 answers ran without any changes. The March version succeeded in 52% of the problems, but this dropped to a pale 10% using the model from June. Why is this happening? We assume that OpenAI pushes changes continuously, but we don't know how the process works and how they evaluate whether the models are improving or regressing. Rumors suggest they are using several smaller and specialized GPT-4 models that act similarly to a large model but are less expensive to run. When a user asks a question, the system decides which model to send the query to. Cheaper and faster, but could this new approach be the problem behind the degradation in quality? In my opinion, this is a red flag for anyone building applications that rely on GPT-4. Having the behavior of an LLM change over time is not acceptable. Have you noticed any issues when using GPT-4 and ChatGPT lately? Do you think these problems are overblown?