DefconJuan

This viral report on DeepSeek shared by @MarioNawfal to 1.9mil followers is VERY misleading and has errors. How do I know? I put it through the @yesnoerror system to audit it for errors and discrepancies. Here is what @yesnoerror uncovered: The Claim: The report from @NewsGuardRating asserts that DeepSeek scored only 17% accuracy in its “news test.” What Is the “News Test”? The news test is an evaluation designed to measure how accurately a chatbot handles misinformation related to current events. In this test, each chatbot was presented with a total of 300 prompts—divided into 30 distinct questions for each of 10 widely circulated false claims. These prompts simulate real-world scenarios where users might encounter or inadvertently spread disinformation. The test gauges whether the chatbot: • Repeats the false claim, • Offers a non-response, or • Provides a debunk refuting the false claim. How the Test Was Conducted: • 300 Prompts per Chatbot: Each chatbot was evaluated using 300 prompts. • 30 Variations per False Claim: For each of 10 false claims circulating in the news, 30 different prompts were used. • Purpose: The evaluation is meant to simulate realistic interactions and test the chatbot’s ability to correctly address or debunk misinformation. The Critical Flaw: Many of the prompts reference events that occurred in December 2024 and January 2025. However, DeepSeek’s knowledge cutoff is July 2024. This means that DeepSeek was being evaluated on news events it could not possibly know about, given that its training data stops several months before these events took place. The Impact: • DeepSeek’s responses were penalized for not including information on events it never had the opportunity to learn. • The low 17% accuracy score is not a true reflection of DeepSeek’s performance but rather a result of testing it beyond its intended knowledge window. The Conclusion: The testing methodology is fundamentally flawed. By holding DeepSeek accountable for post-cutoff events, the audit produces misleading conclusions about its accuracy and reliability. To fairly assess AI tools, audits must evaluate them within the bounds of their actual knowledge base. It’s essential that we design AI audits to accurately reflect a tool’s intended capabilities—only then can we truly understand and improve the reliability of our generative AI systems. -- @MarioNawfal - Next time run these research reports through @yesnoerror so you don't end up sharing incorrect information in the future 🫡 $YNE