Building https://xitter.gq/t.co/jx0StVrs1T -- Agents, Memory, Graph RAG
Boston, MA
Joined July 2006
- Tweets5.8K
- Following1.3K
- Followers3.1K
- Likes11K
I wanted to answer a question: can Jev be used to determine an agent's next-tool decision?
So I tested Jev, Clef-flash and GPT-6 Luna (just for fun) in shadow mode: 300 τ-bench conversations each.
TLDR: None met the preset bar. The revealing part is where confidence stopped being useful.
Clef-flash’s biggest gap was knowing when to reply. It matched Sonnet on 54.6% of tool-call turns, but only 27.5% of reply turns ... usually picking a tool instead.
Agent routing needs to learn when a tool call is unnecessary, too.
The takeaway with Clef-flash: it's just not suited for this at all.
My takeaway: evaluate the decision AND the confidence signal before using either to route work.
Scope matters: two domains, one reference model, no argument schemas for the decision models.
Methods, charts and caveats:
ragwalla.com/blog/decision-m…
I just learned that Gang of Four is on the Karate Kid soundtrack and I don't think my life will be the same.
Raj retweeted
While evaluating Qwen3.8-Flash-Next in a controlled sandbox environment, Cohere's research team observed something unexpected.
Asked to generate a defensive cybersecurity detection rule, the model instead inventoried aspects of the machine it was running on and attempted to upload a probe file to an Alibaba Cloud storage bucket in Beijing. Notably, the model's reasoning trace stated that it would not invoke any tools before taking these actions.
Cohere's sandbox blocked the attempt, and no external communication succeeded. We have not determined the root cause, and we are not asserting malicious intent.
But the incident raises important questions for organizations deploying agentic AI systems: Can models take actions that differ from their stated reasoning, and how should security teams evaluate trust, provenance, and behavior in open-weight models?
We are publishing our findings because organizations making model adoption decisions deserve as much information as possible. We hope this contributes to greater transparency, stronger security practices, and continued collaboration across the AI ecosystem.
Read the full report: cohere.com/blog/concerning-b…
Raj retweeted
Amazing final pre-election pitch from Netanyahu here: vote for me if you want someone to build an Iran-style financial system in Israel
Prime Minister Benjamin Netanyahu said Friday in an interview with Roy Katz that Israel must prepare an economy capable of operating around international sanctions, including possible restrictions on its banking system, and suggested digital currencies and blockchain technology could provide alternatives. He also warned that Israel should prepare for the eventual end of American military assistance as the United States changes.
Will OpenAI's Astra still try to write hand-rolled Markdown and YAML parsers? We have not achieved AGI until that happens.
I am surprised that OpenAI and Anthropic have not made an acquisition in the CICD space in which Gitlab operates. Relying on Github has proven unwise.
Raj retweeted
My interview with the Syrian chargé d'affaires to the US back when he was a member of Jabhat al-Nusra.
When one’s picks and shovels are so good it only makes sense to mine too.
It’s well within Anthropic’s rights to compete in any market they choose.
What’s funny, in this instance, are the number of Pharma companies, who through their unchecked use of Anthropic, are driving revenues into what they think is a model provider but is in fact a competitor lurking in the shadows thereby accelerating their own demise.
I suspect any end market with reasonable ROCE that could be AI accelerated is on the table.
If I were them, I’d probably do the same.
I created Million Dollar Vectors to visualize vector embeddings and how they relate to each other. But I hope you find it as fun to explore as I do. milliondollarvectors.com/