@GetProvably

Making agentic workflows hallucination and poison free powered by our verifiable databases and cryptography tools.

 Switzerland
provably.ai
Joined April 2024
  • Tweets233
  • Following87
  • Followers360
74Media
SourceryKit in early adoption: - detects 100% of agent tool-call errors. - improves accuracy by up to 50% in some models. - works across single-agent and multi-agent workflows. This is why we are building runtime verification for production agents.
1
2
6
233
1/4 AI agents use tools to find information. Retries can recover what was missing. But does the answer use what they found? We examined 24 repaired runs to test whether organising the tool results helped.
1
3
4
290
more replies
3/4 How many answers included every supported fact? • Conversation alone: 12 of 24 • With the organised record: 14 of 24 Two more complete answers. A preliminary result, not proof of higher overall accuracy.
1
7
4/4 Recovering information is only part of the job. We also need to check whether it reaches the answer. SourceryKit makes that gap checkable. Read the experiment: switchboard.provably.ai/blog…

The Evidence Was Recovered. Did It Reach the Answer? | Switchboard

In 9 of 24 repaired runs, no model was asked to write a new answer after recovery. Where synthesis did run, an execution ledger raised supported values reaching the answer from 73.9% to 81.2%, a...

switchboard.provably.ai
4
1/7 Even the best agents get tool calls wrong. What helps them recover? Our previous study showed why a convincing answer isn’t enough. We took 60 failed runs and tried three recovery methods on them. Here’s what we learned.
1
4
6
202
more replies
6/7 One hard task followed a chain from a museum to its architect, his birthplace, a library and the nearest subway. Guided recovery found evidence for all five expected claims. The final answer still omitted them. Finding the evidence isn’t the same as using it.
1
6
7/7 Of 32 guided runs that recovered complete evidence, only 11 also passed the final-answer check. The next challenge: getting that evidence into the answer. These were selected failures, weighted toward missing tool evidence. provably.ai/blogs/the-trace-…

Switchboard | Expert help in your users’ harness

Connect your expert agents to customers’ agents with Switchboard. Share context, resolve issues faster and offer free support or paid expertise.

switchboard.provably.ai
7
Provably retweeted
Shyam
@ProvablyShyamAug 13
anthropic.com/research/multi… This is fantastic read. 1. agents struggle to coordinate with each other especially for common resources. 2. they don't know how to detect when they are being lied to. And as usual Anthropic wants to solve it through more learning..

Patterns and problems in multiagent systems

We ran experiments on swarms of Claude agents and found coordination failures, collusion, and sabotage. Here, we share what they mean for AI safety.

anthropic.com
2
3
51
We spent 9 weeks testing agent reliability during software tool use. Nearly 60% of claims accepted as correct could not be backed by evidence from recorded tool calls. The answer looked right. The evidence wasn't. 🧵
2
5
6
132
more replies
We believe agents should capture evidence natively as they work. SourceryKit captures tool calls in a state of the art verifiable database, allowing agents to evaluate themselves and downstream agents to independently verify their answers.
1
12
SourceryKit detected 100% of the covered deterministic grounding errors in this study. Next, we will test how much accuracy agents can recover through answer healing and targeted retries. Read the study: provably.ai/blogs/The-Agent-…

Switchboard | Expert help in your users’ harness

Connect your expert agents to customers’ agents with Switchboard. Share context, resolve issues faster and offer free support or paid expertise.

switchboard.provably.ai
9
Provably retweeted
Shyam
@ProvablyShyamAug 4
AI is going collaborative. In the next 6 months all we are gonna talk about is verifiability (for safety, security and accuracy), protocols and crazy new use cases.
3
3
45
Load more