Joshua Orlich, CPA

One of the biggest risks of working with AI is how convincingly wrong it can be. I think majority of us that are using AI have experienced this first hand. And that's precisely what makes it dangerous. One question I get asked frequently is: "What do you do to make sure it's right?" One of the ways I address this is through something called adversarial review. Here's how I approach it. 1. Bring in a fresh set of eyes. Let's say I've been using Claude to build an application/workpaper/whatever. Rather than asking Claude to review its own work, I'll bring in Codex. A different model with a completely separate context window. One that has not had any involvement in the original development process. I'll give Codex access to the codebase/folder/docs/etc.., explain what the application is supposed to accomplish, and ask it to critically evaluate the work. The objective isn't to validate Claude's decisions. It's to identify what Claude might have missed or gotten wrong. 2. Make the review adversarial. I don't simply ask the reviewing agent to "check the work." I explicitly instruct it to challenge the work. Look for bugs. Question assumptions. Check for bad formulas. Identify logical inconsistencies, security vulnerabilities, unsupported conclusions, and opportunities for improvement. And importantly, support the findings with evidence. I'm not looking for reassurance here. I want constructive criticism. 3. Sometimes I'll use multiple reviewers. For more complex projects, I'll have several agents independently review the same work. Different models can identify different issues, and independent reviews help reduce the likelihood that something important gets overlooked. But this introduces another challenge. Now I've got multiple lengthy reviews, often containing overlapping findings, conflicting recommendations, and false positives. Which brings me to the next step. 4. Introduce a judge. Rather than manually working through every finding from every reviewer, I'll sometimes introduce another AI agent whose responsibility is to evaluate and consolidate the reviews. Its job is to identify duplicate findings, highlight disagreements, prioritize concerns, and produce a single actionable report (preferably in a visual that is easy to digest). The judge is meant to be impartial. It's purpose is to organize and evaluate the findings, not rule on what's true. 5. Send the findings back to the original agent. I'll give the consolidated report back to the original Claude thread that was doing the original work. I tell it to review the report, which was done on it's work, investigate the findings, challenge anything it believes is incorrect, and address legitimate issues. Sometimes I'll repeat the process. And for consequential work, I still verify important conclusions against actual evidence, source documents, or test results. The goal isn't to eliminate my judgment. It's to make my judgment more effective and easier to validate. As AI becomes increasingly capable, I think we need to spend less time thinking exclusively about how to get AI to produce better outputs and more time thinking about how to evaluate those outputs. This is particularly relevant to us accountants. Independent review, professional skepticism, and risk-based testing are concepts we've understood for decades. We're simply applying those same principles to a new way of working.