Tamer A

Amazing performance, that too with small models! That said, the reported success rate for Agent-E in this plot doesn't align with the success rate mentioned in the Agent-E paper. Here is a screenshot from the Agent-E paper: This highlights a critical challenge: reproducing benchmarking results for the agents in the real world is hard! There are so many variables at play, and I think this is a good example for lack of standardized and robust benchmarking for these tasks. Evals are important. Though people overfit to leaderboards quickly, they serve a good start. For example, lmsys was a good proxy in the starting. I wonder if there will be a lmsys like leaderboard for agents in the near future.

With the announcement of Llama-3 last week, tool usage and agents-based systems are again in the news. Agents are not a new concept, but the current generation of LLMs/VLMs has provided a unique opportunity to tackle automation tasks in a simplified way with better performance. Let me share one of the coolest "agentic" workflows currently SOTA in web automation, presented in the paper that also came out last week titled: Agent-E: Web Navigation to Foundational Design Principles. Here is a summary... 1. The core of web automation A typical web agent needs two foundational capabilities: - Sensing: Sensing the state of the web page, either through DOM or through a screenshot of the web page - Acting: Capability to complete simple tasks like navigating to a URL, selecting/clicking an element on the web page, or performing composite actions with these elements 2. Challenges in Web automation Developing a simple web agent doesn't take much, but building a robust web agent comes with a plethora of challenges. Here are a few examples to give you a bigger picture: - DOM-based interactions can be noisy and expansive, and in most complex scenarios, the context length of current-gen LLMs is not enough (with a few exceptions) to incorporate all the HTML tokens. - The above point forces the developers to simplify the DOM using some sort of denoising step. - Websites are optimized for the human-computer interface (HCI), where visual interaction is of utmost importance. This design simplifies the interaction for humans, but it is not necessarily an optimal design for the interaction of agents in a system. For example, picking a date from a date picker is easy for humans, but for an agent, it involves complex interaction between visual and DOM elements. -Humans can plan or reason on the fly, but the current generation of LLMs is not good enough for both these tasks and requires significantly more tuning to make them work. 3. The nuances of evals for Web automation One of the biggest mistakes people make while evaluating their web agents is the selection of evals. One typical example is reporting the task's success rate alone. Task success rates are necessary but insufficient to evaluate the agent workflow. Why? Because other metrics like task completion time are equally important as the task success rate. A better turnaround time is all we strive for. At the bare minimum, we need evaluation on three aspects: - Task Completion time - Task Completion Cost - Error-awareness 4. Agent-E - a novel hierarchical architecture for web agents - comprises of a planner agent and a browser navigating agent - exploits DOM distillation - built on top of Autogen and uses Playwright for browser control - Given a task, the panner agent lays out the fine-grained steps required for execution, delegated to the browser-navigating agent. - leverages skill harvesting - Packed with sensing skills and action skills 5. Evaluation Setup and Evaluation Measures - follows the Web Voyager benchmark that consists of web navigation tasks across 15 real websites. Each website has about 40-46 tasks resulting in a benchmark dataset of 643 tasks - benchmarking is divided among 5 human evaluators who ran 125-130 tasks each. Each evaluation is tagged as pass or fail with a description in case of a failure - Agent-E used in full autonomous mode (i.e. no human in the loop) - GPT-4-Turbo as the LLM for both planner and browser navigation agent. - The first measure is the task success rate. - The second measure is Self-aware vs Oblivious failure rates: Self-aware failures are failures where an agent is aware of their failure in completing the task and responds with an appropriate explicit failure message. Oblivious failures OTOH are cases where the agent wrongly answers the question or performs the wrong action. - The third measure rate is task completion time - The fourth and the last measure is the number of LLM calls which also contributes a fraction to the overall cost of the system. 6. Results Without even touching multimodality, Agent-E sets a new SOTA on most of the tasks. Take a look at the results below