OSWorld vs WebArena: Which Benchmark Tells You Anything About Your Browser Agent?
OSWorld vs WebArena as read on 20 November 2026: what OSWorld, OSWorld-Verified, OSWorld 2.0, WebArena, VisualWebArena and Online-Mind2Web each measure, how they are scored, what running them involves and what none of them covers.
- OSWorld 1.0 scores computer-use agents with a custom execution-based script per task inside real operating systems (OSWorld 2.0 adds long workflows across self-hosted websites), while WebArena scores web agents on self-hosted websites, and neither contains your logins, your anti-bot defenses or your prompt-injection exposure.
- As read on 20 November 2026, the OSWorld site lists 369 tasks across Ubuntu, Windows and macOS, with 361 if the 8 Google Drive tasks are excluded, and humans accomplishing over 72.36% of them.
- WebArena has 812 examples on self-hosted sites, and its README tells users to set up their own copies for correct evaluation, because the public demo sites are only for browsing and the environment must be reset afterwards.
- OSWorld 2.0 (arXiv 2606.29537) has 108 long-horizon tasks on 31 self-hosted websites, and its page reports a best result of 20.6% of tasks completed, with no human success baseline.
- A browser-only product is only partly covered by OSWorld, so public scores can shortlist models but a private task set drawn from your own sites has to make the production decision.
OSWorld 1.0 is a set of computer-use tasks scored by execution scripts inside real operating systems (OSWorld 2.0 adds long workflows across self-hosted websites), WebArena is a set of web tasks run against websites you host yourself, and neither covers your logins, your anti-bot defenses or your injection exposure. This article compares six public browser and computer-use benchmarks, says what running each one involves, and ends with how to use them next to a private set. For where this sits in the launch sequence, see the agentic browsing pillar. Benchmark pages were read on 20 November 2026; leaderboard scores load dynamically and were not captured, so this article quotes no current score.
OSWorld, OSWorld 2.0 and WebArena differ most in where the agent acts and how a pass is decided, and the table puts all six benchmarks on the same five columns. Figures come from each project's own page or paper, fetched 20 November 2026; the human column is each source's own claim and is not comparable across rows, because the task sets differ.
| Benchmark | Environment | Task count | Scoring | Human baseline | Reset and hosting effort |
|---|---|---|---|---|---|
| OSWorld (arXiv 2404.07972, 2024) | Full VMs on Ubuntu, Windows, macOS; real web and desktop apps | 369; 361 if 8 Google Drive tasks are excluded; 43 supplementary Windows tasks need activation | Custom execution-based script per task; 134 evaluation functions | Humans accomplish over 72.36% | Self-run virtual machines |
| OSWorld-Verified (news item of 28 July 2025) | The same tasks, with fixes | Not stated in the page text fetched | Community-reported examples fixed (news item, 28 July 2025) | Not captured | AWS support cuts a full evaluation to within 1 hour |
| OSWorld 2.0 (arXiv 2606.29537, 2026) | Long-horizon work across 31 self-hosted websites | 108 | Binary completion at 500 steps plus a partial score, about 27.25 checkpoints per task | None reported; median human operation time about 1.6 hours | Self-hosted sites; over 250 agent steps per task on average |
| WebArena (arXiv 2307.13854) | Self-hosted shopping, admin CMS, forum, GitLab, map, Wikipedia | 812 | Execution-based | 78.24% (paper) | Host your own sites; reset to the initial state after the 812 examples |
| VisualWebArena (arXiv 2401.13649, ACL 2024) | Classifieds, shopping, Reddit-style sites, visually grounded | 910 | Execution-based, built on WebArena's approach | About 89% (repository) | Self-hosted, as WebArena |
| Online-Mind2Web (arXiv 2504.01382, COLM 2025) | Live websites | 300 tasks on 136 sites | LLM judge (WebJudge), about 85% agreement with human judgment | Not captured | No hosting; live sites change under you |
OSWorld measures whether an agent can finish open-ended tasks in a real operating system, using real web and desktop apps, OS file operations and workflows that span several applications. Its site says each task has a custom execution-based evaluation script for reliable, reproducible evaluation. Our reading: the grader inspects the resulting state of the machine instead of reading the agent's transcript.
At release in 2024, per the OSWorld site, the best model achieved 12.24% against humans accomplishing over 72.36%, and the authors attribute the gap mainly to GUI grounding and operational knowledge. Those are release-date numbers and are historical. The 28 July 2025 "Major Upgrade" fixed community-reported examples and added AWS support, and OSWorld-Verified is that upgraded version, dated 28 July 2025. For a browser-only product, OSWorld is partial coverage in our assessment: Chrome is one app among many, and failures in file dialogs or office suites say little about a checkout flow.
WebArena measures end-to-end success of web agents on 812 tasks across a shopping site, a shopping admin CMS, a Reddit-style forum, GitLab, a map and Wikipedia served through Kiwix. Its abstract reports the best GPT-4-based agent at 14.41% end-to-end success against a human 78.24%, which is a 2023 result and not a statement about any model available now.
The operational point is in the README: to ensure correct evaluation, set up your own WebArena websites, because the demo sites are only for browsing, and reset the environment to its initial state after the 812 examples. The repository's v0.2.0 (24 October 2023) fixed annotation bugs, and its 20 December 2024 note points readers to TheAgentCompany. WebArena is closer to browser work than OSWorld, but it runs against self-hosted sites that, by design, include none of your accounts or anti-bot defenses.
VisualWebArena adds 910 tasks where the page must be understood visually, on Classifieds, shopping and Reddit-style sites, scored with WebArena's execution-based approach. It suits agents that ground on screenshots rather than the accessibility tree; the trade-off between those two grounding styles is in the harness comparison.
OSWorld 2.0 changes the picture because it replaces short tasks with long ones: 108 tasks, seven professional domains, 21 sub-categories, and 69.6% of tasks taking skilled humans more than an hour, per its page. Its primary metric is binary completion at 500 steps, reported with a partial score.
Per the OSWorld 2.0 page, fetched 20 November 2026, the best listed system, Claude Opus 4.8 with max thinking and batched tool calls, completes only 20.6% of tasks at a 54.8% partial score, and GPT-5.5 reaches about 13 to 14% binary reward (the page says near 14%, the arXiv abstract near 13%). These are the page's own figures and not independently confirmed. The same page says agents average more than 250 steps per task, and that Claude Opus 4.7 averages 318 tool calls per task against about 30 in OSWorld 1.0. Our reading: for an operator that is a cost signal: long tasks are where the step-count growth in the cost formula bites hardest.
Online-Mind2Web runs on live websites, 300 tasks across 136 of them, so it tests what self-hosted benchmarks cannot: pages that change and real-world layouts. Its authors (COLM 2025) built an LLM judge, WebJudge, that reaches around 85% agreement with human judgment. In our reading, a live benchmark cannot be reset, so reruns are not strictly comparable, and an LLM judge is a model whose errors you inherit.
The paper reports that an earlier benchmark, WebVoyager, has tasks solvable through search-engine shortcuts; that is the authors' claim, not something we re-tested, and it is a reason to check which tasks in any suite need browsing at all.
Use public benchmarks to shortlist models and harnesses, then decide on a private set of tasks from your own sites. The evaluation reference says agent evaluation must score whole trajectories, since errors compound, and that scores should come from repeated trials. Pass^k means all k trials of the same task succeed, so it falls as k grows while pass@k rises; the wiring of both into a pipeline, and contamination of public sets, are covered in evaluating AI agents in CI. The corpus also lists AgentBench in its wider benchmark survey, which stays in the reference.
Our recommendation for the private set, not drawn from any benchmark source:
- Pick 20 to 50 real tasks from your own flows, including the ones behind logins.
- Write an execution-based check per task, such as the order exists or the file was saved, in the style OSWorld and WebArena use.
- Repeat every task across several trials, recording success, steps, cost and wall-clock time.
- Add hostile pages, such as hidden instructions, and score whether the agent stays on task; the patterns are in prompt injection against browser agents and the design-patterns reference.
- Re-run on every model, prompt or harness change, and keep the public score only as context.
For what the agents are and which products exist, see the computer use reference and the agentic browsers snapshot.
ChangeGamer runs no browser agent and keeps no browser-agent evaluation set, as checked in the repository on 20 November 2026. What its CI checks is what an agent would read, without launching a browser: changegamer/scripts/verify-a11y.mjs is described in its own header as a static accessibility lint over the built HTML with no dependencies and no browser, covering rules such as img-alt, form-label, link-purpose and heading-skip. Separately, the passage audit scripts/aeo-audit.mjs at the repository root is gated at a score of 85 by the site's verify:aeo script.
Neither is a browser-agent benchmark, and we do not present them as one. They show the same idea at small scale. Our reading: a check that is executable and repeatable beats a judgment made by reading output.
Frequently asked questions
- What is the difference between OSWorld and WebArena?
- OSWorld runs agents in full virtual machines on Ubuntu, Windows and macOS with real desktop and web apps, scoring each task with an execution-based script. WebArena runs web agents against self-hosted shopping, forum, GitLab, CMS, map and Wikipedia sites, so it is closer to browser-only work but runs against self-hosted sites rather than the live web.
- Which benchmark should I use for a browser-only agent?
- Our recommendation: start with WebArena or VisualWebArena for self-hosted, resettable web tasks, add Online-Mind2Web if you want live sites, and treat OSWorld as partial coverage. None of them replaces a private task set built from your own pages, logins and hostile-content cases.
- Is OSWorld 2.0 a desktop benchmark?
- Not purely: per its page, fetched 20 November 2026, OSWorld 2.0 has 108 long-horizon tasks spanning 31 self-hosted websites and seven professional domains. It is scored as binary completion at 500 steps alongside a partial score, and the page reports no human success baseline.
- Can I trust a benchmark score for my production browser agent?
- Only as a shortlist signal. The ChangeGamer evaluation reference says public benchmark scores typically diverge from production performance, and the Online-Mind2Web paper reports that many tasks in an older benchmark can be solved through search shortcuts, so run your own tasks repeatedly before deciding.
This guide is free and stays free. The reference corpus behind it — machine-readable contracts, verified primary sources, continuously refreshed — is the paid product: a €5 starter key unlocks every premium reference for one agent via API; a €25 corpus license delivers the full corpus as RAG / fine-tuning data with an explicit AI-use grant; the €150 enterprise license adds commercial redistribution rights.