If you're building an AI agent that needs information from the web, choosing a retrieval provider can look deceptively simple.
Pick an API. Send it a URL. Get the page.
But real websites are not equally easy to retrieve.
A SaaS pricing page behaves differently from product pages, documentation, JavaScript-heavy sites, and pages protected by anti-bot systems.
So I wanted to test a more practical question:
When does each web retrieval method actually win?
For CAutomates Agent Utility Index Benchmark #001B, I tested five retrieval approaches against the same 50 real-world pages:
The benchmark covered five different workloads, with 10 pages in each:
Across those pages, I defined 141 expected fields that the retrieval methods needed to surface.
The result was not simply "Provider A is best."
Firecrawl won overall. But the workload changed the picture.
| Retrieval method | Pages retrieved | Expected fields found | Field coverage | Median latency |
|---|---|---|---|---|
| Firecrawl | 50/50 | 106/141 | 75.2% | 6,506 ms |
| Bright Data | 47/50 | 94/141 | 66.7% | 5,211 ms |
| Tavily | 35/50 | 69/141 | 48.9% | 310 ms |
| ScrapingBee | 32/50 | 67/141 | 47.5% | 3,835.5 ms |
| Direct Fetch | 34/50 | 44/141 | 31.2% | 652.5 ms |
At first glance, Firecrawl is the obvious winner.
It retrieved all 50 pages and found 106 of the 141 expected fields.
But that overall ranking hides some of the most useful findings.
Firecrawl was the only method in this benchmark to successfully retrieve all 50 pages under the final locked execution methodology.
It also achieved the highest overall field coverage:
106/141 fields, or 75.2%.
Its strongest category was Ecommerce/Product, where it retrieved all 10 pages and found 28/30 expected fields.
It also led JavaScript-heavy pages on overall retrieval and field coverage, and it produced the strongest overall result on difficult/anti-bot pages.
If I had to select one retrieval provider based only on this benchmark and had no information about the incoming workload, Firecrawl produced the strongest general-purpose result.
But that does not mean it was best at everything.
Tavily's overall numbers were lower:
35/50 pages retrieved and 69/141 fields found.
But Documentation told a different story.
Tavily led that category on field coverage.
It was also by far the fastest method in the benchmark based on median measured latency:
310 ms median
compared with:
This is an important distinction.
A provider that loses an overall benchmark may still be a very attractive option for a specific workload.
If an AI system primarily needs technical documentation, the overall leaderboard may not be the right way to choose its retrieval method.
Bright Data finished second overall:
47/50 retrieval and 94/141 fields.
It did not beat Firecrawl on total field coverage.
But on successfully retrieved difficult/anti-bot pages, Bright Data had the strongest conditional field coverage.
Conditional field coverage asks a different question:
When the provider actually retrieves the page, how much of the expected information does it find?
That distinction matters.
A provider can be excellent at extracting information once it gains access to a page while still experiencing more retrieval failures than another provider.
Bright Data's results demonstrate why I don't think retrieval success and extraction quality should be treated as the same metric.
ScrapingBee produced one of the more surprising results.
Overall, it retrieved only:
32/50 pages
and found:
67/141 expected fields.
But among pages it successfully retrieved, its conditional field coverage was 77.0%, the highest overall conditional coverage of the five methods.
Its SaaS pricing performance was particularly strong:
18/21 expected fields
and its Documentation result reached:
23/30 fields.
The problem was reliability across harder workloads.
ScrapingBee retrieved 0/10 difficult/anti-bot pages in this benchmark.
So simply looking at its 47.5% overall field coverage misses part of the story.
When retrieval worked, extraction was often strong.
When retrieval failed, there was nothing to extract.
Direct Fetch was intentionally the simple baseline.
No specialized scraping provider. No browser-rendering rescue path. No fallback provider.
It retrieved:
34/50 pages
and found:
44/141 expected fields.
That is only 31.2% overall field coverage.
But something interesting happened when I analyzed the individual workloads.
There were 10 benchmark rows where the simple-first criteria indicated that Direct Fetch could potentially be sufficient before escalating to a paid retrieval provider.
Those appeared primarily in:
This does not prove monetary savings. Infrastructure and network costs for Direct Fetch were not measured.
But it does support an interesting architecture question:
Does every URL need an expensive retrieval path first?
Maybe not.
For some workloads, a system could attempt simple retrieval first and escalate when the result is insufficient.
That is something I want to investigate further.
The difficult/anti-bot category produced one of the clearest differences between methods.
Direct Fetch managed only:
1/10 retrievals and 0/30 expected fields.
ScrapingBee retrieved:
0/10.
Firecrawl, under its final deterministic pacing methodology, retrieved:
10/10.
Bright Data also performed strongly on difficult pages, particularly when looking at conditional extraction quality.
This category is a useful reminder that "fetching a URL" and "reliably accessing the modern web" are very different problems.
Another important lesson came from Tavily.
Its full run returned HTTP 200 responses, but 15 rows were still marked as retrieval failures because the responses did not contain usable benchmark content.
This is why the Agent Utility Index does not treat HTTP success alone as retrieval success.
For an AI agent, this:
200 OK + no useful information
can still be a failed action.
The benchmark evaluates whether the retrieval produced information the agent could actually use.
Across the frozen benchmark executions, observed failure patterns included:
Different retrieval methods failed in different ways.
Direct Fetch encountered many 403 responses.
ScrapingBee experienced substantial 500-level failures.
Tavily sometimes returned successful HTTP responses without usable content.
Bright Data experienced several MCP timeouts.
Those differences matter when designing an AI system because reliability is not simply a single success percentage.
The type of failure determines what a useful fallback strategy might look like.
I also wanted this benchmark to eventually measure cost efficiency.
But the current data does not support a fair dollar comparison.
Tavily reported 15 usage credits during its full run.
ScrapingBee reported 160 credits.
Firecrawl and Bright Data did not expose comparable monetary usage data through these benchmark executions.
Direct Fetch recorded an external retrieval-provider cost of zero, but its infrastructure, networking, and compute costs were not measured.
Those units are not directly comparable.
So I am not converting credits into dollars or publishing a "cost per correct field" ranking yet.
Doing that would create precision the experiment did not actually measure.
For Benchmark #001B:
Firecrawl won overall.
It delivered the highest retrieval reliability and the highest overall field coverage.
But the more useful conclusion is not simply "use Firecrawl."
The benchmark showed several different strengths:
Firecrawl
Strongest general-purpose performance and the best overall field coverage.
Bright Data
Strong overall performance with particularly interesting conditional performance on difficult pages.
Tavily
Extremely fast in this benchmark and strongest on Documentation field coverage.
ScrapingBee
Strong extraction quality when retrieval succeeded, but significantly weaker retrieval reliability on harder workloads.
Direct Fetch
Weakest overall, but potentially sufficient for some simpler pages without immediately invoking a specialized retrieval provider.
This is the finding I find most interesting.
Many AI systems effectively make a static infrastructure decision:
Use Provider X for web retrieval.
But these results suggest another architecture may be worth investigating:
Understand the workload -> try the appropriate retrieval method -> validate the result -> escalate when necessary.
A documentation page may not need the same retrieval strategy as a JavaScript-heavy product configurator.
A straightforward SaaS pricing page may not need the same infrastructure as a heavily protected marketplace page.
And some pages may not require a paid retrieval provider at all.
Benchmark #001B does not prove that an automatic retrieval router will outperform a single provider.
But it provides enough evidence to justify testing the hypothesis.
That is where the CAutomates Agent Utility Index goes next.
Benchmark #001B used:
50 human-reviewed URLs
divided equally across five workload categories:
The locked corpus contained 141 human-reviewed expected fields.
Each retrieval method was evaluated against the same approved corpus.
Provider execution artifacts were kept separate from the locked dataset.
Results were analyzed across:
A total of 250 provider-page observations were included in the final cross-provider analysis.
Provider-specific execution controls were introduced only when necessary for reproducible execution. For example, Firecrawl's final run used a deterministic rate-limit policy after an earlier execution encountered HTTP 429 responses.
That earlier rate-limit-affected execution was preserved rather than silently replaced.
This is Benchmark #001B, not a universal ranking of retrieval providers.
The dataset contains 50 pages, not the entire web.
Provider behavior, websites, APIs, anti-bot systems, pricing, and infrastructure can change.
The benchmark also measures defined expected-field presence rather than every dimension of retrieval quality.
Latency measurements reflect the benchmark execution environment and provider methodology used during these runs.
And because comparable monetary usage information was not available for every provider, this benchmark does not establish a cost winner.
The results should therefore be interpreted as evidence about these workloads under this methodology, not as a claim that one provider will always outperform another.
Benchmark #001B started with a simple question:
When does each web retrieval provider win?
The answer appears to be more nuanced than a single leaderboard.
The next research question is more interesting:
Can an AI system identify the workload first and choose a better retrieval strategy automatically?
That could mean trying simple retrieval on straightforward pages, using specialized retrieval when necessary, and escalating when the first result fails validation.
I am not building that conclusion into a router yet.
First, I want more evidence.
That's the purpose of the CAutomates Agent Utility Index: testing the infrastructure behind AI agents on real workloads and measuring what actually works.
For the earlier cost-focused retrieval benchmark, see Benchmark #001: I Tested AI Web Retrieval Tools on Real Pages. Here Is What a Correct Result Actually Costs.
Chrissa
Chrissa Ibiernas is a Marketing Automation, Lead Generation & AI Workflow Specialist who documents practical AI, automation, agent-readiness, and search experiments through Chrissa Automates. Contact: hello@chrissaautomates.com
Looking for creator-led content for a technology product?
See how I approach product demonstrations, tutorials, and ongoing content for AI, SaaS, automation, and technology brands.
