How to give ChatGPT, Claude or your AI agent live web access (search + read any page)

Short answer: an LLM can only use the live web through a tool that does the fetching for it. There are three practical options:

1. The chat app's built-in search. ChatGPT and Claude both offer it. It's fine for questions in the chat window, but you can't control the sources or reuse the results in your own code.
2. A search API. You get structured results, but usually only titles and snippets, and you still have to fetch and clean each page yourself.
3. An MCP tool or scraper that searches and reads pages. The model gets the top results plus each page's main text as clean Markdown, ready to quote and cite.

For agents and RAG pipelines, option 3 gives the model the most to work with. Below is how to set it up in a couple of minutes, without a search API key.

What "live web access" actually needs

To answer from the web, a model needs four things:

1. Search: turn the question into ranked URLs.
2. Fetch: download each page, including JavaScript-heavy or protected pages.
3. Clean: strip navigation, cookie banners and ads so only the main content is left.
4. Fit: trim to a token budget and keep the source URL for citations.

Built-in browsing hides all of this. If you build it yourself, step 3 is where most of the work goes: raw HTML is mostly noise, and noise in the context window means worse answers and higher cost.

Option 3 in practice: one tool for search + clean pages

The AI Web Search Scraper does steps 1–4 in one run. You give it a query and it returns the top results with title, URL, domain, snippet, published date (when the engine shows one) and, optionally, each page as Markdown with a token estimate.

A real result for the query best open source vector database 2026 (trimmed):

{
  "position": 1,
  "title": "9 Best Open Source Vector Databases in 2026 (Tested) - Geekflare",
  "url": "https://geekflare.com/open-source/vector-database/",
  "date": "Oct 19, 2026",
  "snippet": "I looked at 31 open source vector databases, Postgres extensions and search engines with vector search...",
  "markdown": "| Qdrant | Yes — sparse and dense vectors fused in one query | ...",
  "tokensApprox": 200
}

Connect it to Claude, ChatGPT or Cursor (no code)

Add the Apify MCP server with just this tool:

{ "mcpServers": { "apify": { "url": "https://mcp.apify.com/?tools=get_anything/ai-web-search-scraper" } } }

In Claude Desktop: Settings → Connectors → Add custom connector, then paste the URL. Now you can ask: "Search the web for this week's news about EU AI Act enforcement and summarise the top 5 sources with links."

Call it from your own agent (Python)

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("get_anything/ai-web-search-scraper").call(run_input={
    "queries": ["best open source vector database 2026"],
    "maxResults": 5,
    "fetchPageContent": True,
    "maxCharsPerPage": 3000,
})
results = list(client.dataset(run["defaultDatasetId"]).iterate_items())
context = "\n\n".join(f"[{r['position']}] {r['title']} ({r['url']})\n{r.get('markdown', '')}" for r in results)

Then pass context to the model with an instruction to answer only from the sources and cite them as [1], [2].

When you already know the URLs

If the agent needs to read specific pages (a docs site, a list of articles, a competitor's pricing page), skip search and use the LLM Web Scraper. It converts any URL to clean Markdown, can crawl a whole site, and reports tokensApprox per page so you can budget the context window.

For sites that block normal fetchers, the Web Unblocker renders the page in a hardened browser and reports which anti-bot system protected it.

Which option should you use?

You want to…	Use
Ask a quick question in a chat window	The app's built-in search
Give an agent search results with full page text and citations	AI Web Search Scraper (MCP or API)
Load specific pages or a whole docs site into RAG	LLM Web Scraper
Read a page that blocks normal requests	Web Unblocker

FAQ

Can ChatGPT or Claude browse the web on their own?

In their chat apps, yes, through built-in search. Through the API or in your own agent, the model only sees what a tool fetches for it. That's what MCP tools and scrapers provide.

Do I need a Google or Bing search API key?

Not with the AI Web Search Scraper. It uses DuckDuckGo by default, with Google available through a residential proxy. You only need a free Apify account.

How do I keep the context window small?

Set maxCharsPerPage to cap each page, and use tokensApprox on every result to budget. Five pages at 3,000 characters is about 4,000 tokens.

What does it cost?

You pay per result, with no subscription. Apify's free plan includes monthly credit, which covers a lot of testing.