AI Web Search Scraper — SERP & Page Content for LLMs & RAG

Web search for LLMs. Give it a query (or several) and get back clean, ranked results — title, URL, snippet, position, domain — and optionally the full content of each result page as LLM-ready Markdown. It's the search half of a RAG web browser: pair search with readable pages to ground ChatGPT / Claude or feed a RAG pipeline. No API key.

What it does

• Query → ranked results with title, URL, snippet, domain and position.
• Optional page content — fetches each result and extracts clean Markdown (boilerplate stripped) with a token count.
• Engines: DuckDuckGo (fast, reliable, browser-rendered) or Google (Camoufox-rendered; use a residential proxy).
• Multiple queries per run; choose language and country.
• Export to JSON, CSV, or Markdown, or pull via the Apify API.

Input

Field	Description
queries	One or more search queries.
engine	duckduckgo (default) or google.
maxResults	Results per query.
language / country	hl / gl codes.
fetchPageContent	Also fetch each result page as Markdown.
maxCharsPerPage	Truncate page content to fit token budgets.
proxyConfiguration	Proxy (residential recommended for Google).

Example

{ "queries": ["retrieval augmented generation"], "engine": "duckduckgo",
  "maxResults": 10, "fetchPageContent": true }

Output

{ "query": "retrieval augmented generation", "position": 1,
  "title": "Retrieval-augmented generation - Wikipedia",
  "url": "https://en.wikipedia.org/wiki/...", "domain": "en.wikipedia.org",
  "contentSuccess": true, "markdown": "# Retrieval-augmented generation ...", "tokensApprox": 1800 }

Use cases

• Ground an LLM / agent with live web results and readable page text.
• RAG ingestion — search a topic, fetch top pages to Markdown, chunk into a vector DB.
• Research & monitoring — bulk queries → structured SERP for analysis.

FAQ

Do I need an API key? No. DuckDuckGo works out of the box; for Google, keep the residential proxy on.

Use responsibly and in line with each search engine's terms.

🤖 Use with Claude or ChatGPT (MCP)

Run this actor from Claude, ChatGPT, Cursor or any MCP client via the Apify MCP server. In Claude Desktop: Settings → Connectors → Add custom connector → https://mcp.apify.com. Or expose just this tool:

{ "mcpServers": { "apify": { "url": "https://mcp.apify.com?tools=get_anything/ai-web-search-scraper" } } }

Full guide: Connect Apify actors to Claude & ChatGPT.

⭐ Found this useful?

If this Actor saved you time, please leave a rating on its Apify page — reviews genuinely help others discover it and help me keep improving it.