SERP Scraper — Google, Bing & DuckDuckGo for LLM / RAG

Scrape search engine results by keyword and get clean, structured data — no API key, any country and language. Built as the reliable "search" half of a RAG web browser: feed live search results into ChatGPT / Claude, do rank tracking, or mine keyword ideas.

Engines

Engine	Reliability	Notes
Bing (default)	✅ High	High-quality organic results, rendered in a hardened browser. Recommended for production.
DuckDuckGo	✅ High	Privacy-friendly results, browser-rendered.
Google	⚠️ Best-effort	Google aggressively CAPTCHA-walls cloud/datacenter IPs. This engine renders the basic (gbv=1) page and retries on a fresh residential IP, but success is not guaranteed — use Bing for reliable results.

Why not Google-only? Google serves a /sorry CAPTCHA to virtually all cloud IPs. No scraper can reliably beat that without a paid CAPTCHA-solving service, so this actor defaults to Bing, which returns comparable results on every run.

What you get

One item per organic result:

Field	Description
query	The search query
engine	Engine used (bing / duckduckgo / google)
position	Rank on the SERP (1-based, across pages)
title	Result title
url	Destination URL
displayedUrl	Breadcrumb URL shown by the engine (when available)
domain	Hostname of the result
snippet	Result snippet text
relatedSearches	(first result only) related search phrases
scrapedAt	ISO 8601 timestamp

Input

{
  "queries": ["best web scraping tools", "apify alternatives"],
  "engine": "bing",
  "maxResults": 20,
  "country": "us",
  "language": "en",
  "includeRelatedSearches": true,
  "maxRetries": 3,
  "proxyConfiguration": { "useApifyProxy": true, "apifyProxyGroups": ["RESIDENTIAL"] }
}

Field	Type	Default	Description
queries	array	—	One or more search queries (supports site:, intitle:, quotes)
engine	string	bing	bing, duckduckgo, or google
maxResults	integer	10	Organic results per query (paginates to reach it)
country	string	us	Two-letter country code
language	string	en	Two-letter language code
includeRelatedSearches	boolean	true	Collect "Related searches"
maxRetries	integer	3	Retries on a fresh IP when blocked (mainly Google)
proxyConfiguration	object	Residential	Proxy — residential strongly recommended

Use cases

• LLM / RAG grounding — feed live search results into ChatGPT / Claude pipelines.
• SEO & rank tracking — monitor rankings for target keywords across countries.
• Market & competitor research — capture Related Searches for content and keyword ideas.
• Content & lead discovery — harvest ranking pages for a topic at scale.

🤖 Use with Claude or ChatGPT (MCP)

Run this actor straight from Claude, ChatGPT, Cursor or any MCP client via the Apify MCP server. In Claude Desktop: Settings → Connectors → Add custom connector → https://mcp.apify.com, then ask it to search the web. Or expose just this tool:

{ "mcpServers": { "apify": { "url": "https://mcp.apify.com?tools=get_anything/serp-scraper" } } }

Full guide: Connect Apify actors to Claude & ChatGPT.

Notes

• Only public search-results-page data is collected.
• Residential proxy is strongly recommended for all engines and is what makes Google's retry logic effective.