Web data for AI agents, RAG and ChatGPT or Claude
Language models are only as current as the data you give them. These tools turn web pages, search results, YouTube videos, GitHub repositories and research papers into clean, LLM-ready text.
Every one of them can be called directly from Claude, ChatGPT, Cursor or your own agent through the Apify MCP server. No glue code needed.
Who uses it
- Developers building RAG pipelines
- AI agent and automation builders
- Researchers feeding papers and videos to LLMs
What data you get
- Clean Markdown or text per page
- Search results with page content
- Video transcripts with timestamps
- Repo README, file tree and code
- Token counts for budgeting
Ask your AI assistant
Connect the scraper to Claude, ChatGPT or Cursor through the Apify MCP server, then ask:
“Search the web for the latest EU AI Act guidance and summarize the three most relevant pages.”
https://mcp.apify.com/?tools=get_anything/llm-web-scraper
Scrapers for this
LLM Web Scraper - Website to Markdown for RAG & ChatGPT
Turn any URL or website into clean, LLM-ready Markdown or text - boilerplate, ads and nav stripped - with metadata and token count. Feed ChatGPT/Claude context or build RAG pipelines. HTTP-fast with a hardened-browser fallback for JS-heavy pages. Export JSON, CSV or Markdown.
AI Web Search Scraper - SERP & Page Content for LLM RAG
Web search API for LLMs: query -> clean results (title, URL, snippet, position, domain) and OPTIONALLY each page's full content as LLM-ready Markdown. Ground ChatGPT/Claude and RAG pipelines with live web results. Google (browser-rendered) or DuckDuckGo engine. Export JSON, CSV or Markdown.
YouTube Transcript Scraper - LLM-Ready Video Text
Get clean, LLM-ready transcripts and metadata from any YouTube video, playlist, channel or search query. Feed videos straight into ChatGPT / Claude for summaries, research and RAG. Timestamps, chapters, plain text and Markdown included.
GitHub Scraper - Repo Code, Docs & Metadata to LLM Text
Chat with a repo: turn any GitHub repository into LLM-ready text - README, file tree and source files as clean text with token counts, plus repo metadata and optional issues. Feed a codebase to ChatGPT/Claude or a RAG pipeline. Official GitHub API, no token required. Export JSON, CSV or Markdown.
arXiv Scraper - Research Papers & Full Text for LLM/RAG
Search arXiv or fetch paper IDs and get clean records - title, authors, abstract, categories, dates, DOI, PDF & abstract URLs - plus OPTIONAL full paper text as LLM-ready Markdown. Feed research into ChatGPT/Claude or a RAG pipeline. Official arXiv API, no key. Export JSON, CSV or Markdown.
Web Unblocker - Anti-Bot Scraper (Cloudflare, DataDome)
Fetch any URL even behind anti-bot walls (Cloudflare, DataDome, PerimeterX, Akamai, Imperva). Returns HTML plus optional Markdown/text and a screenshot, detects which protection guards the site, and retries via a hardened browser on a fresh residential IP when blocked. No API key.
FAQ
How do I connect these to Claude or ChatGPT?
Add the Apify MCP server URL as a custom connector and sign in with your Apify account. Each scraper page here shows its exact URL.
What about sites with bot protection?
Web Unblocker fetches pages behind Cloudflare and DataDome checks and returns the HTML or Markdown.
Need a different source or field?
I build custom scrapers and data feeds, with a focus on Gulf and Middle East sites.