GitHub Scraper — Repo Code, Docs & Metadata to LLM Text

Chat with a repo. Turn any GitHub repository into clean, LLM-ready text: the README, the file tree, and the contents of selected source/doc files — each with a token count — plus repo metadata and (optionally) recent issues. Built for feeding a codebase into ChatGPT / Claude or a RAG pipeline.

Why this one

AI coding assistants and RAG-over-code are one of the biggest LLM use cases. This Actor gives you the whole repo as structured text in one run — no cloning, no git, no parsing HTML. It uses GitHub's official REST API plus raw.githubusercontent.com, so it's fast and reliable. Works without a token (60 req/h); add one for 5,000 req/h.

What it does

• Scrape multiple repos by owner/repo or URL.
• README + file tree + file contents (filtered by glob patterns).
• Sensible include/exclude defaults (code + docs in; node_modules, lockfiles, binaries out).
• Caps for max files, file size, and chars per file to control token cost.
• Optional issues as text.
• Export to JSON, CSV, or Markdown, or pull via the Apify API.

Input

Field	Description
repos	Repositories as owner/repo or GitHub URLs.
includeReadme / includeFileTree / includeFileContents	What to return.
includePatterns / excludePatterns	Glob filters (defaults provided).
maxFiles / maxFileSizeKB / maxCharsPerFile	Token-budget caps.
includeIssues / maxIssues	Optionally pull issues.
githubToken	Optional PAT for a higher rate limit.

Example

{
  "repos": ["apify/apify-sdk-python"],
  "includeReadme": true,
  "includeFileContents": true,
  "includePatterns": ["*.py", "*.md"],
  "maxFiles": 50,
  "maxFileSizeKB": 100
}

Output

One repo record (metadata + README + tree), then one file record per selected file:

{ "resultType": "file", "repo": "apify/apify-sdk-python", "path": "src/apify/actor.py",
  "content": "from __future__ ...", "tokensApprox": 1820,
  "url": "https://github.com/apify/apify-sdk-python/blob/master/src/apify/actor.py" }

Use cases

• RAG over a codebase — chunk file.content into a vector DB, ask questions about the repo.
• LLM code review / onboarding — paste README + key files into ChatGPT/Claude.
• Docs generation — summarise a repo from its README and source.
• Dependency / license audits — pull metadata across many repos.

FAQ

Do I need a token? No — but a token raises the limit to 5,000 req/h for large runs.

How are binary files handled? Skipped (size cap + a binary guard); only text is returned.

🤖 Use with Claude or ChatGPT (MCP)

Run this actor from Claude, ChatGPT, Cursor or any MCP client via the Apify MCP server. In Claude Desktop: Settings → Connectors → Add custom connector → https://mcp.apify.com. Or expose just this tool:

{ "mcpServers": { "apify": { "url": "https://mcp.apify.com?tools=get_anything/github-repo-scraper" } } }

Full guide: Connect Apify actors to Claude & ChatGPT.

⭐ Found this useful?

If this Actor saved you time, please leave a rating on its Apify page — reviews genuinely help others discover it and help me keep improving it.