> ## Documentation Index
> Fetch the complete documentation index at: https://firecrawl-claude-eager-dijkstra-u6tgua.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Scraping GitHub

> Learn how to scrape and search GitHub using Firecrawl's core features and the Research Index

Learn how to use Firecrawl's core features to scrape GitHub repositories, issues, and documentation.

<Info>
  There are two ways to get GitHub data with Firecrawl. You can **scrape, map, or crawl** a specific `github.com` URL (what the rest of this page covers), or you can use the [Research Index](/features/research) GitHub search to semantically discover relevant issues, pull requests, discussions, and READMEs across GitHub without knowing the exact URL.
</Info>

## Setup

```bash theme={null}
npm install firecrawl zod
```

## Scrape with JSON Mode

Extract structured data from repositories using Zod schemas.

```typescript theme={null}
import { Firecrawl } from 'firecrawl';
import { z } from 'zod';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const result = await firecrawl.scrape('https://github.com/firecrawl/firecrawl', {
    formats: [{
        type: 'json',
        schema: z.object({
            name: z.string(),
            description: z.string(),
            stars: z.number(),
            forks: z.number(),
            language: z.string(),
            topics: z.array(z.string())
        })
    }]
});

console.log(result.json);
```

## Search

Find repositories, issues, or documentation on GitHub.

```typescript theme={null}
import { Firecrawl } from 'firecrawl';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const searchResult = await firecrawl.search('machine learning site:github.com', {
    limit: 10,
    sources: [{ type: 'web' }], // { type: 'news' }, { type: 'images' }
    scrapeOptions: {
        formats: ['markdown']
    }
});

console.log(searchResult);
```

## Search GitHub with the Research Index

Firecrawl's [Research Index](/features/research) includes a GitHub index that semantically searches indexed public GitHub issues, pull requests, discussions, and repository READMEs. It's useful for finding implementation details and engineering prior art when you don't know the exact repository or URL.

```typescript theme={null}
import { Firecrawl } from 'firecrawl';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const results = await firecrawl.research.searchGithub(
    'how to handle rate limit retries with exponential backoff',
    { k: 10 }
);

console.log(results);
```

You can also reach the same GitHub index via the general [`/search`](/features/search) endpoint with the `github` category.

```typescript theme={null}
import { Firecrawl } from 'firecrawl';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const searchResult = await firecrawl.search('exponential backoff retry', {
    categories: ['github'],
    limit: 10
});

console.log(searchResult);
```

See the [Research Index guide](/features/research) for Python, cURL, CLI, and MCP usage.

## Scrape

Scrape a single GitHub page - repository, issue, or file.

```typescript theme={null}
import { Firecrawl } from 'firecrawl';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const result = await firecrawl.scrape('https://github.com/firecrawl/firecrawl', {
    formats: ['markdown'] // i.e. html, links, etc.
});

console.log(result);
```

## Map

Discover all available URLs in a repository or documentation site. Note: Map returns URLs only, without content.

```typescript theme={null}
import { Firecrawl } from 'firecrawl';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const mapResult = await firecrawl.map('https://github.com/vercel/next.js/tree/canary/docs');

console.log(mapResult.links);
// Returns array of URLs without content
```

## Crawl

Crawl multiple pages from a repository or documentation.

```typescript theme={null}
import { Firecrawl } from 'firecrawl';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

const crawlResult = await firecrawl.crawl('https://github.com/facebook/react/wiki', {
    limit: 10,
    scrapeOptions: {
        formats: ['markdown']
    }
});

console.log(crawlResult.data);
```

## Batch Scrape

Scrape multiple GitHub URLs simultaneously.

```typescript theme={null}
import { Firecrawl } from 'firecrawl';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

// Wait for completion
const job = await firecrawl.batchScrape([
    'https://github.com/vercel/next.js',
    'https://github.com/facebook/react',
    'https://github.com/microsoft/typescript'],
    {
        options: {
            formats: ['markdown']
        },
        pollInterval: 2,
        timeout: 120
    }
);


console.log(job.status, job.completed, job.total);

console.log(job);
```

## Batch Scrape with JSON Mode

Extract structured data from multiple repositories at once.

```typescript theme={null}
import { Firecrawl } from 'firecrawl';
import { z } from 'zod';

const firecrawl = new Firecrawl({ apiKey: process.env.FIRECRAWL_API_KEY });

// Wait for completion
const job = await firecrawl.batchScrape([
    'https://github.com/vercel/next.js',
    'https://github.com/facebook/react'],
    {
        options: {
            formats: [{
                type: 'json',
                schema: z.object({
                    name: z.string(),
                    description: z.string(),
                    stars: z.number(),
                    language: z.string()
                })
            }]
        },
        pollInterval: 2,
        timeout: 120
    }
);


console.log(job.status, job.completed, job.total);

console.log(job);
```