A Fast Website Reader MCP Server

@just-every

リポジトリ

基本情報

カテゴリ

ブラウザ自動化

トランスポート

stdio

作者

just-every

投稿者

James Peter

A Fast Website Reader MCP Server について

A Markdown Content Preprocessor that fetches web pages, strips noise, and converts content to clean Markdown while preserving links. Designed for with minimal token footprint so entire pages can be read at once. Crawl and scrape webpage and whole sites locally with minimal depend

#webpage #crawl #scrape #markdown #fast

A Fast Website Reader MCP Server の設定

以下の設定を使って、このサーバーを MCP 対応クライアントに追加してください。

{
  "mcpServers": {
    "read-website-fast": {
      "command": "npx",
      "args": [
        "-y",
        "github:just-every/mcp-read-website-fast",
        "serve"
      ]
    }
  }
}

A Fast Website Reader MCP Server のツール

Scrape content from a single URL with advanced options. **Best for:** Single page content extraction, when you know exactly which page contains the information. **Not recommended for:** Multiple pages (use batch_scrape), unknown page (use search), structured data (use extract). **Common mistakes:** Using scrape for a list of URLs (use batch_scrape instead). **Prompt Example:** "Get the content of the page at https://example.com." **Usage Example:** ```json { "name": "firecrawl_scrape", "arguments": { "url": "https://example.com", "formats": ["markdown"] } } ``` **Returns:** Markdown, HTML, or other formats as specified.

Map a website to discover all indexed URLs on the site. **Best for:** Discovering URLs on a website before deciding what to scrape; finding specific sections of a website. **Not recommended for:** When you already know which specific URL you need (use scrape or batch_scrape); when you need the content of the pages (use scrape after mapping). **Common mistakes:** Using crawl to discover URLs instead of map. **Prompt Example:** "List all URLs on example.com." **Usage Example:** ```json { "name": "firecrawl_map", "arguments": { "url": "https://example.com" } } ``` **Returns:** Array of URLs found on the site.

Starts an asynchronous crawl job on a website and extracts content from all pages. **Best for:** Extracting content from multiple related pages, when you need comprehensive coverage. **Not recommended for:** Extracting content from a single page (use scrape); when token limits are a concern (use map + batch_scrape); when you need fast results (crawling can be slow). **Warning:** Crawl responses can be very large and may exceed token limits. Limit the crawl depth and number of pages, or use map + batch_scrape for better control. **Common mistakes:** Setting limit or maxDepth too high (causes token overflow); using crawl for a single page (use scrape instead). **Prompt Example:** "Get all blog posts from the first two levels of example.com/blog." **Usage Example:** ```json { "name": "firecrawl_crawl", "arguments": { "url": "https://example.com/blog/*", "maxDepth": 2, "limit": 100, "allowExternalLinks": false, "deduplicateSimilarURLs": true } } ``` **Returns:** Operation ID for status checking; use firecrawl_check_crawl_status to check progress.

Check the status of a crawl job. **Usage Example:** ```json { "name": "firecrawl_check_crawl_status", "arguments": { "id": "550e8400-e29b-41d4-a716-446655440000" } } ``` **Returns:** Status and progress of the crawl job, including results if available.

Search the web and optionally extract content from search results. **Best for:** Finding specific information across multiple websites, when you don't know which website has the information; when you need the most relevant content for a query. **Not recommended for:** When you already know which website to scrape (use scrape); when you need comprehensive coverage of a single website (use map or crawl). **Common mistakes:** Using crawl or map for open-ended questions (use search instead). **Prompt Example:** "Find the latest research papers on AI published in 2023." **Usage Example:** ```json { "name": "firecrawl_search", "arguments": { "query": "latest AI research papers 2023", "limit": 5, "lang": "en", "country": "us", "scrapeOptions": { "formats": ["markdown"], "onlyMainContent": true } } } ``` **Returns:** Array of search results (with optional scraped content).

Extract structured information from web pages using LLM capabilities. Supports both cloud AI and self-hosted LLM extraction. **Best for:** Extracting specific structured data like prices, names, details. **Not recommended for:** When you need the full content of a page (use scrape); when you're not looking for specific structured data. **Arguments:** - urls: Array of URLs to extract information from - prompt: Custom prompt for the LLM extraction - systemPrompt: System prompt to guide the LLM - schema: JSON schema for structured data extraction - allowExternalLinks: Allow extraction from external links - enableWebSearch: Enable web search for additional context - includeSubdomains: Include subdomains in extraction **Prompt Example:** "Extract the product name, price, and description from these product pages." **Usage Example:** ```json { "name": "firecrawl_extract", "arguments": { "urls": ["https://example.com/page1", "https://example.com/page2"], "prompt": "Extract product information including name, price, and description", "systemPrompt": "You are a helpful assistant that extracts product information", "schema": { "type": "object", "properties": { "name": { "type": "string" }, "price": { "type": "number" }, "description": { "type": "string" } }, "required": ["name", "price"] }, "allowExternalLinks": false, "enableWebSearch": false, "includeSubdomains": false } } ``` **Returns:** Extracted structured data as defined by your schema.

Conduct deep web research on a query using intelligent crawling, search, and LLM analysis. **Best for:** Complex research questions requiring multiple sources, in-depth analysis. **Not recommended for:** Simple questions that can be answered with a single search; when you need very specific information from a known page (use scrape); when you need results quickly (deep research can take time). **Arguments:** - query (string, required): The research question or topic to explore. - maxDepth (number, optional): Maximum recursive depth for crawling/search (default: 3). - timeLimit (number, optional): Time limit in seconds for the research session (default: 120). - maxUrls (number, optional): Maximum number of URLs to analyze (default: 50). **Prompt Example:** "Research the environmental impact of electric vehicles versus gasoline vehicles." **Usage Example:** ```json { "name": "firecrawl_deep_research", "arguments": { "query": "What are the environmental impacts of electric vehicles compared to gasoline vehicles?", "maxDepth": 3, "timeLimit": 120, "maxUrls": 50 } } ``` **Returns:** Final analysis generated by an LLM based on research. (data.finalAnalysis); may also include structured activities and sources used in the research process.

Generate a standardized llms.txt (and optionally llms-full.txt) file for a given domain. This file defines how large language models should interact with the site. **Best for:** Creating machine-readable permission guidelines for AI models. **Not recommended for:** General content extraction or research. **Arguments:** - url (string, required): The base URL of the website to analyze. - maxUrls (number, optional): Max number of URLs to include (default: 10). - showFullText (boolean, optional): Whether to include llms-full.txt contents in the response. **Prompt Example:** "Generate an LLMs.txt file for example.com." **Usage Example:** ```json { "name": "firecrawl_generate_llmstxt", "arguments": { "url": "https://example.com", "maxUrls": 20, "showFullText": true } } ``` **Returns:** LLMs.txt file contents (and optionally llms-full.txt).

概要

What is A Fast Website Reader MCP?

A Fast Website Reader MCP is an MCP server that fetches web pages, strips noise, and converts content to clean Markdown while preserving links. It is designed for LLM pipelines with minimal token footprint and can crawl sites locally with minimal dependencies.

How to use A Fast Website Reader MCP?

Install and run the server using npx -y github:just-every/mcp-read-website-fast serve. Configure it in MCP clients (Claude Desktop, Cursor, VS Code, JetBrains IDEs) by adding a JSON entry with the command npx and args ["-y","github:just-every/mcp-read-website-fast","serve"]. The tool read_website_fast fetches a URL and returns clean markdown; optional parameters include: depth (crawl depth, default 0), respectRobots (default true). Two resources are available: read-website-fast://status (cache statistics) and read-website-fast://clear-cache (clear cache).

Key features of A Fast Website Reader MCP

Content extraction using Mozilla Readability (same engine as Firefox Reader View)
HTML to Markdown conversion with Turndown + GFM support
Smart caching with SHA-256 hashed URLs
Polite crawling with robots.txt support and rate limiting
Concurrent fetching with configurable depth crawling
Stream-first design for low memory usage
Link preservation for knowledge graphs
Optional chunking for downstream processing

Use cases of A Fast Website Reader MCP

Fetch a single webpage and get clean Markdown for LLM input
Crawl a site up to a configurable depth for knowledge graph construction
Extract and preserve links from pages for graph-based analysis
Clear cache or inspect cache statistics via MCP resources

よくある質問

What runtime or dependencies are required?

The server requires Node.js and is installed via npm. Run `npm install` then `npm run build` to build from source.

Where does the cache live?

Cache is stored in a local directory (default `.cache`) and can be configured via the `--cache-dir` CLI option.

Does the server respect robots.txt?

Yes, by default it respects robots.txt. Use the `--no-robots` flag or set `respectRobots` to `false` to ignore it.

What are the configurable crawling limits?

You can set crawl depth (`--depth`, default 0 for single page), max concurrent requests (`--concurrency`, default 3), and request timeout (`--timeout`, default 30000 ms). These are available as both CLI and tool parameters.

Does the server require authentication or API keys?

No authentication or API keys are mentioned. The server runs locally and fetches public web pages.

「ブラウザ自動化」の他のコンテンツ

Apify Model Context Protocol (MCP) Server

apify

The Apify MCP server enables your AI agents to extract data from social media, search engines, maps, e-commerce sites, or any other website using thousands of ready-made scrapers, crawlers, and automation tools available on the Apify Store.

1.7K