🤖 Perfect for AI Coding Assistants!
Want to supercharge your AI coding experience? mdCrawler helps you provide the perfect context to your AI coding agents:
- Run mdCrawler to download documentation for the technologies you're using
- Place the generated Markdown docs in your project's
/docsfolder - Now tools like Cursor, Augment, Windsurf, Traycer, Codebuff and others can access the full context of your dependencies!
No more incomplete or outdated context - give your AI the documentation it needs to help you code better! PSSTTT : We already have fetched some docs for you :), check out our output folder to see the result PSSSTTT2 : Don't know where to begin ? Check out our easycommand.txt
Built with ❤️ using Crawl4AI, the powerful open-source web crawler (30.5k+ ⭐)
Transform entire documentation websites into a clean, organized collection of Markdown files with a single command. mdCrawler leverages the power of Crawl4AI to automatically process documentation sites and convert them into well-structured Markdown files while preserving the original structure.
Join our vibrant Discord community AiCodingBattle where we:
- Share daily news about AI and coding tools
- Exchange expertise and best practices
- Connect with fellow AI enthusiasts
Give it a documentation URL, and mdCrawler will:
- 🔍 Automatically discover all documentation pages
- 📥 Download and process each page
- ✨ Convert the content to clean Markdown
- 📁 Save everything in an organized folder structure
Perfect for:
- 📖 Offline documentation reading
- 📝 Content migration projects
- 🔄 Documentation backups
- 🎨 Custom documentation styling
# Clone the repository
git clone https://github.com/AiCodingBattle/mdCrawler.git
cd mdCrawler
# Create and activate a virtual environment
python -m venv venv
source venv/bin/activate # On Windows use: .\venv\Scripts\activate
# Install dependencies
pip install -r requirements.txt
# Install browser dependencies
python -m playwright installYou can use either format:
# Using positional arguments (original format)
python main.py https://docs.python.org/fr/3/ python-docs
# Using named arguments
python main.py --url https://docs.python.org/fr/3/ --name python-docs# Convert multiple documentation sites using a YAML config
python main.py --config libraries_extended.yamlYour converted documentation will be available in the docs/ directory!
Here are some examples of how to use mdCrawler with popular documentation sites:
# Convert Supabase docs to Markdown
python main.py https://supabase.com/docs supabase-docsThis will create a complete Markdown version of Supabase's documentation, including:
- Authentication guides
- Database documentation
- API references
- Getting started guides
# Convert Stripe docs to Markdown
python main.py https://docs.stripe.com stripe-docsPerfect for creating offline copies of:
- Payment integration guides
- API documentation
- Product documentation
- Testing guides
# Convert Crawl4AI docs to Markdown
python main.py https://docs.crawl4ai.com crawl4ai-docsUseful for:
- Offline reference
- Contributing to the project
- Custom documentation styling
# Convert SvelteKit docs to Markdown
python main.py https://svelte.dev/docs/kit sveltekit-docsGreat for:
- Learning materials
- Framework documentation
- Component guides
Each conversion will maintain the original documentation structure and create clean, well-formatted Markdown files in their respective output directories.
- Python 3.8 or higher (Install via Windows Store for Windows users)
- pip (Python package manager)
mdCrawler now includes an extensive library of pre-configured documentation sources covering:
- Programming Languages: Python, Java, C++, Rust, Go
- Web Frameworks: Django, Flask, FastAPI, Express.js, Rails
- Database Systems: PostgreSQL, MySQL, MongoDB, Redis
- AI/ML Frameworks: TensorFlow, PyTorch, Hugging Face, LangChain
- Cloud & DevOps: Docker, Kubernetes, Terraform
- Frontend Frameworks: React, Vue, Angular, Svelte
- And many more!
To use the extended library:
- Check
libraries_extended.yamlfor available documentation sources - Run with the config file to process multiple sites:
python main.py --config libraries_extended.yamldocs/
└── your-docs/
├── index.md # Main documentation page
├── getting-started.md # Each page becomes a Markdown file
└── api/ # Original structure is preserved
├── overview.md
└── reference.md
- Crawl4AI - The backbone of our crawler, providing powerful web crawling capabilities
- Playwright - Browser automation
- Python 3.8+
This project would not be possible without:
- Crawl4AI and its amazing community
- The incredible work of @unclecode and all Crawl4AI contributors
- Everyone who has contributed to making web crawling more accessible
This project is licensed under the MIT License - see the LICENSE file for details.
Contributions are welcome! Please feel free to submit a Pull Request.
If you find this tool useful, please consider giving it a star on GitHub!