The Firecrawl Toolkit is the easiest way for developers to interact with web content through crawling, scraping, and mapping capabilities. It offers seamless integration for web crawling, content extraction, and site mapping, allowing you to process websites with advanced features like custom actions, multiple output formats, and batch processing—all in one comprehensive package with minimal dependencies.
- 🕷️ Web Crawling: Traverse websites with customizable depth and path controls, supporting both internal and external link processing
- 📄 Content Extraction: Extract content in multiple formats (Markdown, HTML, raw HTML) with smart content filtering
- 🗺️ Site Mapping: Generate comprehensive site maps with advanced search and subdomain capabilities
- 🔄 Batch Processing: Process multiple URLs simultaneously with unified configurations
- 🤖 Custom Actions: Automate complex interactions (clicking, scrolling, form filling) during scraping
- 📱 Device Emulation: Switch between mobile and desktop views with customizable headers
- 🌎 Geolocation: Simulate different locations with country and language preferences
- ⚡ Smart Retry Logic: Built-in retry mechanism with real-time status monitoring and webhooks
- 🪶 Lightweight Design: Minimal dependencies powered by requests for easy setup and deployment
- 🔒 Robust Error Handling: Comprehensive error catching and validation system
- 🎯 Parameter Validation: Extensive validation for all API inputs and configurations
- 📊 Multiple Output Formats: Support for various output types (Markdown, HTML, screenshots, etc.)
- Installation
- API Key Configuration
- Usage
- Advanced Configuration
- Available Formats 8000
- Supported Actions
- Error Handling and Safety
- Contributing
- Issues and Support
- Feature Requests
- Versioning and Changelog
- Security
- License
-
Clone the repository:
git clone https://github.com/RMNCLDYO/firecrawl-toolkit.git
-
Navigate to the repository folder:
cd firecrawl-toolkit -
Install the required dependencies:
pip install -r requirements.txt
-
Obtain an API key from Firecrawl.
-
You have three options for managing your API key:
Click here to view the API key configuration options
-
Setting it as an environment variable on your device (recommended for everyday use)
- Navigate to your terminal.
- Add your API key like so:
export FIRECRAWL_API_KEY=your_api_key
This method allows the API key to be loaded automatically when using the wrapper.
-
Using an .env file (recommended for development):
- Install python-dotenv if you haven't already:
pip install python-dotenv. - Create a .env file in the project's root directory.
- Add your API key to the .env file like so:
FIRECRAWL_API_KEY=your_api_key
This method allows the API key to be loaded automatically when using the wrapper.
- Install python-dotenv if you haven't already:
-
Direct Input:
-
If you prefer not to use a
.envfile, you can directly pass your API key as an argument to the wrapper function.Wrapper
api_key="your_api_key"
This method requires manually inputting your API key each time you initiate an API call.
-
-
For traversing websites and extracting content with customizable depth and path controls.
import firecrawl
# Basic crawling
firecrawl.crawl(
url="https://example.com",
formats=["markdown", "html"]
)For extracting content from specific URLs with custom actions and formatting.
import firecrawl
# Single URL scraping
firecrawl.scrape(
url="https://example.com",
formats=["markdown", "html"],
onlyMainContent=True
)For processing multiple URLs simultaneously with shared configurations.
import firecrawl
# Batch scraping
firecrawl.batch_scrape(
urls=["https://example.com", "https://sitemaps.org"],
formats=["markdown", "html"]
)For generating comprehensive site maps with search capabilities.
import firecrawl
firecrawl.map(
url="https://example.com",
includeSubdomains=True,
limit=1000
)| Description | Parameter | Type | Example |
|---|---|---|---|
| Output Formats | formats |
List | ["markdown", "html", "rawHtml"] |
| Main Content Only | onlyMainContent |
Boolean | True |
| Include Tags | includeTags |
List | ["article", "main"] |
| Exclude Tags | excludeTags |
List | ["nav", "footer"] |
| Custom Headers | headers |
Dict | {"User-Agent": "Custom"} |
| Wait Time | waitFor |
Integer | 1000 |
| Mobile View | mobile |
Boolean | False |
| Custom Actions | actions |
List | [{"type": "click", "selector": "#btn"}] |
| Location | location |
Dict | {"country": "US", "languages": ["en-US"]} |
markdown: Formatted Markdown contenthtml: Clean HTML contentrawHtml: Original HTML contentlinks: Extracted linksextract: Custom content extractionscreenshot: Page screenshotscreenshot@fullPage: Full page screenshot
The toolkit supports various page interactions:
| Action Type | Description | Parameters |
|---|---|---|
wait |
Wait for element/time | milliseconds, selector |
click |
Click elements | selector |
write |
Input text | selector, text |
press |
Press keyboard keys | key |
scroll |
Scroll page | direction, amount |
screenshot |
Take screenshots | fullPage |
scrape |
Get current page state | None |
| Error Type | Description | Solution |
|---|---|---|
| ConfigurationError | Missing or invalid configuration | Check config.yaml and API key |
| ValidationError | Invalid request parameters | Verify parameter values |
| APIError | API-related issues | Check error message for details |
| NetworkError | Connection problems | Verify internet connection |
| ResponseError | Invalid API response | Check response format expectations |
Contributions are welcome!
Please refer to CONTRIBUTING.md for detailed guidelines on how to contribute to this project.
Encountered a bug? We'd love to hear about it. Please follow these steps to report any issues:
- Check if the issue has already been reported.
- Use the Bug Report template to create a detailed report.
- Submit the report here.
Your report will help us make the project better for everyone.
Got an idea for a new feature? Feel free to suggest it. Here's how:
- Check if the feature has already been suggested or implemented.
- Use the Feature Request template to create a detailed request.
- Submit the request here.
Your suggestions for improvements are always welcome.
Stay up-to-date with the latest changes and improvements in each version:
- CHANGELOG.md provides detailed descriptions of each release.
Your security is important to us. If you discover a security vulnerability, please follow our responsible disclosure guidelines found in SECURITY.md. Please refrain from disclosing any vulnerabilities publicly until said vulnerability has been reported and addressed.
Licensed under the MIT License. See LICENSE for details.