Tooling for the Every Election project. A small scraper app to:
- Scrape information on current and recent boundary reviews from LGBCE
- Store it in a SQLite database
- Commit the data to a GitHub repo
- Raise slack notifications about status changes
- Raise GitHub issues about completed reviews on the Every Election repo
It can optionally be run on morph.io
pip install -r requirements.txt
Configuration is performed using env vars.
-
To commit the scraped data to a GitHub repo, set the following env vars:
MORPH_GITHUB_BOUNDARY_REPO = "DemocracyClub/boundary-data" MORPH_GITHUB_USERNAME = "polling-bot-4000" MORPH_GITHUB_EMAIL = "user@example.com" MORPH_GITHUB_API_KEY = "abc123"
MORPH_GITHUB_API_KEYwill need push access to the repo. -
To raise slack notifications about status changes, set:
MORPH_BOUNDARY_BOT_SLACK_WEBHOOK_URL = "https://hooks.slack.com/services/foo/bar/baz" -
To raise GitHub issues about completed reviews on the Every Election repo, set:
MORPH_GITHUB_ISSUE_ONLY_API_KEY = "abc123"MORPH_GITHUB_ISSUE_ONLY_API_KEYdoes not need any special permissions.
When running for the first time, set BOOTSTRAP_MODE = True in scraper.py
For all future runs, set BOOTSTRAP_MODE = False
python scraper.py