(See also lambda-scraper)
Use AWS Lambda functions as a proxy to scrape web pages with Selenium. This is a cost effective way to have access to a large pool of IP addresses. Run the following to create as many Lambda functions as you need (one for each IP address). The number of functions as well as the region can be specified in variables.tf. Each Lambda function changes IP address after approximately 6 minutes of inactivity. For example, you could create 360 Lambda functions which you cycle through one per second, while making as many requests as possible via each corresponding IP address. Note that, in practice, AWS will sometimes assign the same IP address to more than one Lambda function.
You will need to have installed Terraform and Docker.
git clone https://github.com/teticio/lambda-selenium.git
cd lambda-selenium
terraform init
terraform apply -auto-approve
# run "terraform apply -destroy -auto-approve" in the same directory to tear all this down againYou can specify an AWS_PROFILE and AWS_REGION with
terraform apply -auto-approve -var 'region=AWS_REGION' -var 'profile=AWS_PROFILE'An example of how to use this from Python is provided in test_selenium.py. It runs the script in example.py to search for descriptions of dog breeds in Google.
AWS_DEFAULT_REGION=AWS_REGION python test_selenium.pyThere are also examples of running the Lambda functions in parallel and asynchronously, which greatly speed up the process.
# Multi-processing (uses multiple CPU cores)
AWS_DEFAULT_REGION=AWS_REGION python test_selenium_parallel.py
# Asynchronous
AWS_DEFAULT_REGION=AWS_REGION python test_selenium_async.py