- Overview
- Features
- Requirements
- Installation
- Usage
- Configuration
- Performance Tests for aws sync command
s3sync.py is a utility created to sync files to/from S3 as a continuously running process, without having to manually take care of managing the sync. It internally uses the aws s3 sync command to do the sync, and uses the python module watchdog to listen to filesystem events on the monitored path and push changes to S3. For pull, there is no listener implemented, and it does simple interval based pull. Therefore for pull it is recommended to use s3fs instead - just mount the s3 bucket on your filesystem.
- Rate limiting using the python tocken-bucket module. You can set
max_syncs_per_minutein the config yaml and the file system watcher triggered pushes will be throttled as per that limit. - Optional Reporting of runtime stats for the sync operation using the pyformance module
- Ability to filter by
include_patternsandexclude_patternsor toexclude_directoriescompletely, or make the filtercase_sensitive - Automated setup of AWS CLI config by creating a separate named profile for the utility with ability to tune performance by setting
max_concurrent_requests,max_queue_size, etc. - Setuptools integration, python click based command line interface
Requires AWS CLI version 2 to be installed and available in the path
pip install pys3sync
s3sync --help
Usage: s3sync.py [OPTIONS] COMMAND [ARGS]...
A utility created to sync files to/from S3 as a continuously running
process, without having to manually take care of managing the sync. It
internally uses the aws s3 sync command to do the sync and uses python's
watchdog listener to get notified of any changes to the watched folder.
Options:
--config PATH Path to the config.yaml file containing configuration
params for this utility [required]
-v, --verbosity LVL Either CRITICAL, ERROR, WARNING, INFO or DEBUG
--help Show this message and exit.
Commands:
init Initial setup.
pull One-way continuous sync from s3 path to local path (based on
polling...
push One-way continuous sync from localpath to s3 path (uses a file...s3sync --config config.yaml push --help
Usage: s3sync.py push [OPTIONS]
One-way continuous sync from localpath to s3 path (uses a file watcher
called watchdog)
Options:
--s3path PATH Full s3 path to sync to/from [required]
--localpath PATH Local directory path which you want to sync [required]
--help Show this message and exit.s3sync --config config.yaml push --help
Usage: s3sync.py pull [OPTIONS]
One-way continuous sync from s3 path to local path (based on polling on an
interval)
Options:
--s3path PATH Full s3 path to sync to/from [required]
--localpath PATH Local directory path which you want to sync [required]
--interval INTEGER S3 polling interval in seconds [required]
--help Show this message and exit.s3sync --config-yaml config.yaml init
This utility creates a named profile for your AWS CLI so that the parameters required for the S3 cli for the utility are isolated from your regular AWS CLI profile. The first time you nee to run the init command, which will create the named profile s3sync in your local aws config (~/.aws/config), with the parameters configured in config.yaml and credentials copied from your default AWS credentials file.
You run one instance of this utility per localpath<>s3path combination that you want to continuously sync
s3sync --config config.yaml -v DEBUG push --s3path s3://<bucket>/<path> --localpath ./
s3sync --config config.yaml -v DEBUG pull --s3path s3://<bucket>/<path> --localpath ./sync --interval 2
global:
max_syncs_per_minute: 10
report_stats: False
watcher:
include_patterns:
exclude_patterns: ["*.git/*"]
exclude_directories: False
case_sensitive: False
s3:
max_concurrent_requests: 20
max_queue_size: 2000
multipart_threshold: 8MB
multipart_chunksize: 8MB
max_bandwidth:
use_accelerate_endpoint: "false"
region: ap-south-1Include/excluse patterns are implemented using pathtools.match_any_path, which ultimately supports unix glob pattern syntax. You can test your patterns using the provided script patternhelper.py. These patterns are passed to the watchdog as well as aws cli, which also uses the same syntax. Both properties accept a list of patterns.
Please change these values carefully. They depend on your machine and your internet connection. Read more about improving s3 sync transfer speeds here
Passed through to your ~/.aws/config via aws configure set default.s3.max_concurrent_requests command. Read about the parameter here
Passed through to your ~/.aws/config via aws configure set default.s3.max_queue_size command. Read about the parameter here
Passed through to your ~/.aws/config via aws configure set default.s3.multipart_threshold command. Read about the parameter here
Passed through to your ~/.aws/config via aws configure set default.s3.multipart_chunksize command. Read about the parameter here
Passed through to your ~/.aws/config via aws configure set default.s3.max_bandwidth command. Read about the parameter here
Passed through to your ~/.aws/config via aws configure set default.s3.use_accelerate_endpoint command. Read about the parameter here
Network: home/BLR (Airtel 1 Gbps Xtreme Fiber)
WiFi: 5 GHZ, RSS: -38 dbM, Tx rate: 1300 Mbps (802.11 ac)
Upload speed to s3: 18 MB/s
Download speed from s3: 15 MB/s
Number of threads for s3 sync command: 10 (default)
Test artifact 1: https://github.com/apache/storm
Size: 224M
Number of files: 3571
time aws s3 sync --storage-class REDUCED_REDUNDANCY ./ s3://psm-poc-dmp-temp/codesync
real 0m45.543s
user 0m14.755s
sys 0m3.685scp -rf ../intSDK .
time aws s3 sync --storage-class REDUCED_REDUNDANCY ./ s3://psm-poc-dmp-temp/codesync
real 0m3.141s
user 0m1.887s
sys 0m0.405srm -rf examples/
time aws s3 sync --storage-class REDUCED_REDUNDANCY --delete ./ s3://psm-poc-dmp-temp/codesync
real 0m3.436s
user 0m2.276s
sys 0m0.406stouch README.markdown
time aws s3 sync --storage-class REDUCED_REDUNDANCY --delete --exact-timestamps ./ s3://psm-poc-dmp-temp/codesync
real 0m2.602s
user 0m1.492s
sys 0m0.296stime aws s3 sync --storage-class REDUCED_REDUNDANCY --delete --exact-timestamps ./ s3://psm-poc-dmp-temp/codesync
real 0m2.442s
user 0m1.469s
sys 0m0.294sSize: 400M
Number of files: 577
time aws s3 sync --storage-class REDUCED_REDUNDANCY --delete --exact-timestamps ./ s3://psm-poc-dmp-temp/codesync
real 0m22.015s
user 0m5.972s
sys 0m2.516sTest artifact 1: https://github.com/apache/storm
Size: 224M
Number of files: 3571
time aws s3 sync --storage-class REDUCED_REDUNDANCY --delete --exact-timestamps s3://psm-poc-dmp-temp/codesync ./
real 0m26.448s
user 0m14.544s
sys 0m3.794sSize: 400M
Number of files: 577
time aws s3 sync --storage-class REDUCED_REDUNDANCY --delete --exact-timestamps s3://psm-poc-dmp-temp/codesync ./
real 0m29.268s
user 0m6.131s
sys 0m2.855s