Simple script for synchronizing files over FTPS and SFTP. In addition to the files themselves, the script uploads an index file containing hashes of all uploaded files. This index file is then used to skip unchanged files during the next synchronization.
This strategy was designed specifically for uploading the files of a static website from a continuous integration system. It avoids some drawbacks of similar tools in this scenario.
lftp'smirrorcommand only checks the modification dates of the files. This is useless for files produced by a CI system, which might often be the same but get a new modification date for each run.rsyncneeds to be installed on both machines, which is usually not possible if the target machine is a web hosting space.rclonecan skip files based on checksums, but only when connected through SFTP. On web hosting spaces, you probably don't get SSH access. Even if you do, the checksums get computed remotely and expensive computations on the web server are often not accepted by the providers.
- An FTPS or SFTP connection.
- Python 3 on the uploading machine.
- The Python package
paramikowhen using SFTP.
That's it. In particular, the script doesn't require or compute anything on the target machine.
For the most simple use, run
ftp_sync.py --host HOST --user FTP_USER --password PASSWORD some/directoryThis will upload all files from some/directory to the root of the FTP user. Note that files that were uploaded previously, but don't exist locally anymore, will not be deleted by default. If you want to delete these files, specify the --delete-files parameter.
These are the complete parameters of the script:
usage: ftp_sync.py [-h] --host HOST --user USER --password PASSWORD [--sftp]
[--allow-any-host-key] [--delete-files] [--dry-run]
[--index-file FILE]
LOCAL_PATH [REMOTE_PATH]
positional arguments:
LOCAL_PATH directory on the local machine to synchronize
REMOTE_PATH path on the remote machine where files get uploaded
(default: /)
optional arguments:
-h, --help show this help message and exit
--host HOST FTP host
--user USER FTP user
--password PASSWORD FTP password
--sftp Connect using SFTP instead of FTPS
--allow-any-host-key Allow any host key when connecting to SFTP
--delete-files delete files that were uploaded but do not exist
locally anymore
--dry-run print output only, do not modify any files
--index-file FILE filename for the index file (default:
ftp_sync.json.gz)