With this project you can define a time interval to fetch logs from AWS Cloudwatch, and ship them to Logz.io.
Before using this tool, you'll need to make sure that you have AWS access keys with permissions to:
logs:FilterLogEventssts:GetCallerIdentity
Note: This solution can handle one AWS account per container. If you wish to follow multiple accounts, you'll need to create multiple containers (one container per AWS account).
docker pull logzio/cloudwatch-fetcher:latestThis directory will hold the configuration and the position file for the fetcher. The position file will allow the fetcher to resume to the point it last fetched, in case the container was stopped.
mkdir logzio-cloudwatch-fetcher \
&& cd logzio-cloudwatch-fetcherIn the directory you created in the previous step, create a configuration file and name it config.yaml.
| Field | Description | Required/Default |
|---|---|---|
aws_region |
The AWS region your log groups are in. Note that all log groups should be in the same region | Required |
log_groups |
An array of log group configuration | Required |
log_groups.path |
The AWS Cloudwatch log group you want to tail | Required |
log_groups.custom_fields |
Array of key-value pairs, for adding custom fields to the logs from the log group | - |
collection_interval |
Interval IN MINUTES to fetch logs from Cloudwatch. Minimum value is 5, maximum value is 1380 | Default: 5 |
See this config sample for example.
docker run --name logzio-cloudwatch-fetcher \
-e AWS_ACCESS_KEY_ID=<<AWS-ACCESS-KEY>> \
-e AWS_SECRET_ACCESS_KEY=<<AWS-SECRET-KEY>> \
-e LOGZIO_LOG_SHIPPING_TOKEN=<<LOGZIO-LOGS-SHIPPING-TOKEN>> \
-e LOGZIO_LISTENER=https://<<LOGZIO-LISTENER>>:8071 \
-v "$(pwd)":/logzio/src/shared \
logzio/cloudwatch-fetcher:latestReplace the following:
| Parameter | Description |
|---|---|
<<AWS-ACCESS-KEY>> |
Your AWS access key |
<<AWS-SECRET-KEY>> |
Your AWS secret key |
<<LOGZIO-LOGS-SHIPPING-TOKEN>> |
Your Logz.io logs shipping token |
<<LOGZIO-LISTENER>> |
Your logz.io listener url, for example: listener.logz.io |
Give your logs some time to get from your system to ours, and then open Logz.io.
NOTE that the logs will have the original timestamp from Cloudwatch, so when you're searching for them, make sure that you're viewing the relevant time frame.
When you stop the container, the code will run until completion of the iteration.
To make sure it will finish the iteration on time, please give it a grace period of 30 seconds when you run the docker stop command:
docker stop -t 30 logzio-cloudwatch-fetcherAfter every successful iteration of each log group, the latest time & next token we got from AWS will be written to a file name position.yaml
You can find the file inside your mounted host directory that you created.
If you stopped the container, the file will allow the fetcher to continue from the exact place it stopped.
- 0.0.1: Initial release.