owTo: Rotate Logs to S3

This article will talk about how to use logrotate to rotate your logs to S3. Here we specifically are using Gentoo Linux, and we can find EC2 AMIs on the Gentoo in the Cloud page. We will be using s3cmd to actually move the files to S3,

Configuration

/etc/logrotate.conf

This is the original /etc/logrotate.conf file we are going to edit:

/etc/logrotate.conf (Original)

We want to have the log files rotate hourly, but that is not an option for the configuration file, so we will remove:

# rotate log files weekly
weekly
#daily

and setup a cron job to take care of doing the rotation.

We just want the latest hours worth of backlogs, because they will be uploaded to S3. So we will change

# keep 4 weeks worth of backlogs
rotate 4

to

# keep 1 hours worth of backlogs
rotate 1

The default date extension is not fine grained enough for hourly rotations, so after

# use date as a suffix of the rotated file
dateext

we will add

dateformat -%Y-%m-%d-%s

Compression happens after the postrotate script, which is when the upload occurs, and since we are not keeping a significant backlog of logs, then we just do nocompress. We will replace:

# uncomment this if you want your log files compressed
compress

with

# because compression would happen after upload
nocompress

This leads to our final /etc/logrotate.conf:

/etc/logrotate.conf

/etc/logrotate.d/syslog-ng

The syslog-ng logrotate config file doesn’t change except for the s3cmd command

/etc/logrotate.d/syslog-ng

In the example here, it is using the bucket, logging-bucket, change it to a bucket that you control.

And for INSTANCE_ID we are getting the instance id of the instance and dropping the i- off the front (e.g. i-12345678becomes 12345678). The reason for this is to have the logs of different instances spread across S3 partitions. Amazon S3 Performance Tips & Tricks gives a good writeup on the issue.

/etc/logrotate.d/apache2

The apache logrotate config file doesn’t change except for adding the s3cmd commands.

/etc/logrotate.d/apache2

This example will upload into the bucket, logging-bucket, and is for the site www.example.com. The bucket needs to be changed to a bucket you control, and the site should be changed to the site it is logging.

The reason for INSTANCE_ID is so if we have multiple web servers for the same site, they will not step on each others’ toes.

The purpose for having site= before the site and instance= before the instance is to make it easy for running a hive script against the logs.

Rotating Logs

Hourly

/etc/cron.hourly/logrotate.cron

The easy way to do this is to move logrotate.cron from /etc/cron.daily to /etc/cron.hourly

Console – user@hostname ~ $

And here we’ll look at what it actually does:

/etc/cron.hourly/logrotate.cron

/etc/crontab

If we are trying to set this up on a system that doesn’t have /etc/cron.hourly or if we want to rotate on a different increment, we could instead edit the /etc/crontab file. This example shows will have it rotate hourly still.

/etc/crontab

On Shutdown

When the system is shutting down, we want the current logs to be rotated to S3. On Gentoo, we can make an/etc/local.d script for this; other Linux distributions may need a different method for this.

/etc/local.d/logrotate.stop

Dry Run

To make sure that our configuration files are set up correctly and things will do what we expect we can do a dry run like so:

Console – user@hostname ~ $

Run Manually

We can manually run the rotate, and if the conditions for rotating the logs are met they will be rotated.

Console – user@hostname ~ $

Force Rotate

If we want the log to be rotated no matter what, we can force a rotate:

Console – user@hostname ~ $

IAM Policies

We need to make sure our IAM role or user has permissions to S3.

If we have a dedicated bucket

Dedicated Bucket Policy

If we have to share a bucket

Shared Bucket Policy

Views: 16