wiki'd

by JoKeru

Put your logs on Google Cloud Storage

You're running a service and you need to keep all the logs for a certain period of time (legal constraints or just internal retention policy). The logs are pretty big to be stored locally, so you need a big disk somewhere in the cloud - Google Cloud Storage comes to the rescue (this is just one of the options available, Amazon also has similar products: S3 or Glacier).

These are the steps to follow:
0 - you must have a Google Account
1 - create a new project within the Google Developer Console and activate the Google Cloud Storage API
2 - install and configure gsutil on you server
3 - create the job to copy/move your logs to cloud

[cc lang="bash"]
\$ cat \< move-to-cloud.sh
#!/bin/bash

# this script will move to cloud the logs older than 20 days
# we keep 20 days locally because 95% of queries are run against last 2 weeks logs
# we store each server's logs in a separate bucket
# we also configure a TTL for every log file (170 days)

KeepLocal=20

BucketList=`/root/gsutil/gsutil ls`
for log in `/usr/bin/find /logs/ -type f -name access.log-*.gz -mtime +\$KeepLocal -exec /bin/ls {} \;`
do
#
bucket="logs_"`/bin/echo "\$log" | /usr/bin/cut -d '/' -f 4`
bucket_exists=`/bin/echo "\$BucketList" | /bin/grep -c "gs://\$bucket/"`
if [ "\$bucket_exists" == "0" ]
then
/root/gsutil/gsutil mb gs://\$bucket
/root/gsutil/gsutil lifecycle set lifecycle.xml gs://\$bucket
fi
#
/root/gsutil/gsutil -q cp \$log gs://\$bucket/
if [ \$? -eq 0 ]
then
/bin/rm \$log
fi
done
EOF

\$ cat \< lifecycle.xml
<?xml version="1.0" ?>






170



EOF
[/cc]

Comments