You're running a service and you need to keep all the logs for a certain period of time (legal constraints or just internal retention policy). The logs are pretty big to be stored locally, so you need a big disk somewhere in the cloud - Google Cloud Storage comes to the rescue (this is just one of the options available, Amazon also has similar products: S3 or Glacier).
These are the steps to follow:
0 - you must have a Google Account
1 - create a new project within the Google Developer Console and
activate the Google Cloud Storage API
2 - install and configure
gsutil
on you server
3 - create the job to copy/move your logs to cloud
[cc lang="bash"]
\$ cat \<
#!/bin/bash
# this script will move to cloud the logs older than 20 days
# we keep 20 days locally because 95% of queries are run against last
2 weeks logs
# we store each server's logs in a separate bucket
# we also configure a TTL for every log file (170 days)
KeepLocal=20
BucketList=`/root/gsutil/gsutil ls`
for log in `/usr/bin/find /logs/ -type f -name access.log-*.gz -mtime
+\$KeepLocal -exec /bin/ls {} \;`
do
#
bucket="logs_"`/bin/echo "\$log" | /usr/bin/cut -d '/' -f 4`
bucket_exists=`/bin/echo "\$BucketList" | /bin/grep -c
"gs://\$bucket/"`
if [ "\$bucket_exists" == "0" ]
then
/root/gsutil/gsutil mb gs://\$bucket
/root/gsutil/gsutil lifecycle set lifecycle.xml gs://\$bucket
fi
#
/root/gsutil/gsutil -q cp \$log gs://\$bucket/
if [ \$? -eq 0 ]
then
/bin/rm \$log
fi
done
EOF
\$ cat \<
<?xml version="1.0" ?>
EOF
[/cc]