Munin is a great resource monitoring tool that is simple to install and configure. It shows data in form of graphs, which once you’ve learned to interpret often allow you to pinpoint an issue in one instant.
Among other things, we use Munin to monitor the disk usage on servers storing backups of a few MySQL database clusters. These clusters are managed by ClusterControl.
One cluster is set up to make a full backup every four hours, which gives in Munin graphs the shape of a sawtooth wave. The « drops » in the wave correspond to the times ClusterControl applies the backup retention policy and deletes the old backups, around 1 AM every night:
On another cluster the database is backed up via mysqldump daily, around 9 PM. This should give an horizontal line with « bumps » in it, where the start of the bump corresponds to the start of the backup, and the end of the bump corresponds to the deletion of the old daily backup:
However, here you can immediately notice that the line is tilted, which means that the disk space usage is increasing. Is the database (and hence the backups) growing up in size? No, because in this case the bumps would grow bigger with time, while here they’re all identical.
The only explanation is that something is left over from the old backups. And upon checking the old backup directories we found what it was.
ClusterControl makes a mysqldump backup by creating a directory BACKUP-<nnnn> with three files in it: <timestamp>_mysqldb.sql.gz, <timestamp>_data.sql.gz, and <timestamp>_schema.sql.gz. Because of a small bug in v1.2.9, ClusterControl only deletes the <timestamp>_data.sql.gz file when wiping out the old backups.
While awaiting for a bugfix, here’s a Bash script that can be set up as a (daily) cron job to clean up the old backup directories.
cd /backup/mysqldump
for dir in `ls`
do
[ -e $dir/*_mysqldb.sql.gz -a ! -e $dir/*_data.sql.gz ] && rm -f $dir/*.sql.gz && echo "$dir cleaned"
done
Laisser un commentaire
Participez-vous à la discussion?N'hésitez pas à contribuer!