I found the same a while ago on mine. It reduced a many GB hourly diff to a few MB which including the snapshot was ~1/5 the size.

Deanna Earley

----- Reply message -----
From: "Adam Spiers" <bitfolk@adamspiers.org>
Date: Thu, Mar 24, 2011 12:02
Subject: [bitfolk] Individual snapshot disk usage report added to panel/backups
To: "Andy Smith" <andy@bitfolk.com>
Cc: <users@lists.bitfolk.com>

On 24 March 2011 11:38, Andy Smith <andy@bitfolk.com> wrote:
> Hi Adam,
> On Thu, Mar 24, 2011 at 11:26:53AM +0000, Adam Spiers wrote:
>> This is great - thanks!  However I'm struggling to understand the
>> numbers shown.  On my VPS I see hourly snapshots consuming 2.5GB -
>> what does this mean exactly?  Presumably not that my VPS is churning
>> that much data per hour, because it should be idle most of the time
> You refer to the differential report, so there are ~2.5GiB of diffs
> per snapshot. Since the per-snapshot report shows each snapshot
> coming in at ~9.5GiB that means you have the same amount of files
> but with ~2.5GiB changing content every time.

Wow, OK - something's not right for sure, as there shouldn't be
anywhere near that much churn on my VPS.

> Bear in mind that any time a file changes (even if it's just
> metadata), a copy of both the old and new version will be stored in
> their entirety.

By metadata presumably you mean inode updates?  So I can search for
churn via find -ctime?  In that case, could I expect remounting
partitions with the noatime or relatime options to drastically reduce
the size of incrementals?


> I can have a look and see what I can make of it if you like. I'd
> rather not look at customer data without permission.
> If you want to look yourself, look for files that change metadata
> (mtime, ownership) all the time, like if they're being checked out
> of version control or downloaded from somewhere without preserving
> mtimes. Something like that.

Ahah.  I think I may have located the culprits: huge log files.

-rw-r--r--  1 root     root      67797975 Mar 24 11:47
-rw-r--r--  1 root     root     141526573 Mar 24 11:55
-rw-r--r--  1 www-data www-data 314066833 Mar 24 11:55
-rw-r--r--  1 www-data www-data 895519982 Mar 24 11:49
-rw-r--r--  1 root     root     899657028 Mar 24 11:56

If I start rotating these regularly, then I could expect the
incrementals to significantly shrink, right?

For the benefit of anyone else who wishes to sanity check their churn,
I did something like this:

 backup_dirs="/boot /etc /home /initrd /root /srv /usr/local /var"
 find $backup_dirs -ctime -2 >& /tmp/find_-ctime_-2.out
 xargs ls -ld < /tmp/find_-ctime_-2.out | sort -n -k5

If there's a better way then please do share it.

users mailing list