Re: [bitfolk] Individual snapshot disk usage report added to panel/backups - BitFolk Users

24 Mar 2011

I found the same a while ago on mine. It reduced a many GB hourly diff to a few MB which
including the snapshot was ~1/5 the size.

-- 
Deanna Earley

----- Reply message -----
From: "Adam Spiers" &lt;bitfolk(a)adamspiers.org&gt;
Date: Thu, Mar 24, 2011 12:02
Subject: [bitfolk] Individual snapshot disk usage report added to	panel/backups
To: "Andy Smith" &lt;andy(a)bitfolk.com&gt;
Cc: &lt;users(a)lists.bitfolk.com&gt;

On 24 March 2011 11:38, Andy Smith &lt;andy(a)bitfolk.com&gt; wrote:
...
  Hi Adam,

 On Thu, Mar 24, 2011 at 11:26:53AM +0000, Adam Spiers wrote:
  This is great - thanks!  However I'm
struggling to understand the
 numbers shown.  On my VPS I see hourly snapshots consuming 2.5GB -
 what does this mean exactly?  Presumably not that my VPS is churning
 that much data per hour, because it should be idle most of the time
 AFAIK. 
 You refer to the differential report, so there are ~2.5GiB of diffs
 per snapshot. Since the per-snapshot report shows each snapshot
 coming in at ~9.5GiB that means you have the same amount of files
 but with ~2.5GiB changing content every time. 
Wow, OK - something's not right for sure, as there shouldn't be
anywhere near that much churn on my VPS.

...
  Bear in mind that any time a file changes (even if
it's just
 metadata), a copy of both the old and new version will be stored in
 their entirety. 
By metadata presumably you mean inode updates?  So I can search for
churn via find -ctime?  In that case, could I expect remounting
partitions with the noatime or relatime options to drastically reduce
the size of incrementals?

http://en.wikipedia.org/wiki/Atime_(Unix)#ctime
http://lwn.net/Articles/244829/

...
  I can have a look and see what I can make of it if you
like. I'd
 rather not look at customer data without permission.

 If you want to look yourself, look for files that change metadata
 (mtime, ownership) all the time, like if they're being checked out
 of version control or downloaded from somewhere without preserving
 mtimes. Something like that. 
Ahah.  I think I may have located the culprits: huge log files.

-rw-r--r--  1 root     root      67797975 Mar 24 11:47
/home/adam/web/adamspiers.org/logs/error.log
-rw-r--r--  1 root     root     141526573 Mar 24 11:55
/var/www/redmine/log/mod_rails/error.log
-rw-r--r--  1 www-data www-data 314066833 Mar 24 11:55
/var/www/redmine/log/production.log
-rw-r--r--  1 www-data www-data 895519982 Mar 24 11:49
/var/www/wordpress/wp-content/debug.log
-rw-r--r--  1 root     root     899657028 Mar 24 11:56
/home/adam/web/adamspiers.org/logs/access.log

If I start rotating these regularly, then I could expect the
incrementals to significantly shrink, right?

For the benefit of anyone else who wishes to sanity check their churn,
I did something like this:

  backup_dirs="/boot /etc /home /initrd /root /srv /usr/local /var"
  find $backup_dirs -ctime -2 >& /tmp/find_-ctime_-2.out
  xargs ls -ld < /tmp/find_-ctime_-2.out | sort -n -k5

If there's a better way then please do share it.

_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users