BitFolk Users March 2011

users@mailman.bitfolk.com

18 participants
17 discussions

by Adam Sweet

Hi everyone My VPS running Debian Lenny has recently started to crash frequently, about 6 times in the last week. I've had out of memory conditions in the past, though reasonably infrequently, so that was my first I thought. I've bumped my RAM allocation, added extra swap and removed a few services which weren't strictly necessary but the crashing continues. My resource graphs don't suggest any memory usage build up before the crash or anything else untoward going on and the system logs report nothing, it seems to just go more or less instantaneously. I managed to capture some of the console output when I got to the machine: http://pastebin.com/TjZRCebQ But it had been dead for about 6 hours as I'd been out for the evening, so it's possible the crucial part was lost before I got to it. I'm not much of an expert on interpreting kernel panics so I thought I'd ask if anybody has any greater insight on the cause or how I can capture the output to disk. At this point, my only options seem to be to migrate services off to another system until the system stops crashing, or add more RAM, or start afresh on a new VPS to see if it goes away. Regards, Adam Sweet -- http://blog.adamsweet.org/

13 years, 11 months

Re: [bitfolk] Individual snapshot disk usage report added to panel/backups

by Deanna Earley

I found the same a while ago on mine. It reduced a many GB hourly diff to a few MB which including the snapshot was ~1/5 the size. -- Deanna Earley ----- Reply message ----- From: "Adam Spiers" <bitfolk(a)adamspiers.org> Date: Thu, Mar 24, 2011 12:02 Subject: [bitfolk] Individual snapshot disk usage report added to panel/backups To: "Andy Smith" <andy(a)bitfolk.com> Cc: <users(a)lists.bitfolk.com> On 24 March 2011 11:38, Andy Smith <andy(a)bitfolk.com> wrote: > Hi Adam, > > On Thu, Mar 24, 2011 at 11:26:53AM +0000, Adam Spiers wrote: >> This is great - thanks! However I'm struggling to understand the >> numbers shown. On my VPS I see hourly snapshots consuming 2.5GB - >> what does this mean exactly? Presumably not that my VPS is churning >> that much data per hour, because it should be idle most of the time >> AFAIK. > > You refer to the differential report, so there are ~2.5GiB of diffs > per snapshot. Since the per-snapshot report shows each snapshot > coming in at ~9.5GiB that means you have the same amount of files > but with ~2.5GiB changing content every time. Wow, OK - something's not right for sure, as there shouldn't be anywhere near that much churn on my VPS. > Bear in mind that any time a file changes (even if it's just > metadata), a copy of both the old and new version will be stored in > their entirety. By metadata presumably you mean inode updates? So I can search for churn via find -ctime? In that case, could I expect remounting partitions with the noatime or relatime options to drastically reduce the size of incrementals? http://en.wikipedia.org/wiki/Atime_(Unix)#ctime http://lwn.net/Articles/244829/ > I can have a look and see what I can make of it if you like. I'd > rather not look at customer data without permission. > > If you want to look yourself, look for files that change metadata > (mtime, ownership) all the time, like if they're being checked out > of version control or downloaded from somewhere without preserving > mtimes. Something like that. Ahah. I think I may have located the culprits: huge log files. -rw-r--r-- 1 root root 67797975 Mar 24 11:47 /home/adam/web/adamspiers.org/logs/error.log -rw-r--r-- 1 root root 141526573 Mar 24 11:55 /var/www/redmine/log/mod_rails/error.log -rw-r--r-- 1 www-data www-data 314066833 Mar 24 11:55 /var/www/redmine/log/production.log -rw-r--r-- 1 www-data www-data 895519982 Mar 24 11:49 /var/www/wordpress/wp-content/debug.log -rw-r--r-- 1 root root 899657028 Mar 24 11:56 /home/adam/web/adamspiers.org/logs/access.log If I start rotating these regularly, then I could expect the incrementals to significantly shrink, right? For the benefit of anyone else who wishes to sanity check their churn, I did something like this: backup_dirs="/boot /etc /home /initrd /root /srv /usr/local /var" find $backup_dirs -ctime -2 >& /tmp/find_-ctime_-2.out xargs ls -ld < /tmp/find_-ctime_-2.out | sort -n -k5 If there's a better way then please do share it. _______________________________________________ users mailing list users(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/users

14 years

Individual snapshot disk usage report added to panel/backups

by Andy Smith

Hi, We've added a report of each snapshot's disk usage to: https://panel.bitfolk.com/backups/ This will make it easier for you to spot which snapshot introduced lots of files, for example. Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

14 years

Ubuntu 10.04 Logwatch

by Robert Gauld

Hi folks I'm after either some signposting to a more appropriate place to ask or a solution. The problem I have is that I'm trying to setup logwatch to monitor a log file for a script I've written and am having no joy whatsoever. Here's the steps I've taken and content, ownership and permissions of relevant files (ask if I've missed something useful). 1. Setup script to log to /var/log/ssh-validate 2. Create script to process logfile (/etc/logwatch/scripts/services/ssh-validate) 3. Create logfile definition (/etc/logwatch/logfiles/ssh-validate) 4. Create service definition (/etc/logwatch/services/ssh-validate.conf) File /etc/logwatch/scripts/services/ssh-validate root:root -rwxr-xr-x Perl script, reads from STDIN, outputs to STDOUT, works from command line (using cat /var/log/ssh-validate | ./ssh-validate) File /etc/logwatch/logfiles/ssh-validate root:root -rw-r--r-- LogFile = ssh-validate Archive = ssh-validate.*.gz File /etc/logwatch/services/ssh-validate.conf root:root -rw-r--r-- Title = ssh-validate LogFile = ssh-validate -- Robert Gauld http://www.robertgauld.co.uk

14 years

Clarification regarding "reinstall on system compromise" has been added to Terms and Conditions

by Andy Smith

Hello, A clarification has today been added to BitFolk's Terms and Conditions: http://bitfolk.com/policy/terms.html * BitFolk reserves the right to require that The Customer perform a reinstall of The Service if at any time BitFolk believes that The Service has been seriously compromised. Whether the compromise is serious enough to require reinstall will be at BitFolk's sole discretion. I refer to this as a clarification because it has always been the case that BitFolk has required reinstallation of seriously compromised VPSes. This has previously been enforced under the "block or terminate service at any time for any reason" clause, but it was suggested that it would be preferable to explicitly list it, and I agree with that suggestion. I hope you will agree in any case that responsible systems administration practice is to reinstall when a root-level compromise is discovered or strongly suspected. Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

14 years

Problems with the authoritative nameservers and names with \032 in them

by Andy Smith

Hello, It was brought to my attention earlier today that a zone file correctly loaded into a customer's BIND nameserver and transferred to BitFolk's servers was not being served correctly by two of the servers. On further investigation it seems that there's some bug in PowerDNS regarding DNS names (the bit on the left hand side) with "\032" in them. Last year I fixed a bug in the sqlite backend of PowerDNS regarding incorrect escaping of "\", which had prevented "\032" and anything else with "\" in it being stored correctly as DNS content (the right hand side). I'm not yet sure if this is related or more fundamental. I've reported the bug. At the moment the impact is that if you have a DNS zone with something like this in it: foo\032bar._http._tcp SRV 0 0 10 baz.example.com. then it will get stored in b.authns.bitfolk.com and c.authns.bitfolk.com as: foo bar._http._tcp SRV 0 0 10 baz.example.com. This is incorrect. BIND servers taking the same AXFR would store "foo\032bar._http._tcp". Anything correctly querying for "foo\032bar._http._tcp" will not get answers from the affected servers. Therefore for the moment if you have zones that use these types of record on the left hand side, you should not delegate them to {b,c}.authns.bitfolk.com. "\032" on the right hand side is okay. If you use PowerDNS yourselves I'd be interested to know if you can replicate this with backends other than gsqlite (e.g. gmysql or gpsql), as this would indicate whether it's a problem with the gsqlite backend or something else. FWIW this type of DNS name seems to be used for DNS-based service discovery: http://www.dns-sd.org/ Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

14 years

How to verify the SSH fingerprints of BitFolk's hosts

by Andy Smith

Hello, When connecting via SSH to a BitFolk host for the first time (e.g. your Xen Shell), you may be wondering how to verify that the SSH fingerprint is genuine. To assist you in this, a list of them has been securely published and also uploaded into the OpenPGP web of trust. Please see: https://tools.bitfolk.com/wiki/Verifying_BitFolk%27s_SSH_fingerprints for more details. Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

14 years

No more support for Gentoo

by Andy Smith

Hi folks, I've had to remove support for Gentoo because I don't know how to compile a kernel in Gentoo that has Xen support; I'm not a Gentoo user and am not willing/able to put the time in at the moment to learn that. Hardly anyone wants Gentoo[1] so it's hard to justify. This doesn't mean that you can't install Gentoo. There are still some customers at BitFolk using Gentoo, and I find it hard to believe that you can't compile a Xen kernel "the Gentoo way", I just don't personally know how. I'm not willing to call Gentoo supported when the only way I know to get it going is to copy in a kernel and initrd from elsewhere. If you know better and are willing to write a wiki article[2] giving step by step instructions on how to compile a Xen kernel "the Gentoo way" then firstly, some other customers would be very happy, because they don't seem to know either; and secondly, I'd appreciate that a lot too because I'll feel comfortable offering Gentoo as a supported option again. If the article is of good quality I may also be convinced to give you some service credit. Cheers, Andy [1] http://strugglers.net/~andy/bitfolk/distros.html [2] https://tools.bitfolk.com/wiki/Compiling_a_Xen_kernel_under_Gentoo for example? Whatever you think is best. -- http://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

14 years

Cacti bandwidth graphs with "total transferred" figures now available

by Andy Smith

Hello, Recently it was pointed out that the Cacti bandwidth graphs that BitFolk provides do not contain a figure for the total data transferred in/out. It turns out that the graph template was poorly chosen, since a template with this information does already exist. Here's an example of the style of bandwidth graph almost all of you have: http://tools.bitfolk.com/cacti/graph_910.html Here's an example of one with total transferred figures: http://tools.bitfolk.com/cacti/graph_2472.html Unfortunately it is not possible to just switch existing graphs to the different template. So, if it seems like something you'd like to have, please send an email to support(a)bitfolk.com asking for new bandwidth graphs. You can of course always work out roughly what the figures would be, from the average rate figures: $ units '10.03kilobit/sec' 'megabytes/day' * 108.324 / 0.0092315646 Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

14 years

BitFolk's Ubuntu VPSes do not run update-grub after a kernel install/upgrade

by Andy Smith

Hello, A while back someone pointed out to me that BitFolk's Ubuntu VPSes don't run update-grub after a kernel upgrade/install. This can lead to some annoyance if you reboot expecting a new kernel, or even if you removed the one that you were using before. We don't tend to keep test VPSes around for very long, so I didn't notice this myself. The reason for it is that /etc/kernel-img.conf does not contain: postinst_hook = update-grub postrm_hook = update-grub So if you want update-grub to be run after upgrade/install/removal, you should add the above to the file. The default install images have now had this set. Cheers, Andy -- http://bitfolk.com/ -- No-nonsense VPS hosting _______________________________________________ announce mailing list announce(a)lists.bitfolk.com https://lists.bitfolk.com/mailman/listinfo/announce

14 years

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

BitFolk Users March 2011