17 Apr
2010
17 Apr
'10
4:41 a.m.
Hello,
As you may know, ext3 filesystem is by default set to require a fsck
at boot time based on both elapsed time since last fsck and number
of mounts since last fsck.
The time-based fsck can be painful, because when one of BitFolk's
servers is rebooted a high proportion of the VPSes on it won't have
been rebooted inside this time period (typically 6 months).
Therefore almost every VPS will fsck its filesystems at the same
time, causing massive IO load and a slow boot for everyone.
Having now realised this, I'm considering disabling the time-based
fsck by default.
Would it bother you to discover your VPS had been provided with
time-based fsck disabled?
Would it bother you if you one day discovered that time-based fsck
had been disabled for you without your knowledge since you aren't on
this mailing list?
In case you're interested, you can see the current settings like
this:
$ sudo tune2fs -l /dev/xvda | grep -i 'check\|mount count'
Mount count: 2
Maximum mount count: 34
Last checked: Sat Oct 17 09:10:33 2009
Check interval: 15552000 (6 months)
Next check after: Thu Apr 15 09:10:33 2010
And you can disable time-based fsck like this:
$ sudo tune2fs -i 0 /dev/xvda
tune2fs 1.41.3 (12-Oct-2008)
Setting interval between checks to 0 seconds
$ sudo tune2fs -l /dev/xvda | grep -i 'check\|mount count'
Mount count: 2
Maximum mount count: 34
Last checked: Sat Oct 17 09:10:33 2009
Check interval: 0 (<none>)
(replace /dev/xvda with whatever your partitions are -- see
/proc/partitions)
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
You dont have to be illiterate to use the Internet, but it help's.
-- Mike Bristow
17 Apr
17 Apr
8:09 a.m.
On 17/04/10 05:41, Andy Smith wrote:
Would it bother you if you one day discovered that
time-based fsck
had been disabled for you without your knowledge since you aren't on
this mailing list?
It probably would actually - even though ext3 rarely goes wrong it has
happened to me a couple of times before.
It could be possible to take a leaf out of the cicada's book and set the
time / mount count to primes (7/31 maybe?) to help things along - and I
don't know if you keep stats on how often users reboot their VMs (you'd
hope more than once a year) but if so setting the time to some point in
the 60-70% percentile or something would improve things without getting
rid of the checking altogether.
You could always mail customers directly and have them deliberately turn
it back on if they want it, but I would imagine people will care much
more after the fs goes wrong and they remember checking was off at some
point a year or so from now ;)
Cheers
Alex.
--
This message was scanned by Better Hosted and is believed to be clean.
http://www.betterhosted.com
8:46 a.m.
Hi Andy,
On Sat, April 17, 2010 5:41 am, Andy Smith wrote:
The time-based fsck can be painful,
Indeed, and they always seem to occur when you really down want them to!
Having now realised this, I'm considering
disabling the time-based
fsck by default.
Good idea - fully support this.
Would it bother you to discover your VPS had been
provided with
time-based fsck disabled?
Not at all, particularly if you provided instructions for people to enable
it.
Would it bother you if you one day discovered that
time-based fsck
had been disabled for you without your knowledge since you aren't on
this mailing list?
Without knowledge, perhaps. For me personally I wouldn't mind because I'm
not keen on time-based checks however I can see why someone else might
mind if they see things from a different perspective.
However, whilst not everyone is on the mailing list you've still got
contact e-mail addresses for all your customers? If so, can you mailshot
us all to advise us of such an intended change? As long as users can
change it back (and are told how to do so) I don't see it being a problem.
The key issue is that the default is a pain shouldered by everyone so if
you can reduce this pain in one fell swoop whilst still allowing anyone to
change their own configuration if desired then surely everyone's a winner.
Mathew
9:14 a.m.
Hello,
On Sat, Apr 17, 2010 at 09:09:52AM +0100, Alex Hudson wrote:
On 17/04/10 05:41, Andy Smith wrote:
You *can* force a fsck on next boot by doing:
$ sudo shutdown -r -F
You can also do an fsck in read-only mode and if it finds
problems you know then to do the reboot:
$ sudo e2fsck -fvn /dev/xvda
So these are ways for people to periodically do an fsck or know they
need to do a single user mode one, without waiting until the host is
rebooted.
So anyway, say I went through and disabled time-based fsck and told
everyone I'd done so by emailing them directly, explained I don't
mind if you switched it back on, and said how. Would that be OK?
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
Would it bother you if you one day discovered
that time-based fsck
had been disabled for you without your knowledge since you aren't on
this mailing list?
It probably would actually - even though ext3 rarely goes wrong it has
happened to me a couple of times before.
9:15 a.m.
Hi Mathew,
On Sat, Apr 17, 2010 at 09:46:45AM +0100, Mathew Newton wrote:
However, whilst not everyone is on the mailing list
you've still got
contact e-mail addresses for all your customers? If so, can you mailshot
us all to advise us of such an intended change? As long as users can
change it back (and are told how to do so) I don't see it being a problem.
Good point.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
"It is I, Simon Quinlank. The chief conductor on the bus that is called
hobby." -- Simon Quinlank
11:41 a.m.
Hi,
I
don't know if you keep stats on how often users reboot their VMs (you'd
hope more than once a year)
What? Oh how things have changed. Time was you'd expect to go three or
four years between reboots.
Regards,
@ndy
--
andyjpb(a)ashurst.eu.org
http://www.ashurst.eu.org/
0x7EBA75FF
12:37 p.m.
On 17/04/10 12:41, Andy Bennett wrote:
Not sure about when "time was" actually occurred, but I would consider
three to four years between reboot essentially dereliction of sysadmin
duties. If anything I would say kernel security etc. has improved over
the years - I would have said you can go longer now than you could in
the past between upgrades, but that you need to be much more timely
about applying updates because the internet is that much more hostile.
I would also say that I don't see the point of not rebooting. If doing
maintenance on machines results in problematic service downtime then the
problem is the design of the service, not the need to do maintenance.
Cheers
Alex.
--
This message was scanned by Better Hosted and is believed to be clean.
http://www.betterhosted.com
I don't know if you keep stats on how often
users reboot their VMs
(you'd hope more than once a year)
What? Oh how things have changed. Time was you'd expect to go three or
four years between reboots. 
7:10 p.m.
On 17 Apr 10, at 04:41 , Andy Bennett wrote:
What? Oh how things have changed. Time was you'd
expect to go three or four years between reboots.
Perhaps on systems in completely controlled / closed networks that might have been true,
but in the 18 years I've been administering systems on the internet there have always
been times when it's been occasionally necessary to reboot in order to apply security-
or stability-related bugfixes. And that's not even taking into account downtime due
to failures or maintenance with facilities. (Server hosting facilities didn't used
to be as robust as they are now.)
I don't mean to start an argument, but I'm always suspicious of assertions that
things in the past were better than they really are. :) Progress is real.
Graham
18 Apr
18 Apr
2:34 p.m.
Hello,
On Sat, Apr 17, 2010 at 04:41:47AM +0000, Andy Smith wrote:
The time-based fsck can be painful, because when one
of BitFolk's
servers is rebooted a high proportion of the VPSes on it won't have
been rebooted inside this time period (typically 6 months).
Therefore almost every VPS will fsck its filesystems at the same
time, causing massive IO load and a slow boot for everyone.
Having now realised this, I'm considering disabling the time-based
fsck by default.
By coincidence, this just popped up in my feeds:
http://utcc.utoronto.ca/~cks/space/blog/linux/TunefsOffExt3Checks
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
19 Apr
19 Apr
9 a.m.
Hi,
Your reply is, of course, reasoned and sensible. :-) Whilst multi-year
uptimes were not unheard of, they certainly weren't the "norm" either.
I was just attacking the assertion that "reboot" was something that
should be expect or even engaged in lightly.
There sees to be a move towards "change and patch" for the sake of it as
opposed to "run what you need, keep it secure with multiple layers of
defense and be wary of making any changes at all when something is working".
It saddens me when I hear people ruing each other with chants of "oooo,
legacy" as it almost always shows a misunderstanding of the other
party's reasoning or a complete misunderstanding of the actual business
requirements of a system rather than just the fact the software or
hardware is more than 2 weeks old.
These days system stability seems to be actively discouraged in a flurry
of changes and tweaks. Worse, often these changes are couched in terms
of those factors but without any clear indication that there is a
problem or the problem is big enough to warrant the disruption caused by
the "fix".
My friend's company once "lost" a server. They could ping it but no one
could remember where it was. Eventually they discovered that it had been
plastered up behind a partition during an office refurbishment a few
years previously.
Regards,
@ndy
--
andyjpb(a)ashurst.eu.org
http://www.ashurst.eu.org/
0x7EBA75FF
What? Oh how
things have changed. Time was you'd expect to go three or four years between reboots.
Perhaps on systems in completely controlled / closed networks that might have been true,
but in the 18 years I've been administering systems on the internet there have always
been times when it's been occasionally necessary to reboot in order to apply security-
or stability-related bugfixes. And that's not even taking into account downtime due
to failures or maintenance with facilities. (Server hosting facilities didn't used
to be as robust as they are now.)
I don't mean to start an argument, but I'm always suspicious of assertions that
things in the past were better than they really are. :) Progress is real.
9:19 a.m.
On 19/04/10 10:00, Andy Bennett wrote:
These days system stability seems to be actively
discouraged in a
flurry of changes and tweaks. Worse, often these changes are couched
in terms of those factors but without any clear indication that there
is a problem or the problem is big enough to warrant the disruption
caused by the "fix".
My friend's company once "lost" a server. They could ping it but no
one could remember where it was. Eventually they discovered that it
had been plastered up behind a partition during an office
refurbishment a few years previously.
Surely this is an anecdote which would make any good sysadmin recoil in
horror?!
Imagine running important services on such a server and it blowing a
power supply. I wouldn't want to be the person to explain to the MD of
the company that not only was an important service not running, that I
couldn't even locate the machine it was supposed to be running on. Inept
doesn't even begin to describe it.
(It sounds apocryphal to me anyway, but it would certainly be a good
example of 'worst IT practice' I could think of)
Cheers
Alex.
--
This message was scanned by Better Hosted and is believed to be clean.
http://www.betterhosted.com
9:43 a.m.
I don't usually jump in on these things, but this thread is starting to get
away from the point.
I've head that story before talking about Netware servers, sounds
"embelished" to me.
As a security paranoid sysadmin person who has run small Internet facing
Linux / BSD systems for years, including when working at an ISP some years
back, here is my two-penneth
1) FSCK - I don't do enough serious admin to run it manually - I wouldn't be
keen on it being removed from boot unless replaced with something else.
2) Messing with my server config without my knowledge - please don't -
that's not why I pay for a VPS for - otherwise I'd just get hosting.
3) Uptime - On a server with 2 or 3 visible services - I see no reason why a
year + uptime is not realistic. Most updates to Sendmail, Mailman, Apache,
PHP etc can be achieved without a reboot
4) My biggest gripe with Bitfolk is the number of crashes / reboots etc -
FAR more than my previous VPS provider
Even when I ran an old Pentium box co-located at a friends company it was up
for over a year, and that was on standard desktop hardware.
Seldom do systems at this sort of scale desperately need a kernel update in
my experience - we're not plugging in weird hardward devices that need
support, all drivers are pretty much static. There was a serious kernel
priv-esc a year or so back, but that was about it in my experience. If
something is working, stable and sufficiently secure, what good sysadmin
would mess about with it on the off chance of making it slightly better.
Stuart
On 19 April 2010 10:19, Alex Hudson <home(a)alexhudson.com> wrote:
On 19/04/10 10:00, Andy Bennett wrote:
These days system stability seems to be actively
discouraged in a flurry
of changes and tweaks. Worse, often these changes are couched in terms of
those factors but without any clear indication that there is a problem or
the problem is big enough to warrant the disruption caused by the "fix".
My friend's company once "lost" a server. They could ping it but no one
could remember where it was. Eventually they discovered that it had been
plastered up behind a partition during an office refurbishment a few years
previously.
Surely this is an anecdote which would make any good sysadmin recoil in
horror?!
Imagine running important services on such a server and it blowing a power
supply. I wouldn't want to be the person to explain to the MD of the company
that not only was an important service not running, that I couldn't even
locate the machine it was supposed to be running on. Inept doesn't even
begin to describe it.
(It sounds apocryphal to me anyway, but it would certainly be a good
example of 'worst IT practice' I could think of)
Cheers
Alex.
--
This message was scanned by Better Hosted and is believed to be clean.
http://www.betterhosted.com
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
10:08 a.m.
Hi Stuart,
On Mon, Apr 19, 2010 at 10:43:46AM +0100, Stuart Criddle wrote:
2) Messing with my server config without my knowledge
- please don't -
that's not why I pay for a VPS for - otherwise I'd just get hosting.
OK. You've clearly stated that you wouldn't want time-based fsck
turned off for yourself, so I won't consider doing it on your block
device(s).
Imagine you're not on this mailing list though, so haven't had the
opportunity to find out about this. If I were to disable the fsck
and contact you to say what has been done, and how to re-enable it
if desired, would that be an issue for you in those circumstances?
4) My biggest gripe with Bitfolk is the number of
crashes / reboots etc -
FAR more than my previous VPS provider
Taken to support queue.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
6 May
6 May
2:37 a.m.
Hello,
On Sat, Apr 17, 2010 at 04:41:47AM +0000, Andy Smith wrote:
The time-based fsck can be painful, because when one
of BitFolk's
servers is rebooted a high proportion of the VPSes on it won't have
been rebooted inside this time period (typically 6 months).
Therefore almost every VPS will fsck its filesystems at the same
time, causing massive IO load and a slow boot for everyone.
Having now realised this, I'm considering disabling the time-based
fsck by default.
I've started disabling this in new installs and have written this:
http://bitfolk.com/customer_information.html#toc_3_Why_has_time_based_files…
I've not yet decided whether I will go through and disable it for
existing customers yet, but if I do I will send a direct email
saying what, why and how to undo.
Cheers,
Andy
--
http://bitfolk.com/ -- No-nonsense VPS hosting
5115
days inactive
5134
days old
13 comments
6 participants
participants (6)
-
Alex Hudson -
Andy Bennett -
Andy Smith -
Graham Freeman -
Mathew Newton -
Stuart Criddle