21 Nov
2019
21 Nov
'19
11:02 p.m.
Hi,
Briefly:
At around 2153Z today we started receiving alerts for basically
"everything" flapping up and down due to between 20 and 30% packet
loss. In short, this was due to a distributed denial of service
attack aimed at another customer of our colo provider. Around 2240Z
they deployed some mitigation and hopefully we don't see any more
issues.
TL;DR:
I'd only got as far as determining that it affected "everything"
before it stopped again shortly before 2200. It then started up
again around 2215 and I was able to see that it also affected our
colo provider.
I was able to make contact and they started investigating around
2215. There wasn't anything for me to do at this point except watch.
It was a large UDP DDoS, random source and destination ports.
At about 2240 they put in some mitigation. The degree to which you
were affected during this time will vary based on how your legit
traffic reached us (or didn't reach us).
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
21 Nov
21 Nov
11:16 p.m.
On Thu, Nov 21, 2019 at 11:02:05PM +0000, Andy Smith wrote:
At around 2153Z today we started receiving alerts for
basically
"everything" flapping up and down due to between 20 and 30% packet
loss. [...]
Thanks for the explanation. I was definitely getting very slow responses to
everything around that time and wondered if it was something like that.
--
Anahata
anahata(a)treewind.co.uk -+- http://www.treewind.co.uk
Home: 01535 501017 Mob: 07976 263827
11:39 p.m.
Hello,
On Thu, Nov 21, 2019 at 11:22:14PM +0000, Dom Latter wrote:
Did I notice?
No.
:)
I know you're not looking for an explanation but in terms of who was
affected and to what degree, it really will vary quite a lot here.
BitFolk's servers are split between two different racks at the colo
provider, with a pair of switches in each rack. One of those pairs
of switches also hosts the target of this attack, so that pair was
affected a bit worse. That covers servers:
hen
hobgoblin
leffe
macallan
paradox
In addition, some of the colo provider's private peering comes in
there, so that was affected quite badly. That covers my home for
example, since peering with Andrews&Arnold is there. So for me from
my home it looked like everything was down.
Some of their transit was affected, to a degree I do not know yet,
but LINX peering was probably unaffected.
So customers on servers on the other pair of switches, which are:
elephant
limoncello
snaps
talisker
…may not have seen any issue at all, depending on the route their
legit traffic was taking.
On top of that, our internal monitoring node is on a server on the
target switch pair, so from its point of view everything was also
flapping up and down. So even if you did get alerts from us, they
were not necessarily reflective of full reality.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
11:43 p.m.
In addition, some of the colo provider's private
peering comes in
there, so that was affected quite badly. That covers my home for
example, since peering with Andrews&Arnold is there. So for me from
my home it looked like everything was down.
Same here, also on Andrews&Arnold.
Funny actually, IMHO both Bitfolk and Andrews&Arnold are super reliable
providers, so I naturally expected my network cable at home to be the
culprit ;)
Good to hear an alternative explanation. Thank you!
Conrad
22 Nov
22 Nov
midnight
On Thu, 21 Nov 2019 at 23:43, Conrad Wood <cnw(a)conradwood.net> wrote:
Another supremely content customer of both A&A and Bitfolk here ;-)
Something tells me the subscribers to this list tend to have excellent
taste in IT service providers!
In addition,
some of the colo provider's private peering comes in
there, so that was affected quite badly. That covers my home for
example, since peering with Andrews&Arnold is there. So for me from
my home it looked like everything was down.
Same here, also on Andrews&Arnold.
Funny actually, IMHO both Bitfolk and Andrews&Arnold are super reliable
providers, so I naturally expected my network cable at home to be the
culprit ;)
Good to hear an alternative explanation. Thank you!
Conrad 
6:38 a.m.
On Fri, 22 Nov 2019, at 00:00, Adam Spiers wrote:
Another supremely content customer of both A&A and
Bitfolk here ;-)
Something tells me the subscribers to this list tend to have excellent
taste in IT service providers!
Yet another A&A and Bitfolk customer here but, boy - did I ever pick the wrong time to
upgrade from Stretch to Buster!
I spent large chunks of the that hour watching nothing happen, trying to figure out why
top was telling me everything was good but performance was decidedly not. All worked out
in the end, though.
Cheers,
jmi
--
Jamie MacIsaac
jamie(a)macisa.ac
2:15 p.m.
I have been considering moving to A&A for a bit after a recommendation or 3.
The topic so far reinforces that.
Cheers
Kirbs
On 22/11/2019 06:38, Jamie MacIsaac wrote:
On Fri, 22 Nov 2019, at 00:00, Adam Spiers wrote:
Another supremely content customer of both
A&A and Bitfolk here ;-)
Something tells me the subscribers to this list tend to have excellent
taste in IT service providers!
Yet another A&A and Bitfolk customer here but,
boy - did I ever pick the wrong time to upgrade from Stretch to Buster!
I spent large chunks of the that hour watching nothing happen, trying to figure out why
top was telling me everything was good but performance was decidedly not. All worked out
in the end, though.
Cheers,
jmi
23 Nov
23 Nov
12:29 a.m.
Hmm I am beginning to see an emerging pattern to the naming of
servers/services here.
It is not a displeasing one.....
LOL
Kirbs
On 21/11/2019 23:39, Andy Smith wrote:
Hello,
On Thu, Nov 21, 2019 at 11:22:14PM +0000, Dom Latter wrote:
--
admins(a)sheffieldhackspace.org.uk
www.sheffieldhackspace.org.uk
Did I notice?
No.
:)
I know you're not looking for an explanation but in terms of who was
affected and to what degree, it really will vary quite a lot here.
BitFolk's servers are split between two different racks at the colo
provider, with a pair of switches in each rack. One of those pairs
of switches also hosts the target of this attack, so that pair was
affected a bit worse. That covers servers:
hen
hobgoblin
leffe
macallan
paradox
In addition, some of the colo provider's private peering comes in
there, so that was affected quite badly. That covers my home for
example, since peering with Andrews&Arnold is there. So for me from
my home it looked like everything was down.
Some of their transit was affected, to a degree I do not know yet,
but LINX peering was probably unaffected.
So customers on servers on the other pair of switches, which are:
elephant
limoncello
snaps
talisker
…may not have seen any issue at all, depending on the route their
legit traffic was taking.
On top of that, our internal monitoring node is on a server on the
target switch pair, so from its point of view everything was also
flapping up and down. So even if you did get alerts from us, they
were not necessarily reflective of full reality.
Cheers,
Andy
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
12:32 a.m.
BTW shouldn't it be schnapps/schnaps ??
Where is noninio or grappa
Kirbs
On 23/11/2019 00:29, admins wrote:
Hmm I am beginning to see an emerging pattern to the naming of
servers/services here.
It is not a displeasing one.....
LOL
Kirbs
On 21/11/2019 23:39, Andy Smith wrote:
--
admins(a)sheffieldhackspace.org.uk
www.sheffieldhackspace.org.uk
Hello,
On Thu, Nov 21, 2019 at 11:22:14PM +0000, Dom Latter wrote:
--
admins(a)sheffieldhackspace.org.uk
www.sheffieldhackspace.org.uk
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users Did I notice?
No.
:)
I know you're not looking for an explanation but in terms of who was
affected and to what degree, it really will vary quite a lot here.
BitFolk's servers are split between two different racks at the colo
provider, with a pair of switches in each rack. One of those pairs
of switches also hosts the target of this attack, so that pair was
affected a bit worse. That covers servers:
hen
hobgoblin
leffe
macallan
paradox
In addition, some of the colo provider's private peering comes in
there, so that was affected quite badly. That covers my home for
example, since peering with Andrews&Arnold is there. So for me from
my home it looked like everything was down.
Some of their transit was affected, to a degree I do not know yet,
but LINX peering was probably unaffected.
So customers on servers on the other pair of switches, which are:
elephant
limoncello
snaps
talisker
…may not have seen any issue at all, depending on the route their
legit traffic was taking.
On top of that, our internal monitoring node is on a server on the
target switch pair, so from its point of view everything was also
flapping up and down. So even if you did get alerts from us, they
were not necessarily reflective of full reality.
Cheers,
Andy
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
8:29 a.m.
Hi Kirbs,
On Sat, Nov 23, 2019 at 12:32:22AM +0000, admins wrote:
BTW shouldn't it be schnapps/schnaps ??
Many of them are specific product names. There's a link here from
the name to what it actually is.
https://tools.bitfolk.com/wiki/BitFolk_server_naming#Naming_leader_board
Where is noninio or grappa
grappa is already in the list of suggestions:
https://tools.bitfolk.com/wiki/BitFolk_server_naming#Current_naming_contests
please feel free to add more…
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
24 Nov
24 Nov
10:32 a.m.
I think Grappa is a touch generic then, where as Nonino is a specific
Grappa product.
Hmm will have a think about some more....
Research is needed, LOL
Kirbs
On 23/11/2019 08:29, Andy Smith wrote:
Hi Kirbs,
On Sat, Nov 23, 2019 at 12:32:22AM +0000, admins wrote:
--
admins(a)sheffieldhackspace.org.uk
www.sheffieldhackspace.org.uk
BTW shouldn't it be schnapps/schnaps ??
Many of them are specific product names. There's a link here from
the name to what it actually is.
https://tools.bitfolk.com/wiki/BitFolk_server_naming#Naming_leader_board
Where is noninio or grappa
grappa is
already in the list of suggestions:
https://tools.bitfolk.com/wiki/BitFolk_server_naming#Current_naming_contests
please feel free to add more…
Cheers,
Andy
_______________________________________________
users mailing list
users(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/users
1613
days inactive
1616
days old
11 comments
7 participants
participants (7)
-
Adam Spiers -
admins -
Anahata -
Andy Smith -
Conrad Wood -
Dom Latter -
Jamie MacIsaac