Hi,
Briefly:
At around 2153Z today we started receiving alerts for basically
"everything" flapping up and down due to between 20 and 30% packet
loss. In short, this was due to a distributed denial of service
attack aimed at another customer of our colo provider. Around 2240Z
they deployed some mitigation and hopefully we don't see any more
issues.
TL;DR:
I'd only got as far as determining that it affected "everything"
before it stopped again shortly before 2200. It then started up
again around 2215 and I was able to see that it also affected our
colo provider.
I was able to make contact and they started investigating around
2215. There wasn't anything for me to do at this point except watch.
It was a large UDP DDoS, random source and destination ports.
At about 2240 they put in some mitigation. The degree to which you
were affected during this time will vary based on how your legit
traffic reached us (or didn't reach us).
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
Hi All,
It has come to my attention - thanks Andy - that my IP6 configuration
may be wrong. My VPS is receiving IP6 traffic, but sending none, and is
(perhaps sometimes) unable to route to an IP6 address.
This is way beyond my pay grade, so I need some help here.
When I look in my cpanel, it says my IP6 address is
2001:ba8:1f1:f00d::/64
But ip a gives ....
ian@hobsoni:~$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP
group default qlen 1000
link/ether 00:16:5e:00:04:89 brd ff:ff:ff:ff:ff:ff
inet 85.119.82.210/21 brd 85.119.87.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 2001:ba8:1f1:f06e::2/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::216:5eff:fe00:489/64 scope link
valid_lft forever preferred_lft forever
ian@hobsoni:~$ ip -6 route show
2001:ba8:1f1:f06e::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via 2001:ba8:1f1:f06e::1 dev eth0 metric 1024 pref medium
ian@hobsoni:~$ ip -6 address show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 state UNKNOWN qlen 1000
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 state UP qlen 1000
inet6 2001:ba8:1f1:f06e::2/64 scope global
valid_lft forever preferred_lft forever
inet6 fe80::216:5eff:fe00:489/64 scope link
valid_lft forever preferred_lft forever
ian@hobsoni:~$
And here is the netplan config
ian@hobsoni:/etc/netplan$ cat 01-netcfg.yaml
network:
version: 2
renderer: networkd
ethernets:
eth0:
addresses:
- "85.119.82.210/21"
- "2001:ba8:1f1:f00d::2/64"
gateway4: 85.119.80.1
gateway6: "2001:ba8:1f1:f00d::1"
nameservers:
addresses: [85.119.80.232 85.119.80.233]
ian@hobsoni:/etc/netplan$
I'm guessing that f06e is not f00d for netplan and its barfing on it.
(Sorry).
But I don't know which is correct.
Regards
Ian
--
Ian Hobson
Tel (+351) 910 418 473
Hi All,
A website on my VPS has been receiving HEAD requests,
which cause it to fail, and are cluttering up my error logs. They are
coming from IPs all over the place, but only a few (3 to 5) a day.
What would generate them?
How should the software respond?
Regards
Ian
--
Ian Hobson
Tel (+351) 910 418 473
Hi All,
I have had a user complain they got no response from a website on my
VPN. It appears that from about 9:07 until after 9:44 my front-end
(nginx) could not talk to the back-end (php). It cleared before I became
aware of the problem.
I can find nothing in the logs, other than the failed reads, so I am at
a loss to diagnose things.
Environment - The O/S is Ubuntu 18.04 LTS.
nginx is version 1.16.1 compiled to include a couple of extra
modules...standard stuff plus...
--with-pcre=../pcre-8.42 \
--with-zlib=../zlib-1.2.11 \
--add-module=../nchan-1.2.6
nginx talks to php7.2-fpm using fastcgi-pass to 127.0.0.1:9000.
Any ideas gratefully received!
Regards
Ian
--
Ian Hobson
--
This email has been checked for viruses by AVG.
https://www.avg.com
On 5 Nov 2019, at 15:13, Andy Smith <andy(a)bitfolk.com> wrote:
>
> On Tue, Nov 05, 2019 at 03:02:39PM +0000, Chris Smith wrote:
>> I’m also a mild -1 for much the same reasons already mentioned, but out of interest do you know why it doesn’t work great for those people? I can understand that people might think a mailing list a bit old fashioned, but I’m a bit baffled at how someone could try to use one and fail in the attempt.
>
> Well, did you intend to send this to me alone? If not, that's one
> example of how someone can fail. Perhaps you did intend to send it
> only to me, but 4 different people this past week have done that and
> did not intend to send it just to me, so that's one thing.
Ha! What a brilliant example. I only wish it was deliberate. Yes, I did intend to reply to the list, and I’ve corrected that now so that everyone else can have a quick giggle at my expense.
> Most of the replies I see on the list are more like forum responses
> not emails. I find them hard to read as emails. The top posting,
> terribly broken quoting. I'm seeing less and less point in expecting
> clear emails from people who don't see value in sending clear emails.
I’m with you on that. My mail client has no option to turn off top posting so I have to go to the effort of formatting manually, and I only go to that effort for technical mailing lists because those are the only people who care.
> I have used email since 1994, and FidoNet for a couple of years
> prior to that. But I wonder how far the tide is turning.
Sadly I think the only thing keeping email alive is the fact that almost every non-email platform requires an email address for user identification and authentication. There’s a big shift now for linking to Facebook and Google etc. for doing that, but those are obviously very specific and platform dependent mechanisms, and even those require an email address. Sooner or later there will be a common, platform-agnostic mechanism that will become dominant, and at that point I suspect email die.
Chris
—
Chris Smith <space.dandy(a)icloud.com>
Hi,
I've added a warning to the Xen Shell "install" command that new
installs of 32-bit guests are now deprecated, and pointing you to:
https://tools.bitfolk.com/wiki/64-bit_guests
Those of you with existing 32-bit guests, which I know is many of
you, should be okay for a few years yet. Even after the Xen
hypervisor officially ditches support for 32-bit guests we will
ensure they are still bootable. But I would encourage you to choose
64-bit at your next install or operating system upgrade for an
easier life.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
_______________________________________________
announce mailing list
announce(a)lists.bitfolk.com
https://lists.bitfolk.com/mailman/listinfo/announce
Hello,
I've been noticing SYN flooding¹ on various TCP services going on for
weeks now, not just against BitFolk hosts but against all hosts I
have access to, worldwide. Others have noticed it too.
Today I also noticed that some customer DNS and HTTP servers were
occasionally taking a long time (10+ seconds) to respond and this was
generating an alert from our monitoring, which would then clear, and
then trigger again.
I've done a tcpdump against two such customer services so far and I
see stuff like this:
$ sudo tcpdump -vpni v-[redacted] 'tcp and dst port 53'
00:53:33.622739 IP (tos 0x8, ttl 79, id 47798, offset 0, flags [DF], proto TCP (6), length 40)
185.90.116.150.52945 > 85.119.[redacted].53: Flags [S], cksum 0x2dca (correct), seq 3396968370, win 29200, length 0
00:53:34.451820 IP (tos 0x8, ttl 75, id 64961, offset 0, flags [DF], proto TCP (6), length 40)
185.90.116.137.42540 > 85.119.[redacted].53: Flags [S], cksum 0x74cf (correct), seq 1691936512, win 29200, length 0
00:53:34.636166 IP (tos 0x0, ttl 62, id 38397, offset 0, flags [DF], proto TCP (6), length 60)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [S], cksum 0x144a (correct), seq 3351471429, win 29200, options [mss 1460,sackOK,TS val 2923360440 ecr 0,nop,wscale 7],
length 0
00:53:34.792156 IP (tos 0x8, ttl 55, id 51802, offset 0, flags [DF], proto TCP (6), length 40)
185.90.118.36.58121 > 85.119.[redacted].53: Flags [S], cksum 0x5b25 (correct), seq 3802350695, win 29200, length 0
00:53:35.659295 IP (tos 0x0, ttl 62, id 38398, offset 0, flags [DF], proto TCP (6), length 60)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [S], cksum 0x134a (correct), seq 3351471429, win 29200, options [mss 1460,sackOK,TS val 2923360696 ecr 0,nop,wscale 7],
length 0
00:53:37.247124 IP (tos 0x8, ttl 68, id 41159, offset 0, flags [DF], proto TCP (6), length 40)
185.90.117.12.41829 > 85.119.[redacted].53: Flags [S], cksum 0xf1bb (correct), seq 805085492, win 29200, length 0
00:53:37.675366 IP (tos 0x0, ttl 62, id 38399, offset 0, flags [DF], proto TCP (6), length 60)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [S], cksum 0x1152 (correct), seq 3351471429, win 29200, options [mss 1460,sackOK,TS val 2923361200 ecr 0,nop,wscale 7],
length 0
00:53:37.756095 IP (tos 0x8, ttl 78, id 5747, offset 0, flags [DF], proto TCP (6), length 40)
185.90.116.142.65402 > 85.119.[redacted].53: Flags [S], cksum 0xf3b5 (correct), seq 2604980314, win 29200, length 0
00:53:37.760805 IP (tos 0x8, ttl 73, id 5112, offset 0, flags [DF], proto TCP (6), length 40)
185.90.116.61.62352 > 85.119.[redacted].53: Flags [S], cksum 0x5be0 (correct), seq 636416449, win 29200, length 0
00:53:37.860091 IP (tos 0x8, ttl 55, id 53714, offset 0, flags [DF], proto TCP (6), length 40)
185.90.118.42.47449 > 85.119.[redacted].53: Flags [S], cksum 0x3f4d (correct), seq 2136599859, win 29200, length 0
00:53:39.582555 IP (tos 0x8, ttl 73, id 11769, offset 0, flags [DF], proto TCP (6), length 40)
185.90.116.251.54065 > 85.119.[redacted].53: Flags [S], cksum 0xfd05 (correct), seq 2936203048, win 29200, length 0
00:53:40.357845 IP (tos 0x8, ttl 66, id 24626, offset 0, flags [DF], proto TCP (6), length 40)
185.90.116.143.49296 > 85.119.[redacted].53: Flags [S], cksum 0x6e13 (correct), seq 3886043274, win 29200, length 0
00:53:40.369750 IP (tos 0x8, ttl 62, id 39030, offset 0, flags [DF], proto TCP (6), length 40)
185.90.118.213.43688 > 85.119.[redacted].53: Flags [S], cksum 0x2a92 (correct), seq 231047561, win 29200, length 0
00:53:41.477175 IP (tos 0x8, ttl 53, id 47222, offset 0, flags [DF], proto TCP (6), length 40)
185.90.116.92.32796 > 85.119.[redacted].53: Flags [S], cksum 0x2417 (correct), seq 2911442245, win 29200, length 0
00:53:41.562028 IP (tos 0x8, ttl 54, id 46361, offset 0, flags [DF], proto TCP (6), length 40)
185.90.116.136.63986 > 85.119.[redacted].53: Flags [S], cksum 0x6088 (correct), seq 3811125553, win 29200, length 0
00:53:41.839400 IP (tos 0x0, ttl 62, id 38400, offset 0, flags [DF], proto TCP (6), length 60)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [S], cksum 0x0d42 (correct), seq 3351471429, win 29200, options [mss 1460,sackOK,TS val 2923362240 ecr 0,nop,wscale 7],
length 0
00:53:41.839884 IP (tos 0x0, ttl 62, id 38401, offset 0, flags [DF], proto TCP (6), length 52)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [.], cksum 0x1d5b (correct), ack 3745901387, win 229, options [nop,nop,TS val 2923362241 ecr 41403076], length 0
00:53:41.839934 IP (tos 0x0, ttl 62, id 38402, offset 0, flags [DF], proto TCP (6), length 52)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [F.], cksum 0x1d5a (correct), seq 0, ack 1, win 229, options [nop,nop,TS val 2923362241 ecr 41403076], length 0
00:53:41.842751 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto TCP (6), length 52)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [.], cksum 0x1d58 (correct), ack 2, win 229, options [nop,nop,TS val 2923362241 ecr 41403077], length 0
00:53:41.938989 IP (tos 0x8, ttl 56, id 57551, offset 0, flags [DF], proto TCP (6), length 40)
185.90.118.99.40999 > 85.119.[redacted].53: Flags [S], cksum 0xeefb (correct), seq 137547173, win 29200, length 0
00:53:42.234411 IP (tos 0x8, ttl 60, id 59687, offset 0, flags [DF], proto TCP (6), length 40)
185.90.117.200.49353 > 85.119.[redacted].53: Flags [S], cksum 0x875f (correct), seq 216731778, win 29200, length 0
00:53:43.192600 IP (tos 0x8, ttl 79, id 8563, offset 0, flags [DF], proto TCP (6), length 40)
185.90.116.208.47026 > 85.119.[redacted].53: Flags [S], cksum 0xd35e (correct), seq 1815703363, win 29200, length 0
00:53:43.355109 IP (tos 0x8, ttl 80, id 37254, offset 0, flags [DF], proto TCP (6), length 40)
185.90.118.237.43294 > 85.119.[redacted].53: Flags [S], cksum 0xd9af (correct), seq 848473872, win 29200, length 0
So I'm seeing a lot of TCP packets with the SYN flag set ("[S]")
coming from hosts all over 185.90.116.0/22, but they never actually
establish a connection and exchange data. I think that's a SYN
flood.
If you are seeing flapping alerts for your TCP-based services, can
you have a look to see if same is happening to you?
If using tcpdump you'd want something like:
# tcpdump -vpni eth0 'tcp and dst port 53'
(For looking at TCP traffic to your DNS server)
The source IPs are probably going to change rapidly, so I'm not sure
what configuration if any is needed in the OS or DNS server to
prevent this from starving out legit connections, e.g. the ones from
our monitoring!
Against BitFolk's own infrastructure I do see this too, but I'm
maybe not using the same software as you because it's not starving
out legit connections. So if you are affected by this and you do
find a way to block this or increase your limits or whatever, I'd
appreciate you letting me know what you're running and what you
needed to do.
Cheers,
Andy
¹ https://en.wikipedia.org/wiki/SYN_flood
--
https://bitfolk.com/ -- No-nonsense VPS hosting
