1 Nov
2019
1 Nov
'19
1:01 a.m.
Hello,
I've been noticing SYN flooding¹ on various TCP services going on for
weeks now, not just against BitFolk hosts but against all hosts I
have access to, worldwide. Others have noticed it too.
Today I also noticed that some customer DNS and HTTP servers were
occasionally taking a long time (10+ seconds) to respond and this was
generating an alert from our monitoring, which would then clear, and
then trigger again.
I've done a tcpdump against two such customer services so far and I
see stuff like this:
$ sudo tcpdump -vpni v-[redacted] 'tcp and dst port 53'
00:53:33.622739 IP (tos 0x8, ttl 79, id 47798, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.150.52945 > 85.119.[redacted].53: Flags [S], cksum 0x2dca (correct), seq
3396968370, win 29200, length 0
00:53:34.451820 IP (tos 0x8, ttl 75, id 64961, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.137.42540 > 85.119.[redacted].53: Flags [S], cksum 0x74cf (correct), seq
1691936512, win 29200, length 0
00:53:34.636166 IP (tos 0x0, ttl 62, id 38397, offset 0, flags [DF], proto TCP (6), length
60)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [S], cksum 0x144a (correct), seq
3351471429, win 29200, options [mss 1460,sackOK,TS val 2923360440 ecr 0,nop,wscale 7],
length 0
00:53:34.792156 IP (tos 0x8, ttl 55, id 51802, offset 0, flags [DF], proto TCP (6), length
40)
185.90.118.36.58121 > 85.119.[redacted].53: Flags [S], cksum 0x5b25 (correct), seq
3802350695, win 29200, length 0
00:53:35.659295 IP (tos 0x0, ttl 62, id 38398, offset 0, flags [DF], proto TCP (6), length
60)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [S], cksum 0x134a (correct), seq
3351471429, win 29200, options [mss 1460,sackOK,TS val 2923360696 ecr 0,nop,wscale 7],
length 0
00:53:37.247124 IP (tos 0x8, ttl 68, id 41159, offset 0, flags [DF], proto TCP (6), length
40)
185.90.117.12.41829 > 85.119.[redacted].53: Flags [S], cksum 0xf1bb (correct), seq
805085492, win 29200, length 0
00:53:37.675366 IP (tos 0x0, ttl 62, id 38399, offset 0, flags [DF], proto TCP (6), length
60)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [S], cksum 0x1152 (correct), seq
3351471429, win 29200, options [mss 1460,sackOK,TS val 2923361200 ecr 0,nop,wscale 7],
length 0
00:53:37.756095 IP (tos 0x8, ttl 78, id 5747, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.142.65402 > 85.119.[redacted].53: Flags [S], cksum 0xf3b5 (correct), seq
2604980314, win 29200, length 0
00:53:37.760805 IP (tos 0x8, ttl 73, id 5112, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.61.62352 > 85.119.[redacted].53: Flags [S], cksum 0x5be0 (correct), seq
636416449, win 29200, length 0
00:53:37.860091 IP (tos 0x8, ttl 55, id 53714, offset 0, flags [DF], proto TCP (6), length
40)
185.90.118.42.47449 > 85.119.[redacted].53: Flags [S], cksum 0x3f4d (correct), seq
2136599859, win 29200, length 0
00:53:39.582555 IP (tos 0x8, ttl 73, id 11769, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.251.54065 > 85.119.[redacted].53: Flags [S], cksum 0xfd05 (correct), seq
2936203048, win 29200, length 0
00:53:40.357845 IP (tos 0x8, ttl 66, id 24626, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.143.49296 > 85.119.[redacted].53: Flags [S], cksum 0x6e13 (correct), seq
3886043274, win 29200, length 0
00:53:40.369750 IP (tos 0x8, ttl 62, id 39030, offset 0, flags [DF], proto TCP (6), length
40)
185.90.118.213.43688 > 85.119.[redacted].53: Flags [S], cksum 0x2a92 (correct), seq
231047561, win 29200, length 0
00:53:41.477175 IP (tos 0x8, ttl 53, id 47222, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.92.32796 > 85.119.[redacted].53: Flags [S], cksum 0x2417 (correct), seq
2911442245, win 29200, length 0
00:53:41.562028 IP (tos 0x8, ttl 54, id 46361, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.136.63986 > 85.119.[redacted].53: Flags [S], cksum 0x6088 (correct), seq
3811125553, win 29200, length 0
00:53:41.839400 IP (tos 0x0, ttl 62, id 38400, offset 0, flags [DF], proto TCP (6), length
60)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [S], cksum 0x0d42 (correct), seq
3351471429, win 29200, options [mss 1460,sackOK,TS val 2923362240 ecr 0,nop,wscale 7],
length 0
00:53:41.839884 IP (tos 0x0, ttl 62, id 38401, offset 0, flags [DF], proto TCP (6), length
52)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [.], cksum 0x1d5b (correct), ack
3745901387, win 229, options [nop,nop,TS val 2923362241 ecr 41403076], length 0
00:53:41.839934 IP (tos 0x0, ttl 62, id 38402, offset 0, flags [DF], proto TCP (6), length
52)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [F.], cksum 0x1d5a (correct), seq
0, ack 1, win 229, options [nop,nop,TS val 2923362241 ecr 41403076], length 0
00:53:41.842751 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto TCP (6), length
52)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [.], cksum 0x1d58 (correct), ack
2, win 229, options [nop,nop,TS val 2923362241 ecr 41403077], length 0
00:53:41.938989 IP (tos 0x8, ttl 56, id 57551, offset 0, flags [DF], proto TCP (6), length
40)
185.90.118.99.40999 > 85.119.[redacted].53: Flags [S], cksum 0xeefb (correct), seq
137547173, win 29200, length 0
00:53:42.234411 IP (tos 0x8, ttl 60, id 59687, offset 0, flags [DF], proto TCP (6), length
40)
185.90.117.200.49353 > 85.119.[redacted].53: Flags [S], cksum 0x875f (correct), seq
216731778, win 29200, length 0
00:53:43.192600 IP (tos 0x8, ttl 79, id 8563, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.208.47026 > 85.119.[redacted].53: Flags [S], cksum 0xd35e (correct), seq
1815703363, win 29200, length 0
00:53:43.355109 IP (tos 0x8, ttl 80, id 37254, offset 0, flags [DF], proto TCP (6), length
40)
185.90.118.237.43294 > 85.119.[redacted].53: Flags [S], cksum 0xd9af (correct), seq
848473872, win 29200, length 0
So I'm seeing a lot of TCP packets with the SYN flag set ("[S]")
coming from hosts all over 185.90.116.0/22, but they never actually
establish a connection and exchange data. I think that's a SYN
flood.
If you are seeing flapping alerts for your TCP-based services, can
you have a look to see if same is happening to you?
If using tcpdump you'd want something like:
# tcpdump -vpni eth0 'tcp and dst port 53'
(For looking at TCP traffic to your DNS server)
The source IPs are probably going to change rapidly, so I'm not sure
what configuration if any is needed in the OS or DNS server to
prevent this from starving out legit connections, e.g. the ones from
our monitoring!
Against BitFolk's own infrastructure I do see this too, but I'm
maybe not using the same software as you because it's not starving
out legit connections. So if you are affected by this and you do
find a way to block this or increase your limits or whatever, I'd
appreciate you letting me know what you're running and what you
needed to do.
Cheers,
Andy
¹ https://en.wikipedia.org/wiki/SYN_flood
--
https://bitfolk.com/ -- No-nonsense VPS hosting
1 Nov
1 Nov
8:09 a.m.
On 2019-11-01 01:01+0000, Andy Smith wrote:
So I'm seeing a lot of TCP packets with the SYN
flag set ("[S]")
coming from hosts all over 185.90.116.0/22, but they never actually
establish a connection and exchange data. I think that's a SYN
flood.
If you are seeing flapping alerts for your TCP-based services, can
you have a look to see if same is happening to you?
If using tcpdump you'd want something like:
# tcpdump -vpni eth0 'tcp and dst port 53'
(For looking at TCP traffic to your DNS server)
One of the counter measures (and thing I didn't suggest during an
interview for an SRE at a large search company) is to enable SYN
cookies, look at drawbacks though. This turns the three-way handshake
into four way, and doesn't steal space from the state table until the
third state of the now four-way handshake.
https://en.wikipedia.org/wiki/SYN_cookies
--
Best regards,
Ed http://www.s5h.net/
2:08 p.m.
Hello,
On Fri, Nov 01, 2019 at 08:09:49AM +0000, ed-bitfolk(a)s5h.net wrote:
One of the counter measures (and thing I didn't
suggest during an
interview for an SRE at a large search company) is to enable SYN
cookies, look at drawbacks though. This turns the three-way handshake
into four way, and doesn't steal space from the state table until the
third state of the now four-way handshake.
https://en.wikipedia.org/wiki/SYN_cookies
I do actually use SYN cookies myself so this may explain why I don't
see this for any of BitFolk's own stuff.
Those of you who are seeing alerts, do you have SYN cookies enabled
and if not, does enabling them stop it happening?
Of course nothing will stop the actual spoofed SYN packets, but they
aren't very high rate so as long as your SYN queue is not filling up
you probably don't need to care.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
2:40 p.m.
On Fri, Nov 01, 2019 at 02:08:24PM +0000, Andy Smith wrote:
Those of you who are seeing alerts, do you have SYN
cookies enabled
and if not, does enabling them stop it happening?
In a support ticket I had a confirmation that they did not have SYN
cookies enabled, and enabling them made the problem [of the SYN queue
filling] go away.
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
3 Nov
3 Nov
7 p.m.
On 01/11/2019 14:08, Andy Smith wrote:
On Fri, Nov 01, 2019 at 08:09:49AM +0000,
ed-bitfolk(a)s5h.net wrote:
This is appearing on my console:
TCP: request_sock_TCP: Possible SYN flooding on port 25. Sending
cookies. Check SNMP counters.
And SYN cookies are enabled on my server.
Gavin
One of the counter measures (and thing I
didn't suggest during an
interview for an SRE at a large search company) is to enable SYN
cookies, look at drawbacks though. This turns the three-way handshake
into four way, and doesn't steal space from the state table until the
third state of the now four-way handshake.
https://en.wikipedia.org/wiki/SYN_cookies
I do actually use SYN cookies myself
so this may explain why I don't
see this for any of BitFolk's own stuff.
Those of you who are seeing alerts, do you have SYN cookies enabled
and if not, does enabling them stop it happening?
8:20 p.m.
Hello,
On Sun, Nov 03, 2019 at 07:00:54PM +0000, Gavin Westwood wrote:
On 01/11/2019 14:08, Andy Smith wrote:
But I am not seeing alerts for your DNS server, so they are
probably doing their job.
If your SYN queue was being filled no new connections to your DNS
server would be possible. So this is more evidence that what is
being seen is a SYN flood and SYN cookies do help.
Another customer is seeing flapping alerts for Apache httpd. SYN
cookies did not help there, but I do not know the exact nature of
the problem there (eve if they are seeing a SYN flood).
Cheers,
Andy
--
https://bitfolk.com/ -- No-nonsense VPS hosting
Those of you who are seeing alerts, do you have
SYN cookies enabled
and if not, does enabling them stop it happening?
This is appearing on my console:
TCP: request_sock_TCP: Possible SYN flooding on port 25. Sending
cookies. Check SNMP counters.
And SYN cookies are enabled on my server. 
1 Nov
1 Nov
8:15 a.m.
On 1 Nov 2019, at 01:01, Andy Smith <andy(a)bitfolk.com> wrote:
I've been noticing SYN flooding¹ on various TCP
services going on for
weeks now, not just against BitFolk hosts but against all hosts I
have access to, worldwide. Others have noticed it too.
Today I also noticed that some customer DNS and HTTP servers were
occasionally taking a long time (10+ seconds) to respond and this was
generating an alert from our monitoring, which would then clear, and
then trigger again.
I have three servers (one at BitFolk) with ports 80 and 443 open and I don’t see any SYN
flooding directed at me but all three servers seem to be taking part in SYN flooding
against others. This has been going on for months and the traffic is low-level.
All three servers are sending SYN ACKs to addresses in 185.90.0.0/16 at the moment but it
often changes.
Mike
8:47 a.m.
On 2019-11-01 08:15+0000, Mike Zanker wrote:
I have three servers (one at BitFolk) with ports 80
and 443 open and I
don’t see any SYN flooding directed at me but all three servers seem
to be taking part in SYN flooding against others. This has been going
on for months and the traffic is low-level.
All three servers are sending SYN ACKs to addresses in 185.90.0.0/16
at the moment but it often changes.
The SYN ACK is a response to a SYN packet. This would signify that the
three machines are responding to SYN requests from 185.90.0.0/16. If no
ACK returns from the SYN ACK, then the initial SYN was probably spoofed.
There's an animation at this address that shows the event sequence:
<https://www.inetdaemon.com/tutorials/internet/tcp/3-way_handshake.shtml>
--
Best regards,
Ed http://www.s5h.net/
9:29 a.m.
On 1 Nov 2019, at 08:47, ed-bitfolk(a)s5h.net wrote:
The SYN ACK is a response to a SYN packet. This would
signify that the
three machines are responding to SYN requests from 185.90.0.0/16. If no
ACK returns from the SYN ACK, then the initial SYN was probably spoofed.
Yes, I’m aware of that. These are definitely spoofed SYNs.
Mike
1629
days inactive
1631
days old
9 comments
4 participants
participants (4)
-
Andy Smith -
ed-bitfolk＠s5h.net -
Gavin Westwood -
Mike Zanker