1 Nov
2019
1 Nov
'19
1:01 a.m.
Hello,
I've been noticing SYN flooding¹ on various TCP services going on for
weeks now, not just against BitFolk hosts but against all hosts I
have access to, worldwide. Others have noticed it too.
Today I also noticed that some customer DNS and HTTP servers were
occasionally taking a long time (10+ seconds) to respond and this was
generating an alert from our monitoring, which would then clear, and
then trigger again.
I've done a tcpdump against two such customer services so far and I
see stuff like this:
$ sudo tcpdump -vpni v-[redacted] 'tcp and dst port 53'
00:53:33.622739 IP (tos 0x8, ttl 79, id 47798, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.150.52945 > 85.119.[redacted].53: Flags [S], cksum 0x2dca (correct), seq
3396968370, win 29200, length 0
00:53:34.451820 IP (tos 0x8, ttl 75, id 64961, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.137.42540 > 85.119.[redacted].53: Flags [S], cksum 0x74cf (correct), seq
1691936512, win 29200, length 0
00:53:34.636166 IP (tos 0x0, ttl 62, id 38397, offset 0, flags [DF], proto TCP (6), length
60)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [S], cksum 0x144a (correct), seq
3351471429, win 29200, options [mss 1460,sackOK,TS val 2923360440 ecr 0,nop,wscale 7],
length 0
00:53:34.792156 IP (tos 0x8, ttl 55, id 51802, offset 0, flags [DF], proto TCP (6), length
40)
185.90.118.36.58121 > 85.119.[redacted].53: Flags [S], cksum 0x5b25 (correct), seq
3802350695, win 29200, length 0
00:53:35.659295 IP (tos 0x0, ttl 62, id 38398, offset 0, flags [DF], proto TCP (6), length
60)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [S], cksum 0x134a (correct), seq
3351471429, win 29200, options [mss 1460,sackOK,TS val 2923360696 ecr 0,nop,wscale 7],
length 0
00:53:37.247124 IP (tos 0x8, ttl 68, id 41159, offset 0, flags [DF], proto TCP (6), length
40)
185.90.117.12.41829 > 85.119.[redacted].53: Flags [S], cksum 0xf1bb (correct), seq
805085492, win 29200, length 0
00:53:37.675366 IP (tos 0x0, ttl 62, id 38399, offset 0, flags [DF], proto TCP (6), length
60)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [S], cksum 0x1152 (correct), seq
3351471429, win 29200, options [mss 1460,sackOK,TS val 2923361200 ecr 0,nop,wscale 7],
length 0
00:53:37.756095 IP (tos 0x8, ttl 78, id 5747, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.142.65402 > 85.119.[redacted].53: Flags [S], cksum 0xf3b5 (correct), seq
2604980314, win 29200, length 0
00:53:37.760805 IP (tos 0x8, ttl 73, id 5112, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.61.62352 > 85.119.[redacted].53: Flags [S], cksum 0x5be0 (correct), seq
636416449, win 29200, length 0
00:53:37.860091 IP (tos 0x8, ttl 55, id 53714, offset 0, flags [DF], proto TCP (6), length
40)
185.90.118.42.47449 > 85.119.[redacted].53: Flags [S], cksum 0x3f4d (correct), seq
2136599859, win 29200, length 0
00:53:39.582555 IP (tos 0x8, ttl 73, id 11769, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.251.54065 > 85.119.[redacted].53: Flags [S], cksum 0xfd05 (correct), seq
2936203048, win 29200, length 0
00:53:40.357845 IP (tos 0x8, ttl 66, id 24626, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.143.49296 > 85.119.[redacted].53: Flags [S], cksum 0x6e13 (correct), seq
3886043274, win 29200, length 0
00:53:40.369750 IP (tos 0x8, ttl 62, id 39030, offset 0, flags [DF], proto TCP (6), length
40)
185.90.118.213.43688 > 85.119.[redacted].53: Flags [S], cksum 0x2a92 (correct), seq
231047561, win 29200, length 0
00:53:41.477175 IP (tos 0x8, ttl 53, id 47222, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.92.32796 > 85.119.[redacted].53: Flags [S], cksum 0x2417 (correct), seq
2911442245, win 29200, length 0
00:53:41.562028 IP (tos 0x8, ttl 54, id 46361, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.136.63986 > 85.119.[redacted].53: Flags [S], cksum 0x6088 (correct), seq
3811125553, win 29200, length 0
00:53:41.839400 IP (tos 0x0, ttl 62, id 38400, offset 0, flags [DF], proto TCP (6), length
60)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [S], cksum 0x0d42 (correct), seq
3351471429, win 29200, options [mss 1460,sackOK,TS val 2923362240 ecr 0,nop,wscale 7],
length 0
00:53:41.839884 IP (tos 0x0, ttl 62, id 38401, offset 0, flags [DF], proto TCP (6), length
52)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [.], cksum 0x1d5b (correct), ack
3745901387, win 229, options [nop,nop,TS val 2923362241 ecr 41403076], length 0
00:53:41.839934 IP (tos 0x0, ttl 62, id 38402, offset 0, flags [DF], proto TCP (6), length
52)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [F.], cksum 0x1d5a (correct), seq
0, ack 1, win 229, options [nop,nop,TS val 2923362241 ecr 41403076], length 0
00:53:41.842751 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto TCP (6), length
52)
85.119.80.238.42534 > 85.119.[redacted].53: Flags [.], cksum 0x1d58 (correct), ack
2, win 229, options [nop,nop,TS val 2923362241 ecr 41403077], length 0
00:53:41.938989 IP (tos 0x8, ttl 56, id 57551, offset 0, flags [DF], proto TCP (6), length
40)
185.90.118.99.40999 > 85.119.[redacted].53: Flags [S], cksum 0xeefb (correct), seq
137547173, win 29200, length 0
00:53:42.234411 IP (tos 0x8, ttl 60, id 59687, offset 0, flags [DF], proto TCP (6), length
40)
185.90.117.200.49353 > 85.119.[redacted].53: Flags [S], cksum 0x875f (correct), seq
216731778, win 29200, length 0
00:53:43.192600 IP (tos 0x8, ttl 79, id 8563, offset 0, flags [DF], proto TCP (6), length
40)
185.90.116.208.47026 > 85.119.[redacted].53: Flags [S], cksum 0xd35e (correct), seq
1815703363, win 29200, length 0
00:53:43.355109 IP (tos 0x8, ttl 80, id 37254, offset 0, flags [DF], proto TCP (6), length
40)
185.90.118.237.43294 > 85.119.[redacted].53: Flags [S], cksum 0xd9af (correct), seq
848473872, win 29200, length 0
So I'm seeing a lot of TCP packets with the SYN flag set ("[S]")
coming from hosts all over 185.90.116.0/22, but they never actually
establish a connection and exchange data. I think that's a SYN
flood.
If you are seeing flapping alerts for your TCP-based services, can
you have a look to see if same is happening to you?
If using tcpdump you'd want something like:
# tcpdump -vpni eth0 'tcp and dst port 53'
(For looking at TCP traffic to your DNS server)
The source IPs are probably going to change rapidly, so I'm not sure
what configuration if any is needed in the OS or DNS server to
prevent this from starving out legit connections, e.g. the ones from
our monitoring!
Against BitFolk's own infrastructure I do see this too, but I'm
maybe not using the same software as you because it's not starving
out legit connections. So if you are affected by this and you do
find a way to block this or increase your limits or whatever, I'd
appreciate you letting me know what you're running and what you
needed to do.
Cheers,
Andy
¹ https://en.wikipedia.org/wiki/SYN_flood
--
https://bitfolk.com/ -- No-nonsense VPS hosting