Awesome response Andy!

One thing I’d add regarding logs - you could create another partition for /var/logs (or some other path and then symlink back into /var/logs), and put "aggressive" size-based log rotation to prevent them spinning out of control. Even if they do, they'll only fill the /var/logs partition and your websites should continue to serve as normal. 

Kind regards,
Paul

+44 (0) 773 996 2121

Sent from my iPhone. Please excuse brevity, spelling, and punctuation. 

From: Andy Smith via BitFolk Users <users@mailman.bitfolk.com>
Sent: Monday, July 25, 2022 2:31:40 AM
To: users@mailman.bitfolk.com <users@mailman.bitfolk.com>
Cc: Andy Smith <andy@bitfolk.com>
Subject: [bitfolk] Re: Troublesome activity
 
Hi Ian,

On Sun, Jul 24, 2022 at 01:05:55PM +0100, Ian Bowden via BitFolk Users wrote:
> My VPS is receiving 250 connections per second from an IP 51.81.86.37.

As this is UDP, you don't actually know that the packets are coming
from 51.81.86.37 - they just say they are. Someone with admin access
to a machine on the Internet can craft packets with any source and
destination addresses in them.

The TCP three way handshake prevents most such spoofing for TCP
protocols, but UDP applications can end up sending replies to things
that never actually talked to them in the first place, so software
needs to be configured with such attacks in mind.

> Sample from syslog:
>
> Jul 24 12:39:52 buddhismwithoutboundaries named[629]: client
> 51.81.86.37#44122 (.): query (cache) './ANY/IN' denied

I'm guessing that you have a nameserver listening on port 53 because
you are authoritative for one or more domains. In that case,
recursive queries should be disabled, so that it doesn't try to
provide answers for any other query than one relating to the
domain(s) it is configured to serve.

You appear to have done that, as these strange queries are being
denied.

If they are trying to do an amplification attack, it is failing;
your bandwidth graphs at:

    https://tools.bitfolk.com/grafana/

show that you're seeing about 170kb/sec inbound traffic and about
the same amount outbound, so it's not amplifying at all. That's
probably because you have correctly configured your nameserver to
refuse those queries. If you had not then it might have been worse,
as you would be sending bigger answers back to 51.81.86.37.

They might not care about amplification. If they forge queries to
look like they came from 51.81.86.37 and send them to a big list of
known nameservers, then all those widely-=spread nameservers will
reply to 51.81.86.37 and it will be in effect some kind of DDoS that
actually need not be launched from multiple places. You are only
sending 170kb/s or so, but if they have 1,000 hosts like yours doing
it then it's 160Mb/s that's harder to block. It's still pretty small
fry though.

You could configure your nameserver to not log denied queries. That
would solve your most pressing concern - it won't be filling your
storage with logs about denied queries. Though on the other hand you
may not then know if this happens again. If you decide to do this
then you should think about how you would tell if someone started
doing this again.

Now that you know that 51.81.86.37 is either a victim or an
attacker, I agree with others who have said you should firewall off
DNS queries from it. That will stop your replies to it also. Don't
forget to work out how to re-apply firewall rules after a reboot, as
just adding them with the "iptables" command doesn't make them
persist.

You might like to look into higher-level firewall tools like ufw:

    https://wiki.debian.org/Uncomplicated%20Firewall%20%28ufw%29

or firewalld:

    https://firewalld.org/

If it worries you that anyone can send any kind of query to your DNS
server, you might prefer to take your DNS server private. I guess
you already have secondary DNS servers provided by BitFolk or
others. You could have those servers be the only ones visible, and
firewall off DNS from anything but them. This is referred to as a
hidden primary setup:

    https://second.wiki/wiki/hidden_primary
    https://tools.bitfolk.com/wiki/Secondary_DNS_service#Add_the_BitFolk_DNS_servers_to_your_domain.27s_NS_list

If you *are* going to accept queries on your server then you will
have to think about how you will detect and prevent dodgy behaviour
such as this.

There may be some rate limiting settings you can use in your name
server to slow the rate of responses to identical questions. Some
people put "dnsdist" in front of their nameservers as it's very
flexible:

    https://dnsdist.org/advanced/qpslimits.html

Finally, you may want to use tcpdump to see exactly what queries
are being asked. Something like:

# tcpdump -s 0 -vpni eth0 'host 51.81.86.37'

Or to save to a capture file instead of printing to the terminal:

# tcpdump -w /var/tmp/dns.pcap -s 0 -vpni eth0 'host 51.81.86.37'

That can be later examined with tcpdump or other tools like
wireshark, tshark etc.

Cheers,
Andy

--
https://bitfolk.com/ -- No-nonsense VPS hosting