The earliest possible point of dropping packets is the iptables
raw table, as shown in the diagram in https://unix.stackexchange.com/questions/243079/netfilter-iptables-why-not-using-the-raw-table
You can drop packets there in the
PREROUTING chain like this:
iptables -t raw -A PREROUTING -p udp -j DROP
However, with this approach you are also dropping DNS responses for the requests initiated by your server, since processing of the
raw table occurs before connection tracking takes place.
You can add allowed UDP hosts like this:
iptables -t raw -A PREROUTING -p udp -s !nnn.nnn.nnn.nnn -j DROP
nnn.nnn.nnn.nnn is the IP address of the host where you want to receive UDP traffic with.
There can also be other consequences when disabling UDP traffic before connection tracking, depending on the server.