--syn flag is useful to check TCP traffic, but the
NEW state can be used for other protocols (including
ICMP. I can say that
NEW is more general than
--syn TCP option.
From the iptables manual, you can read:
NEW meaning that the packet has started a new connection,
or otherwise associated with a connection which has not seen packets in both directions
An example, a DNS request will match the
NEW state, but it will not match a rule with
--syn option. Simply, it is a UDP datagram.
--syn option can be used to check for TCP packets with bad flags combination to drop them.
Also, you can use both of these options together to check for
NEW TCP flows without
--syn as first packet and drop them such as:
$ sudo iptables -A bad_tcp_packets -p tcp ! --syn -m state --state NEW -j LOG --log-prefix "New not syn:"
Here, we are adding this types of packets to a user-defined chain called
bad_tcp_packets to be dropped/logged, etc...