What you are asking about is typically called an "application layer" firewall or "layer 7" firewall because it operates at layer 7 of the OSI model. A layer 7 filter operates by inspecting all packets passing through the firewall in order to identify the protocol in use.
There is a layer 7 classifier available for use with Netfilter, the Linux firewall implementation. It doesn't appear to be an especially active project, but neither is completely stale -- the last update on the main page appears to be from January of last year.
I don't have any personal experience with this module.