It should work if all involved parties use really modern software.
While SMTP works well layered on TCP,it is, at least in its original form, not in itself a protocol BASED on TCP/IP. If you look at the original RFC 821, a "TCP transport" is defined.... in an appendix.
RFC 2821 (from 1989) considers using numerical addresses "discouraged".
Even far more modern versions of the specs uphold that philosophy to some degree, from RFC5321: "SMTP is independent of the particular transmission subsystem and requires only a reliable ordered data stream channel. While this document specifically discusses transport over TCP, other transports are possible. Appendices to RFC 821  describe some of them."
However, this RFC - from 2008, which actually makes it very NEW, does sanction the use of "address literals" as "allowed" ("To bypass this barrier, a special literal form of the address is allowed as an alternative to a domain name.") in Section 4.1.3 but still discourages it as a "SHOULD NOT" in 2.1.4.
SMTP, and much of the software built around it, uses hosts, not ip addresses, as its "native currency" - if an "address literal" is usable as a "host", so be it. And so did the (mostly outmoded) non-SMTP protocols (eg UUCP mail) that were used in the email ecosystem of old together with SMTP-based systems.
Relying on every involved system being in full compliance with a 2008 standard might be more risky than it seems.