Postfix doesn't allow defining multiple entries for
relayhosts for redundancy. Have a look at the page you have referenced:
In the case of SMTP, specify a domain name, hostname, hostname:port,
[hostname]:port, [hostaddress] or [hostaddress]:port. The form
[hostname] turns off MX lookups.
As you can see, there is no way to define multiple addresses/hostnames here.
if I give relayhost a made-up hostname that have multiple mx records,
will it try all?
I guess so, but if it requires authentication, then it may not work with the secondary server anyway and fail.
If I give relayhost a made-up [hostname] that have multiple a records,
how will it handle it?
It will use the first address (A record) returned by the dns in round robin method. But this also doesn't imply that it will try another one in case the one returned by the dns server fails.
Having said that, you may have a look at this postfix option: smtp_fallback_relay