The answer to your questions is maybe. It depends on the spec's of the memory, if the memory matches the specification then there is no reason why it will not work.
Your system supports the following memory -
Registered ECC (RDIMMs):
- Up to six single-rank RDIMMs for a maximum of 6 GB (6x 1 GB)
- Up to six dual-rank RDIMMs for a maximum of 24 GB (6x 4 GB)
- Up to six quad-rank RDIMMs for a maximum of 48 GB (6x 8 GB)
ECC Unbuffered (UDIMMs):
- Up to four single-rank UDIMMs for a maximum of 4 GB (4x 1 GB)
- Up to four dual-rank UDIMMs for a maximum of 16 GB (4x 4 GB)
In short, max is 48GB with Registered ECC memory (RDIMM) and 16GB with ECC unbuffered memory (UDIMM). It has 6 slots. Supported speeds are PC3-8500(1066MHz), PC3-10600 (1333MHz). 2GB is the minimum RAM requirement for ESXi 5.1, however 8GB is recommended. IMO, you should look for anything above 8GB.
Note: Registered ECC memory is supported only by systems with Intel Xeon 3400 Series processors, so make sure what type of processor you have before purchasing the memory.
Hope that answers your question!!
Source - http://www.redbooks.ibm.com/redbooks.nsf/RedbookAbstracts/tips0802.html