I have a very interesting and frustrating issue. The company that I work for uses an application to do simulation and verification for electrical engineering. This application produces output based on a random seed. Once you have the random seed for a given simulation, you can feed that random seed back into the application, and you should get identical results. Unfortunately we're running into issues where different Linux distributions are producing different results given the same random seed. This is problematic because the engineers utilizing these tools rely upon the ability to re-produce exact results every time. I should mention that we're using exactly the same version of the application across the different distributions - in fact, it is NFS-mounted, so it's exactly the same install point that gets tested every time.
At this point, I'm testing with the following operating systems:
CentOS 5 CentOS 6 SLES 11 openSuSE 11.4 openSuSE 12.1 openSuSE 12.2
The high-level results that I'm seeing are that CentOS 5 and SLES 11 produce identical results, and CentOS 6 and openSuSE 11.4/12.1/12.2 produce identical results. CentOS 5 and SLES 11 are the distributions listed as supported by the software manufacturer, so, at this point, we're looking at the result that those two distributions produce as the "correct" one. I haven't been able to find a consistent set of similarities or differences at this point that tells me what's going on. For example, here are the kernel versions in use on the different systems:
CentOS 5 - 2.6.18 CentOS 6 - 2.6.32 SLES 11 - 3.0.51 openSuSE 11.4 - 22.214.171.124 openSuSE 12.1 - 3.1.10 openSuSE 12.2 - 3.4.11
So, without digging into details about what patches might be applied to which kernel versions in these distributions, I can already say that it isn't a major change between 2.6 and 3.0, for example, because SLES 11 and CentOS 5 produce consistent results, yet have differing major kernel versions. I've looked at glibc, as well, and struck out there - openSuSE 11.4 and SLES 11 have very close versions of glibc packages (2.11.3).
My question is this: where are the next places I should look to try to track down this problem? I've started into getting strace output and trying to compare that, but that's very time-consuming and of little value when the strace output is vastly different even between distributions that produce identical output. I cannot very well go to the application manufacturer since the platforms that I'm having issues with are listed as unsupported, so they will just tell me to get lost. Any hints on where to go from here in tracking down the problem?