Server Fault is a question and answer site for system and network administrators. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I have a very interesting and frustrating issue. The company that I work for uses an application to do simulation and verification for electrical engineering. This application produces output based on a random seed. Once you have the random seed for a given simulation, you can feed that random seed back into the application, and you should get identical results. Unfortunately we're running into issues where different Linux distributions are producing different results given the same random seed. This is problematic because the engineers utilizing these tools rely upon the ability to re-produce exact results every time. I should mention that we're using exactly the same version of the application across the different distributions - in fact, it is NFS-mounted, so it's exactly the same install point that gets tested every time.

At this point, I'm testing with the following operating systems:

CentOS 5
CentOS 6
openSuSE 11.4
openSuSE 12.1
openSuSE 12.2

The high-level results that I'm seeing are that CentOS 5 and SLES 11 produce identical results, and CentOS 6 and openSuSE 11.4/12.1/12.2 produce identical results. CentOS 5 and SLES 11 are the distributions listed as supported by the software manufacturer, so, at this point, we're looking at the result that those two distributions produce as the "correct" one. I haven't been able to find a consistent set of similarities or differences at this point that tells me what's going on. For example, here are the kernel versions in use on the different systems:

CentOS 5 - 2.6.18
CentOS 6 - 2.6.32
SLES 11 - 3.0.51
openSuSE 11.4 -
openSuSE 12.1 - 3.1.10
openSuSE 12.2 - 3.4.11

So, without digging into details about what patches might be applied to which kernel versions in these distributions, I can already say that it isn't a major change between 2.6 and 3.0, for example, because SLES 11 and CentOS 5 produce consistent results, yet have differing major kernel versions. I've looked at glibc, as well, and struck out there - openSuSE 11.4 and SLES 11 have very close versions of glibc packages (2.11.3).

My question is this: where are the next places I should look to try to track down this problem? I've started into getting strace output and trying to compare that, but that's very time-consuming and of little value when the strace output is vastly different even between distributions that produce identical output. I cannot very well go to the application manufacturer since the platforms that I'm having issues with are listed as unsupported, so they will just tell me to get lost. Any hints on where to go from here in tracking down the problem?

share|improve this question

closed as off topic by EEAA, mdpc, John Gardeniers, TheCleaner, Khaled Feb 8 '13 at 18:25

Questions on Server Fault are expected to relate to server, networking, or related infrastructure administration within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

Why not attach gdb and start stepping though the critical functions comparing until you see which function calls start diverging? – kormoc Feb 8 '13 at 0:31
What libraries is this program linked against? – mattdm Feb 8 '13 at 1:25
An ldd on the primary binary shows this: ` => (0x00007fff3b199000) => /lib64/ (0x00007f61ef15c000) /lib64/ (0x00007f61ef4c9000) ` However, I'm still trying to figure out how many other binaries might be called in the process, so this may not be a complete list. – Nick Feb 8 '13 at 14:57
Please elaborate on the reasoning for closing: this is a legitimate production environment question, and one that an engineer would be required to prove is not the fault of the servers. The code is likely to blame, but an engineer would need to know where to go from here. – Andrew B Feb 11 '13 at 20:05
@AndrewB "Why is X generating different binary code than Y" is not really a system administration question - it's a software development one (more appropriate to Stack Overflow than Server Fault). – voretaq7 Feb 11 '13 at 20:29

The last time I ran into this particular issue, it ended up being a difference in whether the two operating systems were running in 32-bit mode or 64-bit mode. (@kormoc's suggestion is useful, as gdb was how I isolated it) Since you did not validate that this condition was the same between the servers, I would recommend that you begin your search there.

If the architecture does differ, you can then confirm that as the source of your problem by setting up one of your problem OSes with a matching architecture and seeing if the results converge once more. That should provide your developers enough information to work on implementing platform independent seeds.

share|improve this answer
Sorry about that - meant to add that, as well. All of the distributions that I'm evaluating are 64-bit distributions, and I'm running them all as Xen PV domUs on a single piece of hardware. So, I don't think it's architecture or hardware at this point - something else. I will work on the gdb stuff tonight or tomorrow... – Nick Feb 8 '13 at 0:35
Shucks, was worth a shot. Take a look at cgdb if you don't have X installed, it splits your screen and tracks the current execution position within the code in a separate pane. I've never gone back. – Andrew B Feb 8 '13 at 0:39
I've taken the liberty of firing up a chat in the event that you need assistance stepping through gdb. If you haven't gotten 4 upvotes by the time you need to use it, visit the URL and reply with your profile URL on that site so that I can override the low rep filter. – Andrew B Feb 8 '13 at 0:59
Thanks for the offer of a chat - I think I got gdb working. Unfortunately it seems that the moderators believe my question does not belong here, so I will go on a search for a better place to ask it. – Nick Feb 11 '13 at 19:52

Not the answer you're looking for? Browse other questions tagged or ask your own question.