0

So I have a bit of a mystery. We have an in-house built video processing application that is, naturally, very process intensive. I got my hands on a server with 2x E5-2640v3 processors and I was able to get the average FPS on the processing to be around 7 to 8 FPS with peaks up to 11 FPS, depending on how complex the scenes are.

My tests in the past have shown a MASSIVE boost when going from v2 Xeons to v3 Xeons.

Just to point out something, this software does a lot of complex computational work. Yes, we know GPU processing would be more ideal, but due to the way our system works there's more to it than simply porting the code over. However, the work is under way, but a usable version won't be ready anytime soon.

I have a single 1230v5 in my desktop and those numbers are around 10 to 12 FPS average with peaks at 15 FPS.

Now the mystery. I managed to get a few servers with 2x E5-2640v4 procs for testing. On this server (using the same product version, performance settings, OS version, etc, etc.) the average FPS drops to around 2 to 3, with the peaks at 9 FPS.

Here are the specs for the 2640v3 and the 2640v4:

Specs------------2640v3-----2640v4

  • Arch--------------22nm------------14nm
  • Cores------------8------------------10
  • Threads---------16-----------------20
  • Base Freq------2.6GHz----------2.4GHz
  • Max Turbo------3.4GHz---------3.4GHz

In the past, the newer versions have always shown a significant improvement in performance for the same product line (ei. 2640 --> 2640v2 --> 2640v3, etc) but it doesn't seem to be the case for the v4 that I have.

All the online benchmarks show that the single core performance of the v4's are about 90% of the v3's. But I'm only seeing about 40% of the performance. A 0.2GHz drop in the base frequency should not result in such a difference. At most I would expect the overall FPS to drop at most 1 FPS, if not just stay the same from the increase in processing efficiency.

Now I have researched, changed, and tweaked the performance settings in the BIOS, hypervisor (I've tried ESXi 5.5 U1, which is what the server has listed in its OS compatibility chart, and ESXi 6.5), and in the OS.

I'm at my wits end to find out what and why there is such a drop in performance. Does anyone have any suggestions?

  • Is hyperthreading on or off? – Michael Hampton Dec 16 '16 at 21:01
  • Are you sure the power settings (both in BIOS and OS) are the same? – Mark Wagner Dec 16 '16 at 21:25
  • @MichaelHampton it's on. I have 40 logical cores available. – Chris Dec 16 '16 at 23:07
  • @MarkWagner Yup. I'm extremely sure. I spent 10 hours investigating the possibility of incorrect power/performance settings. – Chris Dec 16 '16 at 23:10
  • Have you ruled out hardware faults that may cause thermal throttling? – Mark Wagner Dec 17 '16 at 0:42
0

Put aside the specifications of the processor for now. To understand what is going on may require some analysis of your application's execution.

On Linux, for example, the perf command can show counters to get instructions per clock. Or, profiling and sample stack traces to see where time was spent. See Brendan Gregg's perf_events page. He talks about thread state analysis as a general approach to this kind of methodology which can work for any platform.

In addition to the microbenchmarks, keep trying different variables and looking for differences. Be methodical. RAM size and speed, storage system speed, disabling power saving modes, NUMA effects, there are any number of reasons for performance issues.

  • Thanks for the suggestions. I actually did a comprehensive battery of tests earlier this year. I created very detailed charts showing the differences in RAM configuration, vCPUs, number of instances, etc. I did this with v2 and v3 procs from the 2640 to the 2697. All of our tests showed that proc generation is the most important, then higher frequencies over lower frequencies. So based on all that data, the v4 should be king. However, this is not at all what we're seeing. And this is with identical configurations on the software side. – Chris Dec 19 '16 at 15:21
  • Profile the application and instrument the OS. Find where the CPU time is going. Find how many instructions per second the CPU is doing, or it it is stalled on memory or pipeline. If source code is available, try rebuilds with different compilers and optimizations. – John Mahowald Dec 19 '16 at 22:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.