4

I have a Linux (RHEL 5.8) server with Java 7 and Tomcat 7.

The performance is poor, and I am pretty sure it's the DB queries that are slow.

We have 2 cores right now, and load average never exceeds 1.5, the second core usage is often 0%. They want to try adding a core to see if it will help. I think it won't. Typically I would only add another if I see that all the cores are maxed out at least some of time.

What's your thinking on this? When do you say it's time to add more cores?

More info DB is on another box managed by a DBA. I am Linux Sysadmin.
CPU: 2 cores of Intel(R) Xeon(R) CPU X5675 @ 3.07GHz 16 GB RAM with 2 GB swap 8 GB allocated to heap, usage for heap is generally less than 4 GB, spikes are around 6GB CPU usage spikes at around 120%

8
  • 1
    Who are they?
    – ewwhite
    Apr 18, 2014 at 9:30
  • @ewwhite “They” are most likely co-workers who are making the request. Apr 18, 2014 at 13:54
  • "They" are developers for the app which runs on that Tomcat instance. Apr 18, 2014 at 18:53
  • 6
    You do realize that your whole post is fundamentally flawed, right? Your think the DB is slow but you're asking about cores on the application server. Sounds to me like you haven't done your homework. You need to do application profiling and figure out what parts of the application are taking so long.
    – longneck
    Apr 18, 2014 at 19:07
  • 1
    @longneck You are misunderstanding the question more than it being flawed. I am saying, I don't think we need any more cores, and I think the problem is on the DB, which is a separate server. I actually think a big part of the problem are the dynamically created queries. They want to put more cores on the app server, which doesn't see more then 1.5 load with 2 cores. I think that is reasoning enough to say we don't need cores (regardless of whether the app is cpu or IO bound). I want to know if that is sound thinking regarding when to add cores in general. The rest is backstory. Apr 18, 2014 at 19:49

6 Answers 6

7

Your performance may have nothing to do with core-count. Yo could be I/O-constrained or just not have enough RAM on the system.

What are the specifications of the underlying system hardware? E.g. CPU model, clock speed, RAM amount.

Is the performance poor compared to another metric? Your expectations? Has the performance always been bad?

In general, I look at system load as the way to determine if the core count is sufficient. It sounds like it's at the right level in your situation.

2

The performance is poor, and I am pretty sure it's the DB queries that are slow.

If you are confident that is the issue, then have you optimized your MySQL or MSSQL install? Simply installing software without performance tuning will not be solved by just throwing more resources at it.

I recommend using the MySQL Tuning Primer Script located here. Very easy to use & the recommendations are pretty spot on.

Depending on your setup you might need to learn how to performance tune by hand—meaning you learn to interpret MySQL output yourself and act on it—but this script works quite well for 95% of the setups I have ever used them on. The other 5% are database idiosyncratic setups that required more custom care. I highly recommend tutorials like this one on the MySQL performance blog.

3
  • DB is on another box managed by a DBA. I am Linux Sysadmin. Apr 18, 2014 at 18:54
  • First you say, “DB is on another box managed by a DBA.” And you say, “I am pretty sure it's the DB queries that are slow.” Then do you think this problem is simply a hot potato to be passed around? If the DB queries are slow & it’s on another box, then why are you mentioning it? Apr 18, 2014 at 20:48
  • The point I was making is that I am certain in this case (lack of) cores are not the issue, but the developers still want so see if it helps. I gave them the reasoning based on the CPU usage statistics, and then wondered about (aside from the very obvious cases) what others are using as a metric that more cores are needed. I guess the DB backstory is confusing much more than helping in this question. Apr 18, 2014 at 21:33
0

Prior to adding more cores, it would be a good idea to look at what resources are actually being used - for example if the system is IO bound, adding more cores is throwing away money, but upgrading your disks or RAM will probably solve the problem. Try running vmstat (for example "vmstat 2") and monitoring the io, memory and CPU to get a bit of a feel for whats going on.

0

I/O constraints are most likely.

SSD?

1
  • There's no proper way to know that from the information we have.
    – MichelZ
    Apr 18, 2014 at 19:38
0

Since you are in a virtualized environment, adding/removing CPU and memory is so easy it's basically a no-op. So just do it. Then when there is no performance improvement, you point to your application profiling reports and say "I told you so".

0

There are multiple metrics to use when determining when to add cores to a VM. Core usage, as you mentioned, is one of the most obvious metrics. Others include actual core performance and multi-threading capability of the applications.

A CPU can be a performance bottleneck without ever going above 10% usage. Usually this is the case when your CPU is just plain slow (1ghz vs 2.5ghz), and you'll want to upgrade your underlying host hardware. (My 1.5ghz tablet is painfully slow, even though CPU usage never rises above 10%. It's just a slow CPU!)

There are still a lot of applications out there that are not multi-threaded, so no matter how many CPUs (cores) you throw at them they will never use more than one. Usually in this case you'll want to investigate upgrading your application to take advantage of multiple CPUs before you add more to the VM. Also, some operating systems have restrictions as well (usually based on licensing); for example, Windows Server 2003 Standard only supports 4 physical CPUs, but doesn't care about cores per CPU. This can affect whether you add virtual CPUs or cores-per-vCPU to a VM.

Assuming that my host resource usage is minimal, I would add cores to a VM when my performance metrics show an average (mean) CPU usage greater than 50% during regular business hours. It's an arbitrary number, and depending on your applications and usage patterns you may want more cores at 25% or 75% mean CPU usage, though by the time your mean hits 75% it's probably way past time to make any changes. You'll want to pay attention to consistent usage vs usage spikes. If your CPU is normally at 5% but spikes to 100% every few minutes, you don't want to add cores until you've determined the cause of the spikes. FYI, vSphere has these performance metrics built in.
EDITORIAL NOTE: This is probably most appropriate to an application that has a fairly constant load, such as a web server. Applications with a hit-and-miss load, such as a dev server that compiles code for 20 minutes every 4 hours, would be very different. In that case you would look at the performance of the application (the compiler) and the peak CPU usage when that application is running (compiling). 100% CPU during the entire compiling operation would probably benefit from more CPU, as long as the compiler is multi-threaded. But then, you have to worry about your devs getting mad at you for cutting their play time in half.

Some applications, such as MS SQL Server and MS Exchange, will intentionally hog your resources (memory is the most noticeable one) and this is by design, so you need to be aware of what your applications are supposed to be doing. You need to weigh the actual application performance against its resource usage - if SQL server is using 100% CPU and responding quickly, that's different from a SQL server using 100% CPU and responding slowly.

If application performance is slow, but your metrics and benchmarks reveal that your CPU is not the bottleneck, then don't worry about adding cores - though, if downtime and host resources are no concern, then temporarily adding cores to prove a point to your users doesn't take anything but 10 minutes worth of time.

Also, keep in mind that a good hypervisor, like VMWare ESXi, allows you to over-provision. That means you can provision 4 VMs with 4 cores each (16 cores total) even though your host only has 8 cores. The hypervisor dynamically allocates host resources to the VM that needs them the most, so with 3 idle VMs your 4th VM can chew on as much of your host as you can allocate to it. The catch you want to watch out for is licensing, as some applications are licensed per core or per CPU and not per machine or OS installation.

Not the answer you're looking for? Browse other questions tagged .