Tag Info

Hot answers tagged

19

Is it ever sane to use a Virtualized solution when performing I/O heavy workloads? Yep, very sane indeed, in fact for most organisations now virtual is the default and doing things on physical boxes is the very much the exception. We have over 100k VMs of all forms and many of them are >40k IOPS with no issue at all. What are the best practices ...


10

Is it ever sane to use a Virtualized solution when performing I/O heavy workloads? Does a database server regularly pulling 1gb/second random IO count? Have one here. Or a virtual file server delivering up to 600mb/second to a HPC cluster. That one is running off 8 Velicoraptors in a Raid 10, dedicated. What are the best practices around this ...


9

At first, the client requests will get queued, until there is a process/thread that gets free on the apache server. So, the clients will see a delay in loading the page. See the MaxClients parameter documentation for more information. When placed in the backlog queue, the client request can eventually time out on the client side. Then the user will see ...


6

somaxconn determines the maximum number of backlogged connections allowed for each TCP port on the system. Increasing it (recommended for servers) can prevent "connection refused" messages, but it can result in slow connections if the server can't handle the increased load. You can check the current backlog with netstat -ant | grep -c SYN_REC according to ...


5

Recognize that Ubuntu 14.04 uses Apache 2 with PHP running through an mpm_prefork module, of which an editable file is in /etc/apache2/mods-enabled/mpm_prefork.conf. Also, recognize that starting in Apache 2.4, MaxClients is now renamed as MaxRequestWorkers, and so any documentation regarding MaxClients needs to be switched to MaxRequestWorkers. Stop the ...


5

What's the question here? You don't seem to be comparing apples-to-apples. So I'm not sure if you just want this system to run like your other systems, or if you really care about the cause of the performance difference. You could obtain another of the server model and SKU that you're familiar with. You could use the same type of disks you used in ...


5

That image says that the maximum UDP transfer rate over 1gbps ethernet is 119635891 Bytes per second. Multiple 119635891 with 8 (one byte is 8 bits) and you get 957087128 == 957,08mbps


4

I don't think it's possible to say: Buy X, install Y, and your problem will be instantly solved. This is going to take several iterations to get through, and you'll need to put more work into identifying the bottlenecks when they inevitably appear. For the most part I'm going to avoid recommending specific hardware, as that would be out of date by the time I ...


4

Congratulations, you've managed to use nearly all of your swap space. The first obvious problem here is that you went very deep into swap. This is probably what's causing the system to thrash so hard (lots of time spent in system, I/O wait and software interrupts). First thing to do is to reduce the number of Apache processes that are running. You don't ...


3

Before anything, based on the screenshot you have posted with htop output, it seems you have 512MB of RAM on a site running WordPress? I have never seen WordPress happy on servers less than 1GB of RAM. Maybe if you are running a test or development site, 512MB is adequate, but for a production site you need 1GB of RAM. That is the root of your problem. That ...


3

What you can raise MaxRequestWorkers to depends on how much RAM each of your httpd / apache processes takes up. If each one takes up 50MB (just picking a random number), then every 20 request workers you have in use at once (i.e. concurrent connections) will take up 1GB. So in that example, you could have at most 8GB * 20 = 160 MaxRequestWorkers. However, ...


2

I suspect that you have several issues here that are stacking up and causing problems. Disk i/o is fragmented, and probably inefficient. OPTIMIZE TABLE on each of your tables should sort that out and help a bit. Memory is tuned horribly, likely causing issues with having to swap frequently. Slow queries - this is the big suspect for your high CPU usage. ...


2

You can use iotop command. It's top-like I/O monitor


2

Besides the basic math & concept that you still need the same IOs as non-virtualised, there is also QOS/prioritisation. Most virtualisation platforms offer at least a basic support for this, will help out a lot to prevent the misbehaving dev VM stalling your prod DB.


2

The reason for using noatime or relatime on an actual disk is that without it reads will also involve writing to disk. This additional writing can slow down the system, cause disks to spin up which would otherwise have remained spun down for much longer, as well as cause wear on flash media. But with tmpfs no data ever has to be written to disk. The data ...


2

Since you're running your environment on AWS, I'm hoping that you're environment is setup using CloudFormation templates or any of the services built upon it (e.g. Elastic Beanstalk). With that assumption, I'm gonna propose the following: Spin up an exact copy of your environment in a separate VPC Create your load test scripts, starting with a lower amount ...


2

As far as I know, listen block maxconn is different then global maxconn. With global maxconn you limit the max number of connections you let the haproxy process handle. Listen / frontend section has its own maxconn, which limits the nubmer of connections per listener. So, try to set up maxconn in your frontend sections too, or at least set it up in default ...


2

You seem to be obsessed with nginx tuning while not even knowing which part of your architecture is the slowest. It's very unlikely that nginx would be the first thing to tune. Put your fastcgi targets in one upstream block and append $upstream_addr and $upstream_response_time to your log format. If not already present, append $request_time to your log ...


2

Late for this party, but might benefit others, so... Re. cluster size, first and most important, you'd need to look at the distribution of file sizes, so you could optimize for both low fragmentation and disk space waste so you'd resize clusters close to this size, not overall avg - e.g.: if most files fall near 2k, a 2k cluster size would be optimal, if ...


2

Separate your application and display logic (front) from your data access logic and storage (back). The database is going to generate a lot of IO activity, and as such will slow down other things inside the same server. Add RAM. no, seriously, add RAM. Accessing data stored in memory is faster than accessing data stored inside a flash-drive, which is ...


2

It makes no sense to try to keep the load low at all costs. What is important is that your process steps back if something more important needs to make use of the resources offered by your system. ionice and nice / renice can be used to reduce the priority so that it only runs when the system is otherwise idle.


2

This boils down to file system and procfs tuning. What you explain as 'high' load is situation where other normal processes on system are starved from reads and they forced to wait. Situation is characterized by high share of CPU wait time (check top %wa) many processes in D state (uninterruptible sleep due to waiting for reads from disk) Using noop ...


1

You are on right track. Here are some things which should help you figure out remaining details. You need detailed monitoring of Apache on all stats provided by mod_status "_" Waiting for Connection "S" Starting up "R" Reading Request "W" Sending Reply "K" Keepalive (read) "D" DNS Lookup "C" Closing connection "L" Logging "G" Gracefully finishing "I" Idle ...


1

As for performance, you must consider two aspects of it. First there is the processing performance in the server, and secondly the performance that end users experience. If you use any sort of redirect, whether by PHP or Apache's Redirect rule (which can be in .htaccess or Apache configuration file, end client must first wait for the first response ...


1

Gluster is terrible at large numbers of small files. At least, as of version 3.6. They claim that the beta version of 3.7 that's already been released greatly improves this issue. I'd think that this version will be released as stable soon, since the Beta was released on April 28th, and they planned on the stable version being released earlier this month.


1

Are you running your benchmark tool from multiple machines or from a single one ? It seems you are running from a single one and so the local port range is the culprit.


1

You'd want to make sure your MaxClients is set to something comfortably above the typical peaks (which is sounds like is 10 in your case but you saw a burst up to 86 once, so maybe 150 to allow some breathing room in case you get a bigger burst). This way you won't run out of connections should that happen and users experience slowness. Be sure you have ...


1

Pacoman, It seems that because you have two two WD-RED drives in a mirror, the IO to write the ZIL consistency group to disk is causing high IO. There is always a ZIL (Write-Cache). If you do not have any LOG devices, then the log device is on the pool itself, and can be as large as maximum write speed * 5 seconds. Your probably reading from the ZIL, ...


1

From a system point of view, there is no inherent differences in running 4 masters with 4 server sections, or a single master with 16 server sections. It implements the same architecture : parallelized event-based processes. The worker/core ratio must be accounted wrt the total of workers across all of your masters if you have several ones. This comes from ...


1

You can use atop. It's the top-alike tool that will show you all kind of stuff about your running processes. It will also highlight the resources that are hogged (being used heavily or becoming bottleneck for the processes). It can also record the state of the system every X minutes, and that feature can be used as "poor man performance monitor". To see ...



Only top voted, non community-wiki answers of a minimum length are eligible