Tag Info

Hot answers tagged

34

Lowering the value is pretty trivial without a mysql restart Let's say you want to lower timeouts to 30 seconds First, add this to my.cnf [mysqld] interactive_timeout=30 wait_timeout=30 Then, you can do something like this mysql -uroot -ppassword -e"SET GLOBAL wait_timeout=30; SET GLOBAL interactive_timeout=30" All DB Connections after this will ...


30

I finally found the setting that was really limiting the number of connections: net.ipv4.netfilter.ip_conntrack_max. This was set to 11,776 and whatever I set it to is the number of requests I can serve in my test before having to wait tcp_fin_timeout seconds for more connections to become available. The conntrack table is what the kernel uses to track the ...


23

I'm not at all surprised. SAN arrays typically have a LOT of disks involved. The limiting factor for disk I/O is the speed of the individual disk, and these stack. 6 drives locally in a RAID10 will perform better than 2, and 80 drives on a SAN will perform better than 10 drives locally. There are variables of course, but that's how it's supposed to work. ...


23

Based on the info in the MySQL Documentation you should do the following: Find out what the highest number of simultaneous connections mysqld has had using Connections, Threads_created, and Max_used_connections, SHOW GLOBAL STATUS LIKE 'Connections'; SHOW GLOBAL STATUS LIKE 'Threads_created'; SHOW GLOBAL STATUS LIKE 'Max_used_connections'; Try ...


18

You really want to look at what the /proc filesystem has to offer you in this regard. TCP Tuning Guide at the US Dept. of Energy TCP Tuning parameters for different OS:es IBM's "Administer Linux on the fly" Documentation at LinuxInsight.com vis-a-vis /proc/sys/net/ipv4 On that last page, you might find the following to be of interest to you: ...


15

It's almost certainly due to caching. The DAS probable has minimal caching, where most Enterprise SANs have multiple gigabytes of cache. I'd guess the app is saturating the DAS's cache, but not the SAN's.


11

kjournald is responsible for the journal of ext3 (journaling filesystem). It's known to use a lot of CPU under certain loads. There's not much to do except use another filesystem or disable journaling (effectively making the fs ext2). Theoretically you can use one of the other modes of ext3 journaling and check if the CPU usage goes down, but remember that ...


10

You probably shouldn't consider MyISAM, INNODB will work for you. MyISAM is maybe faster when it comes to SELECT but (for example) it locks your full table on updates. As for INNODB: generally, always consider more RAM before you go into sharding (size of the DB =~ RAM) take a look at the following variables: innodb_buffer_pool_size (we use roughly ...


10

I wrote a comprehensive guide to tracking down performance bottlenecks on Linux systems for work: http://web.archive.org/web/20101028025942/https://anchor.com.au/hosting/development/HuntingThePerformanceWumpus . Covers more than you asked for, but it'll (hopefully) help you track down the problem you're seeing regardless of the actual source.


10

Top has a field called "iowait". If your system is seeing a lot of that, you know something's up. There's also iotop! Package: iotop: Description: simple top-like I/O monitor iotop does for I/O usage what top(1) does for CPU usage. It watches I/O usage information output by the Linux kernel (requires 2.6.20 or later) and displays a table of current I/O ...


9

Conceptually it always feels like serving disk from SAN should be slower than serving it locally. However, there are plenty of factors which can reverse this and result in the SAN being a much faster option. Some of these factors are: Does your workload require fast seek time or fast throughput, or both? How many spindles on the SAN LUN versus the local ...


7

Disable the NTFS Last Access Time Stamp; Turn off indexing and speed up Windows XP; and Turn off unnecessary services.


7

Spend money on RAM first, disk second, and CPU speed third. Use CVS or some other software version control system even if you're the only programmer. Back up frequently. Actually, spend money on a good monitor and keyboard first.


6

It sounds like the bottleneck is the app powering the socket rather than it being Nginx itself. We see this a lot with PHP when used with sockets versus a TCP/IP connection. In our case, PHP bottlenecks much earlier than Nginx ever would though. Have you checked over the sysctl.conf connection tracking limit, socket backlog limit net.core.somaxconn ...


5

Not all application traffic is highly paralellizeable. We have a web-application internally that is resolutely single-threaded, even when running on a 24-core monster. In our case, the web-server will happily use as many cores as we throw at it for static serving; but when bound to a specific app-server it stays with one. If we look at per-core usage when ...


4

Disable last access time on your filesystem;


4

By default your ext3 filesystem is going to be mounted with atimes turned on. Each time a file or directory is read/accessed the filesystem will have to write back to the disks to update this atime record. This means that even if your workload is mostly read based you'll still need to hit the disks to update the access times of each file & directory, and ...


4

The requirements and constraints: 50:50 read:write ratio Files being written will range from way larger than the block size to vastly larger than the block size. Individual requests will range from 128KB to 4MB On Linux The file-system will be pretty large, at 14TB. Unknowns that would help: Whether or not the random I/O is within files, or is purely ...


4

If there are processes binding to INADDR_ANY, then some systems will try to pick ports only from the range of 49152 to 65535. That could account for your ~15k limit as the range is exactly 16384 ports. Wikipedia: Ephemeral port You may be able to expand that range by finding the instructions for your OS here: The Ephemeral Port Range


4

The command show status like 'Com_select%'; is at the session level. You probably want the server level. Try show global status like 'Com_select';


3

Some suggestions Look at the read queue size, if your application is highly random, then tweak the readahead in /sys/block/<< dev >>/queue/read_ahead_kb to ensure your reading data you need not data the os thinks you need. Switch to the deadline scheduler if you haven't already use the noatime mount option unless you're hosting a mail spool mount ...


3

Either don't install antivirus software or disable scan-on-access (and if not system-wide, then at least for your development directories). This can make a big difference in compile time. See also this answer.


3

I don't think there is a tunable to set that directly. This falls under the category of TCP/IP tuning. To find out what you can tune, try 'man 7 tcp'. The sysctl ( 'man 8 sysctl' ) is used to set these. 'sysctl -a | grep tcp' will show you most of what you can tune, but I am not sure if it will show all of them. Also, unless this has changed, TCP/IP ...


3

It sounds like you need greater concurrency. First ensure that you aren't bound by disk activity. atop is a good interactive method for this. sar and other tools are available as well. Given the ramdisk, this shouldn't be the major issue. If your queue is full of mail going to many different domains, that is a sign that you aren't running enough server ...


3

You are allowing Apache to spawn up to 1500 children to serve requests (ServerLimit / MaxClients) -- It's no wonder your server load (the number of processes waiting in the run queue) is getting enormous! My first suggestions at 400 requests per second with the numbers you're quoting in your question would be "Move the MySQL server to its own box" or ...


3

Don't forget that InnoDB Buffer Pool Houses Data and Indexes Please run this query SELECT CONCAT(ROUND(KBS/POWER(1024, IF(PowerOf1024<0,0,IF(PowerOf1024>3,0,PowerOf1024)))+0.49999), SUBSTR(' KMG',IF(PowerOf1024<0,0, IF(PowerOf1024>3,0,PowerOf1024))+1,1)) recommended_innodb_buffer_pool_size FROM (SELECT SUM(data_length+index_length) KBS FROM ...


2

You could reduce the time spent in the TIME_WAIT state (Set net.ipv4.tcp_fin_timeout). You could replace Apache with YAWS or nginx or something similar. Tradeoffs of more connections generally involve memory usage, and if you have a forking process, lots of child processes which swamp your CPU.


2

Try setting the following as well setting tcp_fin_timeout. This should close out TIME_WAIT more quickly. net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_tw_recycle = 1


2

It sounds like you're doing OK. Yes. I'd drop the RAM allocated to the main instance until you start to see cache hits drop below 99% (a flawed metric I know, but can be used with care) and look for increases in disk IO rates (disk queue length, average response times etc) as well as the good old "user test" - Does it seem to run slower to the user or do ...


2

450 children with an RSS of around 10mb each is over 4GB of potential memory usage. More than enough to cause your c1.small instance to swap. Swapping is almost always a downward spiral for apache servers. I'd say the next few things I'd be looking to check are - does the apache error log mention hitting maxclients - does dmesg or /var/log/messages ...



Only top voted, non community-wiki answers of a minimum length are eligible