0

I have a relatively busy application running on a dedicated server. The spec is

8-core Xeon(R) CPU E3-1240 v3 @ 3.40Ghz 
16GB RAM
8TB RAID 10
80GB SSD for MySQL

I use Wowza Streaming Server to stream audio and have a website running on this machine as well.The application had been running fine with no performance issues.

I have since moved to a high availability cloud solution which consisted of the following.

2 x Web nodes - 2 x CPU Core / 12GB RAM each
1 x DB node (MySQL) - 2 x CPU Core / 12GB RAM / 30GB SAS Storage
1 x Load balancer
2TB SAN Storage 

The website and audio files are stored on the SAN storage and the MySQL on the SAS storage on the DB node.

As soon as I moved to this solution I noticed massive drop in performance, site load times became really slow and at times takes 30sec - 1min to load. I have constant audio streaming and moderate website traffic. Interesting to note is there appears to be no performance issues with the audio streaming, audio begins quickly and there's no stopping and starting, it's only the website that runs slow.

The server management team have since upgraded the cores on the machine with the web servers now having 6 cores each and the db node with 24 cores. There has been a performance increase since the upgrade but it's still a seriously degraded performance compared to the dedicated server.

The management team are working on it and are making tweaks to increase performance but I wanted to open the question up to the community to see what ideas anyone has as to why the performance is so degraded. They have mentioned there are I/O issues with the SAN storage, but have yet to find a solution.

Is SAN storage suitable for heavy audio streaming and a busy web application?

UPDATE

Response time before on dedicated server

enter image description here

After on new server

enter image description here

So as you can see the application is a lot slower on the new set up.

5
  • 1
    If they see I/O issues on the storage end, that is going to be the obvious place to fix first. Without knowing anything about their storage infrastructure (fiber? iscsi? direct SAS?, RAID levels, drive types, number of spindles, volume layout, etc) or the load you are putting on the storage, I don't think I would be able to provide any real help as it would just be random shots in the dark. SAN storage can be fine if configured correctly.
    – Rex
    Jul 24, 2015 at 15:15
  • 1
    Have you profiled the web app to see what's taking so long?
    – GregL
    Jul 24, 2015 at 15:22
  • 2
    Unless you have admin access to the SAN and know how it's configured, this is the wrong question to ask.
    – Basil
    Jul 24, 2015 at 15:26
  • The way I read this, the DB having the performance problem is trying to use local SAS hard drives, not the SAN storage? Jul 26, 2015 at 2:29
  • Was there any form of SLA with the service provider for their SAN?
    – Chopper3
    Jul 26, 2015 at 8:33

1 Answer 1

2

Absolutely. You can use any kind of storage for these kinds of applications. But I suspect your SAN is not configured properly for what you need. For something like this you don't just need space, you need speed, which means physical disks to share the load. If they are giving you a slice of a RAID5 array shared with 20 other hosts...that's not going to work too well.

The audio streaming is probably being affected by the slow storage, but because audio streaming is designed to function across narrow connections, it is able to compensate better. So I wouldn't be too sure that storage is "happy" right now.

The web site is slow probably because your database is very unhappy, almost certainly with its storage.

From what I can see, your storage is substantially less fast than it was on the dedicated hardware (SSD to SAS for the DB from what I can see).

Storage slowness often shows up as high CPU utilization as the CPU is in wait state. Depending on OS, you should be able to see this and characterize where it is worst.

The good news is, assuming you are using Fiber Channel connections, it is possible, in theory, to get you the throughput you need from the SAN. But you'll need to have a better disk configuration or selection in order for that to happen. If possible, put your DB onto SSDs again. If you can't get that, try to get a larger number of physical disks in a disk group for you to pull from.

If all else fails, look into increasing RAM on the DB server, which can compensate for inadequate storage.

1
  • Thanks you Mary that's a really helpful answer. I'm going to find out more about the disk configuration to get a better idea.
    – Wasim
    Jul 25, 2015 at 11:36

Not the answer you're looking for? Browse other questions tagged .