8

Okay, so first of all, allow me to say that I'm not an operations person, but a developer. So I'm going into a bit of unknown land here, so please bear with me.

I would like to use a Azure Virtual Machine for extracting a 50 GB XML file from a 1.9 GB zip file. So I've been testing which instance size on Azure I should use to get good performance while not paying for more than I need.

However, the disk performance of Azure VM's have not been amazing, and I would like to know if it's me who's doing something wrong, or if my results are what can be expected.

First of all, what I have been testing with? I have a custom .NET console application that does nothing but take a zip file as an argument and immediately starts extracting the zip file to the same directory that the zip file resides in. While the extraction is going on, the application calculates how many megabytes the application have written to the target file per second and outputs it.

On my local development machine, I get pretty good performance with this application, 160-210 MB/s written. So the whole extraction process takes around 8 minutes. The specs of my local machine is Intel Core i7 950, 3 GHz, 4 cores (8 logical), 12 GB RAM, Samsung SSD 830 series 250 GB.

Okay, so I started testing different instance sizes, and here is my results.

  • On a A4 instance with Windows Server 2012 Datacenter R2 (8 cores, 14 GB RAM) with a striped RAID of 4 virtual disks using the same storage account, without host caching, I got steady 30-35 MB/s, which means the whole extraction took 24 minutes and 48 seconds. I also tried enabling host caching, but it didn't really make any difference.
  • On a D4 instance with Windows Server 2012 Datacenter (8 cores, 28 GB RAM, 500 GB local SSD disk) i got really good performance (150+ MB/s) for the first minutes, and then varying performance with peaks at 200 MB/s and valleys at 9 MB/s. Average performance was between 70 and 100 MB/s. The extraction took 9 min and 40 seconds.
  • On a D3 instance with Windows Server 2012 Datacenter (4 cores, 14 GB RAM, 250 GB local SSD disk) i got really good performance (150+ MB/s) the first minute, but then the performance declined to steady 20-40 MB/s, making the extraction process take 21 minutes and 49 seconds.

On a D2 and D1 instance, the disk performance is worse than on the D3.

And this really surprises me. How can a local SSD disk perform so badly, as it does on the D1, D2 and D3 instances? And does anyone know why the disk performance differs so much between D1 to D4? Is it a memory issue? When I look at the task manager while the extraction os going on, the memory usage is exploding. I suspect it's because Windows is caching the written data, but when it runs out of memory, it has to flush the data to disk. When this happens, the disk performance declines. But this doesn't happen on my local machine, so why is such aggresive caching necessary on these VM's?

I know there's differences between my local machine and a virtual machine hosted in Azure, but is the disk performance I'm experiencing really to be expected?

(I originally posted my question on Stackoverflow, as I suspected it was my application that was the cause. But I'm not so sure anymore)

6

Bit late to the party here, but for what it's worth "attached" SSDs in Azure have their IOPs throttled based on the machine size. It wasn't mentioned in the pricing anywhere but I raised this issue as a ticket with technical support when they referred me to the blog post below.

See this link: http://azure.microsoft.com/blog/2014/10/06/d-series-performance-expectations/

  • Great info, thanks Richard! I suspected it had to do with throttling, but I couldn't at that time find any specific information about it. That blog post answers my questions. – René Apr 29 '15 at 4:23
  • azure limit disk io..what the heck..I switch to google cloud ..they don't limit this and price are much cheaper. – tyan Mar 3 '18 at 3:57
4

Maybe it's not related to the memory. "Physical" disks on Azure uses what's called Blob storage and that's the reason the performance is not the same as in your local computer even with similar hardware. You can find more information on this link. This seems to be a common problem for the SQL Server community who relies heavily on disk performance. I'll quote an article i found about it

The Performance Guidance for SQL Server in Azure VMs whitepaper mentions (on pages 15 and 26) that in Azure VMs, Storage Spaces can be used to improve storage performance for SQL Server workloads

Storage Spaces is a built in feature of Windows Server 2012 that lets you use several physical disks to create one single virtual disk, hence as a work around for the I/O limitations of the VM disk itself. Quoting again:

With larger VMs that support many (up to 16) physical disks, a very significant increase in storage performance is possible. This can enable Azure as a viable platform for many SQL workloads that would be too constrained by the I/O performance of a single Azure disk otherwise.

This article will show you the steps to create a storage space.

You should try this first. Probably you can even use a smaller instance as the storage service is a separate service from the instance itself.

  • You're right that standard disks on Azure actually uses Blob Storage, but I don't believe that's the case with the SSD disks. According to the information given my Microsoft, these are local physical SSD disks. And when that's the case, I think it's odd that the performance differs so much between the D instance sizes. Regarding Storage Spaces: I explored this and did some tests, but unfortunately it didn't really give me any noticeable performance gain. In the end, I've come to the conclusion that I just have to accept that I need to use a larger instance size for this task. – René Nov 3 '14 at 1:28
  • 1
    @René did you had a chance to test the new Azure Storage Premium? Sorry for bumping this old topic but i'm quite curious to see how this new storage would perform in your ambient. – Bruno Faria Jan 9 '15 at 11:48
  • Sorry for my late reply. Great tip about Azure Storage Premium. I haven't tried it as it's not yet available in my region. But I'll keep an eye on it. Thanks again! – René Apr 29 '15 at 4:30

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.