0

I'm trying to improve the performance of server for nginx 100TB/month images distributions. But I have two problems: high CPU load and periodical splashes (I don't know why)

Hardware is quite poor (this is the challenge, 100Tb on sub-standard hardware):

  • AMD Opteron X2150 APU, 4 cores 1.1GHz
  • SATA-SSD 256 GB ZTC-SM201-256G
  • 8 GB RAM
  • Broadcom Corporation NetXtreme BCM5720 Gigabit Ethernet PCIe

Ubuntu 14.04

Nginx 1.12.2

Content: images with 250k-2M size. around 200Gb. Seldom write.

Nginx cache config:

proxy_cache_path /home/XXX levels=1:2 keys_zone=manga:90m max_size=230G inactive=15d;

fstab:

UUID=34d74174-f584-42aa-b838-57b37b89007a / ext2 errors=remount-ro 0 1

UUID=c033e0f8-5034-4076-9aad-f238bbbecedf /boot ext2 defaults 0 2

UUID=1e5cd397-e881-4075-8b21-9c8e62fa6045 /home ext2
defaults,noatime,nodiratime,discard 0 2

UUID=693c70c7-4f09-4c2f-8ba1-c6c4de7c1b70 none swap sw 0 0

/etc/sysctl.conf

vm.swappiness=0 vm.vfs_cache_pressure=50

In atop I see nginx on top

enter image description here enter image description here enter image description here

What reason of splashes???

  • I have checked crontab -- nothing
  • No access logs in nginx
  • Enough memory, 717Mb free
  • FS journal has been disabled
  • Nginx cache not full. ~25% free

Any ideas, how to improve performance?

Will it help if I:

  • Install ubuntu 16.04
  • Upgrade cpu from 1.1GHz to 1.5GHz
  • ???

Thx!

UPDATE

I have seen kswapd0 in atop top. But I have set vm.swappiness=0 and reload conf two days ago.

I'm also interested in where the hardware bottleneck is, CPU, SSD or Ethernet

  • 1
    Do you have an actual problem or have you just arbitrarily decided "these numbers are too high"? You have low-end hardware that is seeing some load, but nothing indicates it is actually overloaded. The spikes can come from whatever, cron jobs, traffic spikes, solar flares - you need to know what's running on your machine that could create these spikes. – Sven Jul 17 '18 at 19:43
  • @Sven during splashes nginx sometimes stops serving requests for 5-20sec. Also, I can't wring 60Mbytes/sec from the server. I checked cron, I have same traffic on another servers (different hardware and locations), but see spikes only on the one server. Yes. I know, the hardware is low-end. It simple to have 10Gbps with $10000 budget =) – demon101 Jul 17 '18 at 19:49
  • Can you try to capture more detailed metrics during those splashes / spikes? An obvious mitigation here is to use a content distribution network to reduce load on the server, but that might cost more than the low end server. – Tim Jul 17 '18 at 20:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.