2

I got a simple AWS cloud using nginx servers (EC2s) and NFS drive to store CMS files (EFS). Amazon has famous credits and I ran out of credits on a very low traffic cloud. After investigating it turns out that EC2 is flooding EFS with metadata requests. I just don't know how and why?

All the files in total are 8GB on NFS, they're all PHP files. Static files are offloaded to S3. The traffic between EC2 and EFS is about 1MB per second, however, metadata traffic is 5MB per second! That comes to between 100-300MB per minute of metadata traffic only!!!

After checking nfsstat and dumping the traffic it turns out that 90% of that traffic is GETATTR request and response.

But observing EC2, the processors on are almost asleep so processing 5MB per second sounds surreal. Not mentioning 5MB metadata for 1MB transfer.

I'd like to do some more search but my knowledge and Google has ended so don't know where from here. Tried to utilise lsof and nsfstats and can't find what in world is needing that much of metadata. Any help would be much appreciated.

EDIT: If you're looking for a solution for running out of AWS EFS credits because of this or similar reasons - EFS credits are based on the space you're taking. The solution (from AWS support team) is to create dummy files to increase your credit balance. In my case adding 50GB of "empty" taken space did the trick via command:

dd if=/dev/zero of=/path/to/efs/dummyfile bs=1G count=10

Above will create 10GB "empty" file on EFS that will increase your credits

6
  • Is your socket for communicating with the php daemon also on NFS or just the files?
    – David
    Aug 25, 2017 at 6:05
  • Is find running on the front end for any reason? Aug 25, 2017 at 7:06
  • @david no NFS stores only files there's no logic on NFS all processing is on EC2 server
    – Ralf
    Aug 26, 2017 at 8:43
  • @cedric do you mean find as unix find? No there's barely anything going on on the server and nginx. The more I look at it I suspect php-fpm socket asking constantly for file permissions but I can't figure out how to read those metadata packets to find out what they are. They're unreadable on wire shark :(
    – Ralf
    Aug 26, 2017 at 8:47
  • Yes, I meant GNU find utility. Something like mlocate.updatedb uses it and generates a lot of stat() calls for a filesystem. iotop, iostat, 'lsof' and strace can all be useful for looking at filesystem activity. Have you tried tcpdump instead of wireshark? Aug 26, 2017 at 12:36

0

You must log in to answer this question.

Browse other questions tagged .