0

Client/Serrver application -- we are having a problem with records failing to upload. It appears that the method I am using in my communication plug-in is that it is trying to open too many connections (one for each Action to be uploaded) and the gateway is refusing them after about 300 connection attempts. During my testing The problem only occurred after about 300 connection attempts. However, I believe that the upper limit is based on total connections used by Apache. I think that Apache is only allowing a total of 500 connection. When there are more actions to be uploaded than there are available connections from Apache then our problem appears.

Need advice/suggestions/solutions, etc..... Linux Server w/ 8GB

One suggestion we looked at wa to implement involves caching the records to be uploaded to a special folder on each client machine ([APP_DATA]/SCI/CACHED_UPLOADS ) Once cached the records will remain in the folder until it is successfully uploaded to the server. It will then be removed from the folder. There will also be an 'attempts' counter. If after a certain number of attempts the records still cannot be uploaded it will be moved to a 'Failed' folder. We can later modify our app to look at this folder for reporting purposes. This method will use a single connection to upload all of the cached files.

I don't like this solution because we need to push script to thousands of client computers. I'd prefer to have solution on the Linux server.

1 Answer 1

1

You haven't really provided enough detail to give a good response, unfortunately. Are you just POST'ing data to a webapp/service? Is there a reason you need to run apache? What's this "plugin" running on the client-side? What sort of data is it sending? What exactly is a "record," etc.

At any rate, I can tell you right now that your app's probably going to need refactoring. You'll want some sort of persistent queue in front of your app, so that you can absorb requests from your client machines and feed them into your app at a rate it can handle. There's plenty of reading material on the internet for this sort of stuff, but zeromq's docs are pretty accessible if you're not familiar and need a place to start ( http://www.zeromq.org/intro:read-the-manual )

If refactoring isn't an option, you'll probably need to triage the issue by sticking a bunch of servers behind a load balancer.

In the immediate term, though.. take a look at the footprint of your average apache process, and get as many child processes spun up as will fit into memory. Then figure out how long your average requests takes, and lower your KeepAlive time. If you're running at the default values, then each connection from a client machine's going to occupy a child process for 120seconds or so. Switching to something like nginx might be an option, too; it threads better than Apache, but I only have experience using it to speedup static content delivery; I haven't tried it out in this use case. Someone else might have more info here.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .