0

I have a web-based application that conducts some linguistic analysis of user-submitted texts. This is a rather memory-intensive task and typically takes an extended period of time (e.g., up to 3 minutes for processing 30 files). I'm using Django's StreamingHttpResponse function to do the job, but noticed that nginx is dropping user's request after processing about 7 files (less than 50 seconds). I tried to adjust the both nginx and Gunicorn keep_alive settings, but it seems not working. I wonder if anyone here could give me some pointers on this?

I'm also wondering what is the best approach to tackle a task that takes a long time to compute? Asynchronously?

2 Answers 2

0
I'm also wondering what is the best approach to tackle a task that takes a long time to compute? Asynchronously?

This is what worker queues are for. You should consider separating the submission of the files from processing. Let the user submit the files, save them off, add a message to a worker queue to process them, indeed asynchronously. The user gets on with their business, may see a loading screen, but it's no longer related to that web session.

In the meanwhile, a separate process picks new tasks from the worker queue, processing each independently of whatever the user is doing. There are many such queueing systems, like Amazon AWS SQS:

https://aws.amazon.com/sqs/

2
  • Thank you for the explanation. I think a worker queue is what I really need in my applications. If I could follow up on this, how can I notify users that their jobs are done? By email? Or setup a result page and let them refresh from time to time? What's the usual way to handle this?
    – aihaiyang
    Jan 13, 2016 at 3:53
  • Indeed, there are various ways, you named two very good options. See for example webpagetest.org - submit any URL there. It submits to its unique URL, and refreshes that URL until it finds that the other worker process has indeed done its job. Other methods may include an AJAX refresh block somewhere, a push notification, whatever your logic demands.
    – JayMcTee
    Jan 13, 2016 at 8:19
0

I see no-one has answered, so I typed your question into Google and found this page. Have you tried increasing all applicable timeouts as it suggests? That seems like the first thing to do.

0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .