I haven't found an solution so far, so I'm asking here:
This is the situation I'm locking an solution for:
We are planing an architecture that contains the following components
- Docker Container (auto load balancing as load requires it) Application Server that Handles incoming Data (import, backend storage, web api)
- Queue Server
- Docker Container (auto load balancing as load requires it) Application that Processes each item on the queue and stores it to an elastic search server and MySQL Database (ES is used for queries and output to customers)
- Docker Container (auto load balancing as load requires it) Application that Handles outgoing data collection
I have almost everything completely setup. The only thing I don't have an solution for is the Load Balancing to the Frontend (Frontend Nginx Server). I only require this for the Communication to 1 and from 4. Their Count can vary so Nginx should be capable to know how many containers exist. Session Handling is working (Redis Backend) (Tested with Load Balancing on 8 App Servers and an Stress test Tool.). But I don't have an clue on how to handle the communication between the Varying number of app servers and Nginx. If any one could give me an clue how to configure this. Any clue on solving this is appreciated.