I have the following setup: 1 server with application X, that runs a crontab once an hour. The crontab connects to the database and runs some heavy calculations and exports data to a special data file and restarts the application. The export can run anywhere from 10 minutes to 40 minutes.
I want to:
Move that server to AWS and use auto-scaling group.
The crontab to run on only one server, do the calculation export the data and somehow sync it to all other live servers.
All servers should automatically detect the new data and restart themselves safely (not while syncing for example).
New servers that starts from the auto-scaling groups show automatically fetch the data files on startup before starting the actual application.
I don't have a "simple" idea on how to do it or any AWS specific solution.
This is my idea:
Run one server outside auto-scaling group. Execute crontab only on that server. All data files will be uploaded to S3.
All autoscaling servers will have a crontab that runs every minute and check for an unique file "please_download_me_TIMESTAMP"
Once the files are downloaded the script will restart the service.
If a new server is started, on startup it will automatically fetch all files from S3.
Would you think this would work?