Is there any problem in doing this with lots of data?

mysqldump ... | gzip | s3cmd put - s3://bucket/file.sql.gz

The MySQL dump is about 100GB in size. What happens if gzip or s3cmd can't process the dump fast enough? Will it overflow the pipe buffer?


The writer will block waiting for the reader to become ready.

What you really should worry about is what happens if s3cmd fails. Then you have to start all over.

| improve this answer | |
  • would this construct be signifficantly more efficient than doing in steps? – John W. Nesk Feb 8 '17 at 22:21
  • The MySQL server will kill the client connection if it blocks longer than the value of the net_write_timeout system variable on the server, which defaults (iirc) to 30 seconds. – Michael - sqlbot Feb 8 '17 at 22:37
  • @Michael-sqlbot You could make a whole answer out of that. Though I suspect it won't happen in practice since gzip will just read whenever it can. If it does happen it'll most likely because s3cmd went out to lunch. – Michael Hampton Feb 8 '17 at 22:40
  • You'd be surprised. Well... maybe. :) Gzip, bzip2, xz, and their multi-core friends pigz, pbzip2, and pixz will all block their input once they have a full set of data chunks to crunch and hit 100% CPU, and they'll continue to block their input until they can flush their output. It's least noticeable with gzip because it churns through the input faster as a result of its inferior compression ratio. If they really did read whenever they could, they'd potentially demand infinite memory if they couldn't compress and flush fast enough to keep up with the input data rate. – Michael - sqlbot Feb 8 '17 at 23:44

I'm doing physical backup and upload to s3 bucket in a two separate steps with no issues. Data is about 250 GB and .gz file is about 80 GB

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.