I have to transfer a Drupal site with a 43GB MariaDB DB via ssh/scp to another server.
I have a limited downtime window early in the morning. In the past I transferred smaller DB's (<3GB) directly via pipes like this:
ssh -l webuser 10.0.0.99 "cd /var/www/drupal7/htdocs/site_name && drush sql-dump | xz -1 -" | xz -d - | drush sqlc
But with this bigger DB I am worried that some part of this pipe will fail and I have to restart everything again which almost certainly would mean I will exceed the downtime window.
So the obvious solution would be to split it up in single, independent steps so I could redo parts of the task in case one of them fails:
ssh -l webuser 10.0.0.99 "cd /var/www/drupal7/htdocs/site_name && drush sql-dump | xz -1 - > /home/webuser/db_dump.sql.xz" scp email@example.com:/home/webuser/db_dump.sql.xz . xzcat /home/webuser/db_dump.sql.xz | drush sqlc
But now the steps are sequential and take more time which means less time to redo a step in case something went wrong.
So I guess what I am looking for is a way to create the DB dump on the old server, and have a second, independent process transferring the data to the new server and start restoring the DB on the new server.
scp firstname.lastname@example.org:/home/webuser/db_dump.sql.xz .
on the not yet finished DB dump will not work as scp will not wait until MariaDB finished writing the DB dump.
Does someone know a command, that continues transfering or outputing data until the DB dump is finished?
Or does someone know a better way to transfer the DB?
I figured I could use the
tee command like this:
ssh -l webuser 10.0.0.99 "cd /var/www/drupal7/htdocs/site_name && drush sql-dump | xz -1 - | tee -a /home/webuser/db_dump.sql.xz" | xz -d - | drush sqlc
That would improve the situation, however, imagine the situation where the DB dump is still written to and something goes wrong during network transfer or restore. Then you'd have to restart with the dump file, but you can't just copy it via
scp cause the MariaDB is still writing to it, so same problem as before.