102

Usually after dumping a MySQL database with mysqldump command I immediately tar/gzip the resultant file. I'm looking for a way to do this in one command:

So from this:

mysqldump dbname -u root -p > dbname.sql
tar czvf dbname.sql.tgz dbname.sql
rm dbname.sql

To something like this:

mysqldump dbname -u root -p > some wizardry > dbname.sql.tgz

Or even better (since I'm usually scp'ing the dump file to another server):

mysqldump dbname -u root -p > send dbname.sql.tgz to user@host

I'm running bash on debian.

9 Answers 9

117
mysqldump --opt <database> | gzip -c | ssh user@wherever 'cat > /tmp/yourfile.sql.gz'

You can't use tar in a pipe like this, and you don't need it anyway, as you're only outputting a single file. tar is only useful if you have multiple files.

7
  • 7
    You're right about not needing tar, but you could use it in the pipeline if you did, with mysqldump | tar cf - | gzip -c | ssh ... 'cat > file.tgz' Jan 27, 2010 at 0:15
  • Does that actually work? I'm pretty sure tar needs a list of filenames to work on.
    – James
    Jan 27, 2010 at 10:25
  • 3
    I updated this to work locally (not on a remote ssh server) oh, and I use a dynamic name based on date, thanks to original poster & answerer! mysqldump --opt <database> | gzip -c | cat > $(date +%Y-%m-%d-%H.%M.%S).sql.gz
    – electblake
    Apr 11, 2011 at 13:41
  • 6
    @electblake: you don't need to be using 'cat' if it's local. Just gzip -c > $(date +%Y-%m-%d-%H.%M.%S).sql.gz
    – James
    Jan 3, 2012 at 18:58
  • Just for fun, you could use netcat instead of piping to ssh. You wouold save a little on the encryption overhead of ssh, if it is being transfered over a secure network (or you don't care about security). Nowadays you might also consider using xz instead of gzip.
    – James
    Nov 5, 2012 at 2:10
56

If you are running this locally just use the following command to backup your database & zip it using gzip:

mysqldump -u userName -p (passwordPrompt) yourDatabaseName | gzip -c > output.gz 

(Edit: fixed -c key)

5
  • 2
    Yes, this is the most simple solution. I use it too. Aug 9, 2012 at 13:53
  • 2
    It should probably be gzip -c, right? May 13, 2014 at 9:24
  • nice... but how do I redirect stderr in this command ? If I append 2> /dev/null it doesn't work anymore. And 2> /dev/null before pipe doesn't work either. Apr 10, 2015 at 2:11
  • mysqldump -u userName -p (passwordPrompt) yourDatabaseName 2>/var/log/dump-errors | gzip -v > output.gz
    – undefine
    Mar 20, 2016 at 13:03
  • im using like mysqldump -u root -p databasename --routines | gzip -v >myfile.sql.gz ... i get a part .gz file which im not able to download
    – Sushivam
    Nov 29, 2016 at 8:59
18

Use a named pipe.

mkfifo mysql_pipe
gzip -9 -c < mysql_pipe > name_of_dump.gz &
mysqldump database > mysql_pipe 
rm mysql_pipe

I use it all the time, it'a awesome.

http://en.wikipedia.org/wiki/Named_pipe

3
  • 6
    James does the same thing in 1 line.
    – Jon Haddad
    Feb 3, 2010 at 17:28
  • 16
    ..but learning about named pipes is worth it :-) Feb 15, 2011 at 15:34
  • 1
    mkfifo mysql_pipe; gzip -9 -c < mysql_pipe > name_of_dump.gz &; mysqldump database > mysql_pipe; rm mysql_pipe there, one line. Of course I would keep the pipe around and use it every time.
    – d34dh0r53
    Jul 21, 2012 at 7:20
18

I wrote a quick script to suck down a remote mysql database. It uses mysql compression, gzip and ssh compression. Sucked down a multi GB database at an incredible rate.

    ssh -C user@host "mysqldump --opt --compress database <table> | gzip -9 -c" > outputfile.sql.gz

A side benefit is that it requires no free space on the source database server, so you can use it to backup a database on a server with zero free disk space before going in an pruning your data.

Hope it helps somebody.

2
  • I've created a simple shell script: #!/bin/bash if [ -z "$1" ]; then echo "Usage: ${0} [host] [user] [database] [outputFile]" exit else HOST=$1 fi if [ -z "$2" ]; then echo "Usage: ${0} ${1} [user] [database] [outputFile]" exit else USER=$2 fi if [ -z "$3" ]; then echo "Usage: ${0} ${1} ${2} [database] [outputFile]" exit else DB=$3 fi if [ -z "$4" ]; then OUTFILE="${DB}.sql.gz" else OUTFILE=$4 fi COMMAND="ssh -C ${USER}@${HOST} \"mysqldump --opt ${DB} | gzip -9 -c\" > ${OUTFILE}" ssh -C ${USER}@${HOST} "mysqldump --opt ${DB} | gzip -9 -c" > ${OUTFILE} Sep 12, 2011 at 18:52
  • Two of those compressions are useless: The option to mysqldump compresses the data in the server process and immediately decompresses to again (if the mysqldump is run on the DB server itself). The -C option to ssh activates gzip compression which will waste further CPU-cycles because the data already is gzipped at that point. Jan 10, 2017 at 19:11
5

Use pv and monitor rate!

mysqldump prod_db -h dbslave | pv | gzip -c > prod_2012_08_20.dump.tgz

Or, if you know the size (3GB), get an accurate estimate:

mysqldump prod_db -h dbslave | pv -s 3g | gzip -c > prod_2012_08_20.dump.tgz
4

Try this:

mysqldump --all-databases --password=dbpassword | gzip -c | ssh user@servername "cat >/tmp/filename_of_your_choice.gz"

Please not that I am in no way good at these things, I just combined 2 options on the web into one.

It may very well be better in some other way but this is a one-liner that works for me.

It does however require ssh.keys to be installed and accepted if you want to use it in scripts or crontab or alike.

1
  • 1
    Welcome to ServerFault. It looks perfectly reasonable to me.
    – chicks
    Apr 16, 2017 at 1:04
2

You can do like:

mysqldump --add-drop-table -h dbhost -u dbuser -p dbname (tablename tablename ... ) | gzip -c > wp.sql.gz

e.g.

mysqldump --add-drop-table -h localhost -u root -p wordpress | gzip -c > wp.sql.gz

1

I've been working on this bash script below that tries to put together all the good advices I've seen when it comes to dump/restore with mysql. It is targeted at remote operations.

Just reconfig vars and give it a try. :)

Features are:

  • you can pass a list of tables to dump (selective dump)
  • you can be prompted for passwords (MySQL/SSH) or set them in variables
  • network transmission is gzipped
  • you can opt to save gzipped dump to remote server
  • you can reimport dump to remote server on-the-fly (no temp files on local/remote server)
  • you have visual feedback of what is happening (thanks to echo and pv)
  • you can set mysql variables before and after the dump process

What needs improvement:

  • you need to pass a list of tables (cant dump all tables)
  • MySQL password are the same for source and target
  • you need to GRANT PRIVILEGES manually (looks like MySQL dont let do it remotelly)
  • you need to have installed sshpass
  • some innodb huge compressed tables are slow to dump (may be mysqldump's fault)

I share this script here hoping it can be improved by the community. (best viewed with nano or other editor that colors the code)

--------------------------------- cut here ----------------------------------

#!/bin/bash
#set -x

#REQUIRED VARS
SOURCE_USER=root   #MySQL user
SOURCE_HOST=localhost
SOURCE_PASSWORD=yourmysqlpass  #optional
SOURCE_DBNAME=yourdbname
TARGET_HOST=192.168.1.2
TARGET_DBNAME=yourdbname
TARGET_SSHUSER=root
TARGET_SSHPASSWORD=yoursshpass  #optional
TABLES='table1 table2 table3 table4'
TARGET_DIR="/data/dumpfiles"
EXEC_ACTION_TEXT[0]='Reimport TABLES directly into remote MySQL database'
EXEC_ACTION_TEXT[1]='Backup gzipped data to TARGED_DIR on remote TARGET_HOST'
EXEC_ACTION=0

#print config
echo "---------------------------------"
echo " SOURCE_USER:    $SOURCE_USER (MySQL)"
if [ "SOURCE_PASSWORD" != "" ]; then
echo " SOURCE_PASSWORD:<present>        "; else
echo " SOURCE_PASSWORD:<to be asked>    "
fi
echo " SOURCE_HOST:    $SOURCE_HOST     "
echo " SOURCE_DBNAME:  $SOURCE_DBNAME   "
echo " TARGET_HOST:    $TARGET_HOST     "
echo " TARGET_DBNAME:  $TARGET_DBNAME   "
echo " TARGET_SSHUSER: $TARGET_SSHUSER  "
if [ "TARGET_SSHPASSWORD" != "" ]; then
echo " TARGET_SSHPASS: <present>     "; else
echo " TARGET_SSHPASS: <to be asked>    "
fi
echo " TABLES:         $TABLES          "
echo " EXEC_ACTION:    $EXEC_ACTION - ${EXEC_ACTION_TEXT[$EXEC_ACTION]}"
echo " TARGET_DIR:     $TARGET_DIR (only for action 1)"
echo "---------------------------------"
echo "PRESS <ENTER> to continue...";  read;  echo

#read the mysql password from command-line (SOURCE and TARGET uses the same password)
if [ "$SOURCE_PASSWORD" == "" ]; then
     echo -n "Type $SOURCE_USER password for MySQL servers: "; read -s SOURCE_PASSWORD; echo
fi
echo "Creating database $TARGET_DBNAME on $TARGET_HOST if not exists ... "
mysql \
--user=$SOURCE_USER \
--password=$SOURCE_PASSWORD \
--host=$TARGET_HOST \
--execute "create database if not exists $TARGET_DBNAME;"

echo '--------------------------------------------------------------------------------------'
echo "**** ATTENTION ****: execute this command on mysql server at  $TARGET_HOST :"
echo "GRANT ALL PRIVILEGES ON $TARGET_DBNAME.* TO '$SOURCE_USER'@'%' IDENTIFIED BY 'yourpass';"
echo '--------------------------------------------------------------------------------------'
echo "PRESS <ENTER> to continue...";  read;  echo

#read the password from command-line
if [ "$TARGET_SSHPASSWORD" == "" ]; then
     echo -n "Type the password for remote SSH Server (TARGET) ['$TARGET_SSHUSER'@'$TARGET_HOST']: "; read -s TARGET_SSHPASSWORD; echo
fi

for thistable in $TABLES
do
     case "$EXEC_ACTION" in
         0)
         thisaction="gunzip | mysql --user=$SOURCE_USER --password=$SOURCE_PASSWORD -D $TARGET_DBNAME"
         endmessage='remote reimporting has finished'
         ;;
         1)
         thisaction="cat > $TARGET_DIR/`date +%Y.%m.%d`-"$thistable".gz"
         endmessage="$thisaction has finished"
         ;;
         *)   echo "EXEC_ACTION=$EXEC_ACTION not supported" && exit 1
     esac

     echo "---------------------------------------------------------------------"
     echo "-- table $thistable"
     echo "---------------------------------------------------------------------"
     (
       echo -n "-- setting variables... " > /dev/stderr  #talk to user via stderr
       echo "SET AUTOCOMMIT=0; SET UNIQUE_CHECKS=0; SET FOREIGN_KEY_CHECKS=0;"
       echo -n "starting mysqldump... " > /dev/stderr
       mysqldump --opt --user=$SOURCE_USER --password=$SOURCE_PASSWORD --host=$SOURCE_HOST $SOURCE_DBNAME $thistable
       echo -n "done mysqldump, reseting variables... " > /dev/stderr
       echo "SET FOREIGN_KEY_CHECKS=1; SET UNIQUE_CHECKS=1; SET AUTOCOMMIT=1;"
       echo -n "commiting... " > /dev/stderr
       echo "COMMIT;"
       echo "done!" > /dev/stderr
     ) | \
     gzip -c -2 | \
     pv | \
     sshpass -p $TARGET_SSHPASSWORD ssh $TARGET_SSHUSER'@'$TARGET_HOST $thisaction
     echo $endmessage ' with exit status '$?
done
0

You can also store your password in a config file and use this option --defaults-extra-file:

mysqldump --defaults-extra-file=mysqldump.cnf DataBaseName | gzip -c > DBOutputName.sql.gz

The config file can look like this:

[mysqldump]
host = localhost
user = username
password = "password"

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .