MySQL Backup to FTP and Email Shell Script for Cron v2.1

23 09 2006

I updated the script just a tad bit. Using the magic of sed, this version of the script makes a backup of each database individually when you choose to backup all of your databases. The previous version dumped all of the databases into a single file. When you run this script and have it emailed to you, you'll end up with multiple attachments, each attachment being a single database name. Why did I do this? Well, I myself needed to restore a database and I had all the databases in a single file. I ended up searching throughout the file to find where surrounding databases started or stopped and copied the info in between. It wasn't very fun, especially since the databases weren't very small. This script gets around that headache completely. If you still prefer to have all of your DB data dumped into a single file, search for "MySQL Backup to FTP and Email Shell Script for Cron v2". Both scripts are exactly the same besides this feature.


CODE:
#! /bin/bash

# Ameir Abdeldayem
# http://www.ameir.net
# You are free to modify and distribute this code,
# so long as you keep my name and URL in it.

# your MySQL server's name
SERVER=ameir.net

# directory to backup to
BACKDIR=~/backups

# date format that is appended to filename
DATE=`date +'%m-%d-%Y'`

#----------------------MySQL Settings--------------------#

# your MySQL server's location (IP address is best)
HOST=localhost

# MySQL username
USER=username

# MySQL password
PASS=password

# List all of the MySQL databases that you want to backup in here, 
# each separated by a space
DBS="db1 db2"

# set to 'y' if you want to backup all your databases. this will override
# the database selection above.
DUMPALL=y


#----------------------Mail Settings--------------------#

# set to 'y' if you'd like to be emailed the backup (requires mutt)
MAIL=y

# email addresses to send backups to, separated by a space
EMAILS="1@gmail.com 2@inbox.com 3@goowy.com"

SUBJECT="MySQL backup on $SERVER ($DATE)"

#----------------------FTP Settings--------------------#

# set "FTP=y" if you want to enable FTP backups
FTP=n

# FTP server settings; should be self-explanatory
FTPHOST="ftp.server.com"
FTPUSER="username"
FTPPASS="pass"

# directory to backup to. if it doesn't exist, file will be uploaded to 
# first logged-in directory
FTPDIR="backups"

#-------------------Deletion Settings-------------------#

# delete old files?
DELETE=y

# how many days of backups do you want to keep?
DAYS=3

#----------------------End of Settings------------------#

# check of the backup directory exists
# if not, create it
if  [ -e $BACKDIR ]
then
echo Backups directory already exists
else
mkdir $BACKDIR
fi

if  [ $DUMPALL = "y" ]
then
echo "Creating list of all your databases..."

mysql -h $HOST --user=$USER --password=$PASS -e "show databases;" > dbs_on_$SERVER.txt

# redefine list of databases to be backed up
DBS=`sed -e ':a;N;$!ba;s/\n/ /g' -e 's/Database //g' dbs_on_$SERVER.txt`
fi

echo "Backing up MySQL databases..."
for database in $DBS
do
mysqldump -h $HOST --user=$USER --password=$PASS $database > \
$BACKDIR/$SERVER-mysqlbackup-$database-$DATE.sql
gzip -f -9 $BACKDIR/$SERVER-mysqlbackup-$database-$DATE.sql
done

# if you have the mail program 'mutt' installed on
# your server, this script will have mutt attach the backup
# and send it to the email addresses in $EMAILS

if  [ $MAIL = "y" ]
then
BODY="Your backup is ready! Find more useful scripts and info at http://www.ameir.net"
ATTACH=`for file in $BACKDIR/*$DATE.sql.gz; do echo -n "-a ${file} ";  done`

echo "$BODY" | mutt -s "$SUBJECT" $ATTACH $EMAILS
        
echo -e "Your backup has been emailed to you! \n"
fi

if  [ $FTP = "y" ]
then
echo "Initiating FTP connection..."
cd $BACKDIR
ATTACH=`for file in *$DATE.sql.gz; do echo -n -e "put ${file}\n"; done`

ftp -nv <<EOF
open $FTPHOST
user $FTPUSER $FTPPASS
cd $FTPDIR
$ATTACH
quit
EOF
echo -e  "FTP transfer complete! \n"
fi

if  [ $DELETE = "y" ]
then
find $BACKDIR -name "*.sql.gz" -mtime $DAYS -exec rm {} \;

if  [ $DAYS = "1" ]
then
echo "Yesterday's backup has been deleted."
else
echo "The backup from $DAYS days ago has been deleted."
fi
fi

echo Your backup is complete!

Download backupmysql-2.sh



Trackbacks


28 08 2009
PingBack
Weblog: denajs.se
Tracked: Aug 28, 18:22
08 10 2009
PingBack
Weblog: www.syslogs.org
Tracked: Oct 08, 05:43

Comments

Display comments as (Linear | Threaded)
09 10 2006
#1 Rob Ludlow (Reply)

The MySQL backup script looks GREAT. I've been searching around and have found a few solutions, but none of them do exactly what I need. This script is very close, but is missing:

1) Option to have compression of the individual SQL files
2) Option to remove the local backup files (all other backup scripts I've tried leave a copy locally AND on the FTP server). I don't need to clutter my local server with backups if there are stored remotely.

Keep up the great work!
10 10 2006
#1.1 Ameir Abdeldayem (Reply)

Thanks for checking out the site. The coding might not look clear but this script actually does compress each individual SQL file. With this script you can also remove the local backups, keeping only the backups within the days you specify. For example, if you set the number of days to 3, the script will keep the backups from the last three days. You can set it to 1 to delete all previous backups and only keep the current day's backup. I have not implemented a way to remove old backups from the FTP server. Do you think that's a useful enough feature to look into implementing? Let me know what else is missing so I can make sure this script fits your and everyone else's needs.
10 10 2006
#1.1.1 Rob Ludlow (Reply)

Ameir,

Thanks for the quick reply, and thanks for the clarification! If your script does everything I just mentioned then I'm pretty sure you're all set! Now that I know it compresses the files and deletes the local copies I'm going to load it on my server.

Regarding the feature to delete FTP files: If it isn't hard to implement then I think it would be a cool feature, but definately not a requirement. My backup plan for my DB's has been to FTP all backups (using another MySQL backup app) to another host, and then use SyncBackSE to backup those files to my local maching (you can never have too much redundency).

For the past year I've been using a program called "MySQL database backup" which worked great for a year, but then I started having a TON of issues with it and the author is MIA.

Anywho... I'm going to give this a try. If it works out how I need it to, then I'd like to give it a good rating at hotscripts and I'd hope you have a "donate" link somewhere on the site!
10 10 2006
#1.1.2 Rob Ludlow (Reply)

Ameir,

So far it works like a charm! I've setup the script to backup my top 3 DB's and have tested having the script FTP to an offsite server and email to me. Both worked flawlessly.

For a novice (like me) it would be good to have some more detailed help / instructions, but not a huge deal. Right now my questions / suggestions are:

1) What exact CRON command do I use to run this script in CPanel? I've used "curl http://www.site/script.php to run my other backup scripts, but this script isn't in PHP and it isn't located in the public_html directory, but the /home/account/ directory.

2) Today's files say ...10-10-2006.sql.gz. Can I rename a file to test (right now) if it will really be deleted in 3 days? Maybe ...10-05-2006.sql.gz ?

3) I just noticed that you had tried phpMyBackupPro. I know why I didn't end up using it, but am curious why you ended up writing your own script.

4) The main thing almost all backup scripts like this are missing is basic instructions on how to RESTORE these files if something bad happens. Instructions via ssh and phpmyadmin would be excellent.

Well, just posting my general thoughts, use them however you like, but keep up the great work!
10 10 2006
#1.1.2.1 Ameir Abdeldayem (Reply)

Rob,

I am happy to hear that the script works flawlessly for you. I like hearing that my scripts help others out there besides me.

I should hurry up and finish up the README I've been updating off and on for a while, as it would answer a lot of your questions. Thanks for asking these questions; I'll certainly include those answers in the README when I get a chance.
Anyways, here are the answers you were looking for:

1) There are multiple ways to execute the script, but the easiest way for me is to call it by its absolute or relative filename. For example, if the script is in your home directory of your cPanel account, you can (in the cron module), call it by ~/backupmysql.sh. You can also call it by /home/user/backupmysql.sh. It's really a matter of preference in this case. It's also good that you don't have the script in a publicly-accessible folder like public_html.

2) If you rename the file to another date, it still will not be deleted by the script. The script deletes files based on the REAL time it was created/modified. If you want to test the deletion capability of the script, you can set the value to "1" and see if the previous backup gets replaced by a more current one.

3) Before I wrote this script, I thought phpMyBackupPro was a dream come true, and to an extent it is. It does just about everything I need, but the gripes I had with it are what compelled me to write this script. First off, it took long enough to setup and configure. Secondly, the backups were stored locally in a folder that was publicly accessible. I had to create .htaccess files for the export directory, and all that configuration stuff was more time consuming than I liked, especially since I needed to backup databases on multiple servers. That, combined with my interest to create such a script, made this script happen. Oh by the way, I also prefer the way dates are displayed with this script in emails and filenames, as opposed to a time hash.

4) I figured that if anyone was able to use cron and configure this script properly, they could find out how to restore their databases too. There's a lot of info out there on the web as well, so I figured I'd be writing a howto for nothing since no one would probably read it. It's certainly no trouble for me to write a howto on how to restore the DB's, and maybe I can write a script to ease this process.

Thanks a lot for your interest and comments, Rob. I hope the script works out well for you and I hope you tell others about it as well. If you have any problems or general questions please let me know.

BTW, thanks for your donation suggestion :-) I may add a donation link up sometime, but in the meantime, you can click on the Google ads at the right of the page. Thanks for your support.

--Ameir
11 10 2006
#1.1.2.1.1 Rob Ludlow (Reply)

Excellent! I've created a cron job to run in the wee hours of the morning. I can't believe how quick this script runs... just perfect.

RE DELETIONS: In my impatience I ran a test of the deletions by setting the days to 0... worked as expected. No local file, but FTPd to the server.

Also, I misspoke when I said I got email working. I guess I don't have "mutt" but I'm not going to backup via email anyway. Along those lines, how do I setup the script, the cron job, or CPanel so when the cron runs that an email isn't sent to my root account with the results of the backup?

No worries about creating something to ease the process of restoring a db. I'm no SQL expert, but I think this will work:
mysql -u USERNAME -p -D DATABASENAME < backupfile.sql

I think a fatal flaw many people make is they never CHECK that they can actually restore their backups. How would it feel to create backups for a year only to find out the backups can't be restored when you need them. I'd suggest you encourage people to run an "acid test" to make sure their solution really works (I know I'll be doing that soon with the backups from your script).
12 10 2006
#1.1.2.1.1.1 Ameir (Reply)

I'm glad to hear that the script works well for you. If you'd like to disable the emails that cron sends, you can add >& /dev/null at the end of the cron entry.
11 10 2006
#1.1.2.1.2 Rob Ludlow (Reply)

Sorry I'm totally monopolizing your site with my posts Ameir.

I totally screwed up one of my DBs (on purpose for the acid test) and then did the following:

1) "Dropped" all tables in the DB via PhpMyAdmin
2) Renamed the file and used cPanel's file manager to "Extract File Contents"
3) Via SSH ran the following: mysql -u USERNAME -p -D DATABASENAME < backupfile.sql
4) Checked the DB and everything was perfect.

A few notes:
1) Should I have done something different than "dropping" the tables or did I do the right thing? Also, what's the command to drop tables via SSH?

2) I couldn't extract the file contents with the existing name of the file (I'm guessing it was the .sql.gz). I renamed it to DB.gz and it extracted perfectly.
3) I was happy to discover I could use the root MySQL password and user to restore any DB... no more fumbling for a bunch of different user names / passwords.

Related to the name item above, if I had my choice I'd rename my backup files from:

hostname-mysqlbackup-DB-10-11-2006.sql.gz
to
DB-10-11-2006.gz (no dash, no mysqlbakup and no .sql)
12 10 2006
#1.1.2.1.2.1 Ameir (Reply)

Don't worry about the posting; I'm glad to see someone is interested in my work.

Since you are using phpMyAdmin already, you shouldn't have to worry about dropping tables or anything of the sort. All you have to do is select the DB you want to import data to, then click the "SQL" button. Here you will be presented with a box to type in SQL, and where you can also upload a .sql file. If your copy of phpMyAdmin isn't too old, the option to upload a compressed file is probably there. If this doesn't sound right, then your version of phpMyAdmin might have that option under the "Import" tab after selecting a DB. You shouldn't have to worry about dropping tables, etc.

If you don't have the option to upload a compressed file, you can extract the file contents and upload it to the "SQL" section of the DB. I don't know why your file could not extract under cPanel, but I just tried extracting my files on my cPanel server and it worked fine. The result was a file with the same name, but with a .sql extension. I'm not too sure what is going on at your end. If you want to extract the file in a linux shell, you can do so with the command gunzip. Just type gunzip filename.sql.gz or whatever the filename is.

In regards to the filename preference, I made this script to be friendly with multiple servers. I manage a few websites, and I like to keep track of what DBs belong to what server. If you only have one server, this is certainly not an issue, and you can change the script to fit your needs. Just change
CODE:
$BACKDIR/$SERVER-mysqlbackup-$database-$DATE.sql
gzip -f -9 $BACKDIR/$SERVER-mysqlbackup-$database-$DATE.sql

to
CODE:
$BACKDIR/$database-$DATE.sql
gzip -f -9 $BACKDIR/$database-$DATE.sql

and you should be fine.
12 10 2006
#1.1.2.1.2.1.1 Rob Ludlow (Reply)

Ameir, everything is working just fine now.

I changed the code as you have above and now the name is perfect and the file can now be extracted in cPanel (I must have messed it up before)!

The only question I have left is how to edit my setup so I don't get emails to my default email address when the cron job runs. Any thoughts?

Thanks again for your great work. I'll definitely be telling all my hosting friends!

BTW, it's a great thing that you have that pointer about opening/saing a the file in cPanel if it doesn't work. If you hadn't had that note I would have thought the script wasn't working and would have abandoned it.
12 10 2006
#1.1.2.1.2.1.1.1 Rob Ludlow (Reply)

I did a bunch of searching and finally think I found the answer:
I need to add the following to the end of my cron job:

>/dev/null 2>&1

"This line tells crontab to keep ALL output quiet and not email me "
14 10 2006
#1.1.2.1.2.1.1.1.1 Ameir Abdeldayem (Reply)

Thanks for your support Rob and thanks for telling others about this script. In addition to that, please vote for my scripts at hotscripts.com so others can see that this script is actually worth looking at. Is everything okay with your cron emails now?
17 10 2006
#1.1.2.1.2.1.1.1.1.1 Rob Ludlow (Reply)

Ameir,

It looks like the cron/email issue is fixed now, thanks.

I'll definately put in a review / vote on hotscripts!

A few general items:
1) I'm not receiving any emails from your blog software and I'm subscribed to all entries.
2) I shot you an email about a week ago with some additional info I didn't want on the blog, did you receive the email? If not, could you shoot me an email and I'll reply?
12 12 2006
#2 Guy (Reply)

Thanks Amir.

There is always somebody smarter than me out there, and today its you!
I googled for something like mysql cron shell script and found your excellent work.
Just the right balance of features and ease of use.
Good work!
07 07 2007
#3 manny (Reply)

Great script!!

After looking for a long time....found the perfect 10.

Keep up the good work.
01 09 2007
#4 Rob Ludlow (Reply)

Ameir,

Long time no write! It's been about a year using your script and everything is still working like a charm. I've now got a good size DB on www.BackYardChickens.com (250 mb) and your script compresses and FTP's the files like butter!

I spot check the backups from time to time (always a great idea when using an automated backup system) and everything was perfect.
01 09 2007
#4.1 Ameir Abdeldayem (Reply)

Hi Rob! Yes, it has been a long time hasn't it? I am very glad to see that you are still using my script and it is working well for you. I use this script at work as well with a fairly large DB and it is working great for us too. The only problem is that mutt doesn't do a great job if you want the DB emailed--but who can handle a 250MB attachment anyways?

Let me know if there's anything I can do to make the script better. I'm always looking for good things to code in my spare time.
30 09 2007
#5 Rob Ludlow (Reply)

Hey Ameir,

My DB has gotten pretty big. Even though the backups from your cool script are only 50 mb, my unzipped sql file is about 300 mb.

From time to time I like to do a restore just to make sure backups are functioning properly (highly recommended process).

Well, today I tried and it shot my load avg through the roof as my server worked to insert all of the millions of records.

The majority (probably 60%) of the data is in the search tables. Is there a way to either:

1) Backup everything in the DB except the search tables, or
2) Restore all of the tables except the search tables from the backup sql file

I'm guessing the former is probably easier than the latter.

Right now I'm using the following command to restore my DB:

mysql -u admin -p forumtest < mydb-09-30-2007.sql

Is there a magic way to edit that to ignore the search table?
11 10 2007
#6 Dman (Reply)

Great script not tested but it looks like i have found whatever i wanted
11 10 2007
#7 Dman (Reply)

can you write this in PHP?
10 11 2007
#7.1 Ameir Abdeldayem (Reply)

I can work on a PHP port, but this should also work by being called with PHP's exec() function. I'll test it out when I get some time and report back with the results.
10 11 2007
#8 Rob Ludlow (Reply)

Hi Ameir,

I got to thinking that it would be more effective to just exclude the search tables from my backups. These tables aren't necessary for the restore so I might as well exclude them.

Is there a way to use your script to only backup specific tables in a DB?
10 11 2007
#8.1 Ameir Abdeldayem (Reply)

This can definitely be done with a couple of changes in the code. Are you backing up a single DB or multiple? Also, do you want to exclude a single table or multiple tables? When I get some time I can make some changes to the script to allow this functionality.
10 11 2007
#8.1.1 Rob Ludlow (Reply)

Ameir,

That's a great question. I'm currently backing up 6 DB's with your excellent script. But only one of them has the tables I want to exclude. I was thinking about running two version of your script... the one that works perfectly and backs up 5 DB's in their entirety and another that is a hacked version and only backs up the non-search tables of my huge DB.

I have looked all over for a mysqldump command that allows me to exclude specific tables from the DB backup but couldn't find anything. I could only find the command to include specific tables:

mysqldump --add-drop-table -u user -p db table1 table2 table3 > /home/acct/forumdb.sql

One person suggested that I dump the DB, restore the DB into a copy, remove/drop the tables I don't want, then dump the copy DB and delete the files. This seemed like a huge drain on resources and I figured there had to be a better way.
10 11 2007
#8.1.1.1 Ameir Abdeldayem (Reply)

Hi Rob,

I also think it might be best to keep two versions of the scripts, one for a full backup, and another that backs up selected tables. You don't want to have a bunch of extra code in your script if you only need full backups.

Although mysqldump only allows you to include specific tables, in a script this becomes irrelevant. It's fairly easy to spit out a list of tables, and filter the list so only the ones you want to backup are shown. For example, the mysqlshow command can list all the tables of a DB, and all we have to do is grep -v the output to filter out the tables we don't want to backup.

CODE:
mysqlshow -u user -p pass "db_name" % | grep -v -e bigtable1 -e bigtable2


We still have to filter out the "pretty" output of mysqlshow to make it script-friendly. Don't worry, I hope to start working on this script over the weekend and will keep you updated with the progress.

Is there anything else in the current script that you think needs improvement?
10 11 2007
#9 Rob Ludlow (Reply)

Ameir, I've got a guy at work that I consider an expert programer. He suggested the same thing I think you are suggesting above, which is basically:

1) Get a list of all table names that are in the DB
2) Remove the names of the tables to be excluded from the list
3) Do a mysqldump of the remaining table names.

Regarding the status of the script... I can't think of a single thing that I'd like it to do more than it is currently doing. If I think of anything I'll definitely ping you!
10 11 2007
#9.1 Ameir Abdeldayem (Reply)

Hi Rob,

I came up with a script that should do the trick for you. It's a modified version of the previous script. I'll probably polish it a bit in the future in terms of comments and variable names, but it works on my end regardless. If you could test it out with your large DB I'd greatly appreciate it. Please let me know how it goes.

http://www.ameir.net/files/backupmysql-tbls.sh

Thanks,

Ameir
11 11 2007
#10 Rob Ludlow (Reply)

Ameir,

It worked PERFECTLY!! Here's what I did:

1) Edited the config settings. Set toggle to "on" and entered my 3 search tables
2) Ran the file in shell and saw it list out all the tables to be backed up (cool)!
3) Watched in my FTP program and noticed the DB was only 94 mb and the zip file was only 32 mb (vs the backup with the tables that was 53mb).
4) Dropped the tables from my test db
5) Unzipped the backup
6) Restored the backup.sql file into my test DB: It only too 3.5 minutes to restore. It was actually hanging / freezing on the restore in the past, so I'm super glad it restored so quickly!

Notes:
1) I don't think I mentioned this before, but I tried to ftp the file.sh to my server via ASCII and then BINARY and neither allowed me to run the script. Each time I tried I got: file.sh permission denied. The only way I could execute the script is by simply clicking edit/save via cPanel's file manager. Any ideas?

2) When I tried to gunzip the file (with the default name -my.....) I got the error that -y isn't a valid switch. I'd suggest you remove the "-" by default for the script when you do your cleanup.


I'm on my way to set this up as a cron job. Thanks so much!

(BTW, email me so I can start using you for contract work)!
28 12 2007
#10.1 Ameir Abdeldayem (Reply)

Hi Rob,

Sorry for taking so long to get back to you; I've been extremely busy with school :-( . I'm glad to hear that the script worked out well for you. I may make a couple of changes to it and offer it here; I'm sure it'll be useful to others.

In response to note 1), this is a common problem (if it's the same problem I experienced before). I'm assuming that you downloaded the script, changed the settings, and FTPd it to your server. The problem with this is that when you save it within Windows, the carriage returns/line feeds follow the DOS format. When trying to run the script in Linux, it isn't interpreted correctly. You can get around this by modifying the script within Linux or using a text editor within Windows that can maintain the Linux format. Also, make sure you do chmod +x file.sh on the server before trying to execute. One or both of these situations might be your problem.

Also, I'll debug the issue you mentioned in note 2), especially before putting it on the site officially. Oh, and if you need me to help you out with anything please let me know. Coding isn't my specialty but it can be fun (sometimes :-p ). Just email me if you ever need anything.

Thanks for the feedback!
24 07 2009
#10.1.1 Chris (Reply)

"Find" command is disabled for most hared hostings for security reasons.

Any alternative ?
26 07 2009
#10.1.1.1 Ameir Abdeldayem (Reply)

This can probably be done (in a convoluted way) with the stat command. You can create a function that uses stat -c %Y filename to find the time since Epoch, and delete files with an age greater than what you're looking for. It's definitely not nearly as convenient as the find command, but it can be done.

I guess I've been lucky by never having a host that's blocked the find command.

Let me know how it goes.
21 11 2007
#11 Shila (Reply)

Hi Ameir

Excellent script, just what I wanted. Before I test this script with my two very large databases (nearly 1 Gb each). Typically how long should it take to complete the backup and ftp to a remote site with e-mail notifications but no attachment!!

Currently I am using 'mysqldumper' application which does almost everything that your script does but the actual dumping process takes too long, generally about 1 and half hours for each database plus the ftp transfer time.

Thanks for making this script available
Shila
28 12 2007
#11.1 Ameir Abdeldayem (Reply)

Hi Shila,

The time it takes for the backup can widely vary based on many factors, so there's no real way to estimate how long a full database backup would take for you. I can tell you that that this script uses native MySQL functions to do the backup, so you can't really get more efficient than that. I Googled for "mysqldumper" but couldn't tell from their site if they use MySQL's mysqldump feature.

In short, this script will be as fast or faster than "mysqldumper," and the FTP transfer will take the same amount of time, providing that dumps from both programs are the same size. The best way to find out is to give it a shot ;-)

Let me know how it goes!
02 01 2008
#12 Rob Ludlow (Reply)

Ameir, I've reviewed your great script here: http://www.hotscripts.com/review/71564.html
20 01 2008
#12.1 Ameir Abdeldayem (Reply)

Thanks Rob! That's just what I needed! Excellent review! Thank you for all the kind words. I'm glad the script helps you out, and thanks to your review hopefully others will be able to make use of it as well. Thanks again!
05 01 2008
#13 Tom (Reply)

Great script but the resulting sql.gz file is always missing data !!

If I do an SQL dump and view the data everything is there, but when I open the downloaded file created by this script data is missing.

How can I fix this ??
20 01 2008
#13.1 Ameir Abdeldayem (Reply)

That's interesting to hear. I've never encountered this issue and don't know of anyone who has. What commands are you using to perform the SQL dump? This script uses native MySQL commands to perform the dump, so I don't see where the difference could be. Report back with the command you use and I'll be sure to assist you where I can. Thanks for your interest.
19 01 2008
#14 Hussein (Reply)

Wonderful script Ameir, thank you for sharing.

I needed to back up 3 tables from 1 database and the script you revised for Rob above seems to work perfectly.

I've disabled both mail and ftp, and currently only backing the tables locally. that's all I need at the moment, and the script does it beautifully. Great job
20 01 2008
#14.1 Ameir Abdeldayem (Reply)

Thanks for your comment Hussein. I'm glad to hear that the script is working well for you. If you find any bugs in that version please do let me know; I might publish that copy sometime in the near future and it'd be great to hear about any bugs in it. Thanks again, and be sure to rate it at http://www.hotscripts.com/Detailed/71564.html if you think it's worthy.
10 02 2008
#15 Norm (Reply)

Ameir, great script and can't wait to use it. Just one thing, have you thought of adding a directory list include. So we could include a directory (or three) into the backup?

Would be very nice.

Cheers
08 03 2008
#16 Julien (Reply)

Congrats for this script! The backup works fine for me, but the email doesn't get sent, although I have mutt installed (it's there by default on my shared hosting server). Does mutt need some special configuration?
08 03 2008
#16.1 Ameir Abdeldayem (Reply)

It could be that your email service is not accepting the emails from mutt. Mutt oftentimes sends emails with a from name that's not valid (like server.local). What service are you using? Try this via command line:
echo "test" | mutt -s test email@yahoo.com email2@gmail.com

This will send an email via mutt to multiple addresses. Try a Gmail one if you can, as they accept just about everything. Let me know how this goes for you; perhaps just changing the email address will do the trick.
24 03 2008
#17 Joe Jebara (Reply)

Many Many thanks for this script , just what i needed and works Perfect !

one little question :
when the MAIL=y is it supposed to email the file only and not write to the backup directory?
because it email the files to me but i see nothing in the backup directory which is fine for me , i just want to make sure it is supposed to work this way !



Shukran ameir
30 03 2008
#17.1 Ameir Abdeldayem (Reply)

Hi Joe,

I'm glad the script is working well for you.

When MAIL=y, a backup of the databases should be in the backups directory. It is very strange for it not to be there but still be emailed the backup, as the script looks in the backup directory in order to send the email. Can you check your backup directory path once more just to confirm this? Also, if your backup path has a "~" in it, make sure that you are looking in the correct user's folder. It's peculiar for you not to find it, but I can assure you that the backups are somewhere on the drive, as the script can find it fine.

-Ameir
31 03 2008
#17.1.1 Joe jebara (Reply)

Yes you were right, i just left the backup directory as ~/backup instead of ~/public_html/backup and as you can see the backup directory was created above the public_html folder and all backups were there.

Thanks
25 03 2008
#18 Rob Ludlow (Reply)

Hey Ameir,

Can't believe it's been about 1.5 years since I first found this great script. My DB is huge and the edit you made to slim it down (by avoiding the search tables) is working like a charm.

Today I did a sanity check and dropped all my tables, unzipped a backup file from a day ago, and restored it. Worked PERFECTLY!

I also linked to you from my blog with details on my backup process:

http://www.nifty-stuff.com/daily-mysql-database-backup.php

Great work as always and thanks again for such an excellent script!
30 03 2008
#18.1 Ameir Abdeldayem (Reply)

Hi Rob,

As always, thank you for your kind words. I really like your article; I actually wrote this and another backup script because I ran into a similar situation, but where I lost EVERYTHING (my host disappeared into thin air)! Hopefully others will learn from our experiences. I send my backups to several email accounts from different services, as well as FTP them over to a filesharing service that promises to never delete files (bitroad.net). So far, so good!

Thanks again for your support (and the link)! Nice theme btw.

-Ameir
12 05 2008
#19 midshipman (Reply)

This is the single most useful script I used on my linux machine in the last months. Thank you so much for sharing it with the rest of us!
27 09 2008
#20 Original Sin (Reply)

Ameir, excellent work. EXCELLENT!
29 09 2008
#20.1 Ameir Abdeldayem (Reply)

I'm glad my script was helpful to you. Thanks for your input!
29 09 2008
#21 Dustin Debris (Reply)

I had an issue with the script that I'm hoping you can explain.

On line 116, variable ATTACH is defined. Character 47 and 48 were the echo switch "-e". I have no idea what this is, but apparently my shell (ubuntu gutsy) doesn't think that it's a valid switch, and instead dumped it as part of the ATTACH string.

The result was that your script was sending "-e put etc..." to the FTP server, which it didn't like and couldn't interpret. Then iyour script reported all was well, which it wasn't.

By removing the -e switch (I honestly haven't even looked it up to know what it is) on a whim, I found that the script fires PERFECTLY!

Thanks again for such a great tool!
29 09 2008
#21.1 Ameir Abdeldayem (Reply)

That's strange, as I'm using this script on Ubuntu Hardy Server without any issues. The "-e" switch to echo causes it to interpret backslash escapes, such as the "\n" on the line with ATTACH. By removing the -e, you (should) get "\n" printed out. It may be something specific to the version of echo you're using, although that sounds a bit unlikely. I'm glad it's working fine for you now, though.
You're correct about the error detection; I need to catch those as they come up. I might just whip up another version with a couple of tweaks here and there in the coming weeks.
Thanks for trying out the script!
29 09 2008
#21.1.1 Dustin Debris (Reply)

I stuck

echo "$ATTACH"

After echo FTP transfer completed... and before fi
It echoes

"put localhost-mysqlbackup-pointofsale-09-29-2008.sql.gz"

Then there's a blank line..
and then "The backup from 120 days ago has been deleted."

Not sure if that blank line is meant to be there or if it's the \newline...
Can you clarify, knowing your script inside out?

At any rate, that was on my development machine. I installed it on one of my live units about an hour ago (ubuntu Gutsy as well) and had to do the same thing. I've got one more machine to get it running on downtown tomorrow (gutsy as well), I'll let you know if it's 3 out of 3 :-)
30 11 2008
#21.1.1.1 Ameir Abdeldayem (Reply)

Hi Dustin,

I'm assuming you put echo "$ATTACH" there just to see what happens, which is cool. The blank line is actually intended to be there, and is indeed due to the newline character in the script.

Are you having any issues getting the script working correctly?

-Ameir
17 10 2008
#22 Daniel Sousa (Reply)

Hi, your script looks great, but I'm getting a time out error on the FTP connection. Does anyone have any idea of what can be causing this?
30 11 2008
#22.1 Ameir Abdeldayem (Reply)

Hi Daniel,

Are you sure that you're able to connect to your FTP server without issues? If you are, you might want to try passive mode FTP in the script (change ftp -nv to ftp -pnv). Make sure all the settings are correct, including port numbers. If you continue to have issues, let me know. It may help to try connecting to your FTP server via the Linux commandline to see if you have issues as well.

-Ameir
22 10 2008
#23 Adrian Nadeau (Reply)

This script is great! Thank you so much. This is exactly what we were looking to do. I love the FTP part as I've found many to automate backup but none to actually move the backup file somewhere. Thank you again so much!

Adrian
23 10 2008
#24 James (Reply)

Thank you. This was the first script that I found that actually worked to backup my database, and I've looked through several... I've tried a few written in PHP, and I was getting errors trying to send them about there being no remote address, and other sh scripts that had syntax errors (and I'm not good at sh).

Again, thank you very much :-)
28 11 2008
#25 Vishal Rao | Stress Management Forum (Reply)

The script is working fine but sending the backups through FTP to another server is not working. Here's the log after initial logging:

Using binary mode to transfer files.
250 OK. Current directory is /database_backups/wahf
?Invalid command.
221-Goodbye. You uploaded 0 and downloaded 0 kbytes.
221 Logout.
FTP transfer complete!
30 11 2008
#25.1 Ameir Abdeldayem (Reply)

Hi Vishal,

What type of FTP server are you trying to connect to? I'm thinking that you have a stripped-down FTP server (like what many hosts provide) that doesn't offer many commands beyond put/get. Maybe if you delete the line cd $FTPDIR you'll have some luck. Let me know what turns up.

-Ameir
01 12 2008
#25.1.1 Vishal | Stress Management Forum (Reply)

I'm using ftp server provided with my hosting account. I deleted the cd $FTPDIR line but no luck. I think the ATTACH line is not working. Any reference for a good and cheap FTP server?

Thanks,

Vishal
26 12 2008
#25.2 Dustin Debris (Reply)

Vishal,
Your log is identical to mine before I ditched the -e switch in the echo line.
Look up line 116, character 47 and 48. Should be "-e"

Remove those 2 characters and try again.
17 12 2008
#26 Tesna (Reply)

Thanks for the script! Very useful and simple! I've modified your script to allow the files compressed in tar.gz format to allow better compression than zip.

I also modify the script to split the backups files before sends them via email since my google apps only accepts messages not larger than 20 MB.

I'm using email accounts hosted on google apps to store my backup files :-)

To see the modified script you can see my homepage :-)
13 03 2009
#27 Elessar (Reply)

Hi! Thanks for the script! Its great!
i have opinion, can you modify script to delete backup file on remote ftp server, after one, two or tree day, such as first server, when delete option is selected? because my ftp server space is going to full very soon after run this script and i must delete older backup form ftp server manually every day.
tnx.
22 03 2009
#27.1 Ameir Abdeldayem (Reply)

Thanks for your comment. That does sound like a great idea. I don't currently know of a way to delete files over FTP based on their age without using a more advanced FTP program (perhaps ncftp or lftp). If you don't mind installing an extra program, then this can likely be done--I doubt it can be done otherwise. I'll try to look into this and let you know.
05 04 2009
#28 Cagri Ersen (Reply)

Hello,

It's a great script.

I've add "prompt" paramater to FTP upload section for disabling Interactive mode. Becouse the script didn't upload the backups when interactive mode is enabled on the server.
11 04 2009
#29 Rob Ludlow (Reply)

Hey Ameir,

It's been 2.5 years (time flies) and my DB has gone from 300 MB when I first contacted you to now being about 2gb!

The ability to exclude certain tables has made ALL the difference in the size of the backup files, but also how long it takes to restore my monster DB.

This script has saved my butt MANY times and makes my ability to restore DB's to production or to test environments super easy and fast.

Thanks again!
25 04 2009
#29.1 Ameir Abdeldayem (Reply)

Hey Rob,

Wow! That's a huge increase in filesize! Sounds like your site is prospering.

I'm glad to hear the script is still working out well for you. I too am using the same script (albeit with a couple of changes here and there) on several servers I deal with. It's certainly helped me out tons as well.

I'm glad the script has been able to help others as it's helped me.

Good to hear from you again.

-Ameir
14 06 2009
#29.1.1 Fred (Reply)

Hello Ameir,

I've run across this page via google. I run a vbulletin site that has a large database and want to find a way to automatically back it up every 3 days or so.

This seems to do what I need but being a newbie to this kind of stuff, I have no idea how to set it all up?

Can you give me a step-by-step? Do I put upload the .sh code you gave above to the host server (a level above public html)?

I'm unsure if you're familiar with vbulletin, but if so, what settings do I need to do in there? I keep seeing "CRON" mentioned but I'm unfamiliar with what it is.

Thanks for any info / walk-through you can provide.
17 07 2009
#29.1.1.1 Ameir Abdeldayem (Reply)

Hi Fred,
You're definitely on the right track. This script should work fine so long as Vbulletin is using MySQL.

You should find out if your server supports CRON (which is an automatic scheduler). If not, you'll either have to initiate the backups manually through a shell, or upload the script to a cgi-bin or something of the sort and run the script through a web browser. I doubt my reply is detailed enough, so let me know where you're stuck and I'll try my best to help.

-Ameir
22 05 2009
#30 satla zone (Reply)

Hello Ameir,
Your script is very useful for me but only one thing is annoying me.
I want to tar gzip the backup and not only gzip.
What should I change in your script then?
17 07 2009
#30.1 Ameir Abdeldayem (Reply)

Hi Satla,
This change is pretty easy. I tested it and it takes a tad bit longer to compress and the filesizes are slightly larger, so keep that in mind. All you have to do is change the for loop to:
CODE:
echo "Backing up MySQL databases..."
for database in $DBS
do
        mysqldump -h $HOST --user=$USER --password=$PASS $database > \
        $BACKDIR/$SERVER-mysqlbackup-$database-$DATE.sql

        cd $BACKDIR
        tar cvfz $SERVER-mysqlbackup-$database-$DATE.sql.tgz $SERVER-mysqlbackup-$database-$DATE.sql
        rm $BACKDIR/$SERVER-mysqlbackup-$database-$DATE.sql
done

and change all .sql.gz references to .sql.tgz or .sql.tar.gz (your preference). Hope that helps!
04 09 2009
#30.1.1 Brandon H (Reply)

Hi - I too am new to this so I am going to test it out when I get a chance. I am really trying to find a reliabel database and file backup system (automatic). I was wondering if there was a way to include file backups into the script and include the same backup features for FTP to remote server?

Thanks!
04 09 2009
#30.1.1.1 Ameir Abdeldayem (Reply)

Hi Brandon,

Check out this post:
http://www.ameir.net/blog/index.php?/archives/15-Folder-Backup-to-FTP-and-Email-Shell-Script-for-Cron-v2.html

That script backs up whatever directories you specify (and its subfolders). I personally use that and the MySQL backup script, and everything has been working well for years.

-Ameir

Add Comment


Enclosing asterisks marks text as bold (*word*), underscore are made via _word_.
Standard emoticons like :-) and ;-) are converted to images.
BBCode format allowed

To prevent automated Bots from commentspamming, please enter the string you see in the image below in the appropriate input box. Your comment will only be submitted if the strings match. Please ensure that your browser supports and accepts cookies, or your comment cannot be verified correctly.
CAPTCHA