What is MyBack
MyBack is a fully-automated backup solution I’ve implemented recently. After running for years now with mostly manual-triggered, non-incremental and redundant backups I reached the point where I was finally willing to invest some time for a proper and appropriate solution that can run in the background on a daily base. I also wanted a way to backup my several VirtualBox images, incrementally as well.
Requirements
- Ubuntu (tested with Kubuntu 12.10)
- rsync
- libnotify-bin
- The partition of the backup destination directory must support hard-links. See here for a list of supported file storage systems (like ext3, ext4, NTFS): http://en.wikipedia.org/wiki/Comparison_of_file_systems#Features
Installation
- Download the tarball from here or use your terminal
wget http://blog.cwill-dev.com/downloads/myback/myback.tar.gz
- Open a terminal and make yourself sudo
sudo su
- Move downladed tarball to /opt/
mv ~/Downloads/myback.tar.gz /opt/
- Navigate to /opt/ and unpack the file
cd /opt/ tar -zxvf ./myback.tar.gz
- Make sh-files executable
cd /opt/myback find . -name '*.sh' -exec chmod +x \{\} \;
Configuration
Execution parameters
There is only one file you have to configure for MyBack to do its job – namely the file that calls the application’s main script with the individual parameters that fit to your system. So open the exec.sh file now
nano /opt/myback/exec.sh
You will find following one-liner:
/opt/myback/app/myback.sh -s -d /media/BACKUP_STORAGE / # |---------- 1 --------] [2] [3] [-------- 4 --------] [5]
1: The path to the myback.sh main script
2: If -s is specified, the script will support incremental VirtualBox backup (see chapter Insights->VirtualBox)
3: -d is required and must be followed by the destination folder
4: The destination folder, into which the script will backup
5: The folder to backup (in this case “/”, so root – the entire system (see chapter excludes / includes)
When you are done press [CTRL][O] followed by [CTRL][X] to save your settings.
Of course the destination folder must be mounted and accessible.
Partitions
If you have mounted some of the system directories to several partitions – for instance /opt or /home – then you have to specify those folders explicitly (even you think “/” might contain these folders already) . This is based on the underlying rsync application, which will not copy the data inside those folders, in fact it will just create those folders but not copy the files.
This is an example in case you have both /opt and /home on different partitions
/opt/myback/app/myback.sh -s -d /media/BACKUP_STORAGE / /home /opt
Excludes / Blacklist
As already pointed out MyBack uses a blacklist for excluding several files and directories. This blacklist is located in the root directory of the unpacked myback folder. So open this file:
nano /opt/myback/excludes.txt
The provided blacklist covers the default case in which the backup script gets to backup the entire system (“/”).
Cache, tmp or redundant folders are filtered automatically. Also some default applications – like GIMP, DropBox, Firefox or Thunderbird - got considered already. For the latter one MyBack will not backup the emails, only the account settings. I made this decision to save storage and for better performance, since the common case is that emails are kept on the server anyway.
Usually there should be no need to edit these settings. But if you are interested in adjusting the backup structure individually to fit to your system’s individual configuration have a look at these pages:
http://rsync.samba.org/ftp/rsync/rsync.html
http://programmersnotebook.wordpress.com/2010/03/20/rsync-and-exclude-from
http://itefix.no/i2/content/excluding-directories-cwrsync
http://serverfault.com/questions/150269/complex-includes-excludes-with-rsync
http://stackoverflow.com/questions/13659202/rsync-complex-filter
So if you backup more than one directory (see chapter partitions above) be aware of this certainty.
Run the backup
For that we have already configured the execution parameters and the blacklist we just have to run the exec.sh script
/opt/myback/exec.sh
Now lean back, the initial backup may take some time. The output will be printed to the terminal.
Configure Cronjob
Since a backup does not make much sense if it’s not running automatically frequently, we should set up a cronjob now.
This gets done by using Ubuntu’s crontab.
crontab -e
At this point we can tell the system on what time and frequency the backup should get executed.
add 30 15 * * * /opt/myback/exec.sh
In this example the backup will run every day at half past three afternoon.
For further information you might want to check crontab’s manpage:
http://unixhelp.ed.ac.uk/CGI/man-cgi?crontab+5
See also http://askubuntu.com/questions/23009/reasons-why-crontab-does-not-work
Recovery
Since MyBack backups all required system data and information (when running with “/”) the recovery procedure is pretty simple, yet a long time process.
- Install plain Ubuntu on the new system
- Mount the backup drive
- Run the recovery script
/media/BACKUP_STORAGE/current/_myback_/restore/restore.sh
The recovery script does following things:
- Replace package information
- Add repository keys
- Update source lists
- Restore and install packages
- Restore system
- Reboot
See this thread: http://ubuntuforums.org/showthread.php?t=2133648
Reading package lists... Done Building dependency tree Reading state information... Done You might want to run 'apt-get -f install' to correct these. The following packages have unmet dependencies: firefox-globalmenu : Depends: firefox (= 20.0+build1-0ubuntu0.12.10.3) but it is not installed E: Unmet dependencies. Try using -f.
So the only solution to deal with this is to remove the following packages from the generated apt-installed.lst (/BAK_DESTINATION_DIR/current/_myback_/recover)
- firefox-globalmenu
- kubuntu-firefox-installer
- firefox
Insights / Advanced
VirtualBox
If the backup script gets called with the -s parameter MyBack will backup all VirtualBox images of ALL user’s as well, incrementally. This is done by creating a snapshot of each VM. If a VM is running on the time of the backup the VM will be paused for the time of the snapshot creation.
This behaviour will lead to a huge list of snapshots after a while. So you may want to remove/merge those snapshots frequently. Either you do this manually in the VirtualBox-UI, or you can use the script I provided:
/opt/myback/tools/vbox_merge_snapshots.sh
Notification
The script uses notifiy-send to notifiy the desktop environment about the status of a backup. This will also work if the backup got called by crontab. The notification will look like this:
Logs
MyBack will copy its terminal output to the destination backup directory. It will be located in the directory
/BACKUP_DIR/current/_myback_/log/myback.log
MyBack
- MyBack uses the hardlink-feature of rsync. So even your file browser will show all files in each backup-folder, this does not mean that they exist multiple times.
- MyBack will copy itself to the destination backup directory as well. All of its files will be located under
/BACKUP_DIR/current/_myback_/ - The structure of the backup directory will be as follow:
As you can see, MyBack creates for each backup an individual directory, named by the date of execution.
The soft link current will always lead to the last created backup. - All basic files, that will be required to start the recovery of your system will be stored here as well (ie sources.list)
Restore
I went through several hardly failures during the recovery process. As you can read everywhere there should be two possibilities to backup installed software and to re-import those on the new system – but both do not work!
dpkg –get-selection and dpkg –set-selection
dpkg --clear-selections dpkg --set-selections installed-packages.lst apt-get update apt-get dselect-upgrade
apt-mark showauto showmanual and auto manual
apt-mark auto $(cat ${BAK_SOURCES}_myback_/restore/pkgs_auto.lst) apt-mark manual $(cat ${BAK_SOURCES}_myback_/restore/pkgs_manual.lst)
So the only way that did the jpb properly was to apt-get install each package automatically one by one. Maybe in the upcoming Ubuntu release this problem might be fixed.
Code
myback.sh
#!/bin/bash # Copyright (c) 2013 Christopher Will<dev@cwill-dev.com> # # Permission is hereby granted, free of charge, to any person obtaining a copy of # this software and associated documentation files (the "Software"), to deal in the # Software without restriction, including without limitation the rights to use, # copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the # Software, and to permit persons to whom the Software is furnished to do so, subject # to the following conditions: # # The above copyright notice and this permission notice shall be included in all # copies or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, # INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A # PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT # HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. ##################################################################################### # # VARIABLES # ##################################################################################### # Script name and version app='MyBack v0.1beta' # Setting display properties for the desktop notifications DISPLAY=":0.0" export DISPLAY # File defining the backup excludes MYBACK_BASEDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )/../" && pwd )/" EXCLUDE_FILE=${MYBACK_BASEDIR}'excludes.txt' # Will contain all users with a home directory (used for impersonification later on) declare -a users # Virtual Box support # If set (see --help) then the script will search for all available VirtualBox images # of all users. For each image snapshots will get created, so the backup just uses # incremental changes instead of the complete ie VHD virtual_box_support=false # Specify the root directory in which rsync will store the backup (with $date as sub- # directory). The given storage must be mounted correctly. # # As advice, the destination directory should not be located on the same device where # the data to backup is located. Because in case of hardware failure you will then # not be able to access your backups. Better prefer an external storage. # Example: /mnt/myBackupDevice/rsync dest_dir='' # This array will contain all positional arguments (=source directories) specifying # each directory that has to get considered during the backup process. # In a default and simple environment it should be enough to specify the personal # home directory. But it may also list any other directories. # See the myback_excludes for a default case dealing with a root (/) source directory. declare -a source_directories # Will contain the temporary log file (which gets copied to the destination myback # directory afterwards) tmp_log_file='' # We use the current date for each backup as sub-directory-name within the given # $dest_dir date=`date "+%Y-%m-%dT%H:%M:%S"` ##################################################################################### # # FUNCTIONS # ##################################################################################### # Initialize basic variables. ##################################################################################### function init() { # Logfile # Redirect stdout & stderr to tmp logfile # Gets copied later to the destination directory. tmp_rand=$(cat /dev/urandom | tr -cd [:alnum:] | head -c 4) tmp_log_file="/tmp/myback_"$date"_"$tmp_rand".log" exec > >(tee -a ${tmp_log_file}) exec 2> >(tee -a ${tmp_log_file} >&2) exec 2>&1 # Set user list to $users # Gets used for snapshotting the virtual box images as well for the # UI notification tmp_users="$( getent passwd | grep /home/ | cut -d ':' -f 1 )" for u in ${tmp_users} do tmp_dir="/home/${u}" if [ -d "$tmp_dir" ]; then users+=(${u}) fi done } # Displays a notification baloon in desktop environment for each user # Args # 1 - String - Title # 2 - String - Content # 3 - String - System icon (ie dialog-ok, dialog-error, dialog-information etc) ##################################################################################### function display_notification() { icon_dialog="dialog-information" if [[ -n "$3" ]]; then icon_dialog=${3} fi for ((i=0;i<${#users[@]};i++)); do $(su -c "notify-send -u critical \"${1}\" \"${2}\" --icon=${icon_dialog}" -s /bin/sh ${users[i]}) done } # Exits script with failure code 1 and displays a desktop notification of type "error" # as well. # Args: # $1 - String - Error message ##################################################################################### function failure() { echo $1 display_notification "MyBack - Error occurred" "${1}" "dialog-error" exit 1 } # Prepares the script arguments. ##################################################################################### function setup_args() { while getopts "h?sd:" opt; do case "$opt" in # help h|\?) echo $app echo 'Simple yet effective rsync backup wrapper' echo '' echo '[OPTIONS] [DIR1[DIR2[DIRn..]]]' echo '' echo 'Options:' echo '-h Show brief help' echo '-d Backups destination directory' echo '-s Virtual Box support' exit 0 ;; # destination directory d) if [[ -n "$OPTARG" ]]; then dest_dir=${OPTARG%/} fi ;; # virtual_box_support s) virtual_box_support=true ;; esac done # Reset argument index pointer for accessing positional parameters shift $(( OPTIND-1 )) # Iterate through each positional parameter and add it to the source_directories # array for var in "$@" do # Do not remove trailing slash if root directory (/) if [ ${var} != '/' ]; then var=(${var%/}) fi source_directories+=(${var}) done } # Validates all variables ##################################################################################### function validate { # Check whether this script got called as super user. If not, we can not # proceed. if [ "$(whoami)" != "root" ]; then failure 'Aborting: You must be root to execute this script.' fi # Check dest_dir to be valid if [ ! -d ${dest_dir} ]; then failure 'Aborting: Destination directory "'${dest_dir}'" does not exist. Maybe it is not mounted?' fi # Check exclude file if [ ! -f ${EXCLUDE_FILE} ]; then failure 'Aborting: Exclude file "'${EXCLUDE_FILE}'" does not exist.' fi # Check source directories to be set if [ ${#source_directories[@]} == 0 ]; then failure 'Aborting: No source directories specified.' fi # Check source directories to be valid for ((i=0;i<${#source_directories[@]};i++)); do if [ ! -d ${source_directories[i]} ]; then failure 'Aborting: Source directory "'${source_directories[i]}'" does not exist. Maybe it is not mounted?' fi done } ##################################################################################### # # STARTUP # ##################################################################################### # Hello World echo '========================================' echo '= THIS SCRIPT COMES WITH NO WARRANTY =' echo '= Bash version '${BASH_VERSION}' =' echo '= [CTRL] + [Z] to kill process =' echo '========================================' # Do some basic initialization init # Notify UI process_start_date_time=`date "+%Y-%m-%dT%H:%M:%S"` display_notification "MyBack - Backup started" "Backup process started at ${process_start_date_time}" # Handle input parameters setup_args "$@" # Validate environment validate # Convert source_directories to string (used as rsync parameter) source_directories_string='' for ((i=0;i<${#source_directories[@]};i++)); do source_directories_string+=${source_directories[i]}' ' done # The main symlink pointing to the latest (this) backup latest_backup_dir=${dest_dir}/current # The full path to the backup destination directory. Keep hierarchies. current_bak_dest_dir=${dest_dir}/${date}/ # Do not set the link-dest argument on the first run, since there will be no # referencing sub-directories within the "latest backup" directory yet. link_dest='' if [ -e ${latest_backup_dir} ] then link_dest='--link-dest='${latest_backup_dir}/ fi # Create the directory hierarchy for the current backup # Create myback special folder MYBACK_DEST_DIR=${current_bak_dest_dir}_myback_/ # Display working target directory echo '> Timestamp: '${date} echo '> Sources: '${source_directories_string} echo '> Destination: '${dest_dir}/${date} echo '> Logfile: '${MYBACK_DEST_DIR}myback.log if ${virtual_box_support}; then echo '> VirtualBox support enabled' else echo '> VirtualBox support disabled' fi echo '----------------------------------------' ##################################################################################### # # VIRTUALBOX # Create snapshots of each installed VM, so rsync is able to backup the VirtualBox # images incrementally # ##################################################################################### if ${virtual_box_support}; then echo '> Creating VirtualBox snapshots' # Do snapshots for all users for ((i=0;i<${#users[@]};i++)); do echo "> > User: ${users[i]}" # Get all installed VMs vms=$(su -c "VBoxManage list vms | sed -E 's/^\"(.*)\".*/\1/g'" -s /bin/sh ${users[i]}) # Create new snapshot of each for vm_name in $vms; do echo "> > > Processing ${vm_name}.." $(su -c "VBoxManage snapshot ${vm_name} take ${date} --description \"Generated by myback automatically at {$date}\" --pause" -s /bin/sh ${users[i]}) echo "> > > [DONE] ${vm_name}" done done echo '> [DONE] Creating VirtualBox snapshots' fi ##################################################################################### # # RSYNC # ##################################################################################### # Let the user know which directory we are currently process echo '> Backing up '${source_directories_string} # Call rsync - this is the main logic! # link_dest - Is only set if the initial backup was processed already # current_bak_dest_dir - The backup location (of structure destination/date/hierarchy) rsync -axPv --exclude-from ${EXCLUDE_FILE} ${link_dest} ${source_directories_string} ${current_bak_dest_dir} # Keep symlink up to date if [ -e ${current_bak_dest_dir} ] then rm -f ${latest_backup_dir} # Remove ln -s ${dest_dir}/${date} ${latest_backup_dir} # Re-create fi ##################################################################################### # # RESTORE PREPARATION # We copy all files that will be required during the recovery to the destination # folder as well. # ##################################################################################### # Create directories mkdir -p ${MYBACK_DEST_DIR} mkdir ${MYBACK_DEST_DIR}'restore' mkdir ${MYBACK_DEST_DIR}'restore/apt' mkdir ${MYBACK_DEST_DIR}'log' mkdir ${MYBACK_DEST_DIR}'app' mkdir ${MYBACK_DEST_DIR}'tools' # Export list of all installed applications and repository keys apt_dir=${MYBACK_DEST_DIR}'restore/apt/' cp /etc/apt/sources.list ${apt_dir}'sources.list' if [ -f '/etc/apt/apt.conf' ]; then cp /etc/apt/apt.conf ${apt_dir}'apt.conf' fi if [ -f '/etc/apt/preferences' ]; then cp /etc/apt/preferences ${apt_dir}'preferences' fi cp -R /etc/apt/sources.list.d/ ${apt_dir} cp -R /etc/apt/apt.conf.d/ ${apt_dir} cp -R /etc/apt/preferences.d/ ${apt_dir} cp -R /var/lib/apt/lists/ ${apt_dir} apt-key key exportall > ${MYBACK_DEST_DIR}'restore/repositories.keys' # Solution 1: dpkg selectiom -> Buggy (as of 12.10), re-import won't work dpkg --get-selections > ${MYBACK_DEST_DIR}'restore/installed-packages.lst' # Solution 2: apt-mark -> Buggy as well (as of 12.10) apt-mark showauto > ${MYBACK_DEST_DIR}'restore/pkgs_auto.lst' apt-mark showmanual > ${MYBACK_DEST_DIR}'restore/pkgs_manual.lst' # Solution 3: Manual approach -> Create list of packages to re-install one by on recovery package_list=$(dpkg-query -Wf '${Package} ') package_list=${package_list// /$'\n'} # change the semicolons to white space for package in $package_list; do echo "$package" >> ${MYBACK_DEST_DIR}'restore/apt-install.lst' done # Copy tmp logfile to current destination directory mv ${tmp_log_file} ${MYBACK_DEST_DIR}"log/myback.log" # Copy myback system files to backup as well cp -R ${MYBACK_BASEDIR}'app/' ${MYBACK_DEST_DIR} cp -R ${MYBACK_BASEDIR}'tools/' ${MYBACK_DEST_DIR} cp -R ${MYBACK_BASEDIR}'restore/' ${MYBACK_DEST_DIR} cp ${MYBACK_BASEDIR}"excludes.txt" ${MYBACK_DEST_DIR}'/' cp ${MYBACK_BASEDIR}"exec.sh" ${MYBACK_DEST_DIR}'/' ##################################################################################### # # FINISH # ##################################################################################### # Notify UI (multiline intended) process_end_date_time=`date "+%Y-%m-%dT%H:%M:%S"` finish_msg="Backup process finished at ${process_end_date_time} Destination directory: ${current_bak_dest_dir} Logfile: ${MYBACK_DEST_DIR}log/myback.log" display_notification "MyBack - Backup finished" "${finish_msg}" "dialog-ok"
excludes.txt
# ======================================================= # This is the exclude file for the rsync backup. # # Notice: # All path entries are interpreted relative to the source # directories. # ======================================================= # Universal excludes ######################## lost+found ld.so.cache *.log *.bak # Root file system ######################## - /bin/ - /boot/ - /cdrom/ - /dev/ - /etc/modules.conf - /lib/ - /lib32/ - /lib64/ - /media/ - /mnt/ - /proc/ - /run/ - /sbin/ - /selinux/ - /sys/ - /tmp/ + /usr/ + /usr/share/ + /usr/local/ + /usr/local/share/ - /usr/local/* - /usr/* - /var/cache/ - /var/crash/ - /var/lock/ - /var/log/ - /var/run/ - /var/tmp/ - /var/lib/sudo/ - initrd.img.old - initrd.img - vmlinuz - vmlinuz.old # Filters for home dirs ######################## # Cache - /home/*/.cache/ # Downloads - /home/*/Downloads/ # Dropbox - /home/*/Dropbox # Temporary files / cache - /home/*/.local/share/Trash - /home/*/.cache - /home/*/.Trash # X Windows System - /home/*/.xsession-errors* # Several ######################## # Exclude backup text files - *~ - \#*\# # Commonly distributed Mac OS X cache - .DS_Store # Commonly distributed Windows cache - Thumbs.db # Common Applications ######################## # Adobe Reader - /home/*/.adobe/**/AssetCache/ - /home/*/.adobe/**/Cache/ - /home/*/.adobe/**/Temp/ - /home/*/.adobe/**/UserCache.bin # Dropbox temp stuff - /home/*/.dropbox/ - /home/*/.dropbox-dist/ # Gimp - /.gimp-*/tmp - /.gimp-*/swap # Mozilla Firefox - /home/*/.mozilla/firefox/*/Cache/ - /home/*/.mozilla/firefox/*/lock - /home/*/.mozilla/firefox/*/.parentlock # Mozilla Thunderbird - Do NOT backup emails (profiles only) - /home/*/.thunderbird/*/lock - /home/*/.thunderbird/*/.parentlock - /home/*/.thunderbird/*/ImapMail/ # Pidgin (accounts.xml contains passwords in clear text) - /home/*/.purple/accounts.xml
restore.sh
#!/bin/bash # Copyright (c) 2013 Christopher Will<dev@cwill-dev.com> # # Permission is hereby granted, free of charge, to any person obtaining a copy of # this software and associated documentation files (the "Software"), to deal in the # Software without restriction, including without limitation the rights to use, # copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the # Software, and to permit persons to whom the Software is furnished to do so, subject # to the following conditions: # # The above copyright notice and this permission notice shall be included in all # copies or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, # INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A # PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT # HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Functions ######################################################################## # Helper method to request user interaction # Args # 1: Message to display function pause() { read -p "$*" } # Initialization ######################################################################## # We mast be lord of the system if [ "$(whoami)" != "root" ]; then echo 'Aborting: You must be root to execute this script.' exit 1 fi # Root directory of the backedup files (per definition two levels up) BAK_SOURCES="$( cd "$( dirname "${BASH_SOURCE[0]}" )/../../../current/" && pwd )/" BAK_APT_DIR=${BAK_SOURCES}'_myback_/restore/apt/' # Hello World ######################################################################## echo "======================================" echo "MyBack - Restore Script" echo "======================================" echo "" echo "Step 1: Replace package information" echo "Step 2: Add repository keys" echo "Step 3: Update source lists" echo "Step 4: Restore and install packages" echo "Step 5: Restore system" echo "Step 6: Reboot" echo "" echo "Good luck my friend - may the force be with you!" echo "--------------------------------------" echo "" # Replace local package sources information with our backup ######################################################################## echo "" echo "> ------------------------------------" echo "> 1/6: Replace package information" echo "> ------------------------------------" echo "" pause "> Press [Enter] key to continue..." cp ${BAK_APT_DIR}'sources.list' /etc/apt/sources.list if [ -f ${BAK_APT_DIR}'apt.conf' ]; then cp ${BAK_APT_DIR}'apt.conf' /etc/apt/apt.conf fi if [ -f ${BAK_APT_DIR}'preferences' ]; then cp ${BAK_APT_DIR}'preferences' /etc/apt/preferences fi cp -R ${BAK_APT_DIR}sources.list.d/ /etc/apt/sources.list.d/ cp -R ${BAK_APT_DIR}apt.conf.d/ /etc/apt/apt.conf.d/ cp -R ${BAK_APT_DIR}preferences.d/ /etc/apt/preferences.d/ cp -R ${BAK_APT_DIR}lists/ /var/lib/apt/lists/ echo "" echo "> [DONE]" echo "" # Add repository keys to system ######################################################################## echo "" echo "> ------------------------------------" echo "> 2/6: Add repository keys" echo "> ------------------------------------" echo "" pause "> Press [Enter] key to continue..." apt-key add ${BAK_SOURCES}_myback_/restore/repositories.keys echo "> [DONE]" echo "" # Update the sources list ######################################################################## echo "" echo "> ------------------------------------" echo "> 3/6: Update and upgrade packages" echo "> ------------------------------------" echo "" pause "> Press [Enter] key to continue..." apt-get update apt-get -y upgrade echo "> [DONE]" echo "" # Restore backuped packages ######################################################################## echo "" echo "> ------------------------------------" echo "> 4/6: Restore packages (long process)" echo "> ------------------------------------" echo "" pause "> Press [Enter] key to continue..." #S olution 1: dpkg selection # Damn dpkg.. too bugy to use this simple stuff! #dpkg --clear-selections #dpkg --set-selections < ${BAK_SOURCES}_myback_/restore/installed-packages.lst #apt-get update #apt-get dselect-upgrade # Solution 2: apt-mark # Damn apt-mark #apt-mark auto $(cat ${BAK_SOURCES}_myback_/restore/pkgs_auto.lst) #apt-mark manual $(cat ${BAK_SOURCES}_myback_/restore/pkgs_manual.lst) # Solution 3: Manual install packages one-by-one while read p; do if [[ -n "$p" ]]; then echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>" echo "> "${p} echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>" apt-get -y install $p fi done < ${BAK_SOURCES}_myback_/restore/apt-install.lst # Remove unneeded packages apt-get -y autoremove echo "> [DONE]" echo "" # Copy backup ######################################################################## echo "" echo "> ------------------------------------" echo "> 5/6: Restore system (long process)" echo "> ------------------------------------" echo "" pause "> Press [Enter] key to continue..." rsync -av --exclude="/_myback_/" ${BAK_SOURCES} / echo "> [DONE]" echo "" # Reboot ######################################################################## echo "" echo "> ------------------------------------" echo "> 6/6: Reboot" echo "> ------------------------------------" echo "" echo "Awesome! We are done." echo "Hope to see you back after reboot :-)" echo "" pause "> Press [Enter] key to finish!" reboot echo "> [DONE]"
vbox_merge_snapshots.sh
#!/bin/bash # Copyright (c) 2013 Christopher Will<dev@cwill-dev.com> # # Permission is hereby granted, free of charge, to any person obtaining a copy of # this software and associated documentation files (the "Software"), to deal in the # Software without restriction, including without limitation the rights to use, # copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the # Software, and to permit persons to whom the Software is furnished to do so, subject # to the following conditions: # # The above copyright notice and this permission notice shall be included in all # copies or substantial portions of the Software. # # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, # INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A # PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT # HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. # Hello World echo '========================================' echo '= THIS SCRIPT COMES WITH NO WARRANTY =' echo '= Bash version '${BASH_VERSION}' =' echo '= [CTRL] + [Z] to kill process =' echo '========================================' # Exit with message and error code function failure() { echo '' echo 'ERROR:'$1 echo '' exit 1 } # Check whether this script got called as super user. If not, we can not # proceed. if [ "$(whoami)" != "root" ]; then failure 'Aborting: You must be root to execute this script.' fi # Get all users declare -a users tmp_users="$( getent passwd | grep /home/ | cut -d ':' -f 1 )" for u in ${tmp_users}; do tmp_dir="/home/${u}" if [ -d "$tmp_dir" ]; then users+=(${u}) fi done # Do delete/merge all snapshots for ((i=0;i<${#users[@]};i++)); do echo "> Processing user: ${users[i]}" # Get all installed VMs vms=$(su -c "VBoxManage list vms | sed -E 's/^\"(.*)\".*/\1/g'" -s /bin/sh ${users[i]}) # Loop through each VM for vm_name in $vms; do echo "> > Processing VM ${vm_name}.." # Loop through each snapshot and delete/merge it snapshots=$(su -c "VBoxManage showvminfo ${vm_name} --machinereadable | grep SnapshotName | cut -d '\"' -f2" -s /bin/sh ${users[i]}) for snapshot_name in $snapshots; do echo "> > > Merging snapshot: ${snapshot_name}.." snapshots=$(su -c "VBoxManage snapshot ${vm_name} delete ${snapshot_name}" -s /bin/sh ${users[i]}) done done done
Thanks
Further thanks to following posts, pages, documentations etc
http://stackoverflow.com/questions/13659202/rsync-complex-filter
http://askubuntu.com/questions/9135/best-way-to-backup-all-settings-list-of-installed-packages-tweaks-etc/99151#99151
http://askubuntu.com/questions/183010/apt-get-unmet-dependencies-try-apt-get-f-install-with-no-packages-or-speci
http://askubuntu.com/questions/140246/how-do-i-resolve-unmet-dependencies
http://askubuntu.com/questions/243387/how-can-i-backup-my-programs-applications-so-that-after-i-format-my-linux-and-i
http://serverfault.com/questions/150269/complex-includes-excludes-with-rsync
http://itefix.no/i2/content/excluding-directories-cwrsync
http://ubuntuforums.org/showthread.php?t=35087
http://ubuntuforums.org/showthread.php?t=2045187
http://ubuntuforums.org/showthread.php?t=1533494
http://ubuntuforums.org/showthread.php?t=2133648
http://www.virtualbox.org/manual/ch08.html#idp21979584
http://ubuntuforums.org/showthread.php?t=1071892
https://help.ubuntu.com/12.10/serverguide/automatic-updates.html
Conclusion
Please feel free to commit bugs-reports, things that might be done better or any kind of improvement. Thanks.
Royal Continental Suites, Dubai is an ideal starting point from which to discover the city’s dynamic energy, where modern architecture stands side-by-side with traditional souk markets. Strategically located on Business Bay Downtown, minutes away from Dubai Mall and City Walk, the hotel offers spacious rooms with stunning views of the ever-evolving skyline or the Arabian Sea. Asian-inspired dining options and a pool offering unobstructed views of the Burj Khalifa complete the experience.
https://royalcontinentalhotels.com/rcs/about-us/
Hello, thanks for the detailed writeup, very useful.
I have a situation where the data to backup is on a remote machine so I want to rsync it from a remote location. Is there a way to achieve this using your script?
Thank you
Hi Jason,
thanks for your comment.
First I would like to recommend you the new version hosted on Github:
https://github.com/cwilldev/PUBS
No, I did not consider remote backups, yet.
But on a gut level: Did you try already to mount the remote location into your local system? In this case it should not matter if it’s remote or not.
Or you could run the backup script on the remote server, and moving the created backup via rsync manually to you system.
Hello
Sorted it for now with the old script.
I commented out the validation of source directories as this was causing issues with the source I entered into the exec.sh (user@remotelocation.com/folder)
I also had to configure pre shared keys for SSH auto login.
As my remote system uses a non standard port for SSH I altered the rsync line in script to:
rsync -axPv –exclude-from ${EXCLUDE_FILE} ${link_dest} –rsh=’ssh -p19′ ${source_directories_string} ${current_bak_dest_dir}
Prior to this I had tried mounting the remote directory locally as you suggested, this did work but the speed was incredibly slow compared to the way I have now achieved it.
I will try the new script soon too.
Tnank you
Hi Json,
cool that you found a solution.
Feel free to send pull requests for the PUBS project :-)
Have a nice day,
Chris
PS: Thanks for sharing.
One thing to add is that I am not using it on Ubuntu, I am running the script from Gentoo without issue apart from the dialog boxes which do not display just because I don’t have X Windows.
Thx. Nice script and walkthrough. Works perfect out the box.