Name: Christopher, aka "cwilldev"

Posts by cwilldev:

    How to Properly Set Up VirtualBox Host/Guest System with NFS

    April 30th, 2013

    Usecase

    Since I always have to google for a good solution to configure a guest/host-system properly I will now just post the most usefull links.

    Solution

    Basic setup (creating VSI, installing Debian, configure network, setting up shares)
    http://www.uvd.co.uk/blog/labs/setting-up-a-debian-virtualbox-for-web-development

    Cloning correctly (considering UDEV-rules)
    http://lastkth-en.blogspot.de/2008/04/virtualbox-clone-virtual-machine-ubuntu.html

    NFS Sharing #1
    http://www.howtoforge.com/setting-up-an-nfs-server-and-client-on-ubuntu-10.04

    NFS Sharing #2
    https://tuleap.net/wiki/?group_id=101&pagename=Development+environment%2FVirtualBox

    NFS mount system call failed
    http://ryanak.ca/2009/09/21/fixing-nfs-on-debian.html

    So thanks to you guys writing those great articles!

    7 Comments "

    Merge VirtualBox Snapshots via Bash Script

    April 13th, 2013

    Use case

    My backup script automatically creates daily snapshots of each VM images (for incremental backups). Thus I have a huge amount of snapshots after a while. Unfortunately the VirtualBox-UI does not provide a method of removing/mergin all snapshots at once.

    Solution

    So I wrote a small script that is doing this job for me (and configured it via cronjob to be run once a week).
    The code must be executed as sudo.

    # Hello World
    echo '========================================'
    echo '= THIS SCRIPT COMES WITH NO WARRANTY   ='
    echo '= Bash version '${BASH_VERSION}'       ='
    echo '= [CTRL] + [Z] to kill process         ='
    echo '========================================'
    
    # Exit with message and error code
    function failure() {	
    	echo ''
    	echo 'ERROR:'$1
    	echo ''
    	exit 1
    }
    
    # Check whether this script got called as super user. If not, we can not 
    # proceed.
    if [ "$(whoami)" != "root" ]; then
    	failure 'Aborting: You must be root to execute this script.'
    fi
      
    # Get all users
    declare -a users 	
    tmp_users="$( getent passwd | grep /home/ | cut -d ':' -f 1 )"
    for u in ${tmp_users}; do
    	tmp_dir="/home/${u}"
    	if [ -d "$tmp_dir" ]; then
    		users+=(${u})
    	fi
    done	
    
    # Do delete/merge all snapshots
    for ((i=0;i<${#users[@]};i++)); do
    
    	echo "> Processing user: ${users[i]}"
    
    	# Get all installed VMs
    	vms=$(su -c "VBoxManage list vms | sed -E 's/^\"(.*)\".*/\1/g'" -s /bin/sh ${users[i]})
    
    	# Create new snapshot of each
    	for vm_name in $vms; do 
    		
    		echo "> > Processing VM ${vm_name}.."
    
    		# Loop through each snapshot and delete/merge it
    		snapshots=$(su -c "VBoxManage showvminfo ${vm_name} --machinereadable | grep SnapshotName | cut -d '\"' -f2" -s /bin/sh ${users[i]})
    		for snapshot_name in $snapshots; do 
    			echo "> > > Merging snapshot: ${snapshot_name}.."
    			snapshots=$(su -c "VBoxManage snapshot ${vm_name} delete ${snapshot_name}" -s /bin/sh ${users[i]})
    		done
    	done
    	
    done

    Conclusion

    The script tries to remove all snapshots of all images of all users.
    In VirtualBox terms “removing an snapshot” actually means that they get merged into the main virtual file – without any loss of data.

     

    4 Comments "

    MyBack: Incremental Ubuntu Backup Script including VirtualBox using a rsync Wrapper

    April 13th, 2013
    This is an outdated version. I fully refactored the tool and published it on Github.

    If you are just interested in the sourcecode please jump directly to the sourcecode-section.

    What is MyBack

    MyBack is a fully-automated backup solution I’ve implemented recently. After running for years now with mostly manual-triggered, non-incremental and redundant backups I reached the point where I was finally willing to invest some time for a proper and appropriate solution that can run in the background on a daily base. I also wanted a way to backup my several VirtualBox images, incrementally as well.

    ALL commands in this post are meant to be executed as root! So either prepend a “sudo” to each command, or impersonate your user as root by executing “sudo su”/”sudo -i” once.

    Requirements

    Installation

    1. Download the tarball from here or use your terminal
      wget http://blog.cwill-dev.com/downloads/myback/myback.tar.gz
    2. Open a terminal and make yourself sudo
      sudo su
    3. Move downladed tarball to /opt/
      mv ~/Downloads/myback.tar.gz /opt/
    4. Navigate to /opt/ and unpack the file
      cd /opt/
      tar -zxvf ./myback.tar.gz
    5. Make sh-files executable
      cd /opt/myback
      find . -name '*.sh' -exec chmod +x \{\} \;

    Configuration

    Execution parameters

    There is only one file you have to configure for MyBack to do its job – namely the file that calls the application’s main script with the individual parameters that fit to your system. So open the exec.sh file now

    nano /opt/myback/exec.sh

    You will find following one-liner:

    /opt/myback/app/myback.sh  -s -d  /media/BACKUP_STORAGE /
    # |---------- 1 --------] [2] [3] [-------- 4 --------] [5]

    1: The path to the myback.sh main script
    2: If -s is specified, the script will support incremental VirtualBox backup (see chapter Insights->VirtualBox)
    3: -d is required and must be followed by the destination folder
    4: The destination folder, into which the script will backup
    5: The folder to backup (in this case “/”, so root – the entire system (see chapter excludes / includes)

    When you are done press [CTRL][O] followed by [CTRL][X] to save your settings.
    Of course the destination folder must be mounted and accessible.

    Partitions

    If you have mounted some of the system directories to several partitions – for instance /opt or /home – then you have to specify those folders explicitly (even you think “/” might contain these folders already) . This is based on the underlying rsync application, which will not copy the data inside those folders, in fact it will just create those folders but not copy the files.

    This is an example in case you have both /opt and /home on different partitions

    /opt/myback/app/myback.sh -s -d /media/BACKUP_STORAGE / /home /opt

    Excludes / Blacklist

    As already pointed out MyBack uses a blacklist for excluding several files and directories. This blacklist is located in the root directory of the unpacked myback folder. So open this file:

    nano /opt/myback/excludes.txt

    The provided blacklist covers the default case in which the backup script gets to backup the entire system (“/”).
    Cache, tmp or redundant folders are filtered automatically. Also some default applications – like GIMP,  DropBox, Firefox or Thunderbird - got considered already. For the latter one MyBack will not backup the emails, only the account settings. I made this decision to save storage and for better performance, since the common case is that emails are kept on the server anyway.

    Usually there should be no need to edit these settings. But if you are interested in adjusting the backup structure individually to fit to your system’s individual configuration have a look at these pages:

    http://rsync.samba.org/ftp/rsync/rsync.html
    http://programmersnotebook.wordpress.com/2010/03/20/rsync-and-exclude-from
    http://itefix.no/i2/content/excluding-directories-cwrsync
    http://serverfault.com/questions/150269/complex-includes-excludes-with-rsync
    http://stackoverflow.com/questions/13659202/rsync-complex-filter

    One important information is, that each item within the blacklist is always seen relative to the directory rsync is backuping.
    So if you backup more than one directory (see chapter partitions above) be aware of this certainty.
    Be aware of the fact that a traiding space of any path definition inside the exclude list might lead to unwanted errors.

    Run the backup

    For that we have already configured the execution parameters and the blacklist we just have to run the exec.sh script

    /opt/myback/exec.sh

    Now lean back, the initial backup may take some time. The output will be printed to the terminal.

    Configure Cronjob

    Since a backup does not make much sense if it’s not running automatically frequently, we should set up a cronjob now.
    This gets done by using Ubuntu’s crontab.

    crontab -e

    At this point we can tell the system on what time and frequency the backup should get executed.

    add 30 15 * * * /opt/myback/exec.sh

    In this example the backup will run every day at half past three afternoon.
    For further information you might want to check crontab’s manpage:
    http://unixhelp.ed.ac.uk/CGI/man-cgi?crontab+5

    As already mentioned in the intro, this command MUST absolutely get executed as sudo.
    There are some bugs in some environments where cronjobs won’t get executed if there is no blank line at the end of the file. In my case I also faced some problems writing some extra comments to the file.
    See also http://askubuntu.com/questions/23009/reasons-why-crontab-does-not-work
    Ubuntu’s crontab uses Anacron internally. Anacron will take take that the backup gets called to the specified intervals accordingly. So you do not have to take care to turn on your computer on that specific time.

    Recovery

    Since MyBack backups all required system data and information (when running with “/”) the recovery procedure is pretty simple, yet a long time process.

    1. Install plain Ubuntu on the new system
    2. Mount the backup drive
    3. Run the recovery script
      /media/BACKUP_STORAGE/current/_myback_/restore/restore.sh

    The recovery script does following things:

    1. Replace package information
    2. Add repository keys
    3. Update source lists
    4. Restore and install packages
    5. Restore system
    6. Reboot
    As of the current Kubuntu version (12.10) the firefox package will lead to failures during the recovery.
    See this thread: http://ubuntuforums.org/showthread.php?t=2133648

    Reading package lists... Done
    Building dependency tree 
    Reading state information... Done
    You might want to run 'apt-get -f install' to correct these.
    The following packages have unmet dependencies:
    firefox-globalmenu : Depends: firefox (= 20.0+build1-0ubuntu0.12.10.3) but it is not installed
    E: Unmet dependencies. Try using -f.

    So the only solution to deal with this is to remove the following packages from the generated apt-installed.lst (/BAK_DESTINATION_DIR/current/_myback_/recover)

    • firefox-globalmenu
    • kubuntu-firefox-installer
    • firefox

    Insights / Advanced

    VirtualBox

    If the backup script gets called with the -s parameter MyBack will backup all VirtualBox images of ALL user’s as well, incrementally. This is done by creating a snapshot of each VM. If a VM is running on the time of the backup the VM will be paused for the time of the snapshot creation.

    VBox Snapshot Example

    This behaviour will lead to a huge list of snapshots after a while. So you may want to remove/merge those snapshots frequently. Either you do this manually in the VirtualBox-UI, or you can use the script I provided:

    /opt/myback/tools/vbox_merge_snapshots.sh

    Notification

    The script uses notifiy-send to notifiy the desktop environment about the status of a backup. This will also work if the backup got called by crontab. The notification will look like this:
    Notification example

    Logs

    MyBack will copy its terminal output to the destination backup directory. It will be located in the directory
    /BACKUP_DIR/current/_myback_/log/myback.log

    MyBack

    • MyBack uses the hardlink-feature of rsync. So even your file browser will show all files in each backup-folder, this does not mean that they exist multiple times.
    • MyBack will copy itself to the destination backup directory as well. All of its files will be located under
      /BACKUP_DIR/current/_myback_/
    • The structure of the backup directory will be as follow:
      Structure example
      As you can see, MyBack creates for each backup an individual directory, named by the date of execution.
      The soft link current will always lead to the last created backup.
    • All basic files, that will be required to start the recovery of your system will be stored here as well (ie sources.list)

    Restore

    I went through several hardly failures during the recovery process. As you can read everywhere there should be two possibilities to backup installed software and to re-import those on the new system – but both do not work!

    dpkg –get-selection and dpkg –set-selection

    dpkg --clear-selections
    dpkg --set-selections installed-packages.lst 
    apt-get update
    apt-get dselect-upgrade

    apt-mark showauto showmanual and auto manual

    apt-mark auto $(cat ${BAK_SOURCES}_myback_/restore/pkgs_auto.lst)
    apt-mark manual $(cat ${BAK_SOURCES}_myback_/restore/pkgs_manual.lst)

    So the only way that did the jpb properly was to apt-get install each package automatically one by one. Maybe in the upcoming Ubuntu release this problem might be fixed.

    Code

    myback.sh

    #!/bin/bash
    
    # Copyright (c) 2013 Christopher Will<dev@cwill-dev.com>
    # 
    # Permission is hereby granted, free of charge, to any person obtaining a copy of 
    # this software and associated documentation files (the "Software"), to deal in the 
    # Software without restriction, including without limitation the rights to use, 
    # copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the 
    # Software, and to permit persons to whom the Software is furnished to do so, subject 
    # to the following conditions:
    # 
    # The above copyright notice and this permission notice shall be included in all 
    # copies or substantial portions of the Software.
    # 
    # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 
    # INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 
    # PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 
    # HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 
    # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 
    # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
    
    #####################################################################################
    #
    # VARIABLES
    #
    #####################################################################################
    
    # Script name and version
    app='MyBack v0.1beta'
    
    # Setting display properties for the desktop notifications
    DISPLAY=":0.0"
    export DISPLAY
    
    # File defining the backup excludes
    MYBACK_BASEDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )/../" && pwd )/"
    EXCLUDE_FILE=${MYBACK_BASEDIR}'excludes.txt'
    
    # Will contain all users with a home directory (used for impersonification later on)
    declare -a users 
    
    # Virtual Box support
    # If set (see --help) then the script will search for all available VirtualBox images 
    # of all users. For each image snapshots will get created, so the backup just uses
    # incremental changes instead of the complete ie VHD
    virtual_box_support=false
    
    # Specify the root directory in which rsync will store the backup (with $date as sub-
    # directory). The given storage must be mounted correctly.
    #
    # As advice, the destination directory should not be located on the same device where 
    # the data to backup is located. Because in case of hardware failure you will then 
    # not be able to access your backups. Better prefer an external storage.
    # Example: /mnt/myBackupDevice/rsync 
    dest_dir='' 
    
    # This array will contain all positional arguments (=source directories) specifying 
    # each directory that has to get considered during the backup process. 
    # In a default and simple environment it should be enough to specify the personal 
    # home directory. But it may also list any other directories.
    # See the myback_excludes for a default case dealing with a root (/) source directory.
    declare -a source_directories
    
    # Will contain the temporary log file (which gets copied to the destination myback
    # directory afterwards)
    tmp_log_file=''
    
    # We use the current date for each backup as sub-directory-name within the given 
    # $dest_dir
    date=`date "+%Y-%m-%dT%H:%M:%S"` 
    
    #####################################################################################
    #
    # FUNCTIONS
    #
    #####################################################################################
    
    # Initialize basic variables.
    #####################################################################################
    function init() {
    
    	# Logfile
    	# Redirect stdout & stderr to tmp logfile
    	# Gets copied later to the destination directory.
    	tmp_rand=$(cat /dev/urandom | tr -cd [:alnum:] | head -c 4)
    	tmp_log_file="/tmp/myback_"$date"_"$tmp_rand".log"
    	exec >  >(tee -a ${tmp_log_file})
    	exec 2> >(tee -a ${tmp_log_file} >&2)
    	exec 2>&1
    
    	# Set user list to $users
    	# Gets used for snapshotting the virtual box images as well for the
    	# UI notification	
    	tmp_users="$( getent passwd | grep /home/ | cut -d ':' -f 1 )"
    	for u in ${tmp_users}
    	do
    		tmp_dir="/home/${u}"
    		if [ -d "$tmp_dir" ]; then
    			users+=(${u})
    		fi
    	done	
    }
    
    # Displays a notification baloon in desktop environment for each user
    # Args
    # 1 - String - Title
    # 2 - String - Content
    # 3 - String - System icon (ie dialog-ok, dialog-error, dialog-information etc)
    #####################################################################################
    function display_notification() {
    	icon_dialog="dialog-information"
    	if [[ -n "$3" ]]; then
    		icon_dialog=${3}
    	fi  
    	for ((i=0;i<${#users[@]};i++)); do 
    		$(su -c "notify-send -u critical \"${1}\" \"${2}\" --icon=${icon_dialog}" -s /bin/sh ${users[i]})
    	done  
    } 
    
    # Exits script with failure code 1 and displays a desktop notification of type "error" 
    # as well.
    # Args:
    # $1 - String - Error message
    #####################################################################################
    function failure() {	
    	echo $1
    	display_notification "MyBack - Error occurred" "${1}" "dialog-error"
    	exit 1
    }
    
    # Prepares the script arguments.
    #####################################################################################
    function setup_args() { 
    	while getopts "h?sd:" opt; do
    		case "$opt" in
    
    			# help
    			h|\?)
    				echo $app
    				echo 'Simple yet effective rsync backup wrapper'
    				echo ''
    				echo '[OPTIONS] [DIR1[DIR2[DIRn..]]]'
    				echo ''
    				echo 'Options:'
    				echo '-h Show brief help'
    				echo '-d Backups destination directory'
    				echo '-s Virtual Box support'
    				exit 0
    				;;
    
    			# destination directory
    			d) 
    				if [[ -n "$OPTARG" ]]; then
    					dest_dir=${OPTARG%/}
    				fi 
    				;;  
    
    			# virtual_box_support
    			s) 
    				virtual_box_support=true
    				;;  
    		esac
    	done  
    
    	# Reset argument index pointer for accessing positional parameters
    	shift $(( OPTIND-1 ))
    
    	# Iterate through each positional parameter and add it to the source_directories 
    	# array
    	for var in "$@"
    	do
    		# Do not remove trailing slash if root directory (/)
    		if [ ${var} != '/' ]; then
    			var=(${var%/})
    		fi
    		source_directories+=(${var})
    	done
    }
    
    # Validates all variables
    #####################################################################################
    function validate {
    
    	# Check whether this script got called as super user. If not, we can not 
    	# proceed.
    	if [ "$(whoami)" != "root" ]; then
    		failure 'Aborting: You must be root to execute this script.'
    	fi
    
    	# Check dest_dir to be valid
    	if [ ! -d ${dest_dir} ]; then
    		failure 'Aborting: Destination directory "'${dest_dir}'" does not exist. Maybe it is not mounted?'
    	fi  
    
    	# Check exclude file
    	if [ ! -f ${EXCLUDE_FILE} ]; then
    		failure 'Aborting: Exclude file "'${EXCLUDE_FILE}'" does not exist.'
    	fi 
    
    	# Check source directories to be set
    	if [ ${#source_directories[@]} == 0 ]; then
    		failure 'Aborting: No source directories specified.'
    	fi
    
    	# Check source directories to be valid
    	for ((i=0;i<${#source_directories[@]};i++)); do
    		if [ ! -d ${source_directories[i]} ]; then
    			failure 'Aborting: Source directory "'${source_directories[i]}'" does not exist. Maybe it is not mounted?'
    		fi
    	done  
    }
    
    #####################################################################################
    #
    # STARTUP
    #
    #####################################################################################
    
    # Hello World
    echo '========================================'
    echo '= THIS SCRIPT COMES WITH NO WARRANTY   ='
    echo '= Bash version '${BASH_VERSION}'       ='
    echo '= [CTRL] + [Z] to kill process         ='
    echo '========================================'
    
    # Do some basic initialization 
    init
    
    # Notify UI
    process_start_date_time=`date "+%Y-%m-%dT%H:%M:%S"`
    display_notification "MyBack - Backup started" "Backup process started at ${process_start_date_time}"
    
    # Handle input parameters
    setup_args "$@"
    
    # Validate environment
    validate 
    
    # Convert source_directories to string (used as rsync parameter)
    source_directories_string=''
    for ((i=0;i<${#source_directories[@]};i++)); do
    	source_directories_string+=${source_directories[i]}' '
    done  
    
    # The main symlink pointing to the latest (this) backup
    latest_backup_dir=${dest_dir}/current
    
    # The full path to the backup destination directory. Keep hierarchies.
    current_bak_dest_dir=${dest_dir}/${date}/
    
    # Do not set the link-dest argument on the first run, since there will be no 
    # referencing sub-directories within the "latest backup" directory yet.
    link_dest=''
    if [ -e ${latest_backup_dir} ]
    then 
    	link_dest='--link-dest='${latest_backup_dir}/
    fi
    
    # Create the directory hierarchy for the current backup
    # Create myback special folder
    MYBACK_DEST_DIR=${current_bak_dest_dir}_myback_/
    
    # Display working target directory
    echo '> Timestamp: '${date}
    echo '> Sources: '${source_directories_string}
    echo '> Destination: '${dest_dir}/${date}
    echo '> Logfile: '${MYBACK_DEST_DIR}myback.log
    if ${virtual_box_support}; then
    	echo '> VirtualBox support enabled'
    else	
    	echo '> VirtualBox support disabled'
    fi
    echo '----------------------------------------'
    
    #####################################################################################
    #
    # VIRTUALBOX
    # Create snapshots of each installed VM, so rsync is able to backup the VirtualBox
    # images incrementally
    #
    #####################################################################################
    if ${virtual_box_support}; then
    	echo '> Creating VirtualBox snapshots'
    
    	# Do snapshots for all users
    	for ((i=0;i<${#users[@]};i++)); do
    
    		echo "> > User: ${users[i]}"
    
    		# Get all installed VMs
    		vms=$(su -c "VBoxManage list vms | sed -E 's/^\"(.*)\".*/\1/g'" -s /bin/sh ${users[i]})
    
    		# Create new snapshot of each
    		for vm_name in $vms; do 
    			echo "> > > Processing ${vm_name}.."
    			$(su -c "VBoxManage snapshot ${vm_name} take ${date} --description \"Generated by myback automatically at {$date}\" --pause" -s /bin/sh ${users[i]})
    			echo "> > > [DONE] ${vm_name}"
    		done
    
    	done
    
    	echo '> [DONE] Creating VirtualBox snapshots'
    fi
    
    #####################################################################################
    #
    # RSYNC
    #
    #####################################################################################
    
    # Let the user know which directory we are currently process
    echo '> Backing up '${source_directories_string}
    
    # Call rsync - this is the main logic! 
    # link_dest             - Is only set if the initial backup was processed already
    # current_bak_dest_dir - The backup location (of structure destination/date/hierarchy)
    rsync -axPv --exclude-from ${EXCLUDE_FILE} ${link_dest} ${source_directories_string} ${current_bak_dest_dir} 
    
    # Keep symlink up to date
    if [ -e ${current_bak_dest_dir} ]
    then
    	rm -f ${latest_backup_dir} # Remove
    	ln -s ${dest_dir}/${date} ${latest_backup_dir} # Re-create
    fi 
    
    #####################################################################################
    #
    # RESTORE PREPARATION
    # We copy all files that will be required during the recovery to the destination 
    # folder as well.
    #
    #####################################################################################
    
    # Create directories
    mkdir -p ${MYBACK_DEST_DIR} 
    mkdir ${MYBACK_DEST_DIR}'restore'
    mkdir ${MYBACK_DEST_DIR}'restore/apt'
    mkdir ${MYBACK_DEST_DIR}'log'
    mkdir ${MYBACK_DEST_DIR}'app'
    mkdir ${MYBACK_DEST_DIR}'tools'
    
    # Export list of all installed applications and repository keys
    apt_dir=${MYBACK_DEST_DIR}'restore/apt/'
    cp /etc/apt/sources.list ${apt_dir}'sources.list'
    if [ -f '/etc/apt/apt.conf' ]; then
       cp /etc/apt/apt.conf ${apt_dir}'apt.conf'
    fi
    if [ -f '/etc/apt/preferences' ]; then
    	cp /etc/apt/preferences ${apt_dir}'preferences'
    fi	
    cp -R /etc/apt/sources.list.d/ ${apt_dir}
    cp -R /etc/apt/apt.conf.d/ ${apt_dir}
    cp -R /etc/apt/preferences.d/ ${apt_dir}
    cp -R /var/lib/apt/lists/ ${apt_dir}
    
    apt-key key exportall > ${MYBACK_DEST_DIR}'restore/repositories.keys'
    
    # Solution 1: dpkg selectiom -> Buggy (as of 12.10), re-import won't work
    dpkg --get-selections > ${MYBACK_DEST_DIR}'restore/installed-packages.lst' 
    
    # Solution 2: apt-mark -> Buggy as well (as of 12.10)
    apt-mark showauto > ${MYBACK_DEST_DIR}'restore/pkgs_auto.lst'
    apt-mark showmanual > ${MYBACK_DEST_DIR}'restore/pkgs_manual.lst'
    
    # Solution 3: Manual approach -> Create list of packages to re-install one by on recovery
    package_list=$(dpkg-query -Wf '${Package} ')
    package_list=${package_list// /$'\n'}  # change the semicolons to white space
    for package in $package_list; do
    	echo "$package" >> ${MYBACK_DEST_DIR}'restore/apt-install.lst'
    done
    
    # Copy tmp logfile to current destination directory
    mv ${tmp_log_file} ${MYBACK_DEST_DIR}"log/myback.log"
    
    # Copy myback system files to backup as well
    cp -R ${MYBACK_BASEDIR}'app/' ${MYBACK_DEST_DIR}
    cp -R ${MYBACK_BASEDIR}'tools/' ${MYBACK_DEST_DIR}
    cp -R ${MYBACK_BASEDIR}'restore/' ${MYBACK_DEST_DIR}
    cp ${MYBACK_BASEDIR}"excludes.txt" ${MYBACK_DEST_DIR}'/'
    cp ${MYBACK_BASEDIR}"exec.sh" ${MYBACK_DEST_DIR}'/'
    
    #####################################################################################
    #
    # FINISH
    #
    #####################################################################################
    
    # Notify UI (multiline intended)
    process_end_date_time=`date "+%Y-%m-%dT%H:%M:%S"`
    finish_msg="Backup process finished at ${process_end_date_time}
    
    Destination directory: ${current_bak_dest_dir}
    
    Logfile: ${MYBACK_DEST_DIR}log/myback.log"
    display_notification "MyBack - Backup finished" "${finish_msg}" "dialog-ok"

    excludes.txt

    # =======================================================
    # This is the exclude file for the rsync backup.
    # 
    # Notice:
    # All path entries are interpreted relative to the source 
    # directories.
    # =======================================================
    
    # Universal excludes
    ########################
    
    lost+found
    ld.so.cache
    *.log
    *.bak
    
    # Root file system 
    ########################
    
    - /bin/
    - /boot/
    - /cdrom/
    - /dev/
    - /etc/modules.conf
    - /lib/
    - /lib32/
    - /lib64/
    - /media/
    - /mnt/
    - /proc/
    - /run/
    - /sbin/
    - /selinux/
    - /sys/
    - /tmp/
    + /usr/
    + /usr/share/
    + /usr/local/
    + /usr/local/share/
    - /usr/local/*
    - /usr/*
    - /var/cache/
    - /var/crash/
    - /var/lock/
    - /var/log/
    - /var/run/
    - /var/tmp/
    - /var/lib/sudo/
    
    - initrd.img.old
    - initrd.img
    - vmlinuz
    - vmlinuz.old
    
    # Filters for home dirs
    ########################
    
    # Cache
    - /home/*/.cache/
    
    # Downloads
    - /home/*/Downloads/ 
    
    # Dropbox
    - /home/*/Dropbox
    
    # Temporary files / cache
    - /home/*/.local/share/Trash
    - /home/*/.cache
    - /home/*/.Trash
    
    # X Windows System
    - /home/*/.xsession-errors* 
    
    # Several
    ########################
    
    # Exclude backup text files
    - *~
    - \#*\#
    
    # Commonly distributed Mac OS X cache
    - .DS_Store
    
    # Commonly distributed Windows cache
    - Thumbs.db
    
    # Common Applications
    ########################
    
    # Adobe Reader
    - /home/*/.adobe/**/AssetCache/
    - /home/*/.adobe/**/Cache/
    - /home/*/.adobe/**/Temp/
    - /home/*/.adobe/**/UserCache.bin
    
    # Dropbox temp stuff
    - /home/*/.dropbox/
    - /home/*/.dropbox-dist/
    
    # Gimp
    - /.gimp-*/tmp
    - /.gimp-*/swap
    
    # Mozilla Firefox
    - /home/*/.mozilla/firefox/*/Cache/
    - /home/*/.mozilla/firefox/*/lock
    - /home/*/.mozilla/firefox/*/.parentlock
    
    # Mozilla Thunderbird - Do NOT backup emails (profiles only)
    - /home/*/.thunderbird/*/lock
    - /home/*/.thunderbird/*/.parentlock
    - /home/*/.thunderbird/*/ImapMail/
    
    # Pidgin (accounts.xml contains passwords in clear text)
    - /home/*/.purple/accounts.xml

    restore.sh

    #!/bin/bash
    
    # Copyright (c) 2013 Christopher Will<dev@cwill-dev.com>
    # 
    # Permission is hereby granted, free of charge, to any person obtaining a copy of 
    # this software and associated documentation files (the "Software"), to deal in the 
    # Software without restriction, including without limitation the rights to use, 
    # copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the 
    # Software, and to permit persons to whom the Software is furnished to do so, subject 
    # to the following conditions:
    # 
    # The above copyright notice and this permission notice shall be included in all 
    # copies or substantial portions of the Software.
    # 
    # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 
    # INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 
    # PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 
    # HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 
    # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 
    # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
    
    # Functions
    ########################################################################
    
    # Helper method to request user interaction
    # Args
    # 1: Message to display
    function pause() {
       read -p "$*"
    }
    
    # Initialization
    ########################################################################
    
    # We mast be lord of the system
    if [ "$(whoami)" != "root" ]; then
    	echo 'Aborting: You must be root to execute this script.'
    	exit 1
    fi
    
    # Root directory of the backedup files (per definition two levels up)
    BAK_SOURCES="$( cd "$( dirname "${BASH_SOURCE[0]}" )/../../../current/" && pwd )/"
    BAK_APT_DIR=${BAK_SOURCES}'_myback_/restore/apt/'
    
    # Hello World
    ########################################################################
    
    echo "======================================"
    echo "MyBack - Restore Script"
    echo "======================================"
    echo ""
    echo "Step 1: Replace package information"
    echo "Step 2: Add repository keys"
    echo "Step 3: Update source lists"
    echo "Step 4: Restore and install packages"
    echo "Step 5: Restore system"
    echo "Step 6: Reboot"
    echo ""
    echo "Good luck my friend - may the force be with you!"
    echo "--------------------------------------"
    echo ""
    
    # Replace local package sources information with our backup
    ########################################################################
    echo ""
    echo "> ------------------------------------"
    echo "> 1/6: Replace package information"
    echo "> ------------------------------------"
    echo ""
    pause "> Press [Enter] key to continue..."
    cp ${BAK_APT_DIR}'sources.list' /etc/apt/sources.list
    if [ -f ${BAK_APT_DIR}'apt.conf' ]; then
    	cp ${BAK_APT_DIR}'apt.conf' /etc/apt/apt.conf 
    fi
    if [ -f ${BAK_APT_DIR}'preferences' ]; then
    	cp ${BAK_APT_DIR}'preferences' /etc/apt/preferences
    fi
    cp -R ${BAK_APT_DIR}sources.list.d/ /etc/apt/sources.list.d/ 
    cp -R ${BAK_APT_DIR}apt.conf.d/ /etc/apt/apt.conf.d/
    cp -R ${BAK_APT_DIR}preferences.d/ /etc/apt/preferences.d/
    cp -R ${BAK_APT_DIR}lists/ /var/lib/apt/lists/
    echo ""
    echo "> [DONE]" 
    echo "" 
    
    # Add repository keys to system
    ########################################################################
    echo ""
    echo "> ------------------------------------"
    echo "> 2/6: Add repository keys"
    echo "> ------------------------------------"
    echo ""
    pause "> Press [Enter] key to continue..."
    apt-key add ${BAK_SOURCES}_myback_/restore/repositories.keys
    echo "> [DONE]" 
    echo ""
    
    # Update the sources list
    ########################################################################
    echo ""
    echo "> ------------------------------------"
    echo "> 3/6: Update and upgrade packages"
    echo "> ------------------------------------"
    echo ""
    pause "> Press [Enter] key to continue..."
    apt-get update
    apt-get -y upgrade 
    echo "> [DONE]" 
    echo ""
    
    # Restore backuped packages
    ########################################################################
    echo ""
    echo "> ------------------------------------"
    echo "> 4/6: Restore packages (long process)"
    echo "> ------------------------------------"
    echo ""
    pause "> Press [Enter] key to continue..."
    
    #S olution 1: dpkg selection
    # Damn dpkg.. too bugy to use this simple stuff!
    #dpkg --clear-selections
    #dpkg --set-selections < ${BAK_SOURCES}_myback_/restore/installed-packages.lst 
    #apt-get update
    #apt-get dselect-upgrade
    
    # Solution 2: apt-mark
    # Damn apt-mark
    #apt-mark auto $(cat ${BAK_SOURCES}_myback_/restore/pkgs_auto.lst)
    #apt-mark manual $(cat ${BAK_SOURCES}_myback_/restore/pkgs_manual.lst)
    
    # Solution 3: Manual install packages one-by-one
    while read p; do
    	if [[ -n "$p" ]]; then
    
    		echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
    		echo "> "${p}
    		echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
    
    		apt-get -y install $p
    	fi
    done < ${BAK_SOURCES}_myback_/restore/apt-install.lst
    
    # Remove unneeded packages
    apt-get -y autoremove
    
    echo "> [DONE]" 
    echo ""
    
    # Copy backup
    ########################################################################
    echo ""
    echo "> ------------------------------------"
    echo "> 5/6: Restore system (long process)"
    echo "> ------------------------------------"
    echo ""
    pause "> Press [Enter] key to continue..."
    rsync -av --exclude="/_myback_/" ${BAK_SOURCES} /
    echo "> [DONE]" 
    echo ""
    
    # Reboot
    ########################################################################
    echo ""
    echo "> ------------------------------------"
    echo "> 6/6: Reboot"
    echo "> ------------------------------------"
    echo ""
    echo "Awesome! We are done."
    echo "Hope to see you back after reboot :-)"
    echo ""
    pause "> Press [Enter] key to finish!"
    reboot
    echo "> [DONE]"

    vbox_merge_snapshots.sh

    #!/bin/bash
    
    # Copyright (c) 2013 Christopher Will<dev@cwill-dev.com>
    # 
    # Permission is hereby granted, free of charge, to any person obtaining a copy of 
    # this software and associated documentation files (the "Software"), to deal in the 
    # Software without restriction, including without limitation the rights to use, 
    # copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the 
    # Software, and to permit persons to whom the Software is furnished to do so, subject 
    # to the following conditions:
    # 
    # The above copyright notice and this permission notice shall be included in all 
    # copies or substantial portions of the Software.
    # 
    # THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, 
    # INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A 
    # PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT 
    # HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION 
    # OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE 
    # SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
    
    # Hello World
    echo '========================================'
    echo '= THIS SCRIPT COMES WITH NO WARRANTY   ='
    echo '= Bash version '${BASH_VERSION}'       ='
    echo '= [CTRL] + [Z] to kill process         ='
    echo '========================================'
    
    # Exit with message and error code
    function failure() {	
    	echo ''
    	echo 'ERROR:'$1
    	echo ''
    	exit 1
    }
    
    # Check whether this script got called as super user. If not, we can not 
    # proceed.
    if [ "$(whoami)" != "root" ]; then
    	failure 'Aborting: You must be root to execute this script.'
    fi
    
    # Get all users
    declare -a users 	
    tmp_users="$( getent passwd | grep /home/ | cut -d ':' -f 1 )"
    for u in ${tmp_users}; do
    	tmp_dir="/home/${u}"
    	if [ -d "$tmp_dir" ]; then
    		users+=(${u})
    	fi
    done	
    
    # Do delete/merge all snapshots
    for ((i=0;i<${#users[@]};i++)); do
    
    	echo "> Processing user: ${users[i]}"
    
    	# Get all installed VMs
    	vms=$(su -c "VBoxManage list vms | sed -E 's/^\"(.*)\".*/\1/g'" -s /bin/sh ${users[i]})
    
    	# Loop through each VM
    	for vm_name in $vms; do 
    
    		echo "> > Processing VM ${vm_name}.."
    
    		# Loop through each snapshot and delete/merge it
    		snapshots=$(su -c "VBoxManage showvminfo ${vm_name} --machinereadable | grep SnapshotName | cut -d '\"' -f2" -s /bin/sh ${users[i]})
    		for snapshot_name in $snapshots; do 
    			echo "> > > Merging snapshot: ${snapshot_name}.."
    			snapshots=$(su -c "VBoxManage snapshot ${vm_name} delete ${snapshot_name}" -s /bin/sh ${users[i]})
    		done
    	done
    
    done

    Thanks

    Further thanks to following posts, pages, documentations etc

    http://stackoverflow.com/questions/13659202/rsync-complex-filter
    http://askubuntu.com/questions/9135/best-way-to-backup-all-settings-list-of-installed-packages-tweaks-etc/99151#99151
    http://askubuntu.com/questions/183010/apt-get-unmet-dependencies-try-apt-get-f-install-with-no-packages-or-speci
    http://askubuntu.com/questions/140246/how-do-i-resolve-unmet-dependencies
    http://askubuntu.com/questions/243387/how-can-i-backup-my-programs-applications-so-that-after-i-format-my-linux-and-i
    http://serverfault.com/questions/150269/complex-includes-excludes-with-rsync
    http://itefix.no/i2/content/excluding-directories-cwrsync
    http://ubuntuforums.org/showthread.php?t=35087
    http://ubuntuforums.org/showthread.php?t=2045187
    http://ubuntuforums.org/showthread.php?t=1533494
    http://ubuntuforums.org/showthread.php?t=2133648
    http://www.virtualbox.org/manual/ch08.html#idp21979584
    http://ubuntuforums.org/showthread.php?t=1071892
    https://help.ubuntu.com/12.10/serverguide/automatic-updates.html

    Conclusion

    Please feel free to commit bugs-reports, things that might be done better or any kind of improvement. Thanks.

    6 Comments "

    Installing Oracle’s Java (7) on Ubuntu (12.10)

    April 3rd, 2013

    Use case

    Since the Java installation handling changed during the last Ubuntu releases (due to Oracle’s politics) most parts of my old post are out of date now.

    Solution

    I found a great instruction about how to install Oracle’s Java JDK on Ubuntu.
    As I could not write it better I will just post the link here:

    http://www.wikihow.com/Install-Oracle-Java-JDK-on-Ubuntu-Linux

    Side note

    I prefer archiving several JDK versions, so I put each JDK under /opt/java/JDK-{VERSION}.
    In order to not be forced to update my alternatives everytime when a new update gets released I created a symbolic link called “current” in the same directory (/opt/java/).
    So in case of a new version I just need to update the symlink once.

    1 Comment "

    German Umlauts on US-Keyboard on (K)Ubuntu

    April 3rd, 2013

    Use case

    As a software developer I prefer using keyboards with US-layout. Mainly because special characters (like []{}\|<>) are more easy to write/reach – some even without a modifier.
    But, for email and stuff, I still rely on German Umlauts.

    Solution

    So this is what I did do use German Umlauts even with US-keyboard.

    Create a .Xmodmap file in your home directory

    nano .Xmodmap

    And add following lines:

    keycode 108 = Mode_switch
    keycode 26 = e E EuroSign EuroSign e E
    keycode 30 = u U udiaeresis Udiaeresis u U
    keycode 32 = o O odiaeresis Odiaeresis o O
    keycode 38 = a A adiaeresis Adiaeresis a A
    keycode 39 = s S ssharp ssharp s S

    Conclusion

    That’s it. Now you can access both the lower- and upper-case Umlauts by using the right ALT key.

    If you don’t want to restart your X-server to see the changes you can re-import the xmodmap settings as follow (again in your home directory):

    xmodmap .Xmodmap

    1 Comment "

    Lenovo T530, Kubuntu 12.10, NVIDIA Optimus, External Monitors

    April 3rd, 2013
    April 16, 2013
    Just a couple of days after I wrote this post NVIDIA released a new version of its driver that is optimized for the Optimus technology. See http://steamforlinux.com/?q=en/node/210 for further information.

    Use case

    I bought a new Lenovo Thinkpad T530 laptop incl. a docking station recently. I was facing several problems regarding the docking station, the grafix card and external monitors. After spending couple of hours to get this all up and running I decided to share my experiences.

    Maybe it will help someone, or maybe someone may help me by providing alternatives and/or better solutions.

    Specs

    Hardware

    i7-3630QM, NVIDIA NVS 5400M with Optimus, SATA3 256GB CRUCIAL m4 SSD, 16 GB Kingston Memory KTL-TP3C DDR3 1600 MHz, Mini Dock Plus, Series 3 433810U, Two external Philips 23″ monitors connected to the docking station via DVI

    Software

    3.5.0-17-generic #28-Ubuntu SMP Tue Oct 9 19:31:23 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux, Qt: 4.8.3, KDE Development Platform: 4.9.5, kde4-config: 1.0

    After removing Windows 8 by formatting the SSD and installing Kubuntu 12.10 I was struggling around with som weired problems regarding the NVIDIA Optimus card.

    What happened so far (messed up)

    I tried almost two days to get my two external monitors work together with the laptop display. But, unfortunately, I failed. Well, I even did mess up my system that much that I needed to re-install my system. This mostly is based on the proprietary NVIDIA driver – which broke during both the installation and the de-installation scripts. And I could not figure it out to remove all the NVIDIA stuff manually.

    My sincerely advice: DO NOT USE the official NVIDIA drivers!
    Even though NVIDIA might give you the impression that the specific grafic card will work fine with your specific system. But it won’t.

    For those who might proove me wrong, please try it on your own (and please leave a comment if you succeed):
    http://www.nvidia.com/object/linux-display-amd64-310.44-driver.html

    BIOS settings

    Luckily, the T530 provides an BIOS-option to chose a particular grafic card.

    Integrated – Intel’s integrated grafic, NO NVIDIA
    Discret - The NVIDIA card
    Optimus – Both

    In the latter case the OS decides when to switch between the integrated one and the NVIDIA card. Guess this should be fine running on Windows 7 or 8. But the Linux-support does just barely exist.

    The idea behind the Optimus technology is to prevent the laptop from draining battery: The NVIDIA card gets activated only if it’s necessary. Great idea though.

    Since I use my laptop always on AC (and have a 9-cell battery just for case) I do not care very much about batterie’s lifetime. This is why I’m satisfied by chosing the discrete option.

    Driver installation

    So, the simplest yet most effective and promising way is to install nvidia-current and nvidia-settings.
    Contrary to various recommendations to enable the repository ppa:ubuntu-x-swat/x-updates
    I got no luck with that. So I did not add or change any repositories.

    sudo apt-get install nvidia-current nvidia-settings

    After the installation you have to let NVIDIA configure the xorg.conf for you.

    sudo nvidia-xconfig

    Now you need to reboot your system.

    Setup

    Two external monitors

    No you can run

    nvidia-settings

    to configure your monitors.

    I configured my two external monitors as TwinView. The first (DFP-2) positioned absolutely, the second one (DFP-3) relatively to the first one.
    The third display (laptop, DFP-0) was disabled by default – After playing around a couple of times I accepted that this was the only solution that will work.

    When you are done just save the settings to the xorg.conf (create backup before!) and reboot your system (or just X).

    Laptop display only

    Well, the default scenario just works fine now. But what if you want to undock your laptop or use it without the external monitors.
    Hm, I don’t want to disappoint you, but the former task gave me huge problems (=black screens), and I still can’t switch screens while the system is running. If you have any clue or even a solution please be so kind to leave a comment.

    As you might have seen already the nvidia-settings take use of the metamodes options in the xorg.conf. This is pretty fine, since we can define multiple metamodes options which get handled in sort and as fallbacks.

    So I configured it that way that the X-server will search for my two external monitors on startup, if they don’t exist the laptop’s display gets used.
    These are my settings:

    Option         "metamodes" "DFP-2: 1920x1080 +0+0, DFP-3: 1920x1080 +1920+0"
    Option         "metamodes" "LVDS-0: 1920x1080 +0+0"

    xorg.conf

    This is my entire xorg.conf

    # nvidia-settings: X configuration file generated by nvidia-settings
    # nvidia-settings:  version 304.51  (buildd@batsu)  Fri Oct 12 12:53:54 UTC 2012
    
    Section "ServerLayout"
        Identifier     "Layout0"
        Screen      0  "Screen0" 0 0
        InputDevice    "Keyboard0" "CoreKeyboard"
        InputDevice    "Mouse0" "CorePointer"
        Option         "Xinerama" "0"
    EndSection
    
    Section "Files"
    EndSection
    
    Section "InputDevice"
        # generated from default
        Identifier     "Mouse0"
        Driver         "mouse"
        Option         "Protocol" "auto"
        Option         "Device" "/dev/psaux"
        Option         "Emulate3Buttons" "no"
        Option         "ZAxisMapping" "4 5"
    EndSection
    
    Section "InputDevice"
        # generated from default
        Identifier     "Keyboard0"
        Driver         "kbd"
    EndSection
    
    Section "Monitor"
        # HorizSync source: edid, VertRefresh source: edid
        Identifier     "Monitor0"
        VendorName     "Unknown"
        ModelName      "Philips 234CL"
        HorizSync       30.0 - 83.0
        VertRefresh     56.0 - 76.0
        Option         "DPMS"
    EndSection
    
    Section "Device"
        Identifier     "Device0"
        Driver         "nvidia"
        VendorName     "NVIDIA Corporation"
        BoardName      "NVS 5400M"
    EndSection
    
    Section "Screen"
        Identifier     "Screen0"
        Device         "Device0"
        Monitor        "Monitor0"
        DefaultDepth    24
        Option         "Stereo" "0"
        Option         "nvidiaXineramaInfoOrder" "DFP-2"
        Option         "metamodes" "DFP-2: 1920x1080 +0+0, DFP-3: 1920x1080 +1920+0"
        Option         "metamodes" "LVDS-0: 1920x1080 +0+0"
        SubSection     "Display"
            Depth       24
        EndSubSection
    EndSection

    Side note

    I’m very open minded to mistakes I made, or clue’s to help the (un)docking problems, or better solutions.
    PS: Jep, I played around with bumblebee, but did not end happy either.

    2 Comments "

    Encryption between Java/Android and PHP

    October 9th, 2012

    Use case

    While working on an android app that retrieves it’s data from a PHP-API  I needed to build a both-sided en- and decryption layer to secure the requests and responses properly.

    I decided to use an AES encryption with IvParameterSpec, SecretKeySpec and “AES/CBC/PKCS5Padding” on Java side, and  mcrypt_module “rijndael-128″ on PHP side.

    Implementations

    PHP

    <?
    class ApiCrypter
    {
        private $iv  = 'fdsfds85435nfdfs'; #Same as in JAVA
        private $key = '89432hjfsd891787'; #Same as in JAVA
    
        public function __construct() {
        }
    
        public function encrypt($str) { 
    	  $str = $this->pkcs5_pad($str);   
    	  $iv = $this->iv; 
    	  $td = mcrypt_module_open('rijndael-128', '', 'cbc', $iv); 
    	  mcrypt_generic_init($td, $this->key, $iv);
    	  $encrypted = mcrypt_generic($td, $str); 
    	  mcrypt_generic_deinit($td);
    	  mcrypt_module_close($td); 
    	  return bin2hex($encrypted);
        }
    
        public function decrypt($code) { 
    	  $code = $this->hex2bin($code);
    	  $iv = $this->iv; 
    	  $td = mcrypt_module_open('rijndael-128', '', 'cbc', $iv); 
    	  mcrypt_generic_init($td, $this->key, $iv);
    	  $decrypted = mdecrypt_generic($td, $code); 
    	  mcrypt_generic_deinit($td);
    	  mcrypt_module_close($td); 
    	  $ut =  utf8_encode(trim($decrypted));
    	  return $this->pkcs5_unpad($ut);
        }
    
        protected function hex2bin($hexdata) {
    	  $bindata = ''; 
    	  for ($i = 0; $i < strlen($hexdata); $i += 2) {
    	      $bindata .= chr(hexdec(substr($hexdata, $i, 2)));
    	  } 
    	  return $bindata;
        } 
    
        protected function pkcs5_pad ($text) {
    	  $blocksize = 16;
    	  $pad = $blocksize - (strlen($text) % $blocksize);
    	  return $text . str_repeat(chr($pad), $pad);
        }
    
        protected function pkcs5_unpad($text) {
    	  $pad = ord($text{strlen($text)-1});
    	  if ($pad > strlen($text)) {
    	      return false;	
    	  }
    	  if (strspn($text, chr($pad), strlen($text) - $pad) != $pad) {
    	      return false;
    	  }
    	  return substr($text, 0, -1 * $pad);
        }
    }
    ?>

    JAVA

    package com.cwilldev.crypt;
    
    import java.security.NoSuchAlgorithmException;
    import javax.crypto.Cipher;
    import javax.crypto.NoSuchPaddingException;
    import javax.crypto.spec.IvParameterSpec;
    import javax.crypto.spec.SecretKeySpec;
    
    public class ApiCrypter {
    
        private String iv              = "myuniqueivparam";
        private String secretkey       = "mysecretkey";
        private IvParameterSpec ivspec;
        private SecretKeySpec keyspec;
        private Cipher cipher;
    
        public ApiCrypter()
        {
            ivspec = new IvParameterSpec(iv.getBytes());
            keyspec = new SecretKeySpec(secretkey.getBytes(), "AES");
    
            try {
                cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
            } catch (NoSuchAlgorithmException e) {
                e.printStackTrace();
            } catch (NoSuchPaddingException e) {
                e.printStackTrace();
            }
        }
    
        public byte[] encrypt(String text) throws Exception
        {
            if(text == null || text.length() == 0) {
                throw new Exception("Empty string");
            }
            byte[] encrypted = null;
            try {
                cipher.init(Cipher.ENCRYPT_MODE, keyspec, ivspec);
                encrypted = cipher.doFinal(text.getBytes("UTF-8"));
            }
            catch (Exception e) {
                throw new Exception("[encrypt] " + e.getMessage());
            }
            return encrypted;
        }
    
        public byte[] decrypt(String code) throws Exception
        {
            if(code == null || code.length() == 0) {
                throw new Exception("Empty string");
            }
            byte[] decrypted = null;
            try {
                cipher.init(Cipher.DECRYPT_MODE, keyspec, ivspec);
                decrypted = cipher.doFinal(hexToBytes(code));
            }
            catch (Exception e) {
                throw new Exception("[decrypt] " + e.getMessage());
            }
            return decrypted;
        }
    
        public static String bytesToHex(byte[] data)
        {
            if (data==null) {
                return null;
            }
            int len = data.length;
            String str = "";
            for (int i=0; i<len; i++) {
                if ((data[i]&0xFF)<16) {
                    str = str + "0" + java.lang.Integer.toHexString(data[i]&0xFF);
                }
                else {
                    str = str + java.lang.Integer.toHexString(data[i]&0xFF);
                }
            }
            return str;
        }
    
        public static byte[] hexToBytes(String str) {
            if (str==null) {
                return null;
            }
            else if (str.length() < 2) {
                return null;
            }
            else {
                int len = str.length() / 2;
                byte[] buffer = new byte[len];
                for (int i=0; i<len; i++) {
                    buffer[i] = (byte) Integer.parseInt(str.substring(i*2,i*2+2),16);
                }
                return buffer;
            }
        }
    
    }

    Encryption

    Java

    String encryptedRequest = ApiCrypter.bytesToHex(this.apiCrypter.encrypt(jsonParams.toString()));

    PHP

    $encrypted = $ApiCrypter->encrypt($str);

    Decryption

    Java

    String res = new String( this.apiCrypter.decrypt( text ), "UTF-8" );
    res = URLDecoder.decode(res,"UTF-8");

    PHP

    $decrypted = $ApiCrypter->decrypt($str);

    24 Comments "

    IDEA hook: Grails not found!

    October 9th, 2012

    Use case

    While working with different grails versions and a maven repository in the background I faced some weird problems regarding version problems which IDEA was complaining about.

    Incident

    The stacktrace looked as follow:

    [INFO] ------------------------------------------------------------------------
    [ERROR] FATAL ERROR
    [INFO] ------------------------------------------------------------------------
    [INFO] org.grails.maven.plugin.AbstractGrailsMojo.runGrailsInline(Ljava/lang/String;Ljava/lang/String;Z)V
    [INFO] ------------------------------------------------------------------------
    [INFO] Trace
    java.lang.NoSuchMethodError: org.grails.maven.plugin.AbstractGrailsMojo.runGrailsInline(Ljava/lang/String;Ljava/lang/String;Z)V
    	at org.grails.maven.plugin.GrailsExecMojo.execute(GrailsExecMojo.java:51)
    	at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:490)
    	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:694)
    	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandaloneGoal(DefaultLifecycleExecutor.java:569)
    	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:539)
    	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:387)
    	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:284)
    	at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180)
    	at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328)
    	at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138)
    	at org.apache.maven.cli.MavenCli.main(MavenCli.java:362)
    	at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:60)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    	at java.lang.reflect.Method.invoke(Method.java:597)
    	at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315)
    	at org.codehaus.classworlds.Launcher.launch(Launcher.java:255)
    	at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430)
    	at org.codehaus.classworlds.Launcher.main(Launcher.java:375)
    [INFO] ------------------------------------------------------------------------
    [INFO] Total time: 11 seconds
    [INFO] Finished at: Wed Sep 05 10:44:12 CEST 2012
    [INFO] Final Memory: 33M/743M
    [INFO] ------------------------------------------------------------------------
    java.lang.NullPointerException
    	at org.jetbrains.groovy.grails.rt.Agent$2.run(Agent.java:99)
    	at java.lang.Thread.run(Thread.java:662)
    
    IDEA hook: Grails not found!

    Neither grails clean, nor any re-builds vanished this problem.

    Solution

    After searching the web I could figure out that it has to with the JARs handling and cache of both grails and IDEA.

    Finally, after removing the .ivy2- and maven-cache the problem did not occur anymore.

    1 Comment "

    Enable Zend PHP Debugging under IntelliJ 11.1

    April 23rd, 2012

    Use case

    In this tutorial I assume that you have already installed and configured the Web server and the PHP engine properly.
    If not, please have a look at this tutorial.

    Solution

    In our configuration we use all default properties (like debug port, broadcasting port etc).

    Enable PHP plugin (PHPStorm)

    • Select your project and open the settings menu.
    • Under the project settings you find a PHP entry
    • Select the PHP version
    • Confiure an interpreter
      • Click on the … button
      • Click the plus button in the left-handed upper corner
      • Chose a name (something like PHP<5.3.6>)
      • Under General->PHP Home point to you PHP location (in my case /usr/bin)
      • As debugger choose the Zend Debugger
      • Submit the form
    • If you don’t see the newly configured interpreter directly push the refresh button
    • Now you should see the new interpreter (PHP<5.3.6>)
    • By clicking on the question mark you can see an overview of your PHP configuration.
      Notice that you dont see any debug parts yet. you should not

    Download Zend Debugger

    • Download the Zend Debugger package which corresponds to your operating system here (Studio Web Debugger)
    • Locate the ZendDebugger.so file in the directory which corresponds to your version of PHP (e.g. 4.3.x, 4.4.x, 5.0.x, 5.1.x, 5.2.x)
    • Copy the file to your Web server in a location that is accessible by the Web server (like /home/foo, /opt/bar or whereever you like the file to be stored).

    Enable Zend Debugger in PHP

    • Create a phpinfo-page (HowTo)
    • Search for the key “Loaded Configuration File” to see which php.ini your system is using
    • Edit the php.ini
    • Add or extend the zend-section by adding following settings:
      [Zend]
      zend_extension=<path_to_ZendDebugger.so
      zend_debugger.allow_hosts=127.0.0.1
      ;zend_debugger.allow_hosts=127.0.0.1,localhost
      zend_debugger.expose_remotely=allowed_hosts
    • Restart apache.

    Create Debug-Configuration in IntelliJ

    • Click on “Edit Configurations” (it’s located in the toolbar next to the play-button)
    • Add a new configuration- Chose “PHP Web Application” as type
    • Set a unique name for your project debug configuration
    • Chose “localhost” as server- Point the start URL to the project http root
    • Click OK

    Debug

    • Set a breakpoint somewhere in your code (that of course gets executed)
    • Start the debugger (debug icon next to the play button)
    • Open (if IntelliJ did not already opened a browser on its own) a browser and access the root URL

    Conclusion

    Now IntelliJ should stop at your breakpoint.

    7 Comments "

    Error on SVN checkout: SSL handshake failed: SSL error: Key usage violation in certificate has been detected.

    March 28th, 2012

    Use case

    Lately I received the following error message, while trying to checkout from a SVN repository located on a Windows server from my Kubuntu machine:

    svn: Commit failed (details follow):
    svn: OPTIONS of 'https://HOSTNAME/svn/repo': SSL handshake failed: SSL error: Key usage violation in certificate has been detected. (https://HOSTNAME)

    Solution

    A short search pointed out, that this issue occures because of an unsigned security certificate.Have a look at this post.

    In my case I just needed to install libneon

    $ sudo apt-get install libneon27

    And then to remove the previous symbolic link by replacing it with a new one pointing to gnutls:

    sudo mv /usr/lib/libneon-gnutls.so.27 /usr/lib/libneon-gnutls.so.27.old
    sudo ln -s /usr/lib/libneon.so.27 /usr/lib/libneon-gnutls.so.27

    This worked for me, even though I had to restart my system surprisely.

    5 Comments "

Archive
Categories