Name: Christopher, aka "cwilldev"
Posts by cwilldev:
How to Properly Set Up VirtualBox Host/Guest System with NFS
April 30th, 2013Usecase
Since I always have to google for a good solution to configure a guest/host-system properly I will now just post the most usefull links.
Solution
Basic setup (creating VSI, installing Debian, configure network, setting up shares)
http://www.uvd.co.uk/blog/labs/setting-up-a-debian-virtualbox-for-web-development
Cloning correctly (considering UDEV-rules)
http://lastkth-en.blogspot.de/2008/04/virtualbox-clone-virtual-machine-ubuntu.html
NFS Sharing #1
http://www.howtoforge.com/setting-up-an-nfs-server-and-client-on-ubuntu-10.04
NFS Sharing #2
https://tuleap.net/wiki/?group_id=101&pagename=Development+environment%2FVirtualBox
NFS mount system call failed
http://ryanak.ca/2009/09/21/fixing-nfs-on-debian.html
Merge VirtualBox Snapshots via Bash Script
April 13th, 2013Use case
My backup script automatically creates daily snapshots of each VM images (for incremental backups). Thus I have a huge amount of snapshots after a while. Unfortunately the VirtualBox-UI does not provide a method of removing/mergin all snapshots at once.
Solution
So I wrote a small script that is doing this job for me (and configured it via cronjob to be run once a week).
The code must be executed as sudo.
# Hello World
echo '========================================'
echo '= THIS SCRIPT COMES WITH NO WARRANTY ='
echo '= Bash version '${BASH_VERSION}' ='
echo '= [CTRL] + [Z] to kill process ='
echo '========================================'
# Exit with message and error code
function failure() {
echo ''
echo 'ERROR:'$1
echo ''
exit 1
}
# Check whether this script got called as super user. If not, we can not
# proceed.
if [ "$(whoami)" != "root" ]; then
failure 'Aborting: You must be root to execute this script.'
fi
# Get all users
declare -a users
tmp_users="$( getent passwd | grep /home/ | cut -d ':' -f 1 )"
for u in ${tmp_users}; do
tmp_dir="/home/${u}"
if [ -d "$tmp_dir" ]; then
users+=(${u})
fi
done
# Do delete/merge all snapshots
for ((i=0;i<${#users[@]};i++)); do
echo "> Processing user: ${users[i]}"
# Get all installed VMs
vms=$(su -c "VBoxManage list vms | sed -E 's/^\"(.*)\".*/\1/g'" -s /bin/sh ${users[i]})
# Create new snapshot of each
for vm_name in $vms; do
echo "> > Processing VM ${vm_name}.."
# Loop through each snapshot and delete/merge it
snapshots=$(su -c "VBoxManage showvminfo ${vm_name} --machinereadable | grep SnapshotName | cut -d '\"' -f2" -s /bin/sh ${users[i]})
for snapshot_name in $snapshots; do
echo "> > > Merging snapshot: ${snapshot_name}.."
snapshots=$(su -c "VBoxManage snapshot ${vm_name} delete ${snapshot_name}" -s /bin/sh ${users[i]})
done
done
done
Conclusion
The script tries to remove all snapshots of all images of all users.
In VirtualBox terms “removing an snapshot” actually means that they get merged into the main virtual file – without any loss of data.
MyBack: Incremental Ubuntu Backup Script including VirtualBox using a rsync Wrapper
April 13th, 2013What is MyBack
MyBack is a fully-automated backup solution I’ve implemented recently. After running for years now with mostly manual-triggered, non-incremental and redundant backups I reached the point where I was finally willing to invest some time for a proper and appropriate solution that can run in the background on a daily base. I also wanted a way to backup my several VirtualBox images, incrementally as well.
Requirements
- Ubuntu (tested with Kubuntu 12.10)
- rsync
- libnotify-bin
- The partition of the backup destination directory must support hard-links. See here for a list of supported file storage systems (like ext3, ext4, NTFS): http://en.wikipedia.org/wiki/Comparison_of_file_systems#Features
Installation
- Download the tarball from here or use your terminal
wget http://blog.cwill-dev.com/downloads/myback/myback.tar.gz
- Open a terminal and make yourself sudo
sudo su
- Move downladed tarball to /opt/
mv ~/Downloads/myback.tar.gz /opt/
- Navigate to /opt/ and unpack the file
cd /opt/ tar -zxvf ./myback.tar.gz
- Make sh-files executable
cd /opt/myback find . -name '*.sh' -exec chmod +x \{\} \;
Configuration
Execution parameters
There is only one file you have to configure for MyBack to do its job – namely the file that calls the application’s main script with the individual parameters that fit to your system. So open the exec.sh file now
nano /opt/myback/exec.sh
You will find following one-liner:
/opt/myback/app/myback.sh -s -d /media/BACKUP_STORAGE / # |---------- 1 --------] [2] [3] [-------- 4 --------] [5]
1: The path to the myback.sh main script
2: If -s is specified, the script will support incremental VirtualBox backup (see chapter Insights->VirtualBox)
3: -d is required and must be followed by the destination folder
4: The destination folder, into which the script will backup
5: The folder to backup (in this case “/”, so root – the entire system (see chapter excludes / includes)
When you are done press [CTRL][O] followed by [CTRL][X] to save your settings.
Of course the destination folder must be mounted and accessible.
Partitions
If you have mounted some of the system directories to several partitions – for instance /opt or /home – then you have to specify those folders explicitly (even you think “/” might contain these folders already) . This is based on the underlying rsync application, which will not copy the data inside those folders, in fact it will just create those folders but not copy the files.
This is an example in case you have both /opt and /home on different partitions
/opt/myback/app/myback.sh -s -d /media/BACKUP_STORAGE / /home /opt
Excludes / Blacklist
As already pointed out MyBack uses a blacklist for excluding several files and directories. This blacklist is located in the root directory of the unpacked myback folder. So open this file:
nano /opt/myback/excludes.txt
The provided blacklist covers the default case in which the backup script gets to backup the entire system (“/”).
Cache, tmp or redundant folders are filtered automatically. Also some default applications – like GIMP, DropBox, Firefox or Thunderbird - got considered already. For the latter one MyBack will not backup the emails, only the account settings. I made this decision to save storage and for better performance, since the common case is that emails are kept on the server anyway.
Usually there should be no need to edit these settings. But if you are interested in adjusting the backup structure individually to fit to your system’s individual configuration have a look at these pages:
http://rsync.samba.org/ftp/rsync/rsync.html
http://programmersnotebook.wordpress.com/2010/03/20/rsync-and-exclude-from
http://itefix.no/i2/content/excluding-directories-cwrsync
http://serverfault.com/questions/150269/complex-includes-excludes-with-rsync
http://stackoverflow.com/questions/13659202/rsync-complex-filter
So if you backup more than one directory (see chapter partitions above) be aware of this certainty.
Run the backup
For that we have already configured the execution parameters and the blacklist we just have to run the exec.sh script
/opt/myback/exec.sh
Now lean back, the initial backup may take some time. The output will be printed to the terminal.
Configure Cronjob
Since a backup does not make much sense if it’s not running automatically frequently, we should set up a cronjob now.
This gets done by using Ubuntu’s crontab.
crontab -e
At this point we can tell the system on what time and frequency the backup should get executed.
add 30 15 * * * /opt/myback/exec.sh
In this example the backup will run every day at half past three afternoon.
For further information you might want to check crontab’s manpage:
http://unixhelp.ed.ac.uk/CGI/man-cgi?crontab+5
See also http://askubuntu.com/questions/23009/reasons-why-crontab-does-not-work
Recovery
Since MyBack backups all required system data and information (when running with “/”) the recovery procedure is pretty simple, yet a long time process.
- Install plain Ubuntu on the new system
- Mount the backup drive
- Run the recovery script
/media/BACKUP_STORAGE/current/_myback_/restore/restore.sh
The recovery script does following things:
- Replace package information
- Add repository keys
- Update source lists
- Restore and install packages
- Restore system
- Reboot
See this thread: http://ubuntuforums.org/showthread.php?t=2133648
Reading package lists... Done Building dependency tree Reading state information... Done You might want to run 'apt-get -f install' to correct these. The following packages have unmet dependencies: firefox-globalmenu : Depends: firefox (= 20.0+build1-0ubuntu0.12.10.3) but it is not installed E: Unmet dependencies. Try using -f.
So the only solution to deal with this is to remove the following packages from the generated apt-installed.lst (/BAK_DESTINATION_DIR/current/_myback_/recover)
- firefox-globalmenu
- kubuntu-firefox-installer
- firefox
Insights / Advanced
VirtualBox
If the backup script gets called with the -s parameter MyBack will backup all VirtualBox images of ALL user’s as well, incrementally. This is done by creating a snapshot of each VM. If a VM is running on the time of the backup the VM will be paused for the time of the snapshot creation.
This behaviour will lead to a huge list of snapshots after a while. So you may want to remove/merge those snapshots frequently. Either you do this manually in the VirtualBox-UI, or you can use the script I provided:
/opt/myback/tools/vbox_merge_snapshots.sh
Notification
The script uses notifiy-send to notifiy the desktop environment about the status of a backup. This will also work if the backup got called by crontab. The notification will look like this:
Logs
MyBack will copy its terminal output to the destination backup directory. It will be located in the directory
/BACKUP_DIR/current/_myback_/log/myback.log
MyBack
- MyBack uses the hardlink-feature of rsync. So even your file browser will show all files in each backup-folder, this does not mean that they exist multiple times.
- MyBack will copy itself to the destination backup directory as well. All of its files will be located under
/BACKUP_DIR/current/_myback_/ - The structure of the backup directory will be as follow:
As you can see, MyBack creates for each backup an individual directory, named by the date of execution.
The soft link current will always lead to the last created backup. - All basic files, that will be required to start the recovery of your system will be stored here as well (ie sources.list)
Restore
I went through several hardly failures during the recovery process. As you can read everywhere there should be two possibilities to backup installed software and to re-import those on the new system – but both do not work!
dpkg –get-selection and dpkg –set-selection
dpkg --clear-selections dpkg --set-selections installed-packages.lst apt-get update apt-get dselect-upgrade
apt-mark showauto showmanual and auto manual
apt-mark auto $(cat ${BAK_SOURCES}_myback_/restore/pkgs_auto.lst)
apt-mark manual $(cat ${BAK_SOURCES}_myback_/restore/pkgs_manual.lst)
So the only way that did the jpb properly was to apt-get install each package automatically one by one. Maybe in the upcoming Ubuntu release this problem might be fixed.
Code
myback.sh
#!/bin/bash
# Copyright (c) 2013 Christopher Will<dev@cwill-dev.com>
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of
# this software and associated documentation files (the "Software"), to deal in the
# Software without restriction, including without limitation the rights to use,
# copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
# Software, and to permit persons to whom the Software is furnished to do so, subject
# to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
# PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
#####################################################################################
#
# VARIABLES
#
#####################################################################################
# Script name and version
app='MyBack v0.1beta'
# Setting display properties for the desktop notifications
DISPLAY=":0.0"
export DISPLAY
# File defining the backup excludes
MYBACK_BASEDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )/../" && pwd )/"
EXCLUDE_FILE=${MYBACK_BASEDIR}'excludes.txt'
# Will contain all users with a home directory (used for impersonification later on)
declare -a users
# Virtual Box support
# If set (see --help) then the script will search for all available VirtualBox images
# of all users. For each image snapshots will get created, so the backup just uses
# incremental changes instead of the complete ie VHD
virtual_box_support=false
# Specify the root directory in which rsync will store the backup (with $date as sub-
# directory). The given storage must be mounted correctly.
#
# As advice, the destination directory should not be located on the same device where
# the data to backup is located. Because in case of hardware failure you will then
# not be able to access your backups. Better prefer an external storage.
# Example: /mnt/myBackupDevice/rsync
dest_dir=''
# This array will contain all positional arguments (=source directories) specifying
# each directory that has to get considered during the backup process.
# In a default and simple environment it should be enough to specify the personal
# home directory. But it may also list any other directories.
# See the myback_excludes for a default case dealing with a root (/) source directory.
declare -a source_directories
# Will contain the temporary log file (which gets copied to the destination myback
# directory afterwards)
tmp_log_file=''
# We use the current date for each backup as sub-directory-name within the given
# $dest_dir
date=`date "+%Y-%m-%dT%H:%M:%S"`
#####################################################################################
#
# FUNCTIONS
#
#####################################################################################
# Initialize basic variables.
#####################################################################################
function init() {
# Logfile
# Redirect stdout & stderr to tmp logfile
# Gets copied later to the destination directory.
tmp_rand=$(cat /dev/urandom | tr -cd [:alnum:] | head -c 4)
tmp_log_file="/tmp/myback_"$date"_"$tmp_rand".log"
exec > >(tee -a ${tmp_log_file})
exec 2> >(tee -a ${tmp_log_file} >&2)
exec 2>&1
# Set user list to $users
# Gets used for snapshotting the virtual box images as well for the
# UI notification
tmp_users="$( getent passwd | grep /home/ | cut -d ':' -f 1 )"
for u in ${tmp_users}
do
tmp_dir="/home/${u}"
if [ -d "$tmp_dir" ]; then
users+=(${u})
fi
done
}
# Displays a notification baloon in desktop environment for each user
# Args
# 1 - String - Title
# 2 - String - Content
# 3 - String - System icon (ie dialog-ok, dialog-error, dialog-information etc)
#####################################################################################
function display_notification() {
icon_dialog="dialog-information"
if [[ -n "$3" ]]; then
icon_dialog=${3}
fi
for ((i=0;i<${#users[@]};i++)); do
$(su -c "notify-send -u critical \"${1}\" \"${2}\" --icon=${icon_dialog}" -s /bin/sh ${users[i]})
done
}
# Exits script with failure code 1 and displays a desktop notification of type "error"
# as well.
# Args:
# $1 - String - Error message
#####################################################################################
function failure() {
echo $1
display_notification "MyBack - Error occurred" "${1}" "dialog-error"
exit 1
}
# Prepares the script arguments.
#####################################################################################
function setup_args() {
while getopts "h?sd:" opt; do
case "$opt" in
# help
h|\?)
echo $app
echo 'Simple yet effective rsync backup wrapper'
echo ''
echo '[OPTIONS] [DIR1[DIR2[DIRn..]]]'
echo ''
echo 'Options:'
echo '-h Show brief help'
echo '-d Backups destination directory'
echo '-s Virtual Box support'
exit 0
;;
# destination directory
d)
if [[ -n "$OPTARG" ]]; then
dest_dir=${OPTARG%/}
fi
;;
# virtual_box_support
s)
virtual_box_support=true
;;
esac
done
# Reset argument index pointer for accessing positional parameters
shift $(( OPTIND-1 ))
# Iterate through each positional parameter and add it to the source_directories
# array
for var in "$@"
do
# Do not remove trailing slash if root directory (/)
if [ ${var} != '/' ]; then
var=(${var%/})
fi
source_directories+=(${var})
done
}
# Validates all variables
#####################################################################################
function validate {
# Check whether this script got called as super user. If not, we can not
# proceed.
if [ "$(whoami)" != "root" ]; then
failure 'Aborting: You must be root to execute this script.'
fi
# Check dest_dir to be valid
if [ ! -d ${dest_dir} ]; then
failure 'Aborting: Destination directory "'${dest_dir}'" does not exist. Maybe it is not mounted?'
fi
# Check exclude file
if [ ! -f ${EXCLUDE_FILE} ]; then
failure 'Aborting: Exclude file "'${EXCLUDE_FILE}'" does not exist.'
fi
# Check source directories to be set
if [ ${#source_directories[@]} == 0 ]; then
failure 'Aborting: No source directories specified.'
fi
# Check source directories to be valid
for ((i=0;i<${#source_directories[@]};i++)); do
if [ ! -d ${source_directories[i]} ]; then
failure 'Aborting: Source directory "'${source_directories[i]}'" does not exist. Maybe it is not mounted?'
fi
done
}
#####################################################################################
#
# STARTUP
#
#####################################################################################
# Hello World
echo '========================================'
echo '= THIS SCRIPT COMES WITH NO WARRANTY ='
echo '= Bash version '${BASH_VERSION}' ='
echo '= [CTRL] + [Z] to kill process ='
echo '========================================'
# Do some basic initialization
init
# Notify UI
process_start_date_time=`date "+%Y-%m-%dT%H:%M:%S"`
display_notification "MyBack - Backup started" "Backup process started at ${process_start_date_time}"
# Handle input parameters
setup_args "$@"
# Validate environment
validate
# Convert source_directories to string (used as rsync parameter)
source_directories_string=''
for ((i=0;i<${#source_directories[@]};i++)); do
source_directories_string+=${source_directories[i]}' '
done
# The main symlink pointing to the latest (this) backup
latest_backup_dir=${dest_dir}/current
# The full path to the backup destination directory. Keep hierarchies.
current_bak_dest_dir=${dest_dir}/${date}/
# Do not set the link-dest argument on the first run, since there will be no
# referencing sub-directories within the "latest backup" directory yet.
link_dest=''
if [ -e ${latest_backup_dir} ]
then
link_dest='--link-dest='${latest_backup_dir}/
fi
# Create the directory hierarchy for the current backup
# Create myback special folder
MYBACK_DEST_DIR=${current_bak_dest_dir}_myback_/
# Display working target directory
echo '> Timestamp: '${date}
echo '> Sources: '${source_directories_string}
echo '> Destination: '${dest_dir}/${date}
echo '> Logfile: '${MYBACK_DEST_DIR}myback.log
if ${virtual_box_support}; then
echo '> VirtualBox support enabled'
else
echo '> VirtualBox support disabled'
fi
echo '----------------------------------------'
#####################################################################################
#
# VIRTUALBOX
# Create snapshots of each installed VM, so rsync is able to backup the VirtualBox
# images incrementally
#
#####################################################################################
if ${virtual_box_support}; then
echo '> Creating VirtualBox snapshots'
# Do snapshots for all users
for ((i=0;i<${#users[@]};i++)); do
echo "> > User: ${users[i]}"
# Get all installed VMs
vms=$(su -c "VBoxManage list vms | sed -E 's/^\"(.*)\".*/\1/g'" -s /bin/sh ${users[i]})
# Create new snapshot of each
for vm_name in $vms; do
echo "> > > Processing ${vm_name}.."
$(su -c "VBoxManage snapshot ${vm_name} take ${date} --description \"Generated by myback automatically at {$date}\" --pause" -s /bin/sh ${users[i]})
echo "> > > [DONE] ${vm_name}"
done
done
echo '> [DONE] Creating VirtualBox snapshots'
fi
#####################################################################################
#
# RSYNC
#
#####################################################################################
# Let the user know which directory we are currently process
echo '> Backing up '${source_directories_string}
# Call rsync - this is the main logic!
# link_dest - Is only set if the initial backup was processed already
# current_bak_dest_dir - The backup location (of structure destination/date/hierarchy)
rsync -axPv --exclude-from ${EXCLUDE_FILE} ${link_dest} ${source_directories_string} ${current_bak_dest_dir}
# Keep symlink up to date
if [ -e ${current_bak_dest_dir} ]
then
rm -f ${latest_backup_dir} # Remove
ln -s ${dest_dir}/${date} ${latest_backup_dir} # Re-create
fi
#####################################################################################
#
# RESTORE PREPARATION
# We copy all files that will be required during the recovery to the destination
# folder as well.
#
#####################################################################################
# Create directories
mkdir -p ${MYBACK_DEST_DIR}
mkdir ${MYBACK_DEST_DIR}'restore'
mkdir ${MYBACK_DEST_DIR}'restore/apt'
mkdir ${MYBACK_DEST_DIR}'log'
mkdir ${MYBACK_DEST_DIR}'app'
mkdir ${MYBACK_DEST_DIR}'tools'
# Export list of all installed applications and repository keys
apt_dir=${MYBACK_DEST_DIR}'restore/apt/'
cp /etc/apt/sources.list ${apt_dir}'sources.list'
if [ -f '/etc/apt/apt.conf' ]; then
cp /etc/apt/apt.conf ${apt_dir}'apt.conf'
fi
if [ -f '/etc/apt/preferences' ]; then
cp /etc/apt/preferences ${apt_dir}'preferences'
fi
cp -R /etc/apt/sources.list.d/ ${apt_dir}
cp -R /etc/apt/apt.conf.d/ ${apt_dir}
cp -R /etc/apt/preferences.d/ ${apt_dir}
cp -R /var/lib/apt/lists/ ${apt_dir}
apt-key key exportall > ${MYBACK_DEST_DIR}'restore/repositories.keys'
# Solution 1: dpkg selectiom -> Buggy (as of 12.10), re-import won't work
dpkg --get-selections > ${MYBACK_DEST_DIR}'restore/installed-packages.lst'
# Solution 2: apt-mark -> Buggy as well (as of 12.10)
apt-mark showauto > ${MYBACK_DEST_DIR}'restore/pkgs_auto.lst'
apt-mark showmanual > ${MYBACK_DEST_DIR}'restore/pkgs_manual.lst'
# Solution 3: Manual approach -> Create list of packages to re-install one by on recovery
package_list=$(dpkg-query -Wf '${Package} ')
package_list=${package_list// /$'\n'} # change the semicolons to white space
for package in $package_list; do
echo "$package" >> ${MYBACK_DEST_DIR}'restore/apt-install.lst'
done
# Copy tmp logfile to current destination directory
mv ${tmp_log_file} ${MYBACK_DEST_DIR}"log/myback.log"
# Copy myback system files to backup as well
cp -R ${MYBACK_BASEDIR}'app/' ${MYBACK_DEST_DIR}
cp -R ${MYBACK_BASEDIR}'tools/' ${MYBACK_DEST_DIR}
cp -R ${MYBACK_BASEDIR}'restore/' ${MYBACK_DEST_DIR}
cp ${MYBACK_BASEDIR}"excludes.txt" ${MYBACK_DEST_DIR}'/'
cp ${MYBACK_BASEDIR}"exec.sh" ${MYBACK_DEST_DIR}'/'
#####################################################################################
#
# FINISH
#
#####################################################################################
# Notify UI (multiline intended)
process_end_date_time=`date "+%Y-%m-%dT%H:%M:%S"`
finish_msg="Backup process finished at ${process_end_date_time}
Destination directory: ${current_bak_dest_dir}
Logfile: ${MYBACK_DEST_DIR}log/myback.log"
display_notification "MyBack - Backup finished" "${finish_msg}" "dialog-ok"
excludes.txt
# ======================================================= # This is the exclude file for the rsync backup. # # Notice: # All path entries are interpreted relative to the source # directories. # ======================================================= # Universal excludes ######################## lost+found ld.so.cache *.log *.bak # Root file system ######################## - /bin/ - /boot/ - /cdrom/ - /dev/ - /etc/modules.conf - /lib/ - /lib32/ - /lib64/ - /media/ - /mnt/ - /proc/ - /run/ - /sbin/ - /selinux/ - /sys/ - /tmp/ + /usr/ + /usr/share/ + /usr/local/ + /usr/local/share/ - /usr/local/* - /usr/* - /var/cache/ - /var/crash/ - /var/lock/ - /var/log/ - /var/run/ - /var/tmp/ - /var/lib/sudo/ - initrd.img.old - initrd.img - vmlinuz - vmlinuz.old # Filters for home dirs ######################## # Cache - /home/*/.cache/ # Downloads - /home/*/Downloads/ # Dropbox - /home/*/Dropbox # Temporary files / cache - /home/*/.local/share/Trash - /home/*/.cache - /home/*/.Trash # X Windows System - /home/*/.xsession-errors* # Several ######################## # Exclude backup text files - *~ - \#*\# # Commonly distributed Mac OS X cache - .DS_Store # Commonly distributed Windows cache - Thumbs.db # Common Applications ######################## # Adobe Reader - /home/*/.adobe/**/AssetCache/ - /home/*/.adobe/**/Cache/ - /home/*/.adobe/**/Temp/ - /home/*/.adobe/**/UserCache.bin # Dropbox temp stuff - /home/*/.dropbox/ - /home/*/.dropbox-dist/ # Gimp - /.gimp-*/tmp - /.gimp-*/swap # Mozilla Firefox - /home/*/.mozilla/firefox/*/Cache/ - /home/*/.mozilla/firefox/*/lock - /home/*/.mozilla/firefox/*/.parentlock # Mozilla Thunderbird - Do NOT backup emails (profiles only) - /home/*/.thunderbird/*/lock - /home/*/.thunderbird/*/.parentlock - /home/*/.thunderbird/*/ImapMail/ # Pidgin (accounts.xml contains passwords in clear text) - /home/*/.purple/accounts.xml
restore.sh
#!/bin/bash
# Copyright (c) 2013 Christopher Will<dev@cwill-dev.com>
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of
# this software and associated documentation files (the "Software"), to deal in the
# Software without restriction, including without limitation the rights to use,
# copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
# Software, and to permit persons to whom the Software is furnished to do so, subject
# to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
# PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# Functions
########################################################################
# Helper method to request user interaction
# Args
# 1: Message to display
function pause() {
read -p "$*"
}
# Initialization
########################################################################
# We mast be lord of the system
if [ "$(whoami)" != "root" ]; then
echo 'Aborting: You must be root to execute this script.'
exit 1
fi
# Root directory of the backedup files (per definition two levels up)
BAK_SOURCES="$( cd "$( dirname "${BASH_SOURCE[0]}" )/../../../current/" && pwd )/"
BAK_APT_DIR=${BAK_SOURCES}'_myback_/restore/apt/'
# Hello World
########################################################################
echo "======================================"
echo "MyBack - Restore Script"
echo "======================================"
echo ""
echo "Step 1: Replace package information"
echo "Step 2: Add repository keys"
echo "Step 3: Update source lists"
echo "Step 4: Restore and install packages"
echo "Step 5: Restore system"
echo "Step 6: Reboot"
echo ""
echo "Good luck my friend - may the force be with you!"
echo "--------------------------------------"
echo ""
# Replace local package sources information with our backup
########################################################################
echo ""
echo "> ------------------------------------"
echo "> 1/6: Replace package information"
echo "> ------------------------------------"
echo ""
pause "> Press [Enter] key to continue..."
cp ${BAK_APT_DIR}'sources.list' /etc/apt/sources.list
if [ -f ${BAK_APT_DIR}'apt.conf' ]; then
cp ${BAK_APT_DIR}'apt.conf' /etc/apt/apt.conf
fi
if [ -f ${BAK_APT_DIR}'preferences' ]; then
cp ${BAK_APT_DIR}'preferences' /etc/apt/preferences
fi
cp -R ${BAK_APT_DIR}sources.list.d/ /etc/apt/sources.list.d/
cp -R ${BAK_APT_DIR}apt.conf.d/ /etc/apt/apt.conf.d/
cp -R ${BAK_APT_DIR}preferences.d/ /etc/apt/preferences.d/
cp -R ${BAK_APT_DIR}lists/ /var/lib/apt/lists/
echo ""
echo "> [DONE]"
echo ""
# Add repository keys to system
########################################################################
echo ""
echo "> ------------------------------------"
echo "> 2/6: Add repository keys"
echo "> ------------------------------------"
echo ""
pause "> Press [Enter] key to continue..."
apt-key add ${BAK_SOURCES}_myback_/restore/repositories.keys
echo "> [DONE]"
echo ""
# Update the sources list
########################################################################
echo ""
echo "> ------------------------------------"
echo "> 3/6: Update and upgrade packages"
echo "> ------------------------------------"
echo ""
pause "> Press [Enter] key to continue..."
apt-get update
apt-get -y upgrade
echo "> [DONE]"
echo ""
# Restore backuped packages
########################################################################
echo ""
echo "> ------------------------------------"
echo "> 4/6: Restore packages (long process)"
echo "> ------------------------------------"
echo ""
pause "> Press [Enter] key to continue..."
#S olution 1: dpkg selection
# Damn dpkg.. too bugy to use this simple stuff!
#dpkg --clear-selections
#dpkg --set-selections < ${BAK_SOURCES}_myback_/restore/installed-packages.lst
#apt-get update
#apt-get dselect-upgrade
# Solution 2: apt-mark
# Damn apt-mark
#apt-mark auto $(cat ${BAK_SOURCES}_myback_/restore/pkgs_auto.lst)
#apt-mark manual $(cat ${BAK_SOURCES}_myback_/restore/pkgs_manual.lst)
# Solution 3: Manual install packages one-by-one
while read p; do
if [[ -n "$p" ]]; then
echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
echo "> "${p}
echo ">>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>"
apt-get -y install $p
fi
done < ${BAK_SOURCES}_myback_/restore/apt-install.lst
# Remove unneeded packages
apt-get -y autoremove
echo "> [DONE]"
echo ""
# Copy backup
########################################################################
echo ""
echo "> ------------------------------------"
echo "> 5/6: Restore system (long process)"
echo "> ------------------------------------"
echo ""
pause "> Press [Enter] key to continue..."
rsync -av --exclude="/_myback_/" ${BAK_SOURCES} /
echo "> [DONE]"
echo ""
# Reboot
########################################################################
echo ""
echo "> ------------------------------------"
echo "> 6/6: Reboot"
echo "> ------------------------------------"
echo ""
echo "Awesome! We are done."
echo "Hope to see you back after reboot :-)"
echo ""
pause "> Press [Enter] key to finish!"
reboot
echo "> [DONE]"
vbox_merge_snapshots.sh
#!/bin/bash
# Copyright (c) 2013 Christopher Will<dev@cwill-dev.com>
#
# Permission is hereby granted, free of charge, to any person obtaining a copy of
# this software and associated documentation files (the "Software"), to deal in the
# Software without restriction, including without limitation the rights to use,
# copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the
# Software, and to permit persons to whom the Software is furnished to do so, subject
# to the following conditions:
#
# The above copyright notice and this permission notice shall be included in all
# copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED,
# INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A
# PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
# OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
# SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
# Hello World
echo '========================================'
echo '= THIS SCRIPT COMES WITH NO WARRANTY ='
echo '= Bash version '${BASH_VERSION}' ='
echo '= [CTRL] + [Z] to kill process ='
echo '========================================'
# Exit with message and error code
function failure() {
echo ''
echo 'ERROR:'$1
echo ''
exit 1
}
# Check whether this script got called as super user. If not, we can not
# proceed.
if [ "$(whoami)" != "root" ]; then
failure 'Aborting: You must be root to execute this script.'
fi
# Get all users
declare -a users
tmp_users="$( getent passwd | grep /home/ | cut -d ':' -f 1 )"
for u in ${tmp_users}; do
tmp_dir="/home/${u}"
if [ -d "$tmp_dir" ]; then
users+=(${u})
fi
done
# Do delete/merge all snapshots
for ((i=0;i<${#users[@]};i++)); do
echo "> Processing user: ${users[i]}"
# Get all installed VMs
vms=$(su -c "VBoxManage list vms | sed -E 's/^\"(.*)\".*/\1/g'" -s /bin/sh ${users[i]})
# Loop through each VM
for vm_name in $vms; do
echo "> > Processing VM ${vm_name}.."
# Loop through each snapshot and delete/merge it
snapshots=$(su -c "VBoxManage showvminfo ${vm_name} --machinereadable | grep SnapshotName | cut -d '\"' -f2" -s /bin/sh ${users[i]})
for snapshot_name in $snapshots; do
echo "> > > Merging snapshot: ${snapshot_name}.."
snapshots=$(su -c "VBoxManage snapshot ${vm_name} delete ${snapshot_name}" -s /bin/sh ${users[i]})
done
done
done
Thanks
Further thanks to following posts, pages, documentations etc
http://stackoverflow.com/questions/13659202/rsync-complex-filter
http://askubuntu.com/questions/9135/best-way-to-backup-all-settings-list-of-installed-packages-tweaks-etc/99151#99151
http://askubuntu.com/questions/183010/apt-get-unmet-dependencies-try-apt-get-f-install-with-no-packages-or-speci
http://askubuntu.com/questions/140246/how-do-i-resolve-unmet-dependencies
http://askubuntu.com/questions/243387/how-can-i-backup-my-programs-applications-so-that-after-i-format-my-linux-and-i
http://serverfault.com/questions/150269/complex-includes-excludes-with-rsync
http://itefix.no/i2/content/excluding-directories-cwrsync
http://ubuntuforums.org/showthread.php?t=35087
http://ubuntuforums.org/showthread.php?t=2045187
http://ubuntuforums.org/showthread.php?t=1533494
http://ubuntuforums.org/showthread.php?t=2133648
http://www.virtualbox.org/manual/ch08.html#idp21979584
http://ubuntuforums.org/showthread.php?t=1071892
https://help.ubuntu.com/12.10/serverguide/automatic-updates.html
Conclusion
Please feel free to commit bugs-reports, things that might be done better or any kind of improvement. Thanks.
Installing Oracle’s Java (7) on Ubuntu (12.10)
April 3rd, 2013Use case
Since the Java installation handling changed during the last Ubuntu releases (due to Oracle’s politics) most parts of my old post are out of date now.
Solution
I found a great instruction about how to install Oracle’s Java JDK on Ubuntu.
As I could not write it better I will just post the link here:
http://www.wikihow.com/Install-Oracle-Java-JDK-on-Ubuntu-Linux
Side note
I prefer archiving several JDK versions, so I put each JDK under /opt/java/JDK-{VERSION}.
In order to not be forced to update my alternatives everytime when a new update gets released I created a symbolic link called “current” in the same directory (/opt/java/).
So in case of a new version I just need to update the symlink once.
German Umlauts on US-Keyboard on (K)Ubuntu
April 3rd, 2013Use case
As a software developer I prefer using keyboards with US-layout. Mainly because special characters (like []{}\|<>) are more easy to write/reach – some even without a modifier.
But, for email and stuff, I still rely on German Umlauts.
Solution
So this is what I did do use German Umlauts even with US-keyboard.
Create a .Xmodmap file in your home directory
nano .Xmodmap
And add following lines:
keycode 108 = Mode_switch keycode 26 = e E EuroSign EuroSign e E keycode 30 = u U udiaeresis Udiaeresis u U keycode 32 = o O odiaeresis Odiaeresis o O keycode 38 = a A adiaeresis Adiaeresis a A keycode 39 = s S ssharp ssharp s S
Conclusion
That’s it. Now you can access both the lower- and upper-case Umlauts by using the right ALT key.
If you don’t want to restart your X-server to see the changes you can re-import the xmodmap settings as follow (again in your home directory):
xmodmap .Xmodmap
Lenovo T530, Kubuntu 12.10, NVIDIA Optimus, External Monitors
April 3rd, 2013Just a couple of days after I wrote this post NVIDIA released a new version of its driver that is optimized for the Optimus technology. See http://steamforlinux.com/?q=en/node/210 for further information.
Use case
I bought a new Lenovo Thinkpad T530 laptop incl. a docking station recently. I was facing several problems regarding the docking station, the grafix card and external monitors. After spending couple of hours to get this all up and running I decided to share my experiences.
Maybe it will help someone, or maybe someone may help me by providing alternatives and/or better solutions.
Specs
Hardware
i7-3630QM, NVIDIA NVS 5400M with Optimus, SATA3 256GB CRUCIAL m4 SSD, 16 GB Kingston Memory KTL-TP3C DDR3 1600 MHz, Mini Dock Plus, Series 3 433810U, Two external Philips 23″ monitors connected to the docking station via DVI
Software
3.5.0-17-generic #28-Ubuntu SMP Tue Oct 9 19:31:23 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux, Qt: 4.8.3, KDE Development Platform: 4.9.5, kde4-config: 1.0
After removing Windows 8 by formatting the SSD and installing Kubuntu 12.10 I was struggling around with som weired problems regarding the NVIDIA Optimus card.
What happened so far (messed up)
I tried almost two days to get my two external monitors work together with the laptop display. But, unfortunately, I failed. Well, I even did mess up my system that much that I needed to re-install my system. This mostly is based on the proprietary NVIDIA driver – which broke during both the installation and the de-installation scripts. And I could not figure it out to remove all the NVIDIA stuff manually.
My sincerely advice: DO NOT USE the official NVIDIA drivers!
Even though NVIDIA might give you the impression that the specific grafic card will work fine with your specific system. But it won’t.
For those who might proove me wrong, please try it on your own (and please leave a comment if you succeed):
http://www.nvidia.com/object/linux-display-amd64-310.44-driver.html
BIOS settings
Luckily, the T530 provides an BIOS-option to chose a particular grafic card.
Integrated – Intel’s integrated grafic, NO NVIDIA
Discret - The NVIDIA card
Optimus – Both
In the latter case the OS decides when to switch between the integrated one and the NVIDIA card. Guess this should be fine running on Windows 7 or 8. But the Linux-support does just barely exist.
The idea behind the Optimus technology is to prevent the laptop from draining battery: The NVIDIA card gets activated only if it’s necessary. Great idea though.
Since I use my laptop always on AC (and have a 9-cell battery just for case) I do not care very much about batterie’s lifetime. This is why I’m satisfied by chosing the discrete option.
Driver installation
So, the simplest yet most effective and promising way is to install nvidia-current and nvidia-settings.
Contrary to various recommendations to enable the repository ppa:ubuntu-x-swat/x-updates
I got no luck with that. So I did not add or change any repositories.
sudo apt-get install nvidia-current nvidia-settings
After the installation you have to let NVIDIA configure the xorg.conf for you.
sudo nvidia-xconfig
Now you need to reboot your system.
Setup
Two external monitors
No you can run
nvidia-settings
to configure your monitors.
I configured my two external monitors as TwinView. The first (DFP-2) positioned absolutely, the second one (DFP-3) relatively to the first one.
The third display (laptop, DFP-0) was disabled by default – After playing around a couple of times I accepted that this was the only solution that will work.
When you are done just save the settings to the xorg.conf (create backup before!) and reboot your system (or just X).
Laptop display only
Well, the default scenario just works fine now. But what if you want to undock your laptop or use it without the external monitors.
Hm, I don’t want to disappoint you, but the former task gave me huge problems (=black screens), and I still can’t switch screens while the system is running. If you have any clue or even a solution please be so kind to leave a comment.
As you might have seen already the nvidia-settings take use of the metamodes options in the xorg.conf. This is pretty fine, since we can define multiple metamodes options which get handled in sort and as fallbacks.
So I configured it that way that the X-server will search for my two external monitors on startup, if they don’t exist the laptop’s display gets used.
These are my settings:
Option "metamodes" "DFP-2: 1920x1080 +0+0, DFP-3: 1920x1080 +1920+0" Option "metamodes" "LVDS-0: 1920x1080 +0+0"
xorg.conf
This is my entire xorg.conf
# nvidia-settings: X configuration file generated by nvidia-settings
# nvidia-settings: version 304.51 (buildd@batsu) Fri Oct 12 12:53:54 UTC 2012
Section "ServerLayout"
Identifier "Layout0"
Screen 0 "Screen0" 0 0
InputDevice "Keyboard0" "CoreKeyboard"
InputDevice "Mouse0" "CorePointer"
Option "Xinerama" "0"
EndSection
Section "Files"
EndSection
Section "InputDevice"
# generated from default
Identifier "Mouse0"
Driver "mouse"
Option "Protocol" "auto"
Option "Device" "/dev/psaux"
Option "Emulate3Buttons" "no"
Option "ZAxisMapping" "4 5"
EndSection
Section "InputDevice"
# generated from default
Identifier "Keyboard0"
Driver "kbd"
EndSection
Section "Monitor"
# HorizSync source: edid, VertRefresh source: edid
Identifier "Monitor0"
VendorName "Unknown"
ModelName "Philips 234CL"
HorizSync 30.0 - 83.0
VertRefresh 56.0 - 76.0
Option "DPMS"
EndSection
Section "Device"
Identifier "Device0"
Driver "nvidia"
VendorName "NVIDIA Corporation"
BoardName "NVS 5400M"
EndSection
Section "Screen"
Identifier "Screen0"
Device "Device0"
Monitor "Monitor0"
DefaultDepth 24
Option "Stereo" "0"
Option "nvidiaXineramaInfoOrder" "DFP-2"
Option "metamodes" "DFP-2: 1920x1080 +0+0, DFP-3: 1920x1080 +1920+0"
Option "metamodes" "LVDS-0: 1920x1080 +0+0"
SubSection "Display"
Depth 24
EndSubSection
EndSection
Side note
I’m very open minded to mistakes I made, or clue’s to help the (un)docking problems, or better solutions.
PS: Jep, I played around with bumblebee, but did not end happy either.
Encryption between Java/Android and PHP
October 9th, 2012Use case
While working on an android app that retrieves it’s data from a PHP-API I needed to build a both-sided en- and decryption layer to secure the requests and responses properly.
I decided to use an AES encryption with IvParameterSpec, SecretKeySpec and “AES/CBC/PKCS5Padding” on Java side, and mcrypt_module “rijndael-128″ on PHP side.
Implementations
PHP
<?
class ApiCrypter
{
private $iv = 'fdsfds85435nfdfs'; #Same as in JAVA
private $key = '89432hjfsd891787'; #Same as in JAVA
public function __construct() {
}
public function encrypt($str) {
$str = $this->pkcs5_pad($str);
$iv = $this->iv;
$td = mcrypt_module_open('rijndael-128', '', 'cbc', $iv);
mcrypt_generic_init($td, $this->key, $iv);
$encrypted = mcrypt_generic($td, $str);
mcrypt_generic_deinit($td);
mcrypt_module_close($td);
return bin2hex($encrypted);
}
public function decrypt($code) {
$code = $this->hex2bin($code);
$iv = $this->iv;
$td = mcrypt_module_open('rijndael-128', '', 'cbc', $iv);
mcrypt_generic_init($td, $this->key, $iv);
$decrypted = mdecrypt_generic($td, $code);
mcrypt_generic_deinit($td);
mcrypt_module_close($td);
$ut = utf8_encode(trim($decrypted));
return $this->pkcs5_unpad($ut);
}
protected function hex2bin($hexdata) {
$bindata = '';
for ($i = 0; $i < strlen($hexdata); $i += 2) {
$bindata .= chr(hexdec(substr($hexdata, $i, 2)));
}
return $bindata;
}
protected function pkcs5_pad ($text) {
$blocksize = 16;
$pad = $blocksize - (strlen($text) % $blocksize);
return $text . str_repeat(chr($pad), $pad);
}
protected function pkcs5_unpad($text) {
$pad = ord($text{strlen($text)-1});
if ($pad > strlen($text)) {
return false;
}
if (strspn($text, chr($pad), strlen($text) - $pad) != $pad) {
return false;
}
return substr($text, 0, -1 * $pad);
}
}
?>
JAVA
package com.cwilldev.crypt;
import java.security.NoSuchAlgorithmException;
import javax.crypto.Cipher;
import javax.crypto.NoSuchPaddingException;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
public class ApiCrypter {
private String iv = "myuniqueivparam";
private String secretkey = "mysecretkey";
private IvParameterSpec ivspec;
private SecretKeySpec keyspec;
private Cipher cipher;
public ApiCrypter()
{
ivspec = new IvParameterSpec(iv.getBytes());
keyspec = new SecretKeySpec(secretkey.getBytes(), "AES");
try {
cipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
} catch (NoSuchAlgorithmException e) {
e.printStackTrace();
} catch (NoSuchPaddingException e) {
e.printStackTrace();
}
}
public byte[] encrypt(String text) throws Exception
{
if(text == null || text.length() == 0) {
throw new Exception("Empty string");
}
byte[] encrypted = null;
try {
cipher.init(Cipher.ENCRYPT_MODE, keyspec, ivspec);
encrypted = cipher.doFinal(text.getBytes("UTF-8"));
}
catch (Exception e) {
throw new Exception("[encrypt] " + e.getMessage());
}
return encrypted;
}
public byte[] decrypt(String code) throws Exception
{
if(code == null || code.length() == 0) {
throw new Exception("Empty string");
}
byte[] decrypted = null;
try {
cipher.init(Cipher.DECRYPT_MODE, keyspec, ivspec);
decrypted = cipher.doFinal(hexToBytes(code));
}
catch (Exception e) {
throw new Exception("[decrypt] " + e.getMessage());
}
return decrypted;
}
public static String bytesToHex(byte[] data)
{
if (data==null) {
return null;
}
int len = data.length;
String str = "";
for (int i=0; i<len; i++) {
if ((data[i]&0xFF)<16) {
str = str + "0" + java.lang.Integer.toHexString(data[i]&0xFF);
}
else {
str = str + java.lang.Integer.toHexString(data[i]&0xFF);
}
}
return str;
}
public static byte[] hexToBytes(String str) {
if (str==null) {
return null;
}
else if (str.length() < 2) {
return null;
}
else {
int len = str.length() / 2;
byte[] buffer = new byte[len];
for (int i=0; i<len; i++) {
buffer[i] = (byte) Integer.parseInt(str.substring(i*2,i*2+2),16);
}
return buffer;
}
}
}
Encryption
Java
String encryptedRequest = ApiCrypter.bytesToHex(this.apiCrypter.encrypt(jsonParams.toString()));
PHP
$encrypted = $ApiCrypter->encrypt($str);
Decryption
Java
String res = new String( this.apiCrypter.decrypt( text ), "UTF-8" ); res = URLDecoder.decode(res,"UTF-8");
PHP
$decrypted = $ApiCrypter->decrypt($str);
IDEA hook: Grails not found!
October 9th, 2012Use case
While working with different grails versions and a maven repository in the background I faced some weird problems regarding version problems which IDEA was complaining about.
Incident
The stacktrace looked as follow:
[INFO] ------------------------------------------------------------------------ [ERROR] FATAL ERROR [INFO] ------------------------------------------------------------------------ [INFO] org.grails.maven.plugin.AbstractGrailsMojo.runGrailsInline(Ljava/lang/String;Ljava/lang/String;Z)V [INFO] ------------------------------------------------------------------------ [INFO] Trace java.lang.NoSuchMethodError: org.grails.maven.plugin.AbstractGrailsMojo.runGrailsInline(Ljava/lang/String;Ljava/lang/String;Z)V at org.grails.maven.plugin.GrailsExecMojo.execute(GrailsExecMojo.java:51) at org.apache.maven.plugin.DefaultPluginManager.executeMojo(DefaultPluginManager.java:490) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoals(DefaultLifecycleExecutor.java:694) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeStandaloneGoal(DefaultLifecycleExecutor.java:569) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoal(DefaultLifecycleExecutor.java:539) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeGoalAndHandleFailures(DefaultLifecycleExecutor.java:387) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.executeTaskSegments(DefaultLifecycleExecutor.java:284) at org.apache.maven.lifecycle.DefaultLifecycleExecutor.execute(DefaultLifecycleExecutor.java:180) at org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:328) at org.apache.maven.DefaultMaven.execute(DefaultMaven.java:138) at org.apache.maven.cli.MavenCli.main(MavenCli.java:362) at org.apache.maven.cli.compat.CompatibleMain.main(CompatibleMain.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.codehaus.classworlds.Launcher.launchEnhanced(Launcher.java:315) at org.codehaus.classworlds.Launcher.launch(Launcher.java:255) at org.codehaus.classworlds.Launcher.mainWithExitCode(Launcher.java:430) at org.codehaus.classworlds.Launcher.main(Launcher.java:375) [INFO] ------------------------------------------------------------------------ [INFO] Total time: 11 seconds [INFO] Finished at: Wed Sep 05 10:44:12 CEST 2012 [INFO] Final Memory: 33M/743M [INFO] ------------------------------------------------------------------------ java.lang.NullPointerException at org.jetbrains.groovy.grails.rt.Agent$2.run(Agent.java:99) at java.lang.Thread.run(Thread.java:662) IDEA hook: Grails not found!
Neither grails clean, nor any re-builds vanished this problem.
Solution
After searching the web I could figure out that it has to with the JARs handling and cache of both grails and IDEA.
- http://youtrack.jetbrains.com/issue/IDEA-86261
- http://grails.1312388.n4.nabble.com/Can-not-create-GDK-in-Intellij-IDEA-11-with-2-1-RC1-But-no-problem-while-use-2-0-3-td4628641.html
- http://youtrack.jetbrains.com/issue/IDEA-86250
- http://stackoverflow.com/questions/11980623/sync-error-on-a-grails-project-in-intellij
- http://devnet.jetbrains.net/thread/437217
Finally, after removing the .ivy2- and maven-cache the problem did not occur anymore.
Enable Zend PHP Debugging under IntelliJ 11.1
April 23rd, 2012Use case
In this tutorial I assume that you have already installed and configured the Web server and the PHP engine properly.
If not, please have a look at this tutorial.
Solution
In our configuration we use all default properties (like debug port, broadcasting port etc).
Enable PHP plugin (PHPStorm)
- Select your project and open the settings menu.
- Under the project settings you find a PHP entry
- Select the PHP version
- Confiure an interpreter
- Click on the … button
- Click the plus button in the left-handed upper corner
- Chose a name (something like PHP<5.3.6>)
- Under General->PHP Home point to you PHP location (in my case /usr/bin)
- As debugger choose the Zend Debugger
- Submit the form
- If you don’t see the newly configured interpreter directly push the refresh button
- Now you should see the new interpreter (PHP<5.3.6>)
- By clicking on the question mark you can see an overview of your PHP configuration.
Notice that you dont see any debug parts yet. you should not
Download Zend Debugger
- Download the Zend Debugger package which corresponds to your operating system here (Studio Web Debugger)
- Locate the ZendDebugger.so file in the directory which corresponds to your version of PHP (e.g. 4.3.x, 4.4.x, 5.0.x, 5.1.x, 5.2.x)
- Copy the file to your Web server in a location that is accessible by the Web server (like /home/foo, /opt/bar or whereever you like the file to be stored).
Enable Zend Debugger in PHP
- Create a phpinfo-page (HowTo)
- Search for the key “Loaded Configuration File” to see which php.ini your system is using
- Edit the php.ini
- Add or extend the zend-section by adding following settings:
[Zend] zend_extension=<path_to_ZendDebugger.so zend_debugger.allow_hosts=127.0.0.1 ;zend_debugger.allow_hosts=127.0.0.1,localhost zend_debugger.expose_remotely=allowed_hosts
- Restart apache.
Create Debug-Configuration in IntelliJ
- Click on “Edit Configurations” (it’s located in the toolbar next to the play-button)
- Add a new configuration- Chose “PHP Web Application” as type
- Set a unique name for your project debug configuration
- Chose “localhost” as server- Point the start URL to the project http root
- Click OK
Debug
- Set a breakpoint somewhere in your code (that of course gets executed)
- Start the debugger (debug icon next to the play button)
- Open (if IntelliJ did not already opened a browser on its own) a browser and access the root URL
Conclusion
Now IntelliJ should stop at your breakpoint.
Error on SVN checkout: SSL handshake failed: SSL error: Key usage violation in certificate has been detected.
March 28th, 2012Use case
Lately I received the following error message, while trying to checkout from a SVN repository located on a Windows server from my Kubuntu machine:
svn: Commit failed (details follow): svn: OPTIONS of 'https://HOSTNAME/svn/repo': SSL handshake failed: SSL error: Key usage violation in certificate has been detected. (https://HOSTNAME)
Solution
A short search pointed out, that this issue occures because of an unsigned security certificate.Have a look at this post.
In my case I just needed to install libneon
$ sudo apt-get install libneon27
And then to remove the previous symbolic link by replacing it with a new one pointing to gnutls:
sudo mv /usr/lib/libneon-gnutls.so.27 /usr/lib/libneon-gnutls.so.27.old sudo ln -s /usr/lib/libneon.so.27 /usr/lib/libneon-gnutls.so.27
This worked for me, even though I had to restart my system surprisely.