Moving to KVM virtual machines

Installing VirtualBox is getting increasingly painful on Ubuntu due to the problems with UEFI Secure Boot and the VirtualBox kernel modules. Another reason for an alternative is that running VirtualBox VM’s completely in the background is not as straightforward as it could be.

From the available alternatives I looked into (VMWare, Xen & KVM) it was KVM that fitted my needs (casual VM usage with mostly headless VM’s for testing purposes). Main reasons:

  • Well supported by Ubuntu
  • Easy, straightforward install
  • Background VM’s are simple as
  • Moving VM’s from one host to another is a breeze

Checking system

To check if the CPU can actually support

egrep -c '(svm|vmx)' /proc/cpuinfo

If the number returned is > 0 your systems should be capable to run.

You will also enable your BIOS for virtualisation (in Security settings of most BIOS’s) if that has not already be done. You will get an error if not enabled if you are trying to run an install. The Install of KVM will work fine.

Installation

sudo apt-get install qemu-kvm libvirt-bin bridge-utils virt-manager
sudo addgroup libvirtd
sudo adduser libvirtd
sudo service libvirtd start
sudo service libvirtd status
sudo virt-manager

Moving VM’s to another host

Source Host

virsh shutdown VMNAME
virsh dumpxml VMNAME > /tmp/VMNAME.xml
scp /tmp/VMNAME.xml TARGETHOST:/tmp/VMNAME.xml
scp /var/lib/libvirt/images/VMNAME.qcow2 TARGETHOST:/var/lib/libvirt/images/VMNAME.qcow2

Target Host

virsh define /tmp/vm.xml
virsh start vm

Once you have confirmed operation you probably want to remove the source VM from the Source Host.

virsh undefine VMNAME
rm /var/lib/libvirt/images/VMNAME.qcow2

SShuttle – quick and temporary VPN over SSH

Every once in a while you find a gem. One of these for me is SShuttle – until now I have not known about this one.

Sometimes you need to quickly forward all your traffic via a remote server quickly. And while you can do all of this manually using OpenSSH it’s not a quick one-step process (https://help.ubuntu.com/community/SSH/OpenSSH/PortForwarding). Dynamic SOCKS5 proxies are great if all you need is browser traffic, but there is always software that won’t play ball with SOCKS.

Use-case: I just been trying to get Ubuntu Make to install Eclipse IDE and the local AARNET download mirror is just refusing to cooperate (https://github.com/ubuntu/ubuntu-make/issues/90). A quick forward to a remote VPS fixed the issue without headaches

Install

sudo apt-get install sshuttle

Run

sshuttle -r username@servername.tld 0.0.0.0/0 -vv

That’s all – it sets up routing & iptable rules transparently and removes them after use. Kudos goes to https://github.com/apenwarr – thank you. A VERY useful utility !!!

Source link: https://github.com/apenwarr/sshuttle

Paperless Office using the Raspberry Pi

This is a follow-up on an older blog using Ubuntu.

r by rosmary, on Flickr
Creative Commons Creative Commons Attribution 2.0 Generic License   by  rosmary 

For this purpose I used a Fujitsu ScanSnap S1300i scanner as I really like the features of this series (full duplex scan as well auto document feeder as well for around $250). It’s document feeder is not a good as the S1500 we have in the office, but very compact and can be powered from USB hub.

Raspberry Pi Prerequisites

Since this will be a purely headless install designed to sit in a corner behind the scanner I am using a Base Raspian (Debian Wheezy) install (I personally like the clean minimal install via https://github.com/debian-pi/raspbian-ua-netinst the best).

apt-get install sudo vim wget wput libusb-dev build-essential git-core

Add non-privileged user account(s)

adduser USERNAME
adduser USERNAME sudo
groupadd scanner
usermod -a -G scanner USERNAME

Install Sane

The version of sane from the Raspbian repos is not working with the Fujitsu ScanJet range and needs to be built from source.

git clone git://git.debian.org/sane/sane-backends.git
cd sane-backends
BACKENDS=epjitsu ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
make
make install

Install S1300i Driver

You need to get the driver file (‘1300i_0D12.nal’) from the CD that came with the scanner. If you still have access to a CDROM drive that is. :(

mkdir -p /usr/share/sane/epjitsu/
cp 1300i_0D12.nal /usr/share/sane/epjitsu/

Check /etc/sane.d/epjitsu.conf and see if the following line is there (in my case it was already created by sane build).

# Fujitsu S1300i
firmware /usr/share/sane/epjitsu/1300i_0D12.nal
usb 0x04c5 0x128d

sane-find-scanner -q

found USB scanner (vendor=0x04c5 [FUJITSU], product=0x128d [ScanSnap S1300i]) at libusb:001:004
found USB scanner (vendor=0x0424, product=0xec00) at libusb:001:003

scanimage -L

device `epjitsu:libusb:001:004′ is a FUJITSU ScanSnap S1300i scanner

Copy libsane rules from the sane build directory to udev rules.
sudo cp sane-backends/tools/udev/libsane.rules /etc/udev/rules.d/60-libsane.rules

Logout and log in a the non-privileged user account previously created.

If the scanimage -L command works as above you have fully configured the scanner to work under that user account.

Start saned on boot-up

Edit the /etc/rc.local file and add the following line before the ‘0’ line to ensure saned is running as the non-privileged user when you have to reboot.

saned -a USERNAME

Installing Conversion Tools

sudo apt-get install imagemagick bc exactimage pdftk tesseract-ocr tesseract-ocr-eng unpaper

You can add other languages such as tesseract-ocr-deu if you require OCR support for those.

Scan to Repository Script

The script is hosted on Github: https://github.com/leogaggl/misc-scripts/blob/master/scan2repo.sh

#!/bin/bash
# Thanks to Andreas Gohr (http://www.splitbrain.org/) for the initial work
# https://github.com/splitbrain/paper-backup/
OUT_DIR=~/scan
TMP_DIR=`mktemp -d`
FILE_NAME=scan_`date +%Y%m%d-%H%M%S`
LANGUAGE="eng"
echo 'scanning...'
scanimage --resolution 300 \
--batch="$TMP_DIR/scan_%03d.pnm" \
--format=pnm \
--mode Gray \
--source 'ADF Duplex'
echo "Output saved in $TMP_DIR/scan*.pnm"
cd $TMP_DIR
# cut borders
echo 'cutting borders...'
for i in scan_*.pnm; do
mogrify -shave 50x5 "${i}"
done
# check if there is blank pages
echo 'checking for blank pages...'
for f in ./*.pnm; do
unpaper --size "a4" --overwrite "$f" `echo "$f" | sed 's/scan/scan_unpaper/g'`
#need to rename and delete original since newer versions of unpaper can't use same file name
rm -f "$f"
done
# apply text cleaning and convert to tif
echo 'cleaning pages...'
for i in scan_*.pnm; do
echo "${i}"
convert "${i}" -contrast-stretch 1% -level 29%,76% "${i}.tif"
done
# Starting OCR
echo 'doing OCR...'
for i in scan_*.pnm.tif; do
echo "${i}"
tesseract "$i" "$i" -l $LANGUAGE hocr
hocr2pdf -i "$i" -s -o "$i.pdf" < "$i.html" done # create PDF echo 'Converting PDF...' pdftk *.tif.pdf cat output "$FILE_NAME.pdf" wput $FILE_NAME.pdf ftp://uid:pwd@scanner.domain:21/Alfresco/scans/ cp $FILE_NAME.pdf $OUT_DIR/ rm -rf $TMP_DIR

Thanks go to Andi Gohr @ Splitbrain for the excellent blog that helped me to get over the sane problems and also gave me some ideas to make the scan script better (as unpaper was not doing such a good job): http://www.splitbrain.org/blog/2014-08/23-paper_backup_1_scanner_setup

Installing libdvdcss on Ubuntu 13.10

With the demise of the Medibuntu repository and libdvdcss not being hosted in the main Ubuntu repos due to licensing issues a new repository is needed from 13.10 upwards. Thanks to the good folks at VideoLAN (makers of the awsome VLC Video Player) there is a ready and updated source available.

wget ftp://ftp.videolan.org/pub/debian/videolan-apt.asc | sudo apt-key add -
echo "deb ftp://ftp.videolan.org/pub/debian/stable ./" | sudo tee /etc/apt/sources.list.d/libdvdcss.list
sudo apt-get update
sudo apt-get install libdvdcss2

Darktable – Photo Management under Ubuntu

Whilst I am by no means a photographer I do end up taking quite a few photos (these days pretty much exclusively on my phone) and the management of these photos can be a pain. So far I have never found an program worth the pain over plain old file management.

But having stumbled across Darktable (http://darktable.org/) I think I might have found a worthwile package.

Screenshot

Install on Ubuntu:

sudo add-apt-repository ppa:pmjdebruijn/darktable-release
sudo apt-get update
sudo apt-get install darktable

Bulk converting Office documents to PDF

When you need to convert multiple documents to PDF for distribution (or from one Office format to another) there are a few utilities around. The most workable I found is the UNOCONV utility which is build on top of LibreOffice / OpenOffice. This uses the OpenOffice conversion facilities rather than a simple PDF print driver.

On Ubuntu it can be installed via Software Center or via apt-get from the core repositories.
sudo apt-get install unoconv
Combined with the -exec option of the Unix find command this makes conversion of whole directory structures a breeze.
#find all Word Documents and convert to PDF
find . -name "*.doc*" -exec unoconv -f pdf {} \;
#find all Powerpoint Documents and convert to PDF
find . -name "*.ppt*" -exec unoconv -f pdf {} \;

To show all the possible conversion formats you can use:
unoconv --show
The following list of document formats are currently available:

bib – BibTeX [.bib]
doc – Microsoft Word 97/2000/XP [.doc]
doc6 – Microsoft Word 6.0 [.doc]
doc95 – Microsoft Word 95 [.doc]
docbook – DocBook [.xml]
html – HTML Document (OpenOffice.org Writer) [.html]
odt – ODF Text Document [.odt]
ott – Open Document Text [.ott]
ooxml – Microsoft Office Open XML [.xml]
pdf – Portable Document Format [.pdf]
rtf – Rich Text Format [.rtf]
latex – LaTeX 2e [.ltx]
sdw – StarWriter 5.0 [.sdw]
sdw4 – StarWriter 4.0 [.sdw]
sdw3 – StarWriter 3.0 [.sdw]
stw – Open Office.org 1.0 Text Document Template [.stw]
sxw – Open Office.org 1.0 Text Document [.sxw]
text – Text Encoded [.txt]
mediawiki – MediaWiki [.txt]
txt – Text [.txt]
uot – Unified Office Format text [.uot]
vor – StarWriter 5.0 Template [.vor]
vor4 – StarWriter 4.0 Template [.vor]
vor3 – StarWriter 3.0 Template [.vor]
xhtml – XHTML Document [.html]

The following list of graphics formats are currently available:

bmp – Windows Bitmap [.bmp]
emf – Enhanced Metafile [.emf]
eps – Encapsulated PostScript [.eps]
gif – Graphics Interchange Format [.gif]
html – HTML Document (OpenOffice.org Draw) [.html]
jpg – Joint Photographic Experts Group [.jpg]
met – OS/2 Metafile [.met]
odd – OpenDocument Drawing [.odd]
otg – OpenDocument Drawing Template [.otg]
pbm – Portable Bitmap [.pbm]
pct – Mac Pict [.pct]
pdf – Portable Document Format [.pdf]
pgm – Portable Graymap [.pgm]
png – Portable Network Graphic [.png]
ppm – Portable Pixelmap [.ppm]
ras – Sun Raster Image [.ras]
std – OpenOffice.org 1.0 Drawing Template [.std]
svg – Scalable Vector Graphics [.svg]
svm – StarView Metafile [.svm]
swf – Macromedia Flash (SWF) [.swf]
sxd – OpenOffice.org 1.0 Drawing [.sxd]
sxd3 – StarDraw 3.0 [.sxd]
sxd5 – StarDraw 5.0 [.sxd]
tiff – Tagged Image File Format [.tiff]
vor – StarDraw 5.0 Template [.vor]
vor3 – StarDraw 3.0 Template [.vor]
wmf – Windows Metafile [.wmf]
xhtml – XHTML [.xhtml]
xpm – X PixMap [.xpm]

The following list of presentation formats are currently available:

bmp – Windows Bitmap [.bmp]
emf – Enhanced Metafile [.emf]
eps – Encapsulated PostScript [.eps]
gif – Graphics Interchange Format [.gif]
html – HTML Document (OpenOffice.org Impress) [.html]
jpg – Joint Photographic Experts Group [.jpg]
met – OS/2 Metafile [.met]
odg – ODF Drawing (Impress) [.odg]
odp – ODF Presentation [.odp]
otp – ODF Presentation Template [.otp]
pbm – Portable Bitmap [.pbm]
pct – Mac Pict [.pct]
pdf – Portable Document Format [.pdf]
pgm – Portable Graymap [.pgm]
png – Portable Network Graphic [.png]
pot – Microsoft PowerPoint 97/2000/XP Template [.pot]
ppm – Portable Pixelmap [.ppm]
ppt – Microsoft PowerPoint 97/2000/XP [.ppt]
pwp – PlaceWare [.pwp]
ras – Sun Raster Image [.ras]
sda – StarDraw 5.0 (OpenOffice.org Impress) [.sda]
sdd – StarImpress 5.0 [.sdd]
sdd3 – StarDraw 3.0 (OpenOffice.org Impress) [.sdd]
sdd4 – StarImpress 4.0 [.sdd]
sxd – OpenOffice.org 1.0 Drawing (OpenOffice.org Impress) [.sxd]
sti – OpenOffice.org 1.0 Presentation Template [.sti]
svg – Scalable Vector Graphics [.svg]
svm – StarView Metafile [.svm]
swf – Macromedia Flash (SWF) [.swf]
sxi – OpenOffice.org 1.0 Presentation [.sxi]
tiff – Tagged Image File Format [.tiff]
uop – Unified Office Format presentation [.uop]
vor – StarImpress 5.0 Template [.vor]
vor3 – StarDraw 3.0 Template (OpenOffice.org Impress) [.vor]
vor4 – StarImpress 4.0 Template [.vor]
vor5 – StarDraw 5.0 Template (OpenOffice.org Impress) [.vor]
wmf – Windows Metafile [.wmf]
xhtml – XHTML [.xml]
xpm – X PixMap [.xpm]

The following list of spreadsheet formats are currently available:

csv – Text CSV [.csv]
dbf – dBASE [.dbf]
dif – Data Interchange Format [.dif]
html – HTML Document (OpenOffice.org Calc) [.html]
ods – ODF Spreadsheet [.ods]
ooxml – Microsoft Excel 2003 XML [.xml]
ots – ODF Spreadsheet Template [.ots]
pdf – Portable Document Format [.pdf]
sdc – StarCalc 5.0 [.sdc]
sdc4 – StarCalc 4.0 [.sdc]
sdc3 – StarCalc 3.0 [.sdc]
slk – SYLK [.slk]
stc – OpenOffice.org 1.0 Spreadsheet Template [.stc]
sxc – OpenOffice.org 1.0 Spreadsheet [.sxc]
uos – Unified Office Format spreadsheet [.uos]
vor3 – StarCalc 3.0 Template [.vor]
vor4 – StarCalc 4.0 Template [.vor]
vor – StarCalc 5.0 Template [.vor]
xhtml – XHTML [.xhtml]
xls – Microsoft Excel 97/2000/XP [.xls]
xls5 – Microsoft Excel 5.0 [.xls]
xls95 – Microsoft Excel 95 [.xls]
xlt – Microsoft Excel 97/2000/XP Template [.xlt]
xlt5 – Microsoft Excel 5.0 Template [.xlt]
xlt95 – Microsoft Excel 95 Template [.xlt]

Finding a Notepad++ alternative / replacement on Ubuntu

One of the most important utilities on any computer is a decent text editing tool. Since there are as literally hundreds of text-editors out there you would think it is an easy task to find a similar one for each platform.

After years of trialing on various Windows platforms I have found Notepad++ to be the most useful of them all (closely followed by EMEditor which I used for years before Notepad++). Since I have now switched to Ubuntu completely it was necessary to find a permanent alternative on that platform since Notepad++ is only available on Windows.

The requirements list in order of priority:

  1. Fast & Native (reading NOT Java or similar runtime requirements)
  2. Excellent Search & Replace (which is where most fail ….)
  3. Lightweight
  4. Simple GUI (reading NOT an IDE)

Potential Candidates

Scite

Scite was my first try since it is based on the same engine as Notepad++. I have used it for a number of months but was never really happy with it. Issues with “Find in Files” functionality have been the final issue that make me move on. http://www.scintilla.org/SciTE.html

Scite Screenshot

Geany

I came across this one very recently and after a short trial period grown to really like it. Apart   from it’s silly name and very ugly icon it really has got the nicest overall balance of speed and functionality. And most importantly it also has very decent in-built search & replace functions. This includes “Find in Files” functionality which actually works. http://www.geany.org/

Geany Screenshot

Vim (Cream)

Since vim is my command-line text editor of choice for years I was giving the GUI version a try next to Geany. However since I am not a total vim freak (never had enough time to learn all the power of this tool) I decided to stick to a dedicated GUI text editor for long editing tasks (HTML, CSS coding) and keep vim for when I am on the command-line or a headless server.

Gedit

Being the default editor in Ubuntu I have used it a bit, but always found it very sluggish and not very intuitive to use. Apart from the fact is installed by default I did not find many things I liked about it.

Note: sometimes the find and replace needs are outside of the capabilites of any text editor and you have to resort to a specialised tool. As a GUI I have always found Regexxer (http://regexxer.sourceforge.net/) very useful. Otherwise there is always the full power of grep and sed.