Moving to KVM virtual machines

Installing VirtualBox is getting increasingly painful on Ubuntu due to the problems with UEFI Secure Boot and the VirtualBox kernel modules. Another reason for an alternative is that running VirtualBox VM’s completely in the background is not as straightforward as it could be.

From the available alternatives I looked into (VMWare, Xen & KVM) it was KVM that fitted my needs (casual VM usage with mostly headless VM’s for testing purposes). Main reasons:

  • Well supported by Ubuntu
  • Easy, straightforward install
  • Background VM’s are simple as
  • Moving VM’s from one host to another is a breeze

Checking system

To check if the CPU can actually support

egrep -c '(svm|vmx)' /proc/cpuinfo

If the number returned is > 0 your systems should be capable to run.

You will also enable your BIOS for virtualisation (in Security settings of most BIOS’s) if that has not already be done. You will get an error if not enabled if you are trying to run an install. The Install of KVM will work fine.

Installation

sudo apt-get install qemu-kvm libvirt-bin bridge-utils virt-manager
sudo addgroup libvirtd
sudo adduser libvirtd
sudo service libvirtd start
sudo service libvirtd status
sudo virt-manager

Moving VM’s to another host

Source Host

virsh shutdown VMNAME
virsh dumpxml VMNAME > /tmp/VMNAME.xml
scp /tmp/VMNAME.xml TARGETHOST:/tmp/VMNAME.xml
scp /var/lib/libvirt/images/VMNAME.qcow2 TARGETHOST:/var/lib/libvirt/images/VMNAME.qcow2

Target Host

virsh define /tmp/vm.xml
virsh start vm

Once you have confirmed operation you probably want to remove the source VM from the Source Host.

virsh undefine VMNAME
rm /var/lib/libvirt/images/VMNAME.qcow2

GrovePi Zero – connecting your IoT sensors

I recently purchased a GrovePi Zero and expected this to be a reasonable straight forward way to connect Grove sensors to your Raspberry Pi, read sensor values via Python and pushing them upstream via MQTT. However the software side of things turns out anything but straight forward. Most of the suggestions on the Dexter Industries forum suggest to download some custom OS image – WTF? Hopefully this will save some people time to chase down the same rabbit holes…..

The problem starts with the install scripts which (a pet hate of mine) assume the /home/pi account to be present. There is a variable in at least the 2 of the install scripts (another Github repository – one calls another from a Github URL – like that will ever work ???) which allows you to change the user account but there is also hard-coded references all over the place that still point to that user account. Trying to step manually through the dependency hell in these install scripts that call other scripts ended to be a complete nightmare. It seems to download specific (old !?!) versions of libraries all of which have perfectly fine binaries in the Raspbian repos. Most of the troubleshooting tips use the UI which is not much good on a headless system.

Since all I wanted was to read the sensors from Python I found that there was a library on PyPi (https://pypi.python.org/pypi/grovepi) which should actually do this without installing all this redundant dependencies on what is essentially a remote sensor aggregator. Got hopes up too quickly. Unfortunately turns out that the firmware on these devices is really old and riddled with bugs (could not read from most sensors I tried and all were known issues fixed in later firmware releases. Trying to manually install some of the dependencies (such as avrdude which are all in the Raspbian repositories) for the simple firmware update script turned out useless as they use a custom ‘gpio’ programmer not present. If you are splitting things into multiple Github repositories (which is in itself a good idea) why don’t be consistent about it and create one for the firmware updater alone? You shouldn’t have to compile half of Github on a low-powered device just to flash a firmware file.

Looks like the only solution is to bite the bullet and get another SD card with a default Raspbian image and go through the whole painful experience of installing the whole dependency hell and compiling everything on a RaspberryPi Zero which is like watching paint dry. Hopefully this should then just work on another device that only needs the Python library as a dependency.

Let’s hope so… [will update once finished jumping through all the hoops]

Set up a Raspberry Pi Zero headless

If you are using the GUI (Raspian full download) and want to connect your RPi Zero to a keyboard and monitor there are probably easier ways to do this. These notes are for people that want to use a headless (no monitor and GUI) setup ready to connect to your RPi after first boot via SSH from another terminal.

Download Raspbian Lite

wget -O raspbian-lite-latest.zip https://downloads.raspberrypi.org/raspbian_lite_latest

Download link: https://www.raspberrypi.org/downloads/raspbian/

Write Image to SD Card

dd bs=4M if=2017-08-16-raspbian-stretch-lite.img of=/dev/sdb

After this step there should be 2 additional mounts (if not mount the 2 SD card partitions manually).

Set up network interfaces

Create a new config file for the wireless interface.

cd etc/network/interfaces.d/
vim wlan0.conf

Add the following to the new file (if you want to use DHCP – change to static if you want to fix the IP):

auto wlan0
allow-hotplug wlan0
iface wlan0 inet dhcp
wpa-conf /etc/wpa_supplicant/wpa_supplicant.conf

Set up connection details for local wireless network

vim etc/wpa_supplicant/wpa_supplicant.conf

Check the file and ensure the network settings are as per the Wireless Network you are connecting to.

country=AU
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
ssid="SSID"
psk=HASHED_PSK_VALUE
}

You could use the clear-text PSK passphrase but I prefer not to do that in configuration files. Create a PSK hash using ‘wpa_passphrasethis older article

There could be other config values you might need depending on the network you are connecting to. More info: https://linux.die.net/man/5/wpa_supplicant.conf.

Ensure there are DNS Servers configured

sudo vim etc/resolv.conf

Add the following to the new file (or your own nameservers):

# Google's public DNS servers
nameserver 8.8.8.8
nameserver 8.8.4.4

Enable SSH Access

Create an ssh empty file in the mounted boot partition. This is necessary to enable SSH access as SSH is by default disabled on more recent versions of Raspbian.

cd boot/
sudo touch ssh

Boot RPi

Boot the Rpi by powering up and after ~30secs you should be able to connect to the IP assigned. Check your router or ise ‘nmap’ or similar to check the assigned IP:

sudo nmap -p22 -sV 10.1.1.0/24

Default login details are
UID: pi
PWD: rasbperry

My normal further setup steps for RPi 3 are here.

EDIT [2017-08-21]: One of the things necessary on later RPi’s is to turn off power saving mode on the WLAN interface.

sudo iw dev wlan0 set power_save off

Using DNSMadeEasy as Dynamic DNS provider on Synology Diskstations

Since Synology (despite requests) still has not added DNS Made Easy as a listed provider (despite listing some really obscure services – go figure!) here is the steps to add a custom provider.

DNS Made Easy Setup

Create a new A-Record

  1. Set the name
  2. Set the IP (initial – any valid IP)
  3. Tick the “Dynamic DNS” tickbox
  4. Enter your chosen Dynamic DNS Password
  5. Save the new record

When saving the record you will see a “Dynamic DNS ID” – note down this number. This will become the hostname on the Synology setup.

Synology Setup

Click “Customize” to add a new DDNS provider

Name: DNSMADEEASY
Query URL
http://cp.dnsmadeeasy.com/servlet/updateip?username=__USERNAME__&password=__PASSWORD__&id=__HOSTNAME__&ip=__MYIP__

Click “Add” to add a new DDNS service

  1. Service Provider: *DNSMADEEASY
  2. Hostname: Dynamic DNS ID from DNS Made Easy
  3. Username/Email: your DNS Made Easy email
  4. Password/Key: your chosen DNSMadeEasy DDNS password

Once you save the new DDNS provider you should see the status to go “Normal” in a green color. This means the update was successful. You should now be able to PING the DNS record or if you log in to DNS Made Easy the IP address should have changed to the external IP of your DiskStation.

Getting Fujitsu ScanSnap S1300i to work on Ubuntu 16.04LTS

ScanSnap 1300i

Upgrade or install SANE backends

Since the version of SANE in the Ubuntu 16.04LTS repos is not working for this scanner you either need to install from sources (see this blog) or from this PPA.
sudo add-apt-repository ppa:rolfbensch/sane-git
sudo apt update
sudo apt install sane-backends tesseract-ocr gscan2pdf

Security

Add yourself to the ‘scanner’ group to be able to use the scanner.

sudo usermod -a -G scanner USERNAME

Checking SANE

Check for the libsane version (needs to be at least libsane.so.1.0.26 not libsane.so.1.0.25 which is in the Ubuntu repos)

sudo ldconfig -v | grep libsane

This should show something like: libsane.so.1 -> libsane.so.1.0.26

Add udev rules for the scanner

sudo vim /etc/udev/rules.d/79-scanner.rules
#add the following
Fujitsu ScanSnap S1300i
ATTRS{idVendor}=="04c5", ATTRS{idProduct}=="128d", MODE="0664", GROUP="scanner", ENV{libsane_matched}="yes"

Get the firmware for the FUJITSU ScanSnap S1300i

sudo mkdir /usr/share/sane/epjitsu
cd /usr/share/sane/epjitsu
sudo wget https://github.com/ckunte/scansnap-firmware/raw/master/1300i_0D12.nal

Testing

scanimage -L

>> device `epjitsu:libusb:002:027′ is a FUJITSU ScanSnap S1300i scanner

You can now use gscan2pdf to scan, clean and OCR from the ScanSnap S1300i.

Finding Notebook Hardware for Ubuntu – 2015 Edition

Unfortunately it is still much harder than necessary to find notebook hardware to use with Ubuntu (or other Linux variants). This blog is full of past experiences (some of them quite time-consuming) on finding notebook hardware that will work without too much fiddling. This short note is to document my recent research on that front to help others who want to do the same (as there doesn’t seem to be a lot of good current info around).

There are some vendors that do ship with Ubuntu, however they are generally all based in the US and their pre-sales communications are pretty horrible (I am talking to you ZaReason – still waiting for reply email as well as tweet). Then there is Purism Librem, but unfortunately they have still not shipped their 15″ version and I need a tool now. The jury on this is still out and I don’t really have the time to be a guinea-pig. Maybe next time (as I like what they are doing) …

There was one option from one of the top-tier manufacturers (Dell XPS 13 – Developer Edition) which ships with Ubuntu. But as – per usual – NOT in Australia. However there was a lot of conflicting evidence I found that the Windows Version had some issues with current Ubuntu versions (Dell ship 14.04 LTS – which makes sense from their point).

I ended up buying the Lenovo X1 (3rd Generation) and after the install of Ubuntu I have to say this is the first notebook I just had to plug in the Ubuntu USB (Version 15.04 Vivid Vervet 64bit) and install and everything just worked. No fighting with UEFI firmware, no function keys not working and no issues with sound or other drivers. To be fair – there is one thing I noted (which is irrelevant to me) – the fingerprint reader does need some additional driver installed & configured.

It was also the first notebook I didn’t even bother booting up and create a repair disk first. There is no way I would ever restore something to Windows 8 – however if you are not sure that you will stick with Ubuntu – that’s probably not advisable.

Thank you Lenovo for developing a very decent piece of hardware that just works with Ubuntu ! However I would like to note that the recent Superfish debacle http://support.lenovo.com/us/en/product_security/superfish really left a sour taste and nearly made me take Lenovo off the evaluation list.

Some conversations that might be useful:

https://www.reddit.com/r/Ubuntu/comments/37ukqe/dell_xps_13_vs_lenovo_thinkpad_x1_carbon_3rd_gen/
https://www.reddit.com/r/Ubuntu/comments/37t4re/ubuntu_notebook_hardware_available_in_from/

Installing Ubuntu Phone (Touch) on Nexus 7 LTE

ubuntu phone

Add SDK repository

sudo add-apt-repository ppa:ubuntu-sdk-team/ppa
sudo apt-get update
sudo apt-get install ubuntu-device-flash

Enable USB Debugging on the device

  1. Make sure you have developer mode enabled (see http://developer.android.com/tools/device.html if you are unsure).
  2. Navigate to Settings > Developer options
  3. Enable USB Debugging. When a device is connected, you will be prompted in Android to authorize it.

Unlock Bootloader

adb reboot bootloader
fastboot oem unlock
fastboot reboot

Check that you have the right device

adb shell grep ro.product.name /system/build.prop > mydevicedata \
&& adb shell grep ro.product.device /system/build.prop >> mydevicedata \
&& adb shell grep build.id /system/build.prop >> mydevicedata

ro.product.name=razorg
ro.product.device=deb
ro.build.id=KTU84P

Check which channels are available

ubuntu-device-flash --server="http://system-image.tasemnice.eu" query --list-channels --device=deb

ubuntu-device-flash –server=”http://system-image.tasemnice.eu” query –list-channels –device=deb
ubuntu-touch/ubuntu-rtm/14.09
ubuntu-touch/ubuntu-rtm/14.09-proposed
ubuntu-touch/utopic
ubuntu-touch/utopic-proposed
ubuntu-touch/vivid
ubuntu-touch/vivid-proposed
ubuntu-touch/devel (alias to ubuntu-touch/vivid)
ubuntu-touch/devel-proposed (alias to ubuntu-touch/vivid-proposed)
ubuntu-touch/ubuntu-rtm/devel (alias to ubuntu-touch/ubuntu-rtm/14.09)
ubuntu-touch/ubuntu-rtm/devel-proposed (alias to ubuntu-touch/ubuntu-rtm/14.09-proposed)

In my case I am going for the currently stable ‘ubuntu-touch/vivid’ channel.

Install Ubuntu Touch

ubuntu-device-flash --server="http://system-image.tasemnice.eu" touch --channel="ubuntu-touch/vivid" --bootstrap

References
Ubuntu Devices: https://wiki.ubuntu.com/Touch/Devices

Paperless Office using the Raspberry Pi

This is a follow-up on an older blog using Ubuntu.

r by rosmary, on Flickr
Creative Commons Creative Commons Attribution 2.0 Generic License   by  rosmary 

For this purpose I used a Fujitsu ScanSnap S1300i scanner as I really like the features of this series (full duplex scan as well auto document feeder as well for around $250). It’s document feeder is not a good as the S1500 we have in the office, but very compact and can be powered from USB hub.

Raspberry Pi Prerequisites

Since this will be a purely headless install designed to sit in a corner behind the scanner I am using a Base Raspian (Debian Wheezy) install (I personally like the clean minimal install via https://github.com/debian-pi/raspbian-ua-netinst the best).

apt-get install sudo vim wget wput libusb-dev build-essential git-core

Add non-privileged user account(s)

adduser USERNAME
adduser USERNAME sudo
groupadd scanner
usermod -a -G scanner USERNAME

Install Sane

The version of sane from the Raspbian repos is not working with the Fujitsu ScanJet range and needs to be built from source.

git clone git://git.debian.org/sane/sane-backends.git
cd sane-backends
BACKENDS=epjitsu ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
make
make install

Install S1300i Driver

You need to get the driver file (‘1300i_0D12.nal’) from the CD that came with the scanner. If you still have access to a CDROM drive that is. :(

mkdir -p /usr/share/sane/epjitsu/
cp 1300i_0D12.nal /usr/share/sane/epjitsu/

Check /etc/sane.d/epjitsu.conf and see if the following line is there (in my case it was already created by sane build).

# Fujitsu S1300i
firmware /usr/share/sane/epjitsu/1300i_0D12.nal
usb 0x04c5 0x128d

sane-find-scanner -q

found USB scanner (vendor=0x04c5 [FUJITSU], product=0x128d [ScanSnap S1300i]) at libusb:001:004
found USB scanner (vendor=0x0424, product=0xec00) at libusb:001:003

scanimage -L

device `epjitsu:libusb:001:004′ is a FUJITSU ScanSnap S1300i scanner

Copy libsane rules from the sane build directory to udev rules.
sudo cp sane-backends/tools/udev/libsane.rules /etc/udev/rules.d/60-libsane.rules

Logout and log in a the non-privileged user account previously created.

If the scanimage -L command works as above you have fully configured the scanner to work under that user account.

Start saned on boot-up

Edit the /etc/rc.local file and add the following line before the ‘0’ line to ensure saned is running as the non-privileged user when you have to reboot.

saned -a USERNAME

Installing Conversion Tools

sudo apt-get install imagemagick bc exactimage pdftk tesseract-ocr tesseract-ocr-eng unpaper

You can add other languages such as tesseract-ocr-deu if you require OCR support for those.

Scan to Repository Script

The script is hosted on Github: https://github.com/leogaggl/misc-scripts/blob/master/scan2repo.sh

#!/bin/bash
# Thanks to Andreas Gohr (http://www.splitbrain.org/) for the initial work
# https://github.com/splitbrain/paper-backup/
OUT_DIR=~/scan
TMP_DIR=`mktemp -d`
FILE_NAME=scan_`date +%Y%m%d-%H%M%S`
LANGUAGE="eng"
echo 'scanning...'
scanimage --resolution 300 \
--batch="$TMP_DIR/scan_%03d.pnm" \
--format=pnm \
--mode Gray \
--source 'ADF Duplex'
echo "Output saved in $TMP_DIR/scan*.pnm"
cd $TMP_DIR
# cut borders
echo 'cutting borders...'
for i in scan_*.pnm; do
mogrify -shave 50x5 "${i}"
done
# check if there is blank pages
echo 'checking for blank pages...'
for f in ./*.pnm; do
unpaper --size "a4" --overwrite "$f" `echo "$f" | sed 's/scan/scan_unpaper/g'`
#need to rename and delete original since newer versions of unpaper can't use same file name
rm -f "$f"
done
# apply text cleaning and convert to tif
echo 'cleaning pages...'
for i in scan_*.pnm; do
echo "${i}"
convert "${i}" -contrast-stretch 1% -level 29%,76% "${i}.tif"
done
# Starting OCR
echo 'doing OCR...'
for i in scan_*.pnm.tif; do
echo "${i}"
tesseract "$i" "$i" -l $LANGUAGE hocr
hocr2pdf -i "$i" -s -o "$i.pdf" < "$i.html" done # create PDF echo 'Converting PDF...' pdftk *.tif.pdf cat output "$FILE_NAME.pdf" wput $FILE_NAME.pdf ftp://uid:pwd@scanner.domain:21/Alfresco/scans/ cp $FILE_NAME.pdf $OUT_DIR/ rm -rf $TMP_DIR

Thanks go to Andi Gohr @ Splitbrain for the excellent blog that helped me to get over the sane problems and also gave me some ideas to make the scan script better (as unpaper was not doing such a good job): http://www.splitbrain.org/blog/2014-08/23-paper_backup_1_scanner_setup

Barebone Ubuntu 14.04 Cloud Desktop

Since I have found some issues with my previous LXQT setup in real-life work I decided to fall back to standard Lubuntu for my cloud desktop. As part of this I also switched to TightVNC which seems a lot easier to configure.

Add local user account

adduser USERNAME
adduser USERNAME sudo

Install Lubuntu Desktop

sudo apt-get install --no-install-recommends lubuntu-desktop tightvncserver

TightVNC Configuration

sudo vim /etc/lightdm/lightdm.conf

#
# VNC Server configuration
#
# enabled = True if VNC connections should be allowed
# port = TCP/IP port to listen for connections on
#
[VNCServer]
enabled=true
port=5900
width=1366
height=768
depth=24

sudo /etc/init.d/lightdm restart

Connect to the remote system

ssh -L 5900:localhost:5900 -i /path/to/your/aws/keyfile.pem YOUR.EC2.IP.ADDRESS

tightvnc

 

If you are using a Chromebook then this article might help.

Ubuntu – paperless office on a budget

Since paper and myself have never gotten on well I have always been dreaming of a paperless office. A while ago I purchased a Fujitsu ScanSnap S1500 scanner for the office. I did this after doing some research on which Automatic Document Feed (ADF) multipage & duplex scanners were both affordable as well as supported on Linux.

It took a while for me to get around to set all of this up, but the result now is that this scanner is connected to a headless Ubuntu VM and the press of the scanner button will:

  1. scan the document
  2. perform OCR to convert to text
  3. combine the text with PDF to create a searchable PDF
  4. OPTIONAL – send the resulting document into Alfresco Document Management Server via FTP

Install dependencies

NOTE: PPA is only required for support of Fujitsu ScanSnap S1500
sudo apt-add-repository ppa:rolfbensch/sane-git
sudo apt-get update
sudo apt-get install sane sane-utils imagemagick tesseract-ocr pdftk libtiff-tools libsane-extras exactimage wput

Install scanbuttond

Download the “Debian Experimental” package from http://pkgs.org/download/scanbuttond
sudo dpkg -i scanbuttond_0.2.3.cvs20090713-14_i386.deb

This step is only for the Fujitsu ScanSnap support. For other scanners you can probably install from the Ubuntu Repository

Scanner config

vim 40-libsane.rules
#add this line
ATTRS{idVendor}=="04c5", ATTRS{idProduct}=="11a2", ENV{libsane_matched}="yes"

Permissions

sudo adduser saned scanner

Useful command lines for troubleshooting

Since I had a few trouble getting this scanner to work properly I found the following commands highly useful in locating the issue.
man sane-usb
sane-find-scanner
scanimage -L
dmesg
tail /var/log/udev

NOTE: If you are using a notebook devices be careful as I spent quite a few hours troubleshooting an error when opening the device from saned. It turned out to be that the USB power-management on the Toshiba notebook caused havoc with saned (http://askubuntu.com/questions/55140/error-during-device-i-o-when-using-usb-scanner). Switching to the desktop that is now housing the scanner fixed that problem. Thank you VIRTUALBOX (I ended up setting up a dedicated VM for this task) !

Configure scanbuttond

vim /etc/default/scanbuttond
#change this line from no to yes
RUN=yes

cd /etc/scanbuttond
sudo cp initscanner.sh.example initscanner.sh
sudo vim initscanner.sh

Uncomment or copy any scanner init command(s).

sudo cp buttonpressed.sh.example buttonpressed.sh
sudo vim buttonpressed.sh

Copy the contents of the scan script below. The script is also hosted on GitHub (https://github.com/leogaggl/misc-scripts/blob/master/buttonpressed.sh)

Scan script

#!/bin/bash
OUT_DIR=/output/directory/name
TMP_DIR=`mktemp -d`
FILE_NAME=scan_`date +%Y%m%d-%H%M%S`
cd $TMP_DIR
echo "################## Scanning ###################"
scanimage --resolution 150 --batch=scan_%03d.pnm --format=pnm --mode Gray --device-name "fujitsu:ScanSnap S1500:67953" --source “ADF Duplex” --page-width 210 --page-height 297 --sleeptimer 1 -y 297 -x 210
echo "################## Cleaning ###################"
for f in ./*.pnm; do
unpaper --size "a4" --overwrite "$f" "$f"
done
echo "############## Converting to TIF ##############"
mogrify -format tif *.pnm
echo "################ OCR ################"
for f in ./*.tif; do
tesseract "$f" "$f" -l eng hocr
hocr2pdf -i "$f" -s -o "$f.pdf" < "$f.html" done echo "############## Converting to PDF ##############" pdftk *.tif.pdf cat output "output.pdf" && rm *.tif.pdf && rm *.tif.html echo "############## Copy Output File ##############" cp $FILE_NAME.pdf $OUT_DIR/ echo "############## clean up ##############" cd .. rm -rf $TMP_DIR echo "############## FTP Output File ##############" #wput $OUT_DIR/$FILE_NAME.pdf ftp://user:pwd@ftp.alfrescoserver.com.au:21/autoscan/pdf/

Credits:

A big thank you & hat tip to the following authors of the following pages:


EDIT (2013-09-16): I found this link describing how to remove empty pages: http://philipp.knechtges.com/?p=190 – might have to investigate this when I have some time.