Paperless Office using the Raspberry Pi

This is a follow-up on an older blog using Ubuntu.

r by rosmary, on Flickr
Creative Commons Creative Commons Attribution 2.0 Generic License   by  rosmary 

For this purpose I used a Fujitsu ScanSnap S1300i scanner as I really like the features of this series (full duplex scan as well auto document feeder as well for around $250). It’s document feeder is not a good as the S1500 we have in the office, but very compact and can be powered from USB hub.

Raspberry Pi Prerequisites

Since this will be a purely headless install designed to sit in a corner behind the scanner I am using a Base Raspian (Debian Wheezy) install (I personally like the clean minimal install via https://githutb.com/hifi/raspbian-ua-netins the best).

apt-get install sudo vim wget wput libusb-dev build-essential git-core

Add non-privileged user account(s)

adduser USERNAME
adduser USERNAME sudo
groupadd scanner
usermod -a -G scanner USERNAME

Install Sane

The version of sane from the Raspbian repos is not working with the Fujitsu ScanJet range and needs to be built from source.

git clone git://git.debian.org/sane/sane-backends.git
cd sane-backends
BACKENDS=epjitsu ./configure --prefix=/usr --sysconfdir=/etc --localstatedir=/var
make
make install

Install S1300i Driver

You need to get the driver file (‘1300i_0D12.nal’) from the CD that came with the scanner. If you still have access to a CDROM drive that is. :(

mkdir -p /usr/share/sane/epjitsu/
cp 1300i_0D12.nal /usr/share/sane/epjitsu/

Check /etc/sane.d/epjitsu.conf and see if the following line is there (in my case it was already created by sane build).

# Fujitsu S1300i
firmware /usr/share/sane/epjitsu/1300i_0D12.nal
usb 0x04c5 0x128d

sane-find-scanner -q

found USB scanner (vendor=0x04c5 [FUJITSU], product=0x128d [ScanSnap S1300i]) at libusb:001:004
found USB scanner (vendor=0x0424, product=0xec00) at libusb:001:003

scanimage -L

device `epjitsu:libusb:001:004′ is a FUJITSU ScanSnap S1300i scanner

Copy libsane rules from the sane build directory to udev rules.
sudo cp sane-backends/tools/udev/libsane.rules /etc/udev/rules.d/60-libsane.rules

Logout and log in a the non-privileged user account previously created.

If the scanimage -L command works as above you have fully configured the scanner to work under that user account.

Start saned on boot-up

Edit the /etc/rc.local file and add the following line before the ‘0’ line to ensure saned is running as the non-privileged user when you have to reboot.

saned -a USERNAME

Installing Conversion Tools

sudo apt-get install imagemagick bc exactimage pdftk tesseract-ocr tesseract-ocr-eng

You can add other languages such as tesseract-ocr-deu if you require OCR support for those.

Scan to Repository Script

The script is hosted on Github: https://github.com/leogaggl/misc-scripts/blob/master/scan2repo.sh

#!/bin/bash
# Thanks to Andreas Gohr (http://www.splitbrain.org/) for the initial work
# https://github.com/splitbrain/paper-backup/
OUT_DIR=~/scan
BASE="~/tmp"
cd $BASE
TMP_DIR=`mktemp -d`
FILE_NAME=scan_`date +%Y%m%d-%H%M%S`
LANGUAGE="eng"
echo 'scanning...'
scanimage --resolution 300 \
--batch="$TMP_DIR/scan_%03d.pnm" \
--format=pnm \
--mode Gray \
--source 'ADF Duplex'
echo "Output saved in $TMP_DIR/scan*.pnm"
cd $TMP_DIR
# cut borders
echo 'cutting borders...'
for i in scan_*.pnm; do
mogrify -shave 50x5 "${i}"
done
# check if the page is blank
# http://philipp.knechtges.com/?p=190
echo 'checking for blank pages...'
for i in scan_*.pnm; do
echo "${i}"
histogram=`convert "${i}" -threshold 50% -format %c histogram:info:-`
white=`echo "${histogram}" | grep "#FFFFFF" | sed -n 's/^ *\(.*\):.*$/\1/p'`
black=`echo "${histogram}" | grep "#000000" | sed -n 's/^ *\(.*\):.*$/\1/p'`
blank=`echo "scale=4; ${black}/${white} < 0.005" | bc`
if [ ${blank} -eq "1" ]; then
echo "${i} seems to be blank - removing it..."
rm "${i}"
fi
done
# apply text cleaning and convert to tif
echo 'cleaning pages...'
for i in scan_*.pnm; do
echo "${i}"
convert "${i}" -contrast-stretch 1% -level 29%,76% "${i}.tif"
done
# Starting OCR
echo 'doing OCR...'
for i in scan_*.pnm.tif; do
echo "${i}"
tesseract "$i" "$i" -l $LANGUAGE hocr
hocr2pdf -i "$i" -s -o "$i.pdf" < "$i.html"
done
# create PDF
echo 'Converting PDF...'
pdftk *.tif.pdf cat output "$FILE_NAME.pdf"
wput $FILE_NAME.pdf ftp://uid:pwd@scanner.domain:21/Alfresco/scans/
cp $FILE_NAME.pdf $OUT_DIR/
rm -rf $TMP_DIR

Thanks go to Andi Gohr @ Splitbrain for the excellent blog that helped me to get over the sane problems and also gave me some ideas to make the scan script better (as unpaper was not doing such a good job): http://www.splitbrain.org/blog/2014-08/23-paper_backup_1_scanner_setup

Install Ubuntu 14.04 on a Chromebook

There are plenty of sites out there that give advise on this topic, unfortunately most of them are highly ad-infested to the point of being unreadable as well as only containing single bit rather that an overall picture. This is a collection of useful links to source materials as well as steps necessary to install.

Resource Links

Crouton Github: https://github.com/dnschneid/crouton – Thank you David Schneider for the excellent work !!!
Developer Info for Chromebooks: https://www.chromium.org/chromium-os/developer-information-for-chrome-os-devices

Put Cromebook into “developer mode”

  1. Back up any data as the process wipes the system
  2. Create a restore image for Chrome OS (install the Restore Image Chrome Extension for this task)
  3. Enter Developer Mode – hold down ESC and Refresh (F3) keys and press the Power button

Download Crouton Script

Download link for installer: https://raw.githubusercontent.com/dnschneid/crouton/master/installer/crouton

Installing Crouton

CTRL+Alt+t to open Cronos Prompt + type “shell” to enter proper bash shell.

To see the list of supported releases:
sh -e ~/Downloads/crouton -r list
To see a list of the supported desktop envoironments (target names):
sh -e ~/Downloads/crouton -t help
I generally install LXDE on ‘resource-challenged’ devices.

shell
sudo sh -e ~/Downloads/crouton -r RELEASENAME -t TARGETNAME -e

The ‘-e’ at the end is optional to encrypt the chroot. Which is probably a good idea as the Chromebook in developer mode is completely open and allows any user to access. If you do not specify the Release it defaults to Ubuntu 12.04 (precise)

My default install would be:
sudo sh -e ~/Downloads/crouton -r trusty -t lxde -e

Removing Crouton

The proper way to remove the chroot environment created by Crouton is as follows

sudo delete-chroot CHROOTNAME

CHROOTNAME could be ‘precise’ or ‘trusty’ depending on the installed version and can be found by

ls /mnt/stateful_partition/crouton/chroots/

Run Ubuntu

Depending on your installed shell.
sudo startlxde

Replace with ‘startxfce4′ or ‘startkde’ or ‘startunity’ depending on your target release.

CyanogenMod 12 on Sony Xperia Z2

Just a quick update of the previous article on “Sony Xperia Z2 upgrading to CyanogenMod 11“.

One thing is that CM now included the custom recovery and you do not need to download any other custom recoveries !

Download the CM 12 ZIP file for Sony Xperia Z2 (sirius) and extract the ‘boot.img’ file

Download link: https://download.cyanogenmod.org/?device=sirius

Get the device into fastboot (bootloader mode)

fastboot -i 0xfce flash boot boot.img
fastboot reboot

Install CyanogenMod

Choose the “Install zip from sdcard –> Install from sideload” option

adb sideload cm-12-20150219-NIGHTLY-sirius.zip

After the install has finished choose the “Reboot system now” option.

Ubuntu 14.04 Webmin Install from PPA

On remote systems sometimes a web-based tool can be very handy. Webmin is such a tool that has been well maintained for decades. To install quickly on a Ubuntu Server without having to manage dependencies and keeping it updated as part of normal OS update operations installing from a PPA Repo is handy.

sudo echo "deb http://download.webmin.com/download/repository sarge contrib" >> /etc/apt/sources.list
wget -q http://www.webmin.com/jcameron-key.asc -O- | sudo apt-key add -
#OpenSuse Repo Key
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 977C43A8BA684223
sudo apt-get update
sudo apt-get install webmin

After this you should be able to connect to your webmin instance on port 10000

https://hostname:10000

Security note: I would never allow direct access to Webmin on a remote server but rather tunnel port 10000 over SSH !

Barebone Ubuntu 14.04 Cloud Desktop

Since I have found some issues with my previous LXQT setup in real-life work I decided to fall back to standard Lubuntu for my cloud desktop. As part of this I also switched to TightVNC which seems a lot easier to configure.

Add local user account

adduser USERNAME
adduser USERNAME sudo

Install Lubuntu Desktop

sudo apt-get install --no-install-recommends lubuntu-desktop tightvncserver

TightVNC Configuration

sudo vim /etc/lightdm/lightdm.conf

#
# VNC Server configuration
#
# enabled = True if VNC connections should be allowed
# port = TCP/IP port to listen for connections on
#
[VNCServer]
enabled=true
port=5900
width=1366
height=768
depth=24

sudo /etc/init.d/lightdm restart

Connect to the remote system

ssh -L 5900:localhost:5900 -i /path/to/your/aws/keyfile.pem YOUR.EC2.IP.ADDRESS

tightvnc

 

If you are using a Chromebook then this article might help.

Accessing your cloud desktop from Chromebook

One of the main reasons for setting up a cloud desktop is that I tend to use a lot of different devices some of which are not very powerful.

One of my favorite devices of late has been a HP 11 Chromebook. I originally bought it for a new employee and wanted to check myself how this thing stacks up to do day-to-day computing tasks more efficiently than a standard laptop without all the headaches of running Windows (viruses, endless driver installs, bloatware, malware, …). We already have several people at work working exclusively from Chromebooks and they absolutely love them. Long story short – I ended up keeping the Chromebook for myself as it’s an absolutely great secondary device for me. I can carry it with me everywhere (doesn’t weigh much more than a tablet, roughly the same size as a tablet & has a keyboard and is so much more useful than a tablet).

Initially I was using it more as a secondary device, but lately I have been thinking that I will not even take my main notebook at all for travels. The problem with this is that I do (sometimes – very infrequently) need access to software not available on such a limited device.

Accessing your Cloud Desktop via SSH (I know there seem to be people allowing direct VNC access – but that is just asking for trouble) is highly recommended. I also use RSA keys instead of password authentication.

Copy SSH Private Key (generated on desktop and public key added to authorized_keys on the cloud-server) to the Chromebook,

Unfortunately (unless you want to set you Chromebook into developer mode which has all sorts of other downsides) the inbuilt CROSH shell is very limited. However it gives you enough to securely connect to a cloud-server and you can use the whole tool-set from there

crosh> ssh
ssh> host example.com
ssh> user <username>
ssh> key <rsa_key_filename>
ssh> forward 8000:localhost:5901
ssh> connect

You can now use the RealVNC Chrome App to connect to your cloud desktop.

RealVNC

EDIT (2014-11-10): Found a better Chrome Extension which can handle the port forwarding and RSA certificate authentication which will save the config without having to type the commands each time.

Chrome Secure Shell

ChormeOS - Secure Shell

Vodafone LTE mobile data on Cyanogen Mod

I have had some issues recently with getting LTE (4G) connectivity on the Vodafone Australia Network using CyanogenMod 11 on multiple devices (http://forum.cyanogenmod.org/topic/92919-no-4g-signal-on-vodafone-au/). Turns out that it was an APN issue after all.

The APN provisioned by default when the Voda SIM card is inserted (vfinternet.au) does not work for the LTE Data Network. It works with GPRS & WCDMA, but fails to connect when the phone is set to prefer LTE (4G) Networks and they are actually available. It means that the handset will loose mobile data connectivity altogether. There are various APN Settings floating around on the interwebs, but any I tried previously did not fix the issue.

Ater needing to call Vodafone support for another device (on GPE Stock Android) that was experiencing the same issue I got given these APN Settings:

Settings –> Mobile Networks –> Access Point Names

APN Name: Vodafone AU
APN URI: live.vodafone.com

Leave all other settings at default (for MMS settings see the link to VF Support below).

image

I have tested this for 2 days and works beautifully. The correct APN settings can be found here: http://support.vodafone.com.au/articles/FAQ/APN-Settings

Sony Xperia Z2 upgrading to CyanogenMod 11

Contrary to my normal inclinations not to buy anything but Stock Android phones I ended up with a Sony Xperia Z2 in a hurry over the weekend (it’s a long story…). It appears to be quite a decent handset (with a pretty good camera actually) and one of the main reason to choose this over the other options was that is was one of the few high-end devices which already had a CM snapshot rather than just nightly releases. However similar to their colleagues at Samsung the Sony people also opted to stuff all sorts of crap-ware bloat onto the Android base OS. It appears not to be as bad as the Samsung (who are the kings of crap) mods, but for somebody used to the clean Android experience it’s just very annoying. So I opted to flash it straight to CyanogenMod.

Sony Xperia Z2 2 by epct414, on Flickr
Creative Commons Creative Commons Attribution-No Derivative Works 2.0 Generic License   by  epct414 

Prerequisites

You need the Android SDK installed and on your Path – see one of my other posts on how to do this.

Get Bootloader Unlock Code

Another annoying step which is not required with Google Nexus series.

http://unlockbootloader.sonyericsson.com/instructions

You will need to enter and confirm your email (WTF?) and need your IMEI number and at the end of the process you will get an update key that is required below.

Put phone into bootloader (fastboot) mode

Insert one end of the USB cable into your PC. Turn off the device and wait at least 5 seconds. Press and hold the Volume Down button and plug the other end of the Micro-USB cable in to your device.

fastboot devices
#should display the ID of the connected phone
fastboot -i 0x0fce oem unlock 0xKEY

Download CyanogenMod Release

Extract the boot.img file from the downloaded ZIP file via https://download.cyanogenmod.org/?device=sirius

fastboot -i 0xfce flash boot boot.img
fastboot -i 0xfce reboot

This should boot into Clockwork Mod Recovery and you can use the Volume +/- to navigate and On/Off button to choose.

Select the “Wipe data/factory reset” option

Note: this part was not in the CM Wiki or anywhere else and took me a while to get past. If you see an error “Error Mounting /sdcard/.android_Secure!” during the factory reset procedure you need to format the data partition manually. If you do not do this step the CM install will still work but you will not be able to boot the phone (it gets stuck at the Boot Animation).

Mount /data
Format /data

After the manual format procedure do the “Factory reset” again.

Install CyanogenMod

Choose the “Install zip from sdcard –> Install from sideload” option

adb sideload cm-11-20140804-SNAPSHOT-M9-sirius.zip

After the install has finished choose the “Reboot system now” option.

CM Wiki Entry: http://wiki.cyanogenmod.org/w/Install_CM_for_sirius

Removing ‘Video Call’ default in Google Calendar

This ‘feature’ has been annoying me for a while and after this has caused some confusion with some of my clients I decided to go and look where to disable this. Why this has been made a system wide default is beyond me.

Rather than in the users Calendar Settings this is actually in the Google Apps Admin Console (https://admin.google.com/)

Console –> Google Apps –> Settings for Calendar –> Sharing Settings

Disable the “Automatically add video calls to events created by a user” setting.

Direct Link: https://admin.google.com/AdminHome?fral=1#AppDetails:service=Calendar&flyout=sharing

Finding a private location check-in service

Foursquare decided that it was too hard for them to compete with location services like Yelp and split their app into two separate apps. Whilst that might make sense to the 4Square CEO and his VC masters, it makes no sense from a users perspective. Foursquare can be a bit of a battery hog already, and having 2 apps to open and “annoy” you with notifications is not an improvement by any means. And if I wanted Foursquare to be Yelp – I would have used Yelp in the first place. So no – I do not want to install another separate check-in App (called Swarm). One battery hogging location app was enough.

no checkins here by leogaggl, on Flickr
Creative Commons Creative Commons Attribution-Noncommercial-Share Alike 2.0 Generic License   by  leogaggl 

The other argument used by 4Square’s CEO is that he didn’t want users confused about the “gamification” aspects of 4Square. I personally think that this is highly patronising to the Foursquare user base. I am sure most users would be able to work out what it is useful for.

Since I have always used 4Square mainly as a means to get some analytics of my movements and historic record of where I was at what time (I always downloaded my checkins to Thinkup on my own server) I was trying to find something that would fit the same use case. Meet Ushahidi (http://www.ushahidi.com/) – an excellent geo-coded “reporting” service developed in Kenya. I have been following this project for years already.

Dynamic Timeline

Track your reports on the map and over time, filter your data by time, and see when things happened and where.

Interactive Mapping

One of the most powerful ways to visualize information is to display it on a map. The Ushahidi platform give you rich information mapping tools.

Multiple Data Streams

The Ushahidi Platform allows you to easily collect information via text messages, email, twitter and web-forms.

Free & Open Source

The Ushahidi Platform is free for you to download and use. It is released under the GNU Lesser General Public License (LGPL).

Installation information for your own Ushahidi Server can be found on GitHub – or you can use their hosted service called Crowdmap.

Whilst this is obviously not an option for everybody it works for me. I will miss some the social aspect of 4Square as I had a small number of people I was sharing my check-ins with and it did bring about some by-chance meetups & conversation about other people’s check-ins. But the Ushahidi instance can be shared between multiple people. Maybe some will come along for the ride.

So long Foursquare – it was nice while it lasted ! If you treat your users like sheep – all you will be left with is sheep…

And for those who want to delete their Foursquare account: https://support.foursquare.com/hc/en-us/articles/201065530-How-do-I-delete-my-account-.