Setting up MultiTech LoRaWAN gateway on Ubuntu

As the convener for the Adelaide community of The Things Network, I am frequently setting up Multitech Conduit Gateways. Depending on your PC or notebook hardware you might have some problems with the Exar USB-UART driver on Linux.

Here are the steps to getting this unit setup from an Ubuntu (should work for any other Linux distro) machine.

lsusb

Should show something like this:

Bus 002 Device 006: ID 04e2:1410 Exar Corp. XR21V1410 USB-UART IC

Download the driver for the Exar site: https://www.exar.com/design-tools/software-drivers

unzip xr_usb_serial_common_lnx-3.6-and-newer-pak.zip
cd xr_usb_serial_common_lnx-3.6-and-newer-pak
make
sudo insmod ./xr_usb_serial_common.ko

Ensure driver is loaded at startup

sudo vim /etc/modules
#Add the following
xr_usb_serial_common

ls /dev/tty*

should now show another USB port ttyXRUSB0

For the rest you just need to follow the excellent instructions by Jac Kersing who maintains the Multitech TTN installer and documentation here:
https://www.thethingsnetwork.org/docs/gateways/multitech/mlinux.html

Get EUI of your gateway

mts-io-sysfs show lora/eui 2> /dev/null | sed 's/://g'

Moving to KVM virtual machines

Installing VirtualBox is getting increasingly painful on Ubuntu due to the problems with UEFI Secure Boot and the VirtualBox kernel modules. Another reason for an alternative is that running VirtualBox VM’s completely in the background is not as straightforward as it could be.

From the available alternatives I looked into (VMWare, Xen & KVM) it was KVM that fitted my needs (casual VM usage with mostly headless VM’s for testing purposes). Main reasons:

  • Well supported by Ubuntu
  • Easy, straightforward install
  • Background VM’s are simple as
  • Moving VM’s from one host to another is a breeze

Checking system

To check if the CPU can actually support

egrep -c '(svm|vmx)' /proc/cpuinfo

If the number returned is > 0 your systems should be capable to run.

You will also enable your BIOS for virtualisation (in Security settings of most BIOS’s) if that has not already be done. You will get an error if not enabled if you are trying to run an install. The Install of KVM will work fine.

Installation

sudo apt-get install qemu-kvm libvirt-bin bridge-utils virt-manager
sudo addgroup libvirtd
sudo adduser libvirtd
sudo service libvirtd start
sudo service libvirtd status
sudo virt-manager

Moving VM’s to another host

Source Host

virsh shutdown VMNAME
virsh dumpxml VMNAME > /tmp/VMNAME.xml
scp /tmp/VMNAME.xml TARGETHOST:/tmp/VMNAME.xml
scp /var/lib/libvirt/images/VMNAME.qcow2 TARGETHOST:/var/lib/libvirt/images/VMNAME.qcow2

Target Host

virsh define /tmp/vm.xml
virsh start vm

Once you have confirmed operation you probably want to remove the source VM from the Source Host.

virsh undefine VMNAME
rm /var/lib/libvirt/images/VMNAME.qcow2

Microchip LoRaWAN Development Utility on Ubuntu

Having just wasted a few hours on getting this Java software running on Linux I am documenting this for future reference and hopefully saving other LoRa / TTN folks some time.

Prerequisites

Install a Java JDK + JavaFX. This should work with the default OpenJDK 8 or 9 which comes as part of the Ubuntu repositories. I ended up installing Oracle JDK 8 as well as I thought the error might be related to OpenJDK.

sudo apt install openjdk-8-jre-headless openjfx

Download & install utility

Download location:
LoRa® Technology Evaluation Kit

cd ~/Downloads/
chmod +x LoRaSuite-linux-1.0.run
./LoRaSuite-linux-1.0.run

Fix User Preferences

This step is required for the Utility to run. Unfortunately, this is documented NOWHERE…

Change the following files to include the FilePath entry. The map is empty by default.

  • /home/USERID/.java/.userPrefs/dfu/prefs.xml
  • /home/USERID/.java/.userPrefs/fed/prefs.xml
<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE map SYSTEM "http://java.sun.com/dtd/preferences.dtd">
<map MAP_XML_VERSION="1.0">
  <entry key="FilePath" value="/home/USERID/"/>
</map>

If you chose the default install location you can now start the utility

java -jar ~/Microchip/LoRaSuite/Applications/LoRaDevUtility/LoRaDevUtility.jar

A big thank you goes to The Things Network user JBI – who provided this answer in a TTN forum post.

No thanks to Microchip as their forum and firmware release policy is a bit of a shocker. There are several reports of such problems in the forum with no answers. Unfortunately, this seems to be no exception with electronics manufacturers (Hello Kerlink!).

GrovePi Zero – connecting your IoT sensors

I recently purchased a GrovePi Zero and expected this to be a reasonable straight forward way to connect Grove sensors to your Raspberry Pi, read sensor values via Python and pushing them upstream via MQTT. However the software side of things turns out anything but straight forward. Most of the suggestions on the Dexter Industries forum suggest to download some custom OS image – WTF? Hopefully this will save some people time to chase down the same rabbit holes…..

The problem starts with the install scripts which (a pet hate of mine) assume the /home/pi account to be present. There is a variable in at least the 2 of the install scripts (another Github repository – one calls another from a Github URL – like that will ever work ???) which allows you to change the user account but there is also hard-coded references all over the place that still point to that user account. Trying to step manually through the dependency hell in these install scripts that call other scripts ended to be a complete nightmare. It seems to download specific (old !?!) versions of libraries all of which have perfectly fine binaries in the Raspbian repos. Most of the troubleshooting tips use the UI which is not much good on a headless system.

Since all I wanted was to read the sensors from Python I found that there was a library on PyPi (https://pypi.python.org/pypi/grovepi) which should actually do this without installing all this redundant dependencies on what is essentially a remote sensor aggregator. Got hopes up too quickly. Unfortunately turns out that the firmware on these devices is really old and riddled with bugs (could not read from most sensors I tried and all were known issues fixed in later firmware releases. Trying to manually install some of the dependencies (such as avrdude which are all in the Raspbian repositories) for the simple firmware update script turned out useless as they use a custom ‘gpio’ programmer not present. If you are splitting things into multiple Github repositories (which is in itself a good idea) why don’t be consistent about it and create one for the firmware updater alone? You shouldn’t have to compile half of Github on a low-powered device just to flash a firmware file.

Looks like the only solution is to bite the bullet and get another SD card with a default Raspbian image and go through the whole painful experience of installing the whole dependency hell and compiling everything on a RaspberryPi Zero which is like watching paint dry. Hopefully this should then just work on another device that only needs the Python library as a dependency.

Let’s hope so… [will update once finished jumping through all the hoops]

Connecting your LoPy to The Things Network in Australia

EDIT [2018-06-05]: I have updated the code with the Firmware 1.18.+ releases. The code is available at our Growing Data Foundation Github.

These notes are to assist Australian IoT enthusiasts to get started in connecting a LoPy to The Things Network as it is unfortunately (not yet) straight forward to make them work with the current AU-915 TTN Channel plans. As the initiator of the local Adelaide Community of The Things Network I have been experimenting with a number of devices to connect sensors to #TTNADL. One of my personal favourites is the Pycom LoPy as a nice middle-ground between capabilities and technical complexity.

The Things Network Adelaide

However I ran into a problem where the LoPy would not get a signal back from the TTN network when joining over OTAA even though the TTN Console (http://console.thethingsnetwork.org/) the device showed as connected. With some friendly help from Jose Marcelino at the Pycom Forum this turned out to be an issue with the 915MHz frequency regions. Since there is no actual a standard governing which of the channels are used by TTN gateways, what is outlined here is what is implemented by most TTN communities down under (I know that at least ADL, BNE, SYD and WOL adhere to those). This is the typical Sub-band 2 (Channel 8-15) implementation of AU ISM 915 with TTN Gateways.

This channel plan is also implemented by the install script for the MultiTech Conduit Gateways which are currently the most common gateways installed across Australia (see https://github.com/TheThingsNetwork/gateway-conf/blob/master/AU-global_conf.json)

Channel# Direction Frequency MHz Bandwidth kHz Data rate
8 up 916.8 125 DR0 – DR3
9 up 917.0 125 DR0 – DR3
10 up 917.2 125 DR0 – DR3
11 up 917.4 125 DR0 – DR3
12 up 917.6 125 DR0 – DR3
13 up 917.7 125 DR0 – DR3
14 up 918.0 125 DR0 – DR3
15 up 918.2 125 DR0 – DR3
65 up 917.5 500 DR4
0 down 923.3 500 DR8 – DR13
1 down 923.9 500 DR8 – DR13
2 down 924.5 500 DR8 – DR13
3 down 925.1 500 DR8 – DR13
4 down 925.7 500 DR8 – DR13
5 down 926.3 500 DR8 – DR13
6 down 926.9 500 DR8 – DR13
7 down 927.5 500 DR8 – DR13

To set the LoPy unit up it needs to have the set frequency plan removed and the region specific frequency plan loaded (even though they were bought as 915Mhz units). I have created some quick Python code to set up the LoPy with the above channel structure as the default settings for those devices do not work. Note that you have to remove the default channel settings and add the correct ones before you can successfully register and send data.

Below is my adaptation of the LoPy LoraWAN example to work with TTN in Australia. Please note that the LoPy Lora class only accepts DR values from 1-7. Suggestions more than welcome.

Hope this helps other LoPy owners in Australia to connect to The Things Network. And if you are in Adelaide why don’t you come along to our OpenData and IoT Meetup ? Let’s build this thing together.

RaspberryPi Version 3 SOE

Since there is now a supported Raspbian version without GUI and other unneeded add-ons available as Raspbian Lite the need to use other installers (with sometimes some downsides) is now not a necessity anymore. Below is a list of steps I like to perform before using them for any purpose as my Standard Operating Environment.

Download Raspbian Lite

Download link: https://www.raspberrypi.org/downloads/raspbian/

Write to SD Card

dd bs=4M if=2016-05-27-raspbian-jessie-lite.img of=/dev/sdb

Boot RPi

Default login details are
UID: pi
PWD: rasbperry

Regional settings

locale-gen en_AU.UTF-8
dpkg-reconfigure locales
ln -sf /usr/share/zoneinfo/Australia/Adelaide /etc/localtime

System update

apt update && apt upgrade
apt install vim

Setup SSH keys

As a security precausion it’s a good idea to disable password authentications
ssh-keygen -t rsa
vim /root/.ssh/authorized_keys
# --> add RSA public key

Setup Wireless LAN

iwlist wlan0 scan
wpa_passphrase SSID WPA_KEY
vim /etc/wpa_supplicant/wpa_supplicant.conf

Copy the resulting hash from the previous command

Example config

country=AU
ctrl_interface=DIR=/var/run/wpa_supplicant GROUP=netdev
update_config=1
network={
ssid="YOURSSID"
psk=123456789012345678901234567890
}

Rename default user account

The default Raspbian images use the default user ‘pi’ – I prefer to change this user to another account for security purposes and to be in line with other Linux system I use.

Reboot and log in to the RPi as root – you need to run the following commands as root.

usermod -l NEW_USER_ID pi
usermod -m -d /home/NEW_USER_ID NEW_USER_ID
groupmod --new-name NEW_USER_ID pi
mkdir /home/NEW_USER_ID/.ssh/
vim /home/NEW_USER_ID/.ssh/authorized_keys
# --> add RSA public key
# change owner and permissions on key files
chown -R NEW_USER_ID:NEW_USER_ID /home/NEW_USER_ID/.ssh/
chmod 700 /home/NEW_USER_ID/.ssh/
chmod 600 /home/NEW_USER_ID/.ssh/authorized_keys
rm /etc/sudoers.d/010_pi-nopasswd
echo 'NEW_USER_ID ALL=(ALL) NOPASSWD: ALL' > /etc/sudoers.d/010_NEW_USER_ID-nopasswd

If you are using the desktop version not a headless version you want to make sure you don’t boot into the desktop and if you want to change the autologin ensure you update the username there as well.

/etc/systemd/system/autologin@.service
/etc/lightdm/lightdm.conf

Configure Raspberry Pi settings

raspi-config
I generally need to enable SPI, change the hostname and others depending on the purpose of the unit.

EDIT [2017-08-21]: With the update of Raspbian to Stretch you now have the firmware updater as part of the Raspbian repos:

sudo rpi-update

This will update the Pi Firmware to the latest version as well – which is worth it if you use it for later cameras and peripheral usage.

EDIT [2017-12-27]: change annoying vim defaults in Debian Stretch onwards. Visual mode.

sudo vim /usr/share/vim/vim80/defaults.vim

Comment out the following lines:
“if has(‘mouse’)
” set mouse=a
“endif

Install Hugo on Ubuntu to generate static websites

Whilst there is a .DEB installer to download from the GoHugo sites I get all matter of warnings that the package is of bad quality and I am not comfortable to run these kinds of installers.

Hugo Logo

I rather install from sources in this case which is very straight forward since the main dependencies (largely GO) are in the Ubuntu main repositories.

Install dependencies

sudo apt-get install golang git mercurial python-pygments

Create environment variables

vim ~/.bashrc
#add the following 3 lines
export GOROOT=/usr/lib/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

Update Bash Environment Variables without logging out.

source ~/.bashrc

Install Hugo

go get -u -v github.com/spf13/hugo

Start using Hugo

#create new site
hugo new sitename /path/to/sitename
#change directory to site
cd /path/to/sitename
#create content page
hugo new about.md
#edit content page
vim content/about.md

ChromeOS – removing SSH known_hosts from Chromebook

One of the things that is not implemented in the Secure Shell Chrome extension is the ability to remove know_host fingerprints which alert you if the fingerprint for a specific IP address has changed.

However there are times when you upgrade a systems and this need to be done.

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
d6:be:12:7e:22:23:c3:e1:56:30:d6:cd:65:b7:ab:42.
Please contact your system administrator.
Add correct host key in /.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /.ssh/known_hosts:7
ECDSA host key for xxxxxxxxxxxxx.yyy.au has changed and you have requested strict checking.
Host key verification failed.
NaCl plugin exited with status code 255.
(R)econnect, (C)hoose another connection, or E(x)it?

Here is how to remove a known host fingerprint (from known_hosts) on a Chromebook.

You can find the index of the offending host entry reported by ssh if the connection fails (see above: “Offending ECDSA key in /.ssh/known_hosts:7”). Then you can open the JavaScript console (CTRL +Shift +J) and type the following into the console.

term_.command.removeKnownHostByIndex(INDEX);
Replace INDEX with the number obviously.

To clear all the known hosts:
term_.command.removeAllKnownHosts();

Install Ubuntu 14.04 on a Chromebook

There are plenty of sites out there that give advise on this topic, unfortunately most of them are highly ad-infested to the point of being unreadable as well as only containing single bit rather that an overall picture. This is a collection of useful links to source materials as well as steps necessary to install.

Resource Links

Crouton Github: https://github.com/dnschneid/crouton – Thank you David Schneider for the excellent work !!!
Developer Info for Chromebooks: https://www.chromium.org/chromium-os/developer-information-for-chrome-os-devices

Put Cromebook into “developer mode”

  1. Back up any data as the process wipes the system
  2. Create a restore image for Chrome OS (install the Restore Image Chrome Extension for this task)
  3. Enter Developer Mode – hold down ESC and Refresh (F3) keys and press the Power button

Download Crouton Script

Download link for installer: https://raw.githubusercontent.com/dnschneid/crouton/master/installer/crouton

Installing Crouton

CTRL+Alt+t to open Cronos Prompt + type “shell” to enter proper bash shell.

To see the list of supported releases:
sh -e ~/Downloads/crouton -r list
To see a list of the supported desktop envoironments (target names):
sh -e ~/Downloads/crouton -t help
I generally install LXDE on ‘resource-challenged’ devices.

shell
sudo sh -e ~/Downloads/crouton -r RELEASENAME -t TARGETNAME -e

The ‘-e’ at the end is optional to encrypt the chroot. Which is probably a good idea as the Chromebook in developer mode is completely open and allows any user to access. If you do not specify the Release it defaults to Ubuntu 12.04 (precise)

My default install would be:
sudo sh -e ~/Downloads/crouton -r trusty -t lxde -e

Removing Crouton

The proper way to remove the chroot environment created by Crouton is as follows

sudo delete-chroot CHROOTNAME

CHROOTNAME could be ‘precise’ or ‘trusty’ depending on the installed version and can be found by

ls /mnt/stateful_partition/crouton/chroots/

Run Ubuntu

Depending on your installed shell.
sudo startlxde

Replace with ‘startxfce4’ or ‘startkde’ or ‘startunity’ depending on your target release.

Accessing your cloud desktop from Chromebook

One of the main reasons for setting up a cloud desktop is that I tend to use a lot of different devices some of which are not very powerful.

One of my favorite devices of late has been a HP 11 Chromebook. I originally bought it for a new employee and wanted to check myself how this thing stacks up to do day-to-day computing tasks more efficiently than a standard laptop without all the headaches of running Windows (viruses, endless driver installs, bloatware, malware, …). We already have several people at work working exclusively from Chromebooks and they absolutely love them. Long story short – I ended up keeping the Chromebook for myself as it’s an absolutely great secondary device for me. I can carry it with me everywhere (doesn’t weigh much more than a tablet, roughly the same size as a tablet & has a keyboard and is so much more useful than a tablet).

Initially I was using it more as a secondary device, but lately I have been thinking that I will not even take my main notebook at all for travels. The problem with this is that I do (sometimes – very infrequently) need access to software not available on such a limited device.

Accessing your Cloud Desktop via SSH (I know there seem to be people allowing direct VNC access – but that is just asking for trouble) is highly recommended. I also use RSA keys instead of password authentication.

Copy SSH Private Key (generated on desktop and public key added to authorized_keys on the cloud-server) to the Chromebook,

Unfortunately (unless you want to set you Chromebook into developer mode which has all sorts of other downsides) the inbuilt CROSH shell is very limited. However it gives you enough to securely connect to a cloud-server and you can use the whole tool-set from there

crosh> ssh
ssh> host example.com
ssh> user <username>
ssh> key <rsa_key_filename>
ssh> forward 8000:localhost:5901
ssh> connect

You can now use the RealVNC Chrome App to connect to your cloud desktop.

RealVNC

EDIT (2014-11-10): Found a better Chrome Extension which can handle the port forwarding and RSA certificate authentication which will save the config without having to type the commands each time.

Chrome Secure Shell

ChormeOS - Secure Shell