Raspberry Pi Setup

An Eccentric Anomaly: Ed Davies's Blog

I have an “always-on” Raspberry Pi 3 which does logging of temperature, pressure, power use and so on readings from around the house and county.

I've had it just over a year and it's been running the version of Raspbian (the Debian Linux distribution modified for the Pi) that was current when I got it. The logging has been to flat files containing lines of JSON but recently I started switching to a Sqlite3 database but was having problems with the code running on the Pi which didn't happen on my laptop running Ubuntu 17.04. I strongly suspected the problem was the old version of Sqlite3 or Python3 in that Raspbian distribution so decided to upgrade to the latest Rasbian (2017-11-29, based on Debian 9 Stretch rather than the previous one which was based on Debian 8 Jessie).

During the installation on the new system I made notes of what I needed to do to get my code running. For my own future reference and maybe of help to others, here they are.

Basic Setup

The Pi (called “ripple”) normally runs headless - when I want to do stuff I ssh in. However, its HDMI output is connected to the HDMI input of my main monitor (which gets its VGA input from my laptop which I usually run dual-monitor) and the main keyboard and mouse on my desk (and also the laser printer) can be switched between the two using a USB switch box so it's easy to get to a full desktop environment when needed.

That makes the first few steps of the setup simple.

From Main Menu | Preferences | Raspberry Pi Configuration: in the System tab set the password for the user pi and set underscan to disabled (it's not needed on my monitor and results in a depressing black border) and in the Interfaces tab set SSH to enabled.

From the Wireless and Wired Network Settings (top right) set the Wi-Fi password. Originally this Pi used a wired connection to my router but that stopped working - I suspect I've physically broken the connector at some point while fiddling with the device to get all the USB connections in.


adduser edavies
adduser edavies sudo
adduser mqtt
adduser mqtt dialout

I tend to use my “own” user rather than the default pi just because it makes ssh commands from the laptop a bit shorter. The mqtt user is to run all of the logging code. It's a bit of a misnomer as the code's structured around the MQTT broker and protocol but it's not really central to its actual function and the MQTT broker itself (mosquitto) runs in its own user.

User edavies is in group sudo to do various systems things. User mqtt isn't as it doesn't need it and it's somewhat more vulnerable to weird attacks so not allowing sudo access makes things slightly less rickety.

User mqtt needs to be in group dialout for access to a USB-to-serial adaptor to read household power readings from a Current Cost meter.

Set up .ssh/authorized_keys for edavies and mqtt.

Copy in standard .screenrc for edavies and mqtt.

Clone the software into ~mqtt/projects/mqtt_utils/ and the database into ~mqtt/mqtt/.


apt update and apt upgrade because it's a good idea and also because it's needed to install Sqlite3.

apt install mosquitto
apt install mosquitto-client
apt install owfs
apt install mercurial
apt install sqlite3
apt install screen
apt install nginx
apt install nptdate

Put a copy of websocketd in /usr/local/bin.

1-Wire Setup

I have just one 1-wire sensor actually in use for now: a DS18B20 temperature sensor stuffed up between the leaves of the living room radiator to help work out what the central heating is up to. It's connected via a USB to 1-wire adaptor so in /etc/owfs.conf uncomment out server: usb=all then:

systemctl restart owserver.service

Beyond the low-level TCP 1-wire server the other servers which sit on top are handy for quick testing but for normal production use in my application they're just using up resources and a source of vulnerability so best turned off:

systemctl stop owhttp.service
systemctl disable owhttp.service
systemctl stop owftpd.service
systemctl disable owftpd.service

Nginx Setup

The logging software serves data (JSON usually) over HTTP. To serve fixed files and to buffer the rather wobbly HTTP implementation built into Python I run nginx as a reverse proxy in front of it. One day it'll also provide HTTPS, particularly for control functions.

After that's installed add a copy of the ~mqtt/projects/mqtt_utils/nginx template file with an appropriate name in /etc/nginx/sites-available/. Add a symlink to it in /etc/nginx/sites-enabled/ and get rid of the default symlink there.

systemctl restart nginx.service

Run Software

For user mqtt, crontab -e and add:

# m h  dom mon dow   command
33  *    *   *   *   /home/mqtt/projects/mqtt_utils/checkwx_mqtt.py --logfile /home/mqtt/mqtt/checkwx_mqtt.log --apiKey <apikey>
*/2 *    *   *   *   /home/mqtt/projects/mqtt_utils/mqttrunner.py

<apikey> is the API Key emailed by CheckWX when I registered with them. The checkwx_mqtt.py program downloads the latest METAR for Wick Airport (code EGPC) at 33 minutes past each hour so generally gets the METAR taken at the previous 20 minutes past the hour giving me rough outdoor temperature and wind conditions.

mqttrunner.py is the main module of my logging software. The crontab entry runs it every two minutes. By default the first thing it does is check if there's an existing copy of itself running and, if not, fires up another copy of itself under screen (quick tutorial).

I could run the software via init or systemd but running via the crontab is simple and quite robust. If something goes wrong the program tends to just bail out completely then within the next two minutes another copy is started. When I want to update the software I just kill the running instance, ideally about 50 seconds into an odd-numbered minute, and it magically sorts itself out.

The benefit of running it under a (detached) screen session is that I can easily connect to it from my laptop to see how things are going. A convenient script for the purpose:

ssh -t mqtt@ripple screen -r -S mqttrunner

[Updated 2017-12-30: added details of the nginx setup and a link to Matt Cutts' screen tutorial.]

[Updated 2018-03-21: added ntpdate to list of packages to install as the code now uses ntpdate-debian to check the clock before starting normal logging to deal with the Raspberry Pi not having a real-time clock and sometimes not setting the clock quickly on a boot, e.g., after a power cut.]