I have an “always-on” Raspberry Pi 3 which does logging of temperature, pressure, power use and so on readings from around the house and county.
I've had it just over a year and it's been running the version of Raspbian (the Debian Linux distribution modified for the Pi) that was current when I got it. The logging has been to flat files containing lines of JSON but recently I started switching to a Sqlite3 database but was having problems with the code running on the Pi which didn't happen on my laptop running Ubuntu 17.04. I strongly suspected the problem was the old version of Sqlite3 or Python3 in that Raspbian distribution so decided to upgrade to the latest Rasbian (2017-11-29, based on Debian 9 Stretch rather than the previous one which was based on Debian 8 Jessie).
During the installation on the new system I made notes of what I needed to do to get my code running. For my own future reference and maybe of help to others, here they are.
Basic Setup
The Pi (called “ripple”) normally runs headless - when I want to do stuff I ssh in. However, its HDMI output is connected to the HDMI input of my main monitor (which gets its VGA input from my laptop which I usually run dual-monitor) and the main keyboard and mouse on my desk (and also the laser printer) can be switched between the two using a USB switch box so it's easy to get to a full desktop environment when needed.
That makes the first few steps of the setup simple.
From Main Menu | Preferences | Raspberry Pi Configuration:
in the System tab set the password for the user pi
and set underscan to disabled (it's not needed
on my monitor and results in a depressing black border)
and in the Interfaces tab set SSH to
enabled.
From the Wireless and Wired Network Settings (top right)
set the Wi-Fi password. Originally this Pi used a wired connection to
my router but that stopped working - I suspect I've physically broken
the connector at some point while fiddling with the device to get all
the USB connections in.
Users
adduser edavies adduser edavies sudo adduser mqtt adduser mqtt dialout
I tend to use my “own” user rather than the default pi
just because it makes ssh commands from the laptop a bit shorter.
The mqtt user is to run all of the logging code. It's
a bit of a misnomer as the code's structured around the
MQTT
broker and protocol but it's not really central to its actual
function and the MQTT broker itself
(mosquitto) runs in its own
user.
User edavies is in group sudo to do various
systems things. User mqtt isn't as it doesn't need it and
it's somewhat more vulnerable to weird attacks so not allowing sudo
access makes things slightly less rickety.
User mqtt needs to be in group dialout for
access to a USB-to-serial adaptor to read household power readings
from a
Current Cost meter.
Set up .ssh/authorized_keys for edavies and
mqtt.
Copy in standard .screenrc for edavies and
mqtt.
Clone the software into ~mqtt/projects/mqtt_utils/ and
the database into ~mqtt/mqtt/.
Software
apt update and apt upgrade because it's a
good idea and also because it's needed to install Sqlite3.
apt install mosquitto apt install mosquitto-client apt install owfs apt install mercurial apt install sqlite3 apt install screen apt install nginx apt install nptdate
Put a copy of websocketd in
/usr/local/bin.
1-Wire Setup
I have just one 1-wire sensor actually in use for now: a DS18B20
temperature sensor stuffed up between the leaves of the living room
radiator to help work out what the central heating is up to. It's
connected via a USB to 1-wire adaptor so in /etc/owfs.conf
uncomment out server: usb=all then:
systemctl restart owserver.service
Beyond the low-level TCP 1-wire server the other servers which sit on top are handy for quick testing but for normal production use in my application they're just using up resources and a source of vulnerability so best turned off:
systemctl stop owhttp.service systemctl disable owhttp.service systemctl stop owftpd.service systemctl disable owftpd.service
Nginx Setup
The logging software serves data (JSON usually) over HTTP. To serve fixed files and to buffer the rather wobbly HTTP implementation built into Python I run nginx as a reverse proxy in front of it. One day it'll also provide HTTPS, particularly for control functions.
After that's installed add a copy of the
~mqtt/projects/mqtt_utils/nginx template file with an
appropriate name in
/etc/nginx/sites-available/.
Add a symlink to it in
/etc/nginx/sites-enabled/ and get rid of the default
symlink there.
systemctl restart nginx.service
Run Software
For user mqtt,
crontab -e
and add:
# m h dom mon dow command 33 * * * * /home/mqtt/projects/mqtt_utils/checkwx_mqtt.py --logfile /home/mqtt/mqtt/checkwx_mqtt.log --apiKey <apikey> */2 * * * * /home/mqtt/projects/mqtt_utils/mqttrunner.py
<apikey> is the API Key emailed by
CheckWX when I registered with
them. The checkwx_mqtt.py program downloads the latest
METAR
for
Wick Airport
(code EGPC)
at 33 minutes past each hour so generally gets the METAR taken at the
previous 20 minutes past the hour giving me rough outdoor temperature
and wind conditions.
mqttrunner.py is the main module of my logging software.
The crontab entry runs it every two minutes. By default the first thing
it does is check if there's an existing copy of itself running and, if not,
fires up another copy of itself under
screen
(quick tutorial).
I could run the software via init or systemd but running via the crontab is simple and quite robust. If something goes wrong the program tends to just bail out completely then within the next two minutes another copy is started. When I want to update the software I just kill the running instance, ideally about 50 seconds into an odd-numbered minute, and it magically sorts itself out.
The benefit of running it under a (detached) screen session is that I can easily connect to it from my laptop to see how things are going. A convenient script for the purpose:
#!/bin/bash ssh -t mqtt@ripple screen -r -S mqttrunner
[Updated 2017-12-30: added details of the nginx setup and a link to Matt Cutts' screen tutorial.]
[Updated 2018-03-21: added ntpdate to list of packages to install as the code now uses ntpdate-debian to check the clock before starting normal logging to deal with the Raspberry Pi not having a real-time clock and sometimes not setting the clock quickly on a boot, e.g., after a power cut.]