How to Build a Simple IoT System with Python

Posted on 2 Comments

IoT

The Internet of Things (IoT) movement is in full swing and growing bigger every day. Wireless thermostats and refrigerators, home security systems controlled by a mobile app, scales and health monitors that save your weight and fitness data…these are some examples of the “smart” gadgets coming out of the IoT movement. This blog post describes how to use Python and the MQTT protocol to hook your gadgets up to a network. Message Queuing Telemetry Transport (MQTT) is a lightweight protocol that runs over TCP/IP and uses a publish/subscribe model to pass messages between publishers and consumers. It offers certificate-based SSL/TLS encryption and is often a good transport solution in low bandwidth situations common in IoT networks. This post describes how to set up a basic IoT system using a central message broker to receive and process messages from all the devices on your wireless network. A device is anything that is able to connect to wifi and send/receive an MQTT message: sensors (temperature, light, motion, etc.), controllers (lights on/off, etc.), etc. Although it’s outside of the scope of this article, it’s not that hard to create and program your own device these days, at least if it can run Python on it :) See the links at the end of this article for some ideas. For this tutorial, you can use any wifi-capable device, including a laptop simulating a real device. Here’s the list of what you’ll need:

  • Linux server or VM connected to a wifi network
    • Python
    • Mosquitto MQTT broker
    • MQTT subscriber code
    • Database
  • Device or simulator connected to a wifi network (can be a simulator script running on the above VM)
    • MQTT publisher code

I’ll assume you know some Linux and bash basics and how to set up your Python environment and use pip. I’ll be using Ubuntu 14.04.2 and Python 2.7.9.

The basic system we’re building is simple: a message broker that receives messages from devices, processes them, and saves them in a database. It’d be trivial to add a web interface to view the data, but I’ll leave that as an exercise for the reader.

Setting Up the Message Broker

Mosquitto is an MQTT message broker currently maintained by Eclipse IoT. It is written in C and does not consume much memory, even with lots of clients connected (specifically, “around 3MB RAM with 1000 clients connected“). Mosquitto has the ability to act as a bridge and connect to other MQTT brokers, but our setup will use a single instance as a central receiving point for messages from IoT devices. To install it, run: sudo apt-get install mosquitto

That should install mosquitto as a service and configure it to run on startup, listening for connections on port 1883. For SSL/TLS setup, see here. To test the broker, go ahead and install these command line utilities: sudo apt-get install mosquitto-clients Now you can easily subscribe to a topic with mosquitto_sub -t 'mytopic' and publish a message with mosquitto_pub -t 'mytopic' -m 'mymessage'. See here for more on mosquitto-clients. While developing, I almost always have a terminal window opened that has a subscriber running listening on all topics, represented by “#” and with the -v flag so that the topic is displayed alongside the message: mosquitto_sub -v -t '#'

The configuration file is located by default at /etc/mosquitto/mosquitto.conf on Ubuntu 14.04.2 and it is very well-commented. Note that there is no security or logging enabled by default.

Publishing and Receiving Messages with Python

MQTT messages have a payload and are published to a topic. The payload can be any type of data: text, JSON, XML, binary, etc. The max payload size allowed by Mosquitto can be configured (see message_size_limit), but the default cap is the max allowed by MQTT, around 268MB. MQTT topics are hierarchically structured and delimited by a slash ‘/‘. Some examples are ‘sensor/humidity/12345678’ and ‘device/light/kitchen’. A copy of the message will go to all clients subscribed to the message’s topic at the time of publication. The wildcard characters ‘+’ and ‘#’ can also be used when subscribing to topics. The ‘+’ matches exactly one topic and the ‘#’ matches zero or more topics. Expanding on the examples above, a subscription to ‘sensor/#’ would receive data from all sensors, e.g. messages published to ‘sensor/humidity/12345678’, ‘sensor/light/12345678’, etc. A subscription to ‘device/+/kitchen’ would receive data on all devices in the kitchen, e.g. messages published to ‘device/light/kitchen’, ‘device/refrigerator/kitchen’, etc. There is also a special set of topics beginning in ‘$SYS/‘ where subscribers can get system information like the number of connected clients, and broker uptime. See the documentation for a full listing.

To send and receive MQTT messages, we’ll use the Paho Python client library, also from Eclipse. To install it, run: pip install paho-mqtt or sudo pip install paho-mqtt

The Paho library uses an asynchronous event loop to connect to MQTT brokers. This means you have to create a series of functions, callbacks, that will run when certain events fire. These events include:

  • on_connect = connected to broker
  • on_disconnect = disconnected from broker
  • on_publish = message published
  • on_message = message received
  • on_subscribe = subscribed to topic
  • on_unsubscribe = unsubscribed from topic

You don’t have to set up every callback; for example on_publish will not be used in our subscriber code. After defining the callbacks, the code to connect to the broker is pretty concise. Here’s our simple subscriber (code adapted from Paho source repository):

import paho.mqtt.client as mqtt
from db import save_to_db

def on_connect(mqttc, userdata, rc):
    print('connected...rc=' + str(rc))
    mqttc.subscribe(topic='device/#', qos=0)

def on_disconnect(mqttc, userdata, rc):
    print('disconnected...rc=' + str(rc))

def on_message(mqttc, userdata, msg):
    print('message received...')
    print('topic: ' + msg.topic + ', qos: ' + 
          str(msg.qos) + ', message: ' + str(msg.payload))
    save_to_db(msg)

def on_subscribe(mqttc, userdata, mid, granted_qos):
    print('subscribed (qos=' + str(granted_qos) + ')')

def on_unsubscribe(mqttc, userdata, mid, granted_qos):
    print('unsubscribed (qos=' + str(granted_qos) + ')')

mqttc = mqtt.Client()
mqttc.on_connect = on_connect
mqttc.on_disconnect = on_disconnect
mqttc.on_message = on_message
mqttc.on_subscribe = on_subscribe
mqttc.on_unsubscribe = on_unsubscribe
mqttc.connect(host='localhost', port=1883)
mqttc.loop_forever()

gist

After the callbacks are defined, we create an instance of an MQTT client and assign the callbacks to its on_ properties. Then we connect to the broker, subscribe to a topic, and start the event loop. Because this code is asynchronous, nothing actually happens until the event loop is started. The connect and subscribe methods don’t run when the execution reaches their lines in the code. They are merely scheduled at that time and actually run once the event loop has started. This is the nature of event-driven, asynchronous code. When the loop starts, the connect function runs and its completion triggers on_connect. The subscribe function then runs and its completion triggers on_subscribe. At any point after subscribing, a message received will cause on_message to run. Here I assume you have a db.py module in the same directory as this code with a save_to_db function that parses the message and saves it to your database. I’ll leave that part for you to implement as you like. After a callback returns, the event loop takes over and waits on the next event to fire and next callback to be triggered. The loop_forever call will cause a client to listen until it calls disconnect. Our subscriber code doesn’t disconnect, but it may be interrupted with CTRL+C or killed.

The publisher code is very similar to the subscriber code. The key difference is that it publishes instead of subscribing in the on_connect callback, and disconnects after publishing one message:

import paho.mqtt.client as mqtt

def on_connect(mqttc, userdata, rc):
    print('connected...rc=' + str(rc))
    mqttc.publish(topic='device/sensor/temperature', 
                  payload='80', qos=0)

def on_disconnect(mqttc, userdata, rc):
    print('disconnected...rc=' + str(rc))

def on_message(mqttc, userdata, msg):
    print('message received...')
    print('topic: ' + msg.topic + ', qos: ' + 
          str(msg.qos) + ', message: ' + str(msg.payload))

def on_publish(mqttc, userdata, mid):
    print('message published')
    mqttc.disconnect()

mqttc = mqtt.Client()
mqttc.on_connect = on_connect
mqttc.on_disconnect = on_disconnect
mqttc.on_message = on_message
mqttc.on_publish = on_publish
mqttc.connect(host='localhost', port=1883)
mqttc.loop_forever()

gist

You can test this code all on the same machine using the mosquitto-clients utilities. When deploying, you have several options. In this example, the subscriber code would live on the same Linux machine as Mosquitto. The publisher code would be run on all your devices, periodically sending messages to the central broker. This setup is ideal if you have devices like sensors or monitors that just collect and send in data. For more complicated devices that you’d like to control via MQTT, the setup would be a little different. You could have a subscriber running on the device, and whenever a message is published to a certain topic with a certain payload, this might cause the device to perform an action like turning off a light. Just be careful to secure your IoT devices well or better yet, only expose them to your home network and not to the entire internet.

Go Forth and IoT

The examples in this blog post will hopefully be helpful in getting a basic IoT infrastructure up and running. Using open source tools, it is not too hard to set up an IoT system with a central message broker that receives messages from wifi-connected devices. With some more work, you could develop this base infrastructure into a full system to connect all your gadgets. Admittedly, I didn’t talk much about the “things” in IoT, and these devices are what the movement is all about! Here are some small boards that can run Python and are popular for IoT projects:

They can be connected to just about any device and to wifi. As an example, you could wire a temperature sensor to a PyBoard and program the board to read the temperature and send an MQTT message every hour. With your IoT system connecting all your smart devices, the possibilities are endless!

2 comments

  1. Nice article, I’ve been trying to use MQTT on some of my own projects and I ran into some problems related to connection stability and what exactly to use to ensure as much uptime as possible.
    So my question is if you know how much time the python MQTT library will try to reconnect to the broker if the internet connection of the device was lost and if I should subscribe again to the topics once reconnected (currently I am resubscribing but I read somewhere that this might not be necessary all the time).

  2. Thanks for your comment, Florin. As far as resubscribing to the topics once reconnected, you are correct, it isn’t necessary all the time. However, if you put the call to subscribe in the on_connect callback like the Paho docs, you’ll always call subscribe when you connect/reconnect.

    There is also the distinction between durable and non-durable clients. By default, clients are non-durable, so when they disconnect, all subscriptions are lost if the client reconnects.

    When you instantiate the client in Python, you can set the clean session flag as a keyword argument to make it durable:
    mqttc = mqtt.Client(client_id='device1234', clean_session=False)
    This means that every time that client disconnects, all of its subscriptions will be stored along with subsequent messages (QOS 1 or 2 only). So once this client connects again, it should automatically be subscribed and receive any messages that have come through in its absence. An important thing to point out here is the client_id. This the is used by the broker to uniquely identify the client, so it can look up any prior subscriptions or messages waiting. You’ll always need to use a unique client id with durable clients.

    As far as how long the Python Paho client will try to reconnect if the broker goes down, I’m not 100% sure. I just tested it by running the subscriber program above and stopping the mosquitto service. It was able to reconnect once I restarted mosquitto 10 minutes later. This will depend on the event loop underlying the Paho loop_forever call. For the IoT systems I’ve worked on, this was never an issue because all the clients were small devices that woke up for a few seconds, took a sensor reading, connected and sent a message to the broker, and went back to to sleep. If the broker wasn’t available the device could either discard that sensor reading or store it and send it the next time it connects. Exactly how to handle connectivity issues will depend on the system you’re working with and its requirements.

Leave a Reply

Your email address will not be published. All fields are required.