The Internet of Things (IoT) movement is in full swing and growing bigger every day. Wireless thermostats and refrigerators, home security systems controlled by a mobile app, scales and health monitors that save your weight and fitness data…these are some examples of the “smart” gadgets coming out of the IoT movement. This blog post describes how to use Python and the MQTT protocol to hook your gadgets up to a network. Message Queuing Telemetry Transport (MQTT) is a lightweight protocol that runs over TCP/IP and uses a publish/subscribe model to pass messages between publishers and consumers. It offers certificate-based SSL/TLS encryption and is often a good transport solution in low bandwidth situations common in IoT networks. This post describes how to set up a basic IoT system using a central message broker to receive and process messages from all the devices on your wireless network. A device is anything that is able to connect to wifi and send/receive an MQTT message: sensors (temperature, light, motion, etc.), controllers (lights on/off, etc.), etc. Although it’s outside of the scope of this article, it’s not that hard to create and program your own device these days, at least if it can run Python on it See the links at the end of this article for some ideas. For this tutorial, you can use any wifi-capable device, including a laptop simulating a real device. Here’s the list of what you’ll need:
- Linux server or VM connected to a wifi network
- Mosquitto MQTT broker
- MQTT subscriber code
- Device or simulator connected to a wifi network (can be a simulator script running on the above VM)
- MQTT publisher code
The basic system we’re building is simple: a message broker that receives messages from devices, processes them, and saves them in a database. It’d be trivial to add a web interface to view the data, but I’ll leave that as an exercise for the reader.
Setting Up the Message Broker
Mosquitto is an MQTT message broker currently maintained by Eclipse IoT. It is written in C and does not consume much memory, even with lots of clients connected (specifically, “around 3MB RAM with 1000 clients connected“). Mosquitto has the ability to act as a bridge and connect to other MQTT brokers, but our setup will use a single instance as a central receiving point for messages from IoT devices. To install it, run:
sudo apt-get install mosquitto
That should install mosquitto as a service and configure it to run on startup, listening for connections on port 1883. For SSL/TLS setup, see here. To test the broker, go ahead and install these command line utilities:
sudo apt-get install mosquitto-clients Now you can easily subscribe to a topic with
mosquitto_sub -t 'mytopic' and publish a message with
mosquitto_pub -t 'mytopic' -m 'mymessage'. See here for more on mosquitto-clients. While developing, I almost always have a terminal window opened that has a subscriber running listening on all topics, represented by “#” and with the
-v flag so that the topic is displayed alongside the message:
mosquitto_sub -v -t '#'
The configuration file is located by default at /etc/mosquitto/mosquitto.conf on Ubuntu 14.04.2 and it is very well-commented. Note that there is no security or logging enabled by default.
Publishing and Receiving Messages with Python
MQTT messages have a payload and are published to a topic. The payload can be any type of data: text, JSON, XML, binary, etc. The max payload size allowed by Mosquitto can be configured (see message_size_limit), but the default cap is the max allowed by MQTT, around 268MB. MQTT topics are hierarchically structured and delimited by a slash ‘/‘. Some examples are ‘sensor/humidity/12345678’ and ‘device/light/kitchen’. A copy of the message will go to all clients subscribed to the message’s topic at the time of publication. The wildcard characters ‘+’ and ‘#’ can also be used when subscribing to topics. The ‘+’ matches exactly one topic and the ‘#’ matches zero or more topics. Expanding on the examples above, a subscription to ‘sensor/#’ would receive data from all sensors, e.g. messages published to ‘sensor/humidity/12345678’, ‘sensor/light/12345678’, etc. A subscription to ‘device/+/kitchen’ would receive data on all devices in the kitchen, e.g. messages published to ‘device/light/kitchen’, ‘device/refrigerator/kitchen’, etc. There is also a special set of topics beginning in ‘$SYS/‘ where subscribers can get system information like the number of connected clients, and broker uptime. See the documentation for a full listing.
To send and receive MQTT messages, we’ll use the Paho Python client library, also from Eclipse. To install it, run:
pip install paho-mqtt or
sudo pip install paho-mqtt
The Paho library uses an asynchronous event loop to connect to MQTT brokers. This means you have to create a series of functions, callbacks, that will run when certain events fire. These events include:
- on_connect = connected to broker
- on_disconnect = disconnected from broker
- on_publish = message published
- on_message = message received
- on_subscribe = subscribed to topic
- on_unsubscribe = unsubscribed from topic
You don’t have to set up every callback; for example
on_publish will not be used in our subscriber code. After defining the callbacks, the code to connect to the broker is pretty concise. Here’s our simple subscriber (code adapted from Paho source repository):
import paho.mqtt.client as mqtt from db import save_to_db def on_connect(mqttc, userdata, rc): print('connected...rc=' + str(rc)) mqttc.subscribe(topic='device/#', qos=0) def on_disconnect(mqttc, userdata, rc): print('disconnected...rc=' + str(rc)) def on_message(mqttc, userdata, msg): print('message received...') print('topic: ' + msg.topic + ', qos: ' + str(msg.qos) + ', message: ' + str(msg.payload)) save_to_db(msg) def on_subscribe(mqttc, userdata, mid, granted_qos): print('subscribed (qos=' + str(granted_qos) + ')') def on_unsubscribe(mqttc, userdata, mid, granted_qos): print('unsubscribed (qos=' + str(granted_qos) + ')') mqttc = mqtt.Client() mqttc.on_connect = on_connect mqttc.on_disconnect = on_disconnect mqttc.on_message = on_message mqttc.on_subscribe = on_subscribe mqttc.on_unsubscribe = on_unsubscribe mqttc.connect(host='localhost', port=1883) mqttc.loop_forever()
After the callbacks are defined, we create an instance of an MQTT client and assign the callbacks to its
on_ properties. Then we connect to the broker, subscribe to a topic, and start the event loop. Because this code is asynchronous, nothing actually happens until the event loop is started. The connect and subscribe methods don’t run when the execution reaches their lines in the code. They are merely scheduled at that time and actually run once the event loop has started. This is the nature of event-driven, asynchronous code. When the loop starts, the
connect function runs and its completion triggers
subscribe function then runs and its completion triggers
on_subscribe. At any point after subscribing, a message received will cause
on_message to run. Here I assume you have a
db.py module in the same directory as this code with a
save_to_db function that parses the message and saves it to your database. I’ll leave that part for you to implement as you like. After a callback returns, the event loop takes over and waits on the next event to fire and next callback to be triggered. The
loop_forever call will cause a client to listen until it calls
disconnect. Our subscriber code doesn’t disconnect, but it may be interrupted with CTRL+C or killed.
The publisher code is very similar to the subscriber code. The key difference is that it publishes instead of subscribing in the
on_connect callback, and disconnects after publishing one message:
import paho.mqtt.client as mqtt def on_connect(mqttc, userdata, rc): print('connected...rc=' + str(rc)) mqttc.publish(topic='device/sensor/temperature', payload='80', qos=0) def on_disconnect(mqttc, userdata, rc): print('disconnected...rc=' + str(rc)) def on_message(mqttc, userdata, msg): print('message received...') print('topic: ' + msg.topic + ', qos: ' + str(msg.qos) + ', message: ' + str(msg.payload)) def on_publish(mqttc, userdata, mid): print('message published') mqttc.disconnect() mqttc = mqtt.Client() mqttc.on_connect = on_connect mqttc.on_disconnect = on_disconnect mqttc.on_message = on_message mqttc.on_publish = on_publish mqttc.connect(host='localhost', port=1883) mqttc.loop_forever()
You can test this code all on the same machine using the mosquitto-clients utilities. When deploying, you have several options. In this example, the subscriber code would live on the same Linux machine as Mosquitto. The publisher code would be run on all your devices, periodically sending messages to the central broker. This setup is ideal if you have devices like sensors or monitors that just collect and send in data. For more complicated devices that you’d like to control via MQTT, the setup would be a little different. You could have a subscriber running on the device, and whenever a message is published to a certain topic with a certain payload, this might cause the device to perform an action like turning off a light. Just be careful to secure your IoT devices well or better yet, only expose them to your home network and not to the entire internet.
Go Forth and IoT
The examples in this blog post will hopefully be helpful in getting a basic IoT infrastructure up and running. Using open source tools, it is not too hard to set up an IoT system with a central message broker that receives messages from wifi-connected devices. With some more work, you could develop this base infrastructure into a full system to connect all your gadgets. Admittedly, I didn’t talk much about the “things” in IoT, and these devices are what the movement is all about! Here are some small boards that can run Python and are popular for IoT projects:
They can be connected to just about any device and to wifi. As an example, you could wire a temperature sensor to a PyBoard and program the board to read the temperature and send an MQTT message every hour. With your IoT system connecting all your smart devices, the possibilities are endless!