Skip to main content
Version: 0.13.1

MQTT API

Topic Scheme​

The thin-edge MQTT topic structure adopts a flexible and descriptive semantics, whilst keeping it consistent and predictable to enable other clients/mappers to interact with the thin-edge components.

Whilst the topic structure is flexible, the document will focus on the typical use case. Advanced use-cases can be viewed in the advanced section, but it is strongly encouraged to start out simple and only go to the advanced section if needed.

The MQTT topics are represented by three distinct groups; root, identifier and channel. Each group contains one or more segments.

The typical topic structure is visualized in the following diagram.

Where the groups are described as follows:

GroupDescription
rootBase topic to group the identifier and channel under one common namespace
identifierA descriptor which represents which device/service the channel data is related to
channelRepresents the information, such as telemetry data and commands, related to the identifier. Each channel type defines its own sub topic structure and corresponding payload format.

The specifics of each group are detailed in the following sections.

Root​

The root/base topic prefix is used to group all data related to one thin-edge instance. The root topic is used to avoid conflicts with other clients communicating on the same MQTT broker.

The convention is to use a fixed value of te (short for "thin-edge").

Identifier​

The identifier group is made up of four segments. Together these four segments reflect which device/service the channel information relates to. Such channel data includes telemetry data and commands.

The identifier can be visually represented by the following diagram:

Each segment is described as follows:

SegmentDescriptionRequired
deviceA literal string, "device", indicating that the information is related to a deviceYes
<device_id>id/name of the device. A value of main is used to represent the main device and any other value represents the device id of a child device. e.g. child01, mycustomdeviceYes
serviceA literal string, "service", indicating that the information is related to a serviceNo
<service_id>Service id/name which the information is related toNo
info

An empty segment value is used to represent "not applicable". For instance publishing telemetry data of a device involves setting both the "service" and "service_id" segments to an empty string. See the identifier-example for more details.

Identifier Example​

To help better understand how the identifier is used in the topics, let's image the following setup. You have two devices; the main device and a single child device called "child01". Both the main device (where thin-edge is running), and the child device have a single service called "node_red". Thin-edge will build digital twins for the devices (main and child device), and the "node_red" services associated with each of the devices.

The following diagram details the device hierarchy of the fictional setup.

The diagram shows that there are two devices; the "main" device (the digital twin representation of thin-edge), and a child device called "child01".

The following table shows how the identifier is used to represent the different combination of devices and services, e.g. main device, node_red service running on the main device, child device and the node_red service running on the child device.

EntityStructureIdentifier topic prefix
Main
Devicete/device/main//
Servicete/device/main/service/node_red
Child ("child01")
Devicete/device/child01//
Servicete/device/child01/service/node_red

Channel​

The channel group represents the information which is associated to the identifier. Information includes both telemetry data (e.g. measurement, events and alarms) as well as commands which are used to execute actions on the device.

The channel group is represented by the following segments:

The individual categories dictate the subsequent topic structure and payload schema, however the MQTT topic schema strives to keep some consistency amongst the different categories by applying similar concepts where possible.

SegmentDescription
<category>Category of data (telemetry or commands) which is related to the identifier. A fixed list of categories is used to represent data types such as; measurements, alarms, events, commands etc.
<type>A unique type/name used to identify the information being pushed or received. Types allow users to filter/subscribe to data which interests them. For example, a measurement could be published under a type called "flow_rate", then other clients interested in the flow rate can subscribe to that single typed topic.
...Additional channel specific topic segments. Each category is responsible for defining the number and meaning of the remaining topic segments.

Categories​

The following is an overview of the channel categories which are available.

CategoryDescription
mMeasurements
eEvents
aAlarms
cmdCommands
twinEntity twin metadata
statusService status

Entity registration​

Since thin-edge doesn't enforce what each entity identification level means, an explicit registration is required to register every entity that is going to send data or receive commands. For example, before a measurement can be sent from a service named tedge-agent from the device rpi1001, the entity named rpi1001 must be registered as a device, and tedge-agent must be registered as a service linked to that device.

An entity can be registered with thin-edge by publishing a retained message to the entity identification topic prefix with the entity type and other metadata that defines that entity. To model the example mentioned above, if an entity identification topic scheme like the following is used:

te/device/<device_id>/service/<service_id>

Examples​

Here are a few examples of how various entities can be registered.

Register a device​

tedge mqtt pub -r 'te/device/main//' '{
  "@type": "device",
  "type": "Gateway"
}'

Or the device can be registered using an explicit id:

tedge mqtt pub -r 'te/device/main//' '{
  "@type": "device",
  "@id": "tedge001",
  "type": "Gateway"
}'

Register a service of the main device​

tedge mqtt pub -r 'te/device/main/service/nodered' '{
  "@type": "service",
  "name": "nodered",
  "type": "systemd"
}'

The service is implicitly linked to the parent derived from the topic, main in this example.

But the parent can be explicitly provided as well with the @parent key, if the parent can not be derived from the topic directly:

tedge mqtt pub -r 'te/component_namespace/service/nodered/instance-1' '{
  "@type": "service",
  "@parent": "device/main//",
  "name": "nodered",
  "type": "systemd"
}'

Register a child device​

tedge mqtt pub -r 'te/device/child01//' '{
  "@type": "child-device",
  "name": "child01",
  "type": "SmartHomeHub"
}'

If the @parent info is not provided, it is assumed to be an immediate child of the main device.

Register a nested child device​

Nested child devices are registered in a similar fashion as an immediate child device, however the registration message requires the additional @parent property to be set, indicating which parent the child device should be related to.

tedge mqtt pub -r 'te/device/nested_child01//' '{
  "@type": "child-device",
  "@parent": "device/child01//",
  "name": "nested_child01"
}'

Register a service of a child device​

Service registration for child devices also follow the same rules as the main device, where the @parent device info is derived from the topic itself, by default. But, it is advised to declare it explicitly as follows:

tedge mqtt pub -r 'te/device/child01/service/nodered' '{
  "@type": "service",
  "@parent": "device/child01//",
  "name": "nodered",
  "type": "systemd"
}'
info

A service is always owned by a device. The @parent property in the registration message is used to declare the service's owner. The lifecycle of the service is tied to the device's lifecycle.

For example, a linux service runs on a device as it relies on physical hardware to run. A service can have its own telemetry data (e.g. tracking RAM usage of single process), however when the device ceases to exist, then so does the service.

Register a service of a nested child device​

tedge mqtt pub -r 'te/device/nested_child01/service/nodered' '{
  "@type": "service",
  "@parent": "device/nested_child01//",
  "name": "nodered",
  "type": "systemd"
}'

Auto Registration​

Users can use an auto-registration mechanism for immediate child devices and services, if they conform to the following topic scheme that clearly demarcates the device and service in the topic itself:

te/device/<device_id>/service/<service_id>

Where the second subtopic level is device and 4th level is service.

For example, if the following measurement message is received without any explicit registrations,

te/device/rpi1001/service/collectd/m/cpu_usage

rpi1001 and collectd will be auto-registered as device and service types.

In addition, the main device is associated with the following topic:

te/device/main//

The main device topic follows the same topic schema but the service and <service_id> sections are left blank, whilst keeping the slash separators.

caution

Auto-registration of entities can be enabled/disabled via tedge configuration.

Users are highly encouraged to register the devices manually as it allows devices full control over their registration process. Meta information can also be added to the device to better describe the device's custom type and function.

Entity store​

All the entity registration messages retained with the MQTT broker helps thin-edge components to maintain an entity store with all the registered devices and services along with their metadata.

Components like mappers use such an entity store while processing data from various sources. For example, a mapper component can get info about all the registered entities with a simple subscription for all possible entity identification topic levels as follows:

mosquitto_sub -t 'te/+/+/+/+'
note

The MQTT broker is used as the persistence layer to store the registered entities (assuming the registration messages were published with the MQTT retain flag).

Data types​

Telemetry and commands use the data type topic levels after the entity/component subtopics. Even for the data type levels, a user is free to define those as they wish. But thin-edge has some pre-defined <data-type> subtopics for well-known types like measurements, alarms, events and commands, on which it enforces some constraints, so that it can process them.

Telemetry data​

TypeTopic scheme
Measurementste/<identifier>/m/<measurement-type>
Eventste/<identifier>/e/<event-type>
Alarmste/<identifier>/a/<alarm-type>
Twinte/<identifier>/twin/<data-type>
Statuste/<identifier>/status/<target-type>

Examples: With default device/service topic semantics​

Publish to the main device​

tedge mqtt pub -r 'te/device/main///m/environment' '{
  "temperature": 23.4
}'

If the there is no measurement type, then the type can be left empty, but it must have the trailing slash / (so that the number of topic segments is the same).

tedge mqtt pub -r 'te/device/main///m/' '{
  "temperature": 23.4
}'

Publish to a child device​

tedge mqtt pub -r 'te/device/child01///m/environment' '{
  "temperature": 23.4
}'

Publish to a service on the main device​

tedge mqtt pub -r 'te/device/main/service/nodered/m/environment' '{
  "temperature": 23.4
}'

Any MQTT client can subscribe to all measurements for all entities (devices and services) using the following MQTT topic:

tedge mqtt sub 'te/+/+/+/+/m/+'

If you want to be more specific and only subscribe to the main device, then you can used fixed topic names rather than wildcards:

tedge mqtt sub 'te/device/main/+/+/m/+'

Or to subscribe to a specific type of measurement published to an services on the main device, then use:

tedge mqtt sub 'te/device/main/service/+/m/memory'

Publish to a service on a child device​

tedge mqtt pub -r 'te/device/child01/service/nodered/m/environment' '{
  "temperature": 23.4
}'

Telemetry type metadata​

The data types also may have additional metadata associated with it, which can be added/updated by publishing to /meta subtopics of those data types. For example, the units associated with measurements in the battery_reading measurement type can be updated by publishing the following message:

tedge mqtt pub -r 'te/device/main///m/battery_reading/meta' '{
  "units": {
    "temperature": "°C",
    "voltage": "V",
    "current": "A"
  }
}'

The metadata fields supported by each data type will be defined in detail later.

Twin metadata​

The twin metadata type can be used to store additional information about entities (devices and services). Such information could included: operation system name/version, communication statistics, device status, or any other information that is not suited to be measurements, events or alarms.

Topic (retain=true)
te/device/main///twin/device_OS
Payload
{
  "family": "Debian",
  "version": "11"
}

Commands​

The topic scheme for commands can be visualized using the diagram below.

Where the command segments are describe as follows:

SegmentDescription
<identifier>The identifier (e.g. device/service) associated with the command.
<cmd_type>Command type. Each command can define its own payload schema to allow commands to have parameters related to the command's function.
<cmd_id>Unique command id which is unique for the command instance. e.g. 123456, d511a86cab95be81 etc.

Command examples​

The following table details some example command types which are supported by thin-edge.

Command TypeExample Topic
software_listte/<identifier>/cmd/software_list/<cmd_id>
software_updatete/<identifier>/cmd/software_update/<cmd_id>
config_snapshotte/<identifier>/cmd/config_snapshot/<cmd_id>
config_updatete/<identifier>/cmd/config_update/<cmd_id>
firmware_updatete/<identifier>/cmd/firmware_update/<cmd_id>
restartte/<identifier>/cmd/restart/<cmd_id>
log_uploadte/<identifier>/cmd/log_upload/<cmd_id>
healthte/<identifier>/cmd/health/check

The command would be interpreted differently based on the target entity. For example, the restart could mean either a device restart or a service restart based on the target entity.

Examples: With default device/service topic semantics​

Command to main device​

Command to fetch the software list from the main device:

tedge mqtt pub -r 'te/device/main///cmd/software_list/123' '{
  "status": "init"
}'

The status value will transition from init to the final successful or failed via many intermediate states such as validating, downloading, executing etc.

The status field can even be skipped, which implies init status as follows:

tedge mqtt pub -r 'te/device/main///cmd/software_list/123' '{}'

Command to child device​

Command to update the firmware of a child device:

tedge mqtt pub -r 'te/device/child01///cmd/firmware_update/123' '{
  "status": "init",
  "attempt": 1,
  "name": "OpenWRT",
  "version": "22.03",
  "url": "http://127.0.0.1:8000/tedge/file-transfer/tedge-child/firmware_update/93d50a297a8c235",
  "sha256": "c036cbb7553a909f8b8877d4461924307f27ecb66cff928eeeafd569c3887e29"
}'

Command to a service​

Command to update the configuration of a service:

tedge mqtt pub -r 'te/device/main/service/collectd/cmd/config_update/123' '{
  "status": "init",
  "type": "collectd",
  "path": "/etc/collectd/collectd.conf",
  "url": "http://127.0.0.1:8000/tedge/file-transfer/collectd/config_update/collectd"
}'

Commands metadata​

For commands as well, additional command specific metadata can be registered as retained messages on the <cmd_type> topic.

For example, the supported configuration list of the main device can be declared as follows:

tedge mqtt pub -r 'te/device/main///cmd/config_snapshot' '{
  "description": "Upload a configuration from the device",
  "types": [
    "mosquitto",
    "tedge",
    "collectd"
  ]
}'

Health check​

Services can publish their health status as follows:

tedge mqtt pub -r 'te/device/main/service/tedge-agent/status/health' '{
  "pid": 1234,
  "status": "up",
  "time": 1674739912
}'

Services are responsible for updating their own health status by publishing to the above topic on any status changes. However, other clients can request the service to update its status by sending a health check command as shown below:

tedge mqtt pub -r 'te/device/main/service/tedge-agent/cmd/health/check' '{}'

Services are also expected to react to device-wide health check commands as well (where service and <service_id> segments are left blank):

tedge mqtt pub -r 'te/device/main///cmd/health/check' '{}'

On receipt of the above command, all services on that device should respond with their health status.

The services are also expected to register an MQTT Last Will and Testament (LWT) message with the broker to publish a down status message in the event that the service stops or crashes unexpectedly. The Last Will and Testament message ensures that the down status is published even if the service is not operational. The following example details such a message:

tedge mqtt pub -r 'te/device/main/service/tedge-agent/status/health' '{
  "status": "down"
}'

Backward compatibility for legacy tedge topics​

A compatibility layer should be implemented to map the existing (legacy) tedge topics to the new topic structure. Although the topics for measurements, events etc. can be mapped directly to the new scheme by just changing the topics, certain types like alarms, which had severity in the topic in the old scheme must be mapped to the payload in the new scheme. Even for measurements and alarms, the position of child device id in the topics are also different. So, a compatibility mapper is essential during the transition phase from old scheme to new scheme.

It is important that the layer translate from the legacy to the new so that both existing and new entities/components are both observable via the new topics.

The compatibility layer would be deactivated via configuration after all entities/components have been migrated.

Below details some examples showing the mapping between the legacy and new topics:

Telemetry: Main device​

TypeTopicPayload Changes
Measurements

Legacy

tedge/measurements

New

te/device/main///m/<type>

No Change. If "<type>" is not provided, a default value of "ThinEdgeMeasurement" will be used.

Events

Legacy

tedge/events/<type>

New

te/device/main///e/<type>

No Change

Alarms

Legacy

tedge/alarms/<severity>/<type>

New

te/device/main///a/<type>

The alarm severity should be set in the payload.

{
  "severity": "<severity>"
  // ...
}
Health status

Legacy

tedge/health/<service_name>

New

te/<service_topic_id>/status/health

type property removed from the payload.

Telemetry: Child device​

TypeTopicPayload Changes
Measurements

Legacy

tedge/measurements/<child_id>

New

te/device/<child_id>///m/<type>

No Change. If "<type>" is not provided, a default value of "ThinEdgeMeasurement" will be used.

Events

Legacy

tedge/events/<type>/<child_id>

New

te/device/<child_id>///e/<type>

No Change

Alarms

Legacy

tedge/alarms/<severity>/<type>/<child_id>

New

te/device/<child_id>///a/<type>

The alarm severity should be set in the payload.

{
  "severity": "<severity>"
  // ...
}
Health status

Legacy

tedge/health/<child_id>/<service_name>

New

te/<service_topic_id>/status/health

type property removed from the payload.