Node to Cloud Communication

about

This page outlines how nodes securely interact with the cloud using MQTT over TLS. Each node possesses a unique identifier and credentials established during the claiming process, ensuring secure, authenticated communication. The node configuration, detailing attributes like devices and parameters, is sent to the cloud upon startup, enabling clients such as mobile apps to interpret and utilize this data. Understanding this communication framework is essential for developers to effectively integrate and manage IoT devices within the ESP RainMaker ecosystem.

info

All communication between the node and cloud happens over secure MQTT

How does a Node ensure secured communication with the Cloud?

A node ensures secured communicated with the cloud by providing:

Transport Layer Security (TLS)
MQTT Policies

TLS

The communication between a node and the cloud is secured using TLS encryption with X.509 certificate-based mutual authentication, ensuring only trusted devices can connect.

A generic workflow to establish secured communication:

The node first generates its own private key and then creates a Certificate Signing Request (CSR), which is like applying for an official ID.
This request is sent to a Certification Authority (CA), which verifies it, signs it, and returns a client certificate—essentially an authenticated ID badge.
During the Claiming process, the node receives this certificate, allowing it to establish a secure, encrypted connection with the cloud.
Before any data exchange, both the node and the cloud verify each other’s certificates to confirm they are communicating with the right entity.
This ensures privacy, security, and protection against unauthorized access, preventing fake or malicious devices from infiltrating the system.

Why is this important?

Ensures that only trusted nodes can connect to the cloud.
Prevents hackers or fake devices from pretending to be real nodes.
Keeps all communication private and secure.

MQTT Policies

The MQTT Policy ensures that a node can only connect to the cloud if its Client ID matches its Node ID, preventing unauthorized access. Additionally, the node is restricted to publishing and subscribing only to topics that start with node/<node_id>/ meaning it cannot access topics from other nodes.

tip

For example, if a node has an ID of 1234, it can only interact with topics like node/1234/status or node/1234/command, but it cannot subscribe to node/5678/status.

This policy acts as a security mechanism to keep each node’s communication isolated and protected, ensuring that data remains private and that nodes cannot interfere with each other.

In Summary

Security Aspect	Description
TLS	Locks the communication channel so that no one can listen in or tamper with messages.
MQTT Policies	Sets rules on what a node is allowed to talk about within the secured channel.

MQTT Topics

What is an MQTT Topic?

In ESP RainMaker, an MQTT topic acts as a communication path where a node publishes (sends) updates and subscribes (receives) commands or configurations. Here are the different types of MQTT topics in ESP RainMaker.

Node Configuration
Node Parameters
User-Node Mapping
Time Series
- Time Series Data
- Simple Time Series Data
Alert and Notify
Command Response
- To Node
- From Node
OTA

Node Configuration

In ESP RainMaker, Node Configuration is a collection of all the information about a node (i.e. ESP32-S3). When the node starts up, it sends this configuration to the ESP RainMaker Cloud and also reports it to clients, like mobile apps and command-line interfaces (CLI). This configuration contains everything needed to define the node’s identity, features, and capabilities.

Below is the node configuration layout that ESP RainMaker firmware and phone Apps currently use in the order from top to bottom.

Node Configuration Layout (Click to view)

Entries & Parameters

Box in RED is a mandatory entry in a node configuration.
Box in GREEN is a mandatory parameter of an entry.
The strings below each entries indicate the corresponding key in
(JSON representation , Data type).

1. Node ID

2. Config Version

2. Information

3. Node Attributes

4. Devices & Params

MQTT

Topic node/<node_id>/config
Operation: PUBLISH
Data: Node configuration JSON object described below

JSON Payload (Click to view)

"node_id": "5d898491-a498-48ec-9476-a4e23519fe31",
"config_version": "2019-02-27",
"info": {
    "name": "My_Bridge",
    "fw_version": "1.0",
    "type": "Bridge"
},
"attributes": [{
    "name": "serial_num",
    "value": "123abc"
}, {
    "name": "model",
    "value": "myBridge-2019"
}],
"devices": [{
    "name": "Switch",
    "primary": "power",
    "params": [{
        "name": "name",
        "type": "esp.param.name",
        "data_type": "string",
        "properties": ["read", "write"]
    }, {
        "name": "power",
        "data_type": "bool",
        "properties": ["read", "write", "time_series"],
        "ui_type": "esp.ui.toggle"
    }]
}, {
    "name": "Light",
    "type": "esp.device.lightbulb",
    "attributes": [{
        "name": "serial_number",
        "value": "012345"
    }, {
        "name": "mac",
        "value": "xx:yy:zz:aa:bb:cc"
    }],
    "primary": "power",
    "params": [{
        "name": "name",
        "type": "esp.param.name",
        "data_type": "string",
        "properties": ["read", "write"]
    }, {
        "name": "power",
        "data_type": "bool",
        "properties": ["read", "write", "time_series"],
        "ui_type": "esp.ui.toggle"
    }, {
        "name": "brightness",
        "data_type": "int",
        "properties": ["read", "write"],
        "bounds": {
            "min": 0,
            "max": 100
        },
        "ui_type": "esp.ui.slider"
    }]
}]

Trigger Logic

By default, the device reports once after power on and connection to the cloud. It will also report when the esp_rmaker_report_node_details is called.

Firmware API: esp_rmaker_report_node_details()

Node Parameters

All parameter changes are managed through specific node parameter topics.

Initial Reporting

Initial reporting is done every time the node boots up. Values of all the parameters of all the devices are reported in this.

MQTT

Topic node/<node_id>/params/local/init
Operation: PUBLISH
Data: {"<device-name">:{"<param-name>":<value>,...},...}

JSON Payload (Click to view)

{
  "Lightbulb": {
      "name": "Bedroom Light",
      "power": true,
      "brightness": 55
  }
}

trigger logic

When the node powers on for the first time and connects to the cloud, the node will report the current values of all parameters.
The node can also call the esp_rmaker_report_node_state API to report all parameter values.

Firmware API: esp_rmaker_report_node_state()

Subsequent Reporting

Any subsequent changes on the node, either due to remote control or a local change, are reported on this topic. Only the parameter that has changed is required in the payload. It is used to report parameter values to the cloud.

MQTT

Topic node/<node_id>/params/local
Operation: PUBLISH
Data: {"<device-name">:{"<param-name>":<value>,...},...}

JSON Payload (Click to view)

{
    "Lightbulb": {
        "brightness": 75
    }
}

Trigger Logic

When device parameters change and need to be reported to the cloud, call the esp_rmaker_param_update_and_report API.

Firmware API: esp_rmaker_param_update_and_report()

Remote Control

A node must always stay subscribed to this topic, listening for updates triggered by the clients like phone apps or CLI. Any change in parameter values as per the updates must be reported back using the Subsequent Reporting.

MQTT

Topic node/<node_id>/params/remote
Operation: SUBSCRIBE
Data: {"<device-name">:{"<param-name>":<value>,...},...}

JSON Payload (Click to view)

{
    "Lightbulb": {
        "brightness": 100
    }
}

Trigger Logic

After the node powers on and connects to the cloud, it will automatically subscribe to this topic.

User-Node Mapping

Before the devices under a node can be monitored and controlled remotely, the node should be first mapped to a user so that only that specific user can have permission to access it. This happens in the background during the Wi-Fi provisioning stage. The node sends its node_id to the client and the client sends the user_id and secret_key to the node for mapping. Once connected to the RainMaker Cloud, the node sends the mapping request. Please check here for a detailed workflow.

MQTT

Topic node/<node_id>/user/mapping
Operation: PUBLISH
Data:
{"node_id":"<node_id>","user_id":"<user_id>","secret_key": "<secret_key>"}

JSON Payload (Click to view)

{
    "node_id": "112233AABBCC",
    "user_id": "02e95749-8d9d-4b8e-972c-43325ad27c63",
    "secret_key": "9140ef1d-72be-48d5-a6a1-455a27d77dee"
}

Trigger Logic

Triggered when the device is bound or unbound to the network. The user can also manually trigger it by calling the esp_rmaker_start_user_node_mapping API or if you would like user-node mapping to happen automatically during Wi-Fi setup, call
esp_rmaker_user_mapping_endpoint_register().

Firmware API: esp_rmaker_start_user_node_mapping() or
esp_rmaker_user_mapping_endpoint_register()

Time Series

There are two kinds of time series services that ESP RainMaker offer. One of such is the fully featured time series and the other is simple time series. Please head over to Features to know about the difference of these two services.

Time Series Data

Time series data is used for reporting values that require complete time stamped history and operations to find min, max, average, etc. in a given time window. Fully featured time series allows the reporting of multiple parameters and multiple records in a single MQTT message. it is used for historical record functionality. Compared to the below Simple Time Series Data, Time Series Data uses AWS's native functionality, supporting data statistics and calculations, but at a higher cost.

MQTT

Topic node/<node_id>/tsdata
Operation: PUBLISH

Data template (Click to view)

{
    "ts_data_version": "2021-09-13",
    "ts_data": [
        {
            "name": "<param_name>",
            "dt": "<int/float/bool/string>",
            "records": [
                {
                    "t": <epoch_timestamp>,
                    "v": <value>
                },
                {
                    more records...
                }
            ]
        },
        {
            more parameters...
        }
    ]
}

JSON Payload (Click to view)

{
    "ts_data_version": "2021-09-13",
    "ts_data": [
        {
            "name": "Temperature Sensor.Temperature",
            "dt": "float",
            "records": [
                {
                    "t": 1699468430,
                    "v": 26.5
                }
            ]
        }
    ]
}

Trigger Logic

When a device attribute is set to PROP_FLAG_TIME_SERIES, and the
esp_rmaker_param_update_and_report() API is called, time-series data for that parameter will be reported.

Simple Time Series Data

This allows reporting only a single parameter's single data point in an MQTT message. It is used for historical record functionality. Compared to Time Series Data, Simple Time Series Data only stores raw data and does not support cloud-side statistics and calculations, making it less expensive.

MQTT

Topic node/<node_id>/simple_tsdata
Operation: PUBLISH

Data template (Click to view)

{
    "name": "<param_name>",
    "dt": "<int/float/bool/string>",
    "t": <epoch_timestamp>,
    "v": <value>
}

JSON Payload (Click to view)

{
    "name": "Temperature Sensor.Temperature",
    "dt": "float",
    "t": 1704189730,
    "v": 25.5
}

trigger logic

When a device attribute is set to PROP_FLAG_SIMPLE_TIME_SERIES, and the esp_rmaker_param_update_and_report() is called, simple time-series data for that parameter will be reported.

note for time series data

Note that the parameter names in the examples are a combination of device name and parameter name. This is just for the phone apps to link to appropriate parameter in the node configuration. However, at a specification level, time series data is independent of other RainMaker concepts like device, parameters, etc. allowing to use this independently, if required.

Alert and Notify

This updates a parameter's value and notifies the cloud and connected clients or triggers an alert for a specific event or condition. After publishing, the mobile app will show a notification popup.

MQTT

Topic node/<node_id>/alert
Operation: PUBLISH

trigger logic

When the node is required to send an alert or notify, it calls either call
esp_rmaker_param_update_and_notify or esp_rmaker_raise_alert.

Firmware API: esp_rmaker_param_update_and_notify() or esp_rmaker_raise_alert()

Command Response

Command-Response can be used when a node is offline. The cloud can configure the device via the app, and when the node comes online, it will receive the pushed configuration. The cloud can have a command-response configuration which can be added and modified via the Swagger API at Command Response Requests, and it will push the configuration to the node after the node powers on.

To Node

MQTT

Topic node/<node_id>/to-node
Operation: SUBSCRIBE

Trigger Logic

After enabling the command-response feature, the device will automatically subscribe to this topic after power on and established cloud connection.

From Node

Used for responding to commands from the cloud.

MQTT

Topic node/<node_id>/from-node
Operation: PUBLISH

Trigger Logic

After receiving the Command from the platform, the node will process it and internally respond via the topic stated above.

OTA

OTA uses MQTT topics to manage the OTA process, including reporting OTA status, fetching OTA tasks, and listening for OTA updates.

Report OTA Task Status

Reports the status during the OTA process.

MQTT

Topic node/<node_id>/otastatus
Operation: PUBLISH

Trigger Logic

During the OTA process, the device will report the current OTA status. The user can also call the esp_rmaker_ota_report_status API to report the status.

Firmware API: esp_rmaker_ota_report_status()

Get OTA Task

The device queries OTA tasks. After the platform forces an upgrade or the user confirms the upgrade, if the device is offline, it can fetch the OTA tasks and perform the upgrade when it comes online.

MQTT

Topic node/<node_id>/otafetch
Operation: PUBLISH

Trigger Logic

After the device powers on and connects to the cloud, it will actively fetch the OTA tasks. By default, it fetches the OTA tasks every 24 hours. The user can also manually trigger it by calling the esp_rmaker_ota_fetch or esp_rmaker_ota_fetch_with_delay API.

Firmware API: esp_rmaker_ota_fetch() or esp_rmaker_ota_fetch_with_delay()

Listen OTA Task

When a forced OTA or user-approved OTA is created on the dashboard, and the user triggers the upgrade via the app, the cloud will push the OTA URL, firmware version, firmware size, and other information to this topic.

MQTT

Topic node/<node_id>/otaurl
Operation: SUBSCRIBE

Trigger Logic

After the device connects to the cloud, it will automatically subscribe to the otaurl topic to listen for OTA tasks issued by the cloud.

Documentation Hub

How does a Node ensure secured communication with the Cloud?​

TLS​

MQTT Policies​

MQTT Topics​

Node Configuration​

Node Parameters​

Initial Reporting​

Subsequent Reporting​

Remote Control​

User-Node Mapping​

Time Series​

Time Series Data​

Simple Time Series Data​

Alert and Notify​

Command Response​

To Node​

From Node​

OTA​

Report OTA Task Status​

Get OTA Task​

Listen OTA Task​

How does a Node ensure secured communication with the Cloud?

TLS

MQTT Policies

MQTT Topics

Node Configuration

Node Parameters

Initial Reporting

Subsequent Reporting

Remote Control

User-Node Mapping

Time Series

Time Series Data

Simple Time Series Data

Alert and Notify

Command Response

To Node

From Node

OTA

Report OTA Task Status

Get OTA Task

Listen OTA Task