Skip to main content

Firmware Development Tips

tip

For more information about firmware development, you can refer to this page.

Provisioning（WiFi Provisioning）​

Provisioning is the essential first step in deploying IoT devices, where devices are configured with network credentials and authenticated with cloud services. ESP RainMaker uses ESP-IDF's network provisioning component to simplify this process via Android/iOS phone apps using either BLE or SoftAP transport. An essential step of User-Node mapping is also performed during the provisioning.

Provisioning Transports: BLE/SoftAP​

MethodCommunicationDevice BroadcastUser InteractionProsCons
BLEBLE GATTBLE advertisement, name like PROV_XXXXPhone scans device → establishes BLE connection → Establishes Secure Session → sends Wi-Fi credentials and other infoFast, smooth UX, better feedback, phone stays connected to InternetRequires BLE support, potential compatibility issues
SoftAPHTTP over Wi-FiDevice creates hotspot, SSID like PROV_XXXXPhone connects to hotspot → Establishes Secure Session → sends Wi-Fi credentials and other infoSimple, high compatibilityRequires manual Wi-Fi switch, may disconnect during the process, may lose Internet due to network switch

Provisioning Process (BLE Example)​

  1. Device starts BLE advertising, including:
    • Device Name: PROV_xxxxxx
    • Service UUID: 0xFF50 (Espressif custom provisioning service)
  2. Phone app scans and discovers the device
  3. App establishes GATT communication
  4. App establishes secure session (with optional Proof of Possession pin)
  5. App exchanges data for user-node mapping
  6. App sends Wi-Fi SSID and password
  7. Device attempts connection and reports status to the cloud
  8. App checks Wi-Fi connection status locally and then verifies device status via cloud
info

SoftAP follows a similar flow, but the communication is HTTP instead of BLE.

Provisioning Process (Scanning QR code)​

In production, some devices includes a QR code containing basic provisioning metadata (name, pop, transport). When the user scans this code in the app:

  1. The app parses the QR content which includes transport method(BLE/SoftAP).
  2. Based on transport field, the app selects the corresponding provisioning flow.
  3. This avoids manual device discovery and improves provisioning consistency.

The QR code acts as a guidance mechanism, not a transport. Actual data transfer still happens via BLE or SoftAP. This method also simplifies the user experience by removing the need to manually search or select the device.

Example of QR Code Content​

A RainMaker QR code typically contains JSON or a Base64-encoded URL like:

{
  "ver": "v1",
  "name": "PROV_AB12CD",
  "pop": "abcd1234",
  "transport": "ble"
}

Or:

https://rainmaker.espressif.com/qrcode.html?data={"ver":"v1","name":"PROV_AB12CD","pop":"abcd1234","transport":"ble"}

After scanning the QR code, the app can:

  • Get the device name (for BLE scan or Wi-Fi SSID)
  • Get the POP (if needed)
  • Know the provisioning method (BLE or SoftAP)

There are currently two provisioning interface sets:

  • network_provisioning: a component under the idf-extra-components project. (Recommended)
  • wifi_provisioning: a component under the ESP-IDF. (Legacy)

When the configuration option CONFIG_ESP_RMAKER_USING_NETWORK_PROV is enabled (enabled by default), the system will use the network_provisioning interface. Otherwise, it falls back to using wifi_provisioning.

Key Differences​

  • Functionality:

    The network_provisioning component supports not only Wi-Fi Provisioning but also Thread Provisioning over BLE Transport.

  • API Naming:

    • network_provisioning uses the prefix network_.
    • wifi_provisioning uses the prefix wifi_.
app_network Interfaces​

The underlying network_provisioning component abstracted out via the app_network interface. It provides a more user-friendly interface for provisioning.

FunctionAPI
Initialize provisioningapp_network_init
Start provisioningapp_network_start() (with pop type)
Set custom device type and subtypeapp_network_set_custom_mfg_data()
Set custom POPapp_network_set_custom_pop()

PoP of Provisioning​

In the Wi-Fi provisioning scenario, POP (Proof of Possession) is a crucial security mechanism that serves the following purposes:

  1. Prevent unauthorized users from provisioning/binding the device.
    • If a device is broadcasting BLE openly, anyone can discover it; without POP, others could directly take control of your device.
  2. Ensure the person provisioning is the “real owner”.
    • POP is like a door key—only the person who actually possesses the POP (e.g., obtained via QR code scan) can successfully provision the device.
  3. Authenticate during the provisioning process.
    • After the mobile app establishes a BLE connection with the device, it will provide the POP. The device will then verify if the POP matches. Only if it matches will the provisioning process continue.

When provisioning via the app, the POP value must match the one set on the device in order to establish a valid provisioning channel. There are four types of POP values:

  • POP_TYPE_MAC : Uses the last 3 bytes of the device’s Wi-Fi MAC address, converted to 6 hex characters.
  • POP_TYPE_RANDOM: Uses the first 3 bytes of a random value (random) stored in the fctry partition, which also holds the Rainmaker certificate. When generating the certificate using the admin-cli tool, you can optionally generate a QR code as well. In this case, the POP in the QR code will match the one read from the device.
  • POP_TYPE_NONE: No POP is used. This method applies to scenarios where QR code scanning is not required, and the app can directly proceed with provisioning upon discovering the device via Bluetooth.
  • POP_TYPE_CUSTOM: A custom POP value is used, configured using app_network_set_custom_pop().

Provisioning Advertisement
​

Change the provisioning name prefix​

The provisioning name prefix is configured via the option CONFIG_APP_NETWORK_PROV_NAME_PREFIX, which defaults to PROV_. Users can modify it as needed. The suffix takes either the last three characters of the device's Wi-Fi MAC address or the last three characters of the random field in the RainMaker certificate.

Add Customized Information to BLE Broadcast​

The wifi_provisioning component used in RainMaker supports modifying the Bluetooth manufacturer data. You can refer to the function app_network_set_custom_mfg_data() to set the Bluetooth advertising data. This API just allows modifying the device type and subtype. If other data is to be modified, please check the implementation of app_network_set_custom_mfg_data() and the spec for it as below.

Developers can add product information, type, manufacturer, and other details into the manufacturer data. When the app scans via Bluetooth, it can retrieve this information and display the corresponding product icon accordingly.

note

Field Fixed head + App id + version + customer id + Device type code + Device subtype code + Device extra code + Reserve 0 + Reserve 1 Length 2 3 1 2 2 1 1 1 1 def value 0xe5 0x02 'N' 'o' 'v' 0xFF 0xFF

esp_err_t app_network_set_custom_mfg_data(uint16_t device_type, uint8_t device_subtype)
{
    int8_t mfg_data[] = {MFG_DATA_HEADER, MGF_DATA_APP_ID, MFG_DATA_VERSION, MFG_DATA_CUSTOMER_ID};//{0xe5, 0x02, 'N', 'o', 'v', 'a', 0x00, 0x01}
    size_t mfg_data_len = sizeof(mfg_data) + 4; // 4 bytes of device type, subtype, and extra-code
    custom_mfg_data = (uint8_t *)MEM_ALLOC_EXTRAM(mfg_data_len);
    if (custom_mfg_data == NULL) {
        ESP_LOGE(TAG, "Failed to allocate memory to custom mfg data");
        return ESP_ERR_NO_MEM;
    }
    memcpy(custom_mfg_data, mfg_data, sizeof(mfg_data));
    custom_mfg_data[8] = 0xff & (device_type >> 8);
    custom_mfg_data[9] = 0xff & device_type;
    custom_mfg_data[10] = device_subtype;
    custom_mfg_data[11] = 0;
    custom_mfg_data_len = mfg_data_len;
    ESP_LOG_BUFFER_HEXDUMP("tag", custom_mfg_data, mfg_data_len, 3);
    return ESP_OK;
}

Other configurations and features​

Transport Selection​

CONFIG_APP_NETWORK_PROV_TRANSPORT_BLE is used to configure the BLE transport method for provisioning. CONFIG_APP_NETWORK_PROV_TRANSPORT_SOFTAP is used to configure the SoftAP transport method for provisioning.

Provisioning Timeout​

CONFIG_APP_NETWORK_PROV_TIMEOUT_PERIOD can be used to configure the timeout (in minutes) after which the provisioning will auto stop. A reboot will be required to restart provisioning. It is always recommended to set this to some non zero value, especially if you are not using PoP. Set to 0 if you do not want provisioning to auto stop.

Brute Force Protection​

CONFIG_APP_NETWORK_PROV_MAX_POP_MISMATCH is used to configure the maximum number of POP mismatches allowed before the provisioning process is automatically stopped.

Reset on Failure​

CONFIG_APP_NETWORK_RESET_PROV_ON_FAILURE can be used to configure the reset on failure feature for provisioning. If enabled, the device will be reset if the provisioning fails. CONFIG_APP_NETWORK_PROV_MAX_RETRY_CNT can be used to configure the maximum number of retries allowed before the provisioning process is automatically stopped and reset

Number of SSIDs​

By default, Wi-Fi/Network Provisioning returns 16 SSIDs. This can be configured using
CONFIG_WIFI_PROV_SCAN_MAX_ENTRIES/CONFIG_NETWORK_PROV_SCAN_MAX_ENTRIES.

OTA​

Rollback​

When OTA fails or the new version has an exception, rollback can return the device to the previous version of the firmware. The OTA version rollback function uses the APP rollback function in IDF. In ESP-RainMaker, you only need to enable CONFIG_BOOTLOADER_APP_ROLLBACK_ENABLE. This is enabled by default in ESP RainMaker examples. AFter a reboot, the firmware will wait for CONFIG_ESP_RMAKER_OTA_ROLLBACK_WAIT_PERIOD seconds (90 by default) for MQTT connection and rollback if unable to connect

OTA Fetch​

By default, the firmware will try an OTA fetch once on bootup (configured by CONFIG_ESP_RMAKER_OTA_AUTOFETCH). Optionally, if CONFIG_ESP_RMAKER_OTA_AUTOFETCH_PERIOD is set, the firmware will try an OTA fetch every CONFIG_ESP_RMAKER_OTA_AUTOFETCH_PERIOD hours.

OTA Retries​

CONFIG_ESP_RMAKER_OTA_MAX_RETRIES can be used to configure the maximum number of OTA retry attempts. The device will retry OTA download on failure up to this number of times before giving up. The default value is 3, and the range is 1-10.

CONFIG_ESP_RMAKER_OTA_RETRY_DELAY_MINUTES can be used to configure the delay (in minutes) before re-fetching OTA details after all retry attempts fail. This is specifically for OTA using topics. The default value is 5 minutes, and the range is 1-60 minutes.

Compress OTA​

1. Add include(gen_compressed_ota) in CMakeLists.txt​

Add the following line to your project's CMakeLists.txt:

# The following lines of boilerplate have to be in your project's CMakeLists
# in this exact order for cmake to work correctly
cmake_minimum_required(VERSION 3.16)

# (Not part of the boilerplate)
# This example uses an extra component for common functions such as Wi-Fi and Ethernet connection.
set(EXTRA_COMPONENT_DIRS $ENV{IDF_PATH}/examples/common_components/protocol_examples_common)

include($ENV{IDF_PATH}/tools/cmake/project.cmake)
project(simple_ota)
include(gen_compressed_ota)

2. Add bootloader_components​

Copy the bootloader_components directory from:

https://github.com/espressif/esp-iot-solution/tree/master/examples/ota/simple_ota_example

to your project directory, then delete the idf_component.yml file in this directory.

3. Modify Partition Table​

Compressed OTA requires smaller partition size for the secondary OTA partition. Create a normal ota_0 partition and a compressed firmware ota_1 partition as follows:

# Name,   Type, SubType, Offset,   Size, Flags
esp_secure_cert,  0x3F, ,0xd000,    0x2000, encrypted
nvs,      data, nvs,     0x10000,   0xC000,
nvs_keys, data, nvs_keys,,          0x1000, encrypted
otadata,  data, ota,     ,          0x2000
phy_init, data, phy,     ,          0x1000,
ota_0,    app,  ota_0,   0x20000,   0x120000,
ota_1,    app,  ota_1,   0x140000,  0xBA000,
fctry,    data, nvs,     0x1FA000,  0x6000,

The ota_1 partition only needs to be large enough to hold the compressed firmware.

4. Modify idf_component.yml​

Update the idf_component.yml file in your project's main directory to include bootloader_support_plus:

dependencies:
  bootloader_support_plus:
    version: "0.*"

5. Enable Compressed OTA Feature​

Enable the following configurations in your default configuration:

CONFIG_BOOTLOADER_COMPRESSED_ENABLED=y
CONFIG_BOOTLOADER_DECOMPRESSOR_XZ=y

MQTT Budgeting​

ESP RainMaker implements MQTT budgeting to control the number of MQTT messages sent by a node. This helps prevent excessive cloud communication and manage costs effectively. This is generally expected to be disabed for private deployments assuming that the data reporting rates are already thought through and quality of service is paramount. However, if you still suspect that the firmware code can misbehave and send too many messages, you can enable this feature.

Configuration Options​

Basic Settings​

  • Enable/disable budgeting via CONFIG_ESP_RMAKER_MQTT_ENABLE_BUDGETING (default: enabled)
  • Default budget: CONFIG_ESP_RMAKER_MQTT_DEFAULT_BUDGET (default: 100)
    • Range: 64 to max budget
    • Budget decreases with each MQTT message sent
    • Messages are dropped when budget reaches zero

Budget Limits​

  • Maximum budget: CONFIG_ESP_RMAKER_MQTT_MAX_BUDGET (default: 1024)
    • Range: 64 to 2048
    • No additional budget allocated once maximum is reached

Budget Revival​

  • Revival period: CONFIG_ESP_RMAKER_MQTT_BUDGET_REVIVE_PERIOD (default: 5 seconds)

    • Range: 5 to 600 seconds
    • Period after which budget is increased

  • Revival count: CONFIG_ESP_RMAKER_MQTT_BUDGET_REVIVE_COUNT (default: 1)

    • Range: 1 to 16
    • Number of budget units added each revival period
How It Works​
  1. Node starts with the default budget (100 per 5 seconds)
  2. Each MQTT message sent decreases the budget by 1
  3. Budget automatically increases by revival count every revival period
  4. Messages are dropped if budget reaches 0
  5. Budget never exceeds maximum budget limit

Best Practices​

  • Configure budget based on your application's messaging frequency
  • Set revival period considering your real-time requirements
  • Adjust revival count based on expected message burst patterns
  • Monitor dropped messages and adjust settings if needed
note

MQTT budgeting is crucial for preventing message floods and managing cloud costs. Configure these parameters carefully based on your use case.

On this page