Skip to content

niswitze/Hot-Warm-Cold-On-Azure-IoT

BranchesTags

Repository files navigation

page_type languages products name urlFragment
sample
Python
C#
C
azure-active-directory
azure-iot-hub
azure-data-explorer
azure-app-service
azure-function-app
azure-signalr
Create Hot, Warm, and Cold data paths using Azure IoT Hub, App Service, Functions, SignalR, and Data Explorer
Hot-Warm-Cold-On-Azure-IoT

Create Hot, Warm, and Cold data paths using Azure IoT Hub, App Service, Functions, SignalR, and Data Explorer

  1. Overview
  2. Scenario
  3. Contents
  4. Prerequisites
  5. Setup
  6. About the code
  7. Community Help and Support
  8. Contributing

Overview

This sample demonstrates how to create Hot, Warm, and Cold data paths using Azure IoT Hub or IoT Edge (version 1.1), App Service, Functions, SignalR, and Data Explorer.

To understand more about why this sample was created and what it aims to accomplish, please visit its official article on Tech Community.

Scenario


  1. The ESP8226 connects either directly to an Azure IoT Hub or IoT Edge enabled gateway and submits temperature data produced by a DHT11 sensor.

ℹ️ (NOTE) For this sample, these hardware specifications are not required and any other device or simulator can be used. Just ensure the device or simulator in use has the capability to send temperature data to Azure IoT Hub in the same format as detailed in the IoTHub-TempSensor device code.

  1. This temperature data is then ingested from Azure IoT Hub to an Azure Function and Data explorer.
  2. An instance of Azure Data Explorer ingests all temperature data for long-term storage
  3. An Azure Function broadcasts all temperature data to an Azure SignalR instance
    • This Azure Function also sends out a text alert if the temperature data starts to fall below a certain threshold
  4. The Azure SignalR instance broadcasts temperature data to all clients listening on a WebSocket based connection
  5. An App Service hosts an ASP.NET MVC, .NET 5 based, web app that displays the latest temperature data record per device from the last 24 hours from Azure Data Explorer
  6. Finally, the web app creates a WebSocket connection to the Azure SignalR instance to receive temperature data in real time
    • If set up correctly, the web app will look like the following:

Contents

File/folder Description
IoTFunctionApp/ The Azure Function App that submits all temperature data to SignalR & alerts if the temperature data is below a certain threshold.
IoTHub-TempSensor/ The ESP8226 device code for connecting and sending temperature data to either Azure IoT Hub or IoT Edge.
IoTWebApp/ The web app for the Azure App Service that displays the last 24 hours worth of temperature data store in Data Explorer and all temperature data sent through the SignalR instance.
CODE_OF_CONDUCT.md Guidelines for contributing to the sample.
CONTRIBUTING.md Guidelines for contributing to the sample.
LICENSE The license for the sample.

Prerequisites

The below prerequisites are required for local development:

The below prerequisites are only needed for production use in Azure:

Setup

Step 1: Clone or download this repository

From your shell or command line:

    git clone https://github.com/niswitze/Hot-Warm-Cold-On-Azure-IoT.git

or download and extract the repository .zip file.

⚠️ To avoid path length limitations on Windows, we recommend cloning into a directory near the root of your drive.

Step 2: Configure Azure Resources

  • Azure IoT Hub

    1. After creating an IoT Hub instance, create two consumer groups
      • A seperate consumer group must be created for the Azure Function and Azure Data Explorer services

  • Azure Data Explorer

    1. After creating an Azure Data Explorer instance, create a data connection to the Azure IoT Hub instance
      • The following public documentation is a walk through of how to create an Azure IoT Hub data connection in Data Explorer
      • For this sample, the following KQL statements should be used for creating the table and data mapping
      .create table TemperatureData (DeviceId: string, Temperature: real, TelemetryType: string, EnqueuedTime:datetime)

.create table TemperatureData ingestion json mapping 'TemperatureDataMapping' 
'[{"column":"DeviceId","path":"$.deviceId","datatype":"string"},{"column":"Temperature","path":"$.temperature","datatype":"real"},{"column":"TelemetryType","path":"$.telemetryType","datatype":"string"},{"column":"EnqueuedTime","path":"$.iothub-enqueuedtime","datatype":"datetime"}]'
      • When creating the data connection in Azure Data Explorer, ensure the property iothub-enqueuedtime has been selected from the Event system properties dropdown
    2. For local development, an App Registration needs to be created in Azure Active Directory and given access to the Data Explorer instance
      • The following public documentation is a walk through of how to create an App Registration in Azure Active Directory and assign it access to the Data Explorer instance

  • Azure SignalR

    1. For local development, there is no configuration that needs to take place after creating the Azure SignalR instance. However, during creation ensure the ServiceMode setting for this resource is set to Serverless

Step 3: Configure Applications

  • IoTFunctionApp

    1. Open Visual Studio Code and install the Python and Azure Functions extensions
    2. Create a virtual environment in the IoTFunctionApp directory using Python's venv module
    cd IoTFunctionApp
python3 -m venv venv # only required if you don't have a venv already
    1. Rename the file local.development.settings.json to local.settings.json
    2. Update the following settings in local.settings.json
      Setting Name Value
      AzureWebJobsStorage Connection string to an Azure Storage account. For local development, please use either the Azure Storage Emulator or Azurite and set this value to UseDevelopmentStorage=true
      AzureSignalRHubName Hub name of the Azure SignaR instance
      AzureSignalRConnectionString Connection string of the Azure SignaR instance
      AzureIoTHubName Hub name of the Azure IoTHub instance
      AzureIoTHubConnectionString Connection string of the Azure IoTHub instance
      ConsumerGroup Name of the dedicated consumer group in the Azure IoTHub instance
      TwilioPhoneNumber Phone number created by twilio
      TwilioAccountSid Account SID for Twilio account
      TwilioAuthToken Authentication token for Twilio account
      OnCallPhoneNumber Phone number to send alerts to - Format should be -> (AREACODE) 888 8888
      SendEventUri URI of the HttpTriggerDurable function. Only FUNCTIONAPPNAME and FUNCTIONAPPKEY should be updated with the name of the Azure Function App instance and its primary authorization key. These values can also be updated to their localHost versions for local development.
      StartOrchestrationUri URI of the HttpTriggerDurable function. Only FUNCTIONAPPNAME and FUNCTIONAPPKEY should be updated with the name of the Azure Function App instance and its primary authorization key. These values can also be updated to their localHost versions for local development.
      OrchestratioStatusnUri URI of the event (orchestration) status API, hosted by the Azure Durable Functions framework. Only FUNCTIONAPPNAME, FUNCTIONAPPKEY, and durabletask_extension_key should be updated with the name of the Azure Function App instance, its primary authorization key, and its durabletask_extension key. These values can also be updated to their localHost versions for local development.

  • IoTWebApp

    1. Open the IoTWebApp.sln file using Visual Studio
    2. Run the rebuild command to install dependencies and ensure a successful build
    3. Update the following settings in appsettings.Development.json
      Setting Name Value
      ADXClusterName Name of the ADX cluster
      ADXDatabseName Name of the ADX database
      ADXClusterRegion Name of the region where the ADX cluster instance exists
      ApplicationClientId Client ID of the App Registration that has direct access to the ADX cluster
      ApplicationSecret Secret value for App Registration that has direct access to the ADX cluster
      ApplicationAuthority AAD Tenant ID where this App Registration exists

    ⚠️ Settings ApplicationClientId, ApplicationSecret, and ApplicationAuthority are only required for local development. In production, a Managed Identity should be used instead of an App Registration.

  • IoTHub-TempSensor

    1. Open the IoTHub-TempSensor.ino file using the Arduino IDE
    2. Ensure the ESP8266 platform has been installed
    3. Configure the DHT11 sensor on the ESP8266's GPIO pin 14
      • This value can be updated by changing the value of the TEMP_PIN variable declared in config.h
    4. Upload the this project to your ESP8266 using an Upload Speed of 115200
      • This value can be updated by changing the value of the DATA_RATE variable declared in config.h
    5. Once the upload has been completed, open the Serial Monitor using 115200 as the BAUD rate
    6. In the serial monitor, a prompt will be appear to enter the connection string for an Azure IoT Hub instance and WIFI credentials. These values will then be saved in memory using EEPROM.
      • By default, this prompt will dissapear after 15 seconds. If the prompt does not show or is initially missed, just reset the ESP8226 with the Serial Monitor opened.
    7. For configuring this sample to run with IoT Edge or any additional guidance, please review the commented intro in file IoTHub-TempSensor.ino.

Step 4 (Optional): Deploy Applications to Azure

  • Azure Key Vault

    1. For production use, create an Azure Key Vault instance and move all configuration settings from the IoTFunctionApp and IoTWebApp into it as secrets
      • For the IoTWebApp, configuration settings ApplicationClientId, ApplicationSecret, and ApplicationAuthority do not need to be moved to the Azure Key Vault instance

  • Azure Function App

    1. After creating an Azure Function App instance, publish the IoTFunctionApp code to it from VS Code
    2. Enable the Azure Function App's system managed identity
    3. Assign the Azure Function App's system managed identity to the Azure Key Vault instance using an access policy
      • The minimum level of permissions for this policy should only be Get for the Secrets resource
    4. Configure the Azure Function App's app settings to use Key Vault references
      • An entry for each setting in local.settings.json will need to be added to the Azure Function App's app settings with the same name, but the setting's value will need to be a Key Vault reference

  • Azure App Service

    1. After creating an Azure App Service, publish the IoTWebApp code to it from Visual Studio
    2. Enable the Azure App Service's system managed identity
    3. Assign the Azure App Service's system managed identity to the Azure Key Vault instance using an access policy
      • The minimum level of permissions for this policy should only be Get for the Secrets resource
    4. Configure the Azure App Service's app settings to use Key Vault references
      • An entry for each setting in appsettings.Development.json will need to be added to Azure App Service's app settings with the same name, but the setting's value will need to be a Key Vault reference
    5. Assign the Azure App Service's system managed identity access to the Data Explorer instance
      • The process to assign a system managed identity access at the database level for an Azure Data Explorer instance is the same as the following public documentation. The only difference is, instead of using an App Registration's ApplicationId, use the system managed identity's ID.

  • Azure SignalR

    1. For production use, restrict the CORS setting of the Azure SignalR instance to only allow requests from an origin of the App Service and Function App base URIs.

We'd love your feedback!

ℹ️ Did the sample not work for you as expected? Then please reach out to us using the GitHub Issues page.

About the Code

This sample demonstrates how to create Hot, Warm, and Cold data paths using Azure IoT Hub or IoT Edge (version 1.1), App Service, Functions, SignalR, and Data Explorer. Once the ESP8266 has connected to Azure IoT Hub, using the code from the IoTHub-TempSensor application, it will start sending temperature data reported from the DHT11 sensor. From Azure IoT Hub, the data will be ingested, using two seperate consumer groups, by Azure Data Explorer and an Azure Function. Azure Data Explorer will ingest all temperature data sent to Azure IoT Hub and serve as the Cold, or Hot if using a cache policy, data path. The Azure Function, using the code from the IoTFunctionApp application, will ingest all temperature data sent to Azure IoT Hub and output the data to Azure SignalR. Azure SignalR will in turn output this temperature data to all clients listening on a WebSocket based connection, which allows for data to be reported in real time. Also, the Azure Function will send a text message based alert, using Twilio, if the temperature data falls below a certain threshold. The Azure Function accomplishes this alerting by invoking a Durable Function, named HttpTriggerDurable, which will send a text message every 5 minutes until the alert has been acknowledged. On startup, the web application hosted on Azure App Service, using the code from the IoTWebApp application, will establish a WebSocket based connection to Azure SignalR. Then, the latest temperature data record per device from the last 24 hours, from Azure Data Explorer, will be displayed per device in the web application and serve as the Warm data path. All of these device records displayed in the web application will then be updated in real time using the temperature data received from the Azure SignalR connection.

Community Help and Support

If you find a bug in the sample, raise the issue on GitHub Issues.

To provide feedback on or suggest features for Azure IoT Hub or IoT Edge, visit User Voice page.

Contributing

If you'd like to contribute to this sample, see CONTRIBUTING.MD.

This project has adopted the Microsoft Open Source Code of Conduct. For more information, see the Code of Conduct FAQ or contact opencode@microsoft.com with any additional questions or comments.

About

No description, website, or topics provided.

Resources

Readme

License

MIT license

Code of conduct

Code of conduct
Activity

Stars

1 star

Watchers

1 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages