IOT Edge for Linux on Windows (eflow) - nested VM deployement - proxy settings - corporate firewall #7179
Comments
|
Output when I executed: sudo iotedge check Configuration checks (aziot-identity-service)√ keyd configuration is well-formed - OK Connectivity checks (aziot-identity-service)× host can connect to and perform TLS handshake with iothub AMQP port - Error Configuration checks√ aziot-edged configuration is well-formed - OK Connectivity checks× container on the default network can connect to upstream AMQP port - Error |
|
Your device is unable to connect to IoT Hub, likely due to an error in proxy configuration. On your device, could you check the output of the command: curl -v -x http://123.45.67.89:8080/ \
https://testiothub114.azure-devices.net/devices/myIOThub.azure-devices.net/devices/myEdgeDevice/modules?api-version=2017-11-08-previewYou should get an HTTP 401 Unauthorized response, but it should be able to connect. |
|
Thanks for looking into the matter. Execution result of curl -v -x proxy iot-edge-device is:
|
|
That's not the expected output for a correctly-configured proxy. You'll have to debug your proxy configuration. Check that connections to your IoT Hub are allowed and that TLS certificates are configured correctly if this proxy is intercepting TLS traffic. |
|
Any updates @usman-bin-imran |
|
Hi @gordonwang0 @vadim-kovalyov @jlian Thankyou for the attention to the matter. I am experiencing problems while debugging. Can you give a pathway that I can follow, and see if TLS certificates are properly configured? As, the outbound urls and ports for IoT Hub are already allowed. Looking forward, Usman Bin Imran |
|
@usman-bin-imran, the only thing I see and can suggest to look into is to investigate the TLS error - Make sure that your system supports minimum required version of TLS (1.2) and you have proper root certs installed https://learn.microsoft.com/en-us/azure/iot-hub/iot-hub-tls-support |
|
@usman-bin-imran any updates? |
|
@jlian, we forwarded the request to our organisation's networking department to reconfigure proxy settings properly. Thanks for looking into the matter! Kr, |
|
Hi @gordonwang0 @vadim-kovalyov @jlian Thanks for staying in loop. The proxy has ben set successfully by our networking department. Now we are able to send messages to IoT Hub using python code. While trying to re-communicate IoT Edge for Linux on Windows (EFLOW) with IoT Hub through proxy, we are still encountering errors below (after python code): ------------Python Code Starts Here------------ ------------sudo iotedge check Output Starts Here------------ Configuration checks (aziot-identity-service)√ keyd configuration is well-formed - OK Connectivity checks (aziot-identity-service)× host can connect to and perform TLS handshake with iothub AMQP port - Error Configuration checks√ aziot-edged configuration is well-formed - OK Connectivity checks× container on the default network can connect to upstream AMQP port - Error ------------cURL check suggested by @gordonwang0 ------------ I also re-provisioned the device and restarted it several times. Really appreciate your attention to the matter! Looking forward, Usman Bin Imran |
|
@Azure/iotedge-eflow any ideas? @vadim-kovalyov do you think this might be due to Moby proxy settings misconfiguration, given the check result
|
|
Hey, I think at least two warnings need to be addressed first:
As you can see, Edge Agent and Moby settings are incorrect (empty string), that's why EA can't connect and/or pull any container. |
|
Hi @gordonwang0 @vadim-kovalyov @jlian @PatAltimore Thanks for staying in loop. I am able to send messages to IoT Hub through proxy server using Custom Python Code. I am curious that IoT Edge for Linux on Windows (EFLOW) is not working properly in nested VM situation when outbounds need to be sent through proxy. In response to @vadim-kovalyov previous response, I rechecked the applied configurations for Moby Daemon. Please review my findings below: On the Microsoft Document: https://learn.microsoft.com/en-us/azure/iot-edge/how-to-configure-proxy-support?view=iotedge-1.4#moby-daemon I selected Configure docker daemon on Linux. This took me to the following link: https://docs.docker.com/config/daemon/systemd/#httphttps-proxy In the Docker Daemon Configuration part, it suggested that: "If you're behind an HTTP or HTTPS proxy server, for example in corporate settings, the daemon proxy configurations must be specified in the systemd service file, not in the daemon.json file or using environment variables." I followed the following steps because, I am behind proxy server (corporate settings): This is how my "/etc/systemd/system/docker.service.d/http-proxy.conf" file looks like: Finally, it suggested to run following commands: Output: I think that "Moby Daemon" part on the microsoft document is not addressed properly for IoT Edge for Linux on Windows (nested VM environment). EFLOW is not able to read set configurations from "/etc/systemd/system/docker.service.d/http-proxy.conf". Looking forward to hearing from you! Kind regards, |
|
Hi @gordonwang0 @vadim-kovalyov @jlian @PatAltimore @ksaye The latest sudo iotedge check using proxy gives following results: root@WINDOWSVM-EFLOW [ ~ ]# sudo iotedge check --proxy-uri http://123.45.67.89:8080
|
@Azure/iotedge-eflow and @jagadishmurugan can you help here? Looks like @usman-bin-imran followed the as the docs instructed, which is to |
|
@usman-bin-imran ,
|
|
@usman-bin-imran can you follow up on Jagadish's question above? |
|
@usman-bin-imran Have you had a chance to look at the question above? |
|
Hey @konichi3, Hope this message finds you in good health. Given the limitations of the initial documentation provided by Microsoft for IoT Edge for Linux on Windows (EFLOW), as outlined in the document (https://learn.microsoft.com/en-us/azure/iot-edge/how-to-configure-proxy-support?view=iotedge-1.4), which lacks comprehensive and detailed configuration steps, we have made the decision to implement the transmission of telemetry data from the OPC Server to the IoT Hub programmatically. The solution steps in initial document were incomplete for nested-VM setup. We anticipate @microsoft & @Azure to release an official document detailing steps, addressing the issues highlighted in the aforementioned thread. Gratitude to @jagadishmurugan, @vadim-kovalyov, @jlian, and @gordonwang0 for staying in the loop! A special acknowledgment to @TerryWarwick for engaging with the core team! Kind regards, |
|
Thank you for your feedback, @usman-bin-imran . We appreciate your input and will take it into account as we strive to improve our documentation. Our goal is to make it easier to follow. Since they are no more issues, I'll resolve this thread. Please feel free to re-open if you encounter any issues. |
Greetings Eflow team,
My team is working on achieving following:
Helping Microsoft Article:
https://learn.microsoft.com/en-us/azure/iot-edge/how-to-configure-proxy-support?view=iotedge-1.4
The planned architecture is:
Unfortunately, we are experiencing issues while achieving above. I am attaching logs.
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:26Z [INFO] - Starting Azure IoT Edge Daemon
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:26Z [INFO] - Version - 1.4.20
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:26Z [INFO] - Obtaining Edge device provisioning data...
Dec 26 18:19:26 WINDOWSVM-EFLOW systemd[1]: Started Azure IoT Identity Service.
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Starting service...
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Version - dev build
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Detected HTTPS proxy server http://123.45.67.89:8080/
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Provisioning starting. Reason: Startup
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Updated device info for myEdgeDevice.
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Provisioning complete.
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [INFO] - Identity reconciliation started. Reason: Startup
Dec 26 18:19:26 WINDOWSVM-EFLOW systemd[1]: Started Azure IoT Keys Service.
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - Starting service...
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - Version - dev build
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - Loaded libaziot-keys with version 0x02010000
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - Starting server...
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - --> 200 {"content-type": "application/json"}
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "402"}
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:26Z [INFO] - --> 200 {"content-type": "application/json"}
Dec 26 18:19:26 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:26Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: unexpected EOF
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: unexpected EOF
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [WARN] - Network not available for Identity reconciliation. Using offline backup from last run.
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [INFO] - Starting server...
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [INFO] - <-- POST /identities/device?api-version=2020-09-01 {"content-type": "application/json", "host": "identityd.sock", "content-length": "16"}
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"}
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"}
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Device is myEdgeDevice on myIOThub.azure-devices.net
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Initializing module runtime...
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Using runtime network id azure-iot-edge
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Successfully initialized module runtime
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - <-- POST /keypair?api-version=2020-09-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "61"}
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"}
Dec 26 18:19:30 WINDOWSVM-EFLOW systemd[1]: Started Azure IoT Certificates Service.
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - Starting service...
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - Version - dev build
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - Starting server...
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - <-- GET /certificates/aziot-edged-ca?api-version=2020-09-01 {"host": "certd.sock"}
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"}
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Using existing Edge CA certificate
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - <-- GET /certificates/aziot-edged-ca?api-version=2020-09-01 {"host": "certd.sock"}
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-certd[14179]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"}
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Certificate aziot-edged-ca will be auto-renewed. Next renewal at 2024-03-06T13:02:07+00:00.
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Stopping all modules...
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - All modules stopped
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Detecting if device information has changed...
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Device information has not changed
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Starting management API...
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Starting workload API...
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Starting image garbage collection task...
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Starting watchdog with 60 second period...
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:30Z [INFO] - Watchdog checking Edge runtime status
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [INFO] - <-- PUT /identities/modules/$edgeAgent?api-version=2020-09-01&type=aziot {"content-type": "application/json", "host": "identityd.sock", "content-length": "40"}
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"}
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "402"}
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:30Z [INFO] - --> 200 {"content-type": "application/json"}
Dec 26 18:19:30 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:30Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: unexpected EOF
Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:34Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: unexpected EOF
Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:34Z [INFO] - !!! Hub client error
Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:34Z [INFO] - !!! caused by: error trying to connect: unexpected EOF
Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:34Z [INFO] - !!! caused by: unexpected EOF
Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:34Z [INFO] - --> 404 {"content-type": "application/json"}
Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-edged[14160]: 2023-12-26T18:19:34Z [WARN] - Error in watchdog: Failed to update $edgeAgent identity: Hub client error
Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-edged[14160]: caused by: error trying to connect: unexpected EOF
Dec 26 18:19:34 WINDOWSVM-EFLOW aziot-edged[14160]: caused by: unexpected EOF
Dec 26 18:19:35 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:35Z [INFO] - <-- POST /identities/device/reprovision?api-version=2020-09-01 {"content-type": "application/json", "host": "identityd.sock", "content-length": "16"}
Dec 26 18:19:35 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:35Z [INFO] - Provisioning starting. Reason: Api
Dec 26 18:19:35 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:35Z [INFO] - Updated device info for myEdgeDevice.
Dec 26 18:19:35 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:35Z [INFO] - Provisioning complete.
Dec 26 18:19:35 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:35Z [INFO] - Identity reconciliation started. Reason: Api
Dec 26 18:19:36 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:36Z [INFO] - <-- GET /key/device-id?api-version=2021-05-01 {"host": "keyd.sock"}
Dec 26 18:19:36 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:36Z [INFO] - --> 200 {"content-type": "application/json"}
Dec 26 18:19:36 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:36Z [INFO] - <-- POST /sign?api-version=2021-05-01 {"content-type": "application/json", "host": "keyd.sock", "content-length": "402"}
Dec 26 18:19:36 WINDOWSVM-EFLOW aziot-keyd[14164]: 2023-12-26T18:19:36Z [INFO] - --> 200 {"content-type": "application/json"}
Dec 26 18:19:36 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:36Z [WARN] - Failed to send HTTP request (attempt 1 of 2): error trying to connect: unexpected EOF
Dec 26 18:19:39 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:39Z [WARN] - Failed to send HTTP request (attempt 2 of 2): error trying to connect: unexpected EOF
Dec 26 18:19:39 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:39Z [INFO] - !!! Hub client error
Dec 26 18:19:39 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:39Z [INFO] - !!! caused by: error trying to connect: unexpected EOF
Dec 26 18:19:39 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:39Z [INFO] - !!! caused by: unexpected EOF
Dec 26 18:19:39 WINDOWSVM-EFLOW aziot-identityd[14162]: 2023-12-26T18:19:39Z [INFO] - --> 404 {"content-type": "application/json"}
The text was updated successfully, but these errors were encountered: