[BUG] - Unable to connect to KUKA lbr iisy3 r760 using ROS2 Humble driver

[prasuchit]

Description

Prequisites:

I have ROS2, and all the necessary packages from here cloned and compiled successfully.
I have all the dependencies installed correctly.
I have the KUKA iiQKA.ExternalAPI Control 1.0.3 installed on the robot.
I have installed and verified the RT kernel correctly as described here.

Problem:

Until recently I was able to connect with the robot, send commands through moveit-rviz and make the robot move.

Recently I pulled changes from all the Kroshu GitHub packages and recompiled and suddenly saw it complain about missing ExternalAPI control toolbox. I assumed this was because the external API may have been updated, so I backed up my current installation, cloned a fresh version from here, followed the instructions, and compiled it successfully. After this, I was able to colcon build successfully but started running into communication issues.

I found a very similar issue logged on the GitHub repo [here](https://github.com/kroshu/kuka_drivers/issues/150](https://github.com/kroshu/kuka_drivers/issues/150), and the developer seems to point to the externalAPI as the source of the problem.

I've raised a ticket with KUKA Global (case #: 00973371) but haven't heard back in a few days. Please help. I'm working on a very strict deadline and need to get things operational soon!

@Svastits

KUKA robot OS

iiQKA

KUKA robot OS version

iiQKA.OS 1.2

KUKA external interface version

1.0.3

Affected robot model(s)

lbr iisy 3 r760

Version or commit hash of the driver

No response

Setup

I have all the ROS2 packages installed inside ~/colcon_ws/src and the externalAPI installed at ~/

I put the robot on AUT mode and release SPOC and then on the host machine run:
 
ros2 launch iiqka_moveit_example moveit_planning_example.launch.py with the correct controller and client ip

Reproduction steps

I just followed the step-by-step instructions provided on the Kroshu GitHub readme and instructions provided by customer support representatives from KUKA USA to get everything set up and running.

This problem started last week out of the blue.

Logs

**_CASE 1: When configure succeeds but still can't communicate with the robot_**

    [control_node-1] [INFO] [1712268704.734354956] [control_mode_handler]: Control mode changed to 1
    [control_node-1] [WARN] [1712268704.734893341] [controller_manager]: Could not 'activate' controller with name 'event_broadcaster' because no controller with this name exists
    [control_node-1] [ERROR] [1712268704.734946983] [controller_manager]: Aborting, no controller is switched! ('STRICT' switch)
    [robot_manager_node-2] [ERROR] [1712268704.735179233] [robot_manager]: Could not activate control mode handler or event broadcaster
    [control_node-1] [WARN] [1712268704.735475618] [controller_manager]: Could not 'deactivate' controller with name 'event_broadcaster' because no controller with this name exists
    [control_node-1] [ERROR] [1712268704.735603469] [controller_manager]: Aborting, no controller is switched! ('STRICT' switch)
    [control_node-1] [INFO] [1712268704.736181340] [resource_manager]: 'cleanup' hardware 'lbr_iisy3_r760' 
    [control_node-1] [INFO] [1712268704.736216757] [resource_manager]: Successful 'cleanup' of hardware 'lbr_iisy3_r760'
    [robot_manager_node-2] [ERROR] [1712268704.735858614] [robot_manager]: Could not deactivate control mode handler and event broadcaster
    [control_node-1] [INFO] [1712268723.019401331] [control_mode_handler]: Control mode changed to 1
    [control_node-1] [INFO] [1712268723.019449171] [resource_manager]: 'configure' hardware 'lbr_iisy3_r760' 
    [control_node-1] [INFO] [1712268723.022803538] [KukaEACHardwareInterface]: Set QoS profile with 2 consequent and 1 packet losses allowed in 5000 milliseconds
    [control_node-1] [INFO] [1712268723.022865201] [resource_manager]: Successful 'configure' of hardware 'lbr_iisy3_r760'
    [control_node-1] [WARN] [1712268723.023563206] [controller_manager]: Could not 'activate' controller with name 'event_broadcaster' because no controller with this name exists
    [control_node-1] [ERROR] [1712268723.023642165] [controller_manager]: Aborting, no controller is switched! ('STRICT' switch)
    [robot_manager_node-2] [ERROR] [1712268723.023853978] [robot_manager]: Could not activate control mode handler or event broadcaster
    [control_node-1] [WARN] [1712268723.024017142] [controller_manager]: Could not 'deactivate' controller with name 'event_broadcaster' because no controller with this name exists
    [control_node-1] [ERROR] [1712268723.024057560] [controller_manager]: Aborting, no controller is switched! ('STRICT' switch)
    [robot_manager_node-2] [ERROR] [1712268723.024162152] [robot_manager]: Could not deactivate control mode handler and event broadcaster
    [control_node-1] [INFO] [1712268723.024301990] [resource_manager]: 'cleanup' hardware 'lbr_iisy3_r760' 
    [control_node-1] [INFO] [1712268723.024325660] [resource_manager]: Successful 'cleanup' of hardware 'lbr_iisy3_r760'

    **_CASE 2: When configure fails_**

    [control_node-1] [INFO] [1712347470.988823482] [resource_manager]: 'configure' hardware 'lbr_iisy3_r760' 
    [control_node-1] [INFO] [1712347470.989018744] [control_mode_handler]: Control mode changed to 1
    [control_node-1] [ERROR] [1712347470.990987946] [KukaEACHardwareInterface]: QoS configuration failed, error message: failed to connect to all addresses
    [control_node-1] [INFO] [1712347470.991089417] [resource_manager]: Failed to 'configure' hardware 'lbr_iisy3_r760'
    [robot_manager_node-2] [ERROR] [1712347470.991469636] [robot_manager]: Could not configure hardware interface

    **_Some additional recent logs:_**

     [spawner-8] [INFO] [1712691090.634063134] [spawner_control_mode_handler]: Loaded control_mode_handler
    [spawner-7] [FATAL] [1712691090.634563429] [spawner_effort_controller]: Failed loading controller effort_controller
    [control_node-1] [INFO] [1712691090.635660190] [joint_trajectory_controller]: Controller state will be published at 50.00 Hz.
    [control_node-1] [INFO] [1712691090.637709818] [joint_trajectory_controller]: Action status changes will be monitored at 20.00 Hz.
    [control_node-1] [INFO] [1712691090.645277838] [controller_manager]: Configuring controller 'control_mode_handler'
    [control_node-1] [INFO] [1712691090.646438658] [control_mode_handler]: Control mode handler configured
    [INFO] [spawner-4]: process has finished cleanly [pid 183368]
    [INFO] [spawner-6]: process has finished cleanly [pid 183372]
    [ERROR] [spawner-7]: process has died [pid 183374, exit code 1, cmd '/opt/ros/humble/lib/controller_manager/spawner effort_controller -c /controller_manager -n --inactive --ros-args'].
    [INFO] [spawner-5]: process has finished cleanly [pid 183370]
    [INFO] [spawner-8]: process has finished cleanly [pid 183376]
    [control_node-1] [INFO] [1712691103.646620724] [control_mode_handler]: Control mode changed to 1
    [control_node-1] [WARN] [1712691103.647040309] [controller_manager]: Could not 'activate' controller with name 'event_broadcaster' because no controller with this name exists
    [control_node-1] [ERROR] [1712691103.647126353] [controller_manager]: Aborting, no controller is switched! ('STRICT' switch)
    [robot_manager_node-2] [ERROR] [1712691103.647303042] [robot_manager]: Could not activate control mode handler or event broadcaster
    [robot_manager_node-2] [ERROR] [1712691103.647635759] [robot_manager]: Could not deactivate control mode handler and event broadcaster
    [control_node-1] [WARN] [1712691103.647504963] [controller_manager]: Could not 'deactivate' controller with name 'event_broadcaster' because no controller with this name exists
    [control_node-1] [ERROR] [1712691103.647528412] [controller_manager]: Aborting, no controller is switched! ('STRICT' switch)
    [control_node-1] [INFO] [1712691103.647895328] [resource_manager]: 'cleanup' hardware 'lbr_iisy3_r760' 
    [control_node-1] [INFO] [1712691103.647959225] [resource_manager]: Successful 'cleanup' of hardware 'lbr_iisy3_r760'

[Svastits]

Hi @prasuchit, I will try to solve your problem as soon as possible, although it seems to be a rather strange issue...

The second attempt seems to be a communication issue, maybe your network cable was not correctly connected - I hope this only came up once.

From the other two cases, what I can see is that the event_broadcaster controller is not loaded for some reason. This controller was added not so long ago to the kuka_controllers repo, did you also update that?

The problem seems to be related to the ROS2 side of the setup, so I would also suggest to try it out in mock mode:
ros2 launch kuka_iiqka_eac_driver startup.launch.py use_fake_hardware:=true
If the configuration and activation steps also fail in this case, the problem is certainly with the ROS2 setup, and you should update every package (both apt upgrade + our repos)

[prasuchit]

Hi @Svastits,

Thanks for the prompt response. I don't think it's a ROS issue because for the past week, since the time the driver broke, I've been using this file to continue building and testing my code.

Although I see some warnings and occasionally a couple of errors, I can still see and move the robot correctly on Moveit2-Rviz.

I even tried running the C++ examples and they also work on the fake hardware. So I think the problem is connected to the real robot.

[Svastits]

Hi @prasuchit
Actually the launch file you used for fake hardware is only for trying out moveit and does not use the robot manager node (which tries to activate the controllers and fails in the process).
Please try exactly the same workflow, as with the real robot, with use_fake_hardware:=true. If that is also successful, the problem is indeed with the robot.

[prasuchit]

@Svastits your intuition was correct. It is related to the ROS package. This is what I see when I run the fake hardware:

[spawner-7] [FATAL] [1712847085.991812889] [spawner_effort_controller]: Failed loading controller effort_controller
[INFO] [spawner-8]: process has finished cleanly [pid 613549]
[INFO] [spawner-6]: process has finished cleanly [pid 613545]
[INFO] [spawner-5]: process has finished cleanly [pid 613543]
[ERROR] [spawner-7]: process has died [pid 613547, exit code 1, cmd '/opt/ros/humble/lib/controller_manager/spawner effort_controller -c /controller_manager -n --inactive --ros-args'].
[INFO] [spawner-4]: process has finished cleanly [pid 613541]
[rviz2-10] Warning: class_loader.impl: SEVERE WARNING!!! A namespace collision has occurred with plugin factory for class rviz_default_plugins::displays::InteractiveMarkerDisplay. New factory will OVERWRITE existing one. This situation occurs when libraries containing plugins are directly linked against an executable (the one running right now generating this message). Please separate plugins out into their own library or just don't link against the library and use either class_loader::ClassLoader/MultiLibraryClassLoader to open.
[rviz2-10]          at line 253 in /opt/ros/humble/include/class_loader/class_loader/class_loader_core.hpp
[rviz2-10] [ERROR] [1712847101.466473631] [moveit_ros_visualization.motion_planning_frame]: Action server: /recognize_objects not available
[control_node-1] [INFO] [1712847117.017051283] [control_mode_handler]: Control mode changed to 1
[control_node-1] [INFO] [1712847117.017330586] [resource_manager]: 'deactivate' hardware 'lbr_iisy3_r760' 
[control_node-1] [INFO] [1712847117.017365314] [resource_manager]: Successful 'deactivate' of hardware 'lbr_iisy3_r760'
[control_node-1] [WARN] [1712847117.017713382] [controller_manager]: Could not 'activate' controller with name 'event_broadcaster' because no controller with this name exists
[control_node-1] [ERROR] [1712847117.017758949] [controller_manager]: Aborting, no controller is switched! ('STRICT' switch)
[robot_manager_node-2] [ERROR] [1712847117.017921816] [robot_manager]: Could not activate control mode handler or event broadcaster
[control_node-1] [WARN] [1712847117.018365935] [controller_manager]: Could not 'deactivate' controller with name 'event_broadcaster' because no controller with this name exists
[control_node-1] [ERROR] [1712847117.018427986] [controller_manager]: Aborting, no controller is switched! ('STRICT' switch)
[control_node-1] [INFO] [1712847117.018991883] [resource_manager]: 'cleanup' hardware 'lbr_iisy3_r760' 
[control_node-1] [INFO] [1712847117.019041980] [resource_manager]: Successful 'cleanup' of hardware 'lbr_iisy3_r760'
[robot_manager_node-2] [ERROR] [1712847117.018634546] [robot_manager]: Could not deactivate control mode handler and event broadcaster

Also, what is this SEVERE WARNING namespace collision that keeps popping up? How do we fix it?

Since there are other errors earlier in the log, I decided to upload the complete log via txt file:

Complete_log.txt

[Svastits]

@prasuchit I can see 3 issues in the logs:

• namespace collision for interactive markers, I think this causes the issue of not being able to drag the robot in rviz. I do not know the cause or solution of this, but it does not have another effect on the driver
• hardware interface gets activated immediately - connection to the robot should be only opened if the user requests via the robot manager transitions. This might cause other issues, so I would fix this, but is not the cause of the main issue. Solution: there was an API-breaking change in ros2_control, upgrade the ros-humble-ros2-control package.
• the effort controller cannot be loaded. This is the main issue. For some reason, the effort_controllers/JointGroupPositionController defined in the kuka_controllers repo (effort_controllers package) is not found. Ros2 also provides a package named effort_controllers, for some reason the package in your worskpace does not override this. You could try to rebuild effort_controllers locally, OR uninstall the ros2 version OR just modify the controller config (kuka_iiqka_eac_driver/config/ros2_controller_config.yaml) and change effort_controllers/JointGroupPositionController to e.g. effort_controllers/JointGroupEffortController. (This will only change the bahevior in torque mode)

[prasuchit]

Hi @Svastits,

I appreciate you actively working with me on this to resolve the problem.

1. For the first problem, I found that others have faced the same issue in the past but the latest updates in ros2-control fixed this. I have raised an issue here so even you can track it if you're interested.

2. I just tried upgrading ros2-control but it says that all packages are at their latest version. I run upgrades periodically, so I'm confused why this is happening.

3. The first soultion you suggested didn't work. In fact, colcon identified that there is another package with the same name and asked if I wanted to override it, and I compiled again with the override option, sourced the ws and tried again but in vain.

       [0.445s] WARNING:colcon.colcon_core.package_selection:Some selected packages are already built in one or more underlay workspaces:
    'effort_controllers' is in: /home/psuresh/colcon_ws/install/effort_controllers, /opt/ros/humble
      If a package in a merged underlay workspace is overridden and it installs headers, then all packages in the overlay must sort their include directories by workspace order. Failure to do so may result in build failures or undefined behavior at run time.
      If the overridden package is used by another package in any underlay, then the overriding package in the overlay must be API and ABI compatible or undefined behavior at run time may occur.
      
      If you understand the risks and want to override a package anyways, add the following to the command line:
          --allow-overriding effort_controllers
      
      This may be promoted to an error in a future release of colcon-override-check.
      Starting >>> effort_controllers
      Finished <<< effort_controllers [2.42s] 
   

So I tried the second idea and updated the ros2_controllers.yaml: effort_controllers/JointGroupEffortController

The problem persists. I've attached the updated log.
Complete_log.txt

[Svastits]

@prasuchit this solved the effort_controller issue, now the event_broadcaster is not found and therefore cannot be activated. I have never seen such an issue before, a spawner process is not started, although it is specified in the launch file...
As a workaround you could try to spawn manually before configuring using ros2 run controller_manager spawner event_broadcaster -c controller_manager --inactive

Unfortunately I cannot come up with a non-workaround solution, as everything works fine for us both locally and in CI

[prasuchit]

@Svastits that actually fixed the issue, I was able to plan and move the robot within fake execution!! :)

However, it is puzzling why these issues are happening. I see that the robot manager is specified to start here. Is it dependent on something else? Maybe the order of starting the nodes matters?

For sanity sakes, if you can, please check if someone on your side can set up all the repos on Humble just by following the Readme instructions and see if the same problems can be replicated.

Just to verify with you that everything is fixed now, I've attached the latest log:

Complete_log.txt

[Svastits]

The controller spawners depend on the controller_manager node (and wait until it is available), otherwise there are no other dependencies in the startup procedure, so the order should not really count.

Actually we have set up more machines based on the readme, and the industrial CI that runs for every push and PR is also following those steps, so I am pretty sure, that the issue is not on our side.

The logs now seem mostly right, the second issue is still there that the hardware interface is activated automatically, then is reactivated when you activate manually. What is the ros2_control version you use? It should be 2.40, with that version, this should not happen...

[prasuchit]

Hi @Svastits,

I just checked:

ros-humble-ros2-control is already the newest version (2.40.0-1jammy.20240304.153413).
ros-humble-ros2-control-test-assets is already the newest version (2.40.0-1jammy.20240304.145856).
ros-humble-ros2-controllers is already the newest version (2.33.0-1jammy.20240311.114913).
ros-humble-ros2-controllers-test-nodes is already the newest version (2.33.0-1jammy.20240301.190919).

I'm closing this issue for now since it is working now, albeit through unconventional ways. Thanks for all the help.

Is there a plan to move to Iron? MoveitPy is integrated into Iron and isn't available in Humble.

[Svastits]

Thanks :)
Iron support is not planned, we will only support LTS releases. jazzy will come soon ;)