rmf_traffic_schedule delay messages

[cwrx777]

Hi all,

Any idea what causes the following messages?

1678925981.6027112 [rmf_traffic_schedule-1] [1678925981.602266873][ERROR][rmf_traffic_schedule_primary]: Delay of 7818.776290s for DeliveryRobot_5 of group DeliveryRobot exceeds maximum delay limit of 3600.000000s.
1678925981.7438095 [rmf_traffic_schedule-1] [1678925981.743504022][ERROR][rmf_traffic_schedule_primary]: Delay of -5226.896260s for DeliveryRobot_5 of group DeliveryRobot exceeds maximum delay limit of 3600.000000s.
1678925981.8947752 [rmf_traffic_schedule-1] [1678925981.894420170][ERROR][rmf_traffic_schedule_primary]: Delay of 7818.776290s for DeliveryRobot_5 of group DeliveryRobot exceeds maximum delay limit of 3600.000000s.
1678925981.8950193 [rmf_traffic_schedule-1] [1678925981.894506008][ERROR][rmf_traffic_schedule_primary]: Delay of -5226.896260s for DeliveryRobot_5 of group DeliveryRobot exceeds maximum delay limit of 3600.000000s.
1678925982.1839831 [rmf_traffic_schedule-1] [1678925982.183672793][ERROR][rmf_traffic_schedule_primary]: Delay of 3881.474572s for DeliveryRobot_5 of group DeliveryRobot exceeds maximum delay limit of 3600.000000s.
1678925982.3371518 [rmf_traffic_schedule-1] [1678925982.336514552][ERROR][rmf_traffic_schedule_primary]: Delay of 7818.776290s for DeliveryRobot_5 of group DeliveryRobot exceeds maximum delay limit of 3600.000000s.
1678925982.3375492 [rmf_traffic_schedule-1] [1678925982.336621721][ERROR][rmf_traffic_schedule_primary]: Delay of -5226.896260s for DeliveryRobot_5 of group DeliveryRobot exceeds maximum delay limit of 3600.000000s.

[mxgrey]

This is an indication that the arrival time estimation for your robot is out of whack. Usually this happens because of a miscalculation in the inputs to the next_arrival_estimate, so if this is a problem that you can reproduce then I would do a test where you log those values and sanity check them.

It's not impossible that a bug exists inside of rmf_fleet_adapter itself which could be causing this. We've seen this issue show up due to severe race conditions in updating the schedule. That lead to this PR which I believe should ensure that such race conditions are impossible at this point. So I would recommend carefully examining your arrival estimate inputs.

If you find that your arrival estimates are reasonable even while you're seeing this error, then let us know if you can provide a way to reproduce the problem.

[liugehaizaixue]

I have also encountered similar issues, accompanied by the following information:

I don't understand what this means

[mxgrey]

The information you're showing in that screenshot is unrelated to the arrival estimate error message. What you're showing is simply some terminal output related to the progress of a traffic negotiation. That kind of output happens under normal circumstances.

It appears that you have nine agents in a simultaneous traffic conflict. That's a larger number than what we've historically tested with the airport demo world, so I wouldn't be surprised if the navigation graph for that world is not suited for that many simultaneous robots. You will probably need to add vertices and lanes to the graph, and use additional techniques to help the traffic flow, like putting parallel one-way lanes alongside each other, pointing in opposite directions.

The next generation of RMF will make it easier to design deployments with larger scales of robots, but the current version of RMF will require you to put careful thought into lane design to handle that many.

[liugehaizaixue]

Thank you for your reply. During my testing process, I found that I also encountered this error after running three robots for a period of time. When I test one robot, this problem does not occur. Perhaps not due to the number of robots?

[mxgrey]

When you say "this error" are you talking about the negotiation output in your screenshot? The output you're showing in your screenshot is normal whenever there are multiple robots in an environment. It's not an error, it's just informational.

You won't see this output when there's only one robot because one robot alone will never experience a traffic conflict and therefore will never need to do a traffic negotiation. But whenever there are multiple robots operating in the same space, they will likely need to negotiate their traffic eventually, which is normal and expected, and you'll see these messages when that happens.

[liugehaizaixue]

When you say "this error" are you talking about the negotiation output in your screenshot? The output you're showing in your screenshot is normal whenever there are multiple robots in an environment. It's not an error, it's just informational.

You won't see this output when there's only one robot because one robot alone will never experience a traffic conflict and therefore will never need to do a traffic negotiation. But whenever there are multiple robots operating in the same space, they will likely need to negotiate their traffic eventually, which is normal and expected, and you'll see these messages when that happens.

Sorry, I didn't express myself clearly. I'm not talking about the error, I'm referring to the situation where the robot disappears

[liugehaizaixue]

The detailed problem I encountered was that during my large-scale testing, I actually started multiple robots. These robots will go offline after a period of execution, as if the more robots go offline, the faster.

At the beginning, I set up 10 robots to perform their respective tasks normally, as shown in the following figure.

After a period of time, the robot starts to go offline, 'fleet_states' also stopped updating information, and the purple robot in' rviz 'disappeared, leaving only a few yellow circles. As shown in the following figure.

At the same time, I also want to know what yellow circles and purple balls represent respectively, and what are the differences between the two?

[mxgrey]

The purple dot in RViz indicates where the fleet adapter believes your physical (or simulated) robot is based on the information that your integration code provides to the position update functions. If the purple dot is vanishing then the positions are not being updated correctly.

The yellow dot in RViz indicates the location where your robot is intended to be according to the traffic schedule. This will jump around as the traffic schedule is constantly correcting itself based on incoming data. It should periodically snap back to the purple dot, but obviously it can't do that if the purple dot has been lost due to position update problems.

[liugehaizaixue]

The purple dot in RViz indicates where the fleet adapter believes your physical (or simulated) robot is based on the information that your integration code provides to the position update functions. If the purple dot is vanishing then the positions are not being updated correctly.

The yellow dot in RViz indicates the location where your robot is intended to be according to the traffic schedule. This will jump around as the traffic schedule is constantly correcting itself based on incoming data. It should periodically snap back to the purple dot, but obviously it can't do that if the purple dot has been lost due to position update problems.

Thank you very much. I will take a closer look at the relevant code. Meanwhile, I am very looking forward to the next generation of RMF

[liugehaizaixue]

At the same time, the 'rmf' terminal prompts the following two types of information.


In the next short period of time, the robot may suddenly reappear and disappear again. Finally, it completely disappeared and no longer appeared.

What may be causing this?

[liugehaizaixue]

In the next short period of time, the robot may suddenly reappear and disappear again. Finally, it completely disappeared and no longer appeared.

I assume you're running this in a Gazebo simulation.

When you see this happen, I suggest checking Gazebo itself to see where the robot is within the simulation. I've seen cases where a simulation glitch can cause the robot to "vanish". Usually if you pan the camera to the origin point of the simulation coordinates then you'll see the "vanished" robot rendered there. I don't know why this happens, but it seems to be more common when there are larger numbers of robots in the simulation.

I didn't use Gazebo, I wrote the robot code myself and continuously sent the position of the robot's next coordinate point to simulate the robot's motion process.

[mxgrey]

I didn't use Gazebo, I wrote the robot code myself and continuously sent the position of the robot's next coordinate point to simulate the robot's motion process.

Then I would suggest carefully looking over your simulation code as well as whatever code you're using for your fleet adapter. If the robot is vanishing from RViz then it means your fleet adapter isn't calling the position update functions correctly. This may also explain why you'd be seeing delay error messages.

[liugehaizaixue]

I didn't use Gazebo, I wrote the robot code myself and continuously sent the position of the robot's next coordinate point to simulate the robot's motion process.

Then I would suggest carefully looking over your simulation code as well as whatever code you're using for your fleet adapter. If the robot is vanishing from RViz then it means your fleet adapter isn't calling the position update functions correctly. This may also explain why you'd be seeing delay error messages.

Thank you very much. I will check the code and I would like to ask about the difference between the yellow and purple circles in 'rviz'

[mxgrey]

the difference between the yellow and purple circles in 'rviz'

See here

[StetroF]

红色可以理解为rmf订阅robot_state并实时更新的位置，这个一般是机器人的真实位置，黄色表示rmf预测的你的机器人所在的位置，你可以更改加载的比如tiny_robot.yaml的速度，把他加快，你会发现黄色圈圈明显在红色圈圈前方。
The red one can be seen as the real position of the robot which was subscribed by rmf and update in rviz.The yellow one is the position predicted by rmf that it estimate your position.You can change the tiny_robot.yaml 's speed parameter .And you will find it the yellow one will faster at the begging of the car start to navigate.