LED strobbing. Race condition ?

[Mateo-Lopez-Espejo]

Hello, I have been using Bonsai for the first time this week and it has been amazing how fast you can prototype ideas. Right now I am trying to deliver optogenetic stimulation to a fly when it stops moving or moves in a certain direction. I want to turn the LED so long as the fly is doing the selected action, with a LED pulse minimum duration of 0.5ms, however when the fly does the action for an extended period of time the LED starts strobing (undesirable) after being continuously on for a while (desirable). I bet this has something to do with a race condition, but I cant figure it out.

Furthermore, to terminate the process I had to add a bunch of TakeUntil all over the place, in seemingly arbitrary places: right after sources (like the camera) but also before and after some sinks (Arduino) but not other (VideoWriter). There is likely a better way of dealing with termination and a logic behind it, but right now it eludes me.

Finally, and this is a long shot, I am streaming video at 150 FPS and supposedly recording it at the same frequency, however there is a lot of dropped frames, and a 30 minutes recording becomes 11 minutes since it seems ffmpeg does not account for the frame drop and saves a "150 FPS" video. I suspect I might be doing something silly/inefficient with all the publish/subscriptions that I have places to deal with the spaghetti, and this might be hindering frame rate.

I am attaching the full project, and also a toy example of the LED control which right now is the most critical issue.
Any help is greatly appreciated.

[banchan86]

This seems like a fun project!

Do you mind sharing your workflow as outlined in https://github.com/orgs/bonsai-rx/discussions/864?

With regards to the video FPS, maybe there are a few things we can check first. If you copy over just the video recording pipeline to a new workflow and run it without the rest, are you able to achieve 150fps without any dropped frames (Timestamp + a CsvWriter/MatrixWriter if the CsvWriter can't keep up). How about with the Sleap inference pipeline enabled?

[Mateo-Lopez-Espejo]

Hello Shawn, thanks for your answer.

Sorry I though I had uploaded the zip with the files and did not checked.
The files are attached this time. I wanted to add a video and the SLEAP model used for inference, but both of those are quite heavy and make the archive go over the size limit.
The low hanging fruit right now is the LED strobing, which is better captured by the toy example that I have attached, where the LED turns on for n time after mouse wheel up. If i keep spinning the wheel, the LED should stay on, however, it starts strobing after a while.

Looking at the timestamps of recorded frames, the frame rate clearly drops when there is SLEAP inference in the middle, which surprises me since it should be handled by the GPU (GTX 3070).

edit:
After isolating the video write, it is clear that SLEAP inference is dramatically reducing the ffmpeg encoding speed from >150 Hz to ~ 60. Perhaps I will need to play around with the encoding quality or different codecs.

LED_race_condition.zip

[banchan86]

Thanks Mateo! We can try isolating the SLEAP issue first and maybe that will fix the timing issues for the rest of the workflow. Another thing you could test is SLEAP offline inference (not using Bonsai) with the same files and model you are using for this workflow. Just want to see if the bottleneck is just restricted to Bonsai. Could also try reducing the video dimensions if you aren't losing too much detail.

[banchan86]

Also how fast is the video inference running if you look at the CSV files for the inference?

[Mateo-Lopez-Espejo]

You are right in that the GPU inference is a bottleneck. Doing offline Inference I am reaching 96 Hz, way below the input/desired 150 Hz, however way above the ~ 60 Hz video encoding. I am going to swap he GPU for a beefier one (GTX 3060 -> 4090) which I know can perform inference at 190 Hz and I will let you know if this solves the issue.

What surprises me is that I thought that Inferences was happening in parallel or independently from other processes i.e., video encoding, and that is happening on different hardware (CPU vs GPU).

If the GPU swapping does not solve the issue, do you think would it be worth try to unload some computation from the CPU and use the specialized nvidia video encoding cores, which are different from the cuda cores running inference? ffmpeg -hevc_nvenc

[banchan86]

One optimization for the workflow - I think you can combine the DeltaAngle and Velocity branches together (to make sure they aren't interfering with each other for the LED control) using an ExpressionTransform. Something like this:

The ExpressionTransform node isn't that well documented, we are still revising the documentation but here is a draft.

Within the ExpressionTransform you might write something like this (not sure if this is the right logic you want so check it):

(it.DeltaAngle > 2 or !(it.Velocity > 0.5)) ? true : false

The outputs from the Zip node won't be called DeltaAngle or Velocity, I think they will just be called Item1 or Item2 but for some reason I wasn't able to check the outputs from the Zip node so just name them accordingly.

[Mateo-Lopez-Espejo]

This is a good optimization and I have added it. however I encounter the same strobing problem in the toy example that I sent, where the only condition is if the mouse wheel has been rolled forwards. In theory if you roll forwards faster than the light on timer, the light should stay on at all times, but this is not the case for sustained "valid actions".

In the following image of the toy pipeline, the light duration is 2 seconds, so the light should stay on if I spin the wheel frequently enough, which I am sure I am doing (top graph). However the LED control signal starts ringing after a while and even gets stuck in the off position (bottom graph).

I guess there might be a better way of implementing this LED control than what I am doing currently.

[banchan86]

Hi Mateo will this revised LED logic work for you?

Basically the delay operator is producing a time-delayed version of the original sequence. Set the delay to your desired LED-time off (in the attached example the time is set to 1 second for better visualization). The CombineLatest operator merges the latest version of both sequences, and the ExpressionTransform operator will evaluate both sequences. Within the ExpressionTransform operator we have:
it.Item1 || it.Item2

This only evaluates to false if both items are false, if any item is true it evaluates to true.

In real terms, it only sends the LED off signal if the mousewheel is still not moving 1 sec after.

Workflow attached
#2086 toy_modified_simplest.bonsai.zip

[Mateo-Lopez-Espejo]

Hello Shawn.

Alas, this configuration did not work as intended. However I found a working solutions using CreateObservable and Switch to just keep track of the timer of the last observable

[banchan86]

I am also not sure what you are trying to do with the StartRecording and EndRecording nodes, especially when they are connected to stuff like the CsvWriter. For the CsvWriter I usually prefer just recording all the raw data, and timestamping events when they happen. That may be one way to avoid all the SkipUntil and TakeUntil. Also I usually set up the CsvWriter to record separate CSV files for different data sources, and do postprocessing to integrate them later. I am not sure if something funky is going on there by combining them all together.

[Mateo-Lopez-Espejo]

the start and end recording controls were my attempts to make the runtime finish after 30 minutes. The timers successfully stop the video recording after that time, but video streaming, inference and perhaps other processes kept going preventing the experiment to finish and shut down all hardware automatically. I will separate the CSV writing into multiple files as it seems it should only increase performance, however I did not encounter any video write issues using the combined CSV.