Torchcchio

[lvjonok]

Hello Pinocchio Team,

Around six months ago, when there was no Pinocchio 3 with Casadi support in the Python bindings, I embarked on developing a fork of casadi_kin_dyn to streamline its usage. During this process, I wondered if it would be possible to instantiate templates in Pinocchio using torch variables.

To explore this, I created a small example. However, I encountered an issue where the computational graph was not preserved during computations.

I am reaching out to ask if integrating torch variables in Pinocchio is feasible. There might be a minor issue in my implementation, and if so, I would be happy to assist with integrating torch support into the core codebase.

Additionally, I found this discussion, which suggests the proposal might not be possible. Any insights or guidance would be greatly appreciated.

Thank you!

[simeon-ned]

Hello @jcarpent and @lvjonok,

I hope you don't mind me reviving this discussion. I recently stumbled upon an intriguing implementation from Facebook Research that appears to create a Torch wrapper around Pinocchio's functionality:

• C++ code with bindings
• Python docs

This implementation includes methods for forward kinematics, inverse dynamics, Jacobian computation, and inverse kinematics using Pinocchio, all wrapped to be compatible with PyTorch tensors. It's quite an impressive piece of work!

I wanted to bring this to the attention of the Pinocchio community and ask a few questions:

1. Are the Pinocchio developers aware of this implementation?
2. What are your thoughts on this approach to integrating Pinocchio with PyTorch?
3. Are there any plans within the Pinocchio project to create official PyTorch bindings or integration?
4. If not, would you be open to contributions in this direction?

I believe having PyTorch integration could be incredibly valuable for users who want to combine Pinocchio's efficient rigid body algorithms with PyTorch's machine learning capabilities, especially for applications in reinforcement learning for robotics.

Thank you for your time and consideration. I'm really looking forward to hearing your thoughts on this exciting development!

[jcarpent]

Dear @simeon-ned,

Thanks for coming with these news.

1. Are the Pinocchio developers aware of this implementation?

We were not aware of such initiative, which seems very interesting indeed.
We are very happy to know that Pinocchio has been integrated into PyTorch and could then be easily used inside this deep learning framework.

2. What are your thoughts on this approach to integrating Pinocchio with PyTorch?

I will have a deeper look to the code. But I think we could have a seeless integration, with full support of operations over Lie groups and algebra.

3. Are there any plans within the Pinocchio project to create official PyTorch bindings or integration?

We can think of integrating directly the Torch bindings into the main Pinocchio lib for extended support and integration with all new features, especially constrained dynamics part which seems an important block for future applications.

4. If not, would you be open to contributions in this direction?

We are of course open to external support (as your are currently doing). Thanks in advance for contributing to Pinocchio.
One important remark is that we are mostly missing time to provide proper code review, as we are now focus mostly focus on https://github.com/Simple-Robotics/Simple.

Best,
Justin
PS: and thanks to @lvjonok for starting the discussion

[simeon-ned]

Thank you for the update, @jcarpent! This is great news indeed. I'm excited to hear about the potential for integrating PyTorch support directly into Pinocchio.

@lvjonok and I are looking forward to helping integrate this feature into Pinocchio. We're sure it will be highly appreciated by the RL and robotics communities.

As an interim solution, we could create a small standalone module similar to casadi_kin_dyn, which served as a temporary solution before CasADi support was added to the Python bindings. Our module would instead focus specifically on Torch integration.

Then, when you have more time available after your work on the Simple simulator, we could collaborate on moving this functionality directly into Pinocchio. This approach would allow the community to start benefiting from PyTorch integration sooner while also paving the way for a more seamless integration in the future.

By the way, it's great to hear that you're putting so much attention into Simple. We're eagerly waiting to try it out - the paper looks very promising!

Looking forward to collaborating on this. Let us know how we can best support this effort moving forward.

Best regards,
Simeon