Learning to Simulate Tree-Branch Dynamics for Manipulation

Implements a probabilistic, simulator-driven Bayesian inference framework using SVGD to learn spring-model parameters and predict branch deformation dynamics under robotic manipulation.

Code implementation for the paper: Learning to Simulate Tree-Branch Dynamics for Manipulation: https://sites.google.com/view/lstbdm

Pre requisites

1. Ubuntu 18.04 or 20.04
2. Python 3.8
3. Nvidia Graphics Card to run Isaac Gym
4. Separate Conda or Venv

Installations

Isaac Packages

1. Install Pytorch
2. Download & Install Isaac Gym (simulator): https://docs.robotsfan.com/isaacgym/install.html
3. Set spot_path environment variable e.g.,

   export spot_path="/home/<path>/<to>/lstbdm"

4. Install required libraries (refer versions inside requirements.txt)

Notebooks

To run notebooks first set spot_path in command line and then

cd "$spot_path/notebooks"
jupyter-lab

IDE

   > Note: Set the "$spot_path/source" folder as source to run directly in IDE.       
   > E.g. In Pycharm Settings >> Project Structure >> Source Folders >> add $spot_path/source

Execution

1. Sample URDF files for rigid body tree structure based on configuration file

   $spot_path/source/simulation/basic/urdf/tree

2. Estimation: Source folder for parameter estimation.

   # Estimation sources are generally accessible only via notebooks. To run,
   Isaac Gym: $spot_path/notebooks/work1/isaacgym/*.ipynb

   # Notebooks - simulation parameter inference (run in order a,b,c,d)    
   $spot_path/notebooks/work1/isaacgym/sim parameter estimation - 11*.ipynb

   # Notebooks - real parameter inference (run in order a,b,c,d)     
   $spot_path/notebooks/work1/kinova/tree param estimation - 7*.ipynb

Real Executions ROS & API

Ensure ROS-1 (for Kinova Jaco2) is installed as described in https://github.com/Kinovarobotics/kinova-ros

Modules that will end up being part of external builds. For e.g. part of kinova-ros package to run Jaco arm.

  Note: Requires additional path settings in IDE to be interpretable
  In Pycharm Settings >> Project Structure >> Source Folders >> 
     1. Add local ROS libraries as content root : E.g., "/opt/ros/noetic/share:
     2. Add kinova_msgs as content root from local installation of kinova_ros: E.g., "catkin_ws/src/kinova-ros/kinova-msgs"

To run,

 export spot_path="/home/<path>/<to>/lstbdm"
 export kinova_ros_path="/home/<path>/<to>/catkin_ws/src/kinova-ros"
 bash $spot_path/external/kinova_spot_installer.sh
 
 cd "${kinova_ros_path}/../../"
 # in different terminals
 roslaunch kinova_bringup kinova_robot.launch kinova_robotType:=j2n6s300

Real Kinova Data collection

rosrun spot_control gt_trajectory_tree_pos_control.py -v -r j2n6s300 0

Citation

@article{jacob2024learning,
  title={Learning to Simulate Tree-Branch Dynamics for Manipulation},
  author={Jacob, Jayadeep and Bandyopadhyay, Tirthankar and Williams, Jason and Borges, Paulo and Ramos, Fabio},
  journal={IEEE Robotics and Automation Letters},
  year={2024},
  publisher={IEEE}
}

Copyrights

While most work in this repository is original, some are taken/inspired from external github sources.

https://github.com/MFreidank/pysgmcmc/tree/pytorch
https://github.com/EugenHotaj/pytorch-generative/blob/master/pytorch_generative/models/kde.py
https://github.com/ThomasLENNE/L-system
https://github.com/NVIDIA-Omniverse/IsaacGymEnvs
https://github.com/facebookresearch/pytorch3d/tree/main
https://github.com/NVlabs/storm/tree/main
https://github.com/facebookresearch/differentiable-robot-model

⚠️ Note: This repository is not actively maintained, but pull requests are welcome.