EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild
EasyHOI is a pipeline designed for reconstructing hand-object interactions from single-view images.
- Provide the code for utilizing the Tripo3D API to improve reconstruction quality - Completed on 2024-12-24.
- Resolve issues in segmentation. - Completed on 2025-01-02
- Integrate the code execution environments into one.
- Complete a one-click demo.
Download MANO models from the official website and place the mano folder inside the ./assets directory. After setting up, the directory structure should look like this:
assets/
├── anchor/
├── mano/
│ ├──models/
│ ├──webuser/
│ ├──__init__.py
│ ├──__LICENSE.txt
├── contact_zones.pkl
├── mano_backface_ids.pkl
Create the environment for optimization:
conda create -n easyhoi python=3.9
conda activate easyhoi
conda install -y pytorch=1.13.0 torchvision pytorch-cuda=11.6 -c pytorch -c nvidia
conda install -c fvcore -c iopath -c conda-forge fvcore iopath
conda env update --file environment.yaml
Install pytorch3d follow the official instruction.
Install HaMeR and ViTPose:
cd third_party
git clone https://github.com/ViTAE-Transformer/ViTPose.git
cd ./hamer
pip install -e .[all]
cd ../ViTPose
pip install -v -e .
Since I haven’t resolved the conflict between the environments yet, it’s necessary to create several virtual environments called afford_diff, lisa, and instantmesh. Please refer to the links below to set up these environments.
-
afford_diff: https://github.com/NVlabs/affordance_diffusion/blob/master/docs/install.md
-
instantmesh: https://github.com/TencentARC/InstantMesh?tab=readme-ov-file
Thanks to the authors of these wonderful projects. I will resolve the environment conflicts as soon as possible and provide a more user-friendly demo.
Set the data directory by running the following command:
export DATA_DIR="./data"
Place your images in the $DATA_DIR/images folder. If you prefer a different path, ensure it contains a subfolder named images.
conda activate easyhoi
python preprocess/recon_hand.py --data_dir $DATA_DIR
export TRANSFORMERS_CACHE="/public/home/v-liuym/.cache/huggingface/hub"
conda activate lisa
CUDA_VISIBLE_DEVICES=0 python preprocess/lisa_ho_detect.py --seg_hand --skip --load_in_8bit --data_dir $DATA_DIR
CUDA_VISIBLE_DEVICES=0 python preprocess/lisa_ho_detect.py --skip --load_in_8bit --data_dir $DATA_DIR
conda activate afford_diff
python preprocess/inpaint.py --data_dir $DATA_DIR --save_dir $DATA_DIR/obj_recon/ --img_folder images --inpaint --skip
conda activate easyhoi
python preprocess/seg_image.py --data_dir $DATA_DIR
conda activate instantmesh
export HUGGINGFACE_HUB_CACHE="/public/home/v-liuym/.cache/huggingface/hub"
python preprocess/instantmesh_gen.py preprocess/configs/instant-mesh-large.yaml $DATA_DIR
To use Tripo3D for reconstruction, you need to generate an API key following the instructions in the Tripo AI Docs. Then replace the api_key
in preprocess/tripo3d_gen.py
with your own key.
After updating the API key, execute the following command in your terminal:
python preprocess/tripo3d_gen.py --data_dir $DATA_DIR
conda activate easyhoi
python preprocess/resample_mesh.py --data_dir $DATA_DIR [--resample]
conda activate easyhoi
python src/optim_easyhoi.py -cn optim_teaser #use instantmesh results
python src/optim_easyhoi.py -cn optim_teaser_tripo #use tripo3d results
We would like to express our gratitude to the authors and contributors of the following projects:
If you find our work useful, please consider citing us using the following BibTeX entry:
@article{liu2024easyhoi,
title={EasyHOI: Unleashing the Power of Large Models for Reconstructing Hand-Object Interactions in the Wild},
author={Liu, Yumeng and Long, Xiaoxiao and Yang, Zemin and Liu, Yuan and Habermann, Marc and Theobalt, Christian and Ma, Yuexin and Wang, Wenping},
journal={arXiv preprint arXiv:2411.14280},
year={2024}
}