- Please go to the section "Overview of the original ZS6D-Dino Project" and follow the installation process from there.
- If you are on Ubuntu 22.04, there will be an error, to solve it go to https://stackoverflow.com/questions/72110384/libgl-error-mesa-loader-failed-to-open-iris
- If your graphics card has less than 32 GByte VRAM create 32 GByte swap-memory, we tested it with 16GBVRAM and 32GBSWAP
We demonstrate the effectiveness of deep features extracted from self-supervised, pre-trained Vision Transformer (ViT) for Zero-shot 6D pose estimation. For more detailed information check out the corresponding [[paper](https://arxiv.org/pdf/2309.11986.pdf)].
## How to run ZS6D with CroCo
-**General rule, you will run into several PATH issues! I mentioned the most important ones!**
- Download **CroCo.pth** and **CroCo_V2_ViTLarge_BaseDecoder.pth** from the original croco git and put them into [pretrained_models](pretrained_models)
- Change import statements in the croco (subgit), there should be 4, your IDE will mark it either way
-[prepare_templates_and_gt_croco.py](prepare_templates_and_gt_croco.py) prepares the croco descriptoren, if you want to run it change the paths to your dataset in [cfg_template_gt_generation_ycbv_croco.json](zs6d_configs%2Ftemplate_gt_preparation_configs%2Fcfg_template_gt_generation_ycbv_croco.json)
-[bop_eval_configs](zs6d_configs%2Fbop_eval_configs) check all the paths in the **.json files**
-[test_zs6d_croco.py](test_zs6d_croco.py) is the test script, if you run it, it probably will find no pose because CroCo does not work for the ZS6D pipeline
- To test with **CroCo_V2_ViTLarge_BaseDecoder** just exchange it in the function call with **crocov2**
- evaluation is done with [evaluate_zs6d_bop_croco.py](evaluate_zs6d_bop_croco.py), check also the files in [bop_eval_configs](zs6d_configs%2Fbop_eval_configs)
- analyse the created evaluation data with [analyse_evaluated_zs6d_data.py](analyse_evaluated_zs6d_data.py)
## How to run CroCoM
- Try to run CroCo first, because it also has some **setup steps** which are needed for **CroCoM**, if you are Pro you can try to go straight to CroCoM and debug on the fly
- In [pretrained_models](pretrained_models) is [crocom.py](pretrained_models%2Fcrocom.py) put it into the folder from the **original corco (subgit)** where **croco.py** is found
- Start [croco_match.py](croco_match.py) for testing on **single segmented objects**, by changing the paths in the **main** function
-[evaluate_zs6d_bop_crocom.py](evaluate_zs6d_bop_crocom.py) evaluates on **myset (small ybvc testset)**
-**There is currently no implementation for CroCoM in test_zs6d_crocom.py it does not exist!**
- Evaluation is done with [evaluate_zs6d_bop_crocom.py](evaluate_zs6d_bop_crocom.py), check also the files in [bop_eval_configs](zs6d_configs%2Fbop_eval_configs)
- Analyse the created evaluation data with [analyse_evaluated_zs6d_data.py](analyse_evaluated_zs6d_data.py)
## For what are the additional Scripts?
**No working paths are guaranteed! Personal testing Scripts!**
## Overview of the Pipeline:
Testing a specific layer and token, to find the best matches of the templates to segmented object
Testing all layers and tokens, on one template and one segmented object
-[test_croco_layers.py](test_croco_layers.py)
Testing the output of the CroCoDownstreamMonocularEncoder
-[test_croco_output.py](test_croco_output.py)
## Overview of the original ZS6D-Dino Project:
Note that this repo only deals with 6D pose estimation, you need segmentation masks as input. These can be obtained with supervised trained methods or zero-shot methods. For zero-shot we refer to [cnos](https://github.com/nv-nguyen/cnos).
We demonstrate the effectiveness of deep features extracted from self-supervised, pre-trained Vision Transformer (ViT) for Zero-shot 6D pose estimation. For more detailed information check out the corresponding [[paper](https://arxiv.org/pdf/2309.11986.pdf)].
## Installation:
To setup the environment to run the code locally follow these steps:
