Spconv 2.2.x

README.md

@@ -1,34 +1,61 @@
+## Spconv
+Some issues can occur with newer versions of Spconv 2.x. If you get an error, that precompiled functions are not found, make sure you do not have duplicates of the package cumm-[cuda version]. If issues persists, spconv version 2.2.3 or lower is recommended.
 
-# Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation
+## TODO
+- [ ] PyTorch 2.0 support
+- [ ] Removal of torch-scatter for PyTorch native implementation (package functionalities were integrated into PyTorch as .scatter_reduce_)
+- [X] Fix issue in converted network (see below)
+- [X] Test with Cuda 11.6/11.7 support
+- [X] [Spconv 2.2](https://github.com/traveller59/spconv) (FP32 disabled by default, see their instructions on how to enable FP32 if necessary.)
 
- The source code of our work **"Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation**
-![img|center](./img/pipeline.png)
+## Description
+This repository is forked from the [original implementation](https://github.com/xinge008/Cylinder3D). This repositiory was created to encourage continued research and provide a significantly faster implementation.
 
-## News
-- **2021-03 [NEW:fire:]** Cylinder3D is accepted to CVPR 2021 as an **Oral** presentation
-- **2021-01  [NEW:fire:]** Cylinder3D achieves the **1st place** in the leaderboard of SemanticKITTI **multiscan** semantic segmentation
-<p align="center">
-   <img src="./img/leaderboard2.png" width="30%"> 
-</p>
+[Spconv-V2](https://github.com/traveller59/spconv) made major changes to the convolution operation and how weights are read. [Spconv-V1](https://github.com/traveller59/spconv/tree/v1.2.1) is not supported anymore. Following an unsuccessfull effort to restructure the weights, Cylinder3D was retrained on SemanticKITTI to produced new Spconv-v2 weights.
 
-- **2020-12 [NEW:fire:]** Cylinder3D achieves the 2nd place in the challenge of nuScenes LiDAR segmentation, with mIoU=0.779, fwIoU=0.899 and FPS=10Hz.
-- **2020-12** We release the new version of Cylinder3D with nuScenes dataset support.
-- **2020-11** We preliminarily release the Cylinder3D--v0.1, supporting the LiDAR semantic segmentation on SemanticKITTI and nuScenes.
-- **2020-11** Our work achieves the **1st place** in the leaderboard of SemanticKITTI semantic segmentation (until CVPR2021 DDL, still rank 1st in term of Accuracy now), and based on the proposed method, we also achieve the **1st place** in the leaderboard of SemanticKITTI panoptic segmentation.
+Note: the version released publicly by the authors is from their [first paper](https://arxiv.org/pdf/2008.01550.pdf) and does not include the pointwise refinement module or weighted cross entropy. The CVPR version has not been made publically available. The mIOU of the retrained Spconv2 Version is 63.2, compared to 64.3 mIOU in page 7 in the paper. The original implementation does not contain manual seeding. To achieve the 63.2 mIOU result, the training regimen had to be changed slightly, as the Paper results could not be reproduced (See Training).
 
-<p align="center">
-   <img src="./img/leaderboard.png" width="40%"> 
-</p>
+## Improvements
+- Network code updated to Spconv-v2.1.x Credit here goes to the code by @min2209 in [Issue](https://github.com/xinge008/Cylinder3D/issues/107).
+- Spconv-v2.x receives continued support. Speedup of 50 - 80% compared to original implementation
+- Mixed precision support for further speedup during training
+
+It is likely that further improvements can be made with a more careful choice of hyperparameters when training.
+
+### Issus
+Fixed Typo in converted model. Weights are updated.
+```
+reaA = resA.replace_feature(self.bn1(resA.features)) should be: resA = resA.replace_feature(self.bn1(resA.features))
+```
 
 ## Installation
 
+### Weights
+The weights with mIOU 63.2 (Validation, vs 64.3 Original) can be downloaded [here](https://drive.google.com/file/d/1slLJwOGNO8gJ1oTFTH5yTg9CzM1Ov96J/view?usp=sharing) and should be placed into the ./network folder.
+
+Weights are trained according to the original Cylinder3D, and not according to knowledge distillation (Cylinder3D PVKD).
+
+### Training
+- 40 epochs
+- 0.00707 base LR with sqrt_k scaling rule (equals to original 0.001 at batchsize = 2, equals 0.00489 at batchsize = 24)
+- AdamW with Weight Decay 0.001
+- CosineDecay Schedule
+- Batch Size 24 (Better result possible with lower batch size, batch size chosen for economical reasons.)
+
 ### Requirements
-- PyTorch >= 1.2 
-- yaml
-- Cython
-- [torch-scatter](https://github.com/rusty1s/pytorch_scatter)
+Also tested with CUDA 11.3, just "downgrade" pytorch, spconv and torch-scatter.
+
+Tested on Ubuntu 20.04 LTS. Recommend pip install over conda install.
+- Python 3.8
+- PyTorch == 1.12.1 --extra-index-url https://download.pytorch.org/whl/cu116
+- yaml == 6.0
+- strictyaml == 1.6.1
+- Cython == 0.29.30
+- tqdm == 4.64.0
+- [torch-scatter](https://github.com/rusty1s/pytorch_scatter) == cu116
 - [nuScenes-devkit](https://github.com/nutonomy/nuscenes-devkit) (optional for nuScenes)
-- [spconv](https://github.com/traveller59/spconv) (tested with spconv==1.2.1 and cuda==10.2)
+- [spconv-cu117 == 2.2.3](https://github.com/traveller59/spconv) (different CUDA versions available)
+- numba == 0.55.2 (install last, as this will likely downgrade your numpy version automatically)
 
 ## Data Preparation
 
@@ -54,67 +81,17 @@
 	    └── ...
 ```
 
-### nuScenes
-```
-./
-├── 
-├── ...
-└── path_to_data_shown_in_config/
-		├──v1.0-trainval
-		├──v1.0-test
-		├──samples
-		├──sweeps
-		├──maps
-
-```
-
 ## Training
 1. modify the config/semantickitti.yaml with your custom settings. We provide a sample yaml for SemanticKITTI
 2. train the network by running "sh train.sh"
 
-### Training for nuScenes
-Please refer to [NUSCENES-GUIDE](./NUSCENES-GUIDE.md)
-
-### Pretrained Models
--- We provide a pretrained model for SemanticKITTI [LINK1](https://drive.google.com/file/d/1q4u3LlQXz89LqYW3orXL5oTs_4R2eS8P/view?usp=sharing) or [LINK2](https://pan.baidu.com/s/1c0oIL2QTTcjCo9ZEtvOIvA) (access code: xqmi)
-
--- For nuScenes dataset, please refer to [NUSCENES-GUIDE](./NUSCENES-GUIDE.md)
-
-## Semantic segmentation demo for a folder of lidar scans
-```
-python demo_folder.py --demo-folder YOUR_FOLDER --save-folder YOUR_SAVE_FOLDER
-```
-If you want to validate with your own datasets, you need to provide labels.
---demo-label-folder is optional
-```
-python demo_folder.py --demo-folder YOUR_FOLDER --save-folder YOUR_SAVE_FOLDER --demo-label-folder YOUR_LABEL_FOLDER
-```
-
-## TODO List
-- [x] Release pretrained model for nuScenes.
-- [x] Support multiscan semantic segmentation.
-- [ ] Support more models, including PolarNet, RandLA, SequeezeV3 and etc.
-- [ ] Integrate LiDAR Panotic Segmentation into the codebase.
 
-## Reference
-
-If you find our work useful in your research, please consider citing our [paper](https://arxiv.org/pdf/2011.10033):
+If you find this work useful in your research, please consider citing the original authors [papers](https://arxiv.org/pdf/2011.10033):
 ```
 @article{zhu2020cylindrical,
   title={Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation},
   author={Zhu, Xinge and Zhou, Hui and Wang, Tai and Hong, Fangzhou and Ma, Yuexin and Li, Wei and Li, Hongsheng and Lin, Dahua},
   journal={arXiv preprint arXiv:2011.10033},
   year={2020}
 }
-
-#for LiDAR panoptic segmentation
-@article{hong2020lidar,
-  title={LiDAR-based Panoptic Segmentation via Dynamic Shifting Network},
-  author={Hong, Fangzhou and Zhou, Hui and Zhu, Xinge and Li, Hongsheng and Liu, Ziwei},
-  journal={arXiv preprint arXiv:2011.11964},
-  year={2020}
-}
 ```
-
-## Acknowledgments
-We thanks for the opensource codebases, [PolarSeg](https://github.com/edwardzhou130/PolarSeg) and [spconv](https://github.com/traveller59/spconv)

config/config.py

@@ -61,7 +61,9 @@
         "checkpoint_every_n_steps": Int(),
         "max_num_epochs": Int(),
         "eval_every_n_steps": Int(),
-        "learning_rate": Float()
+        "learning_rate": Float(),
+        "weight_decay": Float(),
+        "mixed_fp16": Bool()
      }
 )
 

config/semantickitti.yaml

@@ -66,3 +66,5 @@ train_params:
   max_num_epochs: 40
   eval_every_n_steps: 4599
   learning_rate: 0.001
+  weight_decay: 0.001
+  mixed_fp16: True