Merge remote-tracking branch 'origin/main' into add_ceph

.gitattributes

@@ -0,0 +1,5 @@
+*.png filter=lfs diff=lfs merge=lfs -text
+*.svg filter=lfs diff=lfs merge=lfs -text
+*.odg filter=lfs diff=lfs merge=lfs -text
+*.gif filter=lfs diff=lfs merge=lfs -text
+*.ico filter=lfs diff=lfs merge=lfs -text

.github/workflows/mkdocs-deploy.yml

@@ -9,12 +9,14 @@ jobs:
   deploy:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
+        with:
+          lfs: true
 
-      - name: Setup Python 3.10
-        uses: actions/setup-python@v4
+      - name: Setup Python 3.11
+        uses: actions/setup-python@v5
         with:
-          python-version: '3.10'
+          python-version: '3.11'
           cache: "pipenv"
           cache-dependency-path: |
             Pipfile.lock

.github/workflows/mkdocs-test.yml

@@ -8,12 +8,14 @@ jobs:
   check:
     runs-on: ubuntu-latest
     steps:
-      - uses: actions/checkout@v3
+      - uses: actions/checkout@v4
+        with:
+          lfs: true
 
-      - name: Setup Python 3.10
-        uses: actions/setup-python@v4
+      - name: Setup Python 3.11
+        uses: actions/setup-python@v5
         with:
-          python-version: '3.10'
+          python-version: '3.11'
           cache: "pipenv"
           cache-dependency-path: |
             Pipfile.lock

Pipfile

@@ -14,4 +14,4 @@ pymdown-extensions = "*"
 mkdocs-git-revision-date-localized-plugin = "*"
 
 [requires]
-python_version = "3.10"
+python_version = "3.11"

bih-cluster/docs/admin/getting-access.md

@@ -52,7 +52,7 @@ The process to create a new group is as follows.
 
 Subsequently, both owner and delegate (but **only owner and delegate**) can initiate changes (new users, resource changes etc.) to the group.
 
-## Form
+### Form
 
 Example values are given in curly braces.
 
@@ -90,7 +90,7 @@ Required resources:
 [etc.]
 ```
 
-## Notes
+### Notes
 
 - Work groups on the cluster must have an owner (its leader, principal investigator, etc.)
 - Group ownership implies control but also accountability for their work group and members.
@@ -118,7 +118,7 @@ Project creation can be initiated by group leaders and group delegates with the
 
 Subsequently, both owner and delegate can initiate changes (new users, resource changes etc.) to the project. 
 
-## Form
+### Form
 
 Example values are given in curly braces.
 
@@ -146,6 +146,8 @@ If you wish to add users to your AG, please use the following form. Note that us
 
 The inquiry has to be send to hpc-helpdesk@bih-charite.de, either by the PI or the delegate.
 
+### Form
+
 ```
 # User 1
 

bih-cluster/docs/best-practice/project-structure.md

@@ -71,7 +71,7 @@ Creating the work directory and copy the input files into `work/input`.
 
 ```bash
 $ mkdir -p project/work/input
-$ cp /fast/projects/cubit/tutorial/input/* project/work/input
+$ cp /data/cephfs-1/work/projects/cubit/tutorial/input/* project/work/input
 ```
 
 Creating the home space.
@@ -159,7 +159,7 @@ EOF
 $ chmod +x scripts/run-mapping.sh
 $ mkdir -p config
 $ cat <<"EOF" >config/default-config.sh
-BWA_INDEX=/fast/projects/cubit/current/static_data/reference/GRCh37/hs37d5/hs37d5.fa
+BWA_INDEX=/data/cephfs-1/work/projects/cubit/current/static_data/reference/GRCh37/hs37d5/hs37d5.fa
 SAMPLES=
 EOF
 $ cat <<"EOF" >config/project-config.sh

bih-cluster/docs/cubit/index.md

@@ -1,6 +1,6 @@
 # Overview
 
-The static data installation can be found at  `/fast/projects/cubit/18.12/static_data`.
+The static data installation can be found at  `/data/cephfs-1/work/projects/cubit/18.12/static_data`.
 
 The static data directory contains a sub-directory for the genomes, the precomputed index files for several different popular mapping tools and associated annotation (GFF and GTF files) from Ensembl and GENCODE for each of the available genomes.
 The top-level directory structure is as follows:

bih-cluster/docs/first-steps/episode-0.md

@@ -23,7 +23,7 @@ introduce the following convention that we use throughout the series:
 $ Commands are prefixed with a little dollar sign
 ```
 
-While file paths are highlighted like this: `/fast/projects/cubit/current`.
+While file paths are highlighted like this: `/data/cephfs-1/work/projects/cubit/current`.
 
 ## Instant Gratification
 

bih-cluster/docs/first-steps/episode-1.md

@@ -22,8 +22,8 @@ $ srun --time 7-00 --mem=8G --ntasks=8 --pty bash -i
 We will provide you with some example FASTQ files, but you can use your own if you like.
 You can find the data here:
 
-- `/fast/projects/cubit/work/tutorial/input/test_R1.fq.gz`
-- `/fast/projects/cubit/work/tutorial/input/test_R2.fq.gz`
+- `/data/cephfs-1/work/projects/cubit/tutorial/input/test_R1.fq.gz`
+- `/data/cephfs-1/work/projects/cubit/tutorial/input/test_R2.fq.gz`
 
 ## Creating a Project Directory
 
@@ -57,11 +57,11 @@ You can create one in your `~/scratch` folder and make it available to the syste
 
 ## Using the Cubit Static Data
 
-The static data is located in `/fast/projects/cubit/current/static_data`.
+The static data is located in `/data/cephfs-1/work/projects/cubit/current/static_data`.
 For our small example, the required reference genome and index can be found at:
 
-- `/fast/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta`
-- `/fast/projects/cubit/current/static_data/precomputed/BWA/0.7.17/GRCh37/g1k_phase1/human_g1k_v37.fasta`
+- `/data/cephfs-1/work/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta`
+- `/data/cephfs-1/work/projects/cubit/current/static_data/precomputed/BWA/0.7.17/GRCh37/g1k_phase1/human_g1k_v37.fasta`
 
 ## Aligning the Reads
 
@@ -70,9 +70,9 @@ Let's align our data:
 ```terminal
 (first-steps) $ bwa mem -t 8 \
     -R "@RG\tID:FLOWCELL.LANE\tPL:ILLUMINA\tLB:test\tSM:PA01" \
-    /fast/projects/cubit/current/static_data/precomputed/BWA/0.7.17/GRCh37/g1k_phase1/human_g1k_v37.fasta \
-    /fast/projects/cubit/work/tutorial/input/test_R1.fq.gz \
-    /fast/projects/cubit/work/tutorial/input/test_R2.fq.gz \
+    /data/cephfs-1/work/projects/cubit/current/static_data/precomputed/BWA/0.7.17/GRCh37/g1k_phase1/human_g1k_v37.fasta \
+    /data/cephfs-1/work/projects/cubit/tutorial/input/test_R1.fq.gz \
+    /data/cephfs-1/work/projects/cubit/tutorial/input/test_R2.fq.gz \
 | samtools view -b \
 | samtools sort -O BAM -T $TMPDIR -o aln.bam
 
@@ -85,7 +85,7 @@ And do the structural variant calling:
 
 ```terminal
 (first-steps) $ delly call \
-    -g /fast/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta \
+    -g /data/cephfs-1/work/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta \
     aln.bam
 ```
 
@@ -97,7 +97,7 @@ And now for the SNP calling (this step will take ~ 20 minutes):
 
 ```terminal
 (first-steps) $ gatk HaplotypeCaller \
-    -R /fast/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta \
+    -R /data/cephfs-1/work/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta \
     -I aln.bam \
     -ploidy 2 \
     -O test.GATK.vcf
@@ -111,5 +111,5 @@ You can access a list of all static data through this wiki, follow this link to
 You can also have a peek via:
 
 ```terminal
-(first-steps) $ tree -L 3 /fast/projects/cubit/current/static_data | less
+(first-steps) $ tree -L 3 /data/cephfs-1/work/projects/cubit/current/static_data | less
 ```

bih-cluster/docs/first-steps/episode-2.md

@@ -33,7 +33,7 @@ You may have noticed that you run `sbatch` with a script, not with regular comma
 The reason is that `sbatch` only accepts bash scripts.
 If you give `sbatch` a normal shell command or binary, it won't work.
 This means that we have to put the command(s) we want to use in a bash script.
-A skeleton script can be found at `/fast/projects/cubit/work/tutorial/skeletons/submit_job.sh`
+A skeleton script can be found at `/data/cephfs-1/work/projects/cubit/tutorial/skeletons/submit_job.sh`
 
 The content of the file:
 
@@ -79,7 +79,7 @@ and copy the wrapper script to this directory:
 
 ```terminal
 (first-steps) $ pushd /fast/users/$USER/work/tutorial/episode2
-(first-steps) $ cp /fast/projects/cubit/work/tutorial/skeletons/submit_job.sh .
+(first-steps) $ cp /data/cephfs-1/work/projects/cubit/tutorial/skeletons/submit_job.sh .
 (first-steps) $ chmod u+w submit_job.sh
 ```
 
@@ -119,14 +119,14 @@ Your file should look something like this:
 export TMPDIR=/fast/users/${USER}/scratch/tmp
 mkdir -p ${TMPDIR}
 
-BWAREF=/fast/projects/cubit/current/static_data/precomputed/BWA/0.7.17/GRCh37/g1k_phase1/human_g1k_v37.fasta
-REF=/fast/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta
+BWAREF=/data/cephfs-1/work/projects/cubit/current/static_data/precomputed/BWA/0.7.17/GRCh37/g1k_phase1/human_g1k_v37.fasta
+REF=/data/cephfs-1/work/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta
 
 bwa mem -t 8 \
     -R "@RG\tID:FLOWCELL.LANE\tPL:ILLUMINA\tLB:test\tSM:PA01" \
     $BWAREF \
-    /fast/projects/cubit/work/tutorial/input/test_R1.fq.gz \
-    /fast/projects/cubit/work/tutorial/input/test_R2.fq.gz \
+    /data/cephfs-1/work/projects/cubit/tutorial/input/test_R1.fq.gz \
+    /data/cephfs-1/work/projects/cubit/tutorial/input/test_R2.fq.gz \
 | samtools view -b \
 | samtools sort -O BAM -T $TMPDIR -o aln.bam
 

bih-cluster/docs/first-steps/episode-3.md

@@ -32,7 +32,7 @@ copy the skeleton:
 ```terminal
 (first-steps) $ mkdir -p /fast/users/${USER}/work/tutorial/episode3
 (first-steps) $ pushd /fast/users/${USER}/work/tutorial/episode3
-(first-steps) $ cp /fast/projects/cubit/work/tutorial/skeletons/Snakefile .
+(first-steps) $ cp /data/cephfs-1/work/projects/cubit/tutorial/skeletons/Snakefile .
 (first-steps) $ chmod u+w Snakefile
 ```
 
@@ -46,8 +46,8 @@ rule all:
 
 rule alignment:
     input:
-        '/fast/projects/cubit/work/tutorial/input/test_R1.fq.gz',
-        '/fast/projects/cubit/work/tutorial/input/test_R2.fq.gz',
+        '/data/cephfs-1/work/projects/cubit/tutorial/input/test_R1.fq.gz',
+        '/data/cephfs-1/work/projects/cubit/tutorial/input/test_R2.fq.gz',
     output:
         bam='alignment/test.bam',
         bai='alignment/test.bam.bai',
@@ -56,7 +56,7 @@ rule alignment:
         export TMPDIR=/fast/users/${{USER}}/scratch/tmp
         mkdir -p ${{TMPDIR}}
 
-        BWAREF=/fast/projects/cubit/current/static_data/precomputed/BWA/0.7.17/GRCh37/g1k_phase1/human_g1k_v37.fasta
+        BWAREF=/data/cephfs-1/work/projects/cubit/current/static_data/precomputed/BWA/0.7.17/GRCh37/g1k_phase1/human_g1k_v37.fasta
 
         bwa mem -t 8 \
             -R "@RG\tID:FLOWCELL.LANE\tPL:ILLUMINA\tLB:test\tSM:PA01" \
@@ -75,7 +75,7 @@ rule structural_variants:
         'structural_variants/test.vcf'
     shell:
         r"""
-        REF=/fast/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta
+        REF=/data/cephfs-1/work/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta
 
         delly call -o {output} -g ${{REF}} {input}
         """
@@ -87,7 +87,7 @@ rule snps:
         'snps/test.vcf'
     shell:
         r"""
-        REF=/fast/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta
+        REF=/data/cephfs-1/work/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta
 
         gatk HaplotypeCaller \
             -R ${{REF}} \
@@ -147,8 +147,8 @@ rule all:
 
 rule alignment:
     input:
-        '/fast/projects/cubit/work/tutorial/input/{id}_R1.fq.gz',
-        '/fast/projects/cubit/work/tutorial/input/{id}_R2.fq.gz',
+        '/data/cephfs-1/work/projects/cubit/tutorial/input/{id}_R1.fq.gz',
+        '/data/cephfs-1/work/projects/cubit/tutorial/input/{id}_R2.fq.gz',
     output:
         bam='alignment/{id}.bam',
         bai='alignment/{id}.bam.bai',
@@ -157,7 +157,7 @@ rule alignment:
         export TMPDIR=/fast/users/${{USER}}/scratch/tmp
         mkdir -p ${{TMPDIR}}
 
-        BWAREF=/fast/projects/cubit/current/static_data/precomputed/BWA/0.7.17/GRCh37/g1k_phase1/human_g1k_v37.fasta
+        BWAREF=/data/cephfs-1/work/projects/cubit/current/static_data/precomputed/BWA/0.7.17/GRCh37/g1k_phase1/human_g1k_v37.fasta
 
         bwa mem -t 8 \
             -R "@RG\tID:FLOWCELL.LANE\tPL:ILLUMINA\tLB:test\tSM:PA01" \
@@ -176,7 +176,7 @@ rule structural_variants:
         'structural_variants/{id}.vcf'
     shell:
         r"""
-        REF=/fast/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta
+        REF=/data/cephfs-1/work/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta
 
         delly call -o {output} -g ${{REF}} {input}
         """
@@ -188,7 +188,7 @@ rule snps:
         'snps/{id}.vcf'
     shell:
         r"""
-        REF=/fast/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta
+        REF=/data/cephfs-1/work/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta
 
         gatk HaplotypeCaller \
             -R ${{REF}} \

bih-cluster/docs/first-steps/episode-4.md

@@ -27,8 +27,8 @@ First, create a new folder for this episode:
 And copy the wrapper script to this folder as well as the Snakefile (you can also reuse the one with the adjustments from the previous [episode](episode-3.md)):
 
 ```terminal
-(first-steps) $ cp /fast/projects/cubit/work/tutorial/skeletons/submit_snakejob.sh .
-(first-steps) $ cp /fast/projects/cubit/work/tutorial/skeletons/Snakefile .
+(first-steps) $ cp /data/cephfs-1/work/projects/cubit/tutorial/skeletons/submit_snakejob.sh .
+(first-steps) $ cp /data/cephfs-1/work/projects/cubit/tutorial/skeletons/Snakefile .
 (first-steps) $ chmod u+w submit_snakejob.sh Snakefile
 ```
 
@@ -109,8 +109,8 @@ rule all:
 
 rule alignment:
     input:
-        '/fast/projects/cubit/work/tutorial/input/{id}_R1.fq.gz',
-        '/fast/projects/cubit/work/tutorial/input/{id}_R2.fq.gz',
+        '/data/cephfs-1/work/projects/cubit/tutorial/input/{id}_R1.fq.gz',
+        '/data/cephfs-1/work/projects/cubit/tutorial/input/{id}_R2.fq.gz',
     output:
         bam='alignment/{id}.bam',
         bai='alignment/{id}.bam.bai',
@@ -123,7 +123,7 @@ rule alignment:
         export TMPDIR=/fast/users/${{USER}}/scratch/tmp
         mkdir -p ${{TMPDIR}}
 
-        BWAREF=/fast/projects/cubit/current/static_data/precomputed/BWA/0.7.17/GRCh37/g1k_phase1/human_g1k_v37.fasta
+        BWAREF=/data/cephfs-1/work/projects/cubit/current/static_data/precomputed/BWA/0.7.17/GRCh37/g1k_phase1/human_g1k_v37.fasta
 
         bwa mem -t 8 \
             -R "@RG\tID:FLOWCELL.LANE\tPL:ILLUMINA\tLB:test\tSM:PA01" \
@@ -146,7 +146,7 @@ rule structural_variants:
         time='2-00:00:00',
     shell:
         r"""
-        REF=/fast/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta
+        REF=/data/cephfs-1/work/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta
 
         delly call -o {output} -g ${{REF}} {input}
         """
@@ -166,7 +166,7 @@ rule snps:
         time='04:00:00',
     shell:
         r"""
-        REF=/fast/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta
+        REF=/data/cephfs-1/work/projects/cubit/current/static_data/reference/GRCh37/g1k_phase1/human_g1k_v37.fasta
 
         gatk HaplotypeCaller \
             -R ${{REF}} \

bih-cluster/docs/how-to/software/cell-ranger.md

@@ -18,7 +18,7 @@ tar -xzvf cellranger-3.0.2.tar.gz
 
 # reference data
 
-will be provided in `/fast/projects/cubit/current/static_data/app_support/cellranger`
+will be provided in `/data/cephfs-1/work/projects/cubit/current/static_data/app_support/cellranger`
 
 # cluster support SLURM
 
@@ -76,7 +76,7 @@ create a script `run_cellranger.sh` with these contents (consult the [documentat
 
 /fast/users/$USER/work/cellranger-3.0.2/cellranger count \
   --id=sample_id \
-  --transcriptome=/fast/projects/cubit/current/static_data/app_support/cellranger/refdata-cellranger-${species}-3.0.0\
+  --transcriptome=/data/cephfs-1/work/projects/cubit/current/static_data/app_support/cellranger/refdata-cellranger-${species}-3.0.0\
   --fastqs=/path/to/fastqs \
   --sample=sample_name \
   --expect-cells=n_cells \

bih-cluster/docs/misc/publication-list.md

@@ -8,6 +8,10 @@ It has been used to support the publications listed below.
 
 ## Articles & Preprints
 
+### 2024
+
+Hollunder, B., Ostrem, J.L., Sahin, I.A., Rajamani, N., Oxenford, S., Butenko, K., Neudorfer, C., Reinhardt, P., Zvarova, P., Polosan, M., Akram, H., Vissani, M., Zhang, C., Sun, B., Navratil, P., Reich, M.M., Volkmann, J., Yeh, F.-C., Baldermann, J.C., Dembek, T.A., Visser-Vandewalle, V., Alho, E.J.L., Franceschini, P.R., Nanda, P., Finke, C., Kühn, A.A., Dougherty, D.D., Richardson, R.M., Bergman, H., DeLong, M.R., Mazzoni, A., Romito, L.M., Tyagi, H., Zrinzo, L., Joyce, E.M., Chabardes, S., Starr, P.A., Li, N., Horn, A., 2024. Mapping dysfunctional circuits in the frontal cortex using deep brain stimulation. Nat. Neurosci. 1–14. [doi: 10.1038/s41593-024-01570-1](https://doi.org/10.1038/s41593-024-01570-1)
+
 ### 2022
 
 Kossen T, Hirzel MA, Madai VI, Boenisch F, Hennemuth A, Hildebrand K, Pokutta S, Sharma K, Hilbert A, Sobesky J, Galinovic I, Khalil AA, Fiebach JB and Frey D. Toward Sharing Brain Images: Differentially Private TOF-MRA Images With Segmentation Labels Using Generative Adversarial Networks. *Frontiers in