20210722 developer call notes #637

tomwhite · 2021-07-22T16:59:01Z

tomwhite
Jul 22, 2021
Maintainer

sgkit PR#620 (Tom) Issue a warning if the number of alt alleles exceeds the maximum specified
sgkit PR#629 (Tom) GWAS tutorial
- sgkit M#1 0.3.0 release

sgkit PR#620 (Tom) Issue a warning if the number of alt alleles exceeds the maximum specified

TW: Approach is now clear, just need to finish it. Follow on discussion about ragged arrays in https://github.com/pystatgen/sgkit/issues/634
JK: OK to have ALT field in a single chunk, can then process as needed.
TW: Agree - store as in VCF (comma-delimited), and also have a fixed variant_allele field containing first n alleles as we do now.
RW: Have you looked at awkward array? Will open issue to discuss more, with some links of previous work.
JK: Awkward array creator came to BDI to talk to Alistair.

sgkit PR#629 (Tom) GWAS tutorial

TW: Finished writing text, then had a long series of tasks to try to fix cyvcf2 problems (reading from https URLs), which resulted in a new cyvcf2 release, but didn’t fix the problem! So did a workaround to download file locally. Text is now ready for review.
JK: Will review.
TW: I will do a release next week.

sgkit#636 Data reading time in benchmark

LL: OK to compare tools each reading from own preferred format?
TW: Yes
JK: Yes
LL: Will read_vcf function be added?
TW: It could, but it might be confusing since you have to convert to Zarr and users might not realize how expensive that it.
JK: You should try to write the VCF only once, then run multiple analyses on that data.
TW: Unless you explicitly write out a new dataset, it won’t be updated on disk, so that works well.
LL: Is PLINK’s --freq the same as sgkit’s summary stats?
JK: hard to make plink do the same thing as we are. Pre-canned options. Aim for it to be as similar as possible.
LL: Are there any resources for how to run sgkit on a cluster?
TW: sgkit uses Dask, and Dask has lots of docs about how to run on a cluster.
- Dask cluster setup:
https://docs.dask.org/en/latest/setup.html
LL: Which functions to focus on?
TW: The original list. No particular order of preference
RW: Some resources about benchmarking:
- https://github.com/pystatgen/sgkit/discussions/528
- https://github.com/pystatgen/sgkit/discussions/481

VCF Zarr presentation for GA4GH Future of VCF Meeting (9 August 2021)

TW: I went to the meeting last week, and have been asked to give a 10-15 min presentation about VCF and Zarr. Will cover what Zarr is, some of the choices made in sgkit and scikit-allel to represent VCF in Zarr (chunks, compression, phasing, missing data, strings, etc). I will write up some slides and share beforehand for discussion.
JK: Would like to be able to simulate data and write to Zarr, so compression size etc can be benchmarked. Path is msprime -> tskit -> zarr. Need help with doing the last step in parallel.
TW: I can help with that.
TW: Indexing VCF. I think our approach is read POS into memory then do binary search to do range queries. Don’t need an index.
RW: Are partial Zarr chunks relevant? May come up in context of reading small numbers of variants. Partial chunks may help avoid reading whole chunk, which might otherwise be expensive compared to other formats.