20240304 developer call notes #1206

hammer · 2024-03-04T17:02:23Z

hammer
Mar 4, 2024
Maintainer

JK’s project “Improved methods for storing and processing large-scale genetic variation data” accepted by Genomics England (project 1062)
Amit’s talk at PyCon Slovakia

Bio2zarr
- Ben ran on GEL and it works!
- Converted 140 GB chunk from chr20 in a few hours
- Each chromosome has ~30 chunks
- JK’s project approval means we can use it in the paper!
- PLINK conversion is in!
- JK: envision a future that could include FASTA e.g.
sgkit PR#1190
- Waiting on Tim’s thumbs up
sgkit PR#1199
- Tim wants us to disable coverage on gu vectorized functions
- https://github.com/pystatgen/sgkit/issues/1194#issuecomment-1976027503
sgkit PR#1205
- Will drop support for Python 3.8 to solve it
0.8.0 release
- Should we fix bugs in old VCF parser?
- JK and TW don’t have time now
- No big urgency on release, so just wait
- When do we want to cut over to bio2zarr?
- TW: what is the feature gap between sgkit io and bio2zarr?
- JK: feature names, truncating based on allele number, etc.
sgkit#1167
- JH will pick it up
Paper
- JK wants to write only about Zarr specification and conversion
- JH: do we need spec for PLINK -> Zarr and the Zarr format itself
- TW: PLINK is really a subset of the Zarr VCF format
- JH: still want a paper on analysis capabilities including QC at scale
- JK: will make a new repo for this more tightly scoped paper
- JK: Will need point release of VCF spec, and 0.8.0 release of sgkit
- JH: do we need to do a pass through Zarr v3?
- TM: VCF 4.4 came out recently
- JK: PL fields do not work well with 1000G data
- TW: just point to local alleles as something to do
- JK: GEL data is biallelic fortunately!