Initial work toward PyTorch data loaders#1

Closed

bkmartinjr wants to merge 71 commits intomainfrom initial-pr

+3,967-1,015

Commits on Sep 18, 2024

initial commit of pytorch datapipe/loader

bkmartinjr
authored and
ryan-williams
committed
update comments

bkmartinjr
authored and
ryan-williams
committed
more lint

bkmartinjr
authored and
ryan-williams
committed
fix typos

bkmartinjr
authored and
ryan-williams
committed
rework for performance

bkmartinjr
authored and
ryan-williams
committed
tuning

bkmartinjr
authored and
ryan-williams
committed
tweaks, checkpoint

bkmartinjr
authored and
ryan-williams
committed
lint

bkmartinjr
authored and
ryan-williams
committed
py 3.8 lint

bkmartinjr
authored and
ryan-williams
committed
rework io and shuffle buffer size params

bkmartinjr
authored and
ryan-williams
committed
lint

bkmartinjr
authored and
ryan-williams
committed
remove encoders; more perf work

bkmartinjr
authored and
ryan-williams
committed
reorganize into separate python package

bkmartinjr
authored and
ryan-williams
committed
fix name

bkmartinjr
authored and
ryan-williams
committed
add more paths to CI

bkmartinjr
authored and
ryan-williams
committed
fix typo in ci

bkmartinjr
authored and
ryan-williams
committed
fix a second typo in ci

bkmartinjr
authored and
ryan-williams
committed
set working dir in CI

bkmartinjr
authored and
ryan-williams
committed
make batched 3.12 compat

bkmartinjr
authored and
ryan-williams
committed
debugging pre-commit failure

bkmartinjr
authored and
ryan-williams
committed
lint, lint, lint

bkmartinjr
authored and
ryan-williams
committed
more CI debugging

bkmartinjr
authored and
ryan-williams
committed
add build test to CI

bkmartinjr
authored and
ryan-williams
committed
add code coverage

bkmartinjr
authored and
ryan-williams
committed
update GHA

bkmartinjr
authored and
ryan-williams
committed
test TypeAlias

bkmartinjr
authored and
ryan-williams
committed
add missing dependencies

bkmartinjr
authored and
ryan-williams
committed
extend tests

bkmartinjr
authored and
ryan-williams
committed
remove coverage reporting from CI for now

bkmartinjr
authored and
ryan-williams
committed
docstrings

bkmartinjr
authored and
ryan-williams
committed
more file organization

bkmartinjr
authored and
ryan-williams
committed
add missing test

bkmartinjr
authored and
ryan-williams
committed
re-run notebook

bkmartinjr
authored and
ryan-williams
committed
update changelog

bkmartinjr
authored and
ryan-williams
committed
add collate unit test

bkmartinjr
authored and
ryan-williams
committed
clean up experiment_dataloader function

bkmartinjr
authored and
ryan-williams
committed
docstrings

bkmartinjr
authored and
ryan-williams
committed
fix typo in notebook name (thanks Ryan!)

bkmartinjr
authored and
ryan-williams
committed
checkpoint updates

bkmartinjr
authored and
ryan-williams
committed
update tests to include _CSR tests

bkmartinjr
authored and
ryan-williams
committed
fix typo in method name

bkmartinjr
authored and
ryan-williams
committed
tuning

bkmartinjr
authored and
ryan-williams
committed
update demo notebook

bkmartinjr
authored and
ryan-williams
committed
add to README

bkmartinjr
authored and
ryan-williams
committed
concurrency tweak

bkmartinjr
authored and
ryan-williams
committed
additional memory reductions

bkmartinjr
authored and
ryan-williams
committed
DDP/multi-GPU support

bkmartinjr
authored and
ryan-williams
committed
add further concurrency to CSR construction

bkmartinjr
authored and
ryan-williams
committed
cleanup

bkmartinjr
authored and
ryan-williams
committed
fix multi-gpu hang due to incorrect __len__ return value

bkmartinjr
authored and
ryan-williams
committed
compat with Lightning

bkmartinjr
authored and
ryan-williams
committed
PR review edits

bkmartinjr
authored and
ryan-williams
committed
formatting

bkmartinjr
authored and
ryan-williams
committed
add py.typed to package

bkmartinjr
authored and
ryan-williams
committed
add sparse support

bkmartinjr
authored and
ryan-williams
committed
start draft of Ligtning notebook

bkmartinjr
authored and
ryan-williams
committed
lint

bkmartinjr
authored and
ryan-williams
committed
update notebook for lightning

bkmartinjr
authored and
ryan-williams
committed
run notebooks

bkmartinjr
authored and
ryan-williams
committed
fix RNG state bug in shuffle; add multi-worker notebook

bkmartinjr
authored and
ryan-williams
committed
add rehome-census.sh, used to construct this repo's history
ryan-williams
committed
update GHA, repo info
ryan-williams
committed
add .pre-commit-config.yaml, run lint
ryan-williams
committed