I think I must be doing something wrong, because when adding new remote files they seem to always get rehashed “only once” several times, which really hurts because they’re big.
One example below with dvc 3.18, you can see it was hashed twice, the scp’ed, then hashed again, each of these seemed to transfer the data once, so essentially 4X slower than needed. Then there was one final local (fast) hash. Dvc file attached below.
Any way to improve this? Thanks.
stages: download_mockobs: cmd: scp scott.grid.uchicago.edu:/sptlocal/analysis/eete+lensing_19-20/resources/sims/planck2018/mockobs/220.127.116.11.1/output/flatsky/seed2/Coadd_allfields_090ghz_flatsky.g3 Coadd_allfields_090ghz_flatsky.g3 deps: - ssh://scott.grid.uchicago.edu/sptlocal/analysis/eete+lensing_19-20/resources/sims/planck2018/mockobs/18.104.22.168.1/output/flatsky/seed2/Coadd_allfields_090ghz_flatsky.g3 outs: - Coadd_allfields_090ghz_flatsky.g3