Hi, I have a pipeline with many stages that write the output in different subdirectories of the same dir.
So basically each stage has as input a dir like this:
- input-dir
- file1
- file2
- fileN
The nth stage writes its output:
- output-dir (same for all the stages)
- results-file1
- output-stage1
- output-stage2
- output-stageN
- results-file2
.
.
.
- results-file1
Actually, results-file* is a zarr group (Groups (zarr.hierarchy) — zarr 2.6.1 documentation), whereas each stage appends new arrays (that correspond to output-stage* dirs).
Is there a way to achieve these with dvc? the command dvc run
complains the output is already specified in another stage.
Thanks in advance