Hello,
I’m in a team of Computer Vision Engineers, and I’m working on improving ours datasets’ management.
Until now, we use git LFS to track our dataset. We reach its limits, as we get huge repository, long pull time. We also want a more secure way to store ours datasets, that could be accessible only by authorize persons. It’s important to precise that our datasets are living, often modified with content added or removed, and we currently do it with PR system.
To improve our process, I found out that DVC may answer some of our issues, and also provides more features afterward (experiments, pipeline…)
Documents are sent to the remote, that will be accessible only by authorized people, so it solves the security part.
DVC is also better for huge files management, I won’t say more about it, I think you already know this.
Now I’ve explained my case, I have some questions:
Relating to PR, I’m aware that it will no longer be possible to use our Git web interface to check PR before approving them, as we won’t see file content change (We also have annotated JSON/TXT files that will be tracked by DVC, it’s not huge file, but they may contain sensitive data, so we prefer to add them in DVC remote for security purpose).
What are best practices about pull requests review with DVC ? DVC is made for git and for versioning, so I suppose it exists a simple way to check PR ? Or do we have to check locally, by pulling everything ?
Another point, we want to be able to easily know what files have changed, so I’m planning on using dvc add with --recursive flag, to ensure each file is tracked individually. I saw on documentation that it’s better to avoid this flag when a folder contains numerous files. What are the limits ? How it will impact performances ?
Thanks !