Get files directly from remote

[qwirdoti]

Is there a way pull files directly from a dvc-managed remote (Google Cloud buckets)? At best, for a specific revision?

For our use case, DVC is managing a dataset composed of media files and json metadata within a common top-level directory, stored in a remote located on a Google Cloud bucket.
Now, consumers of this dataset (outside the DVC git repo) need to pull the data from within Docker containers, which run on Vertex AI.
We can use dvc get <remote-dir-path> to pull the data, but this requires git credentials. If possible, we want to avoid managing SSH keys in our containers. The command dvc get-url <remote-dir-path> gives us the hashed directory. Is there a dvc-internal approach to restore the data directly from the remote, without git credentials (but gcloud auth)? This will probably mean we would lose the option to pull specific revisions, right?

Thanks ahead for any hints!

[shcheklein]

Hi @qwirdoti !

• You could consider using DVC + DataChain approach GitHub - shcheklein/example-datachain-dvc: An example how to use DataChain and DVC to version data, make project reproducible, track experiments and models. DataChain is doing a snapshot of data in the cloud (regular, human-readable format, not DVC). DVC is versioning a parquet file instead which has reference to the specific objects you need. This approach has even more capabilities moving forward (e.g. proper metadata management, etc). I can walk you through it if you want.

• You could read the file that dvc-get url gives you (directory). It’s a JSON AFAIR that has pretty much files names, directory structure + hashes. Hash determines exactly the name in the DVC remote storage.

Let me know if that makes sense.