Is it possible to mount a remote DVC data registry in a local filesystem?

Hi! This is my first post here. I’ve been looking through the docs and topics to find out a best solution.
I think I almost got everything clear in terms of setup. We are currently using a DVC data registry (i.e. a DVC+GIT in charge of tracking changes in a remote storage). We can import this data registry into other DVC projects that just read data from the remote storage.

To reduce data storage (i.e. not creating another cache for each project importing the data registry that includes tracking changes in the data registry — these changes are already taken care of by the data registry itself) we wanted to use the data registry just as a filesystem where we specify the version (so that still the dependency of the code and the version of the registry is kept — though with a somewhat manual supervision instead of automatically).

We find the DVCFileSystem API great for this as this exactly what we need. However it only provides support for python. Furthermore, some python libraries only work with local paths. Is it possible to mount such a DVC filesystem at the OS level. Something with the same functionality as the python API but working in the same way than sshfs for example (with the addition of the revision that we want to mount)?

Thank you!

Hey Pablo,

We don’t provide our own FUSE-filesystem, but DVCFileSystem is a fsspec-compatible filesystem that does have support for it Features of fsspec — fsspec 2022.11.0+27.gb4661bb.dirty documentation I have to disclose that I never tried that feature of fsspec myself and I’m not sure if it will even work with dvcfs or how stable/performant it is going to be, but it is something that you could at least give a try to. We’ll be happy to hear about your experience with it too.

1 Like

I’ll definitely give it a go and let you know! Many thanks!

1 Like