How do I use DVC with SSH remote

Q: I’m just getting started with DVC, but I’d like to use it for multiple developers to access the data and share models and code.

I do own the server, but I’m not sure how to use DVC with SSH remote?


More context to this question and original versions are here.

1 Like

A: SSH is a protocol to connect/logging to a remote computer, and it is commonly used to execute commands or transfer files on a remote machine.

For that to work, you’ll need an SSH client and a server. You can type ssh -V on your terminal to see if you have the client already installed (normally included in OSX or Linux distributions), if you are on windows, the most used client is PuTTY

Now, on the computer that you want to reach, you should have the SSH server, usually sshd (or SSH Daemon), if you are running an operative system with systemd you can check if it is running by executing on the remote server systemctl status sshd (and it should say “Active”).

If everything went well, and you have both, a client on your computer and a server on the other computer that you want to connect to, you can use the client (either ssh or putty by specifying the address of the remote machine - an IP or hostname that resolves to the desire destination).

DVC needs to know the same information in order to establish a connection, and the way you configure it is by using the remote add/modify command. Let’s say that we are using the following configurations:

hostname/ip: example.com
user: mroutis
password: 123456789
port: 22

So the way I can configure the SSH remote with DVC will be the following:

dvc remote add --default ssh-storage ssh://example.com/path/to/storage
dvc remote modify ssh-storage user mroutis
dvc remote modify ssh-storage port 22
dvc remote modify --local ssh-storage password 123456789

:warning: Notice the --local on the last command, this is important, because the dvc remote command modifies the .dvc/config file and this file is stored with Git, so, you can end up with this information on your GitHub/BitBucket/GitLab or whatever you use to distribute your code repository.

Instead of password the recommended way is to use SSH keys.

There’s a difference between the host example.com and the URL I’m passing to DVC ssh://example.com/path/to/storage DVC needs a URL with the following standard: scheme://host/path, in this case, the scheme is ssh, the host is the one we already have, and the path is the absolute path were DVC will be uploading the cache.

DVC will be uploading ( push ) and downloading ( pull) files from your remote, the URL ssh://hostname/path/to/storage is just a way to tell DVC in which directory it will store the data. You can connect to your server with PuTTY and then create a directory with the following command mkdir /tmp/dvc-storage , now, if you configure DVC with the following URL ssh://your-host-name-or-ip/tmp/dvc-storage when you dvc push it will upload files to that directory that you created previously /tmp/dvc-storage and when doing dvc pull it will connect to the remote computer via SSH, go to that directory ( /tmp/dvc-storage/ ) and then download the files via SSH.

1 Like

I don’t think this answers the original question, which is how to set up DVC so that MANY DIFFERENT users can access the ssh server with their own accounts. You just provided an explanation on how to set up the ssh remote. However, your step for the local password is new to me, and might be useful somehow? Can people set up the user for ssh as local as well? Something like
dvc remote modify --local ssh-storage user mroutis
???

1 Like

Hi @fercook !

Yes, you could use --local that way to setup different users, sure. E.g. you would do:

dvc remote modify --local ssh-storage user fercook
dvc remote modify --local ssh-storage password 123456789

and your collegue would do

dvc remote modify --local ssh-storage user mroutis
dvc remote modify --local ssh-storage password 987654321

 Let us know if you have any other questions :slightly_smiling_face:

Do I need to have DVC installed on the remote machine for it to work, or just locally?

P.S.: thanks @shcheklein for this tutorial.

Just locally, @arthurcgusmao. Please refer to https://dvc.org/doc/start/data-versioning#storing-and-sharing for an updated intro to DVC remote storage. Thanks

1 Like

@shcheklein, I wasn’t able to replicate the remote repo configuration using the argument --local as in your answer:

dvc remote add --default ssh-storage ssh://example.com/path/to/storage
dvc remote modify ssh-storage user mroutis
dvc remote modify ssh-storage port 22
dvc remote modify --local ssh-storage password 123456789

When using --local only for configuring the password, DVC says:

ERROR: configuration error - config file error: remote 'ssh-storage' doesn't exist.

What seems to be happening is that DVC looks only on the local config file in that case, disregarding the default (non-local) config.

One can, however, work around that by reconfiguring the steps before the password with the --local argument.

Should I open an issue regarding this or is it the new expected behavior?

Hi @arthurcgusmao! This is a known issue and it will be fixed in the next release in a few days - https://github.com/iterative/dvc/issues/4276 (fix is already merged).

The workaround is to run (along with the regular dvc remote add):

dvc remote add --local ssh-storage ssh://example.com/path/to/storage

to add an entry, so that it is not complaining.

2 Likes

@arthurcgusmao 1.3.1 is out with the fix. Please upgrade and give it a try :slightly_smiling_face:

2 Likes

Hey, I have tried this:
dvc remote add --default ssh-storage ssh://example.com/path/to/storage dvc remote modify ssh-storage user mroutis dvc remote modify ssh-storage port 22 dvc remote modify --local ssh-storage password 123456789

and then this:
dvc remote modify --local ssh-storage user fercook dvc remote modify --local ssh-storage password 123456789

My goal is to use ssh instead of logging in to my account every time I want to do for instance dvc list. After doing the above commands, I still have to log in every time to my github account when I execute a dvc command.

This command by shcheklein gives me the following error:
dvc remote add --local ssh-storage ssh://dvc-data/dataset-registry
ERROR: configuration error - config file error: remote 'ssh-storage' already exists. Use -f|–force to overwrite it.

I have installed dvc with pip, because installing it in Arch with yay failed, as many files were not found in the mirrors. Now that I try to upgrade with pip, I can only get version 1.11 and not 1.3.

How should I proceed? Thanks!

1 Like

Hi @Konstantina!

You have a few things happening at once. No worries, let me try to help you out of it step by step.

  1. SSH storage (dvc remote * set of commands) are not about GitHub access. They are about accessing some server that will be used to store your data files. When you do dvc push, dvc pull, etc

  2. GitHub access (e.g. dvc list might require it internally to access the repository to read .dvc files, etc). Here you don’t need anything special from the DVC perspective. You should setup GH itself to do this - Connecting to GitHub with SSH

Please git it all a try and let us know if that works or not.