Q: I’m just getting started with DVC, but I’d like to use it for multiple developers to access the data and share models and code.
I do own the server, but I’m not sure how to use DVC with SSH remote?
More context to this question and original versions are here.
A: SSH is a protocol to connect/logging to a remote computer, and it is commonly used to execute commands or transfer files on a remote machine.
For that to work, you’ll need an SSH client and a server. You can type ssh -V on your terminal to see if you have the client already installed (normally included in OSX or Linux distributions), if you are on windows, the most used client is PuTTY
Now, on the computer that you want to reach, you should have the SSH server, usually sshd (or SSH Daemon), if you are running an operative system with systemd you can check if it is running by executing on the remote server systemctl status sshd (and it should say “Active”).
If everything went well, and you have both, a client on your computer and a server on the other computer that you want to connect to, you can use the client (either ssh or putty by specifying the address of the remote machine - an IP or hostname that resolves to the desire destination).
DVC needs to know the same information in order to establish a connection, and the way you configure it is by using the remote add/modify command. Let’s say that we are using the following configurations:
hostname/ip: example.com
user: mroutis
password: 123456789
port: 22
So the way I can configure the SSH remote with DVC will be the following:
dvc remote add --default ssh-storage ssh://example.com/path/to/storage
dvc remote modify ssh-storage user mroutis
dvc remote modify ssh-storage port 22
dvc remote modify --local ssh-storage password 123456789
Notice the --local on the last command, this is important, because the dvc remote command modifies the .dvc/config file and this file is stored with Git, so, you can end up with this information on your GitHub/BitBucket/GitLab or whatever you use to distribute your code repository.
Instead of password the recommended way is to use SSH keys.
There’s a difference between the host example.com and the URL I’m passing to DVC ssh://example.com/path/to/storage DVC needs a URL with the following standard: scheme://host/path, in this case, the scheme is ssh, the host is the one we already have, and the path is the absolute path were DVC will be uploading the cache.
DVC will be uploading ( push ) and downloading ( pull) files from your remote, the URL ssh://hostname/path/to/storage is just a way to tell DVC in which directory it will store the data. You can connect to your server with PuTTY and then create a directory with the following command mkdir /tmp/dvc-storage , now, if you configure DVC with the following URL ssh://your-host-name-or-ip/tmp/dvc-storage when you dvc push it will upload files to that directory that you created previously /tmp/dvc-storage and when doing dvc pull it will connect to the remote computer via SSH, go to that directory ( /tmp/dvc-storage/ ) and then download the files via SSH.
I don’t think this answers the original question, which is how to set up DVC so that MANY DIFFERENT users can access the ssh server with their own accounts. You just provided an explanation on how to set up the ssh remote. However, your step for the local password is new to me, and might be useful somehow? Can people set up the user for ssh as local as well? Something like
dvc remote modify --local ssh-storage user mroutis
???
Hi @fercook !
Yes, you could use --local that way to setup different users, sure. E.g. you would do:
dvc remote modify --local ssh-storage user fercook
dvc remote modify --local ssh-storage password 123456789
and your collegue would do
dvc remote modify --local ssh-storage user mroutis
dvc remote modify --local ssh-storage password 987654321
Let us know if you have any other questions :slightly_smiling_face:Do I need to have DVC installed on the remote machine for it to work, or just locally?
P.S.: thanks @shcheklein for this tutorial.
Just locally, @arthurcgusmao. Please refer to https://dvc.org/doc/start/data-versioning#storing-and-sharing for an updated intro to DVC remote storage. Thanks
@shcheklein, I wasn’t able to replicate the remote repo configuration using the argument --local as in your answer:
dvc remote add --default ssh-storage ssh://example.com/path/to/storage
dvc remote modify ssh-storage user mroutis
dvc remote modify ssh-storage port 22
dvc remote modify --local ssh-storage password 123456789
When using --local only for configuring the password, DVC says:
ERROR: configuration error - config file error: remote 'ssh-storage' doesn't exist.
What seems to be happening is that DVC looks only on the local config file in that case, disregarding the default (non-local) config.
One can, however, work around that by reconfiguring the steps before the password with the --local argument.
Should I open an issue regarding this or is it the new expected behavior?
Hi @arthurcgusmao! This is a known issue and it will be fixed in the next release in a few days - https://github.com/iterative/dvc/issues/4276 (fix is already merged).
The workaround is to run (along with the regular dvc remote add):
dvc remote add --local ssh-storage ssh://example.com/path/to/storage
to add an entry, so that it is not complaining.
