SSH login failed after upgrading version 0.24.3=>0.30.1


#1

Hi,

After upgrading to version 0.30.1 I am unable to pull from my ssh server anymore. After downgrading back to 0.24.3 everything goes smooth.

When working with 0.30.1, this is the output I get for dvc pull -r my_remote:

Preparing to download data from ‘ssh://***@***.com/dir1/dir2’
Preparing to collect status from ssh://***@***.com/dir1/dir2
[######### ] 30% Collecting information
e[31mErrore[39m: failed to pull data from the cloud - [Errno 2] No such file

e[33mHaving any troubles?e[39m Hit us up at e[34mhttps://dvc.org/supporte[39m, we are always happy to help!


#2

Hello, Krysiul!

The SSH remote has changed to use SFTP instead of SSH client in some operations, however, I couldn’t managed to replicate the issue, here is my steps:

# Go to working version
git checkout 0.24.3
source .venv/bin/activate
dvc --version  # 0.24.3+9c7381

# Initialize ssh daemon
systemctl start sshd

# Initialize a repository
mkdir /tmp/remote-storage
mkdir /tmp/example
cd /tmp/example
dvc init --no-scm
dvc remote add ssh-remote ssh://localhost/tmp/remote-storage
dvc remote modify ssh-remote ask_password True

# Add a file
echo "hello" > hello
dvc add hello

# Push it
dvc push -r ssh-remote

# Checkout the filthy version
cd -
git checkout 0.30.1
dvc --version  # 0.30.1+97c099

# Try to pull the file without removing it
cd -
dvc pull -r ssh-remote
#     Preparing to download data from 'ssh://localhost/tmp/remote-storage'
#     Preparing to collect status from ssh://localhost/tmp/remote-storage
#     [#########                     ] 30% Collecting information
#     [##############################] 100% Collecting information
#     [##############################] 100% Analysing status.
#     Data '{u'path': u'/tmp/example/hello', u'scheme': u'local'}' didn't change.
#     Everything is up to date.

# Remove the local output to force checkout
rm hello
dvc pull -r ssh-remote
#     Preparing to download data from 'ssh://localhost/tmp/remote-storage'
#     Preparing to collect status from ssh://localhost/tmp/remote-storage
#     [#########                     ] 30% Collecting information
#     [##############################] 100% Collecting information
#     [##############################] 100% Analysing status.
#     Checking out '{u'path': u'/tmp/example/hello', u'scheme': u'local'}' with cache 'b1946ac92492d2347c6235b4d2611184'.
#     Everything is up to date.

# Remove the output and my cache to force checkout
rm -rf hello .dvc/cache
dvc pull -r ssh-remote
#     Preparing to download data from 'ssh://localhost/tmp/remote-storage'
#     Preparing to collect status from ssh://localhost/tmp/remote-storage
#     [#########                     ] 30% Collecting information
#     [##############################] 100% Collecting information
#     [##############################] 100% Analysing status.
#     [##############################] 100% hello
#     Checking out '{u'path': u'/tmp/example/hello', u'scheme': u'local'}' with cache 'b1946ac92492d2347c6235b4d2611184

# Stop sshd and try with a docker instance to test SSH key checking
systemctl stop sshd
systemctl start docker

docker run -d \
           -p 22:22 \
           -v ${HOME}/.ssh/id_rsa.pub:/etc/authorized_keys/${USER} \
           -e SSH_USERS="${USER}:" \
           panubo/sshd

ssh localhost 'mkdir /tmp/remote-storage'

dvc push -r ssh-remote
#     Preparing to upload data to 'ssh://localhost/tmp/remote-storage'
#     Preparing to collect status from ssh://localhost/tmp/remote-storage
#     [#########                     ] 30% Collecting information
#     [##############################] 100% Collecting information
#     [##############################] 100% Analysing status.
#     [##############################] 100% hello

rm -rf hello .dvc/cache
dvc pull -r ssh-remote
#     Preparing to download data from 'ssh://localhost/tmp/remote-storage'
#     Preparing to collect status from ssh://localhost/tmp/remote-storage
#     [#########                     ] 30% Collecting information
#     [##############################] 100% Collecting information
#     [##############################] 100% Analysing status.
#     [##############################] 100% hello
#     Checking out '{u'path': u'/tmp/example/hello', u'scheme': u'local'}' with cache 'b1946ac92492d2347c6235b4d2611184'.

Could you elaborate more about your what was going on when you tried to pull the file?

It would be very helpful f you could run dvc pull --verbose -r my_remote and paste the output, it may lint more information to figure out what was going when the command failed.

Also, did you check if the file was indeed reachable through ssh? (without dvc being involved)

Greetings!

– Outis


#3

Hi. I have the same problem. This happens even after i configured a new Remote and pushed to it again. Or if I execute dvc status -c.

Here is some Output. Maybe that helps figuring out what the problem is?

Error: failed to obtain data status - [Errno 2] No such file
------------------------------------------------------------
Traceback (most recent call last):
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\command\status.py", line 50, in do_run
with_deps=self.args.with_deps,
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\repo\status.py", line 88, in status
all_tags=all_tags,
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\repo\status.py", line 52, in _cloud_status
used, jobs, remote=remote, show_checksums=show_checksums
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\data_cloud.py", line 164, in status
targets, jobs=jobs, remote=cloud, show_checksums=show_checksums
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\remote\local.py", line 583, in status
remote_exists = list(remote.cache_exists(md5s))
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\remote\base.py", line 299, in cache_exists
return list(filter(lambda checksum: checksum in checksums, self.all()))
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\remote\base.py", line 248, in all
self.path_to_checksum(path) for path in self.list_cache_paths()
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\remote\ssh\__init__.py", line 198, in list_cache_paths
return list(ssh.walk_files(self.prefix))
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\remote\ssh\connection.py", line 138, in walk_files
for inner_path in self.walk_files(path):
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\remote\ssh\connection.py", line 130, in walk_files
for entry in self._sftp.listdir_attr(directory):
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\paramiko\sftp_client.py", line 239, in listdir_attr
t, msg = self._request(CMD_OPENDIR, path)
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\paramiko\sftp_client.py", line 813, in _request
return self._read_response(num)
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\paramiko\sftp_client.py", line 865, in _read_response
self._convert_status(msg)
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\paramiko\sftp_client.py", line 894, in _convert_status
raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file

#4

Hi. Me again. After some digging I found the problem:

This line causes problems on windows-systems as it joins a linux-style-path with windows style. This creates a remote-path like this: /opt/dir/hdd/myfolder/dvcremote\9b which is not a valid path.


#5

I created an issue here: https://github.com/iterative/dvc/issues/1701


#6

Hey @Qsi, thanks for your contribution to this topic.I’ve also changed the line you’ve mentioned. It solves one problem, but still I’m not able to use the new version. Likewise, the problem only occurs while using version 0.30.1, and vanishes after switching back to 0.24.3 (haven’t tried any other versions in-between). I think the problem might be with the way directories are created under Windows (maybe posix once again?). Unfortunately for some reason my debugger doesn’t work well with the library, so I’m unable to provide more details. Here is the output of dvc pull --verbose -r some_remote:

e[34mDebuge[39m: PRAGMA user_version;
e[34mDebuge[39m: fetched: [(3,)]
e[34mDebuge[39m: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
e[34mDebuge[39m: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
e[34mDebuge[39m: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
e[34mDebuge[39m: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
e[34mDebuge[39m: PRAGMA user_version = 3;
e[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\02\e5338dcfc6f3bac31b64657b02d811.dir’, md5 ‘02e5338dcfc6f3bac31b64657b02d811.dir’, actual ‘None’
Preparing to download data from ‘ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir’
Preparing to collect status from ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir
[######### ] 30% Collecting informatione[34mDebuge[39m: Establishing ssh connection with ‘some_remote.org’ through port ‘22’ as user ‘some_user’
c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\paramiko\ecdsakey.py:164: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
self.ecdsa_curve.curve_class(), pointinfo
c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\paramiko\kex_ecdh_nist.py:39: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
m.add_string(self.Q_C.public_numbers().encode_point())
c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\paramiko\kex_ecdh_nist.py:96: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
self.curve, Q_S_bytes
c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\paramiko\kex_ecdh_nist.py:111: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
hm.add_string(self.Q_C.public_numbers().encode_point())
[########################### ] 90% Collecting informatione[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\02\e5338dcfc6f3bac31b64657b02d811.dir’, md5 ‘02e5338dcfc6f3bac31b64657b02d811.dir’, actual ‘None’
[##############################] 100% Collecting information
[##############################] 100% Analysing status.
e[34mDebuge[39m: Establishing ssh connection with ‘some_remote.org’ through port ‘22’ as user ‘some_user’
e[34mDebuge[39m: Downloading ‘some_remote.org//home/some_user/some_dir/repo_dir/02/e5338dcfc6f3bac31b64657b02d811.dir’ to ‘C:\Users\some_user\Documents\some_dir.dvc\cache\02\e5338dcfc6f3bac31b64657b02d811.dir’

e[31mErrore[39m: failed to download ‘some_remote.org//home/some_user/some_dir/repo_dir/02/e5338dcfc6f3bac31b64657b02d811.dir’ to ‘C:\Users\some_user\Documents\some_dir.dvc\cache\02\e5338dcfc6f3bac31b64657b02d811.dir’ - [WinError 3] The system cannot find the path specified: ‘’
e[31m------------------------------------------------------------e[39m
Traceback (most recent call last):
File “c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\dvc\remote\ssh_init_.py”, line 157, in download
from_info[“path”], to_info[“path”], progress_title=name
File “c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\dvc\remote\ssh\connection.py”, line 151, in download
makedirs(posixpath.dirname(dest), exist_ok=True)
File “c:\users\some_user\anaconda3\envs\env_name\lib\os.py”, line 220, in makedirs
mkdir(name, mode)
FileNotFoundError: [WinError 3] The system cannot find the path specified: ‘’
e[31m------------------------------------------------------------e[39m

e[33mHaving any troubles?e[39m Hit us up at e[34mhttps://dvc.org/supporte[39m, we are always happy to help!
e[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\02\e5338dcfc6f3bac31b64657b02d811.dir’, md5 ‘02e5338dcfc6f3bac31b64657b02d811.dir’, actual ‘None’
e[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\03\bc18c042d53ffec3374d3ca16b7623.dir’, md5 ‘03bc18c042d53ffec3374d3ca16b7623.dir’, actual ‘None’

Preparing to download data from ‘ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir’

Preparing to collect status from ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir
[######### ] 30% Collecting informatione[34mDebuge[39m: Establishing ssh connection with ‘some_remote.org’ through port ‘22’ as user ‘some_user’
[########################### ] 90% Collecting informatione[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\03\bc18c042d53ffec3374d3ca16b7623.dir’, md5 ‘03bc18c042d53ffec3374d3ca16b7623.dir’, actual ‘None’
[##############################] 100% Collecting information
[##############################] 100% Analysing status.
e[34mDebuge[39m: Establishing ssh connection with ‘some_remote.org’ through port ‘22’ as user ‘some_user’
e[34mDebuge[39m: Downloading ‘some_remote.org//home/some_user/some_dir/repo_dir/03/bc18c042d53ffec3374d3ca16b7623.dir’ to ‘C:\Users\some_user\Documents\some_dir.dvc\cache\03\bc18c042d53ffec3374d3ca16b7623.dir’

e[31mErrore[39m: failed to download ‘some_remote.org//home/some_user/some_dir/repo_dir/03/bc18c042d53ffec3374d3ca16b7623.dir’ to ‘C:\Users\some_user\Documents\some_dir.dvc\cache\03\bc18c042d53ffec3374d3ca16b7623.dir’ - [WinError 3] The system cannot find the path specified: ‘’
e[31m------------------------------------------------------------e[39m
Traceback (most recent call last):
File “c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\dvc\remote\ssh_init_.py”, line 157, in download
from_info[“path”], to_info[“path”], progress_title=name
File “c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\dvc\remote\ssh\connection.py”, line 151, in download
makedirs(posixpath.dirname(dest), exist_ok=True)
File “c:\users\some_user\anaconda3\envs\env_name\lib\os.py”, line 220, in makedirs
mkdir(name, mode)
FileNotFoundError: [WinError 3] The system cannot find the path specified: ‘’
e[31m------------------------------------------------------------e[39m

e[33mHaving any troubles?e[39m Hit us up at e[34mhttps://dvc.org/supporte[39m, we are always happy to help!
e[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\03\bc18c042d53ffec3374d3ca16b7623.dir’, md5 ‘03bc18c042d53ffec3374d3ca16b7623.dir’, actual ‘None’
e[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\2d\699d3476e1071a9ad206bd044f17a9.dir’, md5 ‘2d699d3476e1071a9ad206bd044f17a9.dir’, actual ‘None’

Preparing to download data from ‘ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir’

Preparing to collect status from ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir
[######### ] 30% Collecting informatione[34mDebuge[39m: Establishing ssh connection with ‘some_remote.org’ through port ‘22’ as user ‘some_user’
[########################### ] 90% Collecting informatione[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\2d\699d3476e1071a9ad206bd044f17a9.dir’, md5 ‘2d699d3476e1071a9ad206bd044f17a9.dir’, actual ‘None’
[##############################] 100% Collecting information
[##############################] 100% Analysing status.
e[34mDebuge[39m: Establishing ssh connection with ‘some_remote.org’ through port ‘22’ as user ‘some_user’
e[34mDebuge[39m: Downloading ‘some_remote.org//home/some_user/some_dir/repo_dir/2d/699d3476e1071a9ad206bd044f17a9.dir’ to ‘C:\Users\some_user\Documents\some_dir.dvc\cache\2d\699d3476e1071a9ad206bd044f17a9.dir’

e[31mErrore[39m: failed to download ‘some_remote.org//home/some_user/some_dir/repo_dir/2d/699d3476e1071a9ad206bd044f17a9.dir’ to ‘C:\Users\some_user\Documents\some_dir.dvc\cache\2d\699d3476e1071a9ad206bd044f17a9.dir’ - [WinError 3] The system cannot find the path specified: ‘’
e[31m------------------------------------------------------------e[39m
Traceback (most recent call last):
File “c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\dvc\remote\ssh_init_.py”, line 157, in download
from_info[“path”], to_info[“path”], progress_title=name
File “c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\dvc\remote\ssh\connection.py”, line 151, in download
makedirs(posixpath.dirname(dest), exist_ok=True)
File “c:\users\some_user\anaconda3\envs\env_name\lib\os.py”, line 220, in makedirs
mkdir(name, mode)
FileNotFoundError: [WinError 3] The system cannot find the path specified: ‘’
e[31m------------------------------------------------------------e[39m

e[33mHaving any troubles?e[39m Hit us up at e[34mhttps://dvc.org/supporte[39m, we are always happy to help!
e[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\2d\699d3476e1071a9ad206bd044f17a9.dir’, md5 ‘2d699d3476e1071a9ad206bd044f17a9.dir’, actual ‘None’
e[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\ba\123eb852363aa7775e3c3259257ba5.dir’, md5 ‘ba123eb852363aa7775e3c3259257ba5.dir’, actual ‘None’

Preparing to download data from ‘ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir’

Preparing to collect status from ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir
[######### ] 30% Collecting informatione[34mDebuge[39m: Establishing ssh connection with ‘some_remote.org’ through port ‘22’ as user ‘some_user’


#7

For the record. This was discussed in discord chat and the bug was fixed in 0.31.1 . Thank you guys for the feedback! :slight_smile:


#8

Thanks @kupruser for patching that quickly, the problem I have described in the previous post though persists in 0.31.1 (the first one had indeed vanished). Should I create a new topic for that as it is another bug?


#9

Hey @Krysiul
The same happened with my repo. Found other problems and created a PR. Sadly I can’t link to it due to an error.


#10

@Krysiul Both fixes were kindly submitted by @Qsi :slight_smile: The new one is merged as well, we’ll release a new version later today. Thank you guys for all the feedback! :slightly_smiling_face:


#11

Sorry, it was our auto-moderation system freaking out about links to github for some reason :slight_smile: Unflagged your messages, hopefully, won’t happen next time. Sorry for the inconvenience.


#12

@Krysiul @Qsi Guys, 0.32.0 is out. Please upgrade. :slightly_smiling_face:


#13

Thanks for the update - everything seems to be working OK now :slight_smile: