SSH login failed after upgrading version 0.24.3=>0.30.1

Krysiul · March 4, 2019, 1:25pm

Hi,

After upgrading to version 0.30.1 I am unable to pull from my ssh server anymore. After downgrading back to 0.24.3 everything goes smooth.

When working with 0.30.1, this is the output I get for dvc pull -r my_remote:

Preparing to download data from ‘ssh://@.com/dir1/dir2’
Preparing to collect status from ssh://@.com/dir1/dir2
[######### ] 30% Collecting information
e[31mErrore[39m: failed to pull data from the cloud - [Errno 2] No such file

e[33mHaving any troubles?e[39m Hit us up at e[34mhttps://dvc.org/supporte[39m, we are always happy to help!

mroutis · March 5, 2019, 3:02am

Hello, Krysiul!

The SSH remote has changed to use SFTP instead of SSH client in some operations, however, I couldn’t managed to replicate the issue, here is my steps:

# Go to working version
git checkout 0.24.3
source .venv/bin/activate
dvc --version  # 0.24.3+9c7381

# Initialize ssh daemon
systemctl start sshd

# Initialize a repository
mkdir /tmp/remote-storage
mkdir /tmp/example
cd /tmp/example
dvc init --no-scm
dvc remote add ssh-remote ssh://localhost/tmp/remote-storage
dvc remote modify ssh-remote ask_password True

# Add a file
echo "hello" > hello
dvc add hello

# Push it
dvc push -r ssh-remote

# Checkout the filthy version
cd -
git checkout 0.30.1
dvc --version  # 0.30.1+97c099

# Try to pull the file without removing it
cd -
dvc pull -r ssh-remote
#     Preparing to download data from 'ssh://localhost/tmp/remote-storage'
#     Preparing to collect status from ssh://localhost/tmp/remote-storage
#     [#########                     ] 30% Collecting information
#     [##############################] 100% Collecting information
#     [##############################] 100% Analysing status.
#     Data '{u'path': u'/tmp/example/hello', u'scheme': u'local'}' didn't change.
#     Everything is up to date.

# Remove the local output to force checkout
rm hello
dvc pull -r ssh-remote
#     Preparing to download data from 'ssh://localhost/tmp/remote-storage'
#     Preparing to collect status from ssh://localhost/tmp/remote-storage
#     [#########                     ] 30% Collecting information
#     [##############################] 100% Collecting information
#     [##############################] 100% Analysing status.
#     Checking out '{u'path': u'/tmp/example/hello', u'scheme': u'local'}' with cache 'b1946ac92492d2347c6235b4d2611184'.
#     Everything is up to date.

# Remove the output and my cache to force checkout
rm -rf hello .dvc/cache
dvc pull -r ssh-remote
#     Preparing to download data from 'ssh://localhost/tmp/remote-storage'
#     Preparing to collect status from ssh://localhost/tmp/remote-storage
#     [#########                     ] 30% Collecting information
#     [##############################] 100% Collecting information
#     [##############################] 100% Analysing status.
#     [##############################] 100% hello
#     Checking out '{u'path': u'/tmp/example/hello', u'scheme': u'local'}' with cache 'b1946ac92492d2347c6235b4d2611184

# Stop sshd and try with a docker instance to test SSH key checking
systemctl stop sshd
systemctl start docker

docker run -d \
           -p 22:22 \
           -v ${HOME}/.ssh/id_rsa.pub:/etc/authorized_keys/${USER} \
           -e SSH_USERS="${USER}:" \
           panubo/sshd

ssh localhost 'mkdir /tmp/remote-storage'

dvc push -r ssh-remote
#     Preparing to upload data to 'ssh://localhost/tmp/remote-storage'
#     Preparing to collect status from ssh://localhost/tmp/remote-storage
#     [#########                     ] 30% Collecting information
#     [##############################] 100% Collecting information
#     [##############################] 100% Analysing status.
#     [##############################] 100% hello

rm -rf hello .dvc/cache
dvc pull -r ssh-remote
#     Preparing to download data from 'ssh://localhost/tmp/remote-storage'
#     Preparing to collect status from ssh://localhost/tmp/remote-storage
#     [#########                     ] 30% Collecting information
#     [##############################] 100% Collecting information
#     [##############################] 100% Analysing status.
#     [##############################] 100% hello
#     Checking out '{u'path': u'/tmp/example/hello', u'scheme': u'local'}' with cache 'b1946ac92492d2347c6235b4d2611184'.

Could you elaborate more about your what was going on when you tried to pull the file?

It would be very helpful f you could run dvc pull --verbose -r my_remote and paste the output, it may lint more information to figure out what was going when the command failed.

Also, did you check if the file was indeed reachable through ssh? (without dvc being involved)

Greetings!

– Outis

Qsi · March 8, 2019, 10:13am

Hi. I have the same problem. This happens even after i configured a new Remote and pushed to it again. Or if I execute dvc status -c.

Here is some Output. Maybe that helps figuring out what the problem is?

Error: failed to obtain data status - [Errno 2] No such file
------------------------------------------------------------
Traceback (most recent call last):
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\command\status.py", line 50, in do_run
with_deps=self.args.with_deps,
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\repo\status.py", line 88, in status
all_tags=all_tags,
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\repo\status.py", line 52, in _cloud_status
used, jobs, remote=remote, show_checksums=show_checksums
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\data_cloud.py", line 164, in status
targets, jobs=jobs, remote=cloud, show_checksums=show_checksums
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\remote\local.py", line 583, in status
remote_exists = list(remote.cache_exists(md5s))
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\remote\base.py", line 299, in cache_exists
return list(filter(lambda checksum: checksum in checksums, self.all()))
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\remote\base.py", line 248, in all
self.path_to_checksum(path) for path in self.list_cache_paths()
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\remote\ssh\__init__.py", line 198, in list_cache_paths
return list(ssh.walk_files(self.prefix))
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\remote\ssh\connection.py", line 138, in walk_files
for inner_path in self.walk_files(path):
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\dvc\remote\ssh\connection.py", line 130, in walk_files
for entry in self._sftp.listdir_attr(directory):
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\paramiko\sftp_client.py", line 239, in listdir_attr
t, msg = self._request(CMD_OPENDIR, path)
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\paramiko\sftp_client.py", line 813, in _request
return self._read_response(num)
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\paramiko\sftp_client.py", line 865, in _read_response
self._convert_status(msg)
  File "c:\anaconda3\envs\ift-brillenmatch\lib\site-packages\paramiko\sftp_client.py", line 894, in _convert_status
raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file

Qsi · March 8, 2019, 10:48am

Hi. Me again. After some digging I found the problem:

github.com

iterative/dvc/blob/master/dvc/remote/ssh/connection.py#L131




    if tail:
        self._sftp.mkdir(path)


def walk_files(self, directory):
    from stat import S_ISLNK, S_ISDIR, S_ISREG


    self._sftp_connect()


    for entry in self._sftp.listdir_attr(directory):
        path = os.path.join(directory, entry.filename)


        if S_ISLNK(entry.st_mode):
            path = self._sftp.readlink(directory)
            entry = self._sftp.stat(path)


        if S_ISDIR(entry.st_mode):
            for inner_path in self.walk_files(path):
                yield inner_path


        elif S_ISREG(entry.st_mode):

This line causes problems on windows-systems as it joins a linux-style-path with windows style. This creates a remote-path like this: /opt/dir/hdd/myfolder/dvcremote\9b which is not a valid path.

Qsi · March 8, 2019, 12:10pm

I created an issue here: https://github.com/iterative/dvc/issues/1701

Krysiul · March 13, 2019, 5:36pm

Hey @Qsi, thanks for your contribution to this topic.I’ve also changed the line you’ve mentioned. It solves one problem, but still I’m not able to use the new version. Likewise, the problem only occurs while using version 0.30.1, and vanishes after switching back to 0.24.3 (haven’t tried any other versions in-between). I think the problem might be with the way directories are created under Windows (maybe posix once again?). Unfortunately for some reason my debugger doesn’t work well with the library, so I’m unable to provide more details. Here is the output of dvc pull --verbose -r some_remote:

e[34mDebuge[39m: PRAGMA user_version;
e[34mDebuge[39m: fetched: [(3,)]
e[34mDebuge[39m: CREATE TABLE IF NOT EXISTS state (inode INTEGER PRIMARY KEY, mtime TEXT NOT NULL, size TEXT NOT NULL, md5 TEXT NOT NULL, timestamp TEXT NOT NULL)
e[34mDebuge[39m: CREATE TABLE IF NOT EXISTS state_info (count INTEGER)
e[34mDebuge[39m: CREATE TABLE IF NOT EXISTS link_state (path TEXT PRIMARY KEY, inode INTEGER NOT NULL, mtime TEXT NOT NULL)
e[34mDebuge[39m: INSERT OR IGNORE INTO state_info (count) SELECT 0 WHERE NOT EXISTS (SELECT * FROM state_info)
e[34mDebuge[39m: PRAGMA user_version = 3;
e[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\02\e5338dcfc6f3bac31b64657b02d811.dir’, md5 ‘02e5338dcfc6f3bac31b64657b02d811.dir’, actual ‘None’
Preparing to download data from ‘ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir’
Preparing to collect status from ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir
[######### ] 30% Collecting informatione[34mDebuge[39m: Establishing ssh connection with ‘some_remote.org’ through port ‘22’ as user ‘some_user’
c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\paramiko\ecdsakey.py:164: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
self.ecdsa_curve.curve_class(), pointinfo
c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\paramiko\kex_ecdh_nist.py:39: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
m.add_string(self.Q_C.public_numbers().encode_point())
c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\paramiko\kex_ecdh_nist.py:96: CryptographyDeprecationWarning: Support for unsafe construction of public numbers from encoded data will be removed in a future version. Please use EllipticCurvePublicKey.from_encoded_point
self.curve, Q_S_bytes
c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\paramiko\kex_ecdh_nist.py:111: CryptographyDeprecationWarning: encode_point has been deprecated on EllipticCurvePublicNumbers and will be removed in a future version. Please use EllipticCurvePublicKey.public_bytes to obtain both compressed and uncompressed point encoding.
hm.add_string(self.Q_C.public_numbers().encode_point())
[########################### ] 90% Collecting informatione[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\02\e5338dcfc6f3bac31b64657b02d811.dir’, md5 ‘02e5338dcfc6f3bac31b64657b02d811.dir’, actual ‘None’
[##############################] 100% Collecting information
[##############################] 100% Analysing status.
e[34mDebuge[39m: Establishing ssh connection with ‘some_remote.org’ through port ‘22’ as user ‘some_user’
e[34mDebuge[39m: Downloading ‘some_remote.org//home/some_user/some_dir/repo_dir/02/e5338dcfc6f3bac31b64657b02d811.dir’ to ‘C:\Users\some_user\Documents\some_dir.dvc\cache\02\e5338dcfc6f3bac31b64657b02d811.dir’

e[31mErrore[39m: failed to download ‘some_remote.org//home/some_user/some_dir/repo_dir/02/e5338dcfc6f3bac31b64657b02d811.dir’ to ‘C:\Users\some_user\Documents\some_dir.dvc\cache\02\e5338dcfc6f3bac31b64657b02d811.dir’ - [WinError 3] The system cannot find the path specified: ‘’
e[31m------------------------------------------------------------e[39m
Traceback (most recent call last):
File “c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\dvc\remote\ssh_init_.py”, line 157, in download
from_info[“path”], to_info[“path”], progress_title=name
File “c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\dvc\remote\ssh\connection.py”, line 151, in download
makedirs(posixpath.dirname(dest), exist_ok=True)
File “c:\users\some_user\anaconda3\envs\env_name\lib\os.py”, line 220, in makedirs
mkdir(name, mode)
FileNotFoundError: [WinError 3] The system cannot find the path specified: ‘’
e[31m------------------------------------------------------------e[39m

e[33mHaving any troubles?e[39m Hit us up at e[34mhttps://dvc.org/supporte[39m, we are always happy to help!
e[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\02\e5338dcfc6f3bac31b64657b02d811.dir’, md5 ‘02e5338dcfc6f3bac31b64657b02d811.dir’, actual ‘None’
e[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\03\bc18c042d53ffec3374d3ca16b7623.dir’, md5 ‘03bc18c042d53ffec3374d3ca16b7623.dir’, actual ‘None’

Preparing to download data from ‘ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir’

Preparing to collect status from ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir
[######### ] 30% Collecting informatione[34mDebuge[39m: Establishing ssh connection with ‘some_remote.org’ through port ‘22’ as user ‘some_user’
[########################### ] 90% Collecting informatione[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\03\bc18c042d53ffec3374d3ca16b7623.dir’, md5 ‘03bc18c042d53ffec3374d3ca16b7623.dir’, actual ‘None’
[##############################] 100% Collecting information
[##############################] 100% Analysing status.
e[34mDebuge[39m: Establishing ssh connection with ‘some_remote.org’ through port ‘22’ as user ‘some_user’
e[34mDebuge[39m: Downloading ‘some_remote.org//home/some_user/some_dir/repo_dir/03/bc18c042d53ffec3374d3ca16b7623.dir’ to ‘C:\Users\some_user\Documents\some_dir.dvc\cache\03\bc18c042d53ffec3374d3ca16b7623.dir’

e[31mErrore[39m: failed to download ‘some_remote.org//home/some_user/some_dir/repo_dir/03/bc18c042d53ffec3374d3ca16b7623.dir’ to ‘C:\Users\some_user\Documents\some_dir.dvc\cache\03\bc18c042d53ffec3374d3ca16b7623.dir’ - [WinError 3] The system cannot find the path specified: ‘’
e[31m------------------------------------------------------------e[39m
Traceback (most recent call last):
File “c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\dvc\remote\ssh_init_.py”, line 157, in download
from_info[“path”], to_info[“path”], progress_title=name
File “c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\dvc\remote\ssh\connection.py”, line 151, in download
makedirs(posixpath.dirname(dest), exist_ok=True)
File “c:\users\some_user\anaconda3\envs\env_name\lib\os.py”, line 220, in makedirs
mkdir(name, mode)
FileNotFoundError: [WinError 3] The system cannot find the path specified: ‘’
e[31m------------------------------------------------------------e[39m

e[33mHaving any troubles?e[39m Hit us up at e[34mhttps://dvc.org/supporte[39m, we are always happy to help!
e[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\03\bc18c042d53ffec3374d3ca16b7623.dir’, md5 ‘03bc18c042d53ffec3374d3ca16b7623.dir’, actual ‘None’
e[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\2d\699d3476e1071a9ad206bd044f17a9.dir’, md5 ‘2d699d3476e1071a9ad206bd044f17a9.dir’, actual ‘None’

Preparing to download data from ‘ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir’

Preparing to collect status from ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir
[######### ] 30% Collecting informatione[34mDebuge[39m: Establishing ssh connection with ‘some_remote.org’ through port ‘22’ as user ‘some_user’
[########################### ] 90% Collecting informatione[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\2d\699d3476e1071a9ad206bd044f17a9.dir’, md5 ‘2d699d3476e1071a9ad206bd044f17a9.dir’, actual ‘None’
[##############################] 100% Collecting information
[##############################] 100% Analysing status.
e[34mDebuge[39m: Establishing ssh connection with ‘some_remote.org’ through port ‘22’ as user ‘some_user’
e[34mDebuge[39m: Downloading ‘some_remote.org//home/some_user/some_dir/repo_dir/2d/699d3476e1071a9ad206bd044f17a9.dir’ to ‘C:\Users\some_user\Documents\some_dir.dvc\cache\2d\699d3476e1071a9ad206bd044f17a9.dir’

e[31mErrore[39m: failed to download ‘some_remote.org//home/some_user/some_dir/repo_dir/2d/699d3476e1071a9ad206bd044f17a9.dir’ to ‘C:\Users\some_user\Documents\some_dir.dvc\cache\2d\699d3476e1071a9ad206bd044f17a9.dir’ - [WinError 3] The system cannot find the path specified: ‘’
e[31m------------------------------------------------------------e[39m
Traceback (most recent call last):
File “c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\dvc\remote\ssh_init_.py”, line 157, in download
from_info[“path”], to_info[“path”], progress_title=name
File “c:\users\some_user\anaconda3\envs\env_name\lib\site-packages\dvc\remote\ssh\connection.py”, line 151, in download
makedirs(posixpath.dirname(dest), exist_ok=True)
File “c:\users\some_user\anaconda3\envs\env_name\lib\os.py”, line 220, in makedirs
mkdir(name, mode)
FileNotFoundError: [WinError 3] The system cannot find the path specified: ‘’
e[31m------------------------------------------------------------e[39m

e[33mHaving any troubles?e[39m Hit us up at e[34mhttps://dvc.org/supporte[39m, we are always happy to help!
e[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\2d\699d3476e1071a9ad206bd044f17a9.dir’, md5 ‘2d699d3476e1071a9ad206bd044f17a9.dir’, actual ‘None’
e[34mDebuge[39m: File ‘C:\Users\some_user\Documents\some_dir.dvc\cache\ba\123eb852363aa7775e3c3259257ba5.dir’, md5 ‘ba123eb852363aa7775e3c3259257ba5.dir’, actual ‘None’

Preparing to download data from ‘ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir’

Preparing to collect status from ssh://some_user@some_remote.org/home/some_user/some_dir/repo_dir
[######### ] 30% Collecting informatione[34mDebuge[39m: Establishing ssh connection with ‘some_remote.org’ through port ‘22’ as user ‘some_user’

kupruser · March 13, 2019, 8:34pm

For the record. This was discussed in discord chat and the bug was fixed in 0.31.1 . Thank you guys for the feedback!

Krysiul · March 14, 2019, 8:30am

Thanks @kupruser for patching that quickly, the problem I have described in the previous post though persists in 0.31.1 (the first one had indeed vanished). Should I create a new topic for that as it is another bug?

Qsi · March 14, 2019, 9:30am

Hey @Krysiul
The same happened with my repo. Found other problems and created a PR. Sadly I can’t link to it due to an error.

kupruser · March 14, 2019, 2:06pm

@Krysiul Both fixes were kindly submitted by @Qsi The new one is merged as well, we’ll release a new version later today. Thank you guys for all the feedback!

kupruser · March 14, 2019, 2:09pm

Sorry, it was our auto-moderation system freaking out about links to github for some reason Unflagged your messages, hopefully, won’t happen next time. Sorry for the inconvenience.

kupruser · March 14, 2019, 6:19pm

@Krysiul @Qsi Guys, 0.32.0 is out. Please upgrade.

Krysiul · March 15, 2019, 9:33am

Thanks for the update - everything seems to be working OK now

Topic		Replies	Views
Important: ERROR: failed to pull data from the cloud Questions	10	112	February 16, 2024
DVC pull fails without sudo Bug Reports	8	4343	February 13, 2019
ERROR: unexpected error - OpenSSH private key encryption requires bcrypt with KDF support Questions	2	615	March 4, 2022
ERROR: failed to pull data from the cloud Questions	2	5666	August 24, 2021
ERROR: unexpected error - [Errno 13] Permission denied Questions	0	43	April 3, 2024

SSH login failed after upgrading version 0.24.3=>0.30.1

Related Topics