SSH remote: unexpected error - Permission denied

leandrolanzieri · August 15, 2022, 9:41am

Hi,

I’m trying to configure an SSH remote storage. When I try to push my data directory I get:

$  dvc push
ERROR: unexpected error - Permission denied: Permission denied                                                                    

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

I can confirm that:

I can connect to the server via ssh and sftp
the URL is correct
I have write permissions in the path (already tried by manually uploading the files via sftp.

I can’t really make sense of the debug message:

$  dvc push -v
2022-08-15 11:38:48,112 DEBUG: Lockfile for 'dvc.yaml' not found      
2022-08-15 11:38:48,148 WARNING: Output 'data/numpy'(stage: 'convert_puf_responses') is missing version info. Cache for it will not be collected. Use `dvc repro` to get your pipeline up to date.
2022-08-15 11:38:48,273 DEBUG: Preparing to transfer data from '/home/lanzieri/phd/aging_indicators/aging-indicators-experiments/sram_startup/.dvc/cache' to '/net/archive/rodriguez/aging_indicators_data/sram'
2022-08-15 11:38:48,273 DEBUG: Preparing to collect status from '/net/archive/rodriguez/aging_indicators_data/sram'
2022-08-15 11:38:48,273 DEBUG: Collecting status from '/net/archive/rodriguez/aging_indicators_data/sram'
2022-08-15 11:38:48,274 DEBUG: Querying 1 oids via object_exists                                                                  
2022-08-15 11:38:48,425 ERROR: unexpected error - Permission denied: Permission denied                                            
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lanzieri/.local/lib/python3.10/site-packages/sshfs/utils.py", line 27, in wrapper
    return await func(*args, **kwargs)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/sshfs/spec.py", line 91, in _connect
    client = await self._stack.enter_async_context(_raw_client)
  File "/usr/lib/python3.10/contextlib.py", line 619, in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/asyncssh/misc.py", line 274, in __aenter__
    self._coro_result = await self._coro
  File "/home/lanzieri/.local/lib/python3.10/site-packages/asyncssh/connection.py", line 7707, in connect
    return await asyncio.wait_for(
  File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/home/lanzieri/.local/lib/python3.10/site-packages/asyncssh/connection.py", line 440, in _connect
    await options.waiter
asyncssh.misc.PermissionDenied: Permission denied

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc/cli/__init__.py", line 185, in main
    ret = cmd.do_run()
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc/cli/command.py", line 22, in do_run
    return self.run()
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc/commands/data_sync.py", line 58, in run
    processed_files_count = self.repo.push(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc/repo/__init__.py", line 48, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc/repo/push.py", line 68, in push
    pushed += self.cloud.push(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc/data_cloud.py", line 109, in push
    return self.transfer(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc/data_cloud.py", line 88, in transfer
    return transfer(src_odb, dest_odb, objs, **kwargs)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_data/transfer.py", line 158, in transfer
    status = compare_status(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_data/status.py", line 179, in compare_status
    dest_exists, dest_missing = status(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_data/status.py", line 136, in status
    exists = hashes.intersection(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_data/status.py", line 56, in _indexed_dir_hashes
    dir_exists.update(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_objects/db.py", line 255, in list_oids_exists
    yield from itertools.compress(oids, in_remote)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 609, in result_iterator
    yield fs.pop().result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 269, in exists
    return self.fs.exists(path)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/funcy/objects.py", line 50, in __get__
    return prop.__get__(instance, type)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/funcy/objects.py", line 28, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_objects/fs/implementations/ssh.py", line 115, in fs
    return _SSHFileSystem(**self.fs_args)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/fsspec/spec.py", line 76, in __call__
    obj = super().__call__(*args, **kwargs)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/sshfs/spec.py", line 76, in __init__
    self._client, self._pool = self.connect(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 86, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 66, in sync
    raise return_result
  File "/home/lanzieri/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 26, in _runner
    result[0] = await coro
  File "/usr/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return fut.result()
  File "/home/lanzieri/.local/lib/python3.10/site-packages/sshfs/utils.py", line 29, in wrapper
    raise PermissionError(exc.reason) from exc
PermissionError: Permission denied
------------------------------------------------------------
2022-08-15 11:38:48,557 DEBUG: [Errno 95] no more link types left to try out: [Errno 95] Operation not supported
------------------------------------------------------------
Traceback (most recent call last):
  File "/home/lanzieri/.local/lib/python3.10/site-packages/sshfs/utils.py", line 27, in wrapper
    return await func(*args, **kwargs)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/sshfs/spec.py", line 91, in _connect
    client = await self._stack.enter_async_context(_raw_client)
  File "/usr/lib/python3.10/contextlib.py", line 619, in enter_async_context
    result = await _cm_type.__aenter__(cm)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/asyncssh/misc.py", line 274, in __aenter__
    self._coro_result = await self._coro
  File "/home/lanzieri/.local/lib/python3.10/site-packages/asyncssh/connection.py", line 7707, in connect
    return await asyncio.wait_for(
  File "/usr/lib/python3.10/asyncio/tasks.py", line 408, in wait_for
    return await fut
  File "/home/lanzieri/.local/lib/python3.10/site-packages/asyncssh/connection.py", line 440, in _connect
    await options.waiter
asyncssh.misc.PermissionDenied: Permission denied

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc/cli/__init__.py", line 185, in main
    ret = cmd.do_run()
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc/cli/command.py", line 22, in do_run
    return self.run()
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc/commands/data_sync.py", line 58, in run
    processed_files_count = self.repo.push(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc/repo/__init__.py", line 48, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc/repo/push.py", line 68, in push
    pushed += self.cloud.push(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc/data_cloud.py", line 109, in push
    return self.transfer(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc/data_cloud.py", line 88, in transfer
    return transfer(src_odb, dest_odb, objs, **kwargs)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_data/transfer.py", line 158, in transfer
    status = compare_status(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_data/status.py", line 179, in compare_status
    dest_exists, dest_missing = status(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_data/status.py", line 136, in status
    exists = hashes.intersection(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_data/status.py", line 56, in _indexed_dir_hashes
    dir_exists.update(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/tqdm/std.py", line 1195, in __iter__
    for obj in iterable:
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_objects/db.py", line 255, in list_oids_exists
    yield from itertools.compress(oids, in_remote)
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 609, in result_iterator
    yield fs.pop().result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 446, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/usr/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 269, in exists
    return self.fs.exists(path)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/funcy/objects.py", line 50, in __get__
    return prop.__get__(instance, type)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/funcy/objects.py", line 28, in __get__
    res = instance.__dict__[self.fget.__name__] = self.fget(instance)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_objects/fs/implementations/ssh.py", line 115, in fs
    return _SSHFileSystem(**self.fs_args)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/fsspec/spec.py", line 76, in __call__
    obj = super().__call__(*args, **kwargs)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/sshfs/spec.py", line 76, in __init__
    self._client, self._pool = self.connect(
  File "/home/lanzieri/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 86, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 66, in sync
    raise return_result
  File "/home/lanzieri/.local/lib/python3.10/site-packages/fsspec/asyn.py", line 26, in _runner
    result[0] = await coro
  File "/usr/lib/python3.10/asyncio/tasks.py", line 445, in wait_for
    return fut.result()
  File "/home/lanzieri/.local/lib/python3.10/site-packages/sshfs/utils.py", line 29, in wrapper
    raise PermissionError(exc.reason) from exc
PermissionError: Permission denied

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 68, in _try_links
    return _link(link, from_fs, from_path, to_fs, to_path)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 28, in _link
    func(from_path, to_path)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_objects/fs/base.py", line 288, in reflink
    return self.fs.reflink(from_info, to_info)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_objects/fs/implementations/local.py", line 157, in reflink
    return system.reflink(path1, path2)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_objects/fs/system.py", line 105, in reflink
    _reflink_linux(source, link_name)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_objects/fs/system.py", line 91, in _reflink_linux
    fcntl.ioctl(d.fileno(), FICLONE, s.fileno())
OSError: [Errno 95] Operation not supported

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 127, in _test_link
    _try_links([link], from_fs, from_file, to_fs, to_file)
  File "/home/lanzieri/.local/lib/python3.10/site-packages/dvc_objects/fs/generic.py", line 76, in _try_links
    raise OSError(
OSError: [Errno 95] no more link types left to try out
------------------------------------------------------------
2022-08-15 11:38:48,558 DEBUG: Removing '/home/lanzieri/phd/aging_indicators/aging-indicators-experiments/.ZsCDGL8UDEPrdDmbm7D6mS.tmp'
2022-08-15 11:38:48,558 DEBUG: Removing '/home/lanzieri/phd/aging_indicators/aging-indicators-experiments/.ZsCDGL8UDEPrdDmbm7D6mS.tmp'
2022-08-15 11:38:48,558 DEBUG: Removing '/home/lanzieri/phd/aging_indicators/aging-indicators-experiments/.ZsCDGL8UDEPrdDmbm7D6mS.tmp'
2022-08-15 11:38:48,558 DEBUG: Removing '/home/lanzieri/phd/aging_indicators/aging-indicators-experiments/sram_startup/.dvc/cache/.QXMYC5qmirjwhLDdpceujW.tmp'
2022-08-15 11:38:48,569 DEBUG: Version info for developers:
DVC version: 2.12.0 (pip)
---------------------------------
Platform: Python 3.10.5 on Linux-5.10.135-1-MANJARO-x86_64-with-glibc2.36
Supports:
	webhdfs (fsspec = 2022.5.0),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.5.0),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.5.0),
	ssh (sshfs = 2022.6.0)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/mapper/luks-9d8db46c-e044-463a-9dda-ad428bb54390
Caches: local
Remotes: ssh
Workspace directory: ext4 on /dev/mapper/luks-9d8db46c-e044-463a-9dda-ad428bb54390
Repo: dvc (subdir), git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2022-08-15 11:38:48,571 DEBUG: Analytics is enabled.
2022-08-15 11:38:48,606 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp8enuwu2w']'
2022-08-15 11:38:48,607 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp8enuwu2w']'

What am I missing? Any ideas?
Thanks!

daavoo · August 17, 2022, 9:27am

Hi @leandrolanzieri !

How did you set up the remote? Could you share the output of dvc config -l --show-origin?

leandrolanzieri · August 17, 2022, 9:46am

Hi @daavoo,

To set up the remote I used:

dvc remote add -d archive_remote ssh://rodriguez@archive.inet.haw-hamburg.de/net/archive/rodriguez/aging_indicators_data/sram

The configuration looks like this:

$ dvc config -l --show-origin
.dvc/config	remote.archive_remote.url=ssh://rodriguez@archive.inet.haw-hamburg.de/net/archive/rodriguez/aging_indicators_data/sram
.dvc/config	core.remote=archive_remote

daavoo · August 17, 2022, 10:49am

Thanks @leandrolanzieri !

When you access the remote via ssh, do you provide any port/password/etc or you can access it directly with
ssh rodriguez@archive.inet.haw-hamburg.de/net/archive/rodriguez/aging_indicators_data/sram
?

p.d. Could you also try to update to the latest version pip install --upgrade 'dvc[ssh]'?

leandrolanzieri · August 18, 2022, 7:32am

Yes @daavoo, I need to enter the passphrase for my RSA key to connect. Although of course I have to connect ssh rodriguez@archive.inet.haw-hamburg.de and then change directories. From the docs I understood that the path to the storing directory should also be part of the URL, right?

It seems I have the latest version

daavoo · August 18, 2022, 7:46am

Got it.
Could you try to set your passphrase in the DVC config:

dvc remote modify --local archive_remote password YOUR_ASSPHRASE

leandrolanzieri · August 18, 2022, 7:53am

Thanks! That did the trick. I see that config.local is ignored, still a bit unsettling to have the passphrase lying around there. I would’ve expected to be prompted for it when attempting to connect. Anyways, thanks again @daavoo !

daavoo · August 18, 2022, 7:55am

I would’ve expected to be prompted for it when attempting to connect.

For that, you can instead set up the ask_password option. See all options here remote modify

leandrolanzieri · August 18, 2022, 7:56am

Awesome! I missed that part

Topic		Replies	Views
ERROR: unexpected error - [Errno 13] Permission denied Questions	0	133	April 3, 2024
Err permission denied on my private repo Questions	0	130	April 12, 2024
DVC push to SSH remote ERROR: No such file Questions	6	1125	June 5, 2024
Facing issue on dvc push Questions	6	106	June 19, 2024
ERROR: failed to reproduce ‘dvc.yaml’: [Errno 13] Permission denied Questions	3	2520	August 2, 2021

SSH remote: unexpected error - Permission denied

Related topics