ERROR: unexpected error - 'retries'

seif · February 10, 2022, 7:16am

Hi, i run dvc from inside a docker container ,i call dvc commands from python script , inside the container ,everything is configured ,repo ,remote storage and remote credentials, when i run the script without the container it works perfectly ,but when using the container it raise this error
when it comes to “dvc push” in the script “ERROR: unexpected error - ‘retries’” ,the error obviously not informative ,so please need a help for that

skshetry · February 10, 2022, 10:20am

Could you please share the output of dvc push -v? It’ll have the error traceback and more logging information. Thanks.

seif · February 10, 2022, 10:47am

here is the complete trace from command line when running the docker container

dvc pushing …
dvc started pushing the data…

2022-02-10 10:42:18,096 DEBUG: Adding '/usr/src/app/.dvc/config.local' to gitignore file.
2022-02-10 10:42:18,112 DEBUG: Adding '/usr/src/app/.dvc/tmp' to gitignore file.
2022-02-10 10:42:18,115 DEBUG: Adding '/usr/src/app/.dvc/cache' to gitignore file.
2022-02-10 10:43:01,608 DEBUG: Preparing to transfer data from '/usr/src/app/.dvc/cache' to 'drayd-pipeline-dev/Data'
2022-02-10 10:43:01,609 DEBUG: Preparing to collect status from 'drayd-pipeline-dev/Data'
2022-02-10 10:43:01,609 DEBUG: Collecting status from 'drayd-pipeline-dev/Data'
2022-02-10 10:43:02,716 ERROR: unexpected error - 'retries'
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/usr/local/lib/python3.7/site-packages/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/usr/local/lib/python3.7/site-packages/dvc/command/data_sync.py", line 67, in run
    glob=self.args.glob,
  File "/usr/local/lib/python3.7/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dvc/repo/push.py", line 57, in push
    obj_ids, jobs, remote=remote, odb=odb or dest_odb
  File "/usr/local/lib/python3.7/site-packages/dvc/data_cloud.py", line 91, in push
    cache_odb=self.repo.odb.local,
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/transfer.py", line 153, in transfer
    status = compare_status(src, dest, obj_ids, check_deleted=False, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/status.py", line 159, in compare_status
    dest, obj_ids, index=dest_index, **kwargs
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/status.py", line 131, in status
    exists.update(odb.hashes_exist(hashes, name=odb.fs_path, **kwargs))
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/db/base.py", line 474, in hashes_exist
    remote_size, remote_hashes = self._estimate_remote_size(hashes, name)
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/db/base.py", line 271, in _estimate_remote_size
    remote_hashes = set(hashes)
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/db/base.py", line 225, in _hashes_with_limit
    for hash_ in self._list_hashes(prefix, progress_callback):
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/db/base.py", line 215, in _list_hashes
    for path in self._list_paths(prefix, progress_callback):
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/db/base.py", line 195, in _list_paths
    for file_info in self.fs.find(fs_path, prefix=prefix):
  File "/usr/local/lib/python3.7/site-packages/dvc/fs/fsspec_wrapper.py", line 179, in find
    files = self.fs.find(with_prefix, prefix=self.path.parts(path)[-1])
  File "/usr/local/lib/python3.7/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/usr/local/lib/python3.7/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 665, in _find
    out = await self._lsdir(path, delimiter="", prefix=prefix)
  File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 579, in _lsdir
    s3 = await self.get_s3(bucket)
  File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 247, in get_s3
    return await self._s3creator.get_bucket_client(bucket)
  File "/usr/local/lib/python3.7/site-packages/s3fs/utils.py", line 65, in get_bucket_client
    response = await general_client.head_bucket(Bucket=bucket_name)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/client.py", line 174, in _make_api_call
    operation_model, request_dict, request_context)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/client.py", line 193, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/endpoint.py", line 83, in _send_request
    exception):
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/endpoint.py", line 216, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/hooks.py", line 29, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/retryhandler.py", line 189, in __call__
    retries_context = kwargs['request_dict']['context']['retries']
KeyError: 'retries'
------------------------------------------------------------
2022-02-10 10:43:03,070 DEBUG: Adding '/usr/src/app/.dvc/config.local' to gitignore file.
2022-02-10 10:43:03,086 DEBUG: Adding '/usr/src/app/.dvc/tmp' to gitignore file.
2022-02-10 10:43:03,091 DEBUG: Adding '/usr/src/app/.dvc/cache' to gitignore file.
2022-02-10 10:43:03,115 DEBUG: [Errno 95] no more link types left to try out: [Errno 95] 'reflink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>: [Errno 18] Invalid cross-device link
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/usr/local/lib/python3.7/site-packages/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/usr/local/lib/python3.7/site-packages/dvc/command/data_sync.py", line 67, in run
    glob=self.args.glob,
  File "/usr/local/lib/python3.7/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dvc/repo/push.py", line 57, in push
    obj_ids, jobs, remote=remote, odb=odb or dest_odb
  File "/usr/local/lib/python3.7/site-packages/dvc/data_cloud.py", line 91, in push
    cache_odb=self.repo.odb.local,
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/transfer.py", line 153, in transfer
    status = compare_status(src, dest, obj_ids, check_deleted=False, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/status.py", line 159, in compare_status
    dest, obj_ids, index=dest_index, **kwargs
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/status.py", line 131, in status
    exists.update(odb.hashes_exist(hashes, name=odb.fs_path, **kwargs))
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/db/base.py", line 474, in hashes_exist
    remote_size, remote_hashes = self._estimate_remote_size(hashes, name)
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/db/base.py", line 271, in _estimate_remote_size
    remote_hashes = set(hashes)
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/db/base.py", line 225, in _hashes_with_limit
    for hash_ in self._list_hashes(prefix, progress_callback):
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/db/base.py", line 215, in _list_hashes
    for path in self._list_paths(prefix, progress_callback):
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/db/base.py", line 195, in _list_paths
    for file_info in self.fs.find(fs_path, prefix=prefix):
  File "/usr/local/lib/python3.7/site-packages/dvc/fs/fsspec_wrapper.py", line 179, in find
    files = self.fs.find(with_prefix, prefix=self.path.parts(path)[-1])
  File "/usr/local/lib/python3.7/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/usr/local/lib/python3.7/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 665, in _find
    out = await self._lsdir(path, delimiter="", prefix=prefix)
  File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 579, in _lsdir
    s3 = await self.get_s3(bucket)
  File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 247, in get_s3
    return await self._s3creator.get_bucket_client(bucket)
  File "/usr/local/lib/python3.7/site-packages/s3fs/utils.py", line 65, in get_bucket_client
    response = await general_client.head_bucket(Bucket=bucket_name)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/client.py", line 174, in _make_api_call
    operation_model, request_dict, request_context)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/client.py", line 193, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/endpoint.py", line 83, in _send_request
    exception):
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/endpoint.py", line 216, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/hooks.py", line 29, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/retryhandler.py", line 189, in __call__
    retries_context = kwargs['request_dict']['context']['retries']
KeyError: 'retries'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/dvc/fs/utils.py", line 28, in _link
    func(from_path, to_path)
  File "/usr/local/lib/python3.7/site-packages/dvc/fs/local.py", line 148, in reflink
    System.reflink(from_info, to_info)
  File "/usr/local/lib/python3.7/site-packages/dvc/system.py", line 112, in reflink
    System._reflink_linux(source, link_name)
  File "/usr/local/lib/python3.7/site-packages/dvc/system.py", line 96, in _reflink_linux
    fcntl.ioctl(d.fileno(), FICLONE, s.fileno())
OSError: [Errno 18] Invalid cross-device link

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/dvc/fs/utils.py", line 69, in _try_links
    return _link(link, from_fs, from_path, to_fs, to_path)
  File "/usr/local/lib/python3.7/site-packages/dvc/fs/utils.py", line 34, in _link
    ) from exc
OSError: [Errno 95] 'reflink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/dvc/fs/utils.py", line 124, in _test_link
    _try_links([link], from_fs, from_file, to_fs, to_file)
  File "/usr/local/lib/python3.7/site-packages/dvc/fs/utils.py", line 79, in _try_links
    ) from error
OSError: [Errno 95] no more link types left to try out
------------------------------------------------------------
2022-02-10 10:43:03,118 DEBUG: Removing '/usr/src/.6xKEDniecvveNNhe48SRTv.tmp'
2022-02-10 10:43:03,126 DEBUG: [Errno 95] no more link types left to try out: [Errno 95] 'hardlink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>: [Errno 18] Invalid cross-device link: '/usr/src/app/.dvc/cache/.Yo26z2kNKRkG4gowQEitca.tmp' -> '/usr/src/.6xKEDniecvveNNhe48SRTv.tmp'
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/usr/local/lib/python3.7/site-packages/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/usr/local/lib/python3.7/site-packages/dvc/command/data_sync.py", line 67, in run
    glob=self.args.glob,
  File "/usr/local/lib/python3.7/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dvc/repo/push.py", line 57, in push
    obj_ids, jobs, remote=remote, odb=odb or dest_odb
  File "/usr/local/lib/python3.7/site-packages/dvc/data_cloud.py", line 91, in push
    cache_odb=self.repo.odb.local,
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/transfer.py", line 153, in transfer
    status = compare_status(src, dest, obj_ids, check_deleted=False, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/status.py", line 159, in compare_status
    dest, obj_ids, index=dest_index, **kwargs
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/status.py", line 131, in status
    exists.update(odb.hashes_exist(hashes, name=odb.fs_path, **kwargs))
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/db/base.py", line 474, in hashes_exist
    remote_size, remote_hashes = self._estimate_remote_size(hashes, name)
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/db/base.py", line 271, in _estimate_remote_size
    remote_hashes = set(hashes)
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/db/base.py", line 225, in _hashes_with_limit
    for hash_ in self._list_hashes(prefix, progress_callback):
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/db/base.py", line 215, in _list_hashes
    for path in self._list_paths(prefix, progress_callback):
  File "/usr/local/lib/python3.7/site-packages/dvc/objects/db/base.py", line 195, in _list_paths
    for file_info in self.fs.find(fs_path, prefix=prefix):
  File "/usr/local/lib/python3.7/site-packages/dvc/fs/fsspec_wrapper.py", line 179, in find
    files = self.fs.find(with_prefix, prefix=self.path.parts(path)[-1])
  File "/usr/local/lib/python3.7/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/usr/local/lib/python3.7/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 665, in _find
    out = await self._lsdir(path, delimiter="", prefix=prefix)
  File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 579, in _lsdir
    s3 = await self.get_s3(bucket)
  File "/usr/local/lib/python3.7/site-packages/s3fs/core.py", line 247, in get_s3
    return await self._s3creator.get_bucket_client(bucket)
  File "/usr/local/lib/python3.7/site-packages/s3fs/utils.py", line 65, in get_bucket_client
    response = await general_client.head_bucket(Bucket=bucket_name)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/client.py", line 174, in _make_api_call
    operation_model, request_dict, request_context)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/client.py", line 193, in _make_request
    return await self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/endpoint.py", line 83, in _send_request
    exception):
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/endpoint.py", line 216, in _needs_retry
    caught_exception=caught_exception, request_dict=request_dict)
  File "/usr/local/lib/python3.7/site-packages/aiobotocore/hooks.py", line 29, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/retryhandler.py", line 189, in __call__
    retries_context = kwargs['request_dict']['context']['retries']
KeyError: 'retries'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/dvc/fs/utils.py", line 28, in _link
    func(from_path, to_path)
  File "/usr/local/lib/python3.7/site-packages/dvc/fs/local.py", line 141, in hardlink
    System.hardlink(from_info, to_info)
  File "/usr/local/lib/python3.7/site-packages/dvc/system.py", line 39, in hardlink
    os.link(src, link_name)
OSError: [Errno 18] Invalid cross-device link: '/usr/src/app/.dvc/cache/.Yo26z2kNKRkG4gowQEitca.tmp' -> '/usr/src/.6xKEDniecvveNNhe48SRTv.tmp'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/dvc/fs/utils.py", line 69, in _try_links
    return _link(link, from_fs, from_path, to_fs, to_path)
  File "/usr/local/lib/python3.7/site-packages/dvc/fs/utils.py", line 34, in _link
    ) from exc
OSError: [Errno 95] 'hardlink' is not supported by <class 'dvc.fs.local.LocalFileSystem'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/dvc/fs/utils.py", line 124, in _test_link
    _try_links([link], from_fs, from_file, to_fs, to_file)
  File "/usr/local/lib/python3.7/site-packages/dvc/fs/utils.py", line 79, in _try_links
    ) from error
OSError: [Errno 95] no more link types left to try out
------------------------------------------------------------
2022-02-10 10:43:03,127 DEBUG: Removing '/usr/src/.6xKEDniecvveNNhe48SRTv.tmp'
2022-02-10 10:43:03,128 DEBUG: Removing '/usr/src/.6xKEDniecvveNNhe48SRTv.tmp'
2022-02-10 10:43:03,131 DEBUG: Removing '/usr/src/app/.dvc/cache/.Yo26z2kNKRkG4gowQEitca.tmp'
2022-02-10 10:43:03,177 DEBUG: Version info for developers:
DVC version: 2.9.3 (pip)
---------------------------------
Platform: Python 3.7.4 on Linux-5.10.16.3-microsoft-standard-WSL2-x86_64-with-debian-10.1
Supports:
        webhdfs (fsspec = 2022.1.0),
        http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
        s3 (s3fs = 2022.1.0, boto3 = 1.20.49)
Cache types: symlink
Cache directory: 9p on C:\
Caches: local
Remotes: s3
Workspace directory: 9p on C:\
Repo: dvc, git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2022-02-10 10:43:03,185 DEBUG: Analytics is enabled.
2022-02-10 10:43:03,276 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpfbhas6pl']'
2022-02-10 10:43:03,277 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpfbhas6pl']'

dvc finished pushing …

Topic		Replies	Views
Facing issue on dvc push Questions	6	106	June 19, 2024
Unexpected error when push to SSH remote Questions	3	1516	June 16, 2023
Error with DVC run in windows docker container Questions	1	845	October 16, 2020
DVC push gives "ERROR: unexpected error - [Errno 21] Is a directory" Bug Reports	3	538	November 21, 2023
ERROR: failed to reproduce 'prepare': [ERRNO 16] Device or resource busy: '/train/data/prepare' Questions	2	540	July 11, 2023

ERROR: unexpected error - 'retries'

Related topics