`dvc import` fails with `Git failed to fetch ref from"

Jonas · March 15, 2024, 8:59pm

Hi,

I want to use dvc import to retrieve dvc-managed data from a remote git repository. The repository is located on a remote server. The data itself is located in an internal S3 remote. The S3 remote’s credentials are configured in the git repository, including the key file location. I can clone the git repo using git and use dvc pull to retrieve the data from the remote.

However, dvc import does not work as expected. It either does not seem to be able to access the remote repo itself or maybe the S3 remote (?). Short version (full output below):

user@machine:~/tmp/try_dvc_data_registry$ dvc import --verbose git@host:repo.git data
2024-03-15 20:27:02,456 DEBUG: v3.48.4 (pip), CPython 3.9.1 on Linux-6.1.0-17-amd64-x86_64-with-glibc2.36
2024-03-15 20:27:02,456 DEBUG: command: /home/user/tmp/try_dvc_data_registry/.venv/bin/dvc import --verbose git@host:repo.git data
2024-03-15 20:27:02,834 DEBUG: Removing output 'file.dill' of stage: 'file.dill.dvc'.
2024-03-15 20:27:02,834 DEBUG: Removing '/home/user/tmp/try_dvc_data_registry/file.dill'
Importing 'data (git@host:repo.git)' -> 'file.dill'
2024-03-15 20:27:02,836 DEBUG: Computed stage: 'file.dill.dvc' md5: 'd6176c15517db1426864eb7f2e39a46a'
2024-03-15 20:27:02,836 DEBUG: 'md5' of stage: 'file.dill.dvc' changed.
2024-03-15 20:27:02,836 DEBUG: Creating external repo git@host:repo.git@None
2024-03-15 20:27:02,836 DEBUG: erepo: git clone 'git@host:repo.git' to a temporary dir
2024-03-15 20:27:09,303 ERROR: failed to import 'data' from 'git@host:repo.git'. - Git failed to fetch ref from 'git@host:repo.git'
...
_pygit2.GitError: authentication required but no callback set
...
scmrepo.exceptions.SCMError: Git failed to fetch ref from 'git@host:repo.git'
...
dvc.scm.SCMError: Git failed to fetch ref from 'git@host:repo.git'

I thought it might be related to these issues

github.com/iterative/dvc

import: Git SSH keyfile auth broken in dvc>=2.44.0

opened 07:51PM - 24 Feb 23 UTC

closed 02:42PM - 03 Mar 23 UTC

ReveStobinson

p0-critical regression git

# Bug Report ## Description `dvc import` fails to authenticate to the spec…ified Git remote with an SSH keyfile (In this case, a GitLab repository on a locally hosted private instance). The following output is produced: ```console Importing 'path/to/data/ (git@internal.gitlab.url:path/to/repo.git)' -> 'rawdata' ERROR: unexpected error - Git failed to fetch ref from 'git@internal.gitlab.url:path/to/repo.git': authentication required but no callback set ``` ### Reproduce The nonfunctioning command is this: ```console dvc import git@internal.gitlab.url:path/to/repo.git \ path/to/data \ -o rawdata/ ``` The directory to be downloaded is located in a .dvc-initialized subdirectory within the remote repository in question. That location has in it's `.dvc/config` the default remote pointing to the ssh path to our dvc remote storage (On a different machine from the one hosting the GitLab instance). Prior to release 2.44.0, as long as we had our SSH config set up properly and used an SSH keyfile that did not require a password, this command would successfully download the files. Now, the above error is always encountered with any DVC version newer than 2.44.0. Downgrading to 2.43.4 or lower resolves the issue. ### Expected `dvc import` to download the files to the `./rawdata` directory, and begin tracking them with DVC (add `rawdata.dvc` file, add `/rawdata` to `.gitignore`, etc.). ### Environment information **Output of `dvc doctor`:** ```console $ dvc doctor DVC version: 2.44.0 (pip) ------------------------- Platform: Python 3.9.16 on Linux-5.15.79.1-microsoft-standard-WSL2-x86_64-with-glibc2.35 Subprojects: dvc_data = 0.39.3 dvc_objects = 0.19.3 dvc_render = 0.1.1 dvc_task = 0.1.11 dvclive = 2.0.2 scmrepo = 0.1.9 Supports: http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3), https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3), ssh (sshfs = 2023.1.0) Cache types: <https://error.dvc.org/no-dvc-cache> Caches: local Remotes: None Workspace directory: ext4 on /dev/sdc Repo: dvc, git ``` **Other potential dependencies of relevance:** ```console dvc-http 2.30.2 dvc-ssh 2.21.0 dvc-studio-client 0.5.0 asyncssh 2.13.1 sshfs 2023.1.0 gitdb 4.0.10 gitpython 3.1.31 pygit2 1.11.1 ``` **Additional Information (if any):** Additional error info from `--verbose`: ```console File "~/mambaforge/envs/dvc_2.44.0/lib/python3.9/site-packages/pygit2/errors.py", line 65, in check_error raise GitError(message) _pygit2.GitError: authentication required but no callback set ``` ### Working environment: <details> <summary>I am able to produce the expected output with the same command in this environment:</summary> ```console DVC version: 2.43.4 (pip) ------------------------- Platform: Python 3.9.16 on Linux-5.15.79.1-microsoft-standard-WSL2-x86_64-with-glibc2.35 Subprojects: dvc_data = 0.37.8 dvc_objects = 0.19.3 dvc_render = 0.1.0 dvc_task = 0.1.11 dvclive = 2.0.2 scmrepo = 0.1.7 Supports: http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3), https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3), ssh (sshfs = 2023.1.0) Cache types: <https://error.dvc.org/no-dvc-cache> Caches: local Remotes: None Workspace directory: ext4 on /dev/sdc Repo: dvc, git ``` **Additional dependencies:** (All the same as non-working env) ```console dvc-http 2.30.2 dvc-ssh 2.21.0 dvc-studio-client 0.5.0 asyncssh 2.13.1 sshfs 2023.1.0 gitdb 4.0.10 gitpython 3.1.31 pygit2 1.11.1 ``` </details>

github.com/iterative/dvc

pull: pulling from private github dataregistry repo with git credential helper not working anymore

opened 02:18PM - 13 Feb 23 UTC

closed 01:33PM - 14 Feb 23 UTC

bakaleks

bug p0-critical regression git

# Bug Report  ## Description With latest 2.44.0 version 'dvc pull' no longer works when pulling assets from private GitHub repo. Using git credential helper for authentication.  ### Reproduce ```console $ dvc pull -v 2023-02-13 15:02:05,305 DEBUG: v2.44.0 (deb), CPython 3.10.8 on Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.31 2023-02-13 15:02:05,305 DEBUG: command: pull -v 2023-02-13 15:02:06,332 DEBUG: Creating external repo https://github.com/company/repo.git@4673682286885bda923460f92cc829602 2023-02-13 15:02:06,332 DEBUG: erepo: git clone 'https://github.com/company/repo.git' to a temporary dir 2023-02-13 15:02:11,410 ERROR: unexpected error - Git failed to fetch ref from 'https://github.com/company/repo.git': remote authentication required but no callback set Traceback (most recent call last): File "funcy/flow.py", line 112, in reraise File "scmrepo/git/backend/pygit2.py", line 508, in fetch_refspecs File "pygit2/remote.py", line 146, in fetch File "pygit2/callbacks.py", line 98, in check_error File "pygit2/errors.py", line 65, in check_error _pygit2.GitError: remote authentication required but no callback set The above exception was the direct cause of the following exception: Traceback (most recent call last): File "dvc/cli/__init__.py", line 205, in main File "dvc/cli/command.py", line 26, in do_run File "dvc/commands/data_sync.py", line 31, in run File "dvc/repo/__init__.py", line 57, in wrapper File "dvc/repo/pull.py", line 34, in pull File "dvc/repo/__init__.py", line 57, in wrapper File "dvc/repo/fetch.py", line 86, in fetch File "dvc/repo/fetch.py", line 142, in _fetch File "dvc/repo/__init__.py", line 449, in used_objs File "dvc/repo/index.py", line 385, in used_objs File "dvc/stage/__init__.py", line 722, in get_used_objs File "dvc/output.py", line 1074, in get_used_objs File "dvc/output.py", line 1133, in get_used_external File "dvc/dependency/repo.py", line 102, in get_used_objs File "dvc/dependency/repo.py", line 116, in _get_used_and_obj File "contextlib.py", line 135, in __enter__ File "dvc/external_repo.py", line 45, in external_repo File "dvc/external_repo.py", line 173, in _cached_clone File "funcy/decorators.py", line 45, in wrapper File "funcy/flow.py", line 274, in wrap_with File "funcy/decorators.py", line 66, in __call__ File "dvc/external_repo.py", line 241, in _clone_default_branch File "dvc/scm.py", line 182, in clone File "dvc/repo/experiments/utils.py", line 288, in fetch_all_exps File "scmrepo/git/__init__.py", line 279, in _backend_func File "scmrepo/git/backend/pygit2.py", line 504, in fetch_refspecs File "contextlib.py", line 153, in __exit__ File "funcy/flow.py", line 116, in reraise File "<string>", line 3, in raise_from scmrepo.exceptions.SCMError: Git failed to fetch ref from 'https://github.com/company/repo.git' ```   ### Expected It worked fine with 2.43.4, suspecting https://github.com/iterative/dvc/pull/8907  ### Environment information  **Output of `dvc doctor`:** ```console $ dvc doctor DVC version: 2.44.0 (deb) ------------------------- Platform: Python 3.10.8 on Linux-3.10.0-1160.15.2.el7.x86_64-x86_64-with-glibc2.31 Subprojects: Supports: azure (adlfs = 2023.1.0, knack = 0.10.1, azure-identity = 1.12.0), gdrive (pydrive2 = 1.15.0), gs (gcsfs = 2023.1.0), hdfs (fsspec = 2023.1.0, pyarrow = 11.0.0), http (aiohttp = 3.8.3, aiohttp-retry = 2.8.3), https (aiohttp = 3.8.3, aiohttp-retry = 2.8.3), oss (ossfs = 2021.8.0), s3 (s3fs = 2023.1.0, boto3 = 1.24.59), ssh (sshfs = 2023.1.0), webdav (webdav4 = 0.9.8), webdavs (webdav4 = 0.9.8), webhdfs (fsspec = 2023.1.0) Cache types: hardlink, symlink Cache directory: xfs on /dev/mapper/rootvg-home_lv Caches: local Remotes: azure Workspace directory: xfs on /dev/mapper/rootvg-home_lv Repo: dvc, git ``` **Additional Information (if any):**

Which however both appear to be long fixed and I am using a much newer version (3.48.4, see dvc doctor below). Interestingly, though, the second issue mentions that dvc==2.45.0 might not have this issue. And indeed when I switch to that version, it seems that at least the git repo can be accessed, indicated by the fact that DVC now complains about unallowed keys that it definitely retrieved from the remote repository:

2024-03-15 20:53:23,812 ERROR: failed to import 'data/output/file.dill' from 'git@host:repo.git'. - '../../../dvc.lock' validation failed in revision '9290b7c': 30 errors: extra keys not allowed @ data['stages']['ltv']['deps'][0]['hash']

So unfortunately I cannot use this older version, as it is not compatible with the data repository that was created with a recent version.

However, this might hint that the issue is related to the linked ones, even though these have been thought fixed. Unless I am doing something horribly wrong, that is.

I’d truly appreciate any help, as we urgently need the import and related functionality. Thank you!

Best regards
Jonas

Dvc doctor

$ dvc doctor
DVC version: 3.48.4 (pip)
-------------------------
Platform: Python 3.9.1 on Linux-6.1.0-17-amd64-x86_64-with-glibc2.36
Subprojects:
  dvc_data = 3.14.1
  dvc_objects = 5.1.0
  dvc_render = 1.0.1
  dvc_task = 0.3.0
  scmrepo = 3.3.0
Supports:
  http (aiohttp = 3.9.3, aiohttp-retry = 2.8.3),
  https (aiohttp = 3.9.3, aiohttp-retry = 2.8.3)
Config:
  Global: /home/user/.config/dvc
  System: /etc/xdg/dvc
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/vda1
Repo: dvc, git

Full traceback

user@machine:~/tmp/try_dvc_data_registry$ dvc import --verbose git@host:repo.git data
2024-03-15 20:27:02,456 DEBUG: v3.48.4 (pip), CPython 3.9.1 on Linux-6.1.0-17-amd64-x86_64-with-glibc2.36
2024-03-15 20:27:02,456 DEBUG: command: /home/user/tmp/try_dvc_data_registry/.venv/bin/dvc import --verbose git@host:repo.git data
2024-03-15 20:27:02,834 DEBUG: Removing output 'file.dill' of stage: 'file.dill.dvc'.
2024-03-15 20:27:02,834 DEBUG: Removing '/home/user/tmp/try_dvc_data_registry/file.dill'
Importing 'data (git@host:repo.git)' -> 'file.dill'
2024-03-15 20:27:02,836 DEBUG: Computed stage: 'file.dill.dvc' md5: 'd6176c15517db1426864eb7f2e39a46a'
2024-03-15 20:27:02,836 DEBUG: 'md5' of stage: 'file.dill.dvc' changed.
2024-03-15 20:27:02,836 DEBUG: Creating external repo git@host:repo.git@None
2024-03-15 20:27:02,836 DEBUG: erepo: git clone 'git@host:repo.git' to a temporary dir
2024-03-15 20:27:09,303 ERROR: failed to import 'data' from 'git@host:repo.git'. - Git failed to fetch ref from 'git@host:repo.git'
Traceback (most recent call last):
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/funcy/flow.py", line 84, in reraise
    yield
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/scmrepo/git/backend/pygit2/__init__.py", line 698, in fetch_refspecs
    for head in remote.ls_remotes(callbacks=cb)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/pygit2/remotes.py", line 171, in ls_remotes
    self.connect(callbacks=callbacks, proxy=proxy)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/pygit2/remotes.py", line 120, in connect
    payload.check_error(err)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/pygit2/callbacks.py", line 99, in check_error
    check_error(error_code)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/pygit2/errors.py", line 65, in check_error
    raise GitError(message)
_pygit2.GitError: authentication required but no callback set

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/scm.py", line 59, in map_scm_exception
    yield
  File "/usr/local/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/repo/open_repo.py", line 23, in _external_repo
    path = _cached_clone(url, rev)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/repo/open_repo.py", line 134, in _cached_clone
    clone_path, shallow = _clone_default_branch(url, rev)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/funcy/decorators.py", line 47, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/funcy/flow.py", line 246, in wrap_with
    return call()
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/funcy/decorators.py", line 68, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/repo/open_repo.py", line 198, in _clone_default_branch
    git = clone(url, clone_path)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/scm.py", line 154, in clone
    fetch_all_exps(git, url, progress=pbar.update_git)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/repo/experiments/utils.py", line 280, in fetch_all_exps
    scm.fetch_refspecs(url, refspecs, progress=progress, **kwargs)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/scmrepo/git/__init__.py", line 359, in fetch_refspecs
    return self._fetch_refspecs(
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/scmrepo/git/__init__.py", line 307, in _backend_func
    result = func(*args, **kwargs)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/scmrepo/git/backend/pygit2/__init__.py", line 703, in fetch_refspecs
    remote.fetch(
  File "/usr/local/lib/python3.9/contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/funcy/flow.py", line 88, in reraise
    raise into from e
scmrepo.exceptions.SCMError: Git failed to fetch ref from 'git@host:repo.git'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/commands/imp.py", line 15, in run
    self.repo.imp(
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/repo/imp.py", line 44, in imp
    return self.imp_url(path, out=out, erepo=erepo, frozen=True, **kwargs)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/repo/__init__.py", line 58, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/repo/scm_context.py", line 143, in run
    return method(repo, *args, **kw)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/repo/imp_url.py", line 86, in imp_url
    stage.run(jobs=jobs, no_download=no_download)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/funcy/decorators.py", line 47, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/stage/decorators.py", line 44, in rwlocked
    return call()
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/funcy/decorators.py", line 68, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/stage/__init__.py", line 603, in run
    self._sync_import(dry, force, kwargs.get("jobs", None), no_download)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/funcy/decorators.py", line 47, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/stage/decorators.py", line 44, in rwlocked
    return call()
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/funcy/decorators.py", line 68, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/stage/__init__.py", line 640, in _sync_import
    sync_import(self, dry, force, jobs, no_download)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/stage/imports.py", line 56, in sync_import
    stage.save_deps()
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/stage/__init__.py", line 496, in save_deps
    dep.save()
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/dependency/repo.py", line 63, in save
    rev = self.fs.repo.get_rev()
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/fs/dvc.py", line 565, in repo
    return self.fs.repo
  File "/usr/local/lib/python3.9/functools.py", line 969, in __get__
    val = self.func(instance)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/fs/dvc.py", line 198, in repo
    repo = self._make_repo(**self._repo_kwargs)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/fs/dvc.py", line 275, in _make_repo
    with Repo.open(uninitialized=True, **kwargs) as repo:
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/repo/__init__.py", line 297, in open
    return open_repo(url, *args, **kwargs)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/repo/open_repo.py", line 60, in open_repo
    return _external_repo(url, *args, **kwargs)
  File "/usr/local/lib/python3.9/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/usr/local/lib/python3.9/contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/user/tmp/try_dvc_data_registry/.venv/lib/python3.9/site-packages/dvc/scm.py", line 64, in map_scm_exception
    raise into  # noqa: B904
dvc.scm.SCMError: Git failed to fetch ref from 'git@host:repo.git'

2024-03-15 20:27:09,309 DEBUG: Analytics is enabled.
2024-03-15 20:27:09,376 DEBUG: Trying to spawn ['daemon', 'analytics', '/tmp/tmpflvigfpp', '-v']
2024-03-15 20:27:09,388 DEBUG: Spawned ['daemon', 'analytics', '/tmp/tmpflvigfpp', '-v'] with pid 1003932

Jonas · March 18, 2024, 3:26pm

Update:

As the error message suggests, the issue is that the git repository cannot be accessed. When I clone the repository manually (no files in dvc cache) and apply dvc import based on that repo the s3 remote can be accessed.
Updating libgit2 to 1.5 did not solve the issue.

skshetry · March 20, 2024, 10:52am

Hi. We use multiple backend implementations for Git operations, mostly dulwich and pygit2.

Looking at your stack trace, it seems the git repository was successfully cloned using dulwich.
But when dvc tried to fetch some references, it tried to use pygit2 and failed.

pygit2 backend does not support ssh remotes, so it is expected to fail. But it should fallback to dulwich backend, which does not seem to be happening. We try to fallback to dulwich if url starts with git@ or has either of git://, ssh://, git+ssh:// schemes.

Could you please share what structure the url has? I believe you are using SCP url that is like git@ but does have a different username(?).

github.com

iterative/scmrepo/blob/1096899158e9d195b4f6fcae9d320ebdbaf0912e/src/scmrepo/git/backend/pygit2/init.py#L639-L655


      
          def _get_remote(self, url: str) -> Generator["Remote", None, None]:
              """Return a pygit2.Remote suitable for the specified Git URL or remote name."""
              try:
                  remote = self.repo.remotes[url]
                  url = remote.url
              except ValueError:
                  pass
              except KeyError as exc:
                  raise SCMError(f"'{url}' is not a valid Git remote or URL") from exc
          
              if os.name == "nt" and url.startswith("file://"):
                  url = url[len("file://") :]
              remote = self.repo.remotes.create_anonymous(url)
              parsed = urlparse(remote.url)
              if parsed.scheme in ("git", "git+ssh", "ssh") or remote.url.startswith("git@"):
                  raise NotImplementedError
              yield remote

Also please check if you can import with urls with either of git://, ssh://, or git+ssh:// schemes.

Jonas · March 20, 2024, 5:49pm

Hi!

Thanks for the hints! It did not solve my problem immediately, but prompted me to play with dulwich a bit, which ultimately lead to the solution!

Two things:

As you mentioned, I added ssh:// to make dvc fall back to dulwich. This was required because my url actually starts with gitea@, not git@, so the fallback did not happen without it. So my url became ssh://gitea@git.something.org:FOO/myrepo.git.
This still did not work, giving gaierror: [Errno -2] Name or service not known. It turns out that dulwich cannot handle a colon between server address and path to the repo, as used in SSH urls I suppose. I changed the colon to a forward slash ssh://gitea@git.something.org/FOO/myrepo.git and that made it work!

Thanks a lot, you have helped me a great deal!

Best regards
Jonas

skshetry · March 21, 2024, 5:36am

Sorry, I forgot to mention that you have to replace the colon with / in ssh urls.

I have created a fix in support scp-style shorthand urls with users other than git@ by skshetry · Pull Request #346 · iterative/scmrepo · GitHub. It’d be great if you can test it out and see if it fixes the issue for you.

To install from the PR, use the following command:

pip install "scmrepo @ git+https://github.com/iterative/scmrepo@refs/pull/346/merge"

Jonas · April 8, 2024, 12:20pm

Hi! Sorry, for the delayed reply and thanks a lot for the fix. I can’t test it right now though, but I’ll come back to this as soon as possible and will let you know then.

Topic		Replies	Views
Import/list + webdavs remote not working Questions	5	544	July 18, 2022
Important: ERROR: failed to pull data from the cloud Questions	10	110	February 16, 2024
Need Help -- Fail to pull data when updating data with DVC v3 but Initialize data with DVC v2 Questions	0	66	February 16, 2024
DVC pull fails without sudo Bug Reports	8	4341	February 13, 2019
"dvc add -external S3://mybucket/data.csv" is failing with access error even after giving correct remote cache configurations Questions	9	2176	April 16, 2021

`dvc import` fails with `Git failed to fetch ref from"

Related Topics