SSL error on getting dataset on Gist URL

flashpixx · February 29, 2024, 9:16am

Hello, I’m starting my first project with DVC. I have installed dvc with Poetry in an VirtualEnv and I have run dvc init , so my .dvc folder has been created in my git repo. I try to get the iris dataset from a github repo with dvc get https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv notebooks/data/iris.csv and I get SSL issues

HTTPSConnectionPool(host='gist.githubusercontent.com', port=443): 
Max retries exceeded with url: /curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv/info/refs?service=git-upload-pack
(Caused by SSLError(SSLError(5, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1007)'))): HTTPSConnectionPool(host='gist.githubusercontent.com', port=443): 
Max retries exceeded with url: /curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv/info/refs?service=git-upload-pack
(Caused by SSLError(SSLError(5, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1007)'))): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1007)

dvd doctor shows:

DVC version: 3.47.0 (pip)
-------------------------
Platform: Python 3.10.13 on macOS-14.2.1-x86_64-i386-64bit
Subprojects:
        dvc_data = 3.13.0
        dvc_objects = 5.0.0
        dvc_render = 1.0.1
        dvc_task = 0.3.0
        scmrepo = 3.1.0
Supports:
        http (aiohttp = 3.9.3, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.9.3, aiohttp-retry = 2.8.3)
Config:
        Global: /Users/me/Library/Application Support/dvc
        System: /Library/Application Support/dvc
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: apfs on /dev/disk1s2s1
Repo: dvc, git
Repo.site_cache_dir: /Library/Caches/dvc/repo/cfda013bb5e55c21fb0c87135b3f016b

I have tried to install certifi but this does not help, also I try import-url with the same issue.
How can I fix this?

shcheklein · February 29, 2024, 10:11pm

Usually, this error that you are getting means that you are within some corporate environment and there custom SSL certificates are involved.

Also, I’m curios why are using https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv as an URL, not the repo URL like it should be?

Also, could you run with -v to see where exactly it happens
.

flashpixx · March 1, 2024, 8:13am

Is there a better way to import a Gist?

Here are the logs:

with dvc get -v

2024-03-01 09:03:53,775 DEBUG: v3.47.0 (pip), CPython 3.10.13 on macOS-14.2.1-x86_64-i386-64bit
2024-03-01 09:03:53,775 DEBUG: command: /Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/bin/dvc get -v https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv notebooks/data/iris.csv
2024-03-01 09:03:53,962 DEBUG: Creating external repo https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv@None
2024-03-01 09:03:53,962 DEBUG: erepo: git clone 'https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv' to a temporary dir
2024-03-01 09:03:54,232 ERROR: failed to get 'notebooks/data/iris.csv' - SCM error: Failed to clone repo 'https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv' to '/var/folders/jf/01x96jrn3yg8jlnt__2yqb3c0000gn/T/tmp_thz0n1idvc-clone': HTTPSConnectionPool(host='gist.githubusercontent.com', port=443): Max retries exceeded with url: /curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv/info/refs?service=git-upload-pack (Caused by SSLError(SSLError(5, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1007)'))): HTTPSConnectionPool(host='gist.githubusercontent.com', port=443): Max retries exceeded with url: /curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv/info/refs?service=git-upload-pack (Caused by SSLError(SSLError(5, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1007)'))): [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1007)
Traceback (most recent call last):
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1099, in _validate_conn
    conn.connect()
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/connection.py", line 653, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/connection.py", line 806, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 465, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 509, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Users/me/.pyenv/versions/3.10.13/lib/python3.10/ssl.py", line 513, in wrap_socket
    return self.sslsocket_class._create(
  File "/Users/me/.pyenv/versions/3.10.13/lib/python3.10/ssl.py", line 1104, in _create
    self.do_handshake()
  File "/Users/me/.pyenv/versions/3.10.13/lib/python3.10/ssl.py", line 1375, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1007)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 793, in urlopen
    response = self._make_request(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 491, in _make_request
    raise new_e
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1007)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dulwich/client.py", line 2279, in _http_request
    resp = self.pool_manager.request(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/_request_methods.py", line 136, in request
    return self.request_encode_url(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/_request_methods.py", line 183, in request_encode_url
    return self.urlopen(method, url, **extra_kw)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/poolmanager.py", line 444, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 877, in urlopen
    return self.urlopen(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 877, in urlopen
    return self.urlopen(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 877, in urlopen
    return self.urlopen(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/connectionpool.py", line 847, in urlopen
    retries = retries.increment(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/urllib3/util/retry.py", line 515, in increment
    raise MaxRetryError(_pool, url, reason) from reason  # type: ignore[arg-type]
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='gist.githubusercontent.com', port=443): Max retries exceeded with url: /curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv/info/refs?service=git-upload-pack (Caused by SSLError(SSLError(5, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1007)')))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 260, in clone
    repo = clone_from()
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dulwich/porcelain.py", line 546, in clone
    return client.clone(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dulwich/client.py", line 753, in clone
    result = self.fetch(path, target, progress=progress, depth=depth)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dulwich/client.py", line 839, in fetch
    result = self.fetch_pack(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dulwich/client.py", line 2156, in fetch_pack
    refs, server_capabilities, url = self._discover_references(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dulwich/client.py", line 2012, in _discover_references
    resp, read = self._http_request(url, headers)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/scmrepo/git/backend/dulwich/client.py", line 50, in _http_request
    result = super()._http_request(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dulwich/client.py", line 2287, in _http_request
    raise GitProtocolError(str(e)) from e
dulwich.errors.GitProtocolError: HTTPSConnectionPool(host='gist.githubusercontent.com', port=443): Max retries exceeded with url: /curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv/info/refs?service=git-upload-pack (Caused by SSLError(SSLError(5, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:1007)')))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/scm.py", line 152, in clone
    git = Git.clone(url, to_path, progress=pbar.update_git, **kwargs)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/scmrepo/git/__init__.py", line 154, in clone
    backend.clone(url, to_path, bare=bare, mirror=mirror, **kwargs)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 268, in clone
    raise CloneError(url, to_path) from exc
scmrepo.exceptions.CloneError: Failed to clone repo 'https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv' to '/var/folders/jf/01x96jrn3yg8jlnt__2yqb3c0000gn/T/tmp_thz0n1idvc-clone'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/commands/get.py", line 31, in _get_file_from_repo
    Repo.get(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/repo/get.py", line 45, in get
    with Repo.open(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/repo/__init__.py", line 295, in open
    return open_repo(url, *args, **kwargs)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/repo/open_repo.py", line 60, in open_repo
    return _external_repo(url, *args, **kwargs)
  File "/Users/me/.pyenv/versions/3.10.13/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/repo/open_repo.py", line 23, in _external_repo
    path = _cached_clone(url, rev)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/repo/open_repo.py", line 134, in _cached_clone
    clone_path, shallow = _clone_default_branch(url, rev)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/funcy/decorators.py", line 47, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/funcy/flow.py", line 246, in wrap_with
    return call()
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/funcy/decorators.py", line 68, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/repo/open_repo.py", line 198, in _clone_default_branch
    git = clone(url, clone_path)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/scm.py", line 157, in clone
    raise CloneError("SCM error") from exc
dvc.scm.CloneError: SCM error

2024-03-01 09:03:54,266 DEBUG: Analytics is enabled.
2024-03-01 09:03:54,317 DEBUG: Trying to spawn ['daemon', 'analytics', '/var/folders/jf/01x96jrn3yg8jlnt__2yqb3c0000gn/T/tmpvwy3yk6m', '-v']
2024-03-01 09:03:54,326 DEBUG: Spawned ['daemon', 'analytics', '/var/folders/jf/01x96jrn3yg8jlnt__2yqb3c0000gn/T/tmpvwy3yk6m', '-v'] with pid 95354

with dvc import-url -v

2024-03-01 09:14:28,834 DEBUG: v3.47.0 (pip), CPython 3.10.13 on macOS-14.2.1-x86_64-i386-64bit
2024-03-01 09:14:28,834 DEBUG: command: /Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/bin/dvc import-url -v https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv notebooks/data/iris.csv
2024-03-01 09:14:29,537 DEBUG: Removing output 'notebooks/data/iris.csv' of stage: 'notebooks/data/iris.csv.dvc'.
2024-03-01 09:14:29,537 DEBUG: Removing '/Users/me/Desktop/myproject/notebooks/data/iris.csv'
Importing 'https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv' -> 'notebooks/data/iris.csv'
2024-03-01 09:14:29,539 DEBUG: Computed stage: 'notebooks/data/iris.csv.dvc' md5: 'ad450e36e243fb64c8b5a3d946881b64'
2024-03-01 09:14:29,539 DEBUG: 'md5' of stage: 'notebooks/data/iris.csv.dvc' changed.
2024-03-01 09:14:30,602 ERROR: failed to import https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv. You could also try downloading it manually, and adding it with `dvc add`. - dependency 'https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv' does not exist
Traceback (most recent call last):
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/commands/imp_url.py", line 13, in run
    self.repo.imp_url(
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/repo/__init__.py", line 58, in wrapper
    return f(repo, *args, **kwargs)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/repo/scm_context.py", line 143, in run
    return method(repo, *args, **kw)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/repo/imp_url.py", line 86, in imp_url
    stage.run(jobs=jobs, no_download=no_download)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/funcy/decorators.py", line 47, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/stage/decorators.py", line 44, in rwlocked
    return call()
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/funcy/decorators.py", line 68, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/stage/__init__.py", line 603, in run
    self._sync_import(dry, force, kwargs.get("jobs", None), no_download)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/funcy/decorators.py", line 47, in wrapper
    return deco(call, *dargs, **dkwargs)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/stage/decorators.py", line 44, in rwlocked
    return call()
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/funcy/decorators.py", line 68, in __call__
    return self._func(*self._args, **self._kwargs)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/stage/__init__.py", line 640, in _sync_import
    sync_import(self, dry, force, jobs, no_download)
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/stage/imports.py", line 56, in sync_import
    stage.save_deps()
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/stage/__init__.py", line 496, in save_deps
    dep.save()
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/dependency/base.py", line 51, in save
    super().save()
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/output.py", line 683, in save
    raise self.DoesNotExistError(self)
dvc.dependency.base.DependencyDoesNotExistError: dependency 'https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv' does not exist

2024-03-01 09:14:30,611 DEBUG: Analytics is enabled.
2024-03-01 09:14:30,687 DEBUG: Trying to spawn ['daemon', 'analytics', '/var/folders/jf/01x96jrn3yg8jlnt__2yqb3c0000gn/T/tmpxms7rjqk', '-v']
2024-03-01 09:14:30,698 DEBUG: Spawned ['daemon', 'analytics', '/var/folders/jf/01x96jrn3yg8jlnt__2yqb3c0000gn/T/tmpxms7rjqk', '-v'] with pid 1105

skshetry · March 1, 2024, 8:25am

Hi, you can import from gist in a couple of ways.

Using git

To import as a git source, you’ll need to copy HTTPS or SSH url from the gist and then:

dvc import <HTTPS_OR_SSH_URL> <file_name>

eg:

dvc import https://gist.github.com/a08a1080b88344b0c8a7.git iris.csv

Using raw url

You can use import-url with GitHub’s raw url (one that starts with gist.githubusercontent.com).

dvc import-url https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv

The dvc get is failing because dvc get works with git repositories. For HTTP URL, you need to use dvc get-url.

You are using import-url correctly. My guess would be that the gist is private? Is that the case?

flashpixx · March 1, 2024, 9:37am

The Gist is public accessable

so I try to use the raw url with

dvc import-url -v https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv notebooks/data/iris.csv

and it shows this error log (shorten)

2024-03-01 10:29:29,993 DEBUG: Removing '/Users/me/Desktop/myproject/notebooks/data/iris.csv'
Importing 'https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv' -> 'notebooks/data/iris.csv'
2024-03-01 10:29:29,994 DEBUG: Computed stage: 'notebooks/data/iris.csv.dvc' md5: 'ad450e36e243fb64c8b5a3d946881b64'
2024-03-01 10:29:29,994 DEBUG: 'md5' of stage: 'notebooks/data/iris.csv.dvc' changed.
2024-03-01 10:29:30,990 ERROR: failed to import https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv. You could also try downloading it manually, and adding it with `dvc add`. - dependency 'https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv' does not exist
[...]
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/dependency/base.py", line 51, in save
    super().save()
  File "/Users/me/Library/Caches/pypoetry/virtualenvs/myproject-ft82oytK-py3.10/lib/python3.10/site-packages/dvc/output.py", line 683, in save
    raise self.DoesNotExistError(self)
dvc.dependency.base.DependencyDoesNotExistError: dependency 'https://gist.githubusercontent.com/curran/a08a1080b88344b0c8a7/raw/0e7a9b0a5d22642a06d3d5b9bcbad9890c8ee534/iris.csv' does not exist

skshetry · March 1, 2024, 1:17pm

hmm, I am able to download it without any issue using the command that you have above. Are you behind any kind of proxy or a firewall?

Are you able to use curl without any issues?

flashpixx · March 1, 2024, 5:35pm

This was a good hint, my CA-Cert-Bundle seems to be corrupted, so I have reinstalled it and this solve the problem. Thanks a lot

Topic		Replies	Views
Unexpected error [Errno2] No storage files available Questions	2	27	October 9, 2024
SSL: Certificate_Verify_Failed Questions	1	31	September 19, 2024
Dvc push error on Self signed certificate Questions	5	1136	March 1, 2023
Important: ERROR: failed to pull data from the cloud Questions	10	352	February 16, 2024
DVC list shows libssl-so.1.1.1k not found Questions	1	495	November 15, 2022

SSL error on getting dataset on Gist URL

Related topics