How use DVC with Azure Blob Storage

DVC supports AWS S3, Google Cloud Storage, Microsoft Azure Blob Storage. However, the instruction is only for AWS S3. How to use DVC with Microsoft Azure Blob Storage?

Hi @felkost !

I’ve seen you asked it on discord too :slightly_smiling_face: , but just for the record: Azure example is available here https://dvc.org/doc/commands-reference/remote-add .

Thanks,
Ruslan

1 Like

The above link doesn’t work.

In particular, I am interested in using dvc remote with a SAS token. When I add the azureblob remote via https:// link to DVC and initiate dvc push, it returns '405 The resource doesn't support specified Http Verb.' error.

To sum app, providing remote as:

  https://wasburl/container/?sig=<sastoken>

results in

  dvc.exceptions.HTTPError: '405 The resource doesn't support specified Http Verb.'

What I read in different forums, it appears Azure requires Cross-Origin Resource Sharing (CORS) headers and DVC doesn’t support it. I guess this post could be a feature request to DVC to include CORS headers.

Any ideas?

===========================
For the record the output log:

SHORT/REGULAR OUTPUT:

(base) ➜  DVC git:(master) ✗ dvc push                                                                                                                                                                                                                                                               [9:36:28]
ERROR: failed to upload '.dvc/cache/38/6f68f067e4c588bf4dcc96afdbece7' to 'https://<wasb>/<folder>/38/6f68f067e4c588bf4dcc96afdbece7' - '405 The resource doesn't support specified Http Verb.'
ERROR: failed to upload '.dvc/cache/d4/1d8cd98f00b204e9800998ecf8427e' to 'https://<wasb>/<folder>/d4/1d8cd98f00b204e9800998ecf8427e' - '405 The resource doesn't support specified Http Verb.'
ERROR: failed to upload '.dvc/cache/66/52640f2cde8646134c9b5cdfb4583f.dir' to 'https://<wasb>/<folder>/66/52640f2cde8646134c9b5cdfb4583f.dir'
ERROR: failed to push data to the cloud - 3 files failed to upload

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

VERBOSE OUTPUT:

(base) ➜  DVC git:(master) ✗ dvc push --verbose                                                                                                                                                                                                                                                     [9:36:40]
2020-09-04 21:40:27,553 DEBUG: Check for update is enabled.
2020-09-04 21:40:27,559 DEBUG: fetched: [(3,)]
2020-09-04 21:40:27,576 DEBUG: Assuming '/Users/<user>/GitHubs/DVC/.dvc/cache/66/52640f2cde8646134c9b5cdfb4583f.dir' is unchanged since it is read-only
2020-09-04 21:40:27,578 DEBUG: Assuming '/Users/<user>/GitHubs/DVC/.dvc/cache/66/52640f2cde8646134c9b5cdfb4583f.dir' is unchanged since it is read-only
2020-09-04 21:40:27,581 DEBUG: Preparing to upload data to 'https://<wasb>/<folder>/?sig=<SAStoken>
2020-09-04 21:40:27,581 DEBUG: Preparing to collect status from https://<wasb>/<folder>/?sig=<SAStoken>
2020-09-04 21:40:27,581 DEBUG: Collecting information from local cache...
2020-09-04 21:40:27,582 DEBUG: Assuming '/Users/<user>/GitHubs/DVC/.dvc/cache/38/6f68f067e4c588bf4dcc96afdbece7' is unchanged since it is read-only
2020-09-04 21:40:27,601 DEBUG: Assuming '/Users/<user>/GitHubs/DVC/.dvc/cache/66/52640f2cde8646134c9b5cdfb4583f.dir' is unchanged since it is read-only
2020-09-04 21:40:27,602 DEBUG: Assuming '/Users/<user>/GitHubs/DVC/.dvc/cache/d4/1d8cd98f00b204e9800998ecf8427e' is unchanged since it is read-only
2020-09-04 21:40:27,602 DEBUG: Collecting information from remote cache...
2020-09-04 21:40:27,603 DEBUG: Querying 1 hashes via object_exists
2020-09-04 21:40:28,030 DEBUG: Matched '0' indexed hashes
2020-09-04 21:40:28,030 DEBUG: Querying 3 hashes via object_exists
2020-09-04 21:40:28,328 DEBUG: Uploading '.dvc/cache/d4/1d8cd98f00b204e9800998ecf8427e' to 'https://<wasb>/<folder>/d4/1d8cd98f00b204e9800998ecf8427e'
2020-09-04 21:40:28,328 DEBUG: Uploading '.dvc/cache/38/6f68f067e4c588bf4dcc96afdbece7' to https://'<wasb>/<folder>/38/6f68f067e4c588bf4dcc96afdbece7'
2020-09-04 21:40:28,556 ERROR: failed to upload '.dvc/cache/38/6f68f067e4c588bf4dcc96afdbece7' to 'https://<wasb>/<folder>/38/6f68f067e4c588bf4dcc96afdbece7' - '405 The resource doesn't support specified Http Verb.'
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/cache/local.py", line 30, in wrapper
    func(from_info, to_info, *args, **kwargs)
  File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/tree/base.py", line 391, in upload
    self._upload(  # noqa, pylint: disable=no-member
  File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/tree/http.py", line 195, in _upload
    raise HTTPError(response.status_code, response.reason)
dvc.exceptions.HTTPError: '405 The resource doesn't support specified Http Verb.'
------------------------------------------------------------
2020-09-04 21:40:28,558 ERROR: failed to upload '.dvc/cache/d4/1d8cd98f00b204e9800998ecf8427e' to 'https://<wasb>/<folder>/d4/1d8cd98f00b204e9800998ecf8427e' - '405 The resource doesn't support specified Http Verb.'
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/cache/local.py", line 30, in wrapper
    func(from_info, to_info, *args, **kwargs)
  File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/tree/base.py", line 391, in upload
    self._upload(  # noqa, pylint: disable=no-member
  File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/tree/http.py", line 195, in _upload
    raise HTTPError(response.status_code, response.reason)
dvc.exceptions.HTTPError: '405 The resource doesn't support specified Http Verb.'
------------------------------------------------------------
2020-09-04 21:40:28,561 DEBUG: failed to upload full contents of 'data/prepared', aborting .dir file upload
2020-09-04 21:40:28,562 ERROR: failed to upload '.dvc/cache/66/52640f2cde8646134c9b5cdfb4583f.dir' to 'https://<wasb>/<folder>/66/52640f2cde8646134c9b5cdfb4583f.dir'
2020-09-04 21:40:28,563 DEBUG: fetched: [(31,)]
2020-09-04 21:40:28,564 ERROR: failed to push data to the cloud - 3 files failed to upload
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/command/data_sync.py", line 50, in run
    processed_files_count = self.repo.push(
  File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/repo/__init__.py", line 34, in wrapper
    ret = f(repo, *args, **kwargs)
  File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/repo/push.py", line 35, in push
    return len(used_run_cache) + self.cloud.push(used, jobs, remote=remote)
  File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/data_cloud.py", line 65, in push
    return self.repo.cache.local.push(
  File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/remote/base.py", line 15, in wrapper
    return f(obj, named_cache, remote, *args, **kwargs)
  File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/cache/local.py", line 417, in push
    return self._process(
  File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/cache/local.py", line 386, in _process
    raise UploadError(fails)
dvc.exceptions.UploadError: 3 files failed to upload
------------------------------------------------------------

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2020-09-04 21:40:28,573 DEBUG: Analytics is enabled.
2020-09-04 21:40:28,708 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/rp/40tr0pm57qlggq2jfnwz81940000gn/T/tmpch5v7r19']'
2020-09-04 21:40:28,709 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/rp/40tr0pm57qlggq2jfnwz81940000gn/T/tmpch5v7r19']'
2 Likes

@paka Hi Krystian!

Could you please create an issue on https://github.com/iterative/dvc/issues ?

Thanks Ruslan; posted: https://github.com/iterative/dvc/issues/4534.