DVC supports AWS S3, Google Cloud Storage, Microsoft Azure Blob Storage. However, the instruction is only for AWS S3. How to use DVC with Microsoft Azure Blob Storage?
Hi @felkost !
I’ve seen you asked it on discord too 
, but just for the record: Azure example is available here https://dvc.org/doc/commands-reference/remote-add .
Thanks,
Ruslan
The above link doesn’t work.
In particular, I am interested in using dvc remote with a SAS token. When I add the azureblob remote via https:// link to DVC and initiate dvc push, it returns '405 The resource doesn't support specified Http Verb.' error.
To sum app, providing remote as:
https://wasburl/container/?sig=<sastoken>
results in
dvc.exceptions.HTTPError: '405 The resource doesn't support specified Http Verb.'
What I read in different forums, it appears Azure requires Cross-Origin Resource Sharing (CORS) headers and DVC doesn’t support it. I guess this post could be a feature request to DVC to include CORS headers.
Any ideas?
===========================
For the record the output log:
SHORT/REGULAR OUTPUT:
(base) ➜ DVC git:(master) ✗ dvc push [9:36:28]
ERROR: failed to upload '.dvc/cache/38/6f68f067e4c588bf4dcc96afdbece7' to 'https://<wasb>/<folder>/38/6f68f067e4c588bf4dcc96afdbece7' - '405 The resource doesn't support specified Http Verb.'
ERROR: failed to upload '.dvc/cache/d4/1d8cd98f00b204e9800998ecf8427e' to 'https://<wasb>/<folder>/d4/1d8cd98f00b204e9800998ecf8427e' - '405 The resource doesn't support specified Http Verb.'
ERROR: failed to upload '.dvc/cache/66/52640f2cde8646134c9b5cdfb4583f.dir' to 'https://<wasb>/<folder>/66/52640f2cde8646134c9b5cdfb4583f.dir'
ERROR: failed to push data to the cloud - 3 files failed to upload
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
VERBOSE OUTPUT:
(base) ➜ DVC git:(master) ✗ dvc push --verbose [9:36:40]
2020-09-04 21:40:27,553 DEBUG: Check for update is enabled.
2020-09-04 21:40:27,559 DEBUG: fetched: [(3,)]
2020-09-04 21:40:27,576 DEBUG: Assuming '/Users/<user>/GitHubs/DVC/.dvc/cache/66/52640f2cde8646134c9b5cdfb4583f.dir' is unchanged since it is read-only
2020-09-04 21:40:27,578 DEBUG: Assuming '/Users/<user>/GitHubs/DVC/.dvc/cache/66/52640f2cde8646134c9b5cdfb4583f.dir' is unchanged since it is read-only
2020-09-04 21:40:27,581 DEBUG: Preparing to upload data to 'https://<wasb>/<folder>/?sig=<SAStoken>
2020-09-04 21:40:27,581 DEBUG: Preparing to collect status from https://<wasb>/<folder>/?sig=<SAStoken>
2020-09-04 21:40:27,581 DEBUG: Collecting information from local cache...
2020-09-04 21:40:27,582 DEBUG: Assuming '/Users/<user>/GitHubs/DVC/.dvc/cache/38/6f68f067e4c588bf4dcc96afdbece7' is unchanged since it is read-only
2020-09-04 21:40:27,601 DEBUG: Assuming '/Users/<user>/GitHubs/DVC/.dvc/cache/66/52640f2cde8646134c9b5cdfb4583f.dir' is unchanged since it is read-only
2020-09-04 21:40:27,602 DEBUG: Assuming '/Users/<user>/GitHubs/DVC/.dvc/cache/d4/1d8cd98f00b204e9800998ecf8427e' is unchanged since it is read-only
2020-09-04 21:40:27,602 DEBUG: Collecting information from remote cache...
2020-09-04 21:40:27,603 DEBUG: Querying 1 hashes via object_exists
2020-09-04 21:40:28,030 DEBUG: Matched '0' indexed hashes
2020-09-04 21:40:28,030 DEBUG: Querying 3 hashes via object_exists
2020-09-04 21:40:28,328 DEBUG: Uploading '.dvc/cache/d4/1d8cd98f00b204e9800998ecf8427e' to 'https://<wasb>/<folder>/d4/1d8cd98f00b204e9800998ecf8427e'
2020-09-04 21:40:28,328 DEBUG: Uploading '.dvc/cache/38/6f68f067e4c588bf4dcc96afdbece7' to https://'<wasb>/<folder>/38/6f68f067e4c588bf4dcc96afdbece7'
2020-09-04 21:40:28,556 ERROR: failed to upload '.dvc/cache/38/6f68f067e4c588bf4dcc96afdbece7' to 'https://<wasb>/<folder>/38/6f68f067e4c588bf4dcc96afdbece7' - '405 The resource doesn't support specified Http Verb.'
------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/cache/local.py", line 30, in wrapper
func(from_info, to_info, *args, **kwargs)
File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/tree/base.py", line 391, in upload
self._upload( # noqa, pylint: disable=no-member
File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/tree/http.py", line 195, in _upload
raise HTTPError(response.status_code, response.reason)
dvc.exceptions.HTTPError: '405 The resource doesn't support specified Http Verb.'
------------------------------------------------------------
2020-09-04 21:40:28,558 ERROR: failed to upload '.dvc/cache/d4/1d8cd98f00b204e9800998ecf8427e' to 'https://<wasb>/<folder>/d4/1d8cd98f00b204e9800998ecf8427e' - '405 The resource doesn't support specified Http Verb.'
------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/cache/local.py", line 30, in wrapper
func(from_info, to_info, *args, **kwargs)
File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/tree/base.py", line 391, in upload
self._upload( # noqa, pylint: disable=no-member
File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/tree/http.py", line 195, in _upload
raise HTTPError(response.status_code, response.reason)
dvc.exceptions.HTTPError: '405 The resource doesn't support specified Http Verb.'
------------------------------------------------------------
2020-09-04 21:40:28,561 DEBUG: failed to upload full contents of 'data/prepared', aborting .dir file upload
2020-09-04 21:40:28,562 ERROR: failed to upload '.dvc/cache/66/52640f2cde8646134c9b5cdfb4583f.dir' to 'https://<wasb>/<folder>/66/52640f2cde8646134c9b5cdfb4583f.dir'
2020-09-04 21:40:28,563 DEBUG: fetched: [(31,)]
2020-09-04 21:40:28,564 ERROR: failed to push data to the cloud - 3 files failed to upload
------------------------------------------------------------
Traceback (most recent call last):
File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/command/data_sync.py", line 50, in run
processed_files_count = self.repo.push(
File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/repo/__init__.py", line 34, in wrapper
ret = f(repo, *args, **kwargs)
File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/repo/push.py", line 35, in push
return len(used_run_cache) + self.cloud.push(used, jobs, remote=remote)
File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/data_cloud.py", line 65, in push
return self.repo.cache.local.push(
File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/remote/base.py", line 15, in wrapper
return f(obj, named_cache, remote, *args, **kwargs)
File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/cache/local.py", line 417, in push
return self._process(
File "/usr/local/Cellar/dvc/1.6.1/libexec/lib/python3.8/site-packages/dvc/cache/local.py", line 386, in _process
raise UploadError(fails)
dvc.exceptions.UploadError: 3 files failed to upload
------------------------------------------------------------
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2020-09-04 21:40:28,573 DEBUG: Analytics is enabled.
2020-09-04 21:40:28,708 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/var/folders/rp/40tr0pm57qlggq2jfnwz81940000gn/T/tmpch5v7r19']'
2020-09-04 21:40:28,709 DEBUG: Spawned '['daemon', '-q', 'analytics', '/var/folders/rp/40tr0pm57qlggq2jfnwz81940000gn/T/tmpch5v7r19']'
2 Likes
@paka Hi Krystian!
Could you please create an issue on https://github.com/iterative/dvc/issues ?
Thanks Ruslan; posted: https://github.com/iterative/dvc/issues/4534.