Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden

naarkhoo · July 24, 2021, 10:48pm

this might sounds similar to what others have asked by I couldn’t really find a solution that fix my problem.

my ~/.aws/credentials looks like

[default]
aws_access_key_id = XYZ
aws_secret_access_key = ABC

[testing]
source_profile = default
role_arn = arn:aws:iam::54:role/ad

I add my remote like

dvc remote add --local -v myremote s3://bib-ds-models-testing/data/dvc-test
and use

I have made my .dvc/config.local to look like

[‘remote “myremote”’]
url = s3://bib-ds-models-testing/data/dvc-test
access_key_id = XYZ
secret_access_key = ABC/h2hOsRcCIFqwYWV7eZaUq3gNmS
profile=‘testing’
credentialpath = /Users/nyt21/.aws/credentials

but still after running dvc push -r myremote I get

ERROR: unexpected error - Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden

kupruser · July 25, 2021, 6:11pm

For the record: replied in dvc - Forbidden: An error occurred (403) when calling the HeadObject operation: - Stack Overflow

clementwalter · April 5, 2022, 11:01am

I’m having a similar issue though I’m perfectly available to use the cli:

dvc push consistently fails while aws s3 cp or aws s3 sync do work.

I don’t know what to try yet

See:

2022-04-05 13:02:29,278 DEBUG: Preparing to transfer data from '/Users/clementwalter/Documents/perso/co-bots-monorepo/packages/co-bots-image-processing/.dvc/cache' to 'co-bots/.dvc'
2022-04-05 13:02:29,278 DEBUG: Preparing to collect status from 'co-bots/.dvc'
2022-04-05 13:02:29,279 DEBUG: Collecting status from 'co-bots/.dvc'
2022-04-05 13:02:29,279 DEBUG: Querying 2 hashes via object_exists
2022-04-05 13:02:30,608 ERROR: unexpected error - Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden
------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/s3fs/core.py", line 261, in _call_s3
    out = await method(**additional_kwargs)
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/aiobotocore/client.py", line 187, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/dvc/main.py", line 55, in main
    ret = cmd.do_run()
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/dvc/command/base.py", line 45, in do_run
    return self.run()
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/dvc/command/data_sync.py", line 57, in run
    processed_files_count = self.repo.push(
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/dvc/repo/__init__.py", line 49, in wrapper
    return f(repo, *args, **kwargs)
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/dvc/repo/push.py", line 56, in push
    pushed += self.cloud.push(
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/dvc/data_cloud.py", line 85, in push
    return transfer(
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/dvc/objects/transfer.py", line 153, in transfer
    status = compare_status(src, dest, obj_ids, check_deleted=False, **kwargs)
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/dvc/objects/status.py", line 158, in compare_status
    dest_exists, dest_missing = status(
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/dvc/objects/status.py", line 122, in status
    exists = hashes.intersection(
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/dvc/objects/status.py", line 48, in _indexed_dir_hashes
    dir_exists.update(odb.list_hashes_exists(dir_hashes - dir_exists))
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/dvc/objects/db/base.py", line 421, in list_hashes_exists
    ret = list(itertools.compress(hashes, in_remote))
  File "/Users/clementwalter/.pyenv/versions/3.8.11/lib/python3.8/concurrent/futures/_base.py", line 619, in result_iterator
    yield fs.pop().result()
  File "/Users/clementwalter/.pyenv/versions/3.8.11/lib/python3.8/concurrent/futures/_base.py", line 444, in result
    return self.__get_result()
  File "/Users/clementwalter/.pyenv/versions/3.8.11/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/Users/clementwalter/.pyenv/versions/3.8.11/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/dvc/objects/db/base.py", line 412, in exists_with_progress
    ret = self.fs.exists(fs_path)
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/dvc/fs/fsspec_wrapper.py", line 91, in exists
    return self.fs.exists(path)
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/fsspec/asyn.py", line 91, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/fsspec/asyn.py", line 71, in sync
    raise return_result
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/fsspec/asyn.py", line 25, in _runner
    result[0] = await coro
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/s3fs/core.py", line 835, in _exists
    await self._info(path, bucket, key, version_id=version_id)
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/s3fs/core.py", line 1029, in _info
    out = await self._call_s3(
  File "/Users/clementwalter/.pyenv/versions/3.8.11/envs/co-bots/lib/python3.8/site-packages/s3fs/core.py", line 281, in _call_s3
    raise err
PermissionError: Forbidden
------------------------------------------------------------
2022-04-05 13:02:30,671 DEBUG: Adding '/Users/clementwalter/Documents/perso/co-bots-monorepo/packages/co-bots-image-processing/.dvc/config.local' to gitignore file.
2022-04-05 13:02:30,674 DEBUG: Adding '/Users/clementwalter/Documents/perso/co-bots-monorepo/packages/co-bots-image-processing/.dvc/tmp' to gitignore file.
2022-04-05 13:02:30,674 DEBUG: Adding '/Users/clementwalter/Documents/perso/co-bots-monorepo/packages/co-bots-image-processing/.dvc/cache' to gitignore file.
2022-04-05 13:02:30,676 DEBUG: Removing '/Users/clementwalter/Documents/perso/co-bots-monorepo/packages/.ALtVEwomf9fsFX2UmKbut9.tmp'
2022-04-05 13:02:30,677 DEBUG: Removing '/Users/clementwalter/Documents/perso/co-bots-monorepo/packages/.ALtVEwomf9fsFX2UmKbut9.tmp'
2022-04-05 13:02:30,678 DEBUG: Removing '/Users/clementwalter/Documents/perso/co-bots-monorepo/packages/.ALtVEwomf9fsFX2UmKbut9.tmp'
2022-04-05 13:02:30,678 DEBUG: Removing '/Users/clementwalter/Documents/perso/co-bots-monorepo/packages/co-bots-image-processing/.dvc/cache/.TZa9m7D429SR85Soc3euje.tmp'
2022-04-05 13:02:30,681 DEBUG: Version info for developers:
DVC version: 2.9.3 (pip)
---------------------------------
Platform: Python 3.8.11 on macOS-11.6.2-x86_64-i386-64bit
Supports:
	webhdfs (fsspec = 2022.1.0),
	http (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	https (aiohttp = 3.8.1, aiohttp-retry = 2.4.6),
	s3 (s3fs = 2022.1.0, boto3 = 1.20.24)
Cache types: reflink, hardlink, symlink
Cache directory: apfs on /dev/disk1s1s1
Caches: local
Remotes: s3
Workspace directory: apfs on /dev/disk1s1s1
Repo: dvc (subdir), git

Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!

and

 (co-bots) packages/co-bots-image-processing   main ±  aws s3 sync .dvc/cache s3://co-bots
upload: .dvc/cache/00/f6b1d1f06b8754d64c452f4ef58f16 to s3://co-bots/00/f6b1d1f06b8754d64c452f4ef58f16
upload: .dvc/cache/00/a7bfb27455742c9479d25779e4214a to s3://co-bots/00/a7bfb27455742c9479d25779e4214a
upload: .dvc/cache/00/377bfc7e3cf1a189380a3e9f52d7c6 to s3://co-bots/00/377bfc7e3cf1a189380a3e9f52d7c6
upload: .dvc/cache/04/e800606cdafbea256f9ff537bc289a to s3://co-bots/04/e800606cdafbea256f9ff537bc289a
upload: .dvc/cache/04/84a80227ef30c3ce9f506d8cce8150 to s3://co-bots/04/84a80227ef30c3ce9f506d8cce8150
upload: .dvc/cache/02/22acd5f40b2cc2073a3e2e9a42828a to s3://co-bots/02/22acd5f40b2cc2073a3e2e9a42828a
upload: .dvc/cache/00/2bbbc2d424b03e445ad9188d5f771c to s3://co-bots/00/2bbbc2d424b03e445ad9188d5f771c
upload: .dvc/cache/01/112074beaadff8b4a6ff1fd27a5dd9 to s3://co-bots/01/112074beaadff8b4a6ff1fd27a5dd9
upload: .dvc/cache/04/2ad2b365565fb1c70773cd51a7f931.dir to s3://co-bots/04/2ad2b365565fb1c70773cd51a7f931.dir
upload: .dvc/cache/06/bdb87ab7ecadfdae2e9d52fa7f8a28 to s3://co-bots/06/bdb87ab7ecadfdae2e9d52fa7f8a28
upload: .dvc/cache/06/5350251c0f1545eb0b48ca698e89f2 to s3://co-bots/06/5350251c0f1545eb0b48ca698e89f2
upload: .dvc/cache/06/f402f0fe899292715a8dc7d393914a to s3://co-bots/06/f402f0fe899292715a8dc7d393914a
...

My dvc config is as follows

[core]
    remote = s3
['remote "s3"']
    url = s3://co-bots/

daavoo · April 8, 2022, 10:41am

Hi @clementwalter , could you fill a bug report in DVC repo (Sign in to GitHub · GitHub)

clementwalter · April 8, 2022, 12:23pm

Hey, sorry I solved myself already, I had to specify the bucket region (which is not the usual default one, I am using eu-west-3 in my case) in the dvc config file. It seams like there is a different behavior when it is not provided between dvc and the aws cli.

cfoo · April 10, 2023, 6:48pm

For me, the issue was version drift. Reinstall the correct version of boto3 for dvc via pip: python3 -m pip install 'dvc[s3]'.

Topic		Replies	Views
Why DVC not working with AWS role on AWS Batch? Questions	1	387	March 11, 2022
Dvc push error, remotedir with minio Questions	0	367	February 21, 2023
Dvc push to public S3 bucket Questions	7	4300	August 17, 2020
AllAccessDisabled Error for mino set as remote Questions	2	2633	April 23, 2021
Dvc import from minIO Questions	0	563	April 17, 2023

Forbidden: An error occurred (403) when calling the HeadObject operation: Forbidden

Related topics