Dvc push to public S3 bucket

Questions
1

Hello there. I’m excited to try DVC’s features but I’m stuck with a basic problem in S3. Ideally, I’d like to push to a private S3 bucket, which I could do with aws-vault. Before doing that, I wanted to check how that would work in a public bucket. But when I try to push to a public S3, I get:

$ dvc remote list
storage	s3://ml-ci
https3	https://ml-ci.s3.amazonaws.com/

$ dvc push -r storage
ERROR: unexpected error - An error occurred (403) when calling the HeadObject operation: Forbidden

So I thought I’d try a suggestion from another thread on this forum and add a HTTPS remote endpoint instead (though I know it was suggested as a read-only solution):

$ dvc push -r https3
ERROR: failed to upload '.dvc/cache/a3/04afb96060aad9017xxxx' to 'https://ml-ci.s3.amazonaws.com/a3/04afb96060aad9017xxxx' - could not perform a POST request
ERROR: failed to push data to the cloud - 1 files failed to upload

What am I doing incorrectly here with remote setup? Is private + aws-vault the only option? I’d really appreciate some help, thank you!

1 Like
2

I’m having a similar issue with pushing to a private bucket. It might be related to IAM roles.

In my case I also get a 404 when I do dvc push. I get the same error if I just run aws s3 ls s3://bucket-name.

However, if I add my work profile name it works: aws s3 ls s3://bucket-name --profile work-profile

I’ve tried setting the AWS_PROFILE env variable to work-profile, but nothing changes. Any idea how I force it to see the right profile?

3

@sumita Are you able to use aws CLI utility with it?

@braaannigan How do you set AWS_PROFILE env var? If awscli with --profile works, then dvc also should. We have a profile config option https://dvc.org/doc/command-reference/remote/modify , e.g.:

dvc remote modify myremote profile work-profile
4

Hi @kupruser

I’ve found the answer on an old thread (with a slight modification of the credential path):
dvc remote modify myremote credentialpath ~/.aws/credentials
dvc remote modify myremote profile profile-name

Thanks

1 Like
5

I eventually solved this with aws-vault, but I missed that there’s documentation about how to use dvc remote to add credentials here: https://dvc.org/doc/command-reference/remote/modify#available-parameters-per-storage-type.

Thank you!

1 Like
6

Hi!

I’ve added this comment about improving the S3 remote error:

Thanks

7

p.s. @sumita you might want to take a look at https://github.com/iterative/dvc.org/issues/1699 and https://github.com/iterative/dvc.org/issues/1700.

Docs contributors are welcome! :slightly_smiling_face:

8

Thanks so much @jorgeorpinel! I looked at both the issues and the comments you left on the post (which I’ll respond to there). The issues make sense and I look forward to those enhancements - thanks again.

1 Like