Specify AWS profile when adding external data from S3

Questions
1

Hi all,

how can I make DVC use a specific AWS-profile when adding external data?

If my credentials are stored in the default-profile I can add external data as follows (just following the S3-Example for Managing External Data)

$ git init
$ dvc init
$ dvc remote add s3cache s3:my-bucket/cache
$ dvc config cache.s3 s3cache
$ dvc add --external s3://my-bucket/remote-data.txt

But I want to use another profile (MyProfile) that I can add to s3cache:

$ dvc remote modify s3cache profile MyProfile

If the default profile is removed or changed from the .aws/credentials the following gives a Bad Request:

$ dvc add --external s3://my-bucket/remote-data.txt
# ERROR: unexpected error - [Errno 22] Bad Request: An error occurred (400) when calling the HeadObject operation: Bad Request

How can I provide the profile information to the add?

2

Hi, I think you’re looking for this:

3

Hi, thanks for your reply. I do have a named profile already (MyProfile). And I know how to add it to a remote

$ dvc remote modify my-remote profile MyProfile

but I don’t know how to specify it for the command:

$ dvc add --external s3://my-bucket/remote-data.txt
4

Have you tried the AWS_PROFILE environment variable?

5

That does work, although I would rather have the profile known to dvc locally instead of changing the default AWS-profile this way.

6

Alright, I found it.

It seems that you can specify a configured remote to the add command via remote://your-remote/your-file (instead of s3://your-bucket/your-file)

So in my case I configured a second remote s3remote for referencing files there:

$ dvc remote add s3remote s3://my-bucket
$ dvc remote modify s3remote profile MyProfile
$ dvc add --external remote://s3remote/remote-data.txt
1 Like