# Configure: DVC + CircleCI + GDRIVE

**URL:** <https://discuss.dvc.org/t/configure-dvc-circleci-gdrive/1011>\
**Category:** Questions\
**Created:** [December 19, 2021, 4:26pm UTC](https://discuss.dvc.org/t/configure-dvc-circleci-gdrive/1011 "2021-12-19T16:26:29Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![eliangius](https://avatars.discourse-cdn.com/v4/letter/e/a6a055/32.png) [@eliangius](https://discuss.dvc.org/u/eliangius)\
**Post date:** [December 19, 2021, 4:26pm UTC](https://discuss.dvc.org/t/configure-dvc-circleci-gdrive/1011/1 "2021-12-19T16:26:29Z")

</div>

Trying to configure DVC to pull gdrive stored artifacts from a CircleCI build? I followed instructions from [docs](https://dvc.org/doc/user-guide/setup-google-drive-remote), hints in this [discussion-question](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795) & this [git-hub issue](https://github.com/iterative/dvc/issues/6230) but there seems to be something missing.

Here is the excerpt of the circle-ci configuration being used:

```auto
- run:
     name: Data checkout
     command: |
        dvc remote modify storage --local gdrive_use_service_account true;
        dvc remote modify storage --local gdrive_service_account_json_file_path /dev/null;
        dvc remote modify storage --local gdrive_service_account_user_email <my@service.iam.gserviceaccount.com>;
        dvc pull --recursive -r storage --verbose;
     environment:
        GDRIVE_CREDENTIALS_DATA: $GDRIVE_CREDENTIALS_DATA

```

The `$GDRIVE_CREDENTIALS_DATA` environment variable was configured in Circle-CI server to be that of my Google’s Service Account JSON to access my google drive. But the CircleCI runner reports this error as if the environment variable did not get read.

```auto
0% 0/1 [00:00<?, ?file/s{'info': ''}]
Go to the following link in your browser:

    https://accounts.google.com/o/oauth2/auth?client_id=710796635688-iivsgbgsb6uv1fap6635dhvuei09o66c.apps.googleusercontent.com&redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fdrive.appdata&access_type=offline&response_type=code&approval_prompt=force

Everything is up to date.              
ERROR: failed to pull data from the cloud - GDrive remote auth failed with credentials in 'GDRIVE_CREDENTIALS_DATA'.
Backup first, remove or fix them, and run DVC again.
It should do auth again and refresh the credentials.

```

Things I am unsure about.

1. Is the `GDRIVE_CREDENTIALS_DATA` environment variable the right name DVC commands looks for?
2. Which JSON content should this environment variable have?
  - `.dvc/tmp/gdrive-user-credentials.json` ?
  - Google Service account JSON looking like this?

```auto
{
  "type": "service_account",
  "project_id": "<my_project>",
  "private_key_id": "<key_id>",
  "private_key": "<key_hash>",
  "client_email": "<my@service.iam.gserviceaccount.com>",
  "client_id": "<client_id>",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
}

```

- Or Google client secret JSON looking like this?

```auto
{
  "installed": {
    "client_id": "<google_client_id>",
    "project_id": "<google_project_id>",
    "auth_uri": "https://accounts.google.com/o/oauth2/auth",
    "token_uri": "https://oauth2.googleapis.com/token",
    "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
    "client_secret": "<client_secret>",
    "redirect_uris": [
      "urn:ietf:wg:oauth:2.0:oob",
      "http://localhost"
    ]
  }
}

```

1. Is the 1st time google browser authentication step still required for CI machines?

Many thanks

---

<div class="post-metadata">

**Author:** ![YanxiangGao](https://avatars.discourse-cdn.com/v4/letter/y/f04885/32.png) [@YanxiangGao](https://discuss.dvc.org/u/YanxiangGao)\
**Post date:** [December 20, 2021, 4:28am UTC](https://discuss.dvc.org/t/configure-dvc-circleci-gdrive/1011/2 "2021-12-20T04:28:09Z")

</div>

I guess you had met a bug , you can try to set any value for `gdrive_service_account_json_file_path` and try again.

> <https://github.com/iterative/dvc/issues/6230>
>
> \## Description
> 
> The \`GDRIVE\_CREDENTIALS\_DATA\` environment variable won't be ho…nored unless the \`gdrive\_service\_account\_json\_file\_path\` configuration option is set beforehand. In most of the habitual use cases, this requirement doesn't make too much sense from the user standpoint.
> 
> Is this the recommended way of passing Google Drive (Service Account) credentials through an environment variable?
> 
> Additionally, after reading the pertaining \[documentation\](https://dvc.org/doc/user-guide/setup-google-drive-remote), I don't understand why I can't use the \`GDRIVE\_CREDENTIALS\_DATA\` environment variable to pass the DVC-specific \`.dvc/tmp/gdrive-user-credentials.json\` file contents, instead of having to pass the raw service account file contents.
> 
> \### How to reproduce
> 
> The following GitHub Actions workflow provides a self-contained example to reproduce this issue. Uncommenting the only line prefixed with a hash \`#\` will make it work as expected.
> 
> \`\`\`yaml
> on: push
> jobs:
> run:
> runs-on: ubuntu-latest
> steps:
> - uses: iterative/setup-dvc@v1
> - run: |
> git init
> dvc init
>           
> dvc remote add origin gdrive://root
> dvc remote modify origin --local gdrive\_use\_service\_account true
> # dvc remote modify origin --local gdrive\_service\_account\_json\_file\_path /dev/null
> 
> date \> file
> dvc add file
> 
> dvc push --verbose --remote origin
> env:
> GDRIVE\_CREDENTIALS\_DATA: ${{ secrets.ORIGINAL\_SERVICE\_ACCOUNT\_JSON }}
> \`\`\`
> 
> \_Note: the \`${{ secrets.ORIGINAL\_SERVICE\_ACCOUNT\_JSON }}\` variable represents a GCP Service Account file in the standard JSON format provided by Google, not a DVC credentials JSON file.\_
> 
> \### Actual output
> 
> \`\`\`console
> $ dvc push --verbose --remote origin
> 2021-06-25 22:04:09,030 ERROR: failed to push data to the cloud - To use service account, set \`gdrive\_service\_account\_json\_file\_path\`, and optionally\`gdrive\_service\_account\_user\_email\` in DVC config
> Learn more at \<https://man.dvc.org/remote/modify\>
> \------------------------------------------------------------
> Traceback (most recent call last):
> File "dvc/command/data\_sync.py", line 57, in run
> File "dvc/repo/\_\_init\_\_.py", line 51, in wrapper
> File "dvc/repo/push.py", line 44, in push
> File "dvc/data\_cloud.py", line 77, in push
> File "dvc/data\_cloud.py", line 38, in get\_remote
> File "dvc/data\_cloud.py", line 58, in \_init\_remote
> File "dvc/remote/\_\_init\_\_.py", line 8, in get\_remote
> File "dvc/fs/gdrive.py", line 128, in \_\_init\_\_
> File "dvc/fs/gdrive.py", line 155, in \_validate\_config
> dvc.exceptions.DvcException: To use service account, set \`gdrive\_service\_account\_json\_file\_path\`, and optionally\`gdrive\_service\_account\_user\_email\` in DVC config
> Learn more at \<https://man.dvc.org/remote/modify\>
> 2021-06-25 22:04:09,033 DEBUG: Analytics is enabled.
> \------------------------------------------------------------
> 2021-06-25 22:04:09,076 DEBUG: Trying to spawn '\['daemon', '-q', 'analytics', '/tmp/tmp8tbexj0k'\]'
> 2021-06-25 22:04:09,077 DEBUG: Spawned '\['daemon', '-q', 'analytics', '/tmp/tmp8tbexj0k'\]'
> \`\`\`
> 
> \### Expected output
> 
> \`\`\`console
> $ dvc push --verbose --remote origin
> 2021-06-25 22:17:58,441 DEBUG: Preparing to upload data to 'gdrive://root'
> 2021-06-25 22:17:58,441 DEBUG: Preparing to collect status from gdrive://root
> 2021-06-25 22:17:58,441 DEBUG: Collecting information from local cache...
> 2021-06-25 22:17:58,441 DEBUG: Collecting information from remote cache...
> 2021-06-25 22:17:58,441 DEBUG: Matched '0' indexed hashes
> 2021-06-25 22:17:58,442 DEBUG: Querying 1 hashes via object\_exists
> 2021-06-25 22:17:58,672 DEBUG: GDrive remote auth with config '\*\*\*'client\_config\_backend': 'settings', 'client\_config\_file': 'client\_secrets.json', 'save\_credentials': True, 'oauth\_scope': \['https://www.googleapis.com/auth/drive', 'https://www.googleapis.com/auth/drive.appdata'\], 'save\_credentials\_backend': 'file', 'save\_credentials\_file': '/home/runner/work/test/test/.dvc/tmp/.5VsCA2nJs9ZQgeBZ8pWNAk.tmp', 'get\_refresh\_token': True, 'service\_config': \*\*\*'client\_user\_email': None, 'client\_json\_file\_path': '/home/runner/work/test/test/.dvc/tmp/.mpgSFHPn7NBgMe35nngenb.tmp'\*\*\*'.
> 2021-06-25 22:17:59,308 DEBUG: Uploading '.dvc/cache/5d/afc8549edbdc83c20f11cbfde93cc4' to 'gdrive://root/5d/afc8549edbdc83c20f11cbfde93cc4'
> 1 file pushed
> 2021-06-25 22:18:01,059 DEBUG: Analytics is enabled.
> 2021-06-25 22:18:01,113 DEBUG: Trying to spawn '\['daemon', '-q', 'analytics', '/tmp/tmp0ocbbm6a'\]'
> 2021-06-25 22:18:01,116 DEBUG: Spawned '\['daemon', '-q', 'analytics', '/tmp/tmp0ocbbm6a'\]'
> \`\`\`
> 
> \### Environment information
> 
> \`\`\`console
> $ dvc doctor
> DVC version: 2.4.1 (deb)
> \---------------------------------
> Platform: Python 3.8.10 on Linux-5.8.0-1033-azure-x86\_64-with-glibc2.7
> Supports: All remotes
> Cache types: \<https://error.dvc.org/no-dvc-cache\>
> Caches: local
> Remotes: gdrive
> Workspace directory: ext4 on /dev/root
> Repo: dvc, git
> \`\`\`
> 
> \### Additional information
> 
> https://github.com/iterative/dvc/blob/4e792ae61c5927ab2e5f6a6914d985d43aa705b4/dvc/fs/gdrive.py#L128
> 
> https://github.com/iterative/dvc/blob/4e792ae61c5927ab2e5f6a6914d985d43aa705b4/dvc/fs/gdrive.py#L149-L162
> 
> \### See also
> 
> \- https://dvc.org/doc/user-guide/setup-google-drive-remote
> \- https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/9
> \- https://iterativeai.slack.com/archives/CB41NAL8H/p1624015633145600
> \- https://iterativeai.slack.com/archives/CNQ95CG1K/p1624187829201500

---

<div class="post-metadata">

**Author:** ![eliangius](https://avatars.discourse-cdn.com/v4/letter/e/a6a055/32.png) [@eliangius](https://discuss.dvc.org/u/eliangius)\
**Post date:** [December 20, 2021, 2:10pm UTC](https://discuss.dvc.org/t/configure-dvc-circleci-gdrive/1011/3 "2021-12-20T14:10:03Z")

</div>

Yea, that git-issue is one of the links I was looking at because it reports a working alternative to the bug. You’ll note I followed the same DVC setup but I am still not able to get DVC pulling data, so I must be missing something…

The problem is that my CI machine still goes through GDrive’s Oauth interactive mechanism to authenticate via the browser, but the GDrive service credentials passed through the environment variable seem are either not read, or are not set properly by me.

Answers to any of my 3 questions would help narrow down things.