# CML + Github actions + Google Drive / Service Account **URL:** \ **Category:** Questions\ **Created:** [June 17, 2021, 11:54am UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795 "2021-06-17T11:54:49Z")\ **Posts on this page:** 17\ **Page:** 1
**Author:** ![mcosta](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mcosta](https://discuss.dvc.org/u/mcosta)\ **Post date:** [June 17, 2021, 11:54am UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/1 "2021-06-17T11:54:49Z")
Hi, What is the correct way to use Github secrets with Google Drive key (json file)? We are getting the following error: ERROR: failed to pull data from the cloud - To use service account, set `gdrive_service_account_json_file_path`, and optionally`gdrive_service_account_user_email` in DVC config Local execution is configured by ```auto dvc remote modify myremote gdrive_use_service_account true ``` and ```auto dvc remote modify myremote --local \ gdrive_service_account_json_file_path path/to/file.json ``` Cheers ---
**Author:** ![0x2b3bfa0](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/0x2b3bfa0/32/119_2.png) [@0x2b3bfa0](https://discuss.dvc.org/u/0x2b3bfa0)\ **Post date:** [June 17, 2021, 12:32pm UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/2 "2021-06-17T12:32:01Z")
👋 Hello, @mcosta! You can [create a repository secret](https://docs.github.com/en/actions/reference/encrypted-secrets#creating-encrypted-secrets-for-a-repository) with the **contents** of the `.dvc/tmp/gdrive-user-credentials.json` file that DVC is going to generate locally after the first successful authentication. Then, you can expose it on your workflow by using the following environment variable, as suggested in [the documentation](https://dvc.org/doc/user-guide/setup-google-drive-remote#authorization): ```auto steps: - run: dvc pull env: GDRIVE_CREDENTIALS_DATA: ${{ secrets.GDRIVE_CREDENTIALS_DATA }} ``` ---
**Author:** ![mcosta](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mcosta](https://discuss.dvc.org/u/mcosta)\ **Post date:** [June 17, 2021, 12:52pm UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/3 "2021-06-17T12:52:25Z")
Hi @0x2b3bfa0 That was our first attempt. It did not work, though. Note that we have our config with gdrive\_use\_service\_account = true ---
**Author:** ![0x2b3bfa0](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/0x2b3bfa0/32/119_2.png) [@0x2b3bfa0](https://discuss.dvc.org/u/0x2b3bfa0)\ **Post date:** [June 17, 2021, 2:44pm UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/4 "2021-06-17T14:44:25Z")
> [@mcosta](#): > > Note that we have our config with gdrive\_use\_service\_account = true Can you please confirm that the `GDRIVE_CREDENTIALS_DATA` variable has been populated with the contents of the `.dvc/tmp/gdrive-user-credentials.json` file and not with the contents of the service account JSON file issued by Google Cloud Platform? If this doesn’t solve your issue, can you please post the output of `dvc doctor` and `dvc pull --verbose --remote myremote`? ---
**Author:** ![mcosta](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mcosta](https://discuss.dvc.org/u/mcosta)\ **Post date:** [June 17, 2021, 2:57pm UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/5 "2021-06-17T14:57:34Z")
@0x2b3bfa0 On our first attempt we added the content from the json directly extracted from Google Coud Platform. Then we did exactly what was mentioned in documentation and updated the secret with the content of the local file (.dvc/tmp/gdrive-user-credentials.json). The error it gave was always the same. The yaml file looks like this name: model-training on: [push] jobs: run: runs-on: [ubuntu-latest] container: docker://dvcorg/cml:0-dvc2-base1 steps: - uses: actions/checkout@v2 - name: Test env: REPO\_TOKEN: ${{ secrets.GITHUB\_TOKEN }} GDRIVE\_CREDENTIALS\_DATA: ${{ secrets.GDRIVE\_CREDENTIALS\_DATA }} ``` run: # Install requirements pip install -r Requirements.txt dvc remote modify filterout --local gdrive_user_credentials_file .dvc/tmp/gdrive-user-credentials.json ``` ---
**Author:** ![mcosta](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mcosta](https://discuss.dvc.org/u/mcosta)\ **Post date:** [June 17, 2021, 3:01pm UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/6 "2021-06-17T15:01:51Z")
Missing this part: dvc pull -r filterout --run-cache dvc repro ---
**Author:** ![0x2b3bfa0](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/0x2b3bfa0/32/119_2.png) [@0x2b3bfa0](https://discuss.dvc.org/u/0x2b3bfa0)\ **Post date:** [June 17, 2021, 3:04pm UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/7 "2021-06-17T15:04:28Z")
> [@mcosta](#): > > The error it gave was always the same. Can you please attach a copy of the error message, preferebly when running your `dvc` commands with the `--verbose` option? > [@mcosta](#): > > ```console > filterout --local gdrive_user_credentials_file .dvc/tmp/gdrive-user-credentials.json > > ``` What did you intend to achieve with this command? DVC should automatically pick the credential file contents from the `GDRIVE_CREDENTIALS_DATA` environment variable. Can you please try to run the workflow after commenting out this line? ---
**Author:** ![mcosta](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mcosta](https://discuss.dvc.org/u/mcosta)\ **Post date:** [June 17, 2021, 3:40pm UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/8 "2021-06-17T15:40:42Z")
@0x2b3bfa0 Sorry, that line was a consequence of a forgotten trial and error flow… After removing it, adding --verbose to dvc pull and forcing a new push, we have the following: > **[Log.rtf](https://drive.google.com/file/d/1aLEe3DQfUakEl_U2YU3qex7n_XcOcX48/view?usp=sharing)** > > Google Drive file. Note: was not able to copy / paste the logs. It did not allow me to do it. It says that new users are only allowed to add 2 links… ---
**Author:** ![0x2b3bfa0](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/0x2b3bfa0/32/119_2.png) [@0x2b3bfa0](https://discuss.dvc.org/u/0x2b3bfa0)\ **Post date:** [June 17, 2021, 4:37pm UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/9 "2021-06-17T16:37:49Z")
I’ve been able to reproduce the issue. 🙈 The following steps work as expected, but they diverge quite a bit from what I recall from previous tests: ```auto on: push jobs: run: runs-on: ubuntu-latest steps: - uses: iterative/setup-dvc@v1 - run: | git init dvc init dvc remote add origin gdrive://root dvc remote modify origin --local gdrive_use_service_account true dvc remote modify origin --local gdrive_service_account_json_file_path /dev/null date > file dvc add file dvc push --remote origin env: GDRIVE_CREDENTIALS_DATA: ${{ secrets.ORIGINAL_SERVICE_ACCOUNT_JSON }} ``` ---
**Author:** ![mcosta](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mcosta](https://discuss.dvc.org/u/mcosta)\ **Post date:** [June 17, 2021, 9:10pm UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/10 "2021-06-17T21:10:00Z")
@0x2b3bfa0 Tried to follow your script. Removed container: docker://dvcorg/cml:0-dvc2-base1 line and added - uses: iterative/setup-dvc@v1 With this change we are having an error upstream in pip install -r Requirements.txt ERROR: Could not open requirements file: [Errno 2] No such file or directory: ‘Requirements.txt’ ---
**Author:** ![mcosta](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mcosta](https://discuss.dvc.org/u/mcosta)\ **Post date:** [June 17, 2021, 10:10pm UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/11 "2021-06-17T22:10:56Z")
@0x2b3bfa0 After looking to [https://github.com/iterative/cml#using-cml-with-dvc](https://github.com/iterative/cml#using-cml-with-dvc) and to your demo code we managed to have it pulling data. We created the secret with Google Service Account key and used as you did. Although it is present in the requirements file, we had to pip install dvc[gdrive] after installing what is in the Requirements.txt. dvc repro fails ERROR: failed to reproduce ‘dvc.yaml’: [Errno 2] No such file or directory: PosixPathInfo: ‘data/Documents’ Theoretical this should exist. For example, in the logs you have things like: 2021-06-17 21:54:54,452 DEBUG: Created ‘copy’: .dvc/cache/0c/048c861ad12cd2bb19e031019c446f → data/documents/train/Document/01.jpg Script Used > name: model-training > on: [push] # Trigger - push event > jobs: > run: > runs-on: [ubuntu-latest] > #container: docker://dvcorg/cml:0-dvc2-base1 > steps: > - uses: actions/checkout@v2 > - uses: iterative/setup-cml@v1 > - name: Model Diff > env: > REPO\_TOKEN: ${{ secrets.GITHUB\_TOKEN }} > GDRIVE\_CREDENTIALS\_DATA: ${{ secrets.ORIGINAL\_SERVICE\_ACCOUNT\_JSON }} > run: | > pip install -r Requirements.txt > pip install dvc[gdrive] > dvc remote modify filterout --local gdrive\_service\_account\_json\_file\_path /dev/null > dvc pull -r filterout --verbose > dvc repro ---
**Author:** ![mcosta](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mcosta](https://discuss.dvc.org/u/mcosta)\ **Post date:** [June 17, 2021, 10:17pm UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/12 "2021-06-17T22:17:32Z")
Umm, it looks that something is case sensitive… data/documents vs data/Documents ---
**Author:** ![0x2b3bfa0](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/0x2b3bfa0/32/119_2.png) [@0x2b3bfa0](https://discuss.dvc.org/u/0x2b3bfa0)\ **Post date:** [June 17, 2021, 10:20pm UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/13 "2021-06-17T22:20:53Z")
> [@mcosta](#): > > Tried to follow your script. Removed > container: docker://dvcorg/cml:0-dvc2-base1 line > and added > > - uses: iterative/setup-dvc@v1 Awesome! You should be able to use GitHub Actions to replace all the useful packages shipped with the Docker images if you want to: ```yaml steps: - uses: actions/setup-python@v2 - uses: iterative/setup-dvc@v1 - uses: iterative/setup-cml@v1 ``` * * * > [@mcosta](#): > > With this change we are having an error upstream in pip install -r Requirements.txt > > ERROR: Could not open requirements file: [Errno 2] No such file or directory: ‘Requirements.txt’ Please keep in mind that the vast majority of Linux distributions rely on [case-sensitive](https://en.wikipedia.org/wiki/Case_sensitivity) filesystems: a file named `requirements.txt` can’t be accessed with any other case combination, like `Requirements.txt` as in your answer. Can you please check if this file begins with an uppercase letter as in your message? ---
**Author:** ![0x2b3bfa0](https://yyz1.discourse-cdn.com/flex035/user_avatar/discuss.dvc.org/0x2b3bfa0/32/119_2.png) [@0x2b3bfa0](https://discuss.dvc.org/u/0x2b3bfa0)\ **Post date:** [June 17, 2021, 10:22pm UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/14 "2021-06-17T22:22:48Z")
> [@mcosta](#): > > Although it is present in the requirements file, we had to pip install dvc[gdrive] after installing what is in the Requirements.txt. If you use our containers or the `iterative/setup-dvc@v1` action, you shouldn’t need/want to install DVC manually to your Python environment. We ship them with all the backends, including `[gdrive]` for convenience. > [@mcosta](#): > > ERROR: failed to reproduce ‘dvc.yaml’: [Errno 2] No such file or directory: PosixPathInfo: ‘data/Documents’ > > Theoretical this should exist. For example, in the logs you have things like: > > 2021-06-17 21:54:54,452 DEBUG: Created ‘copy’: .dvc/cache/0c/048c861ad12cd2bb19e031019c446f → data/documents/train/Document/01.jpg Again, it looks like `data/Documents` is not the same as `data/documents` due to case sensitivity. > [@mcosta](#): > > Umm, it looks that something is case sensitive… > > data/documents vs data/Documents Sorry, I noticed your message after posting the same. As you guessed, **everything** is case sensitive in almost every sane Linux system: even if you’re accustomed to work in case-insensitive filesystems locally, you always try to take that into account for portability. 😅 * * * _Note: I had to consolidate all my messages in a single place because new users can’t post more than three consecutive replies._ ---
**Author:** ![mcosta](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mcosta](https://discuss.dvc.org/u/mcosta)\ **Post date:** [June 17, 2021, 11:45pm UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/16 "2021-06-17T23:45:11Z")
@0x2b3bfa0 Case sensitive issues are solved. It is not issue free, though. Our dvc.yaml has command that looks like cmd: python src/filterout\_train.py -hp …/docs\_params.yaml Then the following error occurs. python: can’t open file ‘src/filterout\_train.py’: [Errno 2] No such file or directory Both Windows and Mac machines run smoothly. So the default working directory is probably defined differently here. **Details of the error** Running stage ‘train\_documents’: [6952](https://github.com/###/Filterout/runs/2853637495?check_suite_focus=true#step:4:6952)\> python src/filterout\_train.py -hp …/docs\_params.yaml [6953](https://github.com/###/Filterout/runs/2853637495?check_suite_focus=true#step:4:6953)python: can’t open file ‘src/filterout\_train.py’: [Errno 2] No such file or directory [6954](https://github.com/###/Filterout/runs/2853637495?check_suite_focus=true#step:4:6954)2021-06-17 22:51:28,195 ERROR: failed to reproduce ‘dvc.yaml’: failed to run: python src/filterout\_train.py -hp …/docs\_params.yaml, exited with 2 ---
**Author:** ![mcosta](https://avatars.discourse-cdn.com/v4/letter/m/6a8cbe/32.png) [@mcosta](https://discuss.dvc.org/u/mcosta)\ **Post date:** [June 18, 2021, 9:43am UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/17 "2021-06-18T09:43:09Z")
Stupid problem with capitalization. We changed the filename to lowercase assuming it would be sent with git push, but in reality no changes were detected. Because the change was made together with several others, we didn’t notice… It’s working! ---
**Author:** ![rmbzmb](https://avatars.discourse-cdn.com/v4/letter/r/ed8c4c/32.png) [@rmbzmb](https://discuss.dvc.org/u/rmbzmb)\ **Post date:** [July 18, 2022, 2:05pm UTC](https://discuss.dvc.org/t/cml-github-actions-google-drive-service-account/795/18 "2022-07-18T14:05:47Z")
It would be nice to have a reference yaml file somewhere.